A data analysis system is a human–machine collaboration：

  • The machine is responsible for collecting data
  • The machine is responsible for preprocessing data
  • The machine presents the results in a way that humans can easily recognize
  • Based on the machine’s work, humans draw conclusions from the data

So if you think of data analysis as a “pipeline,” there are at least two stages：

  • Machine stage：the final output is charts that humans can process1
  • Human stage：the final output is conclusions about the data

In short, for programmers, the main task of data analysis is to make data visualization happen.2

  1. “Humans can process” does not mean “humans can recognize.” After all, hundreds of TB of data are mostly text, and humans can recognize that. The problem is that the data volume already exceeds what humans can process. So even though it may be recognizable, it becomes unprocessable. To make data processable for humans, you must summarize it: the data format should be recognizable to humans, and the amount should not be too large. Given humans’ huge advantage in image recognition, representing data in image form enables the maximum expression of information density. ↩

  2. That’s probably one reason R seems to matter so much. ↩