How "Big Data" will try and put you into a Box

In 1967 I was involved in a design study which involved moving a magnetic tape based application onto direct access storage - and the file size was of the order of 100 megabytes - which was very "big data" fifty years ago. Things have changed and people are now talking of petabytes of data - and I decided to look into the current views on Big Data by logging into a FutureLean online course "Big Data: From Data to Decisions". The course is more about programming systems which use big data - which is not what I was looking for - but there has been some useful discussions on the impact of big data and how far it is can have the effect to feeding people's natural "confirmation bias" to selectively feed material which reinforces their views, and in effect make them more narrow-minded.

My own experiences is that when I decided I needed to loose weight Tesco (my local supermarket) started sending me discount offers on packets of "my favourite" biscuits which I had stopped buying because I decided that they were one of the causes of my being over-weight! Tesco may not have actually decided that I ought to be obese - but if I had made the suggested purchases ... ...

The online course includes a link to a helpful report "The Promise and Perils of Big Data" which includes the following statement.

"The Wall Street Journal once described a TiVO customer who gradually came to realize that his TiVO recommendation system thought he was gay because it kept recommending gay-themes films. When the customer began recording war movies and other “guy stuff” in an effort to change his “reputation,” the system began recommending documentaries about the Third Reich."

The point is that if, for some reason, big data suggests that you have a particular profile the result will be to bombard you with information to reinforce that profile. One of the other students Nathaniel came up with the following check list of questions:

What are the biases?
What are the probable effects?
What are the possible effects?
Who will benefit?
Who will pay a price?
Who will want to use this information?
Who might abuse it?