In 1985, Neil Postman published a book ‘Amusing Ourselves to Death’ which provided an interesting juxtaposition between two authors who predicted dystopian futures, George Orwell (with Nineteen-Eighty Four) and Aldus Huxley (with Brave New World).

“We were keeping our eye on 1984. When the year came and the prophecy didn't, thoughtful Americans sang softly in praise of themselves. The roots of liberal democracy had held. Wherever else the terror had happened, we, at least, had not been visited by Orwellian nightmares.

But we had forgotten that alongside Orwell's dark vision, there was another - slightly older, slightly less well known, equally chilling: Aldous Huxley's Brave New World. Contrary to common belief even among the educated, Huxley and Orwell did not prophesy the same thing. Orwell warns that we will be overcome by an externally imposed oppression. But in Huxley's vision, no Big Brother is required to deprive people of their autonomy, maturity and history. As he saw it, people will come to love their oppression, to adore the technologies that undo their capacities to think.

What Orwell feared were those who would ban books. What Huxley feared was that there would be no reason to ban a book, for there would be no one who wanted to read one. Orwell feared those who would deprive us of information. Huxley feared those who would give us so much that we would be reduced to passivity and egoism. Orwell feared that the truth would be concealed from us. Huxley feared the truth would be drowned in a sea of irrelevance. Orwell feared we would become a captive culture. Huxley feared we would become a trivial culture, preoccupied with some equivalent of the feelies, the orgy porgy, and the centrifugal bumblepuppy. As Huxley remarked in Brave New World Revisited, the civil libertarians and rationalists who are ever on the alert to oppose tyranny "failed to take into account man's almost infinite appetite for distractions." In 1984, Orwell added, people are controlled by inflicting pain. In Brave New World, they are controlled by inflicting pleasure. In short, Orwell feared that what we fear will ruin us. Huxley feared that what we desire will ruin us.”

This juxtaposition provides the perfect metaphor for the debate about data in the world right now. Data is being influenced by ‘the three V’s’:

1. Volume: the sheer amount of data that is being generated digitally at exponential rates
2. Velocity: the unprecedented rate at which data is moving and being collected
3. Variety: the vast and diverse types of data that are being generated from different sources

On each succeeding year, we create more data than all of the preceding years of humanity combined. With the growth of global online access, and new networked channels like social media, this data is also becoming a technologists nightmare - unstructured, complex and variable.

You can’t read any sort of marketing prediction article without stumbling upon the apparent holy grail of “Big Data” - a concept that inevitably touches on the fears of Orwell. On the other side is “Small Data”, or the conventional data that every organization should be focused on. Invariably this suffers from the Huxley metaphor, and the threat of drowning in the sea of irrelevance or egoism.

In this essay, I’ll examine some of the defining factors of big and small data, and run through a simple framework to build a digital data ecosystem that can actually deliver actionable results for organizations.