I enjoyed my first Trip Advisor navigated holiday a few weeks ago. The idea was that we’d go where the wind blew us, booking accommodation on the fly based on Trip Advisor reviews. After the third night we gave up and reverted to ‘turn-up-at-the-door-and-sniff’ instincts. When you find yourself confronted with a couple of hundred reviews of a hotel, many of which seemed to conflict with one another, it becomes very easy to conclude that we were dumber when we’d finished than we were before we’d started looking.

Much as Trip Advisor and the growing number of ‘Big-Data’ providers might not like to hear it, beyond a certain point, the more data there is, the lower its value inherently becomes. It’s a pure fact of nature. Data fundamentally wants and needs to be free.

The converse of this story emerges when we look at wisdom. The more data we have available to us – as I experienced on my holiday – the greater the potential for confusion, and consequently the more we are willing to pay for somebody or something to understand our context and our real needs and have the wisdom to be able to point us to a solution that meets those needs. The core paradox is that the more data we produce – the more hay we add to the haystack – the more difficult it gets to find the needle. The more available and freer data becomes, the greater the value of wisdom.

In the needle-in-a-haystack metaphor, wisdom is all about being able to find the needles. One of the biggest misconceptions of the Big Data industry is that adding more hay to the haystack also increases the quantity of needles.

Not only does this sound nonsensical, it is nonsensical. The reason this is so is because wisdom is all about the ability to contextually distinguish the important from the trivial. And what’s important in life comes from our extremely finite set of needs and desires.

The TRIZ/Systematic-Innovation methodology, unbenknownst to the original researchers as it happens, was the ultimate study of the difference between the important and the trivial. Sixty plus years and 4 million case study analyses later, we can vividly see that the world spends most of its time and energy re-inventing wheels. Which is to all intents and purposes the same as making more hay. The total number of problems in life is measurable in the hundreds. The number of effective solutions to those problems is measurable in the tens. There are, in other words, tens of needles in the haystack. And, moreover, we’ve already found them.

Now the NHS, and much of the rest of the public sector has been tasked by their government masters to ‘measure the experience’ of their patients, staff and customers we’re very likely to see an exponential growth in Big Data projects in the coming months and years.

After that, if I had to make a prediction, someone is going to start asking difficult questions like, ‘now we’ve measured all this data, what are we going to do with it?’ The only point of measuring anything, in the real world at least, is to allow us to improve how things work. We measure patient experience in order to improve patient experience.

And it’s only at that point – the point where we switch from merely measuring to actively intervening to make changes to the system – that we really learn the truth of the Big-Data-Small-Wisdom paradox.

The NHS, being ‘ahead of the game’ is already drowning in Big Data. Terabytes of meaningless hay.

Actually, I suspect the situation is considerably worse than that. The meaningless hay is also toxic hay. Toxic because it hides the truth. Three months before the Mid Staffs debacle hit the public, a Big Data ‘Review’ of the Trust had been given a full set of green lights. Everything, according to the Data, was fine.

Except, of course, it very clearly wasn’t. The needles that would have revealed the real set of problems – and their solutions – had been hidden. The real tragedy when we consider this truth, is that we didn’t need a review at all in order for us to know what and where the needles were, and what could and should have been done to make a meaningfully positive impact on the experience of the patient.