Big Data is great for finding correlations. The Bigger the Data, the better the correlation.

But correlation, no matter how good, has nothing at all to do with causation.

Which is a pity, because causation is the only stuff that any of us ought to act upon. When we confuse correlation and causation, the only certain result is we make matters worse.

When we talk about looking for needles in haystacks, correlation is hay. Needle is causation. Causal relationships between things. Contradictions. Insight. Insight is contradiction is causal relationship.

When we talk about enterprises being able to act with confidence, what we mean is we have found an actual needle, that we have uncovered a meaningful causal relationship that no-one spotted before.

As the world gets ever more complex, busy and interdependent, hay becomes easier and easier to find. Needles, conversely, do not. Moreover, if we believe all of the findings of TRIZ since 1946, that insight is causation is contradiction is needles, the number of needles is finite. And if we already found most of them, there’s not a lot of point in gathering more hay.

Big Data is hay. The best way to find needles is to find them before they get dropped in the hay. Better yet, is go visit the needle factory. You’ll recognize it when you see it, it has a big ‘TRIZ’ sign outside.