We are swamped with data – even our movements are tracked by mobile networks, and stored. Our web tracks are everywhere and our purchasing habits both online and offline are fully traceable. Businesses and security agencies are swamped with data, and yet there is widespread dissatisfaction about query tools and data analytics. Ask the right questions and you will get the right answers. That’s all well and good with SQL, but it has failed us. Knowledgebase tools have failed us too.

/**/

Document libraries have failed us. It is sensemaking that is needed, a way of pulling together disparate data sources to provide meaningful output.

On a battlefield it might called situational awareness, in a futures market it might be called ‘market nous’. Either way, the tools of the late twentieth century are not up to the task of linking structured and unstructured data to provide meaningful output. Big data analytics is the term used to characterize the problem, but linkages across databases have to be established and meaning ascribed to them – whatever ‘meaning’ is.

Business intelligence/analytics tools need to cut through the data and understand the core meanings and implications. Google, with its latent semantic indexing is trying to make sense of textual content, but even that is not enough. That type of analysis can validate a sentence as ‘sensible’, identify keywords and try to provide some kind of content evaluation relative to other sentences, but that is its limit. The usual information toolkit cannot provide insight into data, and that is why there is dissatisfaction.

Dissatisfaction leads to revolution or, in business or technical terms – innovation.

The problem has been recognized and companies such as Saffron Technology are innovating with associative memory constructs which combine structured and unstructured data to provide realtime decision support. The innovators in this field are linking theories of human memory and computer neural network theory in ways which provide real insight into the ‘big data’ that corporations and governments are amassing.