Unusable Data : A Real Problem | BHIVE Workspace

Unusable Data. Scribble Data launched its analytics automation service recently, and I have been on the road talking to various small and big customers about their data challenges.

I had a situation recently where a CEO asked me to help him with analysis of some operational data. I was very puzzled because I know for a fact that his 500 Cr+ organization has a 40-50 people IT team. On top of that they have BI system for a well known vendor, and are receiving services from another analytics company. It struck me as odd that they would bother to talk to a startup like ours that has existed for less than four months.

What it points to more than anything else is that decision makers are faced with more and more difficult decisions every day as competition intensifies, and they would like to draw upon data that they have within their organization, but are unable to for some reason.

I wish to explain why this is happening and suggest that usable data should be a first class requirement for systems.

Complexity : Reason Why Data is Not Usable

The problem is not access to the raw data bits but complexity of the data and the systems in which they live. High-level information such as a sale of a coffee is broken up. It is then distributed across various tables and systems in order to make the end-to-end application work. The understanding of how the bits are laid out and what each bit means is hidden deep inside databases and application code, and takes a specialist to understand and extract.

If decision makers know that they have to consume the data, why aren’t technical systems aligned to the goals?

Systems have been historically designed with the implicit assumption that nobody outside the small team of architects, DBAs, and application development experts will look at them. The idea that a business team might want to look deep inside the application to get insights about their business is a very recent phenomenon. Further most architects have not fully internalized the emerging need to account for frictionless access to data. Decision-makers ask for data is often narrowly interpreted as a request to add some bits to dashboard.

And we are building ever more complex systems. It takes significant skill and experience to  reason about technical systems at the business level.

Usable Data : A First Class Requirement

Complexity is not an accident. I spoke to a large governance platform company recently whose staff mentioned that dont know how to interconnect modules within their platform that they themselves developed. Therefore they cannot correlate data from the two modules. If they can’t do it, how do they expect technology experts, forget business staff, at their customer end to do it? The technology staff that designed and developed the system is world-class. It is not a competence issue. The system was built the way they did is because decision makers prioritized features over depth. Data, analytics, and coherence were not even considered. Now users of the system are stuck with a complex unwieldy platform that provides little or no insights about organizational governance processes or people.

Frictionless access to data should be first-class consideration while building application systems. The critical infrastructure that will make data accessible and usable such as interfaces, descriptions, and access methods should be defined. Even if not implemented due to pressures of the situation. They can’t be easily retrofitted into complex systems.

Alignment : Way Forward

I have started systematically studying what good organizations do in the process of going data-driven. They align three critical pieces of their organization:

(a) People – Ability to frame thoughtful questions against their data

(b) Technology – Ability to reliably service questions on any part of the system in near real time

(c) Process – Ability to take answers and apply them

In future articles, I will expand on each. And, suggest some emerging metrics to measure the progress of a data-driven organization.

About the author:

Dr. Venkata Pingali is the Founder of a Bengaluru-based Data Science Automation startup, Scribble Data. Their cloud-based product AskScribble (‘Google for SQL’) is a customized search engine. It allows anyone to query a database using natural language. It requires no training, no access to data, and 5 mins to setup.

Acknowledgement: Illustration from Wikimedia Commons about Unusable Data

https://commons.wikimedia.org/wiki/File:Data_Mining.svg