Big Data: Focus and Practicality Now Vital

It's been years in the making and has had its fair share of media hype, but according to Gartner's August 'Hype Cycle Special Report for 2014' the concept of Big Data has now entered its aptly named 'Trough of Disillusionment'.

And it's not Gartner alone. Talk to industry stalwarts and a clear message comes back - the honeymoon is over. No longer is it a positive buzzword in meeting rooms. It's becoming tangible...real people with real salaries and real job titles are now associated with the discipline of managing and making the most of a company's big (or small) data, both locally and in the cloud.

We've come to realize there are a number of opportunities for big data and it’s management, as outlined in IBM's August report titled 'The New Hero of Big Data and Analytics'. In it, a new C-suite role is outlined, along with five areas are a Chief Data Officers (CDOs) can optimize and innovate in:

1. Leverage: finding ways to use existing data.
2. Enrichment: existing data is joined up with previously inaccessible (fragmented) data either internal or external.
3. Monetization: using data to find new revenue streams.
4. Protection: ensuring data privacy and security, usually in collaboration with the Chief Information Security Officer.
5. Upkeep: managing the health of the data under governance.

It's a great list of general outcomes for those who manage data to plan around over the coming years, but what might be even more useful is a planning framework to help develop these plans now.

Obviously this framework will evolve, and to some extent there will be a degree of trial and error as organizations try to wrangle increasingly large data-sets. But I thought it'd be useful to make some suggestions for considerations against these outcomes. So I've come up with some key questions to gather information to help in the CDOs strategic planning. Answering yes to most, if not all of these questions is a good indication a CDO in your organization would have a beneficial business impact.

1. People: as mentioned in IBM's report - is the CDO's office a guiding, enforcing authority? Is the office fully aligned to the business and scalable? Are the skills available appropriate? Is the business giving the CDO authority or permission to operate?
2. Compliance: not just with regional and industry regulation but with the company culture.
3. Intelligence: how can the right information reach the right people in a digestible form that catches their attention? Does the information remain useful throughout its lifecycle?
4. CIA: Confidentiality, Integrity and Availability. The triangular cornerstones of any information security policy, no less important. Can your CDO guarantee data CIA, and have board level authority therein?
5. Technology: which technology providers can help support these outcomes today, and well into the future? Does the chosen technology scale in line with the parabolic growth of data, or is it linear or worse, unpredictable?

It's by no means a definitive list, but we hope it helps stimulate the conversation around this emerging discipline of curating data to a commercial end. I look forward to sharing ideas with our customers and partners on this over the next few months. And as always, I’d appreciated any comments under this post.