The importance of dark data in the enterprise with Rich Miller

Introduction

This week I’m joined by Rich Miller, the CEO of Telematica.

We start off by asking if enterprises have the data they need. Rich digs into the importance of understanding dark data and why it is critical. He talks about how more than half of enterprise data is dark data today. At the same time 80% of enterprises rate data as critical to their success. Rich explains how to tap into that dark data and the opportunities that come from it.

---

Speaker Profiles

Rich Miller Twitter: https://twitter.com/rhm2k

Rich Miller LinkedIn: https://www.linkedin.com/in/richmiller/

Telematica: http://www.telematica.com

---

Audio File

---

Featured Hashtags

#CIO #CDO #Leadership #Data #DarkData #DigitalTransformation #CxOitk

---

Podcast Transcript

Tim Crawford 0:01
Companies are looking for new ways to transform their business. Technology plays a critical role in this transformation. Speed and innovation in both technology and thinking are key to this shift. Hello, and welcome to the Cxo in the Know podcast, where I take a provocative but pragmatic look at the intersection of business and technology through the lens of leading CxO executives. I’m your host Tim Crawford, a CIO and strategic advisor at Avoa. This week, I’m joined by Rich Miller, the CEO of Telematica. We start off by asking if enterprises have the data they need. Rich digs into the importance of understanding dark data and why it is critical. He talks about how more than half of enterprise data is dark today. At the same time, 80% of enterprises rate data as critical to their success. Rich explains how to tap into that dark data and the opportunities that come from it. Rich, welcome to the program.

Rich Miller 1:01
Hi, Tim. Nice to be here. Thanks for asking me.

Tim Crawford 1:04
Absolutely. So, Rich, you’re the CEO of Telematica and also the chairman and co-founder of Provident Data. So, to get us started, why don’t you tell us a little bit about who Rich Miller is, who Telematica is, and who Provident Data

Rich Miller 1:20
is? Sure. Well, first of all, I’m a longtime denizen of Silicon Valley, and my technology chops go back to the days of the ARPANET. But what I have been spending the better part of the last 20 years doing is early stage companies, getting them up and running. The focus of those companies has been compute distribution that is data grids, compute grids moving to cloud computing, the management of virtualized systems and distributed systems, and when I say involvement, Telematica is a consultancy and a holding company, and might be thought of as a venture consultancy. We work with early stage companies. Sometimes we invest in them, and mostly what we do is sweat equity, working with top tier technology companies, mostly going into infrastructure or specialized data-oriented offerings to particular segments, one of our earlier companies led, in fact, to an idea that one of my colleagues and I have run with, and we’ve created a new company called Provenant Data, and it’s still pretty much in stealth. We’re not spending a lot of time discussing it, but it’s mostly about managing and versioning data sets in a way that helps you think about the lineage and the provenance of a data set. And we’re hoping that it focuses very heavily on the use of data in situations where data sets have to be managed in accordance with various kinds of regulatory and compliance rules.

Tim Crawford 3:13
That’s great, and you know we’re going to talk a lot about data on the podcast because data is a critical business component, and today, you know, data is critical to business decisions. My first question to kind of get our conversation off the ground is: from an enterprise perspective, do we really have the data we need to make good decisions?

Rich Miller 3:36
What a good one! And the answer is yes and no. We have data often that we need to make good decisions, and don’t know that we have it. But more often than not, in order to have data to make good decisions, you have to first ask the right questions, and then, in a purposeful way, go out and collect or find the data, and then integrate it into your decision processes. So the answer is, I’m sorry, yes and no. There’s a lot of data that’s there, but very often it’s not the right data to answer your questions.

Tim Crawford 4:17
So what do we do about that? And are there components that will help us in terms of driving down into data? And there’s this concept of dark data that I want to drive into as well. So for the folks that are listening to the podcast, they may not be familiar with this term, and there might even be executives listening to the podcast that aren’t familiar with this term. What exactly is dark data, and maybe you could give some examples of what it is.

Rich Miller 4:45
Sure, dark data is a term we usually use to cover all of that unknown and untapped data that already exists inside an organization. Very often, it’s. Generated by systems, devices, maybe interactions and transactions. It’s the data that is collected in an automated fashion in logs, often semi-structured or very structured, but just goes into a bucket somewhere. There’s another aspect of dark data, and that is data that is being developed and used by individuals as part of their job, but never is actually collected and organized and curated. Think of all of the spreadsheets that you have on your computer, all of the versions of the same spreadsheet that you try to go back to later on and figure out which is the authoritative one. There’s a lot of dark data in there as well, so it’s easier than ever to collect the data right now without a specific purpose or with a purpose, but it just falls into a bucket somewhere, and you think I’m done with it. It’ll be there when I need it later.

Tim Crawford 6:08
So, is dark data maybe categorized into two spaces: data that you might be otherwise capturing, but not necessarily using or know exists, and maybe data that I, as an individual, you know, Tim Crawford is using, but Rich Miller doesn’t even know that Tim Crawford has created this data or metadata, and Rich Miller therefore doesn’t have access to that data. Is that a way to kind of think about

Rich Miller 6:36
it? Yes, that’s a very good observation, Tim. If it is dark to me, it might not be dark to you. You know where it is. You use it every day. We might be in the same organization, but unless there is a way in which you can make it known to me that it exists, control it, manage it in terms of what you think my access should be to it. If I don’t know it exists, if I don’t know where to go and look for it, it is dark data, as far as I’m concerned.

Tim Crawford 7:08
Wow! So putting that context in place, that means that there is a lot of dark data that then could be missed opportunities to play into those business decisions.

Rich Miller 7:21
Oh yeah, and as a matter of fact, fairly recent surveys of enterprises globally around dark data. On average, most businesses, the business and the IT decision makers agree that probably more than half the data that exists in their files today is dark. It’s unknown or it’s untapped, and probably 80% rate data as extremely valuable or very valuable to their organization’s overall success, and yet better than half of it is in some unknown place. So,

Tim Crawford 8:01
wow,

Rich Miller 8:02
you’re absolutely right.

Tim Crawford 8:03
So, what are some of the missed opportunities? If you kind of drill into that a bit and talk about, okay, you’ve got this dark data. You may not even know it exists, but what are some examples of missed opportunities that enterprises just aren’t thinking about, and maybe will help them understand where they could go looking for dark data.

Rich Miller 8:24
Well, let’s start with the reason you might want to pick up or find data that already exists in your organization. And at the end of the day, what you’re looking to do is figure out in a methodical, efficient way how to use that data and monetize that data, and when I say monetize, I mean how can you utilize that data in a fashion that either reduces your costs, improves some other metric of performance like latency or time to a decision, or the speed with which transactions are done, or you can turn it around and say this data is capable of being monetized because it has value to someone outside of my enterprise, outside of my organization, and it can be cleaned up, packaged, and offered either in exchange for something else or literally subscribed to, sold, published. There is a business, and there’s a burgeoning business around the packaging up of this kind of data and selling it on subscription basis to other organizations, the financial community in particular is going completely crazy around what’s known as alternate data or alternative data as a means of taking data. That wasn’t immediately designed to help them make a business decision or a financial decision, but it adds enough extra information to their decision that they can do better than the next guy. In the medical and life sciences, they don’t call it alternative data; they call it real-world data or real-world evidence. There’s an enormous amount of data in the medical insurance community that actually can be used to determine the efficacy of pharmaceuticals that are medical processes that have already been put in the market and need to be compared with their competitors or improved. So there is data living in many enterprises that is capable of being monetized. I would even go so far as to say there are places where some of those spreadsheets we were just talking about could be packaged up on a regular basis and monetized for the outside world. So there’s the internal use of the data and the monetization there, and there’s this kind of external monetization, turning data into an actual product or revenue generation-that’s

Tim Crawford 11:24
a fascinating concept to me. Because in conversations that I have, and if I think back to my experience leading IT organizations, you know, the focus is internally: how do we use data internally for our own purposes, our own products, our own services, but what you’re suggesting is yes, do that. But there’s also an opportunity to potentially use data for external purposes as well, and I think that’s just a fascinating opportunity that maybe folks just aren’t thinking as much about, but yet could be a pretty significant financial opportunity. It

Rich Miller 12:02
is increasingly a really incredible business, Tim. It has gotten to the point now where probably five years ago, the whole industry of selling data, kind of the way I’ve just described it, was maybe three to $5 billion a year business, and was pretty much limited to big data-oriented companies like the Thomson Reuters and Bloomberg’s and so forth.

Tim Crawford 12:36
Sure,

Rich Miller 12:36
increasingly that has become a place where other companies, where their primary business is not data, but they have found that they are generating data that is of value, can be put into data markets, sold through data brokers of various kinds, and there are projections of that market being well over $25 billion annually. Wow! Within the next three years.

Tim Crawford 13:10
Wow! Wow! That’s not something to shake a stick at. So when you talk about that data that is being packaged up and potentially sold, you’re not talking about competitive data, right? Or are you?

Rich Miller 13:22
I think there are parts of it that could be competitive data, but I really am not necessarily talking about competitive data or even marketing data to kind of delve into the fine details of a consumer’s purchasing behavior. I literally am talking about the kind of data that can be used in combination with other data to determine, as I was talking about before, for a pharmaceutical company that has put a pharmaceutical into the market to look at, are people using it? Are doctors or medical companies prescribing it? Are the outcomes of using that particular pharmaceutical showing up as having good results in some cases, less good results in others where the determining factor might be geography and weather could be some other aspect that isn’t captured unless you actually had access to a doctor’s medical report safely or an insurance claim. So again, what we’re talking about is how do you improve your primary product by getting data from a variety of sources and using that to focus on improvement in either your processing, your packaging, your target. Targeting all of those things.

Tim Crawford 15:02
Okay, and one of the ways that companies were really trying to get their arms around the data that they had and start to rationalize it was around data lakes and repositories. And there was a lot of money that was spent on data lakes, repositories, and some might even argue data swamps. Were they not intended to bring all types of data together? And what’s the state of those? What’s your take on where those fit in?

Rich Miller 15:30
Well, I think data lakes are one of those concepts that have proven out the Gartner hype cycle. The whole idea that you could take data at great speed, at great velocity-I should say, great volume-and just throw it into a data lake in such a fashion that later on, when you needed it, or when an application needed it, it would go back and find it and make use of it. Quite frankly, that has never worked at scale. I don’t think it ever will. And what it basically says is, as early as possible, when you’re onboarding data, either from your own systems, your in-house manufacturing line, or your transaction flows, or when you’re buying data from somebody else, piping it in. When you’re using in social media the Twitter firehose,

Tim Crawford 16:33
as

Rich Miller 16:33
early as you can, you want to be able to do some recognition of what that data is, and you want to do some pre-processing. You want to do some work on it before you store it anywhere. And what this has done is given rise to tooling like data catalogs, which are very often used at the receiving end of some sort of a stream of data to do at least first order. Think of it as triage on incoming data, so that you don’t lose track of it, and such that it can be utilized effectively later on by an app. So it doesn’t mean that you go back to the days of a data cube, a data mart, where everything is extremely organized and overly structured, mostly in response to a specific question that you’re asking, but it does say that there’s still a lot of work to be done in how you manage onboarding data, and when you apply various kinds of processing or automated processing, hopefully to it, I’d even go so far as to say that when you’re using external data, externally generated data in particular, one of the things that the data engineering responsibility. responsibility should start with: Is what do I know about that data? Do I have the equivalent of a data sheet on it? I would venture to guess that any enterprise that’s in the business of building physical machinery before they purchase a box car full of nuts and bolts. Has a datasheet on all the different sizes, the composition of the metal that’s being used in those nuts and bolts, because it’s important for them to know they’re using the right supplies. They’ve got the right ingredients for whatever it is they’re manufacturing, and so it’s not unusual for an electronics manufacturer, for example, to send you in advance of a purchase. Here’s the data sheet. Here’s what you’re getting. Here’s the composition of it. Here are the inputs. Here are the outputs. Here is the performance tests on it, and here’s my estimate on the number of dead on arrival piece parts. Those are the same kind of things that you would want to have when you’re buying data from some other source.

Tim Crawford 19:15
Interesting. So

Rich Miller 19:16
I guess what I’m coming back to is the whole point is for it to remain visible, illuminated, you need to know about the data as you’re onboarding it. You need to do some pre-processing, or you can spend some time and some money doing pre-processing so that you don’t lose track of it, so it doesn’t end up as dark

Tim Crawford 19:42
data. And as you go down that path, I mean, are there any other gotchas or even liabilities you have to think about as you start to culminate this data, whether it’s internally generated or externally generated?

Rich Miller 19:56
Absolutely, you have to think about it. This is. You know, they you have all of these opportunities we’ve just been talking about, but you also have the liabilities of bringing on data that is dark or remains dark. And in those cases, we’re talking about kind of lack of visibility into all the data that’s being collected internally or that you’re buying and then goes and collects dust on a shelf somewhere. If you do not make it easy to accurately manage the costs of bringing on data and organizing it, this can run away with you. And there are a lot of people who have overdone the data cataloging, tagging areas where they’re spending much more money than they should on the data, and the result is somebody ends up pulling the plug. But the other kind of liability that you have to think about with dark data in particular is if it remains dark, it can hamper your efforts to ensure compliance with regulations and, in particular, data privacy regulations. For example, like the EU’s General Data Protection Regulation GDPR, or what we’re seeing now in California, the California Consumer Privacy Act (CCPA) If in your dark data there is personalized information that should not be retained, or under new regulation is not to be kept, and suddenly you go through an audit, and there it appears there is a hidden liability that no CFO is going to enjoy, no executive is going to enjoy the result of.

Tim Crawford 21:57
Yeah, I would call that a gotcha.

Rich Miller 22:02
It’s a real one, and that goes for you know not just personal privacy, the financial industry, the insurance industry, the health ind, you know, health and medical sciences, life sciences. All of them have regulations that really have to be adhered to, and more often than not, you want to be sure you don’t find that in the dark silos. Once somebody comes in and starts opening the doors,

Tim Crawford 22:33
yeah. As we kind of get toward the end of this episode and we go full circle, I’m kind of wondering: our enterprise is truly prepared to head down this path? I mean, there are some great opportunities. It’s important to think about dark data. It’s important to understand the gotchas and liabilities as you’ve just outlined. But are they truly prepared to go down this path? And if not, what are some things that you think they should be thinking about or things they should be doing.

Rich Miller 23:02
Well, I think it ranges from those companies that are pretty prepared to those that really are unprepared. Know they should be, but don’t know where to start. When I’ve been called in to discuss this kind of situation in particular cases. I’ve kind of come to a small list of best practices. The first order of business is wherever, whenever you acquire data, shine a light on it, illuminate it, know what it is, know something about it, collect the metadata in a fashion that you can find later. Second, is that organizations need to evaluate whether and for how long it makes sense to retain the data. You don’t want to become a data hoarder and simply collect data for the sake of collecting data. This is crucial to avoid incurring rather substantial costs of collecting and storing data that isn’t being used. So you need to have a policy, but also you have to put in to get into place those tools that would allow you to say, you know, this has never been used. It’s never been accessed, and it has to go through a particular process. But let’s not keep that data anymore. So, it’s not keep everything. It’s keep collecting or curating this data, but in a smart way. The third aspect is access. Right now. now because of distributed computing, because of cloud computing, because of edge computing. The biggest challenge working with or dealing with dark data is simply getting access to it. Often, it’s stored in siloed. Repositories. Very often, it’s close to where the data was actually collected, and additionally, it might be stored in systems or in formats that are really hard to query or have limited analytics capabilities at the place where it’s stored. So, this final step or set of steps are what I usually end up closing the conversation with, and that is organize the data as early as possible, as close to the point in time that it’s onboarded. Sometimes these are, and most times now, it’s automated tools, automated tagging, and you do so in a way so that you can put it in places where you can find it later. To that next point, invest in tooling that supports data discovery. You already have a lot of dark data, and you will continue to be looking for data as time goes by. There are some excellent tools for data discovery, or data discovery as a function of a broader data management suite, focus on that. Third issue is invest in a slightly different kind of tooling, and that’s the tooling that can query the data where it is currently stored. If you have to lift up a whole dataset or a database and move it somewhere in order to then investigate it, unpack it, figure out what’s in it. You’re not going to do it. It’s going to be costly. Invest in the tools that allow you to query remotely. So that means your tooling is set up for that, and your data stores are set up for that. And then finally, you want to think about centralization of data storage, but you want to think of it as logical centralization, not necessarily physical centralization. What do I mean by that? What I mean is, wherever the data happens to be stored in any distributed enterprise along the edge, there needs to be a single pane of glass or a single point of view that allows the person searching for or the process searching for that data to have a really full view of it. That’s logical centralization. It doesn’t necessarily mean anything about where you physically locate the data.

Tim Crawford 27:41
It’s not your new data warehouse. No, it is

Rich Miller 27:43
not, Tim.

Tim Crawford 27:45
Yeah, that’s a great place to end. We’re gonna have to end it right there. But Rich, thanks so much for sharing your expertise and data. I’ve known you for a number of years. You’ve always been a great advisor to me, and I look forward to continuing the conversation with you.

Rich Miller 28:03
It’s been a pleasure, and I’m looking forward to having more of the conversations.

Tim Crawford 28:09
For more information on the Cxo in the Know podcast, visit us online at cxo in the know.com You can also find us on Apple Podcasts or wherever you listen to your podcasts, please subscribe and thank you for listening.