How Big Data Got Here, and What's Still Missing

It’s really cool to see Sumo Logic come out with a $15 million series B funding from Sutter Hill Ventures, Greylock, and Shlomo Kramer. Sumo Logic is taking on 2 other companies I know and respect: Splunk and Loggly.

The founders of Sumo Data are Kumar Saurabh and Christian Beedgen, a couple really impressive entrepreneurs who cut their teeth at Arcsight. I had the pleasure of meeting with them a month after they were founded back in April of 2010, when they were still setting up in the TechMart Regis Facility on Great America Parkway in Santa Clara. It was kind of cool because they were moving into the same office that I had used when I was VP of marketing at Zeus Technologies, now an arm (tentacle?) of Riverbed. It’s a small valley.

At the time, I was working as cloud and virtualization Entrepreneur In Residence (EIR) for Trinity Ventures, but Sumo Logic wasn’t quite yet far along enough for VC then. It was pretty exciting when I came across them anyway because I am a long-term believer in this space.

So long-term, in fact, that I invested as an angel in the very 1st blog file management company which was named Addamark. They were ahead of their time and morphed into SEIM vendor SenSage when security became the main use for log management in the mid-2000’s. Then, I backed SenSage’s founder again with another angel investment in the world’s first cloud-resident log management/big data play, hosted on AWS way before it was cool for SaaS companies to do that. Despite some big wins – like Playdom – it was ahead of its time again. Either that, or I’m a sucky investor because I keep thinking things will happen before they do.

Why Big Data Is a Bigger Deal than Cloud Was

My thesis for investing in big data has nothing to do with data from e-commerce or IT management systems. I believe that the volume of data we are generating now from machines absolutely pales in comparison to the volume of data we will soon be generating from our own bodies via new consumer grade medtech offerings.

The Internet of things is limited in scope because we have to make and power things, and we get to define what data they can present. The human body is a blank slate – there is limitless data to gather about electrical, chemical, and physiological states, as well as about behavior and location. That’s not even including 24/7 audio or video.

That’s why I’m one of the leaders in the emerging Quantified Self movement. (Here’s a talk I gave at the first ever QS conference, and that was me on the cover of the Financial Times wearing electrodes on my head.) That’s also why I’m still a huge believer in the log monitoring and analytics part of the cloud computing space. It will be exciting to moderate a panel in New York at the upcoming GigaOm Structure:Data conference, because it gives me a chance to combine my career in cloud security with my weekend passion, biohacking.

What’s Missing in Log Management

There are 2 things that don’t receive enough attention in the log management space. The 1st is real scalability, which means thinking beyond what data centers can do. That inevitably leads to ambient cloud models for log management. Splunk has done an amazing job of pioneering an ambient cloud model with the way they created an eventual consistency model which allows you to make a query to get a “good enough” answer quickly, or a perfect answer in more time. They can do this because the data is spread all over the place but it is controlled centrally, which is a hallmark of ambient cloud architecture. Plus, ambient cloud providers are valued higher than IaaS cloud vendors. That sucks for us infrastructure guys.

The 2nd thing is security. Log data is next to useless if it is not nonrepudiatable. (is that even a word?) Basically, all the log data in the world is not useful as evidence unless you can prove that nobody changed it. That’s why I’m a believer in what Mark Searle, the original Addamark founder, is doing at Kinamik. His experience founding 2 early log management companies has led him to focus on the emerging problem of security for log management. It’s very meta. His 1st start up a decade ago ended up focusing the other way around – on using log management for security.

In any case, I want to see my brain waves, my temperature, my pulse, my heart rate variability, my galvanic skin resistance, the number of steps I take, what I eat, what I breathe, who I talked to, my hormone levels, how happy I was, my brain’s efficiency at any time, and anything else I can think of stored in a very large, very secure, very friendly cloud analytics application. And then I want to share that data anonymously with any researcher who is doing something cool.

So how about it, Sumo Data, Loggly, Splunk, and Kinamik? Are you ready for the onslaught of data? My friends at Quantified Self are building it now. That’s how they roll.

Kudos to GigaOm’s post that inspired me to think of this.