Excerpt cross-posted from DataInspirations.com
by Stacia Misner

I wasn’t going to do it. The whole recap of the year just retired. The goal-setting of the year just arrived. But I did find myself having to plan a few things – plan for customer visits and activities, plan travel for upcoming events, plan time to prepare content for those events, and so on. And as I thought about those plans, I couldn’t help but ponder where I’ve been and where I’m going, and thus this post was born. …

Accidental Business Intelligence? Not Really
I always think of my career in BI as accidental, although taking a longer view I realize that it really wasn’t. In February 2011, I told my story to Andy Leonard (blog | twitter) as part of his SQLPeople series of interviews. There are some karmic aspects to that story that I would never put in print, but I’d be happy to tell you in person someday if you ask nicely.

In that story, I mention working with Lotus Notes. In the late 1990s, I had this feeling that using technology as a knowledge management tool was an attainable goal and put a lot of energy into learning how that should happen. But then I got deflected into business intelligence, which turned into a fascinating and rewarding career. But all along, I had this nagging feeling that BI was just part of the story. I wasn’t satisfied with just delivering on reporting and analysis. As important as that is, I believed additional transformation in the way we work with data and with each other was necessary in order for BI to fulfill its promise.

Collaborative BI
I’ll admit that in the beginning, I wasn’t very impressed with SharePoint – I believe it was SharePoint 2003 when I was first introduced to it. I had been working with Lotus Notes long before that time and felt that it could run circles around that release of SharePoint. Full disclosure – I haven’t looked back at Lotus Notes since I left it, so I have no idea of its capabilities today. But starting with SharePoint 2007, I started thinking beyond the traditional dashboard compilation of scorecards and reports. I was thinking about unstructured data to support the structured and would mention it in my presentations and classes. Then with SharePoint 2010, I started thinking about the collaborative and social aspects and started putting these pieces together with ideas that I had been nurturing since the late 1990s. And so, a presentation was born for a webinar, a few SQLSaturdays, and continues to evolve as I gear up for the PASS Business Analytics Conference in April 2013. Because an hour presentation only sets the stage for some of my ideas, I have set up a Collaborative BI resource page that will grow as I commit these ideas to writing.

Big Data
Meanwhile, the buzz around Big Data became louder in 2012. Now I’ve been around a few years, and I’ve seen buzz come and go. I had plenty to keep myself busy meanwhile during 2012 and just watched and waited to see what would happen. And then things started to get interesting. So much so that it’s time to start talking about it. Consequently, I am working on presentations on this topic throughout the year (keep an eye on Upcoming Events for online and in-person events), including a session at the PASS Business Analytics Conference on Power View and Hadoop in collaboration with Joey D’Antoni (blog | twitter). And that’s just the beginning. I plan to add another resource page for my thoughts on BI and Big Data. Watch for more blog posts and presentations.

When I think back to my “accidental” discovery of BI and the ideas we were throwing around at the time, I realize we were a bit ahead of our time. What we needed at that company to achieve those big ideas was Big Data – we just didn’t call it that then. We started on a much smaller scale and focused on data warehousing and reporting and analysis tools, and we were barely ready for that then. I work with customers today who are still barely ready for that.

Data Science
But now in the era of Big Data and data science, I start thinking about those big ideas again and how much more attainable they are today, 14 years after I started down this path. The BI world is poised for the biggest change I’ve seen in my career. While I cannot share the specifics of what we were thinking about in my R&D days, I can try to explain how I see the difference between BI as we traditionally think of it and where data science can take us.

BI helps us understand what happened or what is happening now, using established processes and tools. Although BI can scale quite dramatically, scale introduces some complexities that in some ways limits the types of reporting and analysis that we can do. Data mining is often included in a discussion of BI technologies, but its use has not been very prevalent in my client base. Data mining can not only be used to explore data to help us understand what happened, but can also be used to predict what might happen. And this is where we see data science come into play now. Data science can help us look forward and to predict an outcome or a correlation. It incorporates many techniques that are common to data mining, but it can go beyond those techniques as well. We can work with larger data sets than ever before because we can store data more cheaply than ever before and we have better tools for dealing with these larger data sets using commodity hardware.

Is the Data Warehouse Dead?
No, I don’t think so. At least not completely. There’s still a place for operational and mission-critical information that’s been consolidated, cleansed, and corporately-sanctioned as truth. I don’t really care what we call that information source – a data warehouse, a data mart, whatever. We need access to that type of information because that’s how we decide what to do today to achieve our goals, respond to specific problems, or show the board (or the world) how we’re doing as a business. The new potential with Big Data and data science is the opportunity to explore data in ways never before possible. We don’t know what the opportunity or business value in that data might be until we examine it in new ways or combine it with other types of data – data that was captured by others and shared publicly is just one example. Creativity is key. But like traditional BI, in my mind, it’s all useless unless we can DO something with that information. And of course, we need to share and collaborate!

What do you think? Is BI as we know it going to die or thrive in this brave new world of big data? I look forward to delving more into these topics more in future posts. I hope you do, too!