By now, Agile development methodologies have more than proven themselves in the software engineering industry. Agile & Scrum has provided development teams with a process that is lightweight, flexible and effective in keeping software projects on-time, on-budget and closer to user expectations.

So why can’t we apply that same approach to data warehouse and BI projects? Well, we can. In this post, I’ll kick-off my conversations with you about how you can apply Agile Scrum for your DW & BI development teams. Many lessons that I’ve learned @ both Oracle and Microsoft will be applied here as well as the practices that we apply @ Razorfish, many of which originate from Ken W. Collier’s exellect book Agile Analytics, which I highly recommend.

One of the effective approaches that Ken takes in his book is the same teaching mechanism that I use, which is to teach the basics of Agile Scrum development and adapt to DW & BI practices. I’ll dive into this throughout my posts to make that more clear. But for now, simply think about the Agile Manifesto and the principles applied therein, namely self-organizing teams, working software for each Sprint review in Scrum and user stories for voice of the customer. From there, we’ll work in the specifics of DW & BI development, which is different in many aspects to developing software in Java or .NET. And since we’re using basic Agile Scrum practices, nearly any good book or article on forming Scrum teams will work.

I also like to emphasize TDD (test-driven development) as an approach that is most effective to quality products in business intelligence and data warehouse solutions. To classic software development professionals, this may seem like a no-brainer. TDD has been also proven very effective at producing quality iterations in development cycles. However, once again, in the DW & BI world, TDD is a bit more difficult. Sticking with data warehouse development for this initial post, it’s important to remember that DW professionals will develop solutions using ETL 4GL tools, SQL code and workflow integration packages. Tool vendors like Microsoft (Visual Studio with Data-Tier Applications, SSIS 2012 and TFS include some support for Agile, versioning, releases and TDD) and open-source tools like Pentaho are moving more & more toward Agile BI.

But generally speaking, it can be challenging for DWBI developers to easily integrate these Agile practices into the development cycle.

My experience has been that it is possible to self-organize, test-first, generate user stories and develop from a backlog in 30-day iterations with working solutions at the end of each iteration in the DWBI world. There is, however, a specific persona in these projects whose job becomes more challenging, as opposed to easier, with Agile, that is not found as much in other software engineering projects. That is the role of the data modeler.

In large and highly-integrated data warehouse solutions, the data modeler is especially important, as you will likely benefit from a canonical model with Web Services integration and contract-first approaches. This can make the reality of a highly-flexible 30-day iteration approach difficult for a role whose intent is to keep things locked-down and with a scope that is quite larger. Certainly much larger than what can be accomplished in 30 days.

Therefore, the approach that we took was to plan a separate set of data modeling sprints that were devised from a backlog coming from user stories that were developed before Sprint Zero even began on the DW project. Within 3 months, we had the model defined such that services, analytics, UX and other areas of the solution could work against a solid set of interfaces. Those Sprints also gave us the opportunity to test and prove-out the model before moving forward by using prototype mock-up reports that used completely auto-generated data in the databases.

So, that’s a starting point for your journey into Agile Analytics from my perspective. I’ve worked on a number of very large DWBI projects where we chose to organize in Agile Scrum teams and it has worked very effectively. There were a number of growing pains, as in each case, many of the team members were new to the approach. I’ll keep posting here and sharing my experiences and lessons learned to help you on your way to successful data warehouse business intelligence projects!

Br, Mark