Why Agile Analytics?

Instead of digging into the weeds of Scrum teams and Agile methodology, let me just give you a quick, short, snappy use case that we are doing @ Razorfish on the BI Platform team that shows true business value for developing data warehouses, analytics and business intelligence solutions in an Agile, iterative manner.

We have several large data warehouses in SQL Server and Teradata Aster along with large distributed file systems using Hadoop for data scientists. When the DW/BI development teams work on a new subject area, new features or other development projects on Big Data and VLDBs, it will take time, care, caution and incremental change.

Now, in our case, we already have an established revenue stream and cost justification for servers & infrastructure for Big Data. But suppose that you are starting a new project that will use data warehouses for BI and analytics and will demonstrate business value such as strategic and marketing advantages that your business cannot achieve today because they cannot mine all that data.

But if you put together a traditional project team and project plan, you may have milestones that drag out on the magnitude of years, not weeks or months. Additionally, those milestones will include gates and those events on the project plan will likely entail project work like requirements analysis, risk assessments, developer checkpoints, analyst checkpoints, etc.

Now suppose instead, that you build the team as a self-evolving group that estimates work based on story points instead of hours and builds only enough backlog that a fully functioning feature can be demonstrated back to the business in 3-4 weeks.

After 2 months (assume 1 sprint for establishing velocity and team establishment), you will have BI reports and access to knowledge mined from Big Data that your business never had before. It will be demonstrable, probably with shims and some prototypes so that it can be demo’d. But it will include data for the demo and will function end-to-end. And after 3 more weeks, you’ll demonstrate more reports with more & more real, live data each iteration.

When you compare that against long development cycles with planned demonstration checkpoints of code that is not fully tested or integrated. Agile will demand that the code is unit, integration and regression tested each sprint.

My experience has shown that Agile approach in DW/BI works and will make you much more successful in keeping the business happy and your Big Data investment safe.