Business Intelligence Nuggets

The Importance of Planning in BI

Derek Comingore — Sun, 17 Aug 2008 19:54:24 GMT

Business Intelligence projects (like software development projects) are prone to bad or completely absent planning prior to their start. Most of us that work in the technology industries are busy professionals that can never seem to find enough time. When you consider that mindset and then the fact that project planning requires additional time...well you get the point. By our very nature we do not want to waste time!

BI projects are large endeavors, rather BI enterprise projects are large endeavors. Personally, I have been fortunate enough to plan, design, and implement several enterprise systems over the years with some of the best BI professionals. And every time I was glad that my team made a plan!

'Scope Creep' is just as common in BI projects as it is in software development. It is easy to not remain focused through the entire lifecycle of an enterprise BI project. The plan can also serve as a deliverable schedule for your project sponsors (usually executives). In a consulting engagement scenario the importance of planning is of even greater importance as the entire project will be based on deliverables and their scheduling.

While I agree that we hate to waste time, I can tell you that proper planning (for any project) is NOT a waste of time.

Value-Add: Data Warehousing & Analysis Services Cube Storage Estimates

Derek Comingore — Tue, 22 Jul 2008 01:45:40 GMT

A common question I hear from my clients is “how much storage do we need?” Now wait a second, I thought storage was cheap these days right? Wrong, consumer storage is cheap; enterprise storage is still not what I would classify as cheap. Add to this fact that Enterprise Data Warehouses (EDWs) commonly break into the several terabytes range and the matter of planning storage capacities becomes greater. Consulting firms that simply ignore this common client question are missing a value-add scenario. Your clients need to know how much space to plan for and if you simply say “roughly X TB” you are creating two net results. The first is that the client may purchase too much storage or may not purchase enough (very bad). Second, the client will have no idea as to how much storage to plan for in the coming years. Do both yourself and the client a favor; research the storage requirements of the BI solution you have proposed!

Estimating Data Warehouse Storage

To estimate a data warehouse’s storage requirements have a dimensional model designed in any number of tools including Visio, Erwin, or Excel. The main point is to have the actual data types required and their storage affecting settings such as variable length string columns designated. When estimating your data warehouse storage requirements make sure you assume the data types have their max allowed values being used for every record. The thinking is that it’s better to overestimate then under when it comes to storage (to a degree).

Now, obtain from the client how many records they believe will be processed and stored in the data warehoused based upon business requirements. Note how the storage requirements of fact tables are directly related to their grain definition. If your fact tables must store data by day you will certainly require more storage than if you only need to store the various measures by week. This is why it’s very important to have a solid dimensional model phase completed prior to entering other data warehousing tasks such as storage estimates. At this point you should instantiate the dimensional model in a SQL Server relational database.

Populate your schema with a sample percentage of the client’s estimates. For example, if the client has said that you can expect one million records per month and you must retain three years worth of data then you can expect to hold a total of around 36 million records. Based upon the estimates populate your fact tables with 1, 5, 10, or even 25 percent of the total records expected to be retained. Rebuild your fact table’s indexes. Finally, obtain the data and index spaced used by your fact tables and multiply those values according to the percentage of sampling you used.

Estimating Analysis Services Cube Storage

Clients not only need to know how much storage to obtain for the relational dimensional model but also the cubes (multidimensional objects). Estimating cube storage requirements is a more difficult task; however the same basic logic applies. Design and build your sample cubes as close as you can to how they will be built in production. The most important settings for estimating cube storage are the partition aggregations & storage mode settings. In a development environment you will probably only have a single partition, so there is an error of margin when estimating cube storage requirements since you will likely have multiple partitions with various storage settings in a production environment. Finally, perform a full process on the Analysis Services database once you are done creating the sample cubes and their related settings.

ROLAP & HOLAP w/0% Aggregations incur zero storage overhead excluding the cube’s metadata. If you apply aggregations to the HOLAP storage mode the aggregations will occur in the partition files (cube), thus consuming space. If you apply aggregations to the ROLAP storage mode, indexed views are created (and populated) in the underlying relational database (which means ROLAP with aggregations does in fact incur some storage overhead). MOLAP stores both grain data as well as aggregations in the partition files (cube). Finally, remote partitions can be used for placing less frequently access data onto cheaper disks.

With your sample cubes built and processed you can begin the cube estimation process. Obtain the sample cubes’ partition(s) estimated size in Management studio. Next, multiply the partition storage estimates according to the percentage of sampling you used (just like we did in the fact table estimates). Note that this method assumes both the storage mode and the aggregations are consistent throughout each cube (which is fairly unlikely in production environments). I spoke about the storage mode margin of error earlier but I should also mention that the % aggregations effect cube partition storage estimates as well. In the real world aggregation designs change, however for estimation purposes it’s easiest to assume a static setting of X%.

By using some variation of the methods mentioned above you will have a solid foundation for providing accurate estimates of storage required for both dimensional data warehouses as well as their associated cubes.

So you want to be a Microsoft DW/BI Pro

Derek Comingore — Tue, 08 Jul 2008 02:25:50 GMT

As a follow up to some recent feedback (actually a question) here is some more in depth information on becoming proficient in the Microsoft Data Warehousing/Business Intelligence stack.

Get the terms straight

Business Intelligence or BI is an umbrella term for various business processes, tools, and methodologies which support improved organizational decision making. Data Warehousing (and thus data warehouses) on the other hand are organizational repositories designed to facilitate highly scalable reporting & analytics. DW and BI go together like PB & J (ah the good ol' days). So is Data Warehousing synonymous with BI...NO! Data Warehousing is a common solution (due to its many advantages) implemented to enable BI in an organization....so when you see DW/BI you will now understand that a) DW/BI doesn’t mean they represent the same thing and b) that DW and BI merely complement each other very well thus the term's existence.

Now, understand before ever cracking open a BI book (or scouring the web all day for free information, who would do such a thing?) that REPORTING <> BI. The BI industry is plagued with terminology problems! So, DW <> BI and Reporting <> BI. And like DW, Reporting is merely one component of the larger field of BI.

Personal Microsoft Business Intelligence Professional Learning Plan

Understand before I dive into this that the words to follow are from a DW/BI architect/engineer and not a fulltime trainer. Actually, I wrote a seperate post on the pros/cons of fulltime trainers prior on this blog here. So with that disclaimer out of the way here are the topics in chronological order I would propose to anyone wishing to learn how to implement DW/BI systems on the Microsoft platform.

Foundations (some of these topics overlap to various degrees)
- Data Warehousing (DW) Fundamentals (Normalized DW Vs Denormalized DW)
- Extract, Transform, and Load (ETL) Fundamentals
- Dimensional Modeling Fundamentals
- OnLine Analytical Processing (OLAP) Fundamentals
- Data Mining Fundamentals (I have more to learn here as well...in fact most do)
- Brief History of DSS (decision support systems) *You could argue you don't need this knowledge and in so far as getting the "job done" you don't.
DW/BI Methodologies
- Kimball (www.kimballgroup.com) is pretty much the defacto standard for HOW to build an enterprise data warehouse on the Microsoft Platform. I personally have not read others work but do know of them and by all means use what ever methodology fits you and your DW/BI projects best.
Microsoft DW/BI Tools (now to the technology)
- SQL Server 2005 (including Analysis/Reporting/Integration Services)
- Microsoft Office 2007 Suite (namely SharePoint, PerformancePoint & Excel)

Best "WWW Stops" (in no particular order)

www.sqlmag.com/businessintelligence/bi
www.microsoft.com/bi
www.microsoft.com/sql/solutions/bi/default.aspx
www.microsoft.com/sql/solutions/dw/default.aspx
www.tdwi.org
Roger Wolter's Blog (MSFT MDM) http://blogs.msdn.com/rogerwolterblog/default.aspx
Chris Web's Blog (MDX) http://cwebbbi.spaces.live.com
Mosha Pasumansky's Blog (MDX) http://sqlblog.com/blogs/mosha/default.aspx
Jesse Orosz's Blog (SSAS) http://jesseorosz.spaces.live.com
Vidas Matelis's Blog (SSAS) www.ssas-info.com/VidasMatelisBlog/
Jamie Thompson's Blog (SSIS) http://blogs.conchango.com/jamiethomson/
Reporting Services' Team Blog (SSRS) http://blogs.msdn.com/sqlrsteamblog/

What you see is what you get

So thats my list (and best "stops") for now. Those of you new to the Microsoft platform may laugh that there are only a handful of products to learn but be forewarned that both SQL Server and the Office products are VERY LARGE offerings. SharePoint & SQL Server in particular can (and often do) fill bookshelves worth of valuable content.

I hope that helps everyone out there who is considering learning more and/or making a career in the MSFT DW/BI space. There is a lot to learn, however you will find that having a solid background in databases (and a bit of programming) goes a long way in learning the actual tools. It also seems to be true according to the TDWI (www.tdwi.org) that most DW/BI Professionals in general (not just those using the Microsoft platform) have lived a "past life" in another part of IT (usually dev/dba). Most people do not go directly into the field of DW/BI (which I personally 100% support)!

The Business Behind BI

Derek Comingore — Thu, 03 Jul 2008 15:13:03 GMT

Hope everyone is doing well out there. A good friend of mine across the pond mentioned that he has started to read a lot of my content across the blogs, my SQL Magazine Articles/BI Column, etc. His feedback was excellent and relates to a general perception issue with the field of BI. "They are highly technology based" was his comment in so many words...

First a few clarifications are in order, writing for SQL Magazine means you will be discussing the actual data platform called SQL Server at some length. This blog and other places though I could write more BI neurtral content. And thus I have created a new post category called "Business Intelligence Nuggets" (which this post is the first in).

The perception issue is that Business Intelligence is all about technology! Business Intelligence is 100% about business, THE TOOLS ARE SIMPLY ENABLERS. A well thought out dimensional model will work well on any database platform. A solid ETL design will run great on any ETL platform it is implemented in. The larger point is, a great BI architect will be good (and thus his designs) on any platform or toolset.

There are layers to any BI solution nomatter how or what it is constructed of. First is the Business and it's logic/requirements. Second, is design and methodologies...the words Kimball & Inmon come to mind. Third, are the tools...in my case SQL Server, Analysis Services, Reporting Services, and Integration Services.

Tools (SQL Server etc.)

Design (Design Patterns, Best Practices, & Methodologies)

Business Requirements (What needs to be measured?)