CrossTabs / Pivoting Data

UNPIVOT: Normalizing data on the fly

Jeff Smith — Wed, 23 Apr 2008 14:33:15 GMT

Everyone seems to want to "pivot" or "cross tab" data, but knowing how to do the opposite is equally important. In fact, I would argue that the skill of "unpivoting" data is more useful and more important and more relevant to a SQL programmer, since pivoting results in denormalized data, while unpivoting can transform non-normalized data into a normalized result set. We all know that there's lots of bad databases designs out there, so this can be a handy technique to know.

Of course, even a well designed, fully normalized database can still benefit from "unpivoting" from time to time, so let's take a look at some common situations and some of the options we have to handle this at our disposal. We will focus on some traditional SQL techniques to do this, and then take a close look at the UNPIVOT operator that was introduced with SQL Server 2005.

Example #1: A Bad database design

Let's start with a commonly bad table design, in which someone has decided to relate a client to multiple contacts by designing their client table like this:

create table Clients
(
    clientID int primary key,
    clientName varchar(100),
    contact1 int,
    contact2 int,
    contact3 int,
    contact4 int
)

insert into Clients
select 1,'ABC Corp',1,34,2,null union all
select 2,'DEF Foundation',6,2,8,9 union all
select 3,'GHI Inc.',5,9,null,null union all
select 4,'XYZ Industries',24,null,6,null

clientID    clientName           contact1    contact2    contact3    contact4
----------- -------------------- ----------- ----------- ----------- -----------
1           ABC Corp             1           34          2           NULL
2           DEF Foundation       6           2           8           9
3           GHI Inc.             5           9           NULL        NULL
4           XYZ Industries       24          NULL        6           NULL

(4 row(s) affected)

(Note: For brevity, I am not including the contact table here, nor the foreign key constraints. Of course, with this table design, it would probably be pretty unlikely to find such constraints in the database anyway)

With this design, it is not very easy or efficient to get a count of all contacts for each client, or to find out which contacts are related to which clients. One thing we can do, however is to "unpivot" this table in a query that returns 1 row per ClientID/ContactID combination. With that result set, we can easily now reference the table as if it were normalized and we can get the information we need.

One way to do is to use UNION ALL to return each row in the clients table 4 times, and each time return a different contactID column:

select clientID, contact1 as ContactID
from clients
where contact1 is not null
union all
select clientID, contact2 as ContactID
from clients
where contact2 is not null
union all
select clientID, contact3 as ContactID
from clients
where contact3 is not null
union all
select clientID, contact4 as ContactID
from clients
where contact4 is not null

clientID    ContactID
----------- -----------
1           1
2           6
3           5
4           24
1           34
2           2
3           9
1           2
2           8
4           6
2           9

(11 row(s) affected)

Another option is to CROSS JOIN the Clients table with a table or resultset that returns 4 rows, which also effectively returns each row in the clients table 4 times. For each of the 4 values in the table we are cross joining, we grab a different contact column:

select *
from
(
    select c.clientID,
       case n.n when 1 then c.contact1
            when 2 then c.contact2
            when 3 then c.contact3
            when 4 then c.contact4 end as ContactID
    from
        clients c
    cross join
        (select 1 as n union all select 2 union all select 3 union all select 4) n
)
    x
where
    x.ContactID is not null

clientID    ContactID
----------- -----------
1           1
1           34
1           2
2           6
2           2
2           8
2           9
3           5
3           9
4           24
4           6

(11 row(s) affected)

(Note that you can use a permanent table of Numbers in your database instead of generating it on the fly with a UNION, as shown)

Finally, however, there is an even eaiser way to handle this: the UNPIVOT operator, new with SQL 2005. UNPIVOT works very efficiently and really allows you to handle this exact situation quite easily:

select clientID, Contact.ContactID
from clients
unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

clientID    ContactID
----------- -----------
1           1
1           34
1           2
2           6
2           2
2           8
2           9
3           5
3           9
4           24
4           6

(11 row(s) affected)

Much shorter to write, and more efficient to execute as well.

Taking a Closer Look at UNPIVOT

The UNPIVOT operator is tricky to get a feel for, however, so let's take a look at it.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

First, the "As Contact" at the end is just labeling the entire unpivot result set with an alias, just as you must alias a derived table. Each column returned by the pivot operator can be referenced by the alias if necessary.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

The "ContactID for" part says that we want to return a column called "ContactID" for each unpivoted row. The IN() list is the columns that we are unpivoting; the values in the 4 columns listed here will be assigned to the ContactID column in the result. So, the first time a particular row is unpivoted, the value of the 'contact1' column is assigned to ContactID, the next time it is the 'contact2' column, then 'contact3', and then finally 'contact4'. Then, the next row is processed and it all begins again.

Thus, because we are unpivoting 4 values, we know that the result of the unpivot will have 4 times as many rows as the source data.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

UNPIVOT returns an additional column as well, which contains the name of the column that was used to produce each unpivoted row. Here, we have specified that to be called ContactNumber. Note that we actually did not return ContactNumber in our example, be we can easily add that in so you can see how it works:

select clientID, Contact.ContactNumber, Contact.ContactID
from clients
unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

clientID    ContactNumber           ContactID
----------- ----------------------- -----------
1           contact1                1
1           contact2                34
1           contact3                2
2           contact1                6
2           contact2                2
2           contact3                8
2           contact4                9
3           contact1                5
3           contact2                9
4           contact1                24
4           contact3                6

(11 row(s) affected)

So, you can see that the code to write is very short, but a little difficult to grasp at first. In the end, though, we are able to take a bad table design and easily "fix it", at least temporarily, so that we can query it using simple and standard SQL statements to get what we need.

Example #2: Normalizing a Transaction Table

Here's another common example:

create table Transactions
(
    TranDate datetime,
    Account varchar(10),
    BudgetAmount money,
    ActualAmount money,
    ProjectionAmount money,
    primary key (TranDate, Account)
)
go
insert into Transactions
select '2008-01-01','0001',354,65,58 union all
select '2008-01-02','0001',14,65,34 union all
select '2008-01-03','0001',0,65,622 union all
select '2008-01-04','0001',9,32,84
go

TranDate                Account    BudgetAmount          ActualAmount          ProjectionAmount
----------------------- ---------- --------------------- --------------------- ---------------------
2008-01-01 00:00:00.000 0001       354.00                65.00                 58.00
2008-01-02 00:00:00.000 0001       14.00                 65.00                 34.00
2008-01-03 00:00:00.000 0001       0.00                  65.00                 622.00
2008-01-04 00:00:00.000 0001       9.00                  32.00                 84.00

(4 row(s) affected)

Notice that we have different columns for Budget, Actual and Projection, which is not really a great database design. Much better would be to break this data out so that we have a single 'Amount' column and a 'TransactionType' column that specifies the type of each transaction. We can transform our Transactions tables into this format using UNION ALL:

select TranDate, Account, 'BudgetAmount' as Type, BudgetAmount as Amount from transactions
union all
select TranDate, Account, 'ActualAmount' as Type, ActualAmount as Amount from transactions
union all
select TranDate, Account, 'ProjectionAmount' as Type, ProjectionAmount as Amount from transactions

TranDate                Account    Type             Amount
----------------------- ---------- ---------------- ---------------------
2008-01-01 00:00:00.000 0001       BudgetAmount     354.00
2008-01-02 00:00:00.000 0001       BudgetAmount     14.00
2008-01-03 00:00:00.000 0001       BudgetAmount     0.00
2008-01-04 00:00:00.000 0001       BudgetAmount     9.00
2008-01-01 00:00:00.000 0001       ActualAmount     65.00
2008-01-02 00:00:00.000 0001       ActualAmount     65.00
2008-01-03 00:00:00.000 0001       ActualAmount     65.00
2008-01-04 00:00:00.000 0001       ActualAmount     32.00
2008-01-01 00:00:00.000 0001       ProjectionAmount 58.00
2008-01-02 00:00:00.000 0001       ProjectionAmount 34.00
2008-01-03 00:00:00.000 0001       ProjectionAmount 622.00
2008-01-04 00:00:00.000 0001       ProjectionAmount 84.00

(12 row(s) affected)

Or, we can use the UNPIVOT operator to do the same much easier:

select TranDate, Account, Type, Amount
from Transactions
unpivot (Amount for Type in (BudgetAmount, ActualAmount, ProjectionAmount)) as Amount

TranDate                Account   Type                     Amount
----------------------- --------- ------------------------ ---------------------
2008-01-01 00:00:00.000 0001      BudgetAmount             354.00
2008-01-01 00:00:00.000 0001      ActualAmount             65.00
2008-01-01 00:00:00.000 0001      ProjectionAmount         58.00
2008-01-02 00:00:00.000 0001      BudgetAmount             14.00
2008-01-02 00:00:00.000 0001      ActualAmount             65.00
2008-01-02 00:00:00.000 0001      ProjectionAmount         34.00
2008-01-03 00:00:00.000 0001      BudgetAmount             0.00
2008-01-03 00:00:00.000 0001      ActualAmount             65.00
2008-01-03 00:00:00.000 0001      ProjectionAmount         622.00
2008-01-04 00:00:00.000 0001      BudgetAmount             9.00
2008-01-04 00:00:00.000 0001      ActualAmount             32.00
2008-01-04 00:00:00.000 0001      ProjectionAmount         84.00

(12 row(s) affected)

Example #3: "Unsummarizing" Data

For those who work with accounting systems, this example may be familiar to you. Many times, Accounting systems have "summary" tables that roll up transactional data into a structure like this:

create table AccountBalances
(
    CompanyID int,
    AccountID int,
    TransactionTypeID int,
    Year int,
    Period1 money,
    Period2 money,
    Period3 money,
    Period4 money,
    Period5 money,
    Period6 money,
    Period7 money,
    Period8 money,
    Period9 money,
    Period10 money,
    Period11 money,
    Period12 money
)

insert into AccountBalances
select 1,1,1,2008,200,300,400,500,400,0,0,0,0,0,0,0 union all
select 1,2,1,2008,100,100,100,100,100,100,100,0,0,0,0,0 union all
select 1,3,1,2008,150,0,50,10,10,200,400,45,0,0,0,0

(As before, let's not worry about those foreign key constraints)

These tables are often calculated when transactions are posted, or periods are closed. Typically, many reports pull from these tables because it is much more efficient than summarizing thousands or millions of transactions, and the data is already "cross-tabbed" the way most reporting tools would like to display it.

We can take this summarized data and "unpivot" it so we can still access the summarized data, but now it will be in a normalized structure. All it takes is a simple UNPIVOT like this:

select CompanyID, AccountID, TransactionTypeID, Year, substring(Period,7,2) as PeriodNo, Amount
from AccountBalances
unpivot (Amount for Period in (Period1,Period2,Period3,Period4,Period5,Period6,Period7,Period8,Period9,Period10,Period11,Period12)) as Amount

CompanyID   AccountID   TransactionTypeID Year        PeriodNo Amount
----------- ----------- ----------------- ----------- -------- ---------------------
1           1           1                 2008        1        200.00
1           1           1                 2008        2        300.00
1           1           1                 2008        3        400.00
1           1           1                 2008        4        500.00
1           1           1                 2008        5        400.00
1           1           1                 2008        6        0.00
1           1           1                 2008        7        0.00
1           1           1                 2008        8        0.00
1           1           1                 2008        9        0.00
1           1           1                 2008        10       0.00
1           1           1                 2008        11       0.00
1           1           1                 2008        12       0.00
1           2           1                 2008        1        100.00
1           2           1                 2008        2        100.00
1           2           1                 2008        3        100.00
1           2           1                 2008        4        100.00
1           2           1                 2008        5        100.00
1           2           1                 2008        6        100.00
1           2           1                 2008        7        100.00
1           2           1                 2008        8        0.00
1           2           1                 2008        9        0.00
1           2           1                 2008        10       0.00
1           2           1                 2008        11       0.00
1           2           1                 2008        12       0.00
1           3           1                 2008        1        150.00
1           3           1                 2008        2        0.00
1           3           1                 2008        3        50.00
1           3           1                 2008        4        10.00
1           3           1                 2008        5        10.00
1           3           1                 2008        6        200.00
1           3           1                 2008        7        400.00
1           3           1                 2008        8        45.00
1           3           1                 2008        9        0.00
1           3           1                 2008        10       0.00
1           3           1                 2008        11       0.00
1           3           1                 2008        12       0.00

(36 row(s) affected)

We can filter so that only periods with a non-zero amount are included, and we can SELECT FROM this result set and get the exact data we need for whatever date range we want without worrying which column the actual data is in.

This can also be done with a CROSS JOIN or a UNION ALL, but with 12 values to pivot, those options become much longer to write and UNPIVOT appears to be the way to go in this case.

Example #4: Multiple unpivots

Finally, let's consider a more complicated example. Here, we have a table that stores games played between two teams, where one team is the HomeTeam and the other is the AwayTeam:

create table Teams
(
    TeamCode char(3) primary key not null,
    TeamName varchar(100) not null
)

create table Games
(
    GameDate datetime,
    HomeTeam char(3) references Teams(TeamCode),
    AwayTeam char(3) references Teams(TeamCode),
    HomeScore int,
    AwayScore int,
    primary key (GameDate, HomeTeam),
    constraint pk2 unique (GameDate, AwayTeam),
    check (HomeTeam <> AwayTeam)
)

insert into Teams
select 'BOS','Boston Red Sox' union all
select 'NYY','New York Yankees'

insert into Games
select '2008-04-01','BOS','NYY',3,1 union all
select '2008-04-02','BOS','NYY',6,4 union all
select '2008-04-03','BOS','NYY',2,3 union all
select '2008-04-08','NYY','BOS',6,0 union all
select '2008-04-09','NYY','BOS',2,6 union all
select '2008-04-10','NYY','BOS',1,10

GameDate                HomeTeam AwayTeam HomeScore   AwayScore
----------------------- -------- -------- ----------- -----------
2008-04-01 00:00:00.000 BOS      NYY      3           1
2008-04-02 00:00:00.000 BOS      NYY      6           4
2008-04-03 00:00:00.000 BOS      NYY      2           3
2008-04-08 00:00:00.000 NYY      BOS      6           0
2008-04-09 00:00:00.000 NYY      BOS      2           6
2008-04-10 00:00:00.000 NYY      BOS      1           10

(6 row(s) affected)

That may or may not be the best possible design for the Games table, but it is a common way to do it. With data in that form, if we want to get the total runs scored per team across all games, or get each teams won-loss record, we need to make two passes through the table. This can be done fairly easily and efficiently with a union:

select GameDate,
    HomeTeam as TeamCode,
    'Home' as HomeOrAway,
    HomeScore as Score,
    case when HomeScore > AwayScore then 1 else 0 end as Win,
    case when HomeScore < AwayScore then 1 else 0 end as Loss,
    case when HomeScore = AwayScore then 1 else 0 end as Tie
from
    Games
union all
select GameDate,
    AwayTeam as TeamCode,
    'Away' as HomeOrAway,
    AwayScore as Score,
    case when HomeScore < AwayScore then 1 else 0 end as Win,
    case when HomeScore > AwayScore then 1 else 0 end as Loss,
    case when HomeScore = AwayScore then 1 else 0 end as Tie
from
    Games

GameDate                TeamCode HomeOrAway Score       Win         Loss        Tie
----------------------- -------- ---------- ----------- ----------- ----------- -----------
2008-04-01 00:00:00.000 BOS      Home       3           1           0           0
2008-04-02 00:00:00.000 BOS      Home       6           1           0           0
2008-04-03 00:00:00.000 BOS      Home       2           0           1           0
2008-04-08 00:00:00.000 NYY      Home       6           1           0           0
2008-04-09 00:00:00.000 NYY      Home       2           0           1           0
2008-04-10 00:00:00.000 NYY      Home       1           0           1           0
2008-04-01 00:00:00.000 NYY      Away       1           0           1           0
2008-04-02 00:00:00.000 NYY      Away       4           0           1           0
2008-04-03 00:00:00.000 NYY      Away       3           1           0           0
2008-04-08 00:00:00.000 BOS      Away       0           0           1           0
2008-04-09 00:00:00.000 BOS      Away       6           1           0           0
2008-04-10 00:00:00.000 BOS      Away       10          1           0           0

(12 row(s) affected)

Now, how can we do this with UNPIVOT? This example is a bit more complicated, because we are "unpivoting" not only the score, but also the TeamCode, and we are also calculating a few extra columns (Win, Loss and Tie).

So, can we UNPIVOT more than 1 column? Let's start simply and do things one at a time. First, let's UNPIVOT just the TeamCode:

select GameDate, HomeOrAway, Team
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team

GameDate                HomeOrAway     Team
----------------------- -------------- ----
2008-04-01 00:00:00.000 HomeTeam       BOS
2008-04-01 00:00:00.000 AwayTeam       NYY
2008-04-02 00:00:00.000 HomeTeam       BOS
2008-04-02 00:00:00.000 AwayTeam       NYY
2008-04-03 00:00:00.000 HomeTeam       BOS
2008-04-03 00:00:00.000 AwayTeam       NYY
2008-04-08 00:00:00.000 HomeTeam       NYY
2008-04-08 00:00:00.000 AwayTeam       BOS
2008-04-09 00:00:00.000 HomeTeam       NYY
2008-04-09 00:00:00.000 AwayTeam       BOS
2008-04-10 00:00:00.000 HomeTeam       NYY
2008-04-10 00:00:00.000 AwayTeam       BOS

(12 row(s) affected)

OK, so far so good. Now, how do we get the score for the team as well? Let's add another UNPIVOT clause to the SELECT, this time for Score:

select GameDate, HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway in (HomeScore, AwayScore)) as Score

Msg 265, Level 16, State 1, Line 1
The column name "HomeOrAway" specified in the UNPIVOT operator conflicts with the existing column name in the UNPIVOT argument.
Msg 8156, Level 16, State 1, Line 1
The column 'HomeOrAway' was specified multiple times for 'Score'.

Hmmm. OK, we cannot specify the same column for both pivots. That is just an alias specification, no big deal, so let's just alias it as "HomeOrAway2" :

select GameDate, HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   BOS 3
2008-04-01 00:00:00.000 HomeTeam   BOS 1
2008-04-01 00:00:00.000 AwayTeam   NYY 3
2008-04-01 00:00:00.000 AwayTeam   NYY 1
2008-04-02 00:00:00.000 HomeTeam   BOS 6
2008-04-02 00:00:00.000 HomeTeam   BOS 4
2008-04-02 00:00:00.000 AwayTeam   NYY 6
2008-04-02 00:00:00.000 AwayTeam   NYY 4
2008-04-03 00:00:00.000 HomeTeam   BOS 2
2008-04-03 00:00:00.000 HomeTeam   BOS 3
2008-04-03 00:00:00.000 AwayTeam   NYY 2
2008-04-03 00:00:00.000 AwayTeam   NYY 3
2008-04-08 00:00:00.000 HomeTeam   NYY 6
2008-04-08 00:00:00.000 HomeTeam   NYY 0
2008-04-08 00:00:00.000 AwayTeam   BOS 6
2008-04-08 00:00:00.000 AwayTeam   BOS 0
2008-04-09 00:00:00.000 HomeTeam   NYY 2
2008-04-09 00:00:00.000 HomeTeam   NYY 6
2008-04-09 00:00:00.000 AwayTeam   BOS 2
2008-04-09 00:00:00.000 AwayTeam   BOS 6
2008-04-10 00:00:00.000 HomeTeam   NYY 1
2008-04-10 00:00:00.000 HomeTeam   NYY 10
2008-04-10 00:00:00.000 AwayTeam   BOS 1
2008-04-10 00:00:00.000 AwayTeam   BOS 10

(24 row(s) affected)

Holy schnikies, it works! We can specify more than one UNPIVOT clause for the same SQL statement! Who would have thunk it?

Uh oh -- wait a second. We have 24 results returned. We should have only 12. Something is not right here. Remember when we aliased our second unpivot column as "HomeOrAway2"? We did not return that anywhere in our results. Let's add that in and take a look:

select GameDate, HomeOrAway, HomeOrAway2, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score

GameDate                HomeOrAway HomeOrAway2 Team Score
----------------------- ---------- ----------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   HomeScore   BOS 3
2008-04-01 00:00:00.000 HomeTeam   AwayScore   BOS 1
2008-04-01 00:00:00.000 AwayTeam   HomeScore   NYY 3
2008-04-01 00:00:00.000 AwayTeam   AwayScore   NYY 1
2008-04-02 00:00:00.000 HomeTeam   HomeScore   BOS 6
2008-04-02 00:00:00.000 HomeTeam   AwayScore   BOS 4
2008-04-02 00:00:00.000 AwayTeam   HomeScore   NYY 6
2008-04-02 00:00:00.000 AwayTeam   AwayScore   NYY 4
2008-04-03 00:00:00.000 HomeTeam   HomeScore   BOS 2
2008-04-03 00:00:00.000 HomeTeam   AwayScore   BOS 3
2008-04-03 00:00:00.000 AwayTeam   HomeScore   NYY 2
2008-04-03 00:00:00.000 AwayTeam   AwayScore   NYY 3
2008-04-08 00:00:00.000 HomeTeam   HomeScore   NYY 6
2008-04-08 00:00:00.000 HomeTeam   AwayScore   NYY 0
2008-04-08 00:00:00.000 AwayTeam   HomeScore   BOS 6
2008-04-08 00:00:00.000 AwayTeam   AwayScore   BOS 0
2008-04-09 00:00:00.000 HomeTeam   HomeScore   NYY 2
2008-04-09 00:00:00.000 HomeTeam   AwayScore   NYY 6
2008-04-09 00:00:00.000 AwayTeam   HomeScore   BOS 2
2008-04-09 00:00:00.000 AwayTeam   AwayScore   BOS 6
2008-04-10 00:00:00.000 HomeTeam   HomeScore   NYY 1
2008-04-10 00:00:00.000 HomeTeam   AwayScore   NYY 10
2008-04-10 00:00:00.000 AwayTeam   HomeScore   BOS 1
2008-04-10 00:00:00.000 AwayTeam   AwayScore   BOS 10

(24 row(s) affected)

Notice what is happening here. Without an UNPIVOT, we had 6 rows. We added one UNPIVOT to pivot two values, and got 12 rows. Then, we added another pivot, again with two values, and got back 24. Looking at the results, we can see that the single game at '2008-04-01' was unpivoted into 4 rows. So, while we can unpivot multiple times, there is no relation between each UNPIVOT. That is, notice there is no correlation between HomeOrAway and HomeOrAway2. Let's look at just that first game:

GameDate                HomeOrAway HomeOrAway2 Team Score
----------------------- ---------- ----------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   HomeScore   BOS 3
2008-04-01 00:00:00.000 HomeTeam   AwayScore   BOS 1
2008-04-01 00:00:00.000 AwayTeam   HomeScore   NYY 3
2008-04-01 00:00:00.000 AwayTeam   AwayScore   NYY 1

Both scores, home and away, are specified for each team -- even if it is not their matching score. What we really want is only these two lines:

How can we get there? Well, there's always a WHERE clause! We can simply filter our result so that the Left 4 characters of HomeOrAway matches the left 4 characters of HomeOrAway2. After all, there is nothing magical about these unpivoted columns, they are data just like anything else.

So, we can now write:

select GameDate, left(HomeOrAway,4) as HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score
where left(HomeOrAway,4) = left(HomeOrAway2,4)

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 Home       BOS 3
2008-04-01 00:00:00.000 Away       NYY 1
2008-04-02 00:00:00.000 Home       BOS 6
2008-04-02 00:00:00.000 Away       NYY 4
2008-04-03 00:00:00.000 Home       BOS 2
2008-04-03 00:00:00.000 Away       NYY 3
2008-04-08 00:00:00.000 Home       NYY 6
2008-04-08 00:00:00.000 Away       BOS 0
2008-04-09 00:00:00.000 Home       NYY 2
2008-04-09 00:00:00.000 Away       BOS 6
2008-04-10 00:00:00.000 Home       NYY 1
2008-04-10 00:00:00.000 Away       BOS 10

(12 row(s) affected)

Now we are back to 12 results returned, the Homes and Aways match up, and things look good! Hey, we just pulled off 2 unpivots at once, not bad! From here, we can add our Win, Loss and Tie calculations as well. Note that we also have removed "HomeOrAway2" from the SELECT, since we don't need to see that column in our result. Finally, we are returning only the left 4 characters of HomeTeam or AwayTeam, so that the word 'Home' or 'Away' is returned in the HomeOrAway column.

This seems like quite a bit of work. Is UNPIVOT really the way to go in this scenario? Maybe not. I have not done full performance testing of UNPIVOT versus the other options shown (this article is way too long as it is!) but as always you should fully test your options and decide which offers the best performance and code simplicity.

Another option if you want to unpivot more than 1 column is to just use one UNPIVOT cause, but then use CASE expressions to return the values you need. For example:

select GameDate, left(HomeOrAway,4) as HomeOrAway,
Team,
case when HomeOrAway='HomeTeam' then HomeScore else AwayScore end as Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 Home       BOS 3
2008-04-01 00:00:00.000 Away       NYY 1
2008-04-02 00:00:00.000 Home       BOS 6
2008-04-02 00:00:00.000 Away       NYY 4
2008-04-03 00:00:00.000 Home       BOS 2
2008-04-03 00:00:00.000 Away       NYY 3
2008-04-08 00:00:00.000 Home       NYY 6
2008-04-08 00:00:00.000 Away       BOS 0
2008-04-09 00:00:00.000 Home       NYY 2
2008-04-09 00:00:00.000 Away       BOS 6
2008-04-10 00:00:00.000 Home       NYY 1
2008-04-10 00:00:00.000 Away       BOS 10

(12 row(s) affected)

This is kind of a combination of using UNPIVOT and a CROSS JOIN; the UNPIVOT doubles-up the results returned just as a CROSS JOIN would, and we just use CASE to return the column that we need depending if the current row is a "HomeTeam" or an "AwayTeam" row. This is a fairly good solution and is probably better than using multiple UNPIVOT clauses.

Summary

UNPIVOT is a really handy operator, but it can be tricky to use. You can UNPIVOT more than 1 column at a time, but beware of how it works -- each UNPIVOT uses it's own "pivot values" and you may get many more results than you intend. Thus, you may need to add criteria to your SELECT to ensure that only the results you are looking for are returned. Alternatively, you can use one UNPIVOT along with CASE expressions to achieve the same results.

UNPIVOT is great, but the main thing is that you are at the mercy of the column names; the naming conventions of the columns used becomes data, and that can really complicate things. In general, we want to keep our object names and our data seperate, so keep that limitation in mind if you wish to employ UNPIVOT in your code. In some cases, an old-school UNION ALL or CROSS JOIN may be a preferrable solution.

Rewriting correlated sub-queries with CASE expressions

Jeff Smith — Wed, 09 Jan 2008 13:42:16 GMT

Here's a very common situation that is very easy to optimize and simplify, submitted via the mailbag:

Nate writes:

Hey, I have a read a bunch of your stuff on your blog and you seem to
be right on the money. I thought maybe you would be able to point me in
the right direction and possibly address this issue on your blog so
others could benefit from your understanding.

I have been searching for the best way to do what I think should be a
simple task in SQL. I have a table full of call history events and I
want to get a summary of some events for each calling party that matches
the form "username(extension)". Using correlated subqueries I do the
following.

SELECT calling_party,

(SELECT SUM(end_time-start_time) AS total_time FROM `event` ei WHERE
event_type=4 AND ei.calling_party = eo.calling_party) AS
total_talking_time,

(SELECT SUM(end_time-start_time) AS total_time FROM `event` ei WHERE
event_type=7 AND ei.calling_party = eo.calling_party) AS
total_ringing_time,

(SELECT MAX(end_time-start_time) AS total_time FROM `event` ei WHERE
event_type=4 AND ei.calling_party = eo.calling_party) AS max_talking_time

FROM `event` eo WHERE calling_party LIKE '%(%)'

GROUP BY calling_party

This works, but ends up taking a really long time to run. I figured
that each subquery was getting executed multiple times. I made a quick php
script that returns the same information but uses multiple queries. I
just query all of the "groups" (SELECT DISTINCT calling_party FROM
`event` WHERE calling_party LIKE '%(%)') and then iterate over that
recordset plugging the actual calling_party value into the where clause of the
aggregate query. The output is identical, but this script runs in well
under a second.

So the question is what is the correct way to do this using SQL? Any
help would be really appreciated.

Hi Nate --

The script as written is making 4 different calls to the calling_party table, and we can reduce this to one call to the table by using CASE expressions to only aggregate the data we need for each column. This is pretty much the standard "static cross-tab" technique to summarize data into multiple columns in a single SELECT.

So, all we need to do is re-write your SELECT like this:

select
calling_party,
sum(case when event_type=4 then end_time-start_time else 0 end) as total_talking_time,
sum(case when event_type=7 then end_time-start_time else 0 end) as total_ringing_time,
max(case when event_type=4 then end_time-start_time else 0 end) as max_talking_time
from
event
where
calling_party LIKE '%(%)'
group by
calling_party

That will return the same results as what you had written, only it should be much more efficient. It is also much shorter. Notice that the CASE expressions ensure that only the data with the specified event_type is including in each aggregate calculation, otherwise we aggregate a value of 0 which has no affect on the results. You can also use a default value of NULL instead of 0 if you'd like, depending on your desired results.

If you have more data in that table for event_types other than 4 and 7, and you only want to return calling_party rows that contain data with those event_types, then be sure to add

and event_type in (4,7)

to your WHERE clause, which will increase the efficiency even further. However, adding that will return different results from your original SELECT (it returns all calling_groups regardless of event_type) so be sure that what you do meets the specifications you require.

As always, be sure that you have proper indexes on all of your tables, and in this case it seems that event_type should certainly be indexed.

see also:

SQL 2005 PIVOT Operator (link)

Jeff Smith — Thu, 12 Jul 2007 13:24:38 GMT

There's two interesting posts over at the MSDN blogs from Craig Freedman about the new PIVOT operator in SQL 2005. First, he gives a nice overview of the operator and how to use it, and then he follows it up with a performance analysis as well. A good read, check it out.

I hope to cover both the PIVOT and also the UNPIVOT operators a little myself at some point in a future post.

see also:

Sorting Columns with the C# Pivot Function

Jeff Smith — Fri, 20 Apr 2007 16:05:42 GMT

Time for another exciting edition of the mailbag!

Maxime writes:

Hi,

First of all, your class is really nice and it handles my problem really well.

I would like to ask you something about the class. I would like to order the pivot columns by column name but I'm unable to do that yet. Do you think you could help me with that ?

Thank you in advance for your time and sorry to bother you with this question but I've tried to post in forums but didn't get any results.

Maxime

(Note: Maxime is referring to my simple Pivot function discussed here. The idea is, as always, to let SQL focus on returning raw, summarized and filtered data in a standard row/column result, and to let your very capable and efficient presentation layer handle all formatting -- which also happens to include pivoting data from rows to columns.)

Sure thing, Maxime, that's a reasonable request and a great idea for improving the code.

If you are using .NET 2.0, you can easily re-arrange columns in a DataTable, but this is not so easy in 1.1. By using the setOrdinal method of a DataColumn, we can ensure that we are creating those new columns in the proper place.

All you need to do is take the existing code for the Pivot function shown here and change just this line:

if (!tmp.Columns.Contains(s))

tmp.Columns.Add(s, dataValues.GetFieldType(pValIndex));

To:

if (!tmp.Columns.Contains(s))
{
DataColumn c = tmp.Columns.Add(s, dataValues.GetFieldType(pValIndex));
// set the index so that it is sorted properly:
int newOrdinal = c.Ordinal;
for (i = newOrdinal - 1; i >= dataValues.FieldCount - 2; i--)
if (c.ColumnName.CompareTo(tmp.Columns[i].ColumnName) < 0)
newOrdinal = i;
c.SetOrdinal(newOrdinal);
}

That should result in "sorted" pivot columns. Use similar tricks if necessary to move other columns around in any DataTable as needed. To sort the columns in descending order, change the CompareTo() method to compare with > 0. In addition, keep in mind that this is a regular text comparison; if you need to sort numeric or date column headers, you should first convert those strings to proper datatypes before doing the compare, or at the very least to a string format that compares and sorts you way you'd like.

If you are using .NET 1.1, your only options are to a) move the data from this DataTable to another one, with columns that are properly sorted as the new table is created, or b) gather the column names in advance and pass those to the function and create the DataTable with those columns already added and ready to be filled in the proper order.

I hope this helps, and congratulations on writing simple, clear, short and efficient SQL code while letting your presentation layer do its job!

see also:

ASP and ADO Pivots -- Old School CrossTabs

Jeff Smith — Mon, 16 May 2005 01:26:00 GMT

OK, so let's sum up my recent flury of crosstab / pivoting posts the past week or so:

Jumping back a bit, I thought I'd also post a bit of code for those still using good old ASP and/or ADO. This is a function writen in VBScript that works much in the same way as the GetRows() method of an ADO recordset -- it returns the entire contents of the recordset in a 2-dimensional array. The difference with this one, of course, is that it will also do a pivot for you.

An entire sample ASP page with the function embedded in it for you to enjoy can be found here. This example, like the others, uses the Northwind database and requires that you have created the view defined here.

Here are the parameters and expected data types of this “old school” Pivot function:

rsData -- (ADODB.Recordset) This is the ADO recordset which contains the data to be pivoted. As before, it must be fully grouped, summarized and sorted properly.
strRow - (String) This is the name of the field that can determines the Rows in the pivot table
strColumn -- (String) The name of the field that will be pivoted into columns
strValue -- (String) The the name of the field that contains the summarized data to be pivoted into columns.
rsColumnVals -- (ADODB.Recordset) This is a new one -- pass in a recordset containing a single column of vals for which to create the pivoted columns.

The last parameter is a new one, and it makes things as efficient as possible to get the list of column values ahead of time. This is similiar to what the dynamic stored procedure method does. Typically, if you are pivoting on field “XYZ”, then you will pass in a recordset that is defined along the lines of “SELECT DISTINCT XYZ FROM ...” for the rsColumnVals parameter. Another thing to note is that the first row of values returned by this function will contain the column headers for the result, which is necessary since those column headers are unknown at runtime unlike when using GetRows(). See the comments and the example for more details.

As before, this method, even when compared to the ultra-speedy GetRows() method in ADO, is clearer, faster, and more efficient than the dynamic sql technique. The main benefit, of course, is that you can use standard SQL with normal grouping and summarizing and you are just returning the data from the database, and not worrying about formatting the results into columns for display purposes.

Feel free to use or modify my code as needed; it is not designed to be a complete ready-to-use libary, but rather an example of ways you can create your own functions at the presentation layer in your applications to do things that the database layer should not have to worry about.

.NET CrossTabs versus SQL Server CrossTabs

Jeff Smith — Thu, 12 May 2005 14:01:00 GMT

As promised in my last post, here is some performance testing to help you determine the performance benefits (if any) of performing your crosstabs at the presentation or code layer, as opposed to forcing SQL Server to do this.

read more...

.NET CrossTabs: Transforming a Normalized DataReader into a Pivoted DataTable

Jeff Smith — Wed, 11 May 2005 19:49:00 GMT

In my last post, I spoke briefly about how I felt that in general crosstabbing data is something that a presentation layer should do and not the database. Consider the result of a crosstab operation -- the columns returned will vary depending on the data. There is pretty much no further manipulation you can do with that result in T-SQL; in the relational database world, the column names of our database objects should be constants and not continually changing as the data changes. Also, in T-SQL there is no easy way to dynamically pivot data, and even doing it with hard-coded pivot values results in long, complicated SELECT statements. So, in my opinion there is really is no benefit of doing this transformation within SQL Server. (Unless you are in the mood for some fun dynamic sql and cursors ...)

Almost all modern report designers allow you to quickly and easily crosstab a standard recordset from SQL Server, so be sure to learn how to use this feature and take advantage of it. Some are more flexible than others, but for the most part they work as you'd expect. Either way, for report-writing crosstabing data at the presentation layer should not be a concern.

The main issue is programmers who are writing their own presentation layer. Many programmers have difficulty writing code to do the transformation; I believe this may be why it is common to be tempted to force SQL Server to do all of the work.

Here's a quick example of an easy way to implement a crosstab at the presentation layer using C#. To keep things short and simple, I have just written a small static function that accepts any object that implements IDataReader and returns a crosstabbed DataTable object. The code for the function is here.

All you need to do is open up an IDataReader object that returns the data you wish to Pivot. The data should be fully summarized by SQL Server already -- i.e., you should GROUP BY your pivot column plus any other columns you wish to return, and you should aggregate your pivot value accordingly. The DataReader should also be sorted so that all rows which will be pivoted into one are sorted together.

For example, consider the Orders table in the Northwind database. Suppose you want to display CompanyNames as rows, ProductNames as columns, and the total quantity of orders per CompanyName per ProductName in your crosstab. The DataReader's recordset should be grouped by CompanyName and ProductName, it should include a SUM(Qty) calculation, and it should be sorted by CompanyName. (see the example for this exact scenario using Northwind).

What's nice is, this is all standard SQL and requires very little effort in terms of writing the SELECT and for the server to process and return the results. We are letting the presentation layer handle the formatting. Another benefit is that we can use this technique with any datasource that ADO.NET can connect to, not just SQL Server. (In fact, we can do this with anything that implements IDataReader)

The arguments for the Pivot() function are as follows:

public static DataTable Pivot (IDataReader dataValues, string keyColumn, string pivotNameColumn, string pivotValueColumn)

dataValues -- this is any open DataReader object, ready to be transformed and pivoted into a DataTable. As mentioned, it should be fully grouped, aggregated, sorted and ready to go.
keyColumn -- This is the column in the DataReader which serves to identify each row. In the previous example, this would be CustomerID. Your DataReader's recordset should be grouped and sorted by this column as well.
pivotNameColumn -- This is the column in the DataReader that contains the values you'd like to transform from rows into columns. In the example, this would be ProductName.
pivotValueColumn -- This is the column that in the DataReader that contains the values to pivot into the appropriate columns. For our example, it would be Qty, which has been defined in the SELECT statement as SUM(Qty).

And, that's it. You just call the Pivot() function on any object that implements IDataReader, specify the pivot information, and a DataTable is returned that you can use however you like. The example uses this to fill up a DataGrid so you can see the results on a form.

Stay tuned for a performance comparison between this method and doing the crosstab transformation on the SQL Server .... You might be surprised by the results.

Another Dynamic SQL CrossTab Stored Procedure

Jeff Smith — Mon, 02 May 2005 18:00:00 GMT

First off, before going any further make sure you have read the hall of fame SQLTeam article by Rob Volk on generating crosstab results using a flexible, dynamic stored procedure that has been viewed over 100,000 times!

read more...