SQL Server 2005

UNPIVOT: Normalizing data on the fly

Jeff Smith — Wed, 23 Apr 2008 14:33:15 GMT

Everyone seems to want to "pivot" or "cross tab" data, but knowing how to do the opposite is equally important. In fact, I would argue that the skill of "unpivoting" data is more useful and more important and more relevant to a SQL programmer, since pivoting results in denormalized data, while unpivoting can transform non-normalized data into a normalized result set. We all know that there's lots of bad databases designs out there, so this can be a handy technique to know.

Of course, even a well designed, fully normalized database can still benefit from "unpivoting" from time to time, so let's take a look at some common situations and some of the options we have to handle this at our disposal. We will focus on some traditional SQL techniques to do this, and then take a close look at the UNPIVOT operator that was introduced with SQL Server 2005.

Example #1: A Bad database design

Let's start with a commonly bad table design, in which someone has decided to relate a client to multiple contacts by designing their client table like this:

create table Clients
(
    clientID int primary key,
    clientName varchar(100),
    contact1 int,
    contact2 int,
    contact3 int,
    contact4 int
)

insert into Clients
select 1,'ABC Corp',1,34,2,null union all
select 2,'DEF Foundation',6,2,8,9 union all
select 3,'GHI Inc.',5,9,null,null union all
select 4,'XYZ Industries',24,null,6,null

clientID    clientName           contact1    contact2    contact3    contact4
----------- -------------------- ----------- ----------- ----------- -----------
1           ABC Corp             1           34          2           NULL
2           DEF Foundation       6           2           8           9
3           GHI Inc.             5           9           NULL        NULL
4           XYZ Industries       24          NULL        6           NULL

(4 row(s) affected)

(Note: For brevity, I am not including the contact table here, nor the foreign key constraints. Of course, with this table design, it would probably be pretty unlikely to find such constraints in the database anyway)

With this design, it is not very easy or efficient to get a count of all contacts for each client, or to find out which contacts are related to which clients. One thing we can do, however is to "unpivot" this table in a query that returns 1 row per ClientID/ContactID combination. With that result set, we can easily now reference the table as if it were normalized and we can get the information we need.

One way to do is to use UNION ALL to return each row in the clients table 4 times, and each time return a different contactID column:

select clientID, contact1 as ContactID
from clients
where contact1 is not null
union all
select clientID, contact2 as ContactID
from clients
where contact2 is not null
union all
select clientID, contact3 as ContactID
from clients
where contact3 is not null
union all
select clientID, contact4 as ContactID
from clients
where contact4 is not null

clientID    ContactID
----------- -----------
1           1
2           6
3           5
4           24
1           34
2           2
3           9
1           2
2           8
4           6
2           9

(11 row(s) affected)

Another option is to CROSS JOIN the Clients table with a table or resultset that returns 4 rows, which also effectively returns each row in the clients table 4 times. For each of the 4 values in the table we are cross joining, we grab a different contact column:

select *
from
(
    select c.clientID,
       case n.n when 1 then c.contact1
            when 2 then c.contact2
            when 3 then c.contact3
            when 4 then c.contact4 end as ContactID
    from
        clients c
    cross join
        (select 1 as n union all select 2 union all select 3 union all select 4) n
)
    x
where
    x.ContactID is not null

clientID    ContactID
----------- -----------
1           1
1           34
1           2
2           6
2           2
2           8
2           9
3           5
3           9
4           24
4           6

(11 row(s) affected)

(Note that you can use a permanent table of Numbers in your database instead of generating it on the fly with a UNION, as shown)

Finally, however, there is an even eaiser way to handle this: the UNPIVOT operator, new with SQL 2005. UNPIVOT works very efficiently and really allows you to handle this exact situation quite easily:

select clientID, Contact.ContactID
from clients
unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

clientID    ContactID
----------- -----------
1           1
1           34
1           2
2           6
2           2
2           8
2           9
3           5
3           9
4           24
4           6

(11 row(s) affected)

Much shorter to write, and more efficient to execute as well.

Taking a Closer Look at UNPIVOT

The UNPIVOT operator is tricky to get a feel for, however, so let's take a look at it.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

First, the "As Contact" at the end is just labeling the entire unpivot result set with an alias, just as you must alias a derived table. Each column returned by the pivot operator can be referenced by the alias if necessary.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

The "ContactID for" part says that we want to return a column called "ContactID" for each unpivoted row. The IN() list is the columns that we are unpivoting; the values in the 4 columns listed here will be assigned to the ContactID column in the result. So, the first time a particular row is unpivoted, the value of the 'contact1' column is assigned to ContactID, the next time it is the 'contact2' column, then 'contact3', and then finally 'contact4'. Then, the next row is processed and it all begins again.

Thus, because we are unpivoting 4 values, we know that the result of the unpivot will have 4 times as many rows as the source data.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

UNPIVOT returns an additional column as well, which contains the name of the column that was used to produce each unpivoted row. Here, we have specified that to be called ContactNumber. Note that we actually did not return ContactNumber in our example, be we can easily add that in so you can see how it works:

select clientID, Contact.ContactNumber, Contact.ContactID
from clients
unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

clientID    ContactNumber           ContactID
----------- ----------------------- -----------
1           contact1                1
1           contact2                34
1           contact3                2
2           contact1                6
2           contact2                2
2           contact3                8
2           contact4                9
3           contact1                5
3           contact2                9
4           contact1                24
4           contact3                6

(11 row(s) affected)

So, you can see that the code to write is very short, but a little difficult to grasp at first. In the end, though, we are able to take a bad table design and easily "fix it", at least temporarily, so that we can query it using simple and standard SQL statements to get what we need.

Example #2: Normalizing a Transaction Table

Here's another common example:

create table Transactions
(
    TranDate datetime,
    Account varchar(10),
    BudgetAmount money,
    ActualAmount money,
    ProjectionAmount money,
    primary key (TranDate, Account)
)
go
insert into Transactions
select '2008-01-01','0001',354,65,58 union all
select '2008-01-02','0001',14,65,34 union all
select '2008-01-03','0001',0,65,622 union all
select '2008-01-04','0001',9,32,84
go

TranDate                Account    BudgetAmount          ActualAmount          ProjectionAmount
----------------------- ---------- --------------------- --------------------- ---------------------
2008-01-01 00:00:00.000 0001       354.00                65.00                 58.00
2008-01-02 00:00:00.000 0001       14.00                 65.00                 34.00
2008-01-03 00:00:00.000 0001       0.00                  65.00                 622.00
2008-01-04 00:00:00.000 0001       9.00                  32.00                 84.00

(4 row(s) affected)

Notice that we have different columns for Budget, Actual and Projection, which is not really a great database design. Much better would be to break this data out so that we have a single 'Amount' column and a 'TransactionType' column that specifies the type of each transaction. We can transform our Transactions tables into this format using UNION ALL:

select TranDate, Account, 'BudgetAmount' as Type, BudgetAmount as Amount from transactions
union all
select TranDate, Account, 'ActualAmount' as Type, ActualAmount as Amount from transactions
union all
select TranDate, Account, 'ProjectionAmount' as Type, ProjectionAmount as Amount from transactions

TranDate                Account    Type             Amount
----------------------- ---------- ---------------- ---------------------
2008-01-01 00:00:00.000 0001       BudgetAmount     354.00
2008-01-02 00:00:00.000 0001       BudgetAmount     14.00
2008-01-03 00:00:00.000 0001       BudgetAmount     0.00
2008-01-04 00:00:00.000 0001       BudgetAmount     9.00
2008-01-01 00:00:00.000 0001       ActualAmount     65.00
2008-01-02 00:00:00.000 0001       ActualAmount     65.00
2008-01-03 00:00:00.000 0001       ActualAmount     65.00
2008-01-04 00:00:00.000 0001       ActualAmount     32.00
2008-01-01 00:00:00.000 0001       ProjectionAmount 58.00
2008-01-02 00:00:00.000 0001       ProjectionAmount 34.00
2008-01-03 00:00:00.000 0001       ProjectionAmount 622.00
2008-01-04 00:00:00.000 0001       ProjectionAmount 84.00

(12 row(s) affected)

Or, we can use the UNPIVOT operator to do the same much easier:

select TranDate, Account, Type, Amount
from Transactions
unpivot (Amount for Type in (BudgetAmount, ActualAmount, ProjectionAmount)) as Amount

TranDate                Account   Type                     Amount
----------------------- --------- ------------------------ ---------------------
2008-01-01 00:00:00.000 0001      BudgetAmount             354.00
2008-01-01 00:00:00.000 0001      ActualAmount             65.00
2008-01-01 00:00:00.000 0001      ProjectionAmount         58.00
2008-01-02 00:00:00.000 0001      BudgetAmount             14.00
2008-01-02 00:00:00.000 0001      ActualAmount             65.00
2008-01-02 00:00:00.000 0001      ProjectionAmount         34.00
2008-01-03 00:00:00.000 0001      BudgetAmount             0.00
2008-01-03 00:00:00.000 0001      ActualAmount             65.00
2008-01-03 00:00:00.000 0001      ProjectionAmount         622.00
2008-01-04 00:00:00.000 0001      BudgetAmount             9.00
2008-01-04 00:00:00.000 0001      ActualAmount             32.00
2008-01-04 00:00:00.000 0001      ProjectionAmount         84.00

(12 row(s) affected)

Example #3: "Unsummarizing" Data

For those who work with accounting systems, this example may be familiar to you. Many times, Accounting systems have "summary" tables that roll up transactional data into a structure like this:

create table AccountBalances
(
    CompanyID int,
    AccountID int,
    TransactionTypeID int,
    Year int,
    Period1 money,
    Period2 money,
    Period3 money,
    Period4 money,
    Period5 money,
    Period6 money,
    Period7 money,
    Period8 money,
    Period9 money,
    Period10 money,
    Period11 money,
    Period12 money
)

insert into AccountBalances
select 1,1,1,2008,200,300,400,500,400,0,0,0,0,0,0,0 union all
select 1,2,1,2008,100,100,100,100,100,100,100,0,0,0,0,0 union all
select 1,3,1,2008,150,0,50,10,10,200,400,45,0,0,0,0

(As before, let's not worry about those foreign key constraints)

These tables are often calculated when transactions are posted, or periods are closed. Typically, many reports pull from these tables because it is much more efficient than summarizing thousands or millions of transactions, and the data is already "cross-tabbed" the way most reporting tools would like to display it.

We can take this summarized data and "unpivot" it so we can still access the summarized data, but now it will be in a normalized structure. All it takes is a simple UNPIVOT like this:

select CompanyID, AccountID, TransactionTypeID, Year, substring(Period,7,2) as PeriodNo, Amount
from AccountBalances
unpivot (Amount for Period in (Period1,Period2,Period3,Period4,Period5,Period6,Period7,Period8,Period9,Period10,Period11,Period12)) as Amount

CompanyID   AccountID   TransactionTypeID Year        PeriodNo Amount
----------- ----------- ----------------- ----------- -------- ---------------------
1           1           1                 2008        1        200.00
1           1           1                 2008        2        300.00
1           1           1                 2008        3        400.00
1           1           1                 2008        4        500.00
1           1           1                 2008        5        400.00
1           1           1                 2008        6        0.00
1           1           1                 2008        7        0.00
1           1           1                 2008        8        0.00
1           1           1                 2008        9        0.00
1           1           1                 2008        10       0.00
1           1           1                 2008        11       0.00
1           1           1                 2008        12       0.00
1           2           1                 2008        1        100.00
1           2           1                 2008        2        100.00
1           2           1                 2008        3        100.00
1           2           1                 2008        4        100.00
1           2           1                 2008        5        100.00
1           2           1                 2008        6        100.00
1           2           1                 2008        7        100.00
1           2           1                 2008        8        0.00
1           2           1                 2008        9        0.00
1           2           1                 2008        10       0.00
1           2           1                 2008        11       0.00
1           2           1                 2008        12       0.00
1           3           1                 2008        1        150.00
1           3           1                 2008        2        0.00
1           3           1                 2008        3        50.00
1           3           1                 2008        4        10.00
1           3           1                 2008        5        10.00
1           3           1                 2008        6        200.00
1           3           1                 2008        7        400.00
1           3           1                 2008        8        45.00
1           3           1                 2008        9        0.00
1           3           1                 2008        10       0.00
1           3           1                 2008        11       0.00
1           3           1                 2008        12       0.00

(36 row(s) affected)

We can filter so that only periods with a non-zero amount are included, and we can SELECT FROM this result set and get the exact data we need for whatever date range we want without worrying which column the actual data is in.

This can also be done with a CROSS JOIN or a UNION ALL, but with 12 values to pivot, those options become much longer to write and UNPIVOT appears to be the way to go in this case.

Example #4: Multiple unpivots

Finally, let's consider a more complicated example. Here, we have a table that stores games played between two teams, where one team is the HomeTeam and the other is the AwayTeam:

create table Teams
(
    TeamCode char(3) primary key not null,
    TeamName varchar(100) not null
)

create table Games
(
    GameDate datetime,
    HomeTeam char(3) references Teams(TeamCode),
    AwayTeam char(3) references Teams(TeamCode),
    HomeScore int,
    AwayScore int,
    primary key (GameDate, HomeTeam),
    constraint pk2 unique (GameDate, AwayTeam),
    check (HomeTeam <> AwayTeam)
)

insert into Teams
select 'BOS','Boston Red Sox' union all
select 'NYY','New York Yankees'

insert into Games
select '2008-04-01','BOS','NYY',3,1 union all
select '2008-04-02','BOS','NYY',6,4 union all
select '2008-04-03','BOS','NYY',2,3 union all
select '2008-04-08','NYY','BOS',6,0 union all
select '2008-04-09','NYY','BOS',2,6 union all
select '2008-04-10','NYY','BOS',1,10

GameDate                HomeTeam AwayTeam HomeScore   AwayScore
----------------------- -------- -------- ----------- -----------
2008-04-01 00:00:00.000 BOS      NYY      3           1
2008-04-02 00:00:00.000 BOS      NYY      6           4
2008-04-03 00:00:00.000 BOS      NYY      2           3
2008-04-08 00:00:00.000 NYY      BOS      6           0
2008-04-09 00:00:00.000 NYY      BOS      2           6
2008-04-10 00:00:00.000 NYY      BOS      1           10

(6 row(s) affected)

That may or may not be the best possible design for the Games table, but it is a common way to do it. With data in that form, if we want to get the total runs scored per team across all games, or get each teams won-loss record, we need to make two passes through the table. This can be done fairly easily and efficiently with a union:

select GameDate,
    HomeTeam as TeamCode,
    'Home' as HomeOrAway,
    HomeScore as Score,
    case when HomeScore > AwayScore then 1 else 0 end as Win,
    case when HomeScore < AwayScore then 1 else 0 end as Loss,
    case when HomeScore = AwayScore then 1 else 0 end as Tie
from
    Games
union all
select GameDate,
    AwayTeam as TeamCode,
    'Away' as HomeOrAway,
    AwayScore as Score,
    case when HomeScore < AwayScore then 1 else 0 end as Win,
    case when HomeScore > AwayScore then 1 else 0 end as Loss,
    case when HomeScore = AwayScore then 1 else 0 end as Tie
from
    Games

GameDate                TeamCode HomeOrAway Score       Win         Loss        Tie
----------------------- -------- ---------- ----------- ----------- ----------- -----------
2008-04-01 00:00:00.000 BOS      Home       3           1           0           0
2008-04-02 00:00:00.000 BOS      Home       6           1           0           0
2008-04-03 00:00:00.000 BOS      Home       2           0           1           0
2008-04-08 00:00:00.000 NYY      Home       6           1           0           0
2008-04-09 00:00:00.000 NYY      Home       2           0           1           0
2008-04-10 00:00:00.000 NYY      Home       1           0           1           0
2008-04-01 00:00:00.000 NYY      Away       1           0           1           0
2008-04-02 00:00:00.000 NYY      Away       4           0           1           0
2008-04-03 00:00:00.000 NYY      Away       3           1           0           0
2008-04-08 00:00:00.000 BOS      Away       0           0           1           0
2008-04-09 00:00:00.000 BOS      Away       6           1           0           0
2008-04-10 00:00:00.000 BOS      Away       10          1           0           0

(12 row(s) affected)

Now, how can we do this with UNPIVOT? This example is a bit more complicated, because we are "unpivoting" not only the score, but also the TeamCode, and we are also calculating a few extra columns (Win, Loss and Tie).

So, can we UNPIVOT more than 1 column? Let's start simply and do things one at a time. First, let's UNPIVOT just the TeamCode:

select GameDate, HomeOrAway, Team
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team

GameDate                HomeOrAway     Team
----------------------- -------------- ----
2008-04-01 00:00:00.000 HomeTeam       BOS
2008-04-01 00:00:00.000 AwayTeam       NYY
2008-04-02 00:00:00.000 HomeTeam       BOS
2008-04-02 00:00:00.000 AwayTeam       NYY
2008-04-03 00:00:00.000 HomeTeam       BOS
2008-04-03 00:00:00.000 AwayTeam       NYY
2008-04-08 00:00:00.000 HomeTeam       NYY
2008-04-08 00:00:00.000 AwayTeam       BOS
2008-04-09 00:00:00.000 HomeTeam       NYY
2008-04-09 00:00:00.000 AwayTeam       BOS
2008-04-10 00:00:00.000 HomeTeam       NYY
2008-04-10 00:00:00.000 AwayTeam       BOS

(12 row(s) affected)

OK, so far so good. Now, how do we get the score for the team as well? Let's add another UNPIVOT clause to the SELECT, this time for Score:

select GameDate, HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway in (HomeScore, AwayScore)) as Score

Msg 265, Level 16, State 1, Line 1
The column name "HomeOrAway" specified in the UNPIVOT operator conflicts with the existing column name in the UNPIVOT argument.
Msg 8156, Level 16, State 1, Line 1
The column 'HomeOrAway' was specified multiple times for 'Score'.

Hmmm. OK, we cannot specify the same column for both pivots. That is just an alias specification, no big deal, so let's just alias it as "HomeOrAway2" :

select GameDate, HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   BOS 3
2008-04-01 00:00:00.000 HomeTeam   BOS 1
2008-04-01 00:00:00.000 AwayTeam   NYY 3
2008-04-01 00:00:00.000 AwayTeam   NYY 1
2008-04-02 00:00:00.000 HomeTeam   BOS 6
2008-04-02 00:00:00.000 HomeTeam   BOS 4
2008-04-02 00:00:00.000 AwayTeam   NYY 6
2008-04-02 00:00:00.000 AwayTeam   NYY 4
2008-04-03 00:00:00.000 HomeTeam   BOS 2
2008-04-03 00:00:00.000 HomeTeam   BOS 3
2008-04-03 00:00:00.000 AwayTeam   NYY 2
2008-04-03 00:00:00.000 AwayTeam   NYY 3
2008-04-08 00:00:00.000 HomeTeam   NYY 6
2008-04-08 00:00:00.000 HomeTeam   NYY 0
2008-04-08 00:00:00.000 AwayTeam   BOS 6
2008-04-08 00:00:00.000 AwayTeam   BOS 0
2008-04-09 00:00:00.000 HomeTeam   NYY 2
2008-04-09 00:00:00.000 HomeTeam   NYY 6
2008-04-09 00:00:00.000 AwayTeam   BOS 2
2008-04-09 00:00:00.000 AwayTeam   BOS 6
2008-04-10 00:00:00.000 HomeTeam   NYY 1
2008-04-10 00:00:00.000 HomeTeam   NYY 10
2008-04-10 00:00:00.000 AwayTeam   BOS 1
2008-04-10 00:00:00.000 AwayTeam   BOS 10

(24 row(s) affected)

Holy schnikies, it works! We can specify more than one UNPIVOT clause for the same SQL statement! Who would have thunk it?

Uh oh -- wait a second. We have 24 results returned. We should have only 12. Something is not right here. Remember when we aliased our second unpivot column as "HomeOrAway2"? We did not return that anywhere in our results. Let's add that in and take a look:

select GameDate, HomeOrAway, HomeOrAway2, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score

GameDate                HomeOrAway HomeOrAway2 Team Score
----------------------- ---------- ----------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   HomeScore   BOS 3
2008-04-01 00:00:00.000 HomeTeam   AwayScore   BOS 1
2008-04-01 00:00:00.000 AwayTeam   HomeScore   NYY 3
2008-04-01 00:00:00.000 AwayTeam   AwayScore   NYY 1
2008-04-02 00:00:00.000 HomeTeam   HomeScore   BOS 6
2008-04-02 00:00:00.000 HomeTeam   AwayScore   BOS 4
2008-04-02 00:00:00.000 AwayTeam   HomeScore   NYY 6
2008-04-02 00:00:00.000 AwayTeam   AwayScore   NYY 4
2008-04-03 00:00:00.000 HomeTeam   HomeScore   BOS 2
2008-04-03 00:00:00.000 HomeTeam   AwayScore   BOS 3
2008-04-03 00:00:00.000 AwayTeam   HomeScore   NYY 2
2008-04-03 00:00:00.000 AwayTeam   AwayScore   NYY 3
2008-04-08 00:00:00.000 HomeTeam   HomeScore   NYY 6
2008-04-08 00:00:00.000 HomeTeam   AwayScore   NYY 0
2008-04-08 00:00:00.000 AwayTeam   HomeScore   BOS 6
2008-04-08 00:00:00.000 AwayTeam   AwayScore   BOS 0
2008-04-09 00:00:00.000 HomeTeam   HomeScore   NYY 2
2008-04-09 00:00:00.000 HomeTeam   AwayScore   NYY 6
2008-04-09 00:00:00.000 AwayTeam   HomeScore   BOS 2
2008-04-09 00:00:00.000 AwayTeam   AwayScore   BOS 6
2008-04-10 00:00:00.000 HomeTeam   HomeScore   NYY 1
2008-04-10 00:00:00.000 HomeTeam   AwayScore   NYY 10
2008-04-10 00:00:00.000 AwayTeam   HomeScore   BOS 1
2008-04-10 00:00:00.000 AwayTeam   AwayScore   BOS 10

(24 row(s) affected)

Notice what is happening here. Without an UNPIVOT, we had 6 rows. We added one UNPIVOT to pivot two values, and got 12 rows. Then, we added another pivot, again with two values, and got back 24. Looking at the results, we can see that the single game at '2008-04-01' was unpivoted into 4 rows. So, while we can unpivot multiple times, there is no relation between each UNPIVOT. That is, notice there is no correlation between HomeOrAway and HomeOrAway2. Let's look at just that first game:

GameDate                HomeOrAway HomeOrAway2 Team Score
----------------------- ---------- ----------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   HomeScore   BOS 3
2008-04-01 00:00:00.000 HomeTeam   AwayScore   BOS 1
2008-04-01 00:00:00.000 AwayTeam   HomeScore   NYY 3
2008-04-01 00:00:00.000 AwayTeam   AwayScore   NYY 1

Both scores, home and away, are specified for each team -- even if it is not their matching score. What we really want is only these two lines:

How can we get there? Well, there's always a WHERE clause! We can simply filter our result so that the Left 4 characters of HomeOrAway matches the left 4 characters of HomeOrAway2. After all, there is nothing magical about these unpivoted columns, they are data just like anything else.

So, we can now write:

select GameDate, left(HomeOrAway,4) as HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score
where left(HomeOrAway,4) = left(HomeOrAway2,4)

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 Home       BOS 3
2008-04-01 00:00:00.000 Away       NYY 1
2008-04-02 00:00:00.000 Home       BOS 6
2008-04-02 00:00:00.000 Away       NYY 4
2008-04-03 00:00:00.000 Home       BOS 2
2008-04-03 00:00:00.000 Away       NYY 3
2008-04-08 00:00:00.000 Home       NYY 6
2008-04-08 00:00:00.000 Away       BOS 0
2008-04-09 00:00:00.000 Home       NYY 2
2008-04-09 00:00:00.000 Away       BOS 6
2008-04-10 00:00:00.000 Home       NYY 1
2008-04-10 00:00:00.000 Away       BOS 10

(12 row(s) affected)

Now we are back to 12 results returned, the Homes and Aways match up, and things look good! Hey, we just pulled off 2 unpivots at once, not bad! From here, we can add our Win, Loss and Tie calculations as well. Note that we also have removed "HomeOrAway2" from the SELECT, since we don't need to see that column in our result. Finally, we are returning only the left 4 characters of HomeTeam or AwayTeam, so that the word 'Home' or 'Away' is returned in the HomeOrAway column.

This seems like quite a bit of work. Is UNPIVOT really the way to go in this scenario? Maybe not. I have not done full performance testing of UNPIVOT versus the other options shown (this article is way too long as it is!) but as always you should fully test your options and decide which offers the best performance and code simplicity.

Another option if you want to unpivot more than 1 column is to just use one UNPIVOT cause, but then use CASE expressions to return the values you need. For example:

select GameDate, left(HomeOrAway,4) as HomeOrAway,
Team,
case when HomeOrAway='HomeTeam' then HomeScore else AwayScore end as Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 Home       BOS 3
2008-04-01 00:00:00.000 Away       NYY 1
2008-04-02 00:00:00.000 Home       BOS 6
2008-04-02 00:00:00.000 Away       NYY 4
2008-04-03 00:00:00.000 Home       BOS 2
2008-04-03 00:00:00.000 Away       NYY 3
2008-04-08 00:00:00.000 Home       NYY 6
2008-04-08 00:00:00.000 Away       BOS 0
2008-04-09 00:00:00.000 Home       NYY 2
2008-04-09 00:00:00.000 Away       BOS 6
2008-04-10 00:00:00.000 Home       NYY 1
2008-04-10 00:00:00.000 Away       BOS 10

(12 row(s) affected)

This is kind of a combination of using UNPIVOT and a CROSS JOIN; the UNPIVOT doubles-up the results returned just as a CROSS JOIN would, and we just use CASE to return the column that we need depending if the current row is a "HomeTeam" or an "AwayTeam" row. This is a fairly good solution and is probably better than using multiple UNPIVOT clauses.

Summary

UNPIVOT is a really handy operator, but it can be tricky to use. You can UNPIVOT more than 1 column at a time, but beware of how it works -- each UNPIVOT uses it's own "pivot values" and you may get many more results than you intend. Thus, you may need to add criteria to your SELECT to ensure that only the results you are looking for are returned. Alternatively, you can use one UNPIVOT along with CASE expressions to achieve the same results.

UNPIVOT is great, but the main thing is that you are at the mercy of the column names; the naming conventions of the columns used becomes data, and that can really complicate things. In general, we want to keep our object names and our data seperate, so keep that limitation in mind if you wish to employ UNPIVOT in your code. In some cases, an old-school UNION ALL or CROSS JOIN may be a preferrable solution.

Top N Percent per Group

Jeff Smith — Thu, 21 Feb 2008 18:05:49 GMT

Here's a good question in the feedback from my post about using the T-SQL 2005 features to return the Top N per Group of a result set:

Sani writes:

What about Top n Percent per Group??? I would greatly appreciate an input on that as well.

That's a good question, and also easily solvable. One way to do this that I thought of was by using a combination of rank() and a count(*) partitioned aggregate function, which is also a new SQL Server 2005 feature.

Simply calculate the rank() of each row in the group, and also the count(*) of all rows in the group. Multiply the count(*) by the percentage of rows you want returned and filter so that your rank() per group is less than that.

Here's an example. Suppose we want to return the newest 10% of all products per region, where the newest product has the latest "AddedDate" column. We can write it like this:

with ProductsByRegion as
(
    select Region, Product, AddedDate,
       rank() over (partition by Region order by AddedDate desc) as AddedRank,
       count(*) over (partition by Region) as RegionProductsCount
from Products
)
select
       Region, Product, AddedDate
from
      ProductsByRegion
where
     AddedRank <= (RegionProductsCount * .10)

That's really all there is to it.

Update: Some great feedback from Geoff N. Hiten in the first comment to this post shows an even easier solution: just use the NTILE() aggregate function. See his comment for an example. I definitely recommend that approach, it is much simpler and certainly more straightforward since that's pretty much what NTILE is designed to do. Thanks Geoff! (So much for my theory that Jeff's who spell their name with a "J" are always smarter than those who spell it with a "G" !)

These partitioned functions are really amazing and so useful, they are hard to live without once you get the hang of them.

Implementing Table Inheritance with SQL Server

Jeff Smith — Thu, 21 Feb 2008 16:05:35 GMT

I have a new article up at SQLTeam:

Implementing Table Inheritance with Sql Server

It discusses the situation where you have multiple entities that are distinct, yet they have many attributes or relations in common. There is an easy way to simplify your database design and your code if you use the concept of a "base table" for that common data, which is very similar to the concept of Inheritance in Object-Oriented programming.

I recently had to do this for a client that tracks Contributions to their foundation; there is a base set of data that all Contributions have, such as who donated, how they paid, the date a tax letter was sent, etc. But for certain contribution types we need additional information, such as if they bought an Auction Item, sponsored an Event, and so on. Thus, we set up a "base table" of Contributions and sub-tables for the different types, and set up a constraint to ensure that the right subtable is used depending on the type.

It sounds complicated and like it may require fancy triggers or at the very least CHECK constraints, but it is very easy to model in T-SQL -- especially in SQL Server 2005+. The end result is that you have a less redundant database design and much less code to write to maintain that data.

Check it out if it sounds interesting, or helpful, or if none of this makes any sense and you're wondering what the heck I'm talking about.

Date Only and Time Only data types in SQL Server 2005 (without the CLR)

Jeff Smith — Wed, 31 Oct 2007 13:13:16 GMT

In this post, I showed a simple way to create simple but useful Date and Time user-defined data types in SQL Server 2000. Here's how to do it in SQL Server 2005, without the need for CLR types or anything fancy.

First, we must create two user defined data types:

create type Date from dateTime
create type Time from dateTime

So, internally (and externally to our clients), these types are really just DateTime. But, we will apply some rules to these types so that the Date data type will always be constrained to a time exactly at midnight, and the the Time data type will always be at the "base date", or 1900-01-01:

create rule DateOnlyRule as
dateAdd(dd,datediff(dd,0,@DateTime),0) = @DateTime

go

create rule TimeOnlyRule as
datediff(dd,0,@DateTime) = 0

go

Finally, we call sp_bindrule to bind the rules to the data types:

EXEC sp_bindrule 'DateOnlyRule', 'Date'
EXEC sp_bindrule 'TimeOnlyRule', 'Time'

That's it! Now, we can create a table that uses our brand-new data types:

create table Trans
(
TranID int identity primary key,
TranAmount money not null,
TranDate Date not null,
TranTime Time not null
)

Notice that TranDate and TranTime are two separate columns here. If we try to insert data, our rules will ensure that our TranDate and TranTime columns only contain the appropriate data, and by doing so we can simply calculate TranDate + TranTime to get the transactions DateTime.

The following inserts will succeed:

insert into Trans (TranAmount, TranDate, TranTime)
values (200, '2005-01-01', '09:00:00 AM')

insert into Trans (TranAmount, TranDate, TranTime)
values (400, '2005-01-03', '7:50:30 PM')

But this will fail, since our rules are violated (getdate() returns the current date and the time):

insert into Trans (TranAmount, TranDate, TranTime)
values (200, getdate(), getdate())

Msg 513, Level 16, State 0, Line 1
A column insert or update conflicts with a rule imposed by a previous CREATE RULE statement. The statement was terminated. The conflict occurred in database 'PlayGround', table 'dbo.Trans', column 'TranDate'.
The statement has been terminated.

Finally, we can select from our table and simply add the two columns together to get each transaction's DateTime:

select *, TranDate + TranTime as TranDateTime
from Trans

TranID TranAmount   TranDate            TranTime            TranDateTime
------- ----------- ------------------- ------------------- -------------------
1       200.00       2005-01-01 00:00:00 1900-01-01 09:00:00 2005-01-01 09:00:00
2       400.00       2005-01-03 00:00:00 1900-01-01 19:50:30 2005-01-03 19:50:30

(2 row(s) affected)

The nice advantage of doing this is that you can quickly and efficiently select all transactions for a single day like this:

select * from Trans
where TranDate = '2005-01-01'

If we stored the Date and Time in the same column, we'd need to use a less efficient range:

select * from Trans
where TranDate >= '2005-01-01' and TranDate < '2005-01-02'

SQL 2008 does finally implement true Date and Time data types, but until then, I hope this gives you some ideas to play with.

Notes:

If you do wish to break apart DateTime data into two columns, another option is to simply use a single, standard DateTime column but then define two extra computed columns in your table, one that returns just the Date, another that returns just the Time. Then, you can index those computed columns and reference them easily and efficiently. If you truly wish to store just a date or just a time, however, these user defined data types are a great way to do it. At the end of this SQLTeam article there's a good example demonstrating this.
You could also just use the expressions for the rules in CHECK constrains on individual table columns, instead of creating the user defined data types and rules. As Books On Line states, CREATE RULE will at some point be removed from future versions of SQL Server, so you may wish to avoid using it.

Taking a look at CROSS APPLY

Jeff Smith — Thu, 18 Oct 2007 21:01:20 GMT

Applying a Sub-Query, Joining a Derived Table ...

I think the easiest way to think of CROSS APPLY is that it is like doing a CROSS JOIN with a correlated sub-query instead of a derived table. Let's see if I can explain that .... A derived table is "self-contained", in that all tables and columns in the parent SELECT are not accessible (though variables and parameters can be referenced). For example, consider:

select A.*, b.X
from A
cross join (select B.X from B where B.Val=A.Val) b

That is not legal because A.Val is out of scope within the derived table; this is because the derived table is evaluated independently of the other tables in the SELECT. To limit the rows in table B so that B.Val = A.Val, we must do that outside of the derived table via a join or in the criteria:

select A.*, b.X
from A
cross join (select * from B) b
where A.Val = b.Val

(Of course, the above is equivalent to doing an INNER JOIN to the derived table, or just joining to the table B.)

Also, keep in mind that the scope-of-derived-tables rule isn't just for CROSS JOINS, it's for all JOINS -- CROSS, INNER, OUTER and even UNION; they all use "self-contained" derived tables.

This is in contrast to a correlated sub-query, where the parent SELECT is in scope for the sub-query; the sub-query is evaluated for each row in the query, so the other tables and columns in the SELECT are all available:

select A.*, (select B.X from B where B.Val=A.Val) as X
from A

(Note: I am ignoring for now the fact that returning multiple rows in a sub-query will return an error.)

This is an easy way to think of the difference between CROSS JOIN and CROSS APPLY. CROSS JOIN, as we saw, joins to a derived table; however, CROSS APPLY, despite looking like a JOIN, actually is applying a correlated sub-query. This imposes both the advantages of a correlated sub-query but also the performance implications.

So, we can simply rewrite our first example using CROSS APPLY like this:

select A.*, b.X
from A
cross apply (select B.X from B where B.Val=A.Val) b

Since we are performing an APPLY and not a JOIN, A.Val is in scope and it works just fine.

Table Valued User Defined Functions

Note that the same rules apply when using Table-Valued User-Defined Functions:

select A.*, B.X
from A
cross join dbo.UDF(A.Val) B

is not legal; once again, A.Val is not in scope for the user-defined function. The best we can do before SQL 2005 was to use a correlated sub-query:

select A.*, (select X from dbo.UDF(A.Val)) X
from A

However, that is not logically equivalent; the UDF cannot return more than 1 row or it will result in an error, and wouldn't make logical sense anyway if it did.

Starting with SQL 2005, we can now use CROSS APPLY and it will work fine:

select A.*, b.X
from A
cross apply dbo.UDF(A.Val) b

So, that is one way to think of the difference between a JOIN and an APPLY; a JOIN combines two separate result sets, but APPLY is more of a loop that evaluates one result set over and over for each row in another. This means that, in general, APPLY will be less efficient than a JOIN, just as, in general, correlated sub-queries are less efficient than derived tables. (The optimizer, however, is generally quite good at optimized correlated sub-queries when possible.)

So, why use CROSS APPLY instead of a correlated sub-query? What's the advantage? Actually, quite a lot -- it is much more powerful!

CROSS APPLY can return multiple rows

Unlike correlate sub-queries, CROSS APPLY works with multiple rows. This allows us to do things like "joining" a table to a function that parses a CSV column in that table into multiple rows:

select A.ID, b.Val
from A
cross apply dbo.ParseCSV(A.CSV) b

When the ParseCSV() function returns multiple rows, it simply acts as if we have joined Table A to the function's return table, duplicating the rows in Table A for each row in the joined table. This is not possible with a correlated sub-query, and will result in an error. This is a quick and easy way to parse a table of data into multiple rows in an efficient "set-based" manner when the algorithm requires a complex User-Defined Function. (The CSV is not a great example, but other parsing routines are not as easily accomplished via a JOIN to a numbers table.)

CROSS APPLY can return multiple columns

Again, in a correlated sub-query, we can only return a single value. If we write a SQL statement that returns a running sum, we can use a correlated sub-query like this:

select o.*,
(select sum(Amount) from Order o
where p.OrderDate <= o.OrderDate) as RunningSum
from Order o

However, what if we'd like to return an additional running sum of Orders based on some other criteria (e.g., for orders with the same "OrderCode")? We'd need another correlated sub-query, greatly reducing the efficiency of our SELECT:

select o.*,
(select sum(Amount) from Order o
where p.OrderDate <= o.OrderDate) as RunningSum,
(select sum(Amount) from Order o
where p.OrderCode = o.OrderCode and p.OrderDate <= o.OrderDate) as SameCode
from Order o

However, we can easily re-write that using a single CROSS APPLY:

select o.*, rs.RunningSum, rs.SameCode
from Order o
cross apply
(
   select
     sum(Amount) as RunningSum,
     sum(case when p.OrderCode = o.OrderCode then Amount else 0 end) as SameCode
   from Order P
   where P.OrderDate <= O.OrderDate
) rs

So, we get the benefit of return multiple columns like a derived table, and we also get the ability to reference outer values in our SELECT to use in criteria and CASE statements. Very, very powerful.

CROSS APPLY also allows us to quickly get columns from the "previous" row in a table quite easily:

select o.*, prev.*
from Order o
cross apply
(
   select top 1 *
   from Order P where P.OrderDate < O.OrderDate
   order by OrderDate DESC
) prev

We can express the running total and counts as CROSS JOINS or INNER JOINS by reworking a correlated sub-query, but using TOP and referencing the outer SELECT is not easily done in that manner.

Note that the above CROSS APPLY will not return any orders without a previous order; we would use OUTER APPLY to ensure that all orders are returned even if no previous orders exist:

select o.*, prev.*
from Order o
outer apply
(
   select top 1 *
   from Order P where P.OrderDate < O.OrderDate
   order by OrderDate DESC
) prev

We can also CROSS APPLY to a Table-Valued User Defined Function that returns exactly one row, but with multiple columns, to return separate pieces of data from a single function call. For example, we can parse an email address into separate username and domain columns. A few months back I wrote a SQL Team article that discusses that concept.

Summary

CROSS and OUTER APPLY are very powerful and can be very useful, but we must be careful to use them only when necessary; I am still testing the possibilities, but in general an APPLY will tend to be not as efficient as a JOIN. In addition, APPLY is mostly demonstrated by applying table-valued user-defined functions, but it can be used with in-line SELECT statements as well.

see also: