Report Writing

UNPIVOT: Normalizing data on the fly

Jeff Smith — Wed, 23 Apr 2008 14:33:15 GMT

Everyone seems to want to "pivot" or "cross tab" data, but knowing how to do the opposite is equally important. In fact, I would argue that the skill of "unpivoting" data is more useful and more important and more relevant to a SQL programmer, since pivoting results in denormalized data, while unpivoting can transform non-normalized data into a normalized result set. We all know that there's lots of bad databases designs out there, so this can be a handy technique to know.

Of course, even a well designed, fully normalized database can still benefit from "unpivoting" from time to time, so let's take a look at some common situations and some of the options we have to handle this at our disposal. We will focus on some traditional SQL techniques to do this, and then take a close look at the UNPIVOT operator that was introduced with SQL Server 2005.

Example #1: A Bad database design

Let's start with a commonly bad table design, in which someone has decided to relate a client to multiple contacts by designing their client table like this:

create table Clients
(
    clientID int primary key,
    clientName varchar(100),
    contact1 int,
    contact2 int,
    contact3 int,
    contact4 int
)

insert into Clients
select 1,'ABC Corp',1,34,2,null union all
select 2,'DEF Foundation',6,2,8,9 union all
select 3,'GHI Inc.',5,9,null,null union all
select 4,'XYZ Industries',24,null,6,null

clientID    clientName           contact1    contact2    contact3    contact4
----------- -------------------- ----------- ----------- ----------- -----------
1           ABC Corp             1           34          2           NULL
2           DEF Foundation       6           2           8           9
3           GHI Inc.             5           9           NULL        NULL
4           XYZ Industries       24          NULL        6           NULL

(4 row(s) affected)

(Note: For brevity, I am not including the contact table here, nor the foreign key constraints. Of course, with this table design, it would probably be pretty unlikely to find such constraints in the database anyway)

With this design, it is not very easy or efficient to get a count of all contacts for each client, or to find out which contacts are related to which clients. One thing we can do, however is to "unpivot" this table in a query that returns 1 row per ClientID/ContactID combination. With that result set, we can easily now reference the table as if it were normalized and we can get the information we need.

One way to do is to use UNION ALL to return each row in the clients table 4 times, and each time return a different contactID column:

select clientID, contact1 as ContactID
from clients
where contact1 is not null
union all
select clientID, contact2 as ContactID
from clients
where contact2 is not null
union all
select clientID, contact3 as ContactID
from clients
where contact3 is not null
union all
select clientID, contact4 as ContactID
from clients
where contact4 is not null

clientID    ContactID
----------- -----------
1           1
2           6
3           5
4           24
1           34
2           2
3           9
1           2
2           8
4           6
2           9

(11 row(s) affected)

Another option is to CROSS JOIN the Clients table with a table or resultset that returns 4 rows, which also effectively returns each row in the clients table 4 times. For each of the 4 values in the table we are cross joining, we grab a different contact column:

select *
from
(
    select c.clientID,
       case n.n when 1 then c.contact1
            when 2 then c.contact2
            when 3 then c.contact3
            when 4 then c.contact4 end as ContactID
    from
        clients c
    cross join
        (select 1 as n union all select 2 union all select 3 union all select 4) n
)
    x
where
    x.ContactID is not null

clientID    ContactID
----------- -----------
1           1
1           34
1           2
2           6
2           2
2           8
2           9
3           5
3           9
4           24
4           6

(11 row(s) affected)

(Note that you can use a permanent table of Numbers in your database instead of generating it on the fly with a UNION, as shown)

Finally, however, there is an even eaiser way to handle this: the UNPIVOT operator, new with SQL 2005. UNPIVOT works very efficiently and really allows you to handle this exact situation quite easily:

select clientID, Contact.ContactID
from clients
unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

clientID    ContactID
----------- -----------
1           1
1           34
1           2
2           6
2           2
2           8
2           9
3           5
3           9
4           24
4           6

(11 row(s) affected)

Much shorter to write, and more efficient to execute as well.

Taking a Closer Look at UNPIVOT

The UNPIVOT operator is tricky to get a feel for, however, so let's take a look at it.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

First, the "As Contact" at the end is just labeling the entire unpivot result set with an alias, just as you must alias a derived table. Each column returned by the pivot operator can be referenced by the alias if necessary.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

The "ContactID for" part says that we want to return a column called "ContactID" for each unpivoted row. The IN() list is the columns that we are unpivoting; the values in the 4 columns listed here will be assigned to the ContactID column in the result. So, the first time a particular row is unpivoted, the value of the 'contact1' column is assigned to ContactID, the next time it is the 'contact2' column, then 'contact3', and then finally 'contact4'. Then, the next row is processed and it all begins again.

Thus, because we are unpivoting 4 values, we know that the result of the unpivot will have 4 times as many rows as the source data.

unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

UNPIVOT returns an additional column as well, which contains the name of the column that was used to produce each unpivoted row. Here, we have specified that to be called ContactNumber. Note that we actually did not return ContactNumber in our example, be we can easily add that in so you can see how it works:

select clientID, Contact.ContactNumber, Contact.ContactID
from clients
unpivot (ContactID for ContactNumber in (contact1, contact2,contact3,contact4)) as Contact

clientID    ContactNumber           ContactID
----------- ----------------------- -----------
1           contact1                1
1           contact2                34
1           contact3                2
2           contact1                6
2           contact2                2
2           contact3                8
2           contact4                9
3           contact1                5
3           contact2                9
4           contact1                24
4           contact3                6

(11 row(s) affected)

So, you can see that the code to write is very short, but a little difficult to grasp at first. In the end, though, we are able to take a bad table design and easily "fix it", at least temporarily, so that we can query it using simple and standard SQL statements to get what we need.

Example #2: Normalizing a Transaction Table

Here's another common example:

create table Transactions
(
    TranDate datetime,
    Account varchar(10),
    BudgetAmount money,
    ActualAmount money,
    ProjectionAmount money,
    primary key (TranDate, Account)
)
go
insert into Transactions
select '2008-01-01','0001',354,65,58 union all
select '2008-01-02','0001',14,65,34 union all
select '2008-01-03','0001',0,65,622 union all
select '2008-01-04','0001',9,32,84
go

TranDate                Account    BudgetAmount          ActualAmount          ProjectionAmount
----------------------- ---------- --------------------- --------------------- ---------------------
2008-01-01 00:00:00.000 0001       354.00                65.00                 58.00
2008-01-02 00:00:00.000 0001       14.00                 65.00                 34.00
2008-01-03 00:00:00.000 0001       0.00                  65.00                 622.00
2008-01-04 00:00:00.000 0001       9.00                  32.00                 84.00

(4 row(s) affected)

Notice that we have different columns for Budget, Actual and Projection, which is not really a great database design. Much better would be to break this data out so that we have a single 'Amount' column and a 'TransactionType' column that specifies the type of each transaction. We can transform our Transactions tables into this format using UNION ALL:

select TranDate, Account, 'BudgetAmount' as Type, BudgetAmount as Amount from transactions
union all
select TranDate, Account, 'ActualAmount' as Type, ActualAmount as Amount from transactions
union all
select TranDate, Account, 'ProjectionAmount' as Type, ProjectionAmount as Amount from transactions

TranDate                Account    Type             Amount
----------------------- ---------- ---------------- ---------------------
2008-01-01 00:00:00.000 0001       BudgetAmount     354.00
2008-01-02 00:00:00.000 0001       BudgetAmount     14.00
2008-01-03 00:00:00.000 0001       BudgetAmount     0.00
2008-01-04 00:00:00.000 0001       BudgetAmount     9.00
2008-01-01 00:00:00.000 0001       ActualAmount     65.00
2008-01-02 00:00:00.000 0001       ActualAmount     65.00
2008-01-03 00:00:00.000 0001       ActualAmount     65.00
2008-01-04 00:00:00.000 0001       ActualAmount     32.00
2008-01-01 00:00:00.000 0001       ProjectionAmount 58.00
2008-01-02 00:00:00.000 0001       ProjectionAmount 34.00
2008-01-03 00:00:00.000 0001       ProjectionAmount 622.00
2008-01-04 00:00:00.000 0001       ProjectionAmount 84.00

(12 row(s) affected)

Or, we can use the UNPIVOT operator to do the same much easier:

select TranDate, Account, Type, Amount
from Transactions
unpivot (Amount for Type in (BudgetAmount, ActualAmount, ProjectionAmount)) as Amount

TranDate                Account   Type                     Amount
----------------------- --------- ------------------------ ---------------------
2008-01-01 00:00:00.000 0001      BudgetAmount             354.00
2008-01-01 00:00:00.000 0001      ActualAmount             65.00
2008-01-01 00:00:00.000 0001      ProjectionAmount         58.00
2008-01-02 00:00:00.000 0001      BudgetAmount             14.00
2008-01-02 00:00:00.000 0001      ActualAmount             65.00
2008-01-02 00:00:00.000 0001      ProjectionAmount         34.00
2008-01-03 00:00:00.000 0001      BudgetAmount             0.00
2008-01-03 00:00:00.000 0001      ActualAmount             65.00
2008-01-03 00:00:00.000 0001      ProjectionAmount         622.00
2008-01-04 00:00:00.000 0001      BudgetAmount             9.00
2008-01-04 00:00:00.000 0001      ActualAmount             32.00
2008-01-04 00:00:00.000 0001      ProjectionAmount         84.00

(12 row(s) affected)

Example #3: "Unsummarizing" Data

For those who work with accounting systems, this example may be familiar to you. Many times, Accounting systems have "summary" tables that roll up transactional data into a structure like this:

create table AccountBalances
(
    CompanyID int,
    AccountID int,
    TransactionTypeID int,
    Year int,
    Period1 money,
    Period2 money,
    Period3 money,
    Period4 money,
    Period5 money,
    Period6 money,
    Period7 money,
    Period8 money,
    Period9 money,
    Period10 money,
    Period11 money,
    Period12 money
)

insert into AccountBalances
select 1,1,1,2008,200,300,400,500,400,0,0,0,0,0,0,0 union all
select 1,2,1,2008,100,100,100,100,100,100,100,0,0,0,0,0 union all
select 1,3,1,2008,150,0,50,10,10,200,400,45,0,0,0,0

(As before, let's not worry about those foreign key constraints)

These tables are often calculated when transactions are posted, or periods are closed. Typically, many reports pull from these tables because it is much more efficient than summarizing thousands or millions of transactions, and the data is already "cross-tabbed" the way most reporting tools would like to display it.

We can take this summarized data and "unpivot" it so we can still access the summarized data, but now it will be in a normalized structure. All it takes is a simple UNPIVOT like this:

select CompanyID, AccountID, TransactionTypeID, Year, substring(Period,7,2) as PeriodNo, Amount
from AccountBalances
unpivot (Amount for Period in (Period1,Period2,Period3,Period4,Period5,Period6,Period7,Period8,Period9,Period10,Period11,Period12)) as Amount

CompanyID   AccountID   TransactionTypeID Year        PeriodNo Amount
----------- ----------- ----------------- ----------- -------- ---------------------
1           1           1                 2008        1        200.00
1           1           1                 2008        2        300.00
1           1           1                 2008        3        400.00
1           1           1                 2008        4        500.00
1           1           1                 2008        5        400.00
1           1           1                 2008        6        0.00
1           1           1                 2008        7        0.00
1           1           1                 2008        8        0.00
1           1           1                 2008        9        0.00
1           1           1                 2008        10       0.00
1           1           1                 2008        11       0.00
1           1           1                 2008        12       0.00
1           2           1                 2008        1        100.00
1           2           1                 2008        2        100.00
1           2           1                 2008        3        100.00
1           2           1                 2008        4        100.00
1           2           1                 2008        5        100.00
1           2           1                 2008        6        100.00
1           2           1                 2008        7        100.00
1           2           1                 2008        8        0.00
1           2           1                 2008        9        0.00
1           2           1                 2008        10       0.00
1           2           1                 2008        11       0.00
1           2           1                 2008        12       0.00
1           3           1                 2008        1        150.00
1           3           1                 2008        2        0.00
1           3           1                 2008        3        50.00
1           3           1                 2008        4        10.00
1           3           1                 2008        5        10.00
1           3           1                 2008        6        200.00
1           3           1                 2008        7        400.00
1           3           1                 2008        8        45.00
1           3           1                 2008        9        0.00
1           3           1                 2008        10       0.00
1           3           1                 2008        11       0.00
1           3           1                 2008        12       0.00

(36 row(s) affected)

We can filter so that only periods with a non-zero amount are included, and we can SELECT FROM this result set and get the exact data we need for whatever date range we want without worrying which column the actual data is in.

This can also be done with a CROSS JOIN or a UNION ALL, but with 12 values to pivot, those options become much longer to write and UNPIVOT appears to be the way to go in this case.

Example #4: Multiple unpivots

Finally, let's consider a more complicated example. Here, we have a table that stores games played between two teams, where one team is the HomeTeam and the other is the AwayTeam:

create table Teams
(
    TeamCode char(3) primary key not null,
    TeamName varchar(100) not null
)

create table Games
(
    GameDate datetime,
    HomeTeam char(3) references Teams(TeamCode),
    AwayTeam char(3) references Teams(TeamCode),
    HomeScore int,
    AwayScore int,
    primary key (GameDate, HomeTeam),
    constraint pk2 unique (GameDate, AwayTeam),
    check (HomeTeam <> AwayTeam)
)

insert into Teams
select 'BOS','Boston Red Sox' union all
select 'NYY','New York Yankees'

insert into Games
select '2008-04-01','BOS','NYY',3,1 union all
select '2008-04-02','BOS','NYY',6,4 union all
select '2008-04-03','BOS','NYY',2,3 union all
select '2008-04-08','NYY','BOS',6,0 union all
select '2008-04-09','NYY','BOS',2,6 union all
select '2008-04-10','NYY','BOS',1,10

GameDate                HomeTeam AwayTeam HomeScore   AwayScore
----------------------- -------- -------- ----------- -----------
2008-04-01 00:00:00.000 BOS      NYY      3           1
2008-04-02 00:00:00.000 BOS      NYY      6           4
2008-04-03 00:00:00.000 BOS      NYY      2           3
2008-04-08 00:00:00.000 NYY      BOS      6           0
2008-04-09 00:00:00.000 NYY      BOS      2           6
2008-04-10 00:00:00.000 NYY      BOS      1           10

(6 row(s) affected)

That may or may not be the best possible design for the Games table, but it is a common way to do it. With data in that form, if we want to get the total runs scored per team across all games, or get each teams won-loss record, we need to make two passes through the table. This can be done fairly easily and efficiently with a union:

select GameDate,
    HomeTeam as TeamCode,
    'Home' as HomeOrAway,
    HomeScore as Score,
    case when HomeScore > AwayScore then 1 else 0 end as Win,
    case when HomeScore < AwayScore then 1 else 0 end as Loss,
    case when HomeScore = AwayScore then 1 else 0 end as Tie
from
    Games
union all
select GameDate,
    AwayTeam as TeamCode,
    'Away' as HomeOrAway,
    AwayScore as Score,
    case when HomeScore < AwayScore then 1 else 0 end as Win,
    case when HomeScore > AwayScore then 1 else 0 end as Loss,
    case when HomeScore = AwayScore then 1 else 0 end as Tie
from
    Games

GameDate                TeamCode HomeOrAway Score       Win         Loss        Tie
----------------------- -------- ---------- ----------- ----------- ----------- -----------
2008-04-01 00:00:00.000 BOS      Home       3           1           0           0
2008-04-02 00:00:00.000 BOS      Home       6           1           0           0
2008-04-03 00:00:00.000 BOS      Home       2           0           1           0
2008-04-08 00:00:00.000 NYY      Home       6           1           0           0
2008-04-09 00:00:00.000 NYY      Home       2           0           1           0
2008-04-10 00:00:00.000 NYY      Home       1           0           1           0
2008-04-01 00:00:00.000 NYY      Away       1           0           1           0
2008-04-02 00:00:00.000 NYY      Away       4           0           1           0
2008-04-03 00:00:00.000 NYY      Away       3           1           0           0
2008-04-08 00:00:00.000 BOS      Away       0           0           1           0
2008-04-09 00:00:00.000 BOS      Away       6           1           0           0
2008-04-10 00:00:00.000 BOS      Away       10          1           0           0

(12 row(s) affected)

Now, how can we do this with UNPIVOT? This example is a bit more complicated, because we are "unpivoting" not only the score, but also the TeamCode, and we are also calculating a few extra columns (Win, Loss and Tie).

So, can we UNPIVOT more than 1 column? Let's start simply and do things one at a time. First, let's UNPIVOT just the TeamCode:

select GameDate, HomeOrAway, Team
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team

GameDate                HomeOrAway     Team
----------------------- -------------- ----
2008-04-01 00:00:00.000 HomeTeam       BOS
2008-04-01 00:00:00.000 AwayTeam       NYY
2008-04-02 00:00:00.000 HomeTeam       BOS
2008-04-02 00:00:00.000 AwayTeam       NYY
2008-04-03 00:00:00.000 HomeTeam       BOS
2008-04-03 00:00:00.000 AwayTeam       NYY
2008-04-08 00:00:00.000 HomeTeam       NYY
2008-04-08 00:00:00.000 AwayTeam       BOS
2008-04-09 00:00:00.000 HomeTeam       NYY
2008-04-09 00:00:00.000 AwayTeam       BOS
2008-04-10 00:00:00.000 HomeTeam       NYY
2008-04-10 00:00:00.000 AwayTeam       BOS

(12 row(s) affected)

OK, so far so good. Now, how do we get the score for the team as well? Let's add another UNPIVOT clause to the SELECT, this time for Score:

select GameDate, HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway in (HomeScore, AwayScore)) as Score

Msg 265, Level 16, State 1, Line 1
The column name "HomeOrAway" specified in the UNPIVOT operator conflicts with the existing column name in the UNPIVOT argument.
Msg 8156, Level 16, State 1, Line 1
The column 'HomeOrAway' was specified multiple times for 'Score'.

Hmmm. OK, we cannot specify the same column for both pivots. That is just an alias specification, no big deal, so let's just alias it as "HomeOrAway2" :

select GameDate, HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   BOS 3
2008-04-01 00:00:00.000 HomeTeam   BOS 1
2008-04-01 00:00:00.000 AwayTeam   NYY 3
2008-04-01 00:00:00.000 AwayTeam   NYY 1
2008-04-02 00:00:00.000 HomeTeam   BOS 6
2008-04-02 00:00:00.000 HomeTeam   BOS 4
2008-04-02 00:00:00.000 AwayTeam   NYY 6
2008-04-02 00:00:00.000 AwayTeam   NYY 4
2008-04-03 00:00:00.000 HomeTeam   BOS 2
2008-04-03 00:00:00.000 HomeTeam   BOS 3
2008-04-03 00:00:00.000 AwayTeam   NYY 2
2008-04-03 00:00:00.000 AwayTeam   NYY 3
2008-04-08 00:00:00.000 HomeTeam   NYY 6
2008-04-08 00:00:00.000 HomeTeam   NYY 0
2008-04-08 00:00:00.000 AwayTeam   BOS 6
2008-04-08 00:00:00.000 AwayTeam   BOS 0
2008-04-09 00:00:00.000 HomeTeam   NYY 2
2008-04-09 00:00:00.000 HomeTeam   NYY 6
2008-04-09 00:00:00.000 AwayTeam   BOS 2
2008-04-09 00:00:00.000 AwayTeam   BOS 6
2008-04-10 00:00:00.000 HomeTeam   NYY 1
2008-04-10 00:00:00.000 HomeTeam   NYY 10
2008-04-10 00:00:00.000 AwayTeam   BOS 1
2008-04-10 00:00:00.000 AwayTeam   BOS 10

(24 row(s) affected)

Holy schnikies, it works! We can specify more than one UNPIVOT clause for the same SQL statement! Who would have thunk it?

Uh oh -- wait a second. We have 24 results returned. We should have only 12. Something is not right here. Remember when we aliased our second unpivot column as "HomeOrAway2"? We did not return that anywhere in our results. Let's add that in and take a look:

select GameDate, HomeOrAway, HomeOrAway2, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score

GameDate                HomeOrAway HomeOrAway2 Team Score
----------------------- ---------- ----------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   HomeScore   BOS 3
2008-04-01 00:00:00.000 HomeTeam   AwayScore   BOS 1
2008-04-01 00:00:00.000 AwayTeam   HomeScore   NYY 3
2008-04-01 00:00:00.000 AwayTeam   AwayScore   NYY 1
2008-04-02 00:00:00.000 HomeTeam   HomeScore   BOS 6
2008-04-02 00:00:00.000 HomeTeam   AwayScore   BOS 4
2008-04-02 00:00:00.000 AwayTeam   HomeScore   NYY 6
2008-04-02 00:00:00.000 AwayTeam   AwayScore   NYY 4
2008-04-03 00:00:00.000 HomeTeam   HomeScore   BOS 2
2008-04-03 00:00:00.000 HomeTeam   AwayScore   BOS 3
2008-04-03 00:00:00.000 AwayTeam   HomeScore   NYY 2
2008-04-03 00:00:00.000 AwayTeam   AwayScore   NYY 3
2008-04-08 00:00:00.000 HomeTeam   HomeScore   NYY 6
2008-04-08 00:00:00.000 HomeTeam   AwayScore   NYY 0
2008-04-08 00:00:00.000 AwayTeam   HomeScore   BOS 6
2008-04-08 00:00:00.000 AwayTeam   AwayScore   BOS 0
2008-04-09 00:00:00.000 HomeTeam   HomeScore   NYY 2
2008-04-09 00:00:00.000 HomeTeam   AwayScore   NYY 6
2008-04-09 00:00:00.000 AwayTeam   HomeScore   BOS 2
2008-04-09 00:00:00.000 AwayTeam   AwayScore   BOS 6
2008-04-10 00:00:00.000 HomeTeam   HomeScore   NYY 1
2008-04-10 00:00:00.000 HomeTeam   AwayScore   NYY 10
2008-04-10 00:00:00.000 AwayTeam   HomeScore   BOS 1
2008-04-10 00:00:00.000 AwayTeam   AwayScore   BOS 10

(24 row(s) affected)

Notice what is happening here. Without an UNPIVOT, we had 6 rows. We added one UNPIVOT to pivot two values, and got 12 rows. Then, we added another pivot, again with two values, and got back 24. Looking at the results, we can see that the single game at '2008-04-01' was unpivoted into 4 rows. So, while we can unpivot multiple times, there is no relation between each UNPIVOT. That is, notice there is no correlation between HomeOrAway and HomeOrAway2. Let's look at just that first game:

GameDate                HomeOrAway HomeOrAway2 Team Score
----------------------- ---------- ----------- ---- -----------
2008-04-01 00:00:00.000 HomeTeam   HomeScore   BOS 3
2008-04-01 00:00:00.000 HomeTeam   AwayScore   BOS 1
2008-04-01 00:00:00.000 AwayTeam   HomeScore   NYY 3
2008-04-01 00:00:00.000 AwayTeam   AwayScore   NYY 1

Both scores, home and away, are specified for each team -- even if it is not their matching score. What we really want is only these two lines:

How can we get there? Well, there's always a WHERE clause! We can simply filter our result so that the Left 4 characters of HomeOrAway matches the left 4 characters of HomeOrAway2. After all, there is nothing magical about these unpivoted columns, they are data just like anything else.

So, we can now write:

select GameDate, left(HomeOrAway,4) as HomeOrAway, Team, Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team
unpivot (Score for HomeOrAway2 in (HomeScore, AwayScore)) as Score
where left(HomeOrAway,4) = left(HomeOrAway2,4)

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 Home       BOS 3
2008-04-01 00:00:00.000 Away       NYY 1
2008-04-02 00:00:00.000 Home       BOS 6
2008-04-02 00:00:00.000 Away       NYY 4
2008-04-03 00:00:00.000 Home       BOS 2
2008-04-03 00:00:00.000 Away       NYY 3
2008-04-08 00:00:00.000 Home       NYY 6
2008-04-08 00:00:00.000 Away       BOS 0
2008-04-09 00:00:00.000 Home       NYY 2
2008-04-09 00:00:00.000 Away       BOS 6
2008-04-10 00:00:00.000 Home       NYY 1
2008-04-10 00:00:00.000 Away       BOS 10

(12 row(s) affected)

Now we are back to 12 results returned, the Homes and Aways match up, and things look good! Hey, we just pulled off 2 unpivots at once, not bad! From here, we can add our Win, Loss and Tie calculations as well. Note that we also have removed "HomeOrAway2" from the SELECT, since we don't need to see that column in our result. Finally, we are returning only the left 4 characters of HomeTeam or AwayTeam, so that the word 'Home' or 'Away' is returned in the HomeOrAway column.

This seems like quite a bit of work. Is UNPIVOT really the way to go in this scenario? Maybe not. I have not done full performance testing of UNPIVOT versus the other options shown (this article is way too long as it is!) but as always you should fully test your options and decide which offers the best performance and code simplicity.

Another option if you want to unpivot more than 1 column is to just use one UNPIVOT cause, but then use CASE expressions to return the values you need. For example:

select GameDate, left(HomeOrAway,4) as HomeOrAway,
Team,
case when HomeOrAway='HomeTeam' then HomeScore else AwayScore end as Score
from Games
unpivot (Team for HomeOrAway in (HomeTeam, AwayTeam)) as Team

GameDate                HomeOrAway Team Score
----------------------- ---------- ---- -----------
2008-04-01 00:00:00.000 Home       BOS 3
2008-04-01 00:00:00.000 Away       NYY 1
2008-04-02 00:00:00.000 Home       BOS 6
2008-04-02 00:00:00.000 Away       NYY 4
2008-04-03 00:00:00.000 Home       BOS 2
2008-04-03 00:00:00.000 Away       NYY 3
2008-04-08 00:00:00.000 Home       NYY 6
2008-04-08 00:00:00.000 Away       BOS 0
2008-04-09 00:00:00.000 Home       NYY 2
2008-04-09 00:00:00.000 Away       BOS 6
2008-04-10 00:00:00.000 Home       NYY 1
2008-04-10 00:00:00.000 Away       BOS 10

(12 row(s) affected)

This is kind of a combination of using UNPIVOT and a CROSS JOIN; the UNPIVOT doubles-up the results returned just as a CROSS JOIN would, and we just use CASE to return the column that we need depending if the current row is a "HomeTeam" or an "AwayTeam" row. This is a fairly good solution and is probably better than using multiple UNPIVOT clauses.

Summary

UNPIVOT is a really handy operator, but it can be tricky to use. You can UNPIVOT more than 1 column at a time, but beware of how it works -- each UNPIVOT uses it's own "pivot values" and you may get many more results than you intend. Thus, you may need to add criteria to your SELECT to ensure that only the results you are looking for are returned. Alternatively, you can use one UNPIVOT along with CASE expressions to achieve the same results.

UNPIVOT is great, but the main thing is that you are at the mercy of the column names; the naming conventions of the columns used becomes data, and that can really complicate things. In general, we want to keep our object names and our data seperate, so keep that limitation in mind if you wish to employ UNPIVOT in your code. In some cases, an old-school UNION ALL or CROSS JOIN may be a preferrable solution.

The Mailbag: Referencing Assemblies in Reporting Services; some SQL help

Jeff Smith — Thu, 18 Oct 2007 16:19:54 GMT

As David Letterman would say, wake the kids, call the neighbors, it's time for The Mailbag! Just some quickies today.

Christopher writes:

Greetings Jeff,

First and foremost, great job with all of the blogs. I have a questions
that I cannot seem to get a straight answer for. I am working with SQL
Server Reporting Services (SSRS) and have the need to create VB
functions to customize the reports generated. For example, a setter/getter to
display information that would not be readily available from the
query. SSRS allows this type of custom Visual Basic code to reside in the
report itself, but since most of my code is across multiple reports, it
makes 0 sense to place the same code in each report. Do you know of a
way that I can create a "code library" so that there is one location for
all of the code and all of the reports can access it?

Thank you very much for your time and effort.

Sincerely,
Christopher McGraw

Chris, Here's all the information that you need: Using Custom Assemblies with Reports (from SQL Server 2005 Books On Line). I can't really add much to it -- it explains everything in detail.

Traner writes:

I have two tables, table1's primary key is table2's foreign key.
Table2 has multiple dates that relate to table1.

ie.

Table1 : Table2
JOE : 10/18/07
DAVE : 11/14/07
JOE : 11/27/08

I am trying to get highest date from table2 related to table1. Do you
know how this can be accomplished. Any assistance will be greatly
appreciated.

That really isn't the clearest way to convey your schema, but I think I know what you mean. Let's assume that Table1 has two columns, ID and Name, and that table2 has 3 columns, ID, Date and Value.

First, to get the highest date per ID, you simply use the MAX() aggregate function along with GROUP BY when selecting from Table2:

select ID, Max(Date) as MaxDate
from Table2
group by ID

Be sure to read up and practice with GROUP BY and aggregate functions if you are not familiar with them -- they are very powerful and very fundamental to SQL.

Now, you can select from Table1 and join to the above SQL statement in a derived table to return all rows from Table1 and the maximum date from table 2:

select Table1.ID, Table1.Name, t2.MaxDate
from Table1
inner join
( select ID, Max(Date) as MaxDate
from Table2
group by ID
) t2 on Table1.ID = Table2.ID

As you can see, we have enclosed our previous SQL statement in parenthesis, and assigned it an alias ("t2" in this case), and we can join to it and select from it just like any other table. So, we have simply selected from Table1 and joined to a summarized version of Table2 that only returns the maximum date per ID.

Finally, notice that all we are returning from Table2 is the date; if there are other columns from Table2 that we wish to return (for example, the "Value" column), we must add another join to Table2, that returns only the row with the maximum date per ID:

select Table1.ID, Table1.Name, t2.MaxDate, Table2.Value
from Table1
inner join
( select ID, Max(Date) as MaxDate
from Table2
group by ID
) t2 on Table1.ID = Table2.ID
inner join Table2 on t2.ID = Table2.ID and t2.MaxDate = t2.Date

Notice that the "on" clause indicates that we are joining on ID as you might expect, but also on MaxDate = Date, so that we only return the row in Table2 that has the maximum date per ID.

If you are using SQL Server 2005, this can be done a little easier, with only one join, using the new Rank() function:

select x.ID, x.Name, x.Date, x.Value
from
(
select Table1.ID, Table1.Name, Table2.Date, Table2.Value,
rank() over (partion by Table1.ID order by Table2.Date DESC) as Rank
from Table1
inner join Table2 on Table1.ID = Table2.ID
) x
where x.Rank = 1

More on Rank() and partitions here. Hope this helps you out.

Creating CSV strings in SQL: Should Concatenation and Formatting Be Done at the Database Layer?

Jeff Smith — Tue, 09 Oct 2007 19:02:57 GMT

A question I see very often in the SQLTeam forums is how to return data in a summarized form by concatenating a list of values into a single CSV column. This can be done fairly easily in T-SQL, but as the formatting and concatenation requirements becomes more elaborate, be sure to ask yourself: Am I forcing presentation code into the database layer? read more...

Filter by month (plus other time periods)

Jeff Smith — Fri, 14 Sep 2007 16:21:07 GMT

Introduction

Previously, I wrote about grouping transactions by month. Another common area of difficulty or confusion for SQL beginners is how to efficiently retrieve data just for a single month.

There are two parts to this equation: First, what is the best way to declare parameters that will be used to indicate which month you are looking for? Second, how can we efficiently and easily make use of those parameters to get back the data we need?

Let's take a look at some approaches, both recommended and not.

And, please: if you are not using the DATETIME data type to store dates in your tables, don't even bother reading further -- fix your design first!

Month Parameter Options

@YearMonth as char(6)

If you know me at all, you know that I will advise not to use this approach. We’ll need to parse and validate the values passed into ensure that they are valid, and we need to somehow communicate exactly which format to use when setting the parameter values. For example, is it “MM-YY”, or “MMYYYY”, or “MM/YY”, or “YYYYMM”, etc? It simply makes no sense to introduce string formatting conventions and parsing and validating into the equation when you simply don’t need to. Don’t do it.

@Year as int, @Month as int

This approach works fine, since now we have no string parsing or ambiguity to deal with. It is very clear that Year and Month are both simply numeric values. We can do a quick check to ensure that are @Month value is between 1 and 12, and that our @Year value is within our desired range as well.

As we will see later, an optimal solution will require finding the first day of the month requested. How can we do that given a @Year and a @Month value? Simple, you can either use the Date function here and write:

Declare @FirstDayOfMonth datetime
Set @FirstDayOfMonth = dbo.Date(@Year,@Month,1)

… or you can apply the same logic using DateAdd() like this:

Declare @FirstDayOfMonth datetime

Set @FirstDayOfMonth = dateadd(month,((@Year-1900)*12)+@Month-1,0)

Either way, the result is the same: for a given @Year and @Month, the first day of that month (at midnight) is returned. If you aren’t sure how this formula works (or even if it will work at all), just test it out.

@DateInMonth as datetime

Here, we can simply accept any datetime value, and use the year and month of that date to indicate which month should be returned. A user can pass in the first day of the month, the last day, or any day in between – it doesn’t matter. This works well because we know the parameter will be a valid date, and we can quickly and simply get the year/month of the @DateInMonth parameter using the Year() and Month() functions:

Declare @Year int, @Month int
Select @Year = Year(@DateInMonth), @Month = Month(@DateInMonth)

We can also get the first day of the month for a given @DateInMonth quite easily, using another DateAdd() trick:

Declare @FirstDayOfMonth
Set @FirstDayOfMonth = DateAdd(month, DateDiff(month,0, @DateInMonth), 0)

So, accepting a dateTime parameter to indicate which month you are after is also a good technique to use.

WHERE Clause Options

Now that we have the @Year, @Month and @FirstDayOfMonth values ready to go, we must decide how to use these values to write our query.   Here are some options (again, both good and bad):

where Year(TranDate) =@Year and Month(TranDate) = @Month

This approach will work fine, but it will not be able to use any indexes that might exist on your TranDate column. Thus, it will not be as efficient as possible. Every single TranDate value must be retrieved from your table, the Year() and Month() formulas must be applied, and then the results must be tested to determine if that row should be returned or not.    Overall, avoid wrapping indexable columns in functions such as these in your criteria.

where convert(varchar(6), TranDate, 112) = convert(varchar(6), @FirstDayOfMonth)
-- or --
where convert(varchar(6), TranDate, 112) = left(@Year,4)+right('0' + left(@Month,2),2)
-- or --
where convert(varchar(6), TranDate, 112) = @YearMonth

These WHERE clauses all convert the TranDate to a VARCHAR in the format of YYYMMDD, but truncated at 6 characters, resulting in a YYYMM formatted string. Then, using different methods, that string is compared to a YYYYMM representation of the month we'd like to return. This is a bad approach to take because it requires that every TranDate be converted to a string in order to filter the rows, thus no indexes can be used. Using any conversions or string manipulations to filter data when better options exist is never the way to go.   At worst, it is also very confusing to work with since it can be unclear exactly what some of these convert() functions and string expressions are doing.

where TranDate between @FirstDayOfMonth and @LastDayOfMonth

What if we calculate, for a given @Year and @Month, the first and last day of the month, store them in variables, and then use that range for our criteria? This will let us use indexes, it will be clear and concise, and it makes good sense.

However, there is a minor flaw in this logic: Suppose @Year is 2006, and @Month is 1. If we set @LastDayOfMonth to ‘2006-01-31 12:00:00 AM’, any transactions after 12:00:00 AM on the last day of the month will not be included! We could calculate the very last 1/300th of a millisecond on the last day of the month to use as our upper limit, but there is an even easier way ...

where TranDate >= @FirstDayOfMonth and TranDate < @FirstDayOfNextMonth

Here, we won’t have any issues with times other than midnight leading to missed transactions.

We already have the @FirstDayOfMonth, all we need is the @FirstDayOfNextMonth. That is easily obtained by using DateAdd() to add one month to the @FirstDayOfMonth, as shown:

Declare @FirstDayOfNextMonth
Set @FirstDayOfNextMonth = DateAdd(month, @FirstDayOfMonth,1)

With that, we are all set! Indexes can be used and any datetime value within our range will be properly selected. I highly recommend using this approach whenever you need to select dates for a single month; there is no need to force date computations upon all values in your table when a simple range is all you need.

Other Time Periods

The same approach can be applied if you need to allow for a range of months to be specified, not just a single month. Simply calculate the first day of the starting month, and then calculate the first day of the month after the ending month. Then, you can use that date range to filter your data as shown above.

You should also follow these same guidelines if you'd like to filter data by a specified year, quarter, or week. "Year-to-date" filtering will also work the exact same way.

Conclusion

At the end of the day, accomplishing our task is very simple:

1.    Accept either @Year and @Month parameters or a single @DateInMonth parameter to indicate which month should be returned
2.    Calculate the @FirstDayOfMonth and the @FirstDayOfNextMonth using those parameters
3.    Select your rows with a WHERE clause that returns transaction dates equal to or greater than @FirstDayOfMonth, and less than @FirstDayOfNextMonth, to ensure that all dates at the end of your range are captured and that indexes can be used.

Good luck, and remember: Declare your parameters to be clear and precise, and write your WHERE clauses to be as efficient and accurate as possible.

see also:

Thinking Set-Based .... or not?

Jeff Smith — Mon, 30 Apr 2007 17:03:11 GMT

So, I hear you're a "set-based SQL master" ! As Yoda once said, you've "unlearned what you have learned". You've trained yourself to attack your database code not from a procedural, step-by-step angle, but rather from the set-based "do it all at once" approach. However, don't completely forget the most important skill that you learned in the procedural world!

read more...

Better Alternatives to a FULL OUTER JOIN

Jeff Smith — Thu, 19 Apr 2007 15:56:34 GMT

As many of you know, I strongly recommend that you avoid using RIGHT OUTER JOINS, since they make your SQL code less readable and are easily rewritten as LEFT OUTER JOINs. In addition, I have yet to find a situation where a FULL OUTER JOIN makes sense or is necessary -- I have found that in just about every case other techniques work better.

read more...

How to be an Effective Report Writer

Jeff Smith — Fri, 13 Apr 2007 15:03:18 GMT

A non-technical guide to writing reports, with a focus on managing requirements, results, and expectations.

read more...

Conditional Joins in SQL Server

Jeff Smith — Tue, 03 Apr 2007 16:15:47 GMT

Sometimes, when writing SELECTs, we come across situations in which we we need to write a join containing a condition of some sort. You might think to do this with either a CASE expression or with some OR boolean logic in your join expression. There's a much better way to approach the problem.

read more...

Rendering Excel and Word files with SQL Server

Jeff Smith — Tue, 18 Jul 2006 19:05:00 GMT

I've been playing around with a handy tool for creating Word and Excel files called OfficeWriter that's pretty impressive.

read more...

.NET CrossTabs versus SQL Server CrossTabs

Jeff Smith — Thu, 12 May 2005 14:01:00 GMT

As promised in my last post, here is some performance testing to help you determine the performance benefits (if any) of performing your crosstabs at the presentation or code layer, as opposed to forcing SQL Server to do this.

read more...