Efficient paging of recordsets: SQL Server 2000
Here's the most efficient way that I can think of to return rows @a through @b from a table or query, when each row is ordered and can be uniquely identified by a set of columns.
In this example, we have a table called “Data”, and we wish to sort by TransDate. The primary key of the table is “ID” (yep, an identity !) so we include that as the secondary sort to ensure all rows can be uniquely identified by the sort columns.
The plan: set the rowcount to the starting row number (in this case, @a), and get the value of the “starting points” for each sort column. Then, set the rowcount to the number of rows to return (@b-@a), set up a WHERE clause so that we start at the starting point, and then return the results.
This avoids temp tables and always processes as few rows as possible -- at most, @b rows. So it will get slower as you go. But, it works pretty well so give it a shot. If your ordering needs to be dynamically assigned, then this technique may not do much for you but it may help give you some ideas.
Update #1 (3/22/04) : I have slightly edited the code to fix the "first row" error, as specified in the comments.
Update #2 (4/8/07): If you are using SQL Server 2005, it is much easier ... see this.
-- for each column in your sort, you need a variable to hold
-- the "starting values". In our case, we need two:
declare @startingDate datetime;
declare @startingID int;
-- again, we want to returns results from row @a to row @b:
declare @a int;
declare @b int;
set @a = 200 -- start at row 200
set @b = 250 -- end at row 250
-- get the starting date and starting ID to return results:
set rowcount @a
select @StartingDate = TransDate, @startingID = ID
from Data
order by TransDate ASC,ID ASC
-- find out how many rows to return, and set the rowcount:
set @b = 1 + @b - @a
set rowcount @b
-- now return the results:
select * from Data
where
TransDate > @StartingDate OR
(TransDate = @StartingDate AND ID >= @StartingID)
order by TransDate ASC, ID ASC
-- clean up:
set rowcount 0
The WHERE clause in the final SQL statement is the key, and it is good to be familiar with this type of clause even if you don't use this exact paging technique. Look at it closely, and you will see it expresses logic to return rows within a certain range when that range is defined by more than 1 column.
For example, if you have a column called “YearMonth” with values in a YYYY-MM format, you can easily say:
WHERE YearMonth >= '2003-04'
But if you have two separate columns, one for Month and one for Year, you would express that same WHERE clause this way:
WHERE Year > 2003 OR (Year = 2003 AND Month >= 04)
Legacy Comments
Richard Tallent
2003-12-22 |
re: Efficient paging of recordsets with T-SQL Good job! I posted my own approach (linked to my name here) in response. It is more database independent and reusable, but moves the record-skipping logic to .NET (there are advantages and disadvantages). |
ValterBorges
2003-12-23 |
re: Efficient paging of recordsets with T-SQL what about order desc and using top 1?? |
TheKode
2003-12-29 |
re: Efficient paging of recordsets with T-SQL This code has a bug in it that means it cannot return the very first row. The example starts from row 200 but actually returns the next row, ie 201 (since the first select gets the item at row 200, and the second selects gets the next value > than this value). This means that if you want row 1, you should specify row 0, ie to get the next row. However if you do that, the first select will do a rowcount 0 and set the starting row to the last row of the select. (And if you specify row 1, it will return from row 2.) My fix is to surround the second select with "if @@rowcount >= @nStartRow" so it only does it if the first select returned more rows that the row we want to start from. And change the where clause to "TransDate = @StartingDate AND ID >= @StartingID". It now includes the start row, so the example would return from 200 rather than 201. And so it allows row 1 to be returned. |
Jeff
2003-12-30 |
re: Efficient paging of recordsets with T-SQL Thanks for the fix! sorry I missed that. Valter -- not sure what you mean -- can you elaborate ? |
Jeff
2004-03-24 |
re: Efficient paging of recordsets with T-SQL FYI -- I have altered the code slightly to handle the "skipped first row" issue. It should work quite well now. Check out my dynamic version of this technique in my latest blog .... |
PRINCE OF EGYPT
2004-06-07 |
Creating Effecient Paging with T-SQL and the IDENTITY function Creating Effecient Paging with T-SQL and the IDENTITY function Now we all had the problem of creating a grid with next and back buttons , as it retreieves all the data from the server and filters it on the client site .That is a poor technique if you have a huge table ,few users will hang your machine and result in memory consumption SQL server lacks a way to limit the number of rows returned to the client in ranges ,and you have to write a very complex query to achieve this result ,like getting rows from 5 to 10 ,and you will depend on a primary key for sure .Lets first discuss how MySQL and Oracle solves this problem ... Mysql has a very pretty code to solve this problem you can say : SELECT * FROM foo LIMIT 5,10 this will retrieve the rows starting row number 5 till row number 15 .Very pretty code cause u just drop this { LIMIT N,M } at the end of your most complex query and you are done . Oracle has a similar approach ,it has a psuedoColumn always availiable to you when you issue a query , this column have a name of ROWNUM . if you do something like this SELECT ROWNUM ,* FROM foo you will get ROWNUM COl1 Col2 1 data data 2 data data 3 data data Now to page just say WHERE ROWNUM BETWEEN n AND M Pretty clean code just like the mysql code Now back to my only love SQLserver ,which is the best engine i worked on despite this issue ,there is no direct way to return a range ,you can only return TOP ,but no range Here is a little piece of code that you can compile as a stored procedure SET ROWCOUNT 10 -- set this to your desired page size ,this is for optimization ,the between clause will be enough select IDENTITY(int, 1,1) as 'ROWNUM' ,* into #temptable from foo SELECT * from #temptable will yield this ROWNUM COl1 Col2 1 data data 2 data data 3 data data which is similar to the ORACLE approach ,and the code will be even compatible with ORACLE as i can say SELECT * FROM #temptable where ROWNUM between N and M your code will be easy to read and understand Amr Salah |
Lee Dise
2004-06-08 |
re: Efficient paging of recordsets with T-SQL This thread is old, but interesting, so I thought I'd take a poke... If you know beforehand that your always going to need rows 200 to 250, this would work: SELECT * FROM (SELECT TOP 50 * FROM (SELECT TOP 250 * FROM Data ORDER BY TransDate ASC , ID ASC) a ORDER BY TransDate DESC , ID DESC) b ORDER BY TransDate ASC , ID ASC If the range is to be specified dynamically, then you can make it into dynamic SQL... DECLARE @sql VARCHAR (8000) , @a INT , @b INT SELECT @a = 200 , @b = 250 SELECT @sql = 'SELECT *' + ' FROM (SELECT TOP ' + CONVERT (VARCHAR, @a) + ' *' + ' FROM (SELECT TOP ' + CONVERT (VARCHAR, @b) + ' *' + ' FROM Data' + ' ORDER BY TransDate ASC' + ', ID ASC) a' + ' ORDER BY TransDate DESC' + ', ID DESC) b' + ' ORDER BY TransDate ASC' + ', ID ASC' EXEC (@sql) Better? Worse? I don't know if this approach is more or less efficient. But I think it meets the specification and it's arguably a simpler approach. |
Kevin
2004-07-22 |
re: Efficient paging of recordsets with T-SQL The last solution (reversing the results) doesn't work on the last page. It will always return the number of rows in a page. |
antoine
2004-09-22 |
re: Efficient paging of recordsets with T-SQL There's something I don't like in this 'rowcount' setting. If any error occurs between the 'set rowcount x' statement and 'set rowcount 0' statement, many queries will return wrong results. The rowcount property stays modified for the 'link' between the client and the server and if something wrong occurs -> sh.t happens. This might also lead to security issues (which I already had to handle because of that rowcount conflict). So, when using that property, be really sure your request cannot fail. |
Aatif Chaudhry
2005-09-13 |
re: Efficient paging of recordsets with T-SQL I want to ask how can I get the record of one page in a eperate array. and so on.. Actually I m using a function which returns records in an array. |
Dave
2005-09-14 |
re: Efficient paging of recordsets with T-SQL Any way to get a record count out of this? |
Jason Epstein
2005-10-26 |
re: Efficient paging of recordsets with T-SQL Perfect! Other solutions I have found only work if you wanted to select groups of records using a unique identifier only. The query makes my asp.net custom paging really efficient, it brings back the bare minumum of data required, whilst not not requiring cursors,temp tables, or even Sql2005. |
Lucian Chiriac
2005-12-28 |
re: Efficient paging of recordsets with T-SQL Great! Thank you . I was using a paging solution using temp tables and it reduced the time with at least 25% (recorded with a select on a table with 230.000+ records with joins and left joins with 5-6 other tables). Can't wait to see what happens with a table with over 2 million records and 5-6 joined tables. Best regards, Lucian Chiriac |
Nataly
2006-02-22 |
re: Efficient paging of recordsets with T-SQL I enjoyed! Very good solution!!! |
Jenny
2006-04-14 |
re: Efficient paging of recordsets with T-SQL Nice job! Thanks! |
peleg
2006-05-11 |
re: Efficient paging of recordsets with T-SQL hi "AM Lee Dise" what u do is nice but when ig i have 5 million or more rows! then what will i do? bring 5 million rows and then take 50 of them that slow :( |
Uncle Sam
2006-05-29 |
re: Efficient paging of recordsets with T-SQL PLEASE can anyone tell me how to use NEXT and Back buttons in the query results of large data in SQL 2005 |
annon
2006-08-28 |
re: Efficient paging of recordsets with T-SQL this simply won't work: set rowcount @a select @StartingDate = TransDate, @startingID = ID from Data order by TransDate ASC,ID ASC does not matter what you assign to the rowcount, the select will always return the transdate and id of the first row in the table. |
Jeff
2006-08-28 |
re: Efficient paging of recordsets with T-SQL annon -- Did you try it? |
John Bonds
2007-05-21 |
re: Efficient paging of recordsets: SQL Server 2000 This is a good article. I've also read a few on the topic at the following locations: http://aspnet.4guysfromrolla.com/articles/031506-1.aspx http://www.4guysfromrolla.com/webtech/042606-1.shtml The only thing I'm trying to figure out is how to use this to both sort my result set DESC AND implement search functionality.. p.ProcessID >= @FirstID AND p.IsDeleted = 0 AND p.ProcessID = ISNULL(@ProcessID, p.ProcessID) AND i.VendorID = ISNULL(@VendorID, i.VendorID) AND etc. jB |
Jeff
2007-05-21 |
re: Efficient paging of recordsets: SQL Server 2000 JB -- keep your sorting and your filtering separate; you can use a derived table to help you. i.e., select x.* from ( your filtered select here .... )x where ... your paging condition here .... order by .... |
Ernesto
2007-06-08 |
re: Efficient paging of recordsets: SQL Server 2000 Thanks, I tryed this aproach, but the grid view custom paging also requires the number of rows and I couldn't find a way to get that because of rowcount and making the count before was even more expensive than the temp table. By the way, I think you avoid using the temp table 'cause the table you use has a consecutive id, my table does'nt have that consecutive Id, so I think it's impossible to avoid the temp table unless we had something like Oracle's rownumber. Any way, we all should be in SQL Svr 2005 by now. I f we are into this yet, is becaue our IT departments havn't decided to change yet. :( Bye. |
rüya tabiri
2008-09-07 |
re: Efficient paging of recordsets: SQL Server 2000 does not matter what you assign to the rowcount, the select will always return the transdate and id of the first row in the table. |
Foxchat Sohbet
2009-05-05 |
re: Efficient paging of recordsets: SQL Server 2000 does not matter what you assign to the rowcount, the select will always return the transdate and id of the first row in the table. Thank |
lig tv izle
2009-05-22 |
re: Efficient paging of recordsets: SQL Server 2000 thank you |
Edencity Chat
2009-05-22 |
re: Efficient paging of recordsets: SQL Server 2000 Thank You Admin |
liseli kizlar
2009-06-29 |
re: Efficient paging of recordsets: SQL Server 2000 thank you askim |
chat
2009-10-09 |
re: Efficient paging of recordsets: SQL Server 2000 chat rooms turkish thank you very much.. |
konya chat
2009-11-22 |
konya chat KONYA SOHBET |