SQL General

Speaking Schedule

Geoff N. Hiten — Thu, 27 Mar 2008 14:36:01 GMT

If you feel an uncontrollable urge to listen to my voice or to meet me in person, here is your chance.

You can listen to me on Greg Low's SQL Down Under Podcast. I discuss SQL Clustering, SANs, and ramble on about other topics as well.

http://www.sqldownunder.com/PreviousShows/tabid/98/Default.aspx

I will be joining Kevin Kline and Hilary Cotter on a webcast for Quest Software on April 9^th. We will be discussing features in the forthcoming SQL 2008 product.

http://www.quest.com/events/listdetails.aspx?contentid=7059&searchoff=true&technology=34&prod=&prodfamily=&loc=

And I will be speaking at the SQLSaturday #3 event in Jacksonville, Florida on May 3^rd.

http://www.sqlsaturday.com/default.aspx

Leverage

Geoff N. Hiten — Thu, 07 Feb 2008 02:23:43 GMT

One definition of leverage is "the use of a small initial investment, credit, or borrowed funds to gain a very high return in relation to one's investment, to control a much larger investment, or to reduce one's own liability for any loss." (Courtesy Dictionary.Com).

In programming, the smallest change we can make is a single bit. The next smallest change and the one we can make most effectively in human-readable source code is a single character. This story is how changing one character in a 300 line stored procedure removed 90% of the impact of the worst single query on the entire server.

This process began with a fairly normal analysis of a poorly performing server. As usual, the server was found to be I/O bound. Routine analysis indicated a particular query within a stored procedure to be the worst performing query. Comparing this query to the second worst query on the top ten list revealed it consumed 15 times the CPU and 20 times the IO cycles of the next worst offender. Calling Captain Obvious, please report to Performance Tuning.

I did an index analysis. No joy there. Everything looked correct and the query plans were fine. The query took a lookup string, hit a master lookup table, did key lookups across 4 other tables (three joins deep at most) and returned the extended attributes of the master key. The master table had about 400K rows and was constantly growing with new key values. This lookup was executed hundreds of thousands of times a day, so any improvement would be significant.

Since the schema and data was from an external data provider, I couldn't make any significant changes, nor was there any support for de-normalizing the data for a flat lookup.

My brilliant idea was to create an indexed view and let SQL Enterprise Edition do an under-the-covers substitution. Like most brilliant ideas, this did not work as expected. I could not get SQL to substitute the view for the underlying tables. ARRGHHH!

Finally I realized the problem was not in the query, nor was it in the index. It was something we usually take for granted. An implicit data type conversion was killing me. This was the culprit:

WHERE MasterKey = @LookupKey

Well, it was sort of the culprit. You see, Master_key was defined as varchar(10) by the external data provider and @LookupKey was defined by a wannabe senior developer. Naturally, a real developer uses forward-compatible data types like nvarchar(10) and not nearly obsolete types like varchar(10).

Most of us know not to write something like this:

WHERE Column_Value / 2 = 4

We write something like:

WHERE Column_Value = 8

The idea is to NOT force SQL to scan the column, run every value through a function, and then compare it to a constant. The correct way is to pre-calculate the constant and compare it to the column. Turns scans into seeks, cuts IO, promotes world peace, you get the idea.

So what does this have to do with the first WHERE clause. One rule on implicit conversions to larger data types is that the lower type always gets promoted. Thus the WHERE clause actually became:

WHERE convert(nvarchar(10), MasterKey) = @LookupKey)

Now that is pretty obviously bad.

By now, you have likely figured out the one character fix. Remove the 'N' from the declare function and use the old, bad, non-portable data type. Given the data definition for the real-world data type in this example only uses A-Z and 0-9 characters and the fix was at least a full order of magnitude faster, I can live with that.

Leverage. One character change fixes an entire problem server.

SQL 2008 CTP5 First Glance

Geoff N. Hiten — Fri, 16 Nov 2007 19:11:10 GMT

Microsoft began releasing SQL 2008 CTP5 yesterday through Connect. It should hit MSDN within a few days so everyone can play along. The most obvious change for this release is a much-needed improvement replacement of the installer. As I mentioned in an earlier post, the installer is being totally re-engineered. We still don't have clustering support in this CTP, but the basic installer changes are now included. While there are a few "fit and finish" items that need cleanup, the overall design of the installer is very good. The requirements checker is much lighter and cleaner. The install review presentation page is in a hierarchal form and actually makes sense. Best of all, you can specify separate default directories for system databases, user databases, user database log, tempdb data, tempdb log and backup target locations. Installation is one area where attention to detail and listening to customer feedback has given the SQL development team a significant win.

Online Reindex = ON

Geoff N. Hiten — Thu, 01 Nov 2007 13:35:00 GMT

For the past several editions, SQL Books On-Line (BOL) has helpfully included a script to rebuild or defragment (since 2000) an index. Being Microsoft, this script is NOT located under the reindex or defragmentation topic, it is included in the fragmentation analysis section. For SQL 2000, this is DBCC SHOWCONTIG. For SQL 2005, they rewrote it to use the new system views and stashed it under sys.dm_db_physical_stats. It also uses the new ALTER INDEX command rather than the older DBCC DBREINDEX or DBCC INDEXDEFRAG command. However, this script does not allow you to take advantage of online indexing

My first attempt to modify the script consisted of dropping the defragmentation option and simply adding a WITH ONLINE = ON line to the script. As with most quick and dirty mods, this failed miserably. I soon discovered that some indexes cannot be rebuilt online. There are two conditions that will prevent online reindexing, both of which involve BLOB data types:

You cannot online reindex a non-clustered index that contains a BLOB column.

You cannot online reindex a clustered index of a table that contains a BLOB column.

Pretty straightforward, or so it seems. The code to identify BLOB columns from an index is a bit tricky, but not too impossible. I used actual data type names rather than the internal data type IDs to make the code more readable. I also added a filter condition to remove "trivial" sized indexes. These are typically allocated into mixed extents and do not respond to defragmentation. There is no real harm in including them, but they clutter up the result set when troubleshooting. I kept tweaking the number until I got it as low as possible without an index reappearing in subsequent runs. The test platform was a Microsoft CRM implementation. For those unfamiliar with the table design of MSCRM, it uses GUIDs as primary keys and accepts the default of clustering the primary key. This makes for a nicely fragmented database to test, especially if there is a lot of insert and delete activity.

So without further ado, here is the modified BOL code that allows for scripting online index rebuilds.

-- Adapted from BOL script by Geoff N. Hiten.

USE <Your Database name here>

-- Ensure a USE <databasename> statement has been executed first.

SET NOCOUNT ON;

DECLARE @objectid int;

DECLARE @indexid int;

DECLARE @partitioncount bigint;

DECLARE @schemaname nvarchar(130);

DECLARE @objectname nvarchar(130);

DECLARE @indexname nvarchar(130);

DECLARE @partitionnum bigint;

DECLARE @partitions bigint;

DECLARE @frag float;

DECLARE @command nvarchar(4000);

DECLARE @HasBlobColumn int;

DECLARE @MaxFragmentation int

DECLARE @TrivialPageCount int

-- Tuning constants

SET @MAxFragmentation = 10

SET @TrivialPageCount = 12

-- Conditionally select tables and indexes from the sys.dm_db_index_physical_stats function

-- and convert object and index IDs to names.

SELECT

object_id AS objectid,

index_id AS indexid,

partition_number AS partitionnum,

avg_fragmentation_in_percent AS frag

INTO #work_to_do

FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL , NULL, 'LIMITED')

WHERE avg_fragmentation_in_percent > @MaxFragmentation -- arbitrary threshold. YMMV

AND index_id > 0 -- cannot defrag a heap

and page_count > @TrivialPageCount -- ignore trivial sized indexes

-- Declare the cursor for the list of partitions to be processed.

DECLARE partitions CURSOR FOR SELECT * FROM #work_to_do;

-- Open the cursor.

OPEN partitions;

-- Loop through the partitions.

WHILE (1=1)

BEGIN;

FETCH NEXT

FROM partitions

INTO @objectid, @indexid, @partitionnum, @frag;

IF @@FETCH_STATUS < 0 BREAK;

Set @HasBlobColumn = 0 -- reinitialize

SELECT @objectname = QUOTENAME(o.name), @schemaname = QUOTENAME(s.name)

FROM sys.objects AS o

JOIN sys.schemas as s ON s.schema_id = o.schema_id

WHERE o.object_id = @objectid;

SELECT @indexname = QUOTENAME(name)

FROM sys.indexes

WHERE object_id = @objectid AND index_id = @indexid;

SELECT @partitioncount = count (*)

FROM sys.partitions

WHERE object_id = @objectid AND index_id = @indexid;

-- Check for BLOB columns

if @indexid = 1 -- only check here for clustered indexes ANY blob column on the table counts

Select @HasBlobColumn = case when max(so.object_ID) IS NULL then 0 else 1 end

From sys.objects SO

inner join sys.columns SC

on SO.Object_id = SC.object_id

inner join sys.types ST

on SC.system_type_id = ST.system_type_id

and ST.name in ('text', 'ntext', 'image', 'varchar(max)', 'nvarchar(max)', 'varbinary(max)', 'xml')

where SO.Object_ID = @objectID

else -- nonclustered. Only need to check if indexed column is a BLOB

Select @HasBlobColumn = case when max(so.object_ID) IS NULL then 0 else 1 end

from sys.objects SO

inner join sys.index_columns SIC

on SO.Object_ID = SIC.object_id

inner join sys.Indexes SI

on SO.Object_ID = SI.Object_ID

and SIC.index_id = SI.index_id

inner join sys.columns SC

on SO.Object_id = SC.object_id

and SIC.Column_id = SC.column_id

inner join sys.types ST

on SC.system_type_id = ST.system_type_id

and ST.name in ('text', 'ntext', 'image', 'varchar(max)', 'nvarchar(max)', 'varbinary(max)', 'xml')

where SO.Object_ID = @objectID

SET @command = N'ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REBUILD';

if @HasBlobColumn = 1

Set @command = @command + N' WITH( SORT_IN_TEMPDB = ON) '

else

Set @command = @command + N' WITH( ONLINE = ON, SORT_IN_TEMPDB = ON) '

IF @partitioncount > 1

SET @command = @command + N' PARTITION=' + CAST(@partitionnum AS nvarchar(10));

PRINT N'Executing: ' + @command + ' Has Blob = ' + convert(nvarchar(2),@HasBlobColumn);

EXEC (@command)

-- PRINT N'Executing: ' + @command;

END;

-- Close and deallocate the cursor.

CLOSE partitions;

DEALLOCATE partitions;

-- Drop the temporary table.

DROP TABLE #work_to_do;

BEGIN TRY

Geoff N. Hiten — Tue, 25 Sep 2007 18:40:06 GMT

Hopefully, I won't need a CATCH Block.

In case that opening sentence didn't give you a hint, I am a bit of a tech geek. Bill Graziano talked me into joining this merry band while at the PASS summit last week. While I enjoy answering questions on newsgroups, this will give me a way to initiate conversations. I looked around and decided SQLTeam would be a good place to stand and pontificate.

And now for some real SQL content. Sort of.

Based on a lot of presentations by Microsoft on proposed features and bug fixes, I have to conclude that Connect really matters. Connect is Microsoft's feedback center. While the list of complaints about Connect is quite long, it has one overwhelming benefit. The Microsoft SQL Development team actually pays attention to it. Not only that, but Microsoft is allocating resources based on feedback gathered from Connect. So get over the fact Connect seems modeled after a particularly clumsy session of "Messenger" at a Jr. High School party and use it. Post bugs. Post Suggestions. Vote early and often. You want to influence the product and you don't happen to have Ted Kummert's digits on the speed dial. Here is your chance.