High Availability

Update: SQL 2005 Build 3186 Cluster problem

Geoff N. Hiten — Mon, 12 Nov 2007 21:09:24 GMT

Microsoft does not have a complete resolution for this problem yet, but they have found some more details. Evidently the problem with SQL Agent failure only occurs on systems using a domain admin account for the SQL Agent Service account. Microsoft is not 100% sure yet, so this is just a preliminary finding. However, it does match my own personal experiences. Worst Practices always has a cost. Several DBAs just found that out the hard way. Just as a reminder, this problem only occurs on x64 clusters using SQL Server 2005 build 3186 and higher.

Online Reindex = ON

Geoff N. Hiten — Thu, 01 Nov 2007 13:35:00 GMT

For the past several editions, SQL Books On-Line (BOL) has helpfully included a script to rebuild or defragment (since 2000) an index. Being Microsoft, this script is NOT located under the reindex or defragmentation topic, it is included in the fragmentation analysis section. For SQL 2000, this is DBCC SHOWCONTIG. For SQL 2005, they rewrote it to use the new system views and stashed it under sys.dm_db_physical_stats. It also uses the new ALTER INDEX command rather than the older DBCC DBREINDEX or DBCC INDEXDEFRAG command. However, this script does not allow you to take advantage of online indexing

My first attempt to modify the script consisted of dropping the defragmentation option and simply adding a WITH ONLINE = ON line to the script. As with most quick and dirty mods, this failed miserably. I soon discovered that some indexes cannot be rebuilt online. There are two conditions that will prevent online reindexing, both of which involve BLOB data types:

You cannot online reindex a non-clustered index that contains a BLOB column.

You cannot online reindex a clustered index of a table that contains a BLOB column.

Pretty straightforward, or so it seems. The code to identify BLOB columns from an index is a bit tricky, but not too impossible. I used actual data type names rather than the internal data type IDs to make the code more readable. I also added a filter condition to remove "trivial" sized indexes. These are typically allocated into mixed extents and do not respond to defragmentation. There is no real harm in including them, but they clutter up the result set when troubleshooting. I kept tweaking the number until I got it as low as possible without an index reappearing in subsequent runs. The test platform was a Microsoft CRM implementation. For those unfamiliar with the table design of MSCRM, it uses GUIDs as primary keys and accepts the default of clustering the primary key. This makes for a nicely fragmented database to test, especially if there is a lot of insert and delete activity.

So without further ado, here is the modified BOL code that allows for scripting online index rebuilds.

-- Adapted from BOL script by Geoff N. Hiten.

USE <Your Database name here>

-- Ensure a USE <databasename> statement has been executed first.

SET NOCOUNT ON;

DECLARE @objectid int;

DECLARE @indexid int;

DECLARE @partitioncount bigint;

DECLARE @schemaname nvarchar(130);

DECLARE @objectname nvarchar(130);

DECLARE @indexname nvarchar(130);

DECLARE @partitionnum bigint;

DECLARE @partitions bigint;

DECLARE @frag float;

DECLARE @command nvarchar(4000);

DECLARE @HasBlobColumn int;

DECLARE @MaxFragmentation int

DECLARE @TrivialPageCount int

-- Tuning constants

SET @MAxFragmentation = 10

SET @TrivialPageCount = 12

-- Conditionally select tables and indexes from the sys.dm_db_index_physical_stats function

-- and convert object and index IDs to names.

SELECT

object_id AS objectid,

index_id AS indexid,

partition_number AS partitionnum,

avg_fragmentation_in_percent AS frag

INTO #work_to_do

FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL , NULL, 'LIMITED')

WHERE avg_fragmentation_in_percent > @MaxFragmentation -- arbitrary threshold. YMMV

AND index_id > 0 -- cannot defrag a heap

and page_count > @TrivialPageCount -- ignore trivial sized indexes

-- Declare the cursor for the list of partitions to be processed.

DECLARE partitions CURSOR FOR SELECT * FROM #work_to_do;

-- Open the cursor.

OPEN partitions;

-- Loop through the partitions.

WHILE (1=1)

BEGIN;

FETCH NEXT

FROM partitions

INTO @objectid, @indexid, @partitionnum, @frag;

IF @@FETCH_STATUS < 0 BREAK;

Set @HasBlobColumn = 0 -- reinitialize

SELECT @objectname = QUOTENAME(o.name), @schemaname = QUOTENAME(s.name)

FROM sys.objects AS o

JOIN sys.schemas as s ON s.schema_id = o.schema_id

WHERE o.object_id = @objectid;

SELECT @indexname = QUOTENAME(name)

FROM sys.indexes

WHERE object_id = @objectid AND index_id = @indexid;

SELECT @partitioncount = count (*)

FROM sys.partitions

WHERE object_id = @objectid AND index_id = @indexid;

-- Check for BLOB columns

if @indexid = 1 -- only check here for clustered indexes ANY blob column on the table counts

Select @HasBlobColumn = case when max(so.object_ID) IS NULL then 0 else 1 end

From sys.objects SO

inner join sys.columns SC

on SO.Object_id = SC.object_id

inner join sys.types ST

on SC.system_type_id = ST.system_type_id

and ST.name in ('text', 'ntext', 'image', 'varchar(max)', 'nvarchar(max)', 'varbinary(max)', 'xml')

where SO.Object_ID = @objectID

else -- nonclustered. Only need to check if indexed column is a BLOB

Select @HasBlobColumn = case when max(so.object_ID) IS NULL then 0 else 1 end

from sys.objects SO

inner join sys.index_columns SIC

on SO.Object_ID = SIC.object_id

inner join sys.Indexes SI

on SO.Object_ID = SI.Object_ID

and SIC.index_id = SI.index_id

inner join sys.columns SC

on SO.Object_id = SC.object_id

and SIC.Column_id = SC.column_id

inner join sys.types ST

on SC.system_type_id = ST.system_type_id

and ST.name in ('text', 'ntext', 'image', 'varchar(max)', 'nvarchar(max)', 'varbinary(max)', 'xml')

where SO.Object_ID = @objectID

SET @command = N'ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REBUILD';

if @HasBlobColumn = 1

Set @command = @command + N' WITH( SORT_IN_TEMPDB = ON) '

else

Set @command = @command + N' WITH( ONLINE = ON, SORT_IN_TEMPDB = ON) '

IF @partitioncount > 1

SET @command = @command + N' PARTITION=' + CAST(@partitionnum AS nvarchar(10));

PRINT N'Executing: ' + @command + ' Has Blob = ' + convert(nvarchar(2),@HasBlobColumn);

EXEC (@command)

-- PRINT N'Executing: ' + @command;

END;

-- Close and deallocate the cursor.

CLOSE partitions;

DEALLOCATE partitions;

-- Drop the temporary table.

DROP TABLE #work_to_do;

SQL 2005 Build 3186 Cluster problem

Geoff N. Hiten — Thu, 11 Oct 2007 17:53:35 GMT

SQL 2005 Build 3186 has a major negative side effect on x64 clusters. Installing it pretty much kills the SQL Agent.. The workaround is to enable unconstrained delegation for the machine and the service accounts. Not exactly a best practices security setting, but necessary. For now, if you are on an x64 cluster and don't have a compelling reason to install this hotfix, you may want to wait until this issue gets resolved.

Here is the MSDN thread discussing the problem and the workaround:

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2242421&SiteID=1

UPDATE: Build 3200 (Cumulative Update 4) has the same issue. Evidently this bug is due to a complex interaction between SQL and Windows Security. I would not expect a quick fix. I can confim that this definitely has major visibility and resource assignment within Microsoft. It just isn't an easy problem to solve.

Patches! We don’t need no stinkeen’ Patches!

Geoff N. Hiten — Tue, 09 Oct 2007 21:31:18 GMT

Well, yes. Actually you do. Patching a SQL Server cluster sounds like a complex endeavor, but it is really a lot simpler than people think. Much of the confusion is due to SQL 2000 and SQL 2005 having slightly different behaviors when it comes to patching. Also, with Clustering now available in Standard Edition, more DBAs find themselves caring for the first cluster in their organization. “Learn as you go” is not nearly as fun as it sounds.

The first rule of patching is to test. Make sure the patches work in a test environment, both with patches for the OS and for SQL. Given the history of unstable SQL Service Packs, I have to suggest waiting for a few weeks after a SP is released to make sure nothing important got broken. The SQL team is very much aware of their past shortcomings in that area and has started to fix the entire SQL servicing model. It will be a while before we see the fruits of those efforts, so for the time being let someone else be the (insert politically correct ethnic minority here) mine detector.

Regardless of the SQL version, the OS will need patching from time to time. Balancing the need to keep a system secure from newly discovered vulnerabilities with the desire to have a stable platform is tricky. You want to stay up-to-date, but not TOO up-to-date. Again, testing is the key. Never deploy anything you have not tested onto a highly-available platform. As such, I never allow Windows or Microsoft update to automatically patch any database host server. I manually inspect and apply patches, although I still use the update tool to recommend and track updates. Only apply patches during a defined maintenance window, even if you just patch a currently inactive node. That way if something goes wrong, you don’t have to explain why you took the accounting server down during month-end closing. If you have an N-1 cluster (N nodes, N-1 instances) then you will always have a free node to start on. Apply the OS patches and reboot if necessary. Failover and repeat as necessary until all nodes are done. OS patches are never cluster aware so everything is in the context of the local node. If you don’t have a free node, say with a two-node, two instance system, then I recommend clearing one node by moving the SQL instances. Always OS patch a node when it is “empty”. Don’t forget to move the Cluster and MSTDC groups too.

SQL 2000 Service Packs and Hotfixes are always cluster-wide for all components. Always start on the node hosting the instance you want to patch. SQL 2000 patches (both Service Packs and Hotfixes) only work on one instance at a time. Pay careful attention to the message boxes indicating completion. They may tell you to reboot the remote node(s). Always reboot when the installer tells you to. The system may be unstable or not failover properly if you do not.

SQL 2005 is a lot trickier. Only the Database Engine and Analysis services are cluster-wide. Everything else is local to the node. So, when you run the installer you may get different options on what to upgrade. If there are one or more instances on the node you execute the update on, you can update those instances. Otherwise you get to update just the local components. Note that this is new for SQL 2005. Before, you had to patch each instance separately. You can update the non-clustered components at the same time you update the cluster-aware components, but only for components on the local node. This is NOT clearly indicated by installer. You must go to each node and update SSIS, workstation components, Notification Services and Reporting Services. Of course, you shouldn’t be running Reporting Services on a failover cluster, but that is another discussion. Fortunately, the installer does have some smarts and will show you what has already been updated so you don’t have to re-run the update for that component. This makes it safe to run the updater if you lose track or have inherited a system where you are not sure what components are at what revision level. Service Packs and Hotfixes both work this same way. And yes, you need to update client tools on all cluster nodes and your workstations. Again, pay attention to the closing messages and reboot when told. Bad Things™ will happen if you don’t.

To Cluster or Not to Cluster, That SSIS the Question

Geoff N. Hiten — Thu, 04 Oct 2007 20:58:55 GMT

Clustering and SSIS have a stormy relationship at best. What you think you get after installing SQL and SSIS on a cluster is not always what you expected. For SQL 2000, the installer was cluster-aware on every component. Tools, code, everything got loaded on all nodes. Now, we aren't so lucky. Only SQL and SSAS install cluster-wide (or at least to all specified nodes). Client tools and most importantly SSIS only install to the local node. In practice, this often means a successful cluster install can leave a DBA with only one node capable of running SSIS packages. What is worse is that the first time this gets noticed is when a production system fails over and SSIS is nowhere to be found. So the first good reason to cluster SSIS is to make sure it actually works on all nodes correctly.

Another good reason to cluster SSIS is if any of your SSIS packages use a clustered resource such as a file share on one of the SQL data drives. You either have to create a clustered file share OR cluster SSIS. I prefer clustering SSIS since it keeps things simpler from a developer and a DBA perspective. Plus, the SQL installer tends to dislike anything "extra" on the core SQL cluster resources.

Microsoft warns that clustering SSIS is not recommended and not supported, but then tells exactly how to do it. Personally, I think they just didn't have time to build this logic into the installer. I have had no problems running clustered SSIS. I have had several issues running SSIS on a cluster without clustering SSIS itself. Either way, here is the link to the TechNet article:

http://technet.microsoft.com/en-us/library/ms345193.aspx

There is one warning about clustering SSIS that I do need to emphasize. Since the SSIS install routines don't support clustering directly, neither do the Service Pack or Hotfix installers. You have to run them independently on each node, preferably with the SSIS service active on that node. This does increase the downtime for applying updates. The alternative is to "uncluster" SSIS, apply the updates, then "recluster" SSIS. Dunno about you, but I just had an ugly flashback to SQL 7.0.

SQL2008 Clustering Changes

Geoff N. Hiten — Wed, 26 Sep 2007 14:25:25 GMT

Last week I stopped by the SQL Lounge at the PASS summit in Denver. I met a Microsoft employee named Max Verun, whose official title is Program Manager - Manageability and Servicing Platform. Translated into ordinary geek-speak, he is in charge of Clustering and Cluster Installation. This was a golden opportunity for me. I spent the next hour or so listening to Max and talking with him about where SQL wants to go with clustering. Since none of that conversation is under NDA, I thought I would share some of the highlights.

Clustering will remain a High Availability failover technology. While I suspect something like what the other guys call "clustering" is on Microsoft's radar, I doubt very much it will be called clustering when it finally surfaces. This isn't any inside knowledge, it is speculation that the dev team is aware of the competition and is not stupid.

Clustering will have some name changes. The "Virtual Server" nomenclature will go away. It is too confusing, especially now that Microsoft has gotten serious about platform Virtualization. So we will just be talking about Clustered SQL Instances in the future. I hope this change is more successful than trying to forget the "Active/Passive..." terminology.

The actual guts of clustering will remain largely unchanged. The failover detection and timeouts work well in most cases so that stays the same. There may be some technical changes to the SQL Cluster Resource DLL to fix some minor issues, but the actual workings of the SQL Clustering won't change.

The big changes in clustering come on the setup side. The most important change is the division of the current installer into two pieces. These two parts are are Setup and Configuration. SQL Cluster install will mirror Windows Cluster Install. With WIndows, you install the base bits including the clustering parts, then configure a cluster. SQL will adopt the same approach. The config tool creates a single-node clustered SQL instance. You then add new nodes to the clustered instance. This is a lot like WIndows Clustering, except that you can't do the SQL clustered install until the WIndows Cluster is built. The config tool will build an INI file so you can clone the installation for the second and ssubsequent nodes.

The ultimate goal here is to be able to sysprep cluster nodes and ship them pre-configured from a system builder. That feature may or may not make it into SQL 2008 due to several other dependencies, but we will get some foundation pieces that should make it easier to install a cluster and ultimately lead up to that possibility

One interesting side effect of splitting the installation functions is eliminating the current dependency on pre-qualifying hardware at the vendor level. Now, the SQL cluster configuration tool will determine if the hardware supports the desired configuration. This should lead to some interesting configurations and lots opportunities for those of us who rescue people from their own cluster follies.

Microsoft is adding iSCSI support for clustering. According to Max, their tests have been good as far as performance and stability. My experience so far with iSCSI and SQL has not been so good. I suspect the difference is due to Microsoft not purchasing their iSCSI equipment from Billy Bob's bargain computer hardware and pool cleaning service. Maybe they will have good enough tests in their tool, but I still don't trust iSCSI on a cluster.

One feature Max seemed very proud of is support for up to 64 nodes in a cluster. I guess 8 nodes isn't confusing enough.

I don't have any time frames for when we will be seeing these changes, whether it is in CTPs or even in the SQL 2008 release. I do know this is the direction the SQL Clustering team is thinking and working, but until the next release ships, nothing is locked in.