SQL Server Stuff

Lessons from a SAN Failure

Bill Graziano — Fri, 26 Feb 2010 14:29:31 GMT

At 1:10AM Sunday morning the main SAN at one of my clients suffered a “partial” failure. Partial means that the SAN was still online and functioning but the LUNs attached to our two main SQL Servers “failed”. Failed means that SQL Server wouldn’t start and the MDF and LDF files mostly showed a zero file size. But they were online and responding and most other LUNs were available.

I’m not sure how SANs know to fail at 1AM on a Saturday night but they seem to. From a personal standpoint this worked out poorly: I was out with friends and after more than a few drinks. From a work standpoint this was about the best time to fail you could imagine. Everything was running well before Monday morning. But it was a long, long Sunday. I started tipsy, got tired and ended up hung over later in the day. Note to self: Try not to go out drinking right before the SAN fails.

This caught us at an interesting time. We’re in the process of migrating to an entirely new set of servers so some things were partially moved. This made it difficult to follow our procedures as cleanly as we’d like. The benefit was that we had much better documentation of everything on the server. I would encourage everyone to really think through the process of implementing your DR plan and document as much as possible. Following a checklist is much easier than trying to remember at night under pressure in a hurry after a few drinks.

I had a series of estimates on how long things would take. They were accurate for any single server failure. They weren’t accurate for a SAN failure that took two servers down. This wasn’t bad but we should have communicated better.

Don’t forget how many things are outside the database. Logins, linked servers, DTS packages (yikes!), jobs, service broker, DTC (especially DTC), database triggers and any objects in the master database are all things you need backed up. We’d done a decent job on this and didn’t find significant problems here. That said this still took a lot of time. There were many annoyances as a result of this. Small settings like a login’s default database had a big impact on whether an application could run. This is probably the single biggest area of concern when looking to recreate a server. I’d encourage everyone to go through every single node of SSMS and look for user created objects or settings outside the database.

Script out your logins with the proper SID and already encrypted passwords and keep it updated. This makes life so much easier. I used an approach based on KB246133 that worked well. I’ll get my scripts posted over the next few days.

The disaster can cause your DR process to fail in unexpected ways. We have a job that scripts out all logins and role memberships and writes it to a file. This runs on the DR server and pulls from the production server. Upon opening the file I found that the contents were a “server not found” error. Fortunately we had other copies and didn’t need to try and restore the master database. This now runs on the production server and pushes the script to the DR site. Soon we’ll get it pushed to our version control software.

One of the biggest challenges is keeping your DR resources up to date. Any server change (new linked server, new SQL Server Agent job, etc.) means that your DR plan (and scripts) is out of date. It helps to automate the generation of these resources if possible.

Take time now to test your database restore process. We test ours quarterly. If you have a large database I’d also encourage you to invest in a compressed backup solution. Restoring backups was the single larger consumer of time during our recovery. And yes, there’s a database mirroring solution planned in our new architecture.

I didn’t have much involvement in things outside SQL Server but this caused many, many things to change in our environment. Many applications today aren’t just executables or web sites. They are a combination of those plus network infrastructure, reports, network ports, IP addresses, DTS and SSIS packages, batch systems and many other things. These all needed a little bit of attention to make sure they were functioning properly.

Profiler turned out to be a handy tool. I started a trace for failed logins and kept that running. That let me fix a number of problems before people were able to report them. I also ran traces to capture exceptions. This helped identify problems with linked servers.

Overall the thing that gave me the most problem was linked servers. In order for a linked server to function properly you need to be pointed to the right server, have the proper login information, have the network routes available and have MSDTC configured properly. We have a lot of linked servers and this created many failure points. Some of the older linked servers used IP addresses and not DNS names. This meant we had to go in and touch all those linked servers when the servers moved.

SQL Server Blogs I Read

Bill Graziano — Fri, 05 Feb 2010 13:16:43 GMT

Peter Larsson recently scrubbed his laptop but didn’t bring over his favorites and made a plea for interesting SQL Server related blogs. My reading list of SQL Server blogs and sites is up to 83 sites in Google Reader and I thought I’d share my list in hopes of finding more. Here’s the link to the OMPL file that you can import into a feed reader: SQL Server Blogs OPML file.

Below is the complete list of SQL Server blogs I read regularly. If you’re writing about SQL Server and not on that list please use the contact me form and let me know about your blog. I’d like to get you added to the list.

Bill subscriptions in Google Reader

[RSS] Allen Kinsel - SQL DBA
[RSS] Andrew Fryer's Blog
[RSS] Bill Ramos on SQL Server
[RSS] Bob Beauchemin's Blog
[RSS] Brent Ozar - SQL Server DBA
[RSS] Carpe Datum
[RSS] Chris Shaw's Weblog
[RSS] Conor Cunningham's Blog
[RSS] Conor vs. SQL
[RSS] Dan's Blog
[RSS] Data Dude
[RSS] Databases, Structures, and the Damn Data Itself
[RSS] DataWorks WebLog
[RSS] Developer hearted / Relational minded
[RSS] Direct Reports (Brian Welcker's Weblog)
[RSS] Esoteric
[RSS] Euan Garden's BLOG
[RSS] Glenn's space
[RSS] Henk's tech blog
[RSS] In Recovery...
[RSS] Jeremiah Peschka, SQL Server Developer
[RSS] Jimmy May, Aspiring Geek: SQL Server Performance, Best Practices, Productivity, etc.
[RSS] Kalen Delaney
[RSS] Kendal Van Dyke
[RSS] Kevin E. Kline
[RSS] Kimberly L. Tripp: Improving *my* SQL skills through your questions!
[RSS] Laurentiu Cristofor's blog
[RSS] Microsoft SQL Server Development Customer Advisory Team
[RSS] Microsoft SQL Server Release Services
[RSS] Microsoft SQL Server Support Blog
[RSS] Midnight DBA
[RSS] Most Recent KBs for SQL Server 2005
[RSS] MSDN: SQL Server
[RSS] MSSQLTips - Latest SQL Server Tips
[RSS] PASS - The Professional Association for SQL Server
[RSS] Patrick Sirr's Blog
[RSS] Pragmatic Works
[RSS] PSS SQL Server Engineers
[RSS] rusanu.com
[RSS] Slava Oks's WebLog
[RSS] SQL Fool
[RSS] SQL Programmability & API Development Team Blog
[RSS] SQL Server
[RSS] SQL Server 2005: CLR Integration
[RSS] SQL Server Blog by Jason Massie
[RSS] SQL Server Community Blogs
[RSS] SQL Server Engine Tips
[RSS] SQL Server Performance
[RSS] SQL Server Policy-Based Management
[RSS] SQL Server Premier Field Engineer Blog
[RSS] SQL Server Storage Engine
[RSS] SQLAndy
[RSS] SQLBatman.com
[RSS] SQLblog.com - The SQL Server blog spot on the web
[RSS] SQLCAT.com
[RSS] SQLJunkies Blogs
[RSS] SQLServerCentral.com
[RSS] SQLServerCentral.com
[RSS] SQLServerCentral.com
[RSS] SQLServerCentral.com
[RSS] SQLServerPedia
[RSS] SQLServerTimes2
[RSS] SQLTeam.com Feed
[RSS] SQLTeam.com Weblogs
[RSS] SQLvariations
[RSS] SSIS Talk
[RSS] StraightPath Solutions SQL Blog
[RSS] The Cranky DBA
[RSS] The Data Platform Insider
[RSS] The Kirkapedia
[RSS] The SQL Doctor is In (Real In)
[RSS] The SQL UPDATE Statement
[RSS] Tips, Tricks, and Advice from the SQL Server Query Optimization Team
[RSS] Tips, Tricks, and Advice from the SQL Server Query Processing Team
[RSS] (title unknown)
[RSS] Tony Rogerson's ramblings on SQL Server
[RSS] Transact-SQL Blog
[RSS] VSTS DB Team Blog
[RSS] Ward Pond's SQL Server blog
[RSS] WebbTech Solutions
[RSS] Write Ahead Blog
[RSS] SQLServerCentral.com Articles

SQL Server Links - 19 January 2010

Bill Graziano — Tue, 19 Jan 2010 15:02:09 GMT

The Bit Bucket (Greg Low): IDisposable : Stored Procedures - Time for a real contract? [sqlblog.com/blogs/greg_low/]

Interesting case of watching log file growth during a perf test [www.sqlskills.com/BLOGS/PAUL/]

Improvement in minimizing lockhash key collisions in SQL Server 2008R2 and its impact on concurrency [blogs.msdn.com/sqlserverstorageengine/]

Microsoft and Intel Push One Million iSCSI IOPS [blog.fosketts.net]

The Truth About Hyper-V Memory Overcommit [www.vcritical.com]

A Few Useful Queries for SQL Server 2008 Integrated Full Text Search (iFTS) [glennberrysqlperformance.spaces.live.com]

SYS2 DMVs on CodePlex [sqlblog.com/blogs/davide_mauri/]

Intel Nehalem-EX and Database Performance [glennberrysqlperformance.spaces.live.com]

Iron and Cloud [www.sqlmag.com] - Interesting editorial about SQLAzure

PASS Processes and Results posted [www.allenkinsel.com]

Free eBook on SQL Server Maintenance Plans Now Available [www.sqlservercentral.com/blogs/aloha_dba/]

SQL Server Waits and Queues [www.sqlservercentral.com/blogs/pearlknows/]

How to increase SQL Database Full Backup speed using compression and Solid State Disks [henkvandervalk.com] - Great post on backup performance.

10 Hot Scalability Links for January 13, 2010 [highscalability.com]

SQL Server Links – 8 Dec 2009

Bill Graziano — Tue, 08 Dec 2009 14:50:48 GMT

I’ve been posting links to interesting blog posts in the SQLTeam.com Newsletter for over a year now. I going to try posting them here also.

Oh what job was it? A procedure to search your jobs for a particular string [www.sqlservercentral.com/blogs/james_howards_sql_blog/]

One Man's Trash... [www.straightpathsql.com] - I really liked this post. A great discussion about the importance of attitude at work.

Bruno Terkaly - Developer Evangelist - bterkaly@microsoft.com : Deploying Azure apps in 1000 words or less [blogs.msdn.com/brunoterkaly/]

Fundamentals of Storage Systems – RAID, An Introduction [www.sqlservercentral.com/blogs/sqlmanofmystery/]

Fundamentals of Storage Systems – Disk Controllers, Host Bus Adapters, and Interfaces [www.sqlservercentral.com/blogs/sqlmanofmystery/]

Troubleshooting Microsoft SQL Server : Transactional Replication Conversations [blogs.msdn.com/chrissk/] - Very detailed article on troubleshooting replication

Report Builder and Firewalls [blogs.msdn.com/psssql/]

How Do You Use SQL Server [facility9.com]

Using Resource Monitor In Windows Server 2008 to Look at I/O Performance for SQL Server 2008 [glennberrysqlperformance.spaces.live.com]

SQL Server 2008 SP1 CU3 and SQL Server 2008 RTM CU6 Released

Bill Graziano — Wed, 22 Jul 2009 03:45:37 GMT

Microsoft continues to release cumulative updates roughly every two months. These two came right on schedule. As always I’ve updated my build list in the SQL Server Version article on SQLTeam.com.

Cost to Compile a Query

Bill Graziano — Wed, 24 Jun 2009 11:49:09 GMT

It’s pretty easy to determine the CPU and disk resources that a given query or stored procedure will use. It’s more difficult to determine the resources that were used to compile that query plan. You can start by looking at sys.dm_exec_cached_plans. It has a column called “size_in_bytes” that will tell you how much memory the query plan is using.

If you generate an XML query plan through SSMS or Profiler you can get some additional information. The XML plan includes this snippet:

If you review the schema for the XML query plan you can find a little bit about these values.

CachedPlanSize is in kilobytes.
CompileTime is in milliseconds (1/1000th of a second) and was introduced in SQL Server 2005 SP2.
CompileCPU is in milliseconds (1/1000th of a second) and was introduced in SQL Server 2005 SP2.
CompileMemory is in kilobytes and was introduced in SQL Server 2005 SP2.

There are also entries for MemoryGrant (KB) and DegreeOfParallelism but I haven’t used those as much.

SQL Server sites I visit

Bill Graziano — Fri, 22 May 2009 12:16:31 GMT

Google Reader just introduced functionality to create “bundles” of feeds. I subscribe to roughly 60 SQL Server blogs that I read on a regular basis. You can download an OPML file of these.

More interestingly you can just subscribe to the individual RSS feeds of these sites. If you subscribe to it in Google Reader it will create create a new tag labeled “SQL Server feeds” which is what I named my bundle. You will be subscribed to each of the individual feeds in my bundle. If you are already subscribed to one of the feeds it will just add this tag to the feed – you won’t be subscribed twice. I’m not sure how readers other than Google Reader will handle this.

Why am I being forced to learn PowerShell?

Bill Graziano — Wed, 13 May 2009 00:41:24 GMT

I spent an annoying afternoon with SQL Server 2008 today. When we installed it on our cluster it created a job called “syspolicy_purge_history”. I assume it does this in most installations but I haven’t checked.

The job was generating an error every time it ran. A quick Google search turned up an article on the problems with syspolicy_purge_history and clusters. Now all I needed was to figure out the PowerShell syntax to correctly refer to this instance.

PowerShell? Oh yes. They couldn’t just call a stored procedure. Oh no. Rather than giving me the choice to learn and then use PowerShell I’m being forced to wade through the syntax while troubleshooting. I’m not exactly happy with that decision right now. I thought this software was supposed to make me productive? Was PowerShell really needed for this?

The KB article does a good job explaining what to do if you’re running on a named instance but there wasn’t an explanation on what to do if you’re not running on a named instance. Technically I guess our instance was named MSSQLSERVER but that didn’t work. It turns out you need to add the keyword DEFAULT as I did below.

(Get-Item SQLSERVER:\SQLPolicy\DBCLUSTER1\DEFAULT).EraseSystemHealthPhantomRecords()

I really hope this job truly needed PowerShell.

Interviewed on RunAs Radio

Bill Graziano — Mon, 13 Apr 2009 12:55:59 GMT

I was recently interviewed on RunAs Radio. We spent quite a bit of time talking about performance tuning. I’m spending quite a bit of time lately looking at how things get into and out of the query cache and we touched on that a bit. We also spent some time talking about ClearTrace and what it can do for you. Take a listen and try not to laugh at a “voice made for print”. I was also interviewed on Greg Low’s SQL Down Under a few years back. That show is in the SQL Down Under archive (show #13).

ClearTrace updated to support SQL Server 2008 trace files

Bill Graziano — Sun, 29 Mar 2009 17:07:01 GMT

It was a long time coming but I finally updated ClearTrace to read SQL Server 2008 trace files. This requires SQL Server 2008 to be installed on the computer where ClearTrace is running. ClearTrace will process traces created in SQL Server 2000, SQL Server 2005 and SQL Server 2008.

ClearTrace is the tool I wrote to make performance tuning easier. Many of my client engagements are to make SQL Server faster and ClearTrace is the tool I use to do that. ClearTrace will read a series of trace files and aggregate the results so you can tune the queries that use the most resources. It’s available free of charge at the link above.

I also cleaned up a bunch of old code that would report errors and performance data back to a web service if you checked the proper boxes. Since I hadn’t looked at that data in over a year I decided not to capture it anymore. Any horrible errors that occur will be saved in a text file near the executable. You can post them to the ClearTrace Support Forum and I’ll try to figure out what’s going on.

Please let me know if you have any problems or how well the software works for you.