Derrick Leggett Blog

Ramblings, grumblings, and other nonintelligible nonsense about SQL Server.

<b><i>My server drives are playing hide-and-seek!!!!</b></i>

While installing the EMC SnapView software, we had a problem with our LUNs not “anchoring”.  We clone the drives, then use mountvol to mount them.  We then change the LUNs to the appropriate drives for our SQL Server instances.  The problem was that the drives would move everytime we rebooted the machine, which is not something you really want happening on a SQL Server installation (or any other type of installation I can think of at the moment).

After EMC researched the issue for several days and discovered they really had no idea what they were doing, I found out that the problem was the registry.  Under HKEY_LOCAL_MACHINE\SYSTEM\MountedDevices\, there are keys for the volume name of a drive and the volume letter of a drive.  For example: 

 VolumeName:  \??\Volume{6f2f7016-57fa-11d8-af91-806e6f6e6963}
 VolumeLetter:  c:
 Key:  0x1DAD3BC2007E000000000000

The problem we were experiencing was caused by Windows changing the drive letters, but not the Key.  Because of this, they VolumeName would still be pointed to the old Key, which was for a different drive letter.  Every time you rebooted, it would resolve back to that old drive.  I fixed the problem by hacking the registry basically.  I deleted the old keys completely, and made sure the Keys matched for the VolumeName and VolumeLetter. 

We have not had the issue with drives moving since this.  Unfortunately though, since I dont' want to take the chance of this happening again when I set up new clones or change things, I need to build the logic into my script to resolve this.  Here is what I'm planning.

Before I change the drive letters I will run:

EXEC master..xp_instance_regread 'HKEY_LOCAL_MACHINE','SYSTEM\MountedDevices','\DosDevices\D:'
EXEC master..xp_instance_regread 'HKEY_LOCAL_MACHINE','SYSTEM\MountedDevices','\??\Volume{6f2f7013-57fa-11d8-af91-806e6f6e6963}'

for example to get the registry keys for both the drive and volume names of the cloned drives.  I will then change the drive letters and get the registry keys for the new drive and volume names.  If there is a mismatch, I will delete the old keys using xp_instance_regdeletekey.  I will then use xp_instance_regwrite to match the drive and volume keys to each other.

We're doing this in SQL because our entire process is already written in SQL and I don't want to introduce anything else into the environment at this stage in the game.

Does anyone have a script written that already does this so I'm not wasting my time?  :)

When I am done, I plan on posting the entire process on my website so somebody else doesn't pay EMC $50k to come in and try to figure out how to get this to work.  Since we wrote all the scripts, EMC hasn't really done anything for us.  They actually had to research how the command line tools worked when they came here, which I just foolishly assumed the experts would already know.  That's kind of like a DBA having to research how the BACKUP command works and charging $50k for it.  They could have given me a book with the examples and we would be further ahead.

Favorite words used: 2 (foolishly, assumed) --Not much of a blog from this perspective.  :)

Mean level (1-10):  2 (I've ranted enough about EMC already.)

Education level (1-10):  7 (It's been a learning experience for me anyway.  Hopefully, someone can pour some knowledge on me.)

Entertainment level (1-10): -5 (If this entertained you, ummmmm I don't really know what to say.)


Legacy Comments

re: <b><i>My server drives are playing hide-and-seek!!!!</b></i>
you guys have some billing problems....

When you hire someone to come out and fix something, if they don't fix it, they don't get paid. I don't care how many meal bills and hotel bills the providing company has to eat...

Derrick Leggett
re: <b><i>My server drives are playing hide-and-seek!!!!</b></i>
We haven't paid them. That's what they charged for the implementation to come out and set this great environment up for us. We have no intention of paying for a failed implementation.

re: <b><i>My server drives are playing hide-and-seek!!!!</b></i>
Glad you got a solution. Yes EMC would try to make the big bucks. The problem is Windows brain deadness though going willy nilly with drive letters. Name ANY other OS used in a data center that changes around drive letters. I work with Windows and it drive me nuts sometimes how not ready for the enterprise it really is. This is a perfect example.

Derrick Leggett
re: <b><i>My server drives are playing hide-and-seek!!!!</b></i>
Actually the problem was how the EMC software interacts with Windows. It wasn't changing the drive letters appropriately in the registry. I created a procedure to fix the registry entries whenever we run their program. Everything works fine now. Since Windows runs many of the largest systems in the world, I'm not too worried about it handling my enterprise. You must have quite the enterprise if it's not ready for yours.

re: <b><i>My server drives are playing hide-and-seek!!!!</b></i>
Why would you even start arguing Windows runs the "Biggest" systems in the world? Sure, if your looking at a stupid fileserver, you might think that but NONE of the Fortune 500 backend "Enterprise" applications run on Windows. None of the Telecom or Finacial data transaction systems either. Pretty much, anywhere where reliability and uptime are essential, Windows is playing in the middle tier as a client interfacing system only. It's like a call center for support, you don't put the smartest people out front, you use cheap idiots to deal with the little stuff... this is where Windows plays in the corporate world, so cope. Two little things about EMC, their software has ALWAYS sucked (That's why they are a Storage Hardware company, and not a Veritas like entity), and you will be paying that bill. If their is one thing EMC does extremely well it is litigation and coersion... non-payment of bills will somehow cause support contracts to be discontinued... Imagine that!

Derrick Leggett
Regarding Why?
You should learn to read. I said "many of the largest" not the "Biggest". Are you a laid-off Oracle DBA because you seem to have a big chip on your shoulder. Get over it.

Also, we don't have a case of non-payment here. We have a case of being refunded for unsuccessful implementation and software we can't use. And, we have already been successful in that. IMAGINE THAT!!!

re: <b><i>My server drives are playing hide-and-seek!!!!</b></i>
Hi Derrick,

I trying to understand the flow here. Typically when we clone the LUNs using SnapView, we cannot mount them on the same server that the source LUNs are mounted. It is usually mounted on some development servers or a backup server.

re: <b><i>My server drives are playing hide-and-seek!!!!</b></i>
Hello Derrick, I will say this, if defiantly take an individual with a “very large pair” to make a chunk of SQL to alter this section in the registry (This is a compliment by the way).

I did a google search for “MountedDevices” and found this thread and I am having a similar issue as you have had.

My issue is I have two computers connected to a Dell 660F I am being forced to use Dell’s “wonderful” storage consolidation software to assign the SAN volumes to a windoz cluster. A couple months back one of the machines had to be powered down and when we just powered this machine backup one of the local volumes failed “D:”. And YEP the all the SAN volumes are nowhere to be found.

I am logged into the SAN from the machine that has missing drive and the I can see the drives online but of course the drive letters themselves are nowhere to be found.

I guess my question would be just as the person above was asking. What was the relationship in this section between the “\DosDevice\x:” entry to the “\\??\Volume\{xxx” entry? If I remember correctly I had to do something really nasty like reconstructing these entries to “align” with the settings in the storage consolidation software.

Also I remember using a couple command line utils that did “Disk Management” functions (i.e. setting partitions sizes and drive letters). Have you ever hear of these?

Signed - Michael “almost ready to eat a bullet” IT guy