VSS Better Than Backups?

Thu Jan 31, 2008 by Travis Laborde

So, I've been dealing with an infrastructure team that didn't have backups of a web server recently… We were told "You don't need backups - that's what you have VSS for, right?"

Hmmm…

Maybe its just because I came up on the "operations" side of IT, and the backups were my highest priority, but it took me a while to realize they were serious. As in "we're gonna argue about this for a while." For better or worse, the infrastructure guys carry a lot of clout at the company I work for now, so I couldn't just scowl at them and tell them how stupid this sounded to me. I had to try and reason about something that just seemed obvious to me.

One thing I've learned is that anything that seems "obvious" to me probably means I don't understand the whole thing :)

I thought I had a perfect, easy to understand example of why that idea doesn't work - config files. We deploy these apps with certain things configurable after the fact, and some other support type person can change these settings without our knowledge, I explained. Our copy in VSS would be unaware of any edits that have gone in since the app was deployed.

No dice. The infrastructure guys were even pushy enough to get the development boss to start wondering if perhaps we should make sure that whenever anyone updates the config files those should be checked into VSS. Forget the fact that different deployments might have different configs, yet only one VSS. "We can make a subdirectory for each one."

I just refused to roll over and die on this one, which got me some nasty comments after the meeting. We did hammer out a tentative "in a catastrophic failure you can beg and plead and we will grab stuff from tape for you - but you had better first help yourself by having copies of your files in email or subdirectories or something."

So please tell me - Am I just wrong on this? Is my own past experience of being responsible for "the backups" making me want to place too much burden on the infrastructure team? After all, this is a different company, with different priorities and challenges, etc.

Thanks!
Travis

Legacy Comments


Mike Swaim 2008-01-31	re: VSS Better Than Backups? I can see their point. Ideally, all of your deployments should be reproducable. As far as I know, we don't back up our production web servers, either. (In our division, at least. I can't speak for the servers controlled by other departments, or by IS.) Also, backing up a web server's not as easy as backing up normal servers, because they're often not part of the same domain as the rest. (Our web server lives in a DMZ, firewalled from the outside world, and our internal network. Doing automated backups would involve setting up special users, and possibly opening ports.) Could you get by with just keeping copies of your installers and config files on a network drive? That's what I've done in a previous life.

Travis Laborde 2008-01-31	re: VSS Better Than Backups? Mike, let me re-iterate: We're talking about configs that have been changed since deployment. So, we've deployed an app, it's been in use for a long while now, the config has been edited over time, etc. All without my input or knowledge even. Now the server dies. Sure I can re-deploy easily once a new server is up, but... what about those files that have been edited since I deployed?

ACALVETT 2008-01-31	re: VSS Better Than Backups? Well, i hope someone is backing up VSS... :D I agree with you. What happens if you have invoked DR and gone off site. VSS may not be available (yet or at all). Even if you then bring back VSS you have had to increase down time because you had to get VSS running before you start recovery. As you say you also get those little changes that then get lost but those are the config files, what about those special application specific OS changes you may have had to deploy over time and so on. Yes, backing up servers in the DMZ does increase the overhead but it is crucial to do so and its not difficult to do it in a secure manner.

i48998 2008-01-31	re: VSS Better Than Backups? Change Management, Change Management, Change Management. You don't change configs on any files without change management. VSS or Harvest is only good if you comply with the change management policies. It doesn't matter how big or small your company is, following change management guidelines are important for all technicians.

Travis Laborde 2008-02-01	re: VSS Better Than Backups? Wow, I must admit that I'm surprised that anyone agrees with the need to use VSS on these files. See what I mean when I say that when something is obvious it might mean you just don't understand? :) Still, I can't help but wonder if anyone is really seeing my point. Maybe my point is I'm just wrong and too dumb to admit it yet :) Let's say I license my app out to different clients. I deploy it for them to their servers, and it "just works." The app has config files that they can edit to change certain preferences. Is it reasonable to expect that they are going to send me copies of these files whenever they make changes, and expect me to hold those in VSS for them? No, of course not. They would be expected to have backups of these things right? If they had a catastrophic failure, I can see myself re-deploying for them, but... Wouldn't it be expected that I don't have their most recent preference updates? I just don't get how this isn't obvious. But I do want to learn and grow from this if I'm wrong.

Mark Harrison 2008-02-02	re: VSS Better Than Backups? I would argue that the speed of recovery is the biggest pro in favor of the backup recovery. You can just pop the tape in and walk away. Config files can be site specific and if they are not backed up then you may end up having to re-create the site specific config files. They probably should be in VSS (or better yet, something less flaky! :-) Usually my config files contain nothing more than a connection string to a database which contains the real system configuration. Still, while the infrastructure guys score a minor point, when the arguments are over, they are responsible for continuity. If they haven't done their best to cover their own backsides they are the most likely candidates to have their backsides kicked!

John Voost 2008-02-02	re: VSS Better Than Backups? If you have an upload section in your website, VSS won't work. You have at least to backup that section.

Malcolm 2008-02-03	re: VSS Better Than Backups? I totally agree with Mark and Andrew. The infrastructure people need to restore systems in the event of a disaster so they should have backups of everything that is necessary so the recovery time is minimised, period. Relying on recreating machines from source control is just idiotic and lazy. I agree with having change control processes but they are there to ensure predictability, auditability, accountability and reliable repetition etc. - they are not effective as a disaster recovery method as speed of recovery is the primary goal.

Terry 2008-02-03	re: VSS Better Than Backups? The process we use is to have the developers keep the app code in VSS. The build engineer pulls and compiles from that VSS. He then makes any necessary environment-specific config changes and places the resulting files and build instructions into a separate VSS that is only used for deployments to staging and production. If a disaster occurs, the platform team is responsible for rebuilding the boxes to a point where the app can be reinstalled. Additionally, nothing in production is changed without going through the Change Management process. All of the servers are also backed up.

Jay 2008-02-03	re: VSS Better Than Backups? A DR strategy should be based on the business's availability requirements and acceptable outage aka service level agreements. If you don’t have SLA’s get them then you will be able to argue your case quite easily, get infrastructure to do a dry restore and if they can’t meet the SLA then the DR solution needs to be re-worked (you will then get what you want) However on the other hand if the business is happy to save a few dollars on stand by disc at the cost of having to restore from tape and VSS this is unfortunately what you will have to live with (at least until the business has to live with a whole day's outage from catastrophic failure), just make sure that business is very aware of the decision's that it has made and they are well documented. In the perfect world (one of unlimited budgets) you are right backup everything to ensure that fast recovery! There is nothing worse than being up till three in the morning running an OS install, patching, tracking down code etc. Still living in that perfect world I would say that you should have your DR on tape (off site), server backups for quick restore on disc, VSS for all of your code backup's and a configuration management DB for all of you config files. All server / system changes should be funnelled through change management framework, to help ensure that your configuration and VSS stores get updated. Check out ITIL and COBIT frameworks.

Alexander 2008-03-27	re: VSS Better Than Backups? As a developer of 12 years, I would say you are both right, but would side on your IT Ops guys. I say keep the "control" in "source control" (VSS). (I presume they keep a delta backup of VSS over some period of time, and say an entire backup every month or so. DVDs are cheap.) Do you have a staging server? Changing config/XML files without a QA/staging server is dangerous, not to mention stupid. How many times have you, say, fat fingered an Oracle connection string? You don't want to find that out when the web app restarts after you change Web.config (or whatever config file you are using). And so this type of deployment really needs source control involved. So, yes, check in your config changes, even if it's the support guy. So I believe "Our copy in VSS would be unaware of any edits that have gone in since the app was deployed." is actually unacceptable. As for getting a fast backup ("Like, yesterday dude.") of the "last known config", or code, or whatever, that ain't a developer problem. That's an Ops problem. Also, do you version "configs" to "binaries"? That may be something you need to do and have seen at places I have worked. Cheers.

Travis Laborde 2008-03-28	re: VSS Better Than Backups? You know, I think that by using the word "config" I've made the waters a little more muddy than I should have. What I'm talking about here are "Preferences." Like many apps that store preferences in INI files or registry entries. These are the "configuration" files I was talking about. The whole idea is that these are to be set/changed at will without needing developer support. Think about when your PC crashes and you have to re-install everything. You have to re-set your Home page, your favorite font in Outlook, whatever. Guess what? People hate that, and software is available to "save" those settings. Or you can make an "image" ahead of time etc. In other words - BACKUPS. Now I'm extending that to a web-based application - but not a public internet site - an internal company run site. If the project budget doesn't allow time for GUI/Database development for certain preferences that we know will change over time, this sort of thing is used. I call them "config" files because IIS protects those files by default. Having said all that, I've decided to build a GUI/Database solution for all of these needs - that serves "all" my apps so that each app doesnt have to develop that functionality on it's own. Problem solved. But I still think its ludicrous to have to argue for backups.