SysAdmins creating software for SysAdmins.
The One Backup Mistake that Can Shut You Down
Organizations, even ones who consider themselves to have solid backup and recovery plans, often make one critical mistake that can render all of their careful planning moot – they fail to back up their infrastructure.
Most companies are good at backing up critical data. Fewer are good at doing full system backups so they can perform a bare metal recovery instead of spending hours/days reinstalling and patching an OS. Only a handful are backing up infrastructure data like that held on routers and switches. If your company has a disaster this could be a real problem.
We recently experienced a pain point in our test labs that is a great illustration of this. We had just moved a bunch of equipment and as a result a lot of the memory on the hardware got flashed. One of our engineers, Dave Kinney spent an entire morning trying to get routing set up on a test configuration and was banging his head (almost literally) against the wall. Everything looked fine. The routing tables were completed correctly. It SHOULD work. Finally after hours of wasted time he realized that one of the routers had iptables that were firewalling all traffic. A small version of Linux was running on that router in the background that he’d forgotten was even there.
That’s when Dave got inspired to make a backup of his config files for the routers. As he put it – “at two in the morning, when the servers are on fire, I don’t want to have to figure this out again.”
That’s really what it comes down to isn’t it? Being as prepared as possible for the unforeseeable. There will be plenty of issues you couldn’t have planned for, so why not do everything you can to free yourself up to deal with them when they happen?
Many devices in our organizations now use config files – and more and more those devices are allowing you to export those files for backup purposes. Think of things such as phones (especially VOIP), security systems, and network hardware.
What about devices that don’t have exportable configuration files? Do it the old fashioned way. Write it down, step-by-step. “To add a new user code to the alarm system, press X, then Y, then Z.” File it all, along with physical media backups of your config files, in a Disaster Recovery binder. Then make sure that binder gets to a safe or offsite. After all, it doesn’t do you any good to have a great DR plan if it gets burned up along with everything else.