Security isn't exactly the most glamorous job in IT, but someone has to do it. After all, servers constantly face an onslaught of hazards, from viruses to worms to denial-of-service attacks, which can bring them -- and your business -- down in a flash.

Software manufacturers, especially Microsoft, aren't helping, finding vulnerabilities and sending out critical patches faster than security managers can deploy them. Which leaves IT departments in a quandary: update servers as soon as a critical patch comes out or wait until the patch is thoroughly tested? Choosing the former can cause applications critical to your business to stop running, but doing the latter, especially if you start falling behind, will leave your servers and network vulnerable to attack.

Whether an IT shop deploys its patches manually or via an automated patch management system, the key to not be buried under a virtual mountain of updates is to have a consistent and well-documented update process. In addition, standardized and hardened server configurations have help security managers keep up.

The onslaught of patches

In 2007, Jennifer Albornoz Mulligan , an analyst with Cambridge, Mass.-based Forrester Research, took a survey of IT security decision makers about how they deal with their patch management issues. She estimated that Microsoft released 49 critical patches for Windows servers in 2006. That was up from about 30 in 2005 and about 20 in 2004. That translates to about one critical patch per week, all of which needed to be evaluated, tested, and distributed by IT security managers and administrators. Many respondents complained that they were falling too far behind.

"Some were very frustrated because they hadn't finished the previous month's patching by the times the new ones came out," she says. "Many are just on a treadmill that won't stop, because they're just constantly behind and can't catch up."

Mulligan found that, on average, her respondents patched servers eight days after their companies' self-imposed patching deadline passed. One of the reasons why these companies fell behind was that 77 percent of the respondents either didn't have a testing environment or had one that only partially replicated their company's production environment.

Standardization helps

What surprised Mulligan the most was that those businesses who had servers that were built to standardized specs, including hardening steps like shutting down services, deleting unnecessary IDs, and turning off unused ports, installed patches much faster than respondents that didn't have that degree of control.

"One of the companies I talked to had a 48-hour patch deadline," she says. "They made it because they had very strict hardening and server configuration requirements."

Jonathan Hassell, president of the Scribner Technology Media Corporation and the author of a number of books on Windows, including the book Hardening Windows (Apress, 2005), thinks a company's patching policies and procedures really come down to priorities. "Something to remember: security is a goal.  It's not a state," he says. "There is no panacea. It's a process, with a lot of moving parts, dependencies, and cost."

Making the mountain into a molehill

So how do you dig yourself out from the pile of patches? Here are some suggestions:

  • Virtualization.  Because money is not always available to replicate the development environment, software like VMWare can help an administrator re-create a particular production environment on demand in order to test a patch. "Some of the most advanced clients I spoke with found it most helpful because it was cost-effective," says Mulligan.
  • Create and follow a documented security policy.  This doesn't just pertain to a regular patching schedule and testing procedure. Hardening servers either manually or via standardized configurations are also a plus. "If you do a good job hardening your systems, you've turned off the services you don't need," says Mulligan. "So if there's a patch for that service, it doesn't matter, since the service is turned off."
  • Use automated patching systems.  Some particularly hands-on IT managers will insist on patching servers by hand. Hassell feels that's a valid method, though he prefers automated tools. "It's not necessary to hand patch them if you spend the time to craft a quality updating policy in whatever software you're using."
  • Prioritize. Which applications are most critical to your business? Those may be the ones where you will want to do the most extensive patch testing. Internet-facing servers, for instance, may need to be patched first because they're the most vulnerable to attack.