In-Depth

Managing to Cope with Patches

• By Johanna Ambrosio
• February 1, 2005

Keeping up with a steady stream of patches to close security loopholes and upgrade apps has become time consuming and costly. In mid-2003, RKA Petroleum, a distributor in Romulous, Mich., went through hell when one of its most critical servers crashed after IT applied a patch to one of its most critical servers. The company had been using Microsoft's Software Update Services, a rudimentary patch manager, which recommended installing an update for Internet Information Services. However, the patch wound up breaking the server instead of fixing it, according to Jason Hittleman, vice president of information services at RKA.

The IT department found out after the fact the server did not require the patch and because the server did not have sufficient disk space, the patch began deleting files, Hittleman recalls. Services began crashing, and employees couldn't log onto the network.

After 15 hours on the phone with Microsoft, the server was still down and RKA wound up having to rebuild the machine. The firm also learned a valuable lesson, Hittleman says. 'It was a wake-up call. We were spending 80 person-hours a month and just barely keeping up with patches. When this happened, we knew there had to be a better way.'

As a result, the company investigated third-party software tools and is now using PatchLink's Update. 'It saves us $30,000 each year in personnel time; we got payback in two months,' Hittleman says.

Another common need for patching results from what Jim Hurley, an analyst with consultancy Aberdeen Group, calls infrastructure alignment. For example, an enterprise decides to deploy a new router but when it does, the new router breaks an application installed some years ago. It's the kind of thing that happens when a new patch in one app requires an update in another app to work correctly.

Witty was not funny
The security angle looms large, of course. Most companies patch to keep viruses and worms at bay, experts say. The malicious software problem is only getting worse-the numbers of viruses and worms released is rising steadily and the time between when a vulnerability is reported and the time rogue programmers release malware designed to exploit is getting shorter all the time.

The Witty worm, for instance, started circulating in March 2004, less than two days after the first public description of a loophole in Internet Security Systems' firewall software. The worm infected as many as 12,000 computers in an hour, according to a report from the Cooperative Association for Internet Data Analysis. The Code Red worm, which hit in January 2003, was even deadlier, CAIDA says, infecting more than 350,000 computers in about 14 hours.

Some software vendors release a steady stream of patches every week, and that exacerbates the problem of keeping up. Some weeks, for example, Microsoft releases as many as five updates. Vendors including Oracle and Microsoft have announced they are moving to quarterly update schedules, but they are in the minority.

The patch problem has gotten out of hand in many enterprises. With, say, even a couple dozen servers and a few hundred desktops, and an average of at least five applications per desktop, the complexity of applying patches escalates rapidly. This is particularly true if there are multiple supported desktop builds for different types of users in various departments, a common scenario in large organizations.

There's one underlying theme to all of these problems, Aberdeen's Hurley says. 'Information is not available' because the correct patch versions are not installed. 'People don't have what they need to get their jobs done.'

Enterprises now look to tackle the problem in a more holistic way. Many find themselves rediscovering the gospel of change management, a staple in the mainframe community that fell out of favor in a Web-focused environment that put a premium on getting an app quickly out the door.

Technology needs an upgrade too
In a perfect world, there would be an IT team of security, development and maintenance experts, responsible for evaluating patches to make sure they are necessary to combat security risks, increase functionality or provide some other benefit. The team would test the patches to make sure they didn't break anything on the production servers.

After a successful test, an automated tool would then push the patch out to all the machines that need it, and a reporting mechanism would attest to how many desktops or servers were successfully updated.

Most important, in this ideal world, patch management would be a piece of two larger processes: layered security and change management. Simply throwing technology at the problem of coping with patches isn't enough, experts say.

The majority of shops are either still patching manually or using Microsoft tools such as Systems Management Server or shareware and freeware. A small number of customers has purchased tools from third parties; at least a dozen are available. (See related story, 'On patch patrol, armed with third-party tools .')

Phebe Waterfield, an analyst with the Yankee Group, estimates that only about 10 percent of the largest firms have a comprehensive patch-management process. These large companies are the ones that really need a coherent approach because of the scale of the problem in a worldwide and distributed environment.

Still, she believes because of security issues and regulatory oversight in many sectors, the market will thrive. She predicts patch management revenue will grow to $300 million in 2008, up from $70 million in 2003. During this time, more customers will come to see patch management is most effectively handled actively and regularly. They will have both best-in-class patch management tools and a structured process for introducing patches into the organization. 'Companies really need to apply patches on a scheduled, routine basis and not as a reaction' to a security threat, Waterfield says.

The 'key determinant' to a successful patch-management program is 'organizational process and not just technology,' Aberdeen's Hurley says. It's the proven way to minimize disruption and ensure business continuity, he adds.

Constructing an upgrade strategy
Barton Malow, a construction management firm based in Southfield, Mich., has some 1,300 desktop machines and 80 servers that it plans to convert to Windows 2003, from a base consisting mostly of Windows 2000 and some Windows XP platforms, according to Paul Johnson, chief network engineer.

The company has implemented patch management as part of its inventory, licensing control and other change-management systems, Johnson says. Barton Malow's patch-management tool of choice is a component of the company's suite of Altiris products. Altiris is a systems management vendor whose products compete with Microsoft's SMS.

So, when Altiris came out with a patch manager as part of its product suite, Barton Malow took a look. Until then, the firm had been using custom Windows scripts to install patches. 'It took a lot of manual effort, 40 hours a week, just to keep our systems patched,' Johnson says.

Now, when a vendor releases a patch, Barton runs it through its test lab, complete with load simulation tools and benchmarking programs that simulate an app's average usage statistics. 'We check there aren't any crashes and odd interactions with the Exchange server and so forth,' Johnson says.

If all goes well, Barton deploys the patch to 20 or 30 machines to make sure it works as intended. After four or five hours, if no problems are reported, the patch is distributed to the rest of the machines that need it.

Optimally, patching is done as part of a regular monthly maintenance cycle. That works most of the time, but if a vendor releases a patch to close a security loophole, a customer can opt to install the patch as soon as possible-even during the middle of the workday. 'We have to assess the likely cost of ignoring the patch for a day versus the costs of shutting down the network,' Johnson says. 'I can't think of the last time we had to do that.'

Patches in a hurry
Ed Bailey works at the University of Florida in Gainesville as IT director at the college of engineering's department of material science. He's responsible for 400 desktops, and he estimates the PatchLink Update tool has saved the university 500 to 600 manhours. 'I can update all the desktops in less than 10 minutes,' he says. 'Before, I did it manually, and if I did it in a real hurry, I could do it in about a week.'

With the department spread out in five buildings on campus, Bailey would walk around to each desktop, install the patch and reboot the PC-a process that took between 10 and 15 minutes for each machine, assuming the update was trouble free.

'I pay $9 per computer for a PatchLink license-that's a minimal amount,' Bailey says. Because he works in a university, security is less stringent and access is more open than in the commercial world. 'I'd have a different approach in a commercial environment,' Bailey says. 'Here, our change-management process is more abbreviated than I'd like.' For instance, the department's 450 students are allowed to plug personal laptops into the network. 'We check out the laptop and register it, and if there are any problems, we turn it off right away,' he says. However, the university doesn't mandate which apps must be installed on the laptops.

Just do it
Mark Mellis, a consultant with System-Experts, a security consulting company based in Sudbury, Mass., says he sees organizations applying a variety of patching strategies. 'It really does go across the spectrum,' he says. One person he knows runs the network at a university and 'is doing something really clever. They're not allowed to apply the standard corporate controls-there's no such thing as a standard desktop build-so what she does is [require] you to sign up for BigFix [a third-party software package] to get a ticket to the network.'

When a new user signs onto the network, the machine is automatically taken to a Web page where BigFix installs itself on the client PC or laptop and checks to make sure everything is okay. 'You can't play on the network unless you're current with all your software,' Mellis explains. It's 'the most aggressive stance I've seen in this arena,' he adds.

In contrast, Mellis says he's worked with a financial institution with 10,000 desktops. This firm uses NetIQ to monitor the patch versions on each desktop, and when required, uses SMS to push the patches.

'Their biggest problem is that the testing cycle for the patches takes a very long time,' Mellis says. 'They have a change-control process and committee to approve it all, and sometimes the bugs appear before the patch-qualification cycle is done.' So the company is inspecting every e-mail and attachment, which emphasizes why it's important to do patch management in the overall context of a layered security approach.

Despite the obvious benefits, it's 'not a no-brainer to get started' on the patch-management path, says Carol Baroudi, founder of Baroudi Bloor, a consulting company based in Arlington, Mass. 'It's quite complex, and you're in a Catch-22 situation for a while' she says. 'Until you get to the point where everything is well defined, it's daunting to get everything into an automated scenario. You need to understand what software is on each client or server, which version it is, what's the critical path for each application on each machine.' Patch management, if done correctly, includes identity management, asset management and configuration management, she says.

Even if companies don't do it all at once, end users and consultants say it's critical to do something. 'It's a mark against IT that patch management was so neglected for so long,' says Barton Malow's Johnson. 'There were so many windows of opportunity that we didn't take advantage of. We were very lucky for a long time, and eventually our luck ran out.'

Also see related article, For every little patch you make.

Discussion Points
Use Discussion Board below

PATCH MANAGEMENT

• Viruses and worms are constant risks and patch management is a key defense against security breaches.

• In an ideal world, patch management would be incorporated into security and change-management processes.

• A comprehensive approach to patch management begins with an IT inventory assessment, which is often the most time-consuming part of the job.

• Only about 10 percent of all Fortune 1000 companies have deployed comprehensive patch-management solutions, according to the Yankee Group.