In-Depth

Why Automated Patch Management Remains Elusive

Patching remains a manual, time-intensive process, despite more automated tools.

• By Mathew Schwartz
• April 25, 2006

What happened to automated patch management? A year ago the subject was all the rage: security managers had to endlessly patch newly discovered vulnerabilities, and they wanted help.

Since then, much of the patch management discussion—at least from the vendor community—has turned to consolidation. Patch management is becoming part of broader enterprise systems management and security event management tools. Assessing patch levels is also a core component of many network access control (NAC) initiatives which scan and quarantine endpoints before they’re granted network access.

The drive to combine enterprise and security event management is understandable: IT managers broadly want to know what’s wrong, fix it, and know it’s fixed. Vetting patches while PCs are quarantined is similarly useful: users of out-of-compliance PCs can be offered a self-help page with links to download the latest patches, install them, then reboot and connect.

Yet both of those things presuppose, or perhaps gloss over, a crucial, lingering element of patch management: having a patch in the first place, and knowing where to apply it.

The Lure of Automated Patching

Patching, it turns out, remains an inherently difficult process. That’s no surprise: vendors typically release patches with little (and sometimes no) real documentation about what they’re fixing, why, and what else might be affected. Unfortunately, patching also remains a fact of life, given the buggy state of today’s software. Gartner predicts that at least until 2008, 90 percent of all computer attacks will simply exploit application vulnerabilities.

Companies know they need help: 66 percent of all firms have deployed patch management—up from 54 percent in 2004—and another 15 percent plan to do so by the end of 2006. Those findings, the result of a Forrester Research survey of 94 IT “decision makers” in North America, also found vulnerability assessment tools are “surprisingly widely deployed” on endpoints (PCs); 49 percent of companies have them.

Security expert Marcus Ranum has described patching as “an endless war that you cannot win,” since security managers never know what they’re fighting until it’s too late. The fight is also getting more difficult, as the window between a vulnerability announcement and an attack crafted to exploit it continues to decrease, leaving security managers with less time to fix vulnerabilities. Witness the Nimba worm unleashed in 2001, almost a year after the vulnerability it exploited was disclosed by Microsoft. Then fast-forward to 2005, when Zotob exploited a vulnerability announced just five days before.

Not surprisingly, companies want a silver bullet, and they look to automation. “They say, ‘I just want to push a button and have it happen,’” notes Eric Herrera, principle product manager for the vulnerability management solutions at CA Inc. (formerly Computer Associates) in Islandia, New York.

According to Forrester, 40 percent of companies currently have automated vulnerability remediation capabilities, though another 16 percent plan to implement them in 2006. “This trend is likely to continue in 2007 as firms struggle to address desktop and laptop security issues,” notes Forrester analyst David Friedlander.

His recommendation: implement automated vulnerability remediation as soon as possible. “Given the need to respond rapidly to vulnerabilities, firms must implement systems that can act on vulnerability assessment data immediately.”

Irreducible Patch Problems

Automation only addresses a fraction of the patch process, however. As Herrera notes, “the patch problem isn’t really a patch problem: it’s a problem of management,” and the patch management lifecycle is complex. It starts with having the security intelligence to know when there’s a vulnerability, then finding a corresponding workaround or patch. You also have to know exactly what the patch or workaround will affect.

Furthermore, organizations must rank vulnerabilities by severity, to know which to patch first. Then they have to test a patch to make sure it doesn’t conflict with in-house or off-the-shelf applications; deploy it only to the correct machines, which can be difficult; and finally, insure the patch installs successfully.

Hence, despite some increased automation for vulnerability notifications, and patch distribution, other irreducible elements remain. For starters, “testing takes time,” notes Herrera. “Everything starts back at the point of: What are my assets? What kind of technology and assets do I have in my infrastructure? What software level are they running, what patches do they have?”

Those can be difficult questions to answer. To be sure, patch management tools can help catalog PCs and what’s running on them. To better manage the whole change process, some organizations also turn to IT process improvement frameworks, such as the IT Infrastructure Library (ITIL), which breaks out IT processes into change management, release management, and configuration management.

How NAC Can Help

Beyond patch testing, new technology is helping facilitate easier patching. For example, various endpoint security initiatives (including Cisco's Network Admission Control and Microsoft's Network Access Protection) may help companies ensure that once a patch is ready it ends up where it's supposed to. This, it turns out, is currently difficult.

While there are no hard-and-fast numbers, available evidence suggests current patch management tools aren’t anywhere near 100 percent effective, at least in large, heterogeneous, distributed environments. “We were surprised when we went out and looked at some of the real-world numbers in the field,” says Mark Beadles, chief architect at EndForce Inc., an independent NAC provider based in Dublin, Ohio. “We suspected this was a problem, but we ended up seeing it was worse than many even feared.”

As an example, he cites a health care company with 35,000 endpoints (an EndForce customer), which prefers to remain anonymous. “From an asset-management perspective, they were more forward-thinking,” he notes. Case in point: the company installed CA Unicenter, and when that alone wasn’t patching sufficient numbers of machines, it also implemented Microsoft’s SUS (Software Update Server).

Yet even after pushing patches with both programs, the health care company regularly found only 50 to 70 percent of all machines got fully patched. “That’s because first, there are machines you just don’t know about,” says Beadles. “Second, these machines are used by humans, and through a combination of deliberant misuse and inadvertent error they don’t get patched.”

Here’s how NAC can help: Once organizations develop, test, and deploy patches, NAC snares machines that might otherwise escape patching, when they request network access. Beadles predicts the overlapping coverage afforded by endpoint security agents running on PCs, plus DHCP network services, plus SSL VPN coverage, will probably catch at least 90 percent of enterprise machines.

While knowing which patches to apply to which machines is difficult, CA’s Herrera predicts that over the next year companies will get much better at writing the rules necessary to automate such processes.

Many organizations will nevertheless shy away from automation where critical or compliance-related processes are involved, simply because automation doesn’t always work. “You’re still going to have the human factor where organizations say, ‘We don’t want to automate the high-risk patches,’” he says.

Related Articles:

• Case Study: Patching the SAFE Federal Credit Union
• The Shape of Endpoint Security to Come
• Untangling Endpoint Security Initiatives