Columns
Credit union takes SAFE approach to patch automation
Some organizational names carry inherent assumptions. Just ask the Sacramento Air Force Employees (SAFE) Federal Credit Union in California, which maintains assets of $1.2 billion for its roughly 120,000 members.
"With a name like SAFE, it implies certain things to our members, and we pride ourselves on protecting our member data and keeping our customers’ data safe," says Marc Buzard, the network services manager for SAFE. "So we’re always looking at what we can do better."
Such concerns drove SAFE to find an alternative to manually patching its 450 Windows and Macintosh computers, and 110 Windows and Unix servers, which are located at 15 different branches. Simply put, managing OS and app upgrades, plus the installation of security patches, took an inordinate amount of time.
"We were starting to grow, and realized it just wasn’t cost-effective to have one or two people running around to all the branches and manually updating them with all the latest patches," Buzard says. The credit union also needed to keep up with new patches, since at the time, "Microsoft was putting them out left and right," he recalls.
About two years ago, SAFE investigated Ecora Software, PatchLink and Microsoft products for automating its patch management process. SAFE dinged Microsoft’s Systems Management Server for being too expensive, and Windows Update over of its lack of reporting and inability to schedule patch installations.
Ultimately, the credit union selected PatchLink’s Update product, impressed by its Web-based interface and reporting capabilities. "One of the key things we liked about PatchLink was its centralized reporting: the ability to run a report and see which machines had been patched," Buzard says.
After a straightforward implementation, including installing the requisite agents on all computers to be patched by PatchLink, the product has functioned as expected. "With any kind of product you always run into issues," Buzard says, "but most of the issues we’ve run into are learning curves."
The credit union patches on a fixed schedule (barring any unexpected, critical updates) based largely on Microsoft’s patch-release schedule—new patches get released on the second Tuesday of every month. The next day, a SAFE IT team meets to plan what needs patching and in what order. Internet-facing servers typically go first. Two people previously handled patching, now it’s just one person’s job. "It’s not even a full-time job for him," Buzard says.
PatchLink is used to update all of the credit union’s Windows, Macintosh, and Linux computers when possible. "Some Linux boxes are running a stripped-down version of Linux provided by one of our vendors, so there are some support issues," Buzard says. Patching the box might produce a version of Linux the vendor doesn’t support. In such cases, "we follow up with the vendor to make sure they’re up to date, as far as deploying patches."
Before distributing any patch, SAFE vets it. "Because we have a test bed we can deploy on and test on, it saves us a lot of downtime," Buzard explains. All patches are tested against the array of front- and back-office apps typical to the credit union. "You walk through the basics, just make sure everything seems to work correctly. There’s still Internet connectivity, applications launch and behave as they’re supposed to, and merely applying the patch doesn’t crash the machine," he explains.
Then the credit union rolls the patch out to just a branch or two at a time—just in case. One particular worry: even with extensive testing, on a certain make and model of machine the credit union uses, applying a patch can sometimes result, after rebooting, in a blue screen. "We’ve gotten to the point where it doesn’t happen often, and when it does we know how to get around it," Buzard points out. "This last go-round, we had two machines out of 450 that exhibited a blue screen." Still, it’s one more reason to play it safe and deploy patches gradually.
Buzard says he may tap a new PatchLink product to update remote or wireless devices that connect to the LAN: "We’re just now staring to deploy some wireless solutions, so we may want to look at that in the future."
—By Mathew Schwartz