In-Depth

Software Configuration Management Undergoes a Restoration

• By Alan Radding
• October 1, 2005

Admittedly, Lockheed Martin Information Technology has a tougher assignment than most, and that’s to clean up the Hanford nuclear waste site. As the IT group serving the project, the Lockheed team manages 2,900 applications. It must keep track of all the information associated with the cleanup, from the tools to the scientific equations and algorithms used on the job. The team consists of as many as 150 people on a given day, not including subcontractors.

With so much at risk, with so many regulators and auditors constantly looking over the work, and with news reporters ready to pounce at the slightest whiff of a problem, “We need to have absolute control of who did what, when, where, and why,” says William Jones, manager of software quality assurance and configuration management. “We need to have a perpetual inventory with granularity.” To do that, the team turned to software vendor MKS to obtain the tool it needed for software configuration management. “We even use MKS to control versions of MKS,” he adds.

Although most IT teams aren’t involved in sanitizing nuclear waste dumps, they are under pressure like never before to meet corporate mandates and to increase their visibility. “Since the dot-com meltdown, there has been a big change in the way businesses are looking at IT,” says Jim Duggan, an analyst at Gartner. With the dot-com collapse, IT lost its magical aura, he says. Now budgets, schedules, service levels and service quality are open to review and evaluation with little tolerance for IT projects that go wrong.

Combined with the need to manage widely dispersed development teams that may include outsourced and offshore programmers, along with the pressures of stringent regulatory compliance, “Management is now interested in getting better at change control and governance,” Duggan says. Suddenly SCM is hot.

SCM has long been regarded as an IT best practice. The products, initially appearing as simple versioning tools, have been sitting in IT shops for decades. The tools protect code through a check-in/out process that prevents programmers from blowing away each other’s work on the same code. More advanced versioning products enable parallel development by tracking and ultimately merging parallel code streams. Today’s SCM products, however, go far beyond basic version control and even parallel development to address everything from business requirements tracking to development process enforcement and automation.

Resistance is futile
Although programmers conceded the benefits of the old versioning products, they resisted using them. The early products were cumbersome and usually required programmers to leave the development environment to fire up the versioning product.

The evolution of SCM in the last few years is dramatic. Even simple versioning products now work from within IDEs. Programmers simply pull down task lists and click checkboxes to input required information, all from within the IDE. “Every product has to integrate with the IDE; that’s the first requirement. Nobody wants to click out of the IDE,” says Carey Schwaber, an analyst at Forrester Research. Managers will likely use an optional Web interface.

The more advanced SCM products—typically packaged as product suites consisting of different combinations of functionality—still perform the basic version control as well as code merging for parallel development. However, they do much more. They can enforce development best practices and processes and provide workflow management. Often packaged as product suites, they encompass business requirements, testing, change management, defect tracking, bug fixing, automated promotion of code to production, and sophisticated reporting that addresses the needs of business executives and auditors, as well as application development managers.

According to Gartner, the SCM market experienced 15 percent growth last year, “double the industry as a whole,” says Duggan. Much of that growth comes from auditors who are pressuring IT departments to upgrade from the simplest version control tools, like Microsoft’s Visual Source Safe or the open-source CVS, he notes, to more advanced tools from companies including MKS, IBM Rational, Serena, Perforce and Telelogic. Avariant of SCM focuses on system configuration management, monitoring and change tracking for systems to ensure high availability.

The Lockheed Martin team at the Hanford site, for example, used VSS before switching to MKS. The simple version control tool could not handle the task the team faced. “We have to maintain a complete audit trail. We need to know any modification to any system,” says Jones. His first step with MKS was to document a complete baseline including every project, subproject, and sub-sub project down to individual lines of code. The system also captures all documents, workflows, change requests and even case tests, which enables complete recursive testing. “This is not just an audit trail but a complete history of the project from birth to death. Who did what and why,” he says.

At first, the programmers resisted the switch. VSS was simple and easy to use. MKS, by comparison, presented a learning curve. Eventually, however, “they have come around to see it as a lifesaver,” Jones adds. Developers can go back to recreate the project at any point in time, and they even maintain all builds on MKS.

Meeting compliance requirements
Regulatory compliance, such as required by Sarbanes-Oxley, is driving much of the most recent interest in advanced SCM products. In general, the mandates require that managers attest to the integrity of the information they are reporting. To do that, the managers need mechanisms that monitor and report what is happening to the systems and applications that generate the information.

An SCM tool, for example, would capture changes to an application that might alter the way financial data is calculated or reported. This could become very important if questions were to arise about the accuracy or integrity of the data. The SCM tool would help auditors and investigators identify which changes were made, when they were made, who made them, and who approved them. Compliance also typically mandates the separation of duties. In application development, that translates into separating programming from testing and promotion of code to production.

“SCM is definitely a solution for some compliance issues, but before you rush out to purchase a product for compliance, we recommend that you talk with your auditors first. There are a lot of SCM tools and a lot of capabilities. You need to figure out which specific compliance needs you want the SCM tool to help you meet,” Schwaber says.

Masterbrand Cabinets turned to the Aldon SCM tool to help with compliance. “Our primary reason for getting SCM was for SOX[Sarbanes-Oxley]. It would guarantee and prove that the objects we created and tested are the same objects we moved into production,” says William Storey, the company’s deputy CIO.

The company chose the Aldon product primarily because it offered an AS/400 version. Since then, the company has expanded its use of the tool to other platforms. Previously, the company relied on an informal process of checks and balances and controls based on the use of a library. Essentially, the developer would pull out a module and put it in a test library. With only about 15 developers on staff, plus periodic contractors, the system worked reasonably well. With the arrival of SOX and the need to separate development from testing, the informal system proved inadequate.

The Aldon tool provides version control through the check-in/out facility and manages the development process from programming through testing to promotion to production. Most importantly to Masterbrand, the tool allows the company to separate duties, isolating development and testing. “If developers could get access to test results, they could go back and change things after users signed off on the code,” Storey explains. The SCM tool, however, “locks the developers out of the test environment,” he notes. With the Aldon tool, critical points in the process are automated, taking it out of the hands of individuals.

Clear case of complexity
As the complexity of development grows, organizations need increasingly advanced SCM tools. “We had once used VSS, but that is version control, not configuration management,” says Arieh Shalem, assistant VP of corporate quality management at TTI Telecom, a provider of operations systems to the telecom industry. Specifically, TTI develops complex operational support systems and business support systems for the telecommunications industry.

These systems monitor and manage multi-vendor, multi-technology wireline and wireless networks covering multiple domains including switching, transport, IP, 2G, 2.5G, 3G, broadband and metro Ethernet. Its 200 developers work mostly in Java and Oracle. It has numerous development teams working on 70 projects simultaneously.

Faced with this challenge, it needed a tool that would support both R&D and current efforts and manage multiple, different projects at the same time. The company used IBM’s ClearCase, which is currently the SCM industry’s market leader, according to Duggan, although he senses market leadership could change as challengers such as MKS, Serena and Telelogic bolster their products. “ClearCase has played the Cadillac card for years, but if you look closely at the technology, there are others who are at the same price or even less expensive and are more capable,” Duggan says.

That may be the case now, but 4 years ago when TTI Telecom opted for ClearCase, there were far fewer options. “ClearCase is not cheap, but at that time, it was the strongest,” Shalem recalls. Once a company makes an SCM choice, changing products is not a trivial decision. The learning curve is steep, resistance can be fierce, and the risk of disruption of projects under development is significant.

In particular, Shalem likes ClearCase’s version tree. “It lets you see immediately where you are in the code, which branch or which entity,” he says. He also likes how it allows the developers to use old versions and old baselines in new development. And he likes the flexibility it allows in supporting almost any development process. His only complaint might be that it is too flexible. “Sometimes it is hard to decide what is the best way for the process,” he says.

For defect tracking and change request management, TTI uses ClearCase’s sister product, ClearQuest. “It is good to have both,” Shalem says.

Trading Technologies builds trading systems for the futures trading industry. Like TTI, it is involved in complex development, with 110 developers working directly on its products. To keep pace with customer demands for new capabilities while addressing problems in older systems, the company finds itself working on as many as four versions of its trading platform at the same time (current production, future beta, development, emergency patch), according to Joanne Wilson, VP, support engineering.

From a technical standpoint, it needed tight source code control for parallel development. It wanted to tie software changes to business requirements and defects. It needed to pursue multiple development paths and rapidly merge changes among the different version branches. Finally, it wanted to “be able to visualize the ancestry of files to determine where a defect was introduced and what development branches it would impact,” says Wilson.

The company looked at ClearCase, MKS and Perforce. In the end, it opted for MKS. “The system seemed to be manageable from both an administrator’s and user’s point of view. Cost was also a big factor,” says Wilson. The learning curve was steep, she concedes, with the need to master new concepts such as sandboxes and development paths. However, the benefits of parallel development quickly became apparent. “Multiple developers could have the same files checked out when they are working on different things,” she notes. The reaction has been so positive that the company is moving all its old software, developed under VSS, to MKS as well.

Open source makes a run for it
With the exception of Microsoft’s VSS, the low end of the SCM market, which consists primarily of version control products, has been taken over by open-source tools. In the open-source arena, the primary version control tool is CVS, although there is a number of projects that are expanding upon the basic CVS code. “CVS is not really that good in a commercial environment because of its lack of tracking. It also allows too many alternatives,” says Duggan.

Collabnet’s Subversion is an open-source SCM tool that provides more robust functionality, Duggan says. Organizations can start with Subversion for free. When they need more functionality, it can be integrated with Collabnet’s Enterprise Edition.

Aegis is another open-source tool. Aegis aspires to move beyond version control to become a software configuration management system through functions that support code integrity, such as registering automated tests and support for code reviews.

“I have been using Aegis for many years. It works, it is reliable, and it is free,” says Jerry Pendergraft, a consultant at Parvenu Systems. For St. Jude Medical, he built mission-critical software under Aegis.

Aegis differs from plain-vanilla version control systems such as CVS. “The big difference with Aegis is that automated tests are part of it,” he says. Code is not allowed to become part of the baseline until it has gone through integration and testing. The tests, he adds, are pretty easy to write and ensure that the baseline is always functional. Like its proprietary counterparts, open-source SCM products can be assembled into suites of products. For example, Aegis integrates with Razor, another open-source tool, to get change and defect management. Underneath, Aegis offers version management capabilities as well.

Free, with strings attached
Open-source SCM tools are free only for those who have the skill, time and patience to download, deploy and integrate them, working only from the raw source code distribution. On the other hand, proprietary SCM tools, especially at the high end, can be quite expensive.

Figuring out exactly what the proprietary tools cost is difficult. “Pricing is complicated. It is based on the number of servers, the number of developers and the number of administrators,” says Schwaber. She advises companies to prepare to spend more than $1,000 per seat, even as high as $4,000 per seat, based on the number of different functional modules desired.

If you are going to spend that kind of money, you need to be sure you will get value from it. “Just buying a tool is not enough to get value,” Duggan says. The trick to capturing the value lies in optimizing your development process. “You need to figure out how you are doing it manually and then go from there. You need agreement on a development process that covers all the risk factors,” he explains. Once you have the process pinned down, you can match the tool to the process. Only then can you be confident you will get value from your SCM investment.

On ADTmag.com
SCM: New tools to streamline development
By Linda L. Briggs

SCM tools offer compliance help
By Linda L. Briggs

Java IDE integrated with Seapine Surround SCM
By Linda L. Briggs


ILLUSTRATION BY RYAN ETTER