Software Configuration Management Undergoes a Restoration
- By Alan Radding
Admittedly, Lockheed Martin Information Technology has a tougher assignment
than most, and that’s to clean up the Hanford nuclear waste site. As
the IT group serving the project, the Lockheed team manages 2,900 applications.
It must keep track of all the information associated with the cleanup, from
the tools to the scientific equations and algorithms used on the job. The team
consists of as many as 150 people on a given day, not including subcontractors.
With so much at risk, with so many regulators and auditors constantly looking
over the work, and with news reporters ready to pounce at the slightest whiff
of a problem, “We need to have absolute control of who did what, when,
where, and why,” says William Jones, manager of software quality assurance
and configuration management. “We need to have a perpetual inventory with
granularity.” To do that, the team turned to software vendor MKS to obtain
the tool it needed for software configuration management. “We even use
MKS to control versions of MKS,” he adds.
Although most IT teams aren’t involved in sanitizing nuclear waste dumps,
they are under pressure like never before to meet corporate mandates and to
increase their visibility. “Since the dot-com meltdown, there has been
a big change in the way businesses are looking at IT,” says Jim Duggan,
an analyst at Gartner. With the dot-com collapse, IT lost its magical aura,
he says. Now budgets, schedules, service levels and service quality are open
to review and evaluation with little tolerance for IT projects that go wrong.
Combined with the need to manage widely dispersed development teams that may
include outsourced and offshore programmers, along with the pressures of stringent
regulatory compliance, “Management is now interested in getting better
at change control and governance,” Duggan says. Suddenly SCM is hot.
SCM has long been regarded as an IT best practice. The products, initially
appearing as simple versioning tools, have been sitting in IT shops for decades.
The tools protect code through a check-in/out process that prevents programmers
from blowing away each other’s work on the same code. More advanced versioning
products enable parallel development by tracking and ultimately merging parallel
code streams. Today’s SCM products, however, go far beyond basic version
control and even parallel development to address everything from business requirements
tracking to development process enforcement and automation.
Resistance is futile
Although programmers conceded the benefits of the old versioning products, they
resisted using them. The early products were cumbersome and usually required
programmers to leave the development environment to fire up the versioning product.
The evolution of SCM in the last few years is dramatic. Even simple versioning
products now work from within IDEs. Programmers simply pull down task lists
and click checkboxes to input required information, all from within the IDE.
“Every product has to integrate with the IDE; that’s the first requirement.
Nobody wants to click out of the IDE,” says Carey Schwaber, an analyst
at Forrester Research. Managers will likely use an optional Web interface.
The more advanced SCM products—typically packaged as product suites consisting
of different combinations of functionality—still perform the basic version
control as well as code merging for parallel development. However, they do much
more. They can enforce development best practices and processes and provide
workflow management. Often packaged as product suites, they encompass business
requirements, testing, change management, defect tracking, bug fixing, automated
promotion of code to production, and sophisticated reporting that addresses
the needs of business executives and auditors, as well as application development
According to Gartner, the SCM market experienced 15 percent growth last year,
“double the industry as a whole,” says Duggan. Much of that growth
comes from auditors who are pressuring IT departments to upgrade from the simplest
version control tools, like Microsoft’s Visual Source Safe or the open-source
CVS, he notes, to more advanced tools from companies including MKS, IBM Rational,
Serena, Perforce and Telelogic. Avariant of SCM focuses on system configuration
management, monitoring and change tracking for systems to ensure high availability.
The Lockheed Martin team at the Hanford site, for example, used VSS before
switching to MKS. The simple version control tool could not handle the task
the team faced. “We have to maintain a complete audit trail. We need to
know any modification to any system,” says Jones. His first step with
MKS was to document a complete baseline including every project, subproject,
and sub-sub project down to individual lines of code. The system also captures
all documents, workflows, change requests and even case tests, which enables
complete recursive testing. “This is not just an audit trail but a complete
history of the project from birth to death. Who did what and why,” he
SCM IS BACK AND BUFF
- Early SCM products were cumbersome and usually required programmers to leave the development environment to fire up the versioning product.
- Today's SCM products, however, go far beyond basic version control and even parallel development to address everything from business requirements tracking to development process enforcement and automation.
- Regulatory compliance, such as required by Sarbanes-Oxley, is driving much of the most recent interest in advanced SCM products.
At first, the programmers resisted the switch. VSS was simple and easy to use.
MKS, by comparison, presented a learning curve. Eventually, however, “they
have come around to see it as a lifesaver,” Jones adds. Developers can
go back to recreate the project at any point in time, and they even maintain
all builds on MKS.
Meeting compliance requirements
Regulatory compliance, such as required by Sarbanes-Oxley, is driving much of
the most recent interest in advanced SCM products. In general, the mandates
require that managers attest to the integrity of the information they are reporting.
To do that, the managers need mechanisms that monitor and report what is happening
to the systems and applications that generate the information.
An SCM tool, for example, would capture changes to an application that might
alter the way financial data is calculated or reported. This could become very
important if questions were to arise about the accuracy or integrity of the
data. The SCM tool would help auditors and investigators identify which changes
were made, when they were made, who made them, and who approved them. Compliance
also typically mandates the separation of duties. In application development,
that translates into separating programming from testing and promotion of code
“SCM is definitely a solution for some compliance issues, but before
you rush out to purchase a product for compliance, we recommend that you talk
with your auditors first. There are a lot of SCM tools and a lot of capabilities.
You need to figure out which specific compliance needs you want the SCM tool
to help you meet,” Schwaber says.
Masterbrand Cabinets turned to the Aldon SCM tool to help with compliance.
“Our primary reason for getting SCM was for SOX[Sarbanes-Oxley]. It would
guarantee and prove that the objects we created and tested are the same objects
we moved into production,” says William Storey, the company’s deputy
The company chose the Aldon product primarily because it offered an AS/400
version. Since then, the company has expanded its use of the tool to other platforms.
Previously, the company relied on an informal process of checks and balances
and controls based on the use of a library. Essentially, the developer would
pull out a module and put it in a test library. With only about 15 developers
on staff, plus periodic contractors, the system worked reasonably well. With
the arrival of SOX and the need to separate development from testing, the informal
system proved inadequate.
The Aldon tool provides version control through the check-in/out facility and
manages the development process from programming through testing to promotion
to production. Most importantly to Masterbrand, the tool allows the company
to separate duties, isolating development and testing. “If developers
could get access to test results, they could go back and change things after
users signed off on the code,” Storey explains. The SCM tool, however,
“locks the developers out of the test environment,” he notes. With
the Aldon tool, critical points in the process are automated, taking it out
of the hands of individuals.
Regulatory compliance is driving much of the interest in advanced SCM products.
Clear case of complexity
As the complexity of development grows, organizations need increasingly advanced
SCM tools. “We had once used VSS, but that is version control, not configuration
management,” says Arieh Shalem, assistant VP of corporate quality management
at TTI Telecom, a provider of operations systems to the telecom industry. Specifically,
TTI develops complex operational support systems and business support systems
for the telecommunications industry.
These systems monitor and manage multi-vendor, multi-technology wireline and
wireless networks covering multiple domains including switching, transport,
IP, 2G, 2.5G, 3G, broadband and metro Ethernet. Its 200 developers work mostly
in Java and Oracle. It has numerous development teams working on 70 projects
Faced with this challenge, it needed a tool that would support both R&D
and current efforts and manage multiple, different projects at the same time.
The company used IBM’s ClearCase, which is currently the SCM industry’s
market leader, according to Duggan, although he senses market leadership could
change as challengers such as MKS, Serena and Telelogic bolster their products.
“ClearCase has played the Cadillac card for years, but if you look closely
at the technology, there are others who are at the same price or even less expensive
and are more capable,” Duggan says.
That may be the case now, but 4 years ago when TTI Telecom opted for ClearCase,
there were far fewer options. “ClearCase is not cheap, but at that time,
it was the strongest,” Shalem recalls. Once a company makes an SCM choice,
changing products is not a trivial decision. The learning curve is steep, resistance
can be fierce, and the risk of disruption of projects under development is significant.
In particular, Shalem likes ClearCase’s version tree. “It lets
you see immediately where you are in the code, which branch or which entity,”
he says. He also likes how it allows the developers to use old versions and
old baselines in new development. And he likes the flexibility it allows in
supporting almost any development process. His only complaint might be that
it is too flexible. “Sometimes it is hard to decide what is the best way
for the process,” he says.
For defect tracking and change request management, TTI uses ClearCase’s
sister product, ClearQuest. “It is good to have both,” Shalem says.
Trading Technologies builds trading systems for the futures trading industry.
Like TTI, it is involved in complex development, with 110 developers working
directly on its products. To keep pace with customer demands for new capabilities
while addressing problems in older systems, the company finds itself working
on as many as four versions of its trading platform at the same time (current
production, future beta, development, emergency patch), according to Joanne
Wilson, VP, support engineering.
From a technical standpoint, it needed tight source code control for parallel
development. It wanted to tie software changes to business requirements and
defects. It needed to pursue multiple development paths and rapidly merge changes
among the different version branches. Finally, it wanted to “be able to
visualize the ancestry of files to determine where a defect was introduced and
what development branches it would impact,” says Wilson.
The company looked at ClearCase, MKS and Perforce. In the end, it opted for
MKS. “The system seemed to be manageable from both an administrator’s
and user’s point of view. Cost was also a big factor,” says Wilson.
The learning curve was steep, she concedes, with the need to master new concepts
such as sandboxes and development paths. However, the benefits of parallel development
quickly became apparent. “Multiple developers could have the same files
checked out when they are working on different things,” she notes. The
reaction has been so positive that the company is moving all its old software,
developed under VSS, to MKS as well.
Open source makes a run for it
With the exception of Microsoft’s VSS, the low end of the SCM market,
which consists primarily of version control products, has been taken over by
open-source tools. In the open-source arena, the primary version control tool
is CVS, although there is a number of projects that are expanding upon the basic
CVS code. “CVS is not really that good in a commercial environment because
of its lack of tracking. It also allows too many alternatives,” says Duggan.
Collabnet’s Subversion is an open-source SCM tool that provides more
robust functionality, Duggan says. Organizations can start with Subversion for
free. When they need more functionality, it can be integrated with Collabnet’s
Aegis is another open-source tool. Aegis aspires to move beyond version control
to become a software configuration management system through functions that
support code integrity, such as registering automated tests and support for
“I have been using Aegis for many years. It works, it is reliable, and
it is free,” says Jerry Pendergraft, a consultant at Parvenu Systems.
For St. Jude Medical, he built mission-critical software under Aegis.
Aegis differs from plain-vanilla version control systems such as CVS. “The
big difference with Aegis is that automated tests are part of it,” he
says. Code is not allowed to become part of the baseline until it has gone through
integration and testing. The tests, he adds, are pretty easy to write and ensure
that the baseline is always functional. Like its proprietary counterparts, open-source
SCM products can be assembled into suites of products. For example, Aegis integrates
with Razor, another open-source tool, to get change and defect management. Underneath,
Aegis offers version management capabilities as well.
Free, with strings attached
Open-source SCM tools are free only for those who have the skill, time and patience
to download, deploy and integrate them, working only from the raw source code
distribution. On the other hand, proprietary SCM tools, especially at the high
end, can be quite expensive.
Figuring out exactly what the proprietary tools cost is difficult. “Pricing
is complicated. It is based on the number of servers, the number of developers
and the number of administrators,” says Schwaber. She advises companies
to prepare to spend more than $1,000 per seat, even as high as $4,000 per seat,
based on the number of different functional modules desired.
If you are going to spend that kind of money, you need to be sure you will
get value from it. “Just buying a tool is not enough to get value,”
Duggan says. The trick to capturing the value lies in optimizing your development
process. “You need to figure out how you are doing it manually and then
go from there. You need agreement on a development process that covers all the
risk factors,” he explains. Once you have the process pinned down, you
can match the tool to the process. Only then can you be confident you will get
value from your SCM investment.
Figuring out exactly what the proprietary tools cost is difficult.
SCM: New tools to streamline
By Linda L. Briggs
SCM tools offer
By Linda L. Briggs
Java IDE integrated
with Seapine Surround SCM
By Linda L. Briggs
ILLUSTRATION BY RYAN ETTER