In-Depth

Software factories come into modern times

• By Stephen Swoyer
• December 1, 2004

What if software were manufactured like an automobile or a refrigerator, using off-the-shelf components and assembly-line production techniques to help automate many of the most repetitive development tasks? Better still, what if the development expertise of whiz-bang developers, or the domain expertise of business managers, could be instantiated — i.e., encapsulated — and widely distributed, or reused, throughout an organization? It’s an idea business leaders have kicked around for about as long as programmers have been plying their craft in enterprise environments. Even after 50 years of incremental software process and quality improvements, the goal of programming automation on a large scale — much less of assembly line software production — has proven to be stubbornly elusive. Not that this has deterred folks from periodically dusting off the concept and dressing it up in the latest management or process improvement-speak, of course.

As some skeptics see it, that’s just what Microsoft did at September’s Object-Oriented Programming, Systems, Languages, and Applications conference, where — in a forum given over to the promotion of its Visual Studio 2005 Team System —it trumpeted a new strategy, called the "software factory," that purports to bring a manufacturing discipline to software development.

Of course, by "software factory," Microsoft isn’t literally describing an assembly line development process — at least, not in the way a car or a refrigerator is manufactured. The software giant does mean a development infrastructure that gives developers a way to encapsulate their knowledge as assets that can be reused by even the least-skilled of their peers.

Similarly, by "software factory," Microsoft means a development strategy in which domain expertise that’s specific to vertical industries (pharmaceuticals, financial services) or horizontal functional areas (accounting, human resources) can also be encapsulated and reused. Finally, Microsoft officials say the software factory concept stresses intelligent automation, in which rote or menial tasks are automated and the divine work of programming is left to enterprise software developers.

The idea, says Keith Short, an architect in the enterprise frameworks and tools group at Microsoft and co-author (with Microsoft colleague Jack Greenfield) of Software Factories: Assembling Applications With Patterns, Models, Frameworks, and Tools, "is to move from a position where a developer is coming to every application fresh" with very little in the way of reusable or encapsulated tooling, to a situation "where the IDE itself can be configured with the knowledge of how to construct specific applications."

This includes, Short says, domain expertise. "Imagine a world where … you’ve got exactly the right set of tools?"

Two programming extremes
As it turns out, Short’s question provokes a range of different, and often irreconcilable, responses from enterprise developers. On one extreme, you’ve got the programming purists, folks who believe software development is as much art as science: No one has yet found a way to automate or instantiate creativity, they argue, and the idea of manufacturing software seems, on its face, ludicrous. On the opposite end of the spectrum, of course, you’ve got an equally purist segment of programmers who believe that much, or nearly all, programming processes can, and should be, instantiated or productized as efficiently as possible. They typically emphasize long-term planning — that Shangri-la of enterprise IT organizations — and effective code reuse.

In programming, as in politics, most constituents fall somewhere in the middle. Of course, they say, some aspects of software development can benefit from more efficient — even automated — production methods, but surely Microsoft doesn’t think it’s possible to automate programming tasks lock, stock, and debug facility?

More to the point, asks Roy Miller, an independent developer and author of Managing Software for Growth: Without Fear, Control and the Manufacturing Mindset, how does Microsoft propose to automate and encapsulate the production of software code, which is often unique and frequently uncontrollable?

"They’re trying to say, 'Here, we’ll give you a suite of tools so your IT shop can just use this tool, and it comes prepackaged with stuff that’s the core of the system.' That’s fine, I don’t see anything necessarily wrong with that, but the devil is always in the details, isn’t it?" Miller asks.

The rub, Miller argues, is that software development is an inherently messy problem — a programmer’s understanding of the problem he or she is trying to solve changes as he or she tries to solve it — and can’t be predicted, much less controlled. Miller acknowledges this isn’t how most organizations approach software development.

"The way most software projects run, the only tool they give management is control. 'We’re going to start with this plan, we’re going to control things to stick to the plan,' but that’s like trying to put your software development on rails like a train," he says. The software factory approach merely perpetuates this problem, he says.

Some programmers say the software factory sounds suspiciously like other quality and process improvement initiatives that have come and gone in their organizations.

In each case, they say, the ultimate goal — to introduce repeatability, automation, or process efficiency into the software development process — has failed to materialize.

Take Jonathan House, a programmer with agile development specialty house SurgeWorks, who once served as a principal with a Java development company called OOP. OOP’s development team was structured around a methodological approach similar to what Microsoft describes as a software factory, House says. The methodology simply didn’t work, he notes.

"We had a central team that was responsible for component development and consisted of the brightest minds in the company," he explains. "They were the vendor to the five development teams that worked on client projects. I know from exposure to other companies in the industry that we were one of the better companies at this model, and even so, we did not see exponential productivity gains over similarly talented teams."

Microsoft’s Short acknowledges the failings of these and other software automation or process improvement initiatives, but says this software factory will be different.

Automation in moderation
Short says the software factory does not instantiate creativity. Take an automobile, for example, which is manufactured on an assembly line. Cars are often assembled almost entirely by robots—but this misses the point. Someone had to design the robots that build the car, just as someone had to design the transmission that transfers power from a car’s engine to its drive-train.

The upshot, say Short and other proponents, is that developer innovation is central to the software factory concept. Developers aren’t going to be replaced by robots; instead, developers are going to help build robots—pieces of code, components, or tools that encapsulate domain expertise—which help make their jobs easier.

"The key is not to think of a factory like a turn-of-the-century sweatshop," Short says. "The image we’d like people to have in mind when they see the word ‘factory’ is much more like a production line staffed by robots, where humans are doing the creative tasks in setting up the production line, but the rote or menial steps are done by robots."

What’s surprising is that so many Microsoft competitors, such as Borland Software and IBM, seem to be on board the software factory bandwagon. Representatives from both vendors agree the future of software development is one of increased automation and more efficient—assembly line—collaboration.

"From Borland’s perspective, the writing of software is, and will continue to be, a very magical thing. If you look at what developers do, it’s this astounding thing where they manage an abstraction, and they worry and work this abstraction until it’s turned into a real-world thing," says Borland CTO Pat Kerpan. "But that’s software creation, not software development. What’s to say that the developers can’t do this even better when they’re not bogged down in the minutiae of writing software today?"

Alan Brown, a director with IBM’s Rational division, agrees. "I don’t necessarily think this is going to be the answer for everybody, but there are people in organizations that need to move to a more consistent approach to applications that help them become more efficient, and that implies a certain amount of [code] reuse, which has traditionally been difficult," he says.

Brown’s remarks cut to the crux of the biggest beef that many critics say they have with the software factory idea: It assumes a degree of code reuse that is optimistic. In real-world situations, critics say, enterprises have had decidedly mixed success reusing code.

"Code reuse is the biggest mistake in the world," says Theodore Woo, a programmer with a prominent U.S. technology and services company. Woo asked not to have the company he works for identified. "Unless it was written specifically for the purpose [of being reused], it can cause no end of headaches."

Component foundry concept flounders
IT organizations emphasize process, repeatability, and efficiency in everything they do. To some extent, they’ve brought the same perspective to their software development efforts as well, although one important goal—far-reaching code reuse—remains tantalizingly elusive. The problem is that the software factory relies extensively on code reuse—in the same way car manufacturers use off-the-shelf components.

It’s a truism that metaphors inevitably come undone when taken too far. In this case, skeptics say, the software factory metaphor founders precisely on the issue of reuse, at least on the scale required in the software factory vision. Critics argue that while it’s possible to design simple components that can be reused — such as login authentication routines, for example — it probably isn’t practical, much less desirable, to design components specifically for reuse. For starters, says SurgeWorks’ House, most enterprise developers just aren’t up to the task of building reusable components that are lean, clean, and imaginatively designed.

"To develop really good reusable software components, you must be a top-tier developer or architect," he says. "The truth is that only about 5 to 10 percent of developers are capable of working at this level of development. This means that 90 to 95 percent of development teams will fail if they are relying on component development."

Aside from this, there’s a simple economic reason that large-scale code reuse isn’t anything more than a pipe dream, he says. "There is a law of diminishing returns on reusable components. The more complex a reusable component is, the less useful it is as a reusable component," he argues.

House cites the example of Log4J, an open-source application logging solution, which has been reused successfully in a variety of different applications. Log4J is very simple code — and is distinct from Hibernate, another open-source tool that stores Java objects in relational tables. Hibernate isn’t reused nearly as much as Log4J, House says: "A more complex component must make certain assumptions about how it will operate to be useful."

Stephan Bren, an MCSD with a prominent services provider based in Southern California, says his organization — which has been assessed as a CMMI Level III development environment — has managed to achieve some degree of code reuse. At the same time, Bren acknowledges, his organization — which he asked not be named — has not been able to reuse code to the extent presupposed by the software factory concept.

"We do employ code-reusability extensively, using a common set of code components across a number of diverse, yet related, applications," Bren explains. "However, our output is not at the level at which a software factory implementation would be cost effective, I think. Software factories are more amenable to the environment that produces a lot of software applications over a given period of time — a high-production environment, where assembly- line software approach would have beneficial impact."

Off-the-shelf components
Microsoft’s Short concedes these points. Critics are right when they cite the problem of diminishing returns in connection with the engineering of highly sophisticated components, Short acknowledges, but they haven’t considered that sophisticated components can be portable between different vertical industries or, even, horizontal function areas. A component that a pharmaceutical company develops to help broker ecommerce transactions, for example, could probably be used by other competitive companies in the same industry. Where critics see complexity and high cost, Short sees a future of off-theshelf components designed by ISVs, systems integrators and even suppliers and business partners. To a large extent, he sees a different metaphor, but one that’s not without parallel in manufacturing history.

CD spin on code reuse
Consider a CD player, for example. A consumer electronics giant like Sony can more readily manufacture its own CD players than niche-oriented vendors that cater to audiophiles. The CD transport, in particular, presents a significant engineering challenge. It’s a composite part, consisting of a laser, servo mechanism, clock, and other components, and requires a substantial investment — relative to a CD player’s other innards — in R&D and production. Sony recoups this investment by reusing the same transport design throughout many of its products — and by selling CD players in massive volumes. However, this isn’t practical for a niche-oriented vendor whose product line may consist of only a single CD player. So, when CD volumes ramped up in the mid-1980s, many niche vendors did an interesting thing: they designed their own unique power supplies, digital-toanalog converter stages, and other valueadded parts, but purchased transports from Sony, Phillips and Pioneer.

To a large degree, that’s the spin on code reuse touted by Microsoft and Short. The CD transport is a highly sophisticated component that’s designed to fulfill a specific purpose within an existing framework — Sony’s CD player products. Nevertheless, the CD transport is designed so that it can be incorporated into the products — or technology frameworks — of other companies, many of whom are Sony’s competitors. That’s Microsoft’s vision of code reuse and software factories in a nutshell.

"This notion of the product line … is the key to this," Short says. "It’s establishing in advance architecture of a group or family of potential applications and then making sure that reusable components fit into that family."

In Microsoft’s vision, an ecosystem of ISVs, integrators, and IT organizations produces components that businesses, their partners, and even their competitors consume. In this respect, Short acknowledges, the software giant is once again bucking a historical trend — the off-the-shelf components business has long been a nifty technology idea in search of a market — but can at least point to a few winning precedents.

"Commercial, off-the-shelf components have been largely unsuccessful but also very successful in a few cases. In the early days of Visual Basic, with OCXs, there was a marvelous aftermarket and reuse, because the domain in which reuse could take place was very, very well constrained," he says. "There was a standard specification for OCX and a whole cottage industry was born around creating OCXs."

Small steps toward the software factory vision
One of the issues with code reuse — in fact, the issue that developers such as House and Bren say causes no end of headaches — is the idea, which is very popular among IT management-types, that existing code can be reused in new software projects — even when it does not fit precisely.

A common complaint is that even if a snatch of code or a component is 85% to 90% appropriate for the project at hand, stitching in the last 10% to 15% can be an incredibly painful proposition in time and money. For this reason, developers are suspicious of Microsoft’s promise of precanned, highly sophisticated components that can just snap into their application infrastructures.

"You don’t see the pain until it comes time to implement the nitty-gritty requirements, and then you say, ‘Oh man, this framework is really being stretched beyond what it was intended to be, and now we’re two-thirds of the way through the project and it’s too late to back out,’" says Miller.

Bren agrees. If the software factory approach is going to work, he argues, organizations need to start making intelligent decisions about code reuse. "High-end applications will still need custom approaches. This is only reasonable. However, common applications are certainly amenable to factory approaches, and this is what the software factory methodology is focused on," he comments. "Keep in mind that software languages themselves are already implementing substantial reusability, with entire suites of objects prepared for implementing common tasks."

See related article, "Model of integrating components."