ADT Q&A: Tom DeMarco, Atlantic Systems Guild -- ADTmag

ADT Q&A: Tom DeMarco, Atlantic Systems Guild

July 13, 2001

It is hard to remember a time when Tom DeMarco was not among the ranks of the great software thinkers. Along with people like Ed Yourdon, Lou Mazzucchelli and Tim Lister (his co-author on a seminal work on software productivity, "Peopleware"), DeMarco has moved from a preoccupation with the structured analysis paradigm.

Now key for consideration are the cultural issues that impact software programming. He recently wrote "The Deadline," (Dorset House Publishing, New York, 1997) a book that uses the novel format to uncover truths about software project management. We spoke by phone with DeMarco at his home in Maine.

Let’s go back a little. A few years ago, you concluded our software problems were ‘people problems,’ not tools problems.

I’d like to say that I always knew it in some way, but no, I think I started out believing the we had tools and methods problems, that the most typical problems were the tools/methods problems, and I came to thinking the exact opposite of that. What changed in my thinking did not exactly invalidate what I wrote in "Structure Analysis and System Specifications" [Prentice-Hall, Englewood Cliffs, N.J., 1979], but it puts it into a very different light. Because, when I first wrote, I thought the nature of problems of poor specification were entirely method problems. We just didn’t know how to write a specification. So, I came up with "Structure Analysis and System Specifications" as a better way to write specifications. And I think it is a better way to write specifications, but I don’t think it addressed the problem as I understood it. And the reason is, when you write a bad specification, it’s not because you don’t know how to write a good one because, no matter what method you use to write a specification, it is far too easy a task to explain our failures. I believe today, when something is specified ambiguously, when it simply doesn’t commit itself on whether a given function is there or whether the ability to alter a database parameter at the local site is in the system or not, what we’ve discovered in that ambiguity is the unresolved conflict among parties to the project. You have one party that wants to maintain total local control of the database, and the other party doesn’t want to allow that. It’s an unresolved conflict. So the specifier can’t commit either way because you get eaten alive by the other side, so what he does, he specifies ambiguously. It’s a survival mechanism. It’s not as though he doesn’t know how to specify it unambiguously. Instead of stating what the system does, [the analyst] papers over using the wonderful possibilities of ambiguity that are built into the English language. We come up with a spec that both parties can sign off on, but which nobody can build. There’s a perfect example of something that I thought in the ‘70s was a methods problem. In the ‘90s it’s a conflict and conflict resolution problem. Methods that I developed to deal with it were tools and techniques that were, I still believe, very good tools and techniques, but just inadequate to and not directed toward conflict resolution. So, I solved the technical problem that wasn’t really the problem, and left unsolved, the people problem that you couldn’t make any progress on without solving. You can write a spec that papers over the problem but eventually you’re going to have to commit one way or another. What the analyst or the specification writer did is to pass on the problem of conflict resolution to the programmer who is probably someone less well equipped to do conflict resolution.

With "The Deadline" you have concluded that a novel is a good way to explore these issues. And you use a character, a ‘Mr. Tompkins,’ to drive the story.

Well, I think the novel I chose to write fills a gap in the tools that managers have available to them. And the gap is often expressed to me in the form of people saying ‘I wish I could look over the shoulder of an experienced manager while he or she solves some of the problems I, as a young manager, am solving in total isolation.’ Managers, particularly young managers, often operate in tremendous isolation. They have access to their boss but seldom to their peers. It’s an odd thing. We manage these programming teams where all the lowest level people are formed into teams, and the managers often try real hard to get them to form into teams, but then we ourselves, recently graduated from the same ranks, are not part of the team in a sense that peer managers don’t undertake to do their work together. They don’t share their work. They’re private. So we don’t have a good opportunity to learn from watching others do a good job. And that’s what I set out to rectify. I set out to build a story that someone could participate in as though he or she were a fly on the wall, watching a good team of managers undertake a really meaty problem. You watch a few dozen projects being run together in an organization that Mr. Tompkins runs. The projects illustrate 30 or 40 matters that I felt were the assets of good management techniques.

I patterned the book on a [1930s] book by George Gamow called "Mr. Tompkins’ Wonderland." There was a pedagogical trick used in that book that I copied. Mr. Tompkins in the book goes into wonderlands, where some of the basic physical rules are different than they are here, and the differences make for very enlightening experiences. He could go someplace where the gravitational constant was different. By comparing experience with the new universe to the experience in the old universe, he comes to a better understanding of the ways that the physical world works. It was a pedagogical trick, and I thought I would put a manager, Mr. Tompkins I called him in deference to Gamow’s work, put a modern Mr. Tompkins into a project setting where some of the fundamental rules had been changed in a way that gave me a chance to explore how those rules affect us.

One of the rules you have in a project is that you have a limited number of people. So I contrived to put Mr. Tompkins in a situation where he has an unlimited number of people. It then allows him to, for instance, throw people at a project to save time. But he can do it in a controlled way. So for instance, he can put a team of five people to work on a project, and another team of 10 people, and another team of 30 people, another team of 50 people, all doing the same work, and let them race to see which gets it done in the first place, and see what quality results. Now, he has no time — he’s desperate to get the work done within a short period of time — but he’s got all these people, so he runs some experiments. He’s running the experiments in hopes that one of the teams will be drastically faster than the others. The other thing I altered, the thing that’s different about the universe where Mr. Tompkins has landed, is that it’s a place where some heavy-handed forces are at work. It’s one of the ex-Soviet block countries trying to become a force in the software industry.

And it’s a country that has been purchased by an individual that resembles Bill Gates …

Well, I wouldn’t want to say that. There are no resemblances in the book. The characters are entirely fictitious, of course. But this American software magnate has indeed acquired this ex-Soviet block country in a leveraged buyout. The country has gone through a totalitarian period of heavy-handed methods, so some of those methods are to be tried out on the project, and one of the most obvious ones is Mr. Tompkins finds out that his life has been put at stake on the deadline. If he doesn’t make the deadline, he’s to forfeit his life. So, it allows him to come to grips with a question that’s interesting in the abstract, but fortunately, none of us actually ever have to deal with, which is "How would you run a project if your life depended on the outcome?" And that’s very interesting because you have very little patience with things that look good on paper or make you look politically correct like ISO 9000 or CMM Advancement — those things he has no patience for at all except to the extent that they’ll actually get him to meet his deadline. He actually comes to a number of slightly unorthodox conclusions — entirely politically incorrect in the sense of corporate politics — on how he ought to run the project.

We got a kick out of a passage where a programmer sort of bases his answers to questions on deadlines based on his read of his boss’s hopes. There’s a wish stage that takes over so many projects.

Well, there’s certainly wishful thinking in projects, but let’s consider something else. Let’s consider what’s really happening there and call it by fair name which is lying. The book is very involved with this complicated subject of lying to your boss. And of course, we all know that lying is not a good thing to do. But there are circumstances in the work world where somebody asks you, for example, to come up with an estimate for when a given piece of work is going to be done, and they are practically begging you to tell a lie. Tompkins finds he has to be careful how he approaches people so that he can learn the truth from them. And that’s a complicated matter because he desperately wants to hear that a particular project is going to finish on time, but he has to ask the question in such a way that he doesn’t make the answer meaningless. So, he comes to grips in a very personal way with how to help people not to lie to their bosses. People don’t want to lie to their bosses, and, of course, bosses don’t want to hear lies. So, what do you have to do in order to make it possible for someone to tell you the truth? Particularly if it’s an uncomfortable truth. The more uncomfortable it is, the more important it is that you learn it as soon as possible because there still may be some options where later on there might not be.

One of the book’s conclusions is that smaller teams do better.

Well, we all knew that happens. We all knew that smaller teams have certain advantages. The book comes to the conclusion that many of the most aggressively scheduled projects end up over staffed. They would have a better chance to accomplish the work with a smaller team. But, unfortunately, the more aggressive your schedule is, the less safe it is to proceed with a small team. Your boss is saying, ‘Look you can have 20 people." You’re saying ‘Well, I don’t have much time. Give me six good people and I’ll take my chances.’ And if you don’t make a schedule, you’re open to the criticism that had you accepted the 20 people, maybe you would have gotten the work done on time. It’s not possible for project managers who are under a lot of pressure to under-staff, it is only possible for them to over-staff. And Tompkins, along with the rest of us, knows that if you need to get something done in 13 months or 10 months or seven months, there’s a maximum number of people you can put on that. Let’s say the maximum number of people is seven for a five-month project, you just can’t absorb people more rapidly that doesn’t prove you can get the job done with seven people. It only proves that if you put nine people on, they’ll get less done than the seven would have.

One of the book’s conclusions is that smaller teams do better.

Some of the books humor indicates you are unimpressed with formal quality awards like the Baldridge Award.

I have not been too impressed. I used the name Baldridge in the book to show that the company that I’m portraying, which is the company that fires Mr. Tompkins, is very interested in things like awards and certifications. And my implication, I guess, is that I don’t approve of that, and I don’t. It’s an example of something that puts all the proper priorities of a corporation upside down.

Our recollection is that one of the early Baldridge winners was "Neutron Jack" Welch of GE. He left the buildings and got rid of the people.

Well, it’s hard to quibble with one kind of success that General Electric has had for stockholders. On the other hand, the meanings of quality as it’s commonly understood are product meanings, and process meanings, which is not a crazy concept, but in some cases, it flies in the face of what we understand to be the protocol here. For example, ... Cadillac was receiving the award at the same time [it had serious problems with its cars’ gaskets]. Well, what’s going on? Here’s a company that is coming to be known for a product that is of less quality, but it’s receiving the award. I personally thought after that the committee should have closed up shop and given up.

Are we getting any better at big projects?

Yes, I do think so. I personally believe that the progress in the software field is astonishing. We are seeing companies that are putting together, in a very short period of time, admirable products. That’s particularly true among the vendors. The operating systems have been developed fairly well and they are enormously ambitious projects. Here I’d include Windows NT.

The upper third is getting substantially better. The average is probably getting better. What you can’t say is that the worst is getting better. If you really want to make progress in the industry, than all your focus should be on improving the worst, improving the people that do things at the lower 30 percentile. They drag down the averages much more than the people at the top are pulling the averages up.

What I can’t say is that the lower quadrant of performers is any better. There are always plenty examples of failures. The FAA has practically an unbroken string in an absolutely essential area if ever there was an area that needed the best and the brightest of software builders and the correct motivations and the proper schedules and the best methods it was these air traffic control systems. And those projects have not illustrated the general trend of getting better and better.

There’s reasons for that. The FAA is, I believe, the problem itself. The consultants who have helped build systems for the FAA reads like a "Who’s Who" of system development, I think. But the companies that built and failed to deliver are in general the companies that are doing the best work in our field. Why did they fail?

Well, most of these projects were hampered by a series of unresolved conflicts between the Regions and the Central FAA that resulted in a development teams setting out to build systems that had forced ambiguity — that is papered-over conflict in them. And it’s a coarse simplification but I think you could almost say that none of those projects had a real spec because of the difficulty of resolving the conflict.

Is there something good coming out of year 2000 problem in that people are looking at methodologies anew?

I wouldn’t expect much of anything good to come out of the year 2000 problem. What I would expect is that some companies will solve the problem admirably, but that just means they’ll be back where they started. There’s no benefit in solving the year 2000 problem, except you avoid a negative benefit, which is the reason we’re having so much difficulty in some companies undertaking the problem at all.

Many companies that declared themselves to be ‘lean and mean’ in the early ‘90s when they laid off all their Cobol programmers are learning that they laid off the very people that they could now use to solve the year 2000 problem. I spent a lot of my time as a litigation consultant over the last few years. Much of it is the direct result of companies that went overboard in becoming lean and mean and declared that they weren’t going to hear any reason why they couldn’t get a system built in two years, or one year, or whatever. So they signed on contracts to do that and told their people to produce or else. Their people didn’t produce and were probably fired, I don’t know, but the companies found themselves being sued left-and-right because they tried to do things that were patently absurd. I spent a lot of my time as a litigation consultant over the last few years I think largely mopping up from the flirtation American industry

I’m going to have a lot of litigation from the year 2000 from the companies that do one of two disastrous things. One is that they either ignore the problem all together because their incentive system is still reminiscent of ‘lean and mean’ and it isn’t allowing them to do something that doesn’t produce real functionality, and they can safely ignore this only if they are willing to leave the company shortly before. I think there are plenty of companies where the CIO are ignoring the problem, collecting big bonuses until 1999 and then they’ll leave. And the other problem comes from the same cause when somebody looks at the possibility of solving the year 2000 problem by pulling all those defects out of the code, and realizes that he’ll be no better off than he is today except he’ll be allowed to proceed into the 21st Century. But decides not to do that because the incentive system doesn’t reward that sort of thing and instead decides to replace the year 2000 non-compliant system with a system that does have some additional features, and the features, therefore, will justify the work. The problem is that that inserts a very fixed deadline when that new project has to be finished, and we all know that many projects do not get finished by time the deadline is no matter how strenuously people are saying it has to be done by the deadline.

Dilbert has brought software project management to the masses. Do you look at that?

Yes, Dilbert is what the 20thcentury is all about. It’s an absolutely ingenious work. There’s actually only one joke that runs through Dilbert. And it is the idea that the two kinds of authority get mismatched. Let me point out to you that the word ‘authority’ in English has two totally different meanings. You may be ‘in authority’ over someone as with a boss. Or you ‘have authority’ — you have some knowledge that the other person doesn’t. You are the authority on function points or C++ or objective analysis.

The running joke in Dilbert that is also the running joke in many companies. That is that sometimes these two different types of authority get completely out of whack. That is that someone who is ‘in authority’ has no natural authority, who is a time-wasting moron, and that is almost the whole humor of Dilbert.