News

Failed project talking or Tales of the FBI

• By Jack Vaughan
• June 14, 2004

Failed projects don't talk, at least not too often. But we should listen to them when they do. People generally want to focus on the positive, but we need to educate ourselves and a trip up a rocky road can be more educational than a cruise down the autobahn. The FBI's Trilogy IT modernization program, heavily documented by now, is a case in point.

Some fumbled cases made it clear to many observers by 2001 that the FBI's computer systems were far from state of the art. A modernization program was launched. Now, in 2004, that program is still struggling to get fully on track.

A report recently released by the Computer Science and Telecommunications Board (CSTB), which is empowered by the National Research Council, points out planning flaws in the Trilogy program. It is a rare and deep look into a failed promise to quickly modernize. The report concludes that, after many improvements, the program is not yet on track to success.

So what went wrong?

Some of the usual things. For example, we know that planning can easily get short shrift when there is a push to get results fast. It seems this happened in the FBI IT organization in a big way.

And requirements change, which happened here in an extraordinary way.

It seems a major shift in focus in the wake of terrorist attacks on the Pentagon and World Trade Center on Sept 11, 2001, led to changes in requirements that heaped new portions of risk on the already-difficult project. Basically, the FBI was asked to do more analysis of intelligence to stop another event such as 9-11 from happening at the same time it was to continue to pursue investigations and mount cases against all sorts of criminals.

We recently had an opportunity to speak with Ken Orr, a member of the CSTB as well as a mainstay of software engineering and data design over the years. A phone conversation with Orr, of the Cutter Consortium and Ken Orr Institute, helped to highlight some of the findings on Trilogy.

'The number one problem is their [the FBI's] enterprise architecture. They didn't have one,' said Orr, who noted problems with the project's data architecture as well. Naturally, this is expanded upon in the full report.

The report notes that management of enterprise architecture design cannot be farmed out of an organization, nor can it be entrusted solely to a CIO within an organization. True management buy-in is also needed. The senior leadership of the FBI needed to be directly involved in creating this strategic view and supporting its implementation.

'It's an old organization. The systems they have been working with have been pretty out of date for a while and are very siloed. They are trying to change their stripes and that is hard,' said Orr.

The change in marching orders after the terrible attack on the U.S. was natural, but it didn't help a project already in trouble. It would also seem to evoke a variation on Fred Brooks's Law, which states that 'Adding manpower to a late software project makes it later.'

'This started before 9-11 and there was a change in charter,' said Orr. The agency's mandate had been 'primarily law enforcement and secondarily domestic counter intelligence. Now that has mushroomed,' he noted.

'After 9-11, they said 'We've got to do it in two or three years, instead of three or four years.' That didn't happen because they tried to do it faster,' said Orr.

At the same time, the agency tried, in effect, to begin to mix operational and analytical databases. Of course this application had very unique requirements and issues. One of these issues was that you could anticipate more than the usual difficulty in gathering useful system requirements.

'If you look at it from a traditional standpoint,' said Orr, 'you see sources are sacrosanct for the operations or law-enforcement people. Meanwhile, the analysts want to look at everything.' This is an issue in the pre-computer world that has ramifications in the present world. Moreover, one can only imagine how difficult it would be under the best of circumstances for requirements gatherers to gain information from people who, by nature, must be close-lipped.

Writer William Saroyan was famously quoted as saying 'We get very little wisdom from success, you know.' One comes to wisdom through failure, he said. There are indications that the FBI is set to put its IT house in order and ready to learn from a study of Trilogy's flaws. But more must be done. Orr said that the FBI has made some good personnel moves in its efforts to get Trilogy on track, although some aspects of the program - notably a propensity to outsource responsibility for architecture - may still bear watching.

A PDF version of the report from the Committee on the FBI's Trilogy Information Technology Modernization Program can be found at http://books.nap.edu/html/FBI/0309092248.pdf .