Make complexity work for you

• By John K. Waters
• October 21, 2002

[PROGRAMMERS REPORT, OCTOBER 22, 2002] -- Medical scientists are using complexity science to understand the patterns that underlie health and disease. Sociologists are using it to figure out human society. Business analysts are using it to gain insights into market structures and to develop competitive strategies. And some software engineers are beginning to use it to quantify the differences between good and bad software.

''The missing piece of the software engineering puzzle,'' Alex Iskold told Programmers Report recently, ''is understanding complexity. That is also what's missing from the existing tools and methodologies.''

Iskold is CEO and co-founder of Information Laboratory, a New York-based analytical software solutions provider. During his career, Iskold has designed a number of large-scale software systems, and he has taught software engineering at New York University. He and co-founder/CTO Dan Kogan started Information Laboratory in 2000 to apply the new ideas of complexity science to many different areas, in particular, to software engineering.

''When we started the company, we had this clear vision that complexity science had a lot to say about how to do software development,'' said Iskold, ''and in particular about large-scale software development.''

Complexity science is described as a multi-disciplinary approach to the study of complex systems. It evolved out of General Systems theory and quantum physics in the mid-20th century. Basically, it is the study of systems composed of many and varied parts that interact in complex and non-linear ways. Complexity science recognizes that such systems can be understood only by looking at the interactions among their parts and the consequences of these interactions.

Iskold explains the concept this way: ''Complex systems are made out of smaller systems, and small systems interact and form a network; the complex system is essentially the emergent property of the interacting subparts.

''The basic insight [of complexity science] for us,'' Iskold said, ''is that, because any kind of large-scale software system is a model of something very real -- a trading system, a car, a supply chain -- it is itself a complex system and must obey the laws of complexity.

''And then we asked the question, 'Can we, by looking at the structure of this complex system, actually distinguish good software from bad software?'

''And, sure enough,'' he asserted, ''we found that you can.''

All software systems have very definite ''structural signatures,'' said Iskold. Good systems are highly modular, with component dependencies kept to a minimum; bad systems have components with excessive responsibilities and involve large-scale cyclical dependencies.

''Once we looked into this closely,'' Iskold said, ''we began to identify simple structural patterns that occur in bad software. For example, when people write large-scale systems, a lot of companies suffer from the problem that code is intermingled. Intermingled code means that every component has many dependencies, most of which are unnecessary -- extraneous dependencies, components that are hooked into tangled webs. This is structurally very bad, because when you make a change to any component it starts traveling and spills all over the system. And that's when people say, 'I fixed this and something else broke.'''

Looking at large-scale software systems within the context of complexity science has led Information Laboratory to invent a new discipline, which the company calls Structural Analysis of Large Scale Systems. ''We are basically claiming that it's not a matter of your technology,'' Iskold said. ''It doesn't matter if it's .NET or J2EE. It boils down to dependency management and the complexity of the system.''

Along with a new way of thinking about software, Information Laboratory's new discipline has produced a tool for analyzing and visualizing the structure and dynamics of large-scale systems. First introduced in September of last year, Small Worlds is one of the first tools to take on the problem of structural dependencies and system complexity.

''What we've done [with Small Worlds] is to create a framework for taking an arbitrary software system and transforming it into a gigantic dependency web, parsing it and then modeling it,'' explained Iskold. ''We also created several structural metrics that help us to very quickly assess the overall health of the system.''

Small Worlds provides users with a variety of visual models and statistics, including views for managing component and package dependencies, tracking the effects of change, system coherency and summary reports. The views help users to identify problem areas in the code that may need system-level refactoring.

The current 2.0 version of Small Worlds allows for analysis of C++ and Java-based systems, Unix shell scripts and a Forte plug-in module. Future releases are planned for database analysis and for systems written in C, C#, Visual Basic, COBOL and J2EE. The 3.0 version, which has no projected release date, will be a ''leap,'' Iskold said, providing overall views of the entire enterprise.

For more information about complexity science, Iskold recommends a book he says changed his life: Complexity: the Emerging Science At the Edge of Order and Chaos , by M. Mitchell Waldrop (Touchstone Books, 1992).

To find out more about the Waldrop book at Amazon.com, go to http://www.amazon.com/exec/obidos/tg/detail/-/0671872346/qid=1034902168/sr=1-1/ref=sr_1_1/103-6737774-1962217?v=glance

To find out more about Small Worlds software, go to http://www.thesmallworlds.com/

Links:
For other Programmer Report articles, please go to http://www.adtmag.com/article.asp?id=6265