In-Depth

Emergence

• By John D. Williams
• May 1, 2002

In my September 2001 column, I wrote about swarm intelligence. I used the example of ants foraging for food to illustrate how the interaction of low-level behaviors can lead to the creation of higher-level behaviors. In the ant's case, a single ant's simple rules for foraging led to higher-level behavior like shortest path optimization. Scientists have taken this behavior and successfully applied it to optimizing the routing of telephone connections.

Swarm intelligence is one element of a phenomenon called emergence, which is a self-organizing behavior. In his book, Emergence: The connected lives of ants, brains, cities, and software (New York: Scribner, 2001), Steven Johnson describes this behavior as follows: "It's what happens when an interconnected system of relatively simple elements self-organizes to form more intelligent, more adaptive higher-level behavior." Emergence is surprising because we typically look for other causes of behavior. It's also counterintuitive because we expect an agent to direct the way things are accomplished. We rarely consider that systems may be self-organizing.

Consider the lowly slime mold. In 2000, a Japanese scientist created a maze with two food sources. The slime mold, without any cognitive capability, found the shortest path to the food. How it did this is related to another amazing capability. At various times, slime mold can exist as thousands of unrelated and non-cooperating cells. At other times, these cells come together, cooperate and behave like one organism. This is all emergent behavior.

So what does this have to do with IT? In business, we are starting to see a trend toward more customer intimacy in systems for personalization, CRM and communities of interest. This trend doesn't necessarily imply emergence, but one example of it, eBay, is an excellent business example of emergent behavior creating self-regulating and self-organizing communities of interest.

On the technology side, Web services are causing a trend toward building composite apps by stringing together already-published Web services to create new business processes. Today, this process is fairly manual, but it won't be for long as we move toward automating the dynamic creation of apps. We already see efforts to automate the discovery of services. This will become crucial as a wider variety of services become available. Similar issues arise in work being done on adaptive mobile networks. What will happen when we start to dynamically discover and select services? Will we see unexpected interactions created when our systems start to dynamically consume services that are automatically discovered? Will these interactions create macro phenomena? More importantly, are the results of all this dynamic interaction within our control? To help answer these questions, let's look at the fundamental principles of emergent systems and how they affect these two areas of IT.

In his book, Johnson defines five principles for emergent systems: More is different, ignorance is useful, encourage random encounters, look for patterns in the signs, and pay attention to your neighbors.

More is different. With foraging ants, emergent behavior won't happen if only one or two ants are looking for food. But it will happen if there are hundreds or thousands of ants. It appears that emergent behavior is more likely when there are many lower-level actors involved.

Ignorance is useful. Think of the slime mold. No one is ordering cells to come together or break apart. Individual actors have no global knowledge. Local knowledge leads to global behavior, and simple actors work best. The more an actor knows, the more complex they become. This is likely to inhibit emergent behavior. Johnson suggests considering what would happen if one of the neurons in your brain became sentient. What if they all became individually sentient? Would there be any "you" left?

Encourage random encounters. Emergent behavior comes from many actors randomly interacting with one another and behaving according to a simple set of rules. The random interaction of many participants seems to be key.

Look for patterns in the signs. When ants forage, they sense the strength of the pheromone other ants have left behind and follow the stronger scent over the weaker one. In other emergent systems, variations within a sign can cause different rules to be executed.

Pay attention to your neighbors. Decisions about what rules to follow are based on an actor's observations of its neighbors. Without neighbors, emergent behavior does not happen. The more neighbors an actor has, the more likely it is emergent behavior will appear.

So how do these rules apply to the way business wants to deal with customers? I've already mentioned companies creating communities of interest around their products. What happens when this is successful? Consider eBay. Pierre Omidyar originally created the site so that his wife could trade Pez dispensers with other collectors. But rather than restrict itself to particular communities, eBay has fostered the creation of hundreds of communities of interest. Buyers and sellers are not concerned with a global view of auction commerce; they are interested only in their chosen domain. Each community is filled with random encounters between buyers and sellers, and each operates at times of his or her own choosing. There is no central planning of what is offered for auction.

We also see patterns emerge -- such as different types of auctions for various items. Finally, by paying attention to neighbors, these communities of interest are both self-sustaining and self-regulating. We see self-regulation happen because each actor is aware of a seller's past behavior. Sellers who commit fraud will find themselves unable to find a buyer for other items.

Because of these behaviors, eBay appears to meet the five principles of emergence. The end emergent behavior is the creation of numerous self-sustaining communities of interest. But can we use this knowledge of how emergent systems work to shape communities of interest that will benefit our businesses? And what types of systems will we have to create to sustain this positive behavior for our companies?

On the technology side, let's look at how the five principles may affect the area of Web services. This is more speculative, since these trends are still emerging. The number of Web services is growing -- and new tools are making it easier to take existing systems and publish capabilities as Web services as well as create new ones. Currently, the selection of Web services is not random; but with automated discovery, the automatic selection of competing services is likely to be determined from a simple rule set that could include some randomness or have an effect much like randomness. In Web services, ignorance equals focus. At this time, companies are not generally interested in services available from around the world, but they do want a range of choices for internal use. This leads them to pay attention to the local environment.

When it comes to seeing patterns in discovery and usage, we are in an area of speculation. I suspect it is likely that patterns of usage and interaction will emerge from the lower-level rules that will be used for automated discovery and assemblage of services into applications. Because of the close parallels to emergent systems behavior, we may well see higher-level emergent effects from dynamically created systems. When we think about paying attention to neighbors, this suggests that the rules we create for discovery and assembly will likely be simple and focused on local conditions. But will we recognize emergent behavior when it pops up unexpectedly? And are we prepared to shape it in ways that benefit the business?

Regardless of which case we consider when thinking about how emergence may affect IT, it seems clear that we need a better understanding of emergent systems for two reasons: We must protect ourselves from the unintended behavior and consequences of emergent systems, and we need to work to shape emerging behavior to benefit the business.