In-Depth
Emergence
In my September 2001 column,
I wrote about swarm intelligence. I used the example of ants foraging for food
to illustrate how the interaction of low-level behaviors can lead to the creation
of higher-level behaviors. In the ant's case, a single ant's simple rules for
foraging led to higher-level behavior like shortest path optimization. Scientists
have taken this behavior and successfully applied it to optimizing the routing
of telephone connections.
Swarm intelligence is one element of a phenomenon called emergence, which is
a self-organizing behavior. In his book, Emergence: The connected lives of
ants, brains, cities, and software (New York: Scribner, 2001), Steven Johnson
describes this behavior as follows: "It's what happens when an interconnected
system of relatively simple elements self-organizes to form more intelligent,
more adaptive higher-level behavior." Emergence is surprising because we
typically look for other causes of behavior. It's also counterintuitive because
we expect an agent to direct the way things are accomplished. We rarely consider
that systems may be self-organizing.
Consider the lowly slime mold. In 2000, a Japanese scientist created a maze
with two food sources. The slime mold, without any cognitive capability, found
the shortest path to the food. How it did this is related to another amazing
capability. At various times, slime mold can exist as thousands of unrelated
and non-cooperating cells. At other times, these cells come together, cooperate
and behave like one organism. This is all emergent behavior.
So what does this have to do with IT? In business, we are starting to see a
trend toward more customer intimacy in systems for personalization, CRM and
communities of interest. This trend doesn't necessarily imply emergence, but
one example of it, eBay, is an excellent business example of emergent behavior
creating self-regulating and self-organizing communities of interest.
On the technology side, Web services are causing a trend toward building composite
apps by stringing together already-published Web services to create new business
processes. Today, this process is fairly manual, but it won't be for long as
we move toward automating the dynamic creation of apps. We already see efforts
to automate the discovery of services. This will become crucial as a wider variety
of services become available. Similar issues arise in work being done on adaptive
mobile networks. What will happen when we start to dynamically discover and
select services? Will we see unexpected interactions created when our systems
start to dynamically consume services that are automatically discovered? Will
these interactions create macro phenomena? More importantly, are the results
of all this dynamic interaction within our control? To help answer these questions,
let's look at the fundamental principles of emergent systems and how they affect
these two areas of IT.
In his book, Johnson defines five principles for emergent systems: More is
different, ignorance is useful, encourage random encounters, look for patterns
in the signs, and pay attention to your neighbors.
More is different. With foraging ants, emergent behavior won't
happen if only one or two ants are looking for food. But it will happen if there
are hundreds or thousands of ants. It appears that emergent behavior is more
likely when there are many lower-level actors involved.
Ignorance is useful. Think of the slime mold. No one is ordering
cells to come together or break apart. Individual actors have no global knowledge.
Local knowledge leads to global behavior, and simple actors work best. The more
an actor knows, the more complex they become. This is likely to inhibit emergent
behavior. Johnson suggests considering what would happen if one of the neurons
in your brain became sentient. What if they all became individually sentient?
Would there be any "you" left?
Encourage random encounters. Emergent behavior comes from many
actors randomly interacting with one another and behaving according to a simple
set of rules. The random interaction of many participants seems to be key.
Look for patterns in the signs. When ants forage, they sense
the strength of the pheromone other ants have left behind and follow the stronger
scent over the weaker one. In other emergent systems, variations within a sign
can cause different rules to be executed.
Pay attention to your neighbors. Decisions about what rules to
follow are based on an actor's observations of its neighbors. Without neighbors,
emergent behavior does not happen. The more neighbors an actor has, the more
likely it is emergent behavior will appear.
So how do these rules apply to the way business wants to deal with customers?
I've already mentioned companies creating communities of interest around their
products. What happens when this is successful? Consider eBay. Pierre Omidyar
originally created the site so that his wife could trade Pez dispensers with
other collectors. But rather than restrict itself to particular communities,
eBay has fostered the creation of hundreds of communities of interest. Buyers
and sellers are not concerned with a global view of auction commerce; they are
interested only in their chosen domain. Each community is filled with random
encounters between buyers and sellers, and each operates at times of his or
her own choosing. There is no central planning of what is offered for auction.
We also see patterns emerge -- such as different types of auctions for various
items. Finally, by paying attention to neighbors, these communities of interest
are both self-sustaining and self-regulating. We see self-regulation happen
because each actor is aware of a seller's past behavior. Sellers who commit
fraud will find themselves unable to find a buyer for other items.
Because of these behaviors, eBay appears to meet the five principles of emergence.
The end emergent behavior is the creation of numerous self-sustaining communities
of interest. But can we use this knowledge of how emergent systems work to shape
communities of interest that will benefit our businesses? And what types of
systems will we have to create to sustain this positive behavior for our companies?
On the technology side, let's look at how the five principles may affect the
area of Web services. This is more speculative, since these trends are still
emerging. The number of Web services is growing -- and new tools are making
it easier to take existing systems and publish capabilities as Web services
as well as create new ones. Currently, the selection of Web services is not
random; but with automated discovery, the automatic selection of competing services
is likely to be determined from a simple rule set that could include some randomness
or have an effect much like randomness. In Web services, ignorance equals focus.
At this time, companies are not generally interested in services available from
around the world, but they do want a range of choices for internal use. This
leads them to pay attention to the local environment.
When it comes to seeing patterns in discovery and usage, we are in an area
of speculation. I suspect it is likely that patterns of usage and interaction
will emerge from the lower-level rules that will be used for automated discovery
and assemblage of services into applications. Because of the close parallels
to emergent systems behavior, we may well see higher-level emergent effects
from dynamically created systems. When we think about paying attention to neighbors,
this suggests that the rules we create for discovery and assembly will likely
be simple and focused on local conditions. But will we recognize emergent behavior
when it pops up unexpectedly? And are we prepared to shape it in ways that benefit
the business?
Regardless of which case we consider when thinking about how emergence may
affect IT, it seems clear that we need a better understanding of emergent systems
for two reasons: We must protect ourselves from the unintended behavior and
consequences of emergent systems, and we need to work to shape emerging behavior
to benefit the business.
About the Author
John D. Williams is a contributor to Application Development Trends. He is president of Blue Mountain Commerce, a Cary, N.C.-based consulting firm specializing in enterprise, domain and application architectures. He can be reached via e-mail at [email protected].