Call for expert systems
- By Ivar Jacobson
The software development process has never been so complex. Software developers
are intensive knowledge workers. Every day, they must apply their hard-earned
development skills to the fullest.
They must continuously learn more because new trends and technologies emerge
while yesterday's become obsolete. They must not only understand new trends
and technologies, they need to know how to adopt them quickly and productively
-- a tough assignment even for the hardest-working experts.
More specifically, relevant knowledge manifests itself in terms of best practices,
guidelines, methodologies and the like. In particular, this kind of knowledge
manifests itself in terms of development standards like UML or the Unified Process;
application platform and middleware standards such as .NET or J2EE; and lower-level
implementation standards such as Java, C#, HTML and XML. Developers must access,
distill and interpret such knowledge -- continuously and just-in-time -- for
the system under construction.
Many of the problems that currently bedevil projects stem from an inability
to apply relevant knowledge. Developers may lack both training and experience,
often because of the shortage of skilled people. And the demand for skilled
people is expected to increase by more than 100% in the next six years.
Projects are always rushed, which leaves developers limited time for learning.
So often, relevant knowledge that can take too much time and effort to access,
distill and interpret remains unexploited.
The consequences of this are devastating. The exclusion of relevant knowledge
results in lower productivity, higher costs and sub-standard quality. It may
even lead to project failure when development tasks become too challenging.
If nothing is done, this situation will only get worse. Technology is getting
more complex, business requirements are becoming harder to fulfill, lead times
have become shorter and quality requirements constantly become higher. There
is no magic way out. The question is: What can be done about it?
The past vision
In 1980 I wrote a letter to my CEO proposing that European high-tech firm Ericsson
leverage its component-based software development approach in three steps: first,
make the Ericsson software modeling language a standard language; second, develop
a standard Ericsson software development process; and third, develop a support
system that employs knowledge-based reasoning.
As for Step 1, Ericsson had already (partly) succeeded by standardizing a language,
SDL, as a world standard for telecom systems (see Specification and Description
Language, ITU-T Recommendation Z.100, March 1993). SDL inspired the development
of the Unified Modeling Language (UML) (see www.omg.org/uml),
which was adopted by the Object Management Group (OMG) as a standard in 1997.
Many of the ideas in UML -- such as sequence diagrams, collaboration diagrams,
activity diagrams and state charts -- were in use at Ericsson in the late 1960s.
As for Step 2, the work process used at Ericsson was actually a forerunner
of the Objectory process that has now evolved into the Rational Unified Process
(RUP) (see www.rational.com/rup),
which is becoming a de facto standard. It also inspired an OMG standard for
process modeling called the Software Process Engineering Metamodel (SPEM) (see
As for Step 3, knowledge-based systems, expert systems, artificial intelligence
(AI) systems and the like were judged to be very promising new technologies
in the 1970s and early 1980s. Unfortunately, many of the companies that were
founded to take advantage of the new technologies failed and were gone by the
late 1980s. The implementation technologies available at that time just did
not fulfill high and sometimes over-hyped expectations.
Two decades later, the time has finally come for knowledge-based systems to
take a stab at the challenges of software development and the chaos these challenges
bring. The standards and technologies required to meet these challenges are
now in place.
The software developer assistant
Thus, we introduce the notion of a developer assistant (DA) to provide some
of the expert knowledge a developer needs, such as development guidelines, language
specifications and process descriptions. The DA can reason with that knowledge
to help achieve results comparable to those of expert developers.
Instead of requiring a software developer to have all requisite knowledge,
the DA can support developers in three ways: by capturing a significant part
of the knowledge, by reasoning based on this knowledge and by automating the
application of this knowledge.
The fundamental requirements are:
* The DA must by its nature be part of an integrated development environment.
It must harmonize, interact and co-exist with other tools for tasks like modeling
* The DA must understand the goals of the developer at any particular
time. For example, a goal could include the software artifacts a developer is
about to produce, and the expected level of quality.
* The DA must be able to reason in the developer's own context. The
reasoning needs to be done as needed, often in dialogue with the developer.
This capability calls for mechanisms within the DA to structure knowledge. An
important coordinate condition is that it does not interfere with other tasks
of the developer.
* The DA must be based on development standards such as UML. Capturing
DA knowledge requires significant effort and investment.
* The DA can be focused and configured to suit the needs of specific
projects or organizations. These needs may, for example, depend on the type
of software system being developed (such as banking, insurance, telecom or air-traffic
control), or on the size and complexity of the project or organization. Therefore,
the knowledge provided by the DA may be used selectively, and that new knowledge
may be added specific to the needs of the project or organization.
* The knowledge provided by a DA must be easy to update, because change
is natural when it comes to knowledge. Simply put, knowledge must be structured
for "plug and play."
We have reached a milestone where such requirements are fully realizable. We
can build a DA for software development that can be useful from day one. We
can also grow through changes to operate over the long run.
Integrating the development environment
The DA is not an island. On the contrary, the DA needs to be a complement to
a more complete environment -- an environment that contains supporting technologies
like modeling and coding tools, configuration management tools and the like.
Existing tools can help developers in specific areas such as requirements capture,
design, implementation, architecture and configuration management. These can
offer sophisticated functionality to describe, maintain and visualize software
artifacts, along with an ability to integrate with other tools. Existing tools
can continue to perform such functions while continuing to be improved through
Current tool strategies focus on supporting developers' knowledge -- there
is plenty of room for improvement. Despite some initiatives to "open up"
knowledge (such as "dynamic help" in Visual Studio), it is often still
locked in passive repositories or static help systems. Existing tools often
only serve up specific "knowledge pages" like tool manuals and development
guidelines, based on explicit requests. Such tools are, in many respects, read-only
dumb terminals -- developers can only browse information when they know what
they need. It is often difficult to apply such knowledge to the problem at hand
because it is not served in a useful way.
Developers must be able to access knowledge in a productive manner, set the
knowledge in context and apply it to the problem at hand. This is the DA's role.
To complete such a mission, the DA must interact with other development tools.
For example, to evaluate knowledge, the DA must detect the state of artifacts
managed within other tools. As the developer applies knowledge, any automated
support provided by the DA needs to refine the artifacts accordingly. Moreover,
the DA may need to refer to specific knowledge, such as help topics, in existing
online documentation within the other tools of the development environment,
instead of having to redefine this information within itself. This integration
will be illustrated later.
The developer's goals
A project's goals are often defined in terms of milestones. A milestone concludes
a phase or iteration, and is more or less standardized.
Based on the overall goals, one must set up the goals for each developer participating
in the project. This is -- from a technical standpoint -- naturally done in
terms of roles, activities and artifacts (R/A/A): Developers adopt roles
as they perform activities that result in artifacts.
Roles: A role is adopted by a developer or by a group of developers
working together. It defines a set of related activities best performed by an
individual or group, and a set of related software artifacts that are the responsibility
of the individual or group adopting the role.
Activities: An activity is performed by the developer and defines
a focused piece of development work; an activity is performed by using some
existing artifacts as input, and produces as results new or refined artifacts
Artifacts: An artifact is created or maintained by a developer,
and can be a software work product like the source code for a class, a use case
(including its description), or any other model element. Such an artifact is
a part of the software system under construction.
Thus, a DA for software development needs to understand the R/A/A that are
relevant to the software developers.
As an example of R/A/A, consider a developer adopting the System Analyst
role and performing its related activities Find Actors and Use Cases and
Structure the Use-Case Model; these activities result in a refined Use-Case
Model artifact that captures requirements made on the software system.
This structure is applicable in any development discipline. Additional examples
are class design activities performed by designers that result in classes, and
test activities performed by testers that result in test specifications and/or
From this, we can draw the following conclusions:
What knowledge to capture. The most essential knowledge to be
captured by the DA and dynamically provided to the developer is how to adopt
roles and perform activities in a productive and focused manner. More specifically,
it is knowledge about how to develop "good" artifacts ("good"
model abstractions and "good" runtime components).
When to apply knowledge. The activities are excellent instruments
for making this determination because, given that the DA understands when the
developer performs an activity, it can support this particular context. The
DA does this by preparing parts of the knowledge significant to the activity
or micro-activity in progress, and by immediately serving it to the developer
"just in time."
What to apply knowledge to. The answer is obvious: Knowledge
should be applied to software artifacts as they are developed in activities.
The application of knowledge then refines existing artifacts in various ways.
Thus, a DA must understand the developer's goals -- the roles, activities and
artifacts (R/A/A) relevant to the developer. As a result, we understand what
knowledge to capture, as well as when and how it should be applied.
Making knowledge useful and accessible
As a vehicle for capturing, structuring and applying knowledge, we employ a
rule-based language so the DA can serve knowledge in the right context (roles
and activities) just when it is needed and apply it automatically to the software
system (artifacts) under development.
When knowledge is to be captured, consider goals in terms of R/A/A. From these
one can derive structured knowledge in terms of rules.
This rule-based language primarily uses ideas from some well-known formal languages,
such as first-order predicate logics, basic collection and set operations, and
the Object Constraint Language (see Reference 1). The rules in this language
basically detect interesting states of software artifacts and then propose new
and better states of those artifacts. The overall purpose, of course, is to
enhance the quality of artifacts under development.
Detecting existing state. The first step in capturing knowledge is to
decide what state of the knowledge domain is interesting to reason about. One
example of an interesting state is faults such as errors and inconsistencies
in artifacts. Another example is a state of incompleteness such as an artifact
that is missing or not fully described.
Proposing new states via rules. The second step of capturing knowledge
is to decide what to do with a detected state of a software artifact, thereby
proposing a new state of the artifact. Such a new state can include, for example,
that we create other (new) artifacts related to the first artifact, that we
provide refinement opportunities in the artifact, or that we fix errors or inconsistencies
in the artifact.
To specify this action, we employ a refined variant of if-then rules, sometimes
also called productions, situation-action rules (see Reference 2) or condition-action
rules. Each such rule is then used as a mechanism to decompose and structure
knowledge into small, manageable and focused chunks that can be automated.
The "then" part of rules can either completely automate reaching
a new state, or can help the developer reach the new state via a dialogue. Moreover,
the "then" part of rules can sometimes propose several alternative
new states, especially when there is no single recommendation on how to handle
To give more concrete examples of if-then rules, let's consider the System
Analyst role and its related activities, Find Actors and Use Cases
and Structure the Use Case Model. This example results in a refined Use
Case Model artifact (as discussed above). From the Find Actors and Use
Cases activity, we derive a simple rule example.
The rule in this example is based on the assumption that every concrete use
case in the use case model should be associated with an actor. Thus, if a concrete
use case not associated with any actor were detected, we could specify a rule
that proposes new states of the use case.
When this rule fires, there may be many reasons why an association is missing.
However, the "then" part proposes a number of alternative recommendations
for correcting the model.
For this rule to be useful to a developer in a concrete situation, it must
describe in more detail the detected (suspect) state of the model and the consequences
of following the corrective recommendations. We still include the simple rule
here to communicate the notion and essence of if-then rules in a real situation.
Some other rule types we have found are:
* Rules to get started (initially) building software system models by
detecting more or fewer empty models and then suggesting ways to start identifying
and describing new elements within the model.
* Rules for model completion that identify missing model elements such
as subsystems, classes, relationships and so on, and then propose corresponding
* Rules for employing software patterns (see Reference 3).
* Rules for evaluations and reviews of models that identify checkpoints
such as formal inconsistencies, errors or anti-patterns in the model, and then
propose corresponding model corrections.
This approach to using rules can support all stages of development: the initial
stage, the more creative stages, the more ongoing work and, finally, the evaluations
and reviews to assure the quality of artifacts before delivery. This approach
is important because almost all adoptions of roles and activities go through
Imagine thousands or even tens of thousands of rules being packaged and served
by the DA just-when-needed within the roles and activities adopted by a software
developer. The DA can help developers make the correct decisions at the right
time by automating the use of relevant expert knowledge. Such rules can let
the developer take action if he or she finds it appropriate; the more and better
rules are, the better a developer can progress.
Intelligent software agents
Since the late 1980s, the development of software-based agents has skyrocketed
in a wide range of application areas (see References 2 and 4). Many important
academic institutes, such as MIT and Carnegie Mellon, have formed significant
agent research groups. There is also standardization work on agents by consortia
like the OMG. There are interest groups that base agent technologies on standard
modeling languages like UML. And on the commercial side, we see agent technology
from key suppliers, like Microsoft Agent, for example. And there are Web sites,
such as www.agentland.com, that list
agent-based companies and products.
Something that can serve as yet another concrete example of this general trend
is an online bibliography like the one on software agents available at the Collection
of Computer Science Bibliographies (see Reference 5). At press time, the site
listed an unbelievable 2,480 agent-related references, most of which were written
between 1997 and 2001.
Key agent characteristics
In general, one can discuss the characteristics of software-based agents, but
an industry-standard definition of an agent has not yet emerged. But here is
what I think are the key characteristics of a software-based agent -- those
that distinguish an agent from "ordinary" objects and components.
These key characteristics are:
* An agent is based on a significant set of knowledge and thus
can reason. Within the DA, agents primarily represent knowledge in terms of
rules. Agents may also represent knowledge in other ways -- in terms of attributes,
states and the like.
* An agent has autonomous behavior in that it is self-driven
and acts on its own behalf without direct external intervention. Based on its
ability to reason, it can decide when to act. When the agent draws conclusions,
it can notify its environment, such as other agents or a developer. Given those
conclusions, it can even take actions by itself. The environment, such as a
developer, can then decide what to do with the agent's conclusions. Within the
agent, this autonomous behavior is driven by rule machinery that proactively
* An agent has a well-defined goal represented by the knowledge
it possesses and by its behavior: that is, its ability to reason given that
knowledge. An agent simply tries to reason to fulfill its goal.
We may also discuss other characteristics of agents, such as collaboration,
adaptability, mobility, life cycle and so on, if required, but we think these
are more like consequences or aspects of the above. Given this limitation, what
do these key characteristics really mean when considering the DA as an agent
The DA is a multi-agent system
The basic idea is that the DA should be implemented as a collection of software-based
agents, or as a multi-agent system. An agent is created to manage a role adopted
by the developer, an activity being performed by the developer, or a software
artifact the developer is responsible for. Thus we have role agents, activity
agents and artifact agents.
The rationale for this model is:
* The role agent's goal is to help developers adopt roles and to be responsible
for artifacts, as well as for starting and performing activities as appropriate.
* The activity agent's goal is to help a developer perform an activity.
It includes providing directions and guidelines within the activity on what
to do, and how and when to do it.
* The artifact agent's goal is to help maintain a correct state of the
artifact. The artifact should be well formed according to its definition of
both syntactical and semantic aspects.
* When an activity is performed, a set of artifacts is input and output
as results. An artifact agent can be created to handle each such artifact.
* Other development tools in the development environment serve as a
repository for artifacts as seen from the DA's perspective (see above).
* The Artifact Proxy entity represents "passive" artifact
data acquired from the other tools, as the original artifact physically lives
in these external tools. The knowledge (rules) of the various agents may need
to access this artifact data as rules are evaluated. Moreover, any change of
this artifact proxy (as initiated by agents) is assumed to be reflected in the
actual artifacts residing in the other development tools.
This agent system is then configurable in the sense that the developer can select
what types of agents to use, and then adjust the knowledge (rules) of these
agents if required.
The agent system: A concrete scenario
The concrete scenario within the agent system illustrates the adoption of a
role. This scenario can be described as follows: Joe, a software developer,
decides to adopt the System Analyst role (adopt role). A corresponding role
agent is created.
To start with, the role agent suggests that Joe should be responsible for the
Use-Case Model artifact. Joe accepts this responsibility, and a corresponding
artifact agent is created to assist Joe (assign responsibility).
Then the role agent suggests that Joe start performing the Find Actors and
Use Cases activity to develop the Use-Case Model. Joe agrees and a corresponding
activity agent is created (perform). Then the activity agent is set up to refer
to the artifact proxy regarding input and output of the activity (set input/output),
and the corresponding artifact agent is initiated to handle the output artifact
(initiate); Joe of course confirms this.
Now, the artifact and activity agents start to reason, given their specific
knowledge of rules. To evaluate the rules, they may access and refine the Use-Case
Model artifact proxy (access/refine). Then, additional support from these agents
is provided in a dialogue with Joe (support).
This is continued as follows: Based on the dialogue with Joe, more refinements
(access/refine) are made on the Use-Case Model artifact. At some point, the
role agent may propose that other activities be performed -- such as Structure
the Use Case Model -- then if Joe accepts, corresponding activity agents
would be created. This way, new agents are created, supported and finally terminated
based on the developers' goals.
Of course, the developer can at any time work with the artifacts directly in
other development tools. However, the basic idea is that the DA is proactive
and continuously provides support (recommendations on how to solve relevant
problems) to the developer, given the developer's goals. The developer may turn
to the DA whenever he or she feels the need.
Each agent can support the developer via a fairly large number of rules. The
agent presents the rules, and then it is often up to the developer to select
a single rule and apply it in context. There are ways to make rules easier to
use. One way, for example, is to categorize them according to steps within an
activity agent. Another way is to prioritize them.
Many rules can sometimes appear simultaneously. For example, within an activity
agent the many-rules approach typically fails to adequately support a novice
developer. The novice may be more passive, which may require the DA to be more
To handle this problem, an agent can group related rules into larger constellations
of rules. The novice can then use the rules in sequence. Each such rule constellation
can support more large-grained modeling tasks than individual rules can do per
se. A constellation also gives a more proactive support by guiding the developer
through several rules in sequence. Thus the developer is not required to decide
individual rules for each task.
There are many kinds of "large-grained modeling tasks," or micro-activities,
that have been identified as appropriate for this kind of support. For example,
one can identify all elements of a certain type (actors) in a model. Another
is to merge or split complex model elements, as in refactoring. A third is to
review a model element by going through a set of checkpoints. All these micro-activities
need to involve several more primitive rules that can be applied in sequence.
To implement the DA as an agent system, a number of components are required
to realize the different agents.
The components are the following:
Rule language compiler and rule machinery. Because rules are specified
using a high-level rule language, they need to be compiled into a format that
can be evaluated by rule machinery. The rule machinery is, in turn, incorporated
and used within each agent, so an agent can evaluate the rules relevant to itself
in its own context.
User experience. The developer experiences the agents through
a user interface that handles all user input to and output from the agents.
Model server. Agents and their rules need to access and refine
the artifact proxies (when driven by a developer). This access, in turn, needs
to be reflected in the development tools hosting the artifacts. Consequently,
the purpose of the model server is to manage all artifact proxies relevant to
the agents together as a complete model, to keep proxies updated and consistent
with the original artifacts hosted in the external development tools, and to
update the artifacts in these tools as requested by agents.
Process configuration. The development process supported by the
agent system often needs to be configured to suit a specific developer and project.
The degree of this adaptation depends, for example, on the software quality
ambitions of the project. It also depends on the complexity and type of software
system being developed. The purpose here is to configure the DA to support a
specific development process personalized to a particular developer.
A transparent development process
Many examples show that adopting a development process can yield success. But
the adoption must be sound and incremental. It must include adequate mentoring
and training of the development team. It must be based on the use of appropriate
development tools. There are, of course, challenges with this approach. One
is the place of "soft factors." The Unified Process, for example,
puts its primary focus on methodology and how to master technology. That still
leaves a wide range of soft factors, including how to treat people as individuals,
how to make them prosper, how to consider their competencies, how to deal with
their attitudes, how to facilitate the means of communication and how to encourage
In particular, there is a significant challenge with the description of a process.
Potential users can perceive a process as too large and too generic. They can
find it out of context and off-target. This difficulty in communication can
make it hard for developers and teams to digest the process. In the worst case,
it can decrease their drive and creativity on the job. Thus there is a risk
that the process can hinder more than it helps. It can become a burden and lead
developers to discard the process and start inventing new processes on the fly,
resulting in an out-of-control project.
The DA does not handle all these challenges by itself. However, the DA can
serve process on a silver platter by making it more alive and by streamlining
it to increase support for the individual developer and his or her team.
One can approach the process issue in terms of transparency. That is, a development
process can be transparent to the developer. Transparency means that developers
face a minimal initial threshold before they can use the process. They should
not have to assimilate a lot of knowledge before they can follow the process.
It should be easy to acquire the knowledge relevant at any specific point in
time. It should be easy to put the knowledge in context and apply it to the
problem at hand.
In this sense, we find it obvious that the DA can help in achieving a transparent
* The DA puts the individual developer in focus. It selectively presents
and applies knowledge for the developer, just in time, based on the developer's
goals to adopt, activities to perform and software artifacts to be responsible
* The DA promotes alignment and supports collaboration among development
teams. It encourages the use of a development process preferred by all team
members. It serves (via agents) as a medium for developer communication.
* The DA can adapt by detecting patterns in developer behavior, and
then refine its support (rules) automatically.
Current DA initiatives
Currently, there are only a few initiatives that fall under the concept of a
DA as outlined above. Two are worth mentioning here: Argo/UML (argouml.tigris.org)
and Jaczone WayPointer.
Jaczone WayPointer (www.jaczone.com) is
an agent system primarily developed on top of products such as Rational Rose
and the Unified Process. Although WayPointer is not a full DA, it is a useful
first step. WayPointer supports developers that adopt roles and perform development
activities within use case modeling, analysis and design.
The software developer assistant (DA) can empower individual developers in
their daily work assignments. It can help them work as a team.
The DA accomplishes this empowerment by understanding the goals of the developer
in terms of roles, activities and software artifacts. It provides streamlined
support in reaching those goals through an implementation consisting of intelligent
software agents and well-oiled rule machinery.
As a result, the development process itself becomes transparent to the developer.
The old days of initial process adoption thresholds, straitjackets and bulkiness
are over. Instead, we see a new era of development processes supporting a just-when-needed
approach served in the context of the developer and the development challenge
BIO: Ivar Jacobson is a thought leader in the software world. He has written
several influential books about components and component architecture, use cases,
modern business engineering, UML and RUP.
1. Warmer, J. B., and A. G. Kleppe. The Object Constraint Language:
Precise Modeling with UML. Reading, Mass.: Addison Wesley Longman, 1999.
2. Russell, S.J., and P. Norvig. Artificial Intelligence: A Modern
Approach. Prentice Hall: Englewood Cliffs, N.J., 1995.
3. Gamma, E., et al. Design Patterns. Reading, Mass.: Addison-Wesley,
4. Murch, R., and T. Johnson. Intelligent Software Agents. Englewood
Cliffs, N.J.: Prentice Hall, 1999.