a. Large, lengthy, expensive government software development projects
have a history of delivering nothing but catastrophic failures that make
industry headlines. In particular, recent years have seen state run child
welfare systems end with millions spent on software that doesn’t work.
(1997 SACSS Project in California, 2004 EDS contract with UK’s Child Support
Agency) However, the State of Ohio’s Statewide Automated Child Welfare
Information System (SACWIS) is different. With a federal mandate to consolidate
88 separate county-based systems into one, the SACWIS project is using
modern development techniques and Java to ensure a different fate. By partitioning
the large project into 5 loosely coupled subsystems that utilize a Service
Oriented Architecture, the SACWIS team is managing complexity. The project
seeks to combine the agile nature of small team development with the overall
architectural control provided by a model driven approach. Through fast
prototyping of functionality and short two week development iterations,
business users are able to verify system functionality incrementally.
b. The system was designed to replace paper based and automated systems
in 88 counties throughout the state of Ohio. The new system is designed
to provide child welfare case workers with secure, readily available access
to 120,000 active case files. The business risks revolve around delivering
a large, user-acceptable system that has been implemented in many different
ways in the past.
a. The new system will benefit case workers by allowing them to track
children on a statewide basis, no matter what county or counties provide
care. In the past, systems in different counties did not necessarily share
information, leading to lost cases and under-served children. Short term
benefits include the ability for case workers to have a consolidated view
of a child’s case over time and geography. The system will not be deployed
until April, 2006, so long term benefits have not been measured. This is,
indeed, a mission critical system that contributes to the fate of tens
of thousands of children in the state of Ohio.
b. The business purpose of the system is to keep track of children
who receive care from the state. This includes children in foster care,
adopted children, and abused children.
c. The system features are quite numerous, implementing 165 course
grained use cases which resulted in over 700 web pages and 250 online reports.
At a high level, features spanned 5 functional areas: Intake, Case, Provider,
Financial, and Administration. These features all revolve around managing
state assistance for children; from entering a child into the system to
tracking how much state money has been spent on them, to which providers
manage the relationship. Other features include a broadcast messaging system
used to communicate with providers and a training module used to track
which providers have received what training.
d. The system implements more than 6,000 functions. The functions are
assembled in various ways to accommodate various use cases in an object
oriented fashion. The application was architected so that function calls
that span multiple functional areas utilize a Service Oriented Architecture
(SOA).
e. The sponsor for this project at the State of Ohio is the Ohio Department
of Jobs & Family Services (ODJFS), the largest state agency in Ohio with
more than 4,000 employees. Their purview includes healthcare, employment
and economic assistance, and child and family services. Opposition to this
project came mostly from those counties within the state that felt they
already had a robust, working system in place for their counties. A federal
mandate to consolidate all county-based child welfare systems kept these
objections in check.
f. State and county subject matter experts (SMEs) were utilized both
during the up-front Rapid Requirements Definition (RRD) stage and in Joint
Analysis & Design (JAD) session which were a part of each two-week iteration.
Especially useful was the combination of SMEs and modelers during the initial
requirements gathering phase. High level business models served as inputs
to each development iteration, so including SMEs in their definition was
crucial.
g. The greatest challenges in completing this project involved size
and complexity combined with an aggressive time schedule and diminutive
budget. Child welfare systems are notoriously complex and have a history
of failure while exceeding their original budgets. Compuware provided a
fixed bid quote for the project, to be delivered in 18 months. The real
innovation here is how Compuware was able to combine rigorous processes
with the tenets of Agile development to drive a team of 45 developers to
success in this goal. By instituting the Rational Unified Process (RUP)
which combined automated Model Driven Development (MDD) with short, two-week
iterations, the project team was able to guide five small functional development
teams toward a single goal. Each two-week iteration yielded working software
that could be verified by business stakeholders.
h. The overall goals for the project did not change, however some architecturally
significant requirements were not defined until well into the development
effort. One such requirement was the security scheme and mechanisms for
the system. These changes were accommodated with minimal adjustments to
the delivery dates. This was possible because of the model-driven approach
and generative approach to development. Vast swaths of functionality could
be changed by updating architectural models and re-generating code. Compuware’s
OptimalJ was critical to the success of this effort and allowed for the
introduction of major changes late in the development cycle.
II. Category: Application Engineering
a. Tools and techniques were combined to great effect on this project.
Compuware’s OptimalJ was used to transform business models into working
code during short two-week iterations. Use of OptimalJ subverted what might
otherwise have been lengthy architecture discussions. Because the architecture
could be codified within the tool, architects had supreme control over
architectural implementation. Also, because many of the transformations
that convert business models into technical architecture come with the
tool, arguments about how to implement certain functionality simply did
not happen. It was easier to go with the out-of-the-box implementations.
Automating the implementation of the fundamental architecture enabled the
project management techniques. The technique called for a working build
of the application at the end of each two-week iteration. This enabled
business stakeholders to evaluate the use cases that were implemented during
the preceding two weeks.
b. Automated functional testing tools were not used to great effect
on this project and their lack of use is noted as a lesson learned from
this project. However load testing tools were used to great effect in order
to test state requirements that called for certain response times. JUnit
and Cactus were also used to great effect by developers for unit testing.
From a project management standpoint, testing teams followed their own
two-week iteration cycle that tested use-cases that were implemented during
the previous two-week cycle. A test-early, test often philosophy was advocated
throughout the project.
c. A 30 month formal software development lifecycle was defined for
this project which began in May, 2004.
d. The project plan included time and staffing allotments for project
planning, rapid requirements definition, 18 two-week development iterations,
system testing, user acceptance testing, pilot and statewide rollout. The
most interesting innovation from a project management standpoint was the
five concurrent teams executing discreet two-week development iteration.
Segmenting the application into five functional areas and creating small
five to eight person teams to tackle two to three use cases per iteration
was the key. Each small team was able to employ agile development techniques
while management retained control over system architecture. Because each
iteration produced working software that could be validated by the business
stakeholders, management always understood how much progress was being
made.
e. Software quality metrics were gathered and reported throughout the
project. Most metrics graphed statistics over number-of-defects figures.
For example, number-of-defects per iteration, number-of-defects per use-case,
number-of-defects per functional development team. Defects were also categorized
into critical, high, medium, and low to allow for an additional plane of
analysis.
a. The major technical challenge encountered on the project involved
data migration from disparate, heterogeneous, legacy data systems. An iterative
approach to data migration was employed. The following is an oversimplification
of the steps, but provides an overview. Notice that the approach transforms
the heterogeneous data environments of the legacy systems into a common/canonical
form in the first stage. This minimizes the amount of code and data manipulations.
It also limits the number of environmental transformations; keeps them
as early in the process as possible. This early decoupling reduces the
impact to the legacy environments while maximizing the availability and
stability of the source data definitions and content needed to support
the remainder of the data Migration/conversion approach. It also establishes
a well defined baseline for the source data definitions that is vital for
requirements and configuration management. The data migration/conversion
approach is basically a 5-stage process – from a technical point of view.
- Transform the legacy source data into a common (or canonical) representation
within the conversion environment. The source data extraction is performed
on the source platform using tools that are commonly in use. The extract
format is provided in a form that is readily imported to the canonical
data environment.
- Map the canonical data environment used to stage the source data
into the SACWIS Target data definition. This mapping provides the specifications
for transforming the source data into the SACWIS application database.
Using analytical tools on the Conversion Environment, data and business
analysts can identify data integrity and completeness issues that need
to be addressed (within the source data, procedurally during conversion,
or as post conversion clean up activities).
- Construct and Test the software and processes for effecting the
transformation from the canonical source data environment into the SACWIS
Target data structures. The target area is a ‘replica’ of the production
environment. The replica environment and data are also used to identify
and define resolutions to data quality issues.
- Exercise and validate the transformation. Using the SACWIS application
and accessing the converted ‘replica’ of the data environment, the business
and technical users can evaluate the validity and completeness of the
process.
- Convert the Live Data. The proven Data Migration/Conversion processes
are utilized to repopulate the ‘replica’ of the production environment
with the most current data. The business and technical users then confirm
the conversion operation. Once confirmed, utilities are used to move
the data into the production environment – ready for operation.
Development Area |
Toolset |
Requirements Analysis/management and
Business Modeling |
Rational Rose, MS-Word |
Creation of Design Artifacts. Technical
and Business Use Case Realizations. |
Rational Rose, MS-Word, SoDA for Word |
GUI and Wireframe creation |
Dreamweaver |
Software Version Control |
CVS 1.11.14 |
Application Development (Web, EJB tiers) |
JDK 1.4.2_04 |
Application code development |
JBuilder 10 |
Development Application Server |
JBoss 3.2.3, Websphere Application Server
5.1 |
Production Application Server |
Websphere Application Server 5.1 |
Development Database |
Oracle 9.2 |
Production Database |
Oracle 9.2 |
Database Design |
ERWin, OptimalJ 3.2 |
PIM Modeling, MDA Based Application Code
Generation |
OptimalJ 3.3.02 SP2 |
Message Oriented Middleware |
IBM MQSeries |
Unit Testing |
JUnit, Struts Unit Testing, Mercury Interactive
Testing Suite. |
Report Generation |
COGNOS 1.1 |
Diagrams |
MS-Visio, MS- Powerpoint |
Images |
PhotoShop |
Load and Stress Testing |
Mercury Load Runner 8.0 |
Spell Checker |
JSpell2504 |
Defect Tracking |
BugRat 2.5.3 |
Third-party tools and add-ins: The System consists
primarily of custom code and open source frameworks. These frameworks include
Struts, Log4J, and Jakarta Commons. The only proprietary components used
are JSpell, the spell checking component, and Cognos, which is the reporting
engine used in Ohio.
Database platform and version: The database is Oracle
9i/10g running on a SUN Solaris platform. The database is configured in
an Oracle RAC (Real Application Cluster) arrangement.
Configuration Management tools and processes: The
SACWIS system employs a robust set of automated Configuration Management
process based on custom “ant” scripts that work in conjunction with the
CVS Software Version Control tool and various Java environments. These
environments include JBoss and IBM WebSphere running on both Unix and Windows
platforms. The processes and procedures support a multi-team development
environment with a fully automated build feature.
The system is architected and built around an ultra-thin client using
the Model View Controller (MVC) implementation pattern. As a result Mobile
devices are readily supported by the system.
There are no maintenance tasks that require the system to be shutdown
or otherwise made unavailable. Due to the clustered environment, the system
can continue to operate at a reduced capacity level, even in the event
of a hardware maintenance event.
Various levels of disaster recovery are supported by the application
including offsite storage of data, remote “warm site”, separate facility
that houses the test and production environments, allowing for the application
to be hosted on the Test Hardware in case of emergencies.
Remote access for various database administration, application server
and operating system support is configured. There are also various administration
tasks that can be done directly through the application over a secure internet
connection.
The system is fully 24x7 capable. An IBM Clustered Application Server
and Web Server farm is configured to communicate to a cluster database
server farm (using Oracle Real Application Clusters).
One of the most challenging aspects of delivering this system was the
budget and timeframe. It was the unique combination of tools and process
that enabled the technology to be delivered. Creating such a large, complex
system in such a short period of time necessitated highly coordinated teams
that could work within a Model Driven Development (MDD) approach. Compuware’s
OptimalJ automated a large portion of the plumbing code needed for the
application, allowing the teams to focus their energy on the presentation
tier and business logic. This was very important in delivering the business
purposes of the system.
a. This development project coordinated the effort of 45 developers.
However small functional teams were formed consisting of five to eight
developers.
b. Each small functional team was tasked with two to three use cases
per two-week iteration. Because code was generated from models, each team
had a head start implementing each use-case. For example, all objects which
persisted data to the database were instantiated in code. Additionally,
all middle-ware glue which conformed to the architecture for the system
was created. This left the task of writing business logic to the developers
on the team. Also necessary was the connecting of html user-interface wire-frames
to the middleware tier. Team members could write their code in unprotected
areas, thus not disturbing generated code, and allowing their work to survive
subsequent code generation.
c. Each team consisted of a team lead, business analyst, EJB/Modeling
expert, one to two Java/JSP developers, and one to two state employee developers
(who will ultimately maintain the application). Tool training was necessary
for all development staff and took place during the rapid requirements
definition phase of the project. One of the lessons learned was that more
than just product training is necessary. When working in an environment
where code is generated, architecture training is necessary in order to
understand what code gets generated and how to use that code. In future
projects, this will be done up front. On this project, it took place as
the project progressed.
d. An exhaustive list of team members will be provided upon request.
e. Person months were not calculated for this project, however, anywhere
from 10 to 60 individuals were involved concurrently over the course of
the 30 month project. Because of the iterative nature of the development
project, and close tracking of implemented use-cases, management was able
to gauge schedule adherence every two weeks. The project was never more
than 10% behind schedule and was ultimately only 2 weeks behind going into
user acceptance testing.
f. The user community has been verifying the functionality that is
being implemented on an ongoing basis. However the application has not
yet been deployed. The pilot period is set to begin in early 2006, with
a staggered county by county rollout which will complete by October, 2006.
g. All of the lessons learned have to been fully compiled. However
many of the changes to the process and tool usage took place during development.
A lessons learned activity was completed at the end of each two week iteration
and many small tweaks were made.