2006 ADT Innovator Award Application
I. Project Information
Company and division name: Digital Insight
About Digital Insight
Digital Insight is the leading online banking provider for financial
institutions. Through its comprehensive portfolio of Internet-based financial
products and services built upon the company's unique architecture, Digital
Insight enables banks and credit unions to become the trusted transaction
hub for their retail and commercial customers. Digital Insight offers consumer
and business Internet banking, online lending, electronic bill payment
and presentment, check imaging, account-to-account transfers, Web site
development and hosting and marketing programs designed to help increase
online banking end user growth and more. Each Digital Insight product and
service reinforces the strength of its financial institution clients.
Project designation: "Corporate Banking - Build
Brief explanation of the goals of the project:
The purpose of this project is to automate builds for the Engineering
ITE and QA groups to provide more efficient, higher quality build processes.
We also wanted to enable engineering to have more control to execute
builds on their own timeframe and in their own environment.
This new system had to be non-disruptive (i.e. adoption could not
create delays for existing projects. It also had to be flexible enough
to adopt to existing tools and practices – significant rework would
be unacceptable. This system would replace the company's internal build
capabilities, so it needed to provide far greater functionality with
minimal implementation time. We knew that if we didn't implement a more
automated solution, the efficiency and scalability of our Software Configuration
Management (SCM) team would be severely limited.
Digital Insight's Corporate Banking provides a complete range of
cash management, information reporting, digital imaging and transaction
initiation capabilities that enable financial institutions to expand into
new markets and serve, retain and grow their commercial client base.
Since we are providing services to the financial market, product quality,
security, and continuous uptime are all critical. We needed to establish
a build automation system that could help us ensure the highest quality
and fastest turnaround within our product delivery processes.
Prior to implementing this new system, we identified some
key gaps in our processes:
- Errors in build process can and do go undetected long after they first
appear – sometimes as late as QA certification. To thoroughly detect
all possible errors, we would have to complete a detailed analysis of
several text files produced during the build. Whatever analysis we do
now is in response to the discovery of a problem. We estimate that this
post-problem analysis is costing the SCM team two person-days per month.
- Gathering metrics on our processes or researching a question is difficult
at best. This is because the get and build results are stored in date-stamped
text files. Certain information, for example the specifics of the package
staging, is not saved, so we have areas where we cannot gather metrics
or do any analysis at all.
- People have no way to check on the status of a build without resorting
to email or a phone call. We have no way of notifying these same people
except in kind.
- People have no way to get a midday build, package or install executed
without us working that request into our work queue at that time. This
often introduces unanticipated delays between request and execution.
How the system helped users – our new system has helped in
some key areas
- It has enabled the SCM team to do more with the same staff – we are
50% more productive
- Developers are more productive because they have control and visibility
into the build process
- We're able to get releases out faster with our accelerated build speeds
through pooling and threading. In addition, standardized, consistent
processes help us avoid or quickly identify errors that could cause project
- We now have much better insight into our development processes, and
we anticipate making further development improvements upon analysis of
II. Organizational Objectives
Short and long term benefits
Better error detection
Our new system has built-in error detection facilities, using both
return codes and log file searches. We will be able to script the error
detection we are currently doing manually.
Ability to gather and report
All of the task information that is gathered by the system is stored
in a database, so queries can be run later to quantify the information
Real-time release status information:
to task status
The system has a stats Web page where anyone can look to see the status
of currently executing builds, as well as the status of the last build
of any project. Built-in email notification at each step will allow us
to custom tailor to whom and when notifications are sent. This will allow
for more timely information sharing
Empowerment of development through "Self service" build
Development managers are able to both fire off their own builds, and
also schedule builds to occur at specific times.
Better lifecycle integration & increased team productivity
We can extend the reach and increase the sophistication of our automated
processes. Instead of limiting such processes to building and preparing
deployments packages, we'll have the capability to deploy and install.
Our group (SCM) was able to take over the installs in the QA environments.
We previously support about fifteen different MIBS environments. After
taking over the QA environments, we were able to support almost one hundred,
and average twenty more installations a week (four more per day).
- Threading – Our new system enables us to run independent tasks in
parallel, and then resume a single project thread to continue on. Our
tasks are all serial now because we're using scripts, not because there's
any dependency between the getting/building/deploying among our servers.
Our experiments indicate significant time savings (20-30%) if each server's
tasks proceed in parallel.
- Distributing tasks – The system can run different pieces of the process
on different machines, better utilizing their capabilities & capacities
- Server pooling – The system can do some basic load balancing, so different
instances of the same task can be directed to different servers in a
pool, based on current load.
Business purpose of the system
- Provide a reliable, high performance system to support our development
team in their delivery of Business Solutions East online banking solutions.
Features/Functions of the new system
- Single, centralized console to access build and release processes
for all project teams
- Customized security settings so each team only sees their projects
- Reusability of project libraries for fast new project setup and leverage
of best practices
- Developer access to standardized CM policies – enables them to run
production processes against their sandbox resources and get immediate
feedback on build success/failure
- Connects processes across our version control (CA Harvest), build
environments, and test systems and protocols (Mercury)
- Distributes complex build activity across a pool of available servers
for highly scalable, parallel processing capabilities.
- Gives development managers a high-level view of their project status
in real-time. More visibility into how the project is going.
- Web-based access to projects so they can be used by distributed teams
- Extensive tracking of development data to analyze the overall development
– the Software Configuration Management
team and the CIO. Initially, some members were hesitant because they had
more of a “build it in house” mentality. After scoping what it would truly
take to build such a system, the choice to buy a robust commercial alternative
- Were users involved during planning and development? We knew what
their main requirements were, and we were able to show them a prototype.
Basically, they just wanted to know that the build function would be
- Challenges: our greatest challenge was time. We had to implement this
system while we were doing ongoing development. This meant the system
had to incorporate our existing tools and processes – we couldn't shut
things down just to implement this new system. This was overcome by doing
a phased approach to the deployment – we tackled one project at a time.
Fortunately, we didn't encounter many problems during implementation.
- Goals – no, they were consistent the entire time
III. Category: Application Engineering
IV. Methodology / Process
- BuildForge FullControl and FullThrottle are the underpinnings of our
SCM Framework solution. FullControl provides the process automation,
security, and control that we needed to standardize and integrate our
processes and share them across the development team. FullThrottle provides
the distributed capability to do our massively parallel processes to
get the build speed improvements.
- Lifecycle methodology used – we have incorporated several different
development methods into our Software Development Lifecycle Process,
depending on the task at hand. Much of our development is waterfall,
but we also employ some of the principles of RUP.
- Project management method – We brought in an external consultant to
manage the process.
- Software quality metrics are a critical component of the system we
were putting in place. Our new system gives us the ability to measure
quality in more meaningful ways throughout the development cycle. We're
able to see if quality is improving or declining throughout the release
cycle, and we have much more data available to analyze our processes.
- Biggest technical challenges – we have not incurred major technical
challenges. The deployment at our first location (Atlanta) took less
than 60 days. We plan to roll the solution out to other locations enterprise
wide – California is the second planned implementation.
- Software tools: BuildForge FullControl and FullThrottle are the cornerstone
of our Business Solutions East Build Automation system. It is the engine
that connects all of our other products into a cohesive process management
system. Using BuildForge, we can connect our existing tools into a service
oriented architecture (SOA) for development using our existing tools.
We've deployed the solution for Windows with a SQL Server back end database.
The main competition was a build vs. buy scenario. BuildForge was chosen
because of its flexible architecture to incorporate our existing tools
and processes – it required minimal rework. We were able to keep the pieces
of our infrastructure that were working, and replace the other aspects
with enhanced capabilities from BuildForge. After looking at the build
vs. buy scenario, we determined that BuildForge was the most cost effective
and robust solution. We also liked that we wouldn't have to provide internal
support for a system – this allows us to focus more on our core competency
of business solutions development.
- BuildForge provides a centralized Management Console on an Apache
server with a resident MySQL database and an embedded execution engine.
Agents then reside on each of the production or scratch servers that
do the actual work. The system is accessible through a web interface.
Connections to other systems/tools/tasks are executed via command line
or calls to the BuildForge API.
- Important characteristics: 1) flexibility to plug in existing tools
and processes, get away from our hard coded approach, 2) repeatability
-- ability to create and enforce standardized processes, 3) reusability
– ability to leverage work across different projects, 4) scalability
– ability to pool server resources, and distribute work in parallel across
server pools, 5) security – ability to tailor access for different project
teams and roles.
VII. Project Team
- Team size – the Atlanta team is 60 developers
- Our team members range in experience level from entry level to sophisticated.
That's why it was important to have standardized processes that mask
some of the complexity behind the scenes. With our new system, less experienced
employees are able to execute tasks with a push of a button.
- The need for training is minimized with the system, as mentioned above.
- The Atlanta deployment took less than two months. This was accomplished
WITHOUT any slips to the existing release schedule.
- Yes, management considered this a great success. We estimate
the new system has saved us significantly in the following areas:
- Increased CM productivity through more efficiency and automation
- Better visibility and control for Development Managers and Team Leads
through real-time project data and self-service capabilities
- Made developers more productive – gave them the ability to do pre-flight
validation of changes and receive immediate feedback on their code.
- Faster release cycles by accelerating builds through parallel processes
- We wouldn't do anything differently – it has been a very successful
At its core, BuildForge acts as the glue between our existing scripts
and processes – initiating, validating results, and reporting status. This
provided a natural transition to a more reliable, repeatable, and auditable
process. Once we implemented basic functionality under BuildForge, it was
easy matter to extend the capabilities by adding more scripts.
BuildForge gave us the ability to grant permission to specific people
to execute tasks. For instance, we are now able to turn over the building/deploying/installing
task for an integration system to a Development Manager, allowing him to
execute this task when needed, without having to work his request into
the SCM work queue.
The better error checking and better notification mechanisms (both
pulling and pushing) allow us to catch errors earlier and notify the right
people without manual intervention or analysis required.
Because information about all the BuildForge activities is captured
in a database, we are much more able to measure these activities and generate
The advanced BuildForge features of threading, task distributing and
server pooling give us the ability to un-serialize tasks and them run in
parallel, to run and coordinate tasks on different machines, and to distribute
task instances among a designated group of machines. These capabilities
have accelerated our release speed and the ability of our group to scale
without adding additional staff resources.