Digital Insight

2006 ADT Innovator Award Application

I. Project Information

Company and division name: Digital Insight

About Digital Insight

Digital Insight is the leading online banking provider for financial institutions. Through its comprehensive portfolio of Internet-based financial products and services built upon the company's unique architecture, Digital Insight enables banks and credit unions to become the trusted transaction hub for their retail and commercial customers. Digital Insight offers consumer and business Internet banking, online lending, electronic bill payment and presentment, check imaging, account-to-account transfers, Web site development and hosting and marketing programs designed to help increase online banking end user growth and more. Each Digital Insight product and service reinforces the strength of its financial institution clients.
Project designation: "Corporate Banking - Build Automation"

Brief explanation of the goals of the project:

The purpose of this project is to automate builds for the Engineering ITE and QA groups to provide more efficient, higher quality build processes. We also wanted to enable engineering to have more control to execute builds on their own timeframe and in their own environment.

Business risks

This new system had to be non-disruptive (i.e. adoption could not create delays for existing projects. It also had to be flexible enough to adopt to existing tools and practices – significant rework would be unacceptable. This system would replace the company's internal build capabilities, so it needed to provide far greater functionality with minimal implementation time. We knew that if we didn't implement a more automated solution, the efficiency and scalability of our Software Configuration Management (SCM) team would be severely limited.

Digital Insight's Corporate Banking provides a complete range of cash management, information reporting, digital imaging and transaction initiation capabilities that enable financial institutions to expand into new markets and serve, retain and grow their commercial client base.
Since we are providing services to the financial market, product quality, security, and continuous uptime are all critical. We needed to establish a build automation system that could help us ensure the highest quality and fastest turnaround within our product delivery processes.
Prior to implementing this new system, we identified some key gaps in our processes:
  • Errors in build process can and do go undetected long after they first appear – sometimes as late as QA certification. To thoroughly detect all possible errors, we would have to complete a detailed analysis of several text files produced during the build. Whatever analysis we do now is in response to the discovery of a problem. We estimate that this post-problem analysis is costing the SCM team two person-days per month.
  • Gathering metrics on our processes or researching a question is difficult at best. This is because the get and build results are stored in date-stamped text files. Certain information, for example the specifics of the package staging, is not saved, so we have areas where we cannot gather metrics or do any analysis at all.
  • People have no way to check on the status of a build without resorting to email or a phone call. We have no way of notifying these same people except in kind.
  • People have no way to get a midday build, package or install executed without us working that request into our work queue at that time. This often introduces unanticipated delays between request and execution.
How the system helped users – our new system has helped in some key areas
  • It has enabled the SCM team to do more with the same staff – we are 50% more productive
  • Developers are more productive because they have control and visibility into the build process
  • We're able to get releases out faster with our accelerated build speeds through pooling and threading. In addition, standardized, consistent processes help us avoid or quickly identify errors that could cause project slips.
  • We now have much better insight into our development processes, and we anticipate making further development improvements upon analysis of this data.

II. Organizational Objectives

Short and long term benefits
Higher quality: Better error detection

Our new system has built-in error detection facilities, using both return codes and log file searches. We will be able to script the error detection we are currently doing manually.

Development visibility: Ability to gather and report metrics

All of the task information that is gathered by the system is stored in a database, so queries can be run later to quantify the information as needed.

Real-time release status information: Easy access to task status

The system has a stats Web page where anyone can look to see the status of currently executing builds, as well as the status of the last build of any project. Built-in email notification at each step will allow us to custom tailor to whom and when notifications are sent. This will allow for more timely information sharing

Empowerment of development through "Self service" build and deploys

Development managers are able to both fire off their own builds, and also schedule builds to occur at specific times.

Better lifecycle integration & increased team productivity
We can extend the reach and increase the sophistication of our automated processes. Instead of limiting such processes to building and preparing deployments packages, we'll have the capability to deploy and install.
Our group (SCM) was able to take over the installs in the QA environments. We previously support about fifteen different MIBS environments. After taking over the QA environments, we were able to support almost one hundred, and average twenty more installations a week (four more per day).
Expanded capacity/performance
  • Threading – Our new system enables us to run independent tasks in parallel, and then resume a single project thread to continue on. Our tasks are all serial now because we're using scripts, not because there's any dependency between the getting/building/deploying among our servers. Our experiments indicate significant time savings (20-30%) if each server's tasks proceed in parallel.
  • Distributing tasks – The system can run different pieces of the process on different machines, better utilizing their capabilities & capacities
  • Server pooling – The system can do some basic load balancing, so different instances of the same task can be directed to different servers in a pool, based on current load.
Business purpose of the system
  • Provide a reliable, high performance system to support our development team in their delivery of Business Solutions East online banking solutions.
Features/Functions of the new system
  • Single, centralized console to access build and release processes for all project teams
  • Customized security settings so each team only sees their projects
  • Reusability of project libraries for fast new project setup and leverage of best practices
  • Developer access to standardized CM policies – enables them to run production processes against their sandbox resources and get immediate feedback on build success/failure
  • Connects processes across our version control (CA Harvest), build environments, and test systems and protocols (Mercury)
  • Distributes complex build activity across a pool of available servers for highly scalable, parallel processing capabilities.
  • Gives development managers a high-level view of their project status in real-time. More visibility into how the project is going.
  • Web-based access to projects so they can be used by distributed teams
  • Extensive tracking of development data to analyze the overall development process.
Internal sponsors – the Software Configuration Management team and the CIO. Initially, some members were hesitant because they had more of a “build it in house” mentality. After scoping what it would truly take to build such a system, the choice to buy a robust commercial alternative was obvious.
  • Were users involved during planning and development? We knew what their main requirements were, and we were able to show them a prototype. Basically, they just wanted to know that the build function would be handled reliably.
  • Challenges: our greatest challenge was time. We had to implement this system while we were doing ongoing development. This meant the system had to incorporate our existing tools and processes – we couldn't shut things down just to implement this new system. This was overcome by doing a phased approach to the deployment – we tackled one project at a time. Fortunately, we didn't encounter many problems during implementation.
  • Goals – no, they were consistent the entire time

III. Category: Application Engineering

IV. Methodology / Process

  • BuildForge FullControl and FullThrottle are the underpinnings of our SCM Framework solution. FullControl provides the process automation, security, and control that we needed to standardize and integrate our processes and share them across the development team. FullThrottle provides the distributed capability to do our massively parallel processes to get the build speed improvements.
  • Lifecycle methodology used – we have incorporated several different development methods into our Software Development Lifecycle Process, depending on the task at hand. Much of our development is waterfall, but we also employ some of the principles of RUP.
  • Project management method – We brought in an external consultant to manage the process.
  • Software quality metrics are a critical component of the system we were putting in place. Our new system gives us the ability to measure quality in more meaningful ways throughout the development cycle. We're able to see if quality is improving or declining throughout the release cycle, and we have much more data available to analyze our processes.

V. Technology

  • Biggest technical challenges – we have not incurred major technical challenges. The deployment at our first location (Atlanta) took less than 60 days. We plan to roll the solution out to other locations enterprise wide – California is the second planned implementation.
  • Software tools: BuildForge FullControl and FullThrottle are the cornerstone of our Business Solutions East Build Automation system. It is the engine that connects all of our other products into a cohesive process management system. Using BuildForge, we can connect our existing tools into a service oriented architecture (SOA) for development using our existing tools. We've deployed the solution for Windows with a SQL Server back end database.
The main competition was a build vs. buy scenario. BuildForge was chosen because of its flexible architecture to incorporate our existing tools and processes – it required minimal rework. We were able to keep the pieces of our infrastructure that were working, and replace the other aspects with enhanced capabilities from BuildForge. After looking at the build vs. buy scenario, we determined that BuildForge was the most cost effective and robust solution. We also liked that we wouldn't have to provide internal support for a system – this allows us to focus more on our core competency of business solutions development.

VI. Architecture

  • BuildForge provides a centralized Management Console on an Apache server with a resident MySQL database and an embedded execution engine. Agents then reside on each of the production or scratch servers that do the actual work. The system is accessible through a web interface. Connections to other systems/tools/tasks are executed via command line or calls to the BuildForge API.
  • Important characteristics: 1) flexibility to plug in existing tools and processes, get away from our hard coded approach, 2) repeatability -- ability to create and enforce standardized processes, 3) reusability – ability to leverage work across different projects, 4) scalability – ability to pool server resources, and distribute work in parallel across server pools, 5) security – ability to tailor access for different project teams and roles.

VII. Project Team

  • Team size – the Atlanta team is 60 developers
  • Our team members range in experience level from entry level to sophisticated. That's why it was important to have standardized processes that mask some of the complexity behind the scenes. With our new system, less experienced employees are able to execute tasks with a push of a button.
  • The need for training is minimized with the system, as mentioned above.
  • The Atlanta deployment took less than two months. This was accomplished WITHOUT any slips to the existing release schedule.
  • Yes, management considered this a great success. We estimate the new system has saved us significantly in the following areas:
    • Increased CM productivity through more efficiency and automation
    • Better visibility and control for Development Managers and Team Leads through real-time project data and self-service capabilities
    • Made developers more productive – gave them the ability to do pre-flight validation of changes and receive immediate feedback on their code.
    • Faster release cycles by accelerating builds through parallel processes
    • We wouldn't do anything differently – it has been a very successful deployment.
At its core, BuildForge acts as the glue between our existing scripts and processes – initiating, validating results, and reporting status. This provided a natural transition to a more reliable, repeatable, and auditable process. Once we implemented basic functionality under BuildForge, it was easy matter to extend the capabilities by adding more scripts.
BuildForge gave us the ability to grant permission to specific people to execute tasks. For instance, we are now able to turn over the building/deploying/installing task for an integration system to a Development Manager, allowing him to execute this task when needed, without having to work his request into the SCM work queue.
The better error checking and better notification mechanisms (both pulling and pushing) allow us to catch errors earlier and notify the right people without manual intervention or analysis required.
Because information about all the BuildForge activities is captured in a database, we are much more able to measure these activities and generate reports.
The advanced BuildForge features of threading, task distributing and server pooling give us the ability to un-serialize tasks and them run in parallel, to run and coordinate tasks on different machines, and to distribute task instances among a designated group of machines. These capabilities have accelerated our release speed and the ability of our group to scale without adding additional staff resources.