Project designation: "CM Framework" – our next generation build and release process management system.
Brief explanation of the goals of the project:
Retool the company’s configuration management framework to improve quality and project speed across all of the company’s game titles. We wanted to significantly increase our development scalability without increasing cost, and institute/enforce consistent development processes across the teams.
Application Engineering
Emphasizes the use of software engineering tools and practices. Includes
the use of Extreme Programming (XP) concepts, the Unified Modeling Language
(UML), Computer-Aided Software Engineering (CASE), object-oriented analysis
and design tools, automated testing tools, methodologies, project management,
process management, configuration management tools and/or productivity
measurement tools in an innovative corporate development project. SRAS
could also be listed as an Application Engineering project.
Business Risk
This new system had to be non-disruptive (i.e. adoption could not create delays for existing projects. It also had to be flexible enough to adopt to each team’s existing tools and practices – significant rework would be unacceptable. Ultimately, the labor required and the quality issues we were experiencing with the existing home grown build system created more risk for the status quo vs. moving to a new system.
In game development, the speed and efficiency of development is critical. Games can cost upwards of $80 million dollars to make. Game development teams can also be extremely large – 200-400 developers per project. If a build error resulted in two hours of downtime, that’s 400-800 man hours lost. Cost overruns like that can cripple a project. What’s more, release schedules are unforgiving because they are typically tied to a specific event (i.e. a movie premiere, the Christmas holiday season, etc.). A late release can cause a company to lose millions of dollars. As a result, our development processes have to be bullet proof.
f. How the system helped users
- Increased CM productivity over 90% through more efficiency and automation
- Better visibility and control for Development Managers and Team Leads through real-time project data and self-service capabilities
- Made developers more productive – gave them the ability to do pre-flight validation of changes and receive immediate feedback on their code.
II. Organizational Objectives
a. Short and long term benefits
- Improved product quality by 70% through standardized, enforceable processes
- Increased build speed by 20x – through server management and mass distribution of parallel tasks. This helps us consistently meet project schedules
- Thousands of dollars saved through better utilization of build system hardware through server pooling
- Made better use of our CM staff resources – projects are now easily portable from one team member to another, and we’ve reduced the headcount requirement on a single project by over 80%
- We are now much more flexible in our development processes because tasks aren’t hard coded into scripts. The process automation we’ve put in place allows for much greater reusability of existing processes.
- Our development process runs much smoother because it is integrated from requirements management through to testing and deployment – data is tracked across the entire lifecycle, and tasks are automated from one stage to another.
- We have much better metrics now that help us refine our development processes. Data is collected for each build run throughout the life of the project, so we can more easily identify and address bottlenecks in the cycle.
- Our new centralized system makes it much easier for geographically dispersed and offshore teams to communicate and run common processes – this has definitely increased the efficiency of our offshore teams, making them a viable alternative to supplement our domestic workforce for round the clock development.
b. Business purpose of the system
- Our CM Framework is essential for our development of high-quality games. It is directly related to the company’s source of income.
- Single, centralized console to access build and release processes for all project teams
- Customized security settings so each team only sees their projects
- Reusability of project libraries for fast new project setup and leverage of best practices
- Developer access to standardized CM policies – enables them to run production processes against their sandbox resources and get immediate feedback on build success/failure
- Connects processes across our version control (Perforce), build environments, and test systems and protocols (Winrunner, Test Director, and instrumented code testing system called “Juice”)
- Distributes complex build activity (like graphics rendering) across a pool of available servers for highly scalable, parallel processing capabilities.
- Gives development managers a high-level view of their project status in real-time. More visibility into how the project is going.
- Web-based access to projects so they can be used by distributed and offshore teams.
- Extensive tracking of development data to analyze the overall development process.
- Correlates source code changes with the build to enable rapid troubleshooting of errors.
Internal sponsors – the Configuration Management team and the CTO. Initially, the development managers and team leaders were hesitant because they didn’t want to give up control. Once we showed them the system’s reliability and value to them, they were on board. They like the fact that they can run CM processes themselves with a push of a button and get immediate feedback on the results. This is a much higher level of service than we were able to give them in the past, and they also have much higher visibility into their project than they did before.
f. Were users involved during planning and development? Not really. We knew what their main requirements were, and we were able to show them a prototype. Basically, they just wanted to know that the build function would be handled reliably.
g. Challenges: our greatest challenge was time. We had to implement this system while we were doing ongoing development. This meant the system had to incorporate our existing tools and processes – we couldn’t shut things down just to implement this new system. This was overcome by doing a phased approach to the deployment – we tackled one project at a time. Fortunately, we didn’t encounter many problems during implementation.
h. Goals – no, they were consistent the entire time
III. Category: Application Engineering
IV. Methodology / Process
a. BuildForge FullControl and FullThrottle are the underpinnings of our CM Framework solution. FullControl provides the process automation, security, and control that we needed to standardize and integrate our processes and share them across the development team. FullThrottle provides the distributed capability to do our massively parallel processes to get the build speed improvements.
b. We incorporated our existing testing tools into the system as part of our overall development lifecycle – see Section IIc.
c. Lifecycle methodology used – we use several development methods, depending on the project and the task at hand. We try not to label ourselves with a specific “doctrine” since each of them have pros and cons. We do employ waterfall in some cases, and others are more Agile and XP focused. The biggest thing for us is fast iterations on our code – we need to know quickly if things are working or not.
d. Project management method – the general plan was based on CMMI principles
e. Software quality metrics are an important bi-product of the system we were putting in place. They weren’t as important for the CM Framework itself. With our new CM Framework in place, we are now able to measure quality in more meaningful ways throughout the development cycle. We track this through the amount of code line changes occurring, the number of product errors, we incorporate test results with the builds to see if errors are reducing, and we tie features/defects to each release to see how many are being resolved in what timeframe.
Based on the information the new system has provided, we were able to make specific recommendations to increase quality. One example was having code checkin curfews – we found that late checkins were almost always the cause of build failures. We’ve also found the developer self-service / pre-flight capability to be very helpful for quality improvement because developers can get early feedback to catch simple errors.
Since implementing the system, our code has been 98% stable over the course of the entire project – i.e. the core functionality was working 98% of the time.
V. Technology
Biggest technical challenges – we did not incur major technical challenges. The implementation of our first project took 2 weeks, and we had it deployed studio wide within 12 moths (across 25 projects and almost 2000 developers). Probably the biggest challenge was just coordinating the phased deployment.
b. Software tools: BuildForge FullControl and FullThrottle are the cornerstone of our CM Framework system. It is the engine that connects all of our other products into a cohesive process management system. Using BuildForge, we’ve essentially created a service oriented architecture (SOA) for development by connecting our build and release processes with the following tools:
Perforce SCM, Winrunner, Test Director, Marimba, our SQL server based data warehouse, and DevTrack for defect tracking. We’re using MySQL and SQL server for the backend databases, running on the Linux, Windows, and Mac platforms.
Other competitors including Electric Cloud and home grown development were also considered. BuildForge was chosen because of its flexible architecture to incorporate our existing tools and processes – it required minimal rework. We were able to keep the pieces of our infrastructure that were working, and replace the other aspects with enhanced capabilities from BuildForge. After looking at the build vs. buy scenario, we determined that BuildForge was the most cost effective and robust solution. We also liked that we wouldn’t have to provide internal support for a system – this allows us to focus more on our core competency of game development.
c.
Architecture
BuildForge provides a centralized Management Console on an Apache server with a resident MySQL database and an embedded execution engine. Agents then reside on each of the production or scratch servers that do the actual work. The system is accessible through a web interface. Connections to other systems/tools/tasks are executed via command line or XML calls to the BuildForge API.
d. Important characteristics: 1) flexibility to plug in existing tools and processes, get away from our hard coded approach, 2) repeatability -- ability to create and enforce standardized processes, 3) reusability – ability to leverage work across different projects, 4) scalability – ability to pool server resources, and distribute work in parallel across server pools, 5) security – ability to tailor access for different project teams and roles.
VI. Project Team
a. Team size – the average team is about 100 developers and graphics designers. In total, we are supporting 2000 developers with our CM Framework.
b. Our team members range in experience level from entry level to sophisticated. That’s why it was important to have standardized processes that mask some of the complexity behind the scenes. With our new system, less experienced employees are able to execute tasks with a push of a button. This has also been helpful to make our remote/offshore teams more productive.
Internally, there were a few concerns over the costs of building and
maintaining all of the components beyond the core rules engine. Sun had
already experience first-hand the necessary upkeep and maintenance on
the in-house system in place.
c. The need for training is minimized with the system, as mentioned above.
d. The first project was up and running in the new system within 2 weeks. The entire deployment of 25 projects took about 9 months. This was accomplished WITHOUT any slips to the existing release schedule.
e. Yes, management considered this a great success. We estimate the new system has saved us millions in the following areas:
- Increased CM productivity over 90% through more efficiency and automation
- Better visibility and control for Development Managers and Team Leads through real-time project data and self-service capabilities
- Made developers more productive (about 25%) – gave them the ability to do pre-flight validation of changes and receive immediate feedback on their code.
- Thousands saved in reduced hardware expenditures
- Faster release cycles by accelerating builds from 60 hours down to 3 hours. This will improve our time to market and increase our competitiveness as a result.
g. If we’d had more time, I would have been able to leverage more of BuildForge FullControl’s capabilities than I did initially. There are a lot of automation features (called “dot commands”) that can really optimize things. I learned about these over time, but I could have implemented them earlier if I didn’t have to balance all of the other work.