In-Depth

Software Engineering Secrets

• By Prasad Kunchakarra, Andrew Turner
• November 1, 2001

As we are all aware, there are no silver bullets in software engineering. But there are techniques, or secrets, that can improve your chances of developing a successful product. Getting the right architecture is the single most important factor for building faster, cheaper and better solutions. The following is a list of best architectural practices for architecting distributed systems.

Secret 1: Use the appropriate number of tiers. Keep the architecture simple by minimizing the number of tiers and the number of components in each tier. Before adding another tier or component, ask yourself the following question: "Will it add any benefit?" If the answer is no, the component or tier should not be introduced. For example, if you need to develop a simple Web-based application that needs to retrieve a few rows of data from databases with no transactions, a separate server that hosts business logic components could be overkill. A JSP-enabled Web server could easily support such functionality.

On the other hand, if you need to develop an enterprise online accounting solution that has to support user authentication/authorization and access to multiple databases on multiple platforms, the functionality should be distributed across multiple services. The services could include service for authentication/authorization, service for supporting database transaction logic, and a separate service for each type of presentation logic.

Distributing the functionality between services makes this solution reusable. The authentication/authorization service can be reused by several applications.

Secret 2: Take advantage of legacy systems. Do not rewrite or duplicate existing application/system functionality. If an existing legacy asset adds value to the solution, the asset can and should be leveraged. Legacy applications can contain large amounts of business logic, but may not have a great deal of documentation that describes all of the business rules precisely and accurately. Rewriting the functionality supported by legacy applications can be a time-consuming, resource-intensive and risk-prone task. Therefore, it is always advisable to use existing legacy applications.

Secret 3: Consider the expected life of the system. Understand and take into consideration the shelf life of the system. If a system is meant to be short-lived, limit the time spent on architectural analysis. If the system has a longer shelf life, an extensive architectural analysis is necessary.

Secret 4: Build services that can be reused. If the system you are building has enterprise-level usability, design the architecture to maximize reuse. If you are developing a service to perform transactions against an enterprise repository, the services can most likely be reused by multiple applications. If you are building a small application whose business value beyond one year is questionable, reuse may not be an important issue.

Secret 5: Make sure you understand the requirements. Thoroughly understand business requirements, and infer key concepts such as business situation and business functions. The technology and architecture being evaluated or selected should be appropriate to the business problem that has to be solved.

For example, a Web-based financial app that serves customers could have different constraints than a Web-based human resources application that serves employees. The high availability and scalability issues associated with the customer-oriented application will lead to different architectural choices compared to an internally facing human resources application.

Secret 6: Use design patterns. Do not try to reinvent the wheel. Follow the concepts of patterns wherever applicable. A patterns-based solution can be easily communicated and has a proven track record. From an architecture point of view, patterns can be applied to identify the number of tiers, and the components within each tier.

Secret 7: Follow the rules. Conform to the organization's enterprise architectural standard, if one is available. Developing, supporting and maintaining applications using multiple and competing architectures is considered expensive and resource-intensive. By sticking to a single architecture and/or platform, you will garner more support from the operations or infrastructure groups, as well as other developers. If a security flaw is found in a Netscape or IE browser, how many people will scramble to analyze and fix the problem compared to a problem with Joe's browser?

Design best practices
It is now time to learn the secrets of the design phase. While this phase is often skipped in favor of starting to write code, that is not how the pros develop systems.

Secret 1: Optimize business rules. The first step in design is to verify the business rules (requirements). Business rules should be consistent with realistic expectations for the number of concurrent users and the transaction response time. Based on the existing infrastructure, chosen architecture, design and system constraints, and time and resources, you should negotiate for realistic and pragmatic business rules that can actually be implemented.

Secret 2: Optimize data design. Carefully analyze the entities and relationships, and design the database schema to meet performance goals. In this phase, determine the primary and foreign key indexes. Initially, you should normalize the database to avoid duplicate data storage. After the normalization, you may need to denormalize for performance reasons.

Secret 3: Proactively tune the database and SQL during development. If the response times for conducting the functionality are critical, performance tuning becomes an important issue. It has been observed that proactive tuning of systems during design and development is exponentially cheaper than reactive tuning of production systems. Therefore, an up-front investment in time and resources for tuning performance is highly recommended.

Secret 4: Optimize application design. Based on the requirements and chosen architecture, try to apply design pattern concepts to identify and design software components. Identifying and implementing the right pattern is an important element of the entire design process. Use object-oriented design concepts to design objects, and the Unified Markup Language (UML) to represent components, objects, relationships and interactions. If you are developing an application that interacts with databases, also take secrets five through nine below into consideration.

Secret 5: Let the database (not the code) do the work. Application developers can perform some of the calculation within the database system (using database functions and advanced SQL queries) or they can retrieve data and perform calculations within the applications. In the majority of situations that need simple calculations—such as min, max, average, ordering and retrieving top 10 values—the database can perform the calculations more efficiently than the applications. Optimize database queries by extensively using SQL optimization techniques for complex queries that deal with large amounts of data. The optimization of algorithms within an application takes longer and is more error-prone than the optimization of SQL queries.

Secret 6: Optimize the design of database schema with indexes. Work with the database administrator and information architect to further optimize database schema by introducing indexes. If you are querying a table with a "where" clause against a column that is neither a primary nor a foreign key, the column would be a good candidate for creating an index. Indexes allow faster retrievals, but slower updates and inserts. Therefore, make sure the data is neither over- nor under-indexed.

Secret 7: Recalculate result sets using materialized views. Queries to a large database involving joins and aggregation such as SUM or both, are very expensive operations in terms of time and processing power. One way to improve query performance is to pre-calculate expensive join and aggregation operations on the database prior to execution time, and to then store these results in the database. Oracle supports this concept through materialized views, a generic database object that summarizes and pre-computes the results. In general, materialized views are suitable in distributed decision support and data warehousing applications.

Secret 8: Model the solution. Use an appropriate amount of modeling (at a minimum, use class diagrams and sequence diagrams). For OO systems, use UML with Rational or Together/J. Class diagrams should at least show public methods with arguments and return values. Sequences should be completed for all nominal use cases and any others that cause the flow to change significantly. How much time will this take? If the component analysis and basic business objects are already documented, it should not take more than a couple of days for a simple system or a couple of weeks for a medium-sized system. We find that doing the sequence diagrams brings some important issues to light. For example, what is the key value passed from the client to the server to retrieve a record? What if the record cannot be found in the database; will an exception be thrown? Is the client developer aware that code must be checked for this exception? These types of realizations often occur during modeling.

Secret 1: Have code inspections early in the game. Peer inspections ... Wait a minute, why are we talking about peer inspections? Why not list some of the coding secrets? Listing coding secrets is impossible, there are just too many things to write about—database connection pooling, JavaDoc, efficient coding, using exceptions, coding style, maintainability and so on. Also, the rules keep changing—the important secrets for JSP writing are different from Servlets, and EJB coding secrets are different from CORBA coding secrets. What we can do is utilize a group of developers to verify that the code includes as many secrets as possible.

Peer inspections should be performed while there is time to affect the product under review. In other words, a code inspection must be performed while it still makes sense to change the code. It is unbelievable how many times we have showed up at a code inspection to find the code is about to be moved into production. Peer code inspections need to be performed as soon as a class in finished being coded. For a junior developer, or a developer new to the underlying technologies or application domain, there may be as many as three code inspections. Inspections should be conducted by peers, other coders within the group and, hopefully, by the group's best coder. The meeting should be spent listing all agreed-upon problems with the code and briefly discussing superior techniques. Deep discussion or issues that cannot be resolved rapidly should be taken off-line. We recommend distributing the product for review a day before the meeting, as neatly printed source code with line and page numbers. Also include some information from the previous phase (i.e., architecture diagrams for design inspection or models for code inspection).

Unit testing best practices
Secret 1: Use automated tools to aid in unit testing. An automated unit-testing tool provides a rapid list of defects that might otherwise be found during a code inspection or unit testing. The types of issues found by a unit-testing tool can include unused variables, resources not closed, methods too complex, failure to check for faulty inputs, and the inefficient use of Strings. A tool such as Parasoft's jTest allows developers to add their own rules. For example, we wrote a custom rule to verify that ResultSets were properly closed; this is a common problem that might otherwise not be detected until production. By default jTest produces lots of output with long lists of definite or potential problems. However, each rule or group of rules can be turned off to produce an efficient listing of defects.

Each method should be executed to verify its nominal path, and each error path should work correctly. In some cases, there may be multiple nominal paths to try. Verify that the outputs are correct, or the database has been modified as expected. For server code, we recommend putting a main method in each class and executing methods outside of the client tier.

Stress testing best practices
Secret 1: Use automated tools for stress testing. With Web-based applications, it is imperative to verify that the application will work properly when many users are hitting it. Tools such as Segue show the turnaround time for x concurrent users or x users hitting repeatedly over a period of time. This type of testing flushes out issues with SQL efficiency, failure to free resources, and general coding efficiency. It is important to have a feel for how many users the system should support within a given response time. Also, Segue allows the tester to verify a Web-based app using different browsers and Internet connections.

Best practices for any phase
Secret 1: Deal with issues as soon as possible. As soon as a problem is found, it should be dealt with. If you realize during the architecture phase that you have never seen a JSP client talk to a CORBA server, do a simple prototype right away. If, during the design phase, you realize that nobody has ever created 1,000 test logins for stress testing before, you should think about how that will be completed. Do not wait until the "proper" development phase to prototype a solution.

Secret 2: Keep a written issue log. Keep a list with the issue, responsible party and resolution. Review the list once or twice a week to ensure that issues will be resolved in a timely fashion. When an issue comes up that requires the business sponsor or project lead to make a decision, you should provide them with the issue, choices and a recommendation. Note if any of the solutions will cause a delay and how long it will take. To ensure that issues do not become lost or put on the back burner, add the following statement to your correspondence: "If a response is not received within 24 hours, we will proceed with Option A."

Secret 3: Help each other. If an individual member of the team is stuck on a problem, get the entire development team into the loop. Sometimes, a senior mid-tier developer may know more about some obscure Oracle feature than a new DBA. It is no consolation to the GUI developer if the server component or the database was responsible for the project's failure. By helping each other we all learn more and become more valuable as developers.