In-Depth
The next step: Validating year 2000 compliance
Note: This article
gives examples of various products from different companies. These references
are not exhaustive and are for illustrative purposes. The choice of the
appropriate product for each company depends significantly on individual
circumstances. Since the intent was not to create an exhaustive list of
all commercially available products in each category, many other products
are not mentioned in this article. If you want an opinion on other products
that you may be considering, please contact the author for information on
additional products. |
Many large I/S organizations are attacking the year 2000 problem in segments
-- working through the inventory analysis, impact analysis and remediation phases
of projects. As the year 2000 projects pass through these phases, managers face
the critical task of verifying the fixed applications are year 2000 compliant.
This article sets out to define the essential components of year 2000 compliance and then describes how verification
can be accomplished. The objective of the author is to shed light on a variety of year 2000 concepts, including
Regression Testing, Date Simulation, Time Machine, Data Aging, Test Coverage Analysis and Test Data Generation.
The first goal of the process is to define year 2000 compliance. An application is year 2000 compliant if it functions
correctly when executed during the year 2000 and beyond with data files containing dates of year 2000 and beyond.
To verify that an application is year 2000 compliant, it must meet the following conditions:
No Regression: If executed today, the modified program performs the same way as the
unmodified program;
Year 2000 Execution: If executed under year 2000 dates, the modified program performs
correctly, producing the expected results;
Year 2000 Data Handling: If executed under year 2000 dates with data files containing
year 2000 dates the modified program performs correctly; and
Sufficiency of Test Data: Sufficient test data is available to exercise all of the
modified application code.
Organizations must ensure developers on year 2000 projects understand these conditions and what work must be done
to meet them. A more complete explanation of each condition follows.
Regression
To meet the "No Regression" condition, experts advise that, except in certain circumstances, no non-year
2000 functional changes be made when applications are modified for year 2000 compliance. Under these conditions,
modified applications executed under current dates (1998 or 1999) should produce the same results as unmodified
programs. In that case, the changed application has not regressed.
One possible exception to that rule is the use of four characters instead of two to display or store dates. In
addition, this may shift the location of other fields on a line or the offset of other fields in a file record.
For the purpose of verifying "No Regression" for year 2000 compliance, these differences should be intelligently
ignored.
Since most applications require continuous maintenance and enhancements, it is essential that, after year 2000
compliance changes are made, the application is returned to production as soon as possible. To get the modified
application into production mode, developers must ensure that it functions identically to current production code,
except for the changes made to gain year 2000
compliance. If maintenance and enhancements were made to the production code while an application is undergoing
year 2000 compliance changes, then the dual changes must be reconciled before the modified application is returned
to production. Experts warn, though, the reconciliation is a difficult, time-consuming, and error-prone process
and should be avoided. To avoid dual projects, developers should enact a code-freeze for a brief period to make
year 2000 compliance changes
and verify there is no regression of the basic functionality. Once that project is completed, the application can
be returned to production and the code freeze can be vacated.
If emergency fixes must be made to an application during a year 2000 compliance project, then the same changes
must be made to both sets of code to avoid any difference in functionality. These differences should clearly be
attributable to the emergency fixes.
Successful regression requires several components, including:
*Test Suites, which consist of Input
information (Test Data Files, and Captured User Screen Input), Output information (Resulting Data Files, Resulting
online screens, and resulting reports) and execution parameters;
*An ability to capture all screen input
and output. This requirement includes the capacity to automatically feed the input to the application when a test
needs to be re-run;
*A facility for comparing the old and
new outputs. Such a comparison facility should be able to handle output screens and output reports as well as output
data files. Each of the comparisons has slightly varying requirements. When comparing screens, developers must
ignore the position of the screens (or Windows), and any fields, such as the time of execution, that are expected
to change. Developers on the project must also ignore similar differences in reports. In case of data files, the
comparison utility must be capable of handling the multitudes of file types, and multiple record types that may
reside within the same file.
Some regression testing products shipping today include, in alphabetical order by company: Computer Associates
International Inc. (CA-Verify and CA-Verify/EEO); Compuware Corp. (QA Hiperstation); Mercury Interactive (TestSuite
2000); Seec Inc. (AccuTest Suite); and Viasoft Inc.(VIA/AutoTest). The list of available file comparison products
include: Computer Associates (CA-Accucheck); Compuware (FileAid); Viasoft Inc. (Comparex); and IBM (Super C).
Year 2000 execution
Once developers can verify that an application has not regressed, the software must be executed with the current
date set to year 2000 dates. Since waiting until January 1, 2000 to test applications is not a viable option, organizations
should create a simulated year 2000 environment. Correct execution under the simulated dates further validates
the year 2000 compliance modifications. Experts suggest the following dates are best for testing: the rollover
from December 31, 1999 to January 1, 2000; February 29, 2000 (it is a leap year); March 31, 2000 (end of the quarter);
and other significant dates.
To execute applications under year 2000 dates, developers must build a simulated environment that
represents dates in the year 2000. Such an environment can be created via one of the following three techniques.
Time Machine -- Utilizing a standalone machine that can duplicate the environment
under which the application executes. The machine is initialized with the appropriate year 2000 dates and all applications
running on that machine see the same date. Overall, the time machine is the best solution, but it does tend to
be fairly expensive due to the cost of the dedicated machine, software,
and resources required for set up.
Logical Time Machine -- This technique calls for the logical partitioning of a machine
to set it up with year 2000 dates. This is primarily applicable for large mainframe computers. All applications
running under that logical partitioning see the same date.
Date Simulation Products -- Date simulation products intercept system date requests
and provide selected applications with dates in a specified range, such as the year 2000 and beyond. For the most
part, date simulation products from various vendors provide the same basic capability. The products differ in the
user interface, and in the ability to select specific users, jobs, steps, tasks and
transactions. In addition some products cannot handle some exceptional coding practices of retrieving the system
date.
Simulation products are shipping today from several suppliers, including Computer Associates (CA-Impact/2000);
Compuware (Xpediter/Xchange); Isogon Corp. (TICTOC); MainWare Inc. (HourGlass); Prince Software Inc. (Simulate
2000); Seec Inc., Journey); and Viasoft (VIA/ValidDate).
Year 2000 data handling
Once the year 2000 Execution Environment is set up and verified, developers should next verify that the application
handles data files containing dates for the year 2000. These would include screen input, databases and transaction
files. Since a significant date related functionality in most applications manipulates dates in data files, it
is necessary that the data files used for the testing
contain year 2000 related dates. Without this, a significant part of the application's functionality may not get
verified.
One example of a lack of verification could be a credit card application designed to compute payments due from
customers, that is run with the current date set to January 1, 2000, while the dates in the data file are left
untouched. Such an application would determine that no payment was
made for a long period of time (March 1998 to January 1, 2000) and will mark every account as past due and thus
delinquent. In reality, though, on January 1, 2000, the data files would reflect payments and transactions throughout
1998 and 1999. This process needs to be simulated.
The simulation process is accomplished by a concept called Data Aging, where all the date files are
uniformly aged (for example, adding a certain number of days). Once that process is completed, applications executed
under a date of January 1, 2000 will display payments made in prior months and would compute accounts correctly.
To ensure that applications can correctly handle data files that contain year 2000 dates requires modification
of the data files. This process can generally be undertaken utilizing a concept called Data Aging, which ages all
dates in the data files by a given number of days to simulate dates around year 2000. An example would be an application
tested on April 15, 1998, using a simulation date of January 1, 2000.
The difference between these two dates, 625 days, is added to all the dates in the data to simulate the data that
would be found around January 1, 2000. In selecting Data Aging products it is necessary to verify that the product
handles the different file formats and databases that are currently in use at your site. Most bridging software
also contains the built-in capability to age the data that is
being read-in.
Companies selling products designed for data aging include: Compuware (File-AID/Data Ager); Mercury Interactive
(Ages Screen Input); Platinum Technology Inc. (TransCentury File Age); Seec (Ages Screen Input); and Serena Software
International (StarWarp).
Sufficiency of test data or test coverage analysis
With the next step, it is essential developers ensure sufficient test data is used to exercise the different pieces
of applications. For the purpose of year 2000 compliance verification, at a minimum, all statements that were modified
must be executed.
Test Coverage Analysis is used to identify the ratio of executed lines of code to total lines of code, expressed
as a percentage. For the purpose of year 2000 Compliance Validation, Test Coverage will be the number of modified
lines that are executed divided by the total number of modified lines.
If the Test Coverage of the modified statements is close to 100%, there is sufficient test data and testing is
satisfactory. On the other hand, if the Test Coverage is significantly below 100%, then it is essential that additional
Test Data is created that can specifically execute the modified statements that were not exercised by the existing
Test Data.
To ensure there is sufficient test data, test coverage analysis must be performed. If 100% of the
modified statements are executed by the test data, then there is sufficient coverage. Otherwise, additional test
data must be generated to ensure 100% coverage of all modified statements.
Test coverage analysis capabilities are incorporated in the products of several vendors, including: Computer Associates
(CA-TestCoverage/2000); Seec Inc. (AccuFix): and Viasoft (VIA/SmartTest and TCA).
Test data can be generated in one of two ways. The first approach is to take the record format of the input records
and randomly generate data values corresponding to the format of the different fields but ensuring that the data
fits the pre-specified data ranges. Although this approach creates additional records, it does not necessarily
ensure the statements that need to be executed will be executed by using this additional data. Test data generation
tools are essential for creating additional test data.
Companies selling this type of test data generation products include: Computer Associates
(CA-Datamacs/II); and Mercury Interactive (TestBytes).
The second approach to test data generation is to identify the specific input data necessary to execute specific
statement(s). This method understands the program logic and identifies the conditions under which the specific
statements will be executed. Examples of companies selling targeted test data generation products include: Seec
Inc. (AccuTest).
For year 2000 compliance validation, Correct Performance is defined as the set of business expectations from the
applications, or what the production code produces. Hence, the modified applications would have Correct Performance
if their results (resulting screens, reports, data files) were identical to what is produced by the current production
version. This definition, if followed
rigorously, will eliminate the tendency to fix pre-existing problems in production code that are unrelated to the
year 2000 problem.
Compliance testing goal
Since significant time, effort and resources are required to perform compliance testing, applications must be prioritized
and the level of compliance verification identified. Applications can require Extensive, Normal or Minimal Level
testing based on the business value, the impact on the bottom line, the magnitude of change required, and the extent
of automation in the remediation process.
The Extensive testing level calls for all testing processes -- regression testing, year 2000 date simulation, data
aging to simulate year 2000 data, test coverage analysis -- for verification of 100% coverage of all modified statements
and generation of additional test data, to reach 100% coverage.
Normal testing includes regression testing, year 2000 date simulation and data aging. The test coverage analysis
and generation of additional test case processes may be minimized.
The Minimal testing level calls for the regression testing, year 2000 date simulation and data aging activities.
The other jobs can be skipped.
This article described the requirements for undertaking a year 2000 compliance validation project and offered examples
of commercially available products that can be used in the process. Although the paper concentrated specifically
on the testing process, it is essential that organizations have a complete methodology for managing the source,
identifying units of work, tracking the changes, the use of automated tools for remediation, year 2000 compliance
validation, and return of code to production. Depending on the level of sophistication of a company's enhancement
and testing process, it may require minimal or extensive changes in the way year 2000 compliance is implemented.
In conclusion, the estimated cost for testing and validation is a significant
factor in the overall cost of year 2000 compliance. In addition, most organizations
have never made such extensive changes to almost every application in a shop
and do not have the experience of testing and verifying every application. Hence,
it is essential that the year 2000 compliance validation be approached in a
systematic way, such as is described here.