Chandra Interactive Analysis of Observations (CIAO) consists of a
suite of tools and applications developed for the analysis of
Chandra X-ray Observatory data. With its unprecedented technical
capabilities and science potential, Chandra requires a novel,
flexible, multi-dimensional set of tools for data analysis. This
mission provides 4-dimensional data (2 spatial-dimensions, time,
& energy), each containing many independent elements. CIAO
tools are able to filter and project the 4-dimensional event
data to manageable sizes and convenient arrays in an easy and
flexible manner. CIAO is a mission independent system and can
be used for data analysis of other X-ray and non- X-ray
observations.
Since launch there have been several major CIAO releases. A major step
in the preparation of each CIAO release is scientific and
functional testing of the software. This is a joint effort by
the Chandra X-ray Center Data Systems (DS) and Science Data
Systems (SDS) teams. We highlight here the stages of CIAO
testing and the procedures we follow at each stage.
CIAO Tool Development
The CIAO tool development process begins
many months before a planned release. SDS scientists identify
and evaluate science and calibration milestones for Chandra data
analysis, and outline the priorities and goals for tool and
application development for a CIAO release.
During the software development process scientists and developers
interact closely to assure CIAO tools are designed and built to
the specified requirements and that they follow CIAO system
standards.
Once implemented, unit tests are developed and run
to assure proper implementation. We recognize that more rigid
tests are required though, and science and development teams
combine efforts to ensure this goal is met.
CIAO Release Testing
CIAO releases involve 4 testing stages.
These are: (1) unit testing, where detailed tests of CIAO tools
verify and validate the correct implementation of scientific
algorithms; (2) mini-testing, where the new functionality of
the system is bundled and tested across ported platforms; (3)
regression testing, where baseline system tests are run on all
ported platforms to validate the system; and (4) package
testing, where tar files are downloaded from the public web page
to validate web links, instructions, and the software
installation.
Science Unit Testing
Unit science tests are developed for each
new or updated tool as they are completed by the software
team. It is in this testing phase that scientists run tools they
have specified for the first time. Unit tool testers verify and
validate that the tool was implemented as they envisioned with
correct and robust algorithms to meet the science goals. They
also run the tools in science data analysis scenarios where the
output file of one tool is provided as the input to the next in
a series of tool steps organized to meet specific science goals.
It is these science scenarios that really separate science
testing from the software team testing.
A tool that is run standalone (as in development unit test) is less
rigorously tested than one that is run as part of a data
analysis thread where tool's functionality is judged, not only
the tool execution and creation of output data, but whether the
data product provides all of the input required to do further
analysis with other tools. The output products of one tool are
often the input to many tools that utilize those data to do
further analysis. It is sometimes found that there are keywords
missing in a file header, or table columns written by a tool
have missing information or are problematic with other programs
that read those data products. Sometimes the software upgrades
are in software libraries. In those cases we identify tools
that explicitly utilize the enhancement or bug fix for this
level of testing.
A science tester first outlines a plan based on the requirements for
the tool, their expectations of the tools flexibility, and the
range of parameters and data the tool was specified to work
with. They have often already provided baseline datasets and
use-cases for the developers unit test. At this stage the goal
is to catch missing details in the specification or
implementation of the tool.
Once there is a plan, the
scientist runs the commands or builds a script to test the tool.
Often iteration with the development team starts at this point.
E-mails are sent to an internal alias where they are being
stored; They are accessible from an e-mail archive. During
this process questions are asked, issues are uncovered, and
interpretation of requirements is discussed. Developers address
reported problems and make bug fixes and enhancements to the
tool. E-mail messages also seed bug reports that track
problems.
The unit testing stage is completed either with successful completion
of the unit test, or by identifying that the uncovered problem
would require a longer time-frame for solving. Each unit tester
captures the testing steps along with the input and output data,
and prepares a unit test worksheet.
The test worksheet is a
means to communicate the statusof unit testing, and when success
is indicated, it is the trigger to have the test added to the
regression suite. Tests are run across all platforms of a CIAO
release to verify that there are no porting issues.
Each test worksheet contains information on the tool status, level of
completion of unit testing, and a log of the test session and/or
a pointer to a test script. Furthermore, it contains pointers to
input data, and to datasets created as output from the tool. The
test worksheets are then used as input to build automated
regression test scripts. Scientists work to validate the tool
on their computer platform of choice. The work to validate the
tools across ported platforms is for most part performed
automatically.
When unit testing is completed, and the issues related to a given tool
are resolved, the tool lead indicates that testing was
successful and the tool is ready for release. Sometimes it is
at this point that a tool is rejected from the upcoming release,
or if the tool is required for a successful release, the
schedule is extended to allow time to address the problem that
must be solved.
Over the few weeks of testing following a code
freeze, theCIAO system is built bi-weekly to bring tools that
have had problems addressed back to the release configuration.
They are re-tested by unit testers until they are ready to
complete a new test worksheet. At this time, a targeted
regression test is built that contains a sample of previously
released CIAO tools, the new or modified tools, and tools that
exploit the functions added to libraries. This test set is
small so it runs quickly and is aimed at exercising the
highest risk areas of the software. Since the test script
consists of a subset of what would become the full regression
test, we refer to this as the mini-test. The mini-test is run
once the first build of the system is completed after a code
freeze, and almost weekly during the period of rebuild and
science unit testing to ensure software stability over all
platforms.
Later in the release cycle, the software tests outlined in the test worksheets are added to the regression test script. The output results of the current execution are compared to data saved from previous runs. Some of these data are from the last CIAO release, while new tests use data provided with the test worksheet. When the mini-test is upgraded with the worksheet tests, we refer to it as the mini+ test.
Full regression testing
Late in the release schedule, a full
regression test is run and results are verified on all
platforms. This test is the culmination of all test scripts
written since the start of CIAO development. New tests outlined
in the test worksheets are added for each release. This test is
usually run once, late in the release cycle, across all ported
platforms, and the results verified against previous runs of the
tools.
An important part of this step is the overall test
evaluation by the SDS testing lead. The unit tests, mini tests,
and full regression tests are reviewed and evaluated, and upon
successful completion, the test lead signs off that science
testing is complete and the tools are ready for release. With
sign-off of science testing, we move into download testing which
is the last step in the release testing
Download testing
In download testing, the system as packaged
for release is posted to a hidden public web page. Scientists
assigned to verify downloads on various platforms supported by
CIAO, practice a download and installation of the system.
Installation instructions are reviewed, web links and tar files
validated, and the system is spot checked. The development team
provides a download test script which runs a sample set of CIAO
tools. The script reports whether the test is successful. At
this stage we evaluate success based on the execution of a tool
and not a comparison to previous results.
Both science and
development leads evaluate the readiness for release, and upon
successful completion of download testing give their concurrence
to make the system available for public download.
Summary
The test procedures described here, have been developed
over the 6 years of the Chandra mission. This framework has
assisted us in making 3 major releases and 6 minor releases of
CIAO. We generally support ports to Sun, several flavors of
Linux, and Mac systems. CIAO consists of approximately 80
command line tools and several large applications. Our release
schedule is well defined.
Our process is sometimes adapted to
specific circumstances but well within the framework described
here. We have found these steps very helpful to achieve stable
and successful CIAO releases.