Testing of CIAO

Previous   Contents   Next

Testing of CIAO

Reprinted by permission from ADASS XV Proceedings (Escorial, Spain, 2005) ASP Conf. Ser., G. Gabriel, C. Arviset, D. Ponz, and E. Solano, eds.


Chandra Interactive Analysis of Observations (CIAO) consists of a suite of tools and applications developed for the analysis of Chandra X-ray Observatory data. With its unprecedented technical capabilities and science potential, Chandra requires a novel, flexible, multi-dimensional set of tools for data analysis. This mission provides 4-dimensional data (2 spatial-dimensions, time, & energy), each containing many independent elements. CIAO tools are able to filter and project the 4-dimensional event data to manageable sizes and convenient arrays in an easy and flexible manner. CIAO is a mission independent system and can be used for data analysis of other X-ray and non- X-ray observations.

Since launch there have been several major CIAO releases. A major step in the preparation of each CIAO release is scientific and functional testing of the software. This is a joint effort by the Chandra X-ray Center Data Systems (DS) and Science Data Systems (SDS) teams. We highlight here the stages of CIAO testing and the procedures we follow at each stage.

CIAO Tool Development

The CIAO tool development process begins many months before a planned release. SDS scientists identify and evaluate science and calibration milestones for Chandra data analysis, and outline the priorities and goals for tool and application development for a CIAO release.

During the software development process scientists and developers interact closely to assure CIAO tools are designed and built to the specified requirements and that they follow CIAO system standards.
Once implemented, unit tests are developed and run to assure proper implementation. We recognize that more rigid tests are required though, and science and development teams combine efforts to ensure this goal is met.

CIAO Release Testing

CIAO releases involve 4 testing stages. These are: (1) unit testing, where detailed tests of CIAO tools verify and validate the correct implementation of scientific algorithms; (2) mini-testing, where the new functionality of the system is bundled and tested across ported platforms; (3) regression testing, where baseline system tests are run on all ported platforms to validate the system; and (4) package testing, where tar files are downloaded from the public web page to validate web links, instructions, and the software installation.

Science Unit Testing

Unit science tests are developed for each new or updated tool as they are completed by the software team. It is in this testing phase that scientists run tools they have specified for the first time. Unit tool testers verify and validate that the tool was implemented as they envisioned with correct and robust algorithms to meet the science goals. They also run the tools in science data analysis scenarios where the output file of one tool is provided as the input to the next in a series of tool steps organized to meet specific science goals. It is these science scenarios that really separate science testing from the software team testing.

A tool that is run standalone (as in development unit test) is less rigorously tested than one that is run as part of a data analysis thread where tool's functionality is judged, not only the tool execution and creation of output data, but whether the data product provides all of the input required to do further analysis with other tools. The output products of one tool are often the input to many tools that utilize those data to do further analysis. It is sometimes found that there are keywords missing in a file header, or table columns written by a tool have missing information or are problematic with other programs that read those data products. Sometimes the software upgrades are in software libraries. In those cases we identify tools that explicitly utilize the enhancement or bug fix for this level of testing.

A science tester first outlines a plan based on the requirements for the tool, their expectations of the tools flexibility, and the range of parameters and data the tool was specified to work with. They have often already provided baseline datasets and use-cases for the developers unit test. At this stage the goal is to catch missing details in the specification or implementation of the tool.
Once there is a plan, the scientist runs the commands or builds a script to test the tool. Often iteration with the development team starts at this point. E-mails are sent to an internal alias where they are being stored; They are accessible from an e-mail archive. During this process questions are asked, issues are uncovered, and interpretation of requirements is discussed. Developers address reported problems and make bug fixes and enhancements to the tool. E-mail messages also seed bug reports that track problems.

The unit testing stage is completed either with successful completion of the unit test, or by identifying that the uncovered problem would require a longer time-frame for solving. Each unit tester captures the testing steps along with the input and output data, and prepares a unit test worksheet.
The test worksheet is a means to communicate the statusof unit testing, and when success is indicated, it is the trigger to have the test added to the regression suite. Tests are run across all platforms of a CIAO release to verify that there are no porting issues.

Each test worksheet contains information on the tool status, level of completion of unit testing, and a log of the test session and/or a pointer to a test script. Furthermore, it contains pointers to input data, and to datasets created as output from the tool. The test worksheets are then used as input to build automated regression test scripts. Scientists work to validate the tool on their computer platform of choice. The work to validate the tools across ported platforms is for most part performed automatically.

When unit testing is completed, and the issues related to a given tool are resolved, the tool lead indicates that testing was successful and the tool is ready for release. Sometimes it is at this point that a tool is rejected from the upcoming release, or if the tool is required for a successful release, the schedule is extended to allow time to address the problem that must be solved.

Over the few weeks of testing following a code freeze, theCIAO system is built bi-weekly to bring tools that have had problems addressed back to the release configuration. They are re-tested by unit testers until they are ready to complete a new test worksheet. At this time, a targeted regression test is built that contains a sample of previously released CIAO tools, the new or modified tools, and tools that exploit the functions added to libraries. This test set is small so it runs quickly and is aimed at exercising the highest risk areas of the software. Since the test script consists of a subset of what would become the full regression test, we refer to this as the mini-test. The mini-test is run once the first build of the system is completed after a code freeze, and almost weekly during the period of rebuild and science unit testing to ensure software stability over all platforms.

Later in the release cycle, the software tests outlined in the test worksheets are added to the regression test script. The output results of the current execution are compared to data saved from previous runs. Some of these data are from the last CIAO release, while new tests use data provided with the test worksheet. When the mini-test is upgraded with the worksheet tests, we refer to it as the mini+ test.

Full regression testing

Late in the release schedule, a full regression test is run and results are verified on all platforms. This test is the culmination of all test scripts written since the start of CIAO development. New tests outlined in the test worksheets are added for each release. This test is usually run once, late in the release cycle, across all ported platforms, and the results verified against previous runs of the tools.
An important part of this step is the overall test evaluation by the SDS testing lead. The unit tests, mini tests, and full regression tests are reviewed and evaluated, and upon successful completion, the test lead signs off that science testing is complete and the tools are ready for release. With sign-off of science testing, we move into download testing which is the last step in the release testing

Download testing

In download testing, the system as packaged for release is posted to a hidden public web page. Scientists assigned to verify downloads on various platforms supported by CIAO, practice a download and installation of the system. Installation instructions are reviewed, web links and tar files validated, and the system is spot checked. The development team provides a download test script which runs a sample set of CIAO tools. The script reports whether the test is successful. At this stage we evaluate success based on the execution of a tool and not a comparison to previous results.
Both science and development leads evaluate the readiness for release, and upon successful completion of download testing give their concurrence to make the system available for public download.


The test procedures described here, have been developed over the 6 years of the Chandra mission. This framework has assisted us in making 3 major releases and 6 minor releases of CIAO. We generally support ports to Sun, several flavors of Linux, and Mac systems. CIAO consists of approximately 80 command line tools and several large applications. Our release schedule is well defined.
Our process is sometimes adapted to specific circumstances but well within the framework described here. We have found these steps very helpful to achieve stable and successful CIAO releases.

M. Karovska, J. D. Evans, and the CIAO testing team


The testing of CIAO is carried out by members of Chandra X-ray Center Science Data Systems and Data Systems teams. Support for this work is provided by the National Aeronautics and Space Administration through the Chandra X-ray Center, which is operated by the Smithsonian Astrophysical Observatory for and on behalf of the National Aeronautics and Space Administration under contract NAS 8-03060.

Detailed information on
Chandra X-ray Observatory and CIAO can be found in: http://cxc.harvard.edu/ciao/.