Celebrating 25 Years of CIAO: A Milestone in Astronomical Data Analysis

Antonella Fruscione, Douglas Burke, Catherine Cranmer, Kenny Glotfelty,  Hans Moritz Günther, David P. Huenemoerder, William Joye, Nicholas P. Lee, Jonathan McDowell, Melania Nynka, David Principe, Aneta Siemiginowska

As we mark the 25th anniversary of the Chandra X-ray Observatory and of the Chandra Interactive Analysis of Observations (CIAO) software, we want to remember its long journey, celebrate its evolution from the beginning to the current version, and highlight some of its significant contributions to the field of X-ray astronomy. Since its inception, CIAO has evolved from a limited set of basic tools for data analysis and proposal preparation into a comprehensive suite of tools and applications that have empowered astronomers worldwide to dig deeper into astronomical data. Today’s CIAO represents a culmination of decades of dedication and innovation in the field of astronomical data analysis.

Bar graph showing Number of CIAO Downloads plotted against Year, from 2011 through 2023. The number slowly rises from just over 1000 to around 2000 in 2019, before spiking to near 3000 in 2020 and beyond.


Figure 1: Number of CIAO software downloads since 2011. CIAO has been downloaded thousands of times per year from all over the world. The spike in 2020 most likely represents the fact that users started to work from home on personal computers rather than institutional machines (where CIAO may be centrally downloaded).

A Giant Leap in Capabilities

First released in October 1999, CIAO began as a modest package, available on only three platforms—Solaris 2.6, Redhat Linux 5.2, and Slackware Linux 3.5—and with a suite of around 30 tools designed to analyze data from the newly launched Chandra X-Ray Observatory (see our first CIAO newsletter article from 2000). Over the years, CIAO has transformed dramatically, keeping pace with technological advancements and the evolving needs of the astronomical community. The word “ciao” comes from an old greeting expression in the Venetian language, “s'ciavo,” that means “I am your servant,” and the name symbolizes the software’s role in serving the astronomical community. This is a feat that, after a quarter century, we can claim we have accomplished!

A vertical timeline titled CIAO Releases showing the version history of the CIAO software from 1.0 in 1999 to 4.16 in 2024. The timeline includes 28 major releases, indicated on the left, and several patch releases in between major releases, shown on the right. The major releases are marked with black dots connected by a vertical line, and the patch releases are labeled with their respective version numbers.


Figure 2: A timeline of CIAO releases from CIAO 1.0 (year 1999) to CIAO 4.16 (year 2023). Patch releases in between major releases are illustrated in the right section.

Fast-forward to today, and CIAO 4.16 is a sophisticated, multi-platform software suite that includes over 150 tools and 100 contributed scripts and supports modern Linux and macOS operating systems. CIAO incorporates advanced data analysis features, makes extensive use of Jupyter notebooks, and has several components, like Sherpa and the contributed scripts, which are developed on Github in an open development model. SAOImageDS9, the premier visualization tool to display and analyze astronomical imaging data, is also distributed with CIAO.

Two images of a supernova remnant side by side. The right image shows the remnant in a predominantly yellow and blue color scheme, highlighting the diffuse structure of the expanding gas and dust. The left image displays the same remnant with a more vibrant color palette, including green, purple, and red, likely representing different elements or temperatures within the remnant. A bright central object is visible in both images, possibly the remnant of the exploded star.


Figure 3: X-ray image of the supernova remnant RCW 103 using two methods of color representation: true color (left) and tri-color (right). The true color image was created using the new energy_hue_map script, which combines the adaptively smoothed counts image with the median energy at each location to create a true color image. Here, the color hues span a continuous range capped by red (median energies below 1.0 keV) and violet (median energies above 1.4 keV). The tri-color image was created using standard Chandra Source Catalog energy bands: soft (0.5–1.2 keV), medium (1.2–2.0 keV), and hard (2.0–7.0 keV). Traditional RGB tri-color images start by creating 3 images in separate energy bands, which are independently scaled to act as the individual color channels of the output image. The combination of primary colors (RGB) leads to the visual interpretation of secondary colors (eg. yellow, cyan etc.). True color images, in contrast, are created from a continuous energy range producing images in the “HSV” (Hue Saturation Value) color system.

A screenshot of the SAOImage DS9 software displaying a Chandra image of SNR G292.0+1.9, a large cloud of expanding gas with several prominent rising columns. The main image shows a colorful representation spanning the rainbow: mostly green, with a red wall to the left and a compact blue and purple core. An inset on the right shows a grayscale broadband image of the same object. The software interface includes various toolbar options and a sidebar displaying image information.


Figure 4: Chandra image of SNR G292.0+1.9 using the latest beta release of SAOImageDS9 v8.6b2, demonstrating some of the program’s new features. In addition to long-term support for RGB frames, DS9 now supports HSV and HLS frames, such as shown here. The colors of this image represent median energies from 1.0 keV (red) to 1.5 keV (purple). The gray scale inset is a broad-band image created with DS9, saved in PNG format, and inserted into the frame using the recently added Illustrate mode. This mode allows images and graphics (text, circles, boxes, polygons) to be placed directly in the display. The DS9 window is shown using the new Advanced View layout (View → Advanced).

An image with 6 frames. All images show the cluster Abell 2052, which consists of a bright, central point-like source, a radially-decreasing brightness, and several cavities, notably to the north and south of the center. Colors move from white at the brightest parts to a dark blue at the faintest, passing through beige, brown, and blue. The upper left image is a standard, binned astronomical image, while the image below it is spatially binned by larger hexagons. In the central column, the image is binned into rectangles of varying sizes on top and angular wedges of different angles below. The right column shows the images in contours above and uniformly-colored irregular polygons below.

Figure 5: Demonstration of some of the advanced adaptive/alternative binning tools available in CIAO. All images show a 620 ks observation of Abell 2052 in the 0.5–7.0 keV band. The left column shows uniform tilings: bins of squares (top, ~1″ on a side) and hexagons (bottom, side lengths of ~4″), the latter generated using the hexgrid tool. In the center column, the images are adaptively binned to have at least 900 counts per bin: rectilinearly (top) using dmnautilus and in polar coordinates (bottom) using dmradar. The right column shows more morphologically complex binning: a contour map (top) generated with dmcontour and mkregmap and a centroid map (bottom) produced by centroid_map, which is based on Voronoi tessellation of the local maxima in the image. Different binning methods can be especially useful when making products such as temperature maps or hardness ratio maps.

Expanding Accessibility and Usability

Initially, CIAO was designed primarily for X-ray astronomers working on Chandra data. However, over the years, the software has evolved on many levels: for example tools have been updated both to respond to changes in Chandra's operations or calibration as well as to support data and users beyond Chandra (reflecting its mission-independent design). CIAO has also become more accessible to a broader audience, including novice astronomers, students at various educational levels, and researchers from small and large institutions worldwide. This inclusivity has been facilitated by extensive documentation—designed to help non X-ray experts—and through 20 CIAO workshops held globally, which played a significant role in educating new generations of astronomers while also gathering valuable user feedback used to continually enhance the software and the documentation.

The CIAO workshops—which blend lectures and hands-on sessions to give real experience with Chandra data—began in 2001, attracting around 30 students per event to the Center for Astrophysics in Cambridge, Massachusetts. Over time, these workshops were extended to various conferences and institutions lacking local expertise, including one in 2017 in Pune, India, and a virtual workshop in 2020 during the Fifth Arab Astronomical Society School for Astrophysics in Egypt. These international experiences, in regions traditionally overlooked by Euro- and US-centric institutions, profoundly broadened the CIAO team’s perspective and underscored the critical importance of extending support to institutions and users lacking the necessary infrastructure, support systems, and expertise to fully leverage the benefits of Chandra's “free and open” data and software. For example, CIAO now includes scripts designed to handle data analysis even in low bandwidth situations.

Adapting to Technological Shifts

The journey of CIAO has seen significant technological shifts, even to some of its very core infrastructure. In the early 2000s, scripting in CIAO used the S-Lang scripting language; however, by 2008 Python became CIAO’s primary scripting language, due to the latter’s widespread adoption in the astronomical community and extensive documentation. This transition has not only modernized CIAO but also enhanced its capabilities, making it more versatile and user-friendly. The version 4.0 update that implemented this change also involved reviewing and rewriting the entire suite of contributed scripts to ensure consistency and maintainability. Moreover, some of the original and obsolete tools were dropped in favor of more modern tools, complex scripts, or applications. CIAO’s Sherpa application, which is now available as a standalone Python package on GitHub, exemplifies the possibilities of an open-source collaboration and modern software practices.

Enhanced User Support and Documentation

The documentation, probably CIAO’s crown jewel, has evolved significantly since 1999. What started as simple text files has grown to become a comprehensive list of data analysis threads and guides and, most recently, Jupyter notebooks. This impressive collection now encompasses over 2100 pages between CIAO and Sherpa, including more than 1160 help pages, 235 bug pages, 181 data analysis threads, 118 dictionary entries, and 101 FAQs, as well as various guides, caveats, galleries, “why” topics, and other written materials.

The Jupyter notebooks provide an interactive environment where users can execute code and visualize results within the same document, making data analysis more intuitive and accessible. This addition represents a major step in making CIAO’s powerful tools even more user-friendly.

A User-Driven Evolution

User support has always been a priority for CIAO. From the early days of ad hoc email support to the robust HelpDesk system today, CIAO has maintained a commitment to assisting its users promptly and effectively.

Bar graph showing Number of Helpdesk tickets plotted against Year from 2012 to 2023. The number of tickets generally ranged between 250 and 350 each year, with peaks of almost 400 in 2020 and 2021. There was a slight decline in recent years, with the number of tickets falling to around 200 by 2022 and 2023.


Figure 6: Number of Helpdesk tickets per year since 2012. Every year hundreds of questions are answered by CIAO experts, most within one day. Questions and answers contribute to the improvement of the software and the documentation.

CIAO’s journey has been characterized by a continuous exchange between its designers and the users. HelpDesk queries, user contributions, feedback from workshops, and the adoption of new technologies have all played crucial roles in shaping and reshaping the software. This need-driven evolution has allowed CIAO to remain an up-to-date software system for the wide multiwavelength astronomical community, capable of meeting the evolving demands of scientific research.

Celebrating Achievements and Looking Ahead

As we celebrate 25 years of CIAO, we know that its legacy will endure well past Chandra, having set a high standard for astronomical data analysis tools and documentation. It is important to recognize that CIAO is the product of a collective effort of a large community of users—software developers, scientists, students—who all have contributed to CIAO’s success in different ways. The software has become an indispensable tool for X-ray astronomers and beyond, facilitating groundbreaking discoveries and, in keeping with one of the grand challenges of the Smithsonian Institution, “unlocking the mysteries of the universe.” CIAO is truly at the service of science!

Acknowledgments

CIAO has been a collaborative effort involving numerous Chandra X-ray Center (CXC) members—too many to list here—over the past 25 years, including members from the CXC Science Data Systems group at the Smithsonian Astrophysical Observatory (SAO) and Massachusetts Institute of Technology (MIT) and from the CXC Data Systems group at SAO. CIAO 4.16 builds on the many contributions and the many achievements along the way.

This work is funded by NASA Contract NAS8-03060, Chandra X-ray Center.