CIAO 4.16 Release Notes
CIAO 4.16 is distributed for the following platforms:
- Linux 64 bit
- Apple macOS 11 (Big Sur) through macOS 14 (Sonoma), Intel/x86 and ARM/M1/M2/M3
CIAO is available using the conda package manager; conda builds are available for Linux and Mac for Python 3.11, 3.10, and 3.9. More information can be found in the conda section of the Installation notes below. The ciao-install script can be used to automate the conda installation process.
There is no support for 32 bit operating systems, older Linux (CentOS 7 era), older macOS platforms (macOSX Catalina and earlier), or Windows (including using WSL). More details can be found on the Platform Support page.
- Notable changes and improvements in CIAO 4.16
- How CALDB 4.11.3 Affects Your Analysis
- How CALDB 4.11.2 Affects Your Analysis
- How CALDB 4.11.1 Affects Your Analysis
- How CALDB 4.11.0 Affects Your Analysis
- Installation and Setup
- Tools
- Parameter Files
- Sherpa
- Graphical User Interfaces
- Analysis Scripts
- Python Modules
- Libraries
- Environment
- Documentation
Notable changes and improvements in CIAO 4.16
CIAO 4.16 includes several important bug fixes and includes updates to support newer compilers. There have been enhancements to Sherpa and CIAO 4.16 provides updates to Off The Shelf (OTS) packages like SAOImage DS9.
-
CIAO 4.16 is now available as native binaries for Apple silicon: M1, M2, M3 ARM processors. (Rosetta2 is no longer required.)
-
CIAO can be installed using the conda package manager. Users can install the conda edition with Python 3.11, 3.10, or 3.9 on Linux and Mac.
The default version of Python is variable and will depend on which other packages are available at the time of installation. Users who want a specific version of python must include python=3.9, python=3.10, or python=3.11 on the conda create command.
Alternatively, the ciao-install script can be used to install CIAO 4.16.
-
Several tools have been updated including important bug fixes and enhancements such as:
- tgextract and mkgrmf have been updated to correct errors with abs(order)>1
- acis_process_events has been updated to support enhanced temperature dependent CTI corrections.
- wcs_match has been updated to to support stacks and range syntax when manually selecting sources.
- Several tools, including dmextract, have been updated to copy the WCS from the input image to the output table.
-
Notable Sherpa changes include:
- Updated XSPEC models to version 12.13.1e
- The grouping commands like group_counts now default to only grouping within the noticed range of channels
- Added support for using arbitrary Python functions when linking model parameter values
- Now supports XSPEC table models with the ESCALE parameter and fixes an issue with models that support a redshift parameter
- Experimental: support for writing out XSPEC table models
- Experimental: added support for plotting using bokeh (not included with CIAO; users need to install separately).
As a reminder, Sherpa is also available as a stand alone system, accessible from the Sherpa GitHub repository or from the standalone Sherpa page.
-
Core CIAO libraries have been updated to
- Improved handling of non-CIAO files which violate the CXC Datamodel including invalid/missing meta-data.
- Corrected bug in parameter interface library affecting some specific user names (starting with par) in particular when running the smoke tests. Also now supports accessing a parameter's data-type/class by using the .p_type modifier.
-
There have been several script updates released with CIAO 4.16 including
- energy_hue_map allows users to create "true" color rather than traditional "tri" color images. Along with this comes several new alternative image binning tools and utilities including: hexgrid, vtbin, centroid_map, pathfinder, mkregmap, merge_too_small, and map2reg,
- blanksky, blanksky_image, and blanksky_sample have been updated to use new CHKVFPHA keyword to determine if the very-faint mode background filtering has been applied.
- specextract now supports MARX event files.
- convert_xspec_script now supports the XSPEC ENERGEIS command.
- srcflux now allows users to create point-source optimized regions using the psf_contour and bkg_fixed_counts tools.
-
CIAO includes version 8.5 of SAOImage ds9. Users are again reminded that they need to change the Edit mode setting in order to select or create regions in ds9. Please see the watchout page for more information on this.
SAOImageDS9 v8.5 highlights include:
- Internal SAMP hubs allows ds9 to communicate directly with other SAMP enabled applications and websites without need of a third party application.
- Added new XPA command to get counts in region.
- Users can now adjust plot symbol sizes.
How CALDB 4.11.3 Affects Your Analysis
CALDB 4.11.3 Release Notes (release 21 August 2024)
HRC Data
-
HRC-S Time-Dependent QEU Files
The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0019) extends the prior N0018 set, adding a new QEU file based on the most recent HZ 43 observation, effective since 2024 March 4. The wavelength-independent drop in QE since the previous November 2023 observation is 2-3.5%, depending on dispersed orders. The wavelength-dependent corrections required on S1 and S3 continue to follow long-observed trends. The data used to derive the previous two wavelength-dependent models (2022-09-09 and 2023-09-08) have considerably better statistics than was previously the case.
These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:
How CALDB 4.11.2 Affects Your Analysis
CALDB 4.11.2 Release Notes (release 6 June 2024)
HRC Data
-
HRC-I Time-Dependent QE Files
The HRC-I quantum efficiency QE has been decaying for some time, particularly since the year 2011, when a 5% to 10% per year reduction had been measured. To address this issue, the HRC-I Calibration Team had introduced a time-dependence into the QE files in 2018 which are now extended in time to the latest epoch of observations. Included in this update is a new file coinciding to the recent HRC-I high-voltage adjustment affecting observations after 2023 September 27. The QE update affects the results from mkinstmap and fluximage and is applicable to all previous epochs of HRC-I observations.
How CALDB 4.11.1 Affects Your Analysis
CALDB 4.11.1 Release Notes (release 18 April 2024)
ACIS Data
-
Warm Temperature ACIS-I CTI Correction Files
Improved time- and temperature-dependent correction to the ACIS charge transfer inefficiency (CTI) correction files (N0012) for the ACIS-S2 and ACIS-S3 chips, joining the four ACIS-I chips with corrections at warm focal-plane temperatures up to -107 C introduced in N0011. No changes are made for data taken with focal-plane temperatures below -117 C, or prior to 2005 on ACIS-I. The CTI corrections for ACIS-S0, -S1, -S4, and -S5 remain the same as the prior N0009 version of the CTI corrections.
Users working with observations using any of the ACIS-S2, ACIS-S3, or any of the chips in the ACIS-I array made after 2005 or with focal-plane temperatures greater than -117 C may wish to run chandra_repro and reprocess the data to improve the CTI gain shift calibration.
HRC Data
-
HRC-S Time-Dependent QEU Files
The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0018) shows decline in the net count rate of 1-2% between the latest set of HZ43 calibration observations, effective since 2023 September 8. The set of QEU files are unevenly distributed in effective dates and affects every HRC-S observation since the start of the mission.
These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:
How CALDB 4.11.0 Affects Your Analysis
CALDB 4.11.0 Release Notes (release 14 December 2023)
ACIS Data
-
Warm Temperature ACIS-I CTI Correction Files
Improved time- and temperature-dependent correction to the ACIS charge transfer inefficiency (CTI) correction file (N0011) for the four ACIS-I chips, in particular, introduction of corrections for for warm focal-plane temperatures up to -107 C. No changes are made for data taken with focal-plane temperatures below -117 C, or prior to 2005 on ACIS-I. The CTI corrections for all six chips on ACIS-S remain the same as the prior N0009 version of the CTI corrections.
Users working with observations using any of the chips in the ACIS-I array made after 2005 or with focal-plane temperatures greater than -117 C may wish to run chandra_repro and reprocess the data to improve the CTI gain shift calibration.
HRC Data
-
HRC-I Time-Dependent QE Files
The HRC-I quantum efficiency (QE) has been decaying for some time, particularly since the year 2011, when a 5% to 10% per year reduction had been measured. To address this issue, the HRC-I Calibration Team had introduced a time-dependence into the QE files in 2018 which are now extended in time to the latest epoch of observations, effective 2022 September 4.
This N0014 version of the QE maps now account for QE loss at higher energies, prior versions were limited to wavelengths >60 Å, but is now extended over the range of 0 Å < λ < 135 Å. The QE update affects the results from mkinstmap and fluximage and is applicable to all previous epochs of HRC-I observations.
Installation and Setup
Users should be aware of these installation items before installing CIAO 4.16. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.
The steps needed to install CIAO have changed in CIAO 4.16 for both installation methods: conda and ciao-install, as discussed below.
Supported Platforms
-
CIAO 4.16 is supported on:
- Linux 64 bit
- Apple macOS 11 (Big Sur) through macOS 14 (Sonoma), Intel/x86 and ARM/M1/M2/M3
Installing with the conda package manager
CIAO can be installed using the conda package and environment management system.
The conda release is available from the custom CXC channel and requires using the conda-forge channel.
$ conda create -n ciao-4.16 -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \ ciao pyciao sherpa ds9 marx ciao-contrib caldb_main python=3.11
See the conda installation instructions for more information, including known problems and incompatibilities.
With the conda edition users can choose to install CIAO with Python 3.11, 3.10, or 3.9. Other OTS such as numpy and matplotlib will also be be different based on the python version.
Install using the ciao-install Installation Script
CIAO 4.16 is only available as a set of conda packages. The ciao-install script has been re-written to allow users to unfamiliar with conda to automate the installation process using a single, simple command:
$ bash ciao-install
Users can download the ciao-install script from the ciao-install download page. Users can customize the download to include additional segments such as the ACIS and HRC blank sky background CALDB files, the CIAO source code, download for different operating systems, link to existing CALDB installations, and more.
Source builds
CIAO 4.16 source code has gone through a significant reorganization and changes to the build system. CIAO is now built using cmake and meson instead of using make and Makefiles.
To download the CIAO source code it can be added with the ciao-install script
$ bash ciao-install --add ciao-src --add sherpa-src
and then built using the ciao_build.sh script
$ ciao $ cd $ASCDS_INSTALL $ bash ./ciao_build.sh --prefix `pwd`
(assuming users have setup the ciao alias as suggested).
Smoke Tests
CIAO 4.16 no longer uses make to run the CIAO smoke tests. The run_smoke_tests.sh script is now used:
$ ciao $ cd $ASCDS_INSTALL $ bash test/smoke/bin/run_smoke_tests.sh
-
Users of previous versions of CIAO will note that the order of the tests is different; in the event of a test failure be sure to include the name of the test, not just the test number when contacting helpdesk.
-
The obsvis smoke tests have been simplified to reduce the amount of windows that are created (and usually taking focus).
Remove old parameter files
With every new CIAO release, some parameter files are changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:
unix% rm ~/cxcds_param4/*
Tools
acis_process_events
-
Added support for new temperature dependent CTI file format.
-
Added a new keyword, CHKVFPHA, to indicate whether the check_vf_pha algorithm has been applied.
acis_build_badpix
-
Fix warning about finding multiple CALDB files.
acis_streak_map
-
Fix for compatibility issue with Numpy 1.25 and newer.
asphist
-
Corrected the units from "degrees" to "deg" as per the FITS standard.
asp_offaxis_corr
-
Fix for problem when dithering across the RA=360|0 boundary.
celldetect
-
Some error handling cleanup.
-
Corrected the units from "degrees" to "deg" as per the FITS standard.
dmarfadd
-
Internal: add missing function prototypes
-
Updates to support newer versions of C-Lang compiler
dmcontour
-
Some error handling cleanup.
dmcoords
-
Code cleanup to remove some compiler warnings.
dmcopy
-
Fixed an issue when copying a FITS image to ASCII (TEXT/DTF) format that messed up the image subspace.
dmextract
-
When using opt=generic or opt=generic2 with images, dmextract now will copy the world coordinate system from the input image to the output table's X and Y columns. The real "RA" and "Dec" columns are now removed as the values can now be accessed as WCS virtual columns.
dmfilth
-
Some error handling cleanup.
-
Corrected the units from "degrees" to "deg" as per the FITS standard.
-
Images that do not provide a full world coordinate system can now be used. The only requirement is that the regions used must be in physical coordinates rather than celestial coordinates.
dmimgfilt
-
Internal: add missing function prototypes
dmimghull
-
Internal: add missing function prototypes
dmimglasso
-
Some error handling cleanup.
dmimgpick
-
The WCS on the input image is now copied to the output table. The X,Y output columns are now a vector column rather than individual scalar columns.
dmlist
-
Fixed issue when listing contents of a file which contains vector columns of either strings or booleans data type.
dmmaskbin
-
Internal: add missing function prototypes
dmnautilus
-
Internal: add missing function prototypes
dmregrid
-
Some error handling cleanup
evalpos
-
Updated to support modified output from dmimgpick tool.
get_src_region
-
The WCS on the input image is now copied to the output table.
-
Some error handling cleanup.
mkgrmf
-
A bug has been fixed which caused the enclosed energy fraction, due to the spectral extraction cross-dispersion width, to be incorrect for orders than +/-1. The magnitude depends on wavlength, but was at maximum 3% in an order's total effective area. (This factor occurs in a grating RMF normalization.)
reproject_image_grid
-
Corrected the units from "degrees" to "deg" as per the FITS standard.
rmfimg
-
Added support for RMF files where the MATRIX and EBOUNDS extensions are in arbitrary order.
sso_freeze
-
Corrects the CRVAL and CRPIX values shown in the verbose output.
tgextract
-
A bug has been fixed in the spectral extraction regions written to the output PHA files for polygon regions. These had first order grids instead of wavelengths scaled by diffraction order. These regions are used by mkgrmf to compute the enclosed energy fraction in the cross-dispersion direction as a function of wavelength. The factor becomes a normalization term in the grating RMF. This error only affected non-standard products. Default grids for HETG are boxes, which have correct coordinates. For LETG/HRC-S, by default, only a first order region is produced.
vtpdetect
-
Some error handling cleanup
wcs_match
-
If the last matching pair was a poor match, the code mistakenly deleted that pair and the prior, valid matched pair. The logic has been updated to prevent the valid pair from being deleted.
-
Fixed a bug when using manual matching and duplicate values were input.
-
In manual matching mode, users can now specify values in an external file using the @file_name syntax as well as supporting ranges in the form a:b.
wrecon
-
Some error handling cleanup.
-
Corrected the units from "degrees" to "deg" as per the FITS standard.
wtransform
-
Some error handling cleanup.
-
Corrected the units from "degrees" to "deg" as per the FITS standard.
Parameter Files
We recommend deleting all the old parameter files or renaming the parameter-file directory before running any new version of CIAO to avoid conflicts. More information is available in the FAQ.
Sherpa
Sherpa is developed as an open project, and is available on GitHub. Comments, fixes, and additions from the community are welcome. There are additional releases for this version throughout the year, and can be installed either directly from GitHub or using the conda package manager, as described on the standalone Sherpa documentation page.
changes
-
The full list Sherpa updates is given in the
4.16.0
and 4.15.1
release notes for the two standalone releases that are incorporated into CIAO 4.16.
The major updates were made to data filtering and grouping, addition of a new plotting backend
and bug fixes. The release highlights include:
- XSPEC
-
-
Updated to version 12.13.1e of the XSPEC model library.
-
Fix an issue with the redshift parameter for XSPEC table models.
-
Allow XSPEC table models which include the ESCALE parameter.
-
Provide experimental support for writing out XSPEC table models.
-
- Filtering and grouping
-
The grouping commands like group_counts and group_snr now default to only grouping within the noticed range of channels, which is a change in behaviour. The filter status is now reported for calls to the group commands and set_analysis, to follow the behavior of notice and ignore which were changed to report the filter status in CIAO 4.15.
- Models
-
Added support for use of arbitrary python functions when linking model parameters.
- Data access
- Random number generation
-
It is now easier to ensure that Sherpa is repeatable, taking advantage of the NumPy Random Number Generator along with the new set_rng and get_rng calls.
- Plotting improvements
-
New plotting backend: Users can now choose between matplotlib and bokeh (experimental, users need to install the package)
A number of plot issues related to PHA data have been addressed.
- Documentation
-
Improved documentation across the system.
Graphical User Interfaces
Obsvis
Obsvis is no longer available as a separate, stand-alone download. Users need to install CIAO to use obsvis.
-
Update default ACIS-I/S y/z offset values to 0.0 to match the obsvis.cal values used since 2015.
-
Updated obsvis to fix an issue with FoV roll being reset when the image frame is binned in ds9
-
Updated code to preserve target string (ie. keep spaces) so that the parser to properly handle target names containing multiple words (ie. 'ar lac' or 'eta carina').
-
Updated to add "Configuration File Version" option to Help pulldown menu. Selecting the option invokes a message dialog indicating the loaded .obsvis,.cal file and latest detected version available for download.
Analysis Scripts
There have been several updates to the contributed scripts package since CIAO 4.15 was released. These changes are specific to the CIAO 4.16.0 contributed scripts release.
dax
- The model editor now checks that any values that have been modified have been set (by pressing the Return or Enter key) before allowing the user to proceed with certain actions such as Fit or Plot.
- The image tasks will now handle images that have been blocked by DS9.
-
Re-ordered and renamed a number of tasks:
Original name New name PI Spectrum (PI) PHA Spectrum (PHA) TIME Ligthcurve (Time) EXPNO Lightcurve (Exposure Number)
energy_hue_map
-
A new technique to create true color images from a continuous energy spectrum rather than traditional tri-color images created using 3 energy bands. The inputs include a map estimating the median (or mean) energy in each pixel (for example created using the statmap tool), and an image of the counts or flux.
hexgrid
-
Creates a regular grid of hexagonal shaped map regions. Hexagons can best approximate circles as they are the highest order regular polygon that can uniformly tile a plane.
vtbin
-
Creates a map file based on the Voronoi Tessellation of points in the input image. By default it uses the location of the local maxima or users can supply their own locations (via the sitefile).
centroid_map
-
This script runs vtbin and then computes the centroid of the pixel values in each map region. It then repeats the Voronoi Tessellation using the centroid values N-many times, where N is the numiter parameter.
pathfinder
-
This script generates map regions based on the local gradient. It follows the steepest ascent to the local maxima. All pixels that reach the same local maxim are grouped together. Better for point sources and other "clumpy" emission.
mkregmap
-
Turn a stack of regions into a map file.
map2reg
-
A utility tool to take a map file and converts it into a region file.
merge_too_small
-
Some of these algorithms can create small regions needed to "fill in" the space between other more statistically significant regions. The merge_too_small script will re-assign small groups to the largest neighboring group; where "small" is either area below threshold or counts below threshold.
apply_fov_limits
-
The script now defaults to creating images with the i4 datatype (4-byte integer) rather than i2 (2-byte integer) it used to use, as the old behavior could lead to data loss. This does increase the memory needed to store the images, so the datatype parameter can be used to change back to the old behaviour.
blanksky
-
blanksky has been updated to make use of the CHKVFPHA keyword, introduced to acis_process_events in CIAO 4.16, to determine if "status=0" event filtering should be applied to ACIS VFaint observations, but will fall back on the old technique of parsing the input event file header to make the determination if the new keyword is not available. The blanksky_sample script will make use of the reference 'infile' keywords for the optional combined infile/sampled background products instead of the 'Merged'-valued keywords as a result of using dmmerge.
check_ciao_version
-
The script has been updated to handle the updated ciao-install script and provide initial support for using mamba/micromamba to install CIAO.
convert_xspec_script
-
Support for the ENERGIES command from XSPEC has been added.
convert_xspec_user_model
-
The script has been updated to match changes to Sherpa in CIAO 4.16.
correct_periscope_drift
-
The script has been updated to support changes to Sherpa plotting in this release.
reproject_obs
-
The script has been updated so that file validation checks at its initialization will not have a dependency on mask files, which are otherwise not used.
specextract
-
The script has been updated so that event files generated by MARX are supported and better handling of the ancillary file parameters when set to "none" values. In particular when the bad pixel file parameter was set to "none", occasionally it would revert to using the default bad pixel file from the CalDB, which is now fixed.
srcflux
- Added new option to create "optimized" source and background regions based on the PSF.
- The marx_root parameter is also added to specify the directory where marx is installed.
- In addition, there is a numerical precision bug fix when computing the combined PSF fraction when merging datasets.
crates_contrib.masked_image_crate
-
This module introduces the MaskedIMAGECrate class. It is a subclass of the standard crates IMAGECrate class but extends it to include a valid method to check if the pixel location is a valid pixel (inside subspace and not NULL/NaN/Inf). It only supports 2D images.
ciao_contrib.runtool
-
The module has been updated to reflect parameter changes and the new scripts added in this release.
sherpa_contrib.profiles
-
The routines have been updated to support changes to Sherpa plotting in this release.
Python Modules
pycrates
-
Internal documentation tweaks.
Libraries
datamodel
-
Fix memory leaks with variable length arrays.
-
Fix for some non-Chandra/CIAO files with bogus DSTYP keywords. Specifically some XMM files filtered with non-Chandra tools had DSTYP keywords exceeding 80 characters which could cause tools to crash.
-
Updated to behave more robustly when a file contains references to subspace blocks: GTIs, MASKs, and REGIONs; but those blocks do not exist. This may happen when using non-CIAO tools to filter or save files.
-
Updated to be more robust when subspace keywords, DSVALn, are blank/invalid.
-
Fixed an issue affecting REGION blocks with non-standard headers (missing HDU keywords).
cxcparam
-
When prompting for a parameter value, if the min or max value was redirected to another parameter, the final redirected value will now be shown in the parameter prompt rather than the redirection.
-
Users can now get the parameters data type using the .p_type modifier. For example:
$ pget dmlist infile.p_type f
which shows the type a "f", ie. a file name.
Users may not set or change the data type of a parameter.
-
Fixed a problem accessing parameter files if the parameter file directory contained the sub-string ".par". This was most often seen during the CIAO smoke tests when the username of the user started with "par"
Environment
Off-the-Shelf (OTS) Package Versions
-
The following OTS packages are included with CIAO 4.16 installed with ciao-install.
Differences in OTS for conda installationThe conda-installed version of CIAO 4.16 has different OTS versions, and it also depends on the Python version you have chosen to use.
Executables
- ds9 8.5
Libraries
- cfitsio 4.3.1
- xpa 2.1.20
- XSPEC 12.13.1e (models only)
Python (major packages)
- Python 3.11.6
- NumPy 1.26
- Matplotlib 3.8.1
- jupyterlab (notebook) 4.0.8
Compiler versions
-
The following versions of the gcc compiler were used to compile the binary CIAO releases installed with ciao-install. The conda-installed compiler versions are different.
System Build System Version Linux RedHat Enterprise Linux 8 gcc 7.4 macOSX Big Sur llvm 11.0.3 -
Several CIAO applications have been updated to support the GNU++14 standard.