CIAO 4.7 Release Notes
Notable changes and improvements in CIAO 4.7:
- This is primarily a maintenance release: bug fixes, supporting new compilers and OTS upgrades.
- CIAO includes version 7.3.2 of SAOImage ds9: users may find that they are unable to select or create region files using this version due to a change in ds9; please see the watchout page for more information on this.
- Sherpa now supports stacks of datasets via the datastack package, which is intended for users who have multiple datasets to analyze, such as multiple observations (split obsid) of the same source or spectra of extended X-ray sources. It provides stack-enabled (i.e. vectorized) versions of the key Sherpa commands used to load data, set source models, get and set parameters, fit, and plot.
- Sherpa has been updated to use version 12.8.2e of the XSPEC model library.
- dmcoords now automatically makes use of the DY_AVG, DZ_AVG, and DTH_AVG keywords in the file headers to correct for the mean SIM drift during an observation. The asolfile parameter is no longer necessary nor required for data processed with ASCDSVER greater than 8.4.2 or has been reprocessed with chandra_repro.
- There have been several new scripts released since the CIAO 4.6 release, including: combine_grating_spectra, which replaces add_grating_orders and add_grating_spectra; convert_xspec_user_model, which compiles XSPEC user models so that they can be used in Sherpa; and monitor_photom, which generates a lightcurve from ACA optical monitor data. There have also been bug fixes and enhancements of existing scripts.
- How CALDB 4.6.9 Affects Your Analysis
- How CALDB 4.6.8 Affects Your Analysis
- How CALDB 4.6.7 Affects Your Analysis
- How CALDB 4.6.5 Affects Your Analysis
- Installation
- Tools
- Parameter Files
- ChIPS
- Sherpa
- Graphical User Interfaces
- Analysis Scripts
- Python Modules
- Libraries
- Environment
- Documentation
How CALDB 4.6.9 Affects Your Analysis
CALDB 4.6.9 Release Notes (release 23 September 2015)
ACIS Imaging and Grating Data
-
Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data
The new, time-dependent ACIS Gain (T_GAIN) files for May-July 2015 (Epoch 62) are introduced and updated for Epoch 61 in this release, and therefore, affects observations taken since 2015 February 01. Other observations are unaffected by these new calibration files.
Users working with ACIS data taken since 2015 February 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.
See the time-dependent ACIS T_GAIN why page for more information.
The gain corrections are at the nominal and expected levels of less than 2% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.
Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.
-
ACIS-S1 (ACIS-5) DET_GAIN and P2_RESP Upgrades
A new position-dependent gain map (DET_GAIN) has been developed for the back-illuminated chip ACIS-5 (ACIS-S1) which makes significant improvements on the calibration of this chips. The new S1 gain map removes most of the irregularities that distort the order-sorting events distribution, allowing for better order separation, and hence much improved spectral extraction and source modeling on the S1 chip. The new S1 gain value imply changes in the the P2_RESP files which are used to derive the RMFs for sources on the S1 chip.
It is important to note that:
- the main improvement is for ACIS-S/LETG grating observations below ~0.5 keV (i.e. dispersed onto the S1 chip).
- ACIS-S/HETG observations are largely unaffected by the gain change.
- imaging spectroscopy for sources on the S1 chip is improved in the soft energy range (in the sense of a more accurate RMF below ~0.5 keV).
- only observations where the S1 chip is fully CTI-corrected (i.e. NOT graded mode) are affected by the gain change.
A comparison of changes in the photon index/normalization/flux between the new and old gain map are given in Figures 7a and 7b of the Technical Details.
These CIAO tools and scripts automatically apply the gain map files:
and the P2_RESP files are used by:
-
ACIS 2009 Blank-sky Background File (Group F ACIS BKGRND)
The ACIS background and stowed background files files from 2009 have corrected DTCOR keyword data type. Background files from other groups have DTCOR values set to REAL8 while the Group F files had been set to INT4, causing annoying warning messages to be thrown when these blanksky files were used together with others from different groups.
The Using the ACIS "Blank-Sky" Background Files thread contains instructions on how to select a file and match it to a specific observation.
Note that the ACIS background files are not included in the main CALDB tarfile. There is a separate background files tarfile available via ciao-install or from the CALDB Download page.
HRC Grating and Imaging Data
-
LETGS 1st-order LSFPARM, HRC-S QE, and HRC-I QE
New line spread function parameter files (LSFPARM) have been introduced for HRC-S/LETG only. These new files, which are inputs to the CIAO mkgrmf tool, incorporate new encircled energy fraction tables which were generated in an effort to improve the signal-to-background ratio in extraction of LETG spectra. When used in conjunction with the optimized source and background extraction regions described in Optimized LETG/HRC-S Extraction Region and Recalibrated EEFRACs, they can result in improved S/N ratios up to ~20% for sources with low counts and high background (e.g. in long exposures, see Figures 13 and 14).
The HRC-S and HRC-I QE files have been modified to account for the changes introduced to the HRC-S/LETG grating RMFs in this release. The HRC-I and HRC-S QE calibration are linked, so the same modification applied to the HRC-S QE is applied to the HRC-I QE. There will be no overall change in the HRC/LETG effective area (EA) because changes in the RMFs are compensating changes in the HRC QE, but there are changes of up to ~12% (largest at low-energies) in EA when the HRC-S or HRC-I are used without a grating.
These CIAO tools and scripts automatically apply the HRC-S or HRC-I QE files when creating response files:
How CALDB 4.6.8 Affects Your Analysis
CALDB 4.6.8 Release Notes (release 1 July 2015)
ACIS Imaging and Grating Data
-
Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data
The new, time-dependent ACIS Gain (T_GAIN) files for February-April 2015 (Epoch 61) are introduced and updated for Epoch 60 in this release, and therefore, affects observations taken since 2014 November 01. Other observations are unaffected by these new calibration files.
Users working with ACIS data taken since 2014 November 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.
See the time-dependent ACIS T_GAIN why page for more information.
The gain corrections are at the nominal and expected levels of less than 2% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.
Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.
-
Time-dependent ACIS Gain (T_GAIN) Files Revised for 2014 ("_revA")
The systematic T_GAIN correction data has been modified for Epochs 56-59, affecting the time period of November 2013-October 2014, fixing errors in the previously generated T_GAIN correction factor values.
The errors primarily affect users during this period with gratings observations on the ACIS-S array or those performing imaging spectroscopy towards the center of ACIS-I3.
For the front-illuminated chips such as ACIS-I3, most of the areas of these chips are unaffected by these T_GAIN updates. However certain locations may be affected above around 5 keV, with changes on the order of 0.5% or less in the PHA/energy values. The back-illuminated S3 chip is not affected by this update. There is significant corrections on the back-illuminated S1 chip (up to 50%, at worst, relative to the original ΔE corrections) in the T_GAIN shifts for each of the affected observing epochs. In the worst cases, the maximum net energy correction introduced by the revised T_GAIN file is 3%; since the typical net energy correction is less than 2%, the revised net change in the correction is no more than 1% of the energy values.
Since existing issues are already found in the S1 gain functions, the newly identified errors in the correction factors is not critical to typical users. It is expected to only affect ACIS-S observations with an inserted grating, which may have the order-sorted events on the S1 chip noticeably affected, particularly those using the GRADED data mode.
How CALDB 4.6.7 Affects Your Analysis
CALDB 4.6.7 Release Notes (release 23 February 2015)
ACIS Imaging and Grating Data
-
Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data
The new, time-dependent ACIS Gain (T_GAIN) files for November 2014-January 2015 (Epoch 60) are introduced and updated for Epoch 59 in this release, and therefore, affects observations taken since 2014 August 01. Other observations are unaffected by these new calibration files.
Users working with ACIS data taken since 2014 August 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.
See the time-dependent ACIS T_GAIN why page for more information.
The gain corrections are at the nominal and expected levels of less than 2% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.
Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.
How CALDB 4.6.5 Affects Your Analysis
CALDB 4.6.5 Release Notes (release 16 December 2014)
ACIS Imaging and Grating Data
-
Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data
The new, time-dependent ACIS Gain (T_GAIN) files for Aug-Oct 2014 (Epoch 59) are introduced and updated for Epoch 58 in this release, and therefore, affects observations taken since 2014 May 01. Other observations are unaffected by these new calibration files.
Users working with ACIS data taken since 2014 May 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.
See the time-dependent ACIS T_GAIN why page for more information.
The gain corrections are at the nominal and expected levels of less than 2% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.
Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.
-
ACIS 2009 Blank-sky Background File (Group F ACIS BKGRND)
The ACIS background files from 2009 have corrected RA_PNT, DEC_PNT, and ROLL_PNT header keywords correcting errors when processing these files for analysis.
The Using the ACIS "Blank-Sky" Background Files thread contains instructions on how to select a file and match it to a specific observation.
Note that the ACIS background files are not included in the main CALDB tarfile. There is a separate background files tarfile available via ciao-install or from the CALDB Download page.
HRC-I Imaging Data
-
HRC-I Gain Maps
The latest time-dependent HRC-I Gain Map (GMAP) and the corresponding PI background spectrum—to be applied to observations after 2014 September 16, have been released.
Note that HRC-I gain maps are only useful to observers doing hardness ratio or PI spectral analysis studies with HRC-I.
As the HRC gain continues to slowly decline, the gain maps are updated annually. The change in gain correction is less than 1% on-axis and roughly 5-10% off-axis. Towards the edge of the microchannel plate, the change is up to 15%. Technical details on the gain maps are described in the gain maps section of the HRC calibration website and the in the HRC-I Gain Map Why Document.
The corresponding background spectrum applicable to the same HRC-I GMAP time period have been generated.
Users working with HRC-I data may wish run chandra_repro to reprocess the data and improve the gain calibration.
HRC-S Gratings Data
-
HRC-S QE and QEU Files
The HRC-S QE files have been updated, reducing the average QE by ~7% for energies above the carbon K-edge (284 eV).
The HRC-S QEU files have been modified to account for the significant loss in the QE in regions of the HRC-S plates that have had the most rapid, localized QE decline. The predictive QEU (up to the year 2020) account for a consistent ~0.5% per year drop in the QE across the first-order LETG spectrum since the mission launch. A somewhat steeper reduction has been observed in the zeroth-order over time.
These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:
Installation
Users should be aware of these installation items before installing CIAO 4.7. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.
Supported Platforms
-
CIAO 4.7 is supported on:
- Linux 32 bit and 64 bit (tested on CentOS 6.5 and Red Hat Enterprise 6 )
- Mac OS X 10.6 & 10.7 Intel 64 bit
- Mac OS X 10.8 & 10.9; & 10.10 Intel 64 bit
![[WARNING]](../imgs/warning.png)
End of Support: 32bit Linux and OSX 10.6.8/10.7CIAO 4.7 will be the last CIAO release that will be available for 32bit Linux and OSX 10.6.8/10.7.
Users are encouraged plan ahead for 2015 if still using any of these operating systems.
Updates to the ciao-install Installation Script
-
There is no separate OSX 10.7 (lion) binary package. Lion users will now download the precompiled 10.6 (Snow Leopard) binary package.
-
OSX Mavericks (10.9) and Yosemite (10.10) users will download the Mountain Lion pre-compiled binaries.
-
Some OSX users reported problems downloading the CIAO tar files. This was frequently tracked to the ciao-install script using an incompatible version of ftp. The script now uses /usr/bin/ftp to avoid this problem.
Remove old parameter files
-
With every new CIAO release, some parameter files are changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:
unix% rm ~/cxcds_param4/*
IPython Settings
-
Users upgrading from CIAO 4.5 or CIAO 4.6 will be prompted to update their IPython settings the first time they start ChIPS and Sherpa.
If upgrading from CIAO 4.4 or earlier, the ipython configuration files stored in the users home directory are incompatible with the new version; a new copy will be installed in a new directory: $HOME/.ipython-ciao4.5. (Note: the file is still called .ipython-ciao4.5!). If you have made customizations with earlier version of CIAO, you will need to manually migrate those changes to work with CIAO 4.7.
Tools
acis_process_events
- Updated the CALDB interface to retrieve the gradefile based on the readout mode. This is necessary to select the correct extension for CC_GRADED mode data taken since 2009-11-01.
- The calc_cc_times parameter has been removed. If the input file was taken in continous clocking mode, the time correction will automatically/always be applied.
ahelp
- Condensed output when re-indexing help files (-r flag).
- Now supports unicode
-
Python wrapper has been updated to accept a function object
as well as the function name. So now both
work.
chips> ahelp("print_window") chips> ahelp(print_window)
arfcorr
- Now supports regions saved in celestial coordinates.
- Bug fix to allow images with a 0 length axis to process.
celldetect
- Fixes a bug in the output X_ERR and Y_ERR columns when the input image does not contain any WCS info.
dmcoords
- The tool will now look for and use the DY_AVG, DZ_AVG, and DTH_AVG keywords in the input file header in lieu of recomputing the values from the aspect solution file. As a result, the asolfile is no longer a required input.
dmdiff
- Correct print format for unsigned long values.
dmellipse
- Fixes a bug when the input image contains separate linear WCS transforms on each axis.
dmextract
- Fix memory issue when using stacks of background regions that could leave too many files left open.
- Corrects the units on the CEL_FLUX and CEL_FLUX_ERR virtual columns.
dmhedit
- More error checking on bad operation strings. Operations are now case insensitive.
- Updated code that guesses datatype to use long integers rather than short integers.
- Added unsigned long, ulong, and unsigned short, ushort, datatypes.
dmimghull
- Fixes a bug when the input image contains separate linear WCS transforms on each axis.
- Clarify output is a FITS, not an ASCII, region file in parameter prompt.
dmimgreproject
- Fixes a bug when the input image contains separate linear WCS transforms on each axis.
dmimgthresh
- Now supports images with unsigned integer datatypes.
- Cleanup of excessive warnings when a problem is encountered.
- Fix a crash when the clobber parameter was set to "no" and the output file contained NULL or NaN values.
dmkeypar
- Correctly print unsinged integer keywords when echo=yes.
dmmerge
- Fixes a bug with long column names (>20 characters).
lim_sens
- No longer restricts snr_limit to be an integer value.
- Modified some parameter's mode value to removed the nonstandard forced learned behavior (nonstandard 'hl' vs 'h').
mkgrmf
- Added standard CXC header to output file.
tgdetect2
- Now allows infile names containing DM filters
wcs_update
- The tool will now update the EQSRC coordinate transform (if present, as in the detect [eg wavdetect] tools output file).
- The tool will update the RA_NOM and DEC_NOM keywords in the input file.
Parameter Files
A summary of parameter files changes is provided in this section. Refer to the Tools and Analysis Scripts sections of these release notes for complete details.
We recommend deleting all the old parameter files or renaming the parameter-file directory before running any new version of CIAO to avoid conflicts. More information is available in the FAQ.
acis_process_events
- The calc_cc_times parameter has been removed.
- If not using gradefile=CALDB, the proper extension name must be supplied: gradefile="custom_grade.fits[GRADE1]"
dmcoords
- The asolfile parameter is no longer required, since the information can be found from the DY_AVG, DZ_AVG, and DTH_AVG keywords of the input file. If users wish to still use the aspect solution file, or files, then the parameter name must be included - e.g. dmcoords evt.fits asolfile=asol1.fits.
dmhedit
- The datatype parameter has two new options - ushort and ulong - for unsigned short and long datatypes, respectively.
lim_sens
- The snr_limit parameter is now a real, float point value instead of being restricted to an integer.
- Several parameter have their mode values changed from hl (hidden plus learn) to the standard h (hidden): expfile, rbkfile, bscale, snr_limit, and const_r.
chandra_repro
- The default value for the verbose parameter is now 1, rather than 0, so users will now see more information on the steps taken by the script.
ChIPS
print_window
-
Anti-aliasing is now applied to bitmap figures - that is, those in PNG or JPEG form - as well as to the on-screen version. This is controlled by the window.smoothing preference, which has now been made the default setting; users who have previously used ChIPS may need to delete, or edit, their $HOME/.chips.rc file to see this change.
Note that this setting can only be changed when a window is created; that is, set_window can not be used to change the value.
load_colormap
- The load_colormap command now checks for invalid input values - that is, values outside the range 0 to 1 - and raises an error.
undo/redo
- Fixed undo/redo commands so that commands within a buffer are rendered as a single entity.
adjust_grid_gaps
- Fixed undo of an adjust_grid_[x|y]gap() call so that subsequent grid calculations utilize the appropriate spacing when calculating positions.
make_script
- Images are saved with a size of (nx,ny) rather than (1,nx,ny).
add_colorbar
- The error message created when trying to add a colorbar while using an invalid image identifier has been improved.
Sherpa
2D Models
- The sigmagauss2d model has been added to sherpa.
XSPEC xset routines
-
The set_xsxset and get_xsxset routines - which are Sherpa's interface to setting and retrieving the value of XSPEC XSET variables - now ignore the case of the name of the keyword.
Note that the set_xsabund, set_xsxsect, and set_xscosmo routines should be used to change the X-Spec abundances, cross-sections, and Cosmology settings.
Upgrade to XSPEC 12.8.2e
- The sherpa interface to the XSPEC models including the required parameter changes have been updated to version 12.8.2e XSPEC model library; CIAO 4.6 used version 12.8.0k.
-
The following models have been added:
-
The soft and hard maximum values for the switch parameter of the XSPEC mekal models has been increased from 1 to 2.
-
The ahelp versions of the XSPEC model descriptions have been reviewed and updated, when necessary, to match the latest version.
datastack
- Datastack is a Sherpa extension package for manipulating a stack of related datasets and simultaneously fitting a model to them. It provides stack-enabled (i.e. vectorized) versions of the key Sherpa commands used to load data, set source models, get and set parameters, fit, and plot.
Graphical User Interfaces
prism
- Fixed a problem displaying byte array columns
- Suppress crates warnings when creating histograms.
Analysis Scripts
apply_fov_limits
- The MASKFILE file - found from the eventfile - is now optional when running the tool with no fovfile parameter.
chandra_repro
- The default verbose level is now 1; it was 0 in CIAO 4.6 and earlier.
- The script has been changed to handle the removal of the calc_cc_times parameter in acis_process_events in CIAO 4.7.
find_chandra_obsid
- The script can now be given an ObsId as well as an object name or location.
add_grating_orders
- This script has been deprecated. Users should use the combine_grating_spectra script instead. It will be removed entirely from a later CIAO release.
add_grating_spectra
- This script has been deprecated. Users should use the combine_grating_spectra script instead. It will be removed entirely from a later CIAO release.
Python Modules
crates
- Memory is now correctly freed when dealing with vector columns; this is most noticeable when handling large images with an associated 2D WCS transformation.
- Improved write speed performance for some files.
- Allow column names which contain a space.
- Corrected a problem reading in some empty tables.
- A warning is now reported when a keyword containing an invalid value - such as NaN - is written out. In such cases the keyword is not written out.
- When opening a file with read-write mode, a warning is now produced if the file can not be written to: for instance, if the file permissions do not allow it, if it is a gzipped file, or if the file name contains Data Model filters. The default mode for the read_file routine is read-only, but the Python object-level interfaces - such as CrateDataset(), TABLECrate(), and IMAGECrate() - have mode='rw' by default.
- Adds capability to generate Virtual columns from component parts. Added two new functions to the HLUI: create_vector_column and create_virtual_column.
- Accommodate nonstandard ROSAT RMF files
- Correctly handle REGION subspace blocks that are present in some XMM files.
- Improved the handling of BIT and BYTE columns.
- When a column is deleted, also remove any virtual columns which are dependent upon it.
- Columns added to a table crate are given a default subspace entry.
- Axis information - including any associated transforms - can now be read from images (and higher dimension datasets) which contain axes with 1D transforms on them. An example are 4D radio data sets which contain STOKES and FREQ axes, often with an associated linear transformation.
- Corrects issue where users cannot access the named attribute for column values (eg: cr.colname) until the data portion has been accessed, such as via a call to the get_colnames() method of a crate.
- Cleanup of some error messages to remove the names of low-level I/O routines.
transform
- Issue warning about CAR projection types. The wcssubs library used by transform may produce bad results of this projection type depending on the algorithm used.
Libraries
datamodel
- Fixes problem with some TSV files (empty comments).
- Fixes crash seen in malformed header (extra TLMIN keywords for nonexistent columns)
- Avoids memory corruption caused by SIMAD TSV format files though due to formatting differences they are not directly usable by the DM.
group
- Repackaged code to support stand alone builds
header
- Add "FAIL" rule for string type value of keyword.
- Improve logic for finding merge rule with exact vs partial matches. Now finds exact match, or longest substring match from lookup table.
region
- Repackaged code to support stand alone builds.
Environment
Off-the-Shelf (OTS) Package Versions
-
The following OTS packages are included with CIAO 4.7. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.
Executables
- ds9 7.3.2
Python
- Python 2.7.6
- NumPy 1.8.1
- ipython 2.0.0
Libraries
- cfitsio v3.360
- fftw 3.3.4
- xpa 2.1.15
- jpeg 9a
- XSPEC v12.8.2e (models only)
- VTK 5.10.1
- CCfits 2.4
- fontconfig 2.8.0
- freetype 2.4.4
- gsl 1.15
- readline 6.2
- vte 0.25.1
- wcssubs 3.8.3
- The ciaover command will now report the version of the Calibration Database as well as the version of the Analysis Scripts package. In addition the full path to the Python executable is shown (even when using the precompiled CIAO version).
Documentation
-
Many pages have been updated to reflect the changes in this release. Several analysis threads are still under review.
-
There are new pages to help users doing source builds of CIAO. There are specific instructions for Linux and OSX.