CIAO 4.4.1 Release Notes
Notable changes and improvements in CIAO 4.4.1:
-
Internal leap second library updated.
-
The Sherpa modeling and fitting package has improved file handling for passing files along to the PyCRATES module. XSpec models have been updated to use the XSpec 12.7.1 models library and seven additive XSpec models have been added.
- How CALDB 4.5.3 Affects Your Analysis
- How CALDB 4.5.2 Affects Your Analysis
- How CALDB 4.5.1 Affects Your Analysis
- How CALDB 4.5.0 Affects Your Analysis
- How CALDB 4.4.10 Affects Your Analysis
- How CALDB 4.4.8 Affects Your Analysis
- How CIAO 4.4 and CALDB 4.4.7 Affect Your Analysis
- Installation
- Tools
- Parameter Files
- Data Model
- ChIPS
- Sherpa
- GUIs (Graphical User Interfaces)
- Analysis Scripts
- Libraries
- Environment
- Documentation
How CALDB 4.5.3 Affects Your Analysis
CALDB 4.5.3 Release Notes (release October 2012)
ACIS Imaging Data and Grating Data
-
Fix to v. N0002 Group E ACIS Blank-sky Background Files
The recently upgraded version N0002 Group E ACIS blank-sky background files, released in CalDB version 4.5.2 to fix a header error in those files, have been corrected again in the current CalDB 4.5.3 release to resolve another error: the GTI extensions were absent from the upgraded files, and instead a second redundant EVENTS list was included. This caused extraneous listings of the files in the ACIS CalDB index, and caused the acis_bkgrnd_lookup script to fail.
The N0003 files listed below have been generated to correct both the header errors present in the N0001 version of the files (the original N0002 upgrade), and the block structure error of the N0002 version described above.
acis2sD2005-09-01bkgrndN0003.fits acis3sD2005-09-01bkgrndN0003.fits acis5sD2005-09-01bkgrndN0003.fits acis6sD2005-09-01bkgrndN0003.fits acis7sD2005-09-01bkgrndN0003.fits acis8sD2005-09-01bkgrndN0003.fits acis0iD2005-09-01bkgrnd_ctiN0003.fits acis1iD2005-09-01bkgrnd_ctiN0003.fits acis2iD2005-09-01bkgrnd_ctiN0003.fits acis2sD2005-09-01bkgrnd_ctiN0003.fits acis3iD2005-09-01bkgrnd_ctiN0003.fits acis3sD2005-09-01bkgrnd_ctiN0003.fits acis5sD2005-09-01bkgrnd_ctiN0003.fits acis6iD2005-09-01bkgrnd_ctiN0003.fits acis6sD2005-09-01bkgrnd_ctiN0003.fits acis7iD2005-09-01bkgrnd_ctiN0003.fits acis7sD2005-09-01bkgrnd_ctiN0003.fits acis8sD2005-09-01bkgrnd_ctiN0003.fits
Users are encouraged to upgrade their CalDB to version 4.5.3 to access the re-corrected N0003 background files, and avoid acis_bkgrnd_lookup errors with the N0002 version of the files.
How CALDB 4.5.2 Affects Your Analysis
CALDB 4.5.2 Release Notes (release September 2012)
ACIS Imaging Data and Grating Data
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for May - Jul 2012 (Epoch 50):
acisD2012-02-01t_gainN0005.fits acisD2012-02-01t_gainN0006.fits acisD2012-05-01t_gain_biN0002.fits acisD2012-05-01t_gainN0002.fits
The files are applicable to
-120 C focal plane temperature only.Users working with ACIS data taken since 01 February 2012 may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
-
ACIS Blank-sky Background File (Group E ACIS BKGRND) Upgrades
The Group E ACIS BKGRND blank-sky background files which are effective for 2005-09-01 through the present, were updated to remove an errant virtual coordinate transform column (pha_COORD) and related EVENTS header keywords (LATP9 and EQUI9). The unnecessary pha_COORD column occurred as a result of the improper inclusion of the the LATP9 and EQUI9 keywords in the BKGRND file headers, and this caused certain CIAO tools and scripts to generate problematic files when run with these background files as input (e.g., dmcopy and reproject_events), or fail completely (e.g., specextract).
The offending keywords were removed from the headers of the 18 affected background files, and the file version number was upgraded to N0002; the new filenames are listed below.
NOTICE as of 10/04/2012: These BKGRND files must be corrected again. A fix is in the works to replace them within about one week.
acis0iD2005-09-01bkgrnd_ctiN0002.fits acis1iD2005-09-01bkgrnd_ctiN0002.fits acis2iD2005-09-01bkgrnd_ctiN0002.fits acis2sD2005-09-01bkgrnd_ctiN0002.fits acis2sD2005-09-01bkgrndN0002.fits acis3iD2005-09-01bkgrnd_ctiN0002.fits acis3sD2005-09-01bkgrnd_ctiN0002.fits acis3sD2005-09-01bkgrndN0002.fits acis5sD2005-09-01bkgrnd_ctiN0002.fits acis5sD2005-09-01bkgrndN0002.fits acis6iD2005-09-01bkgrnd_ctiN0002.fits acis6sD2005-09-01bkgrnd_ctiN0002.fits acis6sD2005-09-01bkgrndN0002.fits acis7iD2005-09-01bkgrnd_ctiN0002.fits acis7sD2005-09-01bkgrnd_ctiN0002.fits acis7sD2005-09-01bkgrndN0002.fits acis8sD2005-09-01bkgrnd_ctiN0002.fits acis8sD2005-09-01bkgrndN0002.fits
Users are encouraged to upgrade their CalDB to version 4.5.2 in order to access the corrected background files and avoid unexpected CIAO tool or script failures with the pre-CalDB 4.5.2 ACIS blank-sky background files.
How CALDB 4.5.1 Affects Your Analysis
CALDB 4.5.1 Release Notes (release 24 July 2012)
HRC Imaging and Grating Data
-
HRC-S T_GMAP Upgrades
On 29 March 2012, the High Voltage (HV) settings of the HRC-S microchannel plates were increased, causing a corresponding rise in the detector gain, and a required update to the HRC-S time-dependent gain map (T_GMAP) calibration. As a result, all observations taken after 01 January 2012 must be reprocessed using the new calibration T_GMAP files available in CalDB 4.5.1.
There are two new T_GMAP files in CalDB 4.5.1, listed below. The first file applies to HRC-S data taken before 29 March 2012, and the second to data taken since then .
hrcsD1999-07-22t_gmapN0002.fits hrcsD2012-03-29t_gmapN0002.fitsUsers with HRC-S observations taken in 2012 must download CalDB v4.5.1 and then reprocess their data in order to apply the updated T_GMAP calibration. Data may be reprocessed using the chandra_repro script in CIAO 4.4, which will automatically apply the newest calibration. Users with pre-2012 data do not need to take any action.
Note: The following observations must be reprocessed manually, not with chandra_repro, by following the manual procedure in the CIAO thread Reprocessing Data to Create a New Level=2 Event File.
OBS_ID HV_set OBJECT GRATING START_DATE ------------------------------------------------------------- 14324 OLD Mkn421 LETG 2012-07-04 04:09:45 14396 OLD Mkn421 LETG 2012-07-03 19:09:24 14397 OLD Mkn421 LETG 2012-07-04 13:10:06 14238 NEW HZ43 LETG 2012-03-18 05:16:00
Before reprocessing these observations with hrc_process_events, the 'gainfile' parameter must be set to the appropriate CalDB path and name, like shown below.
% pset hrc_process_events gainfile ${CALDB}/data/chandra/hrc/t_gmap/hrcsDyyyy-mm-ddt_gmapN0002.fitsHere, the yyyy-mm-dd date depends on the HV setting used for the given ObsID. For ObsID 14238, the date should be 2012-03-29 since it used the new HV setting, while for the other three cases with the old HV setting, 1999-07-22 should be used.
How CALDB 4.5.0 Affects Your Analysis
CALDB 4.5.0 Release Notes (release 27 June 2012)
HRC Imaging and Grating Data
-
HRC-S and HRC-I GAPLOOKUP (degap) Upgrades
The HRC Calibration Team and the HRC Instrument Scientist have developed improved DEGAP solutions for the CALDB 4.5.0 release, currently formatted in the GAPLOOKUP tables in CalDB, for both the HRC-I and HRC-S detectors.
HRC-S
It is recommended that all archival HRC-S data be reprocessed with the new GAPLOOKUP tables, including the more recent HRC-S data that have already been reprocessed with the previous GAPLOOKUP table.
HRC-I
For the HRC-I, the corrections are more modest, with improvement basically in the pre-existing bias toward overestimated degap corrections near the gap boundaries. Hence, users are not likely to see significant changes using the new HRC-I GAPLOOKUP unless they are looking for very small (0.1 arcsecond) deviations, particularly for time-dependent data. In this latter case, users should reprocess their data with the new HRC-I GAPLOOKUP table.
Background InformationDue to the aimpoint drift of the telescope over time, recent observations using the HRC-S (grating or non-grating) may have a small fraction of source (or 0th order) events that fall on the CRSV=100 tap (read here for a short explanation about taps and the degapping process). The degap map available prior to the current CalDB 4.5.0 release - which converts event positions from RAW to CHIP (which are then converted to DET and TDET, and ultimately to SKY) - has residual errors for that tap, leading to asymmetry and broadening of the PSF. It does not affect the positions of events dispersed by the LETG, although the wavelength calibration may be slightly affected. Refer to the calibration page HRC-S Event Position Errors Near the Aim Point for the technical details of this problem.
It is recommended that users with HRC-S observations since the beginning of 2010 check their events using a procedure such as the one shown below, and reprocess their observations if the CRSV=100 events are a significant fraction. Users may determine the fraction of events appearing on CRSV=100 using the following Data Model commands in CIAO:
% dmlist evt1file counts % dmlist evt1file"[CRSV=100]" counts
where the evt1file is filtered for the source(s) of interest, or for the zeroth order image of an LETGS observation. If the fraction is significant, as a general rule, reprocessing using the new HRC-S GAPLOOKUP will be beneficial.
ACIS Imaging and Grating Data
-
Change of Upper Limit of ACIS FP_TEMP Boundary Condition Range
The allowable range of the ACIS focal plane temperature has been changed from 151.16-159.16K to 151.16-164.16K for all relevant ACIS calibration files in CalDB. Only one proprietary observation is affected by this change. See the CalDB 4.5.0 Release notes for full details. The vast majority of users are unaffected by this change.
-
Calibration Validity End Date Correction
The calibration validity end dates were reset in the T_GAIN files associated with 38 ACIS ObsIDs, to address a problem with these files described here. Users analyzing any of the affected observations, who have not recently reprocessed the data, should do so using CalDB 4.5.0 in order for the ACIS time-dependent gain correction to be applied during manual reprocessing with chandra_repro.
The following calibration files also required a validity end date correction to address this issue. Users working with any of the ObsIDs affected by these changes, listed below, should also reprocess their data.
- ACIS CTI file: ObsID 2532
- HRC-I GMAP and PIBGSPEC files: ObsIDs 1364, 4296, 9643, 11909
- HRC-S QEU file: ObsID 3144
How CALDB 4.4.10 Affects Your Analysis
CALDB 4.4.10 Release Notes (release 30 May 2012)
ACIS Imaging Data and Grating Data
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for Feb - Apr 2012 (Epoch 49):
acisD2011-11-01t_gainN0006.fits acisD2011-11-01t_gainN0005.fits acisD2012-02-01t_gain_biN0002.fits acisD2012-02-01t_gainN0002.fits
The files are applicable to
-120 C focal plane temperature only.Users working with ACIS data taken during 01 Nov 2011 through the present may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
-
ACIS QE Contamination Model vN0007
The ACIS QE contamination model has been upgraded to vN0007. This version of the file contains a ~15% change in the optical depth at 700 eV for observations in 2012 as compared with those in early 2009.
These CIAO response tools automatically apply the contamination file when creating ACIS response files:
As well as the scripts which use them:
- specextract (calls mkwarf and mkarf)
- fullgarf (calls mkgarf)
- fluximage (calls mkinstmap)
- merge_all
How CALDB 4.4.8 Affects Your Analysis
CALDB 4.4.8 Release Notes (release 23 Feb 2012)
ACIS Imaging Data and Grating Data
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 Nov 2011 - 31 January 2012 (Epoch 48):
acisD2011-08-01t_gainN0006.fits acisD2011-08-01t_gainN0005.fits acisD2011-11-01t_gain_biN0002.fits acisD2011-11-01t_gainN0002.fitsThe files are applicable to
-120 C focal plane temperature only. The acisD2011-11-01t_gain_biN0002.fits file is the new default processing file in the CALDB.Users working with ACIS data taken during 01 Aug 2011 through the present may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
How CIAO 4.4 and CALDB 4.4.7 Affect Your Analysis
Some tool changes and calibration files released in CIAO 4.4 and CALDB 4.4.7 affect analyses in progress. This section of the release notes describes those changes and which tasks are affected.
ACIS Imaging Data
-
Creating ACIS Bad Pixel Files
The new acis_find_afterglow identifies cosmic-ray afterglows and hot pixels for TIMED mode observations. This tool should be used in place of "acis_run_hotpix" for all analyses.
The chandra_repro script has been updated to use the new tool.
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 August 2011 - 31 October 2011 (Epoch 47):
acisD2011-05-01t_gainN0006.fits acisD2011-05-01t_gainN0005.fits acisD2011-08-01t_gain_biN0002.fits acisD2011-08-01t_gainN0002.fits
The files are applicable to -120 C focal plane temperature only. The acisD2011-08-01t_gain_biN0002.fits file is the new default processing file in the CALDB.
Users working with ACIS data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
ACIS Grating Data
-
Creating ACIS Bad Pixel Files
The new acis_find_afterglow identifies cosmic-ray afterglows and hot pixels for TIMED mode observations. This tool should be used in place of "acis_run_hotpix" for all analyses.
The chandra_repro script has been updated to use the new tool.
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 August 2011 - 31 October 2011 (Epoch 47):
acisD2011-05-01t_gainN0006.fits acisD2011-05-01t_gainN0005.fits acisD2011-08-01t_gain_biN0002.fits acisD2011-08-01t_gainN0002.fits
The files are applicable to -120 C focal plane temperature only. The acisD2011-08-01t_gain_biN0002.fits file is the new default processing file in the CALDB.
Users working with ACIS data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
-
HETG Grating Efficiency (version N0007)
This file was also released in the CALDB 4.4.6.1 patch on 02 December 2011.
The cross-calibration of the first-order HEG and MEG portions of the HETG in the grating efficiency file (GREFF) has been adjusted via analysis of 30 Blazar observations. The new file (N0007) will be used by default when creating HETG response files (ARFs). The CIAO tools and scripts which use this file are:
For instructions on how to continue using the vN0006 file, read the HETG Grating Efficiency why topic.
HRC-I Imaging Data
-
HRC-I Gain Maps
The latest time-dependent gain maps (GMAP) for the HRC-I have been released:
hrciD2010-09-25sampgainN0002.fits # 2010-09-25T12:00:00 to 2011-09-19T12:00:00 hrciD2011-09-19sampgainN0001.fits # 2011-09-19T12:00:00 (open-ended range)
The gain maps are updated annually to maintain 2% consistency in the PI. Technical details on the gain maps are described in the gain maps section of the HRC calibration website.
Users working with HRC-I data should run chandra_repro to reprocess the data and improve the gain calibration in their data.
-
HRC-I background PI spectra
The HRC calibration team has released a set of background spectra from the HRC-I. These spectra describe the particle background of the detector and vary slowly with time which means that they can be used to significantly improve the signal-to-noise of HRC-I imaging data and may be useful for point source analysis (e.g. fields with many point sources).
The HRC-I background spectra files are included in the main CALDB tarfile.
The 2010 HRC-I background PI spectrum file was remade with the new gain map (also released in CALDB 4.4.7), and the 2011 file has been added:
unix% ls -1 $CALDB/data/chandra/hrc/pibgspec/ hrciD1999-10-04pibgspecN0001.fits hrciD2000-12-12pibgspecN0001.fits hrciD2002-01-26pibgspecN0001.fits hrciD2003-02-22pibgspecN0001.fits hrciD2004-11-25pibgspecN0001.fits hrciD2005-10-17pibgspecN0001.fits hrciD2006-09-20pibgspecN0001.fits hrciD2007-09-17pibgspecN0001.fits hrciD2008-09-07pibgspecN0001.fits hrciD2009-09-24pibgspecN0001.fits hrciD2010-09-25pibgspecN0001.fits hrciD2011-09-19pibgspecN0001.fits
For instructions on using the background event files, follow the The HRC-I Background Spectra Files thread.
HRC-S Imaging and Grating Data
-
QE Upgraded to vN0011
A new mean quantum efficiency (QE) file has been released for HRC-S (version N0011).
$CALDB/data/chandra/hrc/qe/hrcsD1999-07-22qeN0011.fits
Following in-flight calibration of the low energy QE, models of flux vs energy and parameter estimates for the two main calibration standards (HZ43 and Sirius B) underwent significant revisions. The latest non-LTE pure hydrogen model atmospheres computed by the Tubingen group have been used to devise a new low energy calibration.
These figures show the residuals between LETG+HRC-S observations and the fiducial model before and after modifications to the HRC-S quantum efficiency.
- LETGS QE pre-correction (vN0010): The fit and residuals to co-added spectra for HZ43 taken since launch, with the previous HRC-S QE (version N0010) in CalDB.
- LETGS QE corrected (vN0011): The fit and residuals to co-added spectra for HZ43 taken since launch, with the new HRC-S QE (version N0011).
The new file will be used by default when creating exposure maps or grating ARFs in CIAO.
Installation
Users should be aware of these installation items before installing CIAO 4.4.1. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.
Supported Platforms
-
CIAO 4.4.1 is supported on:
- Linux 32 bit and 64 bit (tested on CentOS 5.6 and Fedora Core 11)
- Mac OS X 10.6 Intel 32 bit and 64 bit
- Mac OS X 10.7 Intel 32 bit and 64 bit
The following platforms are no longer supported:
- Mac OS X 10.5
- Solaris 10
Updates to the ciao-install Installation Script
-
The default installation directory is $HOME instead of /usr/local.
-
The script will not run if called with sudo, i.e. "sudo bash ciao-install". If ciao-install encounters a permissions problem, it prints instructions on what to fix in order to install CIAO and gives the user the chance to specify a different installation directory.
-
An option was added to check for the required system libraries, e.g. verify that libGL is installed if the chips segment is being installed.
Remove old parameter files
-
With every new CIAO release, some parameter files have changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:
unix% rm ~/cxcds_param4/*
Remove the fontconfig cache
-
The GUI applications (e.g. prism, ChIPS, ds9) used in CIAO may not run correctly if there are incompatible .fontconfig files cached on the user's system. To remove the .fontconfig cache:
unix% rm ~/.fontconfig/*
Tools
acis_build_badpix
-
If a bad pixel or column from the CALDB input file is excluded by the usrfile, the 3x3 island surrounding it is cleared. if badpixel/column in caldib input file is excluded via
acis_find_afterglow
-
acis_find_afterglow identifies cosmic-ray afterglows and hot pixels for TIMED mode observations. This tool should be used in place of "acis_run_hotpix" for all analyses.
acis_process_events
-
Added a check that the bad pixel and event files are from the same ObsID and have the same OBI number. A warning is printed if they don't match.
-
Issue a warning if the bad pixel file is not in time order.
acis_streak_map
-
The tool has been rewritten in Python. The functionality of the tool is unchanged.
acis_run_hotpix
-
Additional acis_build_badpix parameters may be set from the acis_run_hotpix parameter file: this section remove when the tool was deprecated
aconvolve
-
Two new kernel options have been added: hanning and racos.
-
Bug fix: if the axis length was not a power of two, the padding was doubled to the next power of two. The extra padding would create much larger images and require large amounts of memory.
addresp
-
Three new parameters have been added:
-
phafile: the input list of PHA files which are used to calculate the EXPOSURE and ONTIME header values for the output ARF file.
The combined EXPOSURE written to the output files comes from the exposures of the input PHA files, not the ARF input as was done in CIAO 4.3.
-
method: if method=avg, the average of the PHA exposure times is written to the header of the combined ARF. If method=sum, the sum of the PHA exposure times is used.
The addresp tool may be used with method=avg to combine grating ARFs, creating the same output the dmarfadd tool.
-
type: setting type=rsp means that the tool will combine the ARF and RMF into an RSP file instead of making two separate files.
-
-
If the input ARF only contains NaNs, the tool will exit with an error.
aprates
-
Bug fix: the bounds for a value were incorrect when Poisson statistics were used and either the 'n' or 'm' value was greater than approximately 300.
-
New parameter: clobber allows an existing output file to be overwritten.
arfcorr
-
The tool has been rewritten in Python. The functionality of the tool is unchanged.
asphist
-
The tool has been updated to work correctly on reprojected data. Previously, asphist assumed that _NOM and _PNT header values were the same when calculating the offsets, which is not the case for reprojected data.
The updated algorithm uses the full tangent plane offset computations and to correct angle between pointing and North. Both of these affect data that have been reprojected to different tangent points and/or multi-obi observations with significantly different pointings.
celldetect
-
Improvements to recursive blocking mode when the input image is non-square.
-
Bug fix: atan2(~0,~0) gave slightly different results on different platforms.
dither_region
-
New parameter: wcsfile is used when the input region file is in celestial format.
-
Bug fix: TOTLAREA keyword was not being written to the file header. The TOTLAREA keyword records the total area of the region in units of sky pixels.
dmcoords
-
A tolerance check was added to the RA_NOM and DEC_NOM comparisons to improve the accuracy of the WCS comparison.
-
The limits of the chip_id parameter are explicitly included in the parameter file (min=0, max=9).
dmextract
-
Added a check for floating-point NaNs values in images. The output COUNTS column would be NaN if the region enclosed a NaN-value pixel.
-
Fixed a memory problem when working with exposure maps ("DM Error: No kernel found for dataset <exposure map>"). The exposure map was opened once per region, causing dmextract to run out of memory.
-
The output bg_exposure values were incorrect because the background exposure file (bgexp) wasn't being applied in lightcurve mode. Added a warning when exp or bkgexp are specified but missing.
-
dmextract got the exposure wrong when using a DTF file if the time bins were smaller than the resolution of the DTF file (~2 sec).
-
Suppress the warning when no bin column is specified.
dmfilth
-
Bug fix: the random seed is only initialized once. The tool was using the same value for each source/background pair in the stack, which makes the filled values highly correlated (if not identical).
dmhedit
-
An error message is printed if the "move after" rule is used, since that functionality is broken.
-
The filelist parameter can read a file from stdin, i.e. filelist=-.
dmimgfilt
-
Improved error checking for missing input files.
dmimglasso
-
New parameter: maxdepth sets the maximum recursion depth. A warning is issued if the maxdepth is reached.
dmimghull
-
dmimghull creates a convex hull around pixels in image.
dmimgpick
-
Clarified the logic when using method=closest and point is equidistant from pixel centers. This bug was leading to build-specific behavior.
dmimgpm
-
Bug fix: the tool would SegV if the region contained zero pixels.
dmimgthresh
-
Bug fix: the tool checks for integer BLANK values when cut=INDEF.
dmkeypar
-
Bug fix: dmkeypar would return bad data if the filter selected zero rows.
dmlist
-
Using the "raw" mode will suppress the mapping from null value to 'NULL' string, so that the actual value is seen.
dmsort
-
Bug fix: using "copyall=no" with the "opt=text/simple" format.
The text/simple ASCII format does not support IMAGE blocks. When sorting a table with the default value of the dmsort copyall parameter (yes), the tool attempted to copy the other blocks from the input file, including the NULL PRIMARY image.
unix% dmsort simple.fits "simple.sort.dat[opt kernel=text/simple]" keys=y # DMSORT (CIAO): Error adding full header.
dmstat
-
Bug fix: the absolute value was being taken of the coordinate transform, meaning a negative Dec could become postive.
dmtabfilt
-
dmtabfilt applies non-linear filters to columns in a table.
eff2evt
-
Bug fix: the FLUX_LO/HI/WEIGHTED columns should only be created in the output file when the rmfimg parameter is set.
get_src_region
-
Improved behavior for images with linear WCS.
hrc_dtfstats
-
The DTF is computed as a simple mean instead of a variance-weighted mean to address a bug in computing average deadtime corrections for ObsIDs with telemetry saturation, and to make the program consistent with the HRC deadtime correction algorithm used in dmextract.
imgmoment
-
imgmoment computes up to the second moment of an image.
lim_sens
-
The tool checks the units of the psffile and converts them to logical pixels, if necessary. An additional check has been added to confirm that the infile and psffile files are congruent.
mean_energy_map
-
mean_energy_map creates image whose pixels are the exposure-weighted mean energy of the pixel.
mkacisrmf
-
propagate verbose setting to caldb
-
If the upper limit of the energy specification was greater than 12 keV, the tool would segmentation fault.
mkpsf
-
CIAO 4.4 is the last software release that will include the mkpsf tool. It will be removed from the next version of CIAO.
Users can run ChaRT, SAOTrace, and/or MARX to simulate the PSF for their exact source location and spectrum which will yield much better results than the current set of PSF "hypercubes."
mkpsfmap
-
The tool has been updated to work correctly on reprojected data. Previously, mkpsfmap assumed that _NOM and _PNT header values were the same when calculating the offsets, which is not the case for reprojected data.
mkrmf
-
The verbosity setting is propagated to the CALDB lookups.
mkwarf
-
Bug fix: there would be NaN values in the ARF if there are time header keywords with a 0 value. mkwarf has been updated to handle the case where there is flux in the WMAP but zero time for the chip (i.e. it has been filtered out of the file).
pfold
-
pfold folds an event file into a grid of period bins with a fixed number of phase bins.
proposal tools
-
Several bug fixes were applied to the "From Galactic" option. When using the "From Galactic" option in Cycle 13:
- incorrect galactic coordinates were echoed to the output.
- with the NRAO dataset, an incorrect NH was calculated.
- (with the Bell dataset, the correct NH wa returned.)
These issues are fixed in CIAO 4.4/Cycle 14.
-
The prop_dates tool allows numeric months, i.e. "Dec" and "12" are both valid inputs.
reproject_aspect
-
The tool has been rewritten in Python. The functionality of the tool is unchanged.
reproject_events
-
The units and comments for the SIM_X/Y/Z keywords are copied to the output file. This means that the comment and unit fields will be blank if they are blank in the input.
roi
-
roi generates source and background regions from a source list.
sky2tdet
-
New parameter: bin increases the blocking factor of the WMAP in chip coordinates.
The default value of bin=1 will be sufficient for most users. When making a weighted ARF for a large region, on order of a full chip (or larger), mkwarf may run out of memory; in this case, the bin value should be increased to create a coarser WMAP.
-
The tool retains the data subspace and adds a history entry to the header of the output file.
-
Additional error checks have been added to the input files to confirm that the necessary columns are present and that the input and output images are not empty (zero counts).
skyfov
-
If the mask file indicates that the chip was entirely inactive for the observation, the chip is excluded from the output file.
vtpdetect
-
All computations were changed from floats to doubles in order to prevent a loss of precision when computing the exposure-weighted areas.
wavdetect
-
The wavdetect tool takes an observation-specific PSF map file - created by mkpsfmap and set in the new psffile parameter - instead of using the Chandra PSF table data or the tool's internal parameterization of the ROSAT PSF.
To run mkpsfmap, use the eband and eenergy values from your CIAO 4.3 wavdetect command and your input image:
unix% mkpsfmap infile=<image> outfile=psfmap.fits energy=<eband> ecf=<eenergy>
-
The old PSF parameters have been removed: psftable, xoffset, yoffset, eband, and eenergy.
-
increased limits on number of wavelet scales to 127 (from 15)
wrecon
-
The wavdetect tool takes an observation-specific PSF map file - created by mkpsfmap and set in the new psffile parameter - instead of using the Chandra PSF table data or the tool's internal parameterization of the ROSAT PSF.
To run mkpsfmap, use the eband and eenergy values from your CIAO 4.3 wavdetect command and your input image:
unix% mkpsfmap infile=<image> outfile=psfmap.fits energy=<eband> ecf=<eenergy>
-
The old PSF parameters have been removed: psftable, xoffset, yoffset, eband, and eenergy.
-
increased limits on number of wavelet scales to 127 (from 15)
wtransform
-
increased limits on number of wavelet scales to 127 (from 15)
Parameter Files
A summary of parameter files changes is provided in this section. Refer to the Tools and Analysis Scripts sections of these release notes for complete details.
We recommend deleting all the old parameter files or renaming the parameter-file directory before running any new version of CIAO to avoid conflicts. More information is available in the FAQ.
acis_run_hotpix
-
Additional acis_build_badpix parameters may be set from the acis_run_hotpix parameter file: this section remove when the tool was deprecated
addresp
aprates
-
New parameter: clobber allows an existing output file to be overwritten.
combine_spectra
-
New parameter: method specifies whether the script should sum or average the PHA exposure times for the header of the combined spectrum and ARF files.
dither_region
-
New parameter: wcsfile is used when the input region file is in celestial format.
dmimglasso
-
New parameter: maxdepth sets the maximum recursion depth. A warning is issued if the maxdepth is reached.
sky2tdet
-
New parameter: bin increases the blocking factor of the WMAP in chip coordinates.
The default value of bin=1 will be sufficient for most users. When making a weighted ARF for a large region, on order of a full chip (or larger), mkwarf may run out of memory; in this case, the bin value should be increased to create a coarser WMAP.
specextract
-
New parameter: binarfwmap specifies the binning of the sky2tdet WMAP, which is used as input to mkwarf.
wavdetect
-
The wavdetect tool takes an observation-specific PSF map file, created by mkpsfmap and set in the new psffile parameter.
-
The old PSF parameters have been removed: psftable, xoffset, yoffset, eband, and eenergy.
Data Model
PyCrates
-
The CIAO 4.4 release includes a pycrates module which has been reworked and rewritten entirely in Python. This module replaces previous python-wrapped C++ module and library.
The high level user interface (HLUI) and Crates API remain essentially the same, but we have added several new features and capabilities:
-
addition of the CrateDataset which allows users to read and write files with multiple extensions
-
addition of 'update' mode allowing users to update a dataset or extension in place.
-
use lazy initialization to save time and memory resources by reading header and data from a file only when access is needed.
-
named attributes are provided for easy access to regular (non-virtual) columns. For example, access the time column as "tab.time" and the x column of a sky(x,y) vector as "tab.sky.x".
-
Definition of a specialized PHACrateDataset for handling of OGIP compliant PHA files, both Type:I and Type:II.
-
-
Removed support for the ARF routines (read_arf, write_arf, and is_arf), as these are now treated as normal tables.
-
Caveats/Issues:
-
Users may get unexpected results if the same file is opened to more than one variable. Any open done in read-only mode will result in subsequent opens of that file to be invalid, but does not raise an error. Also note that doing so in update mode will open more than one path to modifying file content.
-
PyCrates file handles are not properly closed and garbage-collected when the references to a Crate are freed. As a work-around, the user may:
- explicitly call the CrateDataset.__del__() function to close all file handles.
- or use the CrateDatsaet.snip() method, which breaks the underlying circular dependencies that are preventing normal garbage collection.
Either option should be used only when the user is ready for the objects to be garbage collected.
-
ASCII Kernel
-
Bug fix: the ASCII kernel could not access the last column name or value of a file if it includes the carriage return character (\r). The "\r" is included as part of the text/numerical value, which broke the parsing.
Additional Enhancements & Bug Fixes
-
Using the "raw" mode option will suppress the mapping from null value to 'NULL' string, so that the actual value is seen.
-
When using weighted binning syntax, transfer the units of the weighting column as BUNIT of the image.
-
The syntax [bin x=#10] may be used as shorthand for [bin x=::#10].
-
Bug fix: correct a problem when parsing filter syntax which contains two adjacent separators (commas) or the filter starts with a comma, e.g. file.fits"[ccd_id=7,energy=10000:12000][,sky=region(fov1.fits[ccd_id=7])]".
-
Bug fix: if a filter that contains parentheses is applied to a file, subsequent filtering will not read the subspace correctly, resulting in incorrect GTI information.
-
Bug fix: if an ASCII filter does not contain a newline, the software reports that the filter cannot be read.
ChIPS
ChIPS GUI
-
New in CIAO 4.4, the ChIPS graphical user interface (GUI) allows users to adjust the layout of their plots and make edits to many of the plot properties. A Python terminal is built into the GUI, allowing access to ChIPS and other Python commands. The GUI is available to any process that creates a ChIPS plot, such as Sherpa and Prism.
The GUI is launched by right clicking on an existing ChIPS window and selecting "Show GUI" or by running the show_gui command from within a ChIPS session.
Features include:
- the ability to edit the properties of an object, such as the symbol style of a curve or the font used for an axis label,
- zoom, pan, and select objects in the plot,
- update the plot ranges,
- move plot areas and annotations,
- resize plot areas, labels, and points,
- and an object tree which describes existing plot objects.
Futher information is available from the built-in help menu in the GUI and via "ahelp chipsgui".
Plot Translucency in PDFs
-
The PDF output created by ChIPS supports the transparency and alpha settings.
The Postscript (PS and EPS) output formats do not support the alpha settings of images or the opacity settings of region or histograms. The attribute values are treated as 1 when the objects are displayed in these formats.
Manipulating Plot Objects
-
A new function, highlight_object, can be used identify objects in a window by causing them to blink for approximately three seconds.
-
Window layer grid lines are available to help layout a user's visualization. These lines are for visual cues only and will not appear in printouts. Manipulating the grid lines is available via the GUI.
Plot Labels and Text Formatting
-
Improvements have been made in the parsing of labels. Parsing would fail when escaping underscores, and the superscript option was not being applied if it was at the beginning of the label.
-
In addition, circumflex support for vowels was changed to use math/formula syntax: \hat{a} to disambiguate a circumflex character vs a superscript character. The carat character (e.g. ^{O} or ^a) is used for superscripts. For consistency, \acute, \grave, \tilde, and \ddot support was also added (but the text mode formats \`, \', \~, \", and \c are still supported).
Additional Enhancements & Bug Fixes
-
The get_server_cwd command returns the working directory that is used by the ChIPS server and the GUI print dialog.
-
The numeric accuracy of colorbar labels has been improved to have a nicer display. Due to very small numercial differences in the tick label values, they were not always well-formatted.
The colorbar title now updates its location when the colorbar width is changed.
-
Bug fix: make_script would failed is the current plot was set to "all". The edge.style for regions was also not written to the output script.
-
Bug fix: pick_limits(X_AXIS) would also change the Y axis values.
-
Bug fix: the split command will create a frame if one doesn't exist, using the specified window.
-
Bug fix: set_cascading_property would not work on the properties of annotation objects - including labels, lines, points, regions, and colorbars - when the object was used directly.
-
Bug fix: the add routines for annotations failed when run with a dictionary of attributes instead of a list or a string.
-
Bug fix: hex were values are not being accepted as valid colors in attribute lists when used with any of the add commands.
-
Bug fix: the plot title position (title.ypos and title.xpos) was not saved correctly to the state file.
Sherpa
Bayesian Analysis with pyBLoCXS
-
The pyBLoCXS module is now integrated into Sherpa. pyBLoCXS is a sophisticated Markov chain Monte Carlo based algorithm designed to carry out Bayesian Low-Count X-ray Spectral (BLoCXS) analysis in the Sherpa environment. (For more information on BLoCXS, see van Dyk, D.A., Connors, A., Kashyap, V.L., & Siemiginowska, A. 2001, Ap.J., 548, 224, "Analysis of Energy Spectra with Low Photon Counts via Bayesian Posterior Simulation")
The code is a Python extension to Sherpa that explores parameter space at a suspected minimum using a predefined Sherpa model to high-energy X-ray spectral data.
Visualization of Statistical Distributions from Simulations
-
There is a new PPP statistic function, plot_pvalue(), to compute the likelihood ratios of the null and the alternative models. This function can help users determine if the fitted model is statistically significant compared to the null model.
The get_pvalue_results() command returns the likelihood ratio test results computed by plot_pvalue().
The get_pvalue_plot() command accesses the histogram plot of the likelihood ratios comparing fits of the null model to fits of the alternative model using faked data with Poisson noise.
-
There are new statistics plot functions associated with the pyBLoCXS module to plot statistical distribution functions.
- plot_pdf()
- plot_cdf()
- plot_trace()
- plot_scatter()
- get_pdf_plot()
- get_cdf_plot()
- get_trace_plot()
- get_scatter_plot()
Fitting Template Models to 1D Data
-
Sherpa supports a new template model (an extension of table model). Sherpa can now read in a collection of templates from a directory full of template files, and can compare a data set to all the templates in that collection. Sherpa finds the template that best matches the data, and reports back the parameter values associated with that template.
The files that contain template models can be read into Sherpa via the new load_template_model() function, which has a similar signature to the load_table_model() function. The function loads templates listed by a table-of-contents file (templatefile) into a Sherpa template model (modelname).
The gridsearch method will select the best template model by trying all of the templates.
-
There is new support to overplot template models with plot_source_component().
-
Table and template models now support linear, nearest-neighbor, and polynomial interpolation. (Interpolation is used by both table and template models to match the data grid to the model grid, which must match before the fit statistics can be calculated for fitting.)
Optimization Methods
-
The Monte Carlo optimization method has been significantly improved, and can now find more reasonable parameter values in cases where it formerly came up against parameter limits.
-
The LMDIF optimization method has also been updated to avoid getting "stuck" at the parameter space boundaries for certain cases.
-
There is a new grid searching optimization method, called gridsearch. The user must provide this method with a parameter space grid to search. The method will evaluate the fit statistic for each point in the grid; the best match is the grid point with the lowest value of the fit statistic. The grid search method will report back the parameter values associated with this point.
Modeling and Fitting with PSFs
-
2-D PSF kernels can now be larger than the source data (before we restricted the kernel to be equal to or smaller that the source). FFT padding 2^n occurs using the largest of the two sizes. The PSF padding is removed down to the size of the source data.
-
A fix to the PSF utility 'normalize': it no longer returns NaN when the PSF kernel is all zeros.
-
The WCS in 2-D PSFs kernels is changed according to subkernel extraction. In previous versions of Sherpa, the WCS of subkernel images is wrong. The image sources now line up according to the WCS, consistent with CIAO 3.4. NOTE: this fix only affects imaging. The kernel WCS is not used during fitting.
-
The normalization of a kernel now occurs before kernel extraction for both file-based and model-based PSFs.
-
The coordinate system of kernel plots/contours is now based on the kernel or source data coordinate system. Originally, the coordinate system was based on the number of bins in each dimension.
-
The location of model-based PSFs is now updated based on PSF parameters when set_psf() is used to associate the kernel with a source model. This is simply a convenience if the user forgets to update the kernel model parameters.
-
1-D PSF plots (from file and model) are exclusively scatter plots. (Model PSFs are no longer plotted as a red line).
2-D PSF contours (from file and model) are exclusively scatter plots. (Model PSFs are no longer plotted as a red contour lines).
Models and Parameters
-
![[New]](../imgs/new.gif)
Seven new "additive" Xspec 12.7.1 models have been added to
Sherpa. The models are:
-
The model library has been extended to include 24 optical models that are designed for use with spectra or SEDs at optical wavelengths. These include continuum models, absorption and emission features, and extinction curves for the ISM at optical and UV wavelengths. The position, line width, and other similar model parameters have units of Angstroms where relevant.
-
Several significant bugs associated with the functions set_full_model() and set_bkg_full_model() have been fixed. These functions are intended to allow users to define more complex models (i.e. models with some components that are folded through an RMF or a PSF and other components that are not).
-
Sherpa includes the "additive" and "multiplicative" models of XSpec version 12.7.1:
-
Bug fix: improved the behavior of powlaw1d at x=0.0. Since the model is not defined at that value, attempting to evaluate it would result in a "model evaluation failed" error.
Additional Enhancements & Bug Fixes
-
Using load_pha with some PHA files
would pick up the image block instead of the PHA spectrum when
the primary image was not null, and the image extension was
anything other than WMAP. In such cases, Sherpa
tried and failed to load the image instead of the PHA
spectrum. The PHA spectrum is now always loaded from a PHA
file, regardless of what image exists in the first block of
the file.
-
The DM syntax of an opened file was not passed from Sherpa to
PyCRATES as expected, causing extra, unwanted columns to be
included in the Sherpa data set.
-
The OGIP AREASCAL keyword within Type I and II PHA files is now supported in Sherpa.
-
There is a new math function from the CEPHES library, "inverse erf", in the sherpa.utils module. This function is useful for converting between confidence limits and confidence intervals (sigma/percent).
-
The ATOMDB v2.0.1 database is now distributed with CIAO 4.4, and can be used with XSpec models (such as the APEC models) that depend on the database.
-
PyFITS 3.0 is now supported in the alternate PyFITS I/O backend.
-
The get_arf function includes a reference to the associated RMF (if available) when used in set_full_model.
-
The save command records the XSpec-related settings, such as abundance and xsect.
-
Models defined by set_full_model are correctly written to the save_all output file.
-
Bug fix: The axis scale of plots output by the plot_model_component and plot_source_component commands could not be changed to log scale with the set_xlog and set_ylog command.
-
Bug fix: the log keyword argument was not working correctly for the reg_proj and reg_unc commands.
-
Bug fix: the projection and conf functions were not using the numcores setting in the .sherparc file.
GUIs (Graphical User Interfaces)
prism
-
Bug fix: Quick Histogram would occasionally fail to plot.
Analysis Scripts
chandra_repro
-
The tools acis_find_afterglow (new in CIAO 4.4) and acis_build_badpix are used in place of acis_run_hotpix to identify cosmic ray afterglows and bad pixels in ACIS observations. Details are in the "badpixel" parameter description of the ahelp file.
If badpixel=yes (the default) in CIAO 4.4, a new bad pixel file is made for all ACIS observations, except those taken in continuous-clocking mode.
-
All of the ACIS bad pixel and afterglow status bits are reset with the acis_clear_status_bits script before the new bad pixel file is made. Previously, only the bits set by acis_detect_afterglow were reset.
-
The destreak tool is run before the bad pixel tools are applied. Removing the streaks first improves the accuracy of the afterglow correction.
-
The default filename root is chosen from the first segment of the level=1 event file, e.g. changed from "acisf01838_000N002" to "acisf01838".
combine_spectra
-
New parameter: method specifies whether the script should sum or average the PHA exposure times for the header of the combined spectrum and ARF files.
-
Bug fix: the ANCRFILE header keyword in an output combined background spectrum was incorrectly being set to "NONE" when the input backround response files exactly matched the input source responses.
-
Internal updates to support the new addresp phafiles and method parameters.
specextract
-
New parameter: binarfwmap specifies the binning of the sky2tdet WMAP, which is used as input to mkwarf.
The default value of binarfwmap=1 will be sufficient for most users. When making a weighted ARF for a large region, on order of a full chip (or larger), mkwarf may run out of memory; in this case, the binarfwmap value should be increased to create a coarser WMAP.
The binwmap parameter is stiil used for creating the WMAP used with mkacisrmf.
-
The remove_extra_time_keywords has been removed from specextract. It is no longer needed, since the sky2tdet bug is fixed in CIAO 4.4.
Python 2.7 compatibility
-
Many of the scripts were updated to be compatible with python 2.7. These changes should be transparent to the user.
Removed: psextract
-
The psextract script has been removed from the tarfile. specextract should be used in psextract. A comparison of the specextract and psextract syntax:
specextract infile="evt2.fits[sky=region(src.reg)]" bkgfile="evt2.fits[sky=region(bg.reg)]" \ outroot=point asp=point.asphist pbkfile=acisf063875928N003_pbk0.fits \ weight=no correct=yes \ grouptype=NUM_CTS binspec=15 psextract events="evt2.fits[sky=region(src.reg)]" bgevents="evt2.fits[sky=region(bg.reg)]" bkgresp=yes \ root=point asol=@pcad_asol1.lis pbkfile=acisf063875928N003_pbk0.fits \ gtype=NUM_CTS gspec=15
Contact CXC Helpdesk if you need assistance transitioning from psextract syntax to specextract.
Removed: remove_extra_time_keywords
-
The remove_extra_time_keywords has been removed from the tarfile. It is no longer needed, since the sky2tdet bug is fixed in CIAO 4.4.
Libraries
cxcparam
-
Bug fix: the paccess() routine was ignoring the mode argument as always returning that the file has "rw" access.
dslib
-
If an HDUCLAS keyword is blank, omit it from the output file.
hdrlib
-
If the new SIM average-offset keywords (DY_AVG, DZ_AVG, DTH_AVG) are present in the input file, they are copied in the same component as the SIM keywords.
pixlib
-
Improved error checking on the allowed range of chip ids.
regionlib
-
Added a NULL check to regShapeGetName, e.g. to prevent a tool from crashing if multiple sky(x,y) filters do not intersect.
tcdlib
-
Two new kernel options have been added: hanning and racos; see "ahelp aconvolve" for details.
-
Bug fix: if the axis length was not a power of two, the padding was doubled to the next power of two. The extra padding would create much larger images and require large amounts of memory.
Environment
Off-the-Shelf (OTS) Package Versions
-
The following OTS packages are included with CIAO 4.4. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.
- CCfits v2.3
- cfitsio v3.280
- fftw v3.3
- fontconfig v2.8.0
- freetype v2.4.4
- gsl v1.15
- ipython v0.10.1
- NumPy v1.5.1
- pyds9 v1.3
- Python v2.7.1
- readline v6.2
- vte v0.25.1
- VTK v5.6.1
- wcssubs v3.8.3
- xpa v2.1.13
- XSpec v12_7_0 (models only)
Two OTS packages are no longer included with CIAO:
- ghostscript
- slang
Documentation
Analysis Threads
-
The CIAO, Sherpa, and ChIPS Threads are in the process of being tested and updated for the CIAO 4.4 release. Please check the "Last Update" information at the top of each thread to see if it has been reviewed.