Last modified: 25 March 2022

URL: https://cxc.cfa.harvard.edu/ciao/watchout.html

CIAO "Watch Out" Page

This page lists noteworthy items and issues about the CIAO release. For the full list of known issues please review the:


Files and directories/folders with spaces or other special characters

Users are reminded that CIAO does not work with file names or directories/folders with spaces, " ", or other certain special characters.

Chandra Archive Reprocessing (repro-5)

The CXC is currently in the middle of reprocessing the majority of the data in the Chandra archive, also known as Repro-5. One of the improvements is to correct the detector coordinates to provide a better estimate of the off-axis angle. While celestial coordinates remain unchanged, this update also does change the physical coordinates. As many of the CIAO threads use physical coordinates, users should be aware that there may be a mismatch between the coordinates used in the examples, and the reprocessed data. We will eventually be updating the threads with examples using the reprocessed data when repro-5 is complete.

FTP support discontinued

Upgrade to contrib-4.12.2 or later required to continue to use download_chandra_obsid and download_obsid_caldb.

The CXC is discontinuing all FTP services as of 01 November 2020. Most notably this affects access to the Chandra data archive using the download_chandra_obsid and download_obsid_caldb scripts. These scripts were updated in the CIAO contributed scripts release 4.12.2 to access the Chandra Data Archive via https://. Users will need to use contrib-4.12.2 or later to continue to use these scripts after that date. Users can use the ciaover command to check their version

For user who installed CIAO using ciao-install,

% ciaover -v | egrep -i 'ciao|contrib'
  CIAO        : CIAO 4.12 Monday, November 25, 2019
  Contrib     : Package release 4  Friday, September 18, 2020
  bindir      : /export/ciao-4.12/bin
  Python path : /export/ciao-4.12/ots/bin
CIAO Installation: Linux

For users who installed CIAO using conda

% ciaover -v | grep contrib
ciao-contrib              4.12.3                     py_0    https://cxc.cfa.harvard.edu/conda/ciao

Note: The conda micro version number (.3) is one less than the ciao-install version number; so this is also Release 4.

To upgrade an existing CIAO 4.12 directory installed using ciao-install, simply re-run ciao-install

% bash ciao-install

and answer the questions the same as before. It will verify what packages are already installed and will download the latest contrib release.

To upgrade CIAO 4.12 in conda, activate the CIAO environment and then using the conda update command:

% conda activate ciao
(ciao) % conda update -c https://cxc.cfa.harvard.edu/conda/ciao ciao-contrib --freeze-installed

We recommend using the --freeze-installed flag to avoid updating other package (such as numpy and matplotlib) which may be incompatible with your version of CIAO.

ds9: interruption in archive service

For users with SAOImage ds9 v8.1 and earlier, the Analysis -> Chandra (NASA/CXC) -> Public FTP option will have limited functionality after 01 November 2020. Users can upgrade to ds9 v8.2 (currently in beta) to have access to the new Chandra Public Archive by Obsid and Chandra Public Archive by Cone Search options.


ChIPS withdrawn

ChIPS has been withdrawn from CIAO

The ChIPS plotting package has been withdrawn from CIAO. Plotting in sherpa now uses the Matplotlib plotting package.

The CXC has prepared a ChIPS to Matplotlib conversion guide to assist users to convert their scripts.


Changes to Chandra products in DS10.8.3 requires CIAO 4.12 or workaround

New aspect solution file

Archived Chandra data products created with ASCDSVER=10.8.3 and above now use a different aspect solution file. Users can check the version of the archive products with dmkeypar

% dmkeypar primary/acisf13819N001_evt2.fits ASCDSVER echo+
10.8.3

This new aspect solution file is incompatible with ACIS event processing with CIAO 4.11 and earlier. Users will see the following warning messages when running chandra_repro or acis_process_events

$ chandra_repro 13819 out=13819/repro
...
Running acis_process_events to reprocess the evt1.fits file...
Output from acis_process_events:
# acis_process_events (CIAO 4.11): WARNING: 'CONTENT' keyword '13819/primary/pcadf13819_000N001_asol1.fits' in infile 'ASPSOLOBI' is not valid for aspect solution file, setting file to NONE

# acis_process_events (CIAO 4.11): WARNING: Setting 'stop = none' since acaofffile is none and stop value does not equal none, chip or tdet.

# acis_process_events (CIAO 4.11): WARNING: stop != sky and pix_adj is centroid, edser or randomize, setting pix_adj to 'none'
...

The code will still create an event file, but it will be missing sky coordinates and cannot be used for data analysis.

Workaround

Users are encouraged to upgrade to CIAO 4.12 or later as there are also calibration updates that are incompatible with CIAO 4.11 and earlier. However, users who cannot update to CIAO 4.12 can edit the header of the aspect solution file to change the CONTENT keyword:

$ dmhedit pcadf13819_000N001_asol1.fits file= op=add key=CONTENT value=ASPSOL

If not already done, be sure to gunzip the file before attempting to edit it.


SAOImage DS9

Help, I am unable to create regions in ds9!

By default, ds9 will no longer create regions when it starts; that is, clicking in the ds9 window will not create a region. This is because the ds9 interface has changed in version 7.3.2, as noted above, so that it no longer starts in region editing mode.

To create or edit regions, users must now manually switch to Region mode by selecting

Edit → Region

from the menu bar or the button bar.

Users can make this the default by editing their preferences:

Edit → Preferences ...
Menus and Buttons                  (left panel)
Edit → Menu → Region               (right panel)
Save
Preserve During Load preferences

CIAO users should not enable any of the following options

Menus and Buttons → File → Menu → Preserve During Load → Pan
Menus and Buttons → File → Menu → Preserve During Load → Region

preferences. Due to the way CIAO starts ds9, having these preferences enabled leads to a half-pixel shift in regions. This shift accumulates if the region file is saved and re-loaded.


Installing & Starting CIAO

Python 3.8 support only

In CIAO 4.14 installed using ciao-install only Python 3.8 is provided.

The Python 3 for Scientists website provides guidance on updating code written for Python 2, and the CXC Helpdesk can provide support.

Users of the conda installation can select python 3.9, 3.8, or 3.7.

CIAO on Mac needs recent XQuartz and XCode

OSX users, particularly those that have upgraded from an earlier OSX version, should ensure that compatible versions of XQuartz (at least version 2.7.11) and XCode (also comes as part of Apple Developer Tools) are installed on the system before proceeding with the CIAO installation.

bash vs. zsh

Many macOSX Catalina (10.15) users will use the new default zsh shell when they run the Terminal application. zsh users should follow the same instructions as bash shell users. This includes the same setup script:

zsh$ source /soft/ciao-4.14/bin/ciao.sh
macOSX Catalina: ls: Operation not permitted

In macOSX Catalina, Apple has restricted access to various folders in the user's home directory including Documents, Desktop, and Downloads. By default the Terminal application and Xquart's xterm application do not have access to these folders. Attempting to list the contents from the command line yeilds the following error message:

$ cd Desktop
$ ls
ls: Operation not permitted

This restriction may impede CIAO users who commonly use these folders.

Users can still navigate and display the contents of these folders using the Finder application. CIAO users who want to use these restricted folders need to grant the Terminal and|or xterm applications "Full Disk Access". The setting can be found in the "Security" settings application.

No support for 32-bit Linux, nor OSX 10.13 (High Sierra) and earlier.

CIAO is no longer available for 32-bit Linux, nor is it available for OSX 10.13 (High Sierra) or earlier versions of OSX. The platform support page describes the operating systems that can be used with CIAO.

Fontconfig error: Cannot load default config file

During the smoke tests some users will see the following message during the obsvis test

Fontconfig error: Cannot load default config file

This error is not critical. As long as the test reports a PASS, then users can ignore that message.

Parameter file conflicts with HEAsoft

Both CIAO and FTOOLS use parameter files to store parameter values for tools, and provide tools for manipulating these files (e.g. plist, pset, punlearn; they have the same name in both systems). The location of the parameter files is taken from the $PFILES environment variables for CIAO and FTOOLS (or $UPARM if $PFILES does not exist).

Common errors are:

  • pget: could not get parameter "foo"
    
  • The following parameter:
        "foo,b,l,no,,,"Keyword""
    has the following error(s):
      o Mode field "l" is invalid.
    
  • The following parameter:
    "foo,i,h,INDEF,INDEF,INDEF,"description""
    has the following error(s):
    o Problem converting minimum field "INDEF" to parameter type "i".
    o Problem converting maximum field "INDEF" to parameter type "i".
    

We recommend that you start FTOOLS before CIAO to minimize conflicts; more information on this problem is available in the CIAO and FTOOLS section of the Starting CIAO thread. Also refer to the related FAQ "The tool is failing because ")sval" has somehow shown up in the parameter value.".

Conflicts with Other Software Packages

There are some known conflicts between CIAO and other software or system libraries and tools. Issues not covered in this "Watch Out" page are listed on the "Other Software Packages" bug page.


Analysis

Incorrect GTIs when time-filtering grating data with dmcopy opt=all

When a time filter is applied to a level=1.5 (evt1a.fits) or level=2 (evt2.fits) grating file and the dmcopy opt=all option is used, the GTI block in the output file is not updated to reflect the correct time range. Time-related header keyword values - such as ontime, livetime, and exposure - will be incorrect because they are calculated from the time ranges in the GTI block.

This bug affects all ACIS and HRC grating data analysis. Read the bug report for details and a workaround.

Specifying Aspect Solution Files

The aspect solution files (pcad..._asol1.fits) are used as inputs to several of the CIAO tools - e.g. acis_process_events, asphist, and tg_resolve_events.

When there is a single aspect solution file, it can be provided directly to the tool:

unix% pset asphist infile=pcadf245733620N002_asol1.fits

In rare cases, there will be more than one aspect solution file for an observation. Then all the files must be provided in chronological order. (The time is in the filename, so "ls" lists them in chronological order.) This may be done as a comma-separated list:

unix% pset asphist infile="pcadf063874624N002_asol1.fits,pcadf063875522N002_asol1.fits,pcadf063902942N002_asol1.fits"

The list of aspect solution files may also be given as a stack. Any number of files, including just one, can be specified in the stack:

unix% cat pcad_asol1.lis
pcadf245733620N002_asol1.fits

unix% pset asphist infile=@pcad_asol1.lis

or more than one:

unix% cat pcad_asol1.lis
pcadf063874624N002_asol1.fits
pcadf063875522N002_asol1.fits
pcadf063902942N002_asol1.fits

unix% pset asphist infile=@pcad_asol1.lis

Note that an at symbol ("@") is used before the filename to indicate that it is a stack file.


Python

Installing 3rd party modules with pip3

Users must be careful when using pip3 to install 3rd party modules such as Astropy and Scipy into CIAO. The latest version of these packages MAY require updated versions of packages that CIAO needs; numpy being the most likely conflict.

pip3 installing source code packages

Many of the python packages available via pip are either pure python or are precompiled binary distributions. However, some packages are provided as source code and will need to compile against the libraries on the users system. Users may experience problems building these packages due to some remaining hard-coded paths in some configuration files.