CIAO "Watch Out" Page
This page lists noteworthy items and issues about the CIAO release. For the full list of known issues please review the:
Files and directories/folders with spaces or other special characters
Users are reminded that CIAO does not work with file names or directories/folders with spaces, " ", or other certain special characters.
Chandra Archive Reprocessing (repro-5)
The CXC is currently in the middle of reprocessing the majority of the data in the Chandra archive, also known as Repro-5. One of the improvements is to correct the detector coordinates to provide a better estimate of the off-axis angle. While celestial coordinates remain unchanged, this update also does change the physical coordinates. As many of the CIAO threads use physical coordinates, users should be aware that there may be a mismatch between the coordinates used in the examples, and the reprocessed data. We will eventually be updating the threads with examples using the reprocessed data when repro-5 is complete.
FTP support discontinued
- Upgrade to contrib-4.12.2 or later required to continue to use download_chandra_obsid and download_obsid_caldb.
-
The CXC is discontinuing all FTP services as of 01 November 2020. Most notably this affects access to the Chandra data archive using the download_chandra_obsid and download_obsid_caldb scripts. These scripts were updated in the CIAO contributed scripts release 4.12.2 to access the Chandra Data Archive via https://. Users will need to use contrib-4.12.2 or later to continue to use these scripts after that date. Users can use the ciaover command to check their version
For user who installed CIAO using ciao-install,
% ciaover -v | egrep -i 'ciao|contrib' CIAO : CIAO 4.12 Monday, November 25, 2019 Contrib : Package release 4 Friday, September 18, 2020 bindir : /export/ciao-4.12/bin Python path : /export/ciao-4.12/ots/bin CIAO Installation: Linux
For users who installed CIAO using conda
% ciaover -v | grep contrib ciao-contrib 4.12.3 py_0 https://cxc.cfa.harvard.edu/conda/ciao
Note: The conda micro version number (.3) is one less than the ciao-install version number; so this is also Release 4.
To upgrade an existing CIAO 4.12 directory installed using ciao-install, simply re-run ciao-install
% bash ciao-install
and answer the questions the same as before. It will verify what packages are already installed and will download the latest contrib release.
To upgrade CIAO 4.12 in conda, activate the CIAO environment and then using the conda update command:
% conda activate ciao (ciao) % conda update -c https://cxc.cfa.harvard.edu/conda/ciao ciao-contrib --freeze-installed
We recommend using the --freeze-installed flag to avoid updating other package (such as numpy and matplotlib) which may be incompatible with your version of CIAO.
- ds9: interruption in archive service
-
For users with SAOImage ds9 v8.1 and earlier, the Analysis -> Chandra (NASA/CXC) -> Public FTP option will have limited functionality after 01 November 2020. Users can upgrade to ds9 v8.2 (currently in beta) to have access to the new Chandra Public Archive by Obsid and Chandra Public Archive by Cone Search options.
ChIPS withdrawn
- ChIPS has been withdrawn from CIAO
-
The ChIPS plotting package has been withdrawn from CIAO. Plotting in sherpa now uses the Matplotlib plotting package.
The CXC has prepared a ChIPS to Matplotlib conversion guide to assist users to convert their scripts.
Changes to Chandra products in DS10.8.3 requires CIAO 4.12 or workaround
- New aspect solution file
-
Archived Chandra data products created with ASCDSVER=10.8.3 and above now use a different aspect solution file. Users can check the version of the archive products with dmkeypar
% dmkeypar primary/acisf13819N001_evt2.fits ASCDSVER echo+ 10.8.3
This new aspect solution file is incompatible with ACIS event processing with CIAO 4.11 and earlier. Users will see the following warning messages when running chandra_repro or acis_process_events
$ chandra_repro 13819 out=13819/repro ... Running acis_process_events to reprocess the evt1.fits file... Output from acis_process_events: # acis_process_events (CIAO 4.11): WARNING: 'CONTENT' keyword '13819/primary/pcadf13819_000N001_asol1.fits' in infile 'ASPSOLOBI' is not valid for aspect solution file, setting file to NONE # acis_process_events (CIAO 4.11): WARNING: Setting 'stop = none' since acaofffile is none and stop value does not equal none, chip or tdet. # acis_process_events (CIAO 4.11): WARNING: stop != sky and pix_adj is centroid, edser or randomize, setting pix_adj to 'none' ...
The code will still create an event file, but it will be missing sky coordinates and cannot be used for data analysis.
Workaround
Users are encouraged to upgrade to CIAO 4.12 or later as there are also calibration updates that are incompatible with CIAO 4.11 and earlier. However, users who cannot update to CIAO 4.12 can edit the header of the aspect solution file to change the CONTENT keyword:
$ dmhedit pcadf13819_000N001_asol1.fits file= op=add key=CONTENT value=ASPSOL
If not already done, be sure to gunzip the file before attempting to edit it.
SAOImage DS9
- Help, I am unable to create regions in ds9!
-
By default, ds9 will no longer create regions when it starts; that is, clicking in the ds9 window will not create a region. This is because the ds9 interface has changed in version 7.3.2, as noted above, so that it no longer starts in region editing mode.
To create or edit regions, users must now manually switch to Region mode by selecting
Edit → Regionfrom the menu bar or the button bar.
Users can make this the default by editing their preferences:
Edit → Preferences ... Menus and Buttons (left panel) Edit → Menu → Region (right panel) Save
- Preserve During Load preferences
-
CIAO users should not enable any of the following options
Menus and Buttons → File → Menu → Preserve During Load → Pan Menus and Buttons → File → Menu → Preserve During Load → Region
preferences. Due to the way CIAO starts ds9, having these preferences enabled leads to a half-pixel shift in regions. This shift accumulates if the region file is saved and re-loaded.
Installing & Starting CIAO
- Python 3.8 support only
-
In CIAO 4.14 installed using ciao-install only Python 3.8 is provided.
The Python 3 for Scientists website provides guidance on updating code written for Python 2, and the CXC Helpdesk can provide support.
Users of the conda installation can select python 3.9, 3.8, or 3.7.
- CIAO on Mac needs recent XQuartz and XCode
-
OSX users, particularly those that have upgraded from an earlier OSX version, should ensure that compatible versions of XQuartz (at least version 2.7.11) and XCode (also comes as part of Apple Developer Tools) are installed on the system before proceeding with the CIAO installation.
- bash vs. zsh
-
Many macOSX Catalina (10.15) users will use the new default zsh shell when they run the Terminal application. zsh users should follow the same instructions as bash shell users. This includes the same setup script:
zsh$ source /soft/ciao-4.14/bin/ciao.sh
- macOSX Catalina: ls: Operation not permitted
-
In macOSX Catalina, Apple has restricted access to various folders in the user's home directory including Documents, Desktop, and Downloads. By default the Terminal application and Xquart's xterm application do not have access to these folders. Attempting to list the contents from the command line yeilds the following error message:
$ cd Desktop $ ls ls: Operation not permitted
This restriction may impede CIAO users who commonly use these folders.
Users can still navigate and display the contents of these folders using the Finder application. CIAO users who want to use these restricted folders need to grant the Terminal and|or xterm applications "Full Disk Access". The setting can be found in the "Security" settings application.
- No support for 32-bit Linux, nor OSX 10.13 (High Sierra) and earlier.
-
CIAO is no longer available for 32-bit Linux, nor is it available for OSX 10.13 (High Sierra) or earlier versions of OSX. The platform support page describes the operating systems that can be used with CIAO.
- Fontconfig error: Cannot load default config file
-
During the smoke tests some users will see the following message during the obsvis test
Fontconfig error: Cannot load default config file
This error is not critical. As long as the test reports a PASS, then users can ignore that message.
- Parameter file conflicts with HEAsoft
-
Both CIAO and FTOOLS use parameter files to store parameter values for tools, and provide tools for manipulating these files (e.g. plist, pset, punlearn; they have the same name in both systems). The location of the parameter files is taken from the $PFILES environment variables for CIAO and FTOOLS (or $UPARM if $PFILES does not exist).
Common errors are:
-
pget: could not get parameter "foo"
-
The following parameter: "foo,b,l,no,,,"Keyword"" has the following error(s): o Mode field "l" is invalid. -
The following parameter: "foo,i,h,INDEF,INDEF,INDEF,"description"" has the following error(s): o Problem converting minimum field "INDEF" to parameter type "i". o Problem converting maximum field "INDEF" to parameter type "i".
We recommend that you start FTOOLS before CIAO to minimize conflicts; more information on this problem is available in the CIAO and FTOOLS section of the Starting CIAO thread. Also refer to the related FAQ "The tool is failing because ")sval" has somehow shown up in the parameter value.".
-
- Conflicts with Other Software Packages
-
There are some known conflicts between CIAO and other software or system libraries and tools. Issues not covered in this "Watch Out" page are listed on the "Other Software Packages" bug page.
Analysis
- Incorrect GTIs when time-filtering grating data with dmcopy opt=all
-
When a time filter is applied to a level=1.5 (evt1a.fits) or level=2 (evt2.fits) grating file and the dmcopy opt=all option is used, the GTI block in the output file is not updated to reflect the correct time range. Time-related header keyword values - such as ontime, livetime, and exposure - will be incorrect because they are calculated from the time ranges in the GTI block.
This bug affects all ACIS and HRC grating data analysis. Read the bug report for details and a workaround.
- Specifying Aspect Solution Files
-
The aspect solution files (pcad..._asol1.fits) are used as inputs to several of the CIAO tools - e.g. acis_process_events, asphist, and tg_resolve_events.
When there is a single aspect solution file, it can be provided directly to the tool:
unix% pset asphist infile=pcadf245733620N002_asol1.fits
In rare cases, there will be more than one aspect solution file for an observation. Then all the files must be provided in chronological order. (The time is in the filename, so "ls" lists them in chronological order.) This may be done as a comma-separated list:
unix% pset asphist infile="pcadf063874624N002_asol1.fits,pcadf063875522N002_asol1.fits,pcadf063902942N002_asol1.fits"
The list of aspect solution files may also be given as a stack. Any number of files, including just one, can be specified in the stack:
unix% cat pcad_asol1.lis pcadf245733620N002_asol1.fits unix% pset asphist infile=@pcad_asol1.lis
or more than one:
unix% cat pcad_asol1.lis pcadf063874624N002_asol1.fits pcadf063875522N002_asol1.fits pcadf063902942N002_asol1.fits unix% pset asphist infile=@pcad_asol1.lis
Note that an at symbol ("@") is used before the filename to indicate that it is a stack file.
Python
- Installing 3rd party modules with pip3
-
Users must be careful when using pip3 to install 3rd party modules such as Astropy and Scipy into CIAO. The latest version of these packages MAY require updated versions of packages that CIAO needs; numpy being the most likely conflict.
Many of the python packages available via pip are either pure python or are precompiled binary distributions. However, some packages are provided as source code and will need to compile against the libraries on the users system. Users may experience problems building these packages due to some remaining hard-coded paths in some configuration files.