Last modified: 07 March 2021

URL: https://cxc.cfa.harvard.edu/ciao/threads/ciao_install_conda/

Installing CIAO with conda

CIAO 4.13 Science Threads


Overview

Synopsis:

This thread shows how to install CIAO using the conda package manager. Users learn how to perform the basic steps and then can learn about the additional options they can use to customize their installation.

Related Links:

Last Update: 07 Mar 2021 - Added some FAQs


Contents


Quick Start

The following commands are all that is needed to install CIAO using an existing conda installation and verify that everything is working:

$ conda create -n ciao-4.13 -c https://cxc.cfa.harvard.edu/conda/ciao \
  ciao sherpa ds9 ciao-contrib caldb_main marx
$ conda activate ciao-4.13
$ cd $ASCDS_INSTALL/test
$ make

The rest of this thread discusses these steps in more detail and presents the user with additional options.


Detailed Instructions

Introduction to conda

CIAO 4.13 can be installed using conda.

conda is a package management and environment management system. It is similar to other package managers (such as yum, apt, macports, dnf) but allows for users to create multiple environments with multiple versions of packages which can co-exist on the same system. Users can easily install pre-compiled and pre-configured packages from curated channels (repositories) into various environments. Users then activate the environment to run the applications.

With conda , Linux and Mac users can choose to install CIAO-4.13 using either Python 3.8, 3.7, or 3.6. Using the conda package manager users can also install additional third-party packages.


Install conda

Users must already have conda installed on their systems. This can be either Anaconda or miniconda. CIAO-4.13 is available for Linux and MacOSX. CIAO does not work with Windows, not even when running Windows Subsystem for Linux.

[NOTE]
Minimum conda version?

Some users, macOSX in particular, have reported problems with using old versions of conda. CIAO has successfully be installed using conda 4.7.6 and newer. Users should consider upgrading their base conda installation before installing CIAO.

Users can check if they have conda and the version using

$ conda --version
conda: Command not found.

In this case, conda is not installed or otherwise not setup in the user's PATH.

[IMPORTANT]
Help installing conda

Users should consult the Anaconda troubleshooting guide if they encounter problems installing conda instead of the CXC helpdesk. Timeout's during download are frequent. Users are encouraged to simply try again.

Below are the basic instructions for installing miniconda.

Linux$ curl -o get_conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh 

or 

macOS$ curl -o get_conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh

Users can then run the interactive installation script

$ bash get_conda.sh 

... accept license and answer prompts ... 


Preparing transaction: done
Executing transaction: done
installation finished.
Do you wish the installer to initialize Miniconda3
by running conda init? [yes|no]
[no] >>> 
[IMPORTANT]
tcsh shell users

Users who use the tcsh shell must not run the conda init command if you are using conda 4.7.4 or earlier. There is a serious bug that may prevent users from running commands or may even prevent users from logging into their user account.

This does not affect users using the bash shell. If unsure which you are using check the output from

$ echo $0
bash

After the installer has finished users will want to be sure that the conda command is setup in their PATH environment.

bash$ PATH="${PATH}:<<directory where miniconda was installed>>/bin"

or

tcsh% set path = ( ${path} <<directory where miniconda was installed>>/bin )

Install CIAO into a new conda environment

The CXC is hosting the CIAO-4.13 packages on our own conda channel. This is due to the size of the packages.

To create a new conda environment with CIAO installed users run the following command:

$ conda create -n ciao-4.13 \
  -c https://cxc.cfa.harvard.edu/conda/ciao \
  ciao sherpa ds9 ciao-contrib caldb_main marx

This will create an environment named ciao-4.13 and perform a standard CIAO installation including the standard CALDB. Users who need the ACIS or HRC background event files should replace caldb_main with simply caldb.

Users can choose to install CIAO with either Python 3.8 (default), 3.7, or 3.6. Use the following example to select a different python version:

$ conda create -n ciao-4.13_python35 \
  -c https://cxc.cfa.harvard.edu/conda/ciao \
  python=3.6 ciao sherpa ds9 ciao-contrib caldb_main marx
[TIP]
Tip

Timeouts during download are frequent. Users are encouraged to simply try again.

[NOTE]
Blank sky background files

To install the ACIS and HRC blank sky background files users should replace caldb_main with caldb.

$ conda create -n ciao-4.13 \
  -c https://cxc.cfa.harvard.edu/conda/ciao \
  ciao sherpa ds9 ciao-contrib caldb marx

Alternative: use conda env instead

The conda create example above allows conda to resolve all the package dependencies and will select the latest compatible versions of those packages. There may be circumstances where this causes problems. As an alternative, user can install CIAO using exactly the same version of all the packages that were available when CIAO was released using the conda env command:

$ conda env create -n ciao-4.13 cxc/ciao-4.13-caldb-<platform>
  -or-
$ conda env create -n ciao-4.13 cxc/ciao-4.13-<platform>

where <platform> is either linux or macos. The -caldb environment installs the caldb_main package. Users can then later include the blank sky background files by

$ conda install -n ciao-4.13 -c https://cxc.cfa.harvard.edu/conda/ciao/ \
  acis_bkg_evt hrc_bkg_evt

As with any conda package, this command can be used to install the blank sky background files in any environment at any time.


Activate CIAO environment

Once the CIAO environment is created, it has to be activated. This is equivalent to source'ing the CIAO setup scripts.

For bash shell users, make sure conda's activate script is in your PATH and then simply

$ source activate ciao-4.13

where ciao-4.13 is the name of the environment you just created.

For tcsh shell users

% alias conda source <<directory where miniconda was installed>>/etc/profile.d/conda.csh
% conda activate ciao-4.13

Users may wish to add the alias to their $HOME/.tcshrc startup files.

By default, users will notice the shell prompt has changed to indicate the current conda environment.

$ source activate ciao-4.13
(ciao-4.13) $

Run the smoke tests

Users are encouraged to run the CIAO smoke tests to ensure everything is installed and working correctly.

(ciao-4.13)$ cd $ASCDS_INSTALL/test
(ciao-4.13)$ make

If users choose to omit packages (eg sherpa or ds9), then smoke tests associated with those packages will fail.


Install additional 3rd party packages

Additional packages can then be installed into the CIAO environment using the conda installer.

(ciao-4.13)$ conda install -n ciao-4.13 jupyter astropy scipy

Due to the constraints on python version and numpy version, users may not be able to install certain packages, or possibly the latest versions of certain packages such as astropy or scipy. Similarly, users may not be able to install CIAO into an existing environment with incompatible versions of python or numpy.

If installing additional CIAO packages, users need to remember to include the CXC conda channel. For example to add the ACIS blank sky background files to an existing CIAO installation the command would look like:

(ciao-4.13)$ conda install -n ciao-4.13 -c https://cxc.cfa.harvard.edu/conda/ciao acis_bkg_evt

Users can also install packages using pip or other package managers

(ciao-4.13)$ pip3 install bash_kernel

Leave CIAO environment

Use the deactivate command to leave the conda environment:

(ciao-4.13)$ conda deactivate

or simply exit the terminal window. deactivate can be useful when users need to install additional package which may conflict with the CIAO environment or when updating CIAO components.


Updating CIAO environment

Users can install the latest CIAO and CALDB updates using the conda update command

$ conda update -n ciao-4.13 \
  -c https://cxc.cfa.harvard.edu/conda/ciao \
  ciao sherpa ds9 ciao-contrib caldb_main marx --freeze-installed

Only the packages listed will be updated. To update all the packages in the environment use the --all option. We recommend using the --freeze-installed flag to prevent other packages such as numpy from updating which may could potentially cause a conflict with CIAO.

After updating, users may want to run the conda clean command to completely remove the previous versions. This is especially true when updating the CALDB since each CALDB update is a full release which each requires 5-12Gb of disk space.

$ conda clean --all

Alternative Download Options

Users on slow internet or wireless connections may experience problems downloading the large data files associated with the Chandra CALDB and the XSPEC models. Often users will be able to simply retry again successfully; however, sometimes the server timeouts are persistent.

CALDB alternatives

Partial CALDB download

Users who only need to analyze a limited number of Chandra datasets (OBS_IDs), can use the download_obsid_caldb script to download only the calibrations files associated with individual observations (including optional blank sky background files).

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao ciao sherpa ciao-contrib ds9
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% download_chandra_obsid 4425
...
(ciao_no_caldb) unix% download_obsid_caldb 4425
Output CALDB directory (${ASCDS_INSTALL}/CALDB -> /soft/miniconda/envs/ciao_sans_caldb/CALDB): 
download_obsid_caldb
          infile = 4425
          outdir = /soft/miniconda/envs/ciao_sans_caldb/CALDB
      background = no
         missing = no
         clobber = no
         verbose = 1
            mode = ql

Retrieving files for CALDB_VER = 4.9.4
Retrieving CALDB index files
Processing infile=4425/primary/acisf04425N004_evt2.fits.gz
Retrieving CALDB data files
    Filename:                               0------------------1
    telD1999-07-23geomN0006.fits            ####################
    telD1999-07-23aimptsN0002.fits          ####################
    telD1999-07-23tdetN0001.fits            ####################
    telD1999-07-23skyN0002.fits             ####################
    telD1999-07-23sgeomN0001.fits           ####################
    hrmaD1996-12-20axeffaN0008.fits         ####################
    hrmaD1996-12-20vignetN0003.fits         ####################
    acisD1997-04-17qeN0006.fits             ####################
    acisD2002-02-01qeuN0007.fits            ####################
    acisD2000-11-28badpixN0004.fits         ####################
    acisD1999-08-13contamN0014.fits         ####################
    acisD1996-11-01gradeN0004.fits          ####################
    acisD2000-01-29grdimgN0001.fits         ####################
    acisD2000-01-29gain_ctiN0008.fits       ####################
    acisD1996-11-01evtspltN0002.fits        ####################
    acisD2002-08-01ctiN0009.fits            ####################
    acisD2003-02-01t_gainN0008.fits         ####################
    acisD1999-07-22subpixN0001.fits         ####################
    acisD2000-01-29fef_pha_ctiN0004.fits    ####################
    acisD1999-09-16dead_areaN0001.fits      ####################
    hrmaD1996-12-20reefN0001.fits           ####################
    acisD2000-01-29p2_respN0008.fits        ####################
    hrmaD1996-11-01wpsfN0001.fits           ####################
    acisD2000-01-29osip_ctiN0006.fits       ####################
    acisD2002-11-15gtilimN0004.fits         ####################

Be sure to source the new setup scripts to
use these CALDB files.

(t)csh:
source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.unix

bash:
source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.sh

(ciao_no_caldb) unix% source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.unix
[IMPORTANT]
Important

Users must remember to source the caldbinit setup script every time they activate the environment.

The partial CALDB can be installed anywhere; the default is the same location as the full CALDB. When run again, the script will skip any CALDB files that have already been downloaded and will only download the missing files.

Install separately using the ciao-install tar files

Similar to above, users can download the entire CALDB and install it manually. We have a copy of the CALDB files in our Google Drive in the CALDB folder. The Google Drive downloads are less likely to time out.

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao ciao sherpa ciao-contrib ds9
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% cd $ASCDS_INSTALL
(ciao_no_caldb) unix% mkdir CALDB
(ciao_no_caldb) unix% cd CALDB
(ciao_no_caldb) unix% tar xfz ~/Downloads/caldb_4.9.4_main.tar.gz
(ciao_no_caldb) unix% tar xfz ~/Downloads/acis_bkgrnd_4.9.2.1.tar.gz
(ciao_no_caldb) unix% tar xfz ~/Downloads/hrc_bkgrnd_4.7.7.tar.gz

  -- For bash/dash/zsh users
(ciao_no_caldb) unix% export CALDB=$ASCDS_INSTALL/CALDB
(ciao_no_caldb) unix% export CALDBCONFIG=${CALDB}/software/tools/caldb.config
(ciao_no_caldb) unix% export CALDBALIAS=${CALDB}/software/tools/alias_config.fits

  -- For tcsh users
(ciao_no_caldb) unix% setenv CALDB $ASCDS_INSTALL/CALDB
(ciao_no_caldb) unix% setenv CALDBCONFIG ${CALDB}/software/tools/caldb.config
(ciao_no_caldb) unix% setenv CALDBALIAS ${CALDB}/software/tools/alias_config.fits
[IMPORTANT]
Important

Users must remember to setup these environment variables each time they activate the environment.

Install individual conda tar files

Users can also install the individual conda tar files from our Google Drive in the conda/ciao/noarch folder. The Google Drive downloads are less likely to time out.

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao ciao sherpa ciao-contrib ds9
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% conda install ~/Downloads/caldb_main-4.9.4-0.tar.bz2
(ciao_no_caldb) unix% conda install ~/Downloads/acis_bkg_evt-4.9.2.1-1.tar.bz2   (optional)
(ciao_no_caldb) unix% conda install ~/Downloads/hrc_bkg_evt-4.7.7.1-1.tar.bz2    (optional)

The advantage of this method is that it will also install the activation scripts; however, when updates are requested conda may try to download the entire CALDB files again.

Install from a local channel

Users who have downloaded the conda tar file from the Google Drive (see above) can also setup a local conda repository and install from there. The advantage is that installing using the individual tar files does not check for dependencies; whereas installing from a local repository does resolve dependencies.

unix% conda install conda-build
...
unix% mkdir -p /tmp/my_conda_dir/noarch
unix% cp ~/Downloads/caldb_main-4.9.4-0.tar.bz2 /tmp/my_conda_dir/noarch
unix% cp ~/Downloads/acis_bkg_evt-4.9.2.1-1.tar.bz2 /tmp/my_conda_dir/noarch   (optional)
unix% cp ~/Downloads/hrc_bkg_evt-4.7.7.1-1.tar.bz2 /tmp/my_conda_dir/noarch    (optional)

unix% conda index /tmp/my_conda_dir

unix% conda create -n ciao_my_caldb -c /tmp/my_conda_dir/ -c https://cxc.cfa.harvard.edu/conda/ciao ciao sherpa ciao-contrib ds9 caldb_main
...
  ca-certificates    pkgs/main/linux-64::ca-certificates-2021.1.19-h06a4308_0
  cairo              pkgs/main/linux-64::cairo-1.14.12-h8948797_3
  caldb_main         my_conda_dir/noarch::caldb_main-4.9.4-0
  certifi            pkgs/main/linux-64::certifi-2020.12.5-py38h06a4308_0
  cfitsio            ciao/linux-64::cfitsio-3.490-h14c3975_2
...
unix% conda activate ciao_my_caldb

XSPEC alternatives

Given the platform specific dependencies, the only alternative method to download the sherpa XSPEC model files is to download the correct tar file from our Google Drive and install it using the local repository method described above

unix% conda install conda-build
...

 -- For Linux
unix% mkdir -p /tmp/my_conda_dir/linux-64
unix% cp ~/Downloads/xspec-modelsonly-12.10.1s-h1c2f66e_2.tar.bz2 /tmp/my_conda_dir/linux-64

 -- For macOS
unix% mkdir -p /tmp/my_conda_dir/osx-64
unix% cp ~/Downloads/xspec-modelsonly-12.10.1s-h12d9a10_2.tar.bz2 /tmp/my_conda_dir/osx-64


unix% conda index /tmp/my_conda_dir

unix% conda create -n ciao_my_xspec -c /tmp/my_conda_dir/ -c https://cxc.cfa.harvard.edu/conda/ciao ciao sherpa ciao-contrib ds9 caldb_main
...
  xerces-c           pkgs/main/linux-64::xerces-c-3.2.3-h780794e_0
  xpa                ciao/linux-64::xpa-2.1.19-h14c3975_0
  xspec-modelsonly   my_conda_dir/linux-64::xspec-modelsonly-12.10.1s-h1c2f66e_2
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
...
unix% conda activate ciao_my_xspec

Caveats with Alternative Downloads

Please keep the following caveats in mind

  • Using a local channel will list the package's channel (when running "conda list") as the directory location
  • Using an explicit package will list the package's channel (when running "conda list") as "unknown"
  • If you later try to install from the official channel, you will get messages of the package getting superseded by a higher-priority channel.

Known Problems

Users should be aware of these known issues before using the conda ciao installation.

  1. The conda installation will seem to hang during the installation process when retrieving the largest segments: the XSPEC models and CALDB files. Do not kill the installation. If it times out then it will be able resume and only install the failed packages.

  2. Users who keep the CALDB separate from the CIAO installation must setup the following environment variables independently from the conda setup.

    $ export CALDB=/Some/Other/Directory/CALDB
    $ export CALDBCONFIG=${CALDB}/software/tools/caldb.config
    $ export CALDBALIAS=${CALDB}/software/tools/alias_config.fits
    

    These can be setup in the users' login script ($HOME/.bash_profile), or setup each time the environment is activated. They can be setup either before or after the conda environment is activated.

    The CALDB version will not be reported by the ciaover command if it is not installed by conda

  3. Due to the constraints on python version and numpy version, users may not be able to install CIAO in the same environment as existing packages. Known incompatibilities include:

    • CIAO and StSCI's astroconda which fail due to version incompatibilities in the CFITSIO library.
    • There is a conflict between CIAO and HEASARC's XSPEC package. Users must
      1. First, initialize their HEASARC environment

        $ export HEADAS=/soft/lheasoft/headas/x86_64-pc-linux 
        $ source $HEADAS/headas-init.sh
        
      2. Then activate their CIAO environment

        $ conda activate ciao-4.13
        
      3. Finally, users have to restore the HEADAS environment variable back to it's original value

        $ export HEADAS=/soft/lheasoft/headas/x86_64-pc-linux 
        

      Note: this may cause problems using XSpec models in sherpa. Users must not source the headas-init.sh script after activating the CIAO environment as it will cause incompatibilities in the suite of parameters tools: pset, pget, and plist.

  4. Users must not activate CIAO conda environment in the same terminal (shell) where the classic CIAO setup script (eg ciao.bash) has already been sourced. Similarly, users must not source the CIAO setup scripts in an activate CIAO conda environment. The results in both cases are corrupted environments which can lead to unexpected failures.


History

12 Nov 2020 Moved detailed conda instructions into own thread.
01 Mar 2021 Added new section on Alternative download options.
07 Mar 2021 Added some FAQs