Last modified: 11 Mar 2024

URL: https://cxc.cfa.harvard.edu/ciao/threads/ciao_install_tool/

Installing CIAO 4.16 Using the ciao-install Script

CIAO 4.16 Science Threads


Overview

Synopsis:

The ciao-install script is designed to simplify and automate the installation and upgrade of CIAO and the Chandra Calibration DataBase (CALDB). ciao-install downloads the selected files to your computer, verifies the file integrity, and unpacks them. The script configures CIAO, sets up the command-line ahelp system, and (optionally) runs the CIAO smoke tests.

[IMPORTANT]
[New]Change to how CIAO is distributed

CIAO is no longer being distributed as a set of stand alone tar files. As of CIAO 4.16, CIAO is only being distributed as a set of conda packages. The ciao-install script has been completely re-written to automate the installation of conda and CIAO.

The ciao-install help file has detailed information on the script. For a summary of options, run the script with the "--help" option.

Related Links:

Last Update: 11 Mar 2024 - Added instructions for downloading the ciao-install script via Google Drive as an alternative for users with slow connections.


Contents


Quick Start: Running ciao-install

Just looking for a reminder on how to run the ciao-install script? Here is it:

unix% bash /<path>/ciao-install

or, as described further below,

unix% bash /<path>/ciao-install --help

For a detailed description of running the script, read the rest of this thread.


System Requirements

CIAO is supported on a number of operating systems, which are listed on the Platform Support page. Certain system requirements are necessary to successfully install and run CIAO; review these system requirements before continuing with the installation.


Installation Issues

Users should be aware of these installation items before installing CIAO 4.16. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.

Remove old parameter files

With every new CIAO release, some parameter files have changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:

unix% rm ~/cxcds_param4/*

Download the ciao-install script

download the ciao-install script.

[NOTE]
Note

Note: If you download multiple copies of ciao-install it is likely that your browser will append a number onto the file name, eg "ciao-install(1)". Be sure to check that you are using the last/correct version of this file.


Where should CIAO and CALDB be installed?

The CIAO software package and the Chandra Calibration Database (CALDB) can be installed in any location and do not need to be installed with root or super-user privileges.

CIAO

CIAO unpacks into a directory called ciao-<version>/, i.e. ciao-4.16 for this release. If the current version of CIAO were to be installed into /usr/local/, the full path to CIAO would be /usr/local/ciao-4.16/. This directory (/usr/local/ciao-4.16/) is referred to as the root directory of CIAO in the rest of this thread. It is also the value that the $ASCDS_INSTALL environment variable is set to once CIAO has been started.

In the rest of this thread we will assume CIAO is being installed into the /soft/ directory.


CALDB

The ciao-install script can either install the Chandra CALDB files directly into the CIAO installation directory, or users can use the ciao-install script to create a link to an existing CALDB installation.


Installing CIAO as the root user

It is is strongly suggested that CIAO be installed as a non-privileged, regular user and not as the root user. If CIAO must be installed in a directory owned by root and you are following the manual process, then

  1. create the ciao-4.16/ directory in the desired location, as the root user;
  2. use the chown utility to change ownership of this directory to the installer's username;
  3. unpack CIAO into the newly-created directory as the installer, not as the root user.

Installing the CIAO Software

ciao-install may be run from any location. The script prompts for required information. Here are the prompts and sample answers for a user named "ciaouser"; each prompt is explained in the following section:

unix% bash /<path>/ciao-install
Script Log file is /soft/download/ciao-install-23-12-04.10.20.52.log

Download directory for tar files (/soft/download): /soft/download
CIAO installation directory (/home/ciaouser): /soft
Delete the cache after install? (y|n) (n): 
Run smoke tests? (y|n) (y): 
Save these settings? (y|n) (y): 
Downloading Conda...
Verifying Conda...
Installing: ciao sherpa contrib caldb 
For Linux
Downloading environment files for Linux...
Installing segments...
Fixing install...
Fixing top to: /soft/ciao-4.16
DONE!
CIAO configuration is complete... 
CIAO 4.16.0 Tuesday, December 05, 2023
  bindir      : /soft/ciao-4.16/bin
  CALDB       : 4.11.0
Re-indexing ahelp system
Running smoke tests
 Requested test suite includes 37 tests
   [1/37] Running test ahelp-smoke001 ... PASS
   [2/37] Running test dm-smoke001 ... PASS
   [3/37] Running test dm-smoke002 ... PASS
   [4/37] Running test dm-smoke003 ... PASS
   [5/37] Running test dm-smoke004 ... PASS
   [6/37] Running test dm-smoke005 ... PASS
   [7/37] Running test dm-smoke006 ... PASS
   [8/37] Running test dm-smoke007 ... PASS
   [9/37] Running test ds9-smoke001 ... PASS
   [10/37] Running test ipython-smoke001 ... PASS
   [11/37] Running test ipython-smoke002 ... PASS
   [12/37] Running test obsvis-smoke001 ... PASS
   [13/37] Running test sherpa-smoke001 ... PASS
   [14/37] Running test tools-ape-smoke001 ... PASS
   [15/37] Running test tools-arfcorr-smoke001 ... PASS
   [16/37] Running test tools-asphist-smoke001 ... PASS
   [17/37] Running test tools-asphist-smoke002 ... PASS
   [18/37] Running test tools-csmooth-smoke001 ... PASS
   [19/37] Running test tools-dmcoords-smoke001 ... PASS
   [20/37] Running test tools-dmcoords-smoke002 ... PASS
   [21/37] Running test tools-dme-smoke001 ... PASS
   [22/37] Running test tools-dme-smoke002 ... PASS
   [23/37] Running test tools-dmstat-smoke001 ... PASS
   [24/37] Running test tools-dmstat-smoke002 ... PASS
   [25/37] Running test tools-dmstat-smoke003 ... PASS
   [26/37] Running test tools-hpe-smoke001 ... PASS
   [27/37] Running test tools-mkacisrmf-smoke001 ... PASS
   [28/37] Running test tools-mkacisrmf-smoke002 ... PASS
   [29/37] Running test tools-mkacisrmf-smoke003 ... PASS
   [30/37] Running test tools-mkwarf-smoke001 ... PASS
   [31/37] Running test tools-mkwarf-smoke002 ... PASS
   [32/37] Running test tools-tcm-smoke001 ... PASS
   [33/37] Running test tools-tgdetect-smoke001 ... PASS
   [34/37] Running test tools-tge-smoke001 ... PASS
   [35/37] Running test tools-tge2-smoke001 ... PASS
   [36/37] Running test tools-tre-smoke001 ... PASS
   [37/37] Running test tools-wav-smoke001 ... PASS
 37 smoke tests run: 37 PASSED, 0 FAILED, 0 SKIPPED
To use the install:
	(t)csh: $> source /soft/ciao-4.16/bin/ciao.csh
	bash/zsh: $> source /soft/ciao-4.16/bin/ciao.sh

1. The installation log

The log file - ciao-install-<date>.log - records the script status information; this is also printed to the screen. The log file can be used to help diagnose problems via the CXC Helpdesk.


2. Download directory for tar files

The download directory is a location to which the software and CALDB tarfiles are downloaded before the software is installed. For this installation the directory name is /soft/download/:

Download directory for tar files (/soft/download): /soft/download

3. CIAO installation directory

The next prompt is for the location of CIAO; we want CIAO to be installed into /soft/ciao-4.16 so we answer /soft rather than accept the default value:

CIAO installation directory (/home/ciaouser): /soft

4. Delete tar files after install?

Should the tarfiles be deleted when ciao-install is down running? The default value is n:

Delete the cache after install? (y|n) (n): 

The tarfiles can also be removed manually later.


5. Run smoke tests?

The next prompt asks you whether to run the CIAO smoke tests; these are some simple tests we provide to help validate the installation. Is is strongly suggested that you accept the default value of y here:

Run smoke tests? (y|n) (y): 

6. Save these settings?

The final prompt is to ask you whether the default answers for the prompts should be updated to use your current selection. If you answer "yes" (y), then the values will be written to the text file ~/.ciaoinstall.rc and will be used the next time the tool is run. In this case we accept the default value:

Save these settings? (y|n) (y): 

7. Requested packages

After downloading conda, the script prints the list of packages it will install:

Installing: ciao sherpa contrib caldb

This example is installing the standard installation: CIAO tools, Sherpa, the CALDB (except for the blank-sky files), and contrib (the science scripts).


8. Which build will be installed

The script prints out a line indicating which CIAO build will be installed:

For Linux

At this point, the script begins to download the software.

Downloading environment files for Linux...
Installing segments...
Fixing install...
Fixing top to: /soft/ciao-4.16
DONE!

After the files are downloaded and unpacked, the script configures CIAO and reindexes the CIAO help files ("ahelp"):

CIAO configuration is complete... 
CIAO 4.16.0 Tuesday, December 05, 2023
  bindir      : /soft/ciao-4.16/bin
  CALDB       : 4.11.0
Re-indexing ahelp system

The script will then run the smoke tests. Since the smoke tests include tests of a number of the GUI applications in CIAO, windows will appear and disappear during these tests. This can make using your computer difficult, since these windows may take the mouse focus or overlap your working windows.

The screen output provides a count of how many tests have been run:

Running smoke tests
 Requested test suite includes 37 tests
   [1/37] Running test ahelp-smoke001 ... PASS
   [2/37] Running test dm-smoke001 ... PASS
   [3/37] Running test dm-smoke002 ... PASS
...

This will continue until all the tests have finished, at which point you should see

   [36/37] Running test tools-tre-smoke001 ... PASS
   [37/37] Running test tools-wav-smoke001 ... PASS
 37 smoke tests run: 37 PASSED, 0 FAILED, 0 SKIPPED

Verifying test successes

A successful test is logged with a "... PASS", while an unsuccessful test reports:

   [7/37] Running test <testname> ... FAIL
         Review /tmp/smoke.username/<testname>/diff.log for details

The smoke tests print a summary of successful and failed tests. If all tests were successful:

 37 smoke tests run: 37 PASSED, 0 FAILED, 0 SKIPPED

or if any failed:

 37 smoke tests run: 36 PASSED, 1 FAILED, 0 SKIPPED

       !! CIAO MAY HAVE FAILED TO INSTALL PROPERLY !!
 Please check http://cxc.harvard.edu/ciao/bugs/smoke.html
 for a list of known issues before contacting the Help Desk.

Finished!

The script finished by printing the installation location.

To use the install:
	(t)csh: $> source /soft/ciao-4.16/bin/ciao.csh
	bash/zsh: $> source /soft/ciao-4.16/bin/ciao.sh

Known Smoke Test Issues

There is a bug page for known installation and smoke test issues. If you experience a problem that isn't listed there, submit it to CXC Helpdesk.

You can now proceed to the creating a CIAO alias step.


Create a CIAO alias

It is strongly suggested that you create an alias to make it easy to set up and use CIAO.

The syntax for the alias command depends on the shell (tcsh/csh, bash/zsh/ksh); see this FAQ for help in determining which shell you are using. The actual command to alias will depend on whether CIAO was installed using conda install or with the ciao-install script.

Bourne shell and variants: bash, dash, zsh, and ksh

If CIAO was installed using conda create, then the alias command will be either

alias ciao="conda activate ciao-4.16"
  -or-
alias ciao="source activate ciao-4.16"

Which one depends on how conda is setup on your system.

If CIAO was installed using the ciao-install script then the alias command is

alias ciao="source /soft/ciao-4.16/bin/ciao.sh"

Users should add these alias commands to their startup scripts: bash use $HOME/.bashrc, dash use $HOME/.profile, zsh use $HOME/.zshrc, and ksh use $HOME/.kshrc.

C shell and variants: tcsh

If CIAO was installed using conda create, then the alias command will be

alias ciao "conda activate ciao-4.16"

If CIAO was installed using the ciao-install script then the alias command is

alias ciao "source /soft/ciao-4.16/bin/ciao.csh"

Users should add these alias commands to their startup scripts: csh use $HOME/.cshrc, tcsh use $HOME/.tcshrc (or $HOME/.cshrc if already present).

Users can create the startup file if one does not already exist. These are simply text files that can be edited with the users favorite text editor (emacs, vi, gedit, etc.). The startup files are sourced when you open a new terminal; if you want to start using the alias in a terminal window that is already open then you have to manually source it.

$ source FILENAME

More information is given in the Starting CIAO thread.

Once the alias is defined, you will be able to simply type

unix% ciao

and CIAO will be setup and ready to use in that window.


Clean Up

Finally, you may delete the tarfiles from the original download directory. For this example we would say:

unix% cd /soft/download
unix% /bin/rm -rf conda_constructors conda_installers conda_loc-linux-64 env_files

You may also delete the log file(s) - e.g. ciao-install-<date>.log - once you have reviewed them for errors.


Customizing installation

The ciao-install script has several options. Use --help to review them.

unix% bash /path/ciao-install --help
Usage: ciao-install [options...]

  -h --help          Print this message
  --download-only    Download only (do not install)
  --install-only     Install only (do not download)
  --cache <dir>      Cache directory to use
  --pkg-cache <dir>  (Optional) Override the default package cache
  --prefix <dir>     Install directory
  --with-top <dir>   Fix the install to think it is here
  --link-caldb <dir> Location of CALDB to link in
  --system <system>  System to install
     (linux, macos, or macos-arm)
  --delete-cache     Delete the download cache
  --add <segment>    Add additional segment to CIAO
  --remove <segment> Skip segment.
  --skip-tests       Skip tests after install
  --small-files      Use smaller files to install where possible.
  --force            Overwrite existing install
  --update           Update existing install
  --with-top <dir>   (non-standard) Fix the init scripts
                     to think they are elsewhere
                     (eg. cluster install)
  --download-all <dir> 
                     (non-standard) Generates a complete
                     cache of all products required for install
                     for all supported platforms.
                     Forces --download-only
  --logdir <dir>     Log file directory (defaults to current dir)
  -v, --verbose      Includes more progress output to the screen

Automating installation

Without any command line arguments, ciao-install will prompt the user for answers to several questions. These can be specified on the command line to automate the installation process

unix% yes | bash ciao-install --prefix /soft --cache /soft/download

This will download the conda packages into the /soft/download directory and will install CIAO into /soft/ciao-4.16 directory.


Network installation with different directory mounts

For users installing CIAO in certain network configurations where the install directory is mounted under a different names on host systems, the --with-top option can be used to configure the installation. For example if CIAO is to be installed in /net/ciao-4.16 which is mounted as /soft/ciao-4.16, then

unix% bash ciao-install --prefix /net --with-top /soft --cache /soft/download

Install ACIS and/or HRC blank sky background CALDB files

By default, the ciao-install script does not download neither the ACIS nor the HRC blank sky background CALDB files. To include those packages, use the --add option as shown below:

unix% bash ciao-install --add acis 
  -or-
unix% bash ciao-install --add hrc 
  -or-
unix% bash ciao-install --add acis --add hrc

Some users may wish to install the Chandra CALDB into a completely separate directory. Often this is done so that the same CALDB files can be used with different versions of CIAO, or by multiple users with separate CIAO installations.

To start, before installing CIAO, users will need to download the CALDB files directly from the Downloading CALDB page. For example, install the CALDB files into the /soft/CALDB directory you would

unix% mkdir -p /soft/CALDB
unix% cd /soft/CALDB

unix% wget https://cxc.cfa.harvard.edu/cdaftp/arcftp/caldb/caldb_4.11.0_main.tar.gz
  -or-
unix% curl -O https://cxc.cfa.harvard.edu/cdaftp/arcftp/caldb/caldb_4.11.0_main.tar.gz

unix% tar xfz caldb_4.11.0_main.tar.gz
unix% rm caldb_4.11.0_main.tar.gz   # (optionally cleanup when done)

Then use the --link-caldb option when running ciao-install

unix% bash ciao-install --link-caldb /soft/CALDB
…
Linking CALDB from: /soft/CALDB
To: /soft/ciao-4.16/CALDB
…

Install CIAO source code

The CIAO and Sherpa source code can also be installed using the --add option

unix% bash ciao-install --add ciao-src
   -or-
unix% bash ciao-install --add sherpa-src
   -or-
unix% bash ciao-install --add ciao-src --add sherpa-src

Details about building CIAO from the source code are available in the Building CIAO 4.16 from source thread. Note: "ciao-src" actually downloads two directories: ciao-src and pyciao-src.


Install for different operating system

By default, ciao-install will download the files appropriate for the operating system where it is run. To download the files for a different operating system use the --system command line argument

unix% bash ciao-install --system linux 
   -or-
unix% bash ciao-install --system macos       # Intel/x86
   -or-
unix% bash ciao-install --system macos-arm   # ARM/M1/M2/M3

Note that the smoke tests cannot be run when installing CIAO for a different operating system. The tests can be manually run on the proper OS by setting up for CIAO and running the run_smoke_tests.sh script.

unix% ciao
unix% cd $ASCDS_INSTALL
unix% bash test/smoke/bin/run_smoke_tests.sh

Also, the ahelp files will not be indexed; things like the See Also section and keyword searching will not be available. To manually create the index files on the proper OS run the ahelp -r command:

unix% ciao
unix% ahelp -r

Install a subset of CIAO packages

ciao-install will, by default, install four "segments" ciao (ciao tools and libraries), ciao-contrib (ciao contributed scripts), sherpa, and caldb (standard CALDB without blank sky files). Users can use the --remove command line argument to omit any of the ciao-contrib, sherpa, or caldb segments:

unix% bash ciao-install --remove sherpa …
   -or-
unix% bash ciao-install --remove ciao-contrib …
   -or-
unix% bash ciao-install --remove caldb …

unix% bash ciao-installl --remove sherpa --remove ciao-contrib --remove caldb …

Relocating Installation

It is strongly recommended that users not relocate their CIAO installation after it has been installed. If this is unavoidable, then users can follow the steps outlined below.

  1. After CIAO has been installed and before it is moved/relocated, users need to run the ciao-configure script to reconfigure their installation:

    unix% cd $ASCDS_INSTALL
    unix% bash bin/ciao-configure /<New_Path>/
    Fixing top to: <New/Path/>
    DONE!
    

    Note: <New/Path/> does not need to exist at the time ciao-configure is run.

  2. After it has been reconfigured, then the CIAO installation can be moved

    unix% cd $HOME
    unix% mv $ASCDS_INSTALL /<New_Path>/
    
  3. Users can then source the CIAO setup script an use CIAO from the new location

    unix% source /<New_Path>/bin/ciao.sh -o     # or ciao.csh for tcsh shell users
    CIAO configuration is complete... 
    The CIAO setup for this window has changed from
    CIAO 4.16.0 Tuesday, December 05, 2023
      bindir      : /soft/ciao-4.16/bin
      CALDB       : 4.11.0
    To:
    CIAO 4.16.0 Tuesday, December 05, 2023
      bindir      : /opt/ciao-4.16/bin
      CALDB       : 4.11.0
    

Alternative Download Options

Users on slow internet or wireless connections may experience problems downloading the large data files associated with the Chandra CALDB and the XSPEC models. Often users will be able to simply retry again successfully; however, sometimes the server timeouts are persistent.

Installing CIAO via conda allows for more customization of the download process which may work better for slower connections. However, if the ciao-install format is necessary, the necessary files are also available to download from Google Drive and install manually.

Using ciao-install via Google Drive

Start by downloading the constructor for your system and installing it into a ciao-4.16 directory. Ignore the ahelp warning this triggers. Then, run the following commands in the terminal:

If using macOS:

unix% bash ciao-4.16.0-macOS.sh -b -p ciao-4.16

If using Linux:

unix% bash ciao-4.16.0-Linux.sh -b -p ciao-4.16

Then for either system, navigate to the ciao-4.16 directory and run the ciao-configure script:

unix% cd ciao-4.16/
unix% ./bin/ciao-configure

If you are using a bash/zsh shell, activate CIAO by running:

unix% source bin/ciao.sh

If you are using (t)csh, activate CIAO by running:

unix% source bin/ciao.csh

With CIAO activated, re-index ahelp as follows:

unix% ahelp -r

Note that this approach does not install CALDB. You may download CALDB either from the Download CALDB page or Google Drive. The Google Drive downloads are less likely to time out. CALDB will then need to be unpacked into a CALDB directory under that ciao-4.16 directory created by the installation.


Summary

The CIAO installation is now complete. For help getting started with the software, read the Introductory science threads.


History

20 Apr 2009 The CIAO 4.1.2 release coincides with the beta release of the ciao-install script.
16 Jun 2009 clarified why ciao-install must be run twice to install CALDB in a separate location
15 Dec 2009 updated for the CIAO 4.2 release: ciao-install is the recommended method for installing CIAO. Many enhancements have been made to the script: it now downloads and unpacks the tarfiles, installs the software, and runs the smoke tests.
29 Dec 2009 Added Installing on Snow Leopard note
16 Mar 2010 Added Downloading the CALDB: FTP passive mode note
05 Apr 2010 updated for CALDB 4.2.1: version number of CALDB main tarfile
19 Apr 2010 updated for CALDB 4.2.2: version number of CALDB main tarfile
27 Sep 2010 updated for CALDB 4.3.1: version number of CALDB main
15 Dec 2010 updated for the CIAO 4.3 and CALDB 4.4.1 releases: minor changes to screen output
15 Dec 2011 reviewed for the CIAO 4.4 and CALDB 4.4.7 releases: minor changes to screen output
13 Jun 2012 reviewed for the CIAO 4.4.1 and CALDB 4.4.10 releases: minor changes to screen output
03 Dec 2012 Review for CIAO 4.5
19 Nov 2013 Updated for CIAO 4.6. Cosmetic updates
09 May 2014 Re-order sections. Moved creating ciao alias before check caldb. Added note that check_caldb requires the contrib package to be installed.
11 Dec 2014 Updated for CIAO 4.7. Version updates and minor text edits.
29 Jun 2015 added additional information for passive FTP mode.
15 Dec 2015 Updated for CIAO 4.8. Added information about new CALDB flags.
09 Feb 2016 Updated for CIAO 4.8.1.
24 Feb 2016 Updated for CIAO 4.8.1/4.8.2.
27 May 2016 updated for CALDB 4.7.2: version number of CALDB main
04 May 2017 updated for CALDB 4.7.4.
20 Jul 2017 updated for CALDB 4.7.5.1 and contrib 4.9.4.
18 Aug 2017 updated for CALDB 4.7.6.
13 Dec 2017 updated for CALDB 4.7.7.
09 Apr 2018 Updated for CIAO 4.10, CALDB 4.7.8.
11 Sep 2018 updated for CALDB 4.8.0.
25 Oct 2018 updated for CALDB 4.8.1.
03 Sep 2019 updated for CALDB 4.8.4.
11 Dec 2019 CIAO 4.12 updates.
03 Sep 2019 updated for CALDB 4.9.1 and contrib 4.12.2.
15 Oct 2020 updated for CALDB 4.9.3 and contrib 4.12.4.
16 Dec 2020 updated for CIAO 4.13, contrib 4.13.0, caldb 4.9.4
29 Apr 2021 updated for CALDB 4.9.5 and contrib 4.13.1.
22 Nov 2021 Updated for CIAO 4.14.
30 Nov 2023 The ciao-install script has been completely re-written to make use of the conda package manager. This thread has been significantly updated to reflect the changes to how CIAO is now being distributed.
11 Mar 2024 Added instructions for downloading the ciao-install script via Google Drive as an alternative for users with slow connections.