Skip to the navigation links
Last modified: 16 December 2019

URL: https://cxc.cfa.harvard.edu/ciao/scripting/index.html

Using Python with CIAO


CIAO provides a number of Python modules that support data analysis using CIAO in Python, with the main ones being:

crates

The Python interface to the CIAO Data Model. This module supports reading and writing FITS and ASCII files. The transform module provides support for coordinate conversion (linear and WCS) for tables and images.

Sherpa

The CIAO modeling and fitting package.

More information on Python support in CIAO can be found in "ahelp python", and there is limited customization support as described in users may opt to run their own installation instead; see "ahelp ciaorc".

[NOTE]
Note

We strongly suggest that users install Python packages into their CIAO environment, rather than trying to install CIAO into their Python environent. This has been made significantly easier in CIAO 4.12 with the ability to install CIAO via the conda package manager for Python versions 3.7, 3.6, or 3.5.


What Python packages are provided with CIAO?

The Python packages installed with CIAO depends on how CIAO was installed - whether ciao-install or the conda package manager was used - as well as what packages were installed and, for the conda version, the version of Python.

conda [New]

When using conda, the ciaover command lists the "base" CIAO packages that have been installed (in the example below the CIAO CALDB has not been installed):

unix% ciaover
# packages in environment at /home/ciaouser/anaconda3/envs/ciao412:
#
# Name                    Version                   Build  Channel
ciao                      4.12.0           py37h04863e7_0    https://cxc.cfa.harvard.edu/conda/ciao
ciao-contrib              4.12.0                     py_1    https://cxc.cfa.harvard.edu/conda/ciao
ds9                       8.1                           1    https://cxc.cfa.harvard.edu/conda/ciao
sherpa                    4.12.0           py37h04863e7_0    https://cxc.cfa.harvard.edu/conda/ciao

System information:
Linux <... lots of important information useful for helpdesk questions ...>

A full list of packages can be shown with conda list and pip3 list. The output depends on the OS in use (Linux or macOS), what version of Python was selected (3.5, 3.6, 3.7), and what other packages are installed.

ciao-install

The pip3 tool can be used to report on the Python packages installed in CIAO. For the CIAO 4.12 Linux build this reports (where the major packages are shown in bold):

unix% pip3 list --format=columns
Package            Version
------------------ -------
atomicwrites       1.3.0  
attrs              19.1.0 
backcall           0.1.0  
bash-kernel        0.7.2  
bleach             3.1.0  
caldb4             0.1.0  
ciao-contrib       4.12.0 
cycler             0.10.0 
Cython             0.28.5 
decorator          4.4.0  
defusedxml         0.6.0  
entrypoints        0.3    
importlib-metadata 0.19   
ipykernel          5.1.1  
ipython            7.2.0  
ipython-genutils   0.2.0  
ipywidgets         7.5.1  
jedi               0.14.1 
Jinja2             2.10.1 
jsonschema         3.0.2  
jupyter            1.0.0  
jupyter-client     5.3.1  
jupyter-console    6.0.0  
jupyter-core       4.5.0  
kiwisolver         1.1.0  
MarkupSafe         1.1.1  
matplotlib         3.0.2  
mistune            0.8.4  
more-itertools     7.2.0  
nbconvert          5.5.0  
nbformat           4.4.0  
notebook           6.0.0  
numpy              1.15.4 
pandocfilters      1.4.2  
parso              0.5.1  
pathlib2           2.3.4  
pexpect            4.7.0  
pickleshare        0.7.5  
pip                19.3.1 
pixlib             1.0.0  
pluggy             0.12.0 
prometheus-client  0.7.1  
prompt-toolkit     2.0.9  
ptyprocess         0.6.0  
py                 1.8.0  
Pygments           2.4.2  
pyparsing          2.4.2  
pyrsistent         0.15.4 
pytest             4.0.2  
python-dateutil    2.8.0  
pyzmq              18.0.2 
qtconsole          4.5.2  
Send2Trash         1.5.0  
setuptools         28.8.0 
sherpa             4.12.0 
six                1.12.0 
terminado          0.8.2  
testpath           0.4.2  
tornado            6.0.3  
traitlets          4.3.2  
wcwidth            0.1.7  
webencodings       0.5.1  
widgetsnbextension 3.5.1  
zipp               0.5.2

How do I install a Python package into CIAO?

The method for installing Python packages depends on how CIAO was installed. Although the conda installation method is new with CIAO 4.12, it is the suggested approach for users who want to install Python packages alongside CIAO.

[WARNING]
Updating versions

CIAO depends on a number of libraries and modules (e.g. NumPy), and it has only been tested against the versions available at the time of release. The CXC does not guarantee that CIAO will work correctly if these packages are updated.

Please take care when installing software alongside CIAO as it can result in updating these dependencies.

conda [New]

As you are within a standard conda environment, packages can be installed with conda or pip3.

Conflicts with astroconda

The CIAO installation provides a number of packages that may conflict with those from AstroConda, in particular DS9 and cfitsio. At the time of writing (December 2019), the difference in cfitsio package is known to cause problems. It is suggesttd that at this time CIAO not be combined with AstroConda in the same environment.

ciao-install

For users with CIAO installed via ciao-install, packages should be installed with the pip3 tool, and it requires that you have write access to the CIAO installation directory. If this fails you can try installing with the --user flag.

[WARNING]
Ensure you have the original CIAO installation files

Installing Python packages into the CIAO installation area runs the risk of corrupting the CIAO system. The main concern is accidentally upgrading (or downgrading) a component that CIAO relies on (in particular NumPy. It is strongly suggested that you ensure you have the files that ciao-install downloaded when installing CIAO, so that CIAO can be re-installed if there is a problem.

How to avoid updating NumPy

The version of NumPy provided with CIAO 4.12 is version 1.15.4. Installing recent versions of Python packages - such as AstroPy or PyMC3 - may try to upgrade the installed NumPy. This is currently unsupported, so we suggest that a constraints file is used with pip3 to ensure that NumPy remains at version 1.15.4.

It is safest to not update any Python package provided with CIAO, which can be achieved with the freeze option:

unix% pip3 freeze > $ASCDS_INSTALL/constraints.txt
unix% grep numpy $ASCDS_INSTALL/constraints.txt
numpy==1.15.4

or you can just create a file with this - e.g.

unix% echo "numpy==1.15.4" > $ASCDS_INSTALL/constraints.txt

The location of the file isn't that important, but placing it in $ASCDS_INSTALL means that you have it stored in a memorable location!

This file is then used in a pip3 install line with the -c flag, as will be shown below. Note that not all packages require this set of constraints, but it's hard to know before hand, so it is simplest to always use it.

Example: installing corner

As an example, the following set of routines will install the corner package, which is used to visualize multidimensional samples (in particular the results of Markov Chain Monte Carlo simulations). This package was picked as it is small, Python only, and does not require installing any extra code. Note that the screen output the exact output depends on the version available at the time of installation, and details of the CIAO installation, so may differ slightly from that shown here):

unix% pip3 install -c $ASCDS_INSTALL/constraints.txt corner
Collecting corner
  Downloading https://files.pythonhosted.org/packages/65/af/a7ba022f2d5787f51db91b5550cbe8e8c40a6eebd8f15119e743a09a9c19/corner-2.0.1.tar.gz
Requirement already satisfied: numpy==1.15.4 in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.12/constraints.txt (line 33)) (1.15.4)
Requirement already satisfied: matplotlib==3.0.2 in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.12/constraints.txt (line 27)) (3.0.2)
Requirement already satisfied: python-dateutil==2.8.0 in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.12/constraints.txt (line 49)) (2.8.0)
Requirement already satisfied: cycler==0.10.0 in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.12/constraints.txt (line 8)) (0.10.0)
Requirement already satisfied: kiwisolver==1.1.0 in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.12/constraints.txt (line 25)) (1.1.0)
Requirement already satisfied: pyparsing==2.4.2 in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.12/constraints.txt (line 46)) (2.4.2)
Requirement already satisfied: six==1.12.0 in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.12/constraints.txt (line 54)) (1.12.0)
Requirement already satisfied: setuptools in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from kiwisolver==1.1.0->-c /soft/ciao-4.12/constraints.txt (line 25)) (28.8.0)
Installing collected packages: corner
    Running setup.py install for corner ... done
Successfully installed corner-2.0.1

As a final check we see whether we can import the module (the name used in the import statement depends on how the package is designed and need not match the installation name):

unix% python -c 'import corner'

Example: installing AstroPy and SciPy

Following the previous example, we can install AstroPy and SciPy by saying:

unix% pip3 install -c $ASCDS_INSTALL/constraints.txt astropy scipy
Collecting astropy
  Downloading https://files.pythonhosted.org/packages/25/11/379c5403f59c69f291f962c27c1dc1948f6eb93fd0b63b5473c462ccc53c/astropy-3.2.3-cp35-cp35m-manylinux1_x86_64.whl (6.3MB)
     |################################| 6.3MB 6.5MB/s 
Collecting scipy
  Downloading https://files.pythonhosted.org/packages/1e/9c/4d7a573d49c449c95d33df6648a62a522ec46403f13ffe2c8cd3f094f746/scipy-1.3.3-cp35-cp35m-manylinux1_x86_64.whl (25.1MB)
     |################################| 25.1MB 18.9MB/s 
Requirement already satisfied: numpy==1.15.4 in /soft/ciao-4.12/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.12/constraints.txt (line 33)) (1.15.4)
Installing collected packages: astropy, scipy
Successfully installed astropy-3.2.3 scipy-1.3.3

Problems when compiling code

The steps above work when the module is "pure Python", such as the corner package, or needs minimal compilation on the local machine (e.g. AstroPy). There can be problems when installing modules which require access to system-level - or CIAO-provided - code. Please contact the CXC Helpdesk as we may be able to help in these situations.

Have we broken anything?

It is possible to get into a state where incompatible packages have been installed, so it is worth periodically checking your CIAO installation with the following command:

unix% pip3 check
No broken requirements found.

If there is a problem then it may be possible to remove or change the version of the affected package, or packages, or it may require re-installing CIAO.


About Python

Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools and comes with extensive standard libraries. A list of key software features is available on the About Python webpage.