Using Python with CIAO
- What Python packages are provided with CIAO?
- How do I install a Python package into CIAO?
- About Python
CIAO provides a number of Python modules that support data analysis using CIAO in Python, with the main ones being:
- crates
-
The Python interface to the CIAO Data Model. This module supports reading and writing FITS and ASCII files. The transform module provides support for coordinate conversion (linear and WCS) for tables and images.
- Sherpa
-
The CIAO modeling and fitting package.
More information on Python support in CIAO can be found in "ahelp python", and there is limited customization support as described in users may opt to run their own installation instead; see "ahelp ciaorc".
![[NOTE]](../imgs/note.png)
We strongly suggest that users install Python packages into their CIAO environment, rather than trying to install CIAO into their Python environment.
What Python packages are provided with CIAO?
The Python packages installed with CIAO depends on how CIAO was installed - whether ciao-install or the conda package manager was used - as well as what packages were installed and, for the conda version, the version of Python.
conda
When using conda, the ciaover command lists the "base" CIAO packages that have been installed (in the example below the CIAO CALDB has not been installed):
unix% ciaover # packages in environment at /export/miniconda/envs/ciao-4.15: # # Name Version Build Channel caldb4 0.1.0 pypi_0 pypi ciao 4.15.2 py310h50a7cbf_0 https://cxc.cfa.harvard.edu/conda/ciao ciao-contrib 4.15.3 py_1 https://cxc.cfa.harvard.edu/conda/ciao ds9 8.4.1 0 https://cxc.cfa.harvard.edu/conda/ciao sherpa 4.15.0 pypi_0 pypi System information: Linux <... lots of important information useful for helpdesk questions ...>
A full list of packages can be shown with conda list and pip3 list. The output depends on the OS in use (Linux or macOS), what version of Python was selected (3.10, 3.9, or 3.8), and what other packages are installed.
ciao-install
The pip3 tool can be used to report on the Python packages installed in CIAO. For the CIAO 4.15.2 Linux build this reports (where the major packages are shown in bold):
unix% pip3 list --format=columns Package Version ----------------------------- ----------- anyio 3.6.2 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 asttokens 2.2.1 attrs 22.1.0 backcall 0.2.0 backports.functools-lru-cache 1.6.4 beautifulsoup4 4.11.1 bleach 5.0.1 caldb4 0.1.0 certifi 2022.12.7 cffi 1.15.1 comm 0.1.2 contourpy 1.0.6 cycler 0.11.0 debugpy 1.6.4 decorator 5.1.1 defusedxml 0.7.1 entrypoints 0.4 executing 1.2.0 fastjsonschema 2.16.2 flit_core 3.8.0 fonttools 4.38.0 idna 3.4 importlib-metadata 5.1.0 importlib-resources 5.10.1 ipykernel 6.19.2 ipython 8.7.0 ipython-genutils 0.2.0 ipywidgets 8.0.3 jedi 0.18.2 Jinja2 3.1.2 jsonschema 4.17.3 jupyter 1.0.0 jupyter_client 7.4.8 jupyter-console 6.4.4 jupyter_core 5.1.0 jupyter-events 0.5.0 jupyter_server 2.0.1 jupyter_server_terminals 0.4.2 jupyterlab-pygments 0.2.2 jupyterlab-widgets 3.0.4 kiwisolver 1.4.4 MarkupSafe 2.1.1 matplotlib 3.6.2 matplotlib-inline 0.1.6 mistune 2.0.4 munkres 1.1.4 nbclassic 0.4.8 nbclient 0.7.2 nbconvert 7.2.6 nbformat 5.7.0 nest-asyncio 1.5.6 notebook 6.5.2 notebook_shim 0.2.2 numpy 1.23.5 packaging 22.0 pandocfilters 1.5.0 parso 0.8.3 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.2.0 pip 22.3.1 pixlib 1.0.0 pkgutil_resolve_name 1.3.10 platformdirs 2.6.0 ply 3.11 prometheus-client 0.15.0 prompt-toolkit 3.0.36 psutil 5.9.4 ptyprocess 0.7.0 pure-eval 0.2.2 pycparser 2.21 Pygments 2.13.0 pyparsing 3.0.9 PyQt5 5.15.7 PyQt5-sip 12.11.0 pyrsistent 0.19.2 python-dateutil 2.8.2 python-json-logger 2.0.1 PyYAML 6.0 pyzmq 24.0.1 qtconsole 5.4.0 QtPy 2.3.0 Send2Trash 1.8.0 setuptools 65.5.1 sherpa 4.15.0 sip 6.7.5 six 1.16.0 sniffio 1.3.0 soupsieve 2.3.2.post1 stack-data 0.6.2 terminado 0.17.1 tinycss2 1.2.1 toml 0.10.2 tornado 6.2 traitlets 5.7.0 typing_extensions 4.4.0 unicodedata2 15.0.0 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.4.2 wheel 0.38.4 widgetsnbextension 4.0.4 zipp 3.11.0
It is likely that you will see a note about pip being out of date, as it changes rapidly!
How do I install a Python package into CIAO?
The method for installing Python packages depends on how CIAO was installed. The recommended approach is to use a conda installation of CIAO.
![[WARNING]](../imgs/warning.png)
CIAO depends on a number of libraries and modules (e.g. NumPy), and it has only been tested against the versions available at the time of release. The CXC does not guarantee that CIAO will work correctly if these packages are updated.
Please take care when installing software alongside CIAO as it can result in updating these dependencies.
conda
As you are within a standard conda environment, packages can be installed with conda or pip3.
Conflicts with astroconda
The CIAO installation provides a number of packages that may conflict with those from AstroConda, in particular DS9 and cfitsio. At the time of writing (December 2020), the difference in cfitsio package is known to cause problems. It is suggesttd that at this time CIAO not be combined with AstroConda in the same environment.
ciao-install
For users with CIAO installed via ciao-install, packages should be installed with the pip3 tool, and it requires that you have write access to the CIAO installation directory. If this fails you can try installing with the --user flag.
Installing Python packages into the CIAO installation area runs the risk of corrupting the CIAO system. The main concern is accidentally upgrading (or downgrading) a component that CIAO relies on (in particular NumPy. It is strongly suggested that you ensure you have the files that ciao-install downloaded when installing CIAO, so that CIAO can be re-installed if there is a problem.
How to avoid updating NumPy
The version of NumPy provided with CIAO is version 1.20.3. Installing recent versions of Python packages - such as AstroPy or PyMC3 - may try to upgrade the installed NumPy. This is currently unsupported, so we suggest that a constraints file is used with pip3 to ensure that NumPy remains at version 1.20.3.
It is safest to not update any Python package provided with CIAO, which can be achieved with the freeze option:
unix% pip3 freeze > $ASCDS_INSTALL/constraints.txt unix% grep numpy $ASCDS_INSTALL/constraints.txt numpy==1.20.3
or you can just create a file with this - e.g.
unix% echo "numpy==1.20.3" > $ASCDS_INSTALL/constraints.txt
The location of the file isn't that important, but placing it in $ASCDS_INSTALL means that you have it stored in a memorable location!
This file is then used in a pip3 install line with the -c flag, as will be shown below. Note that not all packages require this set of constraints, but it's hard to know before hand, so it is simplest to always use it.
Example: installing corner
As an example, the following set of routines will install the corner package, which is used to visualize multidimensional samples (in particular the results of Markov Chain Monte Carlo simulations). This package was picked as it is small, Python only, and does not require installing any extra code. Note that the screen output the exact output depends on the version available at the time of installation, and details of the CIAO installation, so may differ slightly from that shown here):
unix% pip3 install -c $ASCDS_INSTALL/constraints.txt corner Collecting corner Using cached corner-2.1.0-py2.py3-none-any.whl (11 kB) Requirement already satisfied: setuptools>=40.6.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from corner) (41.2.0) Requirement already satisfied: matplotlib==3.2.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 25)) (3.2.0) Collecting wheel Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB) Collecting setuptools-scm Using cached setuptools_scm-5.0.1-py2.py3-none-any.whl (28 kB) Requirement already satisfied: numpy==1.18.1 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 31)) (1.18.1) Requirement already satisfied: python-dateutil==2.8.1 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 47)) (2.8.1) Requirement already satisfied: kiwisolver==1.2.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 23)) (1.2.0) Requirement already satisfied: cycler==0.10.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 6)) (0.10.0) Requirement already satisfied: pyparsing==2.4.7 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 44)) (2.4.7) Requirement already satisfied: six==1.14.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 53)) (1.14.0) Installing collected packages: wheel, setuptools-scm, corner Successfully installed corner-2.1.0 setuptools-scm-5.0.1 wheel-0.36.2
As a final check we see whether we can import the module (the name used in the import statement depends on how the package is designed and need not match the installation name):
unix% python -c 'import corner'
Example: installing AstroPy and SciPy
Following the previous example, we can install AstroPy and SciPy by saying:
unix% pip3 install -c $ASCDS_INSTALL/constraints.txt astropy scipy
Collecting astropy
Downloading astropy-4.2-cp37-cp37m-manylinux1_x86_64.whl (9.7 MB)
|################################| 9.7 MB 1.8 MB/s
Collecting scipy
Downloading scipy-1.5.4-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)
|################################| 25.9 MB 8.0 MB/s
Collecting pyerfa
Downloading pyerfa-1.7.1.1-cp37-cp37m-manylinux2010_x86_64.whl (696 kB)
|################################| 696 kB 16.5 MB/s
Requirement already satisfied: numpy==1.18.1 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 31)) (1.18.1)
Installing collected packages: pyerfa, astropy, scipy
Successfully installed astropy-4.2 pyerfa-1.7.1.1 scipy-1.5.4
Problems when compiling code
The steps above work when the module is "pure Python", such as the corner package, or needs minimal compilation on the local machine (e.g. AstroPy). There can be problems when installing modules which require access to system-level - or CIAO-provided - code. Please contact the CXC Helpdesk as we may be able to help in these situations.
Have we broken anything?
It is possible to get into a state where incompatible packages have been installed, so it is worth periodically checking your CIAO installation with the following command:
unix% pip3 check No broken requirements found.
If there is a problem then it may be possible to remove or change the version of the affected package, or packages, or it may require re-installing CIAO.
About Python
Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools and comes with extensive standard libraries. A list of key software features is available on the About Python webpage.
- Python Programming Language website
- Running CIAO tools from Python.
- AstroPython: Python for Astronomers