Last modified: December 2022

URL: https://cxc.cfa.harvard.edu/ciao/ahelp/download_chandra_obsid.html
AHELP for CIAO 4.16

download_chandra_obsid

Context: Tools::Download

Synopsis

Download public data by ObsId from the Chandra archive.

Syntax

download_chandra_obsid obsid
download_chandra_obsid obsid1,..,obsidN
download_chandra_obsid obsid filetype
download_chandra_obsid obsid1,..,obsidN filetype1,..,filetypeN

The supported file types are:
adat aoff aqual arf asol bias bpix cntr_img dtf eph0 eph1 evt1 evt1a
evt2 flt fov full_img msk mtl oif osol pbk pha2 plt readme rmf soff
src2 src_img stat sum vv vvref

The --exclude flag allows you to skip a file type.
The -m or --mirror flags allow you to use a mirror of the Chandra Data
Archive.
The -h or --help flags displays information on the command-line options.
The -q or --quiet flags is used to turn off screen output.

Description

The download_chandra_obsid tool will download the data for the given Chandra Observation Id (ObsId) from the public archive. It can not be used to download proprietary data. Multiple values can be given - for both the ObsId and filetype lists - either as a comma-separated list or in a separate file (using the "@" syntax as described in 'ahelp stack').

Where are the files placed?

The files are stored in the current working directory, using the Observation Id as a directory name: so the data for ObsId 1843 would be placed in the directory 1843/. Within this directory the files are placed as they are in the archive, namely a top-level directory containing the oif.fits and V&V report, with the remaining data in the primary/ and secondary/ sub-directories.

How do I download a specific file?

The optional second argument to download_chandra_obsid allows you to restrict the download to one or more files. A comman-separated list (or stack) of file types can be given, so

unix% download_chandra_obsid 1843 evt2,asol,bpix,fov

will just download the evt2, asol, bpix, and fov files for ObsId 1843.

What happens when files already exist?

The tool will skip downloading any files that already exist on disk as long as they are the correct size. If the on-disk version is smaller than the archive version, then download will be resumed rather than started again. A warning message will be displayed if the on-disk version is larger than the archive version.

The behavior described above means that you can interrupt a download and re-start it and will not have to re-download existing data.

Screen output

Unless the -q or --quiet flag was used, each file that is downloaded will be displayed on screen, giving the "type", format, file type before a progress bar of # marks (each # mark indicates 5% of the file). Once the file has been downloaded an average rate is displayed for the download. Examples of the output for two files are shown below:

Type     Format      Size  0........H.........1  Download Time Average Rate
---------------------------------------------------------------------------
evt2     fits       17 Mb  ####################            2 s  7170.3 kb/s
vv       pdf        78 Kb  ####################          < 1 s  843.1 kb/s

If a file has already been downloaded then the message "already downloaded" will be displayed instead of the progress bar.


Examples

Example 1

unix% download_chandra_obsid 1842

This will download all the data files for ObsId 1842 to the directory 1842/. The screen output for the run will look something like the following (where many lines have been excluded, indicated by the "..." lines):

Downloading files for ObsId 1842, total size is 81 Mb.

Type     Format   Size  0........H.........1  Download Time Average Rate
------------------------------------------------------------------------
evt1     fits    36 Mb  ####################           46 s  818.4 kb/s
evt2     fits    19 Mb  ####################           23 s  809.1 kb/s
...
readme   ascii   11 Kb  ####################          < 1 s  91.7 kb/s
eph1     fits     6 Kb  ####################          < 1 s  109.7 kb/s
fov      fits     6 Kb  ####################          < 1 s  108.1 kb/s
msk      fits     5 Kb  ####################          < 1 s  62.0 kb/s
pbk      fits     4 Kb  ####################          < 1 s  81.4 kb/s

    Total download size for ObsId 1842 = 81 Mb
    Total download time for ObsId 1842 = 1 m 45 s

Example 2

unix% download_chandra_obsid 1842 evt1,bpix,asol,fov,readme

Here we restrict the download to just the evt1, bpix, asol, fov, and readme files from ObsId 1842.

Example 3

unix% download_chandra_obsid 1842 --exclude vvref

Download all but the vvref2.pdf file for obsid 1842. This file contains the detailed V&V data for the observation and can be large.

Example 4

unix% download_chandra_obsid 1842,1843

This will download the data for ObsIds 1842 and 1843 to the directories 1842/ and 1843/ respectively.

Example 5

unix% cat obsids
1842
1843
unix% download_chandra_obsid @obsids

Here we use the support for stacks (see 'ahelp stack' for more information) to download the data for the ObsIds listed in the file obsids, in this case 1842 and 1843.

Example 6

unix% download_chandra_obsid 1842,1843 evt1,asol -q

Here we download only the evt1 and asol files from the two ObsIds and use the -q flag to turn off the screen output.

Example 7

unix% download_chandra_obsid -m https://cxc.cfa.harvard.edu/cdaftp/ 1842

Download ObsId 1842 from the given CDA mirror site (in this example we have used the location of the actual CDA archive). If the observation is not found at the mirror site then the script will skip it (i.e. it does not fall back to trying the CDA site).

An alternative is to use the CDA_MIRROR_SITE environment variable, so that the above could also be written as:

unix% setenv CDA_MIRROR_SITE https://cxc.cfa.harvard.edu/cdaftp/
unix% download_chandra_obsid 1842

Mirror sites of the Chandra Data Archive

If you have set up, or have access to, a mirror of the Chandra Data Archive (it can be only a partial archive), then you can use the --mirror command-line argument, or the CDA_MIRROR_SITE environment variable, to tell the script to use this mirror instead. The mirror option takes precedence over CDA_MIRROR_SITE if both are set.

If an ObsId can not be found on the mirror site then it is skipped; i.e. the script will not also check the CDA site for the data.

The mirror location should include the URL of the HTTP or HTTPS site (the leading https:// or http:// is required) and the path up to, but not including, the byobsid/ directory. So, for the Chandra Data Archive itself you would use

https://cxc.cfa.harvard.edu/cdaftp/

(although obviously in this case you do not need to use the mirror option). Please see the documentation for the mirror site to find out the correct path to use.

Changes in the scripts 4.13.1 (March 2021) release

Validation and Verification files

The Chandra archive contains two V&V files for an observation:

where the first file is intended for users, and the second is only needed in rare situations. The vvref filetype has been added to select this file, and the vv filetype now only refers to the vv2 file.

Excluding file types

The --exclude flag has been added to allow you to ignore certain file types, so to avoid downloading the vvref2 file you can say

download_chandra_obsid 1843 --exclude vvref

Changes in the scripts 4.13.0 (December 2020) release

The script should be more robust to changes made to the Chandra Data Archive web site.

Changes in the scripts 4.12.2 (April 2020) release

Archive has moved

The script now uses the HTTPS service provided by the Chandra Data Archive as the FTP service is being retired. The archive is now available at https://cxc.cfa.harvard.edu/cdaftp/

Order by file size

For each observation, the files are now sorted by decreasing file size before being downloaded.

Support for ARF and RMF access

The next reprocessing of the Chandra archive will add grating ARF and RMF products, so "arf" and "rmf" have been added to the list of supported file types. Note that it will take some time before these files appear in the archive (as the reprocessing has not started yet). Please review the Archive status page for information on data products.

Changes in the scripts 4.9.3 (May 2017) release

Updated to use ftp://cda.cfa.harvard.edu as the default FTP server.

Changes in the scripts 4.8.1 (December 2015) release

The code has been updated to avoid warning messages from NumPy version 1.9. There is no difference to how the script behaves.

Changes in the scripts 4.6.5 (June 2014) release

Added evt1a and adat file types

It is now possible to select evt1a and adat (PCAD Level 1 ACA image) files using the filetype argument. The adat files can be used with the monitor_photom script.

Changes in the scripts 4.5.4 (August 2013) release

Support for CDA mirror sites

The --mirror command-line argument and support for the CDA_MIRROR_SITE environment variable has been added to allow data access from a mirror of the Chandra Data Archive site.

Support for the 00README files

The new 00README files are included in the download, and can be explicitly selected using the readme filetype.

Notes

This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see this page for installation instructions.


Bugs

There are no known bugs for this tool.

See Also

contrib
cda_data, cda_search
tools::download
download_obsid_caldb, find_chandra_obsid, obsid_search_csc, search_csc
tools::utilities
splitobs