How to Download Chandra Data from the Archive
![[CXC Logo]](../../imgs/cxc-logo.gif)
CIAO 4.2 Science Threads
Overview
Last Update: 9 Nov 2010 - the download_chandra_obsid script is a quick and easy method of download data files.
Synopsis:
ChaSeR is a graphical user interface for browsing and/or obtaining data from the Chandra Data Archive. The Archive contains over 800 public-domain datasets, which consist of all observations for which the proprietary period has expired, as well as calibration data. Note that proprietary data may also be accessed via ChaSeR, but it is password-protected from the general community.
A script for downloading Chandra data from the
Archive: download_chandra_obsid.
To use the script, install the contrib
tarfile, then run:
unix% download_chandra_obsid 1843
Purpose:
To download ObsID 1843 (ACIS-I, G21.5-0.9), which is used in all of the Introductory CIAO Threads.
Contents
- Launching ChaSeR
- The Three Steps to Using ChaSeR
- FTP and Unpack the Data
- History
-
Images
- Figure 1: The ChaSer GUI
- Figure 2: The complete form to search for ACIS-I observations of G21.5-0.9
- Figure 3: The Search Results window for ACIS-I observations of G21.5-0.9
- Figure 4: Details of the observation
- Figure 5: Preview images of the observation
- Figure 6: The retrieval list
- Figure 7: Browse Results window
- Figure 8: Completed download screen
Launching ChaSeR
Before beginning, download and install the Java application.
Once installed, launch ChaSeR from the command line:
unix% chaser &
Figure 1 shows the GUI that will appear.
![[Thumbnail image: ChaSer, the GUI used download data from the Chandra Archive]](gui.thumb500.png)
[Version: full-size]
![[Print media version: ChaSer, the GUI used download data from the Chandra Archive]](gui.png)
Figure 1: The ChaSer GUI
The default settings of the ChaSer GUI when it is launched are to search all instrument configurations for all types and categories of observations.
WebChaSeR
If you do not wish to install ChaSeR, data may still be obtained through WebChaSeR, a web version of ChaSeR. While it has the same search capabilities as ChaSeR, it is not possible to select individual files for download. Using this interface, you can only select complete packages of primary and secondary products; see the "Select" step for information on the different packages.
Due to the slightly different functionality, portions of this thread - mainly the Three Steps to Using ChaSeR section - do not apply to users of WebChaSeR. The two applications are similar enough, however, that the thread may serve as a helpful guide.
The Three Steps to Using ChaSeR
Using ChaSeR to obtain data is a simple 3-step process:
1. Browse
Chandra Archive data may be browsed using any of the fields in the ChaSeR GUI, such as Instrument, Grating, ObsID, and Category. Since we are interested in an ACIS-I observation of G21.5-0.9, leave only the appropriate Instrument (ACIS-I) and Grating (NONE) options checked and enter the Target Name.
Figure 2 shows the completed form.
![[Thumbnail image: The Instrument (ACIS-I) and Grating (NONE) options are checked in the ChaSer GUI and the Target Name is set to "G21.5-0.9".]](form.thumb500.png)
[Version: full-size]
![[Print media version: The Instrument (ACIS-I) and Grating (NONE) options are checked in the ChaSer GUI and the Target Name is set to "G21.5-0.9".]](form.png)
Figure 2: The complete form to search for ACIS-I observations of G21.5-0.9
Click the "Submit" button to enter the request.
2. Select
The Search Results window brings up all the observations that match our search criteria, as shown in Figure 3.
![[Thumbnail image: The Search Results window lists all results of the search for ACIS-I observations of G21.5-0.9]](match.thumb500.png)
[Version: full-size]
![[Print media version: The Search Results window lists all results of the search for ACIS-I observations of G21.5-0.9]](match.png)
Figure 3: The Search Results window for ACIS-I observations of G21.5-0.9
Search results may be sorted by the "Order By..." menu.
The "Order By..." menu can be used to sort the results; here we have ordered them by "Observation ID."
Click on ObsID 1843 to highlight the row. After doing so one may:
-
Use the "Detail" button to display specifics about the observation (Figure 4).
[Version: full-size]
Figure 4: Details of the observation
Use the "Save" button to write this information to a text file.
-
Click on the "Images" button to see preview images; Figure 5.
[Version: full-size]
Figure 5: Preview images of the observation
Cycle through the available images with the "Prev Image" and "Next Image" buttons.
-
Write the table information to a file ("Save").
Now the desired data products may be selected for download using the "Select primary products", "Select secondary products", and "Select supporting products" buttons.
The data products are arranged such that the products necessary for most analyses (e.g. the CIAO threads) are in the primary directory. If you are interested in reprocessing your data, you will also need the level 1 files, which are in the secondary directory. Each of the threads also includes a list of the files needed to complete it.
By default, all primary and secondary products are pre-selected. If you are only interested in downloading the specific files need for a thread, but need some help determing what those are, check the Standard Data Distribution Contents webpage. These tables list the file extensions (aka "Type") and the "Content"; the content keyword is how the products are labeled in the ChaSeR interface.
The selections are submitted using the "Add to retrieval list" button. You can then highlight a different observation and add the same (or different) data products to the cart for it. Once you have selected all the data you are interested in, proceed to the next step.
3. Retrieve
Click on "View retrieval list" to view the observations and data products submitted for selection (Figure 6).
![[Thumbnail image: The retrieval list shows which files will be downloaded.]](retlist.thumb500.png)
[Version: full-size]
![[Print media version: The retrieval list shows which files will be downloaded.]](retlist.png)
Figure 6: The retrieval list
Since all the primary and secondary products were selected for retrieval, only the categories are listed. If specific files are chosen, they are listed individually.
One may see the list of data filenames and sizes (Figure 7) by clicking on the "Browse selected products" button.
![[Thumbnail image: The Browse Results window lists the individual files which will be downloaded.]](filesize.thumb500.png)
[Version: full-size]
![[Print media version: The Browse Results window lists the individual files which will be downloaded.]](filesize.png)
Figure 7: Browse Results window
The total file size of the package is listed at the top of the screen. Use this information to make sure there is sufficient disk space to download and unpack the data.
The "Retrieve selected products" button is used to start the retrieval process. The files are compiled into a tarred package which is placed onto the anonymous FTP server (cda.harvard.edu) by default; the tarfile can also be saved directly to a local disk. When the download is complete, the ChaSeR window will look as shown in Figure 8. Note that the location of the logfile (chaser.log) and the FTP directory in which the tarfile was placed are given at the bottom of the GUI.
![[Thumbnail image: The Completed download screen looks the same as the retrieval list, except the location of the tarfile on the FTP is added.]](download.thumb500.png)
[Version: full-size]
![[Print media version: The Completed download screen looks the same as the retrieval list, except the location of the tarfile on the FTP is added.]](download.png)
Figure 8: Completed download screen
When the download has completed, the location of the logfile (chaser.log) and the FTP directory in which the tarfile was placed are displayed at the bottom of the GUI.
The name and location of the tarfile is recorded in chaser.log, along with the ObsID that was downloaded:
unix% cat /export/chaser.log [09/01/2008 14:55:46:310 EST] Request retrieval of 1843 [09/01/2008 14:55:46:366 EST] File(s) package_146_080109145545.tar.* will be available at cda.harvard.edu:/pub/srftp/I6bjzJSZ
When you are finished with ChaSeR, use the "Exit" button to close the GUI.
FTP and Unpack the Data
Get the data from the anonymous FTP server; note that the location of the tarfile will be different for your observation:
unix% ftp cda.harvard.edu Connected to cda.harvard.edu (131.142.185.171). 220 cda2 FTP server (Version wu-2.6.2(1) Mon Oct 21 09:41:35 EDT 2002) ready. Name (cda.harvard.edu:username): anonymous 331 Guest login ok, send your complete e-mail address as password. Password: 230- 230- Welcome to the FTP server at the CXC Science Center 230- 230- This server is cda.harvard.edu 230- 230-If your FTP client crashes or hangs shortly after login please try 230-using a dash (-) as the first character of your password. This will 230-turn off the informational messages that may be confusing your FTP 230-client. 230- 230-Publicly available files are in /pub 230- 230-Problems with this ftp site? Contact cxcftp@head-cfa.harvard.edu 230- 230- +---------------------------------------------------------+ 230- | | 230- | PLEASE NOTE: (ChaSeR users only) | 230- | | 230- | If the directory provided to you by ChaSeR appears | 230- | to be empty, this is a sign that your tar file is | 230- | not yet ready. Please try again in a little while. | 230- | | 230- | If your directory seems to contain several tar | 230- | files, e.g., file.tar.0, file.tar.1, ... | 230- | retrieve all files, then untar as follows: | 230- | cat file.tar.* | tar xvf - | 230- | We are limiting the size of individual files to | 230- | less than 2 GB, but these files need to be | 230- | concatenated before submitting them to tar. | 230- | | 230- +---------------------------------------------------------+ 230- 230 Guest login ok, access restrictions apply. Remote system type is UNIX. Using binary mode to transfer files. ftp> cd /pub/srftp/I6bjzJSZ 250 CWD command successful. ftp> get package_146_080109145545.tar local: package_146_080109145545.tar remote: package_146_080109145545.tar 227 Entering Passive Mode (131,142,185,171,95,112) 150 Opening BINARY mode data connection for package_146_080109145545.tar (68024320 bytes). 226 Transfer complete. 68024320 bytes received in 9.55 secs (7e+03 Kbytes/sec) ftp> bye 221-You have transferred 68024320 bytes in 1 files. 221-Total traffic for this session was 68026426 bytes in 2 transfers. 221-Thank you for using the FTP service on cda2. 221 Goodbye.
Unpack the tarfile:
unix% tar xvf package_0_030616095338.tar 1843/primary/acisf01843N002_src2.fits.gz 1843/primary/acisf01843N002_evt2.fits.gz 1843/primary/acisf01843N002_1_sum2.html.gz 1843/primary/acisf01843N002_2_sum2.html.gz 1843/primary/acisf01843N002_3_sum2.html.gz . . (output omited) . 1843/secondary/acisf084271178N002_5_bias0.fits.gz 1843/secondary/axaff01843N001_VV001_vvref2.pdf.gz 1843/axaff01843N001_VV001_vv2.pdf.gz 1843/oif.fits unix% ls 1843/ package_146_080109145545.tar unix% rm package_146_080109145545.tar
If you downloaded data from multiple ObsIds, a directory will be created for each of them.
After gunzipping all the files, the directory looks like:
unix% cd 1843 unix% ls -1 * axaff01843N001_VV001_vv2.pdf oif.fits primary: acisf01843_000N002_bpix1.fits acisf01843_000N002_fov1.fits acisf01843N002_1_sum2.html acisf01843N002_2_sum2.html acisf01843N002_3_sum2.html acisf01843N002_cntr_img2.fits acisf01843N002_cntr_img2.jpg acisf01843N002_evt2.fits acisf01843N002_full_img2.fits acisf01843N002_full_img2.jpg acisf01843N002_src2.fits acisf01843N002_src_img2.jpg orbitf084197100N001_eph1.fits pcadf084271087N002_asol1.fits secondary: acisf01843_000N002_aoff1.fits acisf01843_000N002_evt1.fits acisf01843_000N002_flt1.fits acisf01843_000N002_msk1.fits acisf01843_000N002_mtl1.fits acisf01843_000N002_soff1.fits acisf01843_000N002_stat1.fits acisf084271178N002_0_bias0.fits acisf084271178N002_1_bias0.fits acisf084271178N002_2_bias0.fits acisf084271178N002_3_bias0.fits acisf084271178N002_4_bias0.fits acisf084271178N002_5_bias0.fits acisf084272477N002_pbk0.fits aspect/ axaff01843N001_VV001_vvref2.pdf ephem/
The data files are now ready to be used in CIAO. For a description of the files, read the Introduction to the Data Products thread.
History
03 Jan 2005 | reviewed for CIAO 3.2: no changes |
01 Dec 2005 | reviewed for CIAO 3.3: no changes |
01 Dec 2006 | reviewed for CIAO 3.4: no changes |
09 Jan 2007 | updated for CIAO 4.0: filenames updated since ObsID 1843 went through Repro III |
09 Jun 2008 | updated image display to place figures inline with text |
29 Sep 2008 | corrected link for downloading Chaser |
30 Dec 2008 | reviewed for CIAO 4.1: no changes |
25 Jan 2010 | reviewed for CIAO 4.2: no changes |
09 Nov 2010 | the download_chandra_obsid script is a quick and easy method of download data files. |