/ ciao4.17 / ahelp / read_file.html

Jump to: Description · Examples · The mode argument · Changes in CIAO 4.11 · Changes in CIAO 4.8 · Bugs · See Also

AHELP for CIAO 4.17

read_file

Context: crates

Synopsis

Read FITS or ASCII data into a crate.

Syntax

read_file(filename, mode='r')

Description

Argument	Description
filename	Name of the file; it can include CIAO Data Model syntax such as filters and binning values
mode	Should the mode be opened for read-only ('r', the default), or for read-write access ('rw').

The read_file command loads the specified file into the appropriate crate, either a TABLECrate or an IMAGECrate. The input file may be in FITS or ASCII format.

Multiple blocks, read_pha and read_rmf

Note that this routine only reads in data from one block of the file (and any related information, such as the GTI blocks of an event file). If you wish to access multiple blocks within a file then see read_dataset or ahelp cratedataset.

Differences to the CIAO Data Model

Users familiar with the CIAO Data Model should note that Crates reads one extension of the input file, not the whole dataset. The file that is written out will only contain the block which was read. If you wish to access all the blocks in the file then you need to use the CrateDataset class.

Examples

Example 1

>>> cr = read_file("tbl.dat")
>>> write_file(cr, "tbl.out")

Read in a file tbl.dat and then write it out to tbl.out, in FITS format.

Example 2

>>> cr = read_file("evt2.fits")
>>> print(cr)

   Crate Type:        <TABLECrate>
   Crate Name:        EVENTS
   Ncols:             16
   Nrows:             248072

Example 3

>>> cr = read_file("evt2.fits[sky=circle(4096,4096,100)][bin sky=::1]")
>>> print(cr)

   Crate Type:        <IMAGECrate>
   Crate Name:        EVENTS_IMAGE

Example 4

>>> f = "evt2.fits[energy=500:7000,sky=region(src.reg)][bin sky=::1]"
>>> cr = read_file(f)
>>> s = get_piximgvals(cr).sum()
>>> print(f"Pixel sum={s}")
Pixel sum=2361

File names can include CIAO Data Model filtering and binning commands (e.g. see "ahelp dm"). In this case energy and spatial filters are applied to an event file before being binned into an image. The pixel values are read in and then summed up.

Example 5

>>> a = read_file("myfile.dat")
>>> x = copy_colvals(a, "col1")
>>> y = copy_colvals(a, "col2")

The read_file command is used to read the contents of the ASCII file myfile.dat into the TABLEcrate a. The copy_colvals command is used to copy the data from the columns col1 and col2 into numpy arrays.

The get_col_names routine can be used to find the columns in a Crate.

Example 6

>>> cr = read_file("src.fits")
>>> x = get_colvals(cr, "x")
>>> rng = np.random.default_rng()
>>> x += rng.random(x.shape) - 0.5
>>> write_file(cr, "xrand.fits")

Here we read in the x column of src.fits, add a random value between -0.5 and 0.5 to each element, then write the results out to xrand.fits.

The above only worked because we used get_colvals to access the data - so that we we working with the data stored in the crate and not a crate - and because the '+=' operator was used to update x; if we had said

>>> x = x + rng.random(x.shape) - 0.5

then xrand.fits would be the same as src.fits. More information on random-number support in NumPy can be found at https://numpy.org/doc/stable/reference/random/index.html.

Notes

Using read_file will only read in a single block of information from the file (as well as ancillary information such as GTI blocks). This means that using read_file followed by write_file will lose information about other blocks in the file. To handle multi-block files, use the CrateDataset class.

Prior to CIAO 4.11, HISTORY and COMMENT records were not guaranteed to be read in by read_file or written out by write_file.

Example 7

>>> a = read_file("myfile.dat[opt colnames=none]")

In this case, the ASCII kernel option 'opt colnames=none' tells the read_file command to skip the header information when reading the contents of the ASCII file myfile.dat. See "ahelp dmascii" for more information on the ASCII file support in CIAO.

Example 8

>>> cr = read_file("srcs.tsv[opt kernel=text/tsv]")
>>> t1 = cr.get_column("tlo").values
>>> t2 = cr.get_column("thi").values
>>> cd = CrateData()
>>> cd.name = "tmid"
>>> cd.values = (t1 + t2) / 2.0
>>> cd.desc = "Average of t1 and t2"
>>> add_col(cr, cd)
>>> cr.write("modsrcs.fits")
>>> cr.write("modsrcs.tsv[opt kernel=text/tsv]")

Here the input file is a tab-seperated file (TSV format); the correct ASCII kernel needs to be specified in this case otherwise the data is not loaded correctly.

The t1 and t2 columns are accessed using the get_column method of the crate; as this returns a CrateData object, the values field is used to extract the column data, since we are not interested in the column metadata.

A new column "tmid" is created, containing the average of the t1 and t2 columns, and added to the crate using add_col(). Finally the new file is written out as in FITS and TSV formats using the write method.

The mode argument

When a file is read in, the write permission is checked against the mode argument and, if it does not match (if mode='rw' but the user does not have write permission, or the file is a gzipped file) then a warning is displayed and the mode is set to 'r'.

When is the mode argument used?

The mode argument is only relevant if you call the write method of the crate with no arguments; that is if you say

>>> cr = read_file('tbl.dat', mode='rw')
UserWarning: File 'tbl.dat' does not have write permission. Changing to
read-only mode.
...
>>> cr.write()
IOError: File is not writeable.

It is not used if you want to write to a new file or one that is not write protected. That is, you can read in a file in read-only mode, change its contents, and write it out to a new file:

>>> cr = read_file('img.fits', mode='r')
>>> ivals = cr.get_image().values
>>> ivals += 1
>>> cr.write('modified.fits')

Changes in CIAO 4.11

History and Comment records

Crates now supports HISTORY and COMMENT records in the file header, so they will now be stored in the Crate when read in, can be added or deleted, and written out to the output file. The history module (which is also new in CIAO 4.11) can be used to create HISTORY records in the format recognized by CIAO tools (such as dmhistory).

Changes in CIAO 4.8

Tri-state logical columns

The FITS convention for boolean columns supports a "null" value, which means that it can represent three states: true, false, and unknown. Any such null, or unknown, value will be converted to False when the data is read in to a Crate (e.g. when read by read_file), and so the information that this cell value was a null will be lost.

Bugs

See the bug pages on the CIAO website for an up-to-date listing of known bugs.

Refer to the CIAO bug pages for an up-to-date listing of known issues.