/ ciao4.16 / ahelp / dmmasks.html

Jump to: Description · Examples · How to Create a Mask · How to Visualize a Mask · How to Manipulate a Mask · Summary of Mask Filters Properties · Caveats · See Also

AHELP for CIAO 4.16

dmmasks

Context: dm

Synopsis

CIAO mask filtering syntax

Description

Masks are two dimensional filters that can be used to include or exclude data from a given file. They can be used as filters similar to the ways regions are used.

Masks are 2D images. Pixel coordinates that map to values equal to 0 are excluded from the input, pixel values not equal to 0 are included. The mask can either be inclusive or exclusive.

The basic mask filtering syntax is

[column=mask(filename)]

Masks complement the existing region filtering syntax. Masks provide a way to create complex 2D filters without needing to express it with a potentially long and complicated region expression. Common uses will be to exclude a large number of point sources embedded in extended emission and to simplify the footprints for overlapping fields of view for merged datasets.

For a general introduction to table and image filtering in CIAO, refer to "ahelp dmfiltering".

Examples

Example 1

unix% dmcopy "evt.fits[sky=mask(mask_file)]" filtered_evt.fits
unix% dmcopy "img.fits[sky=mask(mask_file)]" filtered_img.fits

The same syntax (and mask) is used to filter both tables (eg event files) and images.

Example 2

unix% dmcopy "img.fits[exclude sky=mask(mask_file)]" filtered_img.fits
unix% dmcopy "img.fits[filter sky=mask(mask_file)]" filtered_img.fits

The "exclude" directive inverts the mask logic. The rarely used "filter" directive can also be used; this is the default behavior.

Note that excluded masks are stored in the file subspace with the mask values inverted.

Example 3

unix% dmcopy "evt.fits[ccd_id=3,sky=mask(mask_file)]" out.fits

Mask filters can be combined with other column filters.

Example 4

unix% dmcopy "evt.fits[(x,y)=mask(mask_file)]" out.fits

Mask filters can be applied to vector columns ('sky') or any arbitrary pair of columns.

Example 5

unix% dmcopy "evt.fits[sky=bounds(mask(mask_file))]" out.fits

The "bounds()" directive can be used with masks to generate a rectangular region around the edge of the mask file.

Example 6

unix% dmcopy "evt.fits[sky=mask(mask_file),sky=region(ciao.reg)]"
out.fits
unix% dmcopy "evt.fits[sky=mask(mask_file),sky=circle(1000,1000,20)]"
out.fits

Mask filters and region filters can be used concurrently.

Example 7

unix% dmcopy "evt.fits[sky=mask(mask_file),x=1000:4000]" out.fits

Mask filters can also be used with range filters.

Example 8

unix% dmcopy "evt.fits[sky=mask(first_file)||sky=mask(second_file)]"
out.fits
unix% dmcopy "evt.fits[sky=mask(first_file),sky=mask(second_file)]"
out.fits

Mask filters on the same column can be combined using the logical-and (",") and logical-or ("||") operators.

Example 9

unix% dmcopy
"evt.fits[sky=mask(sky_mask),(time,pi)=mask(another_mask)]" out.fits

Separate mask filters can be applied to multiple columns. Masks are not restricted to spatial axes.

Example 10

unix% dmcopy "evt.fits[sky=mask(mask_file)+region(ciao.reg)]" out.fits
ERROR
unix% dmcopy "evt.fits[sky=mask(mask_file)+circle(4095,4096,100)]"
out.fits
ERROR
unix% dmcopy "evt.fits[sky=field()-mask(mask_file)]" out.fits
ERROR

Masks cannot be logically combined with regions using the region logic operators.

Example 11

unix% dmcopy "evt.fits[sky=mask(mask_file)][exclude
sky=circle(4095,4096,100)]" out.fits
ERROR
unix% dmcopy "evt.fits[exclude
sky=mask(mask_file)][sky=circle(4095,4096,100)]" out.fits
ERROR

Using both include and exclude filters is not supported. This is true for any/all DM filters.

Example 12

unix% dmcopy "evt.fits[sky=mask(sky_mask),(x,time)=mask(another_mask)]"
out.fits
ERROR

A single column, eg 'x', can only be used with one mask at a time.

How to Create a Mask

Masks are simply 2D images, so any CIAO tools can be used. Some common examples are shown below.

Using dmimgthresh

dmimgthresh may be used on an exposure map to create a mask that provides the footprint (field of view) of the observation.

unix% dmimgthresh exposure_map.fits mask.fits cut=10% value=0 clob+

The cutoff value can be expressed in percentage of the max or as an absolute value. Note: dmimgthresh replaces values that are strictly less than cutoff value. Since all the mask filtering logical deals with is pixel values = 0 or those not equal to 0, there is no need to replace the values > the 10% cutoff, however, users wishing to can continue using dmimgthresh

unix% dmimgthresh mask.fits mask_0s_and_1s.fits cut=:10% value=1 clob+

The output image will now contain values equal to 0, 1 (and possibly NaN's if the original input image contained NaN values).

Using dmimgcalc

Another common technique will be to create an image that is all 1's or all 0's and then use DM region filters to include/exclude pixels.

unix% dmimgcalc counts.fits none all_ones.fits op="imgout=((img1-img1)+1)"
unix% dmcopy "all_onts.fits[sky=region(ciao.reg)][opt full]" mask.fits

using [opt full] will preserve the size image. A mask file can be more efficient and faster when dealing with a large number of individual sources (shapes). It is also trivial to invert the mask by simply subtracting it from 1.

unix% dmimgcalc mask.fits none inverted_mask.fits op="imgout=(1-img1)"

One advantage of using the tools in this way is that the WCS will automatically be preserved.

Using crates

Crates allows for easy manipulation of individual mask pixels. When using crates it is often easiest to start with a donor image file and then use crates to read and python to modify the pixel values. This way the image size is already established along with the necessary WCS.

>>> from pycrates import read_file
>>> import numpy as np
>>> img = read_file("img.fits")
>>> orig_pix_vals = img.get_image().values
>>> zeros = np.zeros_like( orig_pix_vals )
>>> zeros[999,999] = 1
>>> img.get_image().values = zeros
>>> img.write("delta.fits", clobber=True)

The output image has all the pixel values replaced with a delta function at pixel 1000,1000 (in image/logical coordinates). Numpy arrays are 0-based indexed, so to get pixel 1000,1000 in logical coordinates you subtract 1. Also note that Numpy arrays are accessed as [y,x] so setting: zeros[199,49]=1, would set logical pixel(50,200)=1.

Since only the pixel values were replaced, the WCS (if any) is preserved. This method can only be used when replacing the pixel value array with an array of the same size and shape. Using a different size/shape array will invalidate the WCS.

How to Visualize a Mask

As masks are simply 2D images, any visualization software should be able to view them.

ds9

SAOImage ds9 can directly display the binary mask by specifying the block name

unix% ds9 "evt.fits[mask]"

and just like other images, the mask and counts data can both be loaded into separate frames and users can alternate/blink between them.

unix% ds9 evt.fits "evt.fits[mask]" -blink

When working with images (not event files), ds9 also has support for overlaying a mask with adjustable transparency

unix% ds9 img.fits -mask color green -mask "img.fits[mask]"

The transparency of the mask can be adjusted by going to Analysis -> Mask Parameter ... Technically, the ds9 mask functionality does also work for event files, but the mask must be full resolution (so for ACIS, would be an 8k x 8k image).

How to Manipulate a Mask

Depending on how the mask is created it may be desirable to adjust the mask pixel values. As these are just 2D images, any tools that operate on images can be used. Below are some common operations that may be performed.

Filter

As shown above, one simple way to modify a mask file is to filter the pixel values. Using [opt full] retains the original size of the mask which is often desireable.

unix% dmcopy "mask.fits[sky=region(ds9.reg)][opt full]" filt_mask.fits

This is only useful to change non-zero pixels to 0.

Threshold

Since masks can be any datatype, it is possible apply thresholds to an image to create different masks. One example of an exposure map was shown above.

unix% dmimgthresh expmap.fits mask.fits cut=10% value=0

Another example may be to select a mask from a map such as is produced by wavdetect (the "cell image"), dmimgblob, or dmnautilus. Each of these tools produce a map where the integer pixel value are used to group pixels together. The values usually go from 1 to N where N is the total number of groups. To turn the map into a mask then requires applying a threshold at the desired group number. A simple example might look like

unix% aconvolve img.fits sm_img.fits "lib:gaus(2,5,5,3,3)" method=fft
unix% dmimgblob sm_img.fits map.fits threshold=0.5 srconly=yes 
unix% dmimgthresh map.fits mask.fits cut=5:5 value=0

dmimgblob identifies groups of connected pixels above the threshold in the smoothed input image and outputs a map that provides the groupings. A single group can be selected with dmimgthresh by setting the upper and lower limits to be the same and setting value=0; so all pixels not associated with, in this example, group #5 would be excluded.

Morphological Operators

Masks can be fine-tuned using common image processing techniques.

An image can be "dilated" using the "max" function in dmimgfilt

unix% dmimgfilt img.fits mask.fits function=max mask="box(0,0,3,3)"

which increases the size of the include area by 1 pixel in all directions. Similarly, the image can be "eroded" using the "min" function

unix% dmimgfilt img.fits mask.fits function=min mask="box(0,0,3,3)"

which decreases the included area by 1 pixel in all directions.

These basic gray-scale morphological operations can be especially useful when used together. For example if the mask contains many isolated points and the desire is to create a mask that includes the entire area, then the "closing" operation can be achieved by dilating then eroding the image

unix% dmimgfilt img.fits - max "box(0,0,3,3)" numiter=10 | \
  dmimgfilt - mask.fits min "box(0,0,3,3)" numiter=10

The "closing" operation acts to fill in small holes in the mask while retaining the original size and outline. The reverse can also be done by "opening" the image

unix% dmimgfilt img.fits - min "box(0,0,3,3)" numiter=10 | \
  dmimgfilt - mask.fits max "box(0,0,3,3)" numiter=10

which is useful when the goal is to increase the excluded regions while retaining the original outline.

Combine

Mask images can be logically combined using ordinary math operations available in dmimgcalc.

To create the union of two masks use the summation, sum, operation. (Note: this assumes that the pixel values are positive!).

unix% dmimgcalc mask1.fits mask2.fits mask_union.fits sum

If the pixel values could be negative, then use the full calculator expression:

unix% dmimgcalc mask1.fits,mask2.fits none mask_union.fits op="imgout=fabs(img1)+fabs(img2)"

To create the intersection of two masks use the multiply, mul, operation

unix% dmimgcalc mask1.fits mask2.fits mask_intersect.fits mul

To combine more than two, the full dmimgcalc operation expression can be used

unix% dmimgcalc m1,m2,m3,m4 none mask_intersect.fits op="imgout=(img1*img2*img3*img4)"
unix% dmimgcalc m1,m2,m3,m4 none mask_union.fits op="imgout=(img1+img2+img3+img4)"

or, if the mask pixel values are >=0 then stacks can be used with dmimgfilt using the "min" function for intersection and "max" function for union.

unix% dmimgfilt @img.stk mask_intersect.fits fun=min mask="point(0,0)"
unix% dmimgfilt @img.stk mask_union.fits fun=max mask="point(0,0)"

where "point(0,0)" tells dmimgfilt to use the same point from each image in the stack when applying the filter.

Summary of Mask Filters Properties

Data Types

The mask file can be any 2D image (integer or floating-point). Pixels value equal to 0 are excluded, pixel values not equal to 0 are included. Negative pixel values, <0, are not equal to 0 and are also included. Pixel locations outside the mask are always excluded. Regardless of the input dataype, masks are always stored as single Byte type images in the output subspace.

Coordinates

If the file being filtered has a World Coordinate System (WCS), and the mask file has a WCS, then the two must be equivalent. That is when filtering images the pixel size must be same and have the same projection parameters (eg tangent point). If the mask or the file being filtered do not contain a WCS then then no consistency checking is performed.

Axes

Any pair of image axes (table columns) can be filtered with masks; masks are not restricted to just spatial filtering.

Filtering Tables vs. Images

Images and tables (eg event files) are filtered using the same syntax. For real-valued table columns rows that have values within +/- 0.5 of the pixel center are filtered with the pixel value.

NaN's and NULL's

Mask pixels with NaN or integer NULL values in the mask are treated as 0. All IEEE special values, eg Inf, are treated as 0.

Storage

The mask filter is stored as part of the data subspace. The mask is always stored as an image in a MASK extension. The Byte type mask will only contain pixel values = 0 or 1.

unix% dmlist evt.fits blocks
 
--------------------------------------------------------------------------------
Dataset: evt.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: EVENTS                         Table        15 cols x 54       rows
Block    3: GTI3                           Table         2 cols x 3        rows
Block    4: GTI2                           Table         2 cols x 2        rows
Block    5: GTI1                           Table         2 cols x 2        rows
Block    6: GTI0                           Table         2 cols x 2        rows
Block    7: MASK                           Image      Byte(2634x2630)

% dmlist evt.fits subspace
 
--------------------------------------------------------------------------------
Data subspace for block EVENTS: Components: 4 Descriptors: 16 
--------------------------------------------------------------------------------
 
 --- Component 1 --- 
 ...
   8 sky                  Real4               TABLE MASK
                                              MASK(MASK)
                                              Field area = 6.71089e+07 Region area = 398
  ...

The WCS, if any, is saved with the mask. Other header keywords in the original mask file are not saved.

bounds()

The bounds() operation returns a rectangle that encloses the sub-image of the mask that is non-zero.

exclude

The inside-outside logic for pixels in the mask is inverted when the [exclude column=mask(file)] syntax is used. Pixels outside the mask are always excluded irrespective of the [exclude ] directive.

Cropping

If a filter is applied to an image with a mask, both the image and mask will be cropped to the size of the bounding-box around the filter.

unix% dmcopy "img.fits[sky=mask(mask.fits)]" a.fits
unix% dmcopy "a.fits[sky=region(ciao.reg)]" b.fits

The output image in b.fits will be cropped to the bounding box around the region file. Simultaneously, the MASK extension will be cropped to the same bounding box.

Merging

When event files (tables) containing masks are merged with dmmerge, the masks will be combined. The output will be the union of the masks from the files in the input stack.

unix% dmcopy "evtA[sky=mask(a.mask)]" a.evt
unix% dmcopy "evtB[sky=mask(b.mask)]" b.evt
unix% dmmerge a.evt,b.evt merged.evt

Area Calculations

Masks are automatically included in the calculation of the area by tools like dmlist and dmextract.

Comparison to Regions

Masks cannot be used with region operators "+", "-", "!", "*", "&", and "|". While this command

unix% dmcopy "img.fits[sky=mask(mask.fits)&region(ciao.reg)]" out.fits
ERROR

does not work, this command

unix% dmcopy "img.fits[sky=mask(mask.fits),sky=region(ciao.reg)]" out.fits

does.

Caveats

Reprojecting

The ciao tools that regrid images and event files do not update the coordinates for regions in the subspace including the mask subspaces. This includes dmregrid, dmregrid2, reproject_image, and reproject_image_grid. Users should regrid their data before filtering.

WCS

CIAO tools that modify the WCS do not automatically update the WCS of the MASK extensions. This includes wcs_update which is run by reproject_aspect. Users should apply any astrometric corrects before filtering.

Combining/Co-Adding/Merging

dmmerge will attempt to merge the region subspaces, including masks, when combining event files. However, other tools that combine data from multiple input files generally do not attempt to combine (union or intersect) the subspace information. For example using dmimgcalc to sum image does not automatically combine the regions|masks.

Multiple FITS Blocks

Users trying to append extra blocks onto a file (eg with dmmappend) should carefully check the output. The results when appending two blocks, each with different masks is error prone.

Higher Dimensions

Masks should not be used to filter two arbitrary axes within a 3D or higher dimension images.

Multiple Subspaces

The mask created when merging multiple event files may be incorrect if the input files have multiple subspaces (ie multiple GTI blocks). Users should apply mask filters after the files are dmmerge'ed together.