CIAO region filtering syntax
Regions are two dimensional filters that can be used to include or exclude data from a given file.
Regions are made up of a number of shapes that are described below. The shapes can either be inclusive or exclusive. The different shapes can then be combined using either a boolean AND or boolean OR operation which leads to a degree of flexibility to design arbitrary regions.
For a general introduction to table and image filtering in CIAO, refer to "ahelp dmfiltering".
The recognized shapes.
Names of shapes
In the table above, the text in capital letters is the minimum allowed for each shape. The shape names are case insensitive, so
both refer to the same shape.
Angles are given in degrees from the X axis, with positive values representing the counter-clockwise direction.
Lengths and Radii
This applies to all regions that take a length parameter (box, rotbox, circle, annulus, ellipse, and pie). The length or radius is interpreted as a physical coordinate value (e.g. sky pixels). If a trailing d, ', or " is added (e.g. 130d), then the value is interpreted as degrees, minutes, or seconds of arc, respectively. Two single quotes will also work for arcseconds.
Coordinates are interpreted as being in physical coordinates unless either the values have a trailing d, e.g. circle(120.896775d,-40.005588d,0.05d), or if the values are expressed in colon separated sexagesimal format, e.g. circle(08:03:35.225,-40:00:20.11,0.05d).
For the polygon shape, if the last point is not equal to the first point, it will add another point onto the end to close the polygon. At present, polygons must be simple; that is they must not intersect themselves.
Regions can be combined using boolean AND (intersection) or boolean OR (union) operations. The syntax for these is given in the table below, where shape1() and shape2() refer to any recognised shape except for "region()".
Region expressions are parsed using the normal precedence rule of the mathematical "*" and "+" operators. The "-" (exclude) operator is replaced by "*!". This means that
is read as
so that the second circle is only excluded from the rectangle, and not the first circle. To exclude the second circle from both shapes you would have to write
An alternative method of excluding shapes is to use the exclude filter, described in "ahelp dmfiltering". So, to exclude all rows within a distance of 20 from (x,y)=(4000,4000) you could use either
The bounds() syntax creates a non-rotated rectangular region that bounds all of the shapes. As the region is not rotated, the boundary can be quite generous compared to the regions it encloses. Valid syntax is:
[sky=bounds(region([filename]))] [sky=bounds(circle(x,y,r)-ellipse(x1,y2,r1,r2,a1))] [ccd_id=3,sky=bounds(region(fov.fits[ccd_id=3]))]
It is not possible to bound only a portion of the region expression; e.g.
is not allowed.
Using region files
Using the region() syntax
Please note that the region() syntax can not be combined with other shapes, so the following expressions are invalid:
If you need to filter by both a region file and other shapes either create a new region file with all the filters in it or apply the filters separately, for instance:
unix% dmcopy "evt2.fits[sky=region(galaxy.reg)]" evt2.galaxy.fits unix% specextract "evt2.galaxy.fits[exclude sky=circle(3845.6,5242.4,12.3)]" ...
Here the dmstat tool is given only those rows of the file evt2.fits that pass the filter, in this case a circle centered at (4000,4000) and with a radius of 20 using the SKY values. Since "sky" is the name of a vector column in the file and translates to "(x,y)", the command could have been written as
where we have also taken advantage of the fact that the region name does not have to be fully spelled out.
Note that the default behavior for region filters is to exclude the entire field and then start adding to it.
The region used to filter the image "img.fits" consists of the whole image (as indicated by the "field()" shape) with a circular region removed from it. Since no coordinate system was given in the filter expression, the logical system is used, so that the above is a short form for
To use other coordinate systems for the filter, include the name of the axes in the expression; for example
Filter the file on the pha and pi columns with the region stored in the file ds9.reg. Although the pha, pi pair do not form a vector column, you are allowed to define a paired column like this on the fly provided their datatypes are the same.
dmcopy "evt2.fits[sky=region(fov1.fits[ccd_id=7]),ccd_id=7]" ccd7.fits
Here we filter an event file so that it contains only events from ccd_id 7. The region() filter is used to ensure that a spatial filter is recorded in the data sub-space, so that future spatial filters will recognize the edges of this chip. Here fov1.fits is the field-of-file provided as part of the Chandra data distribution; this file can also be created by the skfov tool if needed. Since these files contain multiple regions - one per chip - a ccd_id filter is used to select the correct chip.
dmextract "evt2.fits[sky=region(src.fits[#row=2])][bin pi=::5]" spectrum.fits
Create a spectrum from the second source in the FITS region file (src.fits), binning the PI column by a factor of 5.
dmlist "img.fits[sky=circle(4096,4096,100)-sector(4000,4000,-45,45)+box(4100,41 00,50,50)]" complex.fits
In this more complex example the sector is only excluded from the circle. The data inside the box, which overlaps the excluded sector is still included in the overall filter expression. included
Issues with CIAO regions
There are several subtleties with CIAO regions that are worth noting. These arise particularly when using region files made by imagers like DS9.
Celestial coordinate regions
In order for a CIAO tool to make use of a region in world coordinates, e.g.
it must know about the mapping between world and physical coordinates for the dataset in question. Not all CIAO tools have this ability yet. If you have problems with celestial coordinates, try using a region in physical pixel coordinates instead.
DS9 region files saved in celestial coordinates in degrees are not currently supported in CIAO; celestial coordinates are only recognized in sexagesimal format.
Stacks of regions versus regions with many shapes
It's important to understand the difference between two concepts: the REGION STACK and the REGION FILE.
A region file defines a single 'region' which may have multiple shapes. For example, a region file with
is a single region consisting of two circles. Applying a region file to some data lets CIAO decide whether points are in the region, but no knowledge is returned about which of the shapes a point is in.
A region stack contains multiple regions. CIAO will figure out each of the regions separately. The most common use of region stacks is in dmextract, where binning on a region stack creates a histogram where each bin corresponds to one region.
The syntax is
unix% dmextract "evt.fits[bin firstname.lastname@example.org]" ...
to operate on a region stack and make a histogram of counts versus region. In contrast,
unix% dmextract "evt.fits[bin sky=region(reg.stk)]" ...
specifies a single (albeit complicated) region, which will generate a single output bin.
Users may find the various grid() tokens in the stack interface to be useful when they need to perform the same analysis over an grid of regions. For example using the "pgrid" we can extact the counts in different image quadrants as shown here:
unix% dmstat "img.fits[sky=pgrid(4096,4096,0:100:100,0:360:90)]" cen- File=img.fits[sky=pie(4096,4096,0,100,0,90)] EVENTS_IMAGE min: 0 @: ( 4102.5 4098.5 ) max: 5 @: ( 4182.5 4122.5 ) mean: 0.85802469136 sigma: 0.99918042928 sum: 417 good: 486 null: 190 File=img.fits[sky=pie(4096,4096,0,100,90,180)] EVENTS_IMAGE min: 0 @: ( 3998.5 4098.5 ) max: 6 @: ( 4066.5 4178.5 ) mean: 0.66869918699 sigma: 0.86367287106 sum: 329 good: 492 null: 184 File=img.fits[sky=pie(4096,4096,0,100,180,270)] EVENTS_IMAGE min: 0 @: ( 4074.5 3998.5 ) max: 5 @: ( 4066.5 4046.5 ) mean: 0.77510040161 sigma: 0.94083109139 sum: 386 good: 498 null: 178 File=img.fits[sky=pie(4096,4096,0,100,270,360)] EVENTS_IMAGE min: 0 @: ( 4110.5 3998.5 ) max: 6 @: ( 4138.5 4094.5 ) mean: 1.1585365854 sigma: 1.0355805747 sum: 570 good: 492 null: 184
Not every tool accepts stacks so users should check each tools help file to determine if it can accept this kind of syntax.
Includes and excludes
The CIAO syntax requires excluded regions to follow included regions. If you draw four circles on DS9, two includes and two excludes, and then save the region, the resulting file depends on the order the regions were drawn. If the order is -circle+circle-circle+circle, CIAO will not understand because of the leading -circle.
+circle(4096,4096,1000) -circle(4096,4096,250) +circle(4096,4096,50)
will have the inner 50-radius circle included, and the 50-250 radius annulus excluded, but
+circle(4096,4096,50) -circle(4096,4096,250) +circle(4096,4096,1000)
will include everything, as it's interpreted as "(R50 minus R250) OR R1000", and the R1000 includes everything. Therefore, you must be careful about order when generating region files in DS9.
Region areas (BACKSCAL)
For simple regions, the area can be calculated using analytical methods. For complicated regions where shapes overlap, or are close enough that it's not obvious they don't overlap, we must use a different method; CIAO divides up the area into bins (which may be smaller than a single pixel) and counts the number of bins which pass the test 'am I inside the region?'. This approach using discrete bins has a finite error, which is usually of order one percent of the area.
Region areas are most commonly used in the BACKSCAL keyword added to PHA/PI spectral files generated by dmextract. The error in the area is almost always small compared to other calibration uncertainties.
Be particularly careful when making regions comparable in size to the pixel size. When applying such a region to a table which contains fractional pixel positions, filtering does occur in exactly the area specified, although the user should be careful about interpretation - for instance, the true radial profile of a source may be blurred by aspect, PSF and instrument pixelization. When applying the region to an image file, filtering will accept all the counts in a pixel if the pixel center is within the region, and none of them if the pixel center is outside the region; therefore, the effective area sampled is not quite the same as the area calculated.
CIAO understands most of the simple, atomic regions that DS9 can draw including circle, ellipse, point, box, polygons, and annulus. The regions that are purely for annotations such as text, line, vector, ruler, projection, and compass are simply ignored. Note: all 'point' shapes are treated the same when filtering (as a 2D delta-function).
CIAO cannot directly use some of the more exotic ds9 regions. This includes elliptical and box annuli, the panda shape, and format used for writing grids of annuli.
Users need to translate the ds9 regions into CIAO syntax, which while more verbose, is also more flexible. Below are the translations needed to go from ds9 region format to CIAO syntax.
ds9: annulus(xc,yc,r_in,r_out) CIAO: annulus(xc,yc,r_in,r_out)
Where "in" refers to the inner radius and "out" specifies the outer radius. The simple, single annulus is supported in CIAO. If the annulus contain multiple annuli, ds9 will save the region in CIAO format with multiple single annuli.
The dmextract tool also recognizes a special "annulus" shape as part of the binning syntax which supports multiple annulii.
ds9: ellipse(xc,yc,r_major_in,r_minor_in,r_major_out,r_minor_out,angle) CIAO: ellipse(xc,yc,r_major_out,r_minor_out,angle) - ellipse(xc,yc,r_major_in,r_minor_in,angle)
As above and where "major" and "minor" are the ellipse axes.
ds9: box(xc,yc,xlen_in,ylen_in,xlen_out,ylen_out,angle) CIAO: box(xc,yc,xlen_out,ylen_out,angle)-box(xc,yc,xlen_in,ylen_in,angle)
Panda (pie and annulus)
ds9: panda(xc,yc,angle_start,angle_stop,1,r_inner,r_outer,1) CIAO: pie(xc,yc,r_inner,r_outer,angle_start,angle_stop)
ds9 will translate panda shapes into pie shape for a single set of radii and angles. For multiple annuli and/or multiple radii users need to use stacks (if applicable), see pgrid syntax
ds9: epanda(xc,yc,angle_start,angle_stop,1,r_major_in,r_minor_in,r_major_out,r_minor_out,1,angle_ellipse) CIAO: ellipse(xc,yc,r_major_out,r_minor_out,angle_ellipse)*sector(xc,yc,angle_start+angle_ellipse,angle_stop+angle_ellipse)-ellipse(xc,yc,r_major_in,r_minor_in,angle_ellipse)
ds9: bpanda(xc,yc,angle_start,angle_stop,1,r_major_in,r_minor_in,r_major_out,r_minor_out,1,angle_ellipse) CIAO: box(xc,yc,r_major_out,r_minor_out,angle_ellipse)*sector(xc,yc,angle_start+angle_ellipse,angle_stop+angle_ellipse)-box(xc,yc,r_major_in,r_minor_in,angle_ellipse)
The shapes listed above may also contain multiple regions, ie multiple radii or multiple angles. The CIAO equivalent of these is the stack syntax. In particular, the "pgrid" (polar grid) and "rgrid" (rectangular grid) stack syntax can be used to make a grid of regions to input as a stack to those tools that support it.
See the region library bugs page on the CIAO website for an up-to-date listing of known bugs.
Refer to the CIAO bug pages for an up-to-date listing of known issues.
- coords, level, pileup, times
- autoname, ciao, ciao-install, ciaorc, history, parameter, stack, subspace
- dm, dmascii, dmbinning, dmfiltering, dmmasks, dmopt
- convert_ds9_region_to_ciao_stack, dither_region, dmcontour, dmgroupreg, dmimgdist, dmimgfilt, dmimghull, dmimglasso, dmmakereg, get_src_region, mkbgreg, mksubbgreg, roi, splitroi, tg_create_mask