Bugs: Data Model
Table of Contents
Incorrect values from CAR transform.
The WCS library that the DM uses has a problem computing coordinate transforms that involve the CAR transform.
You may get a seg fault if you try to create a very large image. What constitutes "very large" depends on the data type, but for long and float images, 8192x8192 pixels seems to be the threshold.
(The image doesn't have to be square; it just needs to have 8192^2 pixels.)
This condition may be met when the "update=no" option is used. Normally, when you filter a dataset, the data subspace (which describes the boundaries of each column's data and therefore is the intersection of the initial minima and maxima with any subsequent filters) gets updated to reflect the filtering. However, when you give the "update=no" option, you instruct the DM not to update the subspace to reflect the current filter. Therefore, the full ranges for x and y are used in the binning, and you get a 8192x8192 image (and a seg fault, for the reason described above).
TNULL raw header keywords are not copied to the output file (19 Aug 2011)
DM support for FITS TC* keywords (02 May 2013)
There are three issues with the generic use of TC* keywords in FITS files read and written by DM tools.
The first issue: TC*n[A-Z]. The DM function that composes this keyword does not strip off the 'P' from the before adding number and letter to the end.
Second issue: TCTY* keys are not recognized and therefore are not processed.
Third issue: DM is not always retaining the T*NAM information when it stores information on the DM descriptor, resulting in output keywords T*TYP instead.
Omit the "#2=:" from the filter
Specify a range for both elements: [#1=1:20,#2=1:20]
Remove update=no. In this case, the Data Model internally inverts all exclude filters to be an inclusive filter, and correctly filters the image.
Be aware that this process is much slower if the region is large. In that case, it will also add a large region keyword to the file's header, noticeably slowing down any operation on that file.
For ASCII region files, it is also possible to manually invert the filter in the file. The "field()" region syntax is used to include the entire field, then remove the undesired sources. For instance,
# Region file format: CIAO version 1.0 circle(1635.5,4113.5,135.11408) circle(3975,4233,20) circle(2565.5,4129.5,40) circle(2129.5,4007.5,40)
# Region file format: CIAO version 1.0 field() -circle(1635.5,4113.5,135.11408) -circle(3975,4233,20) -circle(2565.5,4129.5,40) -circle(2129.5,4007.5,40)
and the dmcopy filtering command would be
unix% dmcopy "acis_img.fits[sky=region(src.ascii)][opt full,update=no]" filtered.fits
Filtering on some WCS columns produces incorrect results (08 Oct 2012)
When filtering on WCS columns, the range is taken by converting range of the parent columns and using those as the limits of the WCS columns. When the transform is highly non-linear, eg the TAN-P transform used to go from DETX,DETY to THETA,PHI, this can leads to incorrect limits and incorrect filters. Users who want to filter on WCS columns should give explict ranges and not rely on the computed min/maxes.
bad% dmcopy "evt.fits[theta=:1]" good% dmcopy "evt.fits[theta=0:1]"
Multi-valued string filters don't work
"col=foo" is okay, but "col=foo,bar" isn't.
Use "col=foo,col=bar" instead.
Creating a vector on-the-fly when region filtering
When region-filtering images, you can create a vector on the fly from any two axes by using a filter like "(#1,#3)=circle(...)". Although the image is filtered correctly with a temporary vector, the region filter isn't recorded in the subspace. Hence, tools that use the filtered file don't know that pixels outside the filter region are invalid. As a result, dmstat reports no nulls in the filtered image (unless you explicitly tell the DM to set pixels outside the filter to null by using "opt null=...").
Applying a bit-filter expression to an integer column does not work, nor does it cause an error.
Using incorrect syntax with the rectangle shape does not fail when filtering.
For example, setting xmax > xmin and/or ymax > ymin. Instead it appears that the Data Model simply swaps the min and max values.
Filtering an image on logical coordinates causes problems when the short cut of omitting a number (i.e. to indicate the default value) is used.
The exit status of dmcopy is also incorrectly set to 0 (success):
unix% dmcopy "image.fits[#1=1:20,#2=:]" delme.fits # DMCOPY (CIAO): [ftColRead]: FITS error 308 bad first element number in dataset image.fits Block 1 PRIMARY unix% echo $status 0
Trying to exclude a region filter with update=no will cause the image to be filtered by the region's bounding box.
unix% dmcopy "acis_img.fits[exclude sky=region(src.fits)][opt full,update=no]" filtered.fits
The regions are correctly excluded; however, the image is also clipped at the bounding box around all the excluded shapes, so the corners of a few chips are removed.
Using "!=" with string filtering causes dmlist to fail. (05 Mar 2007)
For example, the following commands both fail:
unix% dmlist "region.fits[shape!=Annulus]" data unix% dmlist catalog.fits"[COMMENT!='weak'][cols COMMENT]" data
The "or" syntax ("||") doesn't work on string columns (17 Jul 2009)
For example, this command does not find all instances of "PMterm" in the selected columns:
unix% dmlist stat.fits"[src=PMterm||det=PMterm||mst=PMterm]" counts 13
unix% dmlist stat.fits"[cols det,src,mst]" data,clean | grep PMterm | wc -l 27
The "or" syntax ("||") doesn't work inside a clause (15 Apr 2011)
This example command does not work:
unix% dmlist "evt2.fits[(ccd_id=5||ccd_id=7),pha=2500:3500]" blocks
Rewrite to include the filter conditions in each part of the conditional.
unix% dmlist "evt2.fits[(ccd_id=7,pha=2500:3500)||(ccd_id=5,pha=2500:3500)]" blocks
Virtual columns are not supported. (11 Dec 2007)
In the DM, you can normally do
unix% dmlist evt.fits"[cols ra,dec]" data
even though RA and Dec are just coordinate systems defined on the X and Y columns in the file; the DM applies the transform on the fly. This doesn't work yet for ASCII files.
DTF-FIXED keyword comments may be truncated. (15 Jan 2008)
DTF-FIXED header lines may be up to 1024 characters long. However, if the keyword is longer than the FITS standard, the comment is truncated.
unix% input.txt output.dtf'[opt kernel=text/dtf-fixed]'
TTYPE14 = 'Class' / LV Class Exo: M = missile [B = tactical ballistic missile (except Redstone) apo=80:200] R = research rocket O = orbital LV V = RTV Y = Exo weather rocket X = Big test rocket D= Deep space launch
TTYPE14 = "Class " / LV Class Exo: M = missile [B = tactical ballistic missile (except Redstone) apo
Problems with CSCView .tsv files with 'G' datatypes (02 May 2013)
The text/tsv ASCII parser does not recognize columns with a G format used for floating-point, real valued values such as src_rate_aper_*. The exact behavior will vary based on the other columns returned, and in what order, but generally all columns after the G format will be unreadble by DM tools.
Users can work around the problem by editing the .tsv file and changing any (Gx.y) values to (Fx.y). For example:
#Column src_rate_aper_b (G9.5) Aperture-corrected net count rate in...
#Column src_rate_aper_b (F9.5) Aperture-corrected net count rate in...
Rebinning an image with different values for the two axes causes the coordinate information to be lost
unix% dmcopy acis.img"[bin x=::5,y=::6]" acis5x6.img
Using the same value for both axes works correctly:
unix% dmcopy acis.img"[bin (x,y)=::5]" acis5.img
Running a DM tool on an image where one of the axes has been filtered results in an error. (17 Jul 2009)
unix% dmextract Input event file (ccd3.sky4.fits[y=3767:][bin sky=annulus(3786,3767,0:380:4)]): Enter output file name (rprof.fits): # dmextract (CIAO 4.0 Beta 2): WARNING: Input file, "ccd3.sky4.fits[y=3767:]", has no rows in it. Bus error
Bugs fixed in CIAO 4.5
The following is a list of bugs that were fixed in the CIAO 4.5 software release.
Omitting the cols directive may lead to a crash. (08 Oct 2012)
If a users forgets to specify the cols directive, it may lead to a segmentation violation instead of generating a useful error message.
% dmlist emap.hist"[bin_low, bin_high, counts]" cols # 62735: Received error signal SIGSEGV-segmentation violation. # 62735: An invalid memory reference was made. # 62735: segmentation fault: DMLIST (1) is: exit_upon_error->NULL
BSCALE and BZERO keywords are copied to output file (19 Aug 2011)
The DM doesn't treat BSCALE and BZERO as structural keywords so they get copied to the output. If you have a file which is a floating point image, the addition of these keywords will create incorrect output results.
# DMCOPY (CIAO): Bad data type in filter string formatting
If a logical column exists in the file, the DM will generate this warning. The output is unaffected by this bug.
Incorrect results when using '0' in filters (11 Sep 2012)
The datamodel replaces '0' in filters with a small value, 1e-16. If the other values being filtered are generally smaller than this, as one might have with flux values, then the wrong set of rows or pixels will be returned.
unix% cat foo.dat #data 0.0 1.0e-12 1.0e-14 1.0e-16 1.0e-18 1.0e-20 unix% dmlist foo.dat"[data>0]" data,clean # data 1E-12 1E-14 unix% dmlist foo.dat"[data=0]" data,clean # data 0 1E-16 1E-18 1E-20
For less than and greater than, users can work around this by using a small number other than 0
% dmlist foo.dat"[data>1e-90]" data,clean # data 1E-12 1E-14 1E-16 1E-18 1E-20
There is a bug in region and range filters.
unix% dmcopy "input.fits[sky=circle(4096,4096,100),y=4020:4100,4250,4350]" \ output.fits
fails to do the y filter altogether. This bug also applies to exclude filters.