Modify and create columns in a table
dmtcalc infile outfile expression [clobber] [verbose]
`dmtcalc' is a flexible general purpose table calculation utility. It can change existing columns add new ones using arbitrary expressions which can involve other columns - possibly smoothed - mathematical functions such as sqrt(), random numbers, values taken from the file header, and much more. It uses the same syntax as the dmgti and dmimgcalc tools: this syntax is described in detail in "ahelp dmmath".
The input expression can either be given directly on the command-line or stored in an external file and accessed as a stack - i.e. "@filename". The use of a stack allows more than one column to be created or modified at one time.
When dmtcalc starts, it will make an effort to assure that any data dependencies in the expressions (referenced columns or keywords, or new columns defined in previous expressions) are satisfied, that there are not any data type conflicts within the expressions, and that there are no conflicts of array size or dimensionality. If any of these tests fail, an error message will print, and the tool will exit.
Changing the value of an existing column
dmtcalc cannot be used to change the value of an existing column. Rather, users can create a new column with the same name as an existing column though they need to explicitly copy the values (if desired). For example
Will create a new column "DY" in the output file and the values of the new column will be set equal to the values of the input column. Similarly
Will create a new DZ column that matches the datatype of the expression, in this case an integer datatype. The original values will be truncated and saved in the new output column.
Any/all other column properties are not copied including: column description, column units, WCS, NULL values, etc. As with all new columns, they are added after all the existing columns in the file so the order of the columns is not preserved.
Overwriting times in the GTI blocks
dmtcalc correctly changes the start and stop columns in the file (and thus the order can change). However, it then proceeds to copy the data subspace from the input to the output file and so the values are overwritten with the original values. To override tstart/tstop, they need to be put into a new extension:
Then use the reverse notation when using as a filter:
dmtcalc evt2.fits evt2_edit.fits expression="dtime=(time-TSTART)*1.0e-3"
This time we have used dmtcalc to create a new column called "DTIME" in the output file evt2_edit.fits (new or changed columns always have their names written in upper case in the output file, however they are written in the expression).
The contents of the DTIME column are set equal to
where TSTART is replaced with the value of the TSTART keyword from the header. The case of header keywords is unimportant, so
would give the same result.
For this particular example DTIME can be considered to be the time since the start of the observation in kilo-seconds. In general the expressions used in these examples are just used to illustrate dmtcalc's capabilities and so should not be taken to have any real meaning for Chandra (or any other) data. More details of the allowed expressions are given in "ahelp dmmath".
dmtcalc LV.newer"[cols length,diameter,launch_mass]" calc.out.txt"[opt kernel=text/simple]" expr="result=diameter*length*(diameter/launch_mass)"
Run dmtcalc on a fixed-format text file, creating a new column named "result"; see "ahelp dmascii" for more information on working with ASCII files.
dmtcalc evt2.fits evt2_edit.fits expression="energy=((float)pha)*0.1"
Here we use dmtcalc to over-write the values in the energy column to be ten percent of the PHA values. Since the PHA values are stored as integers we convert them to floating-point numbers - by use of the term "(float)" - before the division.
If we had omitted the "(float)" term, the output would still have been automatically converted to a real since we are multiplying by a real number. It would, however, have changed the ENERGY column to Real8 (ie double) format instead of keeping the Real4 (float) format.
The name of the energy column in the output file (here "evt2_edit.fits") will be in upper case - whatever its case in the input file - since its values have been changed by dmtcalc.
Since "expression" is a required (automatic) parameter, you may be tempted to enter something like:
unix% dmtcalc evt2.fits evt2_edit.fits "energy=((float)pha)*0.1"
This will fail because the parameter parser interprets this as setting a (nonexistent) dmtcalc parameter named "energy" to "((float)pha)*0.1".
dmtcalc evt2.fits evt2_edit.fits expression="newpha=(pha+(long)(1.25e-3*(time-TSTART)))"
The contents of the NEWPHA column are set equal to the PHA column added to the integer part of
The data type of the output column is integer (Int4), since the PHA column is an integer and the time expression has been converted (also called cast) to an integer value (here of type "long").
dmtcalc evt2.obs1,evt2.obs2 evt2.combined expression="newpha=(pha+(long)(1.25e-3*(time-TSTART)))"
This does the same as the previous example except that the input is now two files - evt2.obs1 and evt2.obs2. There is only one output file - evt2.combined - which contains the NEWPHA column. The row order of the output image is determined by the order of the files in the input stack: the first files contents, then the second file, etc until all the input files have been processed. The dmsort tool can be used to change the row order of the output file if required.
dmtcalc lc.orig lc.smoothed expression="rate=(rate:9-0.123)"
Here the file lc.orig (which is assume to be a lightcurve so contains a rate column) is converted to lc.smoothed in which the RATE column is equal to the smoothed - by 9 rows - value of the rate and then a dc offset (0.123) is subtracted from it. The value for the dc offset would have to be calculated prior to running the program - e.g with dmstat as shown in the following csh/tcsh example
unix% dmstat "lc.orig[cols rate]" sigma- unix% set dc=`pget dmstat out_mean` unix% dmtcalc lc.orig lc.smoothed expression="rate=(rate:9-$dc)"
dmtcalc evt2_bary.fits evt2_phase.fits @calcphase.lis
The expression can also be stored in an ASCII file and accessed as a stack. Here we use the contents of calcphase.lis to create an output file in which two new columns have created - PHASE and GPHASE - which contain phase information for the events. The contents of the stack file are:
unix% cat calcphase.lis .dtime=(time-TSTART) GPHASE=.dtime*19.794885 PHASE=GPHASE-(long)GPHASE
We have also included the use of a temporary variable (here .dtime) even though it is not really necessary for this particular example.
The results are only valid if the frequency of the source is 19.794885 Hz (and it's period is not changing, and you have applied the barycenter correction to the events file). For more details on using dmtcalc to add phase information to an event file see the Phase Resolved Spectroscopy thread on the CIAO website.
dmtcalc evt2_bary.fits evt2_phase.fits expression=".dtime=(time-TSTART),GPHASE=.dtime*19.794885,PHASE=GPHASE-(l ong)GPHASE"
Here we repeat the previous example but give the expression directly - i.e. without the use of a stack file.
dmtcalc evt2.fits evt2_edit.fits @randpha.lis
unix% cat randpha.lis newpha= pha + (long) \ ( 1.25e-3*(time-TSTART) + 10*(#rand(1034)-0.5) )
Here we define the expression in an external file and use the "\" character to indicate a continuation line. The contents of the NEWPHA column are set equal to the PHA column added to the integer part of
1.25e-3*(time-TSTART) + 10*(#rand(1034)-0.5)
where #rand(1034) means a random number between 0 and 1 where 1034 is used to seed the random number generator. This seeding only happens once - ie it is not re-set for each row of the table.
The result is similar to the earlier example, except that a random element of +/-5 has been added onto the calculation.
dmtcalc srclist.wav srclist_filt.wav expression="if(psfratio==#nan)then(psfratio=0.0)"
Here we use the support for conditional expressions to filter out the NaN values from the PSRATIO column of srclist.wav, replacing any such values by 0.0.
In order for the expression "if(a)then(b)" to work correctly, the same column must be used in both sections of the comparison.
dmtcalc srclist.wav srclist_filt.wav expression="scol=shape~='cir'"
This expression creates the column SCOL which contains boolean values: TRUE (1) or FALSE (0). The row entry is true if the entry shape column contains the string "cir" and false otherwise. The "~=" operator returns true if the string on the left-hand side contains the string given on the right-hand side.
Detailed Parameter Descriptions
Parameter=infile (file required filetype=input stacks=yes)
Input file to be operated upon. A copy will be made of this file, with any additions/modifications indicated by the expression parameter.
Unless an extension is specified when the file is input, the first table extension will be used. A stack of files may be input, but they must all have the same structure - ie have the same extension name and contain the same columns - since the columns in the first file are assumed to exist - in the same order - in all subsequent files. The behaviour of the tool is not guaranteed if this condition does not hold.
When a stack is supplied, the input files are processed in the order they are supplied to create one output file. The dmsort tool can be used to change the order of rows in the output file.
Parameter=outfile (file required filetype=output)
Name of output file to create.
The output file will have the same extension name as that of the first input file. All the keywords and subspaces in the first file will be copied to the output, in addition to any new columns defined in the expression parameter.
Parameter=expression (string required filetype=input stacks=yes)
A stack of expressions to be evaluated for each row in the input file.
If many expressions are desired, then they should be put into an ascii file, and input as @filename. There is no limit on the number of expressions that can be evaluated. When providing expressions on the command line, note that spaces may be interpreted as stack separators, which can produce unexpected error messages. See "ahelp dmmath" for a description of the syntax allowed here.
Parameter=clobber (boolean default=no)
Clobber the output file?
Parameter=verbose (integer not required default=0 min=0 max=5)
Level of verbose information printed during processing.
Creating Vector Columns
In order to make a new vector column with dmtcalc, it must be used in conjunction with other CIAO tools. Here is a method for creating a vector column "skymod(xmod,ymod)":
First, create the skymod column and generate a new output file (temp1.fits), then filter the original sky column out of that file (temp2.fits):
dmtcalc old.fits"[cols x,y]" temp1.fits expr="xmod=...,ymod=..." dmcopy temp1.fits"[cols -sky]" temp2.fits
The Data Model filter "[cols x,y]" indicates that only those two columns should be included in the new output file. This means that there will be only one vector and one MTYPE/MFORM pair. Update these keywords with the new vector and component names:
dmhedit temp2.fits filel="" op=add key=MTYPE1 value='skymod' dmhedit temp2.fits filel="" op=add key=MFORM1 value='xmod,ymod'
Now paste the rest of the columns from the old file to the new one:
dmpaste old.fits temp2.fits new.fits
Whitespace in Expressions
When an expression is given on the command line - so not as a stack file - then spaces are important. This is because spaces are treated as stack separators - as discussed in "ahelp stack" - which is unlikely to be the desired behaviour here.
expression="energy= (float) pha / 10.0"
will not work. You have to say either
- i.e. remove all the spaces - or surround the expression in ()'s, since this stops the checks for stack separators. This means that
expression="energy=( (float) pha / 10.0 )"
is a valid expression.
An alternative solution is to write the expression in a text file ("expr.lis" say) and then set
since the white space in stack files is not important.
See the bugs page for this tool on the CIAO website for an up-to-date listing of known bugs.
Refer to the CIAO bug pages for an up-to-date listing of known issues.