Using a CSC Save File in CIAO
CSC Threads
Overview
Synopsis:
The results of a catalog search may be written to a file by selecting the Save toolbar button while the the Results tab of CSCView is open, or by clicking the Save Results to File box in the Query tab before actually submitting a query. The save files output in Tab-Separated-Values (TSV) format consist of headers commented with '#', which contain definitions of the CSC columns in the table, as well as data type and format information. There is a row of CSC column names beneath the header, and beneath that are fixed format, tab-separated columns of data values. TSV files are compatible with Vizier and Aladin, and also with the CIAO Data Model.
This file can now be used directly by CIAO tools as well as being imported into SAOImage DS9. DS9 requires RA and Dec coordinates to be in colon-separated sexagesimal format or decimal degrees while CSCView sexagesimal formatting is whitespace-separated. To change the ouput format in CSCView, go to Edit → Preferences:Output Coordinate Format → Decimal
Last Update: 9 Oct 2019 - Figure and text updates.
Contents
Identify ASCII format
The CXC Datamodel can read and write various ASCII file formats. Many of these can be automatically recognized by first few lines in the file; however, the Tab-Separated-Values (TSV) returned by CSCView needs to be explicitly identified.
To use any CSCView TSV format file users need to specify a kernel option which takes the form:
[opt kernel=text/tsv]
So for example if the output from CSCView was saved as m81.tsv, then users can use this in CIAO using the full Virtual File Syntax would be
"m81.tsv[opt kernel=text/tsv]"
Quotes are likely going to be needed when commands are run on the UNIX command line but should be omitted when a CIAO tool prompts for the value.
The VOTable (XML) format output is not supported by CIAO tools; however, one can use the Python XML parsing routines or other third-party modules to read such files into Python applications and can be imported by DS9. One example is the astropy.io.votable subpackage of Astropy.
Using with CIAO tools
With the virtual file specification discussed above, all the CIAO utilities are now available for use.
Suppose we would like to select all CSC 2.0 master sources from the Orion Nebula Cluster as observed in the ACIS broad band (0.5-7.0 keV). Suppose further that we have used the CSC2.0 WWT application to identify the particular ACIS-based master source stack ID acisfJ0535165m052323_001, which includes all the sources we want to consider. If we search for all master sources within this stack using CSCView, we get a table of 1539 unique sources in this region, which we have saved in the tab-separated-value file orion.tsv. We can now use the various CIAO tools to inspect and analyze these data.
We can start with simply running dmlist to see what columns we selected:
unix% dmlist orion.tsv"[opt kernel=text/tsv]" cols -------------------------------------------------------------------------------- Columns for Table Block orion.tsv -------------------------------------------------------------------------------- ColNo Name Unit Type Range 1 name String[22] Source name in the format '2CXO Jhhmmss.s{+|-}ddmmss[X]' 2 detect_stack_id String[24] Stack Source identifier 3 ra String[12] Source position, ICRS right ascension 4 dec String[11] Source position, ICRS declination 5 instrument String[4] Instrument used for the stacked observations; 'ACIS' or 'HRC' 6 region_id Int4 - Detection region identifier (component number) 7 match_type String[11] Type of match 8 src_cnts_aper_b count Real8 -Inf:+Inf Aperture-corrected detection net counts inferred from the source region aperture 9 flux_aper_b erg/s/cm^2 Real4 -Inf:+Inf Aperture-corrected net energy flux inferred from the source region aperture, bes 10 flux_significance_b Real4 -Inf:+Inf Significance of the stacked-observation detection determined from the ratio of t 11 ks_intra_prob_b Real4 -Inf:+Inf Intra-observation Kolmogorov-Smirnov test variability probability (highest value 12 theta_mean arcmin Real4 -Inf:+Inf Mean source region aperture off-axis angle from all stacked observations unix% dmlist orion.tsv"[opt kernel=text/tsv]" counts 1539
In the first example we see which columns were selected, including the units and comments, as well as the number of rows (i.e. number of sources).
If we wanted to check to see how many sources are unambiguous (i.e. not confused with other sources in the stack) we can simply use the dmfiltering syntax.
unix% dmlist orion.tsv"[opt kernel=text/tsv][match_type=u]" counts 1515
which shows us that 1515 out of 1539 sources are not confused.
If we wanted to do a population study of the sources, i.e. a \(\log{(N)}/\log{(S)}\) analysis, we can use the TSV file with several CIAO tools as shown below:
unix% dmstat orion.tsv"[opt kernel=text/tsv][cols flux_significance_b]" flux_significance_b min: 0 @: 10 max: 460.8999939 @: 771 mean: 28.99595175 sigma: 39.520637864 sum: 43261.960011 good: 1492 null: 47 unix% dmextract orion.tsv"[opt kernel=text/tsv][bin flux_significance_b=1:461:1]" src_sig.fits op=generic clob+ # dmextract (CIAO 4.5): WARNING: Keywords LIVETIME and EXPOSURE not found in file 'orion.tsv' LIVETIME set to default 1.0 sherpa In [1]: load_data("src_sig.fits[cols flux_significance_b,counts]") sherpa In [2]: set_source(exp.ee) sherpa In [3]: ignore(0,3) sherpa In [4]: set_method("moncar") sherpa In [5]: fit() Dataset = 1 Method = moncar Statistic = chi2gehrels Initial fit statistic = 878.783 Final fit statistic = 100.296 at function evaluation 1906 Data points = 458 Degrees of freedom = 455 Probability [Q-value] = 1 Reduced statistic = 0.220431 Change in statistic = 778.487 ee.offset -0.0334259 ee.coeff -0.0525678 ee.ampl 72.8004 sherpa In [6]: plot_fit() sherpa In [7]: log_scale(XY_AXIS) sherpa In [8]: limits(Y_AXIS,0.1,AUTO) sherpa In [9]: set_plot_xlabel("Broad Band Source Significance") sherpa In [10]: set_plot_ylabel("Number of Sources") sherpa In [11]: set_plot_title("CSC ACIS Stacked Results for Orion Nebula Cluster") sherpa In [12]: limits(X_AXIS,AUTO,120.)
Here we have determined the range of the broad band source significance values using dmstat, used that range to extract a histogram of the number of sources with flux significances between 1 and 461 with dmextract and then input the data into Sherpa to fit the profile with an exponential model, as shown in Figure 1.
[Version: full-size]
Figure 1: The result of plot_fit()
Using with SAOImage DS9
Returning to our CSCView search, we can select one or more of the sources under the Search Results tab to find and download CSC 2.0 data products, and in this case we choose a Stack Events Image including all of these sources, namely acisfJ0535165m052323_001N020_b_img3.fits. We can then import our orion.tsv table file into DS9's catalog tool either on the command line:
unix% ds9 acisfJ0535165m052323_001N020_b_img3.fits -catalog import tsv orion.tsv &
or by going to Analysis→Catalog Tool ..., then in the Catalog Tool window go to File→Import→Tab-Separated-Value→orion.tsv. The result is shown in Figure 2
[Version: full-size]
Figure 2: Using DS9
The catalog tool can then be used to select sources, create plots of source properties and participate in SAMP enabled analysis sessions with other SAMP aware applications.
History
06 Oct 2008 | original version |
01 Jun 2009 | updated to include recent changes to CSCview |
11 Aug 2010 | updated for CSCview version 1.1: conversion script updated to accept TSV format save files in place of RDB format; 'cone_distance' catalog parameter has been changed to 'separation'. |
13 Aug 2013 | Rewritten to highlight that CIAO can now read TSV format directly. |
11 Apr 2019 | Updated with CSC 2.0 results. |
09 Oct 2019 | Figure and text updates. |