1. Clocking and the DEA.

At the beginning of each science run, low-level CCD clocking sequences are copied from the ACIS Back End Processor (BEP) into Sequencer RAM (SRAM) in the video boards. So is the higher-level Pixel Ram (PRAM) code, which executes SRAM instructions in complete charge transfer sequences.

Charge accumulates in the 1026 row x 1024 column image store on each active CCD. The accumulated charge is transferred, all rows together in parallel, into the 1024 x 1024 frame store. To prevent a sudden large power draw, the transfers are staggered, with only one chip clocked at a time. Each chip's transfer takes 0.04104 seconds, a period referred to as the smear time. To transfer the maximum number of six chips, with five smear intervals, therefore takes 0.2052 seconds.

The data is then transferred from frame store to the video boards in the Detector Electronics Assembly (DEA) while charge for the next frame accumulates in the image store. Transfer to the DEA occurs simultaneously for all active chips. Charge moves down the frame store a row at a time in parallel, and then the bottom row is clocked serially through four readout nodes, a pixel at a time. (In 2x2 summing, which has not yet been used for a science run on orbit, framestore rows move two at a time, and pixels are read out of the nodes two at a time as well.) At the end of each row, the number of overclock pairs requested by the parameter block are clocked through each node. Finally, there's a horizontal synch pseudo-pixel for each row, and a vertical synch for each frame.

The video boards in the DEA digitize the analog charges clocked out of the CCDs. To each 12 bits of digitized pixel data, they append a 4-bit pixel code (start image, start row, pixel value, overclock, or ignore), prescribed by PRAM. The resulting 2-byte words are multiplexed to the appropriate Front End Processer (FEP) in the ACIS Digital Processor Assembly (DPA).

2. FEP processing.

Each FEP contains the following special storage areas:

• The image buffer - ACTEL firmware stores pixel values here as they arrive from the DEA.
• The bias buffer - FEP software stores bias values here, and for each pixel stores a corresponding bit value in the parity plane.
• The parity plane - FEP software checks entries in this plane while processing each exposure, to detect any corruption in the bias map.
• The threshold plane - firmware sets a bit here whenever a pixel is effectively above threshold (specifically, when the pixel's pulse height amplitude > bias value + threshold + delta_overclock).
• The ring buffer - FEP software stores exposure records and science data records in this circular queue, for pickup by the Back End Processor.
• The FEP/BEP mailboxes - These are areas in FEP general-purpose memory that are used by the Mongoose processors in both units to communicate with one another.

The BEP accesses each of the FEP's special storage areas via hardware mapping of FEP to shared BEP memory. The hardened areas of FEP memory (D-cache and I-cache) are not directly shared. When it needs to access those, as for FEP loads or certain readFep commands, the BEP does so by inserting commands into the FEP/BEP mailbox.

The FEP ring buffer is of particular interest, and is described in greater detail in section 4.10 of the Detailed Design Specification. Within the ring buffer, FEP records are laid out in an array of 8184 physical records of 32 4-byte words each. Some FEP-to-BEP record types will utilize the whole 32 words; some (e.g., raw rows and raw histograms) will span several physical records; and some (e.g., event data records and especially exposure records) are less efficient, using only a fraction of a physical record.

The mailbox maintains one pointer to the next ring buffer record to be read by the BEP, and a second pointer to the next record to be written by the FEP. Only the FEP can write to the ring buffer array, and it is inhibited from writing there when the buffer is full (i.e., when the read and write indexes coincide.) The BEP copies each record from shared memory to its own private memory in order to process it, updating the read pointer and thus freeing space again in the ring buffer. When the ring buffer is empty, or the BEP has processed 100 records, it moves on to the next FEP's ring buffer.

Nine ring buffer record types are defined. Of these, one (fiducial pixels) was never implemented in the BEP, one (errors) accounts for a negligible portion of the traffic, and two (histogram and CC 1x3 events) have never been used since orbital activation. Event histograms are compiled by the BEP from TE 3x3 event records in the FEP. That leaves the following types:

• Exposure record. Sent as soon as an image frame is received. Occupies 7 32-bit words, consuming 1 ring buffer record.
• Exposure end record. Sent once an image frame has been processed (or an output node in raw histogram mode), the exposure end record tallies threshold crossings and parity errors. Occupies 4 words, consuming 1 ring buffer record.
• 3x3 event record. Both TE and CC 3x3 events are reported in this format. Each event is placed in its own record of 11 words, consuming 1 ring buffer record.
• 5x5 event record. These are used to report TE 5x5 events. Each event is placed in its own record of 27 words, consuming 1 ring buffer record.
• Raw mode pixel record. Each raw mode pixel record records the pixel values for one row of the CCD, including overclocks. Occupies 573 words (18 ring buffer records), allowing for around 450 rows before the ring buffer fills.

3. BEP processing.

Bias telemetry - Once a bias map is calculated, the bias thief thread in the BEP goes through the FEPs in order and reads their bias maps via the FEP/BEP memory-mapped interface, starting with the row furthest from the framestore. The ring buffer is not involved. For TE modes, unless biasCompressionSlotIndex in the parameter block is 255, the bias thief compresses the data using the Huffman table in the specified slot. From the resulting data, it constructs dataCcBiasMap or dataTeBiasMap telemetry packets. A packet must contain a whole number of rows, and if the "compression" algorithm expands a single-row packet to more than 4096 bytes, that row will be transmitted in an uncompressed packet.

Depending on how successfully compression goes, each dataTeBiasMap packet may hold from 1 to 10 rows of bias data. (Compression has been surprisingly good, and we get 9 or 10 rows pretty regularly, with a compression factor a bit under four.) Total telemetry in the TE case is therefore (# of chips) * (# of rows per chip) * (1 / rows per bias telemetry packet) * 2608 bytes. For a full six-chip run, this works out to aboout 9.5 megabytes uncompressed.

Data packets

Event data - There are three words (12 bytes) of overhead in each data packet for raw and event processing modes. Any one packet contains data for only a single CCD, and apart from event histograms a new packet is started for each frame. Data types consume telemetry at the following rates:

Telemetry buffers.

Each packet class of the BEP software has access to its own fixed pool of telemetry buffers within BEP bulk memory. (Originally, each packet class corresponded to an execution thread. With the advent of the "untricklebias" patch, the science thread began handling both science and biasThief buffers.) A thread will allocate a buffer from its pool, format a packet within the buffer, and "post" the buffer, registering it with an overall telemetry queue. The contents of buffers in the queue get sent to the RCTU on a first-in, first-out basis. The only priority mechanism among threads is the relative size of their buffer allocations. When a packet has been completely read out by the RCTU, its buffer is freed up for re-use by the thread that owns it.

Accordingly, when telemetry gets backed up, a thread may find it has a packet ready to write, but no free buffer within the appropriate telemetry pool. It must wait for a fresh buffer of the proper type to become available. While waiting, command manager and housekeeping threads may be forced to drop further packets. The number of dropped software housekeepig packets, and the opcode of the most recent dropped command, are noted in software housekeeping. When raw or event mode science buffers are sufficiently backed up, FEP ring buffers will fill, and the FEP will drop new frames.

The threads, the size of their buffer allocations, the number of packets those buffers can hold, and the total telemetry involved, are as indicated in the table below, adapted from pp. 125-126 of Reference [2].

Table notes:

1. Sizes are in bytes, and "telemetry when full" is a high-end estimate for packet types of non-deterministic volume.
2. Packets are often much shorter than the allocated buffer into which they are placed, so that, for example, the fact that the total size of the science buffer pool is 819K bytes does not mean that a full telemetry buffer will produce 8192K bytes of telemetry.
3. Messages from the Startup thread, and fatal messages, have their own telemetry buffer pools, but are not included in the table because of their negligible size.
4. The replies to dumpSysConfig and dumpHuffman commands are bepReadReply packets. They are therefore part of the Memory Manager's telemetry allocation pool.

4. Science run completion

The ACIS instrument receives a stopScience command. What kinds and quantities of telemetry will it try to send to close out the run? And what risks are there if the next observation starts too soon?

Residual data

Should the stopScience arrive while bias is being taken or telemetered, ACIS will try to finish sending the entire bias map. Otherwise, the outstanding data may consist of the combination of

• data in the CCD frame store, still being transferred to FEPs from the final frame
• data in the FEP image buffer
• data in the FEP ring buffers
• data in BEP telemetry buffers

Event modes. Faint, very faint and graded modes produce data packets of varying length. A fresh data packet begins with each new frame, and rarely will reported events fill out an entire 2048 byte science buffer. For each frame of each chip, there's also a short 18-word exposure record. Utilization of the science telemetry buffers by these modes is therefore fairly inefficient; their buffers will drain more rapidly than for raw and event histogram modes. Since their "frames" are half the size, CC modes will have only half the events available to fill out a frame's data packet, so their telemetry buffers will drain about twice as fast as for TEs.

Raw mode. The raw mode compression factor as of this writing is about 3.4. Background radiation being around its maximum, we may take that as an effective upper limit for now. Accumulated radiation damage may be expected to lower the compression factor slowly.

The worst case occurs when the ring buffers are full of data from the previous raw frame, and have just begun to receive data from a new frame. Total telemetry consists of six near-full FEP ring buffers (450 rows, compressible to about 200K bytes for each FEP), fullscience telemetry buffers (400 buffers at about 1400 bytes each), and six complete raw frames. For TEs, those frames will occupy ~ 440 * 6 = ~2.7 compressed megabytes; for CC, which uses "frames" of only 512 rows, the figure is ~ 1.9 compressed megabytes. Altogether, that yields a total of 4.6 (resp 3.2) megabytes. In format 2, the data is transmissible in 1570 seconds (26.2 minutes) for TE and 1070 seconds (18 minutes) for CC.

Event Histogram mode. Raw mode and event modes package their data into telemetry packets about as soon as they get it from the FEPs. They will allocate a telemetry buffer, format data into it, put it into the queue, and get ready to allocate the next one.

Event histogram mode uses a different scheme. All of the buffers needed to transmit a set of histograms are allocated at once. Event tallies are updated in place within those buffers until the histogram is complete, then the buffers are placed into the queue. While the histogram is being read out, a second set of buffers is allocated and the accumulation of the next histogram begins.

Accordingly, overall histogram size is limited by the need to store two full histograms within the science telemetry buffers. Each packet uses one buffer; the histogram for each node takes up nine packets (the final one rather small, but it uses a whole buffer). So each chip uses 9 * 4 = 36 science telemetry buffers, and five chips use 180 buffers. Six chips would use 216; since there are only 400 science buffers, that would preclude storing two full sets of histograms simultaneously. Consequently, we have limited our event histogram modes to five chips.

The bottom line

What are the consequences, should telemetry from one observation continue into the next? They aren't dire. In the worst cases, the first science run will be terminated prematurely. The result will be a bias map, or a set of frames, or a histogram, being only partially transmitted, which is likely to confuse CXC ground software; followed by an error flag in the science report packet.

In particular, except temporarily for bias telemetry until the busCrash patch is incorporated, we do not need to be concerned about a FEP bus crash. Those have only occurred, and can only occur, when a FEP becomes unreadable for some reason other than a commanded power shutdown. When the BEP receives a WSPOW command which powers down the FEPs, it notes the change in its own internal tables. Every time the BEP attempts to read data from a FEP, it first checks its own records to see whether that FEP is powered off or otherwise unavailable. (At least, this was the intent of the software design. A bug which causes a bus crash if FEPs are powered down during a bias computation will be fixed by the busCrash patch.)

Likewise, lingering processing from the previous observation cannot trigger the sort of contention for FEP memory access that can lead to latchups. The moment a new startScience command is received, any extant run is clobbered, and stops all attempts at processing, other than issuing the scienceReport packet.

Listed below, in descending order of severity, are the ill effects that can result from too short an interval between a stopScience and the subsequent observation.

The first three items result in different error messages, but are otherwise quite similar: data transfer from the FEPs to the BEP is interrupted, entailing loss of data. If a bias is interrupted, bias maps will be incomplete; if raw or event modes are interrupted, telemetry will include incomplete frames. The sixth column of table 4.1 tells how long it takes, in format 2, for FEPs to clear once the stopScience command arrives. If a bias was in progress, the transmitted bias map will be incomplete. Raw modes and event modes will telemeter complete packets, but only partial data for final frames. Event histograms will be unaffected.

The BEP maintains a pair of system configuration tables, one indicating requested state and one indicating actual state. On receipt of a power command, the BEP will begin switching power on a video board or FEP once a second until the actual state matches the request. If the BEP finds a FEP requested in the parameter block is powered down, it notes a FEP error in software housekeeping (either a SWSTAT_FEPREPLY_POWEROFF, SWSTAT_FEPLOCK_POWEROFF, or SWSTAT_FEPREC_POWEROFF, depending on what the BEP was doing when it found out.) Then it continues polling the remaining boards. If it ever finds all desired FEPs are powered off, it will terminate the run, placing a FEP error termination code in the scienceReport packet.

1. First run clobbered. FEP data access is cut short by the second run's startScience command. The termination code in the science report will be SMTERM_CLOBBERED. In runs with bias, this can't happen, since FEP access will be terminated by the WSPOW00000 command (see case below). In a no-bias run, it will happen only if the first and second runs have at least one FEP in common. The science report's fepErrorCodes will contain 1's for those FEPs which were powered off.
2. First run terminated by FEP errors. FEP data access is cut short by WSPOW commands which leave no active FEPs. The termination code in the science report will be SMTERM_FEP_IO_ERROR. S/W housekeeping contains more detailed error codes. This will be the result when the FEPs are still being accessed when a WSPOW command arrives which powers off all the FEPs used by the first run.
3. First run terminates normally, with FEP errors. This is the no-bias run case in which the first and second runs use different FEPs, but have some in common, the contents of the FEPs unique to the first run are not read out successfully by the time the WSPOW command arrives, but the contents of the common FEPs are successfully read before the second run's startScience. The science report will contain an SMTERM_STOPCMD as usual, but the fepErrorCodes will be toggled on for the FEPs which could not complete.
4. Dropped packets. There are no errors from the science thread, and all the data is intact, but command echoes are dropped past the fourth ACIS command (typically the pblock load.) Recall that the command manager is allotted only four buffers. The first of these will emerge in telemetry only after everything before it in the telemetry queue. That means the entirety of BEP telemetry buffers must clear before the next command echo can be packaged. For bias maps, "saturated" bias thief telemetry buffers will clear in 22 seconds in format 2. For other run types, it is the science buffers which must clear. In format 2, that will take 3.1 minutes for raw modes, 3.8 minutes for event histograms, 1.5 minutes for TE 5x5, and up to 1.2 minutes for other event modes.
   The nine Huffman table dump packets for the second run will also emerge slowly, since the memory manager thread is only allowed one telemetry buffer. They will not be dropped, however. The memory manager will just wait patiently in queue until it gets its chance.
5. Packets appear in telemetry after the obsid change. Officially, this is supposed to be problematic for data processing. According to Joy Nichols, it causes no problems in practice, because CXC data processing doesn't break ACIS telemetry at the change of obsid. Rather, the break comes upon receipt of the scienceReport packet from the earlier observation.

This final table shows how long it takes to drain telemetry for the main modes. The numbers are rough upper limits; under many circumstances, the values will be significantly lower.

Table notes:

1. "Telemetry in each FEP" is an upper bound for contents of the ring buffer, plus contents of one frame which has just begun processing. It estimates not the space occupied within the FEP, but the resulting telemetry, after any event filtering and compression. between two thirds and nine tenths of events will be filtered, with heavier filtering for high background and frontside chips. This table uses a conservative two-thirds estimate.
2. "Telemetry in all FEPs" assumes 6 FEPs are in use. There is some rounding up.
3. "Filled BEP Telemetry buffers" - With particularly bright sources, such as the internal CTI source, faint modes will have more valid events per frame, packing will be more efficient, and the BEP science buffers may hold as much as 210K bytes. Since ring buffers are unaffected, this won't alter the time required between science runs.
4. "Total bytes" is the sum of the preceding two columns, except for raw modes, where a single-chip run is assumed. For your particular observation, multiply "telemetry in each FEP" by number of active chips by the fraction of a chip covered by any subarray, and add that to "Full BEP Telem buffers". (A six-chip TE raw mode with full frames would take up to 25 minutes to clear the FEPs into the telemetry buffers, 28 minutes for a full readout. The corresponding figures for a 6-chip CC raw mode are 14 and 17.1 minutes.)
5. Times assume format 2. To get the corresponding format 1 readout times, multiply by 47.
6. The least conservative figures in the table are the TE biases. Under terrible compression conditions, we have seen a handful of cases which took nearly 15 minutes to transmit.