ACIS Science Run Termination Failure Anomaly

What is it?

The science run fails to terminate when SCS-107 issues two stopScience commands during bias creation.

When did it happen before?

Twice:

  • March 3, 2016, 2016:063:17:11, obsid 17719

  • September 11, 2017, 2017:256:07:53, obsid 19931

Will it happen again?

Hopefully not. The update to buscrash patch, installed in Flight Software Version standard G optional H in 2020 is intended to fix this problem.

How is this anomaly diagnosed?

  • After the execution of SCS-107, the 1STAT1ST ACIS status bit fails to set as expected, from Science Active (0 or RED) to Science Idle (1 or GREEN). The ACIS status bits are available on both the ACIS hardware web pages and the PMON web pages where 1STAT1ST is the seventh digit in the Bilevels display.

  • The 1STAT1ST status bit indicates that the science process failed to terminate and exit when SCS-107 was run.

  • All other indicators (ACIS voltages, input currents, power, temperatures, and SIM position) should indicate that the equipment has shut down as normal for an SCS-107 run, so the instrument is safe. Since SCS-107 issues a WSPOW00000 command that powers down the video boards and the FEPs, the input currents on the DPA side A and B should be:

    • DPA side A, 1DPICACU < 0.6 A

    • DPA side B, 1DPICBCU < 0.4 A

What is the first response?

The anomaly is most likely to be noted by ACIS ops during the first contact after an SCS-107 execution. It does not produce any automated alerts.

We need to:

  • Send an e-mail to the ACIS team (including acisdude, Peter Ford, Bob Goeke, Mark Bautz, and Bev LaMarr) to alert them to the existence of the anomaly.

  • Notify the Chandra Operations team on the telecon after the SCS-107 execution was discovered. If not possible, send email to sot_yellow_alert@cfa describing the situation, including the time of the anomaly and the Obsid when it occured.

  • Prepare the CAPs and submit them for review to capreview AT ipa DOT harvard DOT edu, and cc: acisdude. It will also be necessary to call the OC/CC to determine which number should be used for the CAPs.

    The main recovery CAP will have the following steps:

    • Confirm telemetry format is 1 or 2 and the most recent version of flight software is running

    • Warmboot the BEP and restart DEA housekeeping (SOP_ACIS_WARMBOOT_DEAHOUSKEEPING)

    There is a template CAP in acis_docs/CAPs: CAPXXXX_WMBOOT_HK

    A CAP to update txings values from their defaults to most-recent settings should follow the main recovery CAP if possible. A template for this is in acis_docs/CAPs: CAP1622_TXINGB_SETPARAMS.

  • Execute the CAP at the next available comm.

  • Write a shift report and distribute to sot_shift to inform the project that ACIS is restored to its default configuration.

Impacts

  • The anomaly occurs after SCS-107 execution, so no science is lost.

  • The warmboot of the BEP will reset the parameters of the TXINGS patch to their defaults. If not updated during initial recovery as above, txings settings should be updated as soon as possible via CAP (see CAP 1622) or SAR to prevent undesired radiation shutdown.

  • If a science run is started before a warmboot can be performed, testing on the ACIS Engineering Unit has shown that the initiation of the run will clear the problem, but not without generating an error message due to “clobbering” the hung process.

Note

As of this writing, the latest ACIS Flight Software Patch is standard G, optional I, Version 58. This includes an update to the buscrash patch which should prevent this anomaly going forward.

Relevant Procedures

SOT Procedures

FOT Procedures

  • SOP_ACIS_WARMBOOT_DEAHOUSEKEEPING (PDF) (DOC)

FOT Scripts

CLD Scripts

CAPs

  • CAP 1622 (Update TXINGS Parameter Values) (acis_docs/CAPs/CAP1622_TXINGB_SETPARAMS.pdf) (acis_docs/CAPs/CAP1622_TXINGB_SETPARAMS.docx)

  • CAP 1433 (ACIS BEP Warmboot and DEA housekeeping restart) (PDF) (DOC) (Note the double underscore before wmboot in the filename.)

  • CAP 1381 (ACIS BEP Warmboot and DEA housekeeping restart) (PDF) (DOC)

Relevant Notes/Memos