DEA-A Anomalous Shutdown

What is it?

The DEA-A shuts down anomalously, presumably due to a spurious command.

Note

The diagnosis and response to this anomaly assumes that the active DEA is DEA-A.

When did it happen before?

The DEA-A has shutdown only once in the mission, in 2005:

  • September 15, 2005, 2005:258:23:31:29, obsid 6221

Will it happen again?

It appears it could happen again, but one occurrence provides little guidance.

How is this Anomaly Diagnosed?

Upon review of the telemetry data leading to the anomaly, within a major frame (32.2 seconds), one should see:

  • 1DEPSA (DEA-A Power Supply On/Off) change from 1 to 0 (On to Off)

  • All DEA-A Analog Voltages (1DEP3AVO, 1DEP2AVO, 1DEP1AVO, 1DEP0AVO, 1DEN0AVO, 1DEN1AVO) drop to 0.0 +/- 0.5 V

  • 1DEICACU (DEA-A Input Current) drop to < 0.2 A (this value is noisy, so take an average)

  • DEA-A POWER should go to zero

  • 1DE28AVO (DEA-A +28V Input Voltage) is expected to have a small uptick, ~0.5 V, consistent with the load suddenly dropping to zero

These would be seen to remain in off/powered-down states when reviewing current values on our Real-Time Telemetry pages. All other hardware telemetry should be nominal. Older data can be examined from the dump files or the engineering archive.

To extract information from the dump data, run ACORN on it as per the instructions in “Running ACORN on data dumps in the case of an anomaly (04/06/16)”. Information from the tracelog files written by the ACORN tools can be plotted using the Python or command-line interfaces to ACISpy, see below for details. Alternatively, one may use the MAUDE query utility.

What is the first response?

Our real-time web pages will alert us and the Lead System Engineer will call us. We need to:

  • Send an email to the ACIS team at the official anomaly email address. If it is off-hours, call Peter and Bob.

  • Send an email to sot_red_alert@cfa announcing that the ACIS team is aware of the DEA-A shutdown and is investigating, and that a telecon will be called when more information is available.

  • Contact the GOT Duty Officer to inform that we need the dump data as soon as possible and to email or call us when the dump file is available.

  • Process the dump data and make sure that there is nothing anomalous in the data BEFORE the shutdown. We want to know if a new occurrence looks just like previous occurrences. If yes, it should appear as if in one frame the DEA-A turned off.

  • Once analysis of the dump data is complete, convene a telecon at the next reasonable moment with the ACIS team and review the diagnosis. The MIT ACIS team (Peter Ford, Bob Goeke, Mark Bautz, and Bev LaMarr) should also be included in the discussion, either in the telecon or via email. If the diagnosis is consistent with previous DEA-A anomalies, proceed with the recovery. If the diagnosis is not consistent with previous DEA-A anomalies, stop and start a more involved analysis with the ACIS team.

  • As soon as the time of the shutdown is known, inform sot_yellow_alert@cfa.

  • Identify whether or not additional comm time is needed and if so ask the OC/LSE to request it.

  • Send an email to sot_red_alert@cfa and call a telecon with the FOT, SOT, and FDs to brief them on the diagnosis and the plan to develop CAPs to recover.

  • Prepare the CAPs and submit them for review to capreview AT ipa DOT harvard DOT edu, and cc: acisdude. It will also be necessary to call the OC/CC to determine which number should be used for the CAPs. The CAP should be executed during an interval with no other ACIS commanding. If that is not possible, the team can discuss running an SCS-107. This main recovery CAP will have the following steps:

    A CAP to update txings values from their defaults to most-recent settings should follow the main recovery CAP if possible. A template for this is in acis_docs/CAPs: CAP1622_TXINGB_SETPARAMS.

  • Execute the CAP at the next available comm.

  • Write a shift report and distribute to sot_shift to inform the project that ACIS is restored to its default configuration.

Impacts

  • Until the DEA is powered back on, science operations will be interrupted.

  • After DEA is powered back on, the focal plane temperature will be unregulated and possibly uncalibrated. Recovery requires a BEP warmboot and setting the focal plane temperature to -121 \(^{\circ}\rm{C}\).

  • The warmboot of the BEP will reset the parameters of the TXINGS patch to their defaults. If not updated during initial recovery as above, txings settings should be updated as soon as possible via CAP (see CAP 1622) or SAR to prevent undesired radiation shutdown.

Relevant Procedures

SOT Procedures

FOT Procedures

  • SOP_ACIS_DEAA_ON (PDF) (DOC)

  • SOP_ACIS_WARMBOOT_DEAHOUSEKEEPING (PDF) (DOC)

  • SOT_SI_SET_ACIS_FP_TEMP_TO_M121C (PDF) (DOC)

FOT Scripts

CAPs

Note

The response to the first occurence of the DEA-A anomaly did not include all the steps above and included additional testing of S0 and commanding of a CTI run, neither of which is necessary for an identical repeat occurrence. Response to future DEA anomalies should follow the above steps.

  • CAP 980 (DEA-A Recovery) (PDF)

  • CAP 981 (ACIS DEA Warm Boot) (PDF)

  • CAP 1622 (Update TXINGS Parameter Values) (acis_docs/CAPs/CAP1622_TXINGB_SETPARAMS.pdf) (acis_docs/CAPs/CAP1622_TXINGB_SETPARAMS.docx)

Relevant Notes/Memos