HRC Update
Grant Tremblay, Ralph Kraft, Paul Nulsen, Dan Patnaude, Tom Gauron, Almus Kenter, Ken Gage, Brad Bissell, Gerald Austin, Tom Aldcroft, & Sabina Hurley (on behalf of the Chandra HRC and Flight Operations Teams)
The HRC team is delighted to report that the instrument is healthy, in nominal operations, and continuing to enable world-class Chandra science. This is especially welcome news as it comes in the wake of the most eventful period in the instrument’s long voyage of discovery.
At 3:45:54 UTC on August 24, 2020, after 21 years of essentially nominal operations, the HRC suffered from a major electronics anomaly of unknown cause, leading to off-nominal voltages in its +/- 15 V power supply bus. This caused a cascade of critical issues throughout the instrument, including corruption of all secondary science and housekeeping data, a steep temperature rise and drop (respectively) in the central and forward electronics assemblies, and significant increased noise in the +24 V bus. While the anticoincidence shield did continue to function nominally throughout the anomaly, the instrument itself could not be used for science in this state. A timeline of the initial anomaly is shown in Figures 1 and 2.
Figure 1: The First Anomaly. The analog engineering telemetry MSIDS 2P15VAVL and 2N15VAVL are plotted vs. time (in Date UTC). The gray vertical line marks the time of the anomaly (03:45:54 UTC on August 24, 2020). Note that the scatter in the voltage reading also increased.
Figure 2: Voltages and temperatures during the first anomaly. Top: The +24 V, +15 V, +5 V, and -15 V bus monitor readings are plotted vs. time (Date UTC) in yellow, blue, green, and red points (respectively). The time of the first anomaly is marked by the leftmost vertical line. The rightmost vertical line marks the time we manually shut the instrument down. Bottom: All HRC temperatures plotted along the same time window as shown in the top panel. We also plot the spacecraft pitch as a faint gray line, illustrating that the spacecraft was at a dwell/constant pitch before, throughout, and for several hours after the anomaly. Temperature changes observed immediately following the anomaly are therefore likely due almost entirely to the anomaly itself. Note especially that, while the CEA Box (2CHTRPZT) warmed dramatically following the anomaly, the Forward Electronics Assembly (FEA) Box and +Y temperatures (2FRADPYT and 2DTSTATT) cooled. There appears to be a break in the slope of the CEA Box warming trend.
Our initial fault-tree analysis for the root cause suggested that there was a problem with the +/- 15 V power supply bus on Side A1. Specific hypotheses included:
1. A single event latch in the +/-15V DC-DC Convertor or one of the components downstream on the +/-15V bus.
2. A failure of the +/-15V DC-DC Convertor or one of the components downstream on the +/-15V bus.
3. A hard short to ground, thought to be unlikely since the boards were conformally coated prior to assembly.
In coordination with our colleagues at NASA Headquarters and the NASA Marshall Space Flight Center, the Chandra Team prepared a detailed response plan optimized to minimize the risk posture of the instrument and spacecraft at all times, with the goal of returning both HRC and ACIS to science as soon as safely possible.
Given the likelihood of hypothesis (1) above, our first action was to attempt a low-risk controlled reset of the Side A power supply bus, which we executed successfully via a Command Action Procedure (CAP) on August 26, 2020. To the immense relief of the team, this power cycle indeed returned the bus voltages and all other telemetry to nominal values, and the instrument appeared stable throughout the next morning’s DSN support. However, 11 hours and 13 minutes after the reset, the anomaly occurred for a second time, in a nearly identical manner, effectively ruling out the single event latch hypothesis.
We again initiated a controlled power-down of the instrument immediately upon observing the anomaly’s reappearance at the start of the 5:20pm EDT comm pass on August 27. Following a number of (virtual) meetings with the Chandra Flight Team and Project Office at NASA MSFC, our next course of action was to swap to redundant Side B electronics, a major spacecraft activity that would require an immense amount of planning by the HRC and Flight Operations Teams.
The Side B swap was successfully executed on August 31, 2020, restoring low voltage power to the instrument, and warming it from the mission-low temperatures it had reached during its shutdown period2. Since this was the first time the instrument had ever operated using B-side electronics in flight, this major milestone was the start of a month-long recovery process, during which we carefully monitored instrument stability and carried out the cautious, stepwise reactivation of both HRC detectors with a series of carefully planned, real-time spacecraft activities. In order to allow Chandra to return to science with ACIS alone while we monitored instrument stability and planned these activities, we performed a safe, controlled power-up of the HRC Anticoincidence Shield on September 7. Chandra returned to ACIS-only science on September 12, ending the longest science operations suspension in the history of the mission.
Figure 3: A timeline of the full anomaly recovery period, including stable and nominal telemetry following the swap to Side B electronics on August 31.
On September 28, 2020, we powered-up the HRC-I using the B-side power supply bus for the first time, and performed a second checkout observation on October 18 (Figure 3). One week later, we powered up the HRC-S, and completed its checkout on November 9. On November 18, 2020, with agreement from our colleagues at NASA MSFC, the HRC was formally approved for return to science, ending an intense 86 day anomaly recovery period. With many thousands of lines of code written, thousands of person-hours of work, and hundreds of hours of (virtual!) meetings, Chandra had fully returned to nominal operations with both instruments. The entire Chandra team celebrated this major milestone for our beloved observatory. We show the “second light” images of the HRC-I and HRC-S in Figure 4.
Figure 4: Two of the "second light" images for the HRC-I and HRC-S (left and right, respectively). Technically, the first "second light" image for HRC-I was an observation of AR Lac (just as for HRC-S) — we opt to show the second checkout observation here, simply because Cassiopeia A is spectacular (and also the ceremonial "first light" target of Chandra). The HRC-S image is shown in detector coordinates, and therefore shows AR Lac tracing Chandra's standard Lissajous dither. Instrument performance on B-side electronics has been entirely nominal and stable in the 150+ days since the HRC returned to science.
The HRC has been stable and nominal for all of the 150+ days since we returned to science. As standard operating procedure, we closely monitor all new telemetry from the instrument and spacecraft, learning more about the (minor) differences in instrument behaviour using B-side electronics.
Of course, nearly as important as recovery of the instrument is the investigation of the anomaly itself. This process began in earnest the very first moment the anomaly was noticed, and now continues under the auspices of the HRC Anomaly Working Group, which was formally created in the early days of the recovery period. This group consists of members of the HRC and Flight Operations Teams as well as Engineers from the MSFC Engineering Directorate. We have studied effectively all relevant spacecraft telemetry since launch, looked at quarter-century old ground data from the Integrated Science Instrument Module (ISIM) thermal/vac, spoken with various instrument part vendors, and dusted off old log books from the earliest days of the mission. The HRC “Proof of Concept” hardware in our Cambridge lab, nearly identical to the flight instrument, has also been critical in our investigation. The HRC Anomaly Working Group has now investigated an exquisitely detailed fault tree, ruled out critical scenarios, performed detailed electrical and thermal modeling, and is narrowing down on a fundamental understanding of the anomaly’s root cause. We look forward to updating the community with our findings in the coming months.
Finally, it must be noted that the anomaly recovery period — by far the most complex intervention with the HRC since at least Orbital Activation post-Launch — was conducted almost entirely in a work-from-home environment, given the ongoing pandemic. We are grateful for the deft and flexible response of the entire Chandra team, both to the HRC anomaly and to the COVID-19 pandemic as a whole.
Since the HRC’s welcome return to nominal operations, we have executed two CAPs to increase the HRC-I and HRC-S microchannel plate voltage by two steps each. This change — which was always going to be done, independent of the anomaly — is to proactively manage the long-known HRC gain sag effect, characterized by decreased quantum efficiency (QE) and increased spatial variations across both detectors. These changes have recovered a significant fraction of lost gain in the instrument, and instrument behavior following the voltage increase has been entirely nominal. The HRC stands ready to enable many more years of breakthrough Chandra science.
1 The HRC Central Electronics Assembly (CEA) carries redundant “Side B” electronics. These are effectively identical to the “Side A” electronics that had, prior to the anomaly, been used exclusively since launch. The HRC CEA Side B electronics were tested on the ground, but never tested or used in flight.
2 The FA6 Survival Heater was used to keep the HRC fully within safe operational temperatures throughout the entire anomaly. While we did indeed reach mission-cold temperatures during the recovery period, the instrument was always well within safe limits.