When R/T and PMON Pages are not Updating

Introduction

This SOP describes the procedure to follow if monitoring a comm and R/T pages and PMON pages are not updating.

0. First Double-Check

Has it been at least 5 minutes so that the R/T pages should be updating?
  • PMON should be updating multiple times per minute.
    • Is the DSN station #54/Madrid?
  • If Madrid, benign troubles are more likely.

    • 1. Verify the DSN schedule

    Linked here: https://asc.harvard.edu/mta/DSN.txt.
    If correct, then move to Step 2.

    2. Check the FOT report page

    Follow along the activities and for indications of delays on the pass by the OC/CC, https://occweb.cfa.harvard.edu/occweb/web/webapps/ifot/ifot.php?r=flightops&t=opsreport&ul=14 (find the relevant support) OR https://occweb.cfa.harvard.edu/occweb/web/webapps/ifot/ifot.php?r=flightops&t=shiftreport (click on ''pass plan'').
    If DSN issues have been reported, go on to step 4a, otherwise proceed to step 3.

    3. Check Snapshot and SOH pages for telemetry

    https://cxc.cfa.harvard.edu/mta/Snapshot/snap.cgi?, and https://asc.harvard.edu/cgi-gen/mta/SOH/soh.html. If those are updating, allow a reasonable time for the PMON and RT pages to catch up, ~15 minutes. If problems persist and DSN issues have by now been reported, go on to step 4a; otherwise, go to step 4b.

    4a. Monitor email/text and the FOT pass report for updates

    ONLY in the rare circumstance that a pass is highly desirable for ACIS and ACIS has coordinated with the OC/CC before the pass, call the OC/CC ~30 minutes or a reasonable time after the scheduled start of comm to inquire about the status of the comm. Note that Chandra can safely miss a routine pass with no risk to the mission or data integrity. Passes have been missed and will be missed in the future. A single missed pass due to DSN problems is not a health and safety concern. A missed pass because the spacecraft did not communicate with DSN is a health and safety concern.

    If the pass is missed, send an email to alert the ACIS ops team using the acisdude mailing address (step 5).

    4b. ONLY if the pass report indicates a successful comm, but the R/T and PMON pages are still not updating

    If some of the R/T web pages and/or PMON pages are updating, then you know the OCC is receiving DSN data and the GOT is processing it. Chances are the network links between the GOT and the computers with non-working R/T and/or PMON pages need to have their links reset by the GOT. Send an email to the GOT asking for the links to those computers be reset and be sure to specify the port numbers.

    If none of the 4 computers are updating on either the R/T web page or the PMON page, you can reduce the possibilities by selecting one of the machines and following the instructions here: https://cxc.cfa.harvard.edu/acis/memos/webpage/ACIS_RT_webpages.html#listen.

    If you are not getting any packets then the link between the GOT and the computer is down. Ask the GOT to reset the link. Chances are that if the machine you selected is not getting any packets at all then you need to ask for a link reset for all non-working machine. If you are getting the 58 byte heartbeat packets then you know the link between the GOT and that machine is working ok.

    If you are in the middle of a comm and you are getting the 58 byte heartbeat packet, then the problem is upstream of the computer.
    Proceed to step 5 if the problem is not resolved.

    5. Failing all else

    Jump on Slack and check in for other ideas from the team. Failing that, notify the acisdudes, summarizing the issue for group-wide input and awareness.

    Back To Top