Prev   Index   Next

kadi: Sat May 24 08:36:53 2025

File: /proj/sot/ska/data/kadi/logs/daily.0/kadi_validate.log

Task Status AgeMaxAge Last Date
kadi NOT OK 0.091.5 Sat May 24 08:36:53 2025

Errors:

File contents:

2025-05-24 08:10:02,904 run_validators: Validating pitch
2025-05-24 08:10:02,905 get_command_sheet_exclude_intervals: Getting exclude times from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv&id=19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ&gid=1681877928
_get_data: Getting data for conlofp between 2025:130:11:54:32.904 to 2025:144:12:25:32.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/CONLOFP.h5
_get_msid_data_from_cxc: Slicing CONLOFP arrays [4295:1148480]
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:10:03,911 tlm: Fetching telemetry for ['pitch_comp'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for pitch_comp between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_comp_data: Getting computed values for pitch_comp
_get_data: Getting data for conlofp between 2025:130:11:39:32.904 to 2025:144:12:40:32.904
_get_msid_data_from_cxc: Using times_cache for ccdm8eng 863264442.088 to 864477702.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/CONLOFP.h5
_get_msid_data_from_cxc: Slicing CONLOFP arrays [3416:1148480]
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
_get_data: Getting data for aopcadmd between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOPCADMD.h5
_get_msid_data_from_cxc: Slicing AOPCADMD arrays [4324:1148480]
_get_data: Getting data for orbitephem0_x between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_X.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_X arrays [5756:9799]
_get_data: Getting data for orbitephem0_y between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Using times_cache for orbitephem0 863264652.088 to 864477492.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_Y.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_Y arrays [5756:9799]
_get_data: Getting data for orbitephem0_z between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Using times_cache for orbitephem0 863264652.088 to 864477492.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_Z.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_Z arrays [5756:9799]
_get_data: Getting data for solarephem0_x between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_X.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_X arrays [5756:9799]
_get_data: Getting data for solarephem0_y between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Using times_cache for solarephem0 863264652.088 to 864477492.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_Y.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_Y arrays [5756:9799]
_get_data: Getting data for solarephem0_z between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Using times_cache for solarephem0 863264652.088 to 864477492.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_Z.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_Z arrays [5756:9799]
_get_data: Getting data for aoattqt1 between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT1.h5
_get_msid_data_from_cxc: Slicing AOATTQT1 arrays [3621:1148480]
_get_data: Getting data for aoattqt2 between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863264652.088 to 864477492.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT2.h5
_get_msid_data_from_cxc: Slicing AOATTQT2 arrays [3621:1148480]
_get_data: Getting data for aoattqt3 between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863264652.088 to 864477492.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT3.h5
_get_msid_data_from_cxc: Slicing AOATTQT3 arrays [3621:1148480]
_get_data: Getting data for aoattqt4 between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863264652.088 to 864477492.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT4.h5
_get_msid_data_from_cxc: Slicing AOATTQT4 arrays [3621:1148480]
fetch: Setting bads because of gaps in aoattqt1 between 2025:144:11:10:35.166 to 2025:144:12:37:02.691
fetch: Setting bads because of gaps in aoattqt2 between 2025:144:11:10:35.166 to 2025:144:12:37:02.691
fetch: Setting bads because of gaps in aoattqt3 between 2025:144:11:10:35.166 to 2025:144:12:37:02.691
fetch: Setting bads because of gaps in aoattqt4 between 2025:144:11:10:35.166 to 2025:144:12:37:02.691
interpolate: Interpolating index for orbitephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt1
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt2
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt3
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt4
interpolate: Slicing on indexes
interpolate: Interpolating index for pitch_comp
interpolate: Slicing on indexes
2025-05-24 08:10:08,817 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:10:08,818 get_states: Getting commanded states ['pitch', 'pcad_mode'] between 2025:130:12:10:03.766 - 2025:144:11:10:30.041
2025-05-24 08:10:08,818 get_cmds: Getting commands from 2025:130:12:10:03.766 to 2025:144:11:10:30.041 for scenario=None
2025-05-24 08:10:08,818 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:08,820 get_cmds: Recent commands not in cache: updating local cmd_events, loads and getting recent commands
2025-05-24 08:10:08,820 get_cmd_events_from_sheet: Getting cmd_events from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv
2025-05-24 08:10:09,089 get_cmd_events_from_sheet: Writing 152 cmd_events to /home/aca/.kadi/cmd_events.csv
2025-05-24 08:10:09,101 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/APR with cache=False
2025-05-24 08:10:09,403 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/APR2825A.pkl.gz
2025-05-24 08:10:09,408 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAY with cache=False
2025-05-24 08:10:09,460 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY0525A.pkl.gz
2025-05-24 08:10:09,465 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY1225A.pkl.gz
2025-05-24 08:10:09,471 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY1925A.pkl.gz
2025-05-24 08:10:09,477 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY2625A.pkl.gz
2025-05-24 08:10:09,482 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY2625B.pkl.gz
2025-05-24 08:10:09,486 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/JUN with cache=False
2025-05-24 08:10:09,516 update_cmd_events_and_loads_and_get_cmds_recent: Including loads APR2825A, MAY0525A, MAY1225A, MAY1925A, MAY2625A, MAY2625B
2025-05-24 08:10:09,519 update_cmd_events_and_loads_and_get_cmds_recent: Load APR2825A has 1894 commands with RLTT=2025:118:00:24:12.365
2025-05-24 08:10:09,522 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY0525A has 1709 commands with RLTT=2025:125:01:52:24.962
2025-05-24 08:10:09,526 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY1225A has 2038 commands with RLTT=2025:131:21:15:39.221
2025-05-24 08:10:09,530 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY1925A has 2695 commands with RLTT=2025:139:00:00:00.838
2025-05-24 08:10:09,534 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY2625A has 49 commands with RLTT=None
2025-05-24 08:10:09,538 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY2625B has 1979 commands with RLTT=2025:146:01:43:21.753
2025-05-24 08:10:09,569 update_cmd_events_and_loads_and_get_cmds_recent: Including cmd_events:
Load not run at 2025:146:01:40:21.753
2025-05-24 08:10:09,571 update_cmd_events_and_loads_and_get_cmds_recent: Processing APR2825A with 1894 commands
2025-05-24 08:10:09,572 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1894 commands from APR2825A
2025-05-24 08:10:09,572 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY0525A with 1709 commands
2025-05-24 08:10:09,572 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1709 commands from MAY0525A
2025-05-24 08:10:09,572 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY1225A with 2038 commands
2025-05-24 08:10:09,572 update_cmd_events_and_loads_and_get_cmds_recent: Adding 2038 commands from MAY1225A
2025-05-24 08:10:09,572 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY1925A with 2695 commands
2025-05-24 08:10:09,573 update_cmd_events_and_loads_and_get_cmds_recent: Adding 2695 commands from MAY1925A
2025-05-24 08:10:09,573 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY2625B with 1979 commands
2025-05-24 08:10:09,573 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1979 commands from MAY2625B
2025-05-24 08:10:09,573 update_cmd_events_and_loads_and_get_cmds_recent: Processing CMD_EVT Load_not_run at 2025:146:01:40:21.753 with 1 commands
2025-05-24 08:10:09,573 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1 commands from CMD_EVT Load_not_run at 2025:146:01:40:21.753
2025-05-24 08:10:09,573 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY2625A with 49 commands
2025-05-24 08:10:09,573 update_cmd_events_and_loads_and_get_cmds_recent: Adding 49 commands from MAY2625A
2025-05-24 08:10:09,956 load_pars_dict: Loaded /export/tom/kadi-validate/data/kadi/cmds2.pkl with 180137 pars
2025-05-24 08:10:09,979 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:09,982 get_cmds: Getting commands from 2025:123:12:10:03.766 to 2025:130:12:10:03.766 for scenario=None
2025-05-24 08:10:09,982 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:09,983 get_cmds: Getting recent commands from cache
2025-05-24 08:10:09,984 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:32,446 get_plot_figure: Compressing Telem data for state pitch
2025-05-24 08:10:34,626 get_plot_figure: Creating Telem scatter plot for state pitch
2025-05-24 08:10:34,668 get_plot_figure: Compressing State data for state pitch
2025-05-24 08:10:36,713 get_plot_figure: Creating State scatter plot for state pitch
2025-05-24 08:10:36,796 get_plot_html: Creating HTML for state pitch
2025-05-24 08:10:36,812 run_validators: Validating off_nom_roll
_get_data: Getting data for conlofp between 2025:130:11:54:32.904 to 2025:144:12:25:32.904
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:10:36,994 tlm: Fetching telemetry for ['roll_comp'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for roll_comp between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_comp_data: Getting computed values for roll_comp
_get_data: Getting data for conlofp between 2025:130:11:39:32.904 to 2025:144:12:40:32.904
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
_get_data: Getting data for aopcadmd between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_data: Getting data for orbitephem0_x between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for orbitephem0_y between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for orbitephem0_z between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for solarephem0_x between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for solarephem0_y between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for solarephem0_z between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for aoattqt1 between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for aoattqt2 between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for aoattqt3 between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
_get_data: Getting data for aoattqt4 between 2025:130:11:43:02.904 to 2025:144:12:37:02.904
fetch: Setting bads because of gaps in aoattqt1 between 2025:144:11:10:35.166 to 2025:144:12:37:02.691
fetch: Setting bads because of gaps in aoattqt2 between 2025:144:11:10:35.166 to 2025:144:12:37:02.691
fetch: Setting bads because of gaps in aoattqt3 between 2025:144:11:10:35.166 to 2025:144:12:37:02.691
fetch: Setting bads because of gaps in aoattqt4 between 2025:144:11:10:35.166 to 2025:144:12:37:02.691
interpolate: Interpolating index for orbitephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt1
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt2
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt3
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt4
interpolate: Slicing on indexes
interpolate: Interpolating index for roll_comp
interpolate: Slicing on indexes
2025-05-24 08:10:37,927 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:10:37,927 get_states: Getting commanded states ['off_nom_roll', 'pitch', 'pcad_mode'] between 2025:130:12:10:03.766 - 2025:144:11:10:30.041
2025-05-24 08:10:37,927 get_cmds: Getting commands from 2025:130:12:10:03.766 to 2025:144:11:10:30.041 for scenario=None
2025-05-24 08:10:37,927 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:37,928 get_cmds: Getting recent commands from cache
2025-05-24 08:10:37,929 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:37,931 get_cmds: Getting commands from 2025:123:12:10:03.766 to 2025:130:12:10:03.766 for scenario=None
2025-05-24 08:10:37,932 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:37,932 get_cmds: Getting recent commands from cache
2025-05-24 08:10:37,933 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:42,283 get_plot_figure: Compressing Telem data for state off_nom_roll
2025-05-24 08:10:44,424 get_plot_figure: Creating Telem scatter plot for state off_nom_roll
2025-05-24 08:10:44,447 get_plot_figure: Compressing State data for state off_nom_roll
2025-05-24 08:10:46,483 get_plot_figure: Creating State scatter plot for state off_nom_roll
2025-05-24 08:10:46,532 get_plot_html: Creating HTML for state off_nom_roll
2025-05-24 08:10:46,544 run_validators: Validating dither
_get_data: Getting data for conlofp between 2025:130:11:54:32.904 to 2025:144:12:25:32.904
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:10:46,724 get_cmds: Getting commands from 2025:130:11:50:02.904 to 2025:144:12:10:02.904 for scenario=None
2025-05-24 08:10:46,724 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:46,725 get_cmds: Getting recent commands from cache
2025-05-24 08:10:46,726 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:46,786 add_exclude_interval: dither: excluding interval 2025:132:02:27:38.000 - 2025:132:02:40:38.000: SRDC
2025-05-24 08:10:46,788 add_exclude_interval: dither: excluding interval 2025:135:02:24:59.000 - 2025:135:02:37:59.000: SRDC
2025-05-24 08:10:46,789 add_exclude_interval: dither: excluding interval 2025:137:11:40:31.000 - 2025:137:11:53:31.000: SRDC
2025-05-24 08:10:46,790 add_exclude_interval: dither: excluding interval 2025:137:18:20:03.000 - 2025:137:18:33:03.000: SRDC
2025-05-24 08:10:46,791 tlm: Fetching telemetry for ['aodithen'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for aodithen between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/AODITHEN.h5
_get_msid_data_from_cxc: Slicing AODITHEN arrays [541:143560]
interpolate: Interpolating index for aodithen
interpolate: Slicing on indexes
2025-05-24 08:10:46,945 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:10:46,946 get_states: Getting commanded states ['dither'] between 2025:130:12:10:09.716 - 2025:144:11:10:24.756
2025-05-24 08:10:46,946 get_cmds: Getting commands from 2025:130:12:10:09.716 to 2025:144:11:10:24.756 for scenario=None
2025-05-24 08:10:46,946 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:46,947 get_cmds: Getting recent commands from cache
2025-05-24 08:10:46,947 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:46,949 get_cmds: Getting commands from 2025:123:12:10:09.716 to 2025:130:12:10:09.716 for scenario=None
2025-05-24 08:10:46,950 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:46,951 get_cmds: Getting recent commands from cache
2025-05-24 08:10:46,951 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:47,662 get_plot_figure: Compressing Telem data for state dither
2025-05-24 08:10:47,914 get_plot_figure: Creating Telem scatter plot for state dither
2025-05-24 08:10:47,915 get_plot_figure: Compressing State data for state dither
2025-05-24 08:10:48,172 get_plot_figure: Creating State scatter plot for state dither
2025-05-24 08:10:48,287 get_plot_html: Creating HTML for state dither
2025-05-24 08:10:48,291 run_validators: Validating pcad_mode
2025-05-24 08:10:48,303 get_cmds: Getting commands from 2025:130:11:50:02.904 to 2025:144:12:10:02.904 for scenario=None
2025-05-24 08:10:48,303 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:48,303 get_cmds: Getting recent commands from cache
2025-05-24 08:10:48,304 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:48,349 add_exclude_interval: pcad_mode: excluding interval 2025:132:02:27:38.000 - 2025:132:02:40:38.000: SRDC
2025-05-24 08:10:48,351 add_exclude_interval: pcad_mode: excluding interval 2025:135:02:24:59.000 - 2025:135:02:37:59.000: SRDC
2025-05-24 08:10:48,352 add_exclude_interval: pcad_mode: excluding interval 2025:137:11:40:31.000 - 2025:137:11:53:31.000: SRDC
2025-05-24 08:10:48,353 add_exclude_interval: pcad_mode: excluding interval 2025:137:18:20:03.000 - 2025:137:18:33:03.000: SRDC
2025-05-24 08:10:48,354 tlm: Fetching telemetry for ['aopcadmd'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for aopcadmd between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOPCADMD.h5
_get_msid_data_from_cxc: Slicing AOPCADMD arrays [4324:1148480]
interpolate: Interpolating index for aopcadmd
interpolate: Slicing on indexes
2025-05-24 08:10:48,569 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:10:48,570 get_states: Getting commanded states ['pcad_mode'] between 2025:130:12:10:03.566 - 2025:144:11:10:30.906
2025-05-24 08:10:48,570 get_cmds: Getting commands from 2025:130:12:10:03.566 to 2025:144:11:10:30.906 for scenario=None
2025-05-24 08:10:48,570 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:48,571 get_cmds: Getting recent commands from cache
2025-05-24 08:10:48,571 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:48,573 get_cmds: Getting commands from 2025:123:12:10:03.566 to 2025:130:12:10:03.566 for scenario=None
2025-05-24 08:10:48,574 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:10:48,574 get_cmds: Getting recent commands from cache
2025-05-24 08:10:48,575 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:10:56,693 get_plot_figure: Compressing Telem data for state pcad_mode
2025-05-24 08:10:58,713 get_plot_figure: Creating Telem scatter plot for state pcad_mode
2025-05-24 08:10:58,717 get_plot_figure: Compressing State data for state pcad_mode
2025-05-24 08:11:00,734 get_plot_figure: Creating State scatter plot for state pcad_mode
2025-05-24 08:11:00,786 get_plot_html: Creating HTML for state pcad_mode
2025-05-24 08:11:00,791 run_validators: Validating simpos
2025-05-24 08:11:00,798 tlm: Fetching telemetry for ['3tscpos'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for 3tscpos between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/sim_mrg/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/sim_mrg/3TSCPOS.h5
_get_msid_data_from_cxc: Slicing 3TSCPOS arrays [135:35826]
interpolate: Interpolating index for 3tscpos
interpolate: Slicing on indexes
2025-05-24 08:11:00,909 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:11:00,910 get_states: Getting commanded states ['simpos'] between 2025:130:12:10:17.916 - 2025:144:11:10:17.069
2025-05-24 08:11:00,910 get_cmds: Getting commands from 2025:130:12:10:17.916 to 2025:144:11:10:17.069 for scenario=None
2025-05-24 08:11:00,910 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:00,911 get_cmds: Getting recent commands from cache
2025-05-24 08:11:00,911 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:00,913 get_cmds: Getting commands from 2025:123:12:10:17.916 to 2025:130:12:10:17.916 for scenario=None
2025-05-24 08:11:00,914 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:00,914 get_cmds: Getting recent commands from cache
2025-05-24 08:11:00,915 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:01,053 get_plot_figure: Compressing Telem data for state simpos
2025-05-24 08:11:01,118 get_plot_figure: Creating Telem scatter plot for state simpos
2025-05-24 08:11:01,121 get_plot_figure: Compressing State data for state simpos
2025-05-24 08:11:01,184 get_plot_figure: Creating State scatter plot for state simpos
2025-05-24 08:11:01,193 get_plot_html: Creating HTML for state simpos
2025-05-24 08:11:01,196 run_validators: Validating obsid
2025-05-24 08:11:01,203 tlm: Fetching telemetry for ['cobsrqid'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for cobsrqid between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm10eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm10eng/COBSRQID.h5
_get_msid_data_from_cxc: Slicing COBSRQID arrays [2162:574240]
interpolate: Interpolating index for cobsrqid
interpolate: Slicing on indexes
2025-05-24 08:11:01,400 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:11:01,401 get_states: Getting commanded states ['obsid'] between 2025:130:12:10:03.566 - 2025:144:11:10:30.650
2025-05-24 08:11:01,401 get_cmds: Getting commands from 2025:130:12:10:03.566 to 2025:144:11:10:30.650 for scenario=None
2025-05-24 08:11:01,401 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:01,402 get_cmds: Getting recent commands from cache
2025-05-24 08:11:01,402 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:01,404 get_cmds: Getting commands from 2025:123:12:10:03.566 to 2025:130:12:10:03.566 for scenario=None
2025-05-24 08:11:01,405 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:01,405 get_cmds: Getting recent commands from cache
2025-05-24 08:11:01,406 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:03,555 get_plot_figure: Compressing Telem data for state obsid
2025-05-24 08:11:04,568 get_plot_figure: Creating Telem scatter plot for state obsid
2025-05-24 08:11:04,571 get_plot_figure: Compressing State data for state obsid
2025-05-24 08:11:05,581 get_plot_figure: Creating State scatter plot for state obsid
2025-05-24 08:11:05,597 get_plot_html: Creating HTML for state obsid
2025-05-24 08:11:05,601 run_validators: Validating letg
2025-05-24 08:11:05,608 tlm: Fetching telemetry for ['4ootgsel', '4ootgmtn'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for 4ootgsel between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/4OOTGSEL.h5
_get_msid_data_from_cxc: Slicing 4OOTGSEL arrays [136:35890]
_get_data: Getting data for 4ootgmtn between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Using times_cache for tel2eng 863265372.088 to 864476772.088
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/4OOTGMTN.h5
_get_msid_data_from_cxc: Slicing 4OOTGMTN arrays [136:35890]
interpolate: Interpolating index for 4ootgsel
interpolate: Slicing on indexes
interpolate: Interpolating index for 4ootgmtn
interpolate: Slicing on indexes
2025-05-24 08:11:05,979 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:11:05,980 get_states: Getting commanded states ['letg'] between 2025:130:12:10:17.916 - 2025:144:11:10:06.050
2025-05-24 08:11:05,980 get_cmds: Getting commands from 2025:130:12:10:17.916 to 2025:144:11:10:06.050 for scenario=None
2025-05-24 08:11:05,980 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:05,981 get_cmds: Getting recent commands from cache
2025-05-24 08:11:05,981 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:05,984 get_cmds: Getting commands from 2025:123:12:10:17.916 to 2025:130:12:10:17.916 for scenario=None
2025-05-24 08:11:05,984 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:05,985 get_cmds: Getting recent commands from cache
2025-05-24 08:11:05,985 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:05,995 get_cmds: Getting commands from 2025:100:12:10:17.916 to 2025:130:12:10:17.916 for scenario=None
2025-05-24 08:11:05,995 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:05,996 get_cmds: Getting recent commands from cache
2025-05-24 08:11:05,997 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:10:17.916
2025-05-24 08:11:07,576 load_idx_cmds: Loaded /export/tom/kadi-validate/data/kadi/cmds2.h5 with 1709895 commands
2025-05-24 08:11:07,585 get_matching_block_idx: Selecting commands from cmds_arch[0:]
2025-05-24 08:11:08,012 get_matching_block_idx: Matching blocks for (a) recent commands and (b) existing HDF5
2025-05-24 08:11:08,012 get_matching_block_idx: Match(a=0, b=0, size=3)
2025-05-24 08:11:08,012 get_matching_block_idx: Match(a=6, b=3, size=30)
2025-05-24 08:11:08,012 get_matching_block_idx: Match(a=37, b=33, size=10498)
2025-05-24 08:11:08,012 get_matching_block_idx: Match(a=10535, b=10531, size=0)
2025-05-24 08:11:08,012 get_matching_block_idx: Diffs between (a) recent commands and (b) existing HDF5
2025-05-24 08:11:08,012 get_matching_block_idx: ('equal', 0, 3, 0, 3)
2025-05-24 08:11:08,012 get_matching_block_idx: ('delete', 3, 6, 3, 3)
2025-05-24 08:11:08,012 get_matching_block_idx: ('equal', 6, 36, 3, 33)
2025-05-24 08:11:08,012 get_matching_block_idx: ('delete', 36, 37, 33, 33)
2025-05-24 08:11:08,012 get_matching_block_idx: ('equal', 37, 10535, 33, 10531)
2025-05-24 08:11:08,018 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:11:08,295 get_plot_figure: Compressing Telem data for state letg
2025-05-24 08:11:08,367 get_plot_figure: Creating Telem scatter plot for state letg
2025-05-24 08:11:08,368 get_plot_figure: Compressing State data for state letg
2025-05-24 08:11:08,438 get_plot_figure: Creating State scatter plot for state letg
2025-05-24 08:11:08,445 get_plot_html: Creating HTML for state letg
2025-05-24 08:11:08,448 run_validators: Validating hetg
2025-05-24 08:11:08,455 tlm: Fetching telemetry for ['4ootgsel', '4ootgmtn'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for 4ootgsel between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_data: Getting data for 4ootgmtn between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
interpolate: Interpolating index for 4ootgsel
interpolate: Slicing on indexes
interpolate: Interpolating index for 4ootgmtn
interpolate: Slicing on indexes
2025-05-24 08:11:08,469 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:11:08,470 get_states: Getting commanded states ['hetg'] between 2025:130:12:10:17.916 - 2025:144:11:10:06.050
2025-05-24 08:11:08,470 get_cmds: Getting commands from 2025:130:12:10:17.916 to 2025:144:11:10:06.050 for scenario=None
2025-05-24 08:11:08,470 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:08,471 get_cmds: Getting recent commands from cache
2025-05-24 08:11:08,471 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:08,473 get_cmds: Getting commands from 2025:123:12:10:17.916 to 2025:130:12:10:17.916 for scenario=None
2025-05-24 08:11:08,473 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:08,474 get_cmds: Getting recent commands from cache
2025-05-24 08:11:08,475 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:08,484 get_cmds: Getting commands from 2025:100:12:10:17.916 to 2025:130:12:10:17.916 for scenario=None
2025-05-24 08:11:08,484 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:08,485 get_cmds: Getting recent commands from cache
2025-05-24 08:11:08,485 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:10:17.916
2025-05-24 08:11:08,487 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:11:08,729 get_plot_figure: Compressing Telem data for state hetg
2025-05-24 08:11:08,792 get_plot_figure: Creating Telem scatter plot for state hetg
2025-05-24 08:11:08,794 get_plot_figure: Compressing State data for state hetg
2025-05-24 08:11:08,857 get_plot_figure: Creating State scatter plot for state hetg
2025-05-24 08:11:08,865 get_plot_html: Creating HTML for state hetg
2025-05-24 08:11:08,869 run_validators: Validating sun_pos_mon
2025-05-24 08:11:08,876 tlm: Fetching telemetry for ['aopssupm'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for aopssupm between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/AOPSSUPM.h5
_get_msid_data_from_cxc: Slicing AOPSSUPM arrays [541:143560]
interpolate: Interpolating index for aopssupm
interpolate: Slicing on indexes
filter_bad: Filtering bad values for aopssupm
2025-05-24 08:11:09,372 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:11:09,372 get_states: Getting commanded states ['sun_pos_mon'] between 2025:130:12:10:09.716 - 2025:144:11:10:30.138
2025-05-24 08:11:09,372 get_cmds: Getting commands from 2025:130:12:10:09.716 to 2025:144:11:10:30.138 for scenario=None
2025-05-24 08:11:09,372 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:09,373 get_cmds: Getting recent commands from cache
2025-05-24 08:11:09,373 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:09,375 get_cmds: Getting commands from 2025:123:12:10:09.716 to 2025:130:12:10:09.716 for scenario=None
2025-05-24 08:11:09,376 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:09,376 get_cmds: Getting recent commands from cache
2025-05-24 08:11:09,377 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:09,390 get_cmds: Getting commands from 2025:100:12:10:09.716 to 2025:130:12:10:09.716 for scenario=None
2025-05-24 08:11:09,391 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:09,391 get_cmds: Getting recent commands from cache
2025-05-24 08:11:09,392 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:10:09.716
2025-05-24 08:11:09,394 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:11:09,418 get_cmds: Getting commands from 2024:316:12:10:09.716 to 2025:130:12:10:09.716 for scenario=None
2025-05-24 08:11:09,419 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:09,419 get_cmds: Getting recent commands from cache
2025-05-24 08:11:09,420 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2024:316:12:10:09.716
2025-05-24 08:11:09,424 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:11:10,357 get_plot_figure: Compressing Telem data for state sun_pos_mon
2025-05-24 08:11:10,605 get_plot_figure: Creating Telem scatter plot for state sun_pos_mon
2025-05-24 08:11:10,607 get_plot_figure: Compressing State data for state sun_pos_mon
2025-05-24 08:11:10,857 get_plot_figure: Creating State scatter plot for state sun_pos_mon
2025-05-24 08:11:10,871 get_plot_html: Creating HTML for state sun_pos_mon
2025-05-24 08:11:10,875 run_validators: Validating dpa_power
2025-05-24 08:11:12,321 tlm: Fetching telemetry for ['dpa_power'] between 2025:130:12:10:02.904 and 2025:144:12:10:02.904
_get_data: Getting data for dpa_power between 2025:130:11:55:02.904 to 2025:144:12:25:02.904
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/dp_acispow128/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/dp_acispow128/DP_DPA_POWER.h5
_get_msid_data_from_cxc: Slicing DP_DPA_POWER arrays [305:35990]
interpolate: Interpolating index for dpa_power
interpolate: Slicing on indexes
2025-05-24 08:11:12,516 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:11:12,516 get_states: Getting commanded states ['ccd_count', 'clocking', 'feps', 'ccds', 'fep_count', 'si_mode'] between 2025:130:12:10:14.016 - 2025:144:11:10:09.125
2025-05-24 08:11:12,516 get_cmds: Getting commands from 2025:130:12:10:14.016 to 2025:144:11:10:09.125 for scenario=None
2025-05-24 08:11:12,516 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:12,517 get_cmds: Getting recent commands from cache
2025-05-24 08:11:12,517 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:12,520 get_cmds: Getting commands from 2025:123:12:10:14.016 to 2025:130:12:10:14.016 for scenario=None
2025-05-24 08:11:12,520 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:11:12,521 get_cmds: Getting recent commands from cache
2025-05-24 08:11:12,521 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:11:13,128 get_plot_figure: Compressing Telem data for state dpa_power
2025-05-24 08:11:13,594 get_plot_figure: Creating Telem scatter plot for state dpa_power
2025-05-24 08:11:13,824 get_plot_figure: Compressing State data for state dpa_power
2025-05-24 08:11:13,890 get_plot_figure: Creating State scatter plot for state dpa_power
2025-05-24 08:11:13,939 get_plot_html: Creating HTML for state dpa_power
WARNING: AstropyDeprecationWarning: The pformat_all function is deprecated and may be removed in a future version.
Use Table.pformat instead. [kadi.scripts.validate_states]
2025-05-24 08:11:14,035 main: kadi validate_states processing found state violation(s):

name start stop
--------- --------------------- ---------------------
dpa_power 2025:133:22:37:17.616 2025:133:23:00:48.016

See:https://cxc.harvard.edu/mta/ASPECT/validate_states/

Traceback (most recent call last):
File "/export/tom/kadi-validate/bin/kadi_validate_states", line 8, in
sys.exit(main())
^^^^^^
File "/export/tom/kadi-validate/lib/python3.12/site-packages/kadi/scripts/validate_states.py", line 222, in main
send_alert_email(violations_text, opt)
File "/export/tom/kadi-validate/lib/python3.12/site-packages/kadi/scripts/validate_states.py", line 189, in send_alert_email
from acdc.common import send_mail
ModuleNotFoundError: No module named 'acdc'
2025-05-24 08:26:01,678 run_validators: Validating pitch
2025-05-24 08:26:01,679 get_command_sheet_exclude_intervals: Getting exclude times from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv&id=19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ&gid=1681877928
_get_data: Getting data for conlofp between 2025:130:12:10:31.678 to 2025:144:12:41:31.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/CONLOFP.h5
_get_msid_data_from_cxc: Slicing CONLOFP arrays [5230:1148480]
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:26:02,859 tlm: Fetching telemetry for ['pitch_comp'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for pitch_comp between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_comp_data: Getting computed values for pitch_comp
_get_data: Getting data for conlofp between 2025:130:11:55:31.678 to 2025:144:12:56:31.678
_get_msid_data_from_cxc: Using times_cache for ccdm8eng 863265400.8619998 to 864478660.8619998
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/CONLOFP.h5
_get_msid_data_from_cxc: Slicing CONLOFP arrays [4352:1148480]
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
_get_data: Getting data for aopcadmd between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOPCADMD.h5
_get_msid_data_from_cxc: Slicing AOPCADMD arrays [5259:1148480]
_get_data: Getting data for orbitephem0_x between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_X.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_X arrays [5760:9802]
_get_data: Getting data for orbitephem0_y between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Using times_cache for orbitephem0 863265610.8619998 to 864478450.862
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_Y.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_Y arrays [5760:9802]
_get_data: Getting data for orbitephem0_z between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Using times_cache for orbitephem0 863265610.8619998 to 864478450.862
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_Z.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_Z arrays [5760:9802]
_get_data: Getting data for solarephem0_x between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_X.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_X arrays [5760:9802]
_get_data: Getting data for solarephem0_y between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Using times_cache for solarephem0 863265610.8619998 to 864478450.862
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_Y.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_Y arrays [5760:9802]
_get_data: Getting data for solarephem0_z between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Using times_cache for solarephem0 863265610.8619998 to 864478450.862
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_Z.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_Z arrays [5760:9802]
_get_data: Getting data for aoattqt1 between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT1.h5
_get_msid_data_from_cxc: Slicing AOATTQT1 arrays [4557:1148480]
_get_data: Getting data for aoattqt2 between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863265610.8619998 to 864478450.862
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT2.h5
_get_msid_data_from_cxc: Slicing AOATTQT2 arrays [4557:1148480]
_get_data: Getting data for aoattqt3 between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863265610.8619998 to 864478450.862
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT3.h5
_get_msid_data_from_cxc: Slicing AOATTQT3 arrays [4557:1148480]
_get_data: Getting data for aoattqt4 between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863265610.8619998 to 864478450.862
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT4.h5
_get_msid_data_from_cxc: Slicing AOATTQT4 arrays [4557:1148480]
fetch: Setting bads because of gaps in aoattqt1 between 2025:144:11:10:35.166 to 2025:144:12:53:01.066
fetch: Setting bads because of gaps in aoattqt2 between 2025:144:11:10:35.166 to 2025:144:12:53:01.066
fetch: Setting bads because of gaps in aoattqt3 between 2025:144:11:10:35.166 to 2025:144:12:53:01.066
fetch: Setting bads because of gaps in aoattqt4 between 2025:144:11:10:35.166 to 2025:144:12:53:01.066
interpolate: Interpolating index for orbitephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt1
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt2
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt3
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt4
interpolate: Slicing on indexes
interpolate: Interpolating index for pitch_comp
interpolate: Slicing on indexes
2025-05-24 08:26:07,177 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:07,177 get_states: Getting commanded states ['pitch', 'pcad_mode'] between 2025:130:12:26:02.141 - 2025:144:11:10:30.041
2025-05-24 08:26:07,178 get_cmds: Getting commands from 2025:130:12:26:02.141 to 2025:144:11:10:30.041 for scenario=None
2025-05-24 08:26:07,178 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:07,179 get_cmds: Recent commands not in cache: updating local cmd_events, loads and getting recent commands
2025-05-24 08:26:07,179 get_cmd_events_from_sheet: Getting cmd_events from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv
2025-05-24 08:26:07,472 get_cmd_events_from_sheet: Writing 152 cmd_events to /home/aca/.kadi/cmd_events.csv
2025-05-24 08:26:07,480 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/APR with cache=False
2025-05-24 08:26:07,771 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/APR2825A.pkl.gz
2025-05-24 08:26:07,776 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAY with cache=False
2025-05-24 08:26:07,831 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY0525A.pkl.gz
2025-05-24 08:26:07,837 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY1225A.pkl.gz
2025-05-24 08:26:07,843 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY1925A.pkl.gz
2025-05-24 08:26:07,850 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY2625A.pkl.gz
2025-05-24 08:26:07,856 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY2625B.pkl.gz
2025-05-24 08:26:07,861 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/JUN with cache=False
2025-05-24 08:26:07,891 update_cmd_events_and_loads_and_get_cmds_recent: Including loads APR2825A, MAY0525A, MAY1225A, MAY1925A, MAY2625A, MAY2625B
2025-05-24 08:26:07,894 update_cmd_events_and_loads_and_get_cmds_recent: Load APR2825A has 1894 commands with RLTT=2025:118:00:24:12.365
2025-05-24 08:26:07,897 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY0525A has 1709 commands with RLTT=2025:125:01:52:24.962
2025-05-24 08:26:07,901 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY1225A has 2038 commands with RLTT=2025:131:21:15:39.221
2025-05-24 08:26:07,906 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY1925A has 2695 commands with RLTT=2025:139:00:00:00.838
2025-05-24 08:26:07,910 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY2625A has 49 commands with RLTT=None
2025-05-24 08:26:07,913 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY2625B has 1979 commands with RLTT=2025:146:01:43:21.753
2025-05-24 08:26:07,943 update_cmd_events_and_loads_and_get_cmds_recent: Including cmd_events:
Load not run at 2025:146:01:40:21.753
2025-05-24 08:26:07,945 update_cmd_events_and_loads_and_get_cmds_recent: Processing APR2825A with 1894 commands
2025-05-24 08:26:07,945 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1894 commands from APR2825A
2025-05-24 08:26:07,945 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY0525A with 1709 commands
2025-05-24 08:26:07,946 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1709 commands from MAY0525A
2025-05-24 08:26:07,946 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY1225A with 2038 commands
2025-05-24 08:26:07,946 update_cmd_events_and_loads_and_get_cmds_recent: Adding 2038 commands from MAY1225A
2025-05-24 08:26:07,946 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY1925A with 2695 commands
2025-05-24 08:26:07,946 update_cmd_events_and_loads_and_get_cmds_recent: Adding 2695 commands from MAY1925A
2025-05-24 08:26:07,946 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY2625B with 1979 commands
2025-05-24 08:26:07,947 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1979 commands from MAY2625B
2025-05-24 08:26:07,947 update_cmd_events_and_loads_and_get_cmds_recent: Processing CMD_EVT Load_not_run at 2025:146:01:40:21.753 with 1 commands
2025-05-24 08:26:07,947 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1 commands from CMD_EVT Load_not_run at 2025:146:01:40:21.753
2025-05-24 08:26:07,947 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY2625A with 49 commands
2025-05-24 08:26:07,947 update_cmd_events_and_loads_and_get_cmds_recent: Adding 49 commands from MAY2625A
2025-05-24 08:26:08,328 load_pars_dict: Loaded /export/tom/kadi-validate/data/kadi/cmds2.pkl with 180137 pars
2025-05-24 08:26:08,349 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:08,351 get_cmds: Getting commands from 2025:123:12:26:02.141 to 2025:130:12:26:02.141 for scenario=None
2025-05-24 08:26:08,352 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:08,352 get_cmds: Getting recent commands from cache
2025-05-24 08:26:08,353 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:13,421 get_plot_figure: Compressing Telem data for state pitch
2025-05-24 08:26:15,644 get_plot_figure: Creating Telem scatter plot for state pitch
2025-05-24 08:26:15,682 get_plot_figure: Compressing State data for state pitch
2025-05-24 08:26:17,664 get_plot_figure: Creating State scatter plot for state pitch
2025-05-24 08:26:17,722 get_plot_html: Creating HTML for state pitch
2025-05-24 08:26:17,735 run_validators: Validating off_nom_roll
_get_data: Getting data for conlofp between 2025:130:12:10:31.678 to 2025:144:12:41:31.678
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:26:17,917 tlm: Fetching telemetry for ['roll_comp'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for roll_comp between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_comp_data: Getting computed values for roll_comp
_get_data: Getting data for conlofp between 2025:130:11:55:31.678 to 2025:144:12:56:31.678
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
_get_data: Getting data for aopcadmd between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_data: Getting data for orbitephem0_x between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for orbitephem0_y between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for orbitephem0_z between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for solarephem0_x between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for solarephem0_y between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for solarephem0_z between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for aoattqt1 between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for aoattqt2 between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for aoattqt3 between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
_get_data: Getting data for aoattqt4 between 2025:130:11:59:01.678 to 2025:144:12:53:01.678
fetch: Setting bads because of gaps in aoattqt1 between 2025:144:11:10:35.166 to 2025:144:12:53:01.066
fetch: Setting bads because of gaps in aoattqt2 between 2025:144:11:10:35.166 to 2025:144:12:53:01.066
fetch: Setting bads because of gaps in aoattqt3 between 2025:144:11:10:35.166 to 2025:144:12:53:01.066
fetch: Setting bads because of gaps in aoattqt4 between 2025:144:11:10:35.166 to 2025:144:12:53:01.066
interpolate: Interpolating index for orbitephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt1
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt2
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt3
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt4
interpolate: Slicing on indexes
interpolate: Interpolating index for roll_comp
interpolate: Slicing on indexes
2025-05-24 08:26:18,824 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:18,824 get_states: Getting commanded states ['off_nom_roll', 'pitch', 'pcad_mode'] between 2025:130:12:26:02.141 - 2025:144:11:10:30.041
2025-05-24 08:26:18,825 get_cmds: Getting commands from 2025:130:12:26:02.141 to 2025:144:11:10:30.041 for scenario=None
2025-05-24 08:26:18,825 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:18,826 get_cmds: Getting recent commands from cache
2025-05-24 08:26:18,826 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:18,829 get_cmds: Getting commands from 2025:123:12:26:02.141 to 2025:130:12:26:02.141 for scenario=None
2025-05-24 08:26:18,829 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:18,830 get_cmds: Getting recent commands from cache
2025-05-24 08:26:18,830 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:23,241 get_plot_figure: Compressing Telem data for state off_nom_roll
2025-05-24 08:26:25,305 get_plot_figure: Creating Telem scatter plot for state off_nom_roll
2025-05-24 08:26:25,329 get_plot_figure: Compressing State data for state off_nom_roll
2025-05-24 08:26:27,301 get_plot_figure: Creating State scatter plot for state off_nom_roll
2025-05-24 08:26:27,356 get_plot_html: Creating HTML for state off_nom_roll
2025-05-24 08:26:27,366 run_validators: Validating dither
_get_data: Getting data for conlofp between 2025:130:12:10:31.678 to 2025:144:12:41:31.678
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:26:27,541 get_cmds: Getting commands from 2025:130:12:06:01.678 to 2025:144:12:26:01.678 for scenario=None
2025-05-24 08:26:27,541 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:27,542 get_cmds: Getting recent commands from cache
2025-05-24 08:26:27,542 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:27,602 add_exclude_interval: dither: excluding interval 2025:132:02:27:38.000 - 2025:132:02:40:38.000: SRDC
2025-05-24 08:26:27,603 add_exclude_interval: dither: excluding interval 2025:135:02:24:59.000 - 2025:135:02:37:59.000: SRDC
2025-05-24 08:26:27,605 add_exclude_interval: dither: excluding interval 2025:137:11:40:31.000 - 2025:137:11:53:31.000: SRDC
2025-05-24 08:26:27,606 add_exclude_interval: dither: excluding interval 2025:137:18:20:03.000 - 2025:137:18:33:03.000: SRDC
2025-05-24 08:26:27,606 tlm: Fetching telemetry for ['aodithen'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for aodithen between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/AODITHEN.h5
_get_msid_data_from_cxc: Slicing AODITHEN arrays [658:143560]
interpolate: Interpolating index for aodithen
interpolate: Slicing on indexes
2025-05-24 08:26:27,737 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:27,737 get_states: Getting commanded states ['dither'] between 2025:130:12:26:09.116 - 2025:144:11:10:24.756
2025-05-24 08:26:27,737 get_cmds: Getting commands from 2025:130:12:26:09.116 to 2025:144:11:10:24.756 for scenario=None
2025-05-24 08:26:27,737 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:27,738 get_cmds: Getting recent commands from cache
2025-05-24 08:26:27,739 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:27,741 get_cmds: Getting commands from 2025:123:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:27,741 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:27,742 get_cmds: Getting recent commands from cache
2025-05-24 08:26:27,742 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:28,463 get_plot_figure: Compressing Telem data for state dither
2025-05-24 08:26:28,704 get_plot_figure: Creating Telem scatter plot for state dither
2025-05-24 08:26:28,705 get_plot_figure: Compressing State data for state dither
2025-05-24 08:26:28,948 get_plot_figure: Creating State scatter plot for state dither
2025-05-24 08:26:28,979 get_plot_html: Creating HTML for state dither
2025-05-24 08:26:28,982 run_validators: Validating pcad_mode
2025-05-24 08:26:28,993 get_cmds: Getting commands from 2025:130:12:06:01.678 to 2025:144:12:26:01.678 for scenario=None
2025-05-24 08:26:28,993 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:28,994 get_cmds: Getting recent commands from cache
2025-05-24 08:26:28,994 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:29,052 add_exclude_interval: pcad_mode: excluding interval 2025:132:02:27:38.000 - 2025:132:02:40:38.000: SRDC
2025-05-24 08:26:29,053 add_exclude_interval: pcad_mode: excluding interval 2025:135:02:24:59.000 - 2025:135:02:37:59.000: SRDC
2025-05-24 08:26:29,055 add_exclude_interval: pcad_mode: excluding interval 2025:137:11:40:31.000 - 2025:137:11:53:31.000: SRDC
2025-05-24 08:26:29,056 add_exclude_interval: pcad_mode: excluding interval 2025:137:18:20:03.000 - 2025:137:18:33:03.000: SRDC
2025-05-24 08:26:29,056 tlm: Fetching telemetry for ['aopcadmd'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for aopcadmd between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOPCADMD.h5
_get_msid_data_from_cxc: Slicing AOPCADMD arrays [5259:1148480]
interpolate: Interpolating index for aopcadmd
interpolate: Slicing on indexes
2025-05-24 08:26:29,270 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:29,271 get_states: Getting commanded states ['pcad_mode'] between 2025:130:12:26:01.941 - 2025:144:11:10:30.906
2025-05-24 08:26:29,271 get_cmds: Getting commands from 2025:130:12:26:01.941 to 2025:144:11:10:30.906 for scenario=None
2025-05-24 08:26:29,271 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:29,272 get_cmds: Getting recent commands from cache
2025-05-24 08:26:29,272 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:29,274 get_cmds: Getting commands from 2025:123:12:26:01.941 to 2025:130:12:26:01.941 for scenario=None
2025-05-24 08:26:29,275 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:29,275 get_cmds: Getting recent commands from cache
2025-05-24 08:26:29,276 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:37,421 get_plot_figure: Compressing Telem data for state pcad_mode
2025-05-24 08:26:39,359 get_plot_figure: Creating Telem scatter plot for state pcad_mode
2025-05-24 08:26:39,363 get_plot_figure: Compressing State data for state pcad_mode
2025-05-24 08:26:41,296 get_plot_figure: Creating State scatter plot for state pcad_mode
2025-05-24 08:26:41,348 get_plot_html: Creating HTML for state pcad_mode
2025-05-24 08:26:41,353 run_validators: Validating simpos
2025-05-24 08:26:41,359 tlm: Fetching telemetry for ['3tscpos'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for 3tscpos between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/sim_mrg/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/sim_mrg/3TSCPOS.h5
_get_msid_data_from_cxc: Slicing 3TSCPOS arrays [164:35826]
interpolate: Interpolating index for 3tscpos
interpolate: Slicing on indexes
2025-05-24 08:26:41,493 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:41,493 get_states: Getting commanded states ['simpos'] between 2025:130:12:26:09.116 - 2025:144:11:10:17.069
2025-05-24 08:26:41,493 get_cmds: Getting commands from 2025:130:12:26:09.116 to 2025:144:11:10:17.069 for scenario=None
2025-05-24 08:26:41,493 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:41,494 get_cmds: Getting recent commands from cache
2025-05-24 08:26:41,495 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:41,497 get_cmds: Getting commands from 2025:123:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:41,497 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:41,498 get_cmds: Getting recent commands from cache
2025-05-24 08:26:41,499 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:41,639 get_plot_figure: Compressing Telem data for state simpos
2025-05-24 08:26:41,703 get_plot_figure: Creating Telem scatter plot for state simpos
2025-05-24 08:26:41,706 get_plot_figure: Compressing State data for state simpos
2025-05-24 08:26:41,768 get_plot_figure: Creating State scatter plot for state simpos
2025-05-24 08:26:41,777 get_plot_html: Creating HTML for state simpos
2025-05-24 08:26:41,780 run_validators: Validating obsid
2025-05-24 08:26:41,787 tlm: Fetching telemetry for ['cobsrqid'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for cobsrqid between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm10eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm10eng/COBSRQID.h5
_get_msid_data_from_cxc: Slicing COBSRQID arrays [2630:574240]
interpolate: Interpolating index for cobsrqid
interpolate: Slicing on indexes
2025-05-24 08:26:42,069 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:42,069 get_states: Getting commanded states ['obsid'] between 2025:130:12:26:02.966 - 2025:144:11:10:30.650
2025-05-24 08:26:42,069 get_cmds: Getting commands from 2025:130:12:26:02.966 to 2025:144:11:10:30.650 for scenario=None
2025-05-24 08:26:42,069 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:42,070 get_cmds: Getting recent commands from cache
2025-05-24 08:26:42,070 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:42,073 get_cmds: Getting commands from 2025:123:12:26:02.966 to 2025:130:12:26:02.966 for scenario=None
2025-05-24 08:26:42,073 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:42,074 get_cmds: Getting recent commands from cache
2025-05-24 08:26:42,074 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:44,255 get_plot_figure: Compressing Telem data for state obsid
2025-05-24 08:26:45,220 get_plot_figure: Creating Telem scatter plot for state obsid
2025-05-24 08:26:45,223 get_plot_figure: Compressing State data for state obsid
2025-05-24 08:26:46,190 get_plot_figure: Creating State scatter plot for state obsid
2025-05-24 08:26:46,206 get_plot_html: Creating HTML for state obsid
2025-05-24 08:26:46,210 run_validators: Validating letg
2025-05-24 08:26:46,217 tlm: Fetching telemetry for ['4ootgsel', '4ootgmtn'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for 4ootgsel between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/4OOTGSEL.h5
_get_msid_data_from_cxc: Slicing 4OOTGSEL arrays [165:35890]
_get_data: Getting data for 4ootgmtn between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Using times_cache for tel2eng 863266330.8619998 to 864477730.862
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/4OOTGMTN.h5
_get_msid_data_from_cxc: Slicing 4OOTGMTN arrays [165:35890]
interpolate: Interpolating index for 4ootgsel
interpolate: Slicing on indexes
interpolate: Interpolating index for 4ootgmtn
interpolate: Slicing on indexes
2025-05-24 08:26:46,467 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:46,467 get_states: Getting commanded states ['letg'] between 2025:130:12:26:09.116 - 2025:144:11:10:06.050
2025-05-24 08:26:46,467 get_cmds: Getting commands from 2025:130:12:26:09.116 to 2025:144:11:10:06.050 for scenario=None
2025-05-24 08:26:46,467 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:46,468 get_cmds: Getting recent commands from cache
2025-05-24 08:26:46,469 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:46,471 get_cmds: Getting commands from 2025:123:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:46,471 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:46,472 get_cmds: Getting recent commands from cache
2025-05-24 08:26:46,472 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:46,482 get_cmds: Getting commands from 2025:100:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:46,482 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:46,483 get_cmds: Getting recent commands from cache
2025-05-24 08:26:46,483 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:26:09.116
2025-05-24 08:26:47,681 load_idx_cmds: Loaded /export/tom/kadi-validate/data/kadi/cmds2.h5 with 1709895 commands
2025-05-24 08:26:47,689 get_matching_block_idx: Selecting commands from cmds_arch[0:]
2025-05-24 08:26:48,091 get_matching_block_idx: Matching blocks for (a) recent commands and (b) existing HDF5
2025-05-24 08:26:48,091 get_matching_block_idx: Match(a=0, b=0, size=3)
2025-05-24 08:26:48,091 get_matching_block_idx: Match(a=6, b=3, size=30)
2025-05-24 08:26:48,091 get_matching_block_idx: Match(a=37, b=33, size=10498)
2025-05-24 08:26:48,091 get_matching_block_idx: Match(a=10535, b=10531, size=0)
2025-05-24 08:26:48,091 get_matching_block_idx: Diffs between (a) recent commands and (b) existing HDF5
2025-05-24 08:26:48,091 get_matching_block_idx: ('equal', 0, 3, 0, 3)
2025-05-24 08:26:48,091 get_matching_block_idx: ('delete', 3, 6, 3, 3)
2025-05-24 08:26:48,091 get_matching_block_idx: ('equal', 6, 36, 3, 33)
2025-05-24 08:26:48,091 get_matching_block_idx: ('delete', 36, 37, 33, 33)
2025-05-24 08:26:48,091 get_matching_block_idx: ('equal', 37, 10535, 33, 10531)
2025-05-24 08:26:48,095 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:26:48,335 get_plot_figure: Compressing Telem data for state letg
2025-05-24 08:26:48,396 get_plot_figure: Creating Telem scatter plot for state letg
2025-05-24 08:26:48,398 get_plot_figure: Compressing State data for state letg
2025-05-24 08:26:48,458 get_plot_figure: Creating State scatter plot for state letg
2025-05-24 08:26:48,465 get_plot_html: Creating HTML for state letg
2025-05-24 08:26:48,468 run_validators: Validating hetg
2025-05-24 08:26:48,474 tlm: Fetching telemetry for ['4ootgsel', '4ootgmtn'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for 4ootgsel between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_data: Getting data for 4ootgmtn between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
interpolate: Interpolating index for 4ootgsel
interpolate: Slicing on indexes
interpolate: Interpolating index for 4ootgmtn
interpolate: Slicing on indexes
2025-05-24 08:26:48,488 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:48,489 get_states: Getting commanded states ['hetg'] between 2025:130:12:26:09.116 - 2025:144:11:10:06.050
2025-05-24 08:26:48,489 get_cmds: Getting commands from 2025:130:12:26:09.116 to 2025:144:11:10:06.050 for scenario=None
2025-05-24 08:26:48,489 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:48,489 get_cmds: Getting recent commands from cache
2025-05-24 08:26:48,490 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:48,492 get_cmds: Getting commands from 2025:123:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:48,492 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:48,493 get_cmds: Getting recent commands from cache
2025-05-24 08:26:48,493 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:48,502 get_cmds: Getting commands from 2025:100:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:48,502 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:48,503 get_cmds: Getting recent commands from cache
2025-05-24 08:26:48,503 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:26:09.116
2025-05-24 08:26:48,505 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:26:48,748 get_plot_figure: Compressing Telem data for state hetg
2025-05-24 08:26:48,809 get_plot_figure: Creating Telem scatter plot for state hetg
2025-05-24 08:26:48,811 get_plot_figure: Compressing State data for state hetg
2025-05-24 08:26:48,873 get_plot_figure: Creating State scatter plot for state hetg
2025-05-24 08:26:48,881 get_plot_html: Creating HTML for state hetg
2025-05-24 08:26:48,884 run_validators: Validating sun_pos_mon
2025-05-24 08:26:48,891 tlm: Fetching telemetry for ['aopssupm'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for aopssupm between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/AOPSSUPM.h5
_get_msid_data_from_cxc: Slicing AOPSSUPM arrays [658:143560]
interpolate: Interpolating index for aopssupm
interpolate: Slicing on indexes
filter_bad: Filtering bad values for aopssupm
2025-05-24 08:26:49,069 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:49,069 get_states: Getting commanded states ['sun_pos_mon'] between 2025:130:12:26:09.116 - 2025:144:11:10:30.138
2025-05-24 08:26:49,070 get_cmds: Getting commands from 2025:130:12:26:09.116 to 2025:144:11:10:30.138 for scenario=None
2025-05-24 08:26:49,070 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:49,070 get_cmds: Getting recent commands from cache
2025-05-24 08:26:49,071 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:49,073 get_cmds: Getting commands from 2025:123:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:49,073 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:49,074 get_cmds: Getting recent commands from cache
2025-05-24 08:26:49,074 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:49,087 get_cmds: Getting commands from 2025:100:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:49,088 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:49,088 get_cmds: Getting recent commands from cache
2025-05-24 08:26:49,089 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:26:09.116
2025-05-24 08:26:49,090 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:26:49,115 get_cmds: Getting commands from 2024:316:12:26:09.116 to 2025:130:12:26:09.116 for scenario=None
2025-05-24 08:26:49,115 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:49,116 get_cmds: Getting recent commands from cache
2025-05-24 08:26:49,116 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2024:316:12:26:09.116
2025-05-24 08:26:49,120 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:26:50,053 get_plot_figure: Compressing Telem data for state sun_pos_mon
2025-05-24 08:26:50,292 get_plot_figure: Creating Telem scatter plot for state sun_pos_mon
2025-05-24 08:26:50,294 get_plot_figure: Compressing State data for state sun_pos_mon
2025-05-24 08:26:50,535 get_plot_figure: Creating State scatter plot for state sun_pos_mon
2025-05-24 08:26:50,548 get_plot_html: Creating HTML for state sun_pos_mon
2025-05-24 08:26:50,552 run_validators: Validating dpa_power
2025-05-24 08:26:51,170 tlm: Fetching telemetry for ['dpa_power'] between 2025:130:12:26:01.678 and 2025:144:12:26:01.678
_get_data: Getting data for dpa_power between 2025:130:12:11:01.678 to 2025:144:12:41:01.678
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/dp_acispow128/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/dp_acispow128/DP_DPA_POWER.h5
_get_msid_data_from_cxc: Slicing DP_DPA_POWER arrays [29:35685]
interpolate: Interpolating index for dpa_power
interpolate: Slicing on indexes
2025-05-24 08:26:51,423 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:26:51,423 get_states: Getting commanded states ['ccd_count', 'clocking', 'feps', 'ccds', 'fep_count', 'si_mode'] between 2025:130:12:26:05.216 - 2025:144:11:10:09.125
2025-05-24 08:26:51,424 get_cmds: Getting commands from 2025:130:12:26:05.216 to 2025:144:11:10:09.125 for scenario=None
2025-05-24 08:26:51,424 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:51,424 get_cmds: Getting recent commands from cache
2025-05-24 08:26:51,425 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:51,427 get_cmds: Getting commands from 2025:123:12:26:05.216 to 2025:130:12:26:05.216 for scenario=None
2025-05-24 08:26:51,427 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:26:51,428 get_cmds: Getting recent commands from cache
2025-05-24 08:26:51,429 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:26:52,028 get_plot_figure: Compressing Telem data for state dpa_power
2025-05-24 08:26:52,459 get_plot_figure: Creating Telem scatter plot for state dpa_power
2025-05-24 08:26:52,689 get_plot_figure: Compressing State data for state dpa_power
2025-05-24 08:26:52,754 get_plot_figure: Creating State scatter plot for state dpa_power
2025-05-24 08:26:52,801 get_plot_html: Creating HTML for state dpa_power
WARNING: AstropyDeprecationWarning: The pformat_all function is deprecated and may be removed in a future version.
Use Table.pformat instead. [kadi.scripts.validate_states]
2025-05-24 08:26:52,897 main: kadi validate_states processing found state violation(s):

name start stop
--------- --------------------- ---------------------
dpa_power 2025:133:22:37:17.616 2025:133:23:00:48.016

See:https://cxc.harvard.edu/mta/ASPECT/validate_states/

2025-05-24 08:26:53,019 send_mail: Sent mail to ['aca@cfa.harvard.edu', 'acisdude@cfa.harvard.edu']
2025-05-24 08:36:01,793 run_validators: Validating pitch
2025-05-24 08:36:01,793 get_command_sheet_exclude_intervals: Getting exclude times from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv&id=19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ&gid=1681877928
_get_data: Getting data for conlofp between 2025:130:12:20:31.793 to 2025:144:12:51:31.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/CONLOFP.h5
_get_msid_data_from_cxc: Slicing CONLOFP arrays [5815:1148480]
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:36:03,046 tlm: Fetching telemetry for ['pitch_comp'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for pitch_comp between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_comp_data: Getting computed values for pitch_comp
_get_data: Getting data for conlofp between 2025:130:12:05:31.793 to 2025:144:13:06:31.793
_get_msid_data_from_cxc: Using times_cache for ccdm8eng 863266000.9769996 to 864479260.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm8eng/CONLOFP.h5
_get_msid_data_from_cxc: Slicing CONLOFP arrays [4937:1148480]
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
_get_data: Getting data for aopcadmd between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOPCADMD.h5
_get_msid_data_from_cxc: Slicing AOPCADMD arrays [5845:1148480]
_get_data: Getting data for orbitephem0_x between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_X.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_X arrays [5762:9804]
_get_data: Getting data for orbitephem0_y between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Using times_cache for orbitephem0 863266210.9769994 to 864479050.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_Y.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_Y arrays [5762:9804]
_get_data: Getting data for orbitephem0_z between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Using times_cache for orbitephem0 863266210.9769994 to 864479050.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/orbitephem0/ORBITEPHEM0_Z.h5
_get_msid_data_from_cxc: Slicing ORBITEPHEM0_Z arrays [5762:9804]
_get_data: Getting data for solarephem0_x between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_X.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_X arrays [5762:9804]
_get_data: Getting data for solarephem0_y between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Using times_cache for solarephem0 863266210.9769994 to 864479050.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_Y.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_Y arrays [5762:9804]
_get_data: Getting data for solarephem0_z between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Using times_cache for solarephem0 863266210.9769994 to 864479050.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/solarephem0/SOLAREPHEM0_Z.h5
_get_msid_data_from_cxc: Slicing SOLAREPHEM0_Z arrays [5762:9804]
_get_data: Getting data for aoattqt1 between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT1.h5
_get_msid_data_from_cxc: Slicing AOATTQT1 arrays [5142:1148480]
_get_data: Getting data for aoattqt2 between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863266210.9769994 to 864479050.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT2.h5
_get_msid_data_from_cxc: Slicing AOATTQT2 arrays [5142:1148480]
_get_data: Getting data for aoattqt3 between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863266210.9769994 to 864479050.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT3.h5
_get_msid_data_from_cxc: Slicing AOATTQT3 arrays [5142:1148480]
_get_data: Getting data for aoattqt4 between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_msid_data_from_cxc: Using times_cache for pcad3eng 863266210.9769994 to 864479050.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOATTQT4.h5
_get_msid_data_from_cxc: Slicing AOATTQT4 arrays [5142:1148480]
fetch: Setting bads because of gaps in aoattqt1 between 2025:144:11:10:35.166 to 2025:144:13:03:01.716
fetch: Setting bads because of gaps in aoattqt2 between 2025:144:11:10:35.166 to 2025:144:13:03:01.716
fetch: Setting bads because of gaps in aoattqt3 between 2025:144:11:10:35.166 to 2025:144:13:03:01.716
fetch: Setting bads because of gaps in aoattqt4 between 2025:144:11:10:35.166 to 2025:144:13:03:01.716
interpolate: Interpolating index for orbitephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt1
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt2
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt3
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt4
interpolate: Slicing on indexes
interpolate: Interpolating index for pitch_comp
interpolate: Slicing on indexes
2025-05-24 08:36:07,151 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:07,152 get_states: Getting commanded states ['pitch', 'pcad_mode'] between 2025:130:12:36:02.791 - 2025:144:11:10:30.041
2025-05-24 08:36:07,152 get_cmds: Getting commands from 2025:130:12:36:02.791 to 2025:144:11:10:30.041 for scenario=None
2025-05-24 08:36:07,152 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:07,153 get_cmds: Recent commands not in cache: updating local cmd_events, loads and getting recent commands
2025-05-24 08:36:07,153 get_cmd_events_from_sheet: Getting cmd_events from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv
2025-05-24 08:36:07,437 get_cmd_events_from_sheet: Writing 152 cmd_events to /home/aca/.kadi/cmd_events.csv
2025-05-24 08:36:07,447 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/APR with cache=False
2025-05-24 08:36:07,577 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/APR2825A.pkl.gz
2025-05-24 08:36:07,582 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAY with cache=False
2025-05-24 08:36:07,638 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY0525A.pkl.gz
2025-05-24 08:36:07,645 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY1225A.pkl.gz
2025-05-24 08:36:07,652 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY1925A.pkl.gz
2025-05-24 08:36:07,660 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY2625A.pkl.gz
2025-05-24 08:36:07,667 get_load_cmds_from_occweb_or_local: Already have /home/aca/.kadi/loads/MAY2625B.pkl.gz
2025-05-24 08:36:07,671 get_occweb_page: Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/JUN with cache=False
2025-05-24 08:36:07,721 update_cmd_events_and_loads_and_get_cmds_recent: Including loads APR2825A, MAY0525A, MAY1225A, MAY1925A, MAY2625A, MAY2625B
2025-05-24 08:36:07,724 update_cmd_events_and_loads_and_get_cmds_recent: Load APR2825A has 1894 commands with RLTT=2025:118:00:24:12.365
2025-05-24 08:36:07,727 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY0525A has 1709 commands with RLTT=2025:125:01:52:24.962
2025-05-24 08:36:07,731 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY1225A has 2038 commands with RLTT=2025:131:21:15:39.221
2025-05-24 08:36:07,736 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY1925A has 2695 commands with RLTT=2025:139:00:00:00.838
2025-05-24 08:36:07,741 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY2625A has 49 commands with RLTT=None
2025-05-24 08:36:07,745 update_cmd_events_and_loads_and_get_cmds_recent: Load MAY2625B has 1979 commands with RLTT=2025:146:01:43:21.753
2025-05-24 08:36:07,776 update_cmd_events_and_loads_and_get_cmds_recent: Including cmd_events:
Load not run at 2025:146:01:40:21.753
2025-05-24 08:36:07,778 update_cmd_events_and_loads_and_get_cmds_recent: Processing APR2825A with 1894 commands
2025-05-24 08:36:07,778 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1894 commands from APR2825A
2025-05-24 08:36:07,778 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY0525A with 1709 commands
2025-05-24 08:36:07,778 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1709 commands from MAY0525A
2025-05-24 08:36:07,778 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY1225A with 2038 commands
2025-05-24 08:36:07,779 update_cmd_events_and_loads_and_get_cmds_recent: Adding 2038 commands from MAY1225A
2025-05-24 08:36:07,779 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY1925A with 2695 commands
2025-05-24 08:36:07,779 update_cmd_events_and_loads_and_get_cmds_recent: Adding 2695 commands from MAY1925A
2025-05-24 08:36:07,779 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY2625B with 1979 commands
2025-05-24 08:36:07,779 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1979 commands from MAY2625B
2025-05-24 08:36:07,779 update_cmd_events_and_loads_and_get_cmds_recent: Processing CMD_EVT Load_not_run at 2025:146:01:40:21.753 with 1 commands
2025-05-24 08:36:07,780 update_cmd_events_and_loads_and_get_cmds_recent: Adding 1 commands from CMD_EVT Load_not_run at 2025:146:01:40:21.753
2025-05-24 08:36:07,780 update_cmd_events_and_loads_and_get_cmds_recent: Processing MAY2625A with 49 commands
2025-05-24 08:36:07,780 update_cmd_events_and_loads_and_get_cmds_recent: Adding 49 commands from MAY2625A
2025-05-24 08:36:08,164 load_pars_dict: Loaded /export/tom/kadi-validate/data/kadi/cmds2.pkl with 180137 pars
2025-05-24 08:36:08,186 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:08,188 get_cmds: Getting commands from 2025:123:12:36:02.791 to 2025:130:12:36:02.791 for scenario=None
2025-05-24 08:36:08,189 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:08,190 get_cmds: Getting recent commands from cache
2025-05-24 08:36:08,190 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:13,038 get_plot_figure: Compressing Telem data for state pitch
2025-05-24 08:36:15,281 get_plot_figure: Creating Telem scatter plot for state pitch
2025-05-24 08:36:15,320 get_plot_figure: Compressing State data for state pitch
2025-05-24 08:36:17,293 get_plot_figure: Creating State scatter plot for state pitch
2025-05-24 08:36:17,352 get_plot_html: Creating HTML for state pitch
2025-05-24 08:36:17,366 run_validators: Validating off_nom_roll
_get_data: Getting data for conlofp between 2025:130:12:20:31.793 to 2025:144:12:51:31.793
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:36:17,552 tlm: Fetching telemetry for ['roll_comp'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for roll_comp between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_comp_data: Getting computed values for roll_comp
_get_data: Getting data for conlofp between 2025:130:12:05:31.793 to 2025:144:13:06:31.793
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
_get_data: Getting data for aopcadmd between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_data: Getting data for orbitephem0_x between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for orbitephem0_y between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for orbitephem0_z between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for solarephem0_x between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for solarephem0_y between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for solarephem0_z between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for aoattqt1 between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for aoattqt2 between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for aoattqt3 between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
_get_data: Getting data for aoattqt4 between 2025:130:12:09:01.793 to 2025:144:13:03:01.793
fetch: Setting bads because of gaps in aoattqt1 between 2025:144:11:10:35.166 to 2025:144:13:03:01.716
fetch: Setting bads because of gaps in aoattqt2 between 2025:144:11:10:35.166 to 2025:144:13:03:01.716
fetch: Setting bads because of gaps in aoattqt3 between 2025:144:11:10:35.166 to 2025:144:13:03:01.716
fetch: Setting bads because of gaps in aoattqt4 between 2025:144:11:10:35.166 to 2025:144:13:03:01.716
interpolate: Interpolating index for orbitephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for orbitephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_x
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_y
interpolate: Slicing on indexes
interpolate: Interpolating index for solarephem0_z
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt1
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt2
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt3
interpolate: Slicing on indexes
interpolate: Interpolating index for aoattqt4
interpolate: Slicing on indexes
interpolate: Interpolating index for roll_comp
interpolate: Slicing on indexes
2025-05-24 08:36:18,460 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:18,460 get_states: Getting commanded states ['off_nom_roll', 'pitch', 'pcad_mode'] between 2025:130:12:36:02.791 - 2025:144:11:10:30.041
2025-05-24 08:36:18,461 get_cmds: Getting commands from 2025:130:12:36:02.791 to 2025:144:11:10:30.041 for scenario=None
2025-05-24 08:36:18,461 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:18,461 get_cmds: Getting recent commands from cache
2025-05-24 08:36:18,462 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:18,465 get_cmds: Getting commands from 2025:123:12:36:02.791 to 2025:130:12:36:02.791 for scenario=None
2025-05-24 08:36:18,465 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:18,466 get_cmds: Getting recent commands from cache
2025-05-24 08:36:18,466 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:22,739 get_plot_figure: Compressing Telem data for state off_nom_roll
2025-05-24 08:36:24,801 get_plot_figure: Creating Telem scatter plot for state off_nom_roll
2025-05-24 08:36:24,825 get_plot_figure: Compressing State data for state off_nom_roll
2025-05-24 08:36:26,806 get_plot_figure: Creating State scatter plot for state off_nom_roll
2025-05-24 08:36:26,860 get_plot_html: Creating HTML for state off_nom_roll
2025-05-24 08:36:26,870 run_validators: Validating dither
_get_data: Getting data for conlofp between 2025:130:12:20:31.793 to 2025:144:12:51:31.793
interpolate: Interpolating index for conlofp
interpolate: Slicing on indexes
2025-05-24 08:36:27,048 get_cmds: Getting commands from 2025:130:12:16:01.793 to 2025:144:12:36:01.793 for scenario=None
2025-05-24 08:36:27,048 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:27,049 get_cmds: Getting recent commands from cache
2025-05-24 08:36:27,050 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:27,108 add_exclude_interval: dither: excluding interval 2025:132:02:27:38.000 - 2025:132:02:40:38.000: SRDC
2025-05-24 08:36:27,109 add_exclude_interval: dither: excluding interval 2025:135:02:24:59.000 - 2025:135:02:37:59.000: SRDC
2025-05-24 08:36:27,111 add_exclude_interval: dither: excluding interval 2025:137:11:40:31.000 - 2025:137:11:53:31.000: SRDC
2025-05-24 08:36:27,112 add_exclude_interval: dither: excluding interval 2025:137:18:20:03.000 - 2025:137:18:33:03.000: SRDC
2025-05-24 08:36:27,112 tlm: Fetching telemetry for ['aodithen'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for aodithen between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/AODITHEN.h5
_get_msid_data_from_cxc: Slicing AODITHEN arrays [731:143560]
interpolate: Interpolating index for aodithen
interpolate: Slicing on indexes
2025-05-24 08:36:27,286 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:27,286 get_states: Getting commanded states ['dither'] between 2025:130:12:36:07.716 - 2025:144:11:10:24.756
2025-05-24 08:36:27,286 get_cmds: Getting commands from 2025:130:12:36:07.716 to 2025:144:11:10:24.756 for scenario=None
2025-05-24 08:36:27,287 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:27,287 get_cmds: Getting recent commands from cache
2025-05-24 08:36:27,288 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:27,290 get_cmds: Getting commands from 2025:123:12:36:07.716 to 2025:130:12:36:07.716 for scenario=None
2025-05-24 08:36:27,290 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:27,291 get_cmds: Getting recent commands from cache
2025-05-24 08:36:27,292 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:27,992 get_plot_figure: Compressing Telem data for state dither
2025-05-24 08:36:28,233 get_plot_figure: Creating Telem scatter plot for state dither
2025-05-24 08:36:28,235 get_plot_figure: Compressing State data for state dither
2025-05-24 08:36:28,476 get_plot_figure: Creating State scatter plot for state dither
2025-05-24 08:36:28,506 get_plot_html: Creating HTML for state dither
2025-05-24 08:36:28,510 run_validators: Validating pcad_mode
2025-05-24 08:36:28,521 get_cmds: Getting commands from 2025:130:12:16:01.793 to 2025:144:12:36:01.793 for scenario=None
2025-05-24 08:36:28,521 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:28,522 get_cmds: Getting recent commands from cache
2025-05-24 08:36:28,523 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:28,578 add_exclude_interval: pcad_mode: excluding interval 2025:132:02:27:38.000 - 2025:132:02:40:38.000: SRDC
2025-05-24 08:36:28,579 add_exclude_interval: pcad_mode: excluding interval 2025:135:02:24:59.000 - 2025:135:02:37:59.000: SRDC
2025-05-24 08:36:28,580 add_exclude_interval: pcad_mode: excluding interval 2025:137:11:40:31.000 - 2025:137:11:53:31.000: SRDC
2025-05-24 08:36:28,582 add_exclude_interval: pcad_mode: excluding interval 2025:137:18:20:03.000 - 2025:137:18:33:03.000: SRDC
2025-05-24 08:36:28,582 tlm: Fetching telemetry for ['aopcadmd'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for aopcadmd between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad3eng/AOPCADMD.h5
_get_msid_data_from_cxc: Slicing AOPCADMD arrays [5845:1148480]
interpolate: Interpolating index for aopcadmd
interpolate: Slicing on indexes
2025-05-24 08:36:28,810 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:28,810 get_states: Getting commanded states ['pcad_mode'] between 2025:130:12:36:02.591 - 2025:144:11:10:30.906
2025-05-24 08:36:28,810 get_cmds: Getting commands from 2025:130:12:36:02.591 to 2025:144:11:10:30.906 for scenario=None
2025-05-24 08:36:28,810 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:28,811 get_cmds: Getting recent commands from cache
2025-05-24 08:36:28,811 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:28,813 get_cmds: Getting commands from 2025:123:12:36:02.591 to 2025:130:12:36:02.591 for scenario=None
2025-05-24 08:36:28,814 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:28,814 get_cmds: Getting recent commands from cache
2025-05-24 08:36:28,815 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:36,842 get_plot_figure: Compressing Telem data for state pcad_mode
2025-05-24 08:36:38,787 get_plot_figure: Creating Telem scatter plot for state pcad_mode
2025-05-24 08:36:38,791 get_plot_figure: Compressing State data for state pcad_mode
2025-05-24 08:36:40,752 get_plot_figure: Creating State scatter plot for state pcad_mode
2025-05-24 08:36:40,805 get_plot_html: Creating HTML for state pcad_mode
2025-05-24 08:36:40,810 run_validators: Validating simpos
2025-05-24 08:36:40,816 tlm: Fetching telemetry for ['3tscpos'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for 3tscpos between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/sim_mrg/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/sim_mrg/3TSCPOS.h5
_get_msid_data_from_cxc: Slicing 3TSCPOS arrays [182:35826]
interpolate: Interpolating index for 3tscpos
interpolate: Slicing on indexes
2025-05-24 08:36:40,938 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:40,938 get_states: Getting commanded states ['simpos'] between 2025:130:12:36:32.316 - 2025:144:11:10:17.069
2025-05-24 08:36:40,938 get_cmds: Getting commands from 2025:130:12:36:32.316 to 2025:144:11:10:17.069 for scenario=None
2025-05-24 08:36:40,938 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:40,939 get_cmds: Getting recent commands from cache
2025-05-24 08:36:40,940 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:40,942 get_cmds: Getting commands from 2025:123:12:36:32.316 to 2025:130:12:36:32.316 for scenario=None
2025-05-24 08:36:40,942 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:40,943 get_cmds: Getting recent commands from cache
2025-05-24 08:36:40,943 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:41,081 get_plot_figure: Compressing Telem data for state simpos
2025-05-24 08:36:41,144 get_plot_figure: Creating Telem scatter plot for state simpos
2025-05-24 08:36:41,148 get_plot_figure: Compressing State data for state simpos
2025-05-24 08:36:41,209 get_plot_figure: Creating State scatter plot for state simpos
2025-05-24 08:36:41,218 get_plot_html: Creating HTML for state simpos
2025-05-24 08:36:41,222 run_validators: Validating obsid
2025-05-24 08:36:41,229 tlm: Fetching telemetry for ['cobsrqid'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for cobsrqid between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm10eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/ccdm10eng/COBSRQID.h5
_get_msid_data_from_cxc: Slicing COBSRQID arrays [2923:574240]
interpolate: Interpolating index for cobsrqid
interpolate: Slicing on indexes
2025-05-24 08:36:41,508 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:41,508 get_states: Getting commanded states ['obsid'] between 2025:130:12:36:03.616 - 2025:144:11:10:30.650
2025-05-24 08:36:41,508 get_cmds: Getting commands from 2025:130:12:36:03.616 to 2025:144:11:10:30.650 for scenario=None
2025-05-24 08:36:41,508 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:41,509 get_cmds: Getting recent commands from cache
2025-05-24 08:36:41,509 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:41,512 get_cmds: Getting commands from 2025:123:12:36:03.616 to 2025:130:12:36:03.616 for scenario=None
2025-05-24 08:36:41,512 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:41,513 get_cmds: Getting recent commands from cache
2025-05-24 08:36:41,513 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:43,658 get_plot_figure: Compressing Telem data for state obsid
2025-05-24 08:36:44,639 get_plot_figure: Creating Telem scatter plot for state obsid
2025-05-24 08:36:44,642 get_plot_figure: Compressing State data for state obsid
2025-05-24 08:36:45,604 get_plot_figure: Creating State scatter plot for state obsid
2025-05-24 08:36:45,618 get_plot_html: Creating HTML for state obsid
2025-05-24 08:36:45,622 run_validators: Validating letg
2025-05-24 08:36:45,629 tlm: Fetching telemetry for ['4ootgsel', '4ootgmtn'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for 4ootgsel between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/4OOTGSEL.h5
_get_msid_data_from_cxc: Slicing 4OOTGSEL arrays [183:35890]
_get_data: Getting data for 4ootgmtn between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Using times_cache for tel2eng 863266930.9769996 to 864478330.9769999
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/tel2eng/4OOTGMTN.h5
_get_msid_data_from_cxc: Slicing 4OOTGMTN arrays [183:35890]
interpolate: Interpolating index for 4ootgsel
interpolate: Slicing on indexes
interpolate: Interpolating index for 4ootgmtn
interpolate: Slicing on indexes
2025-05-24 08:36:45,856 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:45,856 get_states: Getting commanded states ['letg'] between 2025:130:12:36:32.316 - 2025:144:11:10:06.050
2025-05-24 08:36:45,856 get_cmds: Getting commands from 2025:130:12:36:32.316 to 2025:144:11:10:06.050 for scenario=None
2025-05-24 08:36:45,856 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:45,857 get_cmds: Getting recent commands from cache
2025-05-24 08:36:45,857 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:45,859 get_cmds: Getting commands from 2025:123:12:36:32.316 to 2025:130:12:36:32.316 for scenario=None
2025-05-24 08:36:45,860 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:45,860 get_cmds: Getting recent commands from cache
2025-05-24 08:36:45,861 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:45,870 get_cmds: Getting commands from 2025:100:12:36:32.316 to 2025:130:12:36:32.316 for scenario=None
2025-05-24 08:36:45,870 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:45,871 get_cmds: Getting recent commands from cache
2025-05-24 08:36:45,871 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:36:32.316
2025-05-24 08:36:47,654 load_idx_cmds: Loaded /export/tom/kadi-validate/data/kadi/cmds2.h5 with 1709895 commands
2025-05-24 08:36:47,663 get_matching_block_idx: Selecting commands from cmds_arch[0:]
2025-05-24 08:36:48,066 get_matching_block_idx: Matching blocks for (a) recent commands and (b) existing HDF5
2025-05-24 08:36:48,066 get_matching_block_idx: Match(a=0, b=0, size=3)
2025-05-24 08:36:48,066 get_matching_block_idx: Match(a=6, b=3, size=30)
2025-05-24 08:36:48,066 get_matching_block_idx: Match(a=37, b=33, size=10498)
2025-05-24 08:36:48,066 get_matching_block_idx: Match(a=10535, b=10531, size=0)
2025-05-24 08:36:48,066 get_matching_block_idx: Diffs between (a) recent commands and (b) existing HDF5
2025-05-24 08:36:48,066 get_matching_block_idx: ('equal', 0, 3, 0, 3)
2025-05-24 08:36:48,066 get_matching_block_idx: ('delete', 3, 6, 3, 3)
2025-05-24 08:36:48,066 get_matching_block_idx: ('equal', 6, 36, 3, 33)
2025-05-24 08:36:48,066 get_matching_block_idx: ('delete', 36, 37, 33, 33)
2025-05-24 08:36:48,066 get_matching_block_idx: ('equal', 37, 10535, 33, 10531)
2025-05-24 08:36:48,070 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:36:48,309 get_plot_figure: Compressing Telem data for state letg
2025-05-24 08:36:48,371 get_plot_figure: Creating Telem scatter plot for state letg
2025-05-24 08:36:48,372 get_plot_figure: Compressing State data for state letg
2025-05-24 08:36:48,434 get_plot_figure: Creating State scatter plot for state letg
2025-05-24 08:36:48,442 get_plot_html: Creating HTML for state letg
2025-05-24 08:36:48,445 run_validators: Validating hetg
2025-05-24 08:36:48,452 tlm: Fetching telemetry for ['4ootgsel', '4ootgmtn'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for 4ootgsel between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_data: Getting data for 4ootgmtn between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
interpolate: Interpolating index for 4ootgsel
interpolate: Slicing on indexes
interpolate: Interpolating index for 4ootgmtn
interpolate: Slicing on indexes
2025-05-24 08:36:48,466 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:48,467 get_states: Getting commanded states ['hetg'] between 2025:130:12:36:32.316 - 2025:144:11:10:06.050
2025-05-24 08:36:48,467 get_cmds: Getting commands from 2025:130:12:36:32.316 to 2025:144:11:10:06.050 for scenario=None
2025-05-24 08:36:48,467 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:48,468 get_cmds: Getting recent commands from cache
2025-05-24 08:36:48,468 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:48,470 get_cmds: Getting commands from 2025:123:12:36:32.316 to 2025:130:12:36:32.316 for scenario=None
2025-05-24 08:36:48,470 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:48,471 get_cmds: Getting recent commands from cache
2025-05-24 08:36:48,472 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:48,481 get_cmds: Getting commands from 2025:100:12:36:32.316 to 2025:130:12:36:32.316 for scenario=None
2025-05-24 08:36:48,481 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:48,482 get_cmds: Getting recent commands from cache
2025-05-24 08:36:48,482 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:36:32.316
2025-05-24 08:36:48,484 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:36:48,725 get_plot_figure: Compressing Telem data for state hetg
2025-05-24 08:36:48,787 get_plot_figure: Creating Telem scatter plot for state hetg
2025-05-24 08:36:48,788 get_plot_figure: Compressing State data for state hetg
2025-05-24 08:36:48,851 get_plot_figure: Creating State scatter plot for state hetg
2025-05-24 08:36:48,859 get_plot_html: Creating HTML for state hetg
2025-05-24 08:36:48,862 run_validators: Validating sun_pos_mon
2025-05-24 08:36:48,870 tlm: Fetching telemetry for ['aopssupm'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for aopssupm between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/pcad8eng/AOPSSUPM.h5
_get_msid_data_from_cxc: Slicing AOPSSUPM arrays [731:143560]
interpolate: Interpolating index for aopssupm
interpolate: Slicing on indexes
filter_bad: Filtering bad values for aopssupm
2025-05-24 08:36:49,060 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:49,060 get_states: Getting commanded states ['sun_pos_mon'] between 2025:130:12:36:07.716 - 2025:144:11:10:30.138
2025-05-24 08:36:49,060 get_cmds: Getting commands from 2025:130:12:36:07.716 to 2025:144:11:10:30.138 for scenario=None
2025-05-24 08:36:49,060 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:49,061 get_cmds: Getting recent commands from cache
2025-05-24 08:36:49,062 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:49,064 get_cmds: Getting commands from 2025:123:12:36:07.716 to 2025:130:12:36:07.716 for scenario=None
2025-05-24 08:36:49,064 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:49,065 get_cmds: Getting recent commands from cache
2025-05-24 08:36:49,065 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:49,078 get_cmds: Getting commands from 2025:100:12:36:07.716 to 2025:130:12:36:07.716 for scenario=None
2025-05-24 08:36:49,079 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:49,079 get_cmds: Getting recent commands from cache
2025-05-24 08:36:49,080 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2025:100:12:36:07.716
2025-05-24 08:36:49,082 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:36:49,106 get_cmds: Getting commands from 2024:316:12:36:07.716 to 2025:130:12:36:07.716 for scenario=None
2025-05-24 08:36:49,106 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:49,107 get_cmds: Getting recent commands from cache
2025-05-24 08:36:49,107 _merge_cmds_archive_recent: Merging cmds_recent with archive commands from 2024:316:12:36:07.716
2025-05-24 08:36:49,112 get_cmds: Getting commands from archive + recent: start < recent loads start + 3 days for scenario=None
2025-05-24 08:36:50,025 get_plot_figure: Compressing Telem data for state sun_pos_mon
2025-05-24 08:36:50,269 get_plot_figure: Creating Telem scatter plot for state sun_pos_mon
2025-05-24 08:36:50,272 get_plot_figure: Compressing State data for state sun_pos_mon
2025-05-24 08:36:50,515 get_plot_figure: Creating State scatter plot for state sun_pos_mon
2025-05-24 08:36:50,528 get_plot_html: Creating HTML for state sun_pos_mon
2025-05-24 08:36:50,532 run_validators: Validating dpa_power
2025-05-24 08:36:50,553 add_exclude_interval: dpa_power: excluding interval 2025:133:22:37:17.000 - 2025:133:23:00:49.000: ?
2025-05-24 08:36:51,168 tlm: Fetching telemetry for ['dpa_power'] between 2025:130:12:36:01.793 and 2025:144:12:36:01.793
_get_data: Getting data for dpa_power between 2025:130:12:21:01.793 to 2025:144:12:51:01.793
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/dp_acispow128/TIME.h5
_get_msid_data_from_cxc: Reading /export/tom/kadi-validate/data/eng_archive/data/dp_acispow128/DP_DPA_POWER.h5
_get_msid_data_from_cxc: Slicing DP_DPA_POWER arrays [47:35685]
interpolate: Interpolating index for dpa_power
interpolate: Slicing on indexes
2025-05-24 08:36:51,325 get_states: Using kadi.commands.states to get cmd_states
2025-05-24 08:36:51,325 get_states: Getting commanded states ['ccd_count', 'clocking', 'feps', 'ccds', 'fep_count', 'si_mode'] between 2025:130:12:36:28.416 - 2025:144:11:10:09.125
2025-05-24 08:36:51,325 get_cmds: Getting commands from 2025:130:12:36:28.416 to 2025:144:11:10:09.125 for scenario=None
2025-05-24 08:36:51,326 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:51,327 get_cmds: Getting recent commands from cache
2025-05-24 08:36:51,327 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:51,330 get_cmds: Getting commands from 2025:123:12:36:28.416 to 2025:130:12:36:28.416 for scenario=None
2025-05-24 08:36:51,330 get_cmds: Cache key: (None, None, 30.0, None)
2025-05-24 08:36:51,331 get_cmds: Getting recent commands from cache
2025-05-24 08:36:51,331 get_cmds: Getting commands from recent only: start and stop are within recent commands for scenario=None
2025-05-24 08:36:52,091 get_plot_figure: Compressing Telem data for state dpa_power
2025-05-24 08:36:52,524 get_plot_figure: Creating Telem scatter plot for state dpa_power
2025-05-24 08:36:52,755 get_plot_figure: Compressing State data for state dpa_power
2025-05-24 08:36:52,822 get_plot_figure: Creating State scatter plot for state dpa_power
2025-05-24 08:36:52,871 get_plot_html: Creating HTML for state dpa_power