Prev
Index
Next
fid_drift_mon3: Mon Oct 20 20:16:23 2025
File: /proj/sot/ska/data/fid_drift_mon3/logs/daily.0/fid_drift_mon.log
| Task | Status | Age | MaxAge | Last Date |
|---|
| fid_drift_mon3 |
NOT OK
|
0.60 | 2.0 |
Mon Oct 20 20:16:23 2025 |
Errors:
- Line 47: <<2025-Oct-20 20:16>> Traceback (most recent call last):
- Line 71: <<2025-Oct-20 20:16>> TimeoutError: The read operation timed out
- Line 73: <<2025-Oct-20 20:16>> The above exception was the direct cause of the following exception:
- Line 75: <<2025-Oct-20 20:16>> Traceback (most recent call last):
- Line 83: <<2025-Oct-20 20:16>> raise reraise(type(error), error, _stacktrace)
- Line 93: <<2025-Oct-20 20:16>> raise ReadTimeoutError(
- Line 94: <<2025-Oct-20 20:16>> urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)
- Line 94: <<2025-Oct-20 20:16>> urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)
- Line 96: <<2025-Oct-20 20:16>> During handling of the above exception, another exception occurred:
- Line 98: <<2025-Oct-20 20:16>> Traceback (most recent call last):
- Line 143: <<2025-Oct-20 20:16>> requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)
File contents:
############################################################
fid_drift_mon_update_fid_stats --data-dir /proj/sot/ska3/flight/data/fid_drift_mon3
############################################################
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,361 process_obs: ***** PROCESSING OBSID 30184 at 2025:290:00:12:52.998 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,520 get_archive_file_data: No FIDPROPS files found for obsid 30184 between 2025:290:00:12:52.998 and 2025:290:08:37:12.746
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,526 process_obs: ***** PROCESSING OBSID 31447 at 2025:290:08:52:14.704 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,600 get_archive_file_data: No FIDPROPS files found for obsid 31447 between 2025:290:08:52:14.704 and 2025:290:10:38:03.746
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,603 process_obs: ***** PROCESSING OBSID 29672 at 2025:290:10:52:35.630 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,695 get_archive_file_data: No FIDPROPS files found for obsid 29672 between 2025:290:10:52:35.630 and 2025:290:17:32:37.023
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,697 process_obs: ***** PROCESSING OBSID 30517 at 2025:291:07:58:14.674 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,746 get_archive_file_data: No FIDPROPS files found for obsid 30517 between 2025:291:07:58:14.674 and 2025:291:13:35:54.423
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,749 process_obs: ***** PROCESSING OBSID 31920 at 2025:291:14:19:28.993 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,791 get_archive_file_data: No FIDPROPS files found for obsid 31920 between 2025:291:14:19:28.993 and 2025:291:17:10:28.742
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,794 process_obs: ***** PROCESSING OBSID 31921 at 2025:291:17:38:21.731 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,839 get_archive_file_data: No FIDPROPS files found for obsid 31921 between 2025:291:17:38:21.731 and 2025:291:22:01:01.480
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,842 process_obs: ***** PROCESSING OBSID 31104 at 2025:291:22:31:39.699 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,892 get_archive_file_data: No FIDPROPS files found for obsid 31104 between 2025:291:22:31:39.699 and 2025:292:04:09:19.448
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,895 process_obs: ***** PROCESSING OBSID 30088 at 2025:292:04:39:15.618 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,962 get_archive_file_data: No FIDPROPS files found for obsid 30088 between 2025:292:04:39:15.618 and 2025:292:13:03:35.367
<<2025-Oct-20 20:12>> 2025-10-20 20:12:11,969 process_obs: ***** PROCESSING OBSID 30533 at 2025:292:13:23:30.929 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,028 get_archive_file_data: No FIDPROPS files found for obsid 30533 between 2025:292:13:23:30.929 and 2025:292:21:47:50.679
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,030 process_obs: ***** PROCESSING OBSID 30233 at 2025:292:23:14:04.029 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,087 get_archive_file_data: No FIDPROPS files found for obsid 30233 between 2025:292:23:14:04.029 and 2025:293:02:05:03.778
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,090 process_obs: ***** PROCESSING OBSID 31925 at 2025:293:02:40:50.237 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,141 get_archive_file_data: No FIDPROPS files found for obsid 31925 between 2025:293:02:40:50.237 and 2025:293:05:31:49.986
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,144 process_obs: ***** PROCESSING OBSID 31931 at 2025:293:06:07:37.103 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,194 get_archive_file_data: No FIDPROPS files found for obsid 31931 between 2025:293:06:07:37.103 and 2025:293:08:53:45.232
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,199 process_obs: ***** PROCESSING OBSID 30518 at 2025:293:22:53:45.883 *****
<<2025-Oct-20 20:12>> 2025-10-20 20:12:12,239 get_archive_file_data: No FIDPROPS files found for obsid 30518 between 2025:293:22:53:45.883 and 2025:294:04:31:25.632
############################################################
fid_drift_mon_plot_drift --data-dir /proj/sot/ska3/flight/data/fid_drift_mon3 --web-dir /proj/sot/ska3/flight/www/ASPECT/fid_drift_mon3
############################################################
<<2025-Oct-20 20:12>> 2025-10-20 20:12:17,456 main: Writing HTML to /proj/sot/ska3/flight/www/ASPECT/fid_drift_mon3/index.html
############################################################
fid_drift_mon_plot_command_vs_telem --data-dir /proj/sot/ska3/flight/data/fid_drift_mon3 --out /proj/sot/ska3/flight/www/ASPECT/fid_drift_mon3/starcheck_telem.png
############################################################
<<2025-Oct-20 20:12>> 2025-10-20 19:12:23,535 get_dwells_with_fids: Getting dwells between 2025:204:00:12:20.330 and 2025:293:14:27:24.146
<<2025-Oct-20 20:13>> 2025-10-20 19:13:58,620 get_dwells_with_fids: Skipping duplicate obsid 28826 for dwell start=2025:244:21:18:35.323 dur=14093
<<2025-Oct-20 20:13>> 2025-10-20 19:13:59,393 get_dwells_with_fids: Skipping duplicate obsid 28826 for dwell start=2025:245:01:52:11.725 dur=20173
<<2025-Oct-20 20:13>> 2025-10-20 19:13:59,561 get_dwells_with_fids: Skipping duplicate obsid 28826 for dwell start=2025:245:08:08:09.926 dur=15112
<<2025-Oct-20 20:13>> 2025-10-20 19:13:59,745 get_dwells_with_fids: Skipping duplicate obsid 28826 for dwell start=2025:245:12:58:39.027 dur=15680
<<2025-Oct-20 20:14>> 2025-10-20 19:14:00,196 get_dwells_with_fids: Skipping duplicate obsid 28826 for dwell start=2025:245:17:59:47.728 dur=11557
<<2025-Oct-20 20:14>> 2025-10-20 19:14:00,457 get_dwells_with_fids: Skipping duplicate obsid 28826 for dwell start=2025:245:22:03:44.729 dur=11902
<<2025-Oct-20 20:15>> 2025-10-20 19:15:51,936 get_fids_commands: Getting fids from commands
<<2025-Oct-20 20:16>> Traceback (most recent call last):
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request
<<2025-Oct-20 20:16>> response = conn.getresponse()
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse
<<2025-Oct-20 20:16>> httplib_response = super().getresponse()
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 1428, in getresponse
<<2025-Oct-20 20:16>> response.begin()
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 331, in begin
<<2025-Oct-20 20:16>> version, status, reason = self._read_status()
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 292, in _read_status
<<2025-Oct-20 20:16>> line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/socket.py", line 720, in readinto
<<2025-Oct-20 20:16>> return self._sock.recv_into(b)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/ssl.py", line 1251, in recv_into
<<2025-Oct-20 20:16>> return self.read(nbytes, buffer)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/ssl.py", line 1103, in read
<<2025-Oct-20 20:16>> return self._sslobj.read(len, buffer)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> TimeoutError: The read operation timed out
<<2025-Oct-20 20:16>>
<<2025-Oct-20 20:16>> The above exception was the direct cause of the following exception:
<<2025-Oct-20 20:16>>
<<2025-Oct-20 20:16>> Traceback (most recent call last):
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
<<2025-Oct-20 20:16>> resp = conn.urlopen(
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
<<2025-Oct-20 20:16>> retries = retries.increment(
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
<<2025-Oct-20 20:16>> raise reraise(type(error), error, _stacktrace)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
<<2025-Oct-20 20:16>> raise value
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
<<2025-Oct-20 20:16>> response = self._make_request(
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
<<2025-Oct-20 20:16>> self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
<<2025-Oct-20 20:16>> raise ReadTimeoutError(
<<2025-Oct-20 20:16>> urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)
<<2025-Oct-20 20:16>>
<<2025-Oct-20 20:16>> During handling of the above exception, another exception occurred:
<<2025-Oct-20 20:16>>
<<2025-Oct-20 20:16>> Traceback (most recent call last):
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/bin/fid_drift_mon_plot_command_vs_telem", line 10, in
<<2025-Oct-20 20:16>> sys.exit(main())
<<2025-Oct-20 20:16>> ^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/fid_drift_mon/plot_command_vs_telem.py", line 277, in main
<<2025-Oct-20 20:16>> fids_commands = get_fids_commands(dwells)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/fid_drift_mon/plot_command_vs_telem.py", line 79, in get_fids_commands
<<2025-Oct-20 20:16>> starcats = observations.get_starcats(obsid=obsid)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/observations.py", line 508, in get_starcats
<<2025-Oct-20 20:16>> obss = get_observations(
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/observations.py", line 681, in get_observations
<<2025-Oct-20 20:16>> cmds = get_cmds(scenario=scenario, event_filter=event_filter)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 283, in get_cmds
<<2025-Oct-20 20:16>> cmds_recent = update_cmd_events_and_loads_and_get_cmds_recent(
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 426, in update_cmd_events_and_loads_and_get_cmds_recent
<<2025-Oct-20 20:16>> loads = update_loads(scenario, lookback=lookback, stop_loads=stop_loads)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 1190, in update_loads
<<2025-Oct-20 20:16>> contents = occweb.get_occweb_dir(dir_year_month)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/occweb.py", line 367, in get_occweb_dir
<<2025-Oct-20 20:16>> html = get_occweb_page(
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/occweb.py", line 328, in get_occweb_page
<<2025-Oct-20 20:16>> req = requests.get(url, auth=auth, timeout=timeout)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/api.py", line 73, in get
<<2025-Oct-20 20:16>> return request("get", url, params=params, **kwargs)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/api.py", line 59, in request
<<2025-Oct-20 20:16>> return session.request(method=method, url=url, **kwargs)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
<<2025-Oct-20 20:16>> resp = self.send(prep, **send_kwargs)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
<<2025-Oct-20 20:16>> r = adapter.send(request, **kwargs)
<<2025-Oct-20 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Oct-20 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
<<2025-Oct-20 20:16>> raise ReadTimeout(e, request=request)
<<2025-Oct-20 20:16>> requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)