Prev
Index
Next
fid_drift_mon3: Mon Apr 14 20:16:17 2025
File: /proj/sot/ska/data/fid_drift_mon3/logs/daily.0/fid_drift_mon.log
| Task | Status | Age | MaxAge | Last Date |
|---|
| fid_drift_mon3 |
NOT OK
|
0.60 | 2.0 |
Mon Apr 14 20:16:17 2025 |
Errors:
- Line 68: <<2025-Apr-14 20:16>> Traceback (most recent call last):
- Line 92: <<2025-Apr-14 20:16>> TimeoutError: The read operation timed out
- Line 94: <<2025-Apr-14 20:16>> The above exception was the direct cause of the following exception:
- Line 96: <<2025-Apr-14 20:16>> Traceback (most recent call last):
- Line 104: <<2025-Apr-14 20:16>> raise reraise(type(error), error, _stacktrace)
- Line 114: <<2025-Apr-14 20:16>> raise ReadTimeoutError(
- Line 115: <<2025-Apr-14 20:16>> urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)
- Line 115: <<2025-Apr-14 20:16>> urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)
- Line 117: <<2025-Apr-14 20:16>> During handling of the above exception, another exception occurred:
- Line 119: <<2025-Apr-14 20:16>> Traceback (most recent call last):
- Line 164: <<2025-Apr-14 20:16>> requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)
File contents:
############################################################
fid_drift_mon_update_fid_stats --data-dir /proj/sot/ska3/flight/data/fid_drift_mon3
############################################################
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,068 process_obs: ***** PROCESSING OBSID 30869 at 2025:083:20:56:01.882 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,070 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,072 process_obs: ***** PROCESSING OBSID 30869 at 2025:084:01:44:08.066 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,074 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,077 process_obs: ***** PROCESSING OBSID 30869 at 2025:084:06:56:58.951 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,079 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,081 process_obs: ***** PROCESSING OBSID 30869 at 2025:084:11:35:15.747 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,083 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,084 process_obs: ***** PROCESSING OBSID 30869 at 2025:084:13:36:22.460 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,086 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,088 process_obs: ***** PROCESSING OBSID 30869 at 2025:084:15:55:40.251 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,090 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,091 process_obs: ***** PROCESSING OBSID 30869 at 2025:084:16:46:42.514 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,093 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,095 process_obs: ***** PROCESSING OBSID 30869 at 2025:084:21:55:40.251 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,097 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,098 process_obs: ***** PROCESSING OBSID 30869 at 2025:085:00:27:11.398 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,100 process_obs: Obsid 30869 has multiple observations, skipping
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,225 process_obs: ***** PROCESSING OBSID 30523 at 2025:101:00:25:59.789 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,391 get_archive_file_data: No FIDPROPS files found for obsid 30523 between 2025:101:00:25:59.789 and 2025:101:13:16:59.538
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,416 process_obs: ***** PROCESSING OBSID 29766 at 2025:102:09:52:37.360 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,559 process_obs: Got 3 rows fidprops and 16902 rows acacen
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,586 process_obs: ***** PROCESSING OBSID 30159 at 2025:102:11:38:55.297 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,851 process_obs: Got 3 rows fidprops and 117542 rows acacen
<<2025-Apr-14 20:12>> 2025-04-14 20:12:13,894 process_obs: ***** PROCESSING OBSID 28521 at 2025:102:20:16:09.455 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,084 process_obs: Got 3 rows fidprops and 72528 rows acacen
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,118 process_obs: ***** PROCESSING OBSID 30508 at 2025:103:11:17:03.468 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,167 get_archive_file_data: No FIDPROPS files found for obsid 30508 between 2025:103:11:17:03.468 and 2025:103:12:44:43.217
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,169 process_obs: ***** PROCESSING OBSID 30888 at 2025:103:13:17:04.928 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,218 get_archive_file_data: No FIDPROPS files found for obsid 30888 between 2025:103:13:17:04.928 and 2025:103:16:08:04.677
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,220 process_obs: ***** PROCESSING OBSID 30889 at 2025:103:17:02:07.327 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,267 get_archive_file_data: No FIDPROPS files found for obsid 30889 between 2025:103:17:02:07.327 and 2025:103:21:58:07.076
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,268 process_obs: ***** PROCESSING OBSID 30454 at 2025:103:22:50:56.065 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,315 get_archive_file_data: No FIDPROPS files found for obsid 30454 between 2025:103:22:50:56.065 and 2025:104:01:52:30.585
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,317 process_obs: ***** PROCESSING OBSID 30085 at 2025:104:03:16:12.273 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,362 get_archive_file_data: No FIDPROPS files found for obsid 30085 between 2025:104:03:16:12.273 and 2025:104:06:07:12.022
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,364 process_obs: ***** PROCESSING OBSID 30781 at 2025:104:07:09:09.441 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,407 get_archive_file_data: No FIDPROPS files found for obsid 30781 between 2025:104:07:09:09.441 and 2025:104:12:46:49.190
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,409 process_obs: ***** PROCESSING OBSID 29928 at 2025:104:13:10:43.426 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,452 get_archive_file_data: No FIDPROPS files found for obsid 29928 between 2025:104:13:10:43.426 and 2025:104:19:44:54.275
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,454 process_obs: ***** PROCESSING OBSID 27993 at 2025:104:19:59:34.161 *****
<<2025-Apr-14 20:12>> 2025-04-14 20:12:14,494 get_archive_file_data: No FIDPROPS files found for obsid 27993 between 2025:104:19:59:34.161 and 2025:104:23:55:39.275
############################################################
fid_drift_mon_plot_drift --data-dir /proj/sot/ska3/flight/data/fid_drift_mon3 --web-dir /proj/sot/ska3/flight/www/ASPECT/fid_drift_mon3
############################################################
<<2025-Apr-14 20:12>> 2025-04-14 20:12:16,751 main: Writing HTML to /proj/sot/ska3/flight/www/ASPECT/fid_drift_mon3/index.html
############################################################
fid_drift_mon_plot_command_vs_telem --data-dir /proj/sot/ska3/flight/data/fid_drift_mon3 --out /proj/sot/ska3/flight/www/ASPECT/fid_drift_mon3/starcheck_telem.png
############################################################
<<2025-Apr-14 20:12>> 2025-04-14 19:12:20,007 get_dwells_with_fids: Getting dwells between 2025:015:00:12:18.664 and 2025:104:12:49:49.369
<<2025-Apr-14 20:12>> 2025-04-14 19:12:25,928 get_dwells_with_fids: Skipping duplicate obsid 28203 for dwell start=2025:015:01:47:57.061 dur=17161
<<2025-Apr-14 20:15>> 2025-04-14 19:15:01,731 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:083:21:42:23.208 dur=12478
<<2025-Apr-14 20:15>> 2025-04-14 19:15:01,950 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:084:01:46:28.409 dur=16119
<<2025-Apr-14 20:15>> 2025-04-14 19:15:02,204 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:084:06:59:18.210 dur=14621
<<2025-Apr-14 20:15>> 2025-04-14 19:15:02,646 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:084:11:36:36.011 dur=6181
<<2025-Apr-14 20:15>> 2025-04-14 19:15:03,129 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:084:13:37:45.311 dur=5922
<<2025-Apr-14 20:15>> 2025-04-14 19:15:03,648 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:084:15:57:38.012 dur=1082
<<2025-Apr-14 20:15>> 2025-04-14 19:15:04,194 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:084:16:49:09.412 dur=13800
<<2025-Apr-14 20:15>> 2025-04-14 19:15:04,739 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:084:20:47:26.113 dur=2376
<<2025-Apr-14 20:15>> 2025-04-14 19:15:04,943 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:084:21:57:36.813 dur=7402
<<2025-Apr-14 20:15>> 2025-04-14 19:15:05,390 get_dwells_with_fids: Skipping duplicate obsid 30869 for dwell start=2025:085:00:28:50.114 dur=22870
<<2025-Apr-14 20:15>> 2025-04-14 19:15:45,969 get_fids_commands: Getting fids from commands
<<2025-Apr-14 20:16>> Traceback (most recent call last):
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request
<<2025-Apr-14 20:16>> response = conn.getresponse()
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse
<<2025-Apr-14 20:16>> httplib_response = super().getresponse()
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 1428, in getresponse
<<2025-Apr-14 20:16>> response.begin()
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 331, in begin
<<2025-Apr-14 20:16>> version, status, reason = self._read_status()
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 292, in _read_status
<<2025-Apr-14 20:16>> line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/socket.py", line 720, in readinto
<<2025-Apr-14 20:16>> return self._sock.recv_into(b)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/ssl.py", line 1251, in recv_into
<<2025-Apr-14 20:16>> return self.read(nbytes, buffer)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/ssl.py", line 1103, in read
<<2025-Apr-14 20:16>> return self._sslobj.read(len, buffer)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> TimeoutError: The read operation timed out
<<2025-Apr-14 20:16>>
<<2025-Apr-14 20:16>> The above exception was the direct cause of the following exception:
<<2025-Apr-14 20:16>>
<<2025-Apr-14 20:16>> Traceback (most recent call last):
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
<<2025-Apr-14 20:16>> resp = conn.urlopen(
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
<<2025-Apr-14 20:16>> retries = retries.increment(
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
<<2025-Apr-14 20:16>> raise reraise(type(error), error, _stacktrace)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
<<2025-Apr-14 20:16>> raise value
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
<<2025-Apr-14 20:16>> response = self._make_request(
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
<<2025-Apr-14 20:16>> self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
<<2025-Apr-14 20:16>> raise ReadTimeoutError(
<<2025-Apr-14 20:16>> urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)
<<2025-Apr-14 20:16>>
<<2025-Apr-14 20:16>> During handling of the above exception, another exception occurred:
<<2025-Apr-14 20:16>>
<<2025-Apr-14 20:16>> Traceback (most recent call last):
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/bin/fid_drift_mon_plot_command_vs_telem", line 10, in
<<2025-Apr-14 20:16>> sys.exit(main())
<<2025-Apr-14 20:16>> ^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/fid_drift_mon/plot_command_vs_telem.py", line 277, in main
<<2025-Apr-14 20:16>> fids_commands = get_fids_commands(dwells)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/fid_drift_mon/plot_command_vs_telem.py", line 79, in get_fids_commands
<<2025-Apr-14 20:16>> starcats = observations.get_starcats(obsid=obsid)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/observations.py", line 508, in get_starcats
<<2025-Apr-14 20:16>> obss = get_observations(
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/observations.py", line 681, in get_observations
<<2025-Apr-14 20:16>> cmds = get_cmds(scenario=scenario, event_filter=event_filter)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 283, in get_cmds
<<2025-Apr-14 20:16>> cmds_recent = update_cmd_events_and_loads_and_get_cmds_recent(
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 426, in update_cmd_events_and_loads_and_get_cmds_recent
<<2025-Apr-14 20:16>> loads = update_loads(scenario, lookback=lookback, stop_loads=stop_loads)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 1190, in update_loads
<<2025-Apr-14 20:16>> contents = occweb.get_occweb_dir(dir_year_month)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/occweb.py", line 367, in get_occweb_dir
<<2025-Apr-14 20:16>> html = get_occweb_page(
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/occweb.py", line 328, in get_occweb_page
<<2025-Apr-14 20:16>> req = requests.get(url, auth=auth, timeout=timeout)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/api.py", line 73, in get
<<2025-Apr-14 20:16>> return request("get", url, params=params, **kwargs)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/api.py", line 59, in request
<<2025-Apr-14 20:16>> return session.request(method=method, url=url, **kwargs)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
<<2025-Apr-14 20:16>> resp = self.send(prep, **send_kwargs)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
<<2025-Apr-14 20:16>> r = adapter.send(request, **kwargs)
<<2025-Apr-14 20:16>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-14 20:16>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
<<2025-Apr-14 20:16>> raise ReadTimeout(e, request=request)
<<2025-Apr-14 20:16>> requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Read timed out. (read timeout=30)