Prev
Index
Next
fid_drift_mon3: Wed Apr 16 20:12:33 2025
File: /proj/sot/ska/data/fid_drift_mon3/logs/daily.0/fid_drift_mon.log
| Task | Status | Age | MaxAge | Last Date |
|---|
| fid_drift_mon3 |
NOT OK
|
0.61 | 2.0 |
Wed Apr 16 20:12:33 2025 |
Errors:
- Line 4: <<2025-Apr-16 20:12>> Traceback (most recent call last):
- Line 12: <<2025-Apr-16 20:12>> TimeoutError: timed out
- Line 14: <<2025-Apr-16 20:12>> The above exception was the direct cause of the following exception:
- Line 16: <<2025-Apr-16 20:12>> Traceback (most recent call last):
- Line 30: <<2025-Apr-16 20:12>> raise ConnectTimeoutError(
- Line 31: <<2025-Apr-16 20:12>> urllib3.exceptions.ConnectTimeoutError: (, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)')
- Line 31: <<2025-Apr-16 20:12>> urllib3.exceptions.ConnectTimeoutError: (, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)')
- Line 33: <<2025-Apr-16 20:12>> The above exception was the direct cause of the following exception:
- Line 35: <<2025-Apr-16 20:12>> Traceback (most recent call last):
- Line 43: <<2025-Apr-16 20:12>> raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
- Line 45: <<2025-Apr-16 20:12>> urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Max retries exceeded with url: /occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAR (Caused by ConnectTimeoutError(, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)'))
- Line 45: <<2025-Apr-16 20:12>> urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Max retries exceeded with url: /occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAR (Caused by ConnectTimeoutError(, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)'))
- Line 47: <<2025-Apr-16 20:12>> During handling of the above exception, another exception occurred:
- Line 49: <<2025-Apr-16 20:12>> Traceback (most recent call last):
- Line 88: <<2025-Apr-16 20:12>> requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Max retries exceeded with url: /occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAR (Caused by ConnectTimeoutError(, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)'))
- Line 88: <<2025-Apr-16 20:12>> requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Max retries exceeded with url: /occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAR (Caused by ConnectTimeoutError(, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)'))
File contents:
############################################################
fid_drift_mon_update_fid_stats --data-dir /proj/sot/ska3/flight/data/fid_drift_mon3
############################################################
<<2025-Apr-16 20:12>> Traceback (most recent call last):
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
<<2025-Apr-16 20:12>> sock = connection.create_connection(
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
<<2025-Apr-16 20:12>> raise err
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
<<2025-Apr-16 20:12>> sock.connect(sa)
<<2025-Apr-16 20:12>> TimeoutError: timed out
<<2025-Apr-16 20:12>>
<<2025-Apr-16 20:12>> The above exception was the direct cause of the following exception:
<<2025-Apr-16 20:12>>
<<2025-Apr-16 20:12>> Traceback (most recent call last):
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
<<2025-Apr-16 20:12>> response = self._make_request(
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
<<2025-Apr-16 20:12>> raise new_e
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
<<2025-Apr-16 20:12>> self._validate_conn(conn)
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
<<2025-Apr-16 20:12>> conn.connect()
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 704, in connect
<<2025-Apr-16 20:12>> self.sock = sock = self._new_conn()
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 207, in _new_conn
<<2025-Apr-16 20:12>> raise ConnectTimeoutError(
<<2025-Apr-16 20:12>> urllib3.exceptions.ConnectTimeoutError: (, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)')
<<2025-Apr-16 20:12>>
<<2025-Apr-16 20:12>> The above exception was the direct cause of the following exception:
<<2025-Apr-16 20:12>>
<<2025-Apr-16 20:12>> Traceback (most recent call last):
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
<<2025-Apr-16 20:12>> resp = conn.urlopen(
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
<<2025-Apr-16 20:12>> retries = retries.increment(
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
<<2025-Apr-16 20:12>> raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Max retries exceeded with url: /occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAR (Caused by ConnectTimeoutError(, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)'))
<<2025-Apr-16 20:12>>
<<2025-Apr-16 20:12>> During handling of the above exception, another exception occurred:
<<2025-Apr-16 20:12>>
<<2025-Apr-16 20:12>> Traceback (most recent call last):
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/bin/fid_drift_mon_update_fid_stats", line 10, in
<<2025-Apr-16 20:12>> sys.exit(main())
<<2025-Apr-16 20:12>> ^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/fid_drift_mon/update_fid_stats.py", line 352, in main
<<2025-Apr-16 20:12>> obss = get_observations(start, stop)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/observations.py", line 681, in get_observations
<<2025-Apr-16 20:12>> cmds = get_cmds(scenario=scenario, event_filter=event_filter)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 283, in get_cmds
<<2025-Apr-16 20:12>> cmds_recent = update_cmd_events_and_loads_and_get_cmds_recent(
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 426, in update_cmd_events_and_loads_and_get_cmds_recent
<<2025-Apr-16 20:12>> loads = update_loads(scenario, lookback=lookback, stop_loads=stop_loads)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/commands_v2.py", line 1190, in update_loads
<<2025-Apr-16 20:12>> contents = occweb.get_occweb_dir(dir_year_month)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/occweb.py", line 367, in get_occweb_dir
<<2025-Apr-16 20:12>> html = get_occweb_page(
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/occweb.py", line 328, in get_occweb_page
<<2025-Apr-16 20:12>> req = requests.get(url, auth=auth, timeout=timeout)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/api.py", line 73, in get
<<2025-Apr-16 20:12>> return request("get", url, params=params, **kwargs)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/api.py", line 59, in request
<<2025-Apr-16 20:12>> return session.request(method=method, url=url, **kwargs)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
<<2025-Apr-16 20:12>> resp = self.send(prep, **send_kwargs)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
<<2025-Apr-16 20:12>> r = adapter.send(request, **kwargs)
<<2025-Apr-16 20:12>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<<2025-Apr-16 20:12>> File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/adapters.py", line 688, in send
<<2025-Apr-16 20:12>> raise ConnectTimeout(e, request=request)
<<2025-Apr-16 20:12>> requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='occweb.cfa.harvard.edu', port=443): Max retries exceeded with url: /occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2025/MAR (Caused by ConnectTimeoutError(, 'Connection to occweb.cfa.harvard.edu timed out. (connect timeout=30)'))