Prev   Index   Next

kadi: Thu Apr 17 08:33:17 2025

File: /proj/sot/ska/data/kadi/logs/daily.0/kadi_validate.log

Task Status AgeMaxAge Last Date
kadi NOT OK 0.091.5 Thu Apr 17 08:33:17 2025

Errors:

File contents:

2025-04-17 08:31:06,861 run_validators: Validating pitch
2025-04-17 08:31:06,862 get_command_sheet_exclude_intervals: Getting exclude times from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv&id=19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ&gid=1681877928
Traceback (most recent call last):
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/ska_helpers/utils.py", line 199, in __getitem__
return super().__getitem__(item)
^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'version'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 445, in request
self.endheaders()
File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 1331, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 1091, in _send_output
self.send(msg)
File "/proj/sot/ska3/flight/lib/python3.12/http/client.py", line 1035, in send
self.connect()
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 276, in connect
self.sock = self._new_conn()
^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connection.py", line 207, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to telemetry.cfa.harvard.edu timed out. (connect timeout=None)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='telemetry.cfa.harvard.edu', port=80): Max retries exceeded with url: /maude/mrest/sysinfo.json (Caused by ConnectTimeoutError(, 'Connection to telemetry.cfa.harvard.edu timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/proj/sot/ska3/flight/bin/kadi_validate_states", line 10, in
sys.exit(main())
^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/scripts/validate_states.py", line 210, in main
validators = run_validators(opt.stop, opt.days, opt.states, opt.no_exclude)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/scripts/validate_states.py", line 98, in run_validators
validator: Validate = cls(stop=stop, days=days, no_exclude=no_exclude)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/validate.py", line 157, in __init__
self.add_exclude_intervals()
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/validate.py", line 561, in add_exclude_intervals
self.exclude_ofp_intervals_except(states_expected=["NRML", "SAFE"])
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/kadi/commands/validate.py", line 288, in exclude_ofp_intervals_except
ofp_states = get_ofp_states(self.start, self.stop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/cheta/utils.py", line 508, in get_ofp_states
tlm = get_telem_table([msid], start - 30 * u.s, stop + 30 * u.s)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/cheta/utils.py", line 469, in get_telem_table
msidset = fetch_module.MSIDset(msids, start - time_pad, stop + time_pad)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/cheta/fetch.py", line 1614, in __init__
new_msids.extend(msid_glob(msid)[0])
^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/cheta/fetch.py", line 464, in msid_glob
ms, MS = _msid_glob(msid, source)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/cheta/fetch.py", line 489, in _msid_glob
source_msids = data_source.get_msids(source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/cheta/fetch.py", line 217, in get_msids
out = list(maude.MSIDS.keys())
^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/maude/maude.py", line 127, in load_msids
msids = {msid: index for index, msid in MSID_FROM_INDEX.items()}
^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/maude/maude.py", line 114, in load_msid_index
with open(get_data_filename("MSIDIDX", "txt"), "r") as fh:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/maude/maude.py", line 89, in get_data_filename
version = MAUDE_SYSINFO["version"]
~~~~~~~~~~~~~^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/ska_helpers/utils.py", line 201, in __getitem__
self._load()
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/maude/maude.py", line 151, in get_maude_sysinfo
r = get_request(url, user, password)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/maude/maude.py", line 516, in get_request
req = func(url, user, password, gzip=gzip, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/maude/maude.py", line 525, in _get_request
req = requests.get(url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.12/site-packages/requests/adapters.py", line 688, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='telemetry.cfa.harvard.edu', port=80): Max retries exceeded with url: /maude/mrest/sysinfo.json (Caused by ConnectTimeoutError(, 'Connection to telemetry.cfa.harvard.edu timed out. (connect timeout=None)'))