Error processing message: 'detector_status' #4
Description
Some snews_pt testing today at the hackathon resulted in the following log messages on the Database container. Logging so I don't forget to look into this.
2025-06-10T13:01:37.816271 (re)Initializing Database Listener System for kafka://kafka.scimma.org/snews.experiments-firedrill
2025-06-17T09:03:45.550320 Error processing message: 'detector_status'
2025-06-17T13:49:28.350541 Error processing message: 'detector_status'
2025-06-17T14:34:19.635023 Error processing message: 'detector_status'
2025-06-18 07:50:04,118 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-06-18 07:50:04,120 INFO sqlalchemy.engine.Engine INSERT INTO coincidence_tier_archive (message_id, message_uuid, received_time_utc, detector_name, machine_time_utc, neutrino_time_utc, p_val, is_test, is_firedrill) VALUES (%(message_id)s, %(message_uuid)s, %(received_time_utc)s, %(detector_name)s, %(machine_time_utc)s, %(neutrino_time_utc)s, %(p_val)s, %(is_test)s, %(is_firedrill)s) RETURNING coincidence_tier_archive.id
2025-06-18 07:50:04,121 INFO sqlalchemy.engine.Engine [generated in 0.00019s] {'message_id': 'DS-20K_CoincidenceTier_None', 'message_uuid': '1a97a950-b712-4899-b28b-e04e11b9e1b3', 'received_time_utc': '2025-06-18 11:50:04.115164+00:00', 'detector_name': 'DS-20K', 'machine_time_utc': None, 'neutrino_time_utc': '2030-01-01T12:34:47.678999000Z', 'p_val': 0.98, 'is_test': 1, 'is_firedrill': 0}
2025-06-18T11:50:04.121490 Error processing message: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
[SQL: INSERT INTO coincidence_tier_archive (message_id, message_uuid, received_time_utc, detector_name, machine_time_utc, neutrino_time_utc, p_val, is_test, is_firedrill) VALUES (%(message_id)s, %(message_uuid)s, %(received_time_utc)s, %(detector_name)s, %(machine_time_utc)s, %(neutrino_time_utc)s, %(p_val)s, %(is_test)s, %(is_firedrill)s) RETURNING coincidence_tier_archive.id]
[parameters: {'message_id': 'DS-20K_CoincidenceTier_None', 'message_uuid': '1a97a950-b712-4899-b28b-e04e11b9e1b3', 'received_time_utc': '2025-06-18 11:50:04.115164+00:00', 'detector_name': 'DS-20K', 'machine_time_utc': None, 'neutrino_time_utc': '2030-01-01T12:34:47.678999000Z', 'p_val': 0.98, 'is_test': 1, 'is_firedrill': 0}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
2025-06-18 07:50:05,127 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-06-18 07:50:05,127 INFO sqlalchemy.engine.Engine INSERT INTO coincidence_tier_archive (message_id, message_uuid, received_time_utc, detector_name, machine_time_utc, neutrino_time_utc, p_val, is_test, is_firedrill) VALUES (%(message_id)s, %(message_uuid)s, %(received_time_utc)s, %(detector_name)s, %(machine_time_utc)s, %(neutrino_time_utc)s, %(p_val)s, %(is_test)s, %(is_firedrill)s) RETURNING coincidence_tier_archive.id
2025-06-18 07:50:05,127 INFO sqlalchemy.engine.Engine [cached since 1.007s ago] {'message_id': 'DUNE_CoincidenceTier_None', 'message_uuid': 'a9cf340e-a4e4-47c3-842d-fc3f135a013c', 'received_time_utc': '2025-06-18 11:50:05.122051+00:00', 'detector_name': 'DUNE', 'machine_time_utc': None, 'neutrino_time_utc': '2030-01-01T12:34:55.678999000Z', 'p_val': 0.98, 'is_test': 1, 'is_firedrill': 0}
2025-06-18 07:50:05,135 INFO sqlalchemy.engine.Engine COMMIT
2025-06-18 07:50:05,136 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-06-18 07:50:05,136 INFO sqlalchemy.engine.Engine INSERT INTO coincidence_tier_archive (message_id, message_uuid, received_time_utc, detector_name, machine_time_utc, neutrino_time_utc, p_val, is_test, is_firedrill) VALUES (%(message_id)s, %(message_uuid)s, %(received_time_utc)s, %(detector_name)s, %(machine_time_utc)s, %(neutrino_time_utc)s, %(p_val)s, %(is_test)s, %(is_firedrill)s) RETURNING coincidence_tier_archive.id
2025-06-18 07:50:05,136 INFO sqlalchemy.engine.Engine [cached since 1.015s ago] {'message_id': 'XENONnT_CoincidenceTier_None', 'message_uuid': '1a8e0350-2d5e-47fe-bd9f-340d4ddaf200', 'received_time_utc': '2025-06-18 11:50:05.135961+00:00', 'detector_name': 'XENONnT', 'machine_time_utc': None, 'neutrino_time_utc': '2030-01-01T12:34:45.678999000Z', 'p_val': 0.98, 'is_test': 1, 'is_firedrill': 0}
2025-06-18 07:50:05,136 INFO sqlalchemy.engine.Engine COMMIT
2025-06-18T17:06:51.054195 Error processing message: 'detector_status'
2025-06-18T17:09:51.072124 Error processing message: 'detector_status'
2025-06-18T17:10:53.078464 Error processing message: 'detector_status'
2025-06-18T17:23:48.149382 Error processing message: 'detector_status'