test failed in CI: `test_self_stat_collection_count`

[jgallagher]

This test failed on a CI run on #8627:

https://github.com/oxidecomputer/omicron/pull/8627/checks?check_run_id=46208988376

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01K0CYCJ10M4HYQ5TJDRNFFTRX/GkU4qFcnvjiE6MOKLpiGpI1Yu5sJbshKlC1KNbKPVprBXJyB/01K0CYDK2VDM6GQS0Z5CQYYQYW

Excerpt from the log showing the failure:

7923	2025-07-17T21:06:32.078Z	  stdout ───
7924	2025-07-17T21:06:32.078Z	
7925	2025-07-17T21:06:32.078Z	    running 1 test
7926	2025-07-17T21:06:32.078Z	    test agent::tests::test_self_stat_collection_count ... FAILED
7927	2025-07-17T21:06:32.078Z	
7928	2025-07-17T21:06:32.078Z	    failures:
7929	2025-07-17T21:06:32.078Z	
7930	2025-07-17T21:06:32.078Z	    failures:
7931	2025-07-17T21:06:32.078Z	        agent::tests::test_self_stat_collection_count
7932	2025-07-17T21:06:32.078Z	
7933	2025-07-17T21:06:32.078Z	    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 7 filtered out; finished in 1.86s
7934	2025-07-17T21:06:32.078Z	    
7935	2025-07-17T21:06:32.078Z	  stderr ───
7936	2025-07-17T21:06:32.079Z	    log file: /var/tmp/omicron_tmp/oximeter_collector-ce92baab1eaac757-test_self_stat_collection_count.4000.0.log
7937	2025-07-17T21:06:32.079Z	    note: configured to log to "/var/tmp/omicron_tmp/oximeter_collector-ce92baab1eaac757-test_self_stat_collection_count.4000.0.log"
7938	2025-07-17T21:06:32.079Z	
7939	2025-07-17T21:06:32.079Z	    thread 'agent::tests::test_self_stat_collection_count' panicked at oximeter/collector/src/agent.rs:844:9:
7940	2025-07-17T21:06:32.079Z	    assertion failed: stats.failed_collections.is_empty()

We previously had this test flaking in #4657 but it looks like this is a different assert failing, so figured it was worth a new issue rather than reopening.

[jgallagher]

@bnaecker I wonder if this is related to #8595? The logs in this test failure look like oximeter is in a bad / wedged state; e.g.,

2025-07-17T21:06:28.359Z	ERRO	test_self_stat_collection_count (collection-task): timer-based collection request queue is full! This may indicate that the producer has a sampling interval that is too fast for the amount of data it generates
    collector_id = 265a9f85-08cc-455a-a86a-e7b9abe08988
    collector_ip = ::1
    interval = 1s
    producer_id = 3f278ece-51f7-4f0a-a6c7-b1ed38be6a39

2025-07-17T21:06:28.381Z	ERRO	test_self_stat_collection_count (collection-task): failed to send value from the collection task to channel for cluster: SendError { .. }
    collector_id = 265a9f85-08cc-455a-a86a-e7b9abe08988
    collector_ip = ::1
    producer_id = 3f278ece-51f7-4f0a-a6c7-b1ed38be6a39

[bnaecker]

It could be! I have no idea what's going in #8595, so your guess is good as mine. Nothing has changed in that code in months, so I wonder if this is a latent bug that some other update revealed.