Report ingestSummary even if AnnData extraction fails (SCP-5973) #2243
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## development #2243 +/- ##
===============================================
+ Coverage 70.77% 70.83% +0.06%
===============================================
Files 331 331
Lines 28306 28311 +5
Branches 2476 2476
===============================================
+ Hits 20033 20055 +22
+ Misses 8130 8113 -17
Partials 143 143
🚀 New features to boost your workflow:
|
bistline
added
the
build failure: false positive
eweitz
approved these changes
Apr 17, 2025
| @@ -1115,10 +1115,20 @@ def extracted_raw_counts?(job) | |||
| extract_params.include?('raw_counts') | |||
| end | |||
|
|
|||
| # determine if this job qualifies for sending an ingestSummary event | |||
| # will return false if summary exists, this is a DE job, or | |||
| # a successful AnnData extract (meaning downstream jobs are running) | |||
There was a problem hiding this comment.
Helpful note! Looking only at the clause in the code, I was confused.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BACKGROUND & CHANGES
This fixes a corner case in Mixpanel reporting where AnnData files that fail during the initial extraction phase do not report an
ingestSummaryevent. These events are the main indication SCP has in Mixpanel for AnnData ingest as only one event is ever sent per AnnData file. The lack of these extraction events means we are underreporting on AnnData ingest attempts. Now, if the initial extraction fails and is not going to be retried due to an OOM exception, the summary is sent.Also, this fixes a regression introduced in #2242 where metadata properties in AnnData files that have names ending in values that match ontology-based properties (e.g.
author_cell_typematching tocell_type) incorrectly attempt to validate that column. Now, exact name matches are enforced rather than relying onendsWith().MANUAL TESTS
raw_location: does_not_existin the Expression tab to ensure the extraction failsingestSummaryeventjobStatus: failedandnumFilesExtracted: 0