FileReadError on a local file, but only when prefetching

[makortel]

I ended up having a local file (actually Run2024 PromptReco MiniAOD file of #49202 copied to have the intended branch splitting structure) that works fine with PoolSource out of the box, but if prefetching is enabled with

process.add_(cms.Service("SiteLocalConfigService",
    overridePrefetching = cms.untracked.bool(True)
))

a reading job fails with

24-Oct-2025 20:51:28 CEST  Successfully opened file file:initialconversion_ttree_miniprod_lzma4.root
R__unzipLZMA: error 9 in lzma_code
24-Oct-2025 20:51:28 CEST  Closed file file:initialconversion_ttree_miniprod_lzma4.root
----- Begin Fatal Exception 24-Oct-2025 20:51:28 CEST-----------------------
An exception of category 'FileReadError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Reading branch EventAuxiliary entry 417
   Additional Info:
      [a] Fatal Root Error: @SUB=TBasket::ReadBasketBuffers
fNbytes = 17844, fKeylen = 83, fObjlen = 101708, noutot = 0, nout=0, nin=17761, nbuf=101708

----- End Fatal Exception -------------------------------------------------

With the application-only storage hint (local files should by default behave as storage-only hint) the reading job succeeds.

I did this test with 15_1_0_pre6_RNTUPLE build.

Given the number of issues we have seen with R__unzipLZMA: error 9 in lzma_code, that we have interpreted as corrupted files, a deeper investigation into this behavior could be worth it.

[makortel]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

Running the same job I occasionally get an exception for different entry (with different numbers)

R__unzipLZMA: error 9 in lzma_code
24-Oct-2025 20:57:45 CEST  Closed file file:initialconversion_ttree_miniprod_lzma4.root
----- Begin Fatal Exception 24-Oct-2025 20:57:45 CEST-----------------------
An exception of category 'FileReadError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Reading branch EventAuxiliary entry 278
   Additional Info:
      [a] Fatal Root Error: @SUB=TBasket::ReadBasketBuffers
fNbytes = 6002, fKeylen = 105, fObjlen = 105296, noutot = 0, nout=0, nin=5897, nbuf=105296

----- End Fatal Exception -------------------------------------------------

(actually the entry 278 repeats much more often than 417)

[makortel]

Comparing IO traces of a no-prefetch and prefetch jobs (where the prefetch job exception message showed the entry 278) the no-prefetch job issues the following singular reads (the first two numbers are the begin and end offsets of the reads)

...
3790389853 3790389973 TBasket name: ThinnedAssociationsHelper  title: MetaData
19130776 19133936 TBasket name: EventAuxiliary    title: Events
#
28710397 28713493 TBasket name: EventAuxiliary    title: Events
38428459 38431635 TBasket name: EventAuxiliary    title: Events
48170335 48173579 TBasket name: EventAuxiliary    title: Events
57935669 57938685 TBasket name: EventAuxiliary    title: Events
63063929 63065561 TBasket name: EventAuxiliary    title: Events
#
101937851 101955375 TBasket name: EventAuxiliary    title: Events
...

whereas the prefetch job does

...
# singular read
3790389853 3790389973 TBasket name: ThinnedAssociationsHelper  title: MetaData
# singular read
19130776 19133936 TBasket name: EventAuxiliary    title: Events
# singular read
1322192660 1322198086 TBasket name: recoJPTJets_slimmedJPTJets__RECO.obj  title: Events
# vector read of many elements
101937851 101955375 TBasket name: EventAuxiliary    title: Events
...

This difference of 5 baskets of EventAuxiliary vs. 1 basket of recoJPTJets_slimmedJPTJets__RECO.obj was the most significant difference between the read patterns that seemed relevant.

The read offset jumping up by ~1.1 GB for ~5 kB read of unrelated to EventAuxiliary right after the first singular read of EventAuxiliary branch seems highly suspicious.

Next step would be to

• Check if the failure occurs in a non-RNTUPLE build (or if ROOT 6.32 vs 6.36 would make a difference)
• Correlate the read of recoJPTJets_slimmedJPTJets__RECO.obj to higher-level activities
• Investigate with a ROOT debug build

[makortel]

The impact of the prefetching configuration flag is this

cmssw/IOPool/Input/src/RootTree.cc

Lines 231 to 235 in 3f4295b

	void RootTree::setCacheSize(unsigned int cacheSize) {
	cacheSize_ = cacheSize;
	treeCache_ = createCacheWithSize(cacheSize);
	if (treeCache_)
	treeCache_->SetEnablePrefetching(enablePrefetching_);

which is called from the RootTree constructor.