PromptReco MiniAOD has unexpected splitting structure

[makortel]

For other reasons I was inspecting a MiniAOD file from Run2024I EGamma0 PromptReco dataset, and noticed the Events TTree contained total of 3.5k branches, whereas a MiniAOD file processed through a runTheMatrix workflow (with AOD file from the same era and PD as an input) has ~500 branches.

The PromptReco MiniAOD file looks (e.g. with edmFileUtil -b) like being fully split.

The PoolOutputModule in a configuration generated with cmsDriver --process PAT --eventcontent MINIAOD --datatier MINIAOD ... sets the splitting as

        overrideBranchesSplitLevel = cms.untracked.VPSet(
            cms.untracked.PSet(
                branch = cms.untracked.string('patPackedCandidates_packedPFCandidates__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('recoGenParticles_prunedGenParticles__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('patTriggerObjectStandAlones_slimmedPatTrigger__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('patPackedGenParticles_packedGenParticles__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('patJets_slimmedJets__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('recoVertexs_offlineSlimmedPrimaryVertices__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('recoVertexs_offlineSlimmedPrimaryVerticesWithBS__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('recoCaloClusters_reducedEgamma_reducedESClusters_*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('EcalRecHitsSorted_reducedEgamma_reducedEBRecHits_*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('EcalRecHitsSorted_reducedEgamma_reducedEERecHits_*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('recoGenJets_slimmedGenJets__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('patJets_slimmedJetsPuppi__*'),
                splitLevel = cms.untracked.int32(99)
            ),
            cms.untracked.PSet(
                branch = cms.untracked.string('EcalRecHitsSorted_reducedEgamma_reducedESRecHits_*'),
                splitLevel = cms.untracked.int32(99)
            )
        ),
        overrideInputFileSplitLevels = cms.untracked.bool(True),
        splitLevel = cms.untracked.int32(0)

this list of branches to be split appears to originate from

cmssw/PhysicsTools/PatAlgos/python/slimming/MicroEventContent_cff.py

Lines 209 to 223 in bcba1a3

	MiniAODOverrideBranchesSplitLevel = cms.untracked.VPSet( [
	cms.untracked.PSet(branch = cms.untracked.string("patPackedCandidates_packedPFCandidates__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("recoGenParticles_prunedGenParticles__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("patTriggerObjectStandAlones_slimmedPatTrigger__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("patPackedGenParticles_packedGenParticles__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("patJets_slimmedJets__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("recoVertexs_offlineSlimmedPrimaryVertices__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("recoVertexs_offlineSlimmedPrimaryVerticesWithBS__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("recoCaloClusters_reducedEgamma_reducedESClusters_*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("EcalRecHitsSorted_reducedEgamma_reducedEBRecHits_*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("EcalRecHitsSorted_reducedEgamma_reducedEERecHits_*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("recoGenJets_slimmedGenJets__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("patJets_slimmedJetsPuppi__*"),splitLevel=cms.untracked.int32(99)),
	cms.untracked.PSet(branch = cms.untracked.string("EcalRecHitsSorted_reducedEgamma_reducedESRecHits_*"),splitLevel=cms.untracked.int32(99)),
	])

that is used in

cmssw/PhysicsTools/PatAlgos/python/slimming/miniAOD_tools.py

Lines 670 to 677 in bcba1a3

	def miniAOD_customizeOutput(out):
	from PhysicsTools.PatAlgos.slimming.MicroEventContent_cff import MiniAODOverrideBranchesSplitLevel
	out.overrideBranchesSplitLevel = MiniAODOverrideBranchesSplitLevel
	out.splitLevel = cms.untracked.int32(0)
	out.dropMetaData = cms.untracked.string('ALL')
	out.fastCloning= cms.untracked.bool(False)
	out.overrideInputFileSplitLevels = cms.untracked.bool(True)
	out.compressionAlgorithm = cms.untracked.string('LZMA')

Maybe Tier0 configuration is somehow not picking this customization up?

[makortel]

assign xpog, operations

[cmsbuild]

New categories assigned: xpog,operations

@battibass,@davidlange6,@fabiocos,@ftenchini,@ftorrresd,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

The miniAOD_customizeOutput() is called from

cmssw/Configuration/Applications/python/ConfigBuilder.py

Lines 715 to 718 in bcba1a3

	if "MINIAOD" in streamType:
	## we should definitely get rid of this customization by now
	from PhysicsTools.PatAlgos.slimming.miniAOD_tools import miniAOD_customizeOutput
	miniAOD_customizeOutput(output)

[makortel]

Inspecting logs from a random PromptReco job (PromptReco_Run398336_JetMET1, CMSSW_15_0_15_patch4) shows the MiniAOD output module configuration to be

process.write_MINIAOD = cms.OutputModule("PoolOutputModule",
    compressionAlgorithm = cms.untracked.string('LZMA'),
    compressionLevel = cms.untracked.int32(4),
    dataset = cms.untracked.PSet(
        dataTier = cms.untracked.string('MINIAOD'),
        filterName = cms.untracked.string('')
    ),
    eventAutoFlushCompressedSize = cms.untracked.int32(-900),
    fileName = cms.untracked.string('write_MINIAOD.root'),
    logicalFileName = cms.untracked.string('/store/unmerged/data/Run2025G/JetMET1/MINIAOD/PromptReco-v1/000/398/336/00000/write_MINIAOD.root'),
    outputCommands = cms.untracked.vstring(<cut long list>),
)

i.e. the splitting customization is clearly not applied.

[makortel]

Processing the PromptReco MiniAOD file through a simple copy job that applies the splitting customization for the OutputModule results in a file that has ~500 branches (instead of 3.5k) and is ~4 % smaller than the original PromtpReco file.

[makortel]

From core perspective not fully defining the MINIAOD output behavior through

cmssw/Configuration/EventContent/python/EventContent_cff.py

Lines 950 to 955 in bcba1a3

	MINIAODEventContent= cms.PSet(
	outputCommands = cms.untracked.vstring('drop *'),
	eventAutoFlushCompressedSize=cms.untracked.int32(-900),
	compressionAlgorithm=cms.untracked.string("LZMA"),
	compressionLevel=cms.untracked.int32(4)
	)

(that is done for everything else) seems problematic in general.

[srimanob]

Is this because when Tier-0 make the config from
https://github.com/cms-sw/cmssw/blob/master/Configuration/DataProcessing/python/Reco.py
https://github.com/cms-sw/cmssw/blob/master/Configuration/DataProcessing/python/Impl/ppEra_Run3_2025.py
The miniaod custmization does not pass through?

[srimanob]

Ping @LinaresToine @jeyserma @Viphava280444 @drkovalskyi
(Please feel free to move to proper cmsTalk, MM)

Note that I try the following command in /Configuration/DataProcessing to see Tier-0 config
python3 RunPromptReco.py --scenario=ppEra_Run3_2025 --reco --aod --miniaod --nanoaod --global-tag 150X_dataRun3_Prompt_v1 --lfn =/store/whatever
Look quickly on pkl file, it seems splitting is there.

[LinaresToine]

Hello @srimanob are you saying that the output from running the job interactively is different than the output of regular data taking?

[srimanob]

Hi @LinaresToine
From @makortel comment here
File size from PromptReco is 4% larger than running interactively. So I try to convert Tier-0 config to config file, for comparison.