docs built + file introspection

Author: ArturU043

docs/_build/doctrees/environment.pickle

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: ca3626de684797951ef03443c45ac66f
+tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/_build/doctrees/file_introspecting.doctree

@@ -0,0 +1,167 @@
+# Remote File Introspecting
+
+The `get_structure()` function allows users to query and inspect the internal structure of datasets available through ServiceX. This is useful for determining which branches exist in a given dataset before running a full transformation with the correct branch labelling and typing.
+
+It is useful for any lightweight exploration when only metadata or structure information is required without fetching event-level data.
+
+---
+
+##  Overview
+
+The function internally issues a ServiceX request, using python function backend, for the specified dataset(s) and returns a simplified summary of the file structure, such as branches and types in a string formatted for readability.
+
+It accepts both programmatic and command-line usage with parametric return types.
+
+---
+
+## Function
+
+```python
+get_structure(datasets, array_out=False, **kwargs)
+```
+
+**Parameters:**
+
+- `datasets` (`dict`, `str`, or `list[str]`): One or more datasets to inspect. Made for Rucio DIDs. If a dictionary is used, keys will be used as labels for each dataset in the output string.
+- `array_out` (`bool`): If True, empty awkward arrays are reconstructed from the structure information. The function will return a dictionary of ak.Array.type objects. This allows for programmatic access to the dataset structure which can be further manipulated.
+- `**kwargs`: Additional arguments forwarded to the helper function `print_structure_from_str`, such as `filter_branch` to apply a filter to displayed branches, `do_print` to print the output during the function call, or `save_to_txt` to save the output to `samples_structure.txt`.
+
+**Returns:**
+- `str`: The formatted file structure string.
+- `None`: If `do_print` or `save_to_txt` is `True`, the function will print or save the output to a file.
+- `dict`: keys are sample names and values are `ak.Array.type` objects with the same dataset structure.
+
+---
+
+## Command-Line Usage
+
+The function is also available as a CLI tool:
+
+```bash
+$ servicex-get-structure "scope:dataset-rucio-id" --filter_branch "el_"
+```
+
+This dumps to the shell a summary of the structure of the dataset, filtered to branches that contain `"el_"` in their names.
+
+```bash
+$ servicex-get-structure "scope:dataset-rucio-id1" "scope:dataset-rucio-id2" --filter_branch "el_"
+```
+
+This will output a combined summary of both datasets with the same filter.
+
+---
+
+### Practical Output Example 
+
+Command:
+
+```bash
+$ servicex-get-structure  user.mtost:user.mtost.all.Mar11 --filter-branch el_pt 
+```
+
+Output on shell: 
+
+```bash
+File structure of all samples with branch filter 'el_pt':
+
+---------------------------
+📁 Sample: user.mtost:user.mtost.all.Mar11
+---------------------------
+
+🌳 Tree: EventLoop_FileExecuted
+   ├── Branches:
+
+🌳 Tree: EventLoop_JobStats
+   ├── Branches:
+
+🌳 Tree: reco
+   ├── Branches:
+   │   ├── el_pt_NOSYS ; dtype: AsJagged(AsDtype('>f4'), header_bytes=10)
+   │   ├── el_pt_EG_RESOLUTION_ALL__1down ; dtype: AsJagged(AsDtype('>f4'), header_bytes=10)
+   │   ├── el_pt_EG_RESOLUTION_ALL__1up ; dtype: AsJagged(AsDtype('>f4'), header_bytes=10)
+   │   ├── el_pt_EG_SCALE_ALL__1down ; dtype: AsJagged(AsDtype('>f4'), header_bytes=10)
+   │   ├── el_pt_EG_SCALE_ALL__1up ; dtype: AsJagged(AsDtype('>f4'), header_bytes=10)
+```
+
+The output lists all trees and branch names matching the specified filter pattern for each requested dataset. 
+It shows the branch data type information as interpreted by `uproot`. This includes the vector nesting level (jagged arrays) and the base type (e.g., f4 for 32-bit floats).
+
+
+#### JSON input 
+
+A json file can be used as input to simplify the command for multiple samples.
+
+```bash
+$ servicex-get-structure "path/to/datasets.jsosn" 
+```
+
+With `datasets.json` containing:
+
+```
+{
+  "Signal": "mc23_13TeV:signal-dataset-rucio-id",
+  "Background W+jets": "mc23_13TeV:background-dataset-rucio-id1",
+  "Background Z+jets": "mc23_13TeV:background-dataset-rucio-id2",
+  "Background Drell-Yan": "mc23_13TeV:background-dataset-rucio-id3",
+}
+```
+
+---
+
+## Programmatic Example
+
+Similarly to the CLI functionality, the output string containing the dataset structure can be retrieved such as:
+
+```python
+from servicex_analysis_utils import get_structure
+
+# Retrieve structure of a specific dataset
+file_structure=get_structure("mc23_13TeV:some-dataset-rucio-id")
+```
+
+### Other options
+
+With `do_print` and `save_to_txt`, the dataset-structure string can instead be routed to std_out or to a text file in the running path.
+
+```python
+from servicex_analysis_utils import get_structure
+
+# Directly dump structure to std_out
+get_structure("mc23_13TeV:some-dataset-rucio-id", do_print=True)
+# Save to samples_summaty.txt
+get_structure("mc23_13TeV:some-dataset-rucio-id", save_to_txt=True)
+```
+
+
+#### Return awkward array type
+
+
+If `array_out` is set to `True` the function reconstructs dummy arrays with the correct structre and return their `Awkward.Array.type` object. 
+
+```python
+from servicex_analysis_utils import get_structure
+
+DS = {"sample1": "user.mtost:user.mtost.all.Mar11"}
+ak_type = get_structure(DS, array_out=True)
+
+rec = ak_type["sample1"].content #get RecordType
+
+# Find index of reco tree and runNumber branch
+reco_idx = rec.fields.index("reco")
+branch_idx = rec.contents[reco_idx].fields.index("runNumber")
+
+print("Type for branch 'runNumber':", rec.contents[reco_idx].contents[branch_idx])
+```
+Output:
+
+```bash
+Type for branch 'runNumber': var * int64
+```
+
+---
+
+## Notes
+
+- The function does not retrieve event data — only structure/metadata.
+- CLI output is printed directly to stdout but can be routed to a file with ` > structure_summary.txt` 
+- Many types will show as None or unknown when they are not interpretable by the uproot or fail to be reconstructed to ak.arrays

docs/_build/doctrees/index.doctree

@@ -0,0 +1,39 @@
+# ServiceX Analysis Utilities
+
+This package provides tools interacting with [ServiceX](https://github.com/ssl-hep/ServiceX_frontend), a data extraction, transformation and delivery system built for ATLAS and CMS analyses on large datasets. The Analysis Utils package offers helper functions that streamline the usage of ServiceX and simplify its integration on workflows. But it also contains specific use case tools that benefit from the service. 
+
+---
+
+## Installation
+
+Install the package from PyPI:
+
+```bash
+pip install servicex-analysis-utils
+```
+More information can be found in [Instalation and requirements](installation.md)
+
+## Documentation Contents
+
+```{toctree}
+:maxdepth: 2
+:caption: Documentation Contents:
+
+installation
+materialization
+file_introspecting
+```
+
+---
+
+## Utility Functions
+
+### `to_awk()`
+Load an Awkward Array from ServiceX output easily.
+
+ See detailed usage here: [Materiazlization documentation](materialization.md)
+
+### `get_structure()`
+Create and send ServiceX requests to retrieve file structures with a CLI implementation.
+
+ See detailed usage here: [File introspection documentation](file_introspecting.md)

docs/_build/doctrees/installation.doctree

@@ -0,0 +1,42 @@
+# Installation
+
+This section provides instructions for installing the ServiceX Analysis Utilities package.
+
+## Prerequisites
+
+Before installing, ensure the following requirements are satisfied:
+
+- Python 3.9 or higher is installed.
+- `pip` is updated to the latest version (`pip install --upgrade pip`).
+- Access to a ServiceX endpoit is granted
+- A valid `servicex.yaml` configuration file is on your local machine.
+
+For instructions on setting up ServiceX, refer to the [ServiceX Installation Guide](https://servicex-frontend.readthedocs.io/en/stable/connect_servicex.html).
+
+## Installation from PyPI
+
+The package is available on PyPI and can be installed via:
+
+```bash
+pip install servicex-analysis-utils
+```
+
+## Installation from Source
+
+Alternatively, the package can be installed from the GitHub repository:
+
+```bash
+git clone https://github.com/ssl-hep/ServiceX_analysis_utils.git
+cd ServiceX_analysis_utils
+pip install .
+```
+
+## Verifying the Installation
+
+After installation, you can verify that the package is accessible by running:
+
+```bash
+python -c "import servicex_analysis_utils"
+```
+
+No output indicates a successful installation.

docs/_build/doctrees/materialization.doctree

@@ -0,0 +1,136 @@
+# Materialization of delivered data
+
+The `to_awk()` function provides a streamlined method to materialize the output of a ServiceX `deliver()` call into Awkward Arrays, Dask arrays, or iterators.
+
+This simplifies workflows by allowing easy manipulation of the retrieved data in various analysis pipelines like in the examples below.
+
+---
+
+## Overview
+
+The `to_awk()` function loads data from the deliver output dictionary, supporting both ROOT (`.root`) and Parquet (`.parquet` or `.pq`) file formats.
+
+It provides flexible options for:
+
+- Direct loading into Awkward Arrays.
+- Lazy loading using Dask for scalable operations.
+- Returning iterator objects for manual control over file streaming.
+
+## Function 
+
+```python
+to_awk(deliver_dict, dask=False, iterator=False, **kwargs)
+```
+
+**Parameters:**
+
+- `deliver_dict` (dict): Dictionary returned by `servicex.deliver()`. Keys are sample names, values are file paths or URLs.
+- `dask` (bool, optional): If True, loads files lazily using Dask. Default is False.
+- `iterator` (bool, optional): If True and not using Dask, returns iterators instead of materialized arrays. Default is False.
+- `**kwargs`: Additional keyword arguments passed to `uproot.dask`, `uproot.iterate`, dak.from_parquet, or `awkward.from_parquet`.
+
+**Returns:**
+
+- `dict`: A dictionary where keys are sample names and values are either Awkward Arrays, Dask Arrays, or iterators. It keeps the same structure as the `deliver` output dict. 
+
+---
+
+## Usage Examples
+
+### Simple Materialization
+
+Load ServiceX deliver results directly into Awkward Arrays:
+
+```python
+from servicex_analysis_utils import to_awk
+from servicex import query, dataset, deliver
+
+spec = {
+    "Sample": [
+        {
+            "Name": "simple_transform",
+            "Dataset": dataset.FileList(
+                ["root://eospublic.cern.ch//eos/opendata/atlas/rucio/data16_13TeV/DAOD_PHYSLITE.37019878._000001.pool.root.1"]  # noqa: E501
+            ),
+            "Query": query.FuncADL_Uproot()
+            .FromTree("CollectionTree")
+            .Select(lambda e: {"el_pt": e["AnalysisElectronsAuxDyn.pt"]}), 
+        }
+    ]
+}
+
+arrays=to_awk(deliver(spec))
+```
+
+### Lazy Loading with Dask
+
+Load results lazily for large datasets using Dask task graphs. Enables parallel execution across multiple workers.
+
+```python
+import dask_awkward as dak
+
+dask_arrays = to_awk(deliver(spec), dask=True)
+el_pt_array = dask_arrays["simple_transform"]["el_pt"]
+mean_el_pt = dak.mean(el_pt_array).compute()
+```
+
+### Using Iterators
+
+Return iterators instead of materialized arrays to avoid loading too much data into memory. Requires `dask=False` (default). Example with loading 10,000 events per chunk:
+
+```python
+iterables = to_awk(deliver(spec), iterator=True, step_size=10000)
+```
+
+You can then manually loop over the data chunks:
+
+```python
+for chunk in iterables['simple_transform']:
+    # process small chunk (~10k events)
+    analyse(chunk) #some function for el_pt
+```
+
+All events can also be loaded by using: 
+
+```python
+import awkward as ak
+arrays= ak.concatenate(list[iterables['simple_transform']])
+```
+
+---
+
+
+## Multiple samples
+
+ServiceX queries allow multiple sample transformations. The `to_awk` allows a straightforward manipulation of such requests. This allows seamless integration with analysis frameworks with multiple samples being manipulated separately after being passing the same transformation using `deliver()`.
+
+```python
+from servicex_analysis_utils import to_awk
+import awkward as ak
+
+# Given a ServiceX deliver return
+deliver_result = {
+    "Signal": ["path/to/signal_file1.root", "path/to/signal_file2.root"],
+    "Background": ["path/to/background_file.root"]
+}
+
+arrays = to_awk(deliver_result)
+
+signal_el_pt = arrays["Signal"]["el_pt"]
+background_el_pt = arrays["Background"]["el_pt"]
+
+mean_signal = ak.mean(signal_el_pt)
+mean_background = ak.mean(background_el_pt)
+
+print(f"Mean electron pT (Signal): {mean_signal:.2f} GeV")
+print(f"Mean electron pT (Background): {mean_background:.2f} GeV")
+```
+
+
+## Notes
+
+- **Multiple samples:** For transformations delivering multiple samples the dask and iterators are applied homegeneously to all.
+- **Error Handling:** In case of loading errors, the affected sample will have `None` as its value in the returned dictionary.
+- **Supported Formats:** A custom dict (non servicex) can be inputed but the paths must point be either ROOT or Parquet format.
+- **Branch Filtering, others:** Additional `**kwargs` allow specifying branch selections or other loading options supported by `uproot`, `awkward` and `dask_awkward`.
+