Adding modules for doc config and materialization

ArturU043 · ArturU043 · commit ff826de9f68d · 2025-04-28T19:11:01.000+02:00
diff --git a/docs/Makefile b/docs/Makefile
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/docs/conf.py b/docs/conf.py
@@ -0,0 +1,45 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+import servicex_analysis_utils
+
+project = "ServiceX Analysis Utils"
+copyright = "2025 Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP)"  # NOQA 501
+author = "Artur Cordeiro Oudot Choi & the ServiceX team"
+release = servicex_analysis_utils.__version__
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = [
+    "sphinx.ext.napoleon",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.viewcode",
+    "myst_parser",
+    "sphinx_copybutton",
+]
+
+source_suffix = {
+    ".rst": "restructuredtext",
+    ".md": "markdown",
+}
+
+templates_path = ["_templates"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+
+html_theme = "furo"
+html_static_path = ["_static"]
+
+copybutton_prompt_text = r">>> |\.\.\. |\$ "
+copybutton_prompt_is_regexp = True
+copybutton_here_doc_delimiter = "EOF"
diff --git a/docs/index.md b/docs/index.md
@@ -0,0 +1,44 @@
+# ServiceX Analysis Utilities
+
+This package provides tools interacting with [ServiceX](https://github.com/ssl-hep/ServiceX_frontend), a data extraction, transformation and delivery system built for ATLAS and CMS analyses on large datasets. The Analysis Utils package offers helper functions that streamline the usage of ServiceX and simplify its integration on workflows. But it also contains specific use case tools that benefit from the service. 
+
+It offers functions for:
+- Materializing ´servicex.deliver()´ results 
+- Retrieving and displaying file structures from datasets remotely with branch filtering. 
+
+---
+
+## Installation
+
+Install the package from PyPI:
+
+```bash
+pip install servicex-analysis-utils
+```
+More information can be found in [Instalation and requirements](installation.md)
+
+## Documentation Contents
+
+```{toctree}
+:maxdepth: 2
+:caption: Documentation Contents:
+
+installation
+usage
+materialization
+file_introspecting 
+```
+
+---
+
+## Utility Functions
+
+### `to_awk()`
+Load an Awkward Array from ServiceX output easily.
+
+ See detailed usage here: [Materiazlization documentation](materialization.md)
+
+### `get_structure()`
+Create and send ServiceX requests to retrieve file structures with a CLI implementation.
+
+ See detailed usage here: [File introspection documentation](file_introspecting.md)
diff --git a/docs/installation.md b/docs/installation.md
@@ -0,0 +1,42 @@
+# Installation
+
+This section provides instructions for installing the ServiceX Analysis Utilities package.
+
+## Prerequisites
+
+Before installing, ensure the following requirements are satisfied:
+
+- Python 3.9 or higher is installed.
+- `pip` is updated to the latest version (`pip install --upgrade pip`).
+- Access to a ServiceX endpoit is granted
+- A valid `servicex.yaml` configuration file is on your local machine.
+
+For instructions on setting up ServiceX, refer to the [ServiceX Installation Guide](https://servicex-frontend.readthedocs.io/en/stable/connect_servicex.html).
+
+## Installation from PyPI
+
+The package is available on PyPI and can be installed via:
+
+```bash
+pip install servicex-analysis-utils
+```
+
+## Installation from Source
+
+Alternatively, the package can be installed from the GitHub repository:
+
+```bash
+git clone https://github.com/ssl-hep/ServiceX_analysis_utils.git
+cd ServiceX_analysis_utils
+pip install .
+```
+
+## Verifying the Installation
+
+After installation, you can verify that the package is accessible by running:
+
+```bash
+python -c "import servicex_analysis_utils"
+```
+
+No output indicates a successful installation.
diff --git a/docs/make.bat b/docs/make.bat
@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd
diff --git a/docs/materialization.md b/docs/materialization.md
@@ -0,0 +1,137 @@
+# Materialization of ServiceX Deliver Results
+
+The `to_awk()` function provides a streamlined method to materialize the output of a ServiceX `deliver()` call into Awkward Arrays, Dask arrays, or iterators.
+
+This simplifies workflows by allowing easy manipulation of the retrieved data in various analysis pipelines like in the examples below.
+
+---
+
+## Functionality Overview
+
+The `to_awk()` function loads data from the deliver output dictionary, supporting both ROOT (`.root`) and Parquet (`.parquet` or `.pq`) file formats.
+
+It provides flexible options for:
+
+- Direct loading into Awkward Arrays.
+- Lazy loading using Dask for scalable operations.
+- Returning iterator objects for manual control over file streaming.
+
+## Function Signature
+
+```python
+to_awk(deliver_dict, dask=False, iterator=False, **kwargs)
+```
+
+**Parameters:**
+
+- `deliver_dict` (dict): Dictionary returned by `servicex.deliver()`. Keys are sample names, values are file paths or URLs.
+- `dask` (bool, optional): If True, loads files lazily using Dask. Default is False.
+- `iterator` (bool, optional): If True and not using Dask, returns iterators instead of materialized arrays. Default is False.
+- `**kwargs`: Additional keyword arguments passed to `uproot.dask`, `uproot.iterate`, dak.from_parquet, or `awkward.from_parquet`.
+
+**Returns:**
+
+- `dict`: A dictionary where keys are sample names and values are either Awkward Arrays, Dask Arrays, or iterators. Keeping the same dict structure as the `deliver` output. 
+
+---
+
+## Usage Examples
+
+### Simple Materialization
+
+Load ServiceX deliver results directly into Awkward Arrays:
+
+```python
+from servicex_analysis_utils import to_awk
+from servicex import query, dataset, deliver
+
+spec = {
+    "Sample": [
+        {
+            "Name": "simple_transform",
+            "Dataset": dataset.FileList(
+                ["root://eospublic.cern.ch//eos/opendata/atlas/rucio/data16_13TeV/DAOD_PHYSLITE.37019878._000001.pool.root.1"]  # noqa: E501
+            ),
+            "Query": query.FuncADL_Uproot()
+            .FromTree("CollectionTree")
+            .Select(lambda e: {"el_pt": e["AnalysisElectronsAuxDyn.pt"]}), 
+        }
+    ]
+}
+
+arrays=to_awk(deliver(spec))
+```
+
+### Lazy Loading with Dask
+
+Load results lazily for large datasets using Dask task graphs. Enables parallel execution across multiple workers.
+
+```python
+import dask_awkward as dak
+
+dask_arrays = to_awk(deliver(spec), dask=True)
+el_pt_array = arrays["simple_transform"]["el_pt"]
+mean_el_pt = dak.mean(el_pt_array).compute()
+```
+
+### Using Iterators
+
+Return iterators instead of materialized arrays to avoid loading too much data into memory. Requires `dask=False` (default). Example with loading 10,000 events per chunk:
+
+```python
+iterables = to_awk(deliver(spec), iterator=True, step_size=10000)
+```
+
+You can then manually loop over the data chunks:
+
+```python
+for chunk in iterables['simple_transform']:
+    # process small chunk (~10k events)
+    analyse(chunk) #some function for el_pt
+```
+
+All events can also be loaded by using: 
+
+```python
+import awkward as ak
+arrays= ak.concatenate(list[iterables['simple_transform']])
+```
+
+---
+
+
+## Multiple samples
+
+ServiceX queries allow multiple sample transformations. The `to_awk` allows a straightforward manipulation of such requests. This allows seamless integration with analysis frameworks with multiple samples being manipulated separately after being passing the same transformation using `deliver()`.
+
+```python
+from servicex_analysis_utils import to_awk
+import awkward as ak
+
+# Given a ServiceX deliver return
+deliver_result = {
+    "Signal": ["path/to/signal_file1.root", "path/to/signal_file2.root"],
+    "Background": ["path/to/background_file.root"]
+}
+
+arrays = to_awk(deliver_result)
+
+signal_el_pt = arrays["Signal"]["el_pt"]
+background_el_pt = arrays["Background"]["el_pt"]
+
+mean_signal = ak.mean(signal_el_pt)
+mean_background = ak.mean(background_el_pt)
+
+print(f"Mean electron pT (Signal): {mean_signal:.2f} GeV")
+print(f"Mean electron pT (Background): {mean_background:.2f} GeV")
+```
+
+
+## Notes
+
+- **Multiple samples:** For transformations delivering multiple samples the dask and iterators are applied homegeneously to all.
+- **Error Handling:** In case of loading errors, the affected sample will have `None` as its value in the returned dictionary.
+- **Supported Formats:** A custom dict (non servicex) can be inputed but the paths must point be either ROOT or Parquet format.
+- **Branch Filtering, others:** Additional `**kwargs` allow specifying branch selections or other loading options supported by `uproot`, `awkward` and `dask_awkward`.
+
+---
