Merge pull request #444 from ssl-hep/doc_week

Overhaul of ServiceX Docs for 3.0 release

Author: kyungeonchoi

.github/workflows/docs.yml

@@ -55,7 +55,7 @@ jobs:
 
   deploy:
     name: Deploy docs to GitHub Pages
-    if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/3.0_develop')
+    if: github.event_name == 'push'
     needs: build
     # Set permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
     permissions:

.readthedocs.yaml

@@ -9,30 +9,24 @@ build:
   os: ubuntu-22.04
   tools:
     python: "3.12"
-    # You can also specify other tool versions:
-    # nodejs: "20"
-    # rust: "1.70"
-    # golang: "1.20"
 
 # Build documentation in the "docs/" directory with Sphinx
 sphinx:
   configuration: docs/conf.py
-  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
+#   # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
   builder: html
   # Fail on all warnings to avoid broken references
   # fail_on_warning: true
 
 # Optionally build your docs in additional formats such as PDF and ePub
 formats:
   - pdf
-#   - epub
 
 # Optional but recommended, declare the Python requirements required
 # to build your documentation
 # See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
 python:
   install:
-    - requirements: docs/requirements.txt
     - method: pip
       path: .
       extra_requirements:

.vscode/settings.json

@@ -55,7 +55,9 @@
         "nqueries",
         "ntuples",
         "numpy",
+        "opendata",
         "pathlib",
+        "PHYSLITE",
         "pnfs",
         "posix",
         "Powheg",
@@ -94,6 +96,7 @@
         "unittests",
         "URL's",
         "xaod",
+        "xaodr",
         "xrootd"
     ],
     "python.analysis.typeCheckingMode": "basic",

README.md

@@ -1,132 +1,7 @@
-# servicex_client
+# ServiceX Client Library
 Python SDK and CLI Client for ServiceX
 
-## Configuration
-The client relies on a YAML file to obtain the URLs of different servicex
-deployments, as well as tokens to authenticate with the service. The file 
-should be named `.servicex` and the format of this file is as follows:
-```yaml
-api_endpoints:
-  - endpoint: http://localhost:5000
-    name: localhost
+## Documentation
+The development documentation is hosted at https://servicex-frontend.readthedocs.io
 
-  - endpoint: https://servicex-release-testing-4.servicex.ssl-hep.org
-    name: testing4
-    token: ...
-
-default_endpoint: testing4
-
-cache_path: /tmp/ServiceX_Client/cache-dir
-shortened_downloaded_filename: true
-
-```
-The `default_endpoint` will be used if otherwise not specified. The cache 
-database and downloaded files will be stored in the directory specified by 
-`cache_path`.
-
-The `shortened_downloaded_filename` property controls whether downloaded files
-will have their names shortened for convenience. Setting to false preserves
-the full filename from the dataset.
-`
-
-The library will search for this file in the current working directory and then
-start looking in parent directories until a file is found.
-
-## Command Line Interface
-When installed, the client provides a new command in your shell, `servicex`.
-This command uses a series of subcommands to work with various functions of
-serviceX.
-
-Common command line arguments:
-
-| Flag | Long Flag | What it does                                         |
-|------|-----------|------------------------------------------------------|
-| -u   | --url     | The url of the serviceX ingress                      |
-| -b   | --backend | Named backend from the .servicex file endpoints list |
-
-If neither url nor backend are specified then the client will attempt to use the
-`default_endpoint` value to determine who to talk to.
-
-### codegens 
-This command will list the code generators deployed.
-
-### transforms
-These commands interact with transforms that have been run
-
-#### list
-List transforms associated with the current user. Add the `--complete` flag to
-only show transforms that have completed.
-
-#### files
-List the files along with their size generated by a transform. Specify the 
-transform request id with the `-t` or `--transform-id` flag
-
-#### download
-Download the files from a transform to a local directory. Specify the transform
-request id with `-t` and the directory to download to with `-d`. Defaults to
-downloading files to the current working directory.
-
-### cache
-These commands allow you to work with the query cache maintained by the serviceX
-client.
-
-#### list
-Show all of the cached transforms along with the run time, code generator, and 
-number of resulting files
-
-#### delete
-Delete a specific transform from the cache. Provide the transform request ID 
-with the `-t` or `--transform-id` arg.
-
-#### clear
-Clear all of the transforms from the cache. Add `-y` to force the operation 
-without confirming with the console.
-
-## Python SDK
-Entry to the SDK starts with constructing an instance of ServiceXClient.  The 
-constructor accepts `backend` argument to specify a named backend from the
-`.servicex` file, or `url` for the direct URL to a serviceX server. With the 
-URL option you can't provide a token from `.servicex` so it must either be an
-unsecured endpoint, or the token must be provided via the WLCG standard of a 
-file pointed to by `BEARER_TOKEN_FILE` environment variable.
-
-With an instance of ServiceXClient you can 
-- List the code generators deployed with the ServiceX instance
-- List the transformers that have been run
-- Get the current status of a specific transform
-
-### Create a Dataset Instance to Run Transforms
-The ServiceX client also can create a `Dataset` instance that 
-allows you to specify a query, provide a dataset identifier,
-and retrieve the results of the resulting transform request.
-
-There are two types of datasets
-- func_adl_dataset
-- Python Function dataset
-
-### Dataset Identifiers
-Before we get too deeply into the dataset classes, we should look
-at how to specify a dataset.
-- RucioDatasetIdentifier - for retrieving data files registered with Rucio
-- FileListDataset - A list of URIs for accessing files using xRootd
-
-### FuncADL Dataset
-This dataset is controlled by the func_adl language. The dataset
-supports the `Select`, `SelectMany`, `Where`, `MetaData`, and `QMetaData` 
-operators from func_adl.
-
-
-### Datasets
-This is the abstract class for requesting data from ServiceX. You have to 
-specify the dataset identifier you want data from and provide some sort of 
-selection query. You can set the result format with the `set_result_format` 
-operator (it's also a factory method arg for the dataset).
-
-Operators that cause the client to interact with the server: These terminal 
-operators will call out to the serviceX server and process results. They
-are all implemented as asynchronous coroutines, but they also come with 
-synchronous versions to make it easy to do easy things.
-
-
-#### 
 

docs/command_line.rst

@@ -0,0 +1,78 @@
+Command Line Interface
+=======================
+*\*Experimental*
+
+The command line interface (CLI) is a text-based interface used to interact with the system.
+When installed, the client provides a new command in your shell,
+``servicex``. This command uses a series of subcommands to work with
+various functions of serviceX.
+
+Common command line arguments:
+
++-----+----------+-----------------------------------------------------+
+| F   | Long     | What it does                                        |
+| lag | Flag     |                                                     |
++=====+==========+=====================================================+
+| -u  | –url     | The url of the serviceX ingress                     |
++-----+----------+-----------------------------------------------------+
+| -b  | –backend | Named backend from the .servicex file endpoints     |
+|     |          | list                                                |
++-----+----------+-----------------------------------------------------+
+
+If neither url nor backend are specified then the client will attempt to
+use the ``default_endpoint`` value to determine who to talk to.
+
+codegens
+~~~~~~~~
+
+This command will list the code generators deployed.
+
+transforms
+~~~~~~~~~~
+
+These commands interact with transforms that have been run
+
+list
+^^^^
+
+List transforms associated with the current user. Add the ``--complete``
+flag to only show transforms that have completed.
+
+files
+^^^^^
+
+List the files along with their size generated by a transform. Specify
+the transform request id with the ``-t`` or ``--transform-id`` flag
+
+download
+^^^^^^^^
+
+Download the files from a transform to a local directory. Specify the
+transform request id with ``-t`` and the directory to download to with
+``-d``. Defaults to downloading files to the current working directory.
+
+cache
+~~~~~
+
+These commands allow you to work with the query cache maintained by the
+serviceX client.
+
+
+list
+^^^^
+
+Show all of the cached transforms along with the run time, code
+generator, and number of resulting files
+
+delete
+^^^^^^
+
+Delete a specific transform from the cache. Provide the transform
+request ID with the ``-t`` or ``--transform-id`` arg.
+
+clear
+^^^^^
+
+Clear all of the transforms from the cache. Add ``-y`` to force the
+operation without confirming with the console.
+

docs/conf.py

@@ -11,9 +11,9 @@
 # sys.path.insert(0, os.path.abspath(".."))
 
 
-project = 'ServiceX Client'
-copyright = '2023, Ben Galewsky, Gordon Watts'
-author = 'Ben Galewsky, Gordon Watts'
+project = 'ServiceX'
+copyright = '2024 Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP)'  # NOQA 501
+author = 'Ben Galewsky, Gordon Watts, KyongEon Choi, Ketan Mahajan, Peter Onyisi'
 release = '3.0.0'
 
 # -- General configuration ---------------------------------------------------
@@ -25,15 +25,26 @@
     'sphinx.ext.viewcode',
     'sphinx.ext.doctest',
     'code_include.extension',
-    'myst_parser'
+    'myst_parser',
+    'sphinxcontrib.autodoc_pydantic',
+    'sphinx_tabs.tabs'
 ]
 
 templates_path = ['_templates']
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'examples']
 
 autoclass_content = 'both'
 
-
+autodoc_pydantic_model_show_json = False
+autodoc_pydantic_field_list_validators = False
+autodoc_pydantic_config_members = False
+autodoc_pydantic_model_members = False
+autodoc_pydantic_model_show_config_summary = False
+autodoc_pydantic_model_show_field_summary = False
+autodoc_pydantic_model_undoc_members = False
+autodoc_pydantic_settings_show_validator_summary = False
+autodoc_pydantic_settings_show_validator_members = False
+autodoc_pydantic_model_member_order = 'bysource'
 # -- Options for HTML output -------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output