Quick Proofread (#472)

* Clarifications

* Fix up bad servicex.yaml file

* Add more robust funcadl query and clean up

* Enhance documentation for FuncADL sequences and Select call usage

* Clean up, filling things out a little bit.

* Updates

* Make it work for just the front end

Author: gordonwatts

docs/command_line.rst

@@ -2,10 +2,10 @@ Command Line Interface (Experimental)
 ======================================
 *\*The command line interface is an under-development feature that is not supported in the 3.0.0 release*
 
-The command line interface (CLI) is a text-based interface used to interact with the system.
-When installed, the client provides a new command in your shell,
+The command line interface (CLI) is a text-based interface used to interact with the ServiceX backend.
+The client provides a new command in your shell,
 ``servicex``. This command uses a series of subcommands to work with
-various functions of serviceX.
+various functions of serviceX. It is installed automatically when you install the servicex frontend package.
 
 Common command line arguments:
 

docs/connect_servicex.rst

@@ -5,8 +5,9 @@ You need a `ServiceX endpoint <select-endpoint_>`_ where transformation is happe
 a `client library <client-installation_>`_ to submit a transformation request.
 
 .. _select-endpoint:
+
 Selecting an ServiceX endpoint
-----------------------
+------------------------------
 
 ServiceX is a hosted service. Each ServiceX instance is deployed at the server
 and dedicated to a specific experiment. Depending on which experiment you work in,
@@ -49,33 +50,41 @@ downloaded to your computer.
 
 
 ServiceX Access File
-~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~
 
 The client relies on a ``servicex.yaml`` file to obtain the URLs of different
 servicex deployments, as well as tokens to authenticate with the
-service. The format of this file is as follows:
+service.
+
+The client library will search for this file in the current working directory
+and then start looking in parent directories and your home directory until a file
+is found.
+
+The format of this file is as follows:
 
 .. code:: yaml
-   
+
+   api_endpoints:
      - endpoint: https://servicex.af.uchicago.edu
        name: servicex-uc-af
        token: <YOUR TOKEN>
 
    cache_path: /tmp/ServiceX_Client/cache-dir
    shortened_downloaded_filename: true
 
+``cache_path`` and ``shortened_downloaded_filename`` are optional fields and default to
+reasonable values.
+
 The cache database and downloaded files will be stored in the directory
 specified by ``cache_path``.
 
 The ``shortened_downloaded_filename`` property controls whether
 downloaded files will have their names shortened for convenience.
-Setting to false preserves the full filename from the dataset. \`
-
-The client library will search for this file in the current working directory
-and then start looking in parent directories until a file is found.
+Setting to false preserves the full filename from the dataset.
 
 
 .. _client-installation:
+
 ServiceX Client Installation
 ----------------------------
 ServiceX client Python package is a python library for users to communicate 
@@ -87,7 +96,7 @@ Prerequisites
 ~~~~~~~~~~~~~
 
 - Python 3.8, or above
-- Access to ServiceX endpoint (Member of the ATLAS or CMS collaborations)
+- Access to ServiceX endpoint
 
 Installation
 ~~~~~~~~~~~~

docs/contribute.rst

@@ -6,65 +6,52 @@ Welcome to the ServiceX contributor guide, and thank you for your interest in co
 Overview
 --------
 
-ServiceX uses a microservice architecture, 
-and is designed to be hosted on a Kubernetes cluster. 
-The ServiceX project uses a polyrepo strategy for source code management: 
-the source code for each microservice is located in a dedicated repo. 
+The ``servicex`` frontend code uses standard python packaging and open-source development methodologies. The code is hosted on GitHub,
+and we use the GitHub issue tracker to manage bugs and feature requests. We also use GitHub pull requests for code review and merging. 
 
-Below is a partial list of these repositories:
-
-- `ServiceX <https://github.com/ssl-hep/ServiceX>`_ - Main repository, contains Helm charts for deployment to Kubernetes
 - `ServiceX_frontend <https://github.com/ssl-hep/ServiceX_frontend>`_ - The ServiceX Python library, which enables users to send requests to ServiceX. Currently, this is the only ServiceX frontend client.
-- `ServiceX_App <https://github.com/ssl-hep/ServiceX_App>`_ - The ServiceX API Server, written in Flask.
 
-Additional repositories related to the project can be found in the `ssl-hep GitHub organization <https://github.com/ssl-hep>`_.
+Additional repositories related to the ServiceX project can be found in the `ssl-hep GitHub organization <https://github.com/ssl-hep>`_.
 
-Please read our `architecture document <https://servicex.readthedocs.io/en/latest/development/architecture/>`_ for more details.
+Join us on Slack
+-----------------
+
+We coordinate our efforts on the `IRIS-HEP Slack <http://iris-hep.slack.com>`_.
+Come join this intellectual hub!
+
+Issues
+------
+
+All development work on the code should start with an issue. Please submit issues for bugs and feature
+requests to the `repository <https://github.com/ssl-hep/ServiceX_frontend>`_.
 
 Branching Strategy
 -------------------
 
-ServiceX uses a slightly modified GitLab flow. Each repository has a main branch, usually named `develop` (or `master` for the Python frontend). All changes should be made on feature branches and submitted as PRs to the main branch. Releases are frozen on dedicated release branches, e.g. `v1.0.0-RC.2`. 
+ServiceX uses a slightly modified GitLab flow. The `master` branch is used for releases, and
+all development work occurs on feature branches.
 
 Development Workflow
 ---------------------
 
 1. Set up a local development environment:
-    - Decide which microservice (or Helm chart) you'd like to change, and locate the corresponding repository.
-    - If you are a not a member of the ``ssl-hep`` GitHub organization, fork the repository.
+    - Fork the ``ServiceX_frontend``
     - Clone the (forked) repository to your local machine:
 
-    .. code-block:: bash
-
-        git clone git@github.com:<GitHub username>/ServiceX_App.git    
-
-    - If you created a fork, add the upstream repository as remote:
-
-    .. code-block:: bash
-
-        git remote add upstream git@github.com:ssl-hep/ServiceX_App.git
-    
     - Set up a new environment via ``conda`` or ``virtualenv``.
     - Install dependencies, including test dependencies:
 
     .. code-block:: bash
 
-        python3 -m pip install -e .[test]
-
-    - If the root directory contains a file named ``.pre-commit-config.yaml``, you can install the `pre-commit <https://pre-commit.com/>`_ hooks with:
-
-    .. code-block:: bash
-
-        pip install pre-commit
-        pre-commit install
+        python3 -m pip install -e .[develop]
 
 2. Develop your contribution:
     - Pull latest changes from upstream:
 
     .. code-block:: bash
 
-        git checkout develop
-        git pull upstream develop
+        git checkout master
+        git pull upstream master
         
     - Create a branch for the feature you want to work on:
 
@@ -77,115 +64,5 @@ Development Workflow
 3. Test your changes:
     - Run the full test suite with ``python -m pytest``, or target specific test files with ``python -m pytest tests/path/to/file.py``.
     - Please write new unit tests to cover any changes you make.
-    - You can also manually test microservice changes against a full ServiceX deployment by building the Docker image, pushing it to DockerHub, and setting the `image` and `tag` values as follows:
-
-    .. code-block:: yaml
-
-        app:
-        image: <organization>/<image repository>
-        tag: my-feature-branch
-    
-    - For more details, please read our full `deployment guide <https://servicex.readthedocs.io/en/latest/deployment/basic>`_. 
 
 4. Submit a pull request to the upstream repository
-    
-
-Issues
-------
-
-Please submit issues for bugs and feature requests to the `main ServiceX repository <https://github.com/ssl-hep/ServiceX>`_, unless the issue is specific to a single microservice.
-
-We manage project priorities with a `ZenHub board <https://app.zenhub.com/workspaces/servicex-5caba4288d0ceb76ea94ae1f/board?repos=180217333,180236972,185614791,182823774,202592339>`_.
-
-Join us on Slack
------------------
-
-We coordinate our efforts on the `IRIS-HEP Slack <http://iris-hep.slack.com>`_.
-Come join this intellectual hub!
-
-Running the Full ServiceX Chart Locally
-----------------------------------------
-
-You can run ServiceX on your laptop using ``docker`` or another similar tool that supports kubernetes.
-
-Prerequisites
---------------
-
-1. ``docker`` is installed and ``kubernetes`` is running (see configuration options).
-2. Make sure ``kubectl`` and ``helm`` are both installed in the shell you'll be doing your development work.
-3. Follow instructions in the deployment guide to install your x509 certificate if you are going to be using any `rucio` or GRID services for your testing.
-
-Running the chart
-------------------
-
-
-1. In the ``Servicex/helm`` directory run ``helm dependency update servicex/``
-2. And install the chart with ``helm install -f values.yaml servicex-testing .\servicex\``
-3. As in the deployment guide, you can now port-forward your servicex ``app`` and ``minio``.
-
-How you write your ``values.yaml`` will depend a lot on what you are testing. Here is an example of a minimal one that will load up the `develop` tag for all the container images, and expects an ATLAS GRID cert:
-
-.. code-block:: yaml
-
-    postgres:
-    enabled: true
-    objectStore:
-    publicURL: localhost:9000
-
-    gridAccount: <your-user>
-
-    x509Secrets:
-    # For ATLAS
-    vomsOrg: atlas
-
-    app:
-    ingress:
-        host: localhost:5000
-
-    transformer:
-    cachePrefix: '""'
-
-
-Making Changes
----------------
-
-
-The best way to work on ServiceX is using the unit tests. That isn't always possible, of course. When it isn't your development cycle will require you to build any changed containers. A possible workflow is:
-
-1. Redeploy the ``helm`` chart (or perhaps use ``upgrade`` rather than ``install`` in the ``helm`` command) and add ``pullPolicy: Never`` to the appropriate app section. For example, add it under ``app:`` in the example file above if you are working on ``servicex_app``.
-2. Change your code (say, in ``servicex_app``).
-3. In the directory for the app should be a ``Dockerfile``. Do the build, and pay attention to the tag. For example, ``docker build -t sslhep/servicex_app:develop .``.
-4. Finally restart the pod, which should cause it to pick up the new build. This might kill a port-forward you have in place, so don't forget to restart that!
-
-Debugging Tips
----------------
-
-Microservice architectures can be difficult to test and debug. Here are some 
-helpful hints to make this easier.
-
-1. Instead of relying on the DID Finder to locate some particular datafile, you
-can mount one of your local directories into the transformer pod and then 
-instruct the DID Finder to always offer up the path to that file regardless of
-the submitted DID. You can use the ``hostMount`` value to have a local directory
-mounted into each transformer pod under ``/data``. You can use the 
-``didFinder.staticFile`` value to instruct DID Finder to offer up a file from that
-directory.
-2. You can use port-forwarding to expose port 15672 from the RabbitMQ pod to 
-your laptop and log into the Rabbit admin console using the username: ``user`` and
-password ``leftfoot1``. From here you can monitor the queues, purge old messages
-and inject your own messages
-
-Notes for Maintainers
----------------------
-
-Hotfixes
---------
-
-If a critical bugfix or hotfix must be applied to a previous release, it should be merged to the main branch and then applied to each affected release branch using 
-
-.. code-block:: bash
-
-    git cherry-pick <merge commit hash> -m 1
-
-Merge commits have 2 parents, so the ``-m 1`` flag is used to specify that the first parent (i.e. previous commit on the main branch) should be used.
-

docs/index.rst

@@ -11,7 +11,8 @@ The High Luminosity Large Hadron Collider (HL-LHC) faces enormous computational
 structure due to high pileup conditions. The ATLAS and CMS experiments will record ~ 10 times as
 much data from ~ 100 times as many collisions as were used to discover the Higgs boson.
 
-ServiceX is a scalable data extraction, transformation and delivery system deployed in a Kubernetes cluster.
+ServiceX is a scalable data extraction, transformation and delivery system deployed in a Kubernetes cluster
+designed to efficiently extract columnar data from large datasets.
 
 .. image:: img/organize2.png
     :alt: organize
@@ -24,39 +25,39 @@ This section describes the concepts that are important to understand when workin
 Datasets
 ^^^^^^^^^
 Datasets are groups of experimental data from which columnar data can be extracted. ServiceX
-supports four sources of of datasets:
+supports four sources of data:
+
 1. Rucio
 2. CERN Open Data Portal
-3. File List
+3. List of File accessible via HTTP or XRootD
 4. EOS Directory
 
 Queries
 ^^^^^^^
 Queries are used to extract data from a dataset. They specify the columns to extract, the events to
 include in the output. There are several types of queries supported by ServiceX:
+
 1. func-adl
 2. Python Function
 3. Dictionary of uproot selections
 
-
 Sample
 ^^^^^^
-A sample is a request to extract columnar data from a given dataset, using a specific
-query. It results in a set of output files that can be used in an analysis.
+A sample is a request to extract columnar data from a specified dataset, using a specific
+query. It results in a set of output files containing the requested data that can be used
+in an analysis via `awkward`, `RDF`, etc..
 
 Transformation Request
 ^^^^^^^^^^^^^^^^^^^^^^
 Multiple samples can be submitted to ServiceX at the same time. Each sample is processed
 independently, and the results can be retrieved as files downloaded to a local directory or
-a list of URLs.
+directly accessed via a URL from ServiceX's output cache.
 
 Local Cache
 ^^^^^^^^^^^
-ServiceX maintains a local cache of the results of queries. This cache can be used to avoid
+ServiceX maintains a local cache of the performed queries and their results. This cache can be used to avoid
 re-running queries that have already been executed.
 
-
-
 .. toctree::
    :maxdepth: 2
    :caption: Contents:

docs/query_types.rst

@@ -53,7 +53,7 @@ This table sumarizes the query types supported by ServiceX and the data formats
 A brief introduction to the query languages
 -------------------------------------------
 
-* **FuncADL** is an Analysis Description Language inspired by functional languages. Sophisticated filtering and computation of new values can be expressed by chaining a series of simple functions. Because FuncADL is written independently of the underlying data libraries, it can run on many data formats.
+* **FuncADL** is an Analysis Description Language inspired by functional languages and C#'s LINQ. Sophisticated filtering and computation of new values can be expressed by chaining a series of simple functions. Because FuncADL is written independently of the underlying data libraries, it can run on many data formats.
 
 * **Uproot-Raw** passes user requests to the ``.arrays()`` function in ``uproot``. In particular, the branches of the input ``TTrees`` can be filtered, cuts can be specified to select events, and additional expressions can be computed. Additional non-``TTree`` objects can be copied from the inputs to the outputs.
 
@@ -107,15 +107,39 @@ Each dictionary either has a ``treename`` key (indicating that it is a query on
 
 FuncADL Query Type
 ------------------
-The FuncADL Query type is very powerful. It is based on functional programming concepts and allows
-the user to specify complex queries in a very compact form. The query is written in a functional
+FuncADL queries are based on functional programming concepts and allow
+the user to specify complex queries in a compact form. The query is written in a functional
 style, with a series of functions that are applied to the data in sequence. The query is written
 in a string or as typed python objects. Depending on the source file format, the query is translated
 into C++ `EventLoop <https://atlassoftwaredocs.web.cern.ch/analysis-software/AnalysisTools/el_intro/>`_
 code, or uproot python code.
 
-Full documentation on the func-adl query language can be found at this `JupyterBook <https://gordonwatts.github.io/xaod_usage/intro.html>`_.
+An example that fetches the :math:`p_T, \eta` and EM fraction of jets from an ATLAS PHYSLITE file is as follows:
+
+.. code-block:: python
+  
+  from func_adl_servicex_xaodr22 import FuncADLQueryPHYSLITE, cpp_float
+
+  query = FuncADLQueryPHYSLITE()
+  jets_per_event = query.Select(lambda e: e.Jets('AnalysisJets'))
+  jet_info_per_event = jets_per_event.Select(
+      lambda jets: {
+          'pt': jets.Select(lambda j: j.pt()),
+          'eta': jets.Select(lambda j: j.eta()),
+          'emf': jets.Select(lambda j: j.getAttribute[cpp_float]('EMFrac'))  # type: ignore
+      }
+  )
+
+FuncADL is based on the concept of sequences. The events in a dataset are a sequence of events. The jets in an event are a sequence of jets.
+The ``Select`` call applies a function that transforms the input sequence, element-by-element, into an output sequence. In the above example,
+the first ``Select`` call is used to transform the sequence of events into a sequence of a sequence of jets (e.g. a sequence of jets representing
+the jets in an event - a 2D array, if you will). The lambda function passed to the ``Select`` call
+is applied to each event in the input sequence, and the result is a sequence of jets for each event.
+
+The dictionary defines the columns of the output file (e.g. the leaves in a ``TTree``). In each case, the three ``lambda`` functions are applied
+to each jet, transforming the sequence of jets into a sequence of :math:`p_T` values, a sequence of :math:`\eta` values, and a sequence of EM fractions.
 
+Full documentation on the func-adl query language can be found at this `JupyterBook <https://gordonwatts.github.io/xaod_usage/intro.html>`_.
 
 Python Function Query Type
 --------------------------