Merge pull request #116 from jku/client-readme-update

Rewrite the Client CLI documentation

Author: jku

CLIENT-CLI.md

@@ -0,0 +1,101 @@
+# tuf-conformance client-under-test CLI
+
+Before a TUF client can be tested with the tuf-conformance test suite, an executable must be provided
+that implements this CLI protocol. There are three required commands: `init`, `refresh` and `download`.
+
+## Commands
+
+### Init command
+
+Initialize the clients local trusted metadata.
+
+`<client> --metadata-dir <METADATA_DIR> init <TRUSTED_ROOT>`
+
+where
+- `METADATA_DIR` is a directory where the client is expected to store metadata it considers valid
+- `TRUSTED_ROOT` is a file that contains initial root.json content
+
+Client must initialize with the given trusted root (this may involve simply copying the
+trusted root into `METADATA_DIR`: no checks are expected). No requests should be made to the repository at this point.
+Client must use exit code 0 if the initialization succeeded and exit code 1 otherwise.
+
+This command will be called only once for a specific test.
+
+**Example**
+
+`my-tuf-client --metadata-dir /tmp/metadata init ./initial_root.json`
+
+
+### Refresh command
+
+Update local metadata from the repository.
+
+`<client> --metadata-dir <METADATA_DIR> --metadata-url <METADATA_URL> refresh`
+
+where
+- `METADATA_DIR` is a directory where the client is expected to read and write trusted local metadata
+- `METADATA_URL` is a URL to the repository metadata store
+
+Client must update its top-level metadata according to the TUF client workflow. Client must use exit code
+0 if the refresh succeeded fully and exit code 1 if any part of the refresh failed. 
+
+Client must use non-versioned filenames for metadata in `METADATA_DIR` (so "root.json" instead of "1.root.json").
+
+This command may be called multiple times in a test.
+
+**Example**
+
+```
+my-tuf-client \
+    --metadata-dir /tmp/metadata \
+    --metadata-url http://localhost:8888/test-repo/metadata \
+    refresh`
+```
+
+
+### Download command
+
+Download an artifact from repository, store it in local disk.
+
+`<client> --metadata-dir <METADATA_DIR> --metadata-url <METADATA_URL> --target-name <TARGET_PATH> --target-base-url <TARGET_URL> --target-dir <TARGET_DIR> download`
+
+where
+- `METADATA_DIR` is a directory where the client is expected to store metadata it considers valid
+- `METADATA_URL` is a URL to the repository metadata store
+- `TARGET_PATH` is the TUF targetpath of the artifact that should be downloaded
+- `TARGET_URL` is the base URL for repository target store
+- `TARGET_DIR`: is a directory where the client should store downloaded and verified files
+
+Client must download the given artifact (targetpath) according to TUF client workflow, and store it in
+given targets directory. Client must ensure that metadata is up-to-date before downloading artifacts
+(see _Refresh command_ for rules on metadata storage in `METADATA_DIR`).
+Client must use exit code 0 if the download succeeded fully and exit code 1 if any part of
+the download failed.
+
+Client may use any filename or directory structure it wants within `TARGET_DIR`: The expectation is that if a 
+targetpath is downloaded twice it is stored with the same filename (even if the content changed).
+
+This command may be called multiple times in a test.
+
+**Example**
+
+```
+my-tuf-client \
+    --metadata-dir /tmp/metadata \
+    --metadata-url http://localhost:8888/test-repo/metadata \
+    --target-name path/to/artifact.txt \
+    --target-base-url http://localhost:8888/test-repo/targets \
+    --target-dir /tmp/targets
+    download
+```
+
+## What does the test suite measure?
+
+The test suite has limited visibility into client decisions so uses various signals to measure conformance:
+* Client command success/failure
+* The metadata the client considers trusted (`METADATA_DIR`)
+* The artifacts the client downloads (`TARGET_DIR`)
+* The HTTP requests made by the client to the repository
+
+Sometimes test suite expectations are not strictly part of TUF specification: As an example the test suite expects
+a specific sequence of metadata HTTP requests to assert that the client handled delegations correctly.

README.md

@@ -22,7 +22,7 @@ The conformance test suite provides a GitHub action that can be used to test a T
 There are two required steps:
 
 1. Include an executable in the client project that implements the client-under-test
-   [CLI protocol](clients/README.md). 
+   [CLI protocol](CLIENT-CLI.md).
 2. Use the `theupdateframework/tuf-conformance` action in your test workflow:
     ```yaml
     jobs:

clients/README.md

@@ -1,50 +1,4 @@
-# Client interface
-
-Clients must implement the following command-line interface:
-
-`init`: Initialises the clients local trusted metadata. It has the following command-line arguments:
-- `--metadata-dir`, Required: The localpath where the client stores metadata.
-- `--trusted-root`, Required: The initial root. The conformance tests generate this.
-
-`refresh`: Updates the local metadata from the repository.
-- `--metadata-dir`, Required: The localpath where the client stores metadata.
-- `--metadata-url`, Required: The URL to the repository. The test suite takes care of passing the URL to the local repository.
-
-`download`: Downloads a target file.
-- `--metadata-dir`, Required: The localpath where the client stores metadata.
-- `--metadata-url`, Required: The URL to the repository. The test suite takes care of passing the URL to the local repository.
-- `--target-name`: The name of the target from the `targets.json` metadata.
-- `--target-base-url`: Base URL to download the target file from. Clients are expected to concatenate `--target-base-url` with `--target-name`.
-- `--target-dir`: The directory in which the client stores downloaded files.
-
-## Integrating a client
-
-The conformance tests implements a `ClientRunner` class which represents the client in the conformance test suite. The goal of the `ClientRunner` is to carry out high-level tasks, for example:
-
-1. Update local metadata
-2. Download target file
-3. Check that the local metadata is what we expect
-4. Check that a downloaded target file is the one we expected.
-
-Some of these tasks are mere assertions against the local metadata. Others require `ClientRunner` to interact with the remote repository. When the `ClientRunner` needs to interact with the repository, it does so through a client wrapper. A client wrapper is a layer between the TUF-conformance test suite and a TUF client implementation. The client wrapper's goal is to invoke the client implementation we want to test. Client wrappers are responsible for:
-
-1. receiving the parameters from the conformance test-suite
-2. invoking the client implementation in a correct manner with the parameters from the test-suite.
-
-We can illustrate the behavior as such:
-
-![Test suite clients workflow](../media/conformance-test-client-workflow.png)
-
-In a conformance test, a `ClientRunner` instructs the client wrapper to fetch the latest metadata. The client wrapper invokes the client implementation - for example python-tuf or go-tuf. The client implementation interacts with the test repository; the client implementation also stores the metadata. The conformance test can then interact directly with the local metadata and verify that it is correct. 
-
-## Where should clients run their tests?
-
-The intention is that TUF client implementations that want to use the test suite will include wrappers like this in their source code, and will run the test suite in their CI systems.
-
-This project may then also run the test suite using the wrappers from the TUF implementations, but the intention is that this repository would not have to e.g. maintain dependency lists for the client wrappers.
-
-That said, some wrappers are included for now:
-* We will likely tweak the "wrapper API" while we build the initial test suite: while this goes on, it makes
-  sense to keep the wrappers here
-* We do need something to test the test suite itself
-* There is no easy way to run the test suite in a TUF implementation CI yet
+This directory contains test clients for the test suite. They are implementations of the
+client-under-test [CLI protocol](../CLIENT-CLI.md) and are intended to help the development
+of the test suite itself and to act as examples for actual client-under-test
+implementations.