|
1 |
| -# servicex_client |
| 1 | +# ServiceX Client Library |
2 | 2 | Python SDK and CLI Client for ServiceX
|
3 | 3 |
|
4 |
| -## Configuration |
5 |
| -The client relies on a YAML file to obtain the URLs of different servicex |
6 |
| -deployments, as well as tokens to authenticate with the service. The file |
7 |
| -should be named `.servicex` and the format of this file is as follows: |
8 |
| -```yaml |
9 |
| -api_endpoints: |
10 |
| - - endpoint: http://localhost:5000 |
11 |
| - name: localhost |
12 |
| - |
13 |
| - - endpoint: https://servicex-release-testing-4.servicex.ssl-hep.org |
14 |
| - name: testing4 |
15 |
| - token: ... |
16 |
| - |
17 |
| -default_endpoint: testing4 |
18 |
| - |
19 |
| -cache_path: /tmp/ServiceX_Client/cache-dir |
20 |
| -shortened_downloaded_filename: true |
21 |
| - |
22 |
| -``` |
23 |
| -The `default_endpoint` will be used if otherwise not specified. The cache |
24 |
| -database and downloaded files will be stored in the directory specified by |
25 |
| -`cache_path`. |
26 |
| - |
27 |
| -The `shortened_downloaded_filename` property controls whether downloaded files |
28 |
| -will have their names shortened for convenience. Setting to false preserves |
29 |
| -the full filename from the dataset. |
30 |
| -` |
31 |
| -
|
32 |
| -The library will search for this file in the current working directory and then |
33 |
| -start looking in parent directories until a file is found. |
34 |
| -
|
35 |
| -## Command Line Interface |
36 |
| -When installed, the client provides a new command in your shell, `servicex`. |
37 |
| -This command uses a series of subcommands to work with various functions of |
38 |
| -serviceX. |
39 |
| - |
40 |
| -Common command line arguments: |
41 |
| - |
42 |
| -| Flag | Long Flag | What it does | |
43 |
| -|------|-----------|------------------------------------------------------| |
44 |
| -| -u | --url | The url of the serviceX ingress | |
45 |
| -| -b | --backend | Named backend from the .servicex file endpoints list | |
46 |
| - |
47 |
| -If neither url nor backend are specified then the client will attempt to use the |
48 |
| -`default_endpoint` value to determine who to talk to. |
49 |
| - |
50 |
| -### codegens |
51 |
| -This command will list the code generators deployed. |
52 |
| - |
53 |
| -### transforms |
54 |
| -These commands interact with transforms that have been run |
55 |
| - |
56 |
| -#### list |
57 |
| -List transforms associated with the current user. Add the `--complete` flag to |
58 |
| -only show transforms that have completed. |
59 |
| - |
60 |
| -#### files |
61 |
| -List the files along with their size generated by a transform. Specify the |
62 |
| -transform request id with the `-t` or `--transform-id` flag |
63 |
| - |
64 |
| -#### download |
65 |
| -Download the files from a transform to a local directory. Specify the transform |
66 |
| -request id with `-t` and the directory to download to with `-d`. Defaults to |
67 |
| -downloading files to the current working directory. |
68 |
| - |
69 |
| -### cache |
70 |
| -These commands allow you to work with the query cache maintained by the serviceX |
71 |
| -client. |
72 |
| - |
73 |
| -#### list |
74 |
| -Show all of the cached transforms along with the run time, code generator, and |
75 |
| -number of resulting files |
76 |
| - |
77 |
| -#### delete |
78 |
| -Delete a specific transform from the cache. Provide the transform request ID |
79 |
| -with the `-t` or `--transform-id` arg. |
80 |
| - |
81 |
| -#### clear |
82 |
| -Clear all of the transforms from the cache. Add `-y` to force the operation |
83 |
| -without confirming with the console. |
84 |
| - |
85 |
| -## Python SDK |
86 |
| -Entry to the SDK starts with constructing an instance of ServiceXClient. The |
87 |
| -constructor accepts `backend` argument to specify a named backend from the |
88 |
| -`.servicex` file, or `url` for the direct URL to a serviceX server. With the |
89 |
| -URL option you can't provide a token from `.servicex` so it must either be an |
90 |
| -unsecured endpoint, or the token must be provided via the WLCG standard of a |
91 |
| -file pointed to by `BEARER_TOKEN_FILE` environment variable. |
92 |
| - |
93 |
| -With an instance of ServiceXClient you can |
94 |
| -- List the code generators deployed with the ServiceX instance |
95 |
| -- List the transformers that have been run |
96 |
| -- Get the current status of a specific transform |
97 |
| - |
98 |
| -### Create a Dataset Instance to Run Transforms |
99 |
| -The ServiceX client also can create a `Dataset` instance that |
100 |
| -allows you to specify a query, provide a dataset identifier, |
101 |
| -and retrieve the results of the resulting transform request. |
102 |
| - |
103 |
| -There are two types of datasets |
104 |
| -- func_adl_dataset |
105 |
| -- Python Function dataset |
106 |
| - |
107 |
| -### Dataset Identifiers |
108 |
| -Before we get too deeply into the dataset classes, we should look |
109 |
| -at how to specify a dataset. |
110 |
| -- RucioDatasetIdentifier - for retrieving data files registered with Rucio |
111 |
| -- FileListDataset - A list of URIs for accessing files using xRootd |
112 |
| - |
113 |
| -### FuncADL Dataset |
114 |
| -This dataset is controlled by the func_adl language. The dataset |
115 |
| -supports the `Select`, `SelectMany`, `Where`, `MetaData`, and `QMetaData` |
116 |
| -operators from func_adl. |
117 |
| - |
118 |
| - |
119 |
| -### Datasets |
120 |
| -This is the abstract class for requesting data from ServiceX. You have to |
121 |
| -specify the dataset identifier you want data from and provide some sort of |
122 |
| -selection query. You can set the result format with the `set_result_format` |
123 |
| -operator (it's also a factory method arg for the dataset). |
124 |
| - |
125 |
| -Operators that cause the client to interact with the server: These terminal |
126 |
| -operators will call out to the serviceX server and process results. They |
127 |
| -are all implemented as asynchronous coroutines, but they also come with |
128 |
| -synchronous versions to make it easy to do easy things. |
129 |
| - |
130 |
| - |
131 |
| -#### Documentation |
132 |
| -The development documentation is hosted at: |
133 |
| -- github pages: https://ssl-hep.github.io/ServiceX_frontend |
134 |
| -- readthedocs: https://servicex-frontend.readthedocs.io |
| 4 | +## Documentation |
| 5 | +The development documentation is hosted at https://servicex-frontend.readthedocs.io |
135 | 6 |
|
136 | 7 |
|
0 commit comments