Skip to content

Commit a3b4e5a

Browse files
PabloPardoGarciaPabloPardoGarcia
authored
Merge pull request #26 from glassflow/release/v2.0.0
Release/v2.0.2
2 parents 72d5a3d + 5a16858 commit a3b4e5a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

51 files changed

+3785
-821
lines changed

‎.github/workflows/on_pr.yaml

Lines changed: 57 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@ on:
44
pull_request:
55
branches:
66
- main
7+
- release/*
8+
9+
permissions:
10+
contents: write
11+
checks: write
12+
pull-requests: write
713

814
jobs:
915
tests:
@@ -24,11 +30,16 @@ jobs:
2430
- name: Install dependencies
2531
run: pip install -e .[dev]
2632

27-
- name: Run Integration Tests
28-
run: pytest tests/glassflow/integration_tests/
33+
- name: Run Tests
34+
run: pytest
2935
env:
30-
PIPELINE_ID: ${{ secrets.PIPELINE_ID }}
31-
PIPELINE_ACCESS_TOKEN: ${{ secrets.PIPELINE_ACCESS_TOKEN }}
36+
PERSONAL_ACCESS_TOKEN: ${{ secrets.INTEGRATION_PERSONAL_ACCESS_TOKEN }}
37+
38+
- name: Upload coverage report
39+
uses: actions/upload-artifact@v4
40+
with:
41+
name: coverageReport
42+
path: tests/reports/
3243

3344
checks:
3445
name: Run code checks
@@ -48,8 +59,46 @@ jobs:
4859
- name: Install dependencies
4960
run: pip install -e .[dev]
5061

51-
- name: Run isort
52-
run: isort .
53-
54-
- name: Run linter and code formatter
62+
- name: Run ruff linter checks
5563
run: ruff check .
64+
65+
- name: Run ruff formatting checks
66+
run: ruff format --check .
67+
68+
coverage:
69+
runs-on: ubuntu-latest
70+
needs: [tests]
71+
steps:
72+
- uses: actions/checkout@v3
73+
with:
74+
persist-credentials: false # otherwise, the token used is the GITHUB_TOKEN, instead of your personal token
75+
fetch-depth: 0 # otherwise, you will fail to push refs to dest repo
76+
77+
- name: Download coverage report
78+
uses: actions/download-artifact@v4
79+
with:
80+
name: coverageReport
81+
82+
- name: Pytest coverage comment
83+
id: coverageComment
84+
uses: MishaKav/pytest-coverage-comment@main
85+
with:
86+
pytest-xml-coverage-path: ./coverage.xml
87+
junitxml-path: ./pytest.xml
88+
89+
- name: Update Readme with Coverage Html
90+
if: ${{ github.ref == 'refs/heads/main' }}
91+
run: |
92+
sed -i '/<!-- Pytest Coverage Comment:Begin -->/,/<!-- Pytest Coverage Comment:End -->/c\<!-- Pytest Coverage Comment:Begin -->\n\${{ steps.coverageComment.outputs.coverageHtml }}\n<!-- Pytest Coverage Comment:End -->' ./README.md
93+
94+
- name: Commit & Push changes to Readme
95+
id: commit
96+
if: ${{ github.ref == 'refs/heads/main' }}
97+
run: |
98+
git config --global user.name 'GitHub Actions Workflow glassflow-python-sdk/.github/workflows/on_pr.yaml'
99+
git config --global user.email 'glassflow-actions-workflow@users.noreply.github.com'
100+
101+
git add README.md
102+
git commit -m "Update coverage on Readme"
103+
104+
git push origin master

‎.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,6 @@ site**
1010
.pypirc
1111
dist/
1212
build
13-
.env
13+
.env
14+
tests/reports
15+
.coverage

‎README.md

Lines changed: 147 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
<img src="https://img.shields.io/badge/slack-join-community?logo=slack&amp;logoColor=white&amp;style=flat"
99
alt="Chat on Slack"></a>
1010
<a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff" style="max-width:100%;"></a>
11+
<!-- Pytest Coverage Comment:Begin -->
12+
<!-- Pytest Coverage Comment:End -->
1113

1214

1315
# GlassFlow Python SDK
@@ -22,11 +24,12 @@ You can install the GlassFlow Python SDK using pip:
2224
pip install glassflow
2325
```
2426

25-
## Available Operations
27+
## Data Operations
2628

2729
* [publish](#publish) - Publish a new event into the pipeline
2830
* [consume](#consume) - Consume the transformed event from the pipeline
2931
* [consume failed](#consume-failed) - Consume the events that failed from the pipeline
32+
* [validate credentials](#validate-credentials) - Validate pipeline credentials
3033

3134

3235
## publish
@@ -36,12 +39,11 @@ Publish a new event into the pipeline
3639
### Example Usage
3740

3841
```python
39-
import glassflow
42+
from glassflow import PipelineDataSource
4043

41-
client = glassflow.GlassFlowClient()
42-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
44+
source = PipelineDataSource(pipeline_id="<str value", pipeline_access_token="<str token>")
4345
data = {} # your json event
44-
res = pipeline_client.publish(request_body=data)
46+
res = source.publish(request_body=data)
4547

4648
if res.status_code == 200:
4749
print("Published sucessfully")
@@ -55,33 +57,163 @@ Consume the transformed event from the pipeline
5557
### Example Usage
5658

5759
```python
58-
import glassflow
60+
from glassflow import PipelineDataSink
5961

60-
client = glassflow.GlassFlowClient()
61-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
62-
res = pipeline_client.consume()
62+
sink = PipelineDataSink(pipeline_id="<str value", pipeline_access_token="<str value>")
63+
res = sink.consume()
6364

6465
if res.status_code == 200:
65-
print(res.body.event)
66+
print(res.json())
6667
```
6768

69+
6870
## consume failed
6971

7072
If the transformation failed for any event, they are available in a failed queue. You can consume those events from the pipeline
7173

7274
### Example Usage
7375

7476
```python
75-
import glassflow
77+
from glassflow import PipelineDataSink
7678

77-
client = glassflow.GlassFlowClient()
78-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
79-
res = pipeline_client.consume_failed()
79+
sink = PipelineDataSink(pipeline_id="<str value", pipeline_access_token="<str value>")
80+
res = sink.consume_failed()
8081

8182
if res.status_code == 200:
82-
print(res.body.event)
83+
print(res.json())
84+
```
85+
86+
87+
## validate credentials
88+
89+
Validate pipeline credentials (`pipeline_id` and `pipeline_access_token`) from source or sink
90+
91+
### Example Usage
92+
93+
```python
94+
from glassflow import PipelineDataSource, errors
95+
96+
try:
97+
source = PipelineDataSource(pipeline_id="<str value", pipeline_access_token="<str value>")
98+
source.validate_credentials()
99+
except errors.PipelineNotFoundError as e:
100+
print("Pipeline ID does not exist!")
101+
raise e
102+
except errors.PipelineAccessTokenInvalidError as e:
103+
print("Pipeline Access Token is invalid!")
104+
raise e
105+
```
106+
107+
108+
## Pipeline Management
109+
110+
In order to manage your pipelines with this SDK, one needs to provide the `PERSONAL_ACCESS_TOKEN`
111+
to the GlassFlow client.
112+
113+
```python
114+
from glassflow import GlassFlowClient
115+
116+
client = GlassFlowClient(personal_access_token="<your personal access token>")
117+
```
118+
119+
Now you can perform CRUD operations on your pipelines:
120+
121+
* [list_pipelines](#list_pipelines) - Returns the list of pipelines available
122+
* [get_pipeline](#get_pipeline) - Returns a pipeline object from a given pipeline ID
123+
* [create](#create) - Create a new pipeline
124+
* [delete](#delete) - Delete an existing pipeline
125+
126+
## list_pipelines
127+
128+
Returns information about the available pipelines. It can be restricted to a
129+
specific space by passing the `space_id`.
130+
131+
### Example Usage
132+
133+
```python
134+
from glassflow import GlassFlowClient
135+
136+
client = GlassFlowClient(personal_access_token="<your access token>")
137+
res = client.list_pipelines()
138+
```
139+
140+
## get_pipeline
141+
142+
Gets information about a pipeline from a given pipeline ID. It returns a Pipeline object
143+
which can be used manage the Pipeline.
144+
145+
### Example Usage
146+
147+
```python
148+
from glassflow import GlassFlowClient
149+
150+
client = GlassFlowClient(personal_access_token="<your access token>")
151+
pipeline = client.get_pipeline(pipeline_id="<your pipeline id>")
152+
153+
print("Name:", pipeline.name)
83154
```
84155

156+
## create
157+
158+
The Pipeline object has a create method that creates a new GlassFlow pipeline.
159+
160+
### Example Usage
161+
162+
```python
163+
from glassflow import Pipeline
164+
165+
pipeline = Pipeline(
166+
name="<your pipeline name>",
167+
transformation_file="path/to/transformation.py",
168+
space_id="<your space id>",
169+
personal_access_token="<your personal access token>"
170+
).create()
171+
```
172+
173+
In the next example we create a pipeline with Google PubSub source
174+
and a webhook sink:
175+
176+
```python
177+
from glassflow import Pipeline
178+
179+
pipeline = Pipeline(
180+
name="<your pipeline name>",
181+
transformation_file="path/to/transformation.py",
182+
space_id="<your space id>",
183+
personal_access_token="<your personal access token>",
184+
source_kind="google_pubsub",
185+
source_config={
186+
"project_id": "<your gcp project id>",
187+
"subscription_id": "<your subscription id>",
188+
"credentials_json": "<your credentials json>"
189+
},
190+
sink_kind="webhook",
191+
sink_config={
192+
"url": "<webhook url>",
193+
"method": "<GET | POST | PUT | PATCH | DELETE>",
194+
"headers": [{"header1": "header1_value"}]
195+
}
196+
).create()
197+
```
198+
199+
## delete
200+
201+
The Pipeline object has a delete method to delete a pipeline
202+
203+
### Example Usage
204+
205+
```python
206+
from glassflow import Pipeline
207+
208+
pipeline = Pipeline(
209+
name="<your pipeline name>",
210+
transformation_file="path/to/transformation.py",
211+
space_id="<your space id>",
212+
personal_access_token="<your personal access token>"
213+
).create()
214+
215+
pipeline.delete()
216+
```
85217

86218
## Quickstart
87219

‎USAGE.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
```python
22
import glassflow
33

4-
client = glassflow.GlassFlowClient()
5-
pipeline = client.pipeline_client(pipeline_id="<pipeline_id>", pipeline_access_token="<pipeline_token>")
4+
source = glassflow.PipelineDataSource(pipeline_id="<pipeline_id>", pipeline_access_token="<pipeline_token>")
65
data = {
76
"name": "Hello World",
87
"id": 1
98
}
10-
res = pipeline.publish(request_body=data)
9+
source_res = source.publish(request_body=data)
10+
11+
sink = glassflow.PipelineDataSink(pipeline_id="<pipeline_id>", pipeline_access_token="<pipeline_token>")
12+
sink_res = sink.consume()
1113
```

0 commit comments

Comments
 (0)