Skip to content

Commit 045a69b

Browse files
PabloPardoGarciaPabloPardoGarcia
authored
Merge pull request #25 from glassflow/dev
Release 2.0.0
2 parents 72d5a3d + 4fc3331 commit 045a69b

36 files changed

+1811
-713
lines changed

‎.github/workflows/on_pr.yaml

Lines changed: 56 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@ on:
44
pull_request:
55
branches:
66
- main
7+
- dev
8+
9+
permissions:
10+
contents: write
11+
checks: write
12+
pull-requests: write
713

814
jobs:
915
tests:
@@ -24,11 +30,18 @@ jobs:
2430
- name: Install dependencies
2531
run: pip install -e .[dev]
2632

27-
- name: Run Integration Tests
28-
run: pytest tests/glassflow/integration_tests/
33+
- name: Run Tests
34+
run: pytest
2935
env:
30-
PIPELINE_ID: ${{ secrets.PIPELINE_ID }}
31-
PIPELINE_ACCESS_TOKEN: ${{ secrets.PIPELINE_ACCESS_TOKEN }}
36+
PIPELINE_ID: ${{ secrets.INTEGRATION_PIPELINE_ID }}
37+
PIPELINE_ACCESS_TOKEN: ${{ secrets.INTEGRATION_PIPELINE_ACCESS_TOKEN }}
38+
PERSONAL_ACCESS_TOKEN: ${{ secrets.INTEGRATION_PERSONAL_ACCESS_TOKEN }}
39+
40+
- name: Upload coverage report
41+
uses: actions/upload-artifact@master
42+
with:
43+
name: coverageReport
44+
path: tests/reports/coverage.xml
3245

3346
checks:
3447
name: Run code checks
@@ -48,8 +61,43 @@ jobs:
4861
- name: Install dependencies
4962
run: pip install -e .[dev]
5063

51-
- name: Run isort
52-
run: isort .
53-
54-
- name: Run linter and code formatter
64+
- name: Run ruff linter checks
5565
run: ruff check .
66+
67+
- name: Run ruff formatting checks
68+
run: ruff format --check .
69+
70+
coverage:
71+
runs-on: ubuntu-latest
72+
needs: [tests]
73+
steps:
74+
- uses: actions/checkout@v3
75+
with:
76+
persist-credentials: false # otherwise, the token used is the GITHUB_TOKEN, instead of your personal token
77+
fetch-depth: 0 # otherwise, you will fail to push refs to dest repo
78+
79+
- name: Download coverage report
80+
uses: actions/download-artifact@master
81+
with:
82+
name: coverageReport
83+
path: tests/reports/coverage.xml
84+
85+
- name: Pytest coverage comment
86+
if: ${{ github.ref == 'refs/heads/main' }}
87+
id: coverageComment
88+
uses: MishaKav/pytest-coverage-comment@main
89+
with:
90+
hide-comment: true
91+
pytest-xml-coverage-path: ./tests/reports/coverage.xml
92+
93+
- name: Update Readme with Coverage Html
94+
if: ${{ github.ref == 'refs/heads/main' }}
95+
run: |
96+
sed -i '/<!-- Pytest Coverage Comment:Begin -->/,/<!-- Pytest Coverage Comment:End -->/c\<!-- Pytest Coverage Comment:Begin -->\n\${{ steps.coverageComment.outputs.coverageHtml }}\n<!-- Pytest Coverage Comment:End -->' ./README.md
97+
98+
- name: Commit & Push changes to Readme
99+
if: ${{ github.ref == 'refs/heads/main' }}
100+
uses: actions-js/push@master
101+
with:
102+
message: Update coverage on Readme
103+
github_token: ${{ secrets.GITHUB_TOKEN }}

‎.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,6 @@ site**
1010
.pypirc
1111
dist/
1212
build
13-
.env
13+
.env
14+
tests/reports
15+
.coverage

‎README.md

Lines changed: 33 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
<img src="https://img.shields.io/badge/slack-join-community?logo=slack&amp;logoColor=white&amp;style=flat"
99
alt="Chat on Slack"></a>
1010
<a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff" style="max-width:100%;"></a>
11+
<!-- Pytest Coverage Comment:Begin -->
12+
<!-- Pytest Coverage Comment:End -->
1113

1214

1315
# GlassFlow Python SDK
@@ -27,6 +29,7 @@ pip install glassflow
2729
* [publish](#publish) - Publish a new event into the pipeline
2830
* [consume](#consume) - Consume the transformed event from the pipeline
2931
* [consume failed](#consume-failed) - Consume the events that failed from the pipeline
32+
* [validate credentials](#validate-credentials) - Validate pipeline credentials
3033

3134

3235
## publish
@@ -38,16 +41,16 @@ Publish a new event into the pipeline
3841
```python
3942
import glassflow
4043

41-
client = glassflow.GlassFlowClient()
42-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
44+
pipeline_source = glassflow.PipelineDataSource(pipeline_id="<str value", pipeline_access_token="<str token>")
4345
data = {} # your json event
44-
res = pipeline_client.publish(request_body=data)
46+
res = pipeline_source.publish(request_body=data)
4547

4648
if res.status_code == 200:
4749
print("Published sucessfully")
4850
```
4951

5052

53+
5154
## consume
5255

5356
Consume the transformed event from the pipeline
@@ -57,14 +60,14 @@ Consume the transformed event from the pipeline
5760
```python
5861
import glassflow
5962

60-
client = glassflow.GlassFlowClient()
61-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
62-
res = pipeline_client.consume()
63+
pipeline_sink = glassflow.PipelineDataSink(pipeline_id="<str value", pipeline_access_token="<str value>")
64+
res = pipeline_sink.consume()
6365

6466
if res.status_code == 200:
65-
print(res.body.event)
67+
print(res.json())
6668
```
6769

70+
6871
## consume failed
6972

7073
If the transformation failed for any event, they are available in a failed queue. You can consume those events from the pipeline
@@ -74,12 +77,31 @@ If the transformation failed for any event, they are available in a failed queue
7477
```python
7578
import glassflow
7679

77-
client = glassflow.GlassFlowClient()
78-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
79-
res = pipeline_client.consume_failed()
80+
pipeline_sink = glassflow.PipelineDataSink(pipeline_id="<str value", pipeline_access_token="<str value>")
81+
res = pipeline_sink.consume_failed()
8082

8183
if res.status_code == 200:
82-
print(res.body.event)
84+
print(res.json())
85+
```
86+
87+
## validate credentials
88+
89+
Validate pipeline credentials (`pipeline_id` and `pipeline_access_token`) from source or sink
90+
91+
### Example Usage
92+
93+
```python
94+
import glassflow
95+
96+
try:
97+
pipeline_source = glassflow.PipelineDataSource(pipeline_id="<str value", pipeline_access_token="<str value>")
98+
pipeline_source.validate_credentials()
99+
except glassflow.errors.PipelineNotFoundError as e:
100+
print("Pipeline ID does not exist!")
101+
raise e
102+
except glassflow.errors.PipelineAccessTokenInvalidError as e:
103+
print("Pipeline Access Token is invalid!")
104+
raise e
83105
```
84106

85107

‎USAGE.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
```python
22
import glassflow
33

4-
client = glassflow.GlassFlowClient()
5-
pipeline = client.pipeline_client(pipeline_id="<pipeline_id>", pipeline_access_token="<pipeline_token>")
4+
source = glassflow.PipelineDataSource(pipeline_id="<pipeline_id>", pipeline_access_token="<pipeline_token>")
65
data = {
76
"name": "Hello World",
87
"id": 1
98
}
10-
res = pipeline.publish(request_body=data)
9+
source_res = source.publish(request_body=data)
10+
11+
sink = glassflow.PipelineDataSink(pipeline_id="<pipeline_id>", pipeline_access_token="<pipeline_token>")
12+
sink_res = sink.consume()
1113
```

‎docs/index.md

Lines changed: 23 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,7 @@ pip install glassflow
1616
* [publish](#publish) - Publish a new event into the pipeline
1717
* [consume](#consume) - Consume the transformed event from the pipeline
1818
* [consume failed](#consume-failed) - Consume the events that failed from the pipeline
19-
* [is access token valid](#is-access-token-valid) - Validates Pipeline Access Token
20-
* [is_valid](#is-valid) - Check if pipeline credentials are valid
19+
* [validate credentials](#validate-credentials) - Validate pipeline credentials
2120

2221

2322
## publish
@@ -27,12 +26,11 @@ Publish a new event into the pipeline
2726
### Example Usage
2827

2928
```python
30-
import glassflow
29+
from glassflow import PipelineDataSource
3130

32-
client = glassflow.GlassFlowClient()
33-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str token>")
31+
source = PipelineDataSource(pipeline_id="<str value", pipeline_access_token="<str token>")
3432
data = {} # your json event
35-
res = pipeline_client.publish(request_body=data)
33+
res = source.publish(request_body=data)
3634

3735
if res.status_code == 200:
3836
print("Published sucessfully")
@@ -47,11 +45,10 @@ Consume the transformed event from the pipeline
4745
### Example Usage
4846

4947
```python
50-
import glassflow
48+
from glassflow import PipelineDataSink
5149

52-
client = glassflow.GlassFlowClient()
53-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
54-
res = pipeline_client.consume()
50+
sink = PipelineDataSink(pipeline_id="<str value", pipeline_access_token="<str value>")
51+
res = sink.consume()
5552

5653
if res.status_code == 200:
5754
print(res.json())
@@ -65,45 +62,33 @@ If the transformation failed for any event, they are available in a failed queue
6562
### Example Usage
6663

6764
```python
68-
import glassflow
65+
from glassflow import PipelineDataSink
6966

70-
client = glassflow.GlassFlowClient()
71-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
72-
res = pipeline_client.consume_failed()
67+
sink = PipelineDataSink(pipeline_id="<str value", pipeline_access_token="<str value>")
68+
res = sink.consume_failed()
7369

7470
if res.status_code == 200:
7571
print(res.json())
7672
```
7773

78-
## is access token valid
74+
## validate credentials
7975

80-
Check if the access token provided is valid
76+
Validate pipeline credentials (`pipeline_id` and `pipeline_access_token`) from source or sink
8177

8278
### Example Usage
8379

8480
```python
85-
import glassflow
86-
87-
client = glassflow.GlassFlowClient()
88-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
89-
90-
if pipeline_client.is_access_token_valid():
91-
print("Pipeline Access Token invalid! Generate a new one from the Webapp!")
92-
```
93-
94-
## is valid
95-
Checks if the pipeline credentials are valid.
96-
97-
### Example Usage
98-
99-
```python
100-
import glassflow
101-
102-
client = glassflow.GlassFlowClient()
103-
pipeline_client = client.pipeline_client(pipeline_id="<str value", pipeline_access_token="<str value>")
104-
105-
if pipeline_client.is_valid():
106-
print("Pipeline credentials are valid")
81+
from glassflow import PipelineDataSource, errors
82+
83+
try:
84+
source = PipelineDataSource(pipeline_id="<str value", pipeline_access_token="<str value>")
85+
source.validate_credentials()
86+
except errors.PipelineNotFoundError as e:
87+
print("Pipeline ID does not exist!")
88+
raise e
89+
except errors.PipelineAccessTokenInvalidError as e:
90+
print("Pipeline Access Token is invalid!")
91+
raise e
10792
```
10893

10994
## SDK Maturity

‎makefile

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
2+
generate-api-data-models:
3+
datamodel-codegen \
4+
--url https://api.glassflow.dev/v1/openapi.yaml \
5+
--output ./src/glassflow/models/api/api.py
6+
sed -i '' -e '1s/^/# ruff: noqa\n/' ./src/glassflow/models/api/api.py

‎pyproject.toml

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,45 @@
11
[build-system]
22
requires = ["setuptools>=61.0"]
3-
build-backend = "setuptools.build_meta"
3+
build-backend = "setuptools.build_meta"
4+
5+
[tool.pytest.ini_options]
6+
addopts = """
7+
--import-mode=importlib \
8+
--cov=src/glassflow \
9+
--cov-report xml:tests/reports/coverage.xml \
10+
-ra -q
11+
"""
12+
testpaths = [
13+
"tests",
14+
]
15+
16+
[tool.ruff.lint]
17+
select = [
18+
# pycodestyle
19+
"E",
20+
# Pyflakes
21+
"F",
22+
# pyupgrade
23+
"UP",
24+
# flake8-bugbear
25+
"B",
26+
# flake8-simplify
27+
"SIM",
28+
# isort
29+
"I",
30+
]
31+
32+
[tool.ruff.lint.pydocstyle]
33+
convention = "google"
34+
35+
[tool.datamodel-codegen]
36+
field-constraints = true
37+
snake-case-field = true
38+
strip-default-none = false
39+
target-python-version = "3.7"
40+
use-title-as-name = true
41+
disable-timestamp = true
42+
enable-version-header = true
43+
use-double-quotes = true
44+
input-file-type = "openapi"
45+
output-model-type = "dataclasses.dataclass"

‎setup.py

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import setuptools
22

33
try:
4-
with open("README.md", "r") as fh:
4+
with open("README.md") as fh:
55
long_description = fh.read()
66
except FileNotFoundError:
77
long_description = ""
@@ -36,7 +36,15 @@
3636
"python-dotenv==1.0.1",
3737
],
3838
extras_require={
39-
"dev": ["pylint==2.16.2", "pytest==8.3.2", "isort==5.13.2", "ruff==0.6.3"]
39+
"dev": [
40+
"pylint==2.16.2",
41+
"pytest==8.3.2",
42+
"pytest-cov==5.0.0",
43+
"datamodel-code-generator[http]==0.26.0",
44+
"requests-mock==1.12.1",
45+
"isort==5.13.2",
46+
"ruff==0.6.3",
47+
]
4048
},
4149
package_dir={"": "src"},
4250
python_requires=">=3.8",

‎src/glassflow/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,5 @@
11
from .client import GlassFlowClient as GlassFlowClient
22
from .config import GlassFlowConfig as GlassFlowConfig
3+
from .models import errors as errors
4+
from .pipeline_data import PipelineDataSink as PipelineDataSink
5+
from .pipeline_data import PipelineDataSource as PipelineDataSource

0 commit comments

Comments
 (0)