Skip to content

Commit 6181921

Browse files
Update documentation (#435)
Co-authored-by: daniil-quix <133032822+daniil-quix@users.noreply.github.com>
1 parent 1b37ede commit 6181921

File tree

3 files changed

+31
-3
lines changed

3 files changed

+31
-3
lines changed

docs/api-reference/application.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ def __init__(broker_address: Optional[Union[str, ConnectionConfig]] = None,
6363
consumer_group: Optional[str] = None,
6464
auto_offset_reset: AutoOffsetReset = "latest",
6565
commit_interval: float = 5.0,
66+
commit_every: int = 0,
6667
consumer_extra_config: Optional[dict] = None,
6768
producer_extra_config: Optional[dict] = None,
6869
state_dir: str = "state",
@@ -109,6 +110,15 @@ Default - "quixstreams-default" (set during init)
109110
>***NOTE:*** Quix Applications will prefix it with the Quix workspace id.
110111
- `commit_interval`: How often to commit the processed messages in seconds.
111112
Default - 5.0.
113+
- `commit_every`: Commit the checkpoint after processing N messages.
114+
Use this parameter for more granular control of the commit schedule.
115+
If the value is > 0, the application will commit the checkpoint after
116+
processing the specified number of messages across all the assigned
117+
partitions.
118+
If the value is <= 0, only the `commit_interval` will be considered.
119+
Default - 0.
120+
>***NOTE:*** Only input offsets are counted, and the application
121+
> may produce more results than the number of incoming messages.
112122
- `auto_offset_reset`: Consumer `auto.offset.reset` setting
113123
- `consumer_extra_config`: A dictionary with additional options that
114124
will be passed to `confluent_kafka.Consumer` as is.

docs/api-reference/dataframe.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -840,7 +840,8 @@ sdf = (
840840
#### StreamingDataFrame.drop
841841

842842
```python
843-
def drop(columns: Union[str, List[str]]) -> Self
843+
def drop(columns: Union[str, List[str]],
844+
errors: Literal["ignore", "raise"] = "raise") -> Self
844845
```
845846

846847
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L969)
@@ -868,6 +869,8 @@ sdf.drop(["x", "y"])
868869
***Arguments:***
869870

870871
- `columns`: a single column name or a list of names, where names are `str`
872+
- `errors`: If "ignore", suppress error and only existing labels are dropped.
873+
Default - `"raise"`.
871874

872875

873876
<br>

docs/api-reference/quixstreams.md

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1344,7 +1344,8 @@ like `sum`, `count`, etc. and applied to the StreamingDataFrame.
13441344
#### StreamingDataFrame.drop
13451345

13461346
```python
1347-
def drop(columns: Union[str, List[str]]) -> Self
1347+
def drop(columns: Union[str, List[str]],
1348+
errors: Literal["ignore", "raise"] = "raise") -> Self
13481349
```
13491350

13501351
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L969)
@@ -1368,6 +1369,8 @@ sdf.drop(["x", "y"])
13681369
**Arguments**:
13691370

13701371
- `columns`: a single column name or a list of names, where names are `str`
1372+
- `errors`: If "ignore", suppress error and only existing labels are dropped.
1373+
Default - `"raise"`.
13711374

13721375
**Returns**:
13731376

@@ -7770,6 +7773,7 @@ def __init__(broker_address: Optional[Union[str, ConnectionConfig]] = None,
77707773
consumer_group: Optional[str] = None,
77717774
auto_offset_reset: AutoOffsetReset = "latest",
77727775
commit_interval: float = 5.0,
7776+
commit_every: int = 0,
77737777
consumer_extra_config: Optional[dict] = None,
77747778
producer_extra_config: Optional[dict] = None,
77757779
state_dir: str = "state",
@@ -7814,6 +7818,15 @@ Default - "quixstreams-default" (set during init)
78147818
>***NOTE:*** Quix Applications will prefix it with the Quix workspace id.
78157819
- `commit_interval`: How often to commit the processed messages in seconds.
78167820
Default - 5.0.
7821+
- `commit_every`: Commit the checkpoint after processing N messages.
7822+
Use this parameter for more granular control of the commit schedule.
7823+
If the value is > 0, the application will commit the checkpoint after
7824+
processing the specified number of messages across all the assigned
7825+
partitions.
7826+
If the value is <= 0, only the `commit_interval` will be considered.
7827+
Default - 0.
7828+
>***NOTE:*** Only input offsets are counted, and the application
7829+
> may produce more results than the number of incoming messages.
78177830
- `auto_offset_reset`: Consumer `auto.offset.reset` setting
78187831
- `consumer_extra_config`: A dictionary with additional options that
78197832
will be passed to `confluent_kafka.Consumer` as is.
@@ -8388,7 +8401,9 @@ def expired() -> bool
83888401

83898402
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/checkpointing/checkpoint.py#L57)
83908403

8391-
Returns `True` if checkpoint deadline has expired.
8404+
Returns `True` if checkpoint deadline has expired OR
8405+
if the total number of processed offsets exceeded the "commit_every" limit
8406+
when it's defined.
83928407

83938408
<a id="quixstreams.checkpointing.checkpoint.Checkpoint.empty"></a>
83948409

0 commit comments

Comments
 (0)