Skip to content

Commit 4f99868

Browse files
Update documentation (#796)
Co-authored-by: daniil-quix <133032822+daniil-quix@users.noreply.github.com>
1 parent f2c9027 commit 4f99868

File tree

2 files changed

+276
-39
lines changed

2 files changed

+276
-39
lines changed

docs/api-reference/application.md

Lines changed: 52 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
class Application()
1111
```
1212

13-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L80)
13+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L90)
1414

1515
The main Application class.
1616

@@ -85,7 +85,7 @@ def __init__(broker_address: Optional[Union[str, ConnectionConfig]] = None,
8585
processing_guarantee: ProcessingGuarantee = "at-least-once")
8686
```
8787

88-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L118)
88+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L128)
8989

9090

9191
<br>
@@ -174,7 +174,7 @@ instead of the default one.
174174
def Quix(cls, *args, **kwargs)
175175
```
176176

177-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L364)
177+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L375)
178178

179179
RAISES EXCEPTION: DEPRECATED.
180180

@@ -197,7 +197,7 @@ def topic(name: str,
197197
timestamp_extractor: Optional[TimestampExtractor] = None) -> Topic
198198
```
199199

200-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L396)
200+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L407)
201201

202202
Create a topic definition.
203203

@@ -279,7 +279,7 @@ def dataframe(topic: Optional[Topic] = None,
279279
source: Optional[BaseSource] = None) -> StreamingDataFrame
280280
```
281281

282-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L476)
282+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L487)
283283

284284
A simple helper method that generates a `StreamingDataFrame`, which is used
285285

@@ -335,7 +335,7 @@ to be used as an input topic.
335335
def stop(fail: bool = False)
336336
```
337337

338-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L532)
338+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L543)
339339

340340
Stop the internal poll loop and the message processing.
341341

@@ -362,7 +362,7 @@ to unhandled exception, and it shouldn't commit the current checkpoint.
362362
def get_producer() -> Producer
363363
```
364364

365-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L577)
365+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L588)
366366

367367
Create and return a pre-configured Producer instance.
368368
The Producer is initialized with params passed to Application.
@@ -397,7 +397,7 @@ with app.get_producer() as producer:
397397
def get_consumer(auto_commit_enable: bool = True) -> Consumer
398398
```
399399

400-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L632)
400+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L643)
401401

402402
Create and return a pre-configured Consumer instance.
403403

@@ -454,7 +454,7 @@ with app.get_consumer() as consumer:
454454
def clear_state()
455455
```
456456

457-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L682)
457+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L693)
458458

459459
Clear the state of the application.
460460

@@ -468,7 +468,7 @@ Clear the state of the application.
468468
def add_source(source: BaseSource, topic: Optional[Topic] = None) -> Topic
469469
```
470470

471-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L688)
471+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L699)
472472

473473
Add a source to the application.
474474

@@ -492,16 +492,41 @@ Note: the names of default topics are prefixed with "source__".
492492
#### Application.run
493493

494494
```python
495-
def run(dataframe: Optional[StreamingDataFrame] = None)
495+
def run(dataframe: Optional[StreamingDataFrame] = None,
496+
timeout: float = 0.0,
497+
count: int = 0)
496498
```
497499

498-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L721)
500+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L732)
499501

500502
Start processing data from Kafka using provided `StreamingDataFrame`
501503

502504
Once started, it can be safely terminated with a `SIGTERM` signal
503505
(like Kubernetes does) or a typical `KeyboardInterrupt` (`Ctrl+C`).
504506

507+
Alternatively, stop conditions can be set (typically for debugging purposes);
508+
has the option of stopping after a number of messages, timeout, or both.
509+
510+
Not setting a timeout or count limit will result in the Application running
511+
indefinitely (expected production behavior).
512+
513+
514+
Stop Condition Details:
515+
516+
A timeout will immediately stop an Application once no new messages have
517+
been consumed after T seconds (after rebalance and recovery).
518+
519+
A count will process N total records from ANY input/SDF topics (so
520+
multiple input topics will very likely differ in their consume total!) after
521+
an initial rebalance and recovery.
522+
THEN, any remaining processes from things such as groupby (which uses internal
523+
topics) will also be validated to ensure the results of said messages are
524+
fully processed (this does NOT count towards the process total).
525+
Note that without a timeout, the Application runs until the count is hit.
526+
527+
If timeout and count are used together (which is the recommended pattern for
528+
debugging), either condition will trigger a stop.
529+
505530

506531

507532
<br>
@@ -518,9 +543,19 @@ topic = app.topic('test-topic')
518543
df = app.dataframe(topic)
519544
df.apply(lambda value, context: print('New message', value)
520545

521-
app.run()
546+
app.run() # could pass `timeout=5` here, for example
522547
```
523548

549+
550+
<br>
551+
***Arguments:***
552+
553+
- `dataframe`: DEPRECATED - do not use; sdfs are now automatically tracked.
554+
- `timeout`: maximum time to wait for a new message.
555+
Default = 0.0 (infinite)
556+
- `count`: how many input topic messages to process before stopping.
557+
Default = 0 (infinite)
558+
524559
<a id="quixstreams.app.Application.setup_topics"></a>
525560

526561
<br><br>
@@ -531,7 +566,7 @@ app.run()
531566
def setup_topics()
532567
```
533568

534-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L846)
569+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L910)
535570

536571
Validate the application topics
537572

@@ -543,7 +578,7 @@ Validate the application topics
543578
class ApplicationConfig(BaseSettings)
544579
```
545580

546-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1014)
581+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1081)
547582

548583
Immutable object holding the application configuration
549584

@@ -566,7 +601,7 @@ def settings_customise_sources(
566601
) -> Tuple[PydanticBaseSettingsSource, ...]
567602
```
568603

569-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1049)
604+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1116)
570605

571606
Included to ignore reading/setting values from the environment
572607

@@ -580,7 +615,7 @@ Included to ignore reading/setting values from the environment
580615
def copy(**kwargs) -> Self
581616
```
582617

583-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1062)
618+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1129)
584619

585620
Update the application config and return a copy
586621

0 commit comments

Comments
 (0)