Skip to content

Commit d43ace3

Browse files
Update documentation (#860)
Co-authored-by: daniil-quix <133032822+daniil-quix@users.noreply.github.com>
1 parent c51cdad commit d43ace3

File tree

3 files changed

+739
-135
lines changed

3 files changed

+739
-135
lines changed

docs/api-reference/application.md

Lines changed: 22 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
class Application()
1111
```
1212

13-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L89)
13+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L94)
1414

1515
The main Application class.
1616

@@ -82,10 +82,11 @@ def __init__(broker_address: Optional[Union[str, ConnectionConfig]] = None,
8282
topic_manager: Optional[TopicManager] = None,
8383
request_timeout: float = 30,
8484
topic_create_timeout: float = 60,
85-
processing_guarantee: ProcessingGuarantee = "at-least-once")
85+
processing_guarantee: ProcessingGuarantee = "at-least-once",
86+
max_partition_buffer_size: int = 10000)
8687
```
8788

88-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L127)
89+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L132)
8990

9091

9192
<br>
@@ -148,6 +149,12 @@ Default - `True`
148149
- `request_timeout`: timeout (seconds) for REST-based requests
149150
- `topic_create_timeout`: timeout (seconds) for topic create finalization
150151
- `processing_guarantee`: Use "exactly-once" or "at-least-once" processing.
152+
- `max_partition_buffer_size`: the maximum number of messages to buffer per topic partition to consider it full.
153+
The buffering is used to consume messages in-order between multiple partitions with the same number.
154+
It is a soft limit, and the actual number of buffered messages can be up to x2 higher.
155+
Lower value decreases the memory use, but increases the latency.
156+
Default - `10000`.
157+
151158
<br><br>***Error Handlers***<br>
152159
To handle errors, `Application` accepts callbacks triggered when
153160
exceptions occur on different stages of stream processing. If the callback
@@ -175,7 +182,7 @@ instead of the default one.
175182
def Quix(cls, *args, **kwargs)
176183
```
177184

178-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L374)
185+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L389)
179186

180187
RAISES EXCEPTION: DEPRECATED.
181188

@@ -198,7 +205,7 @@ def topic(name: str,
198205
timestamp_extractor: Optional[TimestampExtractor] = None) -> Topic
199206
```
200207

201-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L406)
208+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L421)
202209

203210
Create a topic definition.
204211

@@ -280,7 +287,7 @@ def dataframe(topic: Optional[Topic] = None,
280287
source: Optional[BaseSource] = None) -> StreamingDataFrame
281288
```
282289

283-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L486)
290+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L501)
284291

285292
A simple helper method that generates a `StreamingDataFrame`, which is used
286293

@@ -336,7 +343,7 @@ to be used as an input topic.
336343
def stop(fail: bool = False)
337344
```
338345

339-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L542)
346+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L557)
340347

341348
Stop the internal poll loop and the message processing.
342349

@@ -363,7 +370,7 @@ to unhandled exception, and it shouldn't commit the current checkpoint.
363370
def get_producer() -> Producer
364371
```
365372

366-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L587)
373+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L602)
367374

368375
Create and return a pre-configured Producer instance.
369376
The Producer is initialized with params passed to Application.
@@ -398,7 +405,7 @@ with app.get_producer() as producer:
398405
def get_consumer(auto_commit_enable: bool = True) -> Consumer
399406
```
400407

401-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L641)
408+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L657)
402409

403410
Create and return a pre-configured Consumer instance.
404411

@@ -455,7 +462,7 @@ with app.get_consumer() as consumer:
455462
def clear_state()
456463
```
457464

458-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L690)
465+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L706)
459466

460467
Clear the state of the application.
461468

@@ -469,7 +476,7 @@ Clear the state of the application.
469476
def add_source(source: BaseSource, topic: Optional[Topic] = None) -> Topic
470477
```
471478

472-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L696)
479+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L712)
473480

474481
Add a source to the application.
475482

@@ -498,7 +505,7 @@ def run(dataframe: Optional[StreamingDataFrame] = None,
498505
count: int = 0)
499506
```
500507

501-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L729)
508+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L745)
502509

503510
Start processing data from Kafka using provided `StreamingDataFrame`
504511

@@ -565,7 +572,7 @@ Default = 0 (infinite)
565572
class ApplicationConfig(BaseSettings)
566573
```
567574

568-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1070)
575+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1089)
569576

570577
Immutable object holding the application configuration
571578

@@ -588,7 +595,7 @@ def settings_customise_sources(
588595
) -> Tuple[PydanticBaseSettingsSource, ...]
589596
```
590597

591-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1105)
598+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1125)
592599

593600
Included to ignore reading/setting values from the environment
594601

@@ -602,7 +609,7 @@ Included to ignore reading/setting values from the environment
602609
def copy(**kwargs) -> "ApplicationConfig"
603610
```
604611

605-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1118)
612+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1138)
606613

607614
Update the application config and return a copy
608615

docs/api-reference/kafka.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -819,3 +819,31 @@ def consumer_group_metadata() -> GroupMetadata
819819

820820
Used by the producer during consumer offset sending for an EOS transaction.
821821

822+
<a id="quixstreams.kafka.consumer.BaseConsumer.consume"></a>
823+
824+
<br><br>
825+
826+
#### BaseConsumer.consume
827+
828+
```python
829+
def consume(
830+
num_messages: int = 1,
831+
timeout: Optional[float] = None
832+
) -> list[RawConfluentKafkaMessageProto]
833+
```
834+
835+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/consumer.py#L583)
836+
837+
Consumes a list of messages (possibly empty on timeout).
838+
839+
Callbacks may be executed as a side effect of calling this method.
840+
841+
842+
<br>
843+
***Arguments:***
844+
845+
- `num_messages`: The maximum number of messages to return.
846+
Default: `1`.
847+
- `timeout`: The maximum time in seconds to block waiting for message, event or callback.
848+
Default: `None` (infinite).
849+

0 commit comments

Comments
 (0)