Skip to content

Commit b15f21b

Browse files
Update documentation (#770)
Co-authored-by: daniil-quix <133032822+daniil-quix@users.noreply.github.com>
1 parent 664557e commit b15f21b

File tree

5 files changed

+1222
-508
lines changed

5 files changed

+1222
-508
lines changed

docs/api-reference/dataframe.md

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
class StreamingDataFrame()
1111
```
1212

13-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L76)
13+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L80)
1414

1515
`StreamingDataFrame` is the main object you will use for ETL work.
1616

@@ -83,7 +83,7 @@ def apply(func: Union[
8383
metadata: bool = False) -> Self
8484
```
8585

86-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L191)
86+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L195)
8787

8888
Apply a function to transform the value and return a new value.
8989

@@ -141,7 +141,7 @@ def update(func: Union[
141141
metadata: bool = False) -> Self
142142
```
143143

144-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L298)
144+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L302)
145145

146146
Apply a function to mutate value in-place or to perform a side effect
147147

@@ -209,7 +209,7 @@ def filter(func: Union[
209209
metadata: bool = False) -> Self
210210
```
211211

212-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L404)
212+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L408)
213213

214214
Filter value using provided function.
215215

@@ -261,7 +261,7 @@ def group_by(key: Union[str, Callable[[Any], Any]],
261261
key_serializer: Optional[SerializerType] = "json") -> Self
262262
```
263263

264-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L492)
264+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L496)
265265

266266
"Groups" messages by re-keying them via the provided group_by operation
267267

@@ -325,7 +325,7 @@ a clone with this operation added (assign to keep its effect).
325325
def contains(key: str) -> StreamingSeries
326326
```
327327

328-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L570)
328+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L574)
329329

330330
Check if the key is present in the Row value.
331331

@@ -364,7 +364,7 @@ or False otherwise.
364364
def to_topic(topic: Topic, key: Optional[Callable[[Any], Any]] = None) -> Self
365365
```
366366

367-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L595)
367+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L599)
368368

369369
Produce current value to a topic. You can optionally specify a new key.
370370

@@ -417,7 +417,7 @@ the updated StreamingDataFrame instance (reassignment NOT required).
417417
def set_timestamp(func: Callable[[Any, Any, int, Any], int]) -> Self
418418
```
419419

420-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L640)
420+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L644)
421421

422422
Set a new timestamp based on the current message value and its metadata.
423423

@@ -469,7 +469,7 @@ def set_headers(
469469
]) -> Self
470470
```
471471

472-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L681)
472+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L685)
473473

474474
Set new message headers based on the current message value and metadata.
475475

@@ -518,7 +518,7 @@ a new StreamingDataFrame instance
518518
def print(pretty: bool = True, metadata: bool = False) -> Self
519519
```
520520

521-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L732)
521+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L736)
522522

523523
Print out the current message value (and optionally, the message metadata) to
524524

@@ -579,7 +579,7 @@ def print_table(size: int = 5,
579579
column_widths: Optional[dict[str, int]] = None) -> Self
580580
```
581581

582-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L776)
582+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L780)
583583

584584
Print a table with the most recent records.
585585

@@ -672,7 +672,7 @@ sdf.print_table(size=5, title="Live Records", slowdown=1)
672672
def compose(sink: Optional[VoidExecutor] = None) -> dict[str, VoidExecutor]
673673
```
674674

675-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L890)
675+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L896)
676676

677677
Compose all functions of this StreamingDataFrame into one big closure.
678678

@@ -726,7 +726,7 @@ def test(value: Any,
726726
topic: Optional[Topic] = None) -> List[Any]
727727
```
728728

729-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L924)
729+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L930)
730730

731731
A shorthand to test `StreamingDataFrame` with provided value
732732

@@ -766,7 +766,7 @@ def tumbling_window(
766766
) -> TumblingTimeWindowDefinition
767767
```
768768

769-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L961)
769+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L967)
770770

771771
Create a time-based tumbling window transformation on this StreamingDataFrame.
772772

@@ -856,7 +856,7 @@ def tumbling_count_window(
856856
name: Optional[str] = None) -> TumblingCountWindowDefinition
857857
```
858858

859-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1048)
859+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1054)
860860

861861
Create a count-based tumbling window transformation on this StreamingDataFrame.
862862

@@ -923,7 +923,7 @@ def hopping_window(
923923
) -> HoppingTimeWindowDefinition
924924
```
925925

926-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1092)
926+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1098)
927927

928928
Create a time-based hopping window transformation on this StreamingDataFrame.
929929

@@ -1023,7 +1023,7 @@ def hopping_count_window(
10231023
name: Optional[str] = None) -> HoppingCountWindowDefinition
10241024
```
10251025

1026-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1192)
1026+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1198)
10271027

10281028
Create a count-based hopping window transformation on this StreamingDataFrame.
10291029

@@ -1095,7 +1095,7 @@ def sliding_window(
10951095
) -> SlidingTimeWindowDefinition
10961096
```
10971097

1098-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1243)
1098+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1249)
10991099

11001100
Create a time-based sliding window transformation on this StreamingDataFrame.
11011101

@@ -1190,7 +1190,7 @@ def sliding_count_window(
11901190
name: Optional[str] = None) -> SlidingCountWindowDefinition
11911191
```
11921192

1193-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1335)
1193+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1341)
11941194

11951195
Create a count-based sliding window transformation on this StreamingDataFrame.
11961196

@@ -1255,7 +1255,7 @@ def drop(columns: Union[str, List[str]],
12551255
errors: Literal["ignore", "raise"] = "raise") -> Self
12561256
```
12571257

1258-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1382)
1258+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1388)
12591259

12601260
Drop column(s) from the message value (value must support `del`, like a dict).
12611261

@@ -1299,7 +1299,7 @@ a new StreamingDataFrame instance
12991299
def sink(sink: BaseSink)
13001300
```
13011301

1302-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1426)
1302+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1432)
13031303

13041304
Sink the processed data to the specified destination.
13051305

docs/api-reference/kafka.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
class Producer()
1111
```
1212

13-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L42)
13+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L45)
1414

1515
<a id="quixstreams.kafka.producer.Producer.__init__"></a>
1616

@@ -26,7 +26,7 @@ def __init__(broker_address: Union[str, ConnectionConfig],
2626
flush_timeout: Optional[float] = None)
2727
```
2828

29-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L43)
29+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L46)
3030

3131
A wrapper around `confluent_kafka.Producer`.
3232

@@ -61,12 +61,12 @@ def produce(topic: str,
6161
headers: Optional[Headers] = None,
6262
partition: Optional[int] = None,
6363
timestamp: Optional[int] = None,
64-
poll_timeout: float = 5.0,
65-
buffer_error_max_tries: int = 3,
64+
poll_timeout: float = PRODUCER_POLL_TIMEOUT,
65+
buffer_error_max_tries: int = PRODUCER_ON_ERROR_RETRIES,
6666
on_delivery: Optional[DeliveryCallback] = None)
6767
```
6868

69-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L81)
69+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L84)
7070

7171
Produce a message to a topic.
7272

@@ -101,7 +101,7 @@ for the produced message.
101101
def poll(timeout: float = 0)
102102
```
103103

104-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L142)
104+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L145)
105105

106106
Polls the producer for events and calls `on_delivery` callbacks.
107107

@@ -122,7 +122,7 @@ Polls the producer for events and calls `on_delivery` callbacks.
122122
def flush(timeout: Optional[float] = None) -> int
123123
```
124124

125-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L150)
125+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L153)
126126

127127
Wait for all messages in the Producer queue to be delivered.
128128

@@ -147,7 +147,7 @@ number of messages remaining to flush
147147
class TransactionalProducer(Producer)
148148
```
149149

150-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L181)
150+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/kafka/producer.py#L184)
151151

152152
A separate producer class used only internally for transactions
153153
(transactions are only needed when using a consumer).

0 commit comments

Comments
 (0)