Skip to content

Commit 72f658a

Browse files
authored
remove column name deserialization (#392)
* remove everything related to generating column_names during deserialization * remove doc references to column_name deserialization args
1 parent 835ef27 commit 72f658a

File tree

16 files changed

+4385
-4550
lines changed

16 files changed

+4385
-4550
lines changed

docs/api-reference/application.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
class Application()
1111
```
1212

13-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L55)
13+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L55)
1414

1515
The main Application class.
1616

@@ -82,7 +82,7 @@ def __init__(broker_address: Optional[Union[str, ConnectionConfig]] = None,
8282
topic_create_timeout: float = 60)
8383
```
8484

85-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L93)
85+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L93)
8686

8787

8888
<br>
@@ -180,7 +180,7 @@ def Quix(cls,
180180
topic_create_timeout: float = 60) -> Self
181181
```
182182

183-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L313)
183+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L313)
184184

185185
>***NOTE:*** DEPRECATED: use Application with `quix_sdk_token` argument instead.
186186

@@ -290,7 +290,7 @@ def topic(name: str,
290290
timestamp_extractor: Optional[TimestampExtractor] = None) -> Topic
291291
```
292292

293-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L451)
293+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L451)
294294

295295
Create a topic definition.
296296

@@ -371,7 +371,7 @@ topic = app.topic("input-topic", timestamp_extractor=custom_ts_extractor)
371371
def dataframe(topic: Topic) -> StreamingDataFrame
372372
```
373373

374-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L531)
374+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L531)
375375

376376
A simple helper method that generates a `StreamingDataFrame`, which is used
377377

@@ -421,7 +421,7 @@ to be used as an input topic.
421421
def stop(fail: bool = False)
422422
```
423423

424-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L570)
424+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L570)
425425

426426
Stop the internal poll loop and the message processing.
427427

@@ -448,7 +448,7 @@ to unhandled exception, and it shouldn't commit the current checkpoint.
448448
def get_producer() -> Producer
449449
```
450450

451-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L593)
451+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L593)
452452

453453
Create and return a pre-configured Producer instance.
454454
The Producer is initialized with params passed to Application.
@@ -483,7 +483,7 @@ with app.get_producer() as producer:
483483
def get_consumer(auto_commit_enable: bool = True) -> Consumer
484484
```
485485

486-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L623)
486+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L623)
487487

488488
Create and return a pre-configured Consumer instance.
489489
The Consumer is initialized with params passed to Application.
@@ -528,7 +528,7 @@ with app.get_consumer() as consumer:
528528
def clear_state()
529529
```
530530

531-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L666)
531+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L666)
532532

533533
Clear the state of the application.
534534

@@ -542,7 +542,7 @@ Clear the state of the application.
542542
def run(dataframe: StreamingDataFrame)
543543
```
544544

545-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/app.py#L672)
545+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/app.py#L672)
546546

547547
Start processing data from Kafka using provided `StreamingDataFrame`
548548

docs/api-reference/context.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
def set_message_context(context: Optional[MessageContext])
1313
```
1414

15-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/context.py#L20)
15+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/context.py#L20)
1616

1717
Set a MessageContext for the current message in the given `contextvars.Context`
1818

@@ -55,7 +55,7 @@ sdf = sdf.update(lambda value: alter_context(value))
5555
def message_context() -> MessageContext
5656
```
5757

58-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/context.py#L51)
58+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/context.py#L51)
5959

6060
Get a MessageContext for the current message, which houses most of the message
6161

docs/api-reference/dataframe.md

Lines changed: 26 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
class StreamingDataFrame(BaseStreaming)
1111
```
1212

13-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L62)
13+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L62)
1414

1515
`StreamingDataFrame` is the main object you will use for ETL work.
1616

@@ -81,7 +81,7 @@ def apply(func: Union[
8181
metadata: bool = False) -> Self
8282
```
8383

84-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L177)
84+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L177)
8585

8686
Apply a function to transform the value and return a new value.
8787

@@ -139,7 +139,7 @@ def update(func: Union[
139139
metadata: bool = False) -> Self
140140
```
141141

142-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L266)
142+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L266)
143143

144144
Apply a function to mutate value in-place or to perform a side effect
145145

@@ -197,7 +197,7 @@ def filter(func: Union[
197197
metadata: bool = False) -> Self
198198
```
199199

200-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L354)
200+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L354)
201201

202202
Filter value using provided function.
203203

@@ -249,7 +249,7 @@ def group_by(key: Union[str, Callable[[Any], Any]],
249249
key_serializer: Optional[SerializerType] = "json") -> Self
250250
```
251251

252-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L440)
252+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L440)
253253

254254
"Groups" messages by re-keying them via the provided group_by operation
255255

@@ -314,7 +314,7 @@ a clone with this operation added (assign to keep its effect).
314314
def contains(key: str) -> StreamingSeries
315315
```
316316

317-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L518)
317+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L518)
318318

319319
Check if the key is present in the Row value.
320320

@@ -353,7 +353,7 @@ or False otherwise.
353353
def to_topic(topic: Topic, key: Optional[Callable[[Any], Any]] = None) -> Self
354354
```
355355

356-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L543)
356+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L543)
357357

358358
Produce current value to a topic. You can optionally specify a new key.
359359

@@ -396,7 +396,7 @@ By default, the current message key will be used.
396396
def set_timestamp(func: Callable[[Any, Any, int, Any], int]) -> Self
397397
```
398398

399-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L584)
399+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L584)
400400

401401
Set a new timestamp based on the current message value and its metadata.
402402

@@ -449,7 +449,7 @@ def set_headers(
449449
) -> Self
450450
```
451451

452-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L625)
452+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L625)
453453

454454
Set new message headers based on the current message value and metadata.
455455

@@ -500,7 +500,7 @@ def compose(
500500
) -> Dict[str, VoidExecutor]
501501
```
502502

503-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L676)
503+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L676)
504504

505505
Compose all functions of this StreamingDataFrame into one big closure.
506506

@@ -554,7 +554,7 @@ def test(value: Any,
554554
topic: Optional[Topic] = None) -> List[Any]
555555
```
556556

557-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L713)
557+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L713)
558558

559559
A shorthand to test `StreamingDataFrame` with provided value
560560

@@ -591,7 +591,7 @@ def tumbling_window(duration_ms: Union[int, timedelta],
591591
name: Optional[str] = None) -> TumblingWindowDefinition
592592
```
593593

594-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L750)
594+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L750)
595595

596596
Create a tumbling window transformation on this StreamingDataFrame.
597597

@@ -677,7 +677,7 @@ def hopping_window(duration_ms: Union[int, timedelta],
677677
name: Optional[str] = None) -> HoppingWindowDefinition
678678
```
679679

680-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/dataframe.py#L826)
680+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/dataframe.py#L826)
681681

682682
Create a hopping window transformation on this StreamingDataFrame.
683683

@@ -771,7 +771,7 @@ sdf = (
771771
class StreamingSeries(BaseStreaming)
772772
```
773773

774-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L47)
774+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L47)
775775

776776
`StreamingSeries` are typically generated by `StreamingDataframes` when getting
777777
elements from, or performing certain operations on, a `StreamingDataframe`,
@@ -837,7 +837,7 @@ sdf = sdf[["column_a"] & (sdf["new_sum_field"] >= 10)]
837837
def from_apply_callback(cls, func: ApplyWithMetadataCallback) -> Self
838838
```
839839

840-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L107)
840+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L107)
841841

842842
Create a StreamingSeries from a function.
843843

@@ -865,7 +865,7 @@ instance of `StreamingSeries`
865865
def apply(func: ApplyCallback) -> Self
866866
```
867867

868-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L121)
868+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L121)
869869

870870
Add a callable to the execution list for this series.
871871

@@ -917,7 +917,7 @@ a new `StreamingSeries` with the new callable added
917917
def compose_returning() -> ReturningExecutor
918918
```
919919

920-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L155)
920+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L155)
921921

922922
Compose a list of functions from this StreamingSeries and its parents into one
923923

@@ -948,7 +948,7 @@ def compose(
948948
None]] = None) -> VoidExecutor
949949
```
950950

951-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L170)
951+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L170)
952952

953953
Compose all functions of this StreamingSeries into one big closure.
954954

@@ -1006,7 +1006,7 @@ def test(value: Any,
10061006
ctx: Optional[MessageContext] = None) -> Any
10071007
```
10081008

1009-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L214)
1009+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L214)
10101010

10111011
A shorthand to test `StreamingSeries` with provided value
10121012

@@ -1038,7 +1038,7 @@ result of `StreamingSeries`
10381038
def isin(other: Container) -> Self
10391039
```
10401040

1041-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L269)
1041+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L269)
10421042

10431043
Check if series value is in "other".
10441044

@@ -1083,7 +1083,7 @@ new StreamingSeries
10831083
def contains(other: Union[Self, object]) -> Self
10841084
```
10851085

1086-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L296)
1086+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L296)
10871087

10881088
Check if series value contains "other"
10891089

@@ -1128,7 +1128,7 @@ new StreamingSeries
11281128
def is_(other: Union[Self, object]) -> Self
11291129
```
11301130

1131-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L321)
1131+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L321)
11321132

11331133
Check if series value refers to the same object as `other`
11341134

@@ -1170,7 +1170,7 @@ new StreamingSeries
11701170
def isnot(other: Union[Self, object]) -> Self
11711171
```
11721172

1173-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L344)
1173+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L344)
11741174

11751175
Check if series value does not refer to the same object as `other`
11761176

@@ -1213,7 +1213,7 @@ new StreamingSeries
12131213
def isnull() -> Self
12141214
```
12151215

1216-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L368)
1216+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L368)
12171217

12181218
Check if series value is None.
12191219

@@ -1250,7 +1250,7 @@ new StreamingSeries
12501250
def notnull() -> Self
12511251
```
12521252

1253-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L391)
1253+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L391)
12541254

12551255
Check if series value is not None.
12561256

@@ -1287,7 +1287,7 @@ new StreamingSeries
12871287
def abs() -> Self
12881288
```
12891289

1290-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/51c8064d2623b13b3e11c5acbb33409643f66f3c/quixstreams/dataframe/series.py#L414)
1290+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/ea3d07177df3f11deb3c51e8337534408f5f68c1/quixstreams/dataframe/series.py#L414)
12911291

12921292
Get absolute value of the series value.
12931293

0 commit comments

Comments
 (0)