Skip to content

Commit defe2dc

Browse files
Update documentation (#566)
Co-authored-by: daniil-quix <133032822+daniil-quix@users.noreply.github.com>
1 parent 7cb0a1b commit defe2dc

File tree

2 files changed

+184
-146
lines changed

2 files changed

+184
-146
lines changed

docs/api-reference/quixstreams.md

Lines changed: 92 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -9098,38 +9098,58 @@ This is the base class for all sources.
90989098
Sources are executed in a sub-process of the main application.
90999099

91009100
To create your own source you need to implement:
9101-
* `run`
9102-
* `stop`
9103-
* `default_topic`
9101+
9102+
* `start`
9103+
* `stop`
9104+
* `default_topic`
91049105

91059106
`BaseSource` is the most basic interface, and the framework expects every
9106-
sources to implement it. Use `Source` to benefit from a base implementation.
9107+
source to implement it.
9108+
Use `Source` to benefit from a base implementation.
91079109

91089110
You can connect a source to a StreamingDataframe using the Application.
91099111

91109112
Example snippet:
91119113

91129114
```python
9113-
from quixstreams import Application
9114-
from quixstreams.sources import Source
9115+
class RandomNumbersSource(BaseSource):
9116+
def __init__(self):
9117+
super().__init__()
9118+
self._running = False
9119+
9120+
def start(self):
9121+
self._running = True
9122+
9123+
while self._running:
9124+
number = random.randint(0, 100)
9125+
serialized = self._producer_topic.serialize(value=number)
9126+
self._producer.produce(
9127+
topic=self._producer_topic.name,
9128+
key=serialized.key,
9129+
value=serialized.value,
9130+
)
9131+
9132+
def stop(self):
9133+
self._running = False
9134+
9135+
def default_topic(self) -> Topic:
9136+
return Topic(
9137+
name="topic-name",
9138+
value_deserializer="json",
9139+
value_serializer="json",
9140+
)
91159141

9116-
class MySource(Source):
9117-
def run(self):
9118-
for _ in range(1000):
9119-
self.produce(
9120-
key="foo",
9121-
value=b"foo"
9122-
)
91239142

91249143
def main():
9125-
app = Application()
9126-
source = MySource(name="my_source")
9144+
app = Application(broker_address="localhost:9092")
9145+
source = RandomNumbersSource()
91279146

91289147
sdf = app.dataframe(source=source)
91299148
sdf.print(metadata=True)
91309149

91319150
app.run()
91329151

9152+
91339153
if __name__ == "__main__":
91349154
main()
91359155
```
@@ -9142,7 +9162,7 @@ if __name__ == "__main__":
91429162
def configure(topic: Topic, producer: RowProducer) -> None
91439163
```
91449164

9145-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L74)
9165+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L94)
91469166

91479167
This method is triggered when the source is registered to the Application.
91489168

@@ -9157,7 +9177,7 @@ It configures the source's Kafka producer and the topic it will produce to.
91579177
def start() -> None
91589178
```
91599179

9160-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L93)
9180+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L113)
91619181

91629182
This method is triggered in the subprocess when the source is started.
91639183

@@ -9173,29 +9193,12 @@ Use it to fetch data and produce it to Kafka.
91739193
def stop() -> None
91749194
```
91759195

9176-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L102)
9196+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L122)
91779197

91789198
This method is triggered when the application is shutting down.
91799199

91809200
The source must ensure that the `run` method is completed soon.
91819201

9182-
Example Snippet:
9183-
9184-
```python
9185-
class MySource(BaseSource):
9186-
def start(self):
9187-
self._running = True
9188-
while self._running:
9189-
self._producer.produce(
9190-
topic=self._producer_topic,
9191-
value="foo",
9192-
)
9193-
time.sleep(1)
9194-
9195-
def stop(self):
9196-
self._running = False
9197-
```
9198-
91999202
<a id="quixstreams.sources.base.source.BaseSource.default_topic"></a>
92009203

92019204
#### BaseSource.default\_topic
@@ -9205,7 +9208,7 @@ class MySource(BaseSource):
92059208
def default_topic() -> Topic
92069209
```
92079210

9208-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L127)
9211+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L130)
92099212

92109213
This method is triggered when the topic is not provided to the source.
92119214

@@ -9219,24 +9222,53 @@ The source must return a default topic configuration.
92199222
class Source(BaseSource)
92209223
```
92219224

9222-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L135)
9225+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L138)
9226+
9227+
A base class for custom Sources that provides a basic implementation of `BaseSource`
9228+
interface.
9229+
It is recommended to interface to create custom sources.
92239230

9224-
BaseSource class implementation providing
9231+
Subclass it and implement the `run` method to fetch data and produce it to Kafka.
92259232

9226-
Implementation for the abstract method:
9227-
* `default_topic`
9228-
* `start`
9229-
* `stop`
9233+
**Example**:
92309234

9231-
Helper methods
9232-
* serialize
9233-
* produce
9234-
* flush
9235+
9236+
```python
9237+
from quixstreams import Application
9238+
import random
92359239

9236-
Helper property
9237-
* running
9240+
from quixstreams.sources import Source
92389241

9239-
Subclass it and implement the `run` method to fetch data and produce it to Kafka.
9242+
9243+
class RandomNumbersSource(Source):
9244+
def run(self):
9245+
while self.running:
9246+
number = random.randint(0, 100)
9247+
serialized = self._producer_topic.serialize(value=number)
9248+
self.produce(key=str(number), value=serialized.value)
9249+
9250+
9251+
def main():
9252+
app = Application(broker_address="localhost:9092")
9253+
source = RandomNumbersSource(name="random-source")
9254+
9255+
sdf = app.dataframe(source=source)
9256+
sdf.print(metadata=True)
9257+
9258+
app.run()
9259+
9260+
9261+
if __name__ == "__main__":
9262+
main()
9263+
```
9264+
9265+
9266+
Helper methods and properties:
9267+
9268+
* `serialize()`
9269+
* `produce()`
9270+
* `flush()`
9271+
* `running`
92409272

92419273
<a id="quixstreams.sources.base.source.Source.__init__"></a>
92429274

@@ -9246,7 +9278,7 @@ Subclass it and implement the `run` method to fetch data and produce it to Kafka
92469278
def __init__(name: str, shutdown_timeout: float = 10) -> None
92479279
```
92489280

9249-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L155)
9281+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L186)
92509282

92519283
**Arguments**:
92529284

@@ -9262,24 +9294,12 @@ def __init__(name: str, shutdown_timeout: float = 10) -> None
92629294
def running() -> bool
92639295
```
92649296

9265-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L169)
9297+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L200)
92669298

92679299
Property indicating if the source is running.
92689300

92699301
The `stop` method will set it to `False`. Use it to stop the source gracefully.
92709302

9271-
Example snippet:
9272-
9273-
```python
9274-
class MySource(Source):
9275-
def run(self):
9276-
while self.running:
9277-
self.produce(
9278-
value="foo",
9279-
)
9280-
time.sleep(1)
9281-
```
9282-
92839303
<a id="quixstreams.sources.base.source.Source.cleanup"></a>
92849304

92859305
#### Source.cleanup
@@ -9288,13 +9308,13 @@ class MySource(Source):
92889308
def cleanup(failed: bool) -> None
92899309
```
92909310

9291-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L189)
9311+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L208)
92929312

92939313
This method is triggered once the `run` method completes.
92949314

92959315
Use it to clean up the resources and shut down the source gracefully.
92969316

9297-
It flush the producer when `_run` completes successfully.
9317+
It flushes the producer when `_run` completes successfully.
92989318

92999319
<a id="quixstreams.sources.base.source.Source.stop"></a>
93009320

@@ -9304,7 +9324,7 @@ It flush the producer when `_run` completes successfully.
93049324
def stop() -> None
93059325
```
93069326

9307-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L200)
9327+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L219)
93089328

93099329
This method is triggered when the application is shutting down.
93109330

@@ -9318,7 +9338,7 @@ It sets the `running` property to `False`.
93189338
def start()
93199339
```
93209340

9321-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L209)
9341+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L228)
93229342

93239343
This method is triggered in the subprocess when the source is started.
93249344

@@ -9333,7 +9353,7 @@ It marks the source as running, execute it's run method and ensure cleanup happe
93339353
def run()
93349354
```
93359355

9336-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L225)
9356+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L244)
93379357

93389358
This method is triggered in the subprocess when the source is started.
93399359

@@ -9351,7 +9371,7 @@ def serialize(key: Optional[object] = None,
93519371
timestamp_ms: Optional[int] = None) -> KafkaMessage
93529372
```
93539373

9354-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L233)
9374+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L252)
93559375

93569376
Serialize data to bytes using the producer topic serializers and return a `quixstreams.models.messages.KafkaMessage`.
93579377

@@ -9373,7 +9393,7 @@ def produce(value: Optional[Union[str, bytes]] = None,
93739393
buffer_error_max_tries: int = 3) -> None
93749394
```
93759395

9376-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L249)
9396+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L268)
93779397

93789398
Produce a message to the configured source topic in Kafka.
93799399

@@ -9385,7 +9405,7 @@ Produce a message to the configured source topic in Kafka.
93859405
def flush(timeout: Optional[float] = None) -> None
93869406
```
93879407

9388-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L274)
9408+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L293)
93899409

93909410
This method flush the producer.
93919411

@@ -9408,7 +9428,7 @@ None use producer default or -1 is infinite. Default: None
94089428
def default_topic() -> Topic
94099429
```
94109430

9411-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L292)
9431+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L311)
94129432

94139433
Return a default topic matching the source name.
94149434

0 commit comments

Comments
 (0)