|
| 1 | +# Contributing new Connectors |
| 2 | + |
| 3 | +Quix Streams provide APIs to build custom Sources and Sinks. |
| 4 | + |
| 5 | +In this guide, we will explain how to contribute a new Source or Sink to Quix Streams |
| 6 | +and share your work with the community. |
| 7 | + |
| 8 | +## Where to start |
| 9 | +Before submitting a new connector, we encourage you to try out Quix Streams |
| 10 | +and build an application with it. |
| 11 | + |
| 12 | +It will help you to better understand how the library works. |
| 13 | + |
| 14 | +Check out our [Tutorials](../tutorials/README.md) to get started. |
| 15 | + |
| 16 | +## Steps |
| 17 | + |
| 18 | +1. Check the [list of GitHub issues with a `connector` label](https://github.com/quixio/quix-streams/labels/connector) to see if a request for the connector has already been submitted. |
| 19 | +2. If it doesn't exist, create a new one named `"New {source/sink}: {name}"`, and describe your idea. |
| 20 | +3. At this point, you can either ask for help from the Quix Streams maintainers, or implement it yourself and contribute it to the library. |
| 21 | +If you want to develop it yourself, proceed to the next section. |
| 22 | + |
| 23 | + |
| 24 | +### Contributing a new Sink or Source |
| 25 | + |
| 26 | +1. Fork [the Quix Streams repository](https://github.com/quixio/quix-streams/). |
| 27 | + |
| 28 | +2. Clone the repo locally. |
| 29 | + |
| 30 | +3. Create a branch for your connector as a new feature in the format `{feature}/{connector-name}` (e.g. `feature/mqtt-source`) |
| 31 | + |
| 32 | +4. Read ["Creating a Custom Source"](sources/custom-sources.md) and ["Creating a Custom Sink"](sinks/custom-sinks.md) to learn how to build a connector. |
| 33 | + |
| 34 | +5. Create a new Python module in `quixstreams/sources/community/{connector-name}` for Sources or `quixstreams/sinks/community/{connector-name}` for Sinks. |
| 35 | + |
| 36 | +6. In this module, add a new class for your connector following the format `"{Technology}Sink"` or `"{Technology}Source"` (e.g. `MQTTSource` , `PostgreSQLSink` ). |
| 37 | +Read the [Best Practices](#best-practices-for-the-connectors-code) section on how to structure your code and test it. |
| 38 | + |
| 39 | +7. Test your changes locally in some sandbox environment to make sure it works. |
| 40 | + |
| 41 | +8. Commit your changes and push to your fork. |
| 42 | + |
| 43 | +9. Create a Pull Request to Quix Streams `main` branch. |
| 44 | +See [How to describe your PR](#how-to-describe-your-pr) section to learn how to structure the PR. |
| 45 | + |
| 46 | +10. Wait for feedback from a Quix Streams maintainer and adjust the code if necessary. |
| 47 | + |
| 48 | +11. Feel free to join the `#contributors` channel in [the community Slack](https://quix.io/slack-invite) and announce your work! |
| 49 | + |
| 50 | +12. After the PR is accepted, the maintainers will take care of releasing it in the next version of Quix Streams. |
| 51 | + |
| 52 | + |
| 53 | +## Best practices for the connector's code |
| 54 | + |
| 55 | +### Adding external libraries |
| 56 | +In many cases, the connector needs an external library to work. |
| 57 | + |
| 58 | +These libraries are considered optional, and they should not be required for the default |
| 59 | +`quixstreams` installation. |
| 60 | + |
| 61 | +When using an external library: |
| 62 | + |
| 63 | +1. Add new optional dependency to `pyproject.toml` in the `[project.optional-dependencies]` |
| 64 | +section. |
| 65 | +For example, for MQTTSource you need to add `mqtt = [paho-mqtt>=2.1.0,<3]`. |
| 66 | +2. Add a dependency to `tests/requirements.txt` for the tests to run. |
| 67 | +3. To provide nice exception messages for users, wrap the library import in try-except in the connector's code like this: |
| 68 | + |
| 69 | +```python |
| 70 | +try: |
| 71 | + import influxdb_client_3 |
| 72 | +except ImportError as exc: |
| 73 | + raise ImportError( |
| 74 | + 'Package "influxdb3-python" is missing: ' |
| 75 | + "run pip install quixstreams[influxdb3] to fix it" |
| 76 | + ) from exc |
| 77 | +``` |
| 78 | + |
| 79 | + |
| 80 | +### Writing docstrings |
| 81 | +Add a docstring to your connector class briefly describing how it works and covering |
| 82 | +parameters it accepts. |
| 83 | + |
| 84 | +Look at [InfluxDB3Sink](../api-reference/sinks.md#influxdb3sink) as an example. |
| 85 | + |
| 86 | +### Writing tests |
| 87 | +Add unit tests to the `tests/test_quixstreams/test_{sources/sinks}/test_community/test_{connector-name}` folder. |
| 88 | +Use tests for existing connectors in `tests/test_quixstreams/test_{sources/sinks}/test_core/*` folders as examples. |
| 89 | + |
| 90 | + |
| 91 | +## How to describe your PR |
| 92 | +To streamline the review process, describe your PR in the following format |
| 93 | + |
| 94 | +- Link the PR to the previously created issue. |
| 95 | +- Describe new libraries used by the connector. |
| 96 | +- Briefly describe how the connector works and provide some context for how to use it. |
| 97 | +- Describe the processing guarantees the connector provides (e.g. at-least-once, at-most-once, exactly-once). |
| 98 | +- Add examples how to instantiate the connector class. |
| 99 | +- Explain how to set up a sandbox environment for Quix Streams maintainers to test your PR. |
0 commit comments