Skip to content

Commit 899334c

Browse files
mfiedorowiczmfiedorowicz
authored
🚚 release (#364)
2 parents ffd1223 + 0f62ba7 commit 899334c

39 files changed

+925
-197
lines changed

‎.github/workflows/configs/helm-ct.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
validate-maintainers: false
2+
skip-helm-dependencies: false

‎.github/workflows/helm-lint.yaml

Lines changed: 40 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
name: Helm - lint
2+
23
on:
34
push:
45
branches:
@@ -20,16 +21,44 @@ permissions:
2021
jobs:
2122
helm-lint:
2223
runs-on: ubuntu-latest
23-
timeout-minutes: 10
24-
defaults:
25-
run:
26-
working-directory: charts
2724
steps:
2825
- name: Checkout
29-
uses: actions/checkout@v4
30-
- name: Setup Helm
31-
uses: azure/setup-helm@fe7b79cd5ee1e45176fcad797de68ecaf3ca4814 # v4
32-
- name: Update helm dependencies
33-
run: helm dependency update diode
34-
- name: Run helm lint
35-
run: helm lint diode
26+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # 4.2.2
27+
with:
28+
fetch-depth: 0
29+
30+
- name: Set up Helm
31+
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
32+
with:
33+
version: v3.17.0
34+
35+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
36+
with:
37+
python-version: '3.x'
38+
check-latest: true
39+
40+
- name: Extract Helm repo dependencies from Chart.yaml
41+
id: extract_repos
42+
uses: mikefarah/yq@b534aa9ee5d38001fba3cd8fe254a037e4847b37 # v4.45.4
43+
with:
44+
cmd: yq -o=tsv '.dependencies[] | [.name, .repository] | @tsv' charts/diode/Chart.yaml
45+
- name: Add Helm repos from dependencies
46+
run: |
47+
echo "${{ steps.extract_repos.outputs.result }}" | while IFS=$'\t' read -r name repo; do
48+
helm repo add "$name" "$repo"
49+
done
50+
51+
- name: Set up chart-testing
52+
uses: helm/chart-testing-action@0d28d3144d3a25ea2cc349d6e59901c4ff469b3b # v2.7.0
53+
54+
- name: Run chart-testing (list-changed)
55+
id: list-changed
56+
run: |
57+
changed=$(ct list-changed --target-branch release)
58+
if [[ -n "$changed" ]]; then
59+
echo "changed=true" >> "$GITHUB_OUTPUT"
60+
fi
61+
62+
- name: Run chart-testing (lint)
63+
if: steps.list-changed.outputs.changed == 'true'
64+
run: ct lint --config .github/workflows/configs/helm-ct.yaml --target-branch release

‎GET_STARTED.md

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
# Getting Started with Diode
2+
3+
This guide will help you set up and start using Diode to ingest data into NetBox.
4+
5+
## Prerequisites
6+
7+
Before you begin, ensure you have:
8+
9+
- NetBox version 4.2.3 or later
10+
- Docker version 27.0.3 or newer
11+
- bash 4.x or newer
12+
- jq
13+
14+
## Installation Steps
15+
16+
### Deploy Diode server
17+
18+
> **Host**: These steps should be performed on the host where you want to run the Diode server.
19+
20+
> **Note**: For the complete installation instructions, please refer to the [official Diode Server documentation](https://github.com/netboxlabs/diode/tree/develop/diode-server).
21+
22+
We provide a `quickstart.sh` script to automate the setup process. The script will download and configure all necessary files:
23+
24+
- `docker-compose.yaml` — Defines Diode server containers
25+
- `.env` — Environment settings for customization
26+
- `nginx.conf` — Nginx configuration for routing Diode endpoints
27+
- `client-credentials.json` — Defines OAuth2 clients for secure communication
28+
29+
1. Create a working directory:
30+
```bash
31+
mkdir /opt/diode
32+
cd /opt/diode
33+
```
34+
35+
2. Download and prepare the quickstart script:
36+
```bash
37+
curl -sSfLo quickstart.sh https://raw.githubusercontent.com/netboxlabs/diode/release/diode-server/docker/scripts/quickstart.sh
38+
chmod +x quickstart.sh
39+
```
40+
41+
3. Run the script with your NetBox server address:
42+
```bash
43+
./quickstart.sh https://<netbox-server>
44+
```
45+
This should have created an `.env` file for your environment.
46+
47+
4. Start the Diode server:
48+
```bash
49+
docker compose up -d
50+
```
51+
5. Extract the `netbox-to-diode` client secret. This will be needed for the Diode NetBox plugin installation:
52+
```bash
53+
echo $(jq -r '.[] | select(.client_id == "netbox-to-diode") | .client_secret' /opt/diode/oauth2/client/client-credentials.json)
54+
```
55+
> **Note**: This will return a credential that will be used by the Diode NetBox plugin to connect to the Diode server. Store it safely.
56+
57+
### Install Diode NetBox Plugin
58+
59+
> **Host**: These steps should be performed on the host where NetBox is installed.
60+
61+
> **Note**: For the complete installation instructions, please refer to the [official Diode NetBox Plugin documentation](https://github.com/netboxlabs/diode-netbox-plugin/blob/develop/README.md).
62+
63+
1. **Source the NetBox Python Virtual Environment**
64+
```bash
65+
cd /opt/netbox
66+
source venv/bin/activate
67+
```
68+
69+
2. **Install the Plugin Package**
70+
```bash
71+
pip install netboxlabs-diode-netbox-plugin
72+
```
73+
74+
3. **Configure NetBox Settings**
75+
Add the following to your `configuration.py`:
76+
```python
77+
PLUGINS = [
78+
"netbox_diode_plugin",
79+
]
80+
81+
PLUGINS_CONFIG = {
82+
"netbox_diode_plugin": {
83+
# Diode gRPC target for communication with Diode server
84+
"diode_target_override": "grpc://<diode-server:port>/diode",
85+
# NetBox username associated with changes applied via plugin
86+
"diode_username": "diode",
87+
# netbox-to-diode client secret from earlier step
88+
"netbox_to_diode_client_secret": "<netbox-to-diode-secret>"
89+
},
90+
}
91+
```
92+
93+
4. **Apply Database Migrations**
94+
```bash
95+
cd /opt/netbox/netbox
96+
./manage.py migrate netbox_diode_plugin
97+
```
98+
99+
5. **Restart NetBox Services**
100+
```bash
101+
sudo systemctl restart netbox netbox-rq
102+
```
103+
104+
6. **Generate Diode Client Credentials**
105+
> **Note**: These credentials will be used by the Orb agent to send discovery results to NetBox via Diode.
106+
107+
1. Go to your NetBox instance (https://<netbox-server>)
108+
2. In the left-hand pane, navigate to **Diode -> Client Credentials**
109+
3. Click on **+ Add a Credential**
110+
4. For Client Name, enter any name and click **Create**
111+
5. **IMPORTANT**: Copy the _Client ID_ and _Client Secret_ and save them securely
112+
6. Click **Return to List**
113+
114+
You have now created your Diode client credentials. These will be used as environment variables when running the Orb agent.
115+
116+
### Ingest Data with Orb Agent
117+
118+
> **Host**: These steps should be performed on the host where you want to run the Orb agent for network discovery.
119+
120+
> **Note**: For the complete installation instructions, please refer to the [official Orb Agent documentation](https://github.com/netboxlabs/orb-agent).
121+
122+
1. **Export Client Credentials**
123+
```bash
124+
# Export the client credentials you generated in NetBox
125+
export DIODE_CLIENT_ID="<your-client-id>"
126+
export DIODE_CLIENT_SECRET="<your-client-secret>"
127+
```
128+
129+
2. **Create Agent Configuration File**
130+
Create an `agent.yaml` file with the following content:
131+
```yaml
132+
orb:
133+
config_manager:
134+
active: local
135+
backends:
136+
network_discovery: # Enable network discovery backend
137+
common:
138+
diode:
139+
target: grpc://<diode-server:port>/diode
140+
client_id: ${DIODE_CLIENT_ID}
141+
client_secret: ${DIODE_CLIENT_SECRET}
142+
agent_name: my_agent
143+
policies:
144+
network_discovery:
145+
loopback_policy:
146+
config:
147+
scope:
148+
targets:
149+
- 127.0.0.1
150+
```
151+
152+
3. **Run the Agent**
153+
154+
Using host network mode (recommended):
155+
```bash
156+
docker run --net=host \
157+
-v $(pwd):/opt/orb/ \
158+
-e DIODE_CLIENT_ID \
159+
-e DIODE_CLIENT_SECRET \
160+
netboxlabs/orb-agent:latest run -c /opt/orb/agent.yaml
161+
```
162+
163+
Alternative using root user:
164+
```bash
165+
docker run -u root \
166+
-v $(pwd):/opt/orb/ \
167+
-e DIODE_CLIENT_ID \
168+
-e DIODE_CLIENT_SECRET \
169+
netboxlabs/orb-agent:latest run -c /opt/orb/agent.yaml
170+
```
171+
172+
> **Note**: The container needs sufficient permissions to send ICMP and TCP packets. This can be achieved either by:
173+
> - Setting the network mode to `host` (recommended)
174+
> - Running the container as root user
175+
176+
4. **Verify Agent Operation**
177+
- Check the agent logs for successful startup
178+
- Verify data appears in NetBox
179+
180+
## Troubleshooting
181+
182+
### Common Issues
183+
184+
1. **Connection Issues**
185+
- Verify network connectivity between Diode and NetBox
186+
- Check firewall rules
187+
- Validate URLs and ports
188+
189+
### Getting Help
190+
191+
If you encounter issues:
192+
193+
1. Search GitHub: [Issues](https://github.com/netboxlabs/diode/issues)
194+
2. Find us in Slack: [NetDev Community #orb](https://https://netdev-community.slack.com/)

‎README.md

Lines changed: 42 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -1,56 +1,55 @@
11
# Diode
22

3-
Diode is a NetBox data ingestion service that greatly simplifies and enhances the process to add and update network data
4-
in NetBox, ensuring your network source of truth is always accurate and can be trusted to power your network automation
5-
pipelines. Our guiding principle in designing Diode has been to make it as easy as possible to get data into NetBox,
6-
removing as much burden as possible from the user while shifting that effort to technology.
7-
8-
To achieve this, Diode sits in front of NetBox and provides an API purpose built for ingestion of complex network data.
9-
Diode eliminates the need to preprocess data to make it conform to the strict object hierarchy imposed by the NetBox
10-
data model. This allows data to be sent to NetBox in a more freeform manner, in blocks that are intuitive for network
11-
engineers (such as by device or by interface) with much of the related information treated as attributes or properties
12-
of these components of interest. Then, Diode takes care of the heavy lifting, automatically transforming the data to
13-
align it with NetBox’s structured and comprehensive data model. Diode can even create placeholder objects to compensate
14-
for missing information, which means even fragmented information about the network can be captured in NetBox.
15-
16-
## Project status
17-
18-
The Diode project is currently in the _Public Preview_ stage. Please
19-
see [NetBox Labs Product and Feature Lifecycle](https://netboxlabs.com/docs/console/product_feature_lifecycle/) for more
20-
details. We actively welcome feedback to help identify and prioritize bugs, new features and areas of improvement.
21-
22-
## Get started
23-
24-
Diode runs as a sidecar service to NetBox and can run anywhere with network connectivity to NetBox, whether on the same
25-
host or elsewhere. The overall Diode service is delivered through three main components (and a fourth optional
26-
component):
27-
28-
1. Diode plugin - see how to [install the Diode plugin](https://github.com/netboxlabs/diode-netbox-plugin)
29-
2. Diode server - see how
30-
to [run the Diode server](https://github.com/netboxlabs/diode/tree/develop/diode-server#readme)
31-
3. Diode SDK - see how
32-
to [install the Diode Python client SDK](https://github.com/netboxlabs/diode-sdk-python), [download Diode Python script examples](https://github.com/netboxlabs/netbox-learning/tree/develop/diode)
33-
and [use the Diode SDK Go](https://github.com/netboxlabs/diode-sdk-go)
34-
4. Diode agent (optional) - see how
35-
to [install and run the Diode NAPALM discovery agent](https://github.com/netboxlabs/diode-agent/tree/develop/diode-napalm-agent)
3+
Diode is a data ingestion service for NetBox that greatly simplifies and enhances the process of adding and updating data in NetBox, ensuring your network source of truth is always accurate and up to date. Our guiding principle in designing Diode has been to make it as easy as possible to get data into NetBox, removing as much burden as possible from the user while shifting that effort to technology.
4+
5+
## Project Status
6+
7+
The Diode project is currently in the _Public Preview_ stage. Please see [NetBox Labs Product and Feature Lifecycle](https://netboxlabs.com/docs/console/product_feature_lifecycle/) for more details. We actively welcome feedback to help identify and prioritize bugs, new features and areas of improvement.
8+
9+
## Prerequisites
10+
11+
- NetBox 4.2.3 or later
12+
- Python 3.8 or later (for Python SDK)
13+
- Go 1.18 or later (for Go SDK)
14+
- Network connectivity between Diode and NetBox
15+
16+
## Quick Start
17+
18+
For a quick step-by-step guide, see our [Getting Started Guide](./GET_STARTED.md).
19+
20+
1. **Deploy the Diode Server**
21+
See [deployment instructions](https://github.com/netboxlabs/diode/blob/develop/diode-server/README.md)
22+
23+
2. **Install the Diode NetBox Plugin**
24+
See [installation instructions](https://github.com/netboxlabs/diode-netbox-plugin/blob/develop/README.md)
25+
26+
3. **Choose Your Data Ingestion Method**
27+
- Use the NetBox Discovery agent for automated network discovery: see [instructions to run Orb agent](https://github.com/netboxlabs/orb-agent)
28+
- Build custom integrations using our SDKs:
29+
- [Python SDK](https://github.com/netboxlabs/diode-sdk-python)
30+
- [Go SDK](https://github.com/netboxlabs/diode-sdk-go)
31+
32+
## Documentation
33+
34+
- [Diode Protocol Documentation](https://github.com/netboxlabs/diode/blob/develop/docs/diode-proto.md)
3635

3736
## Related Projects
3837

39-
- [diode-netbox-plugin](https://github.com/netboxlabs/diode-netbox-plugin) - The Diode NetBox plugin is a NetBox plugin
40-
and a required component of the Diode ingestion service.
41-
- [diode-sdk-python](https://github.com/netboxlabs/diode-sdk-python) - Diode SDK Python is a Python library for
42-
interacting with the Diode ingestion service utilizing gRPC.
43-
- [diode-sdk-go](https://github.com/netboxlabs/diode-sdk-go) - Diode SDK Go is a Go module for interacting with the
44-
Diode ingestion service utilizing gRPC.
45-
- [diode-agent](https://github.com/netboxlabs/diode-agent) - A collection of agents that leverage the Diode SDK to
46-
interact with the Diode server.
38+
- [diode-netbox-plugin](https://github.com/netboxlabs/diode-netbox-plugin) - The Diode NetBox plugin is a NetBox plugin and a required component of the Diode ingestion service.
39+
- [diode-sdk-python](https://github.com/netboxlabs/diode-sdk-python) - Diode SDK Python is a Python library for interacting with the Diode ingestion service utilizing gRPC.
40+
- [diode-sdk-go](https://github.com/netboxlabs/diode-sdk-go) - Diode SDK Go is a Go module for interacting with the Diode ingestion service utilizing gRPC.
41+
- [orb-agent](https://github.com/netboxlabs/orb-agent) - The NetBox Discovery agent.
42+
43+
## Support
44+
45+
- [GitHub Issues](https://github.com/netboxlabs/diode/issues)
46+
- [Slack NetDev Community (#orb channel)](https://https://netdev-community.slack.com/)
4747

4848
## License
4949

5050
Distributed under the NetBox Limited Use License 1.0. See [LICENSE.md](./LICENSE.md) for more information.
5151

52-
Diode protocol buffers are distributed under the Apache 2.0 License. See [LICENSE.txt](./diode-proto/LICENSE.txt) for
53-
more information.
52+
Diode protocol buffers are distributed under the Apache 2.0 License. See [LICENSE.txt](./diode-proto/LICENSE.txt) for more information.
5453

5554
## Required Notice
5655

‎charts/diode/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ apiVersion: v2
22
name: diode
33
description: A Helm chart for Diode
44
type: application
5-
version: 1.5.0
5+
version: 1.6.0
66
appVersion: "1.2.0"
77
home: https://github.com/netboxlabs/diode
88
sources:

0 commit comments

Comments
 (0)