Skip to content

Commit f0adcfa

Browse files
albinsureshalbinsuresh
committed
feat(kubernetes): Draft Kubernetes deployment of tedge
Signed-off-by: Reuben Miller <reuben.d.miller@gmail.com>
1 parent dc8fcb5 commit f0adcfa

File tree

11 files changed

+439
-0
lines changed

11 files changed

+439
-0
lines changed

‎.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,3 +51,8 @@ containers/tedge/*.tar
5151
Makefile.toml
5252
.vimspector.json
5353
*.env
54+
55+
56+
# thin-edge.io certificate files
57+
tedge-certificate.pem
58+
tedge-private-key.pem

‎kubernetes/README.md

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# thin-edge.io on Kubernetes
2+
3+
:construction: This is a work-in-progress so don't expect everything to work out of the box in all situations. Contributions are welcomed to improve the setup.
4+
5+
This document describes the steps required to get thin-edge.io deployed in a Kubernetes (k8s) cluster.
6+
7+
## Pre-requisites
8+
9+
The following pre-requisites are required to run the example:
10+
11+
* A kubernetes cluster - [k3s](https://k3s.io/) is a good option for a single node cluster if you're just trying things out locally
12+
* kubectl - cli to interact with kubernetes
13+
* [helm](https://helm.sh/)
14+
15+
Kubernetes know-how is also assumed, so if you have trouble with setting up Kubernetes, please consult the public Kubernetes documentation or seek help in community forums (e.g. stack overflow etc.).
16+
17+
## Getting started
18+
19+
The following guide should help you install thin-edge.io as a helm chart.
20+
21+
1. Create the device certificate key pair
22+
23+
**Option 1: Create self-signed certificates using tedge cli**
24+
25+
```sh
26+
mkdir device-certs
27+
tedge --config-dir $(pwd) cert create --device-id mydevice001
28+
```
29+
30+
The certificates will be stored under the `./device-certs` folder.
31+
32+
Alternatively, you can also create device certificates using a CA and openssl. Please consult the openssl documentation on how to do this.
33+
34+
1. Add device certificate to Cumulocity IoT's Trusted Certificates
35+
36+
You can uploaded it to Cumulocity IoT using the Device Management UI, or using the go-c8y-cli tool:
37+
38+
```sh
39+
c8y devicemanagement certificates create --name mydevice001 --file ./device-certs/tedge-certificate.pem --status ENABLED --autoRegistrationEnabled
40+
```
41+
42+
1. Create a kubernetes namespace where the helm chart will be deployed to
43+
44+
```sh
45+
kubectl create namespace tedge
46+
```
47+
48+
1. Create a Secret for the device certificate key pair
49+
50+
```sh
51+
kubectl create secret generic tedge-certs --from-file=./device-certs/tedge-certificate.pem --from-file=./device-certs/tedge-private-key.pem --namespace tedge
52+
```
53+
54+
1. Install the thin-edge.io helm chart
55+
56+
```sh
57+
helm install tedge-chart ./tedge --set c8y.url=example.eu-latest.cumulocity.com --namespace tedge
58+
```
59+
60+
Change the `c8y.url=` value to the MQTT endpoint of your Cumulocity IoT tenant.
61+
62+
1. Validate the device created in Cumulocity
63+
64+
If you are using go-c8y-cli, then you can open the Device Management application to your device using the following command:
65+
66+
```sh
67+
c8y identity get --name mydevice001 | c8y applications open
68+
```
69+
70+
1. The mosquitto broker can be accessed by other pods in the cluster via the `mosquitto.<namespace>.svc.cluster.local` endpoint
71+
72+
**Using an existing service**
73+
74+
The MQTT service endpoint can be verified from within a deployment using the following command:
75+
76+
```sh
77+
kubectl exec -n tedge -it service/mosquitto -- mosquitto_sub -h 'mosquitto.tedge.svc.cluster.local' -p 1883 -t '#' -W 5 -v
78+
```
79+
80+
**Using a new pod**
81+
82+
The MQTT service endpoint can also be verified from other pods by creating a test pod, and executing a command from it.
83+
84+
```sh
85+
kubectl run -n tedge --restart=Never --image eclipse-mosquitto tedge-test -- sleep infinity
86+
kubectl exec -n tedge -it pod/tedge-test -- mosquitto_sub -h 'mosquitto.tedge.svc.cluster.local' -p 1883 -t '#' -W 3 -v
87+
kubectl delete -n tedge pod/tedge-test
88+
```
89+
90+
```sh
91+
te/device/main/service/mosquitto-c8y-bridge {"@id":"example001:device:main:service:mosquitto-c8y-bridge","@parent":"device/main//","@type":"service","name":"mosquitto-c8y-bridge","type":"service"}
92+
te/device/main/service/mosquitto-c8y-bridge/status/health 1
93+
te/device/main/service/tedge-mapper-c8y {"@parent":"device/main//","@type":"service","type":"service"}
94+
te/device/main/service/tedge-mapper-c8y/status/health {"pid":1,"status":"up","time":1710236345.2828279}
95+
te/device/main/service/tedge-agent {"@parent":"device/main//","@type":"service","type":"service"}
96+
te/device/main/service/tedge-agent/status/health {"pid":1,"status":"up","time":1710236338.9187257}
97+
te/device/main///twin/c8y_Agent {"name":"thin-edge.io","url":"https://thin-edge.io","version":"1.0.1"}
98+
te/device/main///cmd/config_snapshot {"types":["tedge-configuration-plugin","tedge-log-plugin","tedge.toml"]}
99+
te/device/main///cmd/config_update {"types":["tedge-configuration-plugin","tedge-log-plugin","tedge.toml"]}
100+
te/device/main///cmd/log_upload {"types":["software-management"]}
101+
te/device/main///cmd/restart {}
102+
te/device/main///cmd/software_list {}
103+
te/device/main///cmd/software_update {}
104+
Timed out
105+
command terminated with exit code 27
106+
```
107+
108+
## Uninstall helm chart
109+
110+
You can uninstall the helm chart using the following command:
111+
112+
```
113+
helm uninstall tedge-chart --namespace tedge
114+
```
115+
116+
## Project structure
117+
118+
Key points about the deployment design are listed below:
119+
120+
* The thin-edge.io helm chart consists of two deployments:
121+
* a multi-container pod with `mosquitto` broker and `tedge-mapper` along with `tedge-bootstrap` init container
122+
that performs the bootstrapping
123+
* an independent deployment of `tedge-agent`
124+
* The `mosquitto` broker and `tedge-mapper` are deployed in the same pod with `/etc/tedge` directory mounted as a shared volume,
125+
so that the mapper and the broker can access the bridge configurations created by the `tedge-bootstrap` container.
126+
* A custom mosquitto configuration is used for the `mosquitto` container so that it picks up configuration extensions
127+
from the `/etc/tedge/mosquitto-conf` directory where the bridge configurations are created.
128+
* The mosquitto broker from the first deployment is exposed as a service so that the `tedge-agent` deployment can access it.
129+
Similarly, the file-transfer-service of the `tedge-agent` is also exposed as a service for the mapper to use it.
130+
* The config settings required by the thin-edge.io components are provided via env variables.
131+
* The device certificate key pair generated and provided to the cluster as a `Secret`.
132+
* The persistent volume for the shared `/etc/tedge` directory is a local volume.
133+
134+
## Misc.
135+
136+
### Using dedicated colima instance on MacOS
137+
138+
If you want to use a dedicated k3s single node cluster on MacOS, then you can use colima to create a dedicated environment (different from the default colima instance).
139+
140+
**Containerd runtime**
141+
142+
You can create a colima instance using containerd as the runtime using the following command:
143+
144+
```sh
145+
colima start k3s --runtime containerd --kubernetes
146+
```
147+
148+
Afterwards you can configure kubectl to use the new context:
149+
150+
```sh
151+
kubectl config use-context colima-k3s
152+
```

‎kubernetes/tedge/.helmignore

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Patterns to ignore when building packages.
2+
# This supports shell glob matching, relative path matching, and
3+
# negation (prefixed with !). Only one pattern per line.
4+
.DS_Store
5+
# Common VCS dirs
6+
.git/
7+
.gitignore
8+
.bzr/
9+
.bzrignore
10+
.hg/
11+
.hgignore
12+
.svn/
13+
# Common backup files
14+
*.swp
15+
*.bak
16+
*.tmp
17+
*.orig
18+
*~
19+
# Various IDEs
20+
.project
21+
.idea/
22+
*.tmproj
23+
.vscode/

‎kubernetes/tedge/Chart.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: v2
2+
name: tedge
3+
description: Device management application using thin-edge.io
4+
type: application
5+
version: "0.1.0"
6+
appVersion: "1.0.1"

‎kubernetes/tedge/templates/_helpers.tpl

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{{/*
2+
Expand the name of the chart.
3+
*/}}
4+
{{- define "tedge.name" -}}
5+
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
6+
{{- end }}
7+
8+
{{/*
9+
Create a default fully qualified app name.
10+
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
11+
If release name contains chart name it will be used as a full name.
12+
*/}}
13+
{{- define "tedge.fullname" -}}
14+
{{- if .Values.fullnameOverride }}
15+
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
16+
{{- else }}
17+
{{- $name := default .Chart.Name .Values.nameOverride }}
18+
{{- if contains $name .Release.Name }}
19+
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
20+
{{- else }}
21+
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
22+
{{- end }}
23+
{{- end }}
24+
{{- end }}
25+
26+
{{/*
27+
Create chart name and version as used by the chart label.
28+
*/}}
29+
{{- define "tedge.chart" -}}
30+
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
31+
{{- end }}
32+
33+
{{/*
34+
Common labels
35+
*/}}
36+
{{- define "tedge.labels" -}}
37+
helm.sh/chart: {{ include "tedge.chart" . }}
38+
{{ include "tedge.selectorLabels" . }}
39+
{{- if .Chart.AppVersion }}
40+
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
41+
{{- end }}
42+
app.kubernetes.io/managed-by: {{ .Release.Service }}
43+
{{- end }}
44+
45+
{{/*
46+
Selector labels
47+
*/}}
48+
{{- define "tedge.selectorLabels" -}}
49+
app.kubernetes.io/name: {{ include "tedge.name" . }}
50+
app.kubernetes.io/instance: {{ .Release.Name }}
51+
{{- end }}
52+
53+
{{/*
54+
Create the name of the service account to use
55+
*/}}
56+
{{- define "tedge.serviceAccountName" -}}
57+
{{- if .Values.serviceAccount.create }}
58+
{{- default (include "tedge.fullname" .) .Values.serviceAccount.name }}
59+
{{- else }}
60+
{{- default "default" .Values.serviceAccount.name }}
61+
{{- end }}
62+
{{- end }}

‎kubernetes/tedge/templates/mosquitto-config.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: mosquitto-config
5+
data:
6+
mosquitto.conf: |
7+
include_dir /etc/tedge/mosquitto-conf

‎kubernetes/tedge/templates/mosquitto.yaml

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: mosquitto
5+
spec:
6+
replicas: 1 # Number of broker pods
7+
selector:
8+
matchLabels:
9+
app: mosquitto
10+
template:
11+
metadata:
12+
labels:
13+
app: mosquitto
14+
spec:
15+
volumes:
16+
- name: tedge-vol
17+
persistentVolumeClaim:
18+
claimName: "tedge-pvc"
19+
- name: mosquitto-config-vol
20+
configMap:
21+
name: mosquitto-config
22+
- name: tedge-certs-vol
23+
secret:
24+
secretName: tedge-certs
25+
initContainers:
26+
- name: tedge-bootstrap
27+
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
28+
imagePullPolicy: {{ .Values.image.pullPolicy }}
29+
securityContext:
30+
runAsUser: 0
31+
command:
32+
- sh
33+
- -xec
34+
- "tedge init && cp /tedge/device-certs/* /etc/tedge/device-certs/ && tedge cert show && tedge connect c8y && cat /etc/tedge/mosquitto-conf/c8y-bridge.conf"
35+
volumeMounts:
36+
- name: tedge-vol
37+
mountPath: /etc/tedge
38+
- name: tedge-certs-vol
39+
mountPath: /tedge/device-certs
40+
env:
41+
- name: TEDGE_C8Y_URL
42+
value: {{ required "A valid c8y.url is required!" .Values.c8y.url }}
43+
containers:
44+
- name: mosquitto
45+
image: eclipse-mosquitto
46+
imagePullPolicy: {{ .Values.image.pullPolicy }}
47+
ports:
48+
- containerPort: 1883
49+
name: mqtt-svc
50+
volumeMounts:
51+
- name: mosquitto-config-vol
52+
mountPath: /mosquitto/config
53+
- name: tedge-vol
54+
mountPath: /etc/tedge
55+
- name: tedge-mapper
56+
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
57+
imagePullPolicy: {{ .Values.image.pullPolicy }}
58+
command: ["tedge-mapper", "c8y"]
59+
ports:
60+
- containerPort: 8001
61+
volumeMounts:
62+
- name: tedge-vol
63+
mountPath: /etc/tedge
64+
env:
65+
- name: TEDGE_MQTT_CLIENT_HOST
66+
value: "localhost"
67+
- name: TEDGE_C8Y_URL
68+
value: {{ required "A valid c8y.url is required!" .Values.c8y.url }}
69+
- name: TEDGE_HTTP_CLIENT_HOST
70+
value: "fts"
71+
- name: TEDGE_C8Y_PROXY_CLIENT_HOST
72+
value: "localhost"
73+
74+
75+
---
76+
77+
apiVersion: v1
78+
kind: Service
79+
metadata:
80+
name: mosquitto
81+
spec:
82+
selector:
83+
app: mosquitto
84+
ports:
85+
- protocol: TCP
86+
port: {{ .Values.services.mqtt.port }}
87+
targetPort: mqtt-svc

‎kubernetes/tedge/templates/tedge-agent.yaml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: tedge-agent
5+
spec:
6+
replicas: 1
7+
selector:
8+
matchLabels:
9+
app: tedge-agent
10+
template:
11+
metadata:
12+
labels:
13+
app: tedge-agent
14+
spec:
15+
containers:
16+
- name: tedge-agent
17+
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
18+
imagePullPolicy: {{ .Values.image.pullPolicy }}
19+
ports:
20+
- containerPort: 8000
21+
name: fts-svc
22+
env:
23+
- name: TEDGE_MQTT_CLIENT_HOST
24+
value: "mosquitto"
25+
- name: TEDGE_HTTP_CLIENT_HOST
26+
value: "localhost"
27+
28+
---
29+
30+
apiVersion: v1
31+
kind: Service
32+
metadata:
33+
name: fts
34+
spec:
35+
selector:
36+
app: tedge-agent
37+
ports:
38+
- name: fts
39+
port: 8000
40+
targetPort: fts-svc

0 commit comments

Comments
 (0)