Skip to content

Commit 38cee73

Browse files
ilia-medvedev-codefreshilia-medvedev-codefresh
committed
initial wip
1 parent 2259ff9 commit 38cee73

31 files changed

+3994
-1
lines changed

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
vendor/
2+
.devspace/

‎README.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,12 @@
1-
# vcluster-prometheus-operator-plugin
1+
# vCluster Prometheus operator plugin
2+
3+
This repository contains a [vCluster plugin](https://www.vcluster.com/docs/v0.19/advanced-topics/plugins-overview) that syncs [Prometheus operator](https://github.com/prometheus-operator/prometheus-operator) resources from virtual clusters to the host cluster.
4+
5+
Currently only the very basic functionality is implemented so the plugin only supports syncing of `PodMonitor` and `ServiceMonitor` resources. This is to allow scraping of metrics from workloads running on virtual clusters from a signle Prometheus or Open Telemetry collector on the host (with [target allocator](https://github.com/open-telemetry/opentelemetry-operator/blob/main/cmd/otel-allocator/README.md) that supports Prometheus operator CRDs).
6+
7+
The repository contains 2 versions of the plugin, each version is compatible with different versions of vCluster, but both versions provide the same functionality.
8+
This compatibility is required as Syncer arhitecture was overhauled by loft.sh in version 0.20.0 and plugin-sdk changed accordingly.
9+
10+
`v1` - Compatible with older versions of vCluster - latest confirmed & tested version is `0.16.4`
11+
12+
`v2` - Compatibe with vCluster version 0.20.0 which was the highest version at the time of writing.

‎v1/.e2e/chainsaw-tests.yaml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
apiVersion: monitoring.coreos.com/v1
2+
kind: ServiceMonitor
3+
metadata:
4+
name: sm-with-ns-selector
5+
labels:
6+
team: frontend
7+
spec:
8+
namespaceSelector:
9+
any: false
10+
matchNames:
11+
- my-virtual-namespace
12+
selector:
13+
matchLabels:
14+
app: example-app
15+
endpoints:
16+
- port: web

‎v1/.e2e/scripts/init.sh

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/bash
2+
# Deploy CRDs and vcluster with the plugin, connect to vcluster and apply resources in resources folder for testing
3+
4+
PLUGIN_IMAGE=$1
5+
MYDIR=$(dirname $0)
6+
ROOT_DIR=$MYDIR/../../
7+
RESOURCES_FILE=$MYDIR/../vcluster-resources.yaml
8+
9+
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.70.0/example/prometheus-operator-crd-full/monitoring.coreos.com_podmonitors.yaml
10+
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.70.0/example/prometheus-operator-crd-full/monitoring.coreos.com_servicemonitors.yaml
11+
helm upgrade --install --repo https://charts.loft.sh vcluster vcluster --version 0.16.4 --values $MYDIR/vcluster-values.yaml --values $ROOT_DIR/plugin.yaml --set plugin.prometheus-operator-resources.image=$PLUGIN_IMAGE --wait
12+
13+
14+
cat $RESOURCES_FILE | vcluster connect vcluster -n default -- kubectl -n default apply -f -

‎v1/.e2e/scripts/vcluster-values.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
serviceAccount:
2+
create: false
3+
name: default
4+
vcluster:
5+
image: rancher/k3s:v1.28.2-k3s1
6+
rbac:
7+
clusterRole:
8+
create: true
9+
role:
10+
extended: true
11+
syncer:
12+
readinessProbe:
13+
enabled: false
14+
livenessProbe:
15+
enabled: false

‎v1/.e2e/vcluster-resources.yaml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: monitoring.coreos.com/v1
2+
kind: ServiceMonitor
3+
metadata:
4+
name: sm-with-ns-selector
5+
namespace: default
6+
labels:
7+
team: frontend
8+
spec:
9+
namespaceSelector:
10+
any: false
11+
matchNames:
12+
- my-virtual-namespace
13+
selector:
14+
matchLabels:
15+
app: example-app
16+
endpoints:
17+
- port: web

‎v1/Dockerfile

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Build the manager binary
2+
FROM golang:1.18 as builder
3+
4+
# Make sure we use go modules
5+
WORKDIR vcluster
6+
7+
# Copy the Go Modules manifests
8+
COPY . .
9+
10+
# Install dependencies
11+
RUN go mod vendor
12+
13+
# Build cmd
14+
RUN CGO_ENABLED=0 GO111MODULE=on go build -mod vendor -o /plugin main.go
15+
16+
# we use alpine for easier debugging
17+
FROM alpine
18+
19+
# Set root path as working directory
20+
WORKDIR /
21+
22+
COPY --from=builder /plugin .
23+
24+
ENTRYPOINT ["/plugin"]

‎v1/Dockerfile.dev

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
FROM golang:1.18 as builder
2+
3+
WORKDIR /plugin
4+
5+
# Install Delve for debugging
6+
RUN go install github.com/go-delve/delve/cmd/dlv@latest
7+
8+
ENV GO111MODULE on
9+
ENV DEBUG true
10+
11+
ENTRYPOINT ["sleep", "999999999999"]

‎v1/README.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
## Development
2+
3+
Before starting to develop, make sure you have installed the following tools on your computer:
4+
- [docker](https://docs.docker.com/)
5+
- [kubectl](https://kubernetes.io/docs/tasks/tools/) with a valid kube context configured
6+
- [helm](https://helm.sh/docs/intro/install/), which is used to deploy vcluster and the plugin
7+
- [vcluster CLI](https://www.vcluster.com/docs/getting-started/setup) v0.6.0 or higher
8+
- [DevSpace](https://devspace.sh/cli/docs/quickstart), which is used to spin up a development environment
9+
10+
If you want to develop within a remote Kubernetes cluster (as opposed to docker-desktop or minikube), make sure to exchange `PLUGIN_IMAGE` in the `devspace.yaml` with a valid registry path you can push to.
11+
12+
After successfully setting up the tools, start the development environment with:
13+
```
14+
devspace dev -n vcluster
15+
```
16+
17+
After a while a terminal should show up with additional instructions. Enter the following command to start the plugin:
18+
```
19+
go run -mod vendor main.go
20+
```
21+
22+
The output should look something like this:
23+
```
24+
I0124 11:20:14.702799 4185 logr.go:249] plugin: Try creating context...
25+
I0124 11:20:14.730044 4185 logr.go:249] plugin: Waiting for vcluster to become leader...
26+
I0124 11:20:14.731097 4185 logr.go:249] plugin: Starting syncers...
27+
[...]
28+
I0124 11:20:15.957331 4185 logr.go:249] plugin: Successfully started plugin.
29+
```
30+
31+
You can now change a file locally in your IDE and then restart the command in the terminal to apply the changes to the plugin.
32+
33+
Delete the development environment with:
34+
```
35+
devspace purge -n vcluster
36+
```
37+
38+
## Using the Plugin in vcluster
39+
40+
### Building the Plugin
41+
To just build the plugin image and push it to the registry, run:
42+
```
43+
# Build
44+
docker build . -t my-repo/my-plugin:0.0.1
45+
46+
# Push
47+
docker push my-repo/my-plugin:0.0.1
48+
```
49+
50+
### Using the plugin
51+
52+
To use the plugin, create a new vcluster with the `plugin.yaml`:
53+
54+
```
55+
# Use local plugin.yaml
56+
vcluster create my-vcluster -n my-vcluster -f ./plugin.yaml
57+
58+
# Use public plugin.yaml
59+
vcluster create my-vcluster -n my-vcluster -f https://raw.githubusercontent.com/codefresh-contrib/vcluster-prometheus-operator-plugin/main/v1/plugin.yaml
60+
```
61+
62+
This will create a new vcluster with the plugin installed.

‎v1/devspace.yaml

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
version: v2beta1
2+
name: vcluster-prom-operator-plugin
3+
4+
vars:
5+
PLUGIN_IMAGE: ilmedcodefreh/vcluster-prom-operator-plugin
6+
# - name: K3S_IMAGE
7+
# value: rancher/k3s:v1.23.1-k3s1
8+
# - name: SERVICE_CIDR
9+
# command: vcluster get service-cidr
10+
images:
11+
vcluster:
12+
image: ${PLUGIN_IMAGE}
13+
dockerfile: ./Dockerfile.dev
14+
rebuildStrategy: ignoreContextChanges
15+
# hooks:
16+
# - name: Install Dependencies & Car Manifests
17+
# events: ["after:deploy"]
18+
# command: |-
19+
# # Install dependencies
20+
# if [ ! -d "vendor" ]; then
21+
# echo "Executing 'go mod vendor'..."
22+
# go mod vendor
23+
# fi
24+
25+
# # Install car crd
26+
# kubectl apply -f manifests/crds.yaml
27+
deployments:
28+
prometheus-operator:
29+
helm:
30+
chart:
31+
name: kube-prometheus-stack
32+
repo: https://prometheus-community.github.io/helm-charts
33+
version: 60.2.0
34+
vcluster:
35+
helm:
36+
chart:
37+
name: vcluster
38+
repo: https://charts.loft.sh
39+
version: 0.16.4
40+
valuesFiles:
41+
- plugin.yaml
42+
values:
43+
plugin:
44+
prometheus-operator-resources:
45+
image: ${PLUGIN_IMAGE}
46+
#serviceCIDR: ${SERVICE_CIDR}
47+
serviceAccount:
48+
create: false
49+
name: default
50+
#vcluster:
51+
#image: ${K3S_IMAGE}
52+
rbac:
53+
clusterRole:
54+
create: true
55+
role:
56+
extended: true
57+
syncer:
58+
readinessProbe:
59+
enabled: false
60+
livenessProbe:
61+
enabled: false
62+
dev:
63+
plugin:
64+
imageSelector: ${PLUGIN_IMAGE}
65+
#command: ["./devspace_start.sh"]
66+
ports:
67+
- port: "2346:2345"
68+
sync:
69+
- path: ./:/plugin
70+
terminal:
71+
command: "./devspace_start.sh"

0 commit comments

Comments
 (0)