Skip to content

Commit 38cee73

Browse files
ilia-medvedev-codefreshilia-medvedev-codefresh
committed
initial wip
1 parent 2259ff9 commit 38cee73

31 files changed

+3994
-1
lines changed

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
vendor/
2+
.devspace/

‎README.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,12 @@
1-
# vcluster-prometheus-operator-plugin
1+
# vCluster Prometheus operator plugin
2+
3+
This repository contains a [vCluster plugin](https://www.vcluster.com/docs/v0.19/advanced-topics/plugins-overview) that syncs [Prometheus operator](https://github.com/prometheus-operator/prometheus-operator) resources from virtual clusters to the host cluster.
4+
5+
Currently only the very basic functionality is implemented so the plugin only supports syncing of `PodMonitor` and `ServiceMonitor` resources. This is to allow scraping of metrics from workloads running on virtual clusters from a signle Prometheus or Open Telemetry collector on the host (with [target allocator](https://github.com/open-telemetry/opentelemetry-operator/blob/main/cmd/otel-allocator/README.md) that supports Prometheus operator CRDs).
6+
7+
The repository contains 2 versions of the plugin, each version is compatible with different versions of vCluster, but both versions provide the same functionality.
8+
This compatibility is required as Syncer arhitecture was overhauled by loft.sh in version 0.20.0 and plugin-sdk changed accordingly.
9+
10+
`v1` - Compatible with older versions of vCluster - latest confirmed & tested version is `0.16.4`
11+
12+
`v2` - Compatibe with vCluster version 0.20.0 which was the highest version at the time of writing.

‎v1/.e2e/chainsaw-tests.yaml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
apiVersion: monitoring.coreos.com/v1
2+
kind: ServiceMonitor
3+
metadata:
4+
name: sm-with-ns-selector
5+
labels:
6+
team: frontend
7+
spec:
8+
namespaceSelector:
9+
any: false
10+
matchNames:
11+
- my-virtual-namespace
12+
selector:
13+
matchLabels:
14+
app: example-app
15+
endpoints:
16+
- port: web

‎v1/.e2e/scripts/init.sh

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/bash
2+
# Deploy CRDs and vcluster with the plugin, connect to vcluster and apply resources in resources folder for testing
3+
4+
PLUGIN_IMAGE=$1
5+
MYDIR=$(dirname $0)
6+
ROOT_DIR=$MYDIR/../../
7+
RESOURCES_FILE=$MYDIR/../vcluster-resources.yaml
8+
9+
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.70.0/example/prometheus-operator-crd-full/monitoring.coreos.com_podmonitors.yaml
10+
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.70.0/example/prometheus-operator-crd-full/monitoring.coreos.com_servicemonitors.yaml
11+
helm upgrade --install --repo https://charts.loft.sh vcluster vcluster --version 0.16.4 --values $MYDIR/vcluster-values.yaml --values $ROOT_DIR/plugin.yaml --set plugin.prometheus-operator-resources.image=$PLUGIN_IMAGE --wait
12+
13+
14+
cat $RESOURCES_FILE | vcluster connect vcluster -n default -- kubectl -n default apply -f -

‎v1/.e2e/scripts/vcluster-values.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
serviceAccount:
2+
create: false
3+
name: default
4+
vcluster:
5+
image: rancher/k3s:v1.28.2-k3s1
6+
rbac:
7+
clusterRole:
8+
create: true
9+
role:
10+
extended: true
11+
syncer:
12+
readinessProbe:
13+
enabled: false
14+
livenessProbe:
15+
enabled: false

‎v1/.e2e/vcluster-resources.yaml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: monitoring.coreos.com/v1
2+
kind: ServiceMonitor
3+
metadata:
4+
name: sm-with-ns-selector
5+
namespace: default
6+
labels:
7+
team: frontend
8+
spec:
9+
namespaceSelector:
10+
any: false
11+
matchNames:
12+
- my-virtual-namespace
13+
selector:
14+
matchLabels:
15+
app: example-app
16+
endpoints:
17+
- port: web

‎v1/Dockerfile

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Build the manager binary
2+
FROM golang:1.18 as builder
3+
4+
# Make sure we use go modules
5+
WORKDIR vcluster
6+
7+
# Copy the Go Modules manifests
8+
COPY . .
9+
10+
# Install dependencies
11+
RUN go mod vendor
12+
13+
# Build cmd
14+
RUN CGO_ENABLED=0 GO111MODULE=on go build -mod vendor -o /plugin main.go
15+
16+
# we use alpine for easier debugging
17+
FROM alpine
18+
19+
# Set root path as working directory
20+
WORKDIR /
21+
22+
COPY --from=builder /plugin .
23+
24+
ENTRYPOINT ["/plugin"]

‎v1/Dockerfile.dev

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
FROM golang:1.18 as builder
2+
3+
WORKDIR /plugin
4+
5+
# Install Delve for debugging
6+
RUN go install github.com/go-delve/delve/cmd/dlv@latest
7+
8+
ENV GO111MODULE on
9+
ENV DEBUG true
10+
11+
ENTRYPOINT ["sleep", "999999999999"]

‎v1/README.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
## Development
2+
3+
Before starting to develop, make sure you have installed the following tools on your computer:
4+
- [docker](https://docs.docker.com/)
5+
- [kubectl](https://kubernetes.io/docs/tasks/tools/) with a valid kube context configured
6+
- [helm](https://helm.sh/docs/intro/install/), which is used to deploy vcluster and the plugin
7+
- [vcluster CLI](https://www.vcluster.com/docs/getting-started/setup) v0.6.0 or higher
8+
- [DevSpace](https://devspace.sh/cli/docs/quickstart), which is used to spin up a development environment
9+
10+
If you want to develop within a remote Kubernetes cluster (as opposed to docker-desktop or minikube), make sure to exchange `PLUGIN_IMAGE` in the `devspace.yaml` with a valid registry path you can push to.
11+
12+
After successfully setting up the tools, start the development environment with:
13+
```
14+
devspace dev -n vcluster
15+
```
16+
17+
After a while a terminal should show up with additional instructions. Enter the following command to start the plugin:
18+
```
19+
go run -mod vendor main.go
20+
```
21+
22+
The output should look something like this:
23+
```
24+
I0124 11:20:14.702799 4185 logr.go:249] plugin: Try creating context...
25+
I0124 11:20:14.730044 4185 logr.go:249] plugin: Waiting for vcluster to become leader...
26+
I0124 11:20:14.731097 4185 logr.go:249] plugin: Starting syncers...
27+
[...]
28+
I0124 11:20:15.957331 4185 logr.go:249] plugin: Successfully started plugin.
29+
```
30+
31+
You can now change a file locally in your IDE and then restart the command in the terminal to apply the changes to the plugin.
32+
33+
Delete the development environment with:
34+
```
35+
devspace purge -n vcluster
36+
```
37+
38+
## Using the Plugin in vcluster
39+
40+
### Building the Plugin
41+
To just build the plugin image and push it to the registry, run:
42+
```
43+
# Build
44+
docker build . -t my-repo/my-plugin:0.0.1
45+
46+
# Push
47+
docker push my-repo/my-plugin:0.0.1
48+
```
49+
50+
### Using the plugin
51+
52+
To use the plugin, create a new vcluster with the `plugin.yaml`:
53+
54+
```
55+
# Use local plugin.yaml
56+
vcluster create my-vcluster -n my-vcluster -f ./plugin.yaml
57+
58+
# Use public plugin.yaml
59+
vcluster create my-vcluster -n my-vcluster -f https://raw.githubusercontent.com/codefresh-contrib/vcluster-prometheus-operator-plugin/main/v1/plugin.yaml
60+
```
61+
62+
This will create a new vcluster with the plugin installed.

‎v1/devspace.yaml

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
version: v2beta1
2+
name: vcluster-prom-operator-plugin
3+
4+
vars:
5+
PLUGIN_IMAGE: ilmedcodefreh/vcluster-prom-operator-plugin
6+
# - name: K3S_IMAGE
7+
# value: rancher/k3s:v1.23.1-k3s1
8+
# - name: SERVICE_CIDR
9+
# command: vcluster get service-cidr
10+
images:
11+
vcluster:
12+
image: ${PLUGIN_IMAGE}
13+
dockerfile: ./Dockerfile.dev
14+
rebuildStrategy: ignoreContextChanges
15+
# hooks:
16+
# - name: Install Dependencies & Car Manifests
17+
# events: ["after:deploy"]
18+
# command: |-
19+
# # Install dependencies
20+
# if [ ! -d "vendor" ]; then
21+
# echo "Executing 'go mod vendor'..."
22+
# go mod vendor
23+
# fi
24+
25+
# # Install car crd
26+
# kubectl apply -f manifests/crds.yaml
27+
deployments:
28+
prometheus-operator:
29+
helm:
30+
chart:
31+
name: kube-prometheus-stack
32+
repo: https://prometheus-community.github.io/helm-charts
33+
version: 60.2.0
34+
vcluster:
35+
helm:
36+
chart:
37+
name: vcluster
38+
repo: https://charts.loft.sh
39+
version: 0.16.4
40+
valuesFiles:
41+
- plugin.yaml
42+
values:
43+
plugin:
44+
prometheus-operator-resources:
45+
image: ${PLUGIN_IMAGE}
46+
#serviceCIDR: ${SERVICE_CIDR}
47+
serviceAccount:
48+
create: false
49+
name: default
50+
#vcluster:
51+
#image: ${K3S_IMAGE}
52+
rbac:
53+
clusterRole:
54+
create: true
55+
role:
56+
extended: true
57+
syncer:
58+
readinessProbe:
59+
enabled: false
60+
livenessProbe:
61+
enabled: false
62+
dev:
63+
plugin:
64+
imageSelector: ${PLUGIN_IMAGE}
65+
#command: ["./devspace_start.sh"]
66+
ports:
67+
- port: "2346:2345"
68+
sync:
69+
- path: ./:/plugin
70+
terminal:
71+
command: "./devspace_start.sh"

‎v1/devspace_start.sh

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/bin/bash
2+
set +e # Continue on errors
3+
4+
COLOR_CYAN="\033[0;36m"
5+
COLOR_RESET="\033[0m"
6+
7+
RUN_CMD="go run -mod vendor main.go"
8+
DEBUG_CMD="dlv debug ./main.go --listen=0.0.0.0:2345 --api-version=2 --output /tmp/__debug_bin --headless --build-flags=\"-mod=vendor\""
9+
10+
echo -e "${COLOR_CYAN}
11+
____ ____
12+
| _ \ _____ __/ ___| _ __ __ _ ___ ___
13+
| | | |/ _ \ \ / /\___ \| '_ \ / _\` |/ __/ _ \\
14+
| |_| | __/\ V / ___) | |_) | (_| | (_| __/
15+
|____/ \___| \_/ |____/| .__/ \__,_|\___\___|
16+
|_|
17+
${COLOR_RESET}
18+
Welcome to your development container!
19+
This is how you can work with it:
20+
- Run \`${COLOR_CYAN}${RUN_CMD}${COLOR_RESET}\` to start the plugin
21+
- ${COLOR_CYAN}Files will be synchronized${COLOR_RESET} between your local machine and this container
22+
23+
If you wish to run the plugin in the debug mode with delve, run:
24+
\`${COLOR_CYAN}${DEBUG_CMD}${COLOR_RESET}\`
25+
Wait until the \`${COLOR_CYAN}API server listening at: [::]:2345${COLOR_RESET}\` message appears
26+
Connect your debugger to localhost:2346
27+
${COLOR_CYAN}Note:${COLOR_RESET} the plugin won't start until you connect with the debugger.
28+
${COLOR_CYAN}Note:${COLOR_RESET} the plugin will be stopped once you detach your debugger session.
29+
30+
${COLOR_CYAN}TIP:${COLOR_RESET} hit an up arrow on your keyboard to find the commands mentioned above :)
31+
"
32+
# add useful commands to the history for convenience
33+
export HISTFILE=/tmp/.bash_history
34+
history -s $DEBUG_CMD
35+
history -s $RUN_CMD
36+
history -a
37+
38+
bash

‎v1/go.mod

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
module github.com/codefresh-contrib/vcluster-prometheus-operator-plugin/v1
2+
3+
go 1.18
4+
5+
require (
6+
github.com/loft-sh/vcluster-sdk v0.4.0
7+
github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.53.0
8+
k8s.io/apimachinery v0.24.0
9+
sigs.k8s.io/controller-runtime v0.12.1
10+
)
11+
12+
require (
13+
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
14+
github.com/MakeNowJust/heredoc v1.0.0 // indirect
15+
github.com/beorn7/perks v1.0.1 // indirect
16+
github.com/cespare/xxhash/v2 v2.2.0 // indirect
17+
github.com/chai2010/gettext-go v0.0.0-20160711120539-c6fed771bfd5 // indirect
18+
github.com/davecgh/go-spew v1.1.1 // indirect
19+
github.com/emicklei/go-restful/v3 v3.10.1 // indirect
20+
github.com/evanphx/json-patch v5.6.0+incompatible // indirect
21+
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect
22+
github.com/fatih/camelcase v1.0.0 // indirect
23+
github.com/fsnotify/fsnotify v1.6.0 // indirect
24+
github.com/fvbommel/sortorder v1.0.2 // indirect
25+
github.com/go-errors/errors v1.4.2 // indirect
26+
github.com/go-logr/logr v1.2.3 // indirect
27+
github.com/go-openapi/jsonpointer v0.19.6 // indirect
28+
github.com/go-openapi/jsonreference v0.20.2 // indirect
29+
github.com/go-openapi/swag v0.22.3 // indirect
30+
github.com/gogo/protobuf v1.3.2 // indirect
31+
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
32+
github.com/golang/protobuf v1.5.2 // indirect
33+
github.com/google/btree v1.1.2 // indirect
34+
github.com/google/gnostic v0.6.9 // indirect
35+
github.com/google/go-cmp v0.5.9 // indirect
36+
github.com/google/gofuzz v1.2.0 // indirect
37+
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
38+
github.com/google/uuid v1.3.0 // indirect
39+
github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
40+
github.com/imdario/mergo v0.3.13 // indirect
41+
github.com/inconshreveable/mousetrap v1.1.0 // indirect
42+
github.com/jonboulle/clockwork v0.3.0 // indirect
43+
github.com/josharian/intern v1.0.0 // indirect
44+
github.com/json-iterator/go v1.1.12 // indirect
45+
github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
46+
github.com/mailru/easyjson v0.7.7 // indirect
47+
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
48+
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
49+
github.com/moby/spdystream v0.2.0 // indirect
50+
github.com/moby/term v0.0.0-20221205130635-1aeaba878587 // indirect
51+
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
52+
github.com/modern-go/reflect2 v1.0.2 // indirect
53+
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
54+
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
55+
github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
56+
github.com/pkg/errors v0.9.1 // indirect
57+
github.com/pmezard/go-difflib v1.0.0 // indirect
58+
github.com/prometheus/client_golang v1.14.0 // indirect
59+
github.com/prometheus/client_model v0.3.0 // indirect
60+
github.com/prometheus/common v0.39.0 // indirect
61+
github.com/prometheus/procfs v0.9.0 // indirect
62+
github.com/russross/blackfriday v1.5.2 // indirect
63+
github.com/spf13/cobra v1.6.1 // indirect
64+
github.com/spf13/pflag v1.0.5 // indirect
65+
github.com/stretchr/testify v1.8.1 // indirect
66+
github.com/xlab/treeprint v1.1.0 // indirect
67+
go.starlark.net v0.0.0-20230128213706-3f75dec8e403 // indirect
68+
golang.org/x/net v0.6.0 // indirect
69+
golang.org/x/oauth2 v0.5.0 // indirect
70+
golang.org/x/sys v0.5.0 // indirect
71+
golang.org/x/term v0.5.0 // indirect
72+
golang.org/x/text v0.7.0 // indirect
73+
golang.org/x/time v0.3.0 // indirect
74+
gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
75+
google.golang.org/appengine v1.6.7 // indirect
76+
google.golang.org/genproto v0.0.0-20230202175211-008b39050e57 // indirect
77+
google.golang.org/grpc v1.53.0 // indirect
78+
google.golang.org/protobuf v1.28.1 // indirect
79+
gopkg.in/inf.v0 v0.9.1 // indirect
80+
gopkg.in/yaml.v2 v2.4.0 // indirect
81+
gopkg.in/yaml.v3 v3.0.1 // indirect
82+
k8s.io/api v0.24.0 // indirect
83+
k8s.io/apiextensions-apiserver v0.24.0 // indirect
84+
k8s.io/cli-runtime v0.24.0 // indirect
85+
k8s.io/client-go v0.24.0 // indirect
86+
k8s.io/component-base v0.24.0 // indirect
87+
k8s.io/klog v1.0.0 // indirect
88+
k8s.io/klog/v2 v2.90.0 // indirect
89+
k8s.io/kube-aggregator v0.24.0 // indirect
90+
k8s.io/kube-openapi v0.0.0-20230202010329-39b3636cbaa3 // indirect
91+
k8s.io/kubectl v0.24.0 // indirect
92+
k8s.io/utils v0.0.0-20230202215443-34013725500c // indirect
93+
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
94+
sigs.k8s.io/kustomize/api v0.11.4 // indirect
95+
sigs.k8s.io/kustomize/kyaml v0.13.6 // indirect
96+
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
97+
sigs.k8s.io/yaml v1.3.0 // indirect
98+
)

0 commit comments

Comments
 (0)