Skip to content

Commit c8a9852

Browse files
lucaspinlucaspin
authored
feat: add new chart for agent-k8s-controller (#23)
1 parent a0cac40 commit c8a9852

File tree

13 files changed

+684
-2
lines changed

13 files changed

+684
-2
lines changed

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
# Semaphore Helm charts
22

33
Helm repository for Semaphore charts. The current charts available are:
4-
- [Semaphore agent](./charts/agent/)
5-
- [External metrics server](./charts/external-metrics-server/)
4+
- [Semaphore controller](./charts/controller/)
5+
- (*deprecated*) [Semaphore agent](./charts/agent/)
6+
- (*deprecated*) [External metrics server](./charts/external-metrics-server/)
67

78
## Usage
89

‎charts/agent/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
> [!WARNING]
2+
> This chart is deprecated. It does not work well with versions of Kubernetes >= 1.26. Please, use the [Semaphore custom controller chart](../controller) to run Semaphore jobs in your Kubernetes cluster.
3+
14
Install one or multiple [Semaphore agent](https://github.com/semaphoreci/agent) pools in a Kubernetes cluster.
25

36
- [Installation](#installation)

‎charts/controller/.helmignore‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Patterns to ignore when building packages.
2+
# This supports shell glob matching, relative path matching, and
3+
# negation (prefixed with !). Only one pattern per line.
4+
.DS_Store
5+
# Common VCS dirs
6+
.git/
7+
.gitignore
8+
.bzr/
9+
.bzrignore
10+
.hg/
11+
.hgignore
12+
.svn/
13+
# Common backup files
14+
*.swp
15+
*.bak
16+
*.tmp
17+
*.orig
18+
*~
19+
# Various IDEs
20+
.project
21+
.idea/
22+
*.tmproj
23+
.vscode/

‎charts/controller/Chart.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
apiVersion: v2
2+
name: controller
3+
description: A Helm chart to install the Semaphore job controller
4+
5+
# A chart can be either an 'application' or a 'library' chart.
6+
#
7+
# Application charts are a collection of templates that can be packaged into versioned archives
8+
# to be deployed.
9+
#
10+
# Library charts provide useful utilities or functions for the chart developer. They're included as
11+
# a dependency of application charts to inject those utilities and functions into the rendering
12+
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
13+
type: application
14+
15+
# This is the chart version. This version number should be incremented each time you make changes
16+
# to the chart and its templates, including the app version.
17+
# Versions are expected to follow Semantic Versioning (https://semver.org/)
18+
version: 0.1.0
19+
20+
# This is the version number of the application being deployed. This version number should be
21+
# incremented each time you make changes to the application. Versions are not expected to
22+
# follow Semantic Versioning. They should reflect the version the application is using.
23+
# It is recommended to use it with quotes.
24+
appVersion: "v0.1.0"

‎charts/controller/README.md‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
Run Semaphore jobs for one or multiple [Semaphore agent type](https://github.com/semaphoreci/agent) pools in a Kubernetes cluster.
2+
3+
- [Installation](#installation)
4+
- [Start jobs for an agent type](#start-jobs-for-an-agent-type)
5+
- [Using multiple agent types](#using-multiple-agent-types)
6+
- [Default pod spec](#default-pod-spec)
7+
- [The pre-job hook](#the-pre-job-hook)
8+
- [Disabling the pre-job hook](#disabling-the-pre-job-hook)
9+
- [Using a custom pre-job hook](#using-a-custom-pre-job-hook)
10+
- [Do not specify a default pod spec](#do-not-specify-a-default-pod-spec)
11+
- [Overriding the default pod spec values](#overriding-the-default-pod-spec-values)
12+
- [Overriding the default pod spec for a single agent type](#overriding-the-default-pod-spec-for-a-single-agent-type)
13+
- [Job retention policies](#job-retention-policies)
14+
- [Job start timeouts](#job-start-timeouts)
15+
- [Logging](#logging)
16+
- [Configuration](#configuration)
17+
18+
## Installation
19+
20+
This chart installs the [agent-k8s-controller](https://github.com/renderedtext/agent-k8s-controller) into your Kubernetes cluster.
21+
22+
You can install it with your Semaphore API token and your Semaphore organization endpoint:
23+
24+
```bash
25+
helm upgrade --install semaphore-controller charts/controller \
26+
--namespace semaphore \
27+
--create-namespace \
28+
--set endpoint=<your-organization>.semaphoreci.com \
29+
--set apiToken=<your-api-token>
30+
```
31+
32+
## Start jobs for an agent type
33+
34+
The controller automatically detects which agent types to monitor by looking at the secrets available in the namespace it is running on. To start monitoring the queue and creating jobs for an agent type, you need to create a Kubernetes secret with the necessary information for the controller to spin new agents for that agent type.
35+
36+
You can follow the guide [here](https://docs.semaphoreci.com/ci-cd-environment/self-hosted-agent-types/) to create an agent type. After doing so, you can start creating jobs for it by creating a secret like this:
37+
38+
```yaml
39+
apiVersion: v1
40+
kind: Secret
41+
metadata:
42+
name: my-semaphore-agent-type
43+
namespace: semaphore
44+
labels:
45+
semaphoreci.com/resource-type: agent-type-configuration
46+
stringData:
47+
agentTypeName: s1-my-agent-type
48+
registrationToken: <agent-type-registration-token>
49+
```
50+
51+
Notice the `semaphoreci.com/resource-type=agent-type-configuration` label. That's how the controller knows this secret has the information needed to start agents for a Semaphore agent type.
52+
53+
## Using multiple agent types
54+
55+
You can start jobs for as many agent types you want. Just be aware that the controller will respect the `parallelism` settings and will not create more than the amount of jobs specified in that settings, for all the agent types you have.
56+
57+
## Default pod spec
58+
59+
The controller is responsible for starting Semaphore agents to run the jobs that appear in the queue for your agent type. Each Semaphore agent that starts will itself create a new pod to run the job that it is assigned. The agent configures that pod configured using a [pod spec](https://github.com/semaphoreci/agent/blob/master/docs/kubernetes-executor.md#--kubernetes-pod-spec) decorator.
60+
61+
This chart provides a default pod spec that the controller will use for all agent types that do not specify it themselves.
62+
63+
### The pre-job hook
64+
65+
By default, the controller's default pod spec includes a pre-job hook used to install the Semaphore toolbox at the beginning of every job.
66+
67+
> [!TIP]
68+
> Pre-installing the Semaphore toolbox (and any other required tools for your builds) in the images used during the jobs is a good way to avoid wasting job running time to install dependencies.
69+
70+
### Disabling the pre-job hook
71+
72+
If you do not want to use the default pre-job hook, you can disable it with the `agent.defaultPodSpec.preJobHook.enabled` value:
73+
74+
```bash
75+
helm upgrade --install semaphore-controller charts/controller \
76+
--namespace semaphore \
77+
--create-namespace \
78+
--set endpoint=<your-organization>.semaphoreci.com \
79+
--set apiToken=<your-api-token> \
80+
--set agent.defaultPodSpec.preJobHook.enabled=false
81+
```
82+
83+
### Using a custom pre-job hook
84+
85+
If the default pre-job hook does not fit your needs, you can use a custom one with the `agent.defaultPodSpec.preJobHook.customScript` value:
86+
87+
```bash
88+
helm upgrade --install semaphore-controller charts/controller \
89+
--namespace semaphore \
90+
--create-namespace \
91+
--set endpoint=<your-organization>.semaphoreci.com \
92+
--set apiToken=<your-api-token> \
93+
--set agent.defaultPodSpec.preJobHook.customScript=$(cat my-custom-script.sh | base64)
94+
```
95+
96+
### Do not specify a default pod spec
97+
98+
If you do not want to have a default pod spec for your agent types, you can disable it with:
99+
100+
```bash
101+
helm upgrade --install semaphore-controller charts/controller \
102+
--namespace semaphore \
103+
--create-namespace \
104+
--set endpoint=<your-organization>.semaphoreci.com \
105+
--set apiToken=<your-api-token> \
106+
--set agent.defaultPodSpec.enabled=false
107+
```
108+
109+
### Overriding the default pod spec values
110+
111+
You can also configure the pod, main container and sidecar containers for the default controller's pod spec, specifying the `agent.defaultPodSpec.pod`, `agent.defaultPodSpec.mainContainer` and `agent.defaultPodSpec.sidecarContainers` parameters.
112+
113+
For example, if you have a `custom-values.yml` file like this:
114+
115+
```yaml
116+
endpoint: <your-organization>.semaphoreci.com
117+
apiToken: <your-api-token>
118+
agent:
119+
defaultPodSpec:
120+
mainContainer:
121+
env:
122+
- name: FOO_1
123+
value: BAR_1
124+
- name: FOO_2
125+
value: BAR_2
126+
```
127+
128+
You can expose `FOO_1` and `FOO_2` environment variables to all Semaphore jobs. To install it:
129+
130+
```bash
131+
helm upgrade --install semaphore-controller charts/controller \
132+
--namespace semaphore \
133+
--create-namespace \
134+
-f custom-values.yml
135+
```
136+
137+
### Overriding the default pod spec for a single agent type
138+
139+
You might need to configure the pods to run the Semaphore jobs differently depending on the agent type. You can do that by specifying a `agentStartupParameters` field in your agent type secret.
140+
141+
For example, if you want to use a different pod spec only for an agent type `s1-my-agent-type-2`, you can do so by specifying the `agentStartupParameters` in the agent type secret:
142+
143+
```yaml
144+
apiVersion: v1
145+
kind: Secret
146+
metadata:
147+
name: custom-pre-job-hook-for-my-agent-type-1
148+
namespace: semaphore
149+
stringData:
150+
pre-job-hook: |-
151+
echo "hello from custom pre-job hook script"
152+
---
153+
apiVersion: v1
154+
kind: ConfigMap
155+
metadata:
156+
name: my-agent-type-2-pod-spec
157+
namespace: semaphore
158+
data:
159+
pod: |-
160+
volumes:
161+
- name: pre-job-hook
162+
secret:
163+
secretName: custom-pre-job-hook-for-my-agent-type-1
164+
defaultMode: 0644
165+
items:
166+
- key: pre-job-hook
167+
path: pre-job-hook
168+
mainContainer: |-
169+
env:
170+
- name: FOO_1
171+
value: DIFFERENT_VALUE_1
172+
- name: FOO_2
173+
value: DIFFERENT_VALUE_2
174+
resources:
175+
limits:
176+
cpu: "0.5"
177+
memory: 500Mi
178+
requests:
179+
cpu: "0.25"
180+
memory: 250Mi
181+
volumeMounts:
182+
- name: pre-job-hook
183+
mountPath: /opt/semaphore/hooks
184+
readOnly: true
185+
---
186+
apiVersion: v1
187+
kind: Secret
188+
metadata:
189+
name: my-semaphore-agent-type-2
190+
namespace: semaphore
191+
labels:
192+
semaphoreci.com/resource-type: agent-type-configuration
193+
stringData:
194+
agentTypeName: s1-my-agent-type-2
195+
registrationToken: <registration-token>
196+
agentStartupParameters: |-
197+
"--kubernetes-pod-spec my-agent-type-2-pod-spec --pre-job-hook-path /opt/semaphore/hooks/pre-job-hook"
198+
```
199+
200+
## Job retention policies
201+
202+
By default, all successful Kubernetes jobs are deleted after they are finished, and all failed jobs are kept for 1 day to help with troubleshooting purposes. However, those values can be configured. For example, you can keep all failed jobs for 7 days, and all successful jobs for 15 minutes, with:
203+
204+
```bash
205+
helm upgrade --install semaphore-controller charts/controller \
206+
--namespace semaphore \
207+
--create-namespace \
208+
--set endpoint=<your-organization>.semaphoreci.com \
209+
--set apiToken=<your-api-token> \
210+
--set keepFailedJobsFor=7d \
211+
--set keepSuccessfulJobsFor=15m
212+
```
213+
214+
## Job start timeouts
215+
216+
By default, if the Kubernetes job created by the controller does not start in 5 minutes, it will cancel it, deleting it. You can override that behavior with the `jobStartTimeout` parameter.
217+
218+
For example, if you want to extend the timeout to 30 minutes instead, you can do it with:
219+
220+
```bash
221+
helm upgrade --install semaphore-controller charts/controller \
222+
--namespace semaphore \
223+
--create-namespace \
224+
--set endpoint=<your-organization>.semaphoreci.com \
225+
--set apiToken=<your-api-token> \
226+
--set jobStartTimeout=30m
227+
```
228+
229+
## Logging
230+
231+
Since agent pods are deleted by the controller when the job finishes, it is recommended to configure your Kubernetes cluster to stream the agent pod logs to an external place, to help with troubleshooting, if needed. [This guide](https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures) describes the usual strategies to accomplish that.
232+
233+
## Configuration
234+
235+
All the available configuration values can be seen with `helm show values renderedtext/controller`.

‎charts/controller/pre-job-hook.sh‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Install the Semaphore toolbox in the job
2+
rm -rf ~/.toolbox
3+
4+
downloadPath="https://github.com/semaphoreci/toolbox/releases/latest/download/self-hosted-linux.tar"
5+
if [ ! -z "${SEMAPHORE_TOOLBOX_VERSION}" ]; then
6+
downloadPath="https://github.com/semaphoreci/toolbox/releases/download/$SEMAPHORE_TOOLBOX_VERSION/self-hosted-linux.tar"
7+
fi
8+
9+
echo "Downloading Semaphore toolbox from $downloadPath..."
10+
curl -sL --retry 5 --connect-timeout 3 $downloadPath -o /tmp/toolbox.tar
11+
tar -xvf /tmp/toolbox.tar
12+
mv toolbox ~/.toolbox
13+
if [ ! -d ~/.toolbox ]; then
14+
echo "Failed to download toolbox."
15+
return 1
16+
fi
17+
18+
echo "Installing..."
19+
bash ~/.toolbox/install-toolbox
20+
if [ "$?" -ne "0" ]; then
21+
echo "Failed to install toolbox."
22+
rm -rf $SEMAPHORE_GIT_DIR
23+
fi
24+
25+
source ~/.toolbox/toolbox
26+
if [ "$?" -ne "0" ]; then
27+
echo "Failed to source toolbox."
28+
rm -rf $SEMAPHORE_GIT_DIR
29+
fi
30+
31+
echo "Semaphore toolbox successfully installed."
32+
33+
# Create SSH configuration.
34+
# This is required in order to avoid having to manually accept the GitHub SSH keys fingerprints on checkout.
35+
# Ideally, we should populate ~/.ssh/known_hosts with the GitHub keys from api.github.com/meta.
36+
mkdir -p ~/.ssh
37+
echo 'Host github.com' | tee -a ~/.ssh/config
38+
echo ' StrictHostKeyChecking no' | tee -a ~/.ssh/config
39+
echo ' UserKnownHostsFile=/dev/null' | tee -a ~/.ssh/config

0 commit comments

Comments
 (0)