Skip to content

re-enable e2e project-monitoring CI workflows #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

re-enable e2e project-monitoring CI workflows #83

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d255be6
re-enable e2e project-monitoring CI workflows
alexandreLamarre Dec 20, 2023
3d0ff90
WIP CI changes
alexandreLamarre Dec 20, 2023
c156a7b
debug
alexandreLamarre Jan 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 31 additions & 33 deletions .github/workflows/e2e-ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,39 +110,37 @@ jobs:
-
name: Check if Project Registration Namespace is auto-created on namespace detection
run: ./.github/workflows/e2e/scripts/create-project-namespace.sh;

# Commenting out for failure in CI but not locally
# -
# name: Create Project Monitoring Stack via ProjectHelmChart CR
# run: ./.github/workflows/e2e/scripts/create-projecthelmchart.sh;
# -
# name: Check if the Project Prometheus Stack is up
# run: ./.github/workflows/e2e/scripts/validate-project-monitoring.sh;
# -
# name: Wait for 8 minutes for enough scraping to be done to continue
# run: |
# for i in {1..48}; do sleep 10; echo "Waited $((i*10)) seconds for metrics to be populated"...; done;
# -
# name: Validate Project Prometheus Targets
# run: ./.github/workflows/e2e/scripts/validate-project-prometheus-targets.sh;
# -
# name: Validate Project Grafana Datasources
# run: ./.github/workflows/e2e/scripts/validate-project-grafana-datasource.sh;
# -
# name: Validate Project Grafana Dashboards
# run: ./.github/workflows/e2e/scripts/validate-project-grafana-dashboards.sh;
# #-
# #name: Validate Project Grafana Dashboard Data
# #run: ./.github/workflows/e2e/scripts/validate-project-grafana-dashboard-data.sh;
# -
# name: Validate Project Prometheus Alerts
# run: ./.github/workflows/e2e/scripts/validate-project-prometheus-alerts.sh;
# -
# name: Validate Project Alertmanager
# run: ./.github/workflows/e2e/scripts/validate-project-alertmanager.sh;
# -
# name: Delete Project Prometheus Stack
# run: ./.github/workflows/e2e/scripts/delete-projecthelmchart.sh;
-
name: Create Project Monitoring Stack via ProjectHelmChart CR
run: ./.github/workflows/e2e/scripts/create-projecthelmchart.sh;
-
name: Check if the Project Prometheus Stack is up
run: ./.github/workflows/e2e/scripts/validate-project-monitoring.sh;
-
name: Wait for 8 minutes for enough scraping to be done to continue
run: |
for i in {1..48}; do sleep 10; echo "Waited $((i*10)) seconds for metrics to be populated"...; done;
-
name: Validate Project Prometheus Targets
run: ./.github/workflows/e2e/scripts/validate-project-prometheus-targets.sh;
-
name: Validate Project Grafana Datasources
run: ./.github/workflows/e2e/scripts/validate-project-grafana-datasource.sh;
-
name: Validate Project Grafana Dashboards
run: ./.github/workflows/e2e/scripts/validate-project-grafana-dashboards.sh;
-
name: Validate Project Grafana Dashboard Data
run: ./.github/workflows/e2e/scripts/validate-project-grafana-dashboard-data.sh;
-
name: Validate Project Prometheus Alerts
run: ./.github/workflows/e2e/scripts/validate-project-prometheus-alerts.sh;
-
name: Validate Project Alertmanager
run: ./.github/workflows/e2e/scripts/validate-project-alertmanager.sh;
-
name: Delete Project Prometheus Stack
run: ./.github/workflows/e2e/scripts/delete-projecthelmchart.sh;
-
name: Uninstall Prometheus Federator
run: ./.github/workflows/e2e/scripts/uninstall-federator.sh;
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/e2e/scripts/create-projecthelmchart.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,16 @@ if [[ "${E2E_CI}" == "true" ]]; then
else
kubectl apply -f ./examples/example.yaml
fi

# debug
kubectl get -n cattle-monitoring-system job || true
kubectl get -A job || true

sleep ${DEFAULT_SLEEP_TIMEOUT_SECONDS};

kubectl get -n cattle-monitoring-system job || true
kubectl get -A job || true

if ! kubectl get -n cattle-monitoring-system job/helm-install-cattle-project-p-example-monitoring; then
echo "ERROR: Helm Install Job for Project Monitoring Stack was never created after ${KUBECTL_WAIT_TIMEOUT} seconds"
exit 1
Expand Down
13 changes: 12 additions & 1 deletion .github/workflows/e2e/scripts/generate-artifacts.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,20 @@ kubectl get helmreleases -n cattle-monitoring-system -o yaml > ${MANIFEST_DIRECT

# Logs

mkdir -p ${LOG_DIRECTORY}/rancher-monitoring

## Rancher logs

mkdir -p ${LOG_DIRECTORY}/rancher


kubectl logs deployment/rancher-webhook -n cattle-system > ${LOG_DIRECTORY}/rancher/rancher_webhook.log || true
kubectl logs deployment/cattle-cluster-agent -n cattle-system > ${LOG_DIRECTORY}/rancher/cluster_agent.log || true
kubectl logs deployment/system-upgrade-controller -n cattle-system > ${LOG_DIRECTORY}/rancher/upgrade_controller.log || true

## Rancher Monitoring

mkdir -p ${LOG_DIRECTORY}/rancher-monitoring

kubectl logs deployment/rancher-monitoring-operator -n cattle-monitoring-system > ${LOG_DIRECTORY}/rancher-monitoring/prometheus_operator.log || true
kubectl logs statefulset/alertmanager-rancher-monitoring-alertmanager -n cattle-monitoring-system > ${LOG_DIRECTORY}/rancher-monitoring/alertmanager.log || true
kubectl logs statefulset/prometheus-rancher-monitoring-prometheus -n cattle-monitoring-system > ${LOG_DIRECTORY}/rancher-monitoring/prometheus.log || true
Expand Down