Skip to content

PropertyBrands/btt-solarwinds-snap-agent-docker

BranchesTags
 
 

Repository files navigation

solarwinds-snap-agent-docker

Docker and Kubernetes assets for running SolarWinds Snap Agent

Table of contents

About

Use the containerized SolarWinds Snap Agent to monitor Docker or Kubernetes environments. Monitor Kubernetes cluster and application health. Aggregate metrics across clusters distributed across multiple data centers and cloud providers. Track pods, deployments, services and more with Kubernetes-integrated service discovery.

Kubernetes assets:

  • Deployment - A single pod to talk to the Kubernetes API to send Kubernetes specific metrics to AppOptics.
  • DaemonSet - A DaemonSet that runs a pod on every node in your cluster and publishes HostAgent and Docker metrics to AppOptics.

A typical cluster will utilize both the Deployment and DaemonSet assets.

Alternatively, you can deploy the containerized agent in a sidecar to run the other AppOptics Integrations and monitor your Kubernetes applications running in the same Pod.

Installation

Kubernetes assests available in this repository expect a solarwinds-token secret to exist. To create this secret run:

kubectl create secret generic solarwinds-token -n kube-system --from-literal=SOLARWINDS_TOKEN=<REPLACE WITH TOKEN>

Deployment

By default, RBAC is enabled in the deploy manifests. If you are not using RBAC you can deploy swisnap-agent-deployment.yaml removing the reference to the Service Account.

In the configMapGenerator section of kustomization.yaml>, you can configure which plugins should be run by setting SWISNAP_ENABLE_<plugin_name> to either true or false. Plugins turned on via environment variables are using default configuration and taskfiles. To see list of plugins currently supported this way please refer to: Environment Parameters.

After configuring deployment to your needs (please refer to Configuration and ensuring that solarwinds-token secret was already created run:

kubectl apply -k ./deploy/overlays/stable/deployment

Finally, check if the deployment is running properly:

kubectl get deployment swisnap-agent-k8s -n kube-system

Enable the Kubernetes plugin in the AppOptics UI and you should start seeing data trickle in.

DaemonSet

The DaemonSet, by default, will give you insight into containers running within its nodes and gather system, processes and docker-related metrics. To deploy the DaemonSet to Kubernetes verify you have an solarwinds-token secret already created and run:

kubectl apply -k ./deploy/overlays/stable/daemonset

Enable the Docker plugin in the AppOptics UI and you should start seeing data trickle in.

Sidecar

Docker

If you wanted to run containerized SolarWinds Snap Agent with custom taskfiles, you can use following snippets:

docker run -d -e SOLARWINDS_TOKEN=token \
           -v my_custom_statsd.yaml:/opt/SolarWinds/Snap/etc/plugins.d/statsd.yaml \
           --name swisnap-agent \
           solarwinds/solarwinds-snap-agent-docker:latest

or using docker-compose:

version: '3'
services:
  swisnap:
    image: solarwinds/solarwinds-snap-agent-docker:latest
    hostname: swisnap-agent
    container_name: swisnap-agent
    volumes:
      - /path/to/my_custom_statsd.yaml:/opt/SolarWinds/Snap/etc/plugins.d/statsd.yaml
    environment:
      - SOLARWINDS_TOKEN=token

Kubernetes

If you wanted to run this on Kubernetes as a sidecar for monitoring specific services, you can follow the instructions below, which use Apache Server as an example. In this setup, the agent will monitor only services running in particular pod(s), not Kubernetes itself.

  • Useful when you want to monitor only specific per-pod-services
  • Configuration is similar to pod setup
  • In order to monitor specific services only, the kubernetes and aosystem plugins should be disabled by setting SWISNAP_ENABLE_KUBERNETES to false and SWISNAP_DISABLE_HOSTAGENT to true in swisnap-agent-deployment.yaml

In order to monitor Apache with the agent in a sidecar, add a second container to your deployment YAML underneath spec.template.spec.containers and the agent should now have access to your service over localhost (notice SWISNAP_ENABLE_APACHE):

Note: Containers inside the same pod can communicate through localhost, so there's no need to pass a static IP - Resource sharing and communication

        containers:
        - name: apache
            imagePullPolicy: Always
            image: '<your-image>'
            ports:
            - containerPort: 80
        - name: swisnap-agent-ds
            image: 'solarwinds/solarwinds-snap-agent-docker:latest'
            imagePullPolicy: Always
            env:
            - name: SOLARWINDS_TOKEN
                value: 'SOLARWINDS_TOKEN'
            - name: APPOPTICS_HOSTNAME
                valueFrom:
                fieldRef:
                    fieldPath: spec.nodeName
            - name: SWISNAP_ENABLE_DOCKER
                value: 'false'
            - name: SWISNAP_ENABLE_APACHE
                value: 'true'
            - name: SWISNAP_DISABLE_HOSTAGENT
                value: 'true'
            - name: HOST_PROC
                value: '/host/proc'

In the example above, the sidecar will run only the Apache plugin. Additionally, if the default Apache Plugin configuration is not sufficient, custom one should be passed to pod running SolarWinds Snap Agent - Configuration.

Configuration

Custom plugins configuration and tasks manifests

SolarWinds Snap Agent image is using default plugins configuration files and tasks manifests. In order to use your own configuration you would have to create Kubernetes configMap. In this example we'll set up two configMaps, one for SolarWinds Snap Agent Kubernetes plugin config and second one for corresponding task.

# create plugin configMap
kubectl create configmap kubernetes-plugin-config --from-file=/path/to/my/plugins.d/kubernetes.yaml --namespace=kube-system

# create task configMap
kubectl create configmap kubernetes-task-manifest --from-file=/path/to/my/tasks.d/task-aokubernetes.yaml --namespace=kube-system

# check if everything is fine
kubectl describe configmaps --namespace=kube-system kubernetes-task-manifest kubernetes-plugin-config

ConfigMaps should be attached to SolarWinds Snap Agent deployment. Here's the example, notice spec.template.spec.containers.volumeMounts and spec.template.spec.volumes:

diff --git a/deploy/base/deployment/kustomization.yaml b/deploy/base/deployment/kustomization.yaml
index 79e0110..000a108 100644
--- a/deploy/base/deployment/kustomization.yaml
+++ b/deploy/base/deployment/kustomization.yaml
@@ -15,7 +15,7 @@ configMapGenerator:
       - SWISNAP_ENABLE_APACHE=false
       - SWISNAP_ENABLE_DOCKER=false
       - SWISNAP_ENABLE_ELASTICSEARCH=false
-      - SWISNAP_ENABLE_KUBERNETES=true
+      - SWISNAP_ENABLE_KUBERNETES=false
       - SWISNAP_ENABLE_PROMETHEUS=false
       - SWISNAP_ENABLE_MESOS=false
       - SWISNAP_ENABLE_MONGODB=false
diff --git a/deploy/base/deployment/swisnap-agent-deployment.yaml b/deploy/base/deployment/swisnap-agent-deployment.yaml
index 294c4b4..babff7d 100644
--- a/deploy/base/deployment/swisnap-agent-deployment.yaml
+++ b/deploy/base/deployment/swisnap-agent-deployment.yaml
@@ -45,6 +45,12 @@ spec:
             - configMapRef:
                 name: swisnap-k8s-configmap
           volumeMounts:
+            - name: kubernetes-plugin-vol
+              mountPath: /opt/SolarWinds/Snap/etc/plugins.d/kubernetes.yaml
+              subPath: kubernetes.yaml
+            - name: kubernetes-task-vol
+              mountPath: /opt/SolarWinds/Snap/etc/tasks.d/task-aokubernetes.yaml
+              subPath: task-aokubernetes.yaml
             - name: proc
               mountPath: /host/proc
               readOnly: true
@@ -56,6 +62,18 @@ spec:
               cpu: 100m
               memory: 256Mi
       volumes:
+        - name: kubernetes-plugin-vol
+          configMap:
+            name: kubernetes-plugin-config
+            items:
+              - key: kubernetes.yaml
+                path: kubernetes.yaml
+        - name: kubernetes-task-vol
+          configMap:
+            name: kubernetes-task-manifest
+            items:
+              - key: task-aokubernetes.yaml
+                path: task-aokubernetes.yaml
         - name: proc
           hostPath:
             path: /proc

Notice that we're not utilizing Environment Parameters to turn on Kubernetes plugin. When you're attaching taskfiles and plugin configuration files through configMaps, there's no need to set environment variables SWISNAP_ENABLE_<plugin-name>. SolarWinds Snap Agent will automatically load plugins based on files stored in configMaps.

Environment Parameters

The following environment parameters are available:

Parameter Description
APPOPTICS_CUSTOM_TAGS Set this to a comma separated K=V list to enable custom tags eg. NAME=TEST,IS_PRODUCTION=false,VERSION=5
SOLARWINDS_TOKEN Your SolarWinds token. This parameter is required.
APPOPTICS_TOKEN Depreciated. Your SolarWinds token. This parameter is used as fallback if SOLARWINDS_TOKEN is not present.
APPOPTICS_HOSTNAME This value overrides the hostname tagged for default host metrics. The DaemonSet uses this to override with Node name.
LOG_LEVEL Expected value: DEBUG, INFO, WARN, ERROR or FATAL. Default value is WARN.
LOG_PATH Set this value to enable SolarWinds Snap Agent logging to file. Default logs are printed to stdout for SolarWinds Snap Agent running in Docker container. Overriding this option disable reading Snap Agent log using docker logs, or kubectl logs.
SWISNAP_SECURE Set this to true to run only signed plugins.
SWISNAP_DISABLE_HOSTAGENT Set this to true to disable the SolarWinds Snap Agent system metrics collection.
SWISNAP_DISABLE_PROCESSES Set this to true to disable the SolarWinds Snap Agent processes metrics collection.
SWISNAP_ENABLE_DOCKER Set this to true to enable the Docker plugin.
SWISNAP_ENABLE_APACHE Set this to true to enable the Apache plugin.
SWISNAP_ENABLE_ELASTICSEARCH Set this to true to enable the Elasticsearch plugin.
SWISNAP_ENABLE_KUBERNETES Set this to true to enable the Kubernetes plugin. Enabling this option on the DaemonSet will cause replication of Kubernetes metrics where the replication count is the number of pods with Kubernetes collection enabled minus one. Typically Kubernetes collection is only enabled on the Deployment asset.
SWISNAP_ENABLE_NGINX Set this to true to enable the Nginx plugin. If enabled the following ENV vars are required to be set:
NGINX_STATUS_URI - one, or multiple space-separeted link(s) to Nginx stub_status URI.
SWISNAP_ENABLE_NGINX_PLUS Set this to true to enable the Nginx Plus plugin. If enabled the following ENV vars are required to be set:
NGINX_PLUS_STATUS_URI - one, or multiple space-separeted link(s) to ngx_http_status_module or status URI.
SWISNAP_ENABLE_NGINX_PLUS_API Set this to true to enable the Nginx Plus Api plugin. If enabled the following ENV vars are required to be set:
NGINX_PLUS_STATUS_URI - one, or multiple space-separeted link(s) to Nginx API URI.
SWISNAP_ENABLE_MESOS Set this to true to enable the Mesos plugin.
SWISNAP_ENABLE_MONGODB Set this to true to enable the MongoDB plugin.
SWISNAP_ENABLE_MYSQL Set this to true to enable the MySQL plugin. If enabled the following ENV vars are required to be set:
MYSQL_USER,
MYSQL_PASS,
MYSQL_HOST
MYSQL_PORT
SWISNAP_ENABLE_PROMETHEUS Set this to true to enable the Prometheus pod annotation scrapping
SWISNAP_ENABLE_POSTGRESQL Set this to true to enable the Postgres plugin. If enabled the following ENV vars are required to be set:
POSTGRES_ADDRESS - specify address for Postgres databse
SWISNAP_ENABLE_RABBITMQ Set this to true to enable the RabbitMQ plugin.
SWISNAP_ENABLE_REDIS Set this to true to enable the Redis plugin. If enabled the following ENV vars are required to be set:
REDIS_SERVERS - one, or multiple space-separeted link(s) to Redis servers
SWISNAP_ENABLE_SOCKET_LISTENER Set this to true to enable the Socket Listener plugin. If enabled the following ENV vars are required to be set:
SOCKET_SERVICE_ADDRESS - URL to listen on,
SOCKET_LISTENER_FORMAT - Data format to consume: "collectd", "graphite", "influx", "json", or "value".
SWISNAP_ENABLE_STATSD Set this to true to enable the Statsd plugin.
SWISNAP_ENABLE_ZOOKEEPER Set this to true to enable the Zookeeper plugin.

If you use SWISNAP_ENABLE_<plugin_name> set to true, then keep in mind that SolarWinds Snap Agent will use default plugins configs and task manifests. For custom configuration see Custom plugins configuration and tasks manifests.

Integrating Kubernetes Cluster Events Collection With Loggly

This documentaton can be also found in Documentation for SolarWinds webpage. Version 22 of Kubernetes collector allows you to collect cluster events and push them to Loggly using logs collector under the hood. To utilize this functionality there is a need to create corresponding configmaps in your cluster, with proper plugins configuration. The example config files can be found in Event collector configs. To enable event collection in your deployment, follow below steps:

  • Create Kubernetes secret for SOLARWINDS_TOKEN:

    kubectl create secret generic solarwinds-token -n kube-system --from-literal=SOLARWINDS_TOKEN=<REPLACE WITH TOKEN>

  • Update Loggly Token in logs-v2.yaml file under v2:collector:logs:all:logging_service:loggly:token.

    v2:
  collector:
    logs:
      all:
        #  [...]
        logging_service:
          # [..]
          ## Sign up for a Loggly account at: https://www.loggly.com
          loggly:

            ## Loggly API token and host
            token: "LOGGLY_TOKEN
            host: "logs-01.loggly.com"

            ## Loggly API port and protocol
            ## use 6514 with TLS or 514 with TCP
            port: 6514
            protocol: tls

  • task-logs-k8s.yaml file configures the logs collector plugin. It is telling logs collector to look for /var/log/SolarWinds/Snap/events.log file:

    ---
version: 2

schedule:
  type: cron
  interval: "0 * * * * *"

plugins:
  - plugin_name: logs

    metrics:
      - /logs/lines_total
      - /logs/lines_forwarded
      - /logs/bytes_forwarded
      - /logs/lines_skipped
      - /logs/lines_failed
      - /logs/bytes_failed
      - /logs/lines_succeeded
      - /logs/bytes_succeeded
      - /logs/attempts_total
      - /logs/failed_attempts_total

    config:
      sources:
        log_files:

          # [...]
          file_paths:
            - Path: /var/log/SolarWinds/Snap/events.log

          # [...]
          exclude_lines_patterns:
            - ".*self-skip-logs-collector.*"

    publish:
      - plugin_name: publisher-appoptics

  • kubernetes.yaml file configures the kubernetes collector plugin. This config contain collector.kubernetes.all.events field with specified filter. With this example filter event collector will watch for normal events in default namespace.

    collector:
  kubernetes:
    all:
      incluster: true
      kubeconfigpath: ""
      interval: "60s"

      events: |
        # Embedded YAML (as a multiline string literal)
        filters:
        - namespace: default
          type: normal

      grpc_timeout: 30

load:
  plugin: snap-plugin-collector-aokubernetes
  task: task-aokubernetes.yaml

  • If you want to monitor events count in AppOptics, then edit task-aokubernetes.yaml task manifest so it contains /kubernetes/events/count metric in workflow.collect.metrics list, and copy it to working directory:

    ---
version: 1
schedule:
  type: streaming
deadline: "55s"
workflow:
  collect:
    config:
      /kubernetes:
        MaxCollectDuration: "2s"
        MaxMetricsBuffer: 250
    metrics:
      /kubernetes/events/count: {}
      /kubernetes/pod/*/*/*/status/phase/Running: {}
    publish:
    - plugin_name: publisher-appoptics
      config:
        period: 60
        floor_seconds: 60

  • Once above steps are finished, create 3 configmaps:

    kubectl create configmap plugin-configs --from-file=./examples/event-collector-configs/logs-v2.yaml --from-file=./examples/event-collector-configs/kubernetes.yaml --namespace=kube-system
kubectl create configmap task-manifests --from-file=./examples/event-collector-configs/task-aokubernetes.yaml --namespace=kube-system
kubectl create configmap task-autoload --from-file=./examples/event-collector-configs/task-logs-k8s.yaml --namespace=kube-system

kubectl describe configmaps -n kube-system plugin-configs task-manifests task-autoload

  • Create Events Collector Deployment (it will automatically create corresponding ServiceAccount):

    kubectl apply -k ./deploy/overlays/stable/events-collector/

  • Watch your cluster events in Loggly

Dashboard

Successful deployments will report metrics in the AppOptics Kubernetes Dashboard.

Development

The included Kubernetes resources rely on a Docker image from Docker Hub, see the Dockerfile for more details. You can build and push this by updating the tag in the Makefile and running:

make build-and-release-docker

After new custom Docker image is succesfuly released to Docker Hub, please remember to update corresponding entry in stable overlay for used Kubernetes objects.

About

Docker and Kubernetes assets for running SolarWinds Snap Agent

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

15 tags

Packages

No packages published

Languages

  • Shell 76.7%
  • Makefile 12.3%
  • Dockerfile 11.0%