Skip to content

Added monitors informations to the OTEL apps #4742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Nov 25, 2024
Merged

Added monitors informations to the OTEL apps #4742

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
4b0d4ba
Added monitors informations to the OTEL apps
JV0812 Nov 8, 2024
7dce140
minor path fix
JV0812 Nov 8, 2024
3d7ee88
added alerts information
JV0812 Nov 22, 2024
d1538cc
Merge branch 'main' into add-create-monitors-section-to-otel-apps
JV0812 Nov 22, 2024
c2ac7e4
Fix broken links
jpipkin1 Nov 22, 2024
4f97d3d
Update docs/integrations/app-development/opentelemetry/jmx-openteleme…
JV0812 Nov 25, 2024
e6d7243
Update docs/integrations/databases/opentelemetry/elasticsearch-opente…
JV0812 Nov 25, 2024
a5e7f05
Update docs/integrations/databases/opentelemetry/mysql-opentelemetry.md
JV0812 Nov 25, 2024
55f811f
Update docs/integrations/hosts-operating-systems/opentelemetry/linux-…
JV0812 Nov 25, 2024
9a3ed32
Update docs/integrations/hosts-operating-systems/opentelemetry/macos-…
JV0812 Nov 25, 2024
62466ee
Update docs/integrations/hosts-operating-systems/opentelemetry/window…
JV0812 Nov 25, 2024
a906423
Update docs/integrations/databases/opentelemetry/mysql-opentelemetry.md
JV0812 Nov 25, 2024
e0aab0b
Update docs/integrations/databases/opentelemetry/mysql-opentelemetry.md
JV0812 Nov 25, 2024
27c88ed
Merge branch 'main' into add-create-monitors-section-to-otel-apps
JV0812 Nov 25, 2024
c1a224c
Update docs/integrations/app-development/opentelemetry/jmx-openteleme…
JV0812 Nov 25, 2024
c34ce31
minor fixes
JV0812 Nov 25, 2024
25d5be0
Merge branch 'main' into add-create-monitors-section-to-otel-apps
JV0812 Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 26 additions & 13 deletions docs/integrations/app-development/opentelemetry/jmx-opentelemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ Java Management Extensions (JMX) is a standard component of the Java Platform. J

The Sumo Logic app for JMX allows you to analyze and gain insights about Java applications. The dashboards provide a quick glance at various deployment metrics like memory, GC performance, and thread behavior, so you can troubleshoot unexpected behavior in your Java environment and the applications running in it.

:::info
This app includes [built-in monitors](#jmx-alerts). For details on creating custom monitors, refer to [Create monitors for JMX app](#create-monitors-for-jmx-app).
:::

## Metrics types

The Sumo Logic app for JMX collects metrics from Java applications via the [JMX Receiver for OpenTelemetry](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jmxreceiver).
Expand All @@ -29,7 +33,7 @@ The following types of metrics are collected from JMX:

For more information on different metrics collected, refer to the [JMX receiver docs](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/jvm.md).

### Sample queries
### Sample metrics query

```sql
sumo.datasource=jmx metric=jvm.memory.heap.used
Expand Down Expand Up @@ -100,7 +104,7 @@ Click on the **Download YAML File** button to get the yaml file.

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/JMX-OpenTelemetry/JMX-OTEL-YAML.png' style={{border:'1px solid gray'}} alt="YAML" />

### Step 3: Send logs and metrics to Sumo
### Step 3: Send logs and metrics to Sumo Logic

import LogsIntro from '../../../reuse/apps/opentelemetry/send-logs-intro.md';

Expand Down Expand Up @@ -180,13 +184,11 @@ If you have multiple Java processes running on the same host, then you need to s
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/JMX-OpenTelemetry/JMX-Add-New-Host.png' alt="Add Another Host" />
:::

## Viewing JMX dashboards


## Viewing JMX Dashboards

:::tip Filter with template variables
Template variables provide dynamic dashboards that can rescope data on the fly. As you apply variables to troubleshoot through your dashboard, you view dynamic changes to the data for a quicker resolution to the root cause. You can use template variables to drill down and examine the data on a granular level. For more information, see [Filter with template variables](/docs/dashboards/filter-template-variables.md).
:::
All dashboards have a set of filters that you can apply to the entire dashboard. Use these filters to drill down and examine the data to a granular level.
- You can change the time range for a dashboard or panel by selecting a predefined interval from a drop-down list, choosing a recently used time range, or specifying custom dates and times. [Learn more](/docs/dashboards/set-custom-time-ranges/).
- You can use template variables to drill down and examine the data on a granular level. For more information, see [Filtering Dashboards with Template Variables](/docs/dashboards/filter-template-variables/).

### Overview

Expand Down Expand Up @@ -222,14 +224,10 @@ Use this dashboard to:

### Class Loading and Threads

The **JMX - Class Loading and Threads** dashboard shows key information about the number and type of threads deadlocked, peak, and GC threads of your java virtual machine running on the deployment.

Use this dashboard to:
* Identify abnormal spikes in Threads and Loaded Classes.
The **JMX - Class Loading and Threads** dashboard shows key information about the number and type of threads deadlocked, peak, and GC threads of your Java virtual machine running on the deployment. Use this dashboard to identify the abnormal spikes in Threads and Loaded Classes.

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/JMX-OpenTelemetry/JMX-Class-Loading-and-Threads.png' alt="Class Loading and Threads" />


### Memory Pool

The **JMX - Memory Pool** dashboard provides key information about the memory pool usage, peak usage, collection usage, garbage collection across various memory pools of your Java virtual machine.
Expand All @@ -239,3 +237,18 @@ Use this dashboard to:
* Gain insights into garbage collection impact on different memory pools.

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/JMX-OpenTelemetry/JMX-Memory-Pool.png' alt="Memory Pool" />

## Create monitors for JMX app

import CreateMonitors from '../../../reuse/apps/create-monitors.md';

<CreateMonitors/>

### JMX alerts

| Alert Name | Alert Description and conditions | Alert Condition | Recover Condition |
|:--|:--|:--|:--|
| `JMX - High Heap Memory Usage Alert` | This alert gets triggered when heap memory usage exceeds threshold. | Count < 20 | Count >= 20 |
| `JMX - High Memory Pool Usage Alert` | This alert gets triggered when memory pool usage exceeds threshold. | Count < 20 | Count >= 20 |
| `JMX - High Non Heap Memory Usage Alert` | This alert gets triggered when non heap memory usage exceeds threshold. | Count < 20 | Count >= 20 |
| `JMX - High Number Of Classes Loaded Alert` | This alert gets triggered when high number of classes are loaded. | Count > 1000 | Count < = 1000 |
93 changes: 51 additions & 42 deletions docs/integrations/containers-orchestration/opentelemetry/activemq-opentelemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,31 +19,34 @@ The diagram below illustrates the components of the ActiveMQ collection for each

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/ActiveMQ-OpenTelemetry/ActiveMQ-OTel-Collection-architecture.png' alt="ActiveMQ OTel Collection architecture" />

This app has been tested with following ActiveMQ versions:
* 5.17.4
* 5.18.2
This app has been tested with the following ActiveMQ versions:
* `5.17.4`
* `5.18.2`

:::info
This app includes [built-in monitors](#activemq-alerts). For details on creating custom monitors, refer to [Create monitors for ActiveMQ app](#create-monitors-for-activemq-app).
:::

## Log and metrics types

The Sumo Logic App for ActiveMQ assumes:
The Sumo Logic app for ActiveMQ uses:

- ActiveMQ app supports the below log format in the log4j2.properties file.
- ActiveMQ app supports the below log format in the `log4j2.properties` file.

* [Audit Logs](https://activemq.apache.org/audit-logging): Every management action made through JMX or Web Console management interface is logged in audit log files.
* [Audit Logs](https://activemq.apache.org/audit-logging). Every management action made through JMX or Web Console management interface is logged in audit log files.

`appender.auditlog.layout.pattern=%d | %-5p | %m | %t%n`

* ActiveMQ Logs:

`appender.logfile.layout.pattern=%d | %-5p | %m | %c | %t%n%throwable{full}`


- For a list of metrics that are collected and used by the app, see [ActiveMQ Metrics](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/activemq.md).
- For a list of metrics that are collected and used by the app, refer to the [ActiveMQ Metrics](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/activemq.md).


## Fields creation in Sumo Logic for ActiveMQ

Following are the [fields](/docs/manage/fields/) which will be created as part of ActiveMQ App install if not already present.
Following are the [fields](/docs/manage/fields/) which will be created as part of ActiveMQ app installation, if not already present.

* `messaging.cluster.name`. User configured. Enter a name to uniquely identify your ActiveMQ cluster. This cluster name will be shown in the Sumo Logic dashboards.
* `messaging.node.name`. Has value of `host name`.
Expand All @@ -55,7 +58,7 @@ If process metrics are enabled it will also create [fields for JMX metrics](/doc

## Prerequisites

### Configure logging in ActiveMQ:
### Configure logging in ActiveMQ

1. By default, ActiveMQ logs (`audit.log` and `activemq.log`) are stored in the directory called `${ACTIVEMQ_HOME}/data/activemq.log`. Make a note of this logs directory.
1. [Enable auditing](https://activemq.apache.org/audit-logging) if not enabled by default.
Expand Down Expand Up @@ -135,7 +138,7 @@ Click on the **Download YAML File** button to get the YAML file.

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/ActiveMQ-OpenTelemetry/ActiveMQ-OTEL-YAML.png' style={{border:'1px solid gray'}} alt="YAML" />

### Step 3: Send logs and metrics to Sumo
### Step 3: Send logs and metrics to Sumo Logic

import LogsIntro from '../../../reuse/apps/opentelemetry/send-logs-intro.md';

Expand Down Expand Up @@ -216,26 +219,32 @@ import LogsOutro from '../../../reuse/apps/opentelemetry/send-logs-outro.md';

## Sample log messages

*activemq.log*

```json
```json title="ActiveMQ Logs"
2021-06-22 15:00:41,922 | DEBUG | Stopping transport tcp:///192.168.100.8:36302@61616 | org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ BrokerService[localhost] Task-15300
```

*audit.log*

```json
```json title="Audit Logs"
2010-12-22 12:12:07,225 | INFO | admin requested /admin/createDestination.action [JMSDestination='test' JMSDestinationType='queue' secret='4eb0bc3e-9d7a-4256-844c-24f40fda98f1' ] from 127.0.0.1 | qtp12205619-39
2010-12-22 12:12:14,512 | INFO | admin requested /admin/purgeDestination.action [JMSDestination='test' JMSDestinationType='queue' secret='eff6a932-1b58-45da-a64a-1b30b246cfc9' ] from 127.0.0.1 | qtp12205619-36
2010-12-22 12:12:17,802 | INFO | admin requested /admin/sendMessage.action [JMSTimeToLive='' JMSXGroupSeq='' AMQ_SCHEDULED_DELAY='' JMSType='' JMSMessageCountHeader='JMSXMessageCounter' JMSXGroupID='' JMSReplyTo='' JMSDestination='test' AMQ_SCHEDULED_PERIOD='' JMSText='Enter some text
here for the message body...' JMSDestinationType='queue' AMQ_SCHEDULED_CRON='' JMSCorrelationID='' AMQ_SCHEDULED_REPEAT='' JMSMessageCount='1' secret='a0e1df62-14d6-4425-82a2-17aa01a16e7d' JMSPriority='' ] from 127.0.0.1 | qtp12205619-37
```

## Sample metrics

```sql
"Query","metric","deployment.environment","host.name","messaging.cluster.name","messaging.node.name","messaging.system","os.type","sumo.datasource","broker","destination","unit","latest"
```

```sql
"#A","activemq.message.wait_time.avg","testprod","ip-10-0-10-92","activemq_cluster","ip-10-0-10-92","activemq","linux","activemq","localhost","testtopic","ms","254.4"
```

## Sample queries

### Log query

This sample Query is from the **Events by Severity** panel of the **ActiveMQ - Logs** dashboard.
This sample log query is from the **Events by Severity** panel of the **ActiveMQ - Logs** dashboard.

```sql
sumo.datasource=activemq deployment.environment={{deployment.environment}} messaging.cluster.name={{messaging.cluster.name}} messaging.node.name={{messaging.node.name}}
Expand All @@ -246,29 +255,17 @@ sumo.datasource=activemq deployment.environment={{deployment.environment}} messa

### Metrics query

Sample query from **Average Enqueue Latency** panel in **ActiveMQ - Destinations** dashboard.
This sample metrics query from the **Average Enqueue Latency** panel of the **ActiveMQ - Destinations** dashboard.

```
```sql
sumo.datasource=activemq deployment.environment={{deployment.environment}} messaging.cluster.name={{messaging.cluster.name}} messaging.node.name={{messaging.node.name}} destination={{destination}} !(destination=activemq.*) metric=activemq.message.wait_time.avg | avg by destination,messaging.cluster.name
```


## Sample metrics

```
"Query","metric","deployment.environment","host.name","messaging.cluster.name","messaging.node.name","messaging.system","os.type","sumo.datasource","broker","destination","unit","latest"
```

```
"#A","activemq.message.wait_time.avg","testprod","ip-10-0-10-92","activemq_cluster","ip-10-0-10-92","activemq","linux","activemq","localhost","testtopic","ms","254.4"
```


## Viewing the ActiveMQ dashboards

### Dashboard filters with template variables

Template variables provide dynamic dashboards that rescope data on the fly. As you apply variables to troubleshoot through your dashboard, you can view dynamic changes to the data for a fast resolution to the root cause. For more information, see the [Filter with template variables](/docs/dashboards/filter-template-variables.md) help page.
All dashboards have a set of filters that you can apply to the entire dashboard. Use these filters to drill down and examine the data to a granular level.
- You can change the time range for a dashboard or panel by selecting a predefined interval from a drop-down list, choosing a recently used time range, or specifying custom dates and times. [Learn more](/docs/dashboards/set-custom-time-ranges/).
- You can use template variables to drill down and examine the data on a granular level. For more information, see [Filtering Dashboards with Template Variables](/docs/dashboards/filter-template-variables/).

### Overview

Expand All @@ -283,7 +280,6 @@ Use this dashboard to:

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/ActiveMQ-OpenTelemetry/ActiveMQ-Overview.png' alt="ActiveMQ dashboards" />


### Brokers

The **ActiveMQ - Brokers** dashboard provides an at-a-glance view of the state of your brokers in the ActiveMQ cluster.
Expand All @@ -295,8 +291,6 @@ Use this dashboard to:

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/ActiveMQ-OpenTelemetry/ActiveMQ-Brokers.png' alt="ActiveMQ dashboards" />



### Destinations

The **ActiveMQ - Destinations** dashboard provides an at-a-glance view of the state of your topics/queues in ActiveMQ clusters.
Expand All @@ -307,10 +301,8 @@ Use this dashboard to:
* Monitor producers, consumers and expired messages on topics/queues.
* Determine the number of topics/queues.


<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/ActiveMQ-OpenTelemetry/ActiveMQ-Destinations.png' alt="ActiveMQ dashboards" />


### Resource utilization

The **ActiveMQ - Resource Utilization** dashboard provides an at-a-glance view of the state of system loads in clusters: Virtual memory usage, physical memory usage, CPU utilization, disk operations, disk writes, and I/O wait.
Expand All @@ -322,8 +314,6 @@ Use this dashboard to:

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/ActiveMQ-OpenTelemetry/ActiveMQ-Resource-Utilization.png' alt="ActiveMQ dashboards" />



### Logs

The **ActiveMQ - Logs** dashboard helps you quickly analyze your ActiveMQ error logs across all clusters.
Expand All @@ -347,3 +337,22 @@ Use this dashboard to:
* Quickly determine patterns across all audit logs in a given ActiveMQ cluster.

<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/ActiveMQ-OpenTelemetry/ActiveMQ-Audit.png' alt="ActiveMQ dashboards" />

## Create monitors for ActiveMQ app

import CreateMonitors from '../../../reuse/apps/create-monitors.md';

<CreateMonitors/>

### ActiveMQ alerts

| Alert Name | Alert Description and conditions | Alert Condition | Recover Condition |
|:--|:--|:--|:--|
| `ActiveMQ - High CPU Usage Alert` | This alert gets triggered when CPU usage on a node in a ActiveMQ cluster is high. | Count >= 80 | Count < 80 |
| `ActiveMQ - High Memory Usage Alert` | This alert gets triggered when memory usage on a node in a ActiveMQ cluster is high. | Count >= 80 | Count < 80 |
| `ActiveMQ - High Storage Used Alert` | This alert gets triggered when there is high store usage on a node in a ActiveMQ cluster. | Count >= 80 | Count < 80 |
| `ActiveMQ - Maximum Connection Alert` | This alert gets triggered when one node in ActiveMQ cluster exceeds the maximum allowed client connection limit. | Count >= 1 | Count < 1 |
| `ActiveMQ - No Consumers on Queues Alert` | This alert gets triggered when a ActiveMQ queue has no consumers. | Count < 1 | Count >= 1 |
| `ActiveMQ - Node Down Alert` | This alert gets triggered when a node in the ActiveMQ cluster is down. | Count >= 1 | Count < 1 |
| `ActiveMQ - Too Many Connections Alert` | This alert gets triggered when there are too many connections to a node in a ActiveMQ cluster. | Count >= 1000 | Count < 1000 |

Loading
Loading