Skip to content

cloudquery-source doc added #3400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
---
id: cloudquery-source
title: CloudQuery Source
sidebar_label: CloudQuery
tags:
- cloud-to-cloud
- cloudquery
description: Learn how to collect inventory from the AWS APIs and transform them into the CloudQuery schema and send it to Sumo Logic.
---

import CodeBlock from '@theme/CodeBlock';
import ExampleJSON from '/files/c2c/cloudquery/example.json';
import MyComponentSource from '!!raw-loader!/files/c2c/cloudquery/example.json';
import TerraformExample from '!!raw-loader!/files/c2c/cloudquery/example.tf';
import useBaseUrl from '@docusaurus/useBaseUrl';

<img src={useBaseUrl('img/send-data/cloudquery-logo.png')} alt="cloudquery-icon" width="70" />

The CloudQuery integration is used to pull inventory from the AWS APIs and transform them into the CloudQuery schema and send it to Sumo Logic.

:::note
This source is available in the [Fed deployment](https://hub.cloudquery.io/plugins/source/cloudquery/aws/v22.19.2/docs).
:::

## Data collected

| Polling Interval | Data |
| :--- | :--- |
| 24 hours | [Data service table data](https://developer.code42.com/api) |

## Setup

### Source configuration

When you create an CloudQuery source, you add it to a Hosted Collector. Before creating the source, identify the Hosted Collector you want to use or create a new Hosted Collector. For instructions, see [Configure a Hosted Collector](/docs/send-data/hosted-collectors/configure-hosted-collector).

To configure a CloudQueryr Source:
1. In Sumo Logic, select **Manage Data** > **Collection** > **Collection**. 
1. On the Collection page, click **Add Source** next to a Hosted Collector.
1. Search for and select **CloudQuery**.
1. Enter a **Name** for the source. The description is optional.
1. (Optional) For **Source Category**, enter any string to tag the output collected from the Source. Category metadata is stored in a searchable field called `_sourceCategory`.
1. (Optional) **Fields**. Click the **+Add** button to define the fields you want to associate. Each field needs a name (key) and value.
* ![green check circle.png](/img/reuse/green-check-circle.png) A green circle with a check mark is shown when the field exists in the Fields table schema.
* ![orange exclamation point.png](/img/reuse/orange-exclamation-point.png) An orange triangle with an exclamation point is shown when the field doesn't exist in the Fields table schema. In this case, an option to automatically add the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo Logic that does not exist in the Fields schema it is ignored, known as dropped.
1. **AWS Access Key ID**. Enter the Access Key ID collected from the [AWS Management Console](#vendor-configuration).
1. **AWS Secret Access Key**. Enter the Secret Access Key collected from the [AWS Management Console](#vendor-configuration).
1. **Regions**. Identify and enter your Region based on your Base URL.
1. **Sevices**. Enter the type of service from which the data needs to be collected.
1. By default, **Polling Interval** is set to 24 hours.
1. When you are finished configuring the Source, click **Save**.

## JSON schema

Sources can be configured using UTF-8 encoded JSON files with the Collector Management API. See [JSON to configure Sources](/docs/send-data/use-json-configure-sources) for details. 

| Parameter | Type | Value | Required | Description |
|:--|:--|:--|:--|:--|
| schemaRef | JSON Object | `{"type":"CloudQuery"}` | Yes | Define the specific schema type. |
| sourceType | String | `"Universal"` | Yes | Type of source. |
| config | JSON Object | [Configuration object](#configuration-object) | Yes | Source type specific values. |

### Configuration Object

| Parameter | Type | Required | Default | Description | Example |
|:--|:--|:--|:--|:--|:--|
| name | String | Yes | `null` | Type a desired name of the source. The name must be unique per Collector. This value is assigned to the [metadata](/docs/search/get-started-with-search/search-basics/built-in-metadata) field `_source`. | `"mySource"` |
| description | String | No | `null` | Type a description of the source. | `"Testing source"`
| category | String | No | `null` | Type a category of the source. This value is assigned to the [metadata](/docs/search/get-started-with-search/search-basics/built-in-metadata) field `_sourceCategory`. See [best practices](/docs/send-data/best-practices) for details. | `"mySource/test"`
| fields | JSON Object | No | `null` | JSON map of key-value fields (metadata) to apply to the Collector or Source. Use the boolean field `_siemForward` to enable forwarding to SIEM.|`{"_siemForward": false, "fieldA": "valueA"}` |
| awsId | String | Yes | `null` | The AWS access ID that you want to use to authenticate collection requests. | |
| awsKey | String | Yes | `null` | The AWS secret key that you want to use to authenticate collection requests. | |
| limitToRegions | Array | Yes | | List of regions for which the data has to be fetched. | |
| limitToServices | Array | Yes | | List of services for which the data has to be fetched. | |
| pollingInterval | Integer | No | 24 | This sets how often the Source checks for data. | |

### JSON example

<CodeBlock language="json">{MyComponentSource}</CodeBlock>

[Download example](/files/c2c/cloudquery/example.json)

### Terraform example

<CodeBlock language="json">{TerraformExample}</CodeBlock>

[Download example](/files/c2c/cloudquery/example.tf)

## Limitations

- The integration lacks support for fetching service table data from multiple AWS accounts.
- The integration does not support to retrieve data from services like Classic load balancer, Network load balancer, Aurora, and VPC.

## FAQ

:::info
Click [here](/docs/c2c/info) for more information about Cloud-to-Cloud sources.
:::
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,12 @@ In this section, we'll introduce the following concepts:
<p>Provides a secure endpoint to receive System Log data from the Citrix Cloud System Log API.</p>
</div>
</div>
<div className="box smallbox card">
<div className="container">
<a href="/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-source"><img src={useBaseUrl('img/send-data/cloudquery-logo.png')} alt="icon" width="80"/><h4>CloudQuery Source</h4></a>
<p>Learn how to collect inventory from the AWS APIs.</p>
</div>
</div>
<div className="box smallbox card">
<div className="container">
<a href="/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloud-to-cloud-source-versions"><img src={useBaseUrl('img/icons/business/SaaS.png')} alt="icon" width="50"/><h4>Cloud-to-Cloud Versions</h4></a>
Expand Down
1 change: 1 addition & 0 deletions sidebars.ts
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,7 @@ module.exports = {
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/cisco-vulnerability-management-source',
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/citrix-cloud-source',
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloud-to-cloud-source-versions',
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-source',
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/code42-incydr-source',
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/crowdstrike-fdr-host-inventory-source',
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/crowdstrike-fdr-source',
Expand Down
23 changes: 23 additions & 0 deletions static/files/c2c/cloudquery/example.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"api.version": "v1",
"source": {
"config": {
"name": "CloudQuery AWS Inventory",
"accessKeyId": "************",
"secretAccessKey": "***************************",
"limitToRegions": [
"us-east-1",
"us-east-2"
],
"awsServices": [
"ECS",
"EC2"
],
"pollInterval": 24
},
"schemaRef": {
"type": "CloudQuery AWS Inventory"
},
"sourceType": "Universal"
}
}
24 changes: 24 additions & 0 deletions static/files/c2c/cloudquery/example.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
resource "sumologic_cloud_to_cloud_source" "cloudquery_source" {
collector_id = sumologic_collector.collector.id
schema_ref = {
type = "CloudQuery"
}
config = jsonencode({
"name": "CloudQuery AWS Inventory",
"accessKeyId": "************",
"secretAccessKey": "***************************",
"limitToRegions": [
"us-east-1",
"us-east-2"
],
"awsServices": [
"ECS",
"EC2"
],
"pollInterval": 24
})
}
resource "sumologic_collector" "collector" {
name = "my-collector"
description = "Just testing this"
}
Binary file added static/img/send-data/cloudquery-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.