Skip to content

Commit 36ce9c6

Browse files
mirii1994mirii1994
authored
S3 fetcher (#154)
* create * resource * resource, datasource, tests * examples * docs, readme * changelog * update logzio client version, fix code * add check for authentication methods, add test for it
1 parent b24892d commit 36ce9c6

15 files changed

+878
-18
lines changed

‎docs/data-sources/s3_fetcher.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# S3 Fetcher Datasource
2+
3+
Use this data source to access information about existing Logz.io S3 Fetchers.
4+
5+
## Argument Reference
6+
7+
* `fetcher_id` - ID of the S3 Fetcher.
8+
9+
## Attribute Reference
10+
11+
* `aws_access_key` - (String) AWS S3 bucket access key. Not applicable if you chose to authenticate with `aws_arn`.
12+
* `aws_arn` - (String) Amazon Resource Name (ARN) to uniquely identify the S3 bucket. Not applicable if you choose to authenticate with AWS keys (access key & secret key).
13+
* `bucket_name` - (String) AWS S3 bucket name.
14+
* `active` - (Boolean) If true, the S3 bucket connector is active and logs are being fetched to Logz.io. If false, the connector is disabled.
15+
* `aws_region` - (String) Bucket's region. Allowed values: `US_EAST_1`, `US_EAST_2`, `US_WEST_1`, `US_WEST_2`, `EU_WEST_1`, `EU_WEST_2`, `EU_WEST_3`, `EU_CENTRAL_1`, `AP_NORTHEAST_1`, `AP_NORTHEAST_2`, `AP_SOUTHEAST_1`, `AP_SOUTHEAST_2`, `SA_EAST_1`, `AP_SOUTH_1`, `CA_CENTRAL_1`.
16+
* `logs_type` - (String) Specifies the log type being sent to Logz.io. Determines the parsing pipeline used to parse and map the logs. [Learn more about parsing options supported by Logz.io](https://docs.logz.io/user-guide/log-shipping/built-in-log-types.html). Allowed values: `elb`, `vpcflow`, `S3Access`, `cloudfront`.
17+
* `prefix` - (String) Prefix of the AWS S3 bucket.
18+
* `add_s3_object_key_as_log_field` - (Boolean) If `true`, enriches logs with a new field detailing the S3 object key.

‎docs/resources/s3_fetcher.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# S3 Fetcher Provider
2+
3+
Provides a Logz.io S3 Fetcher resource. This can be used to create and manage S3 Fetcher instances, that retrieve logs stored in AWS S3 Buckets.
4+
5+
* Learn more about S3 Fetcher in the [Logz.io Docs](https://docs.logz.io/api/#tag/Connect-to-S3-Buckets).
6+
7+
## Example Usage
8+
9+
```hcl
10+
resource "logzio_s3_fetcher" "my_s3_fetcher_keys" {
11+
aws_access_key = "my_access_key"
12+
aws_secret_key = "my_secret_key"
13+
bucket_name = "my_bucket"
14+
active = true
15+
add_s3_object_key_as_log_field = false
16+
aws_region = "EU_WEST_3"
17+
logs_type = "S3Access"
18+
}
19+
20+
resource "logzio_s3_fetcher" "my_s3_fetcher_arn" {
21+
aws_arn = "my_arn"
22+
bucket_name = "my_bucket"
23+
active = true
24+
add_s3_object_key_as_log_field = false
25+
aws_region = "AP_SOUTHEAST_2"
26+
logs_type = "cloudfront"
27+
}
28+
```
29+
30+
## Argument Reference
31+
32+
### Required:
33+
34+
* `aws_access_key` - (String) AWS S3 bucket access key. Not applicable if you choose to authenticate with `aws_arn`. If you choose to authenticate with AWS keys, both `aws_access_key` and `aws_secret key` must be set.
35+
* `aws_secret_key` - (String) AWS S3 bucket secret key. Not applicable if you choose to authenticate with `aws_arn`. If you choose to authenticate with AWS keys, both `aws_access_key` and `aws_secret key` must be set.
36+
* `aws_arn` - (String) Amazon Resource Name (ARN) to uniquely identify the S3 bucket. To generate a new ARN, create a new IAM Role in your AWS admin console. Not applicable if you choose to authenticate with AWS keys (access key & secret key).
37+
* `bucket_name` - (String) AWS S3 bucket name.
38+
* `active` - (Boolean) If true, the S3 bucket connector is active and logs are being fetched to Logz.io. If false, the connector is disabled.
39+
* `aws_region` - (String) Bucket's region. Specify one supported AWS region: `US_EAST_1`, `US_EAST_2`, `US_WEST_1`, `US_WEST_2`, `EU_WEST_1`, `EU_WEST_2`, `EU_WEST_3`, `EU_CENTRAL_1`, `AP_NORTHEAST_1`, `AP_NORTHEAST_2`, `AP_SOUTHEAST_1`, `AP_SOUTHEAST_2`, `SA_EAST_1`, `AP_SOUTH_1`, `CA_CENTRAL_1`.
40+
* `logs_type` - (String) Specifies the log type being sent to Logz.io. Determines the parsing pipeline used to parse and map the logs. [Learn more about parsing options supported by Logz.io](https://docs.logz.io/user-guide/log-shipping/built-in-log-types.html). Allowed values: `elb`, `vpcflow`, `S3Access`, `cloudfront`.
41+
42+
### Optional:
43+
44+
* `prefix` - (String) Prefix of the AWS S3 bucket.
45+
* `add_s3_object_key_as_log_field` - (Boolean) Defaults to `false`. If `true`, enriches logs with a new field detailing the S3 object key.
46+
47+
48+
## Attribute Reference
49+
50+
* `fetcher_id` - (Int) Id of the created fetcher.
51+
52+
### Import S3 Fetcher as Terraform resource
53+
54+
You can import existing S3 Fetcher as follows:
55+
56+
```
57+
terraform import logzio_s3_fetcher.my_fetcher <FETCHER-ID>
58+
```

‎examples/s3_fetcher/main.tf

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
variable "api_token" {
2+
type = string
3+
description = "your logzio API token"
4+
}
5+
6+
provider "logzio" {
7+
api_token = var.api_token
8+
}
9+
10+
resource "logzio_s3_fetcher" "my_s3_fetcher_keys" {
11+
aws_access_key = "my_access_key"
12+
aws_secret_key = "my_secret_key"
13+
bucket_name = "my_bucket"
14+
active = true
15+
add_s3_object_key_as_log_field = false
16+
aws_region = "EU_WEST_3"
17+
logs_type = "S3Access"
18+
}
19+
20+
resource "logzio_s3_fetcher" "my_s3_fetcher_arn" {
21+
aws_arn = "my_arn"
22+
bucket_name = "my_bucket"
23+
active = true
24+
add_s3_object_key_as_log_field = false
25+
aws_region = "AP_SOUTHEAST_2"
26+
logs_type = "cloudfront"
27+
}

‎examples/s3_fetcher/terraform-destroy.sh

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env bash
2+
export TF_LOG=DEBUG
3+
TF_VAR_api_token=${LOGZIO_API_TOKEN} terraform destroy

‎examples/s3_fetcher/terraform.sh

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/usr/bin/env bash
2+
export TF_LOG=DEBUG
3+
terraform init
4+
TF_VAR_api_token=${LOGZIO_API_TOKEN} terraform plan -out terraform.plan
5+
terraform apply terraform.plan

‎go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ require (
77
github.com/hashicorp/go-cty v1.4.1-0.20200414143053-d3edf31b6320
88
github.com/hashicorp/terraform-plugin-log v0.7.0
99
github.com/hashicorp/terraform-plugin-sdk/v2 v2.24.1
10-
github.com/logzio/logzio_terraform_client v1.14.0
10+
github.com/logzio/logzio_terraform_client v1.15.0
1111
github.com/stretchr/testify v1.7.2
1212
)
1313

‎go.sum

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,8 +129,8 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
129129
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
130130
github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348/go.mod h1:B69LEHPfb2qLo0BaaOLcbitczOKLWTsrBG9LczfCD4k=
131131
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
132-
github.com/logzio/logzio_terraform_client v1.14.0 h1:pAmbPdWyYvmuKU8m2vQ/VJJu8WbmfeBsnDTvvBP9wmc=
133-
github.com/logzio/logzio_terraform_client v1.14.0/go.mod h1:hEQixCq9RPpvyzWerxIWKf0SYgangyWpPeogN7nytC0=
132+
github.com/logzio/logzio_terraform_client v1.15.0 h1:FQVhIKkPbwnWSee/j/X7QDULZntY81Kd7H9twPyqG54=
133+
github.com/logzio/logzio_terraform_client v1.15.0/go.mod h1:hEQixCq9RPpvyzWerxIWKf0SYgangyWpPeogN7nytC0=
134134
github.com/matryer/is v1.2.0/go.mod h1:2fLPjFQM9rhQ15aVEtbuwhJinnOqrmgXPNdZsdwlWXA=
135135
github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
136136
github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40=

‎logzio/datasource_s3_fetcher.go

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
package logzio
2+
3+
import (
4+
"context"
5+
"github.com/hashicorp/terraform-plugin-sdk/v2/diag"
6+
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
7+
"github.com/logzio/logzio_terraform_client/s3_fetcher"
8+
"strconv"
9+
)
10+
11+
func dataSourceS3Fetcher() *schema.Resource {
12+
return &schema.Resource{
13+
ReadContext: dataSourceS3FetcherRead,
14+
Schema: map[string]*schema.Schema{
15+
s3FetcherId: {
16+
Type: schema.TypeInt,
17+
Required: true,
18+
},
19+
s3FetcherAccessKey: {
20+
Type: schema.TypeString,
21+
Computed: true,
22+
Optional: true,
23+
Sensitive: true,
24+
},
25+
s3FetcherArn: {
26+
Type: schema.TypeString,
27+
Computed: true,
28+
Optional: true,
29+
Sensitive: true,
30+
},
31+
s3FetcherBucket: {
32+
Type: schema.TypeString,
33+
Computed: true,
34+
},
35+
s3FetcherPrefix: {
36+
Type: schema.TypeString,
37+
Optional: true,
38+
Computed: true,
39+
},
40+
s3FetcherActive: {
41+
Type: schema.TypeBool,
42+
Computed: true,
43+
},
44+
s3FetcherAddS3ObjectKeyAsLogField: {
45+
Type: schema.TypeBool,
46+
Computed: true,
47+
},
48+
s3FetcherRegion: {
49+
Type: schema.TypeString,
50+
Computed: true,
51+
},
52+
s3FetcherLogsType: {
53+
Type: schema.TypeString,
54+
Computed: true,
55+
},
56+
},
57+
}
58+
}
59+
60+
func dataSourceS3FetcherRead(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {
61+
id, ok := d.GetOk(s3FetcherId)
62+
if ok {
63+
client, err := s3_fetcher.New(m.(Config).apiToken, m.(Config).baseUrl)
64+
if err != nil {
65+
return diag.FromErr(err)
66+
}
67+
68+
fetcher, err := client.GetS3Fetcher(int64(id.(int)))
69+
if err != nil {
70+
return diag.FromErr(err)
71+
}
72+
73+
d.SetId(strconv.FormatInt(fetcher.Id, 10))
74+
setS3Fetcher(d, *fetcher)
75+
76+
return nil
77+
}
78+
79+
return diag.Errorf("could not get fetcher id")
80+
}

‎logzio/datasource_s3_fetcher_test.go

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
package logzio
2+
3+
import (
4+
"fmt"
5+
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
6+
"github.com/logzio/logzio_terraform_client/s3_fetcher"
7+
"github.com/logzio/logzio_terraform_provider/logzio/utils"
8+
"testing"
9+
"time"
10+
)
11+
12+
func TestAccDataSourceS3Fetcher(t *testing.T) {
13+
defer utils.SleepAfterTest()
14+
15+
resource.Test(t, resource.TestCase{
16+
PreCheck: func() { testAccPreCheckApiToken(t) },
17+
ProviderFactories: testAccProviderFactories,
18+
Steps: []resource.TestStep{
19+
{
20+
Config: getS3FetcherConfigKeys(false),
21+
Check: resource.ComposeTestCheckFunc(
22+
resource.TestCheckResourceAttr("logzio_s3_fetcher.test_fetcher", s3FetcherBucket, bucketName),
23+
resource.TestCheckResourceAttrSet("logzio_s3_fetcher.test_fetcher", s3FetcherAccessKey),
24+
resource.TestCheckResourceAttrSet("logzio_s3_fetcher.test_fetcher", s3FetcherSecretKey),
25+
resource.TestCheckResourceAttrSet("logzio_s3_fetcher.test_fetcher", s3FetcherId),
26+
resource.TestCheckResourceAttr("logzio_s3_fetcher.test_fetcher", s3FetcherArn, ""),
27+
resource.TestCheckResourceAttr("logzio_s3_fetcher.test_fetcher", s3FetcherActive, "false"),
28+
resource.TestCheckResourceAttr("logzio_s3_fetcher.test_fetcher", s3FetcherAddS3ObjectKeyAsLogField, "false"),
29+
resource.TestCheckResourceAttr("logzio_s3_fetcher.test_fetcher", s3FetcherRegion, s3_fetcher.RegionUsEast1.String()),
30+
resource.TestCheckResourceAttr("logzio_s3_fetcher.test_fetcher", s3FetcherLogsType, s3_fetcher.LogsTypeElb.String()),
31+
),
32+
},
33+
{
34+
PreConfig: func() {
35+
time.Sleep(time.Second * 3)
36+
},
37+
Config: getS3FetcherConfigKeys(false) + getDataSourceS3Fetcher(),
38+
Check: resource.ComposeTestCheckFunc(
39+
resource.TestCheckResourceAttr("data.logzio_s3_fetcher.my_ds_fetcher", s3FetcherBucket, bucketName),
40+
resource.TestCheckResourceAttrSet("data.logzio_s3_fetcher.my_ds_fetcher", s3FetcherAccessKey),
41+
resource.TestCheckResourceAttrSet("data.logzio_s3_fetcher.my_ds_fetcher", s3FetcherId),
42+
resource.TestCheckResourceAttr("data.logzio_s3_fetcher.my_ds_fetcher", s3FetcherArn, ""),
43+
resource.TestCheckResourceAttr("data.logzio_s3_fetcher.my_ds_fetcher", s3FetcherActive, "false"),
44+
resource.TestCheckResourceAttr("data.logzio_s3_fetcher.my_ds_fetcher", s3FetcherAddS3ObjectKeyAsLogField, "false"),
45+
resource.TestCheckResourceAttr("data.logzio_s3_fetcher.my_ds_fetcher", s3FetcherRegion, s3_fetcher.RegionUsEast1.String()),
46+
resource.TestCheckResourceAttr("data.logzio_s3_fetcher.my_ds_fetcher", s3FetcherLogsType, s3_fetcher.LogsTypeElb.String()),
47+
),
48+
},
49+
},
50+
})
51+
}
52+
53+
func getDataSourceS3Fetcher() string {
54+
return fmt.Sprintf(`
55+
data "logzio_s3_fetcher" "my_ds_fetcher" {
56+
fetcher_id = "${logzio_s3_fetcher.test_fetcher.fetcher_id}"
57+
depends_on = ["logzio_s3_fetcher.test_fetcher"]
58+
}`)
59+
}

‎logzio/provider.go

Lines changed: 3 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,20 +22,10 @@ const (
2222
resourceRestoreLogsType = "logzio_restore_logs"
2323
resourceAuthenticationGroupsType = "logzio_authentication_groups"
2424
resourceKibanaObjectType = "logzio_kibana_object"
25+
resourceS3FetcherType = "logzio_s3_fetcher"
2526
envLogzioApiToken = "LOGZIO_API_TOKEN"
2627
envLogzioRegion = "LOGZIO_REGION"
2728
envLogzioBaseURL = "LOGZIO_BASE_URL"
28-
envLogzioAccountId = "LOGZIO_ACCOUNT_ID"
29-
envLogzioS3Path = "S3_PATH"
30-
envLogzioAwsAccessKey = "AWS_ACCESS_KEY"
31-
envLogzioAwsSecretKey = "AWS_SECRET_KEY"
32-
envLogzioAwsArn = "AWS_ARN"
33-
envLogzioAzureAccountName = "AZURE_ACCOUNT_NAME"
34-
envLogzioAzureClientId = "AZURE_CLIENT_ID"
35-
envLogzioAzureClientSecret = "AZURE_CLIENT_SECRET"
36-
envLogzioAzureContainerName = "AZURE_CONTAINER_NAME"
37-
envLogzioAzureTenantId = "AZURE_TENANT_ID"
38-
envLogzioAzurePath = "BLOB_PATH"
3929

4030
baseUrl = "https://api%s.logz.io"
4131
)
@@ -69,6 +59,7 @@ func Provider() *schema.Provider {
6959
resourceRestoreLogsType: dataSourceRestoreLogs(),
7060
resourceAuthenticationGroupsType: dataSourceAuthenticationGroups(),
7161
resourceKibanaObjectType: dataSourceKibanaObject(),
62+
resourceS3FetcherType: dataSourceS3Fetcher(),
7263
},
7364
ResourcesMap: map[string]*schema.Resource{
7465
resourceEndpointType: resourceEndpoint(),
@@ -81,6 +72,7 @@ func Provider() *schema.Provider {
8172
resourceRestoreLogsType: resourceRestoreLogs(),
8273
resourceAuthenticationGroupsType: resourceAuthenticationGroups(),
8374
resourceKibanaObjectType: resourceKibanaObject(),
75+
resourceS3FetcherType: resourceS3Fetcher(),
8476
},
8577
ConfigureContextFunc: providerConfigureWrapper,
8678
}

‎logzio/provider_test.go

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,21 @@ import (
77
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
88
)
99

10+
const (
11+
envLogzioAccountId = "LOGZIO_ACCOUNT_ID"
12+
envLogzioS3Path = "S3_PATH"
13+
envLogzioAwsAccessKey = "AWS_ACCESS_KEY"
14+
envLogzioAwsSecretKey = "AWS_SECRET_KEY"
15+
envLogzioAwsArn = "AWS_ARN"
16+
envLogzioAwsArnS3Fetcher = "AWS_ARN_S3_FETCHER"
17+
envLogzioAzureAccountName = "AZURE_ACCOUNT_NAME"
18+
envLogzioAzureClientId = "AZURE_CLIENT_ID"
19+
envLogzioAzureClientSecret = "AZURE_CLIENT_SECRET"
20+
envLogzioAzureContainerName = "AZURE_CONTAINER_NAME"
21+
envLogzioAzureTenantId = "AZURE_TENANT_ID"
22+
envLogzioAzurePath = "BLOB_PATH"
23+
)
24+
1025
var (
1126
testAccExpectedAlertChannelName string
1227
testAccExpectedApplicationName string

0 commit comments

Comments
 (0)