Modernisation Platform Terraform Module Template

Usage

module "example-s3" {
  source                     = "github.com/ministryofjustice/modernisation-platform-terraform-aws-data-firehose"
  cloudwatch_log_group_names = ["example-1", "example-2", "example-3"]
  destination_bucket_arn     = aws_s3_bucket.example.arn
  name                       = "example-s3" # optionally provide name for more descriptive resource names
  tags                       = local.tags
}

module "example-http" {
  source                       = "github.com/ministryofjustice/modernisation-platform-terraform-aws-data-firehose"
  cloudwatch_log_group_names   = ["example-1", "example-2", "example-3"]
  destination_http_endpoint    = "https://example-url.com/endpoint"
  destination_http_secret_name = "http-api-keys/example" # optionally specify name of secret to create
  name                         = "example-http"          # optionally provide name for more descriptive resource names
  tags                         = local.tags
}

This module creates an AWS Data Stream to be used by a set of AWS CloudWatch Log Groups. Data is streamed from the Log Groups to either a target S3 bucket or HTTP endpoint using a Cloudwatch Log Subscription Filter.

When a HTTP endpoint is specified, an aws_secretsmanager_secret resource is created that is polled at 10 minute intervals for credentials.

The aws_secretsmanager_secret value must be populated independently of this module. See AWS Firehose Secrets for details of the format.

Included in this module are the necessary IAM policy documents and roles for these actions, as well as a KMS key to encrypt the Data Stream.

Looking for issues?

If you're looking to raise an issue with this module, please create a new issue in the Modernisation Platform repository.

Requirements

Name	Version
terraform	~> 1.0
aws	~> 6.0
random	~> 3.4

Providers

Name	Version
aws	~> 6.0
random	~> 3.4

Modules

No modules.

Resources

Name	Type
aws_cloudwatch_log_group.kinesis	resource
aws_cloudwatch_log_subscription_filter.cloudwatch-to-firehose	resource
aws_iam_policy.cloudwatch-to-firehose	resource
aws_iam_policy.firehose	resource
aws_iam_policy_attachment.cloudwatch-to-firehose	resource
aws_iam_policy_attachment.firehose	resource
aws_iam_role.cloudwatch-to-firehose	resource
aws_iam_role.firehose	resource
aws_kinesis_firehose_delivery_stream.firehose	resource
aws_kms_alias.firehose	resource
aws_kms_key.firehose	resource
aws_s3_bucket.firehose-errors	resource
aws_s3_bucket_lifecycle_configuration.firehose-errors	resource
aws_s3_bucket_public_access_block.firehose-errors	resource
aws_s3_bucket_server_side_encryption_configuration.firehose-errors	resource
aws_s3_bucket_versioning.firehose-errors	resource
aws_secretsmanager_secret.firehose	resource
random_id.name	resource
aws_caller_identity.current	data source
aws_iam_policy_document.cloudwatch-logs-role-policy	data source
aws_iam_policy_document.cloudwatch-logs-trust-policy	data source
aws_iam_policy_document.firehose-key-policy	data source
aws_iam_policy_document.firehose-role-policy	data source
aws_iam_policy_document.firehose-trust-policy	data source

Inputs

Name	Description	Type	Default	Required
cloudwatch_filter_pattern	A valid CloudWatch Logs filter pattern for subscribing to a filtered stream of log events.	string	""	no
cloudwatch_log_group_names	List of CloudWatch Log Group names to stream logs from.	list(string)	n/a	yes
destination_bucket_arn	ARN of the bucket for CloudWatch filters.	string	""	no
destination_http_endpoint	HTTP endpoint for CloudWatch filters.	string	""	no
destination_http_secret_name	Name of secret to create for http endpoint. Set the value outside of terraform, see https://docs.aws.amazon.com/firehose/latest/dev/secrets-manager-whats-secret.html	string	null	no
name	Optionally provide unique name to help identify resources when multiple instances of module are created, e.g. 'syslog'	string	null	no
s3_compression_format	Allow optional configuration of AWS Data Stream compression. Log Group subscription filters compress logs by default.	string	"UNCOMPRESSED"	no
tags	Map of tags to be applied to resources.	map(string)	n/a	yes

Outputs

Name	Description
cloudwatch_log_group_name	n/a
data_stream	n/a
firehose_server_side_encryption_key_arn	n/a
iam_roles	n/a
kms_key_arn	n/a
log_subscriptions	n/a
secretsmanager_secret_arn	n/a