Add custom build support for lambdacomponents

README.md

@@ -25,6 +25,8 @@ Some layers include the corresponding OTel language SDK for the Lambda. This all
 
 * **What exporters/receivers/processors are included from the OpenTelemetry Collector?**
     > You can check out [the stripped-down collector's imports](https://github.com/open-telemetry/opentelemetry-lambda/blob/main/collector/lambdacomponents/default.go#L18) in this repository for a full list of currently included components.
+
+    > Self-built binaries of the collector have **experimental** support for a custom set of connectors/exporters/receivers/processors. For more information, see [(Experimental) Customized collector build](./collector/README.md#experimental-customized-collector-build)
 * **Is the Lambda layer provided or do I need to build it and distribute it myself?**
     > This repository provides pre-built Lambda layers, their ARNs are available in the [Releases](https://github.com/open-telemetry/opentelemetry-lambda/releases). You can also build the layers manually and publish them in your AWS account. This repo has files to facilitate doing that. More information is provided in [the Collector folder's README](collector/README.md).
 

collector/Makefile

@@ -7,7 +7,7 @@ LAYER_NAME:=otel-collector
 OTELCOL_VERSION="UNSET" # run recipe set-otelcol-version to get this
 GIT_SHA=$(shell git rev-parse HEAD)
 GOARCH ?= amd64
-GOBUILD=GO111MODULE=on CGO_ENABLED=0 installsuffix=cgo go build -trimpath
+GOBUILD=GO111MODULE=on CGO_ENABLED=0 installsuffix=cgo go build -trimpath$(if ${BUILDTAGS}, -tags "${BUILDTAGS}",)
 BUILD_INFO_IMPORT_PATH=main
 
 LDFLAGS=-ldflags "-s -w -X $(BUILD_INFO_IMPORT_PATH).GitHash=$(GIT_SHA) -X $(BUILD_INFO_IMPORT_PATH).Version=$(OTELCOL_VERSION) \

collector/README.md

@@ -13,6 +13,45 @@ Be sure to:
 * Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
 * Config [AWS credential](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
 
+## (Experimental) Customized collector build
+The collector can be built with a customized set of connectors/exporters/receivers/processors. This feature is **experimental** and is only supported for self-built binaries of the collector.
+
+### Build Tags
+The build-tag `lambdacomponents.custom` must always be provided to opt-in for a custom build.
+Once this build-tag is present, you need provide additional build-tags to include your desired components in the resulting binary:
+
+- `lambdacomponents.all` includes all available components
+- `lambdacomponents.connector.all` includes all available connectors
+- `lambdacomponents.exporter.all` includes all available exporters
+- `lambdacomponents.extension.all` includes all available extensions
+- `lambdacomponents.processor.all` includes all available processors
+- `lambdacomponents.receiver.all` includes all available receivers
+
+Each available component can also be included explicitly by using its specific build-tag. For a full-list of available components, have a look into the [lambdacomponents](./collector/lambdacomponents) package.
+
+As an example, the full command to publish OpenTelemetry Collector Lambda layer in your AWS account and get its ARN including the following components:
+- All receivers
+- All processors
+- No extensions
+- Only the otlphttp exporter
+- Only the spanmetrics connector
+
+would be the following:
+```shell
+cd collector && BUILDTAGS="lambdacomponents.custom,lambdacomponents.receiver.all,lambdacomponents.processor.all,lambdacomponents.exporter.otlphttp,lambdacomponents.connector.spanmetrics" make publish-layer
+```
+
+### Adding additional options
+To add more options for a customized build, you can add your desired component to the [lambdacomponents](./collector/lambdacomponents) package.
+Make sure to always restrict your addition using the appropriate build-tags.
+
+For example, if you want to add the extension `foo`, the file providing this extension should be located in the [extension](./collector/lambdacomponents/extension) directory have the following build restriction:
+```
+//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.extension.all || lambdacomponents.extension.foo)
+```
+
+You can provide your addition as a pull-request to this repository. Before doing so, please also read through the details of [Contributing](#contributing) to this project.
+
 ## Installing
 To install the OpenTelemetry Collector Lambda layer to an existing Lambda function using the `aws` CLI:
 

collector/go.mod

@@ -79,14 +79,17 @@ require (
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 // indirect
 	github.com/hashicorp/go-version v1.7.0 // indirect
+	github.com/hashicorp/golang-lru v1.0.2 // indirect
 	github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
 	github.com/iancoleman/strcase v0.3.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/jonboulle/clockwork v0.4.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/klauspost/compress v1.17.9 // indirect
 	github.com/knadh/koanf/maps v0.1.1 // indirect
 	github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
 	github.com/knadh/koanf/v2 v2.1.1 // indirect
+	github.com/lightstep/go-expohisto v1.0.0 // indirect
 	github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
 	github.com/magefile/mage v1.15.0 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
@@ -95,6 +98,7 @@ require (
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/mostynb/go-grpc-compression v1.2.3 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.107.0 // indirect
 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.107.0 // indirect
 	github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.107.0 // indirect
 	github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.107.0 // indirect

collector/go.sum

@@ -90,6 +90,8 @@ github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 h1:bkypFPDjIYGfCYD5mRBvpqxfYX1
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0/go.mod h1:P+Lt/0by1T8bfcF3z737NnSbmxQAppXMRziHUxPOC8k=
 github.com/hashicorp/go-version v1.7.0 h1:5tqGy27NaOTB8yJKUZELlFAS/LTKJkrmONwQKeRZfjY=
 github.com/hashicorp/go-version v1.7.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
+github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c=
+github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
@@ -98,6 +100,8 @@ github.com/iancoleman/strcase v0.3.0 h1:nTXanmYxhfFAMjZL34Ov6gkzEsSJZ5DbhxWjvSAS
 github.com/iancoleman/strcase v0.3.0/go.mod h1:iwCmte+B7n89clKwxIoIXy/HfoL7AsD47ZCWhYzw7ho=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/jonboulle/clockwork v0.4.0 h1:p4Cf1aMWXnXAUh8lVfewRBx1zaTSYKrKMF2g3ST4RZ4=
+github.com/jonboulle/clockwork v0.4.0/go.mod h1:xgRqUGwRcjKCO1vbZUEtSLrqKoPSsUpK7fnezOII0kc=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
@@ -114,6 +118,8 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
 github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/lightstep/go-expohisto v1.0.0 h1:UPtTS1rGdtehbbAF7o/dhkWLTDI73UifG8LbfQI7cA4=
+github.com/lightstep/go-expohisto v1.0.0/go.mod h1:xDXD0++Mu2FOaItXtdDfksfgxfV0z1TMPa+e/EUd0cs=
 github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
 github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
 github.com/magefile/mage v1.15.0 h1:BvGheCMAsG3bWUDbZ8AyXXpCNwU9u5CB6sM+HNb9HYg=
@@ -135,6 +141,8 @@ github.com/open-telemetry/opentelemetry-collector-contrib/confmap/provider/s3pro
 github.com/open-telemetry/opentelemetry-collector-contrib/confmap/provider/s3provider v0.107.0/go.mod h1:BIZvmh0Jx3qxmmUpQs4Wmp+yw3eZTinKBd1uRAdDRPg=
 github.com/open-telemetry/opentelemetry-collector-contrib/confmap/provider/secretsmanagerprovider v0.107.0 h1:XZGFxuM2NpjCDPKi/zrVXAaf+JN6Rlp/HlkI9lg+jc4=
 github.com/open-telemetry/opentelemetry-collector-contrib/confmap/provider/secretsmanagerprovider v0.107.0/go.mod h1:Fo7KPO1ElNawOHDL4peoRgU3V+DmoRF65R2i7Ox1I/c=
+github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.107.0 h1:1S+oa3bMVrSi7PclgMvxtyPAi8mTdHsuDqAq8XA06qU=
+github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.107.0/go.mod h1:jrqMHeaEBFtmRKKocfoVrfGjRYWp2+nj9C2aGdEN7xo=
 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.107.0 h1:okRANT1e80Lva/ck1gWZXF4Z1caH4SIz5A4crYPfylw=
 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.107.0/go.mod h1:GJoWZ9tTTev7+HjiXW0uV61tYDGTorNmy1Wdg7UmaQU=
 github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.107.0 h1:Xt9yRUYN3vyzjgc3rRwQ10OZ+WTnrIsergAdrEF7usQ=

collector/lambdacomponents/connector/pkg.go

@@ -0,0 +1,19 @@
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package connector
+
+import "go.opentelemetry.io/collector/connector"
+
+var Factories []func(extensionId string) connector.Factory

collector/lambdacomponents/connector/spanmetrics.go

@@ -0,0 +1,28 @@
+//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.connector.all || lambdacomponents.connector.spanmetrics)
+
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package connector
+
+import (
+	"github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector"
+	"go.opentelemetry.io/collector/connector"
+)
+
+func init() {
+	Factories = append(Factories, func(extensionId string) connector.Factory {
+		return spanmetricsconnector.NewFactory()
+	})
+}

collector/lambdacomponents/custom.go

@@ -0,0 +1,83 @@
+//go:build lambdacomponents.custom
+
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package lambdacomponents
+
+import (
+	custom_connector "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/connector"
+	custom_exporter "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/exporter"
+	custom_extension "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/extension"
+	custom_processor "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/processor"
+	custom_receiver "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/receiver"
+
+	"go.opentelemetry.io/collector/component"
+	"go.opentelemetry.io/collector/connector"
+	"go.opentelemetry.io/collector/exporter"
+	"go.opentelemetry.io/collector/extension"
+	"go.opentelemetry.io/collector/otelcol"
+	"go.opentelemetry.io/collector/processor"
+	"go.opentelemetry.io/collector/receiver"
+
+	"go.uber.org/multierr"
+)
+
+func Components(extensionID string) (otelcol.Factories, error) {
+	var errs []error
+
+	receivers, err := makeFactoryMap(custom_receiver.Factories, receiver.MakeFactoryMap, extensionID)
+	if err != nil {
+		errs = append(errs, err)
+	}
+
+	processors, err := makeFactoryMap(custom_processor.Factories, processor.MakeFactoryMap, extensionID)
+	if err != nil {
+		errs = append(errs, err)
+	}
+
+	exporters, err := makeFactoryMap(custom_exporter.Factories, exporter.MakeFactoryMap, extensionID)
+	if err != nil {
+		errs = append(errs, err)
+	}
+
+	extensions, err := makeFactoryMap(custom_extension.Factories, extension.MakeFactoryMap, extensionID)
+	if err != nil {
+		errs = append(errs, err)
+	}
+
+	connectors, err := makeFactoryMap(custom_connector.Factories, connector.MakeFactoryMap, extensionID)
+	if err != nil {
+		errs = append(errs, err)
+	}
+
+	factories := otelcol.Factories{
+		Receivers:  receivers,
+		Processors: processors,
+		Exporters:  exporters,
+		Extensions: extensions,
+		Connectors: connectors,
+	}
+
+	return factories, multierr.Combine(errs...)
+}
+
+func makeFactoryMap[F any](factories []func(extensionId string) F, fn func(...F) (map[component.Type]F, error), extensionId string) (map[component.Type]F, error) {
+	preprocessedFactories := make([]F, len(factories))
+	for i, f := range factories {
+		preprocessedFactories[i] = f(extensionId)
+	}
+
+	return fn(preprocessedFactories...)
+}

collector/lambdacomponents/default.go

@@ -1,3 +1,5 @@
+//go:build !lambdacomponents.custom
+
 // Copyright The OpenTelemetry Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");

collector/lambdacomponents/exporter/logging.go

@@ -0,0 +1,28 @@
+//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.exporter.all || lambdacomponents.exporter.logging)
+
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exporter
+
+import (
+	"go.opentelemetry.io/collector/exporter"
+	"go.opentelemetry.io/collector/exporter/loggingexporter"
+)
+
+func init() {
+	Factories = append(Factories, func(extensionId string) exporter.Factory {
+		return loggingexporter.NewFactory()
+	})
+}

collector/lambdacomponents/exporter/otlp.go

@@ -0,0 +1,28 @@
+//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.exporter.all || lambdacomponents.exporter.otlp)
+
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exporter
+
+import (
+	"go.opentelemetry.io/collector/exporter"
+	"go.opentelemetry.io/collector/exporter/otlpexporter"
+)
+
+func init() {
+	Factories = append(Factories, func(extensionId string) exporter.Factory {
+		return otlpexporter.NewFactory()
+	})
+}

collector/lambdacomponents/exporter/otlphttp.go

@@ -0,0 +1,28 @@
+//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.exporter.all || lambdacomponents.exporter.otlphttp)
+
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exporter
+
+import (
+	"go.opentelemetry.io/collector/exporter"
+	"go.opentelemetry.io/collector/exporter/otlphttpexporter"
+)
+
+func init() {
+	Factories = append(Factories, func(extensionId string) exporter.Factory {
+		return otlphttpexporter.NewFactory()
+	})
+}

collector/lambdacomponents/exporter/pkg.go

@@ -0,0 +1,19 @@
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exporter
+
+import "go.opentelemetry.io/collector/exporter"
+
+var Factories  []func(extensionId string) exporter.Factory