binary size increase

[hrez]

Documentation

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug

I've migrated my cli app from aws-sdk-go to aws-sdk-go-v2 and binary size exploded from 27Mb to 66Mb.
The cli is using spf13/cobra.
While looking for a cause I noticed that if I call functions using aws-sdk-go-v2 directly i.e not from cobra it's the expected size.
I've also verified it with creating an empty cobratest app to which I've added:
func ec2test() {
ctx := context.Background()
cfg, _ := config.LoadDefaultConfig(ctx)
svc := ec2.NewFromConfig(cfg)
fmt.Println(svc)
}

That built into 33Mb. Using google/bloaty, the top line is github.com/aws/aws-sdk-go-v2/service/ec2 at 11.9Mi

If I call ec2test() directly the binary is 9Mb and github.com/aws/aws-sdk-go-v2/service/ec2 is 7.38Ki

Expected behavior

The binary size remains small i.e. 9Mb in cobratest mentioned above. Similar to aws-sdk-go aka v1.

Current behavior

cobratest binary size is 33Mb

Steps to Reproduce

see https://github.com/hrez/cobratest repo

Possible Solution

No response

AWS Go SDK version used

github.com/aws/aws-sdk-go-v2 v1.11.2

Compiler and Version used

go version go1.17.4 linux/amd64

Operating System and version

Ubuntu 18.04.6 LTS

[skmcgrail]

There is a number of conflating factors here that need to be teased apart to understand the behavior you are seeing.

• The Go compiler/linker will attempt to not link in code into your binary that it can prove is not required by your application.
• Removing deadcode like unused methods receivers on types (for example all the API operations on the ec2.Client type requires proof that the type is not used directly, is not reachable as an interface type, or that the reflection package's Method or MethodByName methods are not used which could potentially invoke the methods at runtime.
• The code example you are using to show off the approach is inherently imperfect since it actually is passing the entire client to fmt.Println as an empty interface, which on its own does not trigger the compiler to link in all the associated type methods, but when used with Cobra and/or Viper is enough to trigger the behavior of linking in all the methods reachable from that type. Instead if you just invoke DescribeInstances rather then fmt.Println and still keep the Cobra & Viper components you will notice that the binary size that is 13 MB. A recommendation would be to avoid using the EC2 client as an interface if possible in this scenario to avoid the linker behavior you are running into.
• go build -ldflags "-dumpdep" . provides some useful insight as it will annotate types with UsedInIface when they are used in an interface, though this is only a clue and not necessarily indicative of the linker pulling in all methods on a type.

In short what you've encountered is a behavior in the Go compiler around linking, and not an issue directly with the size of the SDK itself.

[hrez]

Thanks for those details. I tried eliminating one place which used an interface but that didn't lead to any size reduction.
I'd have to dig some more on that.
The fact of the matter, simple v1 to v2 sdk conversion lead to 144% size increase. I am not sure how representative my case here but cobra is pretty common in cli. I'd be curious to hear if anybody else observed the same increase.

[KaibaLopez]

@hrez,
It would seem like this has more or less been responded, I'll move this to discussions since it seems like a better place to gather different opinions on this and I don't want to just close it but also don't want to keep an non-actionable issue open.

[sodul]

We are actually very curious to hear from others about this. In our case a 40MB increase could cause our lambdas to no longer be accepted since our zips would be over the 50MB hard limit.

Are there some typical size differences that the SDK team would be able to share between using v1 and v2, and for larger project that import both v1 and v2?

[djedward]

Lambda does not have a 50MB hard limit for Lambda zips. That limit is when you are directly uploading the zip to Lambda. The documentation covers this here: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html (see the last section: "Deployment packages over 50 MB").

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.