SSM Port Forwarding fails on multiple simultaneous connections

[jylitalo]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
I am trying to establish tunnel from my laptop into EC2 instance that is running mongodb. It works fine, if I do it on command line with

aws ssm start-session --target i-... --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["27017"],"localPortNumber":["27017"]}'

on terminal 1 and

mongodump --port 27017 --db yletunnus --out ./dump --username root --password ... --authenticationDatabase admin --verbose --host localhost

on terminal 2.

If I try to handlee the first command with

// startSession creates tunnel via AWS Systems Manager (SSM)
func (client *SSM) startSession(instanceID, documentName string, params map[string][]string) (*ssm.StartSessionOutput, error) {
	return client.sdk.StartSession(context.TODO(), &ssm.StartSessionInput{
		Target:       &instanceID,
		DocumentName: &documentName,
		Parameters:   params,
	})
}
func (client *SSM) StartPortForwardingSession(instanceID string, localPort, remotePort int) (*ssm.StartSessionOutput, error) {
	params := map[string][]string{
		"portNumber":      {strconv.Itoa(remotePort)},
		"localPortNumber": {strconv.Itoa(localPort)},
	}
	return client.startSession(instanceID, "AWS-StartPortForwardingSession", params)
}
...
	portSession, err := ssm.StartPortForwardingSession(secondaryID, 27017, 27017)

It will create session that appears in SessionManager, but then it disappears and mongodump fails with

2021-03-19T12:17:34.886+0000 will listen for SIGTERM, SIGINT, and SIGKILL
2021-03-19T12:18:04.847+0000 Failed: can't create session: could not connect to server: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: host.docker.internal:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp 192.168.65.2:27017: connect: connection refused }, ] }

and it does exit 1

Version of AWS SDK for Go?
github.com/aws/aws-sdk-go-v2 v1.3.0
github.com/aws/aws-sdk-go-v2/service/ssm v1.2.0

Version of Go (go version)?
1.15.7

To Reproduce (observed behavior)
Assuming you have EC2 running mongodb somewhere, use example above to open port forwarding and issue time.Sleep(120*time.Second) while issueing mongodump command (example also above) on another terminal.

Expected behavior
mongodump should be able to create multiple tcp connection via port forwarding and print out something like:

2021-03-19T14:20:25.103+0200 will listen for SIGTERM, SIGINT, and SIGKILL
2021-03-19T14:20:25.841+0200 enqueued collection 'foo.tempTokens'
2021-03-19T14:20:25.908+0200 enqueued collection 'foo.currentTermsOfService'
...

instead of

Failed: can't create session: could not connect to server

Additional context
This requires amazon-ssm-agent 3.0.222.0 or later on EC2 instance. (https://aws.amazon.com/about-aws/whats-new/2020/10/port-forwarding-sessions-created-sessions-manager-support-multiple-simultaneous-connections/). You also need sessionmanagerplugin (installs under /usr/local in MacOS) on your local machine.

Current work-around is to execute awscli command from golang code.

[mmmorris1975]

I think you may be misunderstanding how the SSM Session API works, or there is some other detail left out of the report. The StartSession API call merely returns some credentials and a websocket endpoint for you to communicate with. It does not actually establish a connection with the host. The disconnect you are seeing is likely the timeout once AWS closes the connection because you aren't following up with the necessary websocket connection.

Take a look at my ssm-session-client repo for how this should be handled.
One note, my code doesn't handle multiplexed port forwarding either. However, the AWS CLI SSM plugin does

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.