AWS Transcribe streaming audio - not working (code provided)

[prazian]

I want to build a web application that needs to transcribe a stream of voice in real-time.
The backend is written in Golang and uses AWS SDK V2.
The connection between frontend and backend is established via websocket.

This is how the backend handles the incoming stream:

// Websocket connection is handled here
.
.
.
	ctx := context.Background()
    ...
	svg := transcribestreaming.NewFromConfig(cfg)
.
.
.
// This code tries to interact with AWS Transcribe (streaming)
	startStreamTranscriptionInput := transcribestreaming.StartStreamTranscriptionInput{
		LanguageCode:                      "en-US",
		MediaEncoding:                     types.MediaEncodingPcm,
		MediaSampleRateHertz:              aws.Int32(16000),
		EnablePartialResultsStabilization: true,
	}

	// Start audio stream with the first chunk of audio
	resp, err := svg.StartStreamTranscription(ctx, &startStreamTranscriptionInput)
	if err != nil {
		log.Println("Transcribe: error in starting the stream...")
		log.Printf("expect no error got, %v", err)
	}

	stream := resp.GetStream()

	log.Println("Transcribe: resp.EnableChannelIdentification", resp.EnableChannelIdentification)
	defer func(stream *transcribestreaming.StartStreamTranscriptionEventStream) {
		err := stream.Close()
		if err != nil {
			log.Printf("stream.Close() expect no error, got %v", err)
		}
	}(stream)
	if err := stream.Err(); err != nil {
		log.Printf("stream.Err() expect no error, %v", err)
	}

	// Receive response from AWS Transcribe
	var wg sync.WaitGroup

	wg.Add(1)
	go func() {
		defer wg.Done()
		expectedServiceEvents := []types.TranscriptResultStream{
			&types.TranscriptResultStreamMemberTranscriptEvent{Value: types.TranscriptEvent{Transcript: &types.Transcript{Results: []types.Result{}}}},
		}
		for i := 0; i < len(expectedServiceEvents); i++ {
			event := <-resp.GetStream().Events()
			if event == nil {
				log.Printf("%d, expect event, got nil", i)
				continue
			}
			if e, a := expectedServiceEvents[i], event; !reflect.DeepEqual(e, a) {
				log.Printf("%d, expect %T %v, got %T %v", i, e, e, a, a)
				continue
			}
			eventType := reflect.TypeOf(event).String()
			if eventType == "*types.TranscriptResultStreamMemberTranscriptEvent" {
				// Print the transcript
				transcriptEvent := event.(*types.TranscriptResultStreamMemberTranscriptEvent)
				transcript := transcriptEvent.Value.Transcript
				for _, result := range transcript.Results {
					log.Printf("Transcript: %s", *result.Alternatives[0].Transcript)
					// TODO: Send the transcript to the client
				}
			}
		}
	}()

	clientEvents := []types.AudioStream{
		&types.AudioStreamMemberAudioEvent{Value: types.AudioEvent{AudioChunk: audioChunk}},
	}
	for _, event := range clientEvents {
		//err = stream.Send(ctx, event)
		log.Printf("Transcribe: sending audio chunk to AWS Transcribe %v Bytes", len(audioChunk))
		err = stream.Writer.Send(ctx, event)
		if err != nil {
			log.Printf("stream.Send expect no error, got %v", err)
		}
	}

	wg.Wait()
	err = stream.Close()
	if err != nil {
		return
	}
	if err := stream.Err(); err != nil {
		log.Printf("stream.Err() expect no error, %v", err)
	}

This is how I send the audio from frontend:

const mediaChunkSize = 1024 * 20; // 20kb
.
.
.
  stream.value = await navigator.mediaDevices.getUserMedia({
    audio: true,
    video: false,
  });
  mediaRecorder = new MediaRecorder(stream.value, {
    mimeType: 'audio/webm;codecs=pcm',
    bitsPerSecond: 16000,
    audioBitsPerSecond: 16000,
  });
.
.
.
  mediaRecorder.ondataavailable = (e) => {
    audioChunks.push(e.data);
    console.log('Data available...');
    console.log('audioChunks.length: ', audioChunks.length);
    console.log('e.data: ', e.data);
    if (
        mediaRecorder && sockets &&
        sockets.transcription &&
        sockets.transcription.readyState === WebSocket.OPEN &&
        stream.value
    ) {
      // sockets.transcription.send(e.data);
      const audioChunksBlob = new Blob(audioChunks);

      if (audioChunksBlob.size >= mediaChunkSize) {
        let audioBlob = new Blob(audioChunks);
        let reader = new FileReader();
        reader.onload = function () {
          let encodedAudio = reader.result;
          // Send audio chunk over websocket
          if (
              sockets &&
              sockets.transcription &&
              sockets.transcription.readyState === WebSocket.OPEN &&
              encodedAudio
          ) {
            console.log('Sending audio chunk to websocket...', encodedAudio);
            sockets.transcription.send(encodedAudio);
          }
        };
        reader.readAsArrayBuffer(audioBlob);
        audioChunks = [];
      }
    }
  };

To write this code, I was inspired by one of the tests in the SDK v2's repo (this one):

aws-sdk-go-v2/service/transcribestreaming/internal/testing/eventstream_test.go

Lines 402 to 493 in 4034d37

	func TestStartStreamTranscription_ReadWrite(t *testing.T) {
	cfg, cleanupFn, err := eventstreamtesting.SetupEventStream(t,
	eventstreamtesting.ServeEventStream{
	T: t,
	ClientEvents: []eventstream.Message{
	{
	Headers: eventstream.Headers{
	{
	Name: eventstreamapi.EventTypeHeader,
	Value: eventstream.StringValue("AudioEvent"),
	},
	eventstreamtesting.EventMessageTypeHeader,
	{
	Name: eventstreamapi.ContentTypeHeader,
	Value: eventstream.StringValue("application/octet-stream"),
	},
	},
	Payload: []byte{0x1, 0x2, 0x3, 0x4},
	},
	},
	Events: []eventstream.Message{
	{
	Headers: eventstream.Headers{
	eventstreamtesting.EventMessageTypeHeader,
	{
	Name: eventstreamapi.EventTypeHeader,
	Value: eventstream.StringValue("TranscriptEvent"),
	},
	},
	Payload: []byte(`{
	"Transcript": {
	"Results": []
	}
	}`),
	},
	},
	BiDirectional: true,
	},
	)
	if err != nil {
	t.Fatalf("expect no error, %v", err)
	}
	defer cleanupFn()
	svc := transcribestreaming.NewFromConfig(cfg)
	resp, err := svc.StartStreamTranscription(context.Background(), &transcribestreaming.StartStreamTranscriptionInput{}, func(options *transcribestreaming.Options) {
	options.APIOptions = append(options.APIOptions, removeValidationMiddleware)
	})
	if err != nil {
	t.Fatalf("expect no error got, %v", err)
	}
	stream := resp.GetStream()
	defer stream.Close()
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
	defer wg.Done()
	expectedServiceEvents := []types.TranscriptResultStream{
	&types.TranscriptResultStreamMemberTranscriptEvent{Value: types.TranscriptEvent{Transcript: &types.Transcript{Results: []types.Result{}}}},
	}
	for i := 0; i < len(expectedServiceEvents); i++ {
	event := <-resp.GetStream().Events()
	if event == nil {
	t.Errorf("%d, expect event, got nil", i)
	}
	if e, a := expectedServiceEvents[i], event; !reflect.DeepEqual(e, a) {
	t.Errorf("%d, expect %T %v, got %T %v", i, e, e, a, a)
	}
	}
	}()
	clientEvents := []types.AudioStream{
	&types.AudioStreamMemberAudioEvent{Value: types.AudioEvent{AudioChunk: []byte{0x1, 0x2, 0x3, 0x4}}},
	}
	for _, event := range clientEvents {
	err = stream.Send(context.Background(), event)
	if err != nil {
	t.Errorf("expect no error, got %v", err)
	}
	}
	wg.Wait()
	stream.Close()
	if err := stream.Err(); err != nil {
	t.Errorf("expect no error, %v", err)
	}
	}

But when I run the code I receive these errors:

• Stream write:
  expect no error, got BadRequestException: Your request timed out because no new audio was received for 15 seconds.

• Stream read:
  0, expect event, got nil

I can see the audio (or at least a binary) is sent to AWS, so I am not sure why I get these errors instead of the transcription text.

I assume the format I am sending the audio should be correct based on this document:
https://docs.aws.amazon.com/transcribe/latest/dg/how-input.html

(I am using Google Chrome, but tested with other browsers as well => same result)

Does anyone know how I can debug this code? Or do you have any suggestion or an example code in Golang (SDK V2) I can use?

PS:

• I see no other logs anywhere else and I don't know how to assure what I send is what AWS Transcribe expects to receive.
• I have done a search for the answer or an example, but unfortunately, I found nothing.

[prazian]

Found this on the old/removed documentation. I'll see if that helps:
https://github.com/aws/aws-sdk-js-v3/tree/d8475f8d972d28fbc15cd7e23abfe18f9eab0644/clients/client-transcribe-streaming#acquire-speech-stream

It uses this package:
https://github.com/microphone-stream/microphone-stream

[ArturoBurela]

Also, have you considered opening an issue/bug report @prazian ? Is this something only happening to you when you stream from the fronted? did you make it work streaming audio files directly?

[prazian]

@ArturoBurela Unfortunately, I got involved in other projects and couldn't spend more time on it or open an issue/bug report.

I remember I could make it work by reading directly from my laptop and using NodeJs (not this code in Golang). The old example from the link I shared in the previous comment could work, but the microphone-stream package that it was using was quite obsolete, and I couldn't make it work with Vue/Vite and the new changes in how browsers handle background jobs (don't remember why and what the problem was). I even asked a few AWS architects at one of their Summit events in Stockholm, but none of them could get it to work or guess what can help.

[tomasmaccarthy]

This example worked!

[CrazyTux]

I believe I'm running into the same issue as you. Did you ever get this to work?

[prazian]

@CrazyTux, my priorities changed, and that project was abandoned. Unfortunately, I couldn't make it work with more modern tooling back then, and even asked a few architects/developers at AWS, but they couldn't help either. One of them at AWS Summit Stockholm even promised to open a ticket internally, but I'm, not sure if anything happened after that.

Based on the fact that they were recommending/promoting more abstracted services, such as Amazon Connect or Amazon Chime (and another one that I don't remember), I suppose they prefer people to use such out-of-the-box (and more expensive for sure) services for specific use cases, rather than directly dealing with Transcribe Streaming to build their own solutions, which was strange, to be honest.

I also learned that debugging video/audio streams is not as easy as I thought it would be!