Adds Opus codec config support

[danrossi]

Fixes Issue #429

Internally opus was supported but a hardcoded config of aac was being sent to the AudioEncoder.

This became a problem attempting to use an AudioEncoder libav polyfill for IOS and Firefox support. Which doesn't support aac by default and requires opus codec.

A new config called opusConfig is added to add opus codec specific settings. Which is configured on the opus property.

[hughfenghen]

@danrossi Thanks for the code contribution, please check out the failing unit tests if you have time.

[danrossi]

I'm so sorry. I'll run the lint check. Somehow after making the change to allow opus codec configs. There is no audio track encoded or muxed.

This is the output I get from the muxer when doing an encode in chrome. Firefox doesn't like the output when loading into a video element the blob also. Firefox and the polyfill for Audioencoder for IOS / macOS need opus audio codec.

Still figuring it out. The main branch build is ok.

VideoEncoder, video track ready, trackId: 1
AudioEncoder, audio track ready, trackId: undefined

[danrossi]

In my WSL environment I get this error running the tests

 pnpm run ci:test

> root@ ci:test /mnt/d/www/players/projects/ming/video-app/WebAV
> pnpm -F @webav/* run ci:test

Scope: 4 of 6 workspace projects
packages/internal-utils ci:test$ vitest run
│  RUN  v1.6.0 /mnt/d/www/players/projects/ming/video-app/WebAV/packages/internal-utils
│       Browser runner started at http://localhost:5055/
│ ⎯⎯⎯⎯⎯⎯ Unhandled Error ⎯⎯⎯⎯⎯⎯⎯
Error: [] Browser name is required. Please, set `test.browser.name` option manually.

[danrossi]

I've figured out the tests run on selenium and requires to be done on windows to connect to a chrome browser. I can see the failure and it's to do with what I saw no audio track is added.

[hughfenghen]

It is recommended to change the audio field type to

export interface ICombinatorOpts {
  width?: number;
  height?: number;
  bitrate?: number;
  fps?: number;
  bgColor?: string;
  videoCodec?: string;
   /**
   * When specifying a composite video file,
   * setting ⁠`audio: false` will ignore the audio track;
   * if you want to include audio, you can configure the codec and other parameters via ⁠`audio: { codec: 'aac' | 'opus'; opusConfig?: OpusEncoderConfig }`.
   * The default value is `audio: { dodec: ‘aac’ }`
   */
  audio?: false | { codec: 'aac' | 'opus'; opusConfig?: OpusEncoderConfig };
  /**
   * 向输出的视频中写入 meta tags 数据
   */
  metaDataTags?: Record<string, string>;
  /**
   * 不安全，随时可能废弃
   */
  __unsafe_hardwareAcceleration__?: HardwarePreference;
}

[hughfenghen]

Then remove the audioCodec and opusConfig fields

[danrossi]

There is a typo in the codec map. The tests are useful to prevent having to do a build importing the dist files then running browser tests which was slow. Has to be done in powershell not WSL though.

[danrossi]

some test files are being tracked

[danrossi]

hooks are being called trying to commit changes which makes it hard to commit.

[danrossi]

I renamed the hooks directory to stop it running. Should be fixed.

[danrossi]

The tests pass , the audio track is created but is corrupted. And so no audio output is made. Firefox can't decode the blob. in ffprobe it shows this error. I think there is a muxer error maybe libav as the muxer might work around.

Invalid chan_config 14

The AAC license makes things complicated and problematic not able to use AAC in firefox or the polyfill for libav.

[danrossi]

The opus codec doesn't work in chrome or firefox. I can't tell if it's the muxer doing that. Polyfill in macOS safari has issues for now also. IOS can't check due to memory issues.

[danrossi]

I'm still working on Opus codec testing to get Firefox working. But just discovered with hardware preference enabled the encoder failed with AV1 due to only supported on certain GPU models like RTX4000+ I am on a 2060. So added an option for that in the checker. the VideoEncoder doesn't return a promise on configure and will only work out it fails on the error callback.

[danrossi]

I'm sorry opus is encoding but it's not being muxed correctly now perhaps. There is an audio track but it's corrupted so there is no audio being previewed.

It's a problem on both Chrome and Firefox.

ffprobe produces an error like

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x609b13378280] Invalid chan_config 14
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x609b13378280] error reading header
just do it-trimmed-1752582129299.mp4: Invalid data found when processing input

https://electroteque.org/dev/webav/opus.html

[danrossi]

I believe this is the issue, it's taking the opus decoder config info and hardcoding aac ?

https://github.com/WebAV-Tech/WebAV/blob/main/packages/internal-utils/src/recodemux.ts#L471

[danrossi]

The Opus codec I discovered is 0xAD. byte muxing is a grey area for me. I hope that is all it needs. Should it somehow get that from the decoder config ?

[danrossi]

Changing that didn't helped. Does the internal muxer need to support Opus ?

[danrossi]

Confirming changing the code to 0xAD works correctly. Not sure how to make that dynamic.

[danrossi]

It should be fixed now but VLC doesn't want to play the file. OK in the browser. There is no fast start on the mp4 it seems so VLC takes a while to play.

[danrossi]

Firefox doesn't like the audio mux code after encoding and previewing. even with the codec code changes. It's ok in Chrome. If audio is disabled it will load the blob. It won't load a file with opus created in chrome. VLC won't play back the audio.

generated video with opus.

https://downloads-electroteque-org.s3.amazonaws.com/just%20do%20it-trimmed-1752600908898_opus.mp4

https://electroteque.org/dev/webav/opus.html
https://electroteque.org/dev/webav/noaudio.html

aac generated file loads ok in firefox. All the generated videos are very slow to load with blocky pixels when seeking.

https://electroteque.org/dev/webav/aac.html

[hughfenghen]

There is no fast start on the mp4 it seems so VLC takes a while to play.

This is a known unfixed issue, see #183

[danrossi]

It's very strange. VLC player won't show the picture of a h264/aac encoded video. It will play the video with opus in it but no picture also. The picture shows up at the end.

Firefox will decode and preview the aac audio video but not the opus audio video. But wont autoload and takes a while to show up. I think there is still muxing issues with the video track also.

https://downloads-electroteque-org.s3.amazonaws.com/just%20do%20it-trimmed-1752600811403.mp4