Skip to content

Use miniaudio for direct decoding flac, mp3, ogg and wav #2759

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Feb 27, 2025
Merged

Use miniaudio for direct decoding flac, mp3, ogg and wav #2759

merged 2 commits into from
Feb 27, 2025

Conversation

data-man
Copy link
Contributor

miniaudio.h taken from mackron/miniaudio@6d5efde. And stb_vorbis.c from nothings/stb@5c20573.
I think miniaudio can be used in many other purposes in the future.
Note: tested in Linux only.

Thank you for the awesome project!

@data-man
Copy link
Contributor Author

It's very unexpected that there's no reaction at all. Doesn't anyone really want it?

@azkadev
Copy link

azkadev commented Feb 5, 2025

It's very unexpected that there's no reaction at all. Doesn't anyone really want it?

If it can be integrated into the shared library without any additional features, I really want this feature, because if I use ffmpeg it will be a problem because of its size.

@data-man
Copy link
Contributor Author

data-man commented Feb 6, 2025

If it can be integrated into the shared library without any additional features

Yes, it is. All OS-specific sound I/O functions are disabled.
But they can be used in the future instead of SDL.

@satmandu
Copy link

satmandu commented Feb 6, 2025

This looks amazing. Any chance of getting this to work on macos?

@data-man
Copy link
Contributor Author

data-man commented Feb 6, 2025

This looks amazing. Any chance of getting this to work on macos?

I can't see any problems.
https://github.com/mackron/miniaudio?tab=readme-ov-file#supported-platforms

Windows
macOS, iOS
Linux
FreeBSD / OpenBSD / NetBSD
Android
Raspberry Pi
Emscripten / HTML5

@satmandu
Copy link

satmandu commented Feb 6, 2025

Does master need to be merged into this? It looks many commits behind. I'd love to try building off of this branch.

@data-man
Copy link
Contributor Author

data-man commented Feb 6, 2025

Does master need to be merged into this? It looks many commits behind. I'd love to try building off of this branch.

Please try now.

@satmandu
Copy link

satmandu commented Feb 6, 2025

Thanks! This worked great! I was able to directly decode a .flac file of an old family interview without any additional conversion!

@ggerganov Any chance of getting this reviewed?

@satmandu
Copy link

satmandu commented Feb 6, 2025

(FYI I ran this branch on a M1 MBP running the current version of macos.)

ggerganov
ggerganov approved these changes Feb 6, 2025
View reviewed changes
Copy link
Member

@ggerganov ggerganov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job and sorry for the delay. We can use this in the examples, but for now I don't plan to add audio decoding directly into whisper.cpp. It seems better to leave this to the user.

Let's resolve the CI and merge. Later on, if we can completely remove SDL dependency, it would be great. But I am not sure how difficult it would be.

@data-man
Copy link
Contributor Author

data-man commented Feb 6, 2025

I hope that I've properly corrected all remarks.

@data-man
Copy link
Contributor Author

data-man commented Feb 6, 2025

Later on, if we can completely remove SDL dependency, it would be great. But I am not sure how difficult it would be.

I think https://github.com/mackron/miniaudio/blob/master/examples/simple_capture.c can be taken as template for audio recording.

foldl
foldl reviewed Feb 7, 2025
View reviewed changes
@data-man data-man force-pushed the miniaudio branch 2 times, most recently from 4d32ac7 to c5f65e1 Compare February 7, 2025 07:48
@data-man
Copy link
Contributor Author

miniaudio 0.11.22 was released, so I've updated this PR.

@ggerganov ggerganov merged commit 7d3da68 into ggml-org:master Feb 27, 2025
@ggerganov ggerganov mentioned this pull request Feb 27, 2025
Merged
@data-man data-man deleted the miniaudio branch February 27, 2025 07:23
@KitaitiMakoto KitaitiMakoto mentioned this pull request Feb 27, 2025
Merged
@data-man data-man mentioned this pull request Feb 28, 2025
Closed
joelvaneenwyk pushed a commit to joelvaneenwyk/whisper-cpp that referenced this pull request Mar 3, 2025
@data-man @joelvaneenwyk
examples : use miniaudio for direct decoding flac, mp3, ogg and wav (g…
7a14947 
…gml-org#2759)
@mrfragger
Copy link

I get why libopus isn't included..read the threads. But still way over my head trying to included it in the souce code for whisper.cpp and custom compile miniaudio with opus support.
Anyway to include opus by default? It's the only audio codec I use for audiobooks.

v0.11.22 - 2025-02-24

In the extras folder, the miniaudio_libvorbis.h and miniaudio_libopus.h files have been deprecated. They have been replaced with versions in the extras/decoders folder. They are now split into a separate .c and .h files. The old files still exist for compatibility, but you need to transition over to the new versions. The transition should be trivial. Compile the .c files like a normal source file, and include the .h file like a normal header.

@WilliamTambellini
Copy link
Contributor

cool

@WilliamTambellini
Copy link
Contributor

FYI @amanda

@ggerganov ggerganov mentioned this pull request May 20, 2025
Merged
danbev added a commit to danbev/whisper.cpp that referenced this pull request Jun 18, 2025
@danbev
examples : add stereo to mono conversion in read_audio_data
1e31080 
This commit adds a conversion from stereo to mono in the
`read_audio_data` function of `common-whisper.cpp`.

The motivation for this change is prior to Commit
7d3da68 ("examples : use miniaudio for
direct decoding flac, mp3, ogg and wav (ggml-org#2759)", there was a step that
read stereo int16 data -> pcm16 (448512 samples), and then converted to
mono (224256 samples), and then also convert to stereo in `pcmf32s.

The middle step here seems to have been missed when rewriting the code to
use Miniaudio and caused issues then transcribing stereo audio files.

For example, currently using the audio sample in the linked issue the
output is:
```console
[00:00:00.000 --> 00:00:03.000]  (speaker 1) Sous-titres réalisés para la communauté d'Amara.org
```

And with the change in this commit the output is:
```
[00:00:00.000 --> 00:00:01.500]  (speaker 1) *sonnerie de téléphone*
[00:00:01.500 --> 00:00:07.000]  (speaker 1) Salut jeune homme !
[00:00:07.000 --> 00:00:08.500]  (speaker 0) C'est vrai que je te dérange ?
[00:00:08.500 --> 00:00:10.500]  (speaker 1) Ah pas du tout, pas du tout, pas du tout !
[00:00:10.500 --> 00:00:12.500]  (speaker 1) J'étais en train de...
[00:00:12.500 --> 00:00:14.500]  (speaker 1) de préparer un courrier
```

Resolves: ggml-org#3092
@danbev danbev mentioned this pull request Jun 18, 2025
Merged
danbev added a commit that referenced this pull request Jun 18, 2025
@danbev
examples : add stereo to mono conversion in read_audio_data (#3266)
ecb8f3c 
This commit adds a conversion from stereo to mono in the
`read_audio_data` function of `common-whisper.cpp`.

The motivation for this change is prior to Commit
7d3da68 ("examples : use miniaudio for
direct decoding flac, mp3, ogg and wav (#2759)", there was a step that
read stereo int16 data -> pcm16 (448512 samples), and then converted to
mono (224256 samples), and then also convert to stereo in `pcmf32s.

The middle step here seems to have been missed when rewriting the code to
use Miniaudio and caused issues then transcribing stereo audio files.

For example, currently using the audio sample in the linked issue the
output is:
```console
[00:00:00.000 --> 00:00:03.000]  (speaker 1) Sous-titres réalisés para la communauté d'Amara.org
```

And with the change in this commit the output is:
```
[00:00:00.000 --> 00:00:01.500]  (speaker 1) *sonnerie de téléphone*
[00:00:01.500 --> 00:00:07.000]  (speaker 1) Salut jeune homme !
[00:00:07.000 --> 00:00:08.500]  (speaker 0) C'est vrai que je te dérange ?
[00:00:08.500 --> 00:00:10.500]  (speaker 1) Ah pas du tout, pas du tout, pas du tout !
[00:00:10.500 --> 00:00:12.500]  (speaker 1) J'étais en train de...
[00:00:12.500 --> 00:00:14.500]  (speaker 1) de préparer un courrier
```

Resolves: #3092
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@foldl foldl foldl left review comments

@ggerganov ggerganov ggerganov approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@data-man @azkadev @satmandu @mrfragger @WilliamTambellini @ggerganov @foldl