v1.7.6

[ggerganov]

Overview

• Add initial VAD support - feedback welcome and appreciated
• Metal FA improvements

M2 Ultra

Flash Attention ON:

CPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M2 ULTRA	METAL	tiny	1	1	7.72	1.05	0.32	0.01	dc8dda6
M2 ULTRA	METAL	tiny-q5_0	1	1	8.20	0.98	0.31	0.01	dc8dda6
M2 ULTRA	METAL	tiny-q5_1	1	1	8.13	0.99	0.31	0.01	dc8dda6
M2 ULTRA	METAL	tiny-q8_0	1	1	7.96	0.93	0.30	0.01	dc8dda6
M2 ULTRA	METAL	base	1	1	13.52	1.39	0.35	0.02	dc8dda6
M2 ULTRA	METAL	base-q5_0	1	1	14.88	1.31	0.34	0.02	dc8dda6
M2 ULTRA	METAL	base-q5_1	1	1	14.76	1.33	0.34	0.02	dc8dda6
M2 ULTRA	METAL	base-q8_0	1	1	14.04	1.28	0.34	0.02	dc8dda6
M2 ULTRA	METAL	small	1	1	38.78	2.72	0.67	0.04	dc8dda6
M2 ULTRA	METAL	small-q5_0	1	1	44.01	2.64	0.69	0.05	dc8dda6
M2 ULTRA	METAL	small-q5_1	1	1	44.02	2.66	0.69	0.05	dc8dda6
M2 ULTRA	METAL	small-q8_0	1	1	40.79	2.49	0.67	0.05	dc8dda6
M2 ULTRA	METAL	medium	1	1	104.48	5.57	1.61	0.10	dc8dda6
M2 ULTRA	METAL	medium-q5_0	1	1	122.24	5.00	1.58	0.12	dc8dda6
M2 ULTRA	METAL	medium-q5_1	1	1	121.99	5.02	1.59	0.12	dc8dda6
M2 ULTRA	METAL	medium-q8_0	1	1	111.68	4.99	1.52	0.11	dc8dda6
M2 ULTRA	METAL	medium-dis	1	1	93.23	0.87	0.21	0.01	dc8dda6
M2 ULTRA	METAL	large-v2	1	1	189.82	8.36	2.35	0.19	dc8dda6
M2 ULTRA	METAL	large-v2-q5_0	1	1	225.73	7.34	2.40	0.22	dc8dda6
M2 ULTRA	METAL	large-v2-q5_1	1	1	225.88	7.60	2.40	0.22	dc8dda6
M2 ULTRA	METAL	large-v2-q8_0	1	1	203.55	7.32	2.26	0.20	dc8dda6
M2 ULTRA	METAL	large-v2-dis	1	1	168.20	0.98	0.24	0.02	dc8dda6
M2 ULTRA	METAL	large-v3-turbo	1	1	170.22	1.46	0.37	0.03	dc8dda6
M2 ULTRA	METAL	large-v3-turbo-q5_0	1	1	201.88	1.27	0.38	0.04	dc8dda6
M2 ULTRA	METAL	large-v3-turbo-q8_0	1	1	182.37	1.24	0.36	0.03	dc8dda6

Flash Attention OFF:

CPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M2 ULTRA	METAL	tiny	1	0	10.15	1.20	0.36	0.01	dc8dda6
M2 ULTRA	METAL	tiny-q5_0	1	0	10.21	1.15	0.39	0.01	dc8dda6
M2 ULTRA	METAL	tiny-q5_1	1	0	9.26	1.15	0.38	0.01	dc8dda6
M2 ULTRA	METAL	tiny-q8_0	1	0	9.00	1.12	0.37	0.01	dc8dda6
M2 ULTRA	METAL	base	1	0	15.77	1.73	0.45	0.02	dc8dda6
M2 ULTRA	METAL	base-q5_0	1	0	16.90	1.63	0.44	0.02	dc8dda6
M2 ULTRA	METAL	base-q5_1	1	0	16.93	1.64	0.44	0.02	dc8dda6
M2 ULTRA	METAL	base-q8_0	1	0	16.13	1.63	0.43	0.02	dc8dda6
M2 ULTRA	METAL	small	1	0	45.15	3.45	0.92	0.05	dc8dda6
M2 ULTRA	METAL	small-q5_0	1	0	50.63	3.36	0.94	0.06	dc8dda6
M2 ULTRA	METAL	small-q5_1	1	0	50.56	3.36	0.94	0.06	dc8dda6
M2 ULTRA	METAL	small-q8_0	1	0	47.52	3.20	0.92	0.05	dc8dda6
M2 ULTRA	METAL	medium	1	0	122.55	7.38	1.95	0.12	dc8dda6
M2 ULTRA	METAL	medium-q5_0	1	0	140.61	6.73	2.02	0.14	dc8dda6
M2 ULTRA	METAL	medium-q5_1	1	0	140.48	6.76	2.04	0.14	dc8dda6
M2 ULTRA	METAL	medium-q8_0	1	0	131.00	6.57	1.96	0.13	dc8dda6
M2 ULTRA	METAL	medium-dis	1	0	110.85	1.00	0.24	0.02	dc8dda6
M2 ULTRA	METAL	large-v2	1	0	222.28	10.96	3.03	0.21	dc8dda6
M2 ULTRA	METAL	large-v2-q5_0	1	0	258.64	9.79	3.04	0.25	dc8dda6
M2 ULTRA	METAL	large-v2-q5_1	1	0	258.32	9.87	3.05	0.24	dc8dda6
M2 ULTRA	METAL	large-v2-q8_0	1	0	236.55	9.61	2.87	0.23	dc8dda6
M2 ULTRA	METAL	large-v2-dis	1	0	199.84	1.14	0.27	0.02	dc8dda6
M2 ULTRA	METAL	large-v3-turbo	1	0	201.52	1.77	0.45	0.03	dc8dda6
M2 ULTRA	METAL	large-v3-turbo-q5_0	1	0	233.14	1.56	0.47	0.04	dc8dda6
M2 ULTRA	METAL	large-v3-turbo-q8_0	1	0	214.23	1.53	0.44	0.04	dc8dda6

What's Changed

• docs : add xcframework section to README.md [no ci] by @danbev in docs : add xcframework section to README.md [no ci] #2997
• sync : ggml by @ggerganov in sync : ggml #2992
• whisper.wasm : fix unknown language issue by @danbev in whisper.wasm : fix unknown language issue #3000
• examples : update server.py to match github pages app [no ci] by @danbev in examples : update server.py to match github pages app [no ci] #3004
• rename : ggerganov -> ggml-org by @ggerganov in rename : ggerganov -> ggml-org #3005
• whisper : fix "bench-all outputs an invalid result on larger models" by @fujimotos in whisper : fix "bench-all outputs an invalid result on larger models" #3002
• tests : add script to benchmark whisper.cpp on LibriSpeech corpus by @fujimotos in tests : add script to benchmark whisper.cpp on LibriSpeech corpus #2999
• ruby : Change homepage URI in Ruby gemspec by @KitaitiMakoto in ruby : Change homepage URI in Ruby gemspec #3007
• fix dead link to models in readme by @gregsadetsky in fix dead link to models in readme #3006
• Update uri.rb by @Olli in Update uri.rb #3016
• Update ruby_whisper_params.c by @Olli in Update ruby_whisper_params.c #3022
• xcf : use check for visionos build version by @danbev in xcf : use check for visionos build version #3021
• Fix README.md by @ekaitz-zarraga in Fix README.md #3024
• docs : document how to use 'WHISPER_FFMPEG' build option by @fujimotos in docs : document how to use 'WHISPER_FFMPEG' build option #3029
• whisper : reduce delta_min from 1000ms to 100ms by @ggerganov in whisper : reduce delta_min from 1000ms to 100ms #3028
• support max_context api for addon.node by @buxuku in support max_context api for addon.node #3025
• Update README.md to note newer NVIDIA GPUs by @jeffklassen in Update README.md to note newer NVIDIA GPUs #3031
• ruby: use CMake in build process by @KitaitiMakoto in ruby: use CMake in build process #3043
• examples : add FFmpeg v7.0 support to ffmpeg-transcode.cpp by @fujimotos in examples : add FFmpeg v7.0 support to ffmpeg-transcode.cpp #3038
• feat: Add no-context option to server by @sachaarbonel in feat: Add no-context option to server #3045
• ruby : make Ruby bindings installed with build options by @KitaitiMakoto in ruby : make Ruby bindings installed with build options #3056
• examples : add HEAPU8 to exported runtime methods by @danbev in examples : add HEAPU8 to exported runtime methods #3062
• ci : disable freeBSD job in build.yml by @danbev in ci : disable freeBSD job in build.yml #3064
• coreml : set convert_to="mlprogram" in convert by @danbev in coreml : set convert_to="mlprogram" in convert #3060
• sync : ggml by @ggerganov in sync : ggml #3071
• ci : enable bindings java job by @danbev in ci : enable bindings java job #3070
• ruby : add encoder begin callback related methods by @KitaitiMakoto in ruby : add encoder begin callback related methods #3076
• Fix deprecated FFmpeg functions by @Podre-Henrique in Fix deprecated FFmpeg functions #3073
• Add Moore Threads GPU support and update GitHub workflow for MUSA build by @yeahdongcn in Add Moore Threads GPU support and update GitHub workflow for MUSA build #3069
• ci : disable publishing of java binding [no ci] by @danbev in ci : disable publishing of java binding [no ci] #3086
• talk-llama : sync llama.cpp by @ggerganov in talk-llama : sync llama.cpp #3084
• whisper : remove empty .gitmodules file [no ci] by @danbev in whisper : remove empty .gitmodules file [no ci] #3085
• feat: expose language detection probabilities to server example by @sachaarbonel in feat: expose language detection probabilities to server example #3044
• whisper : fix grammar advance stack warning by @danbev in whisper : fix grammar advance stack warning #3087
• ggml : suppress Windows compiler warnings by @danbev in ggml : suppress Windows compiler warnings #3075
• make : fix samples glob pattern by @ggerganov in make : fix samples glob pattern #3100
• ruby : ignore "Downloading" output in test_log_suppress by @danbev in ruby : ignore "Downloading" output in test_log_suppress #3106
• server : add --no-gpu option to print usage output by @danbev in server : add --no-gpu option to print usage output #3098
• whisper : add check that target name exists by @danbev in whisper : add check that target name exists #3103
• sync : ggml by @ggerganov in sync : ggml #3108
• ruby : refine HTTP cache feature by @KitaitiMakoto in ruby : refine HTTP cache feature #3109
• feat(server): bump httplib.h to version 0.20.0 by @sachaarbonel in feat(server): bump httplib.h to version 0.20.0 #3101
• removed stdc++fs by @JaredTweed in removed stdc++fs #3097
• docs : Update cli documentation by @antimatter96 in docs : Update cli documentation #3102
• cli : Support "-" for stdout like stdin by @danielzgtg in cli : Support "-" for stdout like stdin #3050
• feat(server): Improve HTTP Connection Abort Handling by @sachaarbonel in feat(server): Improve HTTP Connection Abort Handling #3112
• whisper: remove MSVC warnings pragmas by @danbev in whisper: remove MSVC warnings pragmas #3090
• ci : add zip extension to xcframework artifact name by @danbev in ci : add zip extension to xcframework artifact name #3120
• ci : zip windows artifacts for release uploading by @danbev in ci : zip windows artifacts for release uploading #3124
• sync : ggml by @ggerganov in sync : ggml #3125
• ci : add bindings-java jar artifact to release by @danbev in ci : add bindings-java jar artifact to release #3126
• sync : ggml by @ggerganov in sync : ggml #3127
• stream.wasm : add HEAPU8 to exported runtime methods by @danbev in stream.wasm : add HEAPU8 to exported runtime methods #3130
• whisper : deprecate WHISPER_CCACHE CMake option by @danbev in whisper : deprecate WHISPER_CCACHE CMake option #3131
• wasm : add note about worker.js file generation [no ci] by @danbev in wasm : add note about worker.js file generation [no ci] #3133
• examples : add HEAPU8 to all of the exported runtime methods by @enesgrahovac in examples : add HEAPU8 to all of the exported runtime methods #3134
• ruby : omit test_build_options locally by @danbev in ruby : omit test_build_options locally #3132
• ruby : test extra build options only when env var specified by @KitaitiMakoto in ruby : test extra build options only when env var specified #3136
• examples : update link to Paul Tol's color scheme [no ci] by @danbev in examples : update link to Paul Tol's color scheme [no ci] #3140
• Update Readme to recommend same Openvino as Python tools by @peardox in Update Readme to recommend same Openvino as Python tools #3138
• cli : print color scheme info for --print-colors by @danbev in cli : print color scheme info for --print-colors #3141
• docs : fix -owts flag typo karaoke section [no ci] by @danbev in docs : fix -owts flag typo karaoke section [no ci] #3142
• whisper : remove dummy commit comment [no ci] by @danbev in whisper : remove dummy commit comment [no ci] #3143
• vad : add initial Voice Activity Detection (VAD) support by @danbev in vad : add initial Voice Activity Detection (VAD) support #3065
• docs: Update README.md for whisper.objc app by @tomer-schles in docs: Update README.md for whisper.objc app #2569
• vad : remove shortform for --vad option in cli.cpp by @danbev in vad : remove shortform for --vad option in cli.cpp #3145
• examples : add VAD speech segments example by @danbev in examples : add VAD speech segments example #3147
• sync : ggml by @ggerganov in sync : ggml #3148
• server : add --flash-attn usage output by @danbev in server : add --flash-attn usage output #3152
• vad : add download-vad-model scripts by @danbev in vad : add download-vad-model scripts #3149
• examples : add --print-confidence option to cli by @danbev in examples : add --print-confidence option to cli #3150
• whisper : add build_*/ to .gitignore [no ci] by @danbev in whisper : add build_*/ to .gitignore [no ci] #3157
• vad : store VAD context in whisper_state by @danbev in vad : store VAD context in whisper_state #3156
• vad : return early if no vad segments are detected by @danbev in vad : return early if no vad segments are detected #3158
• examples : add vad-speech-segments to win warns [no ci] by @danbev in examples : add vad-speech-segments to win warns [no ci] #3170
• sync : ggml by @ggerganov in sync : ggml #3171
• ruby : add GGML_SYCL_DNN option to ruby bindings by @danbev in ruby : add GGML_SYCL_DNN option to ruby bindings #3172
• Fix CMakeLists.txt to handle deprecated gpu Warnings by @jugal-sheth in Fix CMakeLists.txt to handle deprecated gpu Warnings #3163
• whisper : remove redundant assignments by @danbev in whisper : remove redundant assignments #3178
• docs: fix very minor typo in README by @alpaim in docs: fix very minor typo in README #3179
• docs : add VAD model download instructions [no ci] by @danbev in docs : add VAD model download instructions [no ci] #3180
• feat(server): Add k6 Load Testing Script by @sachaarbonel in feat(server): Add k6 Load Testing Script #3175
• ci : use dynamic libopenblas.dll for window-blas by @danbev in ci : use dynamic libopenblas.dll for window-blas #3177
• docs : Fix VAD section heading levels in README by @KitaitiMakoto in docs : Fix VAD section heading levels in README #3186
• Quickfix for swedish umlauts + expose model inference settings in talk-llama.cpp by @matteng1 in Quickfix for swedish umlauts + expose model inference settings in talk-llama.cpp #3187
• node : enable no_prints to suppress all output by @danbev in node : enable no_prints to suppress all output #3189
• docs : convert README_sycl.md to utf8 format [no ci] by @danbev in docs : convert README_sycl.md to utf8 format [no ci] #3191
• sync : ggml by @ggerganov in sync : ggml #3193
• ci : update windows artifact uploads steps by @danbev in ci : update windows artifact uploads steps #3192
• tests : add a new benchmark test for long-form audio by @fujimotos in tests : add a new benchmark test for long-form audio #3185
• Install shared libs when using GGML_BACKEND_DL by @peardox in Install shared libs when using GGML_BACKEND_DL #3195
• ruby : add VAD support, migration to Ruby's newer API by @KitaitiMakoto in ruby : add VAD support, migration to Ruby's newer API #3197
• whisper : remove whisper_load_backends function by @danbev in whisper : remove whisper_load_backends function #3196
• sync : ggml by @ggerganov in sync : ggml #3202
• ggml : Fix backtrace breaking Windows build by @danielzgtg in ggml : Fix backtrace breaking Windows build #3203
• ruby : handle build options on installation by @KitaitiMakoto in ruby : handle build options on installation #3206
• vad : revisit timestamp alignment/mapping by @danbev in vad : revisit timestamp alignment/mapping #3173
• ruby : add Core ML support by @KitaitiMakoto in ruby : add Core ML support #3214
• sync : ggml by @ggerganov in sync : ggml #3215
• node : add language detection support by @danbev in node : add language detection support #3190
• ci : use mirrors.kernel.org for Ubuntu packages by @danbev in ci : use mirrors.kernel.org for Ubuntu packages #3220
• musa: correct MUSA SDK rc4.0.1 download URL by @yeahdongcn in musa: correct MUSA SDK rc4.0.1 download URL #3217
• fix: apply whisperParams in fullTranscribeWithTime by @Jo4sDev in fix: apply whisperParams in fullTranscribeWithTime #3201
• ci : add mirror for ports.ubuntu.com (ARM packages) by @danbev in ci : add mirror for ports.ubuntu.com (ARM packages) #3221
• ruby : Add parallel transcription support by @KitaitiMakoto in ruby : Add parallel transcription support #3222
• devops: add main-intel dockerfile by @kwaa in devops: add main-intel dockerfile #3229
• ci(docker): build and publish main-intel image by @kwaa in ci(docker): build and publish main-intel image #3231
• ruby : output format by @KitaitiMakoto in ruby : output format #3237
• sync : whisper.cpp by @ggerganov in sync : whisper.cpp #3239
• ruby : add cleaning of library names in dependencies by @danbev in ruby : add cleaning of library names in dependencies #3241
• ci : update windows runner to windows-2022 by @danbev in ci : update windows runner to windows-2022 #3242
• ruby : add .gitignore entries for ext directory by @danbev in ruby : add .gitignore entries for ext directory #3245
• cli : fix short name conflict for vad options [no ci] by @danbev in cli : fix short name conflict for vad options [no ci] #3247
• server : add Voice Activity Detection (VAD) support by @danbev in server : add Voice Activity Detection (VAD) support #3246
• whisper : fix VAD processing for skipped audio segments by @danbev in whisper : fix VAD processing for skipped audio segments #3230
• feat(server): graceful shutdown, atomic server state, and health endpoint Improvements by @sachaarbonel in feat(server): graceful shutdown, atomic server state, and health endpoint Improvements #3243
• let people select the model the wanna use with yt-wsp.sh using a second argument by @w1redch4d in let people select the model the wanna use with yt-wsp.sh using a second argument #3251
• examples : set the C++ standard to C++17 for server by @danbev in examples : set the C++ standard to C++17 for server #3261
• whisper : clear result_all if vad_samples is empty by @danbev in whisper : clear result_all if vad_samples is empty #3262
• sync : ggml by @ggerganov in sync : ggml #3264
• examples : add stereo to mono conversion in read_audio_data by @danbev in examples : add stereo to mono conversion in read_audio_data #3266
• cmake : fix android build by @ggerganov in cmake : fix android build #3265
• android : update CMakeLists.txt to use FetchContent for ggml by @danbev in android : update CMakeLists.txt to use FetchContent for ggml #3268
• sync : ggml by @ggerganov in sync : ggml #3272
• ruby : specify Apple frameworks explicitly on build by @KitaitiMakoto in ruby : specify Apple frameworks explicitly on build #3270
• ci : add apt-get clean to musa Dockerfile by @danbev in ci : add apt-get clean to musa Dockerfile #3275
• whisper-command: write commands to text file by @aaron-ang in whisper-command: write commands to text file #3273
• chore: add .gitignore entries for OpenVINO support by @mochiya98 in chore: add .gitignore entries for OpenVINO support #3276
• ci : reduce musa image size by @danbev in ci : reduce musa image size #3277
• backport CoreML features to macos < 14 by @glaszig in backport CoreML features to macos < 14 #3255
• feat: update java example by @Jo4sDev in feat: update java example #3281
• ci : enable main-cuda build by @danbev in ci : enable main-cuda build #3282
• stream : add nullptr check of whisper_context by @danbev in stream : add nullptr check of whisper_context #3283

New Contributors

• @fujimotos made their first contribution in whisper : fix "bench-all outputs an invalid result on larger models" #3002
• @gregsadetsky made their first contribution in fix dead link to models in readme #3006
• @Olli made their first contribution in Update uri.rb #3016
• @ekaitz-zarraga made their first contribution in Fix README.md #3024
• @jeffklassen made their first contribution in Update README.md to note newer NVIDIA GPUs #3031
• @Podre-Henrique made their first contribution in Fix deprecated FFmpeg functions #3073
• @yeahdongcn made their first contribution in Add Moore Threads GPU support and update GitHub workflow for MUSA build #3069
• @JaredTweed made their first contribution in removed stdc++fs #3097
• @antimatter96 made their first contribution in docs : Update cli documentation #3102
• @danielzgtg made their first contribution in cli : Support "-" for stdout like stdin #3050
• @enesgrahovac made their first contribution in examples : add HEAPU8 to all of the exported runtime methods #3134
• @peardox made their first contribution in Update Readme to recommend same Openvino as Python tools #3138
• @tomer-schles made their first contribution in docs: Update README.md for whisper.objc app #2569
• @jugal-sheth made their first contribution in Fix CMakeLists.txt to handle deprecated gpu Warnings #3163
• @alpaim made their first contribution in docs: fix very minor typo in README #3179
• @matteng1 made their first contribution in Quickfix for swedish umlauts + expose model inference settings in talk-llama.cpp #3187
• @Jo4sDev made their first contribution in fix: apply whisperParams in fullTranscribeWithTime #3201
• @kwaa made their first contribution in devops: add main-intel dockerfile #3229
• @w1redch4d made their first contribution in let people select the model the wanna use with yt-wsp.sh using a second argument #3251
• @aaron-ang made their first contribution in whisper-command: write commands to text file #3273
• @mochiya98 made their first contribution in chore: add .gitignore entries for OpenVINO support #3276
• @glaszig made their first contribution in backport CoreML features to macos < 14 #3255

Full Changelog: v1.7.5...v1.7.6

---

This discussion was created from the release v1.7.6.

[JulianPscheid]

Nice! Can't wait to try out the VAD feature. Currently doing a simple energy detection to filter out silence and hallucinations. Hopefully the Silero model will be an improvement on that. Thanks to everyone who contributed to this release!