can I enable "-dl" option and detect multiple languages (in one audio-clip)? Thanks!

[minounou]

Hi,

I could run whisper-cli to detect language and it works great (as the following):
whisper-cli -f ./jfk.wav -dl --model ./ggml-small.bin

it works great and I could get the auto-detected language (like the following)::

...
whisper_full_with_state: auto-detected language: en (p = 0.958143)
...

my question is: can I also use the same "-dl" option to detect a wav-file which has multiple languages? in this case will whisper-cli show 2 language-types (similar as the above, but show 2 language-types)? Thanks!

[ggerganov]

whisper-cli does not have this specific function available, but it is something that can be very easily implemented:

whisper.cpp/include/whisper.h

Lines 360 to 372 in b1f5c11

	// Use mel data at offset_ms to try and auto-detect the spoken language
	// Make sure to call whisper_pcm_to_mel() or whisper_set_mel() first
	// Returns the top language id or negative on failure
	// If not null, fills the lang_probs array with the probabilities of all languages
	// The array must be whisper_lang_max_id() + 1 in size
	// ref: https://github.com/openai/whisper/blob/main/whisper/decoding.py#L18-L69
	WHISPER_API int whisper_lang_auto_detect(
	struct whisper_context * ctx,
	int offset_ms,
	int n_threads,
	float * lang_probs);

PRs welcome.

[minounou]

Thanks for the info! I am trying to create a PR and is the following my understanding correct? Thanks!
in here:

whisper.cpp/src/whisper.cpp

Line 5512 in b1f5c11

WHISPER_LOG_INFO("%s: auto-detected language: %s (p = %f)\n", __func__, params.language, probs[whisper_lang_id(params.language)]);

PR will output more probs if (for example) there is a group of probs whose values are much bigger than others, then output all these bigger probs (is this correct?):

        WHISPER_LOG_INFO("%s: auto-detected language: %s (p = %f)\n", __func__, languages[#1], probs[#1]);
        WHISPER_LOG_INFO("%s: auto-detected language: %s (p = %f)\n", __func__, languages[#2], probs[#2]);
         ...
        WHISPER_LOG_INFO("%s: auto-detected language: %s (p = %f)\n", __func__, languages[#k], probs[#k]);