Is possible upload files to speech to text audio

[fagnerdireito]

Is possible upload files to speech to text audio ?

[fagnerdireito]

I don't know if there is a way to transcribe audios in Prism, however, to solve it for now I used the Laravel http client to convert the audio, and then I used Prism to generate an array and use that array to create a calendar event.

public function uploadAudio($audioData)
    {
        // Create a temporary file from the audio data
        $tempFile = tempnam(sys_get_temp_dir(), 'audio_');

        try {
            file_put_contents($tempFile, base64_decode(preg_replace('#^data:audio/\w+;base64,#i', '', $audioData)));

            $response = Http::withHeaders([
                'Authorization' => 'Bearer ' . env('OPENAI_API_KEY'),
            ])->attach(
                'file',
                file_get_contents($tempFile),
                'audio.wav'
            )->post('https://api.openai.com/v1/audio/transcriptions', [
                'model' => 'whisper-1',
                'language' => 'pt',
                'response_format' => 'text',
                'prompt' => 'Transcribe the audio. Response in portuguese of brazil.',
            ]);

            // Only attempt to delete if file exists
            if (file_exists($tempFile)) {
                unlink($tempFile);
            }

            $text = $response->body();

            $this->generateEventFromText($text);
        } catch (\Exception $e) {
            // Only attempt to delete if file exists
            if (file_exists($tempFile)) {
                unlink($tempFile);
            }

            Log::error('Error transcribing audio:', ['error' => $e->getMessage()]);
            throw $e;
        }
    }

    public function generateEventFromText($text)
    {
        $calendars = Calendar::pluck('nome')->toArray();

        $schema = new ObjectSchema(
            name: 'event',
            description: 'A structured calendar event',
            properties: [
                new StringSchema('title', 'The event title'),
                new StringSchema('start', 'The event start date in format YYYY-MM-DD HH:MM:SS'),
                new StringSchema('end', 'The event end date in format YYYY-MM-DD HH:MM:SS is always 15 minutes after start or the end of event in text'),
                new StringSchema('description', 'The event description'),
                new StringSchema('contato', 'The event contact number, is number only if exists on text'),
                new StringSchema('local', 'The event location'),
                new StringSchema('agenda', "The name of the agenda must be the names " . implode(', ', $calendars) . " If you don't set the agenda, use the first name."),
            ],
            requiredFields: ['title', 'start', 'end', 'local', 'agenda']
        );

        $response = Prism::structured()
            ->using(Provider::OpenAI, 'gpt-4o')
            ->withSchema($schema)
            ->withPrompt('Generate array of events from the text: ' . $text . '. To use in a eloquent calendar event model and create the events in the database. Only return the array, no other text or comments. The year of events is ' . now()->year . ' or the year of the event on text. Today is ' . now()->format('d/m/Y') . '.')
            ->generate();

        $review = $response->object;

        // verify if review has ['title', 'start', 'end', 'local', 'agenda']
        if (!isset($review['title']) || !isset($review['start']) || !isset($review['end']) || !isset($review['agenda'])) {
            Notification::make()
                ->title('Erro ao criar evento')
                ->body('Não foi possível criar o evento. O comando não contém os dados necessários.')
                ->danger()
                ->send();
            return;
        }

        // add calendar_id
        $review['calendar_id'] = Calendar::where('nome', $review['agenda'])->first()->id;
        $review['entidade'] = $this->entidade;
        CalendarEvent::create($review);
        $this->dispatch('refresh-calendar');

        Notification::make()
            ->title('Evento criado')
            ->body('O evento foi criado com sucesso.')
            ->success()
            ->send();
    }

[sixlive]

There isn't a way to do audio with Prism. Something I'm willing to explore in the future.

Love the calendar event creation! That is awesome. Love getting to see Prism in action. Great use of structured output.

[ChrisB-TL]

Anthropic also now officially support text/md documents, which I will PR this weekend.

I am pretty sure they support docx documents also, but not documented. Will see if I can get working.

[ChrisB-TL]

@CeriseGoutPelican - I have just PRed txt/md document support for Anthropic in #148

Unfortunately my hunch on docx was wrong. Though as and when Anthropic do start supporting them, you'll be able to do so using the existing Prism API I suspect.

[CeriseGoutPelican]

You both rock!
I need to dive into the library first and then see what I can do to help with this package :)

[orthosie]

We just added PDF (Document) support for Anthropic! I'd happily accept some PRs for more multi-modal capabilities.

OpenAi supports uploading of many different documents. It is a very common use case for document parsing. I would love to see that capability in prism to upload any files and let the providers tell if they don't support the document in the reply.

[ChrisB-TL]

Until someone PRs it, for text based documents you could take a look at markitdown. Convert to markdown and pass in as a message.

You also use a lot less tokens that way!