- \n
- Google Gemini 1.5 Pro can now listen to uploaded audio files and interpret them without referring to written transcripts.<\/li>\n\n\n\n
- The new feature can be used to process information from recorded interviews, videos and earning calls.<\/li>\n\n\n\n
- The model is now available in public preview through its Vertex AI platform to build applications.<\/li>\n<\/ul>\n\n\n\n
Gemini 1.5 Pro Can Now Process Audio Streams<\/strong><\/h2>\n\n\n\n
Google now allows users to prompt audio files to the Gemini 1.5 Pro model. The model then extracts relevant information from the audio inputs, including music and speech, to generate a response based on the prompt. <\/strong><\/p>\n\n\n\n
This feature lets users feed large audio files without the need to provide transcripts for them. Users can also record audio clips instead of typing long prompts.<\/p>\n\n\n\n
Gemini excels at interpreting very large audio files within a matter of seconds. When an audio file is uploaded, it automatically counts the number of tokens and displays it on the interface. It then gives the required response in a structured format.<\/p>\n\n\n\n
It can summarize the audio, extract necessary information, answer direct questions, provide reasoning, and explain concepts. Thus, with Gemini being given \u2018ears,\u2019 it enhances the user experience and makes it extremely convenient for users when they need to upload audio files or speak out lengthy questions.<\/p>\n\n\n\n
The announcement came with many new updates at Next ’24 event<\/a>, including making it available in public preview, inpainting in Imagen 2.0, and new prompt management capabilities for their large models. Here is how they described the new feature:<\/p>\n\n\n\n
\n
“In addition, we are announcing that Gemini 1.5 Pro on Vertex AI now supports the ability to process audio streams including speech, and even the audio portion of videos. This enables seamless cross-modal analysis that provides insights across text, images, videos, and audio \u2014 such as using the model to transcribe, search, analyze, and answer questions across earnings calls or investor meetings.”<\/p>\n<\/blockquote>\n\n\n\n
Now, users can upload their lectures, recorded interviews, conferences, earning calls, music samples, conversations with friends, podcasts or even audio from videos, and get them analyzed through Gemini. After its initial launch, developers found many interesting features in Gemini 1.5 Pro<\/a> after testing.<\/p>\n\n\n\n
How to Access It?<\/strong><\/h2>\n\n\n\n
The new audio processing capabilities to Gemini 1.5 Pro model through their Vertex AI platform. <\/strong><\/p>\n\n\n\n
Anyone can access the model available for free through Google DeepMind\u2019s official website<\/a>. Click on \u2018Try Gemini 1.5.\u2019 You will have to sign in before being redirected to the Google AI studio through which the model can be tested for free. The interface for the model is shown below:<\/p>\n\n\n
\n![\"Google](\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/image12-1024x603.png\")
<\/figure>\n<\/div>\n\n\nNow, to upload audio files, the user will have to first connect your Google Drive. After this, they can record an audio, upload an audio from Drive, or upload an audio from your local machine.<\/p>\n\n\n\n
What You Can Do with this new Listening feature?<\/strong><\/h2>\n\n\n\n
We tested the audio feature for various use cases. They are as follows:<\/p>\n\n\n\n
Example 1: Conversation<\/strong><\/h3>\n\n\n\n
We provided Gemini with an audio of a conversation between a college student and a librarian. We asked it to summarize the audio in a structured format. The audio is of 2 minutes and 36 seconds. <\/p>\n\n\n\n
1) I provided Gemini with an audio of a conversation between a college student and a librarian. I asked Gemini to summarize the audio in a structured format that included all the important information mentioned in the audio. I also asked a follow-up question. pic.twitter.com\/gNCIOTnPsR<\/a><\/p>— Dhruv (@dhruvvvvvvvvv_) April 9, 2024<\/a><\/blockquote>