Run Voice Assistant

This guide will help you to run the Voice Assistant with Large Language Model in the NYRA-Agent Playground.

Prerequisites

Make sure you have the NYRA-Agent playground running. If not, follow the Run Playground guide to start the playground.

You will need following information prepared:

• STT info, any supported STT can be used. Deepgram is easy to register and get started with.

• TTS info, any supported TTS can be used. Fish.Audio is easy to register and get started with.

• LLM info, any supported LLM can be used. It is highly recommended to use OpenAI

• RTC info, currently only Agora RTC is supported. You can register your account at Agora. We assume you have your App ID and App Certificate ready when you configure your .env file.

Steps

To configure your Nyra-Agent, open the playground at localhost:3000.

1. Select the Graph Type: Choose the voice_assistant graph type.

2. Module Selection: Click on the Module Picker to open the module selection menu.

   • If your preferred STT (Speech-to-Text) or TTS (Text-to-Speech) module is not automatically selected, you can choose it from the dropdown list.

   • Note: You’ll need to configure the module with the appropriate details, such as API keys and other required information.

3. LLM Module Selection: For the Large Language Model (LLM) module, open the dropdown and select your preferred model.

4. Apply Module: Click Save Changes to apply the selected module to the graph.

5. Property Configuration: Click on the button next to the graph selection to open the property configuration.

   • You will be presented with a list of configurable properties for the selected Large Language Model.

   • Configure these properties with the relevant information, such as API credentials, model settings, and other configuration details.

6. Save Properties: Click Save Changes to apply the configured properties to the Large Language Model.

7. Confirmation: If you see the success toast, the properties have been successfully applied to the Large Language Model.

Using Azure STT

Azure STT is integrated within RTC extension module. That's why if you want to use Azure STT, you will need to select voice_assistant_integrated_stt graph type.

Bind Weather Tool to your LLM

You can bind the weather tool to your LLM module in the Nyra-Agent Playground. It is recommended to use the OpenAI LLM for this integration.

Once your agent is running, follow these steps:

1. Open Module Picker: Access the Module Picker.

2. Tool Selection: Click on the button next to the LLM module to open the tool selection menu.

3. Select Weather Tool: From the popover list, choose the Weather Tool.

4. Apply Tool: Click Save Changes to bind the weather tool to the LLM module.

5. Confirmation: If you see a success toast, the tool has been successfully applied to the LLM module.

Now you're all set! You can ask the agent about the weather by speaking to it.