Build an AI Buddy with Go, OpenAI, Retell AI, and Twilio Programmable Voice
Time to read: 5 minutes
Build an AI Buddy with Go, OpenAI, Retell AI, and Twilio Programmable Voice
In this tutorial, you will learn how to build an AI buddy that you can call and chat with about how your day went. You will be using OpenAI to set up the chatbot, Twilio Programmable Voice to initiate voice calls, and Retell AI API to convert responses from your chatbot responses to human-like speech.
Prerequisites
To follow along with this tutorial, you need the following:
- A Twilio account
- A Twilio phone number
- An OpenAI account
- A Retell AI account
- Go installed on your local machine
- Experience with the Twilio Voice webhook, Go, and TwiML
- The ngrok CLI (or an alternative tunnel service)
Create the Go app
The first step in this tutorial is to set up your project’s directory and install the required dependencies. First, create a new folder for the project.
Then, initialize a new Go module.
Install the required packages
Now, you need to install four Go packages for this tutorial. These are:
- Gin: An HTTP web framework for Go
- Gorilla WebSocket: A Go package for implementing the WebSocket protocol
- Go OpenAI: An OpenAI wrapper for Go
- Twilio-go: A Go package for interacting with Twilio APIs
- GoDotEnv: A Go package for loading environment variables from a .env file.
To install the packages, run the following command in your terminal.
Retrieve the required environment variables
Then, create a .env file in your project’s root folder and add the following code to the file.
Replace <ENTER_OPENAI_SECRET_KEY>
with your OpenAI API key. Then, replace <ENTER_RETELL_AI_SECRET_KEY>
with your Retell AI API key.
![](/content/dam/twilio-com/global/en/blog/developers/developers-imagery-by-post/a-c/build-an-ai-buddy-with-go,-openai,-retell-ai,-and-twilio-programmable-voice-/openai-project-keys.png/_jcr_content/renditions/compressed-1600.webp)
To retrieve your OpenAI API key, log in to your OpenAI home page, select API Keys in the sidebar, and click "+ Create new secret key". Enter a name for your secret key, set the permissions, and then click Create secret key. Finally, copy your secret key in the modal that appears.
![](/content/dam/twilio-com/global/en/blog/developers/developers-imagery-by-post/a-c/build-an-ai-buddy-with-go,-openai,-retell-ai,-and-twilio-programmable-voice-/retellai-api-keys.png/_jcr_content/renditions/compressed-1600.webp)
To retrieve your Retell AI API key, log in to your Retell AI dashboard, navigate to API Keys, and copy your API key under Credentials.
Write the go code
Following that, create a main.go file, which will serve as the entry point to your application, in your project’s root folder, and add the following code to the file.
The godotenv.Load()
command in the main()
function loads your environment variables from your .env file to your application’s environment using GoDotEnv. The GetOpenAISecretKey()
function will be used to retrieve your OpenAI secret key, while the GetRetellAISecretKey()
function will be used to retrieve your Retell AI secret key.
Set up the Twilio Programmable Voice webhook
The next step is to set up a webhook handler for Twilio phone calls. Webhooks are HTTP handlers that allow for event-driven communications between two applications, i.e., in this project, communications between Twilio and your AI server.
To enable them, add the following code to the bottom of your main.go file.
Then, update the import list to match the following:
The code defines two structs. The first (RegisterCallRequest
) helps pass the required request parameters in the request to Retell AI. The second (RegisterCallResponse
) helps marshall the JSON in the body of the response from the request to Retell AI.
Then, there's an HTTP handler that receives the webhook request from Twilio after someone makes a call to your Twilio phone number. The handler registers a call with Retell AI, setting the call encoding algorithm(AudioEncoding
) to mulaw
and the call sample rate (SampleRate
) to 8000 as required by Twilio Media Streams, and receives the call's ID.
The call ID is then used to configure a websocket URL that will be returned to Twilio via the webhook request. The websocket server will handle all audio exchanges between Twilio and your backend AI server via Retell AI.
Also, to allow for bidirectional audio exchange between Twilio and Retell AI over the websocket, the websocket URL is connected to Twilio using the TwiML markup language Voice Stream instruction wrapped in a <Connect> verb.
Next up, update the main()
function in main.go to match the following. It registers your webhook route and configures the application to listen on port 8081.
Set up your backend AI server
The next step is to set up the backend AI server. When Twilio connects to your Retell AI agent via the Retell AI WebSocket URL returned in the Twilio voice webhook, Retell AI connects the call to your backend AI server using the WebSocket connection that you are about to create.
Add the following code to the bottom of your main.go file to create a websocket route.
Then, add the following to your import list:
In this code, we created a websocket route that returns "Hello, I'm your AI buddy. How did your day go?" on a successful connection, and logs your messages to the terminal.
Now, register the websocket route in your main()
function, by updating the function to match the following code.
Implement your AI server
To do so, add the following code to the bottom of your main.go file.
Then, update the imports list to match the following:
In this code, you created two functions, HandleWebsocketMessages()
and GenerateAIRequest()
, for generating AI responses. The HandleWebsocketMessages()
function:
Takes in messages from your WebSocket handler
Converts the messages to an OpenAI ChatGPT prompt using the
GenerateAIRequest()
functionMakes a call to the OpenAI API to generate a response
Returns the response back to Retell AI via the WebSocket connection
Update your WebSocket handler code
Now, update your Retellwshandler()
websocket handler to match the following so that it calls the new HandleWebsocketMessages()
function. Messages sent to your websocket route will be transferred to the HandleWebsocketMessages()
to generate an AI response with the OpenAI API, which the WebSocket route will return.
Then, update your import code to the following to import the Go packages that you used.
Update your main()
function to the following to register your LLM WebSocket route.
Then, start your Go server.
After that, in a new terminal session or tab run the following command. This connects the server running on your local machine to Twilio and Retell AI.
Create a Retell AI agent for your AI Buddy
Then, run the request in a new terminal session or tab to generate a Retell AI agent for your AI server. Make sure to change <ENTER_RETELL_AI_SECRET_KEY>
to your Retell AI secret key, and <YOUR_LLM_WEBSOCKET_URL>
to your WebSocket URL. Your WebSocket should be in the following format: wss://<ngrok_url>/llm-websocket/
.
This agent will convert the AI response text to speech.
Your AI agent should be added to your Retell AI dashboard if the registration was successful.
![](/content/dam/twilio-com/global/en/blog/developers/developers-imagery-by-post/a-c/build-an-ai-buddy-with-go,-openai,-retell-ai,-and-twilio-programmable-voice-/retellai-ai-agents.png/_jcr_content/renditions/compressed-1600.webp)
Connect your webhook URL to Twilio
To connect your webhook URL to Twilio, open your Twilio Console and navigate to Phone Numbers > Manage > Active Numbers. Select your Twilio phone number, and open the Configure tab.
![](/content/dam/twilio-com/global/en/blog/developers/developers-imagery-by-post/a-c/build-an-ai-buddy-with-go,-openai,-retell-ai,-and-twilio-programmable-voice-/twilio-console-voice-configuration.png/_jcr_content/renditions/compressed-1600.webp)
Under Voice Configuration, set your Configure with dropdown to "Webhook, TwiML Bin, Function, Studio Flow, Proxy Service". Set the "A call comes in" dropdown to "Webhook", add your webhook URL to your webhook URL, and set "HTTP" to HTTP POST. Your webhook URL should be in the following format: https://<ngrok_url>/twilio-webhook/<retell_ai_agent_id>
. You can find the Retell AI agent id in the response from the curl request to Retell AI.
Finally, scroll down and click Save configuration to register your changes.
Chat with your AI Buddy
To chat with your AI buddy, make a call to your Twilio phone number. Your AI buddy should pick up your call and have a chat with you.
That's how to build an AI buddy with Go, OpenAI, Retell AI, and Twilio Programmable Voice
In this tutorial, you learned how to build an AI buddy that you can call and chat with about how your day went, using Twilio Programmable Voice, Retell AI, and OpenAI. You can also edit the AI prompt and convert your AI from a friend to a therapist, a coach, etc.
The entire code for this tutorial can be found on GitHub.
Quadri Sheriff is a Software developer and tech-savvy technical writer who specializes in Developer and API documentation and has strong knowledge of docs-as-code tools.
Related Posts
Related Resources
Twilio Docs
From APIs to SDKs to sample apps
API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
Resource Center
The latest ebooks, industry reports, and webinars
Learn from customer engagement experts to improve your own communication.
Ahoy
Twilio's developer community hub
Best practices, code samples, and inspiration to build communications and digital engagement experiences.