How to Build an AI Messaging App with Spacy, Twilio, and Flask
Time to read: 13 minutes
The power of natural language processing (NLP) allows you to command your computer to effortlessly carry out your instructions and send personalized messages to specific numbers on your behalf. Without the aid of NLP, you'd likely have to adhere to a specific command format and employ regex. This recognition of entities in various sentence structures enhances the application's user-friendliness and intuitiveness, allowing users to interact more naturally.
In this article, we will explore how to leverage the power of Spacy NLP , Twilio , and Flask to develop a system capable of identifying entities within sentences in order to send messages through Twilio, and offering a user-friendly chat interface for seamless interaction.
Prerequisites
- Python 3.x installed.
- Node.js(v16.13.2) and npm installed.
- An SMS enabled cell phone to test out the application.
- A Google account to access Google Colab .
Get a free Twilio number
Twilio provides a number of services and APIs that enable developers to build communication functionality into their applications. In this article, we will be leveraging the power of Programmable Messaging , which enables developers to send and receive SMS and MMS messages.
If you are new to Twilio, follow the steps below to get started. If you already have a Twilio account and Twilio phone number, feel free to skip to the next section.
To get a free Twilio number, sign up for a Twilio account. Sign up for a free Twilio account here . To learn more about Twilio trial accounts, visit this documentation page.
To test a trial account by sending SMS or calls to someone else, you will need to verify your phone number first. Click on the "verify their number" hyperlink to navigate to that screen and verify the number with a one-time password.
Enter your phone number and select a preference for verification.
After phone number verification is complete, the number will be displayed on the Verified Caller IDs screen.
On the Twilio Console , your account information, including the Account SID, the Auth Token, and your Twilio Phone number, will be visible. This will be useful when creating the Flask REST API to send an automated SMS to a number from Twilio.
Set up the project
Once the Twilio Account setup is complete, the next step is to create an AI model and train it with our own custom training samples.
For convenient code maintenance and versioning, you can create a GitHub repository. It can be cloned locally, and all changes made there, which can be pushed to the remote repository periodically. This is so that no progress is lost and changes can be easily rolled back.
To do this, create a new GitHub repository called "Python_Twilio_AI_Messaging_Chatbot". All the code for the next steps is available on this GitHub repository .
Next, follow the instructions in the GitHub documentation to generate a fine-grained personal access token for GitHub. Choose "Python_Twilio_AI_Messaging_Chatbot" under Repository access by selecting Only select repositories and searching for this repository in the list. Then, under Permissions, scroll down to the Contents section and select Read and write in the dropdown field.
Once you have generated your personal access token, copy it and keep it in a safe place, as it will not be shown again.
Clone the repository locally by running the following command, replacing YOUR_GITHUB_USERNAME and YOUR_PERSONAL_ACCESS_TOKEN with your own username and personal access token, respectively.
Once you have cloned the repository, change into the Python_Twilio_AI_Messaging_Chatbot directory:
cd Python_Twilio_AI_Messaging_Chatbot
To set up the remote repository to keep track with the main branch, run this command:
Use Named Entity Recognition (NER) in the application
Named Entity Recognition (NER) is a natural language processing (NLP) task that involves identifying and classifying named entities in text. Named entities are specific words or phrases that represent entities such as people, organizations, locations, dates, quantities, and other types of proper nouns.
The goal of NER is to automatically locate and categorize these named entities within a given text. This task is crucial in various NLP applications, including information extraction, question answering systems, sentiment analysis, and machine translation. By identifying and classifying named entities, NER enables a deeper understanding of the text and helps extract valuable information from unstructured data.
NER is usually implemented using machine learning techniques, particularly supervised learning algorithms. These algorithms are trained on annotated datasets where human annotators label the named entities in a corpus of text. Various machine learning approaches, such as rule-based methods, Hidden Markov Models (HMMs), Conditional Random Fields (CRFs), and deep learning models like Recurrent Neural Networks (RNNs) and Transformer-based architectures, can be used for NER.
Use Spacy to train custom NER models
Spacy , a popular NLP library, implements Named Entity Recognition (NER) using a combination of rule-based matching and statistical models. The models learn to predict the named entity labels based on the contextual information of the words in a sentence. This library provides the capability to train your own custom NER models using your annotated training data.
Spacy provides pretrained models that are already trained on large corpora and can recognize a wide range of named entities. These models are trained using labeled data and can be conveniently loaded into your application.
This library also allows you to define custom rules to match specific patterns of words or phrases as named entities. These rules can be based on lexical patterns, part-of-speech tags, or other linguistic features. Rule-based matching is useful when you have a small set of specific entities that you want to extract.
Train Custom NER Models
To train a custom NER model in Spacy, you need to provide annotated training data where each entity in the text is labeled with its corresponding entity type. Spacy uses a machine learning algorithm, such as a convolutional neural network (CNN) or a transformer-based architecture, to learn the patterns and features that distinguish different named entity types. Training a custom NER model involves iterating over the training data and optimizing the model’s parameters to improve its entity recognition performance.
Fine-tune and evaluate
After training a custom NER model, you can fine-tune it on your specific domain or task by providing additional labeled examples. Spacy provides evaluation metrics to assess the performance of the model, such as precision, recall, and F1-score, which help measure the model’s ability to correctly identify named entities.
Using the data we have about people's phone numbers and text messages, we need to apply Spacy's pre-trained models to achieve good results with fewer training iterations.
Train your custom named entity recognition model
Google Colab lets you train machine learning models online with great speed. Create a new Google colab notebook through Google's research dashboard and add the following code:
Now add your training samples. The training samples are stored as tuples, where the first element in the tuple is a string that represents a sample input sentence.
The next element in the tuple is the dictionary with the key “entities”. The entities key has the list of the named entities in the input sentence. The first element is the starting index of the named entity in the string and the second element is the ending index of the named entity in the string, for example, RECIPIENT_PHONE or MESSAGE.
The third element is the actual named entity that the word between those two indices can be classified as. Let's take an example for better understanding.
"("Send a message to +15551234567 with the message 'Hello'", {"entities": [(18, 30, "RECIPIENT_PHONE"), (49, 54, "MESSAGE")]})"
Here "Send a message to +15551234567 with the message 'Hello'" is the sentence. The word from index 18 till before 30, which is +15551234567 is marked as RECIPIENT_PHONE. The word from index 49 till before 54 is Hello, which is marked as the entity MESSAGE.
Add the following sample training data to your Google Colab notebook:
Execute the above two code cells. Next, add the following code to the Google Colab file:
Next, type in this command and execute it:
Afterwards, execute this command:
Once you have trained this new model to identify named entities in text, you’ll want to test it yourself, with an input sentence. To infer the trained model, execute the below code. You can also replace the sentence in the below code with any other sentence you want to test the model with.
Zip and download your trained model to use in the chatbot application. Be sure to name the zip file as model2.zip :
Once you run this cell, the best model will be downloaded on your system. Awesome, you did it!
Next, we will connect this model to our backend Flask API in order to run input sentences and identify named entities. Then use the recipient number and the message to send a message from the Twilio number to the recipient with the corresponding message.
Build the backend REST API with Flask and Python
Inside the Python_Twilio_AI_Messaging_Chatbot directory, create a folder named backend :
Inside this folder, create and activate a new Python virtual environment. Install the dependencies you will need for this project. If you are on a Unix or macOS system, run the following commands to do this:
If you following this tutorial on a Windows machine, run the following commands instead:
Set the following environment variables, replacing the placeholder text with your values for Account SID, Auth Token, and Twilio phone number. You can find these in the Twilio Console :
You previously downloaded your model as model2.zip into your system at the end of the Train your custom named entity recognition model section. Unzip this file inside the backend folder. This model zipfile is can be referenced from this GitHub link or download it from this Google Drive link.
Create a file called app.py and add the following code . Replace `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN` and `TWILIO_PHONE_NUMBER` with your Twilio account credentials in the code.
This code imports necessary libraries and modules including spacy, Flask, request, jsonify, and MessagingResponse from twilio.twiml.messaging_response. The spacy.cli.CORS (Cross-Origin Resource Sharing) is enabled for the Flask app using the flask_cors extension.
The Python Flask script loads a pre-trained spaCy model using the spacy.load function and assigns it to the variable nlp1. The generate_bot_response function takes a user message as input, processes it using the nlp1 model, and extracts relevant information from the named entities detected in the message. It then calls the send_message function with the recipient number and message extracted from the user's input. If an exception occurs while sending the message, it returns a default response.
The send_message function sends a message using the Twilio service. It uses the Twilio Python SDK to send an SMS message to the specified recipient number. It returns a success message if the message is sent successfully, or an error message if there's an exception.
The /sms route is defined for the Flask app, which handles incoming POST requests. It expects a JSON payload with a "Body" field containing the user's message. It calls the generate_bot_response function with the user's message and constructs a Twilio MessagingResponse object. It then returns a JSON response with the bot's response.
The __name__ == "__main__" condition ensures that the Flask app is only run when the script is executed directly, not when it is imported as a module. The app is run in debug mode for development purposes.
Build the chatbot UI with React
Now it's time to create the React frontend for the user to interact with.
Open a new terminal window. In the root directory of the Python_Twilio_AI_Messaging_Chatbot project, create a new React app called chatbot-frontend:
Once the app has been created, change into the directory, install `react-bootstrap`, and run the app with the following commands:
You can download an image from any site such as https://unsplash.com/. Rename it to chatbot-image.png and store it inside the frontend directory. This is for the logo of the chatbot for the UI.
Verify the app is running by visiting http://localhost:3000 in the browser. The default React app will be visible. Next, update the code in the subdirectory src/App.js to the following code:
This React component represents a chat interface. The component uses the useState hook from React to define two state variables: userInput and chatHistory.
The sendMessage function is an asynchronous function that sends the user's message to a backend API using the fetch function. It then updates the chat history with the user's message and the bot's response. Finally, it clears the input field.
The component returns JSX, which represents the structure and content of the chat interface. The JSX is wrapped in a div with the class name "chat-container" and includes a chat header section with an image and a heading. The chat body section renders the chat history by mapping over the chatHistory state variable and creating a message container for each message. These containers have a dynamic class name based on the sender.
The JSX object includes a chat input section with an input field and a send button. This is bound to the userInput state variable and updates when the user types in the input field. The send button triggers the sendMessage function when clicked.
Now update the App.css file with the following:
- The .chat-container class sets the width, margin, padding, border, border-radius, and box shadow properties for the main container of the chat interface.
- The .chat-header class sets the display and align-items properties for the header section of the chat interface.
- The .chatbot-image class sets the width, height, border-radius, and margin properties for the chatbot's image.
- The .chat-body class sets the margin-top, display, flex-direction, and align-items properties for the body section of the chat interface.
- The .message-container class sets the margin-bottom property for each message container in the chat interface.
- The .user-message class sets the background-color, padding, border-radius, and align-self properties for user messages in the chat interface.
- The .bot-message class sets the background-color, padding, border-radius, and align-self properties for bot messages in the chat interface.
- The .chat-input class sets the margin-top, display, and align-items properties for the input section of the chat interface.
- The .chat-input input class sets the flex-grow, padding, border, border-radius, and margin-right properties for the input field in the chat interface.
- The .chat-input button class sets the padding, background-color, color, border, border-radius, and cursor properties for the send button in the chat interface.
- The .chat-input button:hover class sets the background-color when the send button is being hovered over.
Now it's time to run both the backend and frontend simultaneously and then test the application!
Test the AI powered messaging app
Run both the backend server and frontend app simultaneously, if they are not still already running from earlier.
Once you have unzipped the model, open a new terminal. Change into the backend folder and type:
The server will run on port 5000:
In another terminal, change into the chatbot-frontend directory and type npm start.
Navigate to http://localhost:3000/ to see the Twilio AI Messenger application.
You will first send your message, something like “Can you send the message “Hello, how are you?” to +9197895817457”. You must put your message within quotation marks, or you will run into errors. The chatbot will reply with the status saying “Message sent successfully to recipient +9197895817457 with the message Hello, how are you?”.
To test the application, enter your message in the text box and send it. Make sure the phone number is in E.164 format. Once you send the text to the chatbot, the chatbot will respond if the message was successfully sent or if there was an error.
You will receive an SMS notification with the text you sent to the chatbot. If the message was successful, the chatbot will respond with "successfully sent the message to [GIVEN_PHONE_NUMBER]" along with the message "Hello, how are you?".
This is the message printed in the backend console:
Whenever we hit the endpoint with some data to send a message, it prints the message text and the recipient's phone number. Once the message is successfully delivered, it prints the reply text that will be sent to the chatbot all well. You can see what calls were made to the API endpoint from these server logs.
Try with different prompts and sentences to test out the Twilio AI messenger:
The above image shows a chat with the Twilio AI messenger. It continues from the previous chat where we asked the chatbot to send a message to a certain number. Adding another example, we ask it to send another message to a different number, which it successfully does. The message is “Send the test “Hey what is up” to +919670034500. The reply from the chatbot is “Message sent successfully to recipient +919670034500 with the message Hey what is up? This shows that our model can correctly detect the named entities, message and phone number and send the message successfully!
This is the message printed in the backend console:
A 200 response is displayed along with the server logs for that API call where the message and phone number are printed. The reply that is sent to the chatbot is also logged here on the terminal.
Here are a few more prompt ideas:
The above image displays the Twilio AI Messenger UI and a few prompts that you can test it out with. They include “Send a text to +918795678345 with the message “How are you doing?”. This is just to reaffirm that the model is able to correctly detect the named entities even if we have input sentences which are structured differently.
This shows the messages received on my phone screen from the Twilio AI Messenger. I tried with a set of prompts and the screen shows the messages I received. Every message starts with “Send from your Twilio trial account - “ and then the actual message. I sent messages saying “Hello, how are you?”, “Hello, how are you doing today?”.
The above screenshot is some of the messages received with the Twilio AI messenger.
In case you send any messages without the recipient number, this will be the error screenshot:
If your input is missing a message to send, the chatbot responds with “Couldn’t understand what message you want to send. Please resend.”
In case there are some issues with the Twilio credentials, this is the response:
The Twilio AI Messenger displays a response when the Twilio credentials in the backend are invalid.
What's next for AI messaging apps?
The integration of Spacy NLP, Twilio, and Flask in creating our messaging system has empowered us to forge meaningful connections and deliver a seamless user experience. By leveraging Spacy NLP's entity recognition, Twilio's messaging capabilities, and Flask's backend framework, we have built a powerful application that understands and responds to user input.
Looking forward, there are numerous avenues for enhancing this project. Expanding the application's NLP capabilities by integrating sentiment analysis, intent detection, and language translation can lead to a deeper understanding of user messages and more sophisticated responses. Improving the user interface with features like message threading, emojis, and user profiles, along with real-time updates using web sockets, can create a more dynamic and engaging chat experience for users.
Related Posts
Related Resources
Twilio Docs
From APIs to SDKs to sample apps
API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
Resource Center
The latest ebooks, industry reports, and webinars
Learn from customer engagement experts to improve your own communication.
Ahoy
Twilio's developer community hub
Best practices, code samples, and inspiration to build communications and digital engagement experiences.