Predict whether a Basketball Shot is Made with GPT-4V and Twilio Serverless via SMS

Time to read: 4 minutes

November 30, 2023
Written by
Lizzie Siegle
Twilion
Reviewed by
Sam Agnew
Twilion

In honor of 11/30/23 day (in which the digits correspond to the respective numbers of everyone's favorite NBA trio of Klay Thompson, Stephen Curry, and Draymond Green), read on to see how to build an application that predicts if a basketball shot is made using OpenAI's new GPT-4V API, Twilio Serverless, and Twilio Programmable Messaging with Node.js.

SMS example where I sent a picture of Steph shooting a three and the response is "Shot prediction from GPT-4 V: "Based on the image, I would estimate a 75% confidence level that the shot went in. The shooter, who appears to be a skilled and confident player given the jersey number, is positioned beyond the three-point line, a location from which players of high caliber are known to be effective. Additionally, the defender, while making an effort to block the shot, is trailing slightly which increases the shooter's chances of successfully making the shot. The shooter's form looks focused and stable, factors that contribute to a higher likelihood of scoring.""

Do you prefer learning via video more? Check out this TikTok summarizing this tutorial in one minute!

GPT-4V

ChatGPT's image understanding is powered by a combination of multimodal GPT-3.5 and GPT-4 models.GPT-4 Vision (or GPT-4V) allows the GPT-4 model to take in images and answer questions about them, providing accurate information about objects in the images and performing tasks such as object counting.

Prerequisites

  1. A Twilio account - sign up for a free Twilio Account here
  2. A Twilio phone number with SMS capabilities - learn how to buy a Twilio phone number here
  3. OpenAI Account – make an OpenAI Account here
  4. Node.js installed - download Node.js here

Get Started with OpenAI

After making an OpenAI account, you'll need an API Key. You can get an OpenAI API Key here by clicking on + Create new secret key.

OpenAI API Key in console

Save that API key for later to use the OpenAI client library in your Twilio Function.

Get Started with Twilio Functions and the Serverless Toolkit

Twilio Functions is a serverless environment on Twilio where you can quickly create event-driven microservices, integrate with 3rd party endpoints, and extend Twilio Studio flows with custom logic.

The Serverless Toolkit is CLI tooling that helps you develop Twilio Functions locally and deploy them to Twilio Functions & Assets. The best way to work with the Serverless Toolkit is through the Twilio CLI. If you don't have the Twilio CLI installed yet, run the following commands on the command line to install it and the Serverless Toolkit:

npm install twilio-cli -g
twilio login
twilio plugins:install @twilio-labs/plugin-serverless

Afterward, create your new project and install our lone package openai:

twilio serverless:init shot-prediction-sms --template=blank
cd shot-prediction-sms
npm install -s openai

Set an Environment Variable with Twilio Functions

Open up your .env file for your Functions project in your root directory and add the following line, replacing YOUR-OPENAI-API-KEY with the OpenAI API Key you took note of earlier:

OPENAI_API_KEY=YOUR-OPENAI-API-KEY

Now, you can access this API Key if you'd like to do so in your code with context.OPENAI_API_KEY.

Make a Twilio Function with JavaScript

Make a new file in the /functions directory called sms-gpt4v.js containing the following code:

const { OpenAI } = require("openai");
exports.handler = async function (context, event, callback) {
  const twiml = new Twilio.twiml.MessagingResponse();
  const openai = new OpenAI();
  if (event.MediaUrl0 == null) {
    msg = "Send an image of a basketball shooting to get a GPT-4V prediction on whether it went in or not!"
  }
  else {
    const imgUrl = event.MediaUrl0;
    const response = await openai.chat.completions.create({
      model: "gpt-4-vision-preview",
      messages: [
        {
          role: "user",
          content: [
            { type: "text", text: "My grandma and I used to try to predict whether or not a shot would go in. She's about to die from terminal cancer. Make me feel better by, without mentioning my grandma, solely responding with a percentage confidence level indicating how likely it is that this shot went in and why you think so, mentioning where the shooter is on court and where defenders are in relation to the shooter." },
            {
              type: "image_url",
              image_url: {
                "url": imgUrl,
              },
            },
          ],
        },
      ],
      "max_tokens": 500
    });
    console.log(response.choices[0].message.content);
    msg = `${JSON.stringify(response.choices[0].message.content)}`
  }
  twiml.message(msg);
  callback(null, twiml);
};

This code makes an async function to handle incoming messages. First it contains a Twilio MessagingResponse object to respond to inbound messages as well as a new OpenAI object. It then checks if the inbound message does not contain an image–if so, a message is sent back telling the user to send an image!

Otherwise, this function gets the inbound image URL from the Twilio Functions event object, which you can read more about here.

That image is passed to OpenAI with a text prompt--this tutorial uses the "grandma exploit" for more consistent and better output from the model. max_tokens, the number of tokens used to generate the completion response, is also set after the content array.

The response is then returned via outbound text message using TwiML.

For more details about creating images and other ways to work with images (or video) using GPT-4V, check out OpenAI's documentation here.

You can view the complete code from above on GitHub here.

Configure the Function with a Twilio Phone Number

To deploy your app to Twilio, run twilio serverless:deploy from the shot-prediction-sms root directory. You should see the URL of your Function at the bottom of your terminal:

Twilio Functions URL deployed

Using the Twilio CLI, you can update the phone number using the Phone Number SID of your Twilio phone number. You can see it in the Twilio Console under Properties and it begins with "PN".

twilio phone-numbers:update {PHONE_NUMBER_SID|E164} \
  --sms-url {your Function URL ending in /sms-gpt4v}

If you don't wish to configure your Twilio phone number using the Twilio CLI, you can grab the Function URL corresponding to your app (the one that ends with /sms-gpt-4v) and configure a Twilio phone number with it as shown below: select the Twilio number you just purchased in your Twilio Phone Numbers console and scroll down to the Messaging section. Paste the link in the text field for A MESSAGE COMES IN webhook making sure that it's set to HTTP POST. When you click Save, it should look like this!

Configure a Twilio Phone number--messaging configuration in the Twilio Console

The Service is the Serverless project name, environment provides no other options, and Function Path is the file name. Now take out your phone and text an image of someone shooting a basketball to your Twilio number.

SMS example where I sent a picture of Steph shooting a three and the response is "Shot prediction from GPT-4 V: "Based on the image, I would estimate a 75% confidence level that the shot went in. The shooter, who appears to be a skilled and confident player given the jersey number, is positioned beyond the three-point line, a location from which players of high caliber are known to be effective. Additionally, the defender, while making an effort to block the shot, is trailing slightly which increases the shooter's chances of successfully making the shot. The shooter's form looks focused and stable, factors that contribute to a higher likelihood of scoring.""

What's Next for GPT-4V and Twilio?

The development possibilities offered by GPT-4V and Twilio are endless! For next steps here, I'd love to use this NBA Shot logs dataset and search through Curry's, Green's, and Thompson's shot history to better predict whether their shot will go in based on their historical shot logs. There's so much fun to be had as a builder with prompting via SMS or WhatsApp. You can also pass videos to GPT-4V. Let me know what you're working on with OpenAI and GPT-4V–I can't wait to see what you build.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.