Python Image Recognition with ImageAI and Twilio MMS

January 10, 2020
Written by

image-recog-imageai

Can a computer identify what's in an image? Development libraries like ImageAI make normally-complex Machine Learning tasks including object detection easier. This blog post will show how to build an image classification application using Python, Flask, and ImageAI. This Python image recognition application will receive inbound images with Twilio MMS and respond with a modified image segmented into detected objects and the model's detection confidence percentages.

MMS identifying person and dog

What is Python Image Recognition?

Python image recognition is the process through which artificial intelligence and machine learning technologies detect objects within an image and create corresponding categorizations for the images based on those objects. Image recognition software generally works by analyzing the individual pixels in an image and looking for patterns to help determine what objects are present. This deep learning allows the software to recognize objects over and over again in new images. 

Building an app that utilizes this type of image classification, like the one we’ll build in this tutorial using Twilio MMS, Python, Flask and ImageAI, can be relatively straightforward. Let’s dive into everything you need to know before we get started.

Getting Started With Python Image Recognition

Prerequisites

Setup

Activate a virtual environment in Python 3 and on the command line, run pip install Flask twilio and pip install -U tensorflow keras opencv-python imageai.

Your Flask app will need to be visible from the web so Twilio can send requests to it. Ngrok lets us do this. With it installed, run the following command in your terminal in the directory your code is in. Run ngrok http 5000 in a new terminal tab.

Ngrok with arrow

You should see the screen above. Grab that ngrok URL to configure your Twilio number.

Messaging configuration

Build the Object-Detecting Flask App

Make a directory called imageai and in it, create a file called object_detection.py. At the top of the file be sure to import the following necessary libraries, create a variable with the path to our Python file, and make a Flask app and route to handle inbound MMS/SMS.

from imageai.Detection import ObjectDetection
import os
from twilio.twiml.messaging_response import MessagingResponse
from flask import Flask, request, redirect, send_from_directory
import requests

curr_dir = os.getcwd()
app = Flask(__name__)
@app.route("/sms", methods=['GET', 'POST'])

Right beneath that we'll create a MessagingResponse object for our outbound Twilio message. This will contain the objects the model detected and how confident the model is about them, as well as the modified image. We'll also make a new instance of the ObjectDetection class, set the model type to RetinaNet (a high-density detector which takes longer than other models but is more accurate) and the path to where we downloaded it, and finally load the model. ImageAI supports other models like YOLOv3 and TinyYOLOv3 (You Only Look Once) which offer moderate accuracy but faster detection speeds. There are other RetinaNet implementations for Keras, PyTorch, and more for most of your development needs!

def sms():
    resp = MessagingResponse()
    detector = ObjectDetection()
    detector.setModelTypeAsTinyYOLOv3()
    detector.setModelPath(os.path.join(
        curr_dir, "yolo-tiny.h5"))
    detector.loadModel()

Check if an inbound message is an image with Twilio MMS

This is the complete code for the sms() function.

@app.route("/sms", methods=['GET', 'POST'])
def sms():
    resp = MessagingResponse()
    
    detector = ObjectDetection()
    detector.setModelTypeAsRetinaNet()
    detector.setModelPath(os.path.join(
        curr_dir, "resnet50_coco_best_v2.0.1.h5"))
    detector.loadModel()
    if request.values['NumMedia'] != '0':
        filename = request.values['MessageSid'] + '.jpg'
        resp_str = ''
        with open(filename, 'wb') as f:
            image_url = request.values['MediaUrl0']
            f.write(requests.get(image_url).content)
            detections = detector.detectObjectsFromImage(input_image=filename, output_image_path= filename)
            for each_object in detections:
                perc = each_object["percentage_probability"]
                resp_str += (str(each_object["name"] + " : ") + str(perc) + "%\n")
                print(each_object["name"], " : ", each_object["percentage_probability"])
        msg = resp.message(resp_str)
        msg.media('https://REPLACE-WITH-YOUR-NGROK-URL.ngrok.io/output/{}'.format(filename))
    else:
        resp.message("Try sending a picture message.")
    return str(resp)

Now we check that the 'NumMedia' property isn't zero is how to check that an inbound message is an image. Next we make a string called filename with the MessageSid to distinguish between each inbound image. Then we open the file, write the content of Twilio’s image to it, and pass the image to detector.detectObjectsFromImage to perform object detection on the inbound image, save the output image under the same filename, and returns an array of dictionaries, each of which corresponds to how many objects are detected.

if request.values['NumMedia'] != '0':
        filename = request.values['MessageSid'] + '.jpg'
        sendStr = ''
        with open(filename, 'wb') as f:
            image_url = request.values['MediaUrl0']
            f.write(requests.get(image_url).content)
            detections = detector.detectObjectsFromImage(input_image=filename, output_image_path= filename)

Looping through the array of dictionaries, we find, print out, and create a response string with the objects detected and the percentage probability of the detections.

for eachObject in detections:
    perc = eachObject["percentage_probability"]
    respStr += (str(eachObject["name"] + " : ") + "%.2f" % str(perc) + "%"  + "\n")
    print(eachObject["name"], " : ", eachObject["percentage_probability"])

Finally we return the string and the new image, and handle what happens if the inbound message was not an image.

        msg = resp.message(respStr)
        msg.media('https://REPLACE-WITH-YOUR-NGROK-URL.ngrok.io/output/{}'.format(filename))
    else:
        resp.message("Try sending a picture message.")
    return str(resp)

This is the complete code for the sms() function.

@app.route("/sms", methods=['GET', 'POST'])
def sms():
    resp = MessagingResponse()
    
    detector = ObjectDetection()
    detector.setModelTypeAsRetinaNet()
    detector.setModelPath(os.path.join(
        curr_dir, "resnet50_coco_best_v2.0.1.h5"))
    detector.loadModel()
    if request.values['NumMedia'] != '0':
        filename = request.values['MessageSid'] + '.jpg'
        resp_str = ''
        with open(filename, 'wb') as f:
            image_url = request.values['MediaUrl0']
            f.write(requests.get(image_url).content)
            detections = detector.detectObjectsFromImage(input_image=filename, output_image_path= filename)
            for each_object in detections:
                perc = each_bject["percentage_probability"]
                resp_str += (str(each_bject["name"] + " : ") + str(perc) + "%\n")
                print(each_object["name"], " : ", each_object["percentage_probability"])
        msg = resp.message(resp_str)
        msg.media('https://REPLACE-WITH-YOUR-NGROK-URL.ngrok.io/output/{}'.format(filename))
    else:
        resp.message("Try sending a picture message.")
    return str(resp)

We still need to make that route. Twilio returns a URL to the image, not the actual image itself.

MMS with Bear, Tomomi

Send a Local Image with Twilio

We need a second route called output to help deliver the image we saved containing the objects detected by the model. This URL retrieves that new image and then we finally run our Flask app.

@app.route('/output/<filename>', methods=['GET', 'POST'])
def uploaded_file(filename):
    return send_from_directory(curr_dir, filename)

On the command line open a new tab separate from the one running ngrok. In the folder housing your code, run

export FLASK_APP=objectdetection

export FLASK_ENV=development

flask run  --without-threads

and text your Twilio number an image. You should see something like this:

MMS identifying pizza

Congratulations! Your Python image recognition application is up and running.

What's Next

You could use a custom model or other ones like YOLO and Tiny-YOLO which are faster but less accurate, only taking one forward pass through the network and using fully connected layers instead of convoluted ones. This means they're better-suited for real-time object detection. Some different use cases of object detection include autonomous driving, recognizing characters, facial detection, activity recognition, digital watermarking, and more. Let me know in the comments or online what you're building.