Detect Hair Color with AutoML, MMS, and JavaScript

Time to read: 4 minutes

February 24, 2021
Written by
Lizzie Siegle
Twilion
Reviewed by
Diane Phan
Twilion

newhairheader.png

A few weeks ago, I hosted Anna Lytical on my weekly Twilio TV Twitch show to talk making tech more inclusive and accessible, their path to Google and drag, this cool Google Cloud and Twilio hack we made, and more! Anna Lytical is the sickeningly entertaining and educating coding drag queen dedicated to engaging a more diverse audience with code and technology, especially the LGBTQ+ community. During working hours, they are an engineer at Google working on the developer experience for Google Cloud Platform.

The app we built for the stream was a hair color detector using Google AutoML, Cloud Functions, Twilio MMS, and JavaScript. Read on to learn how to build it yourself and see it in-action by texting a selfie or image with hair to +18182757423!

emily-hair

What you need to get started

To follow along with this post, you will need

Prepare image data to train the model on

The next two sections on preparing data and training the model can be summed up by Anna Lytical in their TikTok video.

It can be a pain to get all of the training images needed for a machine learning model. We could scour a search engine for images of hair, but let's get crafty as developers and builders and use this open source Instagram Scraper. You can search for tags like brownhair on Instagram here. Install the scraper with pip running pip install instagram-scraper.

In a project directory called hair make a directory called data where images corresponding to different hair colors will be saved. You could scrape images from a certain account, but we want a lot of images.

To search hashtags corresponding to hair colors, run the following commands to return 100 images tagged "brownhair", "redhair", and "blondehair" each:

instagram-scraper brownhair --tag --maximum 100
instagram-scraper redhair --tag --maximum 100
instagram-scraper blondehair --tag --maximum 100

 You can check that the images were scraped by running ls - sometimes I run the command a second or third time to get images.

Zip up the data into a folder. If you have a  Mac device, you can do this by right-clicking the folder in Finder and selecting "Compress."

compress data

With this data, we can now train our model.

Build and train the model with AutoML

In your Google Cloud console, select a project or make a new one. For this post, I called the project "AutoML".

Then, search "vision" in the console search bar as shown below.

search "vision" in Google Cloud

Select Get started under the AutoML Vision API.

AutoML Vision
 

Click New dataset, give it a name like "hair". Make sure the model objective is set to single-label classification, and click Create Dataset.

Create single-label classification dataset

You'll be uploading images from your computer. Click on Select files and upload the zipped file (mine is called data.zip.) To save the data to cloud storage, you must make a folder to save it to. Click "Browse" next to "Destination on Cloud Storage" followed by "Create new bucket".

Call the bucket "hair-images" and choose some data storage options according to cost, performance, and availability. I chose the following:

  • multi-region
  • standard
  • fine-grained

Click Create once you have selected them all.

select files to import

Wow! That was a lot. Click the Continue button to upload the data to Google Cloud. This could take quite a bit of time. Go for a skip through the park, make some tea, or watch a few music videos.

lizzie-ricking-skipping

Once all the images are uploaded, you can view them and how they are tagged under the "Images" tab.

tagged images in gcloud console

Click Train next to Images. Then click the blue Start Training button.

train images

Give your model a name like hair-color-detector and make sure "Cloud hosted" is selected. Click Continue, set your node hour budget to 8, and then select Start training when you're done with all the steps.

train new model
 

Training may take a few hours to complete and you will get an email once your model is done training. Time to go for another walk or read a book!

You can test your model in the console by uploading an image under the "Test & Use" tab:

test and use

Write your Google Cloud Function

Cloud Functions allow you to make and deploy a function in Node.js, Python, Go, Java, .NET, or Ruby using just your browser with the Google Cloud Console.

The article will be covering Node.js from hereon. Go to your Cloud console and select Cloud Functions on the side. Make sure you are under the same project in which you built your AutoML model.

 

Select Cloud Functions on the side

Select Create Function at the top of the page.

Create Function

Give the function a name like "hair" and make sure the region is set to "US-central1". Select "Allow unauthenticated invocations" and click Save.

name the Function

After clicking Next, paste the following code in index.js of your function:

const automl = require('@google-cloud/automl');
const predictionClient = new automl.PredictionServiceClient();

const twilio = require('twilio');
const imageToBase64 = require('image-to-base64');

const MessagingResponse = twilio.twiml.MessagingResponse;

const projectId = "REPLACE-WITH-YOUR-PROJECT-ID"; 
const region = 'us-central1';
const modelId = "REPLACE-WITH-YOUR-MODEL-ID" 

/**
 * Responds to any HTTP request.
 *
 * @param {!express:Request} req HTTP request context.
 * @param {!express:Response} res HTTP response context.
 */
exports.reply = async (req, res) => {
  console.log("in reply!");
  console.log("req.body", req.body);
  const twilioResponse = new MessagingResponse();

  // Add text to the twilioResponse
  let url = req.body.MediaUrl0;
  let img = await imageToBase64(url);
  console.log(img);

  const payload = {
    "image": {
      "imageBytes": img
    }
  };
  const reqBody = {
    name: predictionClient.modelPath(projectId, region, modelId),
    payload: payload
  };

  return predictionClient.predict(reqBody)
      .then(responses => {
        console.log('Got a prediction from AutoML API!',
            JSON.stringify(responses));
        let color = responses[0]["payload"][0]["displayName"].replace('hair','');
        console.log(color);
        twilioResponse.message(`I love your ${color} hair`);
        res
            .status(200)
            .type('text/xml')
            .send(twilioResponse.toString());
      })
      .catch(err => {
        console.log('AutoML API Error: ', err);
        twilioResponse.message(`Your hair looks fab, but I couldn't detect the color.`);

        res
            .status(200)
            .type('text/xml')
            .send(twilioResponse.toString());
      });
};

You can find your Project ID by clicking on your project name at the top as shown below:

project ID gif

To get your model ID, go back to the Vision API (search "Vision" in the search bar near the top).  Navigate to Datasets, and copy the string beneath the dataset that begins with "IC."

model ID

The package.json should include the following:

{
  "name": "code",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@google-cloud/automl": "^2.3.1",
    "image-to-base64": "^2.1.1",
    "twilio": "^3.53.0"
  }
}

Click Deploy near the bottom of the dashboard.

Hook up your Cloud Function to a Twilio phone number

Grab your Cloud Function URL by clicking on the three dots next to your deployed function and then click Copy the URL.

Select a purchased Twilio phone number in your Twilio phone number console and scroll down to the Messaging section. Paste the link in the text field for A MESSAGE COMES IN webhook. Make sure that this is set to HTTP POST and text your number an image containing hair!

blond classified as brown hair in an AI-generated image

This image above is AI-generated on thispersondoesnotexist.com using a GAN (generative adversarial network). The model sometimes struggles with blond hair because some of the images tagged #blondehair were people who dyed their blonde hair brown or black, so to improve our model, we'd need to add more images of blond hair.

What's next for detecting hair color

AutoML simplifies building, training, and deploying a machine learning model. You can add other color categories to recognize other hair colors, use the Vision API to recognize other types of images as well, classify curly or straight hair on top of color, use AutoML Natural Language to train a model for language, and more.

Let me know online what you're building and check out my weekly Twitch stream where I host guests like Anna Lytical (check out their educational tech content on TikTok, YouTube, and Twitch), build cool hacks, and more.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.