Gifted: Artificial Intelligence to Hack Gift Giving

Time to read: 3 minutes

December 20, 2016
Written by
Megan Speir
Twilion

AOJ_h3Q2ClGuT5_iDbswNoKyCPiekP3Hr6I55VfXnRXy67kWFoZRtO_sGw_DsauB_uQDgXwJPk-KULHttypdhW_Zi0zuygOLf2sfe8B-0BP_K4xKez_af5xLEpapycbbjQ2OQ9A2

Artificial Intelligence (AI) is coming for us all. Michael Jordan, a mechanical engineer by trade, came to this realization while working as the director of energy and sustainability at a New York City based firm serving large real estate clients.

While busy building energy models, Michael thought that much of his work in applied engineering was actually an optimization problem. Eventually, AI will solve most of these problems through automation.

Sometimes it’s best to face the inevitable and dive head first into the unknown. Embracing the future, Michael decided to do just this by studying more about machine learning and AI.

His tutelage ran the gamut from text-based learning with scikit–learn to genetic algorithms. He was excited by the idea that you could take a product specification and learn the math behind it. This exploration led him to image analysis with the Google Vision API and finally to Clarifai.

The Epic Christmas List

For years, Michael kept what he called, “The Epic Christmas List” mostly to document his heart’s greatest desires. But Michael found he was often asked by others what to gift during the season of giving and he would share the list with any such inquirer.

But Michael would also spend a lot of time following up to help give the gift recommendation a personal touch based upon a photograph or more detailed description of the person. Automating this recommendation method became a top priority because it was kind of a time suck. Certainly his new friend AI could help with that.

Michael applied machine learning to the epic christmas list. Given a corpus (of image data), he aimed to determine the relevance of each gift based upon its tf-idf (term frequency-inverse document frequency) or how well it related to the corpus data based upon tagged attributes (i.e. pretty, fancy, etc.) as defined in Clarifai.

“The tags were awesome, as they were pretty qualitative” he said. “I agreed with them!”

Given an image of a person, using artificial intelligence powered by algorithms and Clarifai, Gifted will suggest a gift for the individual based upon their surroundings or “interests” in a given photograph.

Here’s a sample of the Clarifai code:

# Authenticate with Clarifai 
CLARIFAI_APP_ID = 'CLARIFAI_APP_ID' #Go to https://www.clarifai.com/ and hit "Get your free API key"
CLARIFAI_APP_SECRET = 'CLARIFAI_APP_SECRET' ##Go to https://www.clarifai.com/ and hit "Get your free API key"

# Load the image from Twilio
pic_url = event['image']
twilio_pic = urllib2.Request(pic_url, headers={'User-Agent': "Magic Browser"})
image = urllib2.urlopen(twilio_pic)

#Load the Clarifai API 
api = ClarifaiApi()
# Process the photo with Clarifai
results = api.tag_image_urls(pic_url)['results']
# Process and store the tag confidences (if you need them)
confidence = api.tag_image_urls(pic)['results'][0]['result']['tag']['probs'] 

#Create the photo tag list
tags = []
for result in results:
        tags = tags   result['result']['tag']['classes']
        
#Remove the photo's stop words. These are words that don't provide useful information for your recommendation
stop_words = ['INSERT THE PHOTO TAGS THAT MAKE YOUR RECOMMENDATIONS WORSE']
#Example of stop words I found unhelpful:
stop_words = ['casual','landscape', 'indoors', 'sit', 'wear','horizontal', 'grass','portrait', 'people', 'facial expression', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten', 'monochrome', 'human', 'administration', 'isolated', 'solo']
#Remove the stop words
tag_set = set(tags) - set(stop_words) 

# You now have the tags and confidence for the photo from twilio. You can get creative and decide how you feel to these best match your recommendation database.

Gifted Giving

With the artificial intelligence piece of the product puzzle working, Michael needed to make it usable. “I have zero front-end skills,” he says. As he shopped the idea around, a friend suggested Michael do it through text messaging.

“It’s something everyone knows how to do and is comfortable with,” Michael says of sending messages.

A quick search led him to this blog post, using Amazon Lambda and their API Gateway to send MMS without servers.

The architecture looks something like this:

 


Thanks to that post, Michael was able to throw away a lot of his Python code in one weekend and architect a sort of micro-platform using Amazon and Twilio to build a consumer product with a familiar front-end via messaging.

Unwrapped – How Does it Work?

After testing Gifted with friends, Michael says, “People were blown away by how good some of the suggestions were and by how bad or funny some of the others happened to be.”

The hardest part was manually attaching gifts to a tag in the database.

Sending an MMS through Gifted, creates a Twilio GET request with the image. This goes through the Amazon API Gateway and converts the request into JSON. Then an AWS Lambda function representing the app is where Clarifai receives the image from Twilio, tags it and compares the image to those already in the database. This is where the quality of the recommendation comes in – and returns an Amazon link to a related product from the Pandas dataframe built into the application.

Gifted was built with Python 2.7 as Lambda does not yet support Python 3. But Michael says, “That’s a nice thing about machine learning is that basically everything is in Python.”

Give a Gifted Gift

There’s still time to finish some last minute shopping or try out Gifted for another gift-giving occasion.

This Christmas, Michael sussed out with his own creation that he will be buying a lot of books. “I didn’t know Michael Lewis was writing a new book and my mom is like the all-time biggest Michael Lewis fan!”

To learn more about the project or try it out for yourself, visit http://www.givingiq.com.

If you’re up for building your own artificial intelligence product for “X” like Gifted, check out the following resources:

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.