Pick the Right Meal Using C#, ASP.NET and Nutritionix

October 15, 2014
Written by

Doritos or Granola Bar

Proper nutrition is a topic that is near and dear to my heart. I struggled at one point in my life with making the right food choices and over time the weight piled on. After learning a lot about nutrition, I feel like I have a good handle on making choices in the grocery store. This isn’t always easy though and it is really difficult when traveling.

As an evangelist I travel a lot, so often I come home to an empty fridge. Sometimes I know that I will be leaving in a few days for another trip so I don’t want to do a full fridge restock. A couple quick and easy meals is all I’m after. There be dragons! Convenience food is some of the worst stuff on the planet for you but there are healthier options even in this realm of on-the-go eating. With a couple of quick photos of barcodes and an MMS message, you can get the nutrition decision you need without having to read labels and compare calories, protein, carb and fat totals.

Hot Pockets or Amy's?

Try it yourself! Grab some food items and snap some clear, up close pictures of their barcodes and send them to:

United States: (267) 433-2613

Canada: (450) 954-1629

Recipe (get it?!)

How It Works

Workflow Part 1

Workflow Part 2

Our user will have a few ways to work with our barcode to nutrition info service. The first option is sending in a single barcode. In this case, we will return nutrition information for the single product. Another option is to send in multiple barcodes with an optional keyword to indicate how to process the information. If no keyword is specified we will total up the nutrition information for all of the products send in. This will help the user tell the nutrition details for a meal composed of these items. If the user sends in the ‘compare’ keyword, we will send back a “winner” based on the calories, protein, carbs and fat totals for the given items. This will help the user choose a particular item based on their nutrition goals.

There is a chance that some barcodes will not be recognizable. There is also a chance that some products will not be available in the Nutritionix database. I chose the Zxing.NET barcode reader because it was free, easy to work with and worked on most barcodes I sent to it. I chose Nutritionix because they have a well-populated database of items with the ability to query via barcode. There are many other nutrition databases available which have more items in them. Feel free to explore the options out there.

The full project is available if you want to follow along: Github

Setting Up the Project

Start by creating a new ASP.NET Web Application project from the Empty template with MVC references:

ASP.NET Project

Add a new controller to the project called BarcodeNutrition. Delete the default Index() method since we won’t need it in our application.

Now we need to install some NuGet packages for the libraries we will be using. First we’ll install the Twilio.Mvc package (commands shown for Package Manager Console):

Install-Package Twilio.Mvc

Next, we’ll install the ZXing.NET package:

Install-Package ZXing.Net

Finally, install the Nutritionix package:

Install-Package Nutritionix

Add the following using statements to the top of BarcodeNutritionController.cs:

using System.Drawing;
using System.Net.Http;
using System.Text;

using Twilio;
using Twilio.TwiML;
using Twilio.TwiML.Mvc;
using Nutritionix;
using ZXing;

We also need to add an action method to handle incoming messages from Twilio. This will be the entry point to our service:

public async Task<ActionResult> Inbound(int numMedia, string body)
{
    var response = new TwilioResponse();

    // Trim incoming text to remove whitespace
    body = body.Trim();

    // Code to generate nutrition info result...
    return new TwiMLResult(response);	
}

All the pieces are in place for us to begin writing our barcode to nutrition info service. There are a lot of moving parts in this hack so you’re probably going to want to be able to test things as you build them. When Twilio receives an incoming message at your phone number it will need to make an HTTP request to our Inbound action method. If you want to test locally you will need to expose your localhost to the outside world. I recommend using ngrok for this. You can use this tutorial to help get you set up with ngrok on your Windows machine. Once you have that up and running, configure the Messaging URL for your Twilio number in the numbers portal to point at your local server’s /BarcodeNutrition/Inbound endpoint:

Number config

With this set up you will be able to send barcodes to your Twilio number during the development process so I’d encourage you to set breakpoints and write tests as you follow along.

Reading Barcodes From the Incoming Images

We first check to make sure the user sent in at least one image. If they didn’t we need to let them know to try again with some barcodes:

// No images sent in
if (numMedia == 0)
{
   response.Message("You didn't send any barcodes! Please send in well-focused and zoomed in barcodes with the word 'total' to get total nutrition values or 'compare' to get a comparison of the items.");
   return new TwiMLResult(response);
}

This will generate TwiML to be returned to Twilio. TwiML is XML that tells Twilio how to handle incoming messages and calls. In this case we are returning TwiML that tells Twilio to send an SMS message back to the user with the text specified. Here is the TwiML generated by the code above:

<Response>
   <Message>You didn't send any barcodes! Please send in well-focused and zoomed in barcodes with the word 'total' to get total nutrition values or 'compare' to get a comparison of the items.</Message>
</Response>

Read this document to learn more about TwiML.

With that out of the way we can get down to the business of decoding the barcode images. Add this code to the Inbound method:

// Decode the barcodes
string[] eanCodes = await DecodeBarcodes(numMedia);

if (eanCodes == null)
{
   // There was an error with one of the barcodes. Bail so user can try again.
   response.Message("One of your barcodes was not recognized. Please try cropping your image or taking the picture again closer to the barcode.");
   return new TwiMLResult(response);
}

This code relies on a DecodeBarcodes method that we will add now:

private async Task<string[]> DecodeBarcodes(int numberOfImages)
{
   // Build a List<Bitmap> from incoming images
   var images = new List<Bitmap>(numberOfImages);

   // Build an array of EAN codes from reading the barcodes
   var eanCodes = new string[numberOfImages];

   var httpClient = new HttpClient();
   var reader = new BarcodeReader();

   for (int i = 0; i < numberOfImages; i++)
   {
       Bitmap bitmap;

       using (var stream = await httpClient.GetStreamAsync(Request["MediaUrl" + i]))
       {
           using (bitmap = (Bitmap)Bitmap.FromStream(stream))
           {
               var result = reader.Decode(bitmap);

               if (result == null)
               {
                   // Couldn't read this barcode, we'll return null to indicate we should bail...
                   return null;
               }

               eanCodes[i] = result.Text;
           }
       }
   }

   return eanCodes;
}

For each barcode image that is sent in there will be a corresponding media URL. We use this URL to populate a Bitmap object which is required by the ZXing.NET barcode reader. If the barcode is able to be decoded we add the result string to an array of EAN codes. If any of the barcodes are unreadable we return null from the method. A failed barcode reading means we can’t perform a total or a comparison so we want to give the user a chance to try again.

At this point we have a list of EAN codes that can be used to query the Nutritionix database for nutritional info.

Before moving on, try these barcodes to make sure the reader is working properly:

Hot Pockets

Cheerios

Mmm, Hot Pockets and Cheerios…that’s a meal fit for a king if I’ve ever seen one.

Fetching Nutrition Info From Nutritionix

The Nutritionix API is queryable using the barcode numbers available on our food and drink items. The NutritionixClient object in the NuGet package we added earlier makes it really easy to work with this information. Add the following class level variables to the top of the BarcodeNutritionController making sure to replace the placeholders with your credentials:

private string _nutritionixId = "[Your Nutritionix Application ID]";
private string _nutritionixKey = "[Your Nutritionix Application Key]";

Next we’ll set up a list for food items we successfully query and another for barcodes that are not found in the Nutritionix database:

List<Item> foodItems = new List<Item>();
List<string> skippedBarcodes = new List<string>();

Now we’ll add the method that will populate these lists by querying the Nutritionix API:

private void LookupNutritionInfo(string[] eanCodes, ref List<Item> foodItems, ref List<string> skippedBarcodes)
{
   // Initialize the NutritionixClient
   var nutritionix = new NutritionixClient();
   nutritionix.Initialize(_nutritionixId, _nutritionixKey);

   // Create lists for food items and any barcodes that aren't in the database
   foodItems = new List<Item>(eanCodes.Length);
   skippedBarcodes = new List<string>();

   // Loop through each barcode
   foreach (var barcode in eanCodes)
   {
       Item food = null;

       try
       {
           // Fetch nutrition info by EAN code
           food = nutritionix.RetrieveItemByUPC(barcode);
       }
       catch
       {
           // Invalid barcode format results in a 404. We'll add it to the skipped barcodes list.
           skippedBarcodes.Add(barcode);
           return;
       }

       if (food != null)
       {
           foodItems.Add(food);
       }
       else
       {
           // One of the food items is not available in Nutritionix.
           skippedBarcodes.Add(barcode);
       }
   }
}

 

Note the try/catch block in the middle of the method. This will handle scenarios where the barcode format is not recognized since NutritionixClient will throw an HttpException for the resulting 404 error in this case.

Now we can call the nutrition lookup from our action method passing in the two lists as reference parameters:

LookupNutritionInfo(eanCodes, ref foodItems, ref skippedBarcodes);

If we had to skip any barcodes because they weren’t in the Nutritionix database, we’ll let the user know to try again without these barcodes:

if (skippedBarcodes.Count > 0)
{
   // Let's tell the users that we couldn't find their item(s)...
   var builder = new StringBuilder();

   builder.Append("Sorry but we couldn't find one or more of your items. Please try again without the following EANs which were not found in the Nutritionix database: ");

   foreach (var barcode in skippedBarcodes)
   {
       builder.Append(barcode + " ");
   }

   response.Message(builder.ToString());
   return new TwiMLResult(response);
}

At this point we have all of the food items populated and can process the results based on what the user requested. In the case where we only have one barcode, we’ll return nutrition details for just that item:

private string GetSingleItemNutrition(Item foodItem)
{
   return String.Format(
           "Here are the details for {0} {1}: {2} calories, {3}g protein, {4}g total carbohydrates, {5}g total fat.",
           foodItem.BrandName,
           foodItem.Name,
           foodItem.NutritionFact_Calories,
           foodItem.NutritionFact_Protein,
           foodItem.NutritionFact_TotalCarbohydrate,
           foodItem.NutritionFact_TotalFat
   );
}

If the user wanted the server to total up the nutrition info either by specifying the ‘total’ keyword or by not specifying a keyword we’ll use the following method:

private static string GetTotalNutrition(List<Item> foodItems, string[] eanCodes)
{
   // Default to returning total nutrition info
   var totalCalories = foodItems.Sum((item) => item.NutritionFact_Calories).Value;
   var totalProtein = foodItems.Sum((item) => item.NutritionFact_Protein).Value;
   var totalCarbs = foodItems.Sum((item) => item.NutritionFact_TotalCarbohydrate).Value;
   var totalFat = foodItems.Sum((item) => item.NutritionFact_TotalFat).Value;

   return string.Format("Here are the totals for the items you requested: {0} calories, {1}g protein, {2}g carbohydrates and {3}g total fat.", totalCalories, totalProtein, totalCarbs, totalFat);
}

The last option for processing the results is the ‘compare’ function:

private string CompareNutrition(List<Item> foodItems, string[] eanCodes)
{
   var lowestCalories = foodItems.Aggregate(
                               (item1, item2) => item1.NutritionFact_Calories < item2.NutritionFact_Calories ? item1 : item2
                           );

   var highestProtein = foodItems.Aggregate(
                               (item1, item2) => item1.NutritionFact_Protein > item2.NutritionFact_Protein ? item1 : item2
                           );

   var lowestCarbs = foodItems.Aggregate(
                               (item1, item2) => item1.NutritionFact_TotalCarbohydrate < item2.NutritionFact_TotalCarbohydrate ? item1 : item2
                           );

   var lowestFat = foodItems.Aggregate(
                               (item1, item2) => item1.NutritionFact_TotalFat < item2.NutritionFact_TotalFat ? item1 : item2
                           );

   return string.Format(
       "Lowest calories: {0} {1} (barcode: {2}) with {3} calories. Highest protein: {4} {5} (barcode: {6}) with {7}g of protein. Lowest total carbs: {8} {9} (barcode: {10}) with {11}g carbs. Lowest total fat: {12} {13} (barcode: {14}) with {15}g fat.",
       lowestCalories.BrandName,
       lowestCalories.Name,
       eanCodes[foodItems.IndexOf(lowestCalories)],
       lowestCalories.NutritionFact_Calories,
       highestProtein.BrandName,
       highestProtein.Name,
       eanCodes[foodItems.IndexOf(highestProtein)],
       highestProtein.NutritionFact_Protein,
       lowestCarbs.BrandName,
       lowestCarbs.Name,
       eanCodes[foodItems.IndexOf(lowestCarbs)],
       lowestCarbs.NutritionFact_TotalCarbohydrate,
       lowestFat.BrandName,
       lowestFat.Name,
       eanCodes[foodItems.IndexOf(lowestFat)],
       lowestFat.NutritionFact_TotalFat
   );
}

This method tries to help the user pick the best choice depending on some common nutrition goals – low calorie, high protein, low carb and low fat. It’s not a perfect way to pick the right food but it’ll do when you’re stuck at a gas station trying to decide between cheese puffs and a granola bar.

Now we just need to add a method to decide which calculation to use:

private string GetNutritionInfoResponse(int numMedia, string keyword, List<Item> foodItems, string[] eanCodes)
{
   string responseString = "";

   // Depending on number of items and the keyword in the Body, run some nutrition calculations
   if (numMedia == 1)
   {
       // Single item, just return details for that item.
       responseString = GetSingleItemNutrition(foodItems[0]);
   }
   else if (keyword == String.Empty)
   {
       // Default to totals
       responseString = GetTotalNutrition(foodItems, eanCodes);
   }
   else if (String.Equals(keyword, "total", StringComparison.OrdinalIgnoreCase))
   {
       // User explicitly requested total nutrition info
       responseString = GetTotalNutrition(foodItems, eanCodes);
   }
   else if (String.Equals(keyword, "compare", StringComparison.OrdinalIgnoreCase))
   {
       // User requested item comparison
       responseString = CompareNutrition(foodItems, eanCodes);
   }
   else
   {
       // Invalid keyword
       responseString = String.Format("You sent in '{0}' which is not a valid keyword. Please send in well-focused and zoomed in barcodes with the word 'total' to get total nutrition values or 'compare' to get a comparison of the items.", keyword);
   }

   return responseString;
}

We’ll call this from our Inbound method and return the resulting response string as TwiML to Twilio. The user will get a text message with their requested nutrition information. Armed with this info, hopefully they will make the right choice in the grocery aisle.

Now that the app is completely built, give it a try by sending a couple of barcodes to your new Twilio MMS enabled phone number. Make sure to try out the “compare” feature!

Next Steps

I hope you find this tool as helpful as I did when I wrote it. It definitely served me well this week as I came back from Xamarin Evolve only to have to head off to Connect.js in a few days. With a quick couple of picture messages I was able to grab a few healthy meals for the week!

You don’t have to stop with what I built here though. Here’s some next steps:

  • Render a nutrition label and return it via MMS
  • Send the nutrition data to MyFitnessPal, Fitbit, etc.
  • Make better choices and live a healthier life!

I hope you’re excited to build lots of cool things with MMS using C#. I’m stoked to see what you create so please share them with me on Twitter @brentschooley or through email at brent@twilio.com.