Bargain Hunter: Catch That Amazon Deal with Twilio

Bargain Hunter: Catch That Amazon Deal with Twilio

I love a bargain! And who doesn’t? Every time I buy anything online I find myself looking out for vouchers, promo-codes, or even cheaper prices on other sites.

Nine times out of ten I end up buying what I want from Amazon for convenience and it’s normally cheaper anyway.

But buying from Amazon and not trying to get the best deals out there seems a bit silly to me. Amazon updates their prices many times a day, and being frugal as I am I want to make sure I buy whatever I want when it’s on its lowest price, even if it takes longer for me to get what I want.

To solve this problem I decided to write an application that accepts an Amazon product URL and then polls Amazon to get pricing information. If it finds the prices are lower than the last time it checked, the app then sends an SMS telling me the new price, and by how much it’s gone down. It also gives me a shortened link to the item so I can buy it there and then.

What we’ll need

To complete our bargain-hunter hack I have used Visual Studio 2013 and a few other bits.

• A Twilio number (sign up for a free Twilio account here)
• .NET 4.5 & ASP.NET MVC 5
• Entity Framework 6
• An Amazon account
• A bit.ly account

The first three items will most likely already be available in any .Net development environment, but you can get it from here if you don’t already have it. Additionally we will download a few Nuget packages as  as we progress.

If you don’t want to have to type all the code, feel free to download it from here and all you will need to do is add some environment variables to your system.

Getting your Amazon credentials

I’m pretty sure by now you already have an Amazon account but for this hack we will need access to their Product Advertising API to fetch product prices. So when you get to the website, make sure you click the link Sign Up Now, and complete the registration process as you will need this in order to make requests to the API.

The API requires a combination of three user credentials to make requests. We can get the first two by heading to the Amazon AWS Management Console and generating a couple of keys.

When the page is loaded, click the button Create New Key and click the link Show Access Key.

This will give you an Access Key ID and a Secret Access Key. Make a note of both as we will be using them in a bit.

Next we need get the third user credential, which is your associate tracking ID. You can get one by going to the Amazon Associates page and logging in with your Amazon account details. Once you’re logged in copy your Tracking ID,  which is displayed on the top left.

You’ve now got all the Amazon credentials we will need to complete this hack.

Getting your Bitly credentials

Bitly is the URL shortening service we will use when our bargain hunting application detects a price drop.

Head to Bitly’s website and sign in. You can use your Twitter or Facebook account to sign in or just create a new account if you don’t already have one.

When you’re logged in head to the Settings page and click on Advanced. Scroll to the bottom of the page and take a note of the Login and API Key.

Setting up the initial project

Let’s start by creating a new Solution in Visual Studio by clicking on New Project, and under the tab Visual C# > Web choosing ASP.NET Web Application. Under Name and Solution Name enter BargainHunter. You can save the project to wherever you usually keep your code, I’ve saved mine to the Desktop.

On the next screen, Visual Studio will prompt us to choose what kind of project we want to create. Select the Empty template and check the MVC checkbox to create an empty MVC project.

Once the project is created, right click the project name in the Solution Explorer and choose the option Manage Nuget Packages. This is where we’re going to install all of our dependencies.

Start by installing Twilio, the helper library that will make it easier for us to communicate with Twilio’s API.

Repeat the process for the following libraries:

• EntityFramework – is the ORM that will help us communicate with the database where our deals will be stored.
• Bitly.Net API Wrapper – Is a wrapper to the Bitly API which we will use to generate shortened URLs.
• Hangfire – Is an amazing set of open-source libraries that will help us create, process and manage our background jobs.
• AmazonItemLookup – Is a small library that makes it easier for you to communicate with Amazon Advertising API. This wrapper is not officially supported by Amazon.

Once you have installed them all click on the Updates tab and click the Update All button. This will guarantee we’re running the latest version on everything we’re using.

To bootstrap Hangfire, we will follow its documentation and create a new file on the root of our project called Startup.cs.

In that file add the following initialization code:

Notice we’ve referenced a SQL Server Storage we don’t currently have and will be creating on the next few steps. In the meantime we can add an entry for it in our web.config file that sits on the root of the project. Add the following connection string before the closing tag:

Creating the database

Let’s start by creating a database where all of our deals will be stored. This is also the data we will use to do price-comparison every time we fetch new prices from Amazon.

Right click the app_data folder, choose Add > New Item… > SQL Server Database.

Name this database BargainHunter.mdf and click Add.

Right click on the new database created under app_data and choose Open. This will switch you to the Server Explorer context where you can right click on Tables and choose the option Add New Table.

In the T-SQL tab enter the following code instead of what’s already there:

Click the Update button to create the table.

This table contains information about each one of our products as well as the shortened URL that will be sent to us when prices go below the existing ones.

Once that’s done switch back to Solution Explorer and right click the Models directory. Choose Add > New Item… and click ADO.NET Entity Data Model. Name it Deal and click Add.

The next screen will prompt you to choose your Model Contents. Choose EF Designer from Database and this will create us a model based on the table we have created. Click Next and you will see information about what we’re about to create.

Click Next Again and the wizard will prompt you to choose which database objects you want models created for. Check the Tables checkbox and click Finish.

You will then be taken to a screen containing a diagram of our new entity and if you look under models there will be a file called Deal.edmx. Expand this file and then expand Deal.tt further. You will see another file called Deal.cs, and it will have all the definitions for our entity.

The reason I’m pointing out to this file, is because it clearly states in its comments that any changes made on it could end up overwritten if we decided to regenerate our Deal class. So we need to make sure we steer away from this file and do any further changes we want elsewhere. But where?

Adding metadata to a generated entity

We want to make sure our model has server validation in the event of a malformed URL or in case we forget to add a deal name. To do that create a new class under Models called Metadata.cs. We can then use this class to add metadata to any of our entities as such:

We are adding some data annotations to our model properties as to change how the labels get displayed when we use it on our views. We are also validating the text entered and formatting our data so it gets displayed nicely when it comes back from the database.

But now we need to hook this class up to our model, otherwise this is only a simple class. To do that create yet another class under Models called PartialClasses.cs and replace it’s contents with the following:

The code snippet above takes care of associating our Deal class with the DealMetadata class which lives under the Metadata.cs file. To make our lives easier, we are implementing the ICloneable interface as it will allow us to clone our model later on.

Time for a quick test drive

Let’s create a controller so that we can run our application and see what it looks like.

Right click on the Controllers folder and click Add > Controller… > MVC 5 Controller – Empty > Add. Name it HomeController, and a brand new controller will be created for us.

For the next step we will need to have built the project, so go ahead and hit CTRL+Shift+B to build the project. Now expand the Views folder and right click on the Home folder and choose Add > View…

You will be prompted to name this view, we’ll call it Index. On the Template section choose List and on the Model Class select Deal (BargainHunter.Models). Choose BargainHunterEntities as the Data context class. Uncheck all the checkboxes and click Add.

Comment out all the action links for the time being, so your Index.cshtml file should look like the one below:

We’ve also made a few adjustments to the styles on our page by adding Bootstrap to it and an extra class to our table as it will make it easier to read when we have lots of content.

Let’s go back to the HomeController.cs file and change it to start querying the database that contains our entities.

Run the project, and you should see an empty table. This won’t be empty for too long though as we’re about to start working our way to add some deals to the database.

Let’s add some new deals

We have built all the classes around listing deals but so far haven’t added any deals to our database. We will change this now by creating two new endpoint on our HomeController.cs file. These will handle displaying the form we will use to add new deals and ultimately create it.

There are a couple of things highlighted on the code above which I think are worth going into a bit more detail. The first one is a custom model binder we will create to help us extracting data from our model before we add it to the database.

The second highlighted piece of code refers to a helper class we will create to add or update our deal. It will check whether a deal already exists and in case it’s already there will then update it with the latest price.

Model Binding

When making requests to the Amazon Products Advertising API we need to pass a set of credentials which we have already got from previous steps along with a product ASIN.

A product ASIN is an identifier for a product and we can capture it straight from the product’s URL. We want our form to be clever enough as to capture the ASIN from the URL automatically is paste a URL and our application will figure everything else out.

We also want to store the URL of our product on the database that will be sent out via SMS when the price for that product goes below the one we have stored on the database. A custom model binder is the perfect fit to do this as it will intercept our form, dissect it and get the information it needs from it.

Start by creating a new directory on the root of our project called Infrastructure. Inside this directory create a new class file called DealModelBinder.cs. Change the main class to implement IModelBinder in System.Web.Mvc. Visual Studio will then prompt us to implement the members of this interface which will will do by adding a new method to the class called BindModel as such:

There’s a lot happening on the code above so I thought I’d highlight the important bits and explain them a little bit here.

The way model binders work is by intercepting the request passed by a form and handling the data before anything else has the chance to handle it. Think of it like capturing the current request and making changes before anything else even knows of its existence.

HTTP requests normally terminate the way shown in the image above. A user makes a request, it get intercepted by the server, it communicates with the database, the database responds, and the server responds to tell the user the request has completed.

With Model Binders we introduce a an extra layer, which is where our validation kicks in and also where can put our logic to extract information from the URL and generate a shortened URL.

In our model binder we are using a regular expression to capture the ASIN code from the URL and then using the original URL to create a shortened version using the Bitly API. Make sure you have either replaced the keys with the ones you copied before or have them set in your environment variables.

We then return a new model to our controller to give it a chance to go through it again and perform any validations described in the annotations for that model. Everything working out the way it should our controller then calls the method AddOrUpdateDeal, which we will create next when we add our helper classes to the project.

Helper Classes

Create a new directory called Helpers on the project’s root. In that directory create a new class called AmazonHelper.cs and add the following to it:

The class above communicates with the helper library directly, but exposes a couple of helper methods to allow us to instantiate it and perform lookups.

All the member variables correspond to entries I have configured as environment variables. You can replace them with the appropriate keys but it is good practice for store such values as environment variables. That way you never need to worry about accidentally committing your keys to a public repository. You can check this article out to find more information about how to do it.

Notice I have highlighted the AwsServerUri as it’s an optional parameter. You can completely omit this if you want the library to default to Amazon.com. For a list of valid URL’s head to this page.

Now we need a class to handle the database updates and to alert us when prices become cheaper than the ones we already have on the database.

Create a new class file called TaskHelper.cs under the Infrastructure directory.

At the top of the class create two new member variables called _amazonHelper and _twilio. These will initialize our AmazonHelper class and the Twilio library.

Create a constructor for the class and initialize the member variables.

Like before feel free replace the environment variables with your own Account Sid and Auth Token if you want.

We are very close to being able to create our deals and save them to the database now. Let’s create a new method called AddOrUpdateDeal that returns a Deal and add the following to it.

In the code above we are checking whether a deal already exists, and have some logic to add or update the deal according to what we find. If the deal doesn’t exist we know we are adding it, so will create it in the database and then create a task that checks this deal until it finds the price has gone down. The task is very simple and Hangfire will handle it for us. Create a new method called AddFetchTask on the class.

We are checking for deal updates every hour by using Cron.Hourly. If you wanted to increase or decrease the gap between each check you could change that to any of minutely, hourly, daily, weekly, monthly or even use a Cron Expression for precision.

Conversely, if the deal already exists on the database we need to the update the information we have stored for it and check whether its price is lower than the one we already already have.

To do that we are using IsCheaperDeal to compare the prices and NotifyCheaperDeal to send us an SMS notification with a link when the price has gone down. Let’s implement those two methods on the same class then.

The first method compares the two prices and returns a boolean as to whether the new price is lower than the old price.

The NotifyCheaperDeal method only gets called if the price has gone down, and uses the SendMessage method on the Twilio helper library to send a new SMS message telling you that a price has gone down by a certain amount. Conveniently it will also include a shortened link to the item so even if you’re on the go, you can snap that bargain right out of Amazon.

Let’s add the TaskHelper instance to the top of our controller class. That way we will be  able to use it from within the controller. Open up HomeController.cs and add the following to the very beginning of the class.

Are you ready to roll?

We already have a view that lets us see what’s already in the database, but now we need to create a view that lets us add new items into it. Open up the Views/Home directory and create a new view called Create.

Make sure you specify a Template and a Model Class to match our existing model. That way, the form will be automatically created for us.

In that file you will notice form fields for the price, url and date have been created. We don’t know these yet, so delete them from the form. It should end up looking like this:

I’ve also added a stylesheet and JavaScript file to make the form look a little bit nicer.

Open up the file Index.cshtml under Views/Home and uncomment the Create ActionLink we commented out before.

Also add the following code right under the body tag.

Next time you create a new deal you should see a confirmation message saying the deal has been created.

We are ready to run the project again. When you start it up you should still see that there are no deals on the table but when you click on Create New you will be taken to a form that prompts you to add an Amazon URL and a deal name. It is important that you use a URL from Amazon here as this is where we will get the ASIN code from. You can name the deal anything you like, but make sure it’s something that makes sense to you.

Click create, and now your deal will be added to the database and tracked.

Viewing your tasks

Every time you create a new deal, a task is also created by the method AddFetchTask in the TaskHelper class. To see those tasks, open up a new tab and navigate to http://localhost:[PORT_RUNNING]/hangfire, and this will take you to Hangfire’s dashboard.

In this dashboard, you can see all your tasks and recurring jobs. This will give you real time information about all the tasks you have created, and when they will run. You can also trigger those tasks manually if all you want to do is test them.

What is the most annoying thing?

To me it has to be when I buy something and then find out the price has gone down right after I got it. So for that reason we will also implement a delete function to our application. That way we can delete deals from the database after we buy them.

To do that, we need to modify our controller by adding a Delete endpoint that will then remove the item from the database and consecutively remove the task so it doesn’t keep fetching information for that task.

And with that our deal gets removed from the database and so do any recurring tasks related to it. All you need to do now is uncomment the Delete link on Index.cshtml.

Keep your money where it should be

Saving money is not easy, but with this neat little hack, keeping money in your wallet should be a little bit easier.

We have gone from having to check a product’s price manually multiple times a day with a risk of paying too much for a product, to leaving our application to do the hard work for us and then tell us via SMS when prices have gone down.

But this is only the tip of the iceberg, and we could very easily add other stores and modify the application so it would allow us to update the recurring tasks on a product basis. We could also have it only tell us when prices have gone down by a certain percentage to maximize our savings.

I’d love to hear how you change this hack to make it work for your specific needs. Reach me out on Twitter @marcos_placona, by email on marcos@twilio.com or MarcosPlacona on G+ to tell me more about it.