Craft Beer & SMS: Never Miss A Rare Brew Again

Time to read: 12 minutes

March 05, 2015
Written by
Phil Nash
Twilion

craft

What a time to be alive! Just a couple of weeks ago was London Beer Week. My home city spent a week celebrating everybody’s favourite pint sized beverage. With the focus on great beer from breweries across the capital, the country and the world I’ve been thinking about how I can get more great beer all year round.

You see, there’s this great beer shop, Clapton Craft. It’s near to my flat and sells both superb and sometimes rare beer in growlers*. They have a list of the currently available beers but the problem is that it’s hard to get notified when new beers become available. I’m not one to let that sort of thing cause a problem though, so armed with Ruby, an HTML parser, Redis and Twilio SMS, I’m going to build a growler notification system so that I never miss an exciting IPA, saison or imperial stout again.

*Growlers

For the benefit of those who don’t know, the growler is a wonderful thing. It is a 3 and 1/3 pint bottle typically found in breweries in the US for the purpose of filling up, taking home and enjoying great beer fresh from the keg. Growlers have come to London and there are a few pubs, shops and breweries that are will fill them for you. Clapton Craft is my local growler filler and the inspiration behind this hack.

Getting prepared

To complete today’s hack we’re going to need a few bits:

Once you’re all set up we’ll get started. I have a starter project set up on GitHub with the initial files you’ll need as well as a bunch of tests that we can use to verify we’re doing things right. Clone the project and we can get on with the code.

$ git clone https://github.com/philnash/growler_alerts_starter.git growler_alerts
$ cd growler_alerts
$ bundle install
$ rake spec

Clone the growler alerts starter project, install the dependencies and run the tests.

Failing tests, the best place to start! Before we get on with fixing them we do need a bit of config. We’re going to use envyable to load our development configuration into the environment. Copy config/env.yml.example to config/env.yml and fill in your Twilio Account Sid and Auth Token (you can find this on your account dashboard).

You will also need a Twilio phone number that can send SMS messages. If you’ve already got one carry on, otherwise head on into your account and buy one now.

Once you have those details, enter them into config/env.yml with your own phone number too:

# config/env.yml
development:
  TWILIO_ACCOUNT_SID: YOUR_ACCOUNT_SID
  TWILIO_AUTH_TOKEN: YOUR_AUTH_TOKEN
  TWILIO_NUMBER: YOUR_TWILIO_NUMBER
  MY_NUMBER: YOUR_MOBILE_NUMBER
  REDIS_URL: redis://127.0.0.1:6379/0

I’ve filled in the Redis URL as the system default for now, although if you have installed it with different defaults you will want to update it.

We’re all set to start coding now. Let’s set something up to discover the latest beers available at the Clapton Craft.

There’s no API!

Normally on this blog this is where we reach for an API to help solve our problem. Sadly in this case we don’t have an API, but we do have a website. So, let’s make our own API out of the site.

Since there is no structured data, we need to work with the partially structured data that can be found in the HTML of the Clapton Craft site. This is why I included Nokogiri in the Gemfile. Nokogiri is an XML and HTML parser and we’re going to use its power to reach into the HTML of the Clapton Craft site and extract the beers that are available.

Inspecting the source of their growlers page we can see that there are two blocks, one for growlers and one for bottles. Keep that in mind and let’s create a class to work with this data.

First open up lib/clapton_craft.rb, require the libraries we need and create a class:

# lib/clapton_craft.rb
require "nokogiri"
require "open-uri"

class ClaptonCraft

end

Let’s add an initializer that sets the URI of the page to read the beer details from. Whilst we know what URI we’ll be looking up, this makes it easier to test as we can pass a local file instead.

# lib/clapton_craft.rb
class ClaptonCraft
  def initialize(uri)
    @uri = uri
  end
end

We need a method that will load the page content into Nokogiri so that we can query it. Add the following:

# lib/clapton_craft.rb
class ClaptonCraft
  def get_page
    @page ||= Nokogiri::HTML(open(@uri))
  end
end

The get_page method uses the Ruby standard library’s open-uri to load the contents of the URI straight into Nokogiri’s HTML parser. We now need a method to extract the beers from the page.

# lib/clapton_craft.rb
class ClaptonCraft
  def get_list(type)
    beers = get_page.css(".#{type}_list h4")
    beers.map { |beer| beer.text }
  end
end

We can use a CSS selector to pick items out of the page. With this method we can get either the growlers or the bottles available and then grab the text out of the elements. Let’s open up the terminal and see what that does. Run:

$ bundle console

to open an irb session with our gems already loaded and then run the following:

> require "./lib/clapton_craft"
> clapton_craft = ClaptonCraft.new("http://www.claptoncraft.co.uk/growlers.html")
> clapton_craft.get_list("growlers")
=> ["Burning Sky Session IPA 4.4% £11 ", "Four Pure Oatmeal Stout 5.1% £12.50 ", "Wild Beer Co Madness IPA 6.8% £16 ", "Siren 7 Seas Black IPA6% £14"]

That’s pretty good, but I don’t think we need the ABV or the price. We can select just the first text element of the <h4> and then read its text, stripping any extra whitespace for good. Let’s update the get_list method for this.

# lib/clapton_craft.rb
class ClaptonCraft
  def get_list(type)
    beers = get_page.css(".#{type}_list h4")
    beers.map { |beer| beer.children.first.text.strip }
  end
end

Try that in irb again and we get an array of beers:

> require "./lib/clapton_craft"
> clapton_craft = ClaptonCraft.new("http://www.claptoncraft.co.uk/growlers.html")
> clapton_craft.get_list("growlers")
=> ["Burning Sky Session IPA", "Four Pure Oatmeal Stout", "Wild Beer Co Madness IPA", "Siren 7 Seas Black IPA"]

Great! Let’s just add a couple of convenience methods to get the growlers and the bottles from this page:

# lib/clapton_craft.rb
class ClaptonCraft
  def growlers
    get_list("growlers")
  end

  def bottles
    get_list("bottles")
  end
end

To make sure this is all correct so far, run the tests for the ClaptonCraft class.

$ ruby spec/clapton_craft_spec.rb

Once you have written the code above, the tests for the ClaptonCraft class should pass.

That takes care of getting the latest beers, now we need something to compare against.

Storing beers

We need to keep a record of the beers that are available, so that we can compare them and alert about new beers. I have chosen to do this with Redis as the storage requirements are fairly simple and the Set datatype stops us saving the same beer more than once.

Open up lib/cellar.rb, require the redis library and create a class:

# lib/cellar.rb
require "redis"

class Cellar

end

I called it “Cellar” because that’s where beer is stored.

Sometimes my favourite part of programming is naming things.

We need to initialize the Cellar with a couple of things, a type (Clapton Craft actually serves beer up in growlers or 1 litre bottles) and a redis client to store the data. We can let redis default to a new instance as that will pick up our REDIS_URL environment variable.

# lib/cellar.rb
class Cellar
  def initialize(type, redis=Redis.new)
    @type = type
    @redis = redis
  end
end

We’re going to store two sets of beers for each type of vessel, one for the current beers available and one for all the previous beers. Getting the current beers or previous beer is a case of reading the members from that set.

# lib/cellar.rb
class Cellar
  def current_beers
    @redis.smembers("#{@type}:current")
  end

  def previous_beers
    @redis.smembers("#{@type}:previous")
  end
end

The next method we need is a bit more complicated. It’s going to take the latest set of beers (that we’ll get from the ClaptonCraft class) and set them to the current beers in the Cellar. It will also move any beers that are no longer available into the previous beers list. We’ll use a couple of utility methods for this:

# lib/cellar.rb
class Cellar

  private

  def removed_beers(latest_beers)
    current_beers - latest_beers
  end

  def move_to_previous(beers)
    beers.each do |beer|
      @redis.smove("#{@type}:current", "#{@type}:previous", beer)
    end
  end
end

The removed_beers method takes the latest set of beers and just returns the set of the old beers that are no longer available. The move_to_previous method can then take those removed beers and move them from the current to the previous set. Now, using those private methods, we can build a public current_beers= method:

# lib/cellar.rb
class Cellar
  def current_beers=(latest_beers)
    move_to_previous(removed_beers(latest_beers))
    @redis.sadd("#{@type}:current", latest_beers)
  end
end

This uses our utility functions to move the old beers out of the current set. Then we add all the new beers to the current set. We’re going to add one more method to clear out the Cellar for use later.

# lib/cellar.rb
class Cellar
  def clear
    @redis.del("#{@type}:current")
    @redis.del("#{@type}:previous")
  end
end

With these methods all in place you can now run the tests for the Cellar class and watch them pass:

$ ruby spec/cellar_spec.rb

Running the Cellar tests should pass now too.

In fact, all the tests should pass now:

$ rake spec

Now all the tests pass.

Fantastic! Now we just need to tie this all together and send an SMS if there are new beers available.

Fill it up and take it away

We need three tasks, one to prime Redis with the existing beers, one to run regularly to check if there’s anything new and a last one to clear the cellars. Let’s open up our Rakefile which has only been used to define our test tasks so far.

First, we need to require the files we’ve been working with and our environment variables (if we need them):

# Rakefile
require "rake/testtask"
begin
  require "envyable"
rescue LoadError
end
require "twilio-ruby"
require "./lib/clapton_craft"
require "./lib/cellar"

if defined? Envyable
  Envyable.load("./config/env.yml")
end

Then we add the tasks:

# Rakefile
namespace :beers do
  desc "Loads initial set of beers into Redis"
  task :setup do

  end

  desc "Gets latest beers and sends me an SMS if there are new ones"
  task :alert do

  end

  desc "Clears out all cellars"
  task :clear do

  end
end

The setup task needs to create a Cellar for each available container at the shop and fill it with the currently available beers.

# Rakefile
  desc "Loads initial set of beers into Redis"
  task :setup do
    redis = Redis.new(url: ENV["REDIS_URL"])
    clapton_craft = ClaptonCraft.new("http://www.claptoncraft.co.uk/growlers.html")
    [:growlers, :bottles].each do |type|
      cellar = Cellar.new(type, redis)
      cellar.current_beers = clapton_craft.send(type)
    end
  end

The alert task does a bit more. We need to make a new Twilio REST client to be ready to send messages, then find the latest bottles and growlers, compare to what we have and if there are new beers send an SMS message and update the Cellar.

desc "Gets latest beers and sends me an SMS if there are new ones"
  task :alert do
    client = Twilio::REST::Client.new(ENV["TWILIO_ACCOUNT_SID"], ENV["TWILIO_AUTH_TOKEN"])
    redis = Redis.new(url: ENV["REDIS_URL"])
    clapton_craft = ClaptonCraft.new("http://www.claptoncraft.co.uk/growlers.html")
    [:growlers, :bottles].each do |type|
      cellar = Cellar.new(type, redis)
      latest_beers = clapton_craft.send(type)
      new_beers = latest_beers - cellar.current_beers
      if new_beers.any?
        client.messages.create(
          from: ENV["TWILIO_NUMBER"],
          to: ENV["MY_NUMBER"],
          body: "New #{type} at Clapton Craft: #{new_beers.join(", ")}"
        )
        cellar.current_beers = latest_beers
      end
    end
  end

It’s tough to check if this is working, so I’ve left a couple of example pages with different beers in the spec/fixtures directory that we can use to make sure this is working.

Update the tasks to set up using one of the files and then alert based on the other one.

# Rakefile
  task :setup do
    redis = Redis.new(url: ENV["REDIS_URL"])
    clapton_craft = ClaptonCraft.new("/path/to/growler_alerts/spec/fixtures/clapton_craft.html")
    # ...
  end

  task :alert do
    client = Twilio::REST::Client.new(ENV["TWILIO_ACCOUNT_SID"], ENV["TWILIO_AUTH_TOKEN"])
    redis = Redis.new(url: ENV["REDIS_URL"])
    clapton_craft = ClaptonCraft.new("/path/to/growler_alerts/spec/fixtures/clapton_craft_new.html")
    # ...
  end

Run the two tasks in order, you should receive an SMS about a new beer available for growlers and bottles.

rake beers:setup
rake beers:alert

Two messages alerting me about two new beers from the test.

Great, testing worked! Let’s set this up for the real thing. Reset the URIs for the ClaptonCraft class to the real URL. Now we’re going to create the clear task and use it to clear out our database:

# Rakefile
  task :clear do
    redis = Redis.new(url: ENV["REDIS_URL"])
    [:growlers, :bottles].each do |type|
      cellar = Cellar.new(type, redis)
      cellar.clear
    end
  end

This creates the Cellars for bottles and growlers, then clears them out using the method defined earlier. Run the task:

$ rake beers:clear

Now we are ready to use the scripts for real.

Deploying

You can actually deploy this however you want to. You can leave it on your own machine and run the rake beers:alert task periodically via a cron job or similar. I’ve chosen to deploy to Heroku so that I can get alerts even when my machine is turned off. You can deploy this project to Heroku too if you want. Below is a handy Deploy Button to do all the hard work.

Deploy to Heroku

Once you deploy, you still need to set up the Heroku scheduler. Click on “make your first edit” which will take you to your Heroku dashboard. Click on Add ons and then on the Scheduler and create your task. I have set it to run every hour. The script you need to run is:

bundle exec rake beers:alert

Sit back, grab your favourite craft beer and relax

This is a success for two reasons. Firstly, we can now get updates about new beers at Clapton Craft with no further effort. However this also demonstrates a way of turning websites without APIs into an API that we can use.

That power is in your hands now, you can create alerts for any type of site you want. Personally, I’m going to see if I can find and add other beer shops around London so I can keep up to date on their growler availability. This use of HTML parsing can be quite brittle too, if the site updates the HTML then this will break. I will also be looking into alerting myself if it looks like it has broken.

Have you ever had a site you wished had an API? Do you have a favourite craft beer? Leave me answers in the comments below or grab me on Twitter at @philnash. I might not answer for a little while though, I’ve got to go out and get me some of that Kernel Porter.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.