How I Hacked My University’s Registration System with Python and Twilio

Time to read: 5 minutes

June 21, 2017
Written by
Samuel Taylor
Contributor
Opinions expressed by Twilio contributors are their own

jmu

University students know the pain of trying to register for a class only to realize it’s full. At my university we didn’t even have a waitlist system for most classes. We had to resort to logging in and checking the site multiple times a day. This seemed like something a computer could do, so I set out to automate it with a bit of Python and the Twilio API.

Getting Started

Because the university’s course registration system is behind a password login, we’re going to use a simplified site I set up. For the purposes of this demo, it alternates each minute between having no open seats in CS 101 and having one open seat.

We’re going to use a few libraries to help us with this project. Assuming you already have pip installed, go ahead and install them by running the following pip command: 

pip install requests==2.17.3 beautifulsoup4==4.6.0 redis==2.10.5 twilio==6.3.0 Flask==0.12.2 

We’ll dive into using each one of these libraries as we get further along.

Scraping the Registration System

We need to write a program that can determine whether there are seats available in a given course. To do this, we’ll use a technique called web scraping, in which we download a page from the internet and find the important bits. Two popular libraries that make this easy are Requests and BeautifulSoup. Requests makes it easy to get a web page, and BeautifulSoup can help us find the parts of that page that are important to us.

# scraper.py
import requests
from bs4 import BeautifulSoup

URL = 'http://courses.project.samueltaylor.org/'
COURSE_NUM_NDX = 0
SEATS_NDX = 1

def get_open_seats():
    r = requests.get(URL)
    soup = BeautifulSoup(r.text, 'html.parser')
    courses = {}

    for row in soup.find_all('tr'):
        cols = [e.text for e in row.find_all('td')]
        if cols:
            courses[cols[COURSE_NUM_NDX]] = int(cols[SEATS_NDX])
    return courses

The meat here is in the get_open_seats function. In this function, we use requests.get to download a page’s HTML source, then we parse it with BeautifulSoup. We use find_all('tr') to get all the rows in the table, updating the courses dictionary to indicate the number of seats available in a given course. find_all can be used more powerfully than this simple example, so check out the documentation if you’re interested in learning more. Finally, we return the courses dictionary so that our program can look up how many seats are in a given course (i.e. courses['CS 101'] is the number of seats available in CS 101).

Hooray, now we can determine whether a course has open seats. A great way to test this function out is in the Python interpreter. Save this code into a file called scraper.py, then run the script and drop into interactive mode to see what this function does:

$ python -i scraper.py
>>> get_open_seats()
{'CS 101': 1, 'CS 201': 0}

While this is great, we’re not quite to a solution; we still need some way to notify users when a seat opens up. Twilio SMS to the rescue!

Getting Updates via SMS

When building a user interface, we want simple things to be simple. In this case, users want to get notified when seats in a course open up. The simplest way for them to communicate that intent to us is sharing the course number. Let’s implement a subscription functionality by setting up and handling a webhook. I’m choosing to use Redis (a tool which provides data structures that can be accessed from multiple processes) to store subscriptions.

# sms_handler.py
from flask import Flask, request
import redis
 
twilio_account_sid = 'ACXXXXX'
redis_client = redis.StrictRedis(host='localhost', port=6379, db=0)
app = Flask(__name__)
 
@app.route('/sms', methods=['POST'])
def handle_sms():
    user = request.form['From']
    course = request.form['Body'].strip().upper()
 
    redis_client.sadd(course, user.encode('utf-8'))
 
if __name__ == '__main__':
    app.run(debug=True)

Here we use a web framework for Python called Flask to create a little service that handles SMS messages. After some initial setup, we indicate that requests to the /sms endpoint should be handled by the handle_sms function. In this function, we grab the user’s phone number and the course they were looking for, and store them in a set named after the course.

This is great as far as capturing the subscriptions, but it is a frustrating user interface because it doesn’t provide any feedback to users. We want to get back to users and tell them whether we’re able to service their request as soon as possible. To do that, we’ll provide a TwiML response. The additional lines needed for that are highlighted below.

# sms_handler.py
from flask import Flask, request
import redis
from twilio.twiml.messaging_response import MessagingResponse
 
twilio_account_sid = 'ACXXXXX'
my_number = '+1XXXXXXXXXX'
valid_courses = {'CS 101', 'CS 201'}
 
redis_client = redis.StrictRedis(host='localhost', port=6379, db=0)
app = Flask(__name__)
 
def respond(user, body):
    response = MessagingResponse()
    response.message(body=body)
    return str(response)
 
@app.route('/sms', methods=['POST'])
def handle_sms():
    user = request.form['From']
    course = request.form['Body'].strip().upper()
    if course not in valid_courses:
        return respond(user, body="Hm, that doesn't look like a valid course. Try something like 'CS 101'.")
 
    redis_client.sadd(course, user.encode('utf-8'))
        return respond(user, body=f"Sweet action. We'll let you know when there are seats available in {course}")
 
if __name__ == '__main__':
    app.run(debug=True)

We’ve made two major changes in the code above. First, we validate that the user is asking for a valid course. Second, we respond to users when they ask for updates. In the respond function, we construct a TwiML response to a given number with a given message.

Make sure to install Redis and start it up with the redis-server command. Save the above code into a file called sms_handler.py and then run python sms_handler.py.

Admittedly, the response messages here are a bit silly, but I was surprised to see how much users enjoyed them. In some contexts a personal touch can make for a better user experience.

Let’s extend our earlier scraping script to actually notify those people now that we know who wants to be notified of a course opening up.

# scraper.py
from twilio.rest import Client

client = Client(twilio_account_sid, token)
redis_client = redis.StrictRedis(host='localhost', port=6379, db=0)

def message(recipient, body):
    message = client.messages.create(to=recipient, from_=my_number, body=body)


if __name__ == '__main__':
    courses = get_open_seats()
    for course, seats in courses.items():
        if seats == 0:
            continue

        to_notify = redis_client.smembers(course)
        for user in to_notify:
            message(user.decode('utf-8'), 
                    body=f"Good news! Spots opened up in {course}. " + \
                          "We'll stop bugging you about this one now.")
            redis_client.srem(course, user)

We can run this scraper on a one-off basis to test it by running python scraper.py.

Keeping Tabs on Courses with a Cron Job

While simplifying the process of checking the course registration site into a single script is nice, we want the script to automatically run every few minutes. This problem is easily solved by using Cron. We can add a task to run every three minutes by running crontab -e and adding the following line:

*/3 * * * * /path/to/scraper.py

With that code in place, the Cron daemon will run our scraper every three minutes. We can see the scheduled tasks by running crontab -l. And that’s it! We can subscribe to updates for a class and get back to the important things in life. As a fun side benefit, your friends will be very appreciative when you get them into that packed “Rest and Relaxation” course. While getting into the classes I wanted was plenty of reward for the work, it also ended up helping around a dozen people get their ideal schedules.

Using the techniques from this post, you can set up notifications for a wide variety of things. For instance, someone used Ruby and Twilio to track craft beer availability. To get all the code from this post, check out this gist. You can also contact me via:

Disclaimer: Make sure to check that setting up notifications will not violate your university’s student system terms of service. When in doubt, ask someone who knows!

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.