How to Create a News Feed with Goutte and Twilio's Message Scheduling API

How to Create a News Feed with Goutte and Twilio's Message Scheduling API

I've long been a fan of newsreader applications. Mostly for the convenience of being able to keep up with the stories, writers, and publications that interest me. That said, I'm also aware of the negative impact of such apps on my daily productivity.

The frequent interruption of my concentration by clickbait headlines is proving to be quite frustrating — especially as I fall for it every time. Yet, I'm somewhat hesitant to unsubscribe for the sake of FOMO (Fear of Missing Out).

For this millennial, the struggle is real. So, if you're like me and you share the dilemma of staying socially current while avoiding the trap of sensational headlines, you will find this tutorial incredibly helpful.

What will you build?

You will learn how to create a personalised news feed application that will text the day's headlines at the end of each day. The project will be powered by the Goutte web scraper and Twilio's Message Scheduling API.

The real advantage of building a newsreader lies in the ability to specify when and how you want your news delivered. Once built, your attention will no longer have to decide between your work and keeping up with the latest headline screaming for your attention.

Prerequisites

To follow along, you'll need the following:

• A basic understanding of PHP
• PHP 7.4, though version 8 is preferable.
• Composer installed globally.
• A free or paid Twilio account. If you are new to Twilio, click here to create a free account.
• A smartphone to test your app

Create the project

First, you need to create the project's root directory, named twilio-news-feed, and then switch to it by running the following command in your terminal.

After that, create the following files inside your top-level directory:

• index.php - will contain the code for your Goutte scraper
• sms-scheduler.php -  will contain the code for the Twilio SMS API

To create the files, run the following command:

Then, create a third file, .env, which will contain the Twilio credentials which the code requires to interact with Twilio's Message Scheduling API. In it, paste the following code.

Retrieve your Twilio credentials

Create a Twilio Messaging Service

To be able to use Twilio's Messaging Scheduling, you need to set up a Messaging Service. Messaging Services are containers that bundle messaging functionality for your specific use case or campaign. For instance, you may want to create a Messaging Service to bundle all phone numbers that will send marketing messages to customers.

In this case, we'll be using a Messaging Service to notify users and bundling it with your phone number. To create it, log in to the Twilio Console, click on "Explore Products", and choose the "Messaging" option. Then, click on services and then click the "Create Messaging Service" button.

Type “Twilio News Reader” in the “Messaging Service Friendly Name” input field. Then on the drop-down list for “Select what you want to use Messaging for” choose “Notify my users”. When you’re done, click on the “Create Messaging Service” button in order to proceed to Step 2 of this setup.

Next, you need to add the sender. A sender represents a channel, number type, or messaging type that can send and receive messages using the Twilio Messaging API.

It’s now time to add your Twilio number, so click the “Add Senders” button to begin. Choose the “Phone Number” option and click on “Continue”.

Then click on the checkbox next to the number you want to add and then click the “Add Phone Numbers” button.

Now click on the “Step 3: Set up integration” button. You don’t need to update any information on this page, therefore proceed to Step 4. There are also no changes to be made on Step 4. Click the “Complete Messaging Service Setup” button to finalize this setup.

To test that it works, click on the “Try sending a message” button.

Add your verified personal phone number in the “To phone number” text field. Next, choose your newly created Messaging Service in the “From Messaging Service SID”. For the “Body Text” field, you can add any text you want. Click the “Send test SMS” button and then confirm that your phone has received the message. If it all went as expected, you should receive a text message like the one in the following screenshot.

Congratulations on successfully setting up your new messaging service! With the Messaging Service created, copy the Messaging Service SID and paste it in .env in place of <TWILIO_MESSAGING_SERVICE_SID>.

Next, go to the Console's dashboard and retrieve your Twilio Auth Token, Account SID, and phone number. Copy them and paste them in place of <TWILIO_ACCOUNT_SID>, <TWILIO_AUTH_TOKEN>, and <TWILIO_PHONE_NUMBER> respectively, in .env.

Now, it's time to have some fun creating the newsreader app.

Install the required dependencies

Next, run the following command to install the project’s three dependencies: Goutte, Twilio's PHP SDKs, and PHP Dotenv.

Create the Goutte web scraper

I’ll be scraping the Sky News website, but feel free to choose another website or blog and use my example as a reference guide. Please be aware that, while legal, in some countries and jurisdictions (such as the EU) there are restrictions on what you are allowed to do with scraped data.

Goutte works by extracting data straight from the HTML response. In this example, we want to grab the headline of each story on the front page of Sky News. To do this, you need to open the developer inspector tool on your browser For me, it’s the “Inspect Element” tool on Google chrome. You can find this on "View > Developer > Inspect Elements".

As you can see from the preceding example screenshot, you are looking for the <span> element with the class sdc-site-tile__headline-link. This is the class that contains all of the headlines on the front page of the Sky News website.  With this information, it’s time to write the code.

Copy and paste the following code into index.php:

The code begins by requiring the necessary dependency and file. The function breaking_news() creates a new Goutte Client object and passes it to the variable $client. The Sky News URL is passed to the variable $url and a request is made with the request() method which returns a Crawler object that is assigned to the variable $crawler.

To get the headlines, the class sdc-site-tile__headline-link is passed to the filter() method, and the each() method is used to iterate through the various nodes extracting the headline text.

Once complete, the breaking_news() function returns the Twilio messaging function and passes it the list of headlines as an argument. Next, you need to create the text message function that will use the Twilio API to text the news feed.

As it is now, running this code would result in 62 SMS being sent, so whilst this is great for learning and testing, be careful not to burn through your data too quickly. Twilio currently allows a maximum length of 1,600 characters per message.

Send the news feed as a text message

Next, copy and paste the following code into sms-scheduler.php:

The code begins by requiring all the necessary dependencies. Then, a new Twilio Client object is initialized with your Twilio Account SID and Auth token values. After that, a Twilio message is created with your verified Twilio phone number, your Messaging Service ID, and the body of the text message.

In the body parameter, you’re passing the news headlines that you scraped earlier with Goutte, along with the title Breaking News! Finally, you set the time that you want the message to be sent using PHP’s DateTime function and the scheduleType is  `fixed.

Testing time!

With the code complete, it's time to test it. To do that, run the code by running the command below.

If everything worked as expected, you should receive all the latest news from Sky News as a text message.

Congratulations on completing this tutorial, you’ve done an excellent job! If you’re feeling rather ambitious, you can extend this app to do even more. For instance, I think it’ll be pretty nice to receive delicious recipes and be in the know about the latest vacation deals. You never know what you’ll create if you let your imagination run wild!

I can't wait to see what you build.

Ijeoma Nelson is an ex-Deloitte software developer. She stumbled onto a hidden passion for programming after winning Startup Weekend Silicon Valley. When she’s not tinkering away with code, you can find her writing about code.