How to Allow Users to Upload Media Files to the Cloud

How to Allow Users to Upload Media Files to the Cloud

There are many factors to consider when building a public-facing website. For example, a developer would need to consider what is necessary to protect not only the users but also the project. They would need to figure out how to securely store the data - especially if the project consists of sensitive information.

In this tutorial, you will learn how to develop a functional website to authenticate your users and protect their identity before allowing them to upload an image file to your cloud storage. After authenticating the users, the project can give users the option to upload an image file through the site and store the files in a cloud.

The application will use two key technologies:

• Twilio Verify to generate one-time passcodes delivered to users' mobile phones, so that they can verify their identity and access your app. For even greater security, consider implementing two-factor authentication.
• Amazon Simple Storage Service (Amazon S3) to store the files. Amazon S3 offers fast and inexpensive storage solutions for any project that needs scaling. The Python code will interact with the S3 buckets to store and retrieve objects with flexible permission changing settings.

Tutorial requirements

• Python 3.6 or newer. If your operating system does not provide a Python interpreter, you can go to python.org to download an installer.
• A free or paid Twilio account. If you are new to Twilio get your free account now! (If you sign up through this link, Twilio will give you $10 credit when you upgrade.)
• Create a free account or sign in to your AWS console.
• A credit card linked to your AWS account in case you surpass the Free Tier eligibility options. It is worth noting that you should take extra precautions if you are deploying an app onto AWS. Make sure you stay within the Free Tier limits to avoid surplus charges at the end of the month. Refer to the S3 pricing guide and proper docs to prevent future charges.

Set up the environment

Create a project directory in your terminal called verify-aws-s3 to follow along. If you are using macOS or Linux, enter these commands in your terminal:

Since we will be installing some Python packages for this project, we need to create a virtual environment.

If you are using Linux or macOS system, open a terminal and enter the following commands:

NOTE: Depending on your active version of Python, you might have to specify python3.

If you are on a Windows machine, enter the following commands in a prompt window:

If you are curious to learn more about the packages, you can check them out here:

• The Flask framework, to create the web application that will receive message notifications from Twilio.
• The python-twilio package, to send messages through the Twilio service.
• The python-dotenv package, to read a configuration file.
• AWS' SDK for Python, known as Boto3, to create high-level access to AWS services such as S3.
• The awscli package to gain access to Amazon Web Services from the command line.

Create your first Twilio Verify

To use Twilio Verify, a service must be generated. Head to the Twilio Verify Dashboard, you should be on a page that says Services. Click on the red plus (+) button to create a new service. Give the service a friendly name of "site-verify". The friendly name will show up on the text message that is sent to users' phones, so if you have another name you would like to use, such as "<YOUR_NAME> website verify" feel free to do so.

After you've entered the friendly name, click on the red Create button to confirm.

Creating the Twilio Verify service will lead you to the General Settings page where you can see the properties associated with your new Twilio Verify service.

Open your favorite code editor and create an .env file in the project's root directory. Inside this file, create a new environment variable called VERIFY_SERVICE_SID and copy and paste the SERVICE SID on the web page as the value for this new variable.

To complete the .env file, create two additional environment variables: TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN. You can find the values for these variables on the Twilio Console as seen below:

Set up a development Flask server

Since we will be utilizing Flask throughout the project, we will need to set up the development server. Add a .flaskenv file (make sure you have the leading dot) to your project with the following lines:

These two incredibly helpful lines will save you time when it comes to testing and debugging        your project. The FLASK_APP tells the Flask framework where our application is located and FLASK_ENV configures Flask to run in debug mode.

These lines are convenient because every time you save the source file, the server will reload and reflect the changes.

With .flaskenv created, run the command flask run in your terminal to start the Flask framework. You should see output similar to the screenshot below:

The service is now listening on port 5000 for incoming connections.

Notice that debugging mode is active. When in this mode, the Flask server will automatically restart to incorporate any further changes you make to the source code. However, since you don't have an app.py file yet, nothing will happen. Though, this is a great indicator that everything is installed properly.

Feel free to have Flask running in the background as you explore the code. We will be testing the entire project at the end.

Navigate to the Amazon S3 Dashboard

Log in to the AWS console in your browser and click on the Services dropdown in the top left-hand corner of the webpage. Click on “S3” under the Storage tab or type the name into the search bar to access the S3 dashboard.

Create an S3 Bucket

Click on the orange Create Bucket button as shown below to be redirected to the General Configuration page.

Give the bucket a unique name that does not contain spaces or uppercase letters.

Keep in mind that bucket names have to be creative and unique because Amazon requires unique bucket names across a group of regions. Since this article uses the name "lats-image-data", it is no longer available for any other customer.

It is also important to know that the AWS Region must be set wisely to save costs. Regions are determined by where AWS data centers are located, and thus it's usually recommended to pick the one closest to you.

For example, a US developer would need to make sure their instances are within the United States. Someone living in California might choose "US West (N. California) (us-west-1)" while another developer in Oregon would prefer to choose "US West (Oregeon) (us-west-2)" instead.

The bucket in this tutorial will be named "lats-image-data" and set to the region "US East (Ohio) us-east-2", however please change accordingly to your use case.

Feel free to leave all the settings that follow as default. Then, scroll down and click the orange Create bucket button to see the newly created bucket in the S3 console, which you can see in the image below.

NOTE: Keep a copy of the bucket's AWS Region value (the lower cased, hyphenated string) as we'll need it later.

Create an IAM user on Amazon Web Services

In order for the Flask application to work, an Identity and Management (IAM) User needs to be created. Click the Services dropdown at the top left of the webpage. Scroll down to click on “IAM” under the Security, Identity, & Compliance tab or type the name into the search bar to access the IAM Management Console.

Choose Users on the left side of the console under Dashboard and click on the Add user button as seen in the screenshot below:

Enter a username such as "myfirstIAMuser" and check the box to give the user Programmatic access. This is sufficient enough, as it provides the access key ID and secret access key required to work with AWS SDKs and APIs.  

Click on the blue button at the bottom of the page labelled Next: Permissions. Select the box that says "Attach existing policies directly" and filter the policies by  "AmazonS3FullAccess". When it appears, check the checkbox next to the policy name.

Move forward by clicking the Next: Tags button. Tags are used to categorize AWS resources for different use cases making it more convenient to keep track of them. For example, this would help when you are working on large-scale projects and need to organize the AWS billing costs in a preferred structure. Given this project is relatively small, it's not necessary to add tags to this IAM user - especially if you only plan on using AWS for this specific application.

Go ahead and click Next: Review. Review the details set for "myfirstIAMuser" and finish off by clicking on the Create user button. Click Download.csv to download the CSV file named new_user_credentials.csv containing the access key ID and secret access key variables.

Configure the AWS IAM user credentials

In the terminal, type aws configure and enter the "Access key ID" from the new_user_credentials.csv file when prompted. Press enter and enter the "Secret access key" from the file for "AWS Secret Access Key". Press enter again.

For the "Default region name", enter the S3 Bucket's lowercased, hyphenated region which you made a copy of earlier. For this article, the region is "us-east-2". Press Enter to confirm and press it once more for the "Default output format".

Now that the credentials are configured properly, your project will be able to create connections to the S3 bucket.

Create a database file of eligible users

For the purposes of this tutorial, we will be hardcoding a list of accounts that are allowed to enter the website along with their phone numbers. In a production setting, however, you would likely use a database such as PostgreSQL instead.

We will be skipping passwords altogether solely for the scope of this project tutorial. However, handling passwords is essential in a production setting. Keep in mind that if you were to use your database, you would have to avoid storing passwords as plaintext.

NOTE: There are plenty of libraries that help developers manage passwords in a Flask application such as Flask Security.

Create a file in the project's root directory named settings.py and copy the code below into the file:

The dictionary can be modified to include different emails and phone numbers as you see fit. Make sure the phone numbers are in E.164 format as seen in the settings.py example above.

Be sure to add your phone number to an existing item in the dictionary, or create a new item with your information. Each email address is a unique key. This is helpful in our case because we want to look users up quickly in the login step.

Plan the logic of the project

The flow of logic for the project goes as follows:

1. A user from KNOWN_PARTICIPANTS enters their email address on the website's homepage.
2. The Flask application sends a one time passcode to the user's phone number.
3. The user is prompted to enter the passcode which they received from their phone to verify their identity.
4. If the user enters the passcode correctly, they are redirected to a page to upload a .jpg, .png, or .jpeg image of their choice.
5. The media file will be uploaded to the Amazon S3 bucket that was created earlier in the tutorial.

With that said, let's start coding! 

In your working directory, create a file named app.py and copy and paste the following code:

At the top of the file, we:

• Imported the necessary Python modules and libraries so that the project can load the environment variables.
• Imported the list of participants from settings.py.
• Started the Flask app.

Replace app.secret_key's default value, not-so-secret-key, with a random string. The Flask app uses it for a limited level of security for the user session and because we need to store the users' account information and pass it along to other routes on the site using Flask's session.

Create an empty folder named UPLOADS' in the project's root directory. This is where all uploaded image files will be stored for the sake of simplicity in this tutorial.

To organize the project directory, create another file named s3_functions.py in the project's root directory. This file will contain three helper functions used to connect to the AWS S3 client and utilize the boto3 library.

Then, add the following import statement to the s3_functions.py file:

If you're using Microsoft Windows, use the commands below instead.

Build the user login page

For this project, the user will go to the website and enter their username, which is an email address in this case. Copy and paste the following code at the bottom of your app.py file:

A POST request is made to allow the participant's username to be stored in the Flask session. If the username is in the database, in this case the KNOWN_PARTICIPANTS dictionary, then the username is stored in the current Flask session and the verification token is sent to the corresponding phone number. The participant is redirected to another route where they will see a second form allowing them to submit the verification code.

Create a form that takes in a username input as well as a button to submit the form by copying and pasting the barebones HTML form from this GitHub repo into the index.html file:

Here's an example of what happens when the user enters a username that’s not in the KNOWN_PARTICIPANTS dictionary:

Generate a verification code with Twilio Verify

With the form set up, it’s now time for the fun part - calling the Twilio Verify API! The next step is to build the send_verification function that will fire after the user submits the form. It will send the verification token after the user enters a valid email in the database.

Add the following code to the app.py file under the same route as the login function:

The Twilio Client sends a verification token to the phone number associated with the username stored in the current Flask session. The specified channel in this case is SMS but it can be sent as a call by setting channel='voice' if you prefer.

NOTE: Keep in mind that this is a simple function that sends a verification passcode and does not yet account for error handling.

Time to test it out. On the webpage, enter the email in settings.py that corresponds to your phone number. You should get an SMS with a passcode shortly.

Verify the user's phone number

We will now take the user input from a new form and make sure it is the same exact verification code that Twilio texted via SMS to the mobile phone.

Let's start by creating the form on the HTML side. Copy and paste the HTML from my GitHub repo into the body of verifypage.html:

But wait - how can we verify the 6-digit code if Twilio is the one that sends out the code? We need to define the /verifyme route and define the appropriate functions so that the user can verify the passcode.

Copy and paste the following code to the bottom of the app.py file:

We need to define the check_verification_token() function beneath the verify_passcode_input() code so that this function can be called within this route:

The check_verification_token function takes in the Flask session's phone number and the verification_code that the user typed into the textbox and calls the Verify API to make sure they entered the one time passcode correctly.

Once the user submits the form, which then makes the POST request to the /verifyme route, the verify_passcode_input() function is called. If the passcode was correct, the success page is rendered. Similar to the logic for the login page, if the participant enters an incorrect verification code, the page will refresh and show an error message; the page will also let the user enter the verification code again.

Allow users to submit a photo

At this point, the user has entered their credentials and verification code correctly. It's time to build out the page where users can drag and drop the image file. Copy and paste the HTML below from the GitHub repo into the body of uploadpage.html:

As you might have noticed, the mechanics in the uploadpage.html page are different, as you have to specify the input type that is expected in this POST request. Here, multipart/form-data is used in order to encode the data as a file element.

Once the user clicks on the Submit button, they are rerouted to the /uploader page of the website where they will see a success page if they uploaded an appropriate file with either a .jpg, .png, or .jpeg extension. Otherwise, they will be asked to reupload a valid file.

Here's what the error message could look like:

Go back to the app.py file to build out these two new routes so that the user can upload an image. Copy and paste the following code to the bottom of the file:

If the user makes a POST request on uploadpage.html, then the file is stored in the Flask session's request object. Not only do we check if the user uploaded a valid .jpg, .png, or .jpeg file and that it exists, but we have to make sure that the user input is appropriate.

This is the reason we imported the werkzeug library earlier so that we can utilize the secure_filename function. Our project saves the uploaded files inside the project directory so we must take the precautions to identify file names that may harm our system.

To check the file extensions, we need to define the allowed_file function where the file name uploaded can be parsed and checked against the app.config['UPLOAD_EXTENSIONS']. Scroll up to the top of the app.py file to define a global function.

Place the following code between the definition for the first / route and the KNOWN_PARTICIPANTS:

Once the file input has been checked and approved, the file is saved into the project directory's "UPLOADS" folder, as stated in app.config['UPLOAD_FOLDER'].

Handle file uploads to the cloud database

As seen above, the media file is saved to the local uploads folder in the working directory and then calls another function named s3upload_file(). This file takes in the pathname of the recently added file and inserts it into the bucket name provided in the second parameter. After the upload is complete, the page is refreshed and the user ends up back on the landing page.

Open up the s3_functions.py file again to write the s3upload_file() function to complete the /upload route. Copy and paste the following code under the import statements:

An s3_client object is created to initiate a low-level client that represents the Amazon Simple Storage Service (S3). After creating the connection to S3, the client object uses the upload_file() function, and takes in the path of the filename to figure out which media file to upload to the bucket. The last parameter, object_name, represents the key where the media file will be stored as in the S3 bucket.

Display a success message

You can now redirect them somewhere else as you please, but in this tutorial you’ll redirect them to a success page. Copy and paste the HTML from my GitHub repo into the success.html file within the templates directory:

Authenticate your account with Twilio Verify and submit an image

It's time to test out the app. Feel free to look at the completed code on GitHub. Make sure that Flask is running on your terminal by using flask run. Then visit http://localhost:5000/ and enter any username from the defined dictionary in settings.py.

I'll use the "twilioisthebest@twilio.com" email which is the key for my own phone number for testing:

Check your phone to see the notification for the verification code provided by Twilio Verify. In my case the code was 864831.

After entering the code correctly, you'll see this page:

Select an image in .jpg, .png, or .jpeg format. Feel free to use the classic DRAW_THE_OWL_MEME.png. Click the Upload button and check the uploads folder in the project directory.

Check your project directory and look at the UPLOADS folder, or whichever folder you created to store the pictures. You should see the image you just submitted.

Check the Amazon S3 bucket for the uploaded file

Open a new tab in the web browser and head back to the AWS Console. Navigate to the S3 bucket and click on the bucket name that was used to upload the media files. At this point, there should be one object in the bucket, the uploads folder.

Here's an example of the "lats-image-data" bucket created for this article:

Click on the link for the uploads folder. There should be a new list of objects. The list above shows the file object name, DRAW_THE_OWL_MEME.png, along with the metadata of the object including the file type, date last modified, size, and storage class.

What a success! The media file was uploaded successfully and you have the option to download the file.

What’s next for authenticating users to upload media files to the cloud?

Congratulations on implementing safe practices and incorporating security measures into your project! There are plenty of ways to expand on this project and build a full fledged site to interact with users safely and avoid bad actors that you may come across.

Another way you can use Twilio Verify for authentication is to send a verification code over email using Twilio Verify and SendGrid. In that case, you would use the username in the database or any email address on the user's profile.

You can also increase the amount of safety by checking the user's input through implementing an image recognition API in your project to detect content such as NSFW photos.

If you're ready to expose the app to the world, check out these 3 tips for installing a Python web application on the cloud or read how to redirect a website to another domain name.

Let me know what you're building in your image uploading site and how you are protecting the users in your project by reaching out to me over email!

Diane Phan is a developer on the Developer Voices team. She loves to help programmers tackle difficult challenges that might prevent them from bringing their projects to life. She can be reached at dphan [at] twilio.com or LinkedIn.