Discover projects
News Post

How to Build an Amazon Gallery Full Stack app with DjangoREST Framework API

Tech

Introduction

In this blog post, we will cover how to create your own full-stack Amazon gallery application. The app will scrape real products directly from Amazon and display them on your website.

Since we will talk about all parts of the system - the back- and frontend and how to connect them, we will divide the blog into three main topics:

Creating Backend

API and DjangoREST Framework - these would be the main technologies we use. We will be having endpoint products - a lot of different endpoints.

Coding the scraper script

The second thing we will build is the actual scraper which will get our products' data from Amazon.

We have a separate blog post including a video tutorial for building the scraper script here:

Creating a simple frontend

The third part of our crash course will focus on the frontend. The frontend allows you to see the different categories we want to offer. Of course, each category also shows an exemplary product that it contains. These are not just images, we are showing real products that we are scraping from Amazon.

In this crash course, we will be using the same code base. We will modify it a little. We will also have search functionality, in which we can put “iPhone” for example.

In your app, you can go to categories - something we scrape as well. For them, we need a connector -  a short script that will connect the data we scraped in JSON files, using our applications’ backend.

We are very sure you will learn a lot.

Step 1: Building the backend

Let's start with building the backend - a DjangoREST Framework.

DjangoREST allows you to create the APIs and the Django so that you create totally separate front- and backend.  A lot of companies use this framework to build APIs.

We highly recommend checking out their tutorials. This way, you will perfectly understand everything we’ve covered.

For this project, we will start by creating a Github repository.

Then, we created the folder structure. At this point, we can’t really see the backend or frontend yet. Once the directories are empty, you can't really push them to GitHub but that’s not a problem at all.

Okay, let's get into it.

Creating a VE

First of all, we will need a Virtual Environment for Python.

Second, we will need to create dependencies. We already know which dependencies we will be using and we created the whole file. Feel free to check Jakob's GitHub repository and copy the requirements.txt file.

Okay, let's look at what are the next steps.

We will need the Django, DjangoREST Framework, and some extensions and filters for the functionalities.

We need Selenium for our Amazon scraper web driver manager to install our web driver. This way, we don’t have to work manually.

Installing requirements

Let’s install requirements. First, we update our pip. Now, we start our Django project. We decided to create a whole project called "Amazon Backend". We also used DOTS so we didn’t create an additional level of directories.

Now what we can do is migrate so it will create an SQLite database and we can run our server.

Let’s get the other things.

We have our alias for Pycharm. We will open it on the backend level so this will be our root directory. The next thing we are doing is to find a Python interpreter.

Install products app

For this step, we will need to go back to DjangoREST Framework tutorials. On there, they also tell us how to do it. Let’s call it "Products".

First, we are installing the Products app which is the one we just created. We will handle everything about our Amazon products there. We will use DjangoREST Framework for creating the APIs, and Django extensions which will give us access to shell_plus - a really cool feature.

We will also use Django features for creating the endpoints with filters. This way, we can easily get the products which belong to the category "Phones", for example.

Now, let’s create our Django REST Framework settings. We could do it after the database, but as it could be placed wherever we like, let’s simply add it. We need a couple of settings here:

Default filter backends

Those are settings for the backend so if you got to Django filters (link), here we can see the setting filter backend. That’s it. Then, we will need two more filters.

  • Search Filter
  • Ordering Filter

The next we want to do is go to the URLs and import the include here, as we create a new path - get URLs from the products here. As we don’t have this file yet, now we will create it.

Product list

This will be our first endpoint - we’ll use the class view. We don’t have it yet - so we import it. Now we have to create it in our views file. We will create an API view. We create a class called product list and we inherit it from the API view.

In this particular endpoint, we will be actually able to list our products or create a new product.

If you go to the Django REST Framework tutorial, you will see that there are class views. You can use a function but if you use the class-based views, you will save a lot of time. However, there are even faster solutions, but this will work for now.

We have to implement the get method and the post method for our needs. Before that, we have to actually create our model for a database for our products.

Let’s go to models.

Creating database models

For models, we would have to think for a second.

While we scrape Amazon, we will have multiple products. Each product has its own category. We will have products, categories, and we will add an additional one - a "sellers" category. It’s valuable information we can store, but we don’t have to use it later. Let’s implement those models now, starting with "category".

In the console will see the name, we have the seller, and the product. About the product, there are a few things - a photo, URL, and ASIN - unique ID. If you delete the URL on both sides and leave the ASIN, it’s actually a way to shorten Amazon URLs.

Let’s get back to our script.

We got the seller. In the example in the video, Jakob points out the query for which Amazon chose a certain product as the best one for this query, and not the seller.

Now, we can do “python manage.py makemigrations”

Again, we created all of the migrations and now we’re going to migrate them, so we can have the information in a database. What we need to do now is implement two methods - GET and POST.

When you are creating an API in Django REST Framework - you need a model, view URL, and an important thing - a serializer.  Whenever you have an object, you can serialize it to JSON. This is the representation of the object.

We will use a higher-level serializer than what’s shown in the Django REST Framework tutorial - something faster and cooler.

Let’s create a new file - serializaers.py.

Now, let’s create a couple of serializers.

First of all, this is the "serializer" category. It serializes the categories. We need to create method fields to get a random photo. This will be something we need on the front end - a photo of the first product in the category.

Then we have a basic "seller" serializer - something we probably won’t use.

We also have a "product all info" serializer. This now will serialize "category" and the "seller" in the following way - when we ask the endpoint for data, we will get everything nicely formatted.

Let’s get back to the views.

Alright, here we have another example of a model view set - a level above a simple API view.

The viewset is something high-level - under the hood, there are a ton of possible methods. Get, Pos,t Delete, Update - for those we decided to go with viewsets.

Let’s add the URLs and see how it all works.

Now, we have two options:

  • Create a default router to handle many things for us.
  • Use another solution - describe which functions of the viewset we are going to use in our views.

All right, so this is for categories - it’s one way of using the viewsets. We have multiple categories - we can do a Get and Post there - retrieve or post multiple categories. We can also have a detailed view. Now let's try the other option. (Which one?)

We could recreate it from the tutorial or use it for our "sellers" category. Basically, we bound the views manually, but we got the same outcome.

After running our server, we added a "seller" and a few categories to it.

Let’s go to products.

We can actually post the data here, but we don’t have the form, as we did it manually. This is our manual form, that’s why. We can actually start to create our products here. Now, we see our products.

Let’s go back to our views and create one more view which allows us to filter.  For this one, we will use a particular product serializer, and here will use a custom filter. Those two will use mostly by product frontend ones. We want to get products from one category or actually search for a product with a certain title. Once we actually create the URL for it, we will be able to use those functionalities.

That was very easy and now we can go to the "products" filter. There we can search for “Samsung” for example, but if we do - we won’t get any results. It’s a very simple way to filter data in a DjangoREST Framework.

That would be step one.

What we created here is a simple API so we can go to different endpoints. We wanted to show you different ways of creating viewsets. We have a custom methods viewset, a viewset that uses the route implemented. Then, we have the products and products with filters. Basically, we tried a little bit of everything.

However, for now, our API is pretty much nothing. Without our script and frontends, we don’t really know how to use the API.

The next steps are to create the Amazon Scraper and the Connector Script. It will have a script on its own side, and a new endpoint to append multiple products to our API.

We now have our very simple backend, so we focus on the scraper.

Before we continue, we usually would comment our code as a simple backend and push it.

Building the Amazon scraper

Right now we have a cool directory, let’s create one for the Amazon scraper. We most often have it in the backend directory.

For now, it’s an empty directory. Inside we will have the "scraper.py", we will have a config, or maybe a scraper config - "scraperconfig.py" and the API connector. Those are the three things we will have here. We can have the "init.py" so have it has a python package and also create the reports directory. This will be the directory for Amazon files from the scraper.

Let’s go first with the scraper. Here you can just watch the video that's only about the scraper. Now, we will use the code for it. Again, everything is in a GitHub repository - you can copy it from there.

Let’s implement it now.

So, again, this file is also from a tutorial, this is why we decided to go quickly through it.

You see the options of our web browser and the directory for our reports. We have categories:

  • Base URL
  • API Base URL - if you want to implement the script you have to change it.

Scraper config

You can find all the information about building the scraper in this tutorial.

Reporter class

Now let’s create the "reporter" class. For this directory, we need to get a global path so we always find the reports.

The "reporter" class is the one we will be running in the main file. This is the class that has the whole functionality of scraping. The scraper is a service that scrapes data, but we will include it in our "reporter" class. We have two options - generate one particular category or go all in and look for all the categories.

Generate report functionality

Let’s create the "generate report" functionality - another thing we’ve covered in our Amazon scraper tutorial.

The whole process of running the scraper is in this file. We are initializing a new scraper with the category and it’s running in the background. A cool thing is to create an asynchronous process at the same time.

Alright, so we have the data from "categories" here, now we need to generate a report.

We can now try running the script to look for "PS4"s - running the scraper only.

It took a little longer than we expected because we scraped 50 products. We can change it to 10 as well. Our next step would be to create functionality for all categories. In this case, we will have all of our reports/categories filled up with products. Then, we can connect the connector script.

Let’s run it.

Scraper script

Okay, we are ready to create the connector script. Now, we are going to go through all the JSON report files, grab their content, and then send POST requests to our server so we have it added to our database.

It’s a very common scenario when you have a scraper in a cloud for scraping the data, it's saving somewhere in JSON files, or it’s sending directly to the backend. In our case, we have this additional layer with savings. We want to make you familiar with such small services as they are a very common thing right now.

Let’s get to it.

API connector

Let’s go to API connector and start coding it. We have to iterate and therefore, get the data from every report and push it to the database. This means making a POST request with the data and the body of the request to our API endpoint.

Okay, so we need the files - "create report files" functions that go through the files, get the data, and send data from the files to the API. Here, we will make sure we return only JSON files. Now, let’s send data to the API.

Let’s get the data from the file now. Here you can see that we are using the JSON loads, so as we are doing un the scraper?, here we're using the JSON dump, so here we do the JSON loads. Once we get the data, we are sending it to the API.

Let’s now create a method for posting the data - posting from a single file - X requests when X is a number of categories/reports.

“Addproducts” endpoint

So, our API base URL is this (X). Now we are going to use a new endpoint called "addproducts". If we go to our URLs from the product, we don't have this endpoint yet. so we are adding it right now.

Let’s open some JSON, split the screen, and look at it.

We look at the body - we have a date for when it’s scraped, the category (e.g., "Laptops , e.g.), and the products.

Let’s make the requests now.

This would be our whole function here. Before we run it, we need the endpoint URLs.

Let’s create them now.

Let’s import the report view which we don’t have yet. Now, we are going to actually create it. Let’s do API view. Here, we need a POST method because we will only send data via this function.

Once we get the data from the request, we need to create some type of validation to make sure we get what we wanted. A simple validation. We need a function that gives us the object of the category - once we are adding the product, we need to pass the object of the category. We propose to actually use a Get or Create method. Once we get a new category that we don't have in a database, we create it while creating the products.

Let’s create the "handle products" function which will handle all the products from the JSON and add them to a database.

Let’s define the timezone as well.

Creating objects

Our objects will be unique values we will define a product through. If we find a product that has those 3 things the same as the new one we are adding, we are going to defaults and adding the product, or we are creating a new one.

So, let’s now return a response. This would be the response that we are sending back to our script. We have it created - let’s create a server.

Go to backend directory "python manage.py runserver".

This is a really important step. Once we go to "products", we have only 1 product from 1 category. Let’s give it a try - we have 1 product, 1 category - let’s run our script and see if we get new stuff.

Let’s go to API connector and run our script.

Now, we added some stuff via our API connector script.

Let’s go to product filters and do some filtering (e.g., iPhone). Boom - we have a few searches for “iPhone”.

At this point, we created the Backend API, DjangoREST Framework, the scraper, and the connection script. We now have a backend. Now, it's time for the frontend and integration. When these are done, we will have a full-stack app.

Creating the frontend

Let’s code the frontend. For this one, we will use Visual Studio Code.

For now, it’s empty. What we will need to have is a thing called a live server. You will need to install Live Server so you will be able to deploy. So, yeah, please do it.

Creating the "assets" folder

Let’s start with creating an ”assets” folder. In this directory, we will have two files - a banner, and a profile pic. You can get them on GitHub.

Let’s create the JS directory - an "index.js" and a "category.

Then, we will have a style directory - style.css.

Let’s create the index.html and the category.html.

What we are going to do is to use Live Server instead of a view or react. It’s a very simple solution to what we want. In this course we are focusing mainly on the logic of the solution - the backend.

For our /head HTML section we will have some simple settings - the app name, importing the CSS file, and importing some nice fonts as well.

Let’s create the body now and run the Live Server extension. Obviously, we still have nothing on our live site.

When it comes to styling, feel free to copy the whole CSS file and use it for your application. We are not going to describe every line of the frontend, as it’s not really the purpose of our course.

Now, we can see our basic page on Live Server.

Let’s continue with the categories. Basically, we will just copy the whole "categories" HTML file and use this. We will also have to create the search function there as well.

In order to have it all running, we will have to implement JavaScript.

Before doing that, we will need to make just one last change in the DjangoREST Framework.

After we go to our server and open the pycharm, we have to add the following:

  • CORS_ORIGIN_ALLOW_ALL = True
  • “Corsheaders.middleware.CorsMiddleware”

Those two basically allow us to do a request from the frontend without any problems.

Let’s go to index.js and actually implement our frontend and search functionality.

What we did is get the API URLs to the backend, the git categories - we executed on Load. Whenever we load the page, we are fetching our categories from the API, returning the JSON, and later building the categories from the JSON.

We have a function that builds the categories - we iterate over them. For each category, we create a picture on the page.

That’s it. Now we will ask the backend for each product of a certain category as well.

Let’s run it.

Once we click on Printer - we have the printers, and so on. As you can see, it all works and the "index.js" and "category.js" files are very similar.

On "category.js" we had to do a very basic query to ask the backend to fetch the data filtered by the category id.

Let’s try the search function. It also works because of the condition we set. We can either do a query or filter by category - one of the two functions will be returned and we will fetch the products.

This codebase gives you a ton of possibilities. You can expand it, host it on your favorite cloud, and present it as a resume project. This shows that you can build a full-stack application.

Also, you have implemented microservices such as the web scraper and the script that syncs the database and the scraper.

Need help?

In case you need help setting up your own full-stack Amazon gallery application or you want it configured differenly than what's shown in the tutorial, don't hesitate to get in touch.

Related posts

Keep learning