Webservices et formats de données

TP 3

Before you start

😱 As you can see, the subject of this practical assignment is also quite extensive. Don’t let that scare you.

It combines comprehensive explanations and practical exercises to ensure maximum self-sufficiency.

You probably won’t finish the whole subject, but that’s okay. It’s there to help you with your IT project.

Objectives

In this lab, you will:

• Manually call a web service using an API Client
• Call a web service using the Python library requests
• Discover the Swagger page of a web service
• Manipulate different data formats
• Create a web service using the Python framework FastAPI

1 Calling manually a web service

The first part of this practical assignment does not require you to write any code, but only to make requests to a web service using an API client.

1.1 Webservices

Web service

Web service: The term web service is a broad one, and it would be difficult to provide a concise definition (Wikipedia article).

Within the scope of the project, a web service refers to an application accessible via the HTTP (Hypertext Transfer Protocol) protocol, which generally complies with the REST (Representational State Transfer) architecture.

But there are others, such as SOAP (Simple Object Access Protocol) and RPC (Remote Procedure Call).

In other words, a web service is an application accessible via the web that can be queried either:

• to obtain resources
• to modify accessible resources

A web service can simply be a single, standardised access point to data (such as a database interface), but it can also be a way of controlling an information system (launching jobs, putting them on hold, retrieving results, etc.).

Web services use the HTTP protocol, which is the web protocol (not the internet protocol). It is the one you use without knowing it with your web browser. Requesting a web service is almost like requesting a web page.

To do this, you need:

• the address of the resource, its Uniform Resource Identifier (URI)
  • this is a more general concept than Uniform Resource Locator (URL)
• a method (GET, POST, PUT, DELETE, list of methods)
• and potentially some data

1.2 First GET requests

Tip

In this example, we will use a GET request to read data from a web service. A GET request is like asking a question: “Do you have this information?”

If the data exists, the web service responds with it, usually in JSON format. If not, it may return an error like 404 Not Found.

Several ways to make GET requests to web services (choose one):

• https://hoppscotch.io/ (Online client)
• Bruno (download and install)
• Insomnia (already installed on ENSAI VMs)
• Entering the URL in the address bar of your web browser (Chrome, Firefox…)
• command-line cURL : curl -X GET "https://swapi.info/api/people/35"

Test the web service requests below:

• Look at the response on the right side of your screen
• What are the similarities between the responses?

1.2.1 Carbon intensity

Web service on carbon emissions in the United Kingdom

• Documentation
  • Endpoints
  • Schemas

Requests to test:

• api.carbonintensity.org.uk/intensity
• api.carbonintensity.org.uk/intensity/date/{date}
  • replacing {date} with the date of your choice in the format YYYY-MM-DD

1.2.2 Rennes Métropole open data

Data Rennes Métropole

Retrieve data from various datasets: https://data.rennesmetropole.fr/api/explore/v2.1/catalog/datasets/<dataset_id>/records

Replace <dataset_id> with one of the following values:

• menus-cantines
• arbre
• aire_jeu
• jeux_aires_de_jeux

To vary the number of rows you receive:

• simply add to the end of the URI ?rows=N , where N is the number of rows

1.2.3 Your project’s web service

• Explore some useful methods of your IT project’s web service (discuss this with your tutor).

1.3 Advanced queries

You will use a web service created with FastAPI specifically for this purpose.

Tip

If you want to create this web service yourself using this code : https://github.com/ludo2ne/ENSAI-2A-WS-TP3/

• Create a PostgreSQL service
• Using cloudBeaver create and pop table
• Create a Vscode-python service
  • Network access : For example, open port 5000
  • Clone the repo and create .env file to connect your PostgreSQL database
• python main.py

.env

PASSWORD=
HOST=
PORT=5432
DATABASE=
USER=

After exploring a few external web services, you will work with a web service created specifically for these practical exercises.

Still using your API Client, query the following endpoints:

• GET /attack
• GET /attack/<identifier>
  • replacing identifier with the name or ID of an attack you have just retrieved
• GET /attack?type_attack_id=<id_type>
  • replacing id_type with an integer between 1 and 4
• GET /attack?type_attack_name=<type attack>
  • type attack: special attack, physical attack, fixed damage or status attack

Warning

If you have any issues with the API Client, you can connect directly to Swagger (see instructions below) to execute requests.

Let’s create an attack:

• POST /attack
  • Click on Body, then application/json
  • Paste the text below
  • Custom your own attack replacing attribute values
  • attack_type: physical attack / “special attack” / “fixed damage” / “status attack”

application/body

{
  "name": "<An awesome name>",
  "attack_type": "<An awesome type>",
  "power": 0,
  "accuracy": 0,
  "element": "<An awesome element>",
  "description": "<An awesome description>"
}

• Check that your attack has been created successfully.
  • GET /attack/<attack_name>
  • replacing attack_name with the name of the attack you just created

1.4 Swagger

In your web browser, go to the endpoint /docs.

This will take you to the webservice’s swagger page. This page lists all the webservice endpoints and how to use them. Try using the interface to:

• modify an attack
• delete an attack
• display a list of Pokémon
• add a Pokémon

Further documentation is available at the /redoc endpoint.

2 Python Webservice

Today, the biggest consumers of web services are machines. We will see how to automate calls to a web service in Python.

Note

Today, many web applications (e.g. Facebook, Netflix, Dailymotion, Uber) use what are known as “micro services” architectures.

Communication between their application components (e.g. between their human-machine interface (HMI) and their internal services) is carried out via single-purpose web services. This allows modules to be decoupled from one another, as they communicate solely via HTTP requests or event management systems. They only need to know how to communicate with one another, not the internal workings of other modules.

This requires thorough documentation of web services and the management of a HUGE number of applications in parallel. Amazon, Google and Facebook can afford to do this, but a small company with 10 employees cannot.

2.1 Requests Library

The principle remains the same as making a request manually with an API client. You will use the requests library so that you only have to fill in the relevant parts of your requests.

To make a GET request, simply do the following:

import requests

response = requests.get("http://my-webservice.com") 

Executing this line of code will:

1. Send the GET request to the server you are contacting
2. Store the result in the response variable

This response variable is an object, and like any object, it has attributes and methods, for example:

• response.text: the body of the result in string form, letting requests infer the encoding (this often works)
• Problem: you have a string, and this is not the best data format to work with
• response.json(): the body of the result as a dictionary.
  • This is what you will do most often because the JSON format is easy to manipulate
• response.encoding: the encoding of your request (useful in case of encoding problems)
• response.status_code: the status of the request. The main ones are:

Code	Description
200	General return to say that everything went well
201	Resource created successfully
202	Accepted, with no guarantee of the result (asynchronous system)
400	Bad request (Syntax error)
401	Error, authentication required
403	Resource prohibited (insufficient rights)
404	Resource not found
405	Method Not Allowed
500	Server-side error
503	Service temporarily unavailable

To summarise, the results:

• ✅ 2xx (success): the client’s request was successfully received, understood, and accepted
• ❌ 4xx (Client Errors): this could be due to incorrect syntax, unauthorized access, or the resource not being found
• ❌ 5xx (Server Errors): the server failed to fulfill a valid request

2.2 Exemples d’utilisation

The principle will always be the same:

1. You launch the query
2. You check if the status is OK
3. You retrieve the JSON and process the result (display, conversion to dataframe, etc.)

Template

import requests
import json

response = requests.get("http://my-webservice.com")

if response.status_code != 200:
    raise Exception(
        f"Cannot reach (HTTP {response.status_code}): {response.text}"
    )
else:    
    print(json.dumps(response.json(), indent=2))       # JSON Pretty print

Menus-cantines

import requests
import pandas as pd

url = "https://data.rennesmetropole.fr/api/explore/v2.1/catalog/datasets/menus-cantines/records?limit=20"
req = requests.get(url)

if req.status_code == 200:
    raw_menus = req.json()
    records = raw_menus.get("results")

    if records:
        menus = pd.DataFrame(records)
        print(menus)
    else:
        print("No records found.")
else:
    print(f"Failed to retrieve data. Status code: {req.status_code}.\n{req.text}")

2.3 Tour lab repo

Caution

You will now code on the fork you created in TP2. If you did not do so last time, create a fork.

• Log in to the datalab
• Open a VSCode-python service
  • init script: https://raw.githubusercontent.com/ludo2ne/ENSAI-2A-Projet-info/refs/heads/main/scripts/init-vscode.sh
  • Open port 5000
• Clone your repo
• Open Folder /home/onyxia/work/<repo_name>
• To ensure that everyone starts at the same level, position yourself on the branch tp3-base
  • git switch tp3-base

Verify that everything is working

• Run the unit tests for the business_object package
  • python -m pytest -v src/tests/test_business_object/
• Run the main program : python src/main.py

2.4 Mes premières requêtes en Python

• Open the file /src/client/attack_client.py
• Complete the get_attack(int) method, which:
  • takes an attack ID as a parameter
  • retrieves all available information about this attack
  • returns an object of type AbstractAttack

Tip

To help you, take a look at the instantiate_attack() method of the AttackFactory class (business_object).

See how this method works and use it.

• Create the get_all_attacks() method
• Returns the list of all available attacks in the form of a list of AbstractAttack objects
• Verify that your two methods are functioning
• Run the unit tests for the test_client package

2.5 More complex requests

So far, we have focused on GET queries (simple data reading).

But it is of course possible to do others. For example, POST, PUT or DELETE queries.

Here is the syntax:

post = requests.post("http://example.org", json = {'key':'value'})
put = requests.put("http://example.org", json = {'key':'value'})
delete = requests.delete("http://example.org")

As you can see, the syntaxes are very similar to the syntax of the GET method. We have only added data for certain requests. This is what you did earlier with the API Client. To pass parameters to your request, I recommend using this type of syntax:

# Prepare Arguments
url = "http://example.org"
headers = {'accept': 'application/json'}
data = {'key1':'value1', 'key2':'value2'}

# Launch Request
post = requests.post(url, headers=headers, json = data)

Functionally, it’s the same thing, but it’s better to define the elements outside the query so you don’t get lost.

2.6 Advanced requests in Python

In the attack_client.py module, implement the following methods:

• create_attack(AbstractAttack)
  • takes an AbstractAttack as a parameter
  • creates a new resource in our web service
• update_attack(AbstractAttack)
  • takes an AbstractAttack as a parameter
  • modifies the associated resource in our web service
• delete_attack(AbstractAttack)
  • takes an AbstractAttack as a parameter
  • deletes the associated resource in our web service
• Test your methods

3 Code a web service in Python

Note

Until now, you have been using existing web services. Now you are going to create your own web service.

With the tools available today, it is easy to create a web service yourself.

There are currently three market leaders for creating a REST web service in Python:

• Django REST
• FlaskRESTful
• FastAPI

Each has its advantages and disadvantages. Django is certainly the most comprehensive but also the heaviest, while Flask and FastApi are lighter and quicker to set up. The big advantage of FastApi is the simplicity with which you can create a Swagger documentation page.

Here is the minimal code for a REST web service with FastAPI (official documentation)

app.py

from fastapi import FastAPI

# Instantiate the web service
app = FastAPI()

# Create an endpoint that responds to the GET method at the address "/" and returns the message "Hello World".
@app.get("/")
async def root():
    return {"message": "Hello World"}

# Launching the application on port 5000
if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=5000)

Calling the “/” resource of the web service will return the JSON: {"message": "Hello World"}

Here is a more complete example inspired by the official documentation (you want to create a web service to expose your todos)

from fastapi import FastAPI
from pydantic import BaseModel
from starlette import status
import uvicorn

# On instancie le webservice
app = FastAPI()

class Todo(BaseModel):
    id : int
    content : str

todos = {1 : Todo(1,"Step 1 : Learn python")
        , 2 : Todo(2,"Step 2 : Work on the IT project")
        , 3 : Todo(3,"Step 3 : ???")
        , 4 : Todo(4,"Step 4 : Profit")}

# Endpoint GET /todo
@app.get("/todo")
async def get_all_todo():
    return todos.values()

# Endpoint GET /todo/{id_doto}
@app.get("/todo/{id_toto}")
async def get_todo_by_id(id_toto : int = Path(..., description = "The `id` of the todo you want to get")):
    if todos.get[id_toto] :
        return todos.get[id_toto]
    else :
        return JSONResponse(status_code=status.HTTP_404_NOT_FOUND)

# Endpoint POST /todo
@app.post("/todo", todo, status_code=201)
async def post_todo(todo:Todo):
    if not todos.get(todo.id):
        return JSONResponse(status_code=status.HTTP_409_CONFLICT)
    else :
        todos[todo.id] = todo
        return todo

# Launch the web service
if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=5000)

This code will create a web service that will respond to the following requests:

• GET host/todo: returns all tasks to be done
• GET host/todo/{todo_id}: returns the task behind the ID in the parameter
• POST host/todo/: adds the task passed in the body of the request

FastAPI serialises the objects you return for you. So there’s no need to format your data. However, for clarity, you can use BaseModel classes. These are classes that will only contain attributes that you can declare without a constructor:

class Todo(BaseModel):
    id : int
    content : str

These classes can be used as output from your web service, as well as input (line 33). FastApi will perform a series of checks on the variable types and return an error to the client if their request is not formatted correctly.

Basically, a web service is an application like any other, but instead of having a graphical interface as we humans are used to, the interface is an HTTP interface that will accept requests and send results. Thus, the sequence diagram of the different layers that will be involved in a GET request to retrieve a resource will look like this. Let’s go back to the 3-layer model we saw in class.

sequenceDiagram
    participant U as User
    participant R as Webservice
    participant S as Service
    participant D as DAO
    participant B as Database
    U ->> R: HTTP request
    R ->> S: get_by_id()
    S ->> D: find_by_id()
    D ->> B: SQL query (psycopg)
    B ->> D: SQL cursor (psycopg)
    D ->> S: business object instance
    S ->> R: business object instance
    Note over S,R: the object is potentially altered
    R ->> U: HTTP response

3.1 My first web service

Tip

The web service you will create will be deployed on a specific url. To find this url:

• In the datalab > My services
• VScode-python > Open > your will find a link

• Open the app.py file
• Run this file
• Test the following requests:
• GET hello
• GET hello/everybody
• Stop the web service
  • Click in the VSCode terminal, then press CTRL + C

Using the list of characters defined in the app.py file, add the following endpoints:

• GET character: will return a JSON containing a list of characters
• PUT character/{id}, which will modify the name of the character at index {id} from a body
• DELETE character/{id}, which will delete the item at index {id}

To test endpoints requiring a JSON body, you can use:

{
  "nom":"Agneta",
  "age": 30
}

End of the Lab

Knowing how to use a web service is essential, as they represent a vast source of data accessible in real time. Web services allow you to collect information from multiple sources, such as public APIs or internal systems, without having to manually download files. It is then up to you to convert this data into your preferred format.

Important

• Push your code to GitHub
• Pause or delete your datalab services