Change language

How to handle typing in Python

| |

The first mention of type hints in the Python programming language appeared in the Python Enhancement Proposals database (PEP-483). These type hints are needed to improve static code analysis and autocompletion by editors, which helps reduce the risk of bugs in code.

In this article we will cover the basics of Python code typing and its role in a dynamically typed language, this information will be most useful for novice Python developers.

Typing in Python

The types themselves are used to denote the basic types of variables:

  • str
  • int
  • float
  • bool
  • complex
  • bytes
  • etc.

An example of using base types in a Python function:

def func(a: int, b: float) -> str:
    a: str = f"{a}, {b}"
    return a

In addition, more complex types, such as List, can be parameterized. These types can take parameter values that help describe the function type more precisely. For example, List[int] indicates that a list consists of integer values only.

Example code:

from typing import List

def func(n: int) -> List[int]:
    return list(range(n))

Besides List, there are other types from the typing module that can be parameterized. These types are called Generic types. These types are defined for many of Python's built-in data structures:

  • Set[x].
  • FrozenSet[x].
  • ByteString[x].
  • Dict[x, y]
  • DefaultDict[x, y]
  • OrderedDict[x, y]
  • ChainMap[x,y]
  • Counter[x, int]
  • Deque[x]
  • etc.

As you can see, some types have several parameters that can be described. For example, Dict[x, y] means it will be a dictionary, where keys will have type x, and values will have type y.

There are also more abstract types, such as:

  • Mapping[x, y] - the object has implementations of the getitem method;
  • Iterable[x] - object has an implementation of method iter.

Functions also have their own types. For example, the Callable type can be used to describe a function, where the types of input parameters and return values are specified. Example usage:

from typing import Callable

def func(f: Callable[[int, int], bool]) -> bool:
    return f(1,2)

func(lambda x, y: x == y)
>>> False

The Callable type:

  • Indicates that the object has a call method implemented;
  • Describes the types of parameters to this method.

The first place is an array of input parameter types, the second is the return value type.

You can read about the other abstract container types in the Python documentation.

There are also more specific types, for example Literal[x], where x specifies a particular value rather than a type. For example Literal[3] means number 3. This type is rarely used.

Python also allows you to define your own Generic types.

from typing import TypeVar, Generic

T = TypeVar('T')

class Stack(Generic[T]):
    def __init__(self) -> None:
        # Create an empty list with items of type T
        self.items: List[T] = []

    def push(self, item: T) -> None:
        self.items.append(item)

    def pop(self) -> T:
        return self.items.pop()

    def empty(self) -> bool:
        return not self.items

In this example, TypeVar means a variable of any type that can be substituted when specified. For example:

def func(stack: Stack[int]) -> None:
     stack.push(11)
     stack.push(-2)

s = Stack[int]()
func(s)
s.empty()
>>> False

s.items
>>> [11, -2]

To define your own types, it is possible to inherit not only from Generic, but also from other abstract types, such as Mapping, Iterable.

from typing import Generic, TypeVar, Mapping, Iterator, Dict

KeyType = TypeVar('KeyType')
ValueType = TypeVar('ValueType')

class MyMap(Mapping[KeyType, ValueType]):  # This is a generic subclass of Mapping
    def __getitem__(self, k: KeyType) -> ValueType:
        ...  # Implementations omitted
    def __iter__(self) -> Iterator[KeyType]:
        ...
    def __len__(self) -> int:
        ...

In place of KeyType or ValueType can be specific types.

There are also special constructs that allow you to combine types. For example, Union[x, y, …] is one type. If a variable can be both int and float, you should specify Union[int, float] as the type. If the variable can be both int and None, you can specify Union[int,None] or, preferably, Optional[int] as the type.

Why this is needed

The purpose is to point the developer to the expected data type when receiving or returning data from a function or method. This, in turn, allows you to reduce the number of bugs, speed up code writing and improve its quality.

Suppose you have a user class and a function that converts json to User.

from typing import Dict, Union, Optional

from dataclasses import dataclass

@dataclass
class User:
    name: str
    surname: str
    age: int

def get_user_from_json(json_dict: Dict[str, Optional[Union[int, str]]]) -> User:
    name = json_dict.get("name")
    surname = json_dict.get("surname")
    age = json_dict.get("age")
    if (age is None or
        name is None or
        surname is None):
        raise ValueError("Not enough information")
    return User(age=age, name=name, surname=surname)

Of course, you can write a simpler one:

def get_user_from_json(json_dict: Dict[str, Optional[Union[int, str]]]) -> User:
    return User(age=json_dict["age"], name=json_dict["name"], surname=json_dict["surname"])

However, in both cases an error might occur if the age key is present and yet has a string type. Type validation doesn't add very many lines of code, but with a large number of models it can take up a lot of space in a project.

Using Pydantic helps to validate the data correctly, and the type will automatically change to the required type.

from pydantic import BaseModel

class User(BaseModel):
    name: str
    surname: str
    age: int


def get_user_from_json(json_dict: Dict[str, Optional[Union[int, str]]]) -> User:
    return User(**json_dict)

get_user_from_json({
    "name": "ssa",
    "surname": "ddd",
    "age": 10
})
>>> User(name='ssa', surname='ddd', age=10)

get_user_from_json({
    "name": "ssa",
    "surname": "ddd",
    "age": "10"
 })
>>> User(name='ssa', surname='ddd', age=10)

get_user_from_json({
    "name": "ssa",
    "surname": "ddd",
    "age": "d"
})
--------------------------------------
ValidationError: 1 validation error for User
age
 value is not a valid integer (type=type_error.integer)

As you can see, the stricter typing of the code helps to make it simpler and safer. However, using some of Pydantic's features can have an undesirable effect on the code. For example, data mutation during validation can cause the model value type to be unintelligible. For example:

from pydantic import BaseModel, validator

class User(BaseModel):
    name: str
    age: int

    @validator('age')
    def validate_age(cls, value):
        if int(value) < 10:
            raise ValueError("too low")
        return str(value)

User(name='Brian', age=33)
>>> User(name='Brian', age='33')

In this example, the User created after validation will have a different value than the one specified in the model. This leads to possible major bugs that are best avoided at all times.

Also, the FastAPI framework is now gaining popularity, which, thanks to Pydantic, allows you to quickly write web applications with automatic data validation.

from fastapi import FastAPI
from typing import Optional
from pydantic import BaseModel

app = FastAPI()

class Item(BaseModel):
    name: str
    price: float
    is_offer: Optional[bool] = None

@app.put("/item")
async def put_item(item: Item):
    return {"item_name": item.name, "item_price": item.price}

In this example, the /item endpoint automatically validates the incoming json and passes it to the function as the required model.

Also, to reduce bugs, mypy is used, which allows you to statically analyze code for type matching. This often avoids obvious bugs or type mismatches in functions.

And as a bonus for those who are lazy to support typing manually. MonkeyType allows you to type all functions automatically, although after you run this program you should usually walk through the code and fix some values that aren't detected as expected.

What's New About Python 3.9.0

Starting with the recently released version of Python 3.9, developers no longer have to import abstract collections to describe types. Now dict[x,y] can be used instead of typing.Dict[x,y], the same goes for Deque, List, Counter, etc. You can read a full description of this new feature here: PEP-585.

We have also added type annotations which can later be used by static analysis tools. variable: Annotated[T, x] where T is the variable type and x is some metadata for the variable. According to some authors these metadata can also be used at runtime (see PEP-593 for details).

Conclusion

In this article we have looked at some types in Python. In conclusion, we note that typed code in Python becomes much more readable and obvious, which helps with team review and avoid silly mistakes. A good type description also allows developers to get into the project faster, understand what's going on, and immerse themselves in tasks. Also, by using certain libraries, it is possible to reduce the number of lines of code that were previously required only for validating types and values by several times.