Metaprogramming with Metaclasses in Python



In this article, we`re going to discuss metaclasses , why and when we should use them, and what the alternatives are. This is a fairly advanced Python topic and the following prerequisite is expected —

Note: This article is about Python 3.3 and up

Metaclasses

In Python, everyone has some type associated with it. For example, if we have a variable that has an integer value, then its type — int. You can get the type of anything using the type () function.

num = 23

print ( "Type of num is:" , type (num))

 

lst = [ 1 , 2 , 4 ]

print ( " Type of lst is: " , type (lst))

  

name = "Atul"

print ( "Type of name is:" , type (name))

Output:

 Type of num is: & lt; class `int` & gt; Type of lst is: & lt; class `list` & gt; Type of name is: & lt; class `str` & gt; 

Every type in Python is defined by a class . So in the above example, unlike C or Java, where int, char, float are the main data types, in Python they are objects of class int or str. Thus, we can create a new type by creating a class of that type. For example, we can create a new type Student by creating a class Student .

class Student:

pass

stu_obj = Student ()

 
# Print type of student class object

print ( " Type of stu_obj is: " , type (stu_obj))

Output:

 Type of stu_obj is: & lt; class `__main __. Student` & gt; 

A class is also an object , and like any other object, it is an instance of something called a metaclass . The special type class creates these class objects. The class of type is the default metaclass , which is responsible for creating classes. For example, in the above example, if we try to figure out the type of the class Student , it turns out to be a type .

class Student:

pass

 
# Print type Student class

print ( "Type of Student class is:" , type (Student))

Output:

 Type of Student class is: & lt; class `type` & gt; 

Since classes are also objects, they can be modified in the same way. We can add or subtract fields or methods in the class just like we did with other objects. For example —

# A specific class without any
# class methods and variables

class test: pass

 
# Defining method variables

test.x = 45

 
# Defining class methods

test.foo = lambda self : print ( `Hello` )

  
# object creation

myobj = test ()

 

print (myobj.x)

myobj.foo ()

Output:

 45 Hello 

This whole meta thing can be summarized as — Metaclass create Classes and Classes creates objects

The metaclass is responsible for generating classes, so we can write our own metaclasses to change the way we generate classes by performing additional actions or injecting code. We usually don`t need custom metaclasses, but sometimes we do.
There are problems for which metaclass and nonmetaclass solutions are available (often simpler), but in some cases only a metaclass can solve this problem. We will discuss such a problem in this article.

Creating our own metaclass

To create our own metaclass, our custom the metaclass must inherit the type metaclass and usually override —

  • __new __ (): is the method that is called before __init __ ( ). Creates an object and returns it. We can override this method to control the creation of objects.
  • __init __ (): this method simply initializes the created object passed as a parameter

We can create classes using the type () function directly. It can be called in the following ways —

  1. When called with only one argument, it returns a type. We`ve seen this before in the examples above.
  2. When called with three parameters, it creates a class. The following arguments are passed to it —
    1. Class name
    2. A tuple having base classes inherited from the class
    3. Class dictionary: serves as local a namespace for a class filled with class methods and variables.

Consider this example —

Output:

 Type of Test class: & lt; class `type` & gt; Type of test_obj: & lt; class `__main __. Test` & gt; This is inherited method! This is Test class method! atul 

Now let`s create a metaclass without using type () directly. In the following example, we will create a MultiBases, metaclass that will check if the generated classes inherit from multiple base classes. If so, it will throw an error.

def test_method ( self ):

print ( "This is Test class method!" )

 
# create base class

class Base:

def myfun ( self ):

print ( " This is inherited method! " )

  
# Create Test class dynamically
# type () method directly

Test = type ( `Test` , (Base,), dict (x = " atul " , my_method = test_method))

 
# Test print type

print ( "Type of Test class: " , type (Test))

 
# Create an instance of the Test class

test_obj = Test ()

print ( "Type of test_obj:" , type (test_obj))

 
# calling inherited method
test_obj.myfun ()

 
# calling the Test class method
test_obj. my_method ()

 
# print variable

print (test_obj.x)

# our metaclass

class MultiBases ( type ):

# overriding the __new__ method

def __ new __ ( cls , clsname, bases, clsdict):

# if there are no base classes greater than 1

# raising error

if len (bases) & gt; 1 :

raise TypeError ( " Inherited multiple base classes !!! " )

 

# execute the superclass __new__ method, ie

  # call __init__ of type class

return super () .__ new __ ( cls , clsname, bases, clsdict)

 
# metaclass can be specified using the metaclass keyword
# now the MultiBase class is used to create i classes
# this will be propagated to all Base subclasses

class Base (metaclass = MultiBases):

pass

 
# no error occurs

class A (Base):

pass

 
# no error occurs

class B (Base):

pass

 
# This will throw an error!

class C (A, B):

pass

Output:

 Traceback (most recent call last): File "& lt; stdin & gt;", line 2, in & lt; module & gt; File "& lt; stdin & gt;", line 8, in __new__ TypeError: Inherited multiple base classes !!! 

Solving the metaclass problem

There are some problems that can be solved by both decorators (easily) and metaclasses. But there are several problems that can only be achieved with metaclasses. For example, consider a very simple code repetition problem.

We want to debug class methods, we need to ensure that whenever a class method is executed, it will print the fully qualified name before executing its body.

The very first solution that comes to our mind is — this is using decoration methods , below is some code example:

from functools import wraps

 

def debug (func):

& # 39; & # 39; & # 39; decorator for debugging the passed function & # 39; & # 39; & # 39;

 

@ wraps (func)

  def wrapper ( * args , * * kwargs):

print ( "Full name of this method:" , func .__ qualname__)

return func ( * args, * * kwargs)

return wrapper

  

def debugmethods ( cls ):

class decorator & # 39; & # 39; uses the debug decorator

to debug class methods & # 39; & # 39; & # 39;

 

# check in the class dictionary for any called (method)

# if it exists, replace it with the debugged version

for key, val in vars ( cls ). items ():

if callable (val):

setattr ( cls , key, debug (val))

return cls

  
# sample class
@ debugmethods

class Calc:

def add ( self , x, y):

return x + y

def mul ( self , x, y):

return x * y

def div ( self , x, y):

return x / y

 

mycal = Calc ()

print (mycal.add ( 2 , 3 ))

print ( mycal.mul ( 5 , 2 ))

Output:

 Full name of this method: Calc.add 5 Full name of this method: Calc.mul 10 

This solution works great, but there is one problem: what if we want to apply this method decorator to all subclasses that inherit this class Calc . In this case, we must separately apply the decorator method to each subclass, as we did with the Calc class.

The problem is that if we have there are many such subclasses, then in this case we will not like adding a decorator to each separately. If we know in advance that every subclass should have this debug property, then we should look for a metaclass solution.

Take a look at this metaclass solution , the idea is that the classes will be generated normally and then immediately wrapped with the debug method decorator —

from functools import wraps

 

def debug (func):

& # 39; & # 39; & # 39; decorator for debugging the passed function & # 39; & # 39; & # 39;

 

@ wraps (func)

  def wrapper ( * args , * * kwargs):

print ( "Full name of this method:" , func .__ qualname__)

return func ( * args, * * kwargs)

return wrapper

  

def debugmethods ( cls ):

class decorator & # 39; & # 39; uses the debug decorator

to debug class methods & # 39; & # 39; & # 39;

 

for key, val in vars ( cls ). items ():

if callable (val):

setattr ( cls , key, debug (val))

return cls

  

class debugMeta ( type ):

& # 39; & # 39; & # 39; metaclass that passes the generated class object

debug method to get debug functionality

allowed objects & # 39; & # 39; & # 39;

 

def __ new __ ( cls , clsname, bases , clsdict):

obj = super () .__ new __ ( cls , clsname, bases, clsdict)

obj = debugmethods (obj)

return obj

 
# base class with debugMeta metaclass
# now all subclasses of this
# debugging will be applied

class Base (metaclass = debugMeta): pass

 
# base inheritance

class Calc (Base):

def add ( self , x, y):

return x + y

 
# Calc inheritance

class Calc_adv (Calc):

def mul ( self , x, y ):

return x * y

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
# Now the Calc_adv object shows
# debugging behavior

mycal = Calc_adv ()

print (mycal.mul ( 2 , 3 ))

Output:

 Full name of this method: Calc_adv.mul 6 

When to use metaclasses

Most of the time we do not use metaclasses, they are like black magic and usually for something complex, but there are few cases where we use metaclasses:

  • How do we seen in the above example, metaclasses propagate down the inheritance hierarchy. This will affect all subclasses as well. If this is our situation, then we have to use metaclasses.
  • If we want to change the class automatically when it is created
  • If you are an API developer, you can use metaclasses

According to Tim Peters

Metaclasses are deeper magic that 99% of users should never worry about. If you wonder whether you need them, you don`t (the people who actually need them know with certainty that they need them, and don`t need an explanation about why).

Links

This article courtesy of Atul Kumar . If you are as Python.Engineering and would like to contribute, you can also write an article using contribute.python.engineering or by posting an article contribute @ python.engineering. See my article appearing on the Python.Engineering homepage and help other geeks.

Please post comments if you find anything wrong or if you`d like to share more information on the topic discussed above.