木鸟杂记

大规模数据系统

Python Mix-Ins

Posted on Edited on In ,

python-learn.pngpython-learn.png

Introduction

Once when using Python’s socketserver, I came across ForkingMixIn and ThreadingMixIn. I was fascinated by this plug-in style syntactic sugar. Recently, while writing my own code, I also wanted to create some of these plug-and-play plugin code snippets, so I explored Python’s mix-in mechanism.

Simply put, it is a technique that leverages multiple inheritance to stylishly enhance an original class by plugging in additional code snippets.

Author: 木鸟杂记 https://www.qtmuniao.com, please indicate the source when reprinting

Key Points

To get straight to the point, here are the conclusions. When using MixIn, there are just a few key points to keep in mind:

  1. A Mix-in class is essentially a code snippet and cannot survive on its own.
  2. Enhance functions with the same name in “sibling” [1] classes to achieve a plug-and-play effect.
  3. When defining a subclass, the base class inheritance order cannot be arbitrary; the MixIn class must come before the class being mixed into.

Code Snippet

Unlike static languages, Python uses dynamic binding. Therefore, at definition time, it can independently “exist” as a code snippet (but cannot “survive,” meaning it cannot be used to define class instances for binding), and can reference functions that do not exist in the snippet:

1
2
3
4
5
6
7
8
9
10
class SetOnceMappingMixin:
    '''
    Only allow a key to be set once.
    '''
    __slots__ = ()

    def __setitem__(self, key, value):
        if key in self:
            raise KeyError(str(key) + ' already set')
        return super().__setitem__(key, value)

This code snippet obviously cannot be instantiated on its own, but defining it separately does no harm, and it assumes that the class being mixed into has __setitem__.

Multiple Inheritance Replacement

Since it is plug-and-play, the API of the class being mixed into [2] remains unchanged whether the Mixin is present or not. The principle behind this mechanism is using a function with the same name to replace the original function. Mixin emphasizes its role as a plugin—that is, adding extra functionality on top of the original function implementation. To achieve this, the original function must be reused. In Python, the function that serves this purpose is super().

In programming languages that only allow single inheritance, such as Java, super() is unambiguously a reference to the parent class. But for Python, which supports multiple inheritance, where super() ultimately points to requires some arrangement. Consider the following example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
class Base:
    def __init__(self):
        print('Base.__init__')

class A(Base):
    def __init__(self):
        super().__init__() # 1
        print('A.__init__')

class B(Base):
    def __init__(self):
        super().__init__() # 2
        print('B.__init__')

class C(A,B):
    def __init__(self):
        super().__init__() 
        print('C.__init__')

if __name__ == '__main__':
    c = C()

You can guess what the output will be, then comment out #1 and #2 to see what the output is.

The reason is that for each class, Python computes a Method Resolution Order (MRO) list. The super().func() function traverses this list from front to back, finding the first class that has the func function and then calling it. The rule for constructing the MRO list is quite simple; loosely speaking, it is from left to right, from bottom to top.

Therefore, the MRO list for class C is [3]:

1
2
3
4
>>> C.__mro__
(<class '__main__.C'>, <class '__main__.A'>, <class '__main__.B'>,
<class '__main__.Base'>, <class 'object'>)
>>>

However, if the class inheritance is extremely messy—for example, if a circular dependency occurs—an error will be raised when accessing C.__mro__: TypeError: Cannot create a consistent method resolution. I won’t go into detail here; interested readers can Google it themselves.

Returning to mix-in classes, with this understanding, it becomes clear why the mix-in class must be defined before the class being mixed into when defining a subclass: so that the mix-in class can call the function with the same name in the class being mixed into via the super() method.

A complete example [4] is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class SetOnceMappingMixin:
    '''
    Only allow a key to be set once.
    '''
    __slots__ = ()

    def __setitem__(self, key, value):
        if key in self:
            raise KeyError(str(key) + ' already set')
        return super().__setitem__(key, value)
    
class SetOnceDefaultDict(SetOnceMappingMixin, defaultdict):
    pass


d = SetOnceDefaultDict(list)
d['x'].append(2)
d['x'].append(3)
# d['x'] = 23  # KeyError: 'x already set'

Notes

[1] The “sibling” relationship here refers to the relationship between two classes in multiple inheritance. For example, in class Child(A, B): pass, classes A and B are siblings.

[2] The class being mixed into is the “sibling” of the Mixin. For example, in class Strong(Week, PowerMixIn), Week is the class being mixed into for PowerMixin.

[3] Calling Methods on Parent Classes: https://python3-cookbook.readthedocs.io/zh_CN/latest/c08/p07_calling_method_on_parent_class.html

[4] Extending Classes with Mixins: https://python3-cookbook.readthedocs.io/zh_CN/latest/c08/p18_extending_classes_with_mixins.html

我是青藤木鸟，一个喜欢摄影、专注大规模数据系统的程序员，欢迎关注我的公众号：“木鸟杂记”，有更多的分布式系统、存储和数据库相关的文章，欢迎关注。 关注公众号后，回复“资料”可以获取我总结一份分布式数据库学习资料。 回复“优惠券”可以获取我的大规模数据系统付费专栏《系统日知录》的八折优惠券。

我们还有相关的分布式系统和数据库的群，可以添加我的微信号：qtmuniao，我拉你入群。加我时记得备注：“分布式系统群”。 另外，如果你不想加群，还有一个分布式系统和数据库的论坛（点这里），欢迎来玩耍。

wx-distributed-system-s.jpg

不妨一读