木鸟杂记

大规模数据系统

Python Closures

Posted on Edited on In ,

python-learn.pngpython-learn.png

Introduction

When I first learned about closures while studying JavaScript, I didn’t quite grasp the concept. For interviews, I memorized a vague “definition”: a function nested inside another function, with the inner function returned from the outer function, bringing the outer function’s environment along with it. At the time, I tried to understand closures from the literal meaning and some classic examples. The Chinese translation “闭包” (bìbāo, literally “closed package”) doesn’t easily convey the underlying principle, so I always felt a bit fuzzy about it. Recently, due to work requirements, I’ve been using Python and encountered closures again. This time, I came across some novel and interesting materials, which finally helped me connect several literal concepts (first-class functions, binding, scope, etc.) together and gain a deeper understanding of closures.

The reference materials are listed at the end; I highly recommend reading them.

Author: Muniao’s Notes https://www.qtmuniao.com, please indicate the source when reposting

Overview

Some English technical terms in computing, when translated literally, inevitably appear pale and awkward due to the lack of context. It takes much groundwork before one can appreciate the principles beneath the quirky surface. Closure is one such concept that involves a great deal of context, including the most fundamental concepts in programming languages: binding, environments, variable scope, and functions as first-class citizens.

Binding

In Python, binding is the most basic abstraction technique in a programming language. It binds a value to a variable, which can later be referenced or modified. Below are several levels of binding; each group of statements binds a name and its corresponding value to the environment where it is defined at runtime.

This includes binding a name to a block of memory via assignment statements. Of course, when a function is called, the binding of formal parameters to actual arguments is also a form of binding:

1
In [1]: square = 4

Binding a name to a composite operation, i.e., function definition, using the def keyword:

1
2
In [1]: def square(x):
        return x*x

Binding a name to a data collection, i.e., class definition, using class:

1
2
3
4
5
6
In [1]: class square:
            def __init__(self, x):
                self.x = x

            def value(self):
                return self.x * self.x

According to the execution order, multiple bindings with the same name will cause the later one to override the earlier one:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
In [1]: square = 3

In [2]: square
Out[2]: 3

In [3]: def square(x):
   ...:     return x * x
   ...:
   ...:

In [4]: square
Out[4]: <function __main__.square(x)>

In [5]: class square:
   ...:     def __init__(self, x):
   ...:         self.x = x
   ...:

In [6]: square
Out[6]: __main__.square

All of these are called abstractions because they provide encapsulation of data, composite operations, or data collections—tying a name to complex data or logic so that the caller doesn’t need to care about the implementation details, and using them as basic building blocks for more complex projects. It can be said that binding is the cornerstone of programming.

Returning to the topic of this article, a closure is first and foremost a function, just a special kind of function. As for what makes it special, I’ll explain after introducing a few more concepts.

Scope

Scope, as the name suggests, is the range that a binding can cover, or how far you can access a variable. Each function definition constructs a local scope.

Python, like most programming languages, uses static scoping rules (also known as lexical scoping). When functions are nested, the inner function can access variables from the outer function. Therefore, you can imagine scope as a container that can be nested, with inner scopes extending outer scopes, and the outermost scope being the global scope.

In the previous section, I mentioned that multiple bindings with the same name will cause the later one to override the earlier one. This has an implicit premise: within the same scope. In nested scopes, the relationship is actually one of hiding. A variable definition in an inner function will shadow the definition of the same name in the outer function, but in the outer scope, the variable retains its original value:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
In [16]: a = 4

In [17]: def outer(): 
    ...:     a = 5
    ...:     print(a)
    ...:     def inner():
    ...:         a = 6
    ...:         print(a)
    ...:     inner()
    ...:     print(a)
    ...:

In [18]: outer()
5
6
5

In [19]: print(a)
4

As you can see, scope can also be understood from another angle: in a given environment, when determining the value of a name binding, we search from the innermost scope outward, and the value corresponding to the first binding of that name we find is the value the name refers to.

It is worth emphasizing that nested function definitions cause nested scopes, or an environment extension relationship (the inner extends the outer). Class definitions are slightly different: a class definition introduces a new namespace. Namespace and scope are often compared, but I’ll leave that aside here; feel free to look it up if you’re interested.

Speaking of which, let me mention a commonly discussed example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
In [50]: a = 4

In [51]: def test():
    ...:     print(a) # What should this output?
    ...:     a = 5
    ...:

In [52]: test()
---------------------------------------------------------------------------
UnboundLocalError                         
Traceback (most recent call last)
<ipython-input-52-fbd55f77ab7c> in <module>()
----> 1 test()

<ipython-input-51-200f78e91a1b> in test()
      1 def test():
----> 2     print(a)
      3     a = 5
      4

UnboundLocalError: local variable 'a' referenced before assignment

One might expect the print statement above to output 4 or 5. Why does it throw an error? This is because when the test function is defined, the tokenizer scans all tokens within the function definition. Seeing the assignment statement a = 5, it determines that a is a local variable, so it won’t output 4. When execution reaches print(a), in the local environment, a has not yet been bound, hence the UnboundLocalError.

Digging a bit deeper: although Python is interpreted—meaning it reads, interprets, and executes one statement at a time—for code blocks (compound statements consisting of a header statement, a colon, and its associated indented block, such as function definitions, class definitions, loop statements, etc.), it still performs an overall scan first.

First-Class Functions

Generally, elements that make up a programming language, such as variables, functions, and classes, are subject to different restrictions. The elements with the fewest restrictions are called first-class citizens of that programming language. The most common privileges of first-class citizens are:

  1. Can be bound to a name
  2. Can be passed as an argument to a function
  3. Can be returned as the result of a function
  4. Can be contained within other data structures

Applying this to functions in Python: a function can be assigned to a variable, can be received and returned by a function, and can be defined inside another function (i.e., nested definition):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
In [32]: def test():
    ...:     print('hello world')
    ...:

In [33]: t = test # assign to a variable

In [34]: t()
hello world

In [35]: def wrapper(func):
    ...:     print('wrapper')
    ...:     func()
    ...:

In [36]: wrapper(t) # passed as an argument
wrapper
hello world

In [37]: def add_num(a): 
    ...:     def add(b): # nested definition
    ...:         return a + b
    ...:     return add # returned as the result of a function
    ...:
    ...:

In [38]: add5 = add_num(5)

In [39]: add5(4)
Out[39]: 9

Not all languages treat functions as first-class citizens. For example, Java does not have any of the four privileges listed above.

Here, functions that operate on other functions (i.e., functions that take other functions as arguments or return them as values) are called higher-order functions. Higher-order functions greatly enhance the expressive power of a language, but at the same time, they are not easy to use well.

Stack Calls

Every function call creates a frame in the environment, performs some bindings within that frame, and pushes it onto the function call stack. When the function call ends, the frame is popped off, the bindings within it are released (i.e., garbage collected), and the local scope ceases to exist.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
In [47]: def test():
    ...:     x = 4
    ...:     print(x)
    ...:

In [48]: test()
4

In [49]: x
---------------------------------------------------------------------------
NameError                                 
Traceback (most recent call last)
<ipython-input-49-6fcf9dfbd479> in <module>()
----> 1 x

NameError: name 'x' is not defined

That is, after the call ends, the locally defined variable x cannot be accessed from outside. But as in the earlier example, the returned add function references the variable a from add_num, whose call has already ended. How do we explain this phenomenon? Remember the rule I mentioned before:

1
When functions are nested, the environment of the inner function automatically extends the environment where it was defined

Therefore, after the outer function returns, the returned inner function still maintains the extended environment from when it was defined. In other words, because the inner function holds a reference, all bindings in the outer function’s environment are not reclaimed.

Closure

At long last, it appears. But the climax has actually already passed.

A closure is built upon all the concepts discussed above. The example mentioned earlier:

1
2
3
4
5
6
7
8
9
10
11
In [37]: def add_num(a): 
    ...:     def add(b): # nested definition
    ...:         return a + b
    ...:     return add # returned as the result of a function
    ...:
    ...:

In [38]: add5 = add_num(5)

In [39]: add5(4)
Out[39]: 9

This is a closure. Picking up the foreshadowing from earlier, here is my understanding of a closure: it is a higher-order function in which the outer function (add_num in the example) returns the function it defines internally (add), and because the returned inner function extends the outer function’s environment, it holds a reference to it. When the returned inner function (add5) is called, it can reference the external environment from when it was defined (in this example, the value of a).

Conclusion

After all this, I have only explained what a closure is at a logical or abstract level, and which concepts it is entangled with. But none of this gets to the essence; it is still a castle in the air. If you want to truly understand closures, you can delve into Python’s interpretation and execution mechanism, but that falls into the realm of compiler theory.

References

  1. cs61a course materials: composing programs, which is the companion course for the book SICP. The book is a classic, the course is excellent, and the materials are fascinating—well worth a read.
  2. A nice article I found via Google: A Python Tutorial To Understanding Scopes and Closures

我是青藤木鸟，一个喜欢摄影、专注大规模数据系统的程序员，欢迎关注我的公众号：“木鸟杂记”，有更多的分布式系统、存储和数据库相关的文章，欢迎关注。 关注公众号后，回复“资料”可以获取我总结一份分布式数据库学习资料。 回复“优惠券”可以获取我的大规模数据系统付费专栏《系统日知录》的八折优惠券。

我们还有相关的分布式系统和数据库的群，可以添加我的微信号：qtmuniao，我拉你入群。加我时记得备注：“分布式系统群”。 另外，如果你不想加群，还有一个分布式系统和数据库的论坛（点这里），欢迎来玩耍。

wx-distributed-system-s.jpg

不妨一读