Victor Stinner

Python 3.0 was released 10 years ago. It's time to look back: analyze the migration from Python 2 to Python 3, see the progress we made on the language, list bugs by cannot be fixed in Python 2 because of the backward compatibility, and discuss if it's time or not to bury Python 2.

Python became the defacto language in the scientific world and the favorite programming language as the first language to learn programming.

Draft of the talk:

• Growing populary of the Python programming language

  • Defacto language in the scientific world, replacing other closed source and more specialized but limited language
  • Favorite programming language used as the first language to learn programming

• Port Python 2 code to Python 3

  • Port Python 2 code to Python 3

    • No: "Add support for Python 3", don't loose Python 2 support

  • Port Python 3 code to Python 2

  • Python changes to make the transition smoother:

    • PEP 414: u"syntax" reintroduced in Python 3.3
    • PEP 461: bytes % args, Python 3.5
    • More "Py3k" warnings added to Python 2.7.x
    • Linters like flake8 detect some issues
    • six, futures, modernize, 2to6, etc.

• Analysis of the transition

  • Good: At Pycon US 2014, Guido van Rossum announced that the Python 2.7 support was extended from 2015 to 2020 to give more time to companies to port their applications to Python 3.
  • Bad: 2to3: "drop Python2 support at once", don't work when you have dependencies.
  • If done again, would it be different? Yes, obviously.
  • Python 4 will be different than Python 3: no more "break the world" release, but a "regular deprecation period" release, as any other release. Break things, one by one :-)

• Bugs that won't be fixed in Python 2 anymore

  • Some bugs cannot be fixed without breaking the backward compatibility
  • Unicode Support
  • Python 2 I/O stack bugs: rely on libc stdio.h
  • Security: hash DoS, enabled by default in Python 3.3, Python 3.4 now uses SipHash
  • subprocess is not thread-safe in Python 2. Python 2 subprocess has many race conditions: fixed in Python 3 with a C implementation which has less or no race condition. Handling signals while forking in complex.
  • threading.RLock is not "signal safe" in Python 2
  • Python 2 requires polling to wait for a lock or for I/O. Python 3 uses native threading API with timeout and has asyncio.
  • Python 3 uses a monotonic clock to not crash on system clock update (ex: DST change).
  • Python 3 has a better GIL.
  • Python 2 inherits file descriptors on fork+exec by default. Python 3 don't: PEP 446.
  • Functions can fail with OSError(EINTR) when interrupted by a signal, need to be very careful everywhere. SIGCHLD when a child process completes, SIGWINCH when using ncurses. Python 3.5 restarts the interrupted system call for you.

• Performance

  • Python 3.6 is now faster than Python 2.7
  • https://speed.python.org/

• Evolutions of the Python language

  • Python 3.5

    • PEP 492: async/await "keywords" for asyncio. (Really keywords in Python 3.7.)
    • PEP 461: bytes % args and bytearray % args
    • PEP 465, a new matrix multiplication operator: a @ b.
    • PEP 448: Generalized unpacking: head, *tail = list mylist = [1, 2, **other_list] mydict = {"key": "value", **other_dict}

  • Python 3.6

    • PEP 515: million = 1_000_000
    • PEP 498: f-string: name = "World"; print(f"Hello {name}!")
    • PEP 526, syntax for variable annotations.
    • PEP 525, asynchronous generators.
    • PEP 530: asynchronous comprehensions.

• Bury Python 2?

  • "Python 3 only"
  • https://pythonclock.org/
  • http://www.python3statement.org/
  • Fedora 23, Ubuntu 17.10: no python2 (/usr/bin/python) in the base system
  • 2017, April: IPython 6.0
  • 2017, December: Django 2

Working on optimizations is a task more complex than expected on the first look. Any optimization must be measured to make sure that, in practice, it speeds up the application task. Problem: it is very hard to obtain stable benchmark results.

The stability of a benchmark (performance measurement) is essential to be able to compare two versions of the code and compute the difference (faster or slower?). An unstable benchmark is useless, and is a risk of giving a false result when comparing performance which could lead to bad decisions.

I'm gonna show you the Python project "perf" which helps to launch benchmarks, but also to analyze them: compute the mean and the standard deviation on multiple runs, render an histogram to visualize the probability curve, compare between multiple results, run again a benchmark to collect more samples, etc.

The use case is to measure small isolated optimizations on CPython and make sure that they don't introduce performance regression in term of performance.

The Python language is hard to optimize. Let's see how guards checked at runtime allows to implement new optimizations without breaking the Python semantic.

(Almost) everything in Python is mutable which makes Python a language very difficult to optimize. Most optimizations rely on assumptions, for example that builtin functions are not replaced. Optimizing Python requires a trigger to disable optimization when an assumption is no more true. FAT Python exactly does that with guards checked at runtime. For example, an optimization relying on the builtin len() function is disabled when the function is replaced.

Guards allows to implement various optimizations. Examples: loop unrolling (duplicate the loop body), constant folding (propagates constants), copy builtins to constants, remove unused local variables, etc.

FAT Python implements guards and an optimizer rewriting the Abstract Syntax Tree (AST). The optimizer is implemented in Python so it's easy to enhance it and implement new optimizations.

FAT Python uses a static optimizer, it is less powerful than a JIT compiler like PyPy with tracing, but it was written to be integrated into CPython.

The Python bytecode is inefficient: the compiler generates useless and redundant instructions, dead code and only implement basic optimizations. We can do better! I will present two projects trying to generate better bytecode.

astoptimizer optimizes the abstract syntax tree (AST) of a Python program. It tries to do as much work as possible at compile time. It is able to call builtin functions with immutables arguments, simplifies expressions, optimizes loops, iterators and generators, fold constants and removes dead code. https://bitbucket.org/haypo/astoptimizer

registervm is a fork of Python 3.4 rewriting its evaluation loop to support new instructions using registers instead of a stack. Bytecode using registers is shorter (less instruction) and using registers allows new optimizations. For example, it is much easier to move constants out of loops. registervm removes useless instructions like LOADNAME following STORENAME, and useless jumps. Duplicated constant loads are also removed. http://hg.python.org/sandbox/registervm