Sebastian Witowski

map() vs. List Comprehension

2023-07-31T00:00:00Z

From For Loop vs. List Comprehension, we already know that list comprehension is usually faster than the equivalent for loop. In the article, I also compared list comprehension with the filter() function. I concluded that, while filter() has some justified use cases where it's better than list comprehension (for example, when you want the more memory-efficient generator object that the filter() function returns), list comprehension is usually the faster choice.

About the "Writing Faster Python" series

"Writing Faster Python" is a series of short articles discussing how to solve some common problems with different code structures. I run some benchmarks, discuss the difference between each code snippet, and finish with some personal recommendations.

Are those recommendations going to make your code much faster? Not really.
Is knowing those small differences going to make a slightly better Python programmer? Hopefully!

You can read more about some assumptions I made, the benchmarking setup, and answers to some common questions in the Introduction article. And you can find most of the code examples in this repository.

What about list comprehension vs. map()? Is the map() function faster than list comprehension? And if not, does it make any sense to use it?

I've devised a simple test that compares how map() and list comprehension generate a list of squares for the first million numbers (it also sums up the squares - see the box below the benchmarks for the explanation):

# map_vs_comprehension.py
NUMBERS = range(1_000_001)

def map_lambda():
    return sum(map(lambda x: x * x, NUMBERS))


def comprehension_lambda():
    return sum([x * x for x in NUMBERS])

Here are the benchmarks results:

$ python -m timeit -s "from map_vs_comprehension import map_lambda" "map_lambda()"
5 loops, best of 5: 44.3 msec per loop

$ python -m timeit -s "from map_vs_comprehension import comprehension_lambda" "comprehension_lambda()"
10 loops, best of 5: 36.2 msec per loop

As you can see, map_lambda() is around 20% slower than comprehension_lambda() (44.3/36.2≈1.22).

map() returns a generator

In Python 2, functions like map() or filter() returned lists. But in Python 3, they return generators, so they finish much faster.

There is no free lunch, though. Time saved during the creation of a generator is paid back when we iterate over that generator.

Generators also offer more flexibility. For example, if you only need to grab the first element, creating a generator and calling next() is much faster than creating a list and grabbing the first element with a_list[0].

In my benchmarks, I needed to make sure that both functions did the same amount of work. I could call list(map(...)) to convert a generator to a list, but that would add additional work to the map_lambda() function that the list comprehension doesn't have to do:

def map_lambda():
    return list(map(lambda x: x * x, NUMBERS))


def comprehension_lambda():
    return [x * x for x in NUMBERS]

map_lambda() takes around 47.7 milliseconds to run and comprehension_lambda() takes around 29.8 milliseconds. With list(map(...)) being 60% slower than list comprehension, I felt those benchmarks would not be objective enough.

Instead, I decided to simulate calling another function on the results of a list and a generator. That would force both functions to iterate over all the items. sum() seemed like a good, simple function to achieve that.

Named function

Could the lambda function in map() be the reason why this function is so slow? Let's create another benchmark where we use the math.sqrt() function instead:

from math import sqrt

NUMBERS = range(1_000_001)

def map_sqrt():
    return sum(map(sqrt, NUMBERS))

def comprehension_sqrt():
    return sum([sqrt(x) for x in NUMBERS])

And the results are surprising:

$ python -m timeit -s "from map_vs_comprehension import map_sqrt" "map_sqrt()"
10 loops, best of 5: 31.5 msec per loop

$ python -m timeit -s "from map_vs_comprehension import comprehension_sqrt" "comprehension_sqrt()"
5 loops, best of 5: 45.4 msec per loop

Interesting! If we use an existing function instead of a lambda, map() is faster than list comprehension. This time list comprehension is around 44% slower than map() (45.4/31.5≈1.44).

Conclusions

map() used with a lambda function is usually slower than the equivalent list comprehension. But if you use it with a named function instead, it gets faster.

So which function should you use in your code? That really depends on your personal preference. Some people tend to call map() unpythonic and balk at using it under any circumstances. My rule of thumb is as follows:

I use map() when I can pass an existing function. I find code like map(str, some_text) or map(sqrt, numbers) very readable.
In all other cases, I use list comprehension or a generator expression.

I'm happy to see that my intuitive rule of thumb also coincidentally makes my code faster.

Inlining Functions

2023-07-24T00:00:00Z

In this episode of Writing Faster Python, we will check if we can make the code faster by doing exactly the opposite of what every good programming book suggests – that is, keeping all the code in one, massive function instead of smaller, more manageable functions.

Inlining a function just to make it faster is usually a bad idea and will make your code harder to understand. And for applications that process large amounts of data, it can actually bring the performance down by increasing the memory consumption (thanks Harvey for pointing out this downside!)

I don't recommend doing that unless this small speed improvement of the inlined function is somehow more important to you than a well-designed, readable, and testable code. Proceed with caution.

Let's start by writing a bunch of dummy functions whose only purpose is to call each other multiple times:

# inline_functions.py

def calculate_a():
    return 1


def calculate_b():
    return sum([calculate_a() for _ in range(100)])


def calculate_c():
    return sum([calculate_b() for _ in range(100)])


def calculate_d():
    return sum([calculate_c() for _ in range(100)])

Calling calculate_d() calls calculate_c() 100 times. Each call of calculate_c() calls calculate_b() 100 times. And so on.

In total, the above code performs 1,000,000 function calls. I'm intentionally using a list comprehension (sum([...])) instead of a generator expression (sum(...)) because, as you might know from my Writing Faster Python 3 talk, list comprehension is slightly faster (albeit, at the price of consuming more memory). In this case, the speed difference is tiny (~2%), so it doesn't matter if I stick with the list comprehension or use a generator expression.

Now, let's create two functions. One that calls calculate_d() and another that simply takes the bodies of all those functions and glues them together into a deeply nested list comprehension abomination:

def separate_functions():
    return calculate_d()


def inline_functions():
    return sum([sum([sum([1 for _ in range(100)]) for _ in range(100)]) for _ in range(100)])

Benchmarking time:

$ python -m timeit -s "from inline_functions import separate_functions" "separate_functions()"
10 loops, best of 5: 35.2 msec per loop

$ python -m timeit -s "from inline_functions import inline_functions" "inline_functions()"
20 loops, best of 5: 17.6 msec per loop

If we inline the body of each function, our code will run twice as fast (35.2/17.6=2). And it will be at least twice as hard to read. Maybe more.

In the above examples, the overhead of using a few functions is quite large because the bodies of those functions are small. It takes time to look up a function, but running it is rather fast since each has just one instruction inside. If the functions had much longer bodies, the difference between the above examples would probably be much smaller.

Also, according to this StackOverflow answer to the "is code written inline faster than using function calls?" question, function calls got much faster in CPython 3.10. Before, if your function was accepting positional arguments, CPython had to create dictionaries to handle them for function calls. So there are many factors that can affect the speed of calling a function. But in general, executing a function is slower than executing the code from this function directly.

Using temporary variables

inline_functions() is hard to read with all those nested functions and list comprehensions. And this is still a simple example! I've seen people write code this way but with much more complex functions.

We can make this code easier to follow by assigning the output of each function to a variable (this type of refactoring is called using a temporary variable):

def inline_variables():
    a = 1
    b = sum([a for _ in range(100)])
    c = sum([b for _ in range(100)])
    d = sum([c for _ in range(100)])
    return d

$ python -m timeit -s "from inline_functions import inline_variables" "inline_variables()"
50000 loops, best of 5: 5.43 usec per loop

Using temporary variables takes the execution time down from milliseconds to microseconds (that "u" in "usec" stands for "µ"). So assigning the result of a function call to a variable is a good idea if you know that you will need to reuse that result multiple times. Of course, as long as the function is idempotent (i.e., it always returns the same results).

Conclusions

The fastest code to run is the one that doesn't use variables or functions and contains just one large blob of code. Coincidentally, the most difficult-to-understand code is also the one that doesn't use variables or functions.

Sacrificing the readability of the code just to make it slightly faster is a terrible idea. You should instead consider using a better library (like NumPy), a better algorithm (parallelization or vectorization), or even a faster programming language. The choice depends on how much speed improvement you need to gain.

Still, it was an interesting exercise to see how much the speed varies between inlining code and extracting helpful functions or variables.

Pathlib for Path Manipulations

2023-07-17T00:00:00Z

If I were to name my top ten modules from the standard library, pathlib would be high on that list. It could even make it to the top three.

Manipulating paths was always a tricky problem if your code was supposed to work on different operating systems. If you accidentally hardcoded the ./some/nested/folders path in your Python package, Windows users would complain that your code doesn't work on their computers. And the other way around – a hardcoded some\\nested\\folder path wouldn't work on a Mac or a Linux machine.

Even if you figured out how to make paths work on different operating systems, the functions you can use with file paths are a bit scattered around different modules. Sure, most of them live in the os.path module. But if you want to search for filenames matching a pattern, you must use the glob() function from the glob module. For moving files around, there is os.rename but also shutil.move (which actually calls os.rename unless the destination is on a different disk). When searching for all the places in the code where files are moved, you must remember to check both functions. Unless, you know, someone used the third option: os.replace. Then you have to check all three.

Luckily, thanks to PEP-428, since version 3.4 of CPython, we have a wonderful tool that makes working with paths much easier. Just look at this piece of code:

from pathlib import Path

p = Path('/')
q = p / 'some' / 'nested' / 'folder'
q.resolve() # PosixPath('/some/nested/folder')

Overloading the division operator is a bit unusual, but it's so smart and perfectly suitable for path manipulation that I find this code simply beautiful.

About the "Writing Faster Python" series

Are those recommendations going to make your code much faster? Not really.
Is knowing those small differences going to make a slightly better Python programmer? Hopefully!

The Path object makes working with paths easier in a couple of other ways:

It normalizes paths to platform defaults. Path('some/path') becomes some\\path on Windows, and Path('some\\path') becomes some/path on Linux/Mac.
It ignores extraneous "." path separators, so Path('./some/./path') becomes PosixPath('some/path') on my Macbook. The Path object also tries to be smart about the front slashes. If you use too many (Path('//////some/path')), it removes the redundant ones on Linux or Mac, and returns Path('/some/path').
It unifies the API for various file manipulation operations that previously required using different Python modules. You no longer need the glob module to search for files matching a pattern, and you also don't need the os module to get the names of their directories. All this functionality can now be found in the pathlib module (of course, you can still use the os or glob modules, if you prefer).

But is it faster?

So yeah, all sunshine and rainbows, but we are here to answer one fundamental question: is pathlib faster than os.path?

Before I try to run the benchmarks, my guess is that it's not. Path() is an object-oriented approach to path manipulation. Instantiating an object probably takes longer than calling, for example, os.path.join (which simply spits out a string).

But even if it's slower, I would be curious by how much. Besides, who knows, maybe my gut feeling is wrong?

This time, I'm using a different approach to benchmarking because there is no one standard way to use pathlib. Sure, we can use it to create a path to a file, but we can also use it to print the current directory, list files with names matching a given pattern, or even quickly write text to a file.

I'm going to run a series of benchmarks for different tasks and see how much faster (or slower) it is to use pathlib instead of other functions.

Joining paths

First, let's benchmark probably the most common use case: joining directory names to create a full path to a file.

# pathlib_benchmarks.py

import os
from pathlib import Path

def os_path_join():
    return os.path.join("/", "some", "nested", "path", "to", "a", "file.txt")

def pathlib_join():
    return Path("/") / "some" / "nested" / "path" / "to" / "a" / "file.txt"

$ python -m timeit -s "from pathlib_benchmarks import os_path_join" "os_path_join()"
200000 loops, best of 5: 1.22 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import pathlib_join" "pathlib_join()"
50000 loops, best of 5: 5.74 usec per loop

In a scenario where I initialize Path() instance and then append multiple folders using the / operator, Path can be over four times as slow as using os.path.join (5.74/1.22 ≈ 4.70). And no matter if I create a path from 2 or 20 folders, Path is always around four or five times as slow as os.path.join:

def os_path_join_short():
    return os.path.join("/", "file.txt")

def pathlib_join_short():
    return Path("/") / "file.txt"


def os_path_join_long():
    return os.path.join("/", "an", "even", "longer", "path", "to", "some",
        "nested", "folder", "of", "a", "nested", "and", "nested", "and",
        "nested", "and", "nested", "path", "to", "file.txt",
    )


def pathlib_join_long():
    return (
        Path("/") / "an" / "even" / "longer" / "path" / "to" / "some" / "nested"
        / "folder" / "of" / "a" / "nested" / "and" / "nested" / "and" / "nested"
        / "and" / "nested" / "path" / "to" / "file.txt"
    )

$ python -m timeit -s "from pathlib_benchmarks import os_path_join_short" "os_path_join_short()"
1000000 loops, best of 5: 345 nsec per loop

$ python -m timeit -s "from pathlib_benchmarks import pathlib_join_short" "pathlib_join_short()"
200000 loops, best of 5: 1.69 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import os_path_join_long" "os_path_join_long()"
100000 loops, best of 5: 3.57 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import pathlib_join_long" "pathlib_join_long()"
20000 loops, best of 5: 17.3 usec per loop

Using an existing `Path()` object

What if it's the Path("/") creation that takes a lot of time and the concatenation of folders' names is actually fast? To check this, I will extract Path("/") to a global variable outside of the benchmarked function. Then, I can either reference the global variable directly, or pass it as a parameter to the benchmarked function. No matter which solution I choose, they both take a similar amount of time.

ROOT = Path("/")

def pathlib_join_existing_object(root=ROOT):
    return root / "some" / "nested" / "path" / "to" / "a" / "file.txt"

$ python -m timeit -s "from pathlib_benchmarks import pathlib_join_existing_object" "pathlib_join_existing_object()"
50000 loops, best of 5: 4.85 usec per loop

pathlib_join_existing_object() is slightly faster than pathlib_join (featured in initial benchmarks), but still much slower than using os.path.join (4.85/1.22≈3.98).

As @randallpittman pointed out in the comments, it seems that it's actually the concatenation of paths that makes Path slower in my benchmarks. If I pass all the paths directly as parameters, then it gets faster. Take a look at those two scenarios and their benchmarks:

def pathlib_multiple_args():
    return Path("/", "some", "nested", "path", "to", "a", "file.txt")


def pathlib_full_path():
    return Path("/some/nested/path/to/a/file.txt")

$ python -m timeit -s "from pathlib_benchmarks import pathlib_multiple_args" "pathlib_multiple_args()"
100000 loops, best of 5: 2.21 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import pathlib_full_path" "pathlib_full_path()"
200000 loops, best of 5: 1.4 usec per loop

Both pathlib_multiple_args and pathlib_full_path are now much faster. pathlib_full_path is only 15% slower than os.path (1.4/1.22≈1.15).

Starting from the home folder

One more test - what if we don't want to start from the root folder but from the home folder of the current user? Both modules have functions that return the home folder, so let's combine them with some additional folders and benchmark that:

def os_path_join_home():
    return os.path.join(os.path.expanduser("~"), "some", "nested", "path", "to", "a", "file.txt")


def pathlib_join_home():
    return Path.home() / "some" / "nested" / "path" / "to" / "a" / "file.txt"

$ python -m timeit -s "from pathlib_benchmarks import os_path_join_home" "os_path_join_home()"
100000 loops, best of 5: 2.12 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import pathlib_join_home" "pathlib_join_home()"
50000 loops, best of 5: 8.01 usec per loop

The difference is smaller (8.01/2.12≈3.78), but the os module still wins this round. 1:0 for the os module.

Let's test some other common operations on file paths.

Is it a file?

Time for a second round of benchmarks. Let's compare the performance of functions that check if the object under a given path is a file (and not a directory):

def os_isfile(name):
    return os.path.isfile(f"./{name}")


def pathlib_is_file(name):
    return Path(f"./{name}").is_file()

And to make my benchmarks more complete, I will look for a file that exists but also for one that doesn't:

# First, a file that exists
$ python -m timeit -s "from pathlib_benchmarks import os_isfile" "os_isfile('pathlib_benchmarks.py')"
100000 loops, best of 5: 2.28 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import pathlib_is_file" "pathlib_is_file('pathlib_benchmarks.py')"
50000 loops, best of 5: 4.12 usec per loop

# And a file that doesn't
$ python -m timeit -s "from pathlib_benchmarks import os_isfile" "os_isfile('non-existing-file')"
200000 loops, best of 5: 1.02 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import pathlib_is_file" "pathlib_is_file('non-existing-file')"
100000 loops, best of 5: 2.82 usec per loop

In both scenarios os.path is still faster, although the difference is smaller than in the first set of benchmarks. Path.is_file is around twice as slow when the file exists (4.12/2.28≈1.81) and around three times as slow when it doesn't exist (2.82/1.02≈2.76).

2:0 for os.path.

Get the current directory

How about getting the current directory?

$ python -m timeit -s "import os" "os.getcwd()"
50000 loops, best of 5: 6.75 usec per loop

$ python -m timeit -s "from pathlib import Path" "Path.cwd()"
50000 loops, best of 5: 8.54 usec per loop

os.getcwd() is faster by around 30% this time (8.54/6.75≈1.27).

Find all the files matching a pattern

Let's try something more complex. This time, I want to recursively find all the Python files (that is, files with the ".py" extensions).

If I really need to stick with the os module, I could write something like this:

def os_walk_files():
    python_files = []
    for root, dirs, files in os.walk("."):
        for filename in files:
            if filename.endswith(".py"):
                python_files.append(root + filename)
    return python_files

But it's much easier to use the glob module instead. That way we just need one line of code:

import glob

def glob_find_files():
    glob.glob("./**/*.py", recursive=True)

pathlib comes with a similar function called rglob(). But there are two important distinctions between this function and glob.glob() or os.walk():

Path().rglob() returns Path objects, while os.walk() and glob.glob() return strings. I assume we are ok with Path objects because they work fine for opening files indicated by the file paths or for printing those paths. I don't see a reason to convert them to inferior strings (inferior in terms of what we can do with them). If you really need strings, remember you must additionally call str() on each Path object.
os_walk_files() and glob_find_files() return lists, but Path().rglob() returns a generator. To make the results of all the examples as similar as possible to each other, I will convert this generator to a list (which will slow down my benchmarks). If I don't do this, Path.glob will have an unfair advantage, as creating a generator is much faster than building a list. But in general, if you want to iterate over those files, there is no point in converting a generator to a list first. Moreover, if the list of files is huge, a generator will be much more memory-efficient.

Here is the pathlib version of a function to find all the Python files:

def path_find_files():
    return list(Path().rglob("*.py"))

Let's run the benchmarks:

$ python -m timeit -s "from pathlib_benchmarks import os_walk_files" "os_walk_files()"
5000 loops, best of 5: 80.6 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import glob_find_files" "glob_find_files()"
2000 loops, best of 5: 152 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import path_find_files" "path_find_files()"
2000 loops, best of 5: 156 usec per loop

The most verbose version that includes two loops and an if statement still turns out to be almost twice as fast as using the glob (152/80.6≈1.89) or pathlib (156/80.6 ≈1.94) modules.

That puts our benchmarking score at I-have-lost-track-a-long-time-ago to 0 for the os module.

Quickly write to a file

Another interesting feature of pathlib is that you can quickly write some text or bytes to a file.

Below is a comparison of Path().write_text() and the classic with open() context manager. We open a file (or create it, if it doesn't exist) in write mode and replace the previous content with some simple text:

def classic_write():
    with open("a_file.txt", "w") as f:
        f.write("hello there")

def pathlib_write():
    Path('/a_file.txt').write_text("hello there")

$ python -m timeit -s "from pathlib_benchmarks import classic_write" "classic_write()"
5000 loops, best of 5: 55.3 usec per loop

$ python -m timeit -s "from pathlib_benchmarks import pathlib_write" "pathlib_write()"
5000 loops, best of 5: 55.8 usec per loop

They both take the same amount of time (no matter if the a_file.txt already exists or not). No wonder - write_text() is actually just a nice little wrapper around the with open code.

If you're curious, there is also a wrapper for reading the content from a file. The wrapper is called read_text() and has a similar performance as its with open(<file>, 'r') equivalent.

Conclusions

The list of various tasks we can perform with pathlib can go on for much longer. Creating, deleting, reading, writing, finding, moving, copying, splitting, and whatever other operation you want to perform on a file path or a file itself - pathlib probably has a function for that. Sure, os.path or some other module can do those things faster. But unless file manipulation is the main bottleneck in a program (which I really doubt is a problem for anyone anymore, with large-memory VMs being easily accessible in the cloud), I much more prefer to use pathlib.

It's nice to finally have a single module with all the functionality related to paths and files. And I love this object-oriented approach to file paths. It makes writing scripts for filesystem manipulation much more fun, making Python an even better replacement for bash scripts^[1].

You can find all the code examples from this article in my blog-resources repository.

String Formatting

2023-03-02T00:00:00Z

One of the most well-received features introduced in Python 3.6 were the f-strings. Unlike the walrus operator (introduced in Python 3.8), f-strings quickly became popular - it's hard to find someone who doesn't love them! Officially named literal string interpolation, f-strings are much more readable and faster to write. And if you come from a language like JavaScript, you will feel at home using them because they work the same as template literals introduced in ES6.

If you follow the landscape of string formatting in Python, you've probably already noticed that this brings us a total of four different ways to format strings. Why do we need so many? Let’s quickly review them and find out.

The old style of string formatting with the % operator

name = "Sebastian"

# The standard "old" style
>>> "Hello %s" % name
"Hello Sebastian"

# Or a more verbose way (useful when you pass multiple variables)
>>> "Hello %(name)s" % {"name": name}
"Hello Sebastian"

This formatting style is sometimes called printf-style formatting or %-formatting. It used to be Python's default string formatting style and worked pretty fine. However, it was quite limited - you could only format strings, integers, or doubles (floats or decimal numbers). Each variable was converted to a string by default unless you specified a different output format (e.g., integers could be presented in a binary, octal, decimal, or hex format). If a variable could not be converted to a specific type, you got an error. If you wanted to pass more arguments inside a tuple, but you forgot to write your code in a specific way, you got an error too:

fullname = ('Sebastian', 'Witowski')

# This fails
>>> "Hello %s" % fullname
TypeError: not all arguments converted during string formatting

# This works
>>> "Hello %s" % (fullname,)
"Hello ('Sebastian', 'Witowski')"

There is one interesting feature of the old style formatting that the other methods don't have. It allows you to do some "lazy logging" by only evaluating the string formatting expression when needed. If you write your logging statement like this: log.debug("Some message: a=%s", a), and your logging module is configured not to log out the debug messages, a will never be converted to a string. If for some reason, a takes very long to convert to a string, this might save you some time. But honestly, I can't think of any example of when this might happen. So think of this as a curiosity.

Template strings

In Python 2.4, PEP 292 introduced the template strings formatting. It was added to solve some shortcomings of the old style - template strings were supposed to be simpler and less error-prone.

With template strings, you first create a template, and then you substitute placeholders with variables:

>>> from string import Template
>>> s = Template("Hello ${first} ${last}")
>>> s.substitute(first="Sebastian", last="Witowski")
"Hello Sebastian Witowski"
>>> s.substitute(first="John", last="Doe")
"Hello John Doe"

When you call the substitute method, it returns a new string with all the placeholders (${placeholder_name}) replaced with the specified values. If you forget a mapping for any of the placeholders, you will get a KeyError:

>>> s.substitute(first="Sebastian")
KeyError: 'last'

The new style with str.format()

In Python 3, a new formatting style was introduced with PEP 3101 (and later, it was backported to Python 2.7). This new style was simply the format() function added to the str type. Since format() was a function call, there was no difference in how you would write your code, no matter if you wanted to format a string or a tuple:

name = "Sebastian"
fullname = ('Sebastian', 'Witowski')

>>> "Hello {}".format(name)
"Hello Sebastian"
>>> "Hello {}".format(fullname)
"Hello ('Sebastian', 'Witowski')"

# You can name your arguments:
>>> "Hello {first} {last}".format({"first": "Sebastian", "last": "Witowski"})
"Hello Sebastian Witowski"
# ...or use positions of arguments
>>> "Hello {1} {0}".format("Sebastian", "Witowski")
"Hello Witowski Sebastian"

Similarly to the old style, you could specify the presentation format and pass some additional flags. For example, if you wanted to print an integer and pad it to four digits, you could write it like this:

>>> "The answer is: {answer:04d}".format(answer=42)
"The answer is: 0042"

The new formatting style is much more robust but also a bit more verbose. Even for the simplest situation, you always have to write the .format. And why do we have to repeat ourselves by typing "answer" twice in the above example? Why can't we just tell Python: "Listen, I have this answer variable already defined. Just take it and put it inside this string"?

So, similarly to what exists in other programming languages, literal string interpolation was introduced in Python 3.6 with PEP 498.

f-strings (literal string interpolation)

The newest way of formatting strings in Python is the most convenient one to use. Just prefix a string with the letter "f" (thus the name "f-strings"), and whatever code you put inside the curly brackets gets evaluated. It can be a variable or any kind of Python expression:

name = Sebastian

>>> "Hello {name}"
"Hello {name}" # Nothing happens because we forgot the 'f'!

>>> f"Hello {name}"
"Hello Sebastian"

>>> f"The answer is {40+2}"
"The answer is 42"

import datetime
>>> f"Current year: {datetime.datetime.now():%Y}"
"Current year: 2023"

Which string formatting method is the fastest?

Let's prepare some test functions to see which method is the fastest one.

# string_formatting.py

from string import Template

FIRST = "Sebastian"
LAST = "Witowski"
AGE = 33


def old_style():
    return "Hello %s %s (%i)" % (FIRST, LAST, AGE)


def template_strings():
    return Template("Hello ${first} ${last} (${age})").substitute(first=FIRST, last=LAST, age=AGE)


def new_style():
    return "Hello {} {} ({})".format(FIRST, LAST, AGE)


def f_strings():
    return f"Hello {FIRST} {LAST} ({AGE})"

Here are the benchmark results:

$ python -m timeit -s "from string_formatting import old_style" "old_style()"
2000000 loops, best of 5: 165 nsec per loop

$ python -m timeit -s "from string_formatting import template_strings" "template_strings()"
200000 loops, best of 5: 1.49 usec per loop

$ python -m timeit -s "from string_formatting import new_style" "new_style()"
1000000 loops, best of 5: 200 nsec per loop

$ python -m timeit -s "from string_formatting import f_strings" "f_strings()"
2000000 loops, best of 5: 118 nsec per loop

f-strings are the fastest way of formatting a string. The new string formatting style is around 70% slower (200/118≈1.69), the old style is around 40% slower (165/118≈1.40), and template strings are over ten times slower (1490/118≈12.63).

Someone could argue that in the old_style() function, I'm referencing some global variables, which is not always necessary. Sometimes you might want to pass the variables directly:

def old_style_inline():
    return "Hello %s %s (%i)" % ("Sebastian", "Witowski", 33)

But even in this case, while slightly faster, the old style doesn't beat the f-strings.

$ python -m timeit -s "from string_formatting import old_style_inline" "old_style_inline()"
2000000 loops, best of 5: 149 nsec per loop

Conclusions

Even if f-strings were slower than other formatting styles, I would still keep using them. They are so incredibly convenient that it's hard to justify using other ways of string formatting.

But still, let's try to find use cases for the other methods:

Template strings, as the name suggests, are great when you're writing a template where readability and reusability are more important than performance. Imagine building a large block of text with multiple variables you want to fill in later. You might even want to apply different variables to the same template. This is the perfect use case for template strings. However, this formatting style doesn't make sense for creating small strings. Template strings are slower by an order of magnitude (compared to f-strings), take longer to write and read (the template_strings() example has over twice as many characters as the f_strings() equivalent), and don't have any benefit over the f-strings.
The new style is a bit slower but much more flexible and error-proof compared to the old style. If I couldn't use f-strings, I would choose this option.
Using the old style string formatting is really hard to justify. Of course, if I were to use some ancient Python version (even lower than Python 2.7), this would be my only viable option. The only other scenario where I would choose the old style is formatting a simple string with one variable using a Python version lower than 3.6.

In any other scenario, when the f-strings are available, I would choose them.

Of course, we only looked at formatting strings, that is, putting variables or expressions into a string. However, there are a lot more ways to construct a string. You can add strings together ("answer is " + "42"), join a list ("".join(['answer', ' is', ' 42'])), or probably come up with some even more creative solution. But creating strings effectively is a story for another article.

Compare to None

2023-02-23T00:00:00Z

How do we check if something is None?

With the beauty of the Python language - the code that you would write is literally the same as the above question:

if something is None:

It reminds me of this joke:

- How do you turn pseudocode into Python?
- You add .py at the end of the file.

There is another way in which we could make this comparison:

if something == None:

However, it doesn't make sense to use the second variant. None is a singleton object - there can't be two different None objects in your code. Each time you assign None to a variable, you reference the same None:

>>> a = None
>>> b = None
>>> c = None
>>> a is b is c
True

To compare the identity, you should use is, rather than ==, as I explained in the Checking for True or False article. It's clearer and faster:

$ python -m timeit -s "a = 1" "a is None"
50000000 loops, best of 5: 8.2 nsec per loop

$ python -m timeit -s "a = 1" "a == None"
20000000 loops, best of 5: 13 nsec per loop

As you can see, == is 60% slower than is (13 / 8.2 ≈ 1.59).

Dictionary Comprehension

2023-01-19T00:00:00Z

Apart from the list comprehension method, in Python, we also have dictionary comprehension - a little less known but very useful feature. It's a perfect tool for creating a dictionary from an iterable. Let's see how we can use it and if it's faster than other methods.

About the "Writing Faster Python" series

Are those recommendations going to make your code much faster? Not really.
Is knowing those small differences going to make a slightly better Python programmer? Hopefully!

The simplest way to create a dictionary is to use a for loop:

powers = {}
for n in range(1000):
    powers[n] = n * n

That's not super-elegant. We can simplify our code by passing a list of key-value tuples directly to the dict() function:

dict([(n, n * n) for n in range(1000)])

Before Python 2.7, this was the simplest way to build a dictionary from an iterable. It's not bad, but all those brackets and parentheses can be slightly confusing.

With the release of Python 2.7.3, PEP 274 introduced dictionary comprehension, which lets us simplify our code even further:

{n: n * n for n in range(1000)}

It's certainly much easier to read, but is it faster? Let's look into it.

Dictionary comprehension vs. `dict()` vs. `for` loop

Here are the functions that I'm benchmarking:

# dictionary_comprehension.py

NUMBERS = list(range(1000))

def for_loop():
    powers = {}
    for number in NUMBERS:
        powers[number] = number * number
    return powers


def dict_from_tuples():
    return dict([(n, n * n) for n in NUMBERS])


def dict_comprehension():
    return {i: i * i for i in NUMBERS}

And here are the results for Python 3.11.0:

# Python 3.11.0
$ python -m timeit -s "from dictionary_comprehension import for_loop" "for_loop()"
10000 loops, best of 5: 32.1 usec per loop

$ python -m timeit -s "from dictionary_comprehension import dict_from_tuples" "dict_from_tuples()"
5000 loops, best of 5: 51.3 usec per loop

$ python -m timeit -s "from dictionary_comprehension import dict_comprehension" "dict_comprehension()"
10000 loops, best of 5: 31.2 usec per loop

Interesting! Two things surprised me:

for loop is as fast as dictionary comprehension! I was expecting it to be the slowest function.
Creating a dictionary from a list comprehension is around 60% slower (51.3/31.2≈1.64) than other functions. I expected it to be a bit slower, but not that much.

What happens if we increase the benchmarks to run for more numbers? Let's see:

# dictionary_comprehension.py

MORE_NUMBERS = list(range(1_000_000))

def for_loop2():
    powers = {}
    for number in MORE_NUMBERS:
        powers[number] = number * number
    return powers


def dict_from_tuples2():
    return dict([(n, n * n) for n in MORE_NUMBERS])


def dict_comprehension2():
    return {i: i * i for i in MORE_NUMBERS}

$ python -m timeit -s "from dictionary_comprehension import for_loop2" "for_loop2()"
5 loops, best of 5: 44.9 msec per loop

$ python -m timeit -s "from dictionary_comprehension import dict_from_tuples2" "dict_from_tuples2()"
5 loops, best of 5: 77.9 msec per loop

$ python -m timeit -s "from dictionary_comprehension import dict_comprehension2" "dict_comprehension2()"
5 loops, best of 5: 43.5 msec per loop

Dictionary comprehension and for loop are still equally fast, while dict() is now slightly slower than before (77.9/43/5≈1.79).

I hope I've convinced you by now that dictionary comprehension is one of the best ways to build dictionaries from an iterable. This method is faster than passing a list of tuples to a dict() function. And while it's not really that much faster than a simple for loop, dictionary comprehension is much more readable. Once you understand the syntax, you can immediately see what's happening in that code.

Creating a dictionary from two iterables

What if we want to combine two iterables?

KEYS = list(range(1_000_000))
VALUES = [x * x for x in range(1_000_000)]

Above, we have two iterables we want to use as keys and values in a dictionary. We need to zip the iterables together so we can apply dictionary comprehension:

def comprehension_with_zip():
    return {key: value for key, value in zip(KEYS, VALUES)}

However, here we don't do anything special with key or value. In the initial examples, the value for each key was computed as we were building a dictionary: n: n * n. But now, it's just key: value. In a situation like this, you can pass zipped iterables directly to the dict() function.

def just_zip():
    return dict(zip(KEYS, VALUES))

Let's see the benchmarks:

$ python -m timeit -s "from dictionary_comprehension import comprehension_with_zip" "comprehension_with_zip()"
10 loops, best of 5: 34 msec per loop

$ python -m timeit -s "from dictionary_comprehension import just_zip" "just_zip()"
10 loops, best of 5: 31.4 msec per loop

Calling dict() on zip() directly is slightly faster (34/31.4≈1.08) than using dictionary comprehension. At the same time, it's a bit more concise.

It's very similar to passing an iterable to a list comprehension. In many cases, list comprehension is the best way to create a list, but sometimes you can use an even shorter version if you don't do any processing on the iterable:

# Bad
[x for x in range(1000)]

# Good
list(range(1000))

Conclusions

Dictionary comprehension is one of the cleanest ways to build a dictionary. Compared with the old way of passing a list of tuples (in Python 2.6 and below), it's faster and more readable.

But it only makes sense to use it when you compute a key or a value on the fly or if you want to do some filtering. If both the key and the value are ready (for example, they come from two different iterables), simply passing the zip() function to dict() results in a much faster and more readable code:

# Good use case for dictionary comprehension - we compute the value
{i: i * i for i in range(1000)}

# Good use case for dictionary comprehension - we compute the key
{i * i: i for i in range(1000)}

# Good use case for dictionary comprehension - we filter values
{i: i * i for i in range(1000) if i > 50}

# Bad use case for dictionary comprehension
NUMBERS = range(1000)
SQUARES = [x * x for x in range(1000)]

{key: value for key, value in zip(KEYS, VALUES)}

# Use a zip() instead
dict(zip(NUMBERS, SQUARES))

dict() vs. {}

2022-12-01T00:00:00Z

There are two different ways to create a dictionary. You can call the dict() function or use the literal syntax: {}. And in many cases, these are equivalent choices, so you might give it little thought and assume they both take the same amount of time.

But they don't!

Starting with this article, in my benchmarks, I have switched from Python 3.8 to 3.11. So if you're following the Writing Faster Python series and you're wondering why my code examples suddenly got a bit faster - that's the reason.

Check out the Upgrade Your Python Version article for a comparison of how much faster we can get by simply upgrading the CPython version.

# Python 3.11.0
$ python -m timeit "dict()"
10000000 loops, best of 5: 29.8 nsec per loop

$ python -m timeit "{}"
20000000 loops, best of 5: 14.2 nsec per loop

Benchmarking both versions shows that calling {} is twice as fast as calling dict(). And that's for Python 3.11. If you run the same examples with an older version of Python, dict() is even slower:

# Python 3.8.13
$ python -m timeit "dict()"
5000000 loops, best of 5: 57.2 nsec per loop

$ python -m timeit "{}"
20000000 loops, best of 5: 14.2 nsec per loop

Here dict() is almost four times as slow as {}.

Looking under the hood with the `dis` module

Let's use the disassembler module to compare what's happening when we call dict() and {}:

>>> from dis import dis
>>> dis("dict()")
  0           0 RESUME                   0

  1           2 PUSH_NULL
              4 LOAD_NAME                0 (dict)
              6 PRECALL                  0
             10 CALL                     0
             20 RETURN_VALUE
>>> dis("{}")
  0           0 RESUME                   0

  1           2 BUILD_MAP                0
              4 RETURN_VALUE

The dis module returns the bytecode instructions from a code snippet. It's an excellent way to see what's happening under the hood of your programs. Don't worry if all those cryptic names seem unfamiliar (if you're curious, check out the Python Bytecode Instructions). For us, the important instructions are BUILD_MAP and CALL.

When we call {}, we execute a Python statement, so Python immediately knows what to do - build a dictionary. In comparison, when we call dict(), Python has to find the dict() function and call it. That's because nothing stops you from overriding the dict() function. You can make it do something completely different than creating a dictionary, for example:

def dict(*args, **kwargs):
    # Happy debugging ;)
    return list([1, 2, 3])

Python doesn't stop you from overriding the built-in functions. So when you call dict(), the interpreter has to find this function and call it.

Is there any other difference?

I tried to think of any other reason why you might use dict() over {}, and the only one that came to my mind was for creating a dictionary from an iterator.

Take a look at this example:

>>> iter = zip(['a', 'b', 'c'], [1,2,3])
>>> {iter}
{<zip at 0x102d57b40>}  # This is not really what we want
>>> dict(iter)
{'a': 1, 'b': 2, 'c': 3}  # Much better

We can't use the literal syntax to create a dictionary. We would have to use a dictionary comprehension: {k: v for k, v in iter}. But a simple dict(iter) looks much cleaner. Apart from this use case, I think it's mostly up to your preference which version you use.

There are also some interesting quirks that I found. For example, in CPython 3.6 and below, if you wanted to pass more than 255 arguments to a function, you would get a SyntaxError. So, in this case, dict() is a no-go, but {} should work. However, if you're passing over 255 parameters to a function, you probably have bigger problems in your code than wondering if the literal syntax is a few nanoseconds faster.

[] vs. list(), () vs. tuple, {'x', } vs. set(['x'])

The same rule applies to using [] vs. list(), () vs. tuple(), or {'x',} vs. set(['x']). Using the literal syntax is faster than calling the corresponding function:

$ python -m timeit "list()"
10000000 loops, best of 5: 28.5 nsec per loop

$ python -m timeit "[]"
20000000 loops, best of 5: 12.7 nsec per loop

$ python -m timeit "tuple()"
50000000 loops, best of 5: 9.93 nsec per loop

$ python -m timeit "()"
50000000 loops, best of 5: 4.45 nsec per loop

$ python -m timeit "set(['x'])"
5000000 loops, best of 5: 72.7 nsec per loop

$ python -m timeit "{'x',}"
10000000 loops, best of 5: 29.5 nsec per loop

Of course, if you construct a large data structure, the difference between the two versions becomes unnoticeable:

$ python -m timeit "list(range(1_000_000))"
20 loops, best of 5: 14 msec per loop

$ python -m timeit "[*range(1_000_000)]"
20 loops, best of 5: 14 msec per loop

How to Benchmark (Python) Code

2022-11-17T00:00:00Z

While preparing to write the Writing Faster Python series, the first problem I faced was "How do I benchmark a piece of code in an objective yet uncomplicated way".

I could run python -m timeit <piece of code>, which is probably the simplest way of measuring how long it takes to execute some code^[1]. But maybe it's too simple, and I owe my readers some way of benchmarking that won't be interfered by sudden CPU spikes on my computer?

So here are a couple of different tools and techniques I tried. At the end of the article, I will tell you which one I chose and why. Plus, I will give you some rules of thumb for when each tool might be handy.

python -m timeit

The easiest way to measure how long it takes to run some code is to use the timeit module. You can write python -m timeit your_code(), and Python will print out how long it took to run whatever your_code() does. I like to put the code I want to benchmark inside a function for more clarity, but you don't have to do this. You can directly write multiple Python statements separated by semicolons, and that will work just fine. For example, to see how long it takes to sum up the first 1,000,000 numbers, we can run this code:

python -m timeit "sum(range(1_000_001))"
20 loops, best of 5: 11.5 msec per loop

However, python -m timeit approach has a major drawback - it doesn't separate the setup code from the code you want to benchmark. Let's say you have an import statement that takes a relatively long time to import compared to executing a function from that module. One such import can be import numpy. If we benchmark those two lines of code:

import numpy
numpy.arange(10)

the import will take most of the time during the benchmark. But you probably don't want to benchmark how long it takes to import modules. You want to see how long it takes to execute some functions from that module.

python -m timeit -s "setup code"

To separate the setup code from the benchmarks, timeit supports -s parameter. Whatever code you pass here will be executed but won't be part of the benchmarks. So we can improve the above code and run it like this: python -m timeit -s "import numpy" "numpy.arange(10)".

python -m timeit -s "setup code" -n 10000

We can be a bit more strict and decide to execute our code the same number of times each time. By default, if you don't specify the '-n' (or --number) parameter, timeit will try to run your code 1, 2, 5, 10, 20, ... until the total execution time exceeds 0.2 seconds. A slow function will be executed once, but a very fast one will run thousands of times. If you think executing different code snippets a different number of times affects your benchmarks, you can set this parameter to a predefined number.

docker

One of the issues with running benchmarks with python -m timeit is that sometimes other processes on your computer might affect the Python process and randomly slow it down. For example, I've noticed that if I run my benchmarks with all the usual applications open (multiple Chrome instances with plenty of tabs, Teams and other messenger apps, etc.), they all take a bit longer than when I close basically all the apps on my computer.

So while trying to figure out how to avoid this situation, I decided to try to run my benchmarks in Docker. I came up with the following solution: docker run -w /home -it -v $(pwd):/home python:3.10.4-alpine python -m timeit -s "<some setup code>" "my_function()"

The above code will:

Run Python alpine Docker container (a small, barebones image with Python).
Mount the current folder inside the Docker container (so we can access the files we want to benchmark).
Run the same timeit command as before.

And the results seemed more consistent than without using Docker. Rerunning benchmarks multiple times, I was getting results with smaller deviations. I still had a deviation - some runs were slightly slower, and some were slightly faster. However, that was the case for short code examples (running under 1 second). For longer code examples (running at least a few seconds), the difference between runs was even around 5% (I've tested docker with my bubble sort example from Upgrade Your Python Version article). So, as one vigilant commenter suggested, Docker doesn't really help much here.

Python benchmarking libraries

At some point, you might decide that getting a "best of 5" number that timeit returns by default is not enough. What if I need to know what's the most pessimistic scenario (the maximum time it took to run my code)? Or what's the difference between the slowest and fastest run? Is this difference huge, and my function runs in a completely unpredictable amount of time? Or is it so tiny that it's almost negligible?

There are better benchmarking tools that offer more statistics about your code.

rich-bench

The first tool I checked was the rich-bench package that was created by Anthony Shaw together with his anti-patterns repository for a PyCon talk. This small tool can benchmark a set of files with different code examples and present the results in a nicely formatted table. Each benchmark will compare two different functions and present the mean, min, and max of the results, so you can easily see the spread between the results.

pyperf

If you need a more advanced benchmarking tool, you probably can't go wrong if you choose the official tool used by the Python Performance Benchmark Suite - an authoritative source of benchmarks for all Python implementations. pyperf is an exhaustive tool with many different features, including automatic calibration, detection of unstable results, tracking memory usage, and different modes of work, depending if you want to compare different pieces of code or get a bunch of stats for one function.

Let's see an example. For the benchmarks, I will use a simple but inefficient function to calculate a sum of powers of the first 1,000,000 numbers: sum(n * n for n in range(1_000_001)).

Here is the output from timeit module:

$ python -m timeit "sum(n * n for n in range(1_000_001))"
5 loops, best of 5: 41 msec per loop

And here is the output of the pyperf:

$ python -m pyperf timeit "sum(n * n for n in range(1_000_001))" -o bench.json
.....................
Mean +- std dev: 41.5 ms +- 1.1 ms

The results are very similar, but with the -o parameter, we told pyperf to store the benchmark results in a JSON file, so now we can analyze them and get much more information:

$ python -m pyperf stats bench.json
Total duration: 14.5 sec
Start date: 2022-11-09 18:19:37
End date: 2022-11-09 18:19:53
Raw value minimum: 163 ms
Raw value maximum: 198 ms

Number of calibration run: 1
Number of run with values: 20
Total number of run: 21

Number of warmup per run: 1
Number of value per run: 3
Loop iterations per value: 4
Total number of values: 60

Minimum:         40.8 ms
Median +- MAD:   41.3 ms +- 0.2 ms
Mean +- std dev: 41.5 ms +- 1.1 ms
Maximum:         49.6 ms

  0th percentile: 40.8 ms (-2% of the mean) -- minimum
  5th percentile: 40.9 ms (-1% of the mean)
 25th percentile: 41.2 ms (-1% of the mean) -- Q1
 50th percentile: 41.3 ms (-0% of the mean) -- median
 75th percentile: 41.5 ms (+0% of the mean) -- Q3
 95th percentile: 41.9 ms (+1% of the mean)
100th percentile: 49.6 ms (+20% of the mean) -- maximum

Number of outlier (out of 40.7 ms..41.9 ms): 3

hyperfine

And in case you want to benchmark some code that is not Python code, there is always the hyperfine that can be used to benchmark any CLI command. hyperfine has a similar set of features as the pyperf does. It automatically does warmup runs, clears the cache, and detect statistical outliers. And all that, with nice progress bars and colors, just makes the output looks beautiful.

You can run it for one command, and it will return the usual information like the mean, min, and max time, standard deviation, number of runs, etc. But you can also pass multiple commands, and you will get a comparison of which one was faster:

timeit is just fine...for me

In the end, I chose a very simple way of benchmarking: python -m timeit -s "setup code" "code to benchmark". I don't have to use the perfect benchmarking method (if it even exists). . That would be necessary if I were to benchmark one piece of code and share the results with the world. I couldn't use a random, inefficient method of measuring and tell you "this piece of code is bad because it runs in 15 seconds". You could use a better benchmarking tool, run it on a powerful computer and end up with the same code running in 1.5 seconds.

Comparing two pieces of code is a different story. Sure, a good, reliable benchmarking methodology is important. But in the end, we care about the relative speed difference between the code examples. If my computer runs "Example A" in 10 seconds and "Example B" in 20 seconds, but your computer runs them in 5 and 10 seconds respectively, we can both conclude that "Example B" is twice as slow.

Using timeit is good enough. It lets me separate the setup code from the actual code I want to benchmark. And if you want to run the same benchmarks on your computer, you can do this right away. You already have timeit installed with your distribution of Python. You don't have to install any additional library or set up Docker.

Much more important thing than the most accurate tool is how you set up your benchmarks.

Beware of how you structure your code

Running benchmarks is the easy part. The tricky part is to remember to write your code in a way that won't "cheat". When I first wrote Sorting Lists article, I was so happy to find that sort() was so much faster than sorted(). "OMG, I found the holy grail of sorting in Python" - I thought. Then someone pointed out that list.sort() sorts the list in place. So if I run my benchmarks, the first iteration will sort the list (which is slow), and each next iteration will sort an already sorted list (which is much faster). I had to update my article and start paying more attention to how I organize my benchmarks.

Conclusion

Depending on your use case, you might reach for a different tool to benchmark your code:

python -m timeit "some code" for the simplest, easiest-to-run benchmarks where you just want to get "a number".
python -m timeit -s "setup code" "some code" is a much more useful version if you want to separate some setup code from the actual benchmarks.
docker - while it looked like it did a better job separating my benchmarks from other processes, thus lowering the deviation between runs, after thorough testing, that seemed to be the case for very short examples. For longer ones it didn't really change much.
rich-bench looks like a nice solution if you need a dedicated tool with additional statistics like min, max, median, and nice output formatting. But you will need to set up your benchmarks in a specific structure that rich-bench requires.
pyperf gives you the most advanced set of statistics about your code. And it's used by the official Python benchmarks, so it's an excellent tool for advanced benchmarks.
hyperfine is a great tool to benchmark any command, not only Python code. Or to compare two different commands.

Ok, technically, I could print the current time with time.time(), run my code, print time.time() again, and subtract those two values. But, come on, that's not simple, that's rudimentary. ↩︎

Upgrade Your Python Version

2022-11-14T00:00:00Z

Here is an idea for a completely free^[1] speed improvement for your code - upgrade your Python version!

I started this series of articles using Python 3.8, but today we already have version 3.11. Python 3.11 is the first version of Python that brings pretty significant speed improvements thanks to the Faster CPython project. If you have never heard about it, it started as Mark Shannon's idea to improve the overall performance of CPython, and now a dedicated team of developers (including Guido van Rossum) is working to bring some hefty speed improvements over the next few releases.

So I decided to benchmark some Python scripts to see how much faster they can get by simply updating the Python versions. I will check out some of the examples I described in this "Writing Faster Python" series, but also some random, computationally intensive programs.

Setup

Here are the scripts I will take for a spin. Each link will take you to the corresponding article on that topic.

Ask for Forgiveness or Look Before You Leap - example 2, where we check if all 3 attributes exist (and they do):

# permission_vs_forgiveness.py

class BaseClass:
    hello = "world"
    bar = "world"
    baz = "world"

class Foo(BaseClass):
    pass

FOO = Foo()

# Look before you leap
def test_permission2():
    if hasattr(FOO, "hello") and hasattr(FOO, "bar") and hasattr(FOO, "baz"):
        FOO.hello
        FOO.bar
        FOO.baz

# Ask for forgiveness
def test_forgiveness2():
    try:
        FOO.hello
        FOO.bar
        FOO.baz
    except AttributeError:
        pass

Ask for Forgiveness or Look Before You Leap - example 3, where we check for an attribute, but that attribute doesn't exist:

# permission_vs_forgiveness2.py

class BaseClass:
    pass  # "hello" attribute is now removed

class Foo(BaseClass):
    pass

FOO = Foo()

# Look before you leap
def test_permission3():
    if hasattr(FOO, "hello"):
        FOO.hello

# Ask for forgiveness
def test_forgiveness3():
    try:
        FOO.hello
    except AttributeError:
        pass

Find Item in a List - for loop and a generator expression for finding the first number divisible by 42 and 43. They both use count() function inside:

# find_item.py

from itertools import count

def count_numbers():
    for item in count(1):
        if (item % 42 == 0) and (item % 43 == 0):
            return item

def generator():
    return next(item for item in count(1) if (item % 42 == 0) and (item % 43 == 0))

For Loop vs. List Comprehension - for loop and a list comprehension for creating a filtered list of numbers:

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def for_loop():
    output = []
    for element in MILLION_NUMBERS:
        if not element % 2:
            output.append(element)
    return output

def list_comprehension():
    return [number for number in MILLION_NUMBERS if not number % 2]

Sorting Lists - list.sort() and sorted() for sorting a list of random numbers:

# sorting.py

from random import sample

# List of 1 000 000 integers randomly shuffled
MILLION_RANDOM_NUMBERS = sample(range(1_000_000), 1_000_000)

def test_sort():
    random_list = MILLION_RANDOM_NUMBERS[:]
    return random_list.sort()

def test_sorted():
    random_list = MILLION_RANDOM_NUMBERS[:]
    return sorted(random_list)

Remove Duplicates From a List - removing duplicates from a list with a for loop and by converting list to a set and back to a list:

# duplicates.py

from random import randrange

DUPLICATES = [randrange(100) for _ in range(1_000_000)]

def test_for_loop():
    unique = []
    for element in DUPLICATES:
        if element not in unique:
            unique.append(element)
    return unique

def test_set():
    return list(set(DUPLICATES))

Slower scripts

With the examples from "Writing Faster Python" articles, we have a good variety of common operations. We do attribute lookups, handle exceptions, we test iterators, generators, loops and lists comprehensions, etc.

But all those examples are rather fast to run, so just for good measure, let's add two more functions that are intended to be more computational-heavy and run for at least a few seconds:

Bubble sort - a fairly slow sorting algorithm. Let's run it on a list of 10 000 numbers in descending order, which should take a couple of seconds on my computer:

# bubble_sort.py

DESCENDING_10_000 = list(range(10_000, 0, -1))

def bubble_sort():
    numbers = DESCENDING_10_000[:]
    changed = True
    while changed:
        changed = False
        for i in range(len(numbers) - 1):
            if numbers[i] > numbers[i+1]:
                numbers[i], numbers[i+1] = numbers[i+1], numbers[i]
                changed = True
    return numbers

Monte Carlo estimation of the π number. This is a simple simulation where we draw a square with a side of 1, and inside we draw a circle (so it has a diameter of 1). Then we throw a bunch of darts (or generate random points in case we don't have a large pile of virtual darts) inside that square. This lets us estimate the area of both the square and the circle by simply counting the number of darts that landed inside each of them. By definition, all the darts will end up inside the square, but only some will land in the circle. Finally, we know from school that the circle's area divided by the square's area is equal to π/4. So we do that division, and we get the estimation of π. The more darts we throw, the better the estimation is. Here is a visual explanation of this method.

Again, there are more efficient algorithms to do this simulation (e.g., using NumPy), but I want a slow version on purpose:

# pi_estimation.py

from random import random
from math import sqrt

# Total number of darts to throw.
TOTAL = 100_000_000

def estimate_pi():
    # Number of darts that land inside the circle.
    inside = 0

    for _ in range(TOTAL):
        x2 = random()**2
        y2 = random()**2
        # Check if the x and y points lie inside the circle
        if sqrt(x2 + y2) < 1.0:
            inside += 1
    return (float(inside) / TOTAL) * 4

Benchmarks

With 14 functions to check, we are ready to start our benchmarks. To run all of them at once, I've created a simple bash script to run all functions under different Python versions. I use pyenv to install the latest versions of Python, starting from 3.7, and then I use Python executables from each of those versions. Finally, I print the results in a nice table.

Here is the bash script I came up with. Don't worry if you don't understand how it works. I probably won't understand it one month from now, either.

#!/usr/bin/env bash

# Python versions that we will test
PYENV_VERSIONS=(3.7.14 3.8.14 3.9.14 3.10.7 3.11.0)

# Setup code and the actual functions that we will benchmark
COMMANDS=(
    "-s 'from permission_vs_forgiveness import test_permission2' 'test_permission2()'"
    "-s 'from permission_vs_forgiveness import test_forgiveness2' 'test_forgiveness2()'"
    "-s 'from permission_vs_forgiveness2 import test_permission3' 'test_permission3()'"
    "-s 'from permission_vs_forgiveness2 import test_forgiveness3' 'test_forgiveness3()'"
    "-s 'from find_item import count_numbers' 'count_numbers()'"
    "-s 'from find_item import generator' 'generator()'"
    "-s 'from filter_list import for_loop' 'for_loop()'"
    "-s 'from filter_list import list_comprehension' 'list_comprehension()'"
    "-s 'from sorting import test_sort' 'test_sort()'"
    "-s 'from sorting import test_sorted' 'test_sorted()'"
    "-s 'from duplicates import test_for_loop' 'test_for_loop()'"
    "-s 'from duplicates import test_set' 'test_set()'"
    "-s 'from bubble_sort import bubble_sort' 'bubble_sort()'"
    "-s 'from pi_estimation import estimate_pi' 'estimate_pi()'"
)

OUTPUT="Function,"
# Create a header with version numbers
for v in ${PYENV_VERSIONS[@]}
do
    OUTPUT+="$v,"
done

# Last column will contain difference between 1st and last version of Python in the PYENV_VERSIONS
OUTPUT+="${PYENV_VERSIONS[0]}/${PYENV_VERSIONS[${#PYENV_VERSIONS[@]}-1]}"
OUTPUT+="\n"

for (( i = 0; i < ${#COMMANDS[@]} ; i++ ))
do
    # Remove the single quotes from function name
    OUTPUT+=$(echo ${COMMANDS[$i]##*\ } | tr -d "'")

    for v in ${PYENV_VERSIONS[@]}
    do
        OUTPUT+=","
        OUTPUT+=$(eval "/Users/switowski/.pyenv/versions/$v/bin/python -m timeit ${COMMANDS[$i]}" | sed -e 's/.*: \(.*\) per loop/\1/')
    done
    # Divide timings for the first and last Python version and add it in the last column
    v1=$(eval "/Users/switowski/.pyenv/versions/${PYENV_VERSIONS[0]}/bin/python -m timeit ${COMMANDS[$i]}" | sed -e 's/.*: \(.*\) per loop/\1/' -e 's/[^0-9\.]//g')
    v2=$(eval "/Users/switowski/.pyenv/versions/${PYENV_VERSIONS[${#PYENV_VERSIONS[@]}-1]}/bin/python -m timeit ${COMMANDS[$i]}" | sed -e 's/.*: \(.*\) per loop/\1/' -e 's/[^0-9\.]//g')
    difference=$(echo "scale=2; $v1 / $v2" | bc)
    OUTPUT+=",$difference"

    OUTPUT+="\n"
done

# Print in a table-like format
printf "$OUTPUT" | column -ts,

I've put all the code examples together with the benchmark script and the results in this repository. The actual benchmark script has one more version, in case you don't care about the table, but the raw output from the timeit functions.

Results

Let's see the results. The lower the number, the faster a given code example runs. In the last column, we can see the comparison of how long it takes to run the code in Python 3.7 vs. Python 3.11. "1.68" means this example runs 68% slower in Python 3.7.

I did a bit of cleanup by moving the units next to the function name (instead of next to each number as in the original output).

Function	3.7.14	3.8.14	3.9.14	3.10.7	3.11.0	3.7/3.11
test_permission2() [nsec]	218	145	148	145	140	1.68
test_forgiveness2() [nsec]	91.9	70.4	72	83.1	71.7	1.31
test_permission3() [nsec]	77.4	60.9	61.9	57.1	40.5	1.88
test_forgiveness3() [µsec]	256	251	239	283	307	.83
count_numbers() [µsec]	46.8	47.5	47.4	46.6	41	1.14
generator() [µsec]	47.1	47.7	47.6	45.3	39.5	1.18
for_loop() [msec]	27.2	26.5	26.8	25.6	19.4	1.39
list_comprehension() [msec]	18.3	18	18.6	17.7	17.3	1.04
test_sort() [msec]	175	175	176	176	175	.97
test_sorted() [msec]	183	183	186	183	185	1.00
test_for_loop() [msec]	360	364	316	305	308	1.17
test_set() [msec]	5.59	5.57	5.83	6.09	6.08	.91
bubble_sort() [sec]	8.05	8.24	8.23	7.89	4.69	1.72
estimate_pi() [sec]	17.1	17.9	18.1	17.4	14.3	1.21

We can see that in most cases, our examples run faster as we upgrade the Python version. And Python 3.11 gives us the best improvements. Upgrading your Python version now makes even more sense than before if you're looking for speed improvements.

But for some examples, we see a degradation of performance. The 0.97 for test_sort() and 0.91 for test_set() differences are so small that I assume it's the small randomness of the benchmark results. But the test_forgiveness3() with around 20% decrease in performance in Python 3.11 looked interesting. I checked the release notes for Python 3.11 to find what might be causing this and found nothing. So I decided to compare how Python handles exceptions for the most common example - division by zero:

# division.py
def divide_by_zero():
    try:
        1/0
    except ZeroDivisionError:
        pass

Benchmarking the above code under different Python versions gave me the following results:

Python 3.7.14: 161 nsec
Python 3.8.14: 170 nsec
Python 3.9.14: 165 nsec
Python 3.10.7: 141 nsec
Python 3.11.0: 169 nsec

In Python 3.11.0, it's almost as slow as in Python 3.7 or 3.8. So it seems like the slowdown for my test_forgiveness3() was specific to this one particular example and not something we should be worried about. And while this example is slower, all the other examples of testing permission and forgiveness got much faster in the newer Python versions. In Python 3.11, the "ask for permission" gets an additional speed boost from the "zero cost" exception handling.

"Zero cost" exception handling

Python 3.11 introduced something called "zero cost" exception handling. This Hacker News submission explains how this works in Python and other languages. The gist of this feature is that everything inside the "try" block (the "happy path" of the exception) will now be faster - almost as fast as if there was no try/except block at all.

Let's see this in action!

I created one more short benchmarking script. I took 3 code examples (for loop for filtering a list, bubble sort, and the pi estimation) and wrapped their most inner instructions in a try/except block (so that this try/except block is executed as often as possible). At the same time, since there are no exceptions, the "except" block is never called, so I can just put pass inside.

So, for example, the first test case will compare those two variants:

MILLION_NUMBERS = list(range(1_000_000))

def for_loop():
    output = []
    for element in MILLION_NUMBERS:
        if not element % 2:
            output.append(element)
    return output

def for_loop_with_try_except():
    output = []
    for element in MILLION_NUMBERS:
        if not element % 2:
            try:
                output.append(element)
            except Exception:
                pass
    return output

With zero cost exceptions handling, Python 3.11 should run those code examples faster than Python 3.10 or 3.9.

Let's see the results by running the exceptions_benchmark.sh script:

Function	3.9.14	3.10.7	3.11.0
Filter [msec]	26.7 (28.4)	26 (27.1)	19.6 (20.4)
Pi [sec]	18.4 (19.2)	17.3 (17.5)	14.1 (14.3)
Bubble [sec]	8.26 (8.46)	7.96 (8.06)	4.72 (4.75)

The first number in each column is how long it takes to run the original version (without try/except blocks). The number in parenthesis is how long it takes to run the same function with the try/except blocks called multiple times.

The differences between both variants are tiny for all 3 Python versions. But for Python 3.11 they are even smaller! Take this simple benchmark with a grain of salt, but I hope it helped illustrate what's the benefit of "zero cost" exception handling.

Conclusions

Upgrading Python version is one of a few ways to make your code a bit faster without changing it. And no matter if you upgrade from Python 3.7. to 3.8 or from Python 3.9 to Python 3.10, you will always get some improvements for a large codebase. But it's Python 3.11 where a dedicated effort was made to really speed it up. According to the release notes, it should speed up your code by around 10-60%. So now is a good time to think about upgrading your Python projects.

If you want to run your own benchmarks with more advanced code examples, the Python Performance Benchmark Suite is a good place to look for some inspiration.

Completely free if you have good tests coverage (in case of some subtle bugs between minor Python versions), all the libraries you are using work with newer Python version, and you have a few moments to install new Python version. ↩︎

Python Versions Management With pyenv

2021-02-03T00:00:00Z

Using the latest version of Python is always a good idea. First of all - you get the new features like the f-strings (Python 3.6), ordered dictionaries (officially guaranteed from Python 3.7, but already present in Python 3.6), or the union operator (Python 3.9). But even if you don't use those features, you get plenty of smaller improvements and optimizations. Python is not the language that I would choose when the speed matters, but getting a free speedup here and there only because I updated Python's version is nice to have.

Problems start when you work on multiple projects. Maybe you have one Python project at work and some other side-projects or tutorials you do after work. You can use the same Python version for all of them, but the chances are that the Python version you use at work is not the most recent one. Or rather, it's not even close to the "recent Python version." A lot of projects only update Python when it's absolutely necessary. Or maybe, like me, you have multiple projects at work, and you need to switch between different Python versions.

You could install different Python versions and use the python3.6, python3.7, python3.8, python3.9 commands. Or maybe even do some crazy setup with symlinks and change what the python command points to. But a much better idea is to use a tool called pyenv.

pyenv

pyenv is a tool for managing Python versions. You can use it to install different Python versions and easily switch between them. Need to use Python 3.9? Run pyenv global 3.9.0. Want to use Python 3.6 in a specific folder? Sure, just type pyenv local 3.6.0, and you are all set.

What's really cool about pyenv is that it doesn't touch the Python version installed on your computer (the system Python). It installs every new Python version inside a separate folder. Then it modifies the $PATH environment variable and tells your computer to use those Python versions (and not the system Python). That way, even if you mess up something with pyenv, you can just remove it, and you are back to using whatever Python version you had before installing it. Trust me - you will appreciate this separation on the day when you mess up your Python installation while rushing to fix a bug in production .😉

Installation

When you install pyenv, there are some prerequisites that you need to have. You can check out the installation instructions on GitHub for details, but basically, you need to have all the dependencies for building Python. Otherwise, pyenv won't be able to install any version of Python.

If you are using Windows, check out pyenv-win. It's a port of pyenv to Windows that contains most of its features. It might be missing some of the newest commands, but the most important ones (that I'm showing you here) are present.

You can install pyenv with your package manager, clone it from GitHub or use pyenv-installer. I prefer to use pyenv-installer (even though it requires me to pipe a script from the internet right into bash, which is a big security "no-no"). It automates the whole installation process and installs some additional plugins like pyenv-doctor (to check that pyenv works correctly), pyenv-update (for easy updates), or pyenv-virtualenv (for managing virtual environments). After the installation, you just get short instructions on what code you need to put in your profile script (.bashrc, .zshrc, or config.fish - depending on what type of shell you are using).

Once you finish installing it, make sure you follow the post-installation instructions. You will need to add pyenv init command in the correct place (otherwise, pyenv won't work) and install Python build dependencies (without them, you won't be able to install new Python versions). And you are ready to go!

You can check that pyenv was installed correctly by running pyenv versions (if you don't have any error message, then everything is fine). If you used the pyenv-installer script, you can also run pyenv doctor command. It will perform some checks and hopefully return a "success" message.

pyenv in action

With pyenv installed, you basically do two things:

Install a new Python version (pyenv install <version-number>)
Select that Python version (pyenv [global|local|shell] <version-number>) - I will explain that global/local/shell a bit later.

So, which versions of Python we can install? To get a list, run pyenv install --list:

$ pyenv install --list
Available versions:
  2.1.3
  2.2.3
  2.3.7
  ...
  3.9.0
  3.9-dev
  3.10-dev
  activepython-2.7.14
  activepython-3.5.4
  activepython-3.6.0
  anaconda-1.4.0
  anaconda-1.5.0
  anaconda-1.5.1
  ...
  pypy3.6-7.3.0
  pypy3.6-7.3.1-src
  pypy3.6-7.3.1
  pyston-0.5.1
  pyston-0.6.0
  pyston-0.6.1
  stackless-dev
  stackless-2.7-dev
  stackless-2.7.2
  stackless-2.7.3
  stackless-2.7.4
  stackless-2.7.5
  ...

This list contains the standard CPython versions (those that have just numbers, like 2.1.3, 3.9.0, etc.) and other distributions like activepython, anaconda, or pypy. If you ever wanted to test different Python distributions, now you can easily do this.

You will also notice that some of the latest versions of Python might be missing. That's because they are added manually, so unless someone creates a pull request that adds them, you have to use an older version. If you want to stay on the bleeding edge and install the latest Python version on the day it was released, then pyenv is not a tool for you. But if you don't mind staying one or two minor versions away from the latest one, you should be good.

Let's say we want to install Python 3.9.0. We run pyenv install 3.9.0, and we wait a bit. It can be a slow process (sometimes it takes a few minutes on my computer). To speed it up, make sure you have all the prerequisites installed. For example, if I don't have the openssl and readline already installed on my macOS, each time I try to install a new Python version, pyenv will first download and set up those two packages. So to save yourself some time, go ahead and install all the prerequisites. Otherwise, just go grab a coffee, and after a few minutes, we should be done.

You can see what versions of Python you have installed with pyenv versions command:

$ pyenv versions
  system
  2.7.18
  3.6.9
  3.8.3
* 3.9.0 (set by /Users/switowski/.pyenv/version)

system version is the one that comes with my operating system (by default, macOS comes with Python 2.7), and the rest of them were installed using pyenv.

Once you have some other Python versions available, you can switch between them using pyenv global <version-number>:

$ python --version
Python 3.9.0

$ pyenv global 2.7.18

$ python --version
Python 2.7.18

$ pyenv global 3.6.9

$ python --version
Python 3.6.9

pyenv global changes the global Python version on your computer. In most cases, that's what you want. But there are some other options when you want to switch Python version for a specific case.

local and shell Python versions

If you have a project that uses a specific version of Python (different from the global version), then each time you want to work on this project, you need to switch Python version and then switch it back when you are done. Luckily, pyenv comes with pyenv local command that can help us here:

$ cd python3.6-project/

$ pyenv local 3.6.9

$ python --version
Python 3.6.9

$ cd ..

$ python --version
Python 3.9.0

pyenv local changes the Python version only for the current folder and all the subfolders. That's exactly what you want for your project - you want to use a different Python version in this folder without changing the global one. pyenv local command creates a .python-version file in the current directory and puts the version number inside. When pyenv tries to determine what Python version it should use, it will search for that file in the current folder and all the parent folders. If it finds one, it uses the version specified in that file. And if it gets all the way up to your home folder without finding the .python-version, it will use the global version.

Let's take it one step further. What if you want to change the Python version only temporarily - just to run a few commands? Maybe you want to see how some command works with different Python versions. Or maybe you really miss the times when print was a statement, and you want to feel the nostalgia of Python 2 one more time? That's when you can use the pyenv shell:

$ pyenv shell 2.7.18

$ python --version
Python 2.7.18

$ python -c "print 'Good old times, right?'"
Good old times, right?

pyenv shell changes the Python version for the current session. You can use a different Python version, but when you close your terminal, it gets back to whatever global or local Python version you were using before.

And that's pretty much how you can use pyenv.

A quick troubleshooting tip

It can happen that after you install a new Python version, pyenv won't detect it. So when you try to switch to that version, you will get an error message saying that it's not installed. To fix that, either restart your terminal or run pyenv rehash.

asdf-vm

pyenv is based on rbenv - a version manager for Ruby that works in the same way. And there are similar tools for other languages: nodenv, goenv, and so on.

If you use many different programming languages, installing and managing all those *env tools can be tedious. Luckily, there is a "one tool to rule them all" called asdf-vm. Behind this weird name (after I've heard about it, it took me ages to find it back!), we have a program to manage different versions of programming languages or even tools (you can use it to change what version of CMake, ImageMagic, or kubectl you use).

It works similarly to pyenv. You first install a plugin (for example, for Python), then you install new versions (version 3.9.0 of Python), and you use a set of commands to select a global/local/shell version. It's a super useful tool, and I recommend it if you're tired of this mess with different versions of different programming languages on your computer.

25 IPython Tips for Your Next Advent of Code

2021-01-27T00:00:00Z

I've decided to skip last year's Advent of Code edition. Mostly because I didn't have time, but I also knew that I probably wouldn't finish it. I've never finished any edition. I'm not very good at code katas, and I usually try to brute force them. With AoC, that works for the first ten days, but then the challenges start to get more and more complicated, and adding the @jit decorator to speed up my ugly Python code can only get me so far.

But one thing that helped me a lot with the previous editions was to use IPython. Solving those problems incrementally is what actually makes it fun. You start by hard-coding the simple example that comes with each task. Then you try to find a solution for this small-scale problem. You try different things, you wrangle with the input data, and after each step, you see the output, so you know if you are getting closer to solving it or not. Once you manage to solve the simple case, you load the actual input data, and you run it just to find out that there were a few corner cases that you missed. It wouldn't be fun if I had to use a compiled language and write a full program to see the first results.

This year, instead of doing the "Advent of Code," I've decided to do an "Advent of IPython" on Twitter - for 25 days, I've shared tips that can help you when you're solving problems like AoC using IPython. Here is a recap of what you can do.

1. Display the documentation

In [1]: import re

In [2]: re.findall?
Signature: re.findall(pattern, string, flags=0)
Docstring:
Return a list of all non-overlapping matches in the string.

If one or more capturing groups are present in the pattern, return
a list of groups; this will be a list of tuples if the pattern
has more than one group.

Empty matches are included in the result.
File:      ~/.pyenv/versions/3.9.0/lib/python3.9/re.py
Type:      function

That's one of my favorite features. You can display the documentation of any function, module, and variable by adding the "?" at the beginning or at the end of it. It's called "dynamic object introspection," and I love it because I don't have to leave the terminal to get the documentation. You can use the built-in help() function to get this information with the standard Python REPL, but I find the "?" much more readable. It highlights the most important information like the signature and the docstring, and it comes with colors (even though you can't see them here because my syntax highlighting library doesn't support IPython).

2. Display the source code

In [1]: import pandas

In [2]: pandas.DataFrame??

Init signature:
pandas.DataFrame(
    data=None,
    index: Optional[Collection] = None,
    columns: Optional[Collection] = None,
    dtype: Union[ForwardRef('ExtensionDtype'), str, numpy.dtype, Type[Union[str, float, int, complex, bool]], NoneType] = None,
    copy: bool = False,
)
Source:
class DataFrame(NDFrame):
    """
    Two-dimensional, size-mutable, potentially heterogeneous tabular data.

    Data structure also contains labeled axes (rows and columns).
    Arithmetic operations align on both row and column labels. Can be
    thought of as a dict-like container for Series objects. The primary
    pandas data structure.

    Parameters
    ----------

... and so on

And if you want to see the full source code of a function (or class/module), use two question marks instead (function_name?? or ??function_name).

3. %edit magic function

If you want to write a long function, use the %edit magic command. It will open your favorite editor (or actually the one that you set with the $EDITOR environment variable) where you can edit your code. When you save and close this file, IPython will automatically execute it.

I use it with vim, and it works great when I want to write a bit longer function (with vim I have a lightweight linter, and moving around the code is faster). It's a nice middle ground when you are too lazy to switch to your code editor to write the whole code, but at the same time, the function that you are writing is a bit too big to write it comfortably in IPython.

4. Reopen last file with "%edit -p"

And speaking of the %edit command, you can run %edit -p to reopen the same file that you edited the last time. This is useful if you made a mistake and you want to fix it without having to type everything again or if you want to add more code to the function that you just wrote.

5. Wildcard search

In [1]: import os

In [2]: os.*dir*?
os.__dir__
os.chdir
os.curdir
os.fchdir
os.listdir
os.makedirs
os.mkdir
os.pardir
os.removedirs
os.rmdir
os.scandir
os.supports_dir_fd

In [3]: os.chdir("/some/other/dir")

If you forget the name of some function, you can combine the dynamic object introspection (the "?") and a wildcard (the "*") to perform a wildcard search. For example, I know that the os module has a function to change the current directory, but I don't remember its name. I can list all the functions from the os module, but I'm sure that a function like this must contain "dir" in its name. So I can limit the search and list all the functions from the os module that contain "dir" in their names.

6. post-mortem debugging

In [1]: from solver import solve

In [2]: solve()
IndexError: list index out of range

In [3]: %debug
> /Users/switowski/workspace/iac/solver.py(11)count_trees()
      9         x = (x + dx) % mod
     10         y += dy
---> 11         if values[y][x] == "#":
     12             count += 1
     13     return count

ipdb>

Displaying the documentation is one of my favorite features, but post-mortem debugging is my favorite feature. After you get an exception, you can run %debug, and it will start a debugging session for that exception. That's right! You don't need to put any breakpoints or run IPython with any special parameters. You just start coding, and if when an exception happens, you run this command to start debugging.

7. Start the debugger automatically

In [1]: %pdb
Automatic pdb calling has been turned ON

In [2]: from solver import solve

In [3]: solve()
IndexError: list index out of range

> /Users/switowski/workspace/iac/solver.py(11)count_trees()
      9         x = (x + dx) % mod
     10         y += dy
---> 11         if values[y][x] == "#":
     12             count += 1
     13     return count

ipdb> y
1
ipdb> x
3
ipdb>

And if you want to start a debugger on every exception automatically, you can run %pdb to enable the automatic debugger. Run %pdb again to disable it.

8. Run shell commands

In [1]: !pwd
/Users/switowski/workspace/iac

In [2]: ls -al
total 8
drwxr-xr-x   5 switowski  staff   480 Dec 21 17:26 ./
drwxr-xr-x  55 switowski  staff  1760 Dec 22 14:47 ../
drwxr-xr-x   9 switowski  staff   384 Dec 21 17:27 .git/
drwxr-xr-x   4 switowski  staff   160 Jan 25 11:39 __pycache__/
-rw-r--r--   1 switowski  staff   344 Dec 21 17:26 solver.py

# Node REPL inside IPython? Sure!
In [3]: !node
Welcome to Node.js v12.8.0.
Type ".help" for more information.
> var x = "Hello world"
undefined
> x
'Hello world'
>

You can run shell commands without leaving IPython - you just need to prefix it with the exclamation mark. And the most common shell commands like ls, pwd, cd will work even without it (of course, unless you have a Python function with the same name).

I use it mostly to move between folders or to move files around. But you can do all sorts of crazy things - including starting a REPL for a different programming language inside IPython.

9. Move around the filesystem with %cd

In [1]: !pwd
/Users/switowski/workspace/iac/input_files/wrong/folder

In [2]: %cd ../..
/Users/switowski/workspace/iac/input_files

In [3]: %cd right_folder/
/Users/switowski/workspace/iac/input_files/right_folder

Alternatively, you can also move around the filesystem using the %cd magic command (press Tab to get the autocompletion for the list of available folders). It comes with some additional features - you can bookmark a folder or move a few folders back in the history (run %cd? to see the list of options).

10. %autoreload

Use %autoreload to automatically reload all the imported functions before running them. By default, when you import a function in Python, Python "saves its source code in memory" (ok, that's not what actually happens, but for illustration purposes, let's stick with that oversimplification). When you change the source code of that function, Python won't notice the change, and it will keep using the outdated version.

If you are building a function or a module and you want to keep testing the latest version without restarting the IPython (or using the importlib.reload()), you can use the %autoreload magic command. It will always reload the source code before running your functions. If you want to learn more - I wrote a longer article about it.

11. Change the verbosity of exceptions

By default, the amount of information in IPython's exceptions is just right - at least for me. But if you prefer to change that, you can use the %xmode magic command. It will switch between 4 levels of traceback's verbosity. Check it out - it's the same exception, but the traceback gets more and more detailed:

Minimal

In [1]: %xmode
Exception reporting mode: Minimal

In [2]: solve()
IndexError: list index out of range

Plain

In [3]: %xmode
Exception reporting mode: Plain

In [4]: solve()
Traceback (most recent call last):
File "<ipython-input-6-6f300b4f5987>", line 1, in <module>
    solve()
File "/Users/switowski/workspace/iac/solver.py", line 27, in solve
    sol_part1 = part1(vals)
File "/Users/switowski/workspace/iac/solver.py", line 16, in part1
    return count_trees(vals, 3, 1)
File "/Users/switowski/workspace/iac/solver.py", line 11, in count_trees
    if vals[y][x] == "#":
IndexError: list index out of range

Context (that's the default setting)

In [5]: %xmode
Exception reporting mode: Context

In [6]: solve()
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-8-6f300b4f5987> in <module>
----> 1 solve()

~/workspace/iac/solver.py in solve()
    25 def solve():
    26     vals = getInput()
---> 27     sol_part1 = part1(vals)
    28     print(f"Part 1: {sol_part1}")
    29     print(f"Part 2: {part2(vals, sol_part1)}")

~/workspace/iac/solver.py in part1(vals)
    14
    15 def part1(vals: list) -> int:
---> 16     return count_trees(vals, 3, 1)
    17
    18 def part2(vals: list, sol_part1: int) -> int:

~/workspace/iac/solver.py in count_trees(vals, dx, dy)
    9         x = (x + dx) % mod
    10         y += dy
---> 11         if vals[y][x] == "#":
    12             cnt += 1
    13     return cnt

IndexError: list index out of range

Verbose (like "Context" but also shows the values of local and global variables)

In [7]: %xmode
Exception reporting mode: Verbose

In [8]: solve()
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-10-6f300b4f5987> in <module>
----> 1 solve()
        global solve = <function solve at 0x109312b80>

~/workspace/iac/solver.py in solve()
    25 def solve():
    26     values = read_input()
---> 27     part1 = solve1(values)
        part1 = undefined
        global solve1 = <function solve1 at 0x109f363a0>
        values = [['..##.......', ..., '.#..#...#.#']]
    28     print(f"Part 1: {part1}")
    29     print(f"Part 2: {solve2(values, part1)}")

~/workspace/iac/solver.py in solve1(values=[['..##.......', ..., '.#..#...#.#']])
    14
    15 def solve1(values: list) -> int:
---> 16     return count_trees(values, 3, 1)
        global count_trees = <function count_trees at 0x109f364c0>
        values = [['..##.......', ..., '.#..#...#.#']]
    17
    18 def solve2(values: list, sol_part1: int) -> int:

... and so on

IndexError: list index out of range

12. Rerun commands from the previous sessions

In [1]: a = 10

In [2]: b = a + 20

In [3]: b
Out[3]: 30

# Restart IPython

In [1]: %rerun ~1/
=== Executing: ===
a = 10
b = a + 20
b
=== Output: ===
Out[1]: 30

In [2]: b
Out[2]: 30

You can use the %rerun ~1/ to rerun all the commands from the previous session. That's a great way to get you back to the same place where you left IPython. But it has one huge downside - if you had any exception (and I'm pretty sure you did), the execution will stop there. So you have to remove the lines with exceptions manually. If you are using Jupyter Notebooks, there is a workaround that allows you to tag a notebook cell as "raising an exception." If you rerun it, IPython will ignore this exception. It's not a perfect solution, and an option to ignore exceptions during the %rerun command would be much better.

13. Execute some code at startup

If you want to execute some code each time you start IPython, just create a new file inside the "startup" folder (~/.ipython/profile_default/startup/) and add your code there. IPython will automatically execute any files it finds in this folder. It's great if you want to import some modules that you use all the time, but if you put too much code there, the startup time of IPython will be slower.

14. Use different profiles

Maybe you have a set of modules that you want to import and settings to set in a specific situation. For example, when debugging/profiling, you want to set the exceptions to the verbose mode and import some profiling libraries. Don't put that into the default profile because you don't debug or profile your code all the time. Create a new profile and put your debugging settings inside. Profiles are like different user accounts for IPython - each of them has its own configuration file and startup folder.

15. Output from the previous commands

In [1]: sum(range(1000000))
Out[1]: 499999500000

In [2]: the_sum = _

In [3]: the_sum
Out[3]: 499999500000

In [4]: _1
Out[4]: 499999500000

If you forgot to assign an expression to a variable, use var = _. _ stores the output of the last command (this also works in the standard Python REPL). The results of all the previous commands are stored in variables _1 (output from the first command), _2 (output from the second command), etc.

16. Edit any function or module

You can use %edit to edit any Python function. And I really mean ANY function - functions from your code, from packages installed with pip, or even the built-in ones. You don't even need to know in which file that function is located. Just specify the name (you have to import it first), and IPython will find it for you.

In the above example, I'm breaking the built-in randint() function by always returning 42.

In [1]: welcome = "Welcome to my gist"

In [2]: welcome
Out[2]: 'Welcome to my gist'

In [3]: a = 42

In [4]: b = 41

In [5]: a - b
Out[5]: 1

In [6]: %pastebin 1-5
Out[6]: 'http://dpaste.com/8QA86F776'

If you want to share your code with someone, use the %pastebin command and specify which lines you want to share. IPython will create a pastebin (something similar to GitHub gist), paste selected lines, and return a link that you can send to someone. Just keep in mind that this snippet will expire in 7 days.

18. Use IPython as your debugger

Maybe some of the tips that I've shared convinced you that IPython is actually pretty cool. If that's the case, you can use it not only as a REPL (the interactive Python shell) but also as a debugger. IPython comes with "ipdb" - it's like the built-in Python debugger "pdb", but with some IPython's features on top of it (syntax highlighting, autocompletion, etc.)

You can use ipdb with your breakpoint statements by setting the PYTHONBREAKPOINT environment variable - it controls what happens when you call breakpoint() in your code. This trick requires using Python 3.7 or higher (that's when the breakpoint() statement was introduced).

19. Execute code written in another language

In [1]: %%ruby
   ...: 1.upto 16 do |i|
   ...:   out = ""
   ...:   out += "Fizz" if i % 3 == 0
   ...:   out += "Buzz" if i % 5 == 0
   ...:   puts out.empty? ? i : out
   ...: end
   ...:
   ...:
1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16

Let's say you want to execute some code written in another language without leaving IPython. You might be surprised to see that IPython supports Ruby, Bash, or JavaScript out of the box. And even more languages can be supported when you install additional kernels!

Just type %%ruby, write some Ruby code, and press Enter twice, and IPython will run it with no problem. It also works with Python2 (%%python2).

20. Store variables between sessions

In [1]: a = 100

In [2]: %store a
Stored 'a' (int)

# Restart IPython
In [1]: %store -r a

In [2]: a
Out[2]: 100

IPython uses SQLite for some lightweight storage between sessions. That's where it saves the history of your previous sessions. But you can use it to store your own data. For example, with the %store magic command, you can save variables in IPython's database and restore them in another session using %store -r. You can also set the c.StoreMagics.autorestore = True in the configuration file to automatically restore all the variables from the database when you start IPython.

21. Save session to a file

In [1]: a = 100

In [2]: b = 200

In [3]: c = a + b

In [4]: c
Out[4]: 300

In [5]: %save filename.py 1-4
The following commands were written to file `filename.py`:
a = 100
b = 200
c = a + b
c

You can save your IPython session to a file with the %save command. That's quite useful when you have some working code and you want to continue editing it with your text editor. Instead of manually copying and pasting lines to your code editor, you can dump the whole IPython session and then remove unwanted lines.

22. Clean up ">" symbols and fix indentation

# Clipboard content:
# >def greet(name):
# >    print(f"Hello {name}")

# Just pasting the code won't work
In [1]: >def greet(name):
   ...: >    print(f"Hello {name}")
  File "<ipython-input-1-a7538fc939af>", line 1
    >def greet(name):
    ^
SyntaxError: invalid syntax


# But using %paste works
In [2]: %paste
>def greet(name):
>    print(f"Hello {name}")

## -- End pasted text --

In [3]: greet("Sebastian")
Hello Sebastian

If you need to clean up incorrect indentation or ">" symbols (for example, when you copy the code from a git diff, docstring, or an email), instead of doing it manually, copy the code and run %paste. IPython will paste the code from your clipboard, fix the indentation, and remove the ">" symbols (although it sometimes doesn't work properly).

23. List all the variables

In [1]: a = 100

In [2]: name = "Sebastian"

In [3]: squares = [x*x for x in range(100)]

In [4]: squares_sum = sum(squares)

In [5]: def say_hello():
   ...:     print("Hello!")
   ...:

In [6]: %whos
Variable      Type        Data/Info
-----------------------------------
a             int         100
name          str         Sebastian
say_hello     function    <function say_hello at 0x111b60a60>
squares       list        n=100
squares_sum   int         328350

You can get a list of all the variables from the current session (nicely formatted, with information about their type and the data they store) with the %whos command.

24. Use asynchronous functions

In [1]: import asyncio

In [2]: async def worker():
   ...:     print("Hi")
   ...:     await asyncio.sleep(2)
   ...:     print("Bye")
   ...:

# The following code would fail in the standard Python REPL
# because we can't call await outside of an async function
In [3]: await asyncio.gather(worker(), worker(), worker())
Hi
Hi
Hi
Bye
Bye
Bye

You can speed up your code with asynchronous functions. But the thing about asynchronous code is that you need to start an event loop to call them. However, IPython comes with its own event loop! And with that, you can await asynchronous functions just like you would call a standard, synchronous one.

25. IPython scripts

$ ls
file1.py    file2.py    file3.py    file4.py    wishes.ipy

$ cat wishes.ipy
files = !ls
# Run all the files with .py suffix
for file in files:
    if file.endswith(".py"):
        %run $file

$ ipython wishes.ipy
Have a
Very Merry
Christmas!
🎄🎄🎄🎄🎄🎄

You can execute files containing IPython-specific code (shell commands prefixed with ! or magic methods prefixed with %). Just save the file with the ".ipy" extension and then pass it to the ipython command.

Conclusions

If you have been reading my blog for a bit, you probably already realize that IPython is one of my favorite Python tools. It's an excellent choice for solving code challenges like the Advent of Code, and it has a lot of cool tricks that can help you. Leave a comment if you know some other cool tricks that you want to share!

Remove Duplicates From a List

2020-10-22T00:00:00Z

How do we remove duplicates from a list? One way is to go through the original list, pick up unique values, and append them to a new list.

About the "Writing Faster Python" series

Are those recommendations going to make your code much faster? Not really.
Is knowing those small differences going to make a slightly better Python programmer? Hopefully!

Let's prepare a simple test. I will use the randrange to generate 1 million random numbers between 0 and 99 (this will guarantee some duplicates):

# duplicates.py

from random import randrange

DUPLICATES = [randrange(100) for _ in range(1_000_000)]

Throwaway variable

If you are wondering what's this _ variable - that's a convention used in Python code when you need to declare a variable, but you are not planning to use it (a throwaway variable). In the above code, I want to call randrange(100) 1 million times. I can't omit the variable and just write randrange(100) for range(1_000_000) - I would get a syntax error. Since I need to specify a variable, I name it _ to indicate that I won't use it. I could use any other name, but _ is a common convention.

Keep in mind that in a Python REPL, _ actually stores the value of the last executed expression. Check out this StackOverflow answer for a more detailed explanation.

We have 1 million numbers. Now, let's remove duplicates using a "for loop."

# duplicates.py

def test_for_loop():
    unique = []
    for element in DUPLICATES:
        if element not in unique:
            unique.append(element)
    return unique

Since we are operating on a list, you might be tempted to use list comprehension instead:

>>> unique = []
>>> [unique.append(num) for num in DUPLICATES if num not in unique]

In general, this is not a good way to use a list comprehension because we use it only for the side effects. We don't do anything with the list that we get out of the comprehension. It looks like a nice one-liner (and I might use it in a throwaway code), but:

It hides the intention of the code. List comprehension creates a list. But in our case, we actually hide a "for loop" inside!
It's wasteful - we create a list (because list comprehension always creates a list) just to discard it immediately.

I try to avoid using list comprehension just for the side effects. "For loop" is much more explicit about the intentions of my code.

Remove duplicates with `set()`

There is a much simpler way to remove duplicates - by converting our list to a set. Set, by definition, is a "collection of distinct (unique) items." Converting a list to a set automatically removes duplicates. Then you just need to convert this set back to a list:

# duplicates.py

def test_set():
    return list(set(DUPLICATES))

Which one is faster?

$ python -m timeit -s "from duplicates import test_for_loop" "test_for_loop()"
1 loop, best of 5: 634 msec per loop

$ python -m timeit -s "from duplicates import test_set" "test_set()"
20 loops, best of 5: 11 msec per loop

Converting our list to a set is over 50 times faster (634/11≈57.63) than using a "for loop." And a hundred times cleaner and easier to read 😉.

Unhashable items

This above method of converting a list to a set only works if a list is hashable. So it's fine for strings, numbers, tuples, and any immutable objects. But it won't work for unhashable elements like lists, sets, or dictionaries. So if you have a list of nested lists, your only choice is to use that "bad" for loop. That's why "bad" is in quotes - it's not always bad.

To learn more about the difference between hashable and unhashable objects in Python, check out this StackOverflow question: What does "hashable" mean in Python?

Remove duplicates while preserving the insertion order

There is one problem with sets - they are unordered. When you convert a list to a set, there is no guarantee that it will keep the insertion order. If you need to preserve the original order, you can use this dictionary trick:

# duplicates.py

def test_dict():
    return list(dict.fromkeys(DUPLICATES))

Here is what the above code does:

It creates a dictionary using fromkeys() method. Each element from DUPLICATES is a key with a value of None. Dictionaries in Python 3.6 and above are ordered, so the keys are created in the same order as they appeared on the list. Duplicated items from a list are ignored (since dictionaries can't have duplicated keys).
Then it converts a dictionary to a list - this returns a list of keys. Again, we get those keys in the same order as we inserted into the dictionary in the previous step.

What about the performance?

$ python -m timeit -s "from duplicates import test_dict" "test_dict()"
20 loops, best of 5: 17.9 msec per loop

It's 62% slower than using a set (17.9/11≈1.627), but still over 30 times faster than the "for loop" (634/17.3≈35.419).

The above method only works with Python 3.6 and above. If you are using an older version of Python, replace dict with OrderedDict:

# duplicates.py
from collections import OrderedDict

def test_ordereddict():
    return list(OrderedDict.fromkeys(DUPLICATES))

$ python -m timeit -s "from duplicates import test_ordereddict" "test_ordereddict()"
10 loops, best of 5: 32.8 msec per loop

It's around 3 times as slow as a set (32.8/11≈2.982) and 83% slower than a dictionary (32.8/17.9≈1.832), but it's still much faster than a "for loop" (634/32.8≈19.329). And OrderedDict will work with Python 2.7 and any Python 3 version.

Conclusions

When you need to remove duplicates from a collection of items, the best way to do this is to convert that collection to a set. By definition, the set contains unique items (among other features, like the constant membership testing time). This will make your code faster and more readable.

Downsides? Sets are unordered, so if you need to make sure you don't lose the insertion order, you need to use something else. For example - a dictionary!

type() vs. isinstance()

2020-10-15T00:00:00Z

Python is a dynamically typed language. A variable, initially created as a string, can be later reassigned to an integer or a float. And the interpreter won't complain:

name = "Sebastian"
# Dynamically typed language lets you do this:
name = 42
name = None
name = Exception()

It's quite common to see code that checks variable's type. Maybe you want to accept both a single element and a list of items and act differently in each case. That's what the SMTP.sendmail() from the smtplib does. It checks if the recipient is a string or a list of strings and sends one or more emails.

About the "Writing Faster Python" series

Are those recommendations going to make your code much faster? Not really.
Is knowing those small differences going to make a slightly better Python programmer? Hopefully!

To check the type of a variable, you can use either type() or isinstance() built-in function. Let's see them in action:

>>> variable = "hello"
>>> type(variable) is str
True
>>> isinstance(variable, str)
True

Let's compare both methods' performance:

$ python -m timeit -s "variable = 'hello'" "type(variable) is str"
5000000 loops, best of 5: 52.1 nsec per loop

$ python -m timeit -s "variable = 'hello'" "isinstance(variable, str)"
10000000 loops, best of 5: 35.5 nsec per loop

type is around 40% slower (52.1/35.5≈1.47).

We could use type(variable) == str instead, but it's a bad idea. == should be used when you want to check the value of a variable. We would use it to see if the value of variable is equal to "hello". But when we want to check if variable is a string, is operator is more appropriate. For a more detailed explanation of when to use one or the other, check this article.

Python 3.11 update

In Python 3.11, the difference between the two above code snippets becomes almost negligible:

# Python 3.11.0

$ python -m timeit -s "variable = 'hello'" "type(variable) is str"
20000000 loops, best of 5: 12.3 nsec per loop

$ python -m timeit -s "variable = 'hello'" "isinstance(variable, str)"
20000000 loops, best of 5: 12.7 nsec per loop

That's around a 3% difference. But the following recommendations are still valid no matter which version of Python you are using.

Difference between `isinstance` and `type`

Speed is not the only difference between these two functions. There is actually an important distinction between how they work:

type only returns the type of an object (its class). We can use it to check if variable is of a type str.
isinstance checks if a given object (first parameter) is:
- an instance of a class specified as a second parameter. For example, is variable an instance of the str class?
- or an instance of a subclass of a class specified as a second parameter. In other words - is variable an instance of a subclass of str?

What does it mean in practice? Let's say we want to have a custom class that acts like a list but has some additional methods. So we might subclass the list type and add custom functions inside:

class MyAwesomeList(list):
    # Add additional functions here

But now the type and isinstance return different results if we compare this new class to a list!

>>> my_list = MyAwesomeList()
>>> type(my_list) is list
False
>>> isinstance(my_list, list)
True

We get different results because isinstance checks if my_list is an instance of list (it's not) or a subclass of list (it is, because MyAwesomeList is a subclass of list). If you forget about this difference, it can lead to some subtle bugs in your code.

A better way to create a custom list-like class

If you really need to create a custom class that behaves like a list but has some additional features, check out the collections module. It contains classes like UserList, UserString, or UserDictionary. They are specifically designed to be subclassed when you want to create something that acts like a list, string, or a dictionary. If you try to subclass the list class, you might quickly fall into a rabbit hole of patching and reimplementing the existing methods just to make your subclass work as expected. Trey Hunner as a good article explaining this problem called "The problem with inheriting from dict and list in Python".

Conclusions

isinstance is usually the preferred way to compare types. It's not only faster but also considers inheritance, which is often the desired behavior. In Python, you usually want to check if a given object behaves like a string or a list, not necessarily if it's exactly a string. So instead of checking for string and all it's custom subclasses, you can just use isinstance.

On the other hand, when you want to explicitly check that a given variable is of a specific type (and not its subclass) - use type. And when you use it, use it like this: type(var) is some_type not like this: type(var) == some_type.

And before you start checking types of your variables everywhere throughout your code, check out why "Asking for Forgiveness" might be a better way.

Membership Testing

2020-10-08T00:00:00Z

Membership testing means checking if a collection of items (a list, a set, a dictionary, etc.) contains a specific item. For example, checking if a list of even numbers contains number 42. It's a quite common operation, so let's see how to do it properly.

About the "Writing Faster Python" series

Are those recommendations going to make your code much faster? Not really.
Is knowing those small differences going to make a slightly better Python programmer? Hopefully!

How can we check if a list contains a specific item? There is a terrible way of doing this - iterating through the list in a "for loop":

# membership.py

MILLION_NUMBERS = list(range(1_000_000))

def test_for_loop(number):
    for item in MILLION_NUMBERS:
        if item == number:
            return True
    return False

Here we compare every element of the list with the number we are looking for. If we have a match, we return True. If we get to the end of the list without finding anything, we return False. This algorithm is, to put it mildly, inefficient.

Membership testing operator

Python has a membership testing operator called in. We can simplify our check to one line:

def test_in(number):
    return number in MILLION_NUMBERS

It looks much cleaner and easier to read. But is it faster? Let's check.

We will run two sets of tests - one for a number at the beginning of the list and one for a number at the end:

# Look for the second element in the list
$ python -m timeit -s "from membership import test_for_loop" "test_for_loop(1)"
2000000 loops, best of 5: 180 nsec per loop

$ python -m timeit -s "from membership import test_in" "test_in(1)"
2000000 loops, best of 5: 117 nsec per loop


# Look for the last element in the list
$ python -m timeit -s "from membership import test_for_loop" "test_for_loop(999_999)"
10 loops, best of 5: 26.6 msec per loop

$ python -m timeit -s "from membership import test_in" "test_in(999_999)"
20 loops, best of 5: 13 msec per loop

If we search for the second element in the list, "for loop" is 54% slower (180/117≈1.538). If we search for the last element, it's 105% slower (26.6/13≈2.046).

What if we try to look for an item not included in the list?

$ python -m timeit -s "from membership import test_for_loop" "test_for_loop(-1)"
10 loops, best of 5: 25 msec per loop

$ python -m timeit -s "from membership import test_in" "test_in(-1)"
20 loops, best of 5: 11.4 msec per loop

The results are similar to what we got when the element was at the end of the list. In both cases, Python will check the whole list. Using a "for loop" is 119% slower (25/11.4≈2.193).

List vs. set

Using in is a great idea, but it's still slow because lookup time in a list has O(n) time complexity. The bigger the list, the longer it takes to check all the elements.

There is a better solution - we can use a data structure with a constant average lookup time, such as a set!

# membership.py
MILLION_NUMBERS = set(range(1_000_000))

def test_in_set(number):
    return number in MILLION_NUMBERS

$ python -m timeit -s "from membership import test_in_set" "test_in_set(1)"
2000000 loops, best of 5: 102 nsec per loop

$ python -m timeit -s "from membership import test_in_set" "test_in_set(999_999)"
2000000 loops, best of 5: 121 nsec per loop

$ python -m timeit -s "from membership import test_in_set" "test_in_set(-1)"
2000000 loops, best of 5: 107 nsec per loop

When the element we are looking for is at the beginning of the set, the performance is only slightly better. But if it's at the end of the set (or doesn't belong to the set at all) - the difference is enormous! Using in with a list instead of a set is over 100 000 times slower if the element doesn't exist (11.4ms / 107ns≈106542.056). That's a huge difference, so does it mean that we should always use a set? Not so fast!

Converting a list to a set is not "free"

Set is a perfect solution if we start with a set of numbers. But if we have a list, we first have to convert it to a set. And that takes time.

$ python -m timeit -s "MILLION_NUMBERS = list(range(1_000_000))" "set(MILLION_NUMBERS)"
10 loops, best of 5: 25.9 msec per loop

Converting our list to a set takes more time than a lookup in a list. Even if the element is at the end of the list, lookup takes around 13 msec, while a list-to-set conversion takes 25.9 msec - twice as slow.

If we want to check one element in a list, converting it to a set doesn't make sense. Also, don't forget that sets are unordered, so you may lose the initial ordering by converting a list to a set and back to a list. But if we want to check more than one element and we don't care about the order, this conversion overhead quickly pays off.

Quick lookup time is not the only special power of sets. You can also use them to remove duplicates.

Conclusions

To sum up:

Using a "for loop" to test membership is never a good idea.
Python has a membership testing operator in that you should use instead.
Membership testing in a set is much faster than membership testing in a list. But converting a list to a set also costs you some time!

Selecting an appropriate data structure can sometimes give you a significant speedup. If you want to learn more about the time complexity of various operations in different data structures, the wiki.python.org is a great resource. If you are not sure what the "get slice" or "extend" means in terms of code - here is the same list with code examples.

Checking for True or False

2020-10-01T00:00:00Z

How do you check if something is True in Python? There are three ways:

One "bad" way: if variable == True:
Another "bad" way: if variable is True:
And the good way, recommended even in the Programming Recommendations of PEP8: if variable:

The "bad" ways are not only frowned upon but also slower. Let's use a simple test:

$ python -m timeit -s "variable=False" "if variable == True: pass"
10000000 loops, best of 5: 24.9 nsec per loop

$ python -m timeit -s "variable=False" "if variable is True: pass"
10000000 loops, best of 5: 17.4 nsec per loop

$ python -m timeit -s "variable=False" "if variable: pass"
20000000 loops, best of 5: 10.9 nsec per loop

Using is is around 60% slower than if variable (17.4/10.9≈1.596), but using == is 120% slower (24.9/10.9≈2.284)! It doesn't matter if the variable is actually True or False - the differences in performance are similar (if the variable is True, all three scenarios will be slightly slower).

Similarly, we can check if a variable is not True using one of the following methods:

if variable != True: ("bad")
if variable is not True: ("bad")
if not variable: (good)

$ python -m timeit -s "variable=False" "if variable != True: pass"
10000000 loops, best of 5: 26 nsec per loop

$ python -m timeit -s "variable=False" "if variable is not True: pass"
10000000 loops, best of 5: 18.8 nsec per loop

$ python -m timeit -s "variable=False" "if not variable: pass"
20000000 loops, best of 5: 12.4 nsec per loop

if not variable wins. is not is 50% slower (18.8/12.4≈1.516) and != takes twice as long (26/12.4≈2.016).

The if variable and if not variable versions are faster to execute and faster to read. They are common idioms that you will often see in Python (or other programming languages).

About the "Writing Faster Python" series

Are those recommendations going to make your code much faster? Not really.
Is knowing those small differences going to make a slightly better Python programmer? Hopefully!

"truthy" and "falsy"

Why do I keep putting "bad" in quotes? That's because the "bad" way is not always bad (it's only wrong when you want to compare boolean values, as pointed in PEP8). Sometimes, you intentionally have to use one of those other comparisons.

In Python (and many other languages), there is True, and there are truthy values. That is, values interpreted as True if you run bool(variable). Similarly, there is False, and there are falsy values (values that return False from bool(variable)). An empty list ([]), string (""), dictionary ({}), None and 0 are all falsy but they are not strictly False.

Sometimes you need to distinguish between True/False and truthy/falsy values. If your code should behave in one way when you pass an empty list, and in another, when you pass False, you can't use if not value.

Take a look at the following scenario:

def process_orders(orders=None):
    if not orders:
        # There are no orders, return
        return
    else:
        # Process orders
        ...

We have a function to process some orders. If there are no orders, we want to return without doing anything. Otherwise, we want to process existing orders.

We assume that if there are no orders, then orders parameter is set to None. But, if the orders is an empty list, we also return without any action! And maybe it's possible to receive an empty list because someone is just updating the billing information of a past order? Or perhaps having an empty list means that there is a bug in the system. We should catch that bug before we fill up the database with empty orders! No matter what's the reason for an empty list, the above code will ignore it. We can fix it by investigating the orders parameter more carefully:

def process_orders(orders=None):
    if orders is None:
        # orders is None, return
        return
    elif orders == []:
        # Process empty list of orders
        ...
    elif len(orders) > 0:
        # Process existing orders
        ...

The same applies to truthy values. If your code should work differently for True than for, let's say, value 1, we can't use if variable. We should use == to compare the number (if variable == 1) and is to compare to True (if variable is True). Sounds confusing? Let's take a look at the difference between is and ==.

`is` checks the identity, `==` checks the value

The is operator compares the identity of objects. If two variables are identical, it means that they point to the same object (the same place in memory). They both have the same ID (that you can check with the id() function).

The == operator compares values. It checks if the value of one variable is equal to the value of some other variable.

Some objects in Python are unique, like None, True or False. Each time you assign a variable to True, it points to the same True object as other variables assigned to True. But each time you create a new list, Python creates a new object:

>>> a = True
>>> b = True
>>> a is b
True
# Variables that are identical are always also equal!
>>> a == b
True

# But
>>> a = [1,2,3]
>>> b = [1,2,3]
>>> a is b
False  # Those lists are two different objects
>>> a == b
True  # Both lists are equal (contain the same elements)

It's important to know the difference between is and ==. If you think that they work the same, you might end up with weird bugs in your code:

a = 1
# This will print 'yes'
if a is 1:
    print('yes')

b = 1000
# This won't!
if b is 1000:
    print('yes')

In the above example, the first block of code will print "yes," but the second won't. That's because Python performs some tiny optimizations and small integers share the same ID (they point to the same object). Each time you assign 1 to a new variable, it points to the same 1 object. But when you assign 1000 to a variable, it creates a new object. If we use b == 1000, then everything will work as expected.

Conclusions

To sum up:

To check if a variable is equal to True/False (and you don't have to distinguish between True/False and truthy / falsy values), use if variable or if not variable. It's the simplest and fastest way to do this.
If you want to check that a variable is explicitly True or False (and is not truthy/falsy), use is (if variable is True).
If you want to check if a variable is equal to 0 or if a list is empty, use if variable == 0 or if variable == [].

Sorting Lists

2020-09-24T00:00:00Z

There are at least two common ways to sort lists in Python:

With sorted function that returns a new list
With list.sort method that modifies list in place

Which one is faster? Let's find out!

sorted() vs list.sort()

I will start with a list of 1 000 000 randomly shuffled integers. Later on, I will also check if the order matters.

# sorting.py
from random import sample

# List of 1 000 000 integers randomly shuffled
MILLION_RANDOM_NUMBERS = sample(range(1_000_000), 1_000_000)


def test_sort():
    return MILLION_RANDOM_NUMBERS.sort()

def test_sorted():
    return sorted(MILLION_RANDOM_NUMBERS)

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
1 loop, best of 5: 6 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
1 loop, best of 5: 373 msec per loop

~~When benchmarked with Python 3.8, sort() is around 60 times as fast as sorted() when sorting 1 000 000 numbers (373/6≈62.167).~~

Update: As pointed out by a vigilant reader in the comments section, I've made a terrible blunder in my benchmarks! timeit runs the code multiple times, which means that:

The first time it runs, it sorts the random list in place.
The second and next time, it runs on the same list (that is now sorted)! And sorting an already sorted list is much faster, as I show you in the next paragraph.

We get completely wrong results because we compare calling list.sort() on an ordered list with calling sorted() on a random list.

Let's fix my test functions and rerun benchmarks.

# sorting.py
from random import sample

# List of 1 000 000 integers randomly shuffled
MILLION_RANDOM_NUMBERS = sample(range(1_000_000), 1_000_000)

def test_sort():
    random_list = MILLION_RANDOM_NUMBERS[:]
    return random_list.sort()

def test_sorted():
    random_list = MILLION_RANDOM_NUMBERS[:]
    return sorted(random_list)

This time, I’m explicitly making a copy of the initial shuffled list and then sort that copy (new_list = old_list[:] is a great little snippet to copy a list in Python). Copying a list adds a small overhead to our test functions, but as long as we call the same code in both functions, that’s acceptable.

Let's see the results:

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
1 loop, best of 5: 352 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
1 loop, best of 5: 385 msec per loop

Now, sorted is less than 10% slower (385/352≈1.094). Since we only run one loop, the exact numbers are not very reliable. I have rerun the same tests a couple more times, and the results were slightly different each time. sort took around 345-355 msec and sorted took around 379-394 msec (but it was always slower than sort). This difference comes mostly from the fact that sorted creates a new list (again, as kindly pointed out by a guest reader in the comments).

Initial order matters

What happens when our initial list is already sorted?

MILLION_NUMBERS = list(range(1_000_000))

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
20 loops, best of 5: 12.1 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
20 loops, best of 5: 16.6 msec per loop

Now, sorting takes much less time and the difference between sort and sorted grows to 37% (16.6/12.1≈1.372). Why is sorted 37% slower this time? Well, creating a new list takes the same amount of time as before. And since the time spent on sorting has shrunk, the impact of creating that new list got bigger.

If you want to run the benchmarks on your computer, make sure to adjust the test_sort and test_sorted functions, so they use the new MILLION_NUMBERS variable (instead of the MILLION_RANDOM_NUMBERS). Make sure you do this update for each of the following tests.

And if we try to sort a list of 1 000 000 numbers ordered in descending order:

DESCENDING_MILLION_NUMBERS = list(range(1_000_000, 0, -1))

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
20 loops, best of 5: 11.7 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
20 loops, best of 5: 18.1 msec per loop

The results are almost identical as before. The sorting algorithm is clever enough to optimize the sorting process for a descending list.

For our last test, let’s try to sort 1 000 000 numbers where 100 000 elements are shuffled, and the rest are ordered:

# 10% of numbers are random
MILLION_SLIGHTLY_RANDOM_NUMBERS = [*range(900_000), *sample(range(1_000_000), 100_000)]

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
5 loops, best of 5: 61.2 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
5 loops, best of 5: 71 msec per loop

Both functions get slower as the input list becomes more scrambled.

Using list.sort() is my preferred way of sorting lists - it saves some time (and memory) by not creating a new list. But that's a double-edged sword! Sometimes you might accidentally overwrite the initial list without realizing it (as I did with my initial benchmarks 😅). So, if you want to preserve the initial list's order, you have to use sorted instead. And sorted can be used with any iterable, while sort only works with lists. If you want to sort a set, then sorted is your only solution.

Conclusions

sort is slightly faster than sorted, because it doesn't create a new list. But you might still stick with sorted if:

You don't want to modify the original list. sort performs sorting in-place, so you can't use it here.
You need to sort something else than a list. sort is only defined on lists, so if you want to sort a set or any other collection of items, you have to use sorted instead.

If you want to learn more, the Sorting HOW TO guide from Python documentation contains a lot of useful information.

For Loop vs. List Comprehension

2020-09-17T00:00:00Z

Many simple "for loops" in Python can be replaced with list comprehensions. You can often hear that list comprehension is "more Pythonic" (almost as if there was a scale for comparing how Pythonic something is 😉). In this article, I will compare their performance and discuss when a list comprehension is a good idea, and when it's not.

Filter a list with a "for loop"

Let's use a simple scenario for a loop operation - we have a list of numbers, and we want to remove the odd ones. One important thing to keep in mind is that we can't remove items from a list as we iterate over it. Instead, we have to create a new one containing only the even numbers:

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def for_loop():
    output = []
    for element in MILLION_NUMBERS:
        if not element % 2:
            output.append(element)
    return output

if not element % 2 is equivalent to if element % 2 == 0, but it's slightly faster. I will write a separate article about comparing boolean values soon.

Let's measure the execution time of this function. I'm using Python 3.8 for benchmarks (you can read about the whole setup in the Introduction article):

$ python -m timeit -s "from filter_list import for_loop" "for_loop()"
5 loops, best of 5: 65.4 msec per loop

It takes 65 milliseconds to filter a list of one million elements. How fast will a list comprehension deal with the same task?

Filter a list with list comprehension

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def list_comprehension():
    return [number for number in MILLION_NUMBERS if not number % 2]

$ python -m timeit -s "from filter_list import list_comprehension" "list_comprehension()"
5 loops, best of 5: 44.5 msec per loop

"For loop" is around 50% slower than a list comprehension (65.4/44.5≈1.47). And we just reduced five lines of code to one line! Cleaner and faster code? Great!

Can we make it better?

Filter a list with the `filter` function

Python has a built-in filter function for filtering collections of elements. This sounds like a perfect use case for our problem, so let's see how fast it will be.

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def filter_function():
    return filter(lambda x: not x % 2, MILLION_NUMBERS)

$ python -m timeit -s "from filter_list import filter_function" "filter_function()"
1000000 loops, best of 5: 284 nsec per loop

284 nanoseconds?! That's suspiciously fast! It turns out that the filter function returns an iterator. It doesn't immediately go over one million elements, but it will return the next value when we ask for it. To get all the results at once, we can convert this iterator to a list.

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def filter_return_list():
    return list(filter(lambda x: not x % 2, MILLION_NUMBERS))

$ python -m timeit -s "from filter_list import filter_return_list" "filter_return_list()"
2 loops, best of 5: 104 msec per loop

Now, its performance is not so great anymore. It's 133% slower than the list comprehension (104/44.5≈2.337) and 60% slower than the "for loop" (104/65.4≈1.590).

While, in this case, it's not the best solution, an iterator is an excellent alternative to a list comprehension when we don't need to have all the results at once. If it turns out that we only need to get a few elements from the filtered list, an iterator will be a few orders of magnitude faster than other "non-lazy" solutions.

We could use the filterfalse() function from the itertools library to simplify the filtering condition. filterfalse returns the opposite elements than filter. It picks those elements that evaluate to False. Unfortunately, it doesn't make any difference when it comes to performance:

from itertools import filterfalse

def filterfalse_list():
    return list(filterfalse(lambda x: x % 2, MILLION_NUMBERS))

$ python -m timeit -s "from filter_list import filterfalse_list" "filterfalse_list()"
2 loops, best of 5: 103 msec per loop

Why is list comprehension faster than a for loop?

But why is the list comprehension faster than a for loop? When you use a for loop, on every iteration, you have to look up the variable holding the list and then call its append() function. This doesn't happen in a list comprehension. Instead, there is a special bytecode instruction LIST_APPEND that will append the current value to the list you're constructing.

More than one operation in the loop

List comprehensions are often faster and easier to read, but they have one significant limitation. What happens if you want to execute more than one simple instruction? List comprehension can't accept multiple statements (without sacrificing readability). But in many cases, you can wrap those multiple statements in a function.

Let's use a slightly modified version of the famous "Fizz Buzz" program as an example. We want to iterate over a list of elements and for each of them return:

"fizzbuzz" if the number can be divided by 3 and 5
"fizz" if the number can be divided by 3
"buzz" if the number can be divided by 5
the number itself, if it can't be divided by 3 or 5

Here is a simple solution:

# filter_list.py

def fizz_buzz():
    output = []
    for number in MILLION_NUMBERS:
        if number % 3 == 0 and number % 5 == 0:
            output.append('fizzbuzz')
        elif number % 3 == 0:
            output.append('fizz')
        elif number % 5 == 0:
            output.append('buzz')
        else:
            output.append(number)
    return output

Here is the list comprehension equivalent of the fizz_buzz():

['fizzbuzz' if x % 3 == 0 and x % 5 == 0 else 'fizz' if x % 3 == 0 else 'buzz' if x % 5 == 0 else x for x in MILLION_NUMBERS]

It's not easy to read - at least for me. It gets better if we split it into multiple lines:

[
    "fizzbuzz" if x % 3 == 0 and x % 5 == 0
    else "fizz" if x % 3 == 0
    else "buzz" if x % 5 == 0
    else x
    for x in MILLION_NUMBERS
]

But if I see a list comprehension that spans multiple lines, I try to refactor it. We can extract the "if" statements into a separate function:

# filter_list.py

def transform(number):
    if number % 3 == 0 and number % 5 == 0:
        return 'fizzbuzz'
    elif number % 3 == 0:
        return 'fizz'
    elif number % 5 == 0:
        return 'buzz'
    return number

def fizz_buzz2():
    output = []
    for number in MILLION_NUMBERS:
        output.append(transform(number))
    return output

Now it's trivial to turn it into a list comprehension. And we get the additional benefit of a nice separation of logic into a function that does the "fizz buzz" check and a function that actually iterates over a list of numbers and applies the "fizz buzz" transformation.

Here is the improved list comprehension:

def fizz_buzz2_comprehension():
    return [transform(number) for number in MILLION_NUMBERS]

Let's compare all three versions:

$ python -m timeit -s "from filter_list import fizz_buzz" "fizz_buzz()"
2 loops, best of 5: 191 msec per loop

$ python -m timeit -s "from filter_list import fizz_buzz2" "fizz_buzz2()"
1 loop, best of 5: 285 msec per loop

$ python -m timeit -s "from filter_list import fizz_buzz2_comprehension" "fizz_buzz2_comprehension()"
1 loop, best of 5: 224 msec per loop

Extracting a separate function adds some overhead. List comprehension with a separate transform() function is around 17% slower than the initial "for loop"-based version (224/191≈1.173). But it's much more readable, so I prefer it over the other solutions.

And, if you are curious, the one-line list comprehension mentioned before is the fastest solution:

def fizz_buzz_comprehension():
    return [
        "fizzbuzz" if x % 3 == 0 and x % 5 == 0
        else "fizz" if x % 3 == 0
        else "buzz" if x % 5 == 0
        else x
        for x in MILLION_NUMBERS
    ]

$ python -m timeit -s "from filter_list import fizz_buzz_comprehension" "fizz_buzz_comprehension()"
2 loops, best of 5: 147 msec per loop

Fastest, but also harder to read. If you run this code through a code formatter like black (which is a common practice in many projects), it will further obfuscate this function:

[
    "fizzbuzz"
    if x % 3 == 0 and x % 5 == 0
    else "fizz"
    if x % 3 == 0
    else "buzz"
    if x % 5 == 0
    else x
    for x in MILLION_NUMBERS
]

There is nothing wrong with black here - we are simply putting too much logic inside the list comprehension. If I had to say what the above code does, it would take me much longer to figure it out than if I had two separate functions. Saving a few hundred milliseconds of execution time and adding a few seconds of reading time doesn't sound like a good trade-off 😉.

Clever one-liners can impress some recruiters during code interviews. But in real life, separating logic into different functions makes it much easier to read and document your code. And, statistically, we read more code than we write.

Conclusions

List comprehensions are often not only more readable but also faster than using "for loops." They can simplify your code, but if you put too much logic inside, they will instead become harder to read and understand.

Even though list comprehensions are popular in Python, they have a specific use case: when you want to perform some operations on a list and return another list. And they have limitations - you can't break out of a list comprehension or put comments inside. In many cases, "for loops" will be your only choice.

I only scratched the surface of how useful list comprehension (or any other type of "comprehension" in Python) can be. If you want to learn more, Trey Hunner has many excellent articles and talks on this subject (for example, this one for beginners).

Ordered Dictionaries

2020-09-10T00:00:00Z

If you worked with Python 2 or an early version of Python 3, you probably remember that, in the past, dictionaries were not ordered. If you wanted to have a dictionary that preserved the insertion order, the go-to solution was to use OrderedDict from the collections module.

In Python 3.6, dictionaries were redesigned to improve their performance (their memory usage was decreased by around 20-25%). This change had an interesting side-effect - dictionaries became ordered (although this order was not officially guaranteed). "Not officially guaranteed" means that it was just an implementation detail that could be removed in the future Python releases.

But starting from Python 3.7, the insertion-order preservation has been guaranteed in the language specification. If you started your journey with Python 3.7 or a newer version, you probably don't know the world where you need a separate data structure to preserve the insertion order in a dictionary.

So if there is no need to use the OrderedDict, why is it still included in the collections module? Maybe it's more efficient? Let's find out!

OrderedDict vs dict

For my benchmarks, I will perform some typical dictionary operations:

Create a dictionary of 100 elements
Add a new item
Check if an item exists in a dictionary
Grab an existing and nonexistent item with the get method

To simplify the code, I wrap steps 2-4 in a function that accepts a dictionary (or OrderedDictionary) as an argument.

# dictionaries.py

from collections import OrderedDict

def perform_operations(dictionary):
    dictionary[200] = 'goodbye'
    is_50_included = 50 in dictionary
    item_20 = dictionary.get(20)
    nonexistent_item = dictionary.get('a')

def ordereddict():
    dictionary = OrderedDict.fromkeys(range(100), 'hello world')
    perform_operations(dictionary)

def standard_dict():
    dictionary = dict.fromkeys(range(100), 'hello world')
    perform_operations(dictionary)

Let's compare both functions. I run my benchmarks under Python 3.8 (check out my testing setup in the Introduction article):

$ python -m timeit -s "from dictionaries import ordereddict" "ordereddict()"
50000 loops, best of 5: 8.6 usec per loop

$ python -m timeit -s "from dictionaries import standard_dict" "standard_dict()"
50000 loops, best of 5: 4.7 usec per loop

OrderedDict is over 80% slower than the standard Python dictionary (8.6/4.7≈1.83).

What happens if the dictionary size grows to 10 000 elements?

# dictionaries2.py

from collections import OrderedDict

def perform_operations(dictionary):
    dictionary[20000] = 'goodbye'
    is_5000_included = 5000 in dictionary
    item_2000 = dictionary.get(2000)
    nonexistent_item = dictionary.get('a')

def ordereddict():
    dictionary = OrderedDict.fromkeys(range(10000), 'hello world')
    perform_operations(dictionary)

def standard_dict():
    dictionary = dict.fromkeys(range(10000), 'hello world')
    perform_operations(dictionary)

$ python -m timeit -s "from dictionaries import ordereddict" "ordereddict()"
200 loops, best of 5: 1.07 msec per loop

$ python -m timeit -s "from dictionaries import standard_dict" "standard_dict()"
500 loops, best of 5: 547 usec per loop

After increasing the dictionary size by 100x times, the difference between both functions stays the same. OrderedDict still takes almost twice as long to perform the same operations as a standard Python dictionary.

There is no point in testing even bigger dictionaries. If you need a really big dictionary, you should use more efficient data structures from the Numpy or Pandas libraries.

When to use OrderedDict?

If the OrderedDict is slower, why would you want to use it? I can think of at least two reasons:

You are still using a Python version that doesn't guarantee the order in dictionaries (pre 3.6). In this case, you don't have a choice.
You want to use additional features that OrderedDict offers. For example, it can be reversed. If you try to run reversed() function on a standard dictionary, you will get an error, but OrderedDict will nicely return a reversed version of itself.
You actually care about the ordering when comparing dictionaries. As pointed out by Ned Batchelder in his "Ordered dict surprises" article, when you compare two dictionaries with the same items, but in a different order, Python reports them as equal. But if you compare two OrderedDict objects with the same items in a different order, they are not equal. See this example:
```
>>> d1 = {'a':1, 'b':2}
>>> d2 = {'b':2, 'a':1}
>>> d1 == d2
True

>>> ord_d1 = OrderedDict(a=1, b=2)
>>> ord_d2 = OrderedDict(b=2, a=1)
>>> ord_d1 == ord_d2
False
```

How to stay up to date on Python changes?

If you are using one of the latest versions of Python, dictionaries are ordered by default. But it's easy to miss changes like this, especially if you upgrade Python version by a few releases at once, and you don't read the release notes carefully. I usually read some blog posts when there is a new version of Python coming out (there are plenty of blog posts around that time), so I catch the essential updates.

The best source of information is the official documentation. Unlike a lot of documentation that I have seen in my life, the "What's New in Python 3" page is written in a very approachable language. It's easy to read and grasp the most significant changes. If you haven't done it yet, go check it out. I reread it a few days ago, and I was surprised how many features I forgot about!

Easy Speedup Wins With Numba

2020-09-03T00:00:00Z

If you have functions that do a lot of mathematical operations, use NumPy or rely heavily on loops, then there is a way to speed them up significantly with one line of code. Ok, two lines if you count the import.

Numba and the @jit decorator

Meet Numba and its @jit decorator. It changes how your code is compiled, often improving its performance. You don't have to install any special tools (just the numba pip package), you don't have to tweak any parameters. All you have to do is:

Add the @jit decorator to a function
Check if it's faster

Let's see an example of code before and after applying Numba's optimization.

# numba_testing.py

import math

def compute():
    # Bunch of dummy math operations
    result = 0
    for number in range(1_000_000):
        double = number * 2
        result += math.sqrt(double) + double
    return result

The only purpose of this code is to do some calculations and to "be slow." Let's see how slow (benchmarks are done with Python 3.8 - I describe the whole setup in the Introduction article):

$ python -m timeit -s "from numba_testing import compute" "compute()"
1 loop, best of 5: 217 msec per loop

Now, we add @jit to our code. The body of the function stays the same, and the only difference is the decorator. Don't forget to install Numba package with pip (pip install numba).

# numba_testing.py

import math

from numba import jit

@jit
def compute_jit():
    # Bunch of dummy math operations
    result = 0
    for number in range(1_000_000):
        double = number * 2
        result += math.sqrt(double) + double
    return result

Let's measure the execution time once more:

$ python -m timeit -s "from numba_testing import compute_jit" "compute_jit()"
200 loops, best of 5: 1.76 msec per loop

Using @jit decorator gave us a 120x speedup (217 / 1.76 = 123.295)! That's a huge improvement for such a simple change!

How did I discover Numba?

I first learned about Numba when I was doing code challenges from the Advent of Code a few years ago. I wrote a pretty terrible algorithm, left it running, and went for lunch. When I came back after one hour, my program wasn't even 10% done. I stopped it, added the >@jit decorator to the main function, rerun it, and I had the results in under one minute! Fantastic improvement with almost no work!

This story doesn't mean that it's ok to write sloppy code, and then use hacks to speed it up. But sometimes you just need to make some one-off calculations. You don't want to spend too much time writing the perfect algorithm. Or maybe you can't think of a better algorithm, and the one you have is too slow. Using tools like Numba can be one of the fastest and easiest to apply improvements!

Other features of Numba

@jit is the most common decorator from the Numba library, but there are others that you can use:

@njit - alias for @jit(nopython=True). In nopython mode, Numba tries to run your code without using the Python interpreter at all. It can lead to even bigger speed improvements, but it's also possible that the compilation will fail in this mode.
@vectorize and @guvectorize - produces ufunc and generalized ufunc used in NumPy.
@jitclass - can be used to decorate the whole class.
@cfunc - declares a function to be used as a native callback (from C or C++ code).

There are also advanced features that let you, for example, run your code on GPU with @cuda.jit. This doesn't work out of the box, but it might be worth the effort for some very computational-heavy operations.

Numba has plenty of configuration options that will further improve your code's execution time if you know what you are doing. You can:

Disable GIL (Global Interpreter Lock) with nogil
Cache results with cache
Automatically parallelize functions with parallel.

Check out the documentation to see what you can do. And to see more real-life examples (like computing the Black-Scholes model or the Lennard-Jones potential), visit the Numba Examples page.

Conclusions

Numba is a great library that can significantly speed up your programs with minimal effort. Given that it takes less than a minute to install and decorate some slow functions, it's one of the first solutions that you can check when you want to quickly improve your code (without rewriting it).

It works best if your code:

Uses NumPy a lot
Performs plenty of mathematical operations
Performs operations is a loop

Find Item in a List

2020-08-27T00:00:00Z

Find a number

If you want to find the first number that matches some criteria, what do you do? The easiest way is to write a loop that checks numbers one by one and returns when it finds the correct one.

Let's say we want to get the first number divided by 42 and 43 (that's 1806). If we don't have a predefined set of elements (in this case, we want to check all the numbers starting from 1), we might use a "while loop".

# find_item.py

def while_loop():
    item = 1
    # You don't need to use parentheses, but they improve readability
    while True:
        if (item % 42 == 0) and (item % 43 == 0):
            return item
        item += 1

It's pretty straightforward:

Start from number 1
Check if that number can be divided by 42 and 43.
- If yes, return it (this stops the loop)
Otherwise, check the next number

Least Common Multiple

The examples in this article are intentionally iterating over a list so I can compare the speed of different code constructs. But if you really want to find the least common multiple of two numbers (that is, the smallest number that can be divided by both of them), you're better off:

using the math.lcm() function directly: math.lcm(42, 43) (Python 3.9 and above)
dividing their product by their greatest common divisor: 42 * 43 // math.gcd(42, 43) (Python 3.5 and above)

Both versions will be an order of magnitude faster than my silly examples. Thanks to Dmitry for pointing this out!

Find a number in a list

If we have a list of items that we want to check, we will use a "for loop" instead. I know that the number I'm looking for is smaller than 10 000, so let's use that as the upper limit:

# find_item.py

def for_loop():
    for item in range(1, 10000):
        if (item % 42 == 0) and (item % 43 == 0):
            return item

Let's compare both solutions (benchmarks are done with Python 3.8 - I describe the whole setup in the Introduction article):

$ python -m timeit -s "from find_item import while_loop" "while_loop()"
2000 loops, best of 5: 134 usec per loop

$ python -m timeit -s "from find_item import for_loop" "for_loop()"
2000 loops, best of 5: 103 usec per loop

"While loop" is around 30% slower than the "for loop" (134/103≈1.301).

Loops are optimized to iterate over a collection of elements. Trying to manually do the iteration (for example, by referencing elements in a list through an index variable) will be a slower and often over-engineered solution.

Python 2 flashbacks

In Python 3, the range() function is lazy. It won't initialize an array of 10 000 elements, but it will generate them as needed. It doesn't matter if we say range(1, 10000) or range(1, 1000000) - there will be no difference in speed. But it was not the case in Python 2!

In Python 2, functions like range, filter, or zip were eager, so they would always create the whole collection when initialized. All those elements would be loaded to the memory, increasing the execution time of your code and its memory usage. To avoid this behavior, you had to use their lazy equivalents like xrange, ifilter, or izip.

Out of curiosity, let's see how slow is the for_loop() function if we run it with Python 2.7.18 (the latest and last version of Python 2):

$ pyenv shell 2.7.18
$ python -m timeit -s "from find_item import for_loop" "for_loop()"
10000 loops, best of 3: 151 usec per loop

That's almost 50% slower than running the same function in Python 3 (151/103≈1.4660). Updating Python version is one of the easiest performance wins you can get!

If you are wondering what's pyenv and how to use it to quickly switch Python versions, check out this section of my PyCon 2020 workshop on Python tools.

Let's go back to our "while loop" vs. "for loop" comparison. Does it matter if the element we are looking for is at the beginning or at the end of the list?

def while_loop2():
    item = 1
    while True:
        if (item % 98 == 0) and (item % 99 == 0):
            return item
        item += 1

def for_loop2():
    for item in range(1, 10000):
        if (item % 98 == 0) and (item % 99 == 0):
            return item

This time, we are looking for number 9702, which is at the very end of our list. Let's measure the performance:

$ python -m timeit -s "from find_item import while_loop2" "while_loop2()"
500 loops, best of 5: 710 usec per loop

$ python -m timeit -s "from find_item import for_loop2" "for_loop2()"
500 loops, best of 5: 578 usec per loop

There is almost no difference. "While loop" is around 22% slower this time (710/578≈1.223). I performed a few more tests (up to a number close to 100 000 000), and the difference was always similar (in the range of 20-30% slower).

Find a number in an infinite list

So far, the collection of items we wanted to iterate over was limited to the first 10 000 numbers. But what if we don't know the upper limit? In this case, we can use the count function from the itertools library.

from itertools import count

def count_numbers():
    for item in count(1):
        if (item % 42 == 0) and (item % 43 == 0):
            return item

count(start=0, step=1) will start counting numbers from the start parameter, adding the step in each iteration. In my case, I need to change the start parameter to 1, so it works the same as the previous examples.

count works almost the same as the "while loop" that we made at the beginning. How about the speed?

$ python -m timeit -s "from find_item import count_numbers" "count_numbers()"
2000 loops, best of 5: 109 usec per loop

It's almost the same as the "for loop" version. So count is a good replacement if you need an infinite counter.

What about a list comprehension?

A typical solution for iterating over a list of items is to use a list comprehension. But we want to exit the iteration as soon as we find our number, and that's not easy to do with a list comprehension. It's a great tool to go over the whole collection, but not in this case.

Let's see how bad it is:

def list_comprehension():
    return [item for item in range(1, 10000) if (item % 42 == 0) and (item % 43 == 0)][0]

$ python -m timeit -s "from find_item import list_comprehension" "list_comprehension()"
500 loops, best of 5: 625 usec per loop

That's really bad - it's a few times slower than other solutions! It takes the same amount of time, no matter if we search for the first or last element. And we can't use count here.

But using a list comprehension points us in the right direction - we need something that returns the first element it finds and then stops iterating. And that thing is a generator! We can use a generator expression to grab the first element matching our criteria.

Find item with a generator expression

def generator():
    return next(item for item in count(1) if (item % 42 == 0) and (item % 43 == 0))

The whole code looks very similar to a list comprehension, but we can actually use count. Generator expression will execute only enough code to return the next element. Each time you call next(), it will resume work in the same place where it stopped the last time, grab the next item, return it, and stop again.

$ python -m timeit -s "from find_item import generator" "generator()"
2000 loops, best of 5: 110 usec per loop

It takes almost the same amount of time as the best solution we have found so far. And I find this syntax much easier to read - as long as we don't put too many ifs there!

Generators have the additional benefit of being able to "suspend" and "resume" counting. We can call next() multiple times, and each time we get the next element matching our criteria. If we want to get the first three numbers that can be divided by 42 and 43 - here is how easily we can do this with a generator expression:

def generator_3_items():
    gen = (item for item in count(1) if (item % 42 == 0) and (item % 43 == 0))
    return [next(gen), next(gen), next(gen)]

Compare it with the "for loop" version:

def for_loop_3_items():
    items = []
    for item in count(1):
        if (item % 42 == 0) and (item % 43 == 0):
            items.append(item)
            if len(items) == 3:
                return items

Let's benchmark both versions:

$ python -m timeit -s "from find_item import for_loop_3_items" "for_loop_3_items()"
1000 loops, best of 5: 342 usec per loop

$ python -m timeit -s "from find_item import generator_3_items" "generator_3_items()"
1000 loops, best of 5: 349 usec per loop

Performance-wise, both functions are almost identical. So when would you use one over the other? "For loop" lets you write more complex code. You can't put nested "if" statements or multiline code with side effects inside a generator expression. But if you only do simple filtering, generators can be much easier to read.

Be careful with nested ifs

Nesting too many "if" statements makes code difficult to follow and reason about. And it's easy to make mistakes.

In the last example, if we don't nest the second if, it will be checked in each iteration. But we only need to check it when we modify the items list. It might be tempting to write the following code:

def for_loop_flat():
    items = []
    for item in count(1):
        if (item % 42 == 0) and (item % 43 == 0):
            items.append(item)
        if len(items) == 3:
            return items

This version is easier to follow, but it's also much slower!

$ python -m timeit -s "from find_item import for_loop_3_items" "for_loop_3_items()"
1000 loops, best of 5: 323 usec per loop

$ python -m timeit -s "from find_item import for_loop_flat" "for_loop_flat()"
500 loops, best of 5: 613 usec per loop

If you forget to nest ifs, your code will be 90% slower (613/323≈1.898).

Conclusions

Generator expression combined with next() is a great way to grab one or more elements based on specific criteria. It's memory-efficient, fast, and easy to read - as long as you keep it simple. When the number of "if statements" in the generator expression grows, it becomes much harder to read (and write).

With complex filtering criteria or many ifs, "for loop" is a more suitable choice that doesn't sacrifice the performance.

Ask for Forgiveness or Look Before You Leap?

2020-08-19T00:00:00Z

"Ask for forgiveness" and "look before you leap" (sometimes also called "ask for permission") are two opposite approaches to writing code. If you "look before you leap", you first check if everything is set correctly, then you perform an action. For example, you want to read text from a file. What could go wrong with that? Well, the file might not be in the location where you expect it to be. So, you first check if the file exists:

import os
if os.path.exists("path/to/file.txt"):
    ...

# Or from Python 3.4
from pathlib import Path
if Path("/path/to/file").exists():
    ...

Even if the file exists, maybe you don't have permission to open it? So let's check if you can read it:

import os
if os.access("path/to/file.txt", os.R_OK):
    ...

But what if the file is corrupted? Or if you don't have enough memory to read it? This list could go on. Finally, when you think that you checked every possible corner-case, you can open and read it:

with open("path/to/file.txt") as input_file:
    return input_file.read()

Depending on what you want to do, there might be quite a lot of checks to perform. And even when you think you covered everything, there is no guarantee that some unexpected problems won't prevent you from reading this file. You might have some race conditions if the file is deleted or permissions are changed between one "if" check and the other. So, instead of doing all the checks, you can "ask for forgiveness."

With "ask for forgiveness," you don't check anything. You perform whatever action you want, but you wrap it in a try/catch block. If an exception happens, you handle it. You don't have to think about all the things that can go wrong, your code is much simpler (no more nested ifs), and you will usually catch more errors that way. That's why the Python community, in general, prefers this approach, often called "EAFP" - "Easier to ask for forgiveness than permission."

Here is a simple example of reading a file with the "ask for forgiveness" approach:

try:
    with open("path/to/file.txt", "r") as input_file:
        return input_file.read()
except IOError:
    # Handle the error or just ignore it

Here we are catching the IOError. If you are not sure what kind of exception can be raised, you could catch all of them with the BaseException class, but in general, it's a bad practice. It will catch every possible exception (including, for example, KeyboardInterrupt when you want to stop the process), so try to be more specific.

"Ask for forgiveness" is cleaner. But which one is faster?

"Ask For Forgiveness" vs "Look Before You Leap" - speed

Time for a simple test. Let's say that I have a class, and I want to read an attribute from this class. But I'm using inheritance, so I'm not sure if the attribute is defined or not. I need to protect myself, by either checking if it exists ("look before you leap") or catching the AttributeError ("ask for forgiveness"):

# permission_vs_forgiveness.py

class BaseClass:
    hello = "world"

class Foo(BaseClass):
    pass

FOO = Foo()

# Look before you leap
def test_permission():
    if hasattr(FOO, "hello"):
        FOO.hello

# Ask for forgiveness
def test_forgiveness():
    try:
        FOO.hello
    except AttributeError:
        pass

Let's measure the speed of both functions.

For benchmarking, I'm using the standard timeit module and Python 3.8. I describe my setup and some assumptions in the Introduction to the Writing Faster Python.

$ python -m timeit -s "from permission_vs_forgiveness import test_permission" "test_permission()"
2000000 loops, best of 5: 155 nsec per loop

$ python -m timeit -s "from permission_vs_forgiveness import test_forgiveness" "test_forgiveness()"
2000000 loops, best of 5: 118 nsec per loop

"Look before you leap" is around 30% slower (155/118≈1.314).

What happens if we increase the number of checks? Let's say that this time we want to check for three attributes, not just one:

# permission_vs_forgiveness.py

class BaseClass:
    hello = "world"
    bar = "world"
    baz = "world"

class Foo(BaseClass):
    pass

FOO = Foo()

# Look before you leap
def test_permission2():
    if hasattr(FOO, "hello") and hasattr(FOO, "bar") and hasattr(FOO, "baz"):
        FOO.hello
        FOO.bar
        FOO.baz

# Ask for forgiveness
def test_forgiveness2():
    try:
        FOO.hello
        FOO.bar
        FOO.baz
    except AttributeError:
        pass

$ python -m timeit -s "from permission_vs_forgiveness import test_permission2" "test_permission2()"
500000 loops, best of 5: 326 nsec per loop

$ python -m timeit -s "from permission_vs_forgiveness import test_forgiveness2" "test_forgiveness2()"
2000000 loops, best of 5: 176 nsec per loop

"Look before you leap" is now around 85% slower (326/176≈1.852). So the "ask for forgiveness" is not only much easier to read and robust but, in many cases, also faster. Yes, you read it right, "in many cases," not "in every case!"

The main difference between "EAFP" and "LBYL"

What happens if the attribute is actually not defined? Take a look at this example:

# permission_vs_forgiveness.py

class BaseClass:
    pass  # "hello" attribute is now removed

class Foo(BaseClass):
    pass

FOO = Foo()

# Look before you leap
def test_permission3():
    if hasattr(FOO, "hello"):
        FOO.hello

# Ask for forgiveness
def test_forgiveness3():
    try:
        FOO.hello
    except AttributeError:
        pass

$ python -m timeit -s "from permission_vs_forgiveness import test_permission3" "test_permission3()"
2000000 loops, best of 5: 135 nsec per loop

$ python -m timeit -s "from permission_vs_forgiveness import test_forgiveness3" "test_forgiveness3()"
500000 loops, best of 5: 562 nsec per loop

The tables have turned. "Ask for forgiveness" is now over four times as slow as "Look before you leap" (562/135≈4.163). That's because this time, our code throws an exception. And handling exceptions is expensive.

If you expect your code to fail often, then "Look before you leap" might be much faster.

Verdict

"Ask for forgiveness" results in much cleaner code, makes it easier to catch errors, and in most cases, it's much faster. No wonder that EAFP ("Easier to ask for forgiveness than permission") is such a ubiquitous pattern in Python. Even in the example from the beginning of this article (checking if a file exists with os.path.exists) - if you look at the source code of the exists method, you will see that it's simply using a try/except. "Look before you leap" often results in a longer code that is less readable (with nested if statements) and slower. And following this pattern, you will probably sometimes miss a corner-case or two.

Just keep in mind that handling exceptions is slow. Ask yourself: "Is it more common that this code will throw an exception or not?" If the answer is "yes," and you can fix those problems with a well-placed "if," that's great! But in many cases, you won't be able to predict what problems you will encounter. And using "ask for forgiveness" is perfectly fine - your code should be "correct" before you start making it faster.

Writing Faster Python - Introduction

2020-08-18T00:00:00Z

2022 Update: I started writing these articles in 2020 using Python 3.8 on a 2017 MacBook Pro with Intel CPU. In 2022, I switched to a new MacBook Pro with M1 CPU and decided to also switch to the latest Python 3.11 version as it offers some nice speed-up improvements.

So all the articles written after 2021 use a much faster CPython version and newer laptop than the initial ones.

Writing Faster Python

A few years ago, I made a presentation called "Writing Faster Python," which got quite popular (as for a technical talk). But I made it for Python 2, and even though most advice applies to Python 3, I need to update it at some point. And I will, but first, I need some examples that I can use.

So, today I'm starting a series of articles where I take some common Python code structures and show how they can be improved. In many cases, simply writing idiomatic code and avoiding anti-patterns will result in better and faster code, and that's what I want to focus on. I will also show how you can significantly speed up your programs by using a different interpreter (like PyPy), just-in-time compilers like Numba and other tools. Some code examples are mere curiosities with a marginal impact on the execution time (like replacing dict() with {}), but I want to show you how they work and when I would use one over the other. Finally, there will be cases when the "improved" code is faster but less readable, and I wouldn't use it in my programs - I will clearly warn you when this happens.

This article will be updated with new information as I continue writing the "Writing Faster Python" series. I will answer some common questions, clarify my assumptions (they might change if something doesn't work well), and link to additional resources.

I will try to publish a new article every week or two. Given that so far, I was posting very irregularly, that's a bold statement, and I might need to revalidate it pretty soon 😉.

You can find all the articles published so far in this series here.

The best way to get notifications about new articles is to subscribe to my newsletter (below), follow me on Twitter, or, if you are old fashioned like me, use the RSS (click the icon in the footer of this page).

Assumptions

Here are some assumptions about the code examples, benchmarks, and the overall setup:

I will benchmark the code using the timeit module from the standard library. If the code spans multiple lines, I will wrap it in a separate function. That way, I can import it in the "setup" statement and then benchmark everything easily (without semicolons or weird line breaks). Here is how the benchmarks will look like:
```
$ python -m timeit -s "from my_module import version1" "version1()"
2000000 loops, best of 5: 100 nsec per loop

$ python -m timeit -s "from my_module import version2" "version2()"
2000000 loops, best of 5: 200 nsec per loop
```
The -s parameter specifies the "setup statement" (it's executed once and it's not benchmarked) and the final argument is the actual code to benchmark. timeit module will automatically determine how many times it should run the code to give reliable results.
I will often initialize some setup variables at the beginning of the file and use them in my test functions. Those variables shared between different functions will be written in uppercase letters, for example:
```
MILLION_NUMBERS = range(1_000_000)

def test_version1():
    for number in MILLION_NUMBERS:
        crunch_numbers(number)
```
That's right - I'm using the dreaded global variables. Normally, I would pass those "global variables" as parameters to my functions, but I don't want to do this for two reasons:
- It makes my simple examples harder to follow (now I have to pass arguments around)
- I only wrap code inside functions to split the "setup statement" from the "actual code," so it's easier to benchmark only the relevant code. Usually, in my code "MILLION_NUMBERS" would be in the same scope as the for loop:
```
MILLION_NUMBERS = range(1_000_000)
for number in MILLION_NUMBERS:
    crunch_numbers(number)
```
If you are still not convinced, feel free to pass global variables as parameters in your head while reading the code examples 😉. That won't affect the benchmarks.
I will use one of the latest versions of Python. I start with Python 3.8 and upgrade when the new stable version is released (so no beta or release candidates). Just by updating the Python version, both the "slow" and "fast" code will often run faster. But there is no way that a code example that was "slow" in one Python version will suddenly be "fast" in another.
To ensure that the benchmarks were affected by some process "cutting in," I run them a few times interchangeably ("slow" function, "fast" function, "slow" function, "fast" function, etc.). If they return similar results, I assume that my benchmarks are fine.
I will generally avoid code constructs that improve the speed but sacrifice the readability (so no "replace your Python code with C" advice 😜). Inlining code instead of using functions usually makes it faster, but it turns your programs into blobs of incomprehensible code. And, in most cases, readability of your code is much more important than its speed! I might mention some interesting tips that can be used in specific situations, but I will say explicitly if that's a code that I would use or not.

Code conventions

Code that starts with >>> symbols is executed in an interactive Python shell (REPL). Next line contains the the output of a given command:

>>> 1 + 1
2
>>> print('hello')
hello

Code that starts with $ is executed in shell and results are printed in the next line (or lines):

$ python -m timeit -s "variable = 'hello'" "isinstance(variable, str)"
5000000 loops, best of 5: 72.8 nsec per loop

Code that doesn’t start with any of those is just a standard Python code. Usually, at the top of the file, I put a comment specifying its filename (it will be used when I import modules during the benchmarking):

# my_file.py
def hello():
    return "Hello world!"

You can find most of the code examples in my blog-resources/writing-faster-python repository.

Frequently Asked Questions

"What's the point of these small improvements? Those changes don't matter!"

That’s a very good point. If we take all the code improvements together and apply it to a random Python project, the speed improvement will probably be a fraction of a speed boost that we would get by simply using a much faster computer. Does in mean we can write sloppy code and get away with it? Probably, but if you are reading those words, the chances are that you care about the code that you write. And, like me, you want to learn how to write better code - faster, cleaner, and simpler. So let me show you some ways how our code can be improved without sacrificing its readability.

Every time I'm coding, I keep thinking: "how can I make it better?". I have to stop comparing different code patterns because I could easily waste a few hours every day doing just that. Luckily, at some point, you get a feeling of what will work better. In general, more "Pythonic" solutions will often be faster, so if you come to Python from a different programming language, you might need to adjust the way you write or think about the code.

The whole point of these articles is to learn something new. So if you know any cool tricks to improve Python code, I would love to take them for a spin and share with others! Just leave a comment, drop me an email, or message me on Twitter.

"If function A is 25% faster, then function B is 25% slower, right?"

One of the hardest things in this series is to figure out what’s the least confusing way of saying how much something is faster/slower than something else. It’s easy to get confused about the difference between "faster than" and "as fast as." Does "1.0x faster" actually means "twice as fast" or "identical as"? How do you calculate the percentage for the time difference? Do you compare the difference between two values to the baseline like here, or do you divide one value by the other like here? Can something actually be 200% faster than something else? And can we even say that "something is x times slower than something else" (not really, because "one time less equals zero")?

After going through a bunch of StackOverflow, MathOverflow (1, 2), EnglishOverflow (1) and even some reddit or Hacker News questions, I was just more confused. But luckily, we have Wikipedia explaining how we do percentage increase/decrease and how we calculate speedup in execution times.

As you can see, calculating how many % something is faster is the most confusing. If the initial value is 100%, then the "faster" function can only be up to 100% faster because "faster" means a decrease in time, and we can’t decrease time by more than the initial 100%.

On the other hand, something can be slower by 10%, 100% or 1000% and we can calculate that easily. Take a look at this example. If a "slow" function takes 10 seconds and "fast" function takes 2 seconds, we can say that:

"slow" function is 5 times as slow as "fast" function: 10s / 2s = 5
"slow" function is 4 times slower than the "fast" function: (10s - 2s) / 2s = 4
"slow function is 500% as slow as the "fast" function: 10s/2s * 100%
"slow function is 400% slower than the "fast" function: (10s-2s) / 2s * 100 (alternatively, we can use "10s/2s * 100% - initial 100%" formula)

If I want to say that something is faster, I will avoid using a percentage value and use the speedup instead. The speedup can be defined as "improvement in speed of execution of a task." For example, if a "slow function" takes 2.25s and "fast function" takes 1.50s, we can say that the "fast function" has a 1.5x speedup (2.25 / 1.50 = 1.5).

Conventions that you can expect

If function A takes 10s and function B takes 15s, I will usually say that "function B is 50% slower".
If function A takes 10s and function B takes 30s, I will usually say that "function B is 3 times as slow as A" or that "function B has 3x speedup over the function A".

I hope this makes my calculations clear. In the end, even if I use some incorrect wording or if you think that percentage/speedup should be calculated differently, I provide the raw numbers of each comparison, so everyone can make their own calculations as they like.

"This one function can be improved even more!"

Great, please tell me how! Almost every code can be improved, and there is a huge chance that you might know something that I didn’t think of. I’m always happy to hear how I can improve my code.

Additional resources

Inspiration for the articles comes from my daily work and various parts of the internet, like the StackOverflow questions, PEPs (Python Enhancement Proposals), etc.

If you are looking for more articles about Python best practices, check out the following resources:

The Little Book of Python Anti-Patterns - a free little online book with common Python anti-patterns and how to fix them. It was last updated in 2018, and some tips are specific to Python 2, but I still recommend it to any new Python programmer.
This list will be updated in the future.

18 Plugins for Writing Python in VS Code

2020-04-27T00:00:00Z

VS Code is a great text editor. But when you install it, its functionality is limited. You can edit JavaScript and TypeScript, but for other programming languages, it will be just a text editor. You will need to add some plugins to turn it into a proper IDE.

Luckily, when you open a file in a new language, VS Code will suggest an extension that can help you. With the Python extension, you can already do a lot - you get syntax highlighting, code completion, and many other features that turn a text editor into a code editor.

But there are many other plugins that I discovered when working with Python. Some add entirely new functionality, and others offer just a small improvement here and there. I've decided to write them down. I hope some of you will find them useful!

Python and other language-specific plugins

First and foremost - the Python plugin for VS Code. Out of the box, there is no support for Python in VS Code, but when you open a Python file, VS Code will immediately suggest this plugin. It adds all the necessary features:

Syntax highlighting for Python files
Intellisense (code-completion suggestions)
Ability to start a debugger
Support for collecting and running tests (with different testing frameworks like pytest or unittest)
Different linters
And plenty of other small features that turn VS Code into a proper Python editor

And it's the same with different languages. Each time you open a file that VS Code doesn't support, you get a suggestion of a plugin for that language. It's a great approach! On the one hand, you don't have to figure out which extensions you need to install, but on the other hand, you don't slow down your IDE with plugins that you will never use.

Django and other framework-specific plugins

If you are working with frameworks, there is usually a plugin that will make your life easier, like Django or flask-snippets. They bring some additional improvements for a given framework like:

Better syntax highlighting for framework-specific files (e.g., template files in Django that combine HTML with Django tags)
Additional snippets - especially useful for the templating systems. Being able to insert loops and if-s with a two letter shortcut without opening and closing all those {% tags is a blessing!
Improved support for different functions. For example, Django plugin adds the ability to "Go to definition" from the templates.

IntelliCode

Intellicode makes the autocompletion a bit smarter. It tries to predict which term you are most likely to use in a given situation and puts that term at the top of the list (marked with a ☆ symbol).

It works surprisingly well!

Emmet

Technically, Emmet is not an extension because it's already integrated with VS Code by default (due to its huge popularity). But it still deserves mention, in case there is someone who never heard about it.

Emmet is going to be your best friend if you are writing a lot of HTML and CSS. It lets you expand simple abbreviations into full HTML, it adds CSS prefixes (together with vendor prefixes), and a whole bunch of other useful functions (rename a tag, balance in/out, go to matching pair, etc.)

I absolutely love it when I need to write HTML. I started using it to quickly add a class to a tag (div.header or a.btn.btn-primary) and then I learned new features. With Emmet you can write:

ul>li.list-item*3

and if you press Enter, it will turn into:

<ul>
  <li class="list-item"></li>
  <li class="list-item"></li>
  <li class="list-item"></li>
</ul>

Autodocstring

This plugin speeds up writing Python documentation by generating some of the boilerplate for you.

Write a function signature, type """ to start the docstring, press Enter, and this plugin does the rest. It will take care of copying the arguments from the function signature to the docs. And if you add types to your arguments, it will recognize them and put them in the correct place in the documentation.

Bookmarks

This extension lets you bookmark locations in your code, easily list all your bookmarks in a sidebar, and move between them with keyboard shortcuts.

It's incredibly useful then I'm digging into a new codebase (so I can jump around and not get lost). I also find it helpful when I'm trying to debug some complicated issues - VS Code has a functionality to "Go to Previous/Next location", but without bookmarks, it's easy to get lost.

Dash

With Dash extension, you can access offline documentation for basically any programming language or framework.

It requires installing one of the additional tool to provide the documentation:

Once you download the documentation, you can access it offline.

I'm not using it very often, but it's a great tool if you need to work without access to the internet.

Error Lens

Sometimes the errors marks in VS Code are hard to spot (especially the "info" hints). If you don't wrap lines, it's even worse - the error can be in the part of the code not visible on the screen.

That's why I'm using Error Lens. It lets me modify how the errors should be displayed. It can display the error message next to the line where it occurs and a Sublime-like error icons in the gutter (next to the line number).

File Utils

This small plugin adds a few file-related commands to the Command Palette (normally you can perform them by right-clicking in the sidebar):

Rename
Move
Duplicate
Copy path or name of the file

It also adds a "Move/Duplicate File" option to the context menu.

GitLens

Massive plugin - adds a lot of git integration to VS Code:

Can show blame annotations per line, per file, in the status bar, or on hover.
Provides you with context links to show changes, show diff, copy commit ID.
Brings a sidebar with probably every possible information about the git repository, file and line history, compare and search menus, etc.

It's much more powerful than the default "source control" panel of VS Code. I don't think I'm using even 20% of its features.

indent-rainbow

Very helpful plugin for working with languages like Python, where indentation matters. Every level of indentation gets a slightly different color, so it's easier to see at a glance where a given code block ends.

jumpy (or MetaGo)

jumpy is a very peculiar plugin that takes some time to get used to. Basically, it's supposed to help you move around your code faster.

If you press a keyboard shortcut, jumpy will display a 2-letter code next to every word on the screen. If you type those two letters, your cursor will jump to that location. Similar to what you can do with vim in "normal" mode (with less typing).

Paste and Indent

If you find that VS Code is not doing a good job when you paste code, try this extension. It will let you assign a "Paste and Indent" action to any key shortcut. This command will do its best to indent the code correctly after you paste it (to match the surrounding code). I'm using the "Command+Shift+V" shortcut for it.

Project Manager

VS Code supports the concept of workspaces - you can group some files and folders together and easily switch between them. But you still need to save the workspace configuration, and sometimes it can get lost - I either accidentally remove it or forget where I saved it.

Project Manager takes this hassle away. You can save projects and then open them, no matter where they are located (and you don't have to worry about storing the workspace preference files). Also, it adds a sidebar to browse all your projects.

Quick and Simple Text Selection

I like to use shortcuts that let me select all the text in brackets, tags, etc. By default, VS Code has command to "Expand/Shrink selection" that works ok-ish, but I found the Quick and Simple Text Selection plugin to be a much better way.

It adds a few new shortcuts to select text in:

single/double quotes
parentheses
square/angular/curly brackets
tags

I tried to map them to some intuitive shortcuts and they work like a charm:

Command + ' (⌘ + ') - select text in single quotes
Command + " (⌘ + ⇧ + ')- select text in double quotes
Command + ( (⌘ + ⇧ + 9)- select text in parentheses
Command + < (⌘ + ⇧ + ,)- select text in tag
Command + , (⌘ + ,)- select text in angular brackets

Settings Sync

It's not really related to Python, but it's a very important plugin, so I wanted to mention it.

Settings Sync lets you save the VS Code settings to a private GitHub gist, so you can easily restore them if you switch to a different computer (or if you lose/destroy your current one).

In one of the upcoming versions of VS Code, settings synchronization will become built-in.

TODO Highlight

Highlights all TODO/FIXME/NOTE in the code, so you can easily spot them. You can easily customize it by adding new words and changing the highlight style.

Spell Right

It's strange, but VS Code doesn't have a built-in spell checker. So you have to install one as an extension.

5 Ways of Debugging with IPython

2019-12-23T00:00:00Z

There is a great article from Tenderlove - one of the core Ruby and Rails developers - called "I am a puts debuggerer", that I enjoyed when I played with Ruby. The gist of it is to show you that, in many cases, you don't need a full-fledged debugger. Don't get me (or Tenderlove) wrong - the debugger that comes with a good IDE is one of the most powerful tools that a programmer can have! You can easily put breakpoints in your code, move around the stack trace or inspect and modify variables on the fly. It makes working with large codebase much easier and helps newcomers get up to speed on a new project.

Yet, people still use print statements for debugging their code. I do this all the time. Printing a variable is fast and easy. "I'm going to start a debugging session" sounds heavy. "I think there is a bug with this one variable. I'm going to print it!" doesn't. Never mind that 5 minutes later our one print statement turns into:

print(a_varible)

...

if foo:
    print(">>>>>>>>>>>>>>Inside 3rd IF")

...

    print(">>>>>>>>>>>>>>Inside 37th IF")

print(">>>>>>>>>> #@!?#!!!")

Sounds familiar? There is nothing wrong with using print for debugging. Quite often, it’s all you need to find the bug. And sometimes, it’s the only way that you can debug your code. You can't easily attach a debugger to your production code without impacting your users. But, adding some print statements and then looking at the logs should be fine.

And not everyone is using an IDE with a good debugger. According to the Stack Overflow Developer Survey Results 2019, 30.5% of developers are using Notepad++, 25.4% Vim, and 23.4% Sublime Text. Those are text editors! And even though I have seen people being more productive in Vim than most of the PyCharm or VS Code users, text editors are not created with a powerful debugger in mind. You can always use the standard Python debugger pdb, but a much better alternative is to use IPython as your debugger.

I've been using VS Code for almost two years, but I don't remember when was the last time I used the built-in debugger. I do most of my debugging in IPython. Here is how I'm using it:

Embedding IPython session in the code

The most common case for me is to embed an IPython session in the code. All you need to do is to put the following lines in your code:

from IPython import embed
embed()

I like to put those two statements in the same line:

from IPython import embed; embed()

so I can remove them with one keystroke. And, since putting multiple statements on the same line is a bad practice in Python, every code linter will complain about it. That way, I won't forget to remove it when I'm done 😉.

When you run your code and the interpreter gets to the line with the embed() function, it will open an IPython session. You can poke around and see what's going on in the code. When you are done, you just close the session (Ctrl+d) and the code execution will continue. One nice thing about this approach is that all the modifications done in IPython will persist when you close it. So you can modify some variables or functions (you can even decorate functions with some simple logging) and see how the rest of the code will behave.

Here is a short demo of embed() in action. Let's say we have the following file:

a = 10
b = 15

from IPython import embed; embed()

print(f"a+b = {a+b}")

This is what happens when we run it:

As you can see, I changed the value of the a variable and the new value persisted after I closed the IPython session.

Putting a breakpoint in your code

Embedding an IPython session in the code is fine if you want to see what's going on at a given line. But you can't execute the next lines of code, as a real debugger would do. So a better idea is to put a breakpoint in your code instead. Starting with version 3.7 of Python, there is a new built-in function called breakpoint() that you can use for that. If you are using an older version of Python, you can achieve the same effect by running the following code:

import pdb; pdb.set_trace()

The default debugger (pdb) is pretty rudimentary. Just like in the standard Python REPL, you won't get the syntax highlighting or automatic indentation. A much better alternative is the ipdb. It will use IPython as the debugger. To enable it, use the ipdb instead of pdb:

import ipdb; ipdb.set_trace()

There is also another interesting debugger called PDB++. It has a different set of features than ipdb, for example, a sticky mode that keeps showing you the current location in the code.

No matter which debugger you end up using, they have a pretty standard set of commands. You can execute the next line by calling the next command (or just n), step inside the function with step (or s), continue until the next breakpoint with continue (or c), display where you are in the code with l or ll, etc. If you are new to these CLI debuggers, the "Python Debugging With Pdb" tutorial is a good resource to learn how to use them.

%run -d filename.py

IPython has another way to start a debugger. You don't need to modify the source code of any file as we did before. If you run the %run -d filename.py magic command, IPython will execute the filename.py file and put a breakpoint on the first line there. It's just as if you would put the import ipdb; ipdb.set_trace() manually inside the filename.py file and run it with python filename.py command.

If you want to put the breakpoint somewhere else than the first line, you can use the -b parameter. The following code will put the breakpoint on line 42:

%run -d -b42 filename.py

Keep in mind that the line that you specify has to contain code that actually does something. It can't be an empty line or a comment!

Finally, there might be a situation where you want to put a breakpoint in a different file than the one that you will run. For example, the bug might be hidden in one of the imported modules and you don't want to type next 100 times to get there. The -b option can accept a file name followed by a colon and a line number to specify where exactly you want to put the breakpoint:

%run -d -b myotherfile.py:42 myscript.py

The above code will put a breakpoint on line 42 in a file called myotherfile.py and then start executing file myscript.py. Once the Python interpreter gets to myotherfile.py, it will stop at the breakpoint.

Post-mortem debugging

IPython has 176 features^[1]. Post mortem debugging is the best one. At least for me. Imagine that you are running a script. A long-running script. And suddenly, after 15 minutes, it crashes. Great - you think - now I have to put some breakpoints, rerun it and wait for another 15 minutes to see what's going on. Well, if you are using IPython, then you don't have to wait. All you need to do now is to run the magic command %debug. It will load the stack trace of the last exception and start the debugger (Python stores the last unhandled exception inside the sys.last_traceback variable). It's a great feature that has already saved me hours of rerunning some commands just to start the debugger.

If you are using the standard pdb debugger, you can achieve the same behavior by running the import pdb; pdb.pm() command.

Automatic debugger with %pdb

The only way to make debugging even more convenient is to automatically start a debugger if an exception is raised. And IPython has a magic command to enable this behavior - %pdb.

If you run %pdb 1 (or %pdb on), a debugger will automatically start on each unhandled exception. You can turn this behavior off again with %pdb 0 or %pdb off. Running %pdb without any argument will toggle the automatic debugger on and off.

Photo by Steinar Engeland on Unsplash

This number is totally made up. I'm sorry my data-driven friends. ↩︎

Disable pip Outside of Virtual Environments

2019-11-28T00:00:00Z

Python packages everywhere

I'm a huge fan of virtual environments in Python. They are a convenient way to manage dependencies if you are working on more than one Python project at a time. Well, they are the only way to manage dependencies between projects. In the JavaScript world, if you run npm install it will create a local folder with all the packages and use it in your project (falling back to global packages if a dependency is missing). In Python, all your packages are installed in the same place. And if you want to install a different version of a package, the previous one will be uninstalled:

$ pip install pygments==2.2
Collecting pygments==2.2
  Using cached https://files.pythonhosted.org(...).whl
Installing collected packages: pygments
  Found existing installation: Pygments 2.4.2
    Uninstalling Pygments-2.4.2:
      Successfully uninstalled Pygments-2.4.2
Successfully installed pygments-2.2.0

The best you can do in this situation is to install packages into your user directory (with pip install --user), but that doesn't really solve the problem.

Plenty of tools have been created to solve the dependencies management problem. From the most popular ones like the pipenv or poetry to less popular like hatch (I have yet to meet someone using it) or dephell (that I have heard about at one of the Python conferences). Still, most of the people I know use the same setup as I do - the virtualenv package (with an optional wrapper like virtualenvwrapper or virtualenv burrito). For a long time I didn't even know that since Python 3.3, the virtualenv is baked into Python through the venv module. You can create virtual environments without any external tools by simply running python3 -m venv.

There is even a PEP 582 suggesting to use local packages directory (à la node_modules). So the landscape of Python dependencies managers might change in the future.

I can talk for hours about how to set up the most efficient workflow for Python. In fact, I did - at PyCon 2020! Check out my tutorial on how to set up a Python development environment, which tools to use, and finally - how to make a TODO application from scratch (with tests and documentation).

In my current setup, I'm using virtualenv with virtualfish. I've used virtualenvwrapper and I enjoyed being able to just run workon name-of-environment instead of looking where the activate script is placed. virtualfish is like virtualenvwrapper, but it adds even more short commands like vf ls or vf cd (as for a programmer, I really don't like typing).

And, especially at the beginning, I kept forgetting to activate the virtual environment before I cheerfully ran pip install a-package. Or even worse: pip install -r requirements.txt. Which cluttered my global pip directory with one more package (or hundreds of them in case of requirements.txt file). What's even worse, sometimes it also uninstalled the previous versions of packages. So other projects that I was building stopped working. And if you have the same package installed in a virtual env and globally - it can get messy sometimes.

There had to be a better way!

Make sure that pip only runs in a virtual environment

So one day I said "That's it! There has to be a way to at least get a warning that pip is running outside of a virtual environment!". It turns out that of course there is a way. And it's even built-in into pip! You can set the PIP_REQUIRE_VIRTUALENV environment variable to true and pip will never run outside of a virtual env! Simply add export PIP_REQUIRE_VIRTUALENV=true to your .bashrc or .zshrc (or set -gx PIP_REQUIRE_VIRTUALENV true in config.fish if you use fish shell). Now, each time you try to run pip outside of a virtual env, it will simply refuse to do so:

$ pip install requests
ERROR: Could not find an activated virtualenv (required).

If you want to actually install something outside of a virtual environment, you can temporarily clear that env variable: env PIP_REQUIRE_VIRTUALENV='' pip install request. Why would you ever want to do that? For example, to install the great pipx tool that lets you further isolate your command line Python packages.

You can also create a bash command to install pip packages that ignores this setting:

gpip() {
  PIP_REQUIRE_VIRTUALENV="" pip "$@"
}

Now I no longer have to worry about installing dependencies outside of a virtual environment!

Photo by Tim Evans on Unsplash

You Don't Have to Migrate to Python 3

2019-10-30T00:00:00Z

You can put your pitchforks and torches down - Python 3 is great! If you can migrate your project from Python 2 to Python 3, then by all means, you should do this. But with all the praise of Python 3 and all the great talks on how to migrate, we are forgetting about a huge portion of Python 2 applications. Applications that can't be migrated. Or don't have to be migrated. So let's talk about those.

This article is based on a talk that I gave at PyCon Japan 2019 called "It's 2019 and I'm still using Python 2. Should I be worried?". If you prefer to watch the video instead of reading, you can click the link above.

Python 2 End of Life

Python 3 has been out for over 10 years. The initial EOL (End of Life) for Python 2 was set to 2015, but it was extended until 01.01.2020. Back in 2013 and 2014, people were not ready to move to Python 3. Python 3.0 was pretty much unusable, Python 3.1 and 3.2 were slower than Python 2. But the main problem was that many of the 3rd party libraries were still using Python 2. It wasn't until 2012 that half of the 200 most popular Python packages were migrated to Python 3 (based on the information from the "Python 3 Wall of Shame/Superpowers" website that is no longer working). And by 2018 still, only around 95% of those packages were migrated. And those are the most popular packages! For the more obscure ones, the statistics were probably even worse. So developers were not ready in 2015. Thus, the deadline got extended by another 5 years. During those 5 years, a lot has changed. The latest versions of Python 3 (3.6 and up) are amazing - fast, feature-rich (whether you like the walrus operator or not 😉), and simply a pleasure to work with. Most of the Python packages have been migrated to Python 3. And those that didn't, probably won't. So how come that in 2019 there are still projects that are using Python 2? Well, there are a few reasons that I can think of.

Why do we still have Python 2 projects?

The cost of migration is too high from a business point of view. As developers, we understand that for the past few years, every line of Python 2 code that we write is a technical debt. But most companies are not run by developers. We all have managers that make decisions based on what business value each project brings to the company. And the fact that a programming language will be obsolete in a few months is often not a good enough reason to spend time rewriting everything. Migrating from Python 2 to Python 3 is expensive. And quite often it feels like it won't bring any money to the company. It won't add new features to your product and, while it will bring some speed improvements to your project, if it was the raw speed that you were looking for, you probably wouldn't choose Python in the first place. I have never seen a product that has "Python 3" as one of its features on the landing page. Unless it's a product for developers.

There is always a new feature waiting in the pipeline or an urgent fix that needs to be deployed. And if you are "Agile" (because now everyone is "Agile") and you have a huge backlog, migrating to Python 3 is probably somewhere at the bottom of it. If it was lucky enough to even get into the backlog. If you are a small startup, you need to focus on adding new features and improving users' experience, not on writing the perfect, most up-to-date code. You don't have time for refactoring or rewriting code that just works.

And if you are not a small startup, but a big corporation, you have another problem. A large code base of legacy Python (and by large I mean, for example, 35 000 000 lines of Python 2 code). And migrating old code can be scary. Imagine you have some code written by a developer who left the company a long time ago. There are little or no tests and the documentation is very poor, often outdated (if there is any). The code works, so it's fine. But no one has any idea how it works. So no one has been touching it for years. It's a scary thought that at some point, you will have to rewrite it. So the code stays in Python 2.

Migration to a new version of a programming language is a similar problem to refactoring. In both cases you need to set aside some time to rewrite existing code, hoping that you will make it better in the long run. But refactoring can be done following a "boy scout" rule, that says "you should always leave the place in a better shape than how you found it". So when you are adding a feature to a function, you clean up that function a bit. Migration can't be done like that. Even though you can start writing straddling code (code that will work with both Python 2 and Python 3), you will still have to rewrite other parts of the application at some point.

Risks of staying on Python 2

Let's fast forward 2 months. Python 2 is officially dead, everyone is getting ready for the party to celebrate at PyCon 2020 and you are just sitting there with your production code still running on Python 2. And thinking: "What's the worst that can happen?"

You can get hacked. Well, you can get hacked on Python 3 or any other programming language, but on Python 2 there is a bigger chance of that. Python 2 will not get any updates and this also includes bug fixes. If there is a 0-day for Python 2 discovered on the 2nd of January - good luck and have fun fixing it. No one from the core developers is going to fix it. But it's not the Python interpreter itself that you should be worried about. Your main problem is probably going to be the packages that you are using. Most of them have already abandoned their Python 2 versions and many more will follow in January. The more dependencies you are using, the more likely some of them will have security issues.

Even if there won't be any security issues with your software, as time goes, it will slowly start falling apart. Each time you update part of your system (and you will update them to stay secure), there is a chance that some of the underlying dependencies won't be happy with the new software. And maybe some developers will remove their packages from PyPI, tired of seeing users opening new issues in a project that they decided to deprecated a long time ago. In the end, you will spend more and more time firefighting to keep your project alive.

Removing packages from PyPI makes users angry

What can you do about Python 2 EOL?

So what can you do about the Python 2 End of Life? If you can migrate to Python 3, then do this! Long-term benefits will outweigh the cost of migration. But if you could migrate, you probably would do this long time ago and you wouldn't be reading this article. So I assume that you are looking for other solutions. Here is a list of solutions for Python 2 project, sorted by (my arbitrary feeling of) how difficult it is to implement each of them:

Do nothing

You can pretend that Python 3 never happened and ignore the whole Python 2 EOL problem. As I already mentioned before, by not updating your software you are risking that security vulnerabilities will sneak in (and sneak out your customers' data). Also, some of your dependencies might stop working at some point. But, if the only place where you use Python 2 is some kind of internal script that you run on your computer and it has no dependencies, then nothing is a perfectly fine thing to do! Don't update to Python 3 just because everyone tells you to do this (even though migrating such a simple script would be rather fast and easy). The same if you are expecting that your software will become obsolete next year (maybe you are working on another version already). Weigh the pros and cons of the migration and decide for yourself.

Freeze the state of your application

This is an interesting solution for all sorts of internal tools where you are not concerned about the security (by "internal" I mean - disconnected from the internet), but if some of the dependencies fail, you will be in trouble. Dependencies for Python 2 projects will start breaking next year. People will remove their old projects from GitHub or even PyPI, as I showed you above. Remember when we all laughed at JavaScript when someone removed a library that pads text left and suddenly all the builds started crashing? Well, prepare for that, but this time no one will really care, since "you are using a deprecated version of Python".

Luckily, we have docker! Or any other tool that lets you create immutable containers. Write a Dockerfile that uses Python 2 as a base image. Add all your dependencies there and set up your app as a docker image. Push that image to a public or private repository. And voilà, you have an immutable container with a working application! You can share it, reuse and you don't have to worry that some dependencies are no longer available. It solves most problems for internal tools. And you might want to do this now, not in 2020 when your application will already start giving you trouble.

Change Python interpreter

When I write "Python 2 EOL", I mean "CPython 2". CPython is the most popular Python interpreter, so for many people, Python == CPython. But it's not the only interpreter that we have. There is also, for example, PyPy which is a solid alternative to CPython. And since it's actually built on top of Python 2, PyPy is not planning to deprecate it at any point.

Don't think of PyPy as a "curiosity" that no one is using. PyPy is very mature, it's passing the same test suite as CPython (or as someone once joked "it's bug-to-bug compliant with CPython") and there are companies that have been using it in production for years. So it's a valid replacement for CPython 2. If you search on YouTube, you can find some examples of people happily running it in production - here is one.

So why isn't everyone using PyPy? Because it has some limitations. If your project relies heavily on C extensions, then PyPy might not be a good solution for you. But if you switch to PyPy and everything works fine - which you need to verify with tests - then your app might even run faster than before. Which is a nice side effect to have!

PyPy is not your only alternative. Intel is also maintaining its own distribution of Python called "Intel® Distribution for Python”. It's a free distribution that supports versions 2.7 and 3.6 of Python. When I spoke with one of the people involved in this project they assured me that they are also not planning to deprecate version 2.7 any time soon.

Commercial Python distributions

Finally, there are commercial solutions. One of them is Red Hat Enterprise Linux (RHEL). If you buy version 8, Red Hat will provide you with support for Python 2 until June 2024, as they are ensuring on their website. That could buy you 4 more years of bug fixes and updates for Python 2 ... at the price of switching from a free and open-source programming language to actually paying someone to use their distribution of Python. There are also other commercial vendors (that you can find on the internet) who will offer you paid support for Python 2 versions.

Maintain your own CPython 2 build

If you don't want to pay anyone for fixing Python 2, you can do this yourself! All you need to do is: fork the CPython repository, wait for vulnerabilities to appear, patch them, compile your own CPython version and use this on your production servers. It's exactly as tedious as it sounds and it's probably not the best idea unless you clearly know what you are doing. You don't want to be the one who introduces vulnerabilities on your server!

Migrate to Python 3

If none of the above options works for you, then you might end up migrating to Python 3. There are 2 common ways how you can do this: with straddling code or by rewriting Python 2 code to Python 3.

Straddling code

Straddling code is a code that works with both Python 2 and 3 at the same time. It sounds like more work, as you need to support both major Python versions, but it makes the transition easier - there is no sudden switch from Python 2 to Python 3. You start by running your tests under Python 3 (of course, most of them will fail) and you keep rewriting parts of your application until it works under Python 2 and Python 3. Then you change the Python version in production and finally, you remove the Python 2 code. The biggest advantage of this approach is that you can do this in iterations. You migrate parts of your system and you can keep adding new features to your code at the same time, so your customers will be happy.

Rewriting Python 2 to Python 3

The other option is to rewrite parts of Python 2 code in Python 3. It requires less work, as you don't care about Python 2 anymore. The typical approach is to keep Python 2 version of your app in production and start working on Python 3 version in a separate git branch. You keep testing the new version and when it's ready, you pull the plug on Python 2 code and turn on the Python 3 version. Which is scary as there might be things that you didn't test and then rolling back to Python 2 is going to be painful.

Also, this approach means that you need to stop adding features to your app. Otherwise, you will be doing double work - you will need to add those features to both Python 2 and Python 3 versions of your app.

Rewrite your application

The final and most difficult solution is to rewrite your application from scratch in Python 3 or in any other programming language that you think will work the best. This requires the biggest amount of work and it only makes sense if Python 2 version was just a prototype. But it lets you completely redesign your project, so maybe it will actually work well for you?

Should I migrate or not?

As I said at the beginning if you can migrate to Python 3, do this. Python 3 is faster than Python 2. It has plenty of great features like asyncio, type hints, ordered dictionaries, f-strings or better Unicode support. Most of the packages that were planning to migrate already did it. And those that didn't, probably won't migrate anyway. And finally - you won't be using a programming language that is no longer supported by its creators!

If you want to learn more about how to prepare for the migration process, watch the last part of my talk where I give some ideas or read the Python 3 porting book - it's a great, concise and free guide on how to survive the migration. See you on the other side of Python!

Photo by Nick Fewings on Unsplash

IPython Extensions Guide

2019-10-15T00:00:00Z

Modifying IPython is very easy. Need to execute some code at the startup? Add it to the startup directory. Need to change the caching behavior, exceptions verbosity level or the color theme? Open the .ipython_config.py file and modify everything there. But if you switch to a different computer, you will have to do all the changes again. Or maybe your colleague asks you how to customize his IPython, so it will look "as cool as yours". There is a better way than asking him to modify some configurations files. You can share your modifications as an extension!

What are IPython extensions?

IPython extensions are a great way to solve both problems. Any configuration change can be turned into an extension and shared with others (or simply installed on your second computer). Also, the magic functions that you create can be turned into extensions. Think of extensions as IPython plugins - you can write them yourself or install them from PyPI and, after you enable them, they will modify the behavior of IPython or add some new features.

You can keep the extensions for yourself, by storing them in the ~/.ipython/extensions folder or publish them on PyPI. In this article, I will show you how to install an existing extension and how to write and publish your own.

How to use IPython extensions?

To use an extension, you first need to load it with %load_ext command. IPython comes with 2 extensions bundled by default: %autoreload and %storemagic. There were more in the past, but they were moved to different packages. %autoreload, described in another post, can be used to automatically reload imported modules before executing code. It can be a helpful tool when writing a module. %storemagic is loaded by default and it lets you store variables, macros, and aliases in the SQLite database that comes with IPython. IPython doesn't store those objects between sessions, so unless you want to write and read your variables from a file, using the %storemagic is your best option to preserve and reuse them.

To enable an extension, you just need one command:

%load_ext my_extension

Extensions can have different effects:

Some will work immediately. For example, those that modify the IPython configuration.
Others need to be turned on first. For example, the %autoreload extension by default doesn't do anything. You need to turn on auto-reloading by running %autoreload 1 or %autoreload 2.
And some will add new features to IPython, for example, new magic functions.

Installing extensions from PyPI

Let's see how we can extend the functionality of IPython by adding some new extensions. There are two good ones that I'm using for profiling Python code: line_profiler and memory_profiler. The first one can be used to generate a line-by-line report about the execution time of your code (when you want to pinpoint which line of your code is slow). The second works similar, but this time it shows you a memory usage of your application.

Let's install the line_profiler:

pip install line_profiler

Now we can use this profiler in IPython:

%load_ext line_profiler

Loading the extension will add the %lprun magic function. To use it, we need to provide the names of the functions/modules that we want to profile and then a statement that we want to run.

Let's say we have some slow code that we want to check. I will use the following, pretty useless code, as an example:

def crunch_numbers():
    result = 0
    for x in range(1000):
        result += a_function(x)
        result += b_function(x)
    return result


def a_function(number):
    return number * number


def b_function(number):
    result = 0
    for i in range(number):
        result += i + 5
        if i % 10:
            result += 100 * i
    return result

We can use our newly installed extension to profile this script:

In [1]: from slow_module import crunch_numbers, a_function, b_function

In [2]: %load_ext line_profiler

In [3]: %lprun -f a_function -f b_function crunch_numbers()
Timer unit: 1e-06 s

Total time: 0.000503 s
File: /Users/switowski/workspace/slow_module.py
Function: a_function at line 9

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     9                                           def a_function(number):
    10      1000        503.0      0.5    100.0      return number * number

Total time: 0.698784 s
File: /Users/switowski/workspace/slow_module.py
Function: b_function at line 13

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    13                                           def b_function(number):
    14      1000        412.0      0.4      0.1      result = 0
    15    500500     159589.0      0.3     22.8      for i in range(number):
    16    499500     191225.0      0.4     27.4          result += i + 5
    17    499500     169746.0      0.3     24.3          if i % 10:
    18    449100     177483.0      0.4     25.4              result += 100 * i
    19      1000        329.0      0.3      0.0      return result

The output from the %lprun command will give you detailed information about each line of the function that you specified. You can see how many times this line was executed, what was the total time and "per hit" time, and what percentage of the total time spent in this function was spent on that particular line. If you think there is a problem with a particular line, line_profiler will also show you in which file this function is located, so you don't have to search for it.

In my case, you can see that the whole script was rather fast - it took around 0.6 seconds to finish. Most of the time was spent running this instruction: result += i + 5 on line 16 of slow_module.py file, inside the b_function function.

If you want to look for more IPython extensions, there are 2 good places to find them:

IPython Extensions Index - a wiki page in IPython's GitHub repository that contains a huge list of available extensions. All the entries here are manually curated. Some of them might be outdated, and they won't work anymore since the IPython's API for extensions has changed between major versions. But it's a great place to search for a specific extension, as each entry has a short description of what it's supposed to do. If you find an extension that you want to use and it fails to install or load, try to copy and paste the code of the extension into IPython - it might work that way. And if it does, try turning this code into an extension and submit a Pull Request to update the original version (more on how to create your own extensions below).
Framework::IPython filter on PyPI - sharing extensions on PyPI is now the recommended way. It makes installing extensions much easier. But sometimes the extensions are not properly tagged, so you might also find some by searching for "IPython" or "IPython magic" on PyPI.

Writing an extension

If you can't find an extension that you like, writing your own is very easy. All you need to do is:

Create a file with load_ipython_extension function. This function will be called when you run %load_ext my_extension. Inside this function, you should put all the code that you want to make available after your extension is loaded. For example, if your extension is creating a magic function, put this magic function here.
[Optional] If you want to be able to unload your extension, you can add the unload_ipython_extension function as well. Loading an extension turns it on and unloading - turns it off. It doesn't make sense to unload an extension that adds new magic functions unless you want to disable them for some reason. But it can be useful if your extension is altering the behavior of IPython. For example, if you have an extension that automatically measures the execution of each command that you run, and at some point, you want to get rid of this behavior, you can unload it.
Finally, you need to save the file in a place where IPython can access it. There is a folder inside the .ipython config directory called extensions where you can store your extensions.

Let's say we want to write an extension that will add a new magic function to IPython. Here is all the code that we need:

from IPython.core.magic import register_line_magic


def load_ipython_extension(ipython):
    @register_line_magic("reverse")
    def lmagic(line):
        "Line magic that reverses any string that is passed"
        return line[::-1]

The register_line_magic function will turn our lmagic function into IPython's magic function. Keep in mind that load_ipython_extension has a specific signature that you need to use - it should accept ipython argument. If you don't provide this argument, your extension won't work.

Save this code inside the ~/.ipython/extensions/reverser.py file. The name of the file that you use will be the name of your extension in IPython. You can rename it if you don't like the name reverser, but remember to pass this new name to the %load_ext function.

Now, we can load and test our extension in IPython:

In [1]: %load_ext reverser
Loading extensions from ~/.ipython/extensions is deprecated.
We recommend managing extensions like any other Python packages, in site-packages.

In [2]: %reverse hello world!
Out[2]: '!dlrow olleh'

Great, it works! If we add the unload_ipython_extension, we could also run the %unload_ext reverser, but it doesn't make much sense for an extension that is creating a magic function.

So this is how you can write your own IPython extensions. You might be wondering - what's with this deprecation warning that we saw when we imported our extension:

Loading extensions from ~/.ipython/extensions is deprecated. We recommend managing extensions like any other Python packages, in site-packages.

Does it mean that we did something wrong by putting our extension in the extensions folder? Don't worry, it's the correct folder. This deprecation warning is a suggestion that you should share your extension with others by publishing in on PyPI. If you think that your extension can be useful to others, you should definitely do this! I don't think that my reverser is, but for the illustration purpose, I'm going to publish it anyway 😉.

Publishing extension on PyPI

To publish my extension, I need to turn it into a Python package. There are many great tutorials on how to create Python packages. But to keep my example simple, I will just do the absolutely necessary steps to create a Python package by following the guidelines from the Python Packaging Authority. So please, don't take this article as an example of how to create Python packages 😅.

Here is the structure of the package:

ipython-reverser/
├── LICENSE
├── README.rst
├── ipython_reverser
│   └── __init__.py
└── setup.py

And here is what's inside each of the files:

LICENSE - this is an optional file, but it's a good practice to specify a license for each of your projects. If you don't add a license, no one can actually use it! So don't think that projects without a license are free to copy and reuse!
README.rst - another optional file, but it's good to explain what this project does. The content of this file will be displayed on GitHub.

setup.py containing the following code:

# setup.py
from setuptools import setup

setup(
    name="IPythonReverser",
    version="0.1",
    packages=["ipython_reverser"],
    license="MIT",
    author="Sebastian Witowski",
    author_email="[email protected]",
    url="http://www.github.com/switowski/ipython-reverser",
    description="IPython magic to reverse a string",
    long_description=open("README.rst").read(),
    keywords="ipython reverser reverse",
    install_requires = ['ipython'],
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Intended Audience :: Developers",
        "Framework :: IPython",
        "Programming Language :: Python",
        "Topic :: Utilities",
    ],
)

ipython_reverser/__init__.py - in older versions of Python (before Python 3.3), you had to have an __init__.py file in each of the subdirectories of your package. Without it, you wouldn't be able to import functions from the subdirectories. In the newer versions of Python, they are no longer necessary, but there is a benefit of using them - if you create such a file, it will be automatically executed when you import a module. So, I'm putting the code of my extension inside:
```
# ipython_reverser/__init__.py
from IPython.core.magic import register_line_magic


def load_ipython_extension(ipython):
    @register_line_magic("reverse")
    def lmagic(line):
        "Line magic to reverse a string"
        return line[::-1]
```

You can find the source code of the package on GitHub.

Generating the package

Now, I need to install some tools that I will use in the next step (if you are using a virtual environment, you can skip the python3 -m part of the following commands):

python3 -m pip install --user --upgrade setuptools wheel

Next, I generate the distribution package:

python3 setup.py sdist bdist_wheel

This will create the package inside the dist/ directory.

To publish my package to PyPI, I need to install yet another tool called twine:

python3 -m pip install --user --upgrade twine

[OPTIONAL STEP] If it's the first time you are publishing a package to PyPI, you can do a test run and publish it to TestPyPI. That way you can check if everything is working, without affecting the real PyPI. To publish your package to PyPI, run the following command:

python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*

The first time you interact with twine, it will ask you for your username and password. So make sure to create an account on PyPI. To install a package from TestPyPI, you need to pass --index-url parameter to pip:

python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps your-package

Finally, I can publish the package to PyPI with the following command:

python3 -m twine upload dist/*

Twine will ask you for your username and password, and then you should see a progress bar indicating that everything worked fine.

Now, anyone can install my IPythonReverser package using pip:

python3 -m pip install IPythonReverser

and use it in IPython:

In [1]: %load_ext ipython_reverser

In [2]: %reverse 'hello world from PyPI!'
Out[2]: "'!IPyP morf dlrow olleh'"

One thing to remember - this time we have to use the name of the module when we load our extension. So we use %load_ext ipython_reverser instead of %load_ext reverser.

Conclusions

Extensions are one of the most powerful features of IPython. They are very easy to create and to publish on PyPI, so if you come up with a great extension (something more useful than reversing strings), make sure you share it!

Image from: Unsplash

Automatically Reload Modules with %autoreload

2019-10-01T00:00:00Z

Writing my first module in Python was a confusing experience. As it usually happens, when I was testing it in the interactive Python REPL, the first version turned out to have some bugs (the second and third ones also did 😉).

That's fine - I thought - I will just fix the module and reimport it.

But, to my surprise, calling from my_module import my_function didn't update the code! my_function still had the bug that I just fixed! I double-checked if I modified the correct file, reimported it again and still nothing. It turns out, as StackOverflow kindly explained, that you can't just reimport a module. If you already imported a module (import a_module) or a function (from a_module import a_function) in your Python session and you try to import it again, nothing will happen. It doesn't matter if you use the standard Python REPL or IPython.

How does importing in Python work?

Turns out that, for efficiency reasons, when you import a module in an interactive Python session, Python interpreter does two steps:

First, it checks if the module is already cached in the sys.module dictionary.
And only if it's not there, it actually imports the module.

Which means that, if you already imported the module (or imported a different module that references this one) and you try to import it again, Python will ignore this request. You can read more about how importing works in the documentation.

So, if I can't reimport a module, does it mean that I have to restart Python each time? Not really, that would be very inconvenient.

How to reimport a module?

The easiest way is to quit your interactive session and start it again. It works fine if you don't care about preserving the data that you already have in your session, like the functions that you wrote and the variables that you calculated. But usually you don't want to restart the REPL, so there are better ways.

Since we know that the interpreter will first look for the module in the sys.modules dictionary, we can just delete our module from this dictionary. And it will work in most cases, but there are some caveats. If your module is referenced from another module, there is a chance that you still won't be able to reimport it. So don't do this. There is a better way.

The recommended solution is to use the importlib.reload function. This function is designed exactly for reimporting modules that have already been imported before. To reload your module, you need to run:

import importlib
importlib.reload(my_module)

So that's how you can reimport a module in Python. And if you are not using IPython, this is where your options end. But IPython users have some other interesting solutions to this problem.

%run

If you don't care about actually "importing" your module and all you need is to run some functions defined in a file, you can execute that file instead. It will run all the commands as if you would copy and paste them in your IPython session. You can rerun a file as many times as you want and it will always update all the functions. Running a file in IPython is extremely easy:

%run my_file.py
# You can even skip the ".py" extension:
%run my_file

I cheated a bit when I said that this option is not available in standard Python REPL. It is, but it requires more typing:

exec(open("./my_file.py").read())

To be honest, if I had to type all this, I might as well just use the importlib.reload instead.

All those options are great, but if you are as bad as me when it comes to writing code and you make a lot of mistakes, then it means a lot of reloading. And typing this importlib.reload / %run / exec... is annoying. Wouldn't it be great if there was a way to automatically reload a module? Well, IPython can actually do that!

%autoreload to the rescue

Another one of the magic methods in IPython is related to reloading modules. It's called %autoreload. It's not enabled by default, so you have to load it as an extension:

%load_ext autoreload

Now, you can turn on auto-reloading:

%autoreload 2

And each time you execute some code, IPython will reimport all the modules to make sure that you are using the latest possible versions.

There are 3 configuration options that you can set:

%autoreload 0 - disables the auto-reloading. This is the default setting.
%autoreload 1 - it will only auto-reload modules that were imported using the %aimport function (e.g %aimport my_module). It's a good option if you want to specifically auto-reload only a selected module.
%autoreload 2 - auto-reload all the modules. Great way to make writing and testing your modules much easier.

Great, any caveats? I found 3 minor ones:

IPython with %autoreload enabled will be slightly slower. IPython is quite smart about what to reload. It will check the modification timestamps of the modules and compare them with the time when they are imported. But this checking (and eventually reimporting of the modified modules) will still take some time. It won't be so slow that you will feel it (unless you have modules that take seconds to import), but it will obviously run faster if you disable the auto-reloading.
As pointed out in the documentation, %autoreload is not 100% reliable, and there might be some unexpected behaviors. I never noticed any problems, but some reddit users mentioned that it might not work correctly for the more advanced modules (with classes, etc.).
You need to make sure that you don't have syntax errors in your modules when you are running IPython commands. I often start writing some code in a file and, in the middle of the command, I switch to IPython to quickly test something. And when I execute some code in IPython, it will try to reimport the file that I just modified (the one with the half-written command) and throw a SyntaxError. The good thing is - after the error, you will still get the output of the command that you ran. So for me, it's a minor annoyance, not a real problem. You can easily solve it by running two IPython sessions - one for testing the module (with %autoreload enabled) and the other for running some random commands and looking up things in the documentation.

Here is how %autoreload works in practice (this video is recorded with asciinema, and if you watch it on mobile phone, part of the final comment is cut - it says: #without autoreload, we would still see "hello !"):

So if you don't know %autoreload yet, give it a try the next time you're working on a module in Python!

Image from: Unsplash

It's 2019 and I'm Still Using Python 2

2019-08-28T00:00:00Z

Here are the slides for my talk called "It's 2019 and I'm still using Python 2. Should I be worried?".

Since I update the slides before each conference to incorporate any new ideas that come to my mind and make sure they are up to date, if you are interested in a particular version of the slides, just send me an email and I will sent them your way.

Enjoy!

Wait, IPython Can Do That?!

2019-07-07T00:00:00Z

Here are the slides for my talk called "Wait, IPython can do that?!".

Enjoy!

Slides for a 30-minute-long version of this talk are available here.

Creating Magic Functions in IPython - Part 3

2019-02-15T00:00:00Z

So far in this series, we have covered three different decorators: @register_line_magic (in part1), @register_cell_magic and @register_line_cell_magic (in part2). Which is enough to create any type of magic function in IPython. But, IPython offers another way of creating them - by making a Magics class and defining magic functions within it.

Magics classes

Magics classes are more powerful than functions, in the same way that a class is more powerful than a function. They can hold state between function calls, encapsulate functions, or offer you inheritance. To create a Magics class, you need three things:

Your class needs to inherit from Magics
Your class needs to be decorated with @magics_class
You need to register your magic class using the ipython.register_magics(MyMagicClass) function

In your magic class, you can decorate functions that you want to convert to magic functions with @line_magic, @cell_magic and @line_cell_magic,

Writing a magics class

To show how the magics class works, we will create another version of mypy helper. This time, it will allow us to run type checks on the previous cells. This is how we expect it to work:

In [1]: def greet(name: str) -> str:
   ...:     return f"hello {name}"

In [2]: greet('tom')
Out[2]: 'hello tom'

In [3]: greet(1)
Out[3]: 'hello 1'

In [4]: %mypy 1-2
Out[4]: # Everything should be fine

In [4]: %mypy 1-3
Out[4]: # It should report a problem on cell 3

Here are a few assumptions about the %mypy function:

It should accept all the parameters that the mypy command accepts
It should accept the same range parameters that %history command accepts, but only from the current session. I usually don't reference history from the previous sessions anyway and it will make parsing arguments slightly easier. So 1, 1-5, and 1 2 4-5 are all valid arguments, while 243/1-5 or ~8/1-~6/5 are not.
The order of arguments doesn't matter (and you can even mix ranges with mypy arguments), so we can call our function in the following ways:
- %mypy --ignore-imports 1 2 5-7
- %mypy 1-3
- %mypy 2 4 5-9 --ignore-imports
- %mypy 2 4 --ignore-imports 5-9

With that in mind, let's write the code. The main class looks like this:

from IPython.core.magic import Magics, magics_class, line_magic
import re

# The class MUST call this class decorator at creation time
@magics_class
class MypyMagics(Magics):
    @line_magic
    def mypy(self, line):
        try:
            from mypy.api import run
        except ImportError:
            return "'mypy' not installed. Did you run 'pip install mypy'?"

        if not line:
            return "You need to specify cell range, e.g. '1', '1 2' or '1-5'."

        args = line.split()
        # Parse parameters and separate mypy arguments from cell numbers/ranges
        mypy_arguments = []
        cell_numbers = []
        for arg in args:
            if re.fullmatch(r"\d+(-\d*)?", arg):
                # We matched either "1" or "1-2", so it's a cell number
                cell_numbers.append(arg)
            else:
                mypy_arguments.append(arg)

        # Get commands from a given range of history
        range_string = " ".join(cell_numbers)
        commands = _get_history(range_string)

        # Run mypy on that commands
        print("Running type checks on:")
        print(commands)

        result = run(["-c", commands, *mypy_arguments])

        if result[0]:
            print("\nType checking report:\n")
            print(result[0])  # stdout

        if result[1]:
            print("\nError report:\n")
            print(result[1])  # stderr

        # Return the mypy exit status
        return result[2]


ip = get_ipython()
ip.register_magics(MypyMagics)

We have the MypyMagics class (that inherits from Magics) and in it, we have the mypy line magic that does the following:

checks if mypy is installed
if there were no arguments passed - it returns a short information on how to use it correctly.
parses the arguments and splits those intended for mypy from the cell numbers/ranges. Since mypy doesn't accept arguments that look like a number (1) or range of numbers (1-2), we can safely assume that all arguments that match one of those 2 patterns, are cells.
retrieves the input values from the cells using the _get_history helper (explained below) as a string, and prints that string to the screen, so you can see what code will be checked.
runs the mypy command, prints the report and returns the exit code.

At the end, we need to remember to register the MypyMagics class in IPython.

We are using one helper function on the way:

def _get_history(range_string):
    ip = get_ipython()
    history = ip.history_manager.get_range_by_str(range_string)
    # history contains tuples with the following values:
    # (session_number, line_number, input value of that line)
    # We only need the input values concatenated into one string,
    # with trailing whitespaces removed from each line
    return "\n".join([value.rstrip() for _, _, value in history])

I told you before, that when writing a class, we can put our helper function inside, but I'm purposefully keeping this one outside of the MypyMagics. It's a simple helper that can be used without any knowledge about our class, so it doesn't really belong in it. So, I'm keeping it outside and using the naming convention to suggest that it's a private function.

Coming up with the _get_history helper was quite a pickle, so let's talk a bit more about it.

Approach 1: `_ih`

I needed to retrieve the previous commands from IPython, and I knew that IPython stores them in _ih list (so, if you want to retrieve, let's say, the first command from the current session, you can just run _ih[1]). It sounded easy, but it required some preprocessing. I would first have to translate 1-2 type of ranges into list slices. Then I would have to retrieve all parts of the history, one by one, so for 1 2-3 5, I would need to call _ih[1], _ih[2:4], _ih[5]. It was doable, but I wanted an easier way.

Approach 2: `%history`

My next idea was to reuse the %history magic function. While you can't just write %history in Python code and expect it to work, there is a different way to call magics as standard functions - I had to use the get_ipython().magic(<func_name>) function.

Problem solved! Except that %history magic can either print the output to the terminal or save it in a file. There is no way to convince it to return us a string. Bummer! I could overcome this problem in one of the following 2 ways:

Since by default %history writes to sys.stdout, I could monkey-patch (change the behavior at runtime) the sys.stdout and make it save the content of history output in a variable. Monkey patching is usually not the best idea and I didn't want to introduce bad practices in my code, so I didn't like this solution.
Otherwise, I could save the output of %history to a file and then read it from that file. But creating files on a filesystem just to write something inside and immediately read it back, sounds terrible. I would need to worry about where to create the file, whether or not the file already exists, then remember to delete it. Even with tempfile module that can handle the creation and deletion of temporary file for me, that felt like too much for a simple example.

So the %history function was a no-go.

Approach 3: `HistoryManager`

Finally, I decided to peak inside the %history and use whatever that function was using under the hood - the HistoryManager from IPython.core.history module. HistoryManager.get_range_by_str() accepts the same string formats that %history function does, so no preprocessing was required. That was exactly what I needed! I only had to clean the output a bit (retrieve the correct information from the tuples) and I was done.

Testing time

Now, that our %mypy helper is done (the whole file is available on GitHub) and saved in the IPython startup directory, let's test it:

In [1]: def greet(name: str) -> str:
   ...:     return f"hello {name}"
   ...:

In [2]: greet('Bob')
Out[2]: 'hello Bob'

In [3]: greet(1)
Out[3]: 'hello 1'

In [4]: %mypy 1-3  # this is equivalent to `%mypy 1 2 3`
Running type checks on:
def greet(name: str) -> str:
    return f"hello {name}"
greet('Bob')
greet(1)

Type checking report:

<string>:4: error: Argument 1 to "greet" has incompatible type "int"; expected "str"

Out[4]: 1

# What about passing parameters to mypy?
In [5]: import Flask

In [6]: %mypy 5
Running type checks on:
import flask

Type checking report:

<string>:1: error: No library stub file for module 'flask'
<string>:1: note: (Stub files are from https://github.com/python/typeshed)

Out[6]: 1

In [7]: %mypy 5 --ignore-missing-imports
Running type checks on:
import flask
Out[7]: 0

Perfect, it's working exactly as expected! You now have a helper that will check types of your code, directly in IPython.

There is only one thing that could make this even better - an automatic type checker that, once activated in IPython, will automatically type check your code as you execute it. But that's a story for another article.

Conclusions

This the end of our short journey with IPython magic functions. As you can see, there is nothing magical about them, all it takes is to add a decorator or inherit from a specific class. Magic functions can further extend the already amazing capabilities of IPython. So, don't hesitate to create your own, if you find yourself doing something over and over again. For example, when I was working a lot with SQLAlchemy, I made a magic function that converts an sqlalchemy row object to Python dictionary. It didn't do much, except for presenting the results in a nice way, but boy, what a convenience that was, when playing with data!

Do you know any cool magic functions that you love and would like to share with others? If so, you can always send me an email or find me on Twitter!

Image from: pixabay

Creating Magic Functions in IPython - Part 2

2019-02-08T00:00:00Z

In the previous post, I explained what the magic functions are and why they are cool. We have also created a line magic function that interprets mathematical formulas written in Polish notation. Today, we will talk about cell magic functions.

Cell magics in IPython

Cell magics are similar to line magics, except that they work on cells (blocks of code), not on single lines. IPython comes with a few predefined ones and most of them will let you interpret code written in a different programming language. Need to run some Python 2 code, but IPython is using Python 3 by default? No problem, just type %%python2, paste/type the code and run it:

In [1]: print 'hello there'
  File "<ipython-input-1-202d533f5f80>", line 1
    print 'hello there'
                      ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print('hello there')?

# But!

In [2]: %%python2
   ...: print 'hello there'
   ...:
   ...:
hello there

You can also run code written in Ruby, Bash, JavaScript, and other languages. And those different blocks of code can interact with each other, for example, you can run some JavaScript code and send variables back to Python.

Writing a cell magic function

Now, let's try to write our own cell magic function. I initially wanted to continue with the example of Polish notation from the first part of the series. So I started writing a function that translates all the mathematical operations in a block of code into a Polish notation form. Unfortunately, I quickly realized that if I want to write a good example (not some half-assed code that works only for + and -), I would have to write a proper interpreter. And that would no longer be a simple example^[1]. So this time, we are going to do something different.

One of the new features that came in Python in version 3.5 are type hints. Some people like them, some people don't (which is probably true for every new feature in every programming language). The nice thing about Python type hints is that they are not mandatory. If you don't like them - don't use them. For fast prototyping or a project that you are maintaining yourself, you are probably fine without them. But for a large code base, with plenty of legacy code maintained by multiple developers - type hints can be tremendously helpful!

As you are probably starting to guess, our cell magic function will check types for a block of code. Why? Well, with IPython, you can quickly prototype some code, tweak it and save it to a file using the %save or %%writefile magic functions (or simply copy and paste it, if it's faster for you). But, at the time of writing this article, there is no built-in type checker in Python. The mypy library is a de facto static type checker, but it's still an external tool that you run from shell (mypy filename.py). So let's make a helper that will allow us to type check Python code directly in IPython!

This is how we expect it to work:

In [1]: %%mypy
   ...: def greet(name: str) -> str:
   ...:     return f"hello {name}"
   ...: greet(1)
   ...:
   ...:
Out[1]: # It should print an error message, as 1 is not a string

To achieve this, we will simply call the run function from mypy.api (as suggested in the documentation) and pass the -c PROGRAM_TEXT parameter that checks a string.

Here is the code for the type checker:

from IPython.core.magic import register_cell_magic

@register_cell_magic('mypy')
def typechecker(line, cell):
    try:
        from mypy.api import run
    except ImportError:
        return "'mypy' not installed. Did you run 'pip install mypy'?"
    
    args = []
    if line:
        args = line.split()
    
    result = run(['-c', cell, *args])

    if result[0]:
        print('\nType checking report:\n')
        print(result[0])  # stdout

    if result[1]:
        print('\nError report:\n')
        print(result[1])  # stderr

    # Return the mypy exit status
    return result[2]

Let's go through the code, given that there are a few interesting bits:

@register_cell_magic(mypy)
def typechecker(line, cell):

We start by defining a function called typechecker and registering it as a cell magic function called %%mypy. Why didn't I just define a function called mypy instead of doing this renaming? Well, if I did that, then our mypy function would shadow the mypy module. In this case, it probably won't cause any problems. But in general, you should avoid shadowing variables/functions/modules, because one day, it will cause you a lot of headache.

try:
    from mypy.api import run
except ImportError:
    return "`mypy` not found. Did you forget to run `pip install mypy`?"

Inside our function, we first try to import the mypy module. If it's not available, we inform the user that it should be installed, before this magic function can be used. The nice thing about importing mypy in the typechecker function is that the import error will show up only when you run the magic function. If you put the import at the top of the file, then save the file inside IPython startup directory, and you don't have mypy module installed, you will get the ImportError every time you start IPython. The downside of this approach is that you are running the import code every time you run the typechecker function. This is something that you should avoid doing, if you care about the performance, but in case of our little helper, it's not a big problem.

If you are using Python 3.6 or higher, you can catch the ModuleNotFoundError error instead of ImportError. ModuleNotFoundError is a new subclass of ImportError thrown when a module can't be located. I want to keep my code compatible with lower versions of Python 3, so I will stick to the ImportError.

args = []
if line:
    args = line.split()

result = run(['-c', cell, *args])

Note that the function used for defining a cell magic must accept both a line and cell parameter. Which is great, because this way, we can actually pass parameters to mypy! So here, we are passing additional arguments from the line parameter to the run function. Here is how you could run our magic function with different settings:

In [1]: %%mypy --ignore-missing-imports --follow-imports error
   ...: CODEBLOCK

which is equivalent to running the following command in the command line: mypy --ignore-missing-imports --follow-imports error -c 'CODEBLOCK'.

The rest of the code is quite similar to the example from the documentation.

Testing time

Our cell magic function is ready. Let's save it in the IPython startup directory (what's IPython startup directory?, so it will be available next time we start IPython. In my case, I'm saving it in a file called:

~/.ipython/profile_default/startup/magic_functions.py

Now, let's fire up IPython and see if it works:

In [1]: %%mypy
   ...: def greet(name: str) -> str:
   ...:     return f"hello {name}"
   ...: greet('Bob')
   ...:
   ...:
Out[1]: 0

In [2]: %%mypy
   ...: def greet(name: str) -> str:
   ...:     return f"hello {name}"
   ...: greet(1)
   ...:
   ...:

Type checking report:

<string>:3: error: Argument 1 to "greet" has incompatible type "int"; expected "str"

Out[2]: 1

Great, it works! It returns 0 (which is a standard UNIX exit code for a successful command) if everything is fine. Otherwise, it reports what problems have been found.

How about passing some additional parameters?

In [3]: %%mypy
   ...: import flask
   ...:
   ...:

Type checking report:

<string>:1: error: No library stub file for module 'flask'
<string>:1: note: (Stub files are from https://github.com/python/typeshed)

Out[3]: 1

# Ok, this can happen (https://mypy.readthedocs.io/en/latest/running_mypy.html#ignore-missing-imports)
# Let's ignore this error

In [4]: %%mypy --ignore-missing-imports
   ...: import flask
   ...:
   ...:
Out[4]: 0

Passing additional parameters also works!

Great, we created a nice little helper function that we can use for checking, if the type hints are correct in a given block of code.

Line and cell magic function

There is one more decorator that we didn't discuss yet: @register_line_cell_magic. It's nothing special - especially now that you know how line magic and cell magic works - so there is no need for a separate article. IPython documentation explains this decorator very well:

@register_line_cell_magic
def lcmagic(line, cell=None):
    "Magic that works both as %lcmagic and as %%lcmagic"
    if cell is None:
        print("Called as line magic")
        return line
    else:
        print("Called as cell magic")
        return line, cell

If you run %lcmagic, this function won't receive the cell parameter and it will act as a line magic. If you run %%lcmagic, it will receive the cell parameter and - optionally - the line parameter (like in our last example with %%mypy). So you can check for the presence of cell parameter and based on that, control if it should act as a line or cell magic.

Conclusion

Now you know how to make a line magic and a cell magic functions and how to combine them together into a line and magic function. There is still one more feature that IPython offers - the Magics class. It allows you to write more powerful magic functions, as they can, for example, hold state in between calls. So stay tuned for the last part of this article!

Image from: Pexels

Writing a translator is still a great exercise! I recently followed the Let's Build A Simple Interpreter series, where you would build a Pascal interpreter in Python, and it was a really fun project for someone who never studied the compilers. So, if you are interested in this type of challenge, that blog can help you get started. ↩︎

Creating Magic Functions in IPython - Part 1

2019-02-01T00:00:00Z

IPython magic functions

One of the cool features of IPython are magic functions - helper functions built into IPython. They can help you easily start an interactive debugger, create a macro, run a statement through a code profiler or measure its' execution time and do many more common things.

Don't mistake IPython magic functions with Python magic functions (functions with leading and trailing double underscore, for example __init__ or __eq__) - those are completely different things! In this and next parts of the article, whenever you see a magic function - it's an IPython magic function.

Moreover, you can create your own magic functions. There are 2 different types of magic functions.
The first type - called line magics - are prefixed with % and work like a command typed in your terminal. You start with the name of the function and then pass some arguments, for example:

In [1]: %timeit range(1000)
255 ns ± 10.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

My favorite one is the %debug function. Imagine you run some code and it throws an exception. But given you weren't prepared for the exception, you didn't run it through a debugger. Now, to be able to debug it, you would usually have to go back, put some breakpoints and rerun the same code. Fortunately, if you are using IPython there is a better way! You can run %debug right after the exception happened and IPython will start an interactive debugger for that exception. It's called post-mortem debugging and I absolutely love it!

The second type of magic functions are cell magics and they work on a block of code, not on a single line. They are prefixed with %%. To close a block of code, when you are inside a cell magic function, hit Enter twice. Here is an example of timeit function working on a block of code:

In [2]: %%timeit elements = range(1000)
   ...: x = min(elements)
   ...: y = max(elements)
   ...:
   ...:
52.8 µs ± 4.37 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Both the line magic and the cell magic can be created by simply decorating a Python function. Another way is to write a class that inherits from the IPython.core.magic.Magics. I will cover this second method in a different article.

Creating line magic function

That's all the theory. Now, let's write our first magic function. We will start with a line magic and in the second part of this tutorial, we will make a cell magic.

What kind of magic function are we going to create? Well, let's make something useful. I'm from Poland and in Poland we are use Polish notation for writing down mathematical operations. So instead of writing 2 + 3, we write + 2 3. And instead of writing (5 − 6) * 7 we write * − 5 6 7^[1].

Let's write a simple Polish notation interpreter. It will take an expression in Polish notation as input, and output the correct answer. To keep this example short, I will limit it to only the basic arithmetic operations: +, -, *, and /.

Here is the code that interprets the Polish notation:

def interpret(tokens):
    token = tokens.popleft()
    if token == "+":
        return interpret(tokens) + interpret(tokens)
    elif token == "-":
        return interpret(tokens) - interpret(tokens)
    elif token == "*":
        return interpret(tokens) * interpret(tokens)
    elif token == "/":
        return interpret(tokens) / interpret(tokens)
    else:
        return int(token)

Next, we will create a %pn magic function that will use the above code to interpret Polish notation.

from collections import deque

from IPython.core.magic import register_line_magic


@register_line_magic
def pn(line):
    """Polish Notation interpreter
    
    Usage:
    >>> %pn + 2 2
    4
    """
    return interpret(deque(line.split()))

And that's it. The @register_line_magic decorator turns our pn function into a %pn magic function. The line parameter contains whatever is passed to the magic function. If we call it in the following way: %pn + 2 2, line will contain + 2 2.

To make sure that IPython loads our magic function on startup, copy all the code that we just wrote (you can find the whole file on GitHub) to a file inside IPython startup directory. You can read more about this directory in the IPython startup files post. In my case, I'm saving it in a file called:

~/.ipython/profile_default/startup/magic_functions.py

(name of the file doesn't matter, but the directory where you put it is important).

Ok, it's time to test it. Start IPython and let's do some Polish math:

In [1]: %pn + 2 2
Out[1]: 4

In [2]: %pn * - 5 6 7
Out[2]: -7 

In [3]: %pn * + 5 6 + 7 8
Out[3]: 165

Perfect, it works! Of course, it's quite rudimentary - it only supports 4 operators, it doesn't handle exceptions very well, and given that it's using recursion, it might fail for very long expressions. Also, the queue module and the interpret function will now be available in your IPython sessions, since whatever code you put in the magic_function.py file will be run on IPython startup.
But, you just wrote your first magic function! And it wasn't so difficult!

At this point, you are probably wondering - Why didn't we just write a standard Python function? That's a good question - in this case, we could simply run the following code:

In [1]: pn('+ 2 2')
Out[1]: 4

or even:

In [1]: interpret(deque('+ 2 2'.split()))
Out[1]: 4

As I said in the beginning, magic functions are usually helper functions. Their main advantage is that when someone sees functions with the % prefix, it's clear that it's a magic function from IPython, not a function defined somewhere in the code or a built-in. Also, there is no risk that their names collide with functions from Python modules.

Conclusion

I hope you enjoyed this short tutorial and if you have questions or if you have a cool magic function that you would like to share - drop me an email or ping me on Twitter!

Stay tuned for the next parts. We still need to cover the cell magic functions, line AND cell magic functions and Magic classes.

Image from: Pexels

It's a joke. We don't use Polish notation in Poland 😉 ↩︎

str vs. repr

2019-01-25T00:00:00Z

Every now and then, when I go back to writing Python code after a break, a question comes to mind:

What message should I put into the __str__ and the __repr__ functions?

When you search for the difference between them, you will find out that __str__ should be human readable and __repr__ should be unambiguous (as explained in this StackOverflow question). It's a great, detailed answer. But for some reason, it never really stuck with me. I'm not the smartest developer and sometimes to remember something, I need a very simple example. What I actually found helpful was written straight in the documentation of the repr() function:

For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval()

An excellent example of what it means, is the datetime module:

>>> import datetime
>>> now = datetime.datetime.now()
>>> str(now)
'2019-01-21 19:26:40.820153'
>>> repr(now)
'datetime.datetime(2019, 1, 21, 19, 26, 40, 820153)'

As you can see, the repr function returns a string that can be used to create an object with the same properties as now (not the same as now, but with the same properties). You can verify it by using the following code:

>>> timestamp = datetime.datetime(2019, 1, 21, 19, 26, 40, 820153)
>>> now == timestamp
True
# But!
>>> id(now) == id(timestamp)
False

So how can you use it in your own classes? For instance, if you are writing a class Car that has the attributes color and brand and is initialized in the following way:

red_volvo = Car(brand='volvo', color='red')

then this is what the __repr__ function for the car should return:

>>> repr(red_volvo)
"Car(brand='volvo', color='red')"

It's not always possible to write the __repr__ function that can recreate a given object, but simply keeping in mind those examples with datetime and Car has helped me to remember the difference between the __repr__ and __str__.

I found out about this trick in "Python Tricks" book, by Dan Bader. If you haven't heard of it, it's a great source of intermediate-level pieces of knowledge about Python. I'm in no way associated with Dan, but his book was one of the most enjoyable Python technical reads I've had in a long time.

IPython Startup Files

2019-01-04T00:00:00Z

In one of the companies where I worked, I was a part of a pretty small team of five developers. We had a support rota, so each week, one of us was responsible for handling tickets from users. Apart from requesting new features, users often asked for changes in the system that only admins could do - removing a wrongly submitted comment, replacing a file, editing metadata and so on. Some of those tasks could be done in the browser, but others had to be done by typing commands in IPython. Actually, most of those tasks could be done faster through lPython than in the browser - especially if you had done it before and you'd saved a recipe that you could just copy and paste.

At some point, I noticed that there were two or three commands that I was typing almost every time I started IPython. Those commands were importing functions from various modules. It wasn't a big problem to type them, especially since you can search in IPython history with ctrl+r or with arrows. But I wanted a way to automate it.

My first idea was to put those commands in a file and execute that file when starting IPython. As explained in the documentation, you can easily do this:

ipython -i my_commands.py

where my_commands.py contains all the commands that I want to run. That was not a bad solution as long as you remembered to start IPython including this file. And I was always forgetting to do that. So I made an alias in my .bashrc file that would always start IPython by running the script with my commands:

alias ipython='ipython -i ~/my_commands.py'

This worked pretty well for me until I found out about IPython startup files. IPython startup files are located in the following directory: ~/.ipython/profile_default/startup with a README file explaining that all files with .py or .ipy extension that you put here will be executed when IPython starts (to be more specific - each time IPython starts with this profile - in this case, the default profile). This was a great solution! First of all, you can keep all the startup files in the same place instead of trying to remember where you did put them. Second, thanks to the notion of the profiles, you can define a new profile just for debugging. This profile will import all the modules and functions that you need for debugging.

Importing modules is not the only way you can use the startup files. You can define some functions there or even create your own magic functions.

Here is a short video explaining how a startup file works in IPython:

Image from: Pexels

Sebastian Witowski

map() vs. List Comprehension

About the "Writing Faster Python" series #

Named function #

Conclusions #

Further reading #

Inlining Functions

Using temporary variables #

Conclusions #

Pathlib for Path Manipulations

About the "Writing Faster Python" series #

But is it faster? #

Joining paths #

Using an existing Path() object #

Starting from the home folder #

Is it a file? #

Get the current directory #

Find all the files matching a pattern #

Quickly write to a file #

Conclusions #

Further reading #

String Formatting

The old style of string formatting with the % operator #

Template strings #

The new style with str.format() #

f-strings (literal string interpolation) #

Which string formatting method is the fastest? #

Conclusions #

Further reading #

Compare to None

Dictionary Comprehension

About the "Writing Faster Python" series #

Dictionary comprehension vs. dict() vs. for loop #

Creating a dictionary from two iterables #

Conclusions #

dict() vs. {}

Looking under the hood with the dis module #

Is there any other difference? #

[] vs. list(), () vs. tuple, {'x', } vs. set(['x']) #

How to Benchmark (Python) Code

python -m timeit #

python -m timeit -s "setup code" #

python -m timeit -s "setup code" -n 10000 #

docker #

Python benchmarking libraries #

rich-bench #

pyperf #

hyperfine #

timeit is just fine...for me #

Beware of how you structure your code #

Conclusion #

Upgrade Your Python Version

Setup #

Slower scripts #

Benchmarks #

Results #

"Zero cost" exception handling #

Conclusions #

Python Versions Management With pyenv

pyenv #

Installation #

pyenv in action #

local and shell Python versions #

A quick troubleshooting tip #

asdf-vm #

25 IPython Tips for Your Next Advent of Code

1. Display the documentation #

2. Display the source code #

3. %edit magic function #

4. Reopen last file with "%edit -p" #

5. Wildcard search #

6. post-mortem debugging #

7. Start the debugger automatically #

8. Run shell commands #

9. Move around the filesystem with %cd #

10. %autoreload #

11. Change the verbosity of exceptions #

12. Rerun commands from the previous sessions #

13. Execute some code at startup #

14. Use different profiles #

About the "Writing Faster Python" series

Named function

Conclusions

Further reading

Using temporary variables

Conclusions

About the "Writing Faster Python" series

But is it faster?

Joining paths

Using an existing `Path()` object

Starting from the home folder

Is it a file?

Get the current directory

Find all the files matching a pattern

Quickly write to a file

Conclusions

Further reading

The old style of string formatting with the % operator

Template strings

The new style with str.format()

f-strings (literal string interpolation)

Which string formatting method is the fastest?

Conclusions

Further reading

About the "Writing Faster Python" series

Dictionary comprehension vs. `dict()` vs. `for` loop

Creating a dictionary from two iterables

Conclusions

Looking under the hood with the `dis` module

Is there any other difference?

[] vs. list(), () vs. tuple, {'x', } vs. set(['x'])

python -m timeit

python -m timeit -s "setup code"

python -m timeit -s "setup code" -n 10000

docker

Python benchmarking libraries

rich-bench

pyperf

hyperfine

timeit is just fine...for me

Beware of how you structure your code

Conclusion

Setup

Slower scripts

Benchmarks

Results

"Zero cost" exception handling

Conclusions

pyenv

Installation

pyenv in action

local and shell Python versions

A quick troubleshooting tip

asdf-vm

1. Display the documentation

2. Display the source code

3. %edit magic function

4. Reopen last file with "%edit -p"

5. Wildcard search

6. post-mortem debugging

7. Start the debugger automatically

8. Run shell commands

9. Move around the filesystem with %cd

10. %autoreload

11. Change the verbosity of exceptions

12. Rerun commands from the previous sessions

13. Execute some code at startup

14. Use different profiles

15. Output from the previous commands

16. Edit any function or module

17. Share your code

18. Use IPython as your debugger

19. Execute code written in another language

20. Store variables between sessions

21. Save session to a file

22. Clean up ">" symbols and fix indentation

23. List all the variables

24. Use asynchronous functions

25. IPython scripts

Conclusions