How macOS Broke Python
Written by Barry Warsaw in technology on Tue 27 November 2018. Tags: python, macos,
Here's a common Python idiom that is broken on macOS 10.13 and beyond:
import os, requests
pid = os.fork()
if pid == 0:
# I am the child process.
requests.get('https://www.python.org')
os._exit(0)
else:
# I am the parent process.
do_something()
Now, it's important to stress that this particular code sample may execute just fine. It's an excerpt from some code at work that illustrates a deeper problem that you may or may not encounter in real-world applications. We've seen it crash reproducibly, resulting in core dumps of the Python process, with potentially disk filling dump files in /cores.
In this article, I hope to explain what I know about this problem, with links to information elsewhere on the 'net. Some of those resources include workarounds, but in my experiments, those are not completely reliable in eliminating the core dumps. I'll explain why I think that is.
It's important to stress that at the time of this article's publishing, I do not have a complete solution, and am not even sure one exists. I'll note further that this is not specifically a Python problem and in fact has been described within the Ruby community. It is endemic to common idioms around the use of fork() without exec*() in scripting languages, and is caused by changes in the Objective-C runtime in macOS 10.13 High Sierra and beyond. It can also be observed in certain "prefork" servers.
What is forking?
I won't go into much detail on this, since any POSIX programmer should be well acquainted with the fork(2) system call, and besides, there are tons of other good resources on the 'net that explain fork(). For our purposes here, it's enough to know that fork() is a relatively inexpensive way to make an exact copy of …