Debugging Hung Python Processes with GDB

Debugging Hung Python Processes with GDB

FOSDEM 2017

When things go wrong in production, it can be necessary to troubleshoot problems where they occur, instead of in a development environment. In those situations having a working knowledge of GDB, GDB Python Extensions, and strace is very helpful. You will see some simple techniques to get insight into those situations. This talk outlines several techniques for connecting to an already running, "stuck", or deadlocked Python process using GDB for debugging.

During the talk, we will:

• inspect the current state of threads with
• use and demo the GDB macros for Python
• inspect a locally running process and a core dump collected from a remote machine
• use strace to gather system call information about a process
• discuss the SIGTRAP handler as a proactive way to make rpdb available in production.

I have had to debug several hard-to-find bugs that were very infrequent deadlocks using Python. Furthermore it was happening on remote machines I could not have network access to. This technique was invaluable in those situations.

Everything is in the abstract

Speakers: Brian Bouterse