One of the innovations in InnoDB was the change buffer (originally, insert buffer), which aims to convert random I/O to more sequential I/O, by buffering certain changes to secondary index B-tree leaf pages.
Due to its design and nature, any bugs related to the change buffer are extremely hard to reproduce. The change buffer is also becoming irrelevant, as the difference between random and sequential I/O is disappearing along with rotational storage (HDDs).
Thanks to the rr debugger and some improvements to InnoDB data structures, we have been able to reproduce and fix several tricky bugs related to the InnoDB change buffer.
We shortly explain what the InnoDB change buffer is and how it is supposed to work.
We describe the challenges of testing the change buffer and some bug scenarios at a high level.
Focusing on one example, we show how instead of debugging a core dump and guessing what might have led to the problem, we can use rr replay to analyze a deterministic execution trace from rr record leading to the failure. We can set breakpoints and data watchpoints and examine the state of the traced process at any point of time of execution. This even works across process boundaries, for crash recovery bugs.
Speakers: Marko Mäkelä