It's 2022 and we still use logs to debug production issues?
All the unit tests in the world, the largest QA team still can’t stop bugs from slithering into production. With a distributed microservice architecture debugging becomes much harder. Especially across language & machine boundaries. APMs/Logs have limits. There’s a new generation of tools in town…
Production bugs are the WORST bugs. They got through unit tests, integration tests, QA and staging… They are the spores of software engineering. Yet the only tools most of us use to attack that vermin is quaint little log files and APMs. We cross our fingers and put on the Sherlock Holmes hat hoping that maybe that bug has somehow made it into the log… When it isn’t there our only remedy is guesswork of more logging (which bogs performance for everyone and makes the logs damn near unreadable). But we have no choice other than crossing our fingers and going through CI/CD again.
This is 2021. There are better ways. With modern debugging tools we can follow a specific process as it goes through several different microservices and “step into” as if we were using a local debugger without interrupting the server flow. Magic is possible.
Speakers: Shai Almog