conferences | speakers | series

Scaling your logging infrastructure

home

Scaling your logging infrastructure
FOSDEM 2017

Event logging is important not only for IT security and operations, but also for business decisions. The syslog-ng application is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and finally it stores them or routes them for further analysis. From this session you will learn (using examples from syslog-ng) why and how to parse important information from incoming messages, and how to route logs, feeding downstream systems using arbitrary formats. We will also discuss how the client – relay – server architecture can solve scalability problems. Also, I will present some of the recently introduced “Big Data” destinations of syslog-ng, which can help to scale your infrastructure even further.

Outline:

  • what is syslog-ng
  • log collection
  • introduction to log processing
  • filtering
  • Big Data destinations
  • log formats - the importance of name-value pairs
  • message parsing (both unstructured and some structured message formats)
  • formatting (templates, anonymization)
  • log routing (optimizing analytics infrastructure)
  • client (collection) - relay (processing) - server (storing, routing)
  • summary

Speakers: Peter Czanik