Abhishek Lekshmanan

RGW (Rados Gateway) is the HTTP REST frontend to Ceph, exposing a S3 and Swift API. From the Kraken release of Ceph, RGW introduced the concept of sync modules which allows for forwarding data and metadata to an external tier. This allows for interesting analysis of metadata or archival/backup solutions without the need to support these in Ceph itself. We'll take a brief look into the design of sync modules, peer into the crystal ball for what the future holds and also cover the currently available ElasticSearch and Cloud Sync modules, which allows data to be exported to external clouds supporting a S3 like API including Amazon S3.

Ceph is a distributed storage platform that is a contender to become the future of software defined storage, providing unified access to block, object and file interfaces. Rados Gateway (abbreviated RGW from here on) is the Object storage component of Ceph, exposing an all familiar S3 and Swift APIs for object storage. Since the Kraken release, RGW introduced the concept of sync modules which allows for forwarding data and metadata to an external tier. This allows for interesting analysis of metadata or archival/backup solutions without the need to support these in Ceph itself. We'll look into the basic design of sync modules, pointers to look for should you ever need to write one, cover the existing ElasticSearch & Cloud sync modules, and see what the future holds for RGW Sync modules.

Ceph is a distributed storage platform that is a contender to become the future of software defined storage, providing unified access to block, object and file interfaces. However like any complex systems there are various subsystems that may fail and analyzing logs is generally the first line of action.

This is where the ELK stack comes in; to search, analyze and process logs and metadata. From the Kraken release of Ceph, Ceph's Object Storage (Radosgw/RGW) has integration for metadata search using ElasticSearch making it very easy to get much needed insights into how the object storage is being used for operators and users alike.

Ceph is a distributed storage platform that is a contender to become the future of software defined storage, providing unified access to block, object and file interfaces. However like any complex systems there are various subsystems that may fail and analyzing logs is generally the first line of action.

This is where the ELK (ElasticSearch LogStash & Kibana, often referred to as "ELK stack" or "elastic stack") stack comes in; to search, analyze and process logs and metadata. From the Kraken release of Ceph, Ceph's Object Storage (Radosgw/RGW) has integration for metadata search using ElasticSearch making it very easy to get much needed insights into how the object storage is being used for operators and users alike.

We will cover topics such as: * Current status of ELK and Ceph * Ceph logging and cluster log parsing with Logstash * Future of the ELK for Analyzing and Alerting for Ceph * RGW Metadata export to Elasticsearch - RGW Metadata Search * A few interesting elasticsearch queries on object storage