A proposal to support AWS SQS API natively in Ceph RGW, for advanced Serverless computing use cases.
Bucket notification has become widespread in many applications including AI/ML and Edge Computing. We have demonstrated that Ceph RGW bucket notifications trigger Serverless functions including Knative and AWS Lambda. Such workflows can be characterized as event push and event driven.
We’ve identified yet another workflow that allows Serverless computing frameworks to be more preemptive in auto scaling. The workflow is based on the event queue polling model, the opposite of event push, that allows Serverless computing frameworks, such as KEDA, to preemptively scale the functions based on the queue length. Such workflow is more advantageous and lightweighted in dealing with highly bursty event traffic and reducing Serverless function cold start overhead.
In order to support this workflow, we propose a new set of message queue APIs, modeled after AWS SQS, for Ceph RGW. In this talk, we’ll present the overview and planning of this technology to the community.
Speakers: Yuval Lifshitz Huamin Chen