Cloudflare, a leading internet security, CDN, and DNS provider, faced several challenges with their growing business needs. In this blog post, we will discuss how Apache Kafka emerged as an effective solution for various issues and how the team formed a Connector Framework to streamline the data flow.
Cloudflare’s Operational Challenges
As business requirements expanded, ensuring the operation of both public and private clouds and managing the interconnection between teams became a daunting task for Cloudflare. Matthew Boyle, who leads the team, recognized that implementing the message bus pattern would serve to systematize and harmonize operations.
Choosing Apache Kafka
After evaluating various options, Apache Kafka was identified as an efficient implementation of the message bus pattern. Apache Kafka, an open-source stream-processing software, facilitates handling of real-time data and works particularly well for big data and transactional applications. It offers high-throughput capabilities and is specifically designed to handle real-time data feeds.
Building a Connector Framework
With the increasing adoption of Apache Kafka by various teams across Cloudflare, the need to develop a Connector Framework became evident. Consequently, a universal Connector Framework was designed to simplify the streaming of data between Apache Kafka and other systems while transforming the messages in the process. This facilitated easier integration and communication across different teams.
The Role of JSON and Protobuf
JSON, a widely accepted data interchange format, and Protobuf, a Google-developed language-neutral, platform-neutral mechanism for serializing structured data, have played significant roles in enhancing the performance and interoperability of Apache Kafka at Cloudflare.
Key Lessons Learned
Andrea Medda, a notable figure at Cloudflare, distilled some valuable lessons from their experience with Apache Kafka. They included:
- The importance of balancing between highly configurable and simple standardized methods when providing developer tooling for Apache Kafka.
- Selecting a straightforward and strict 1:1 contract interface to ensure maximum visibility into the workings of topics and their usage.
- Investing in metrics on development tooling to identify problems easily and promptly.
- Prioritizing clear, accessible documentation to facilitate consistent adoption and use of Apache Kafka among application developers.
Gaia: A New Internal Product
Matthew Boyle introduced a new internal product, Gaia, that allows one-click creation of services based on Cloudflare’s best practices. Gaia is expected to further streamline the management of services and accelerate development efforts.
About the Author
This blog post is authored by Nsikan Essien, an Engineering Manager at Field Energy best known for his interest in cloud architectures, platform services, and effective team management. Nsikan is based in London.
Tags: #Cloudflare #ApacheKafka #ConnectorFramework #Gaia
Acknowledgement: This blog post is based on the experiences and insights shared by Andrea Medda and Matthew Boyle at Cloudflare.