Writing applications against streaming or event-driven data is an incredible challenge. Developers are beholden to upstream schemas or have to write a considerable amount of plumbing with streaming systems before getting to the value-add work of their applications. In web development, developers are in control of their schemas. Any time a new feature is built that needs to persist some data, the developer writes a change to the schema and deploys it when they choose to. For streaming applications, the developers aren’t in control. They’re at the mercy of whatever upstream system or process generates the data.
Today we’re happy to announce two new features to the Meroxa platform that aim to make developing against streaming data easier. First, is the Apache Kafka Connector. The concepts and paradigms for streaming data started with Apache Kafka. The second feature is Feature Branch Deploys for streaming data applications. The ability to test a streaming application against staging and a copy of production data is critical. This gives developers the confidence that their changes, once merged to the `main` branch and deployed to production, will work as expected.
Apache Kafka Connector
Many streaming applications start with the core infrastructure of Apache Kafka. Its ability to allow developers to produce and consume data to any number of systems or streaming applications is what’s made it successful. Part of the challenge for any developer learning to build apps off of Apache Kafka is all the new streaming paradigms between delivery semantics and partitions, just to name a few. With support for Apache Kafka on Meroxa, as a source and a destination, it’s never been easier to focus on the business logic instead of all of the plumbing. Check out the feature launch blog post for more details. Support for producing and consuming from Apache Kafka is just the beginning as we work toward the goal of enabling developers to focus on value-add development instead of plumbing.
Feature Branch Deploys
Feature Branch Deploys is the first step on a path to enabling modern continuous delivery practices for streaming data applications. Writing unit and integration tests are important when building any application whether that’s for the web or streaming data applications. They let you know if what you’ve built meets your requirements and expectations. That level of testing is already possible when building Turbine streaming data apps. Nothing compares to taking what you’ve built and testing it against staging or production data. After all, data is what drives streaming applications.
Any time you have a branch in your Turbine application, you’ll be able to deploy that branch directly to Meroxa. Meroxa will do the work of sending the data you want to your application to consume, making sure that it doesn’t impact the production version of the application. Check out our write-up on Feature Branch Deploys.
Get Started with Meroxa
Both features are available today on the Meroxa platform. Get started by creating your own Turbine streaming data application and let us know what you’re building! We’d love to hear about it, so don't forget to share with us on Twitter or in our Discord community.
