Conduit 0.3 is here! Conduit is a tool that helps developers move data within their infrastructure to the places they’re needed.
Getting started is easy as downloading Conduit from the Releases page on GitHub and running:
One of the biggest pieces of work in this release is Conduit’s support for OpenCDC. A gripe that we hear about production data integration tools is that the formats for Change Data Capture (CDC) can be all over the place even between connectors within the same tool! The downstream impact is that developers then need to code toward specific connectors. OpenCDC provides high-level guarantees on the format that you can expect from any connector that has CDC support in Conduit.
OpenCDC represents a breaking change in Conduit’s Connector SDK. This means that any connector that hasn’t been updated to work with 0.3 will only work with 0.2. We’ve updated the connector list in the Conduit repo to reflect which connectors are ready for OpenCDC. You can check out this blog post to learn more about OpenCDC.
In some production situations, you might not want to orchestrate pipelines via an API or a UI. If your data stores don’t change all that much, a static file might be the best way to configure a pipeline. With the release of the Pipeline Config File feature, you have the ability to use `yaml` to configure pipelines. The added benefit of this feature is that you can put the file in source control and have more measured changes to any of your pipelines.
Processors can be injected after data comes from a source connector, during the pipeline itself, or before the data goes to a destination connector. The best way to build processors is to include them as part of your pipeline configuration file like so:
Processors can also be created as part of an API call to Conduit. This is great in cases where you’re building pipelines programmatically as part of your internal processes or even your own product!
If you want to see the full list of what was included in this release, check out the Conduit Changelog and the documentation. This blog post only covers a fraction of what was included. In the coming weeks, we’ll be releasing more blog posts on topics like the performance benchmarks of Conduit 0.3 and connector middleware.
The Conduit team would love to hear about how you’re using Conduit in your setup. Please hit us up on Discord, GitHub Discussions, or Twitter!