Turbine App Visualization is now available in the Meroxa dashboard.

Learn more

Announcing Conduit 0.3

By  Rimas Silkaitis

 27 Sep 2022

Conduit 0.3 is here! Conduit is a tool that helps developers move data within their infrastructure to the places they’re needed. 

Getting started is easy as downloading Conduit from the Releases page on GitHub and running:

$ ./conduit

What’s New

OpenCDC - Consistency in Payloads

One of the biggest pieces of work in this release is Conduit’s support for OpenCDC. A gripe that we hear about production data integration tools is that the formats for Change Data Capture (CDC) can be all over the place even between connectors within the same tool! The downstream impact is that developers then need to code toward specific connectors. OpenCDC provides high-level guarantees on the format that you can expect from any connector that has CDC support in Conduit.Code snippet

OpenCDC represents a breaking change in Conduit’s Connector SDK. This means that any connector that hasn’t been updated to work with 0.3 will only work with 0.2. We’ve updated the connector list in the Conduit repo to reflect which connectors are ready for OpenCDC. You can check out this blog post to learn more about OpenCDC.

Create Pipelines with a Pipeline Config File

In some production situations, you might not want to orchestrate pipelines via an API or a UI. If your data stores don’t change all that much, a static file might be the best way to configure a pipeline. With the release of the Pipeline Config File feature, you have the ability to use `yaml` to configure pipelines. The added benefit of this feature is that you can put the file in source control and have more measured changes to any of your pipelines.Code snippet

JavaScript Processors

Imagine a scenario where you need to drop personally identifiable information before any data reaches less sensitive downstream systems. The best way to do this would be to attach some code to the pipeline. In Conduit 0.3, it’s now possible to use Javascript to transform data. Javascript is the first language that we’re supporting but we plan to provide support for more languages over time. Don’t worry, Conduit does not have an external dependency on Node. Conduit uses goja to make this possible.

Processors can be injected after data comes from a source connector, during the pipeline itself, or before the data goes to a destination connector. The best way to build processors is to include them as part of your pipeline configuration file like so:Code snippet

Processors can also be created as part of an API call to Conduit. This is great in cases where you’re building pipelines programmatically as part of your internal processes or even your own product!

And So Much More

If you want to see the full list of what was included in this release, check out the Conduit Changelog and the documentation. This blog post only covers a fraction of what was included. In the coming weeks, we’ll be releasing more blog posts on topics like the performance benchmarks of Conduit 0.3 and connector middleware.

The Conduit team would love to hear about how you’re using Conduit in your setup. Please hit us up on Discord, GitHub Discussions, or Twitter!

         Conduit, OpenCDC

Rimas Silkaitis

Rimas Silkaitis

vp product @ meroxa. product, data, & dev tools. not necessarily in that order.