Conduit 0.7

Welcome to another release of Conduit! We’ve always thought of Conduit as a Kafka Connect replacement that could do so much more, like move data and run elaborate pipelines. In this release, we get closer to that original goal of a Kafka Connect replacement with our biggest feature, Native Schema Registry support.

Native Schema Registry Support

A schema registry is a great tool to store metadata about the information flowing through pipelines. The metadata can contain information about what fields are required, which fields are optional, and enforce data types. Also, the data in a pipeline can be encoded in a more space efficient format if you have a schema. As a developer, this allows you to be more confident that what gets sent into the pipeline is what you’re expecting.

We’re excited to announce native schema registry support in Conduit. Interacting with a schema registry within a Conduit pipeline is done via one of four built-in processors:

-Decode with Schema Key

-Decode with Schema Payload

-Encode with Schema Key

-Encode with Schema Payload

To add the ability to your pipeline, all you need to do is call one of the aforementioned methods in your pipeline within the processors section:

processors:
  id:   example
  type: decodewithschemakey
  settings:
    url:                 "http://localhost:8085"
    auth.basic.username: "user"
    auth.basic.password: "pass"
    tls.ca.cert:         "/path/to/ca/cert"
    tls.client.cert:     "/path/to/client/cert"
    tls.client.key:      "/path/to/client/key"

Currently, the built-in schema registry processors only supports Avro but we’re looking to include more formats in future releases, like protobuf and JSON schema.

gRPC Connector

While not necessarily part of Conduit itself, we’re excited to announce gRPC Server and Client Conduit connectors. This is super interesting because it now allows Conduit to be used in distributed environments. For example, let’s say you need to aggregate in one place and forward it to another site. The image demonstrates that you can have one Conduit running on Remote Site A and using the Conduit gRPC Server and Client Connectors, you can forward the data to Remote Site B. This is functionality we use internally to move data between regions within AWS. There are still a number of features to be added to these connectors, but it’s a start at enabling these distributed scenarios.

We’d love your feedback too!

As always, we’d love to get your feedback! If you want to see the full list of what is included in this release, check out the Conduit Changelog and the documentation. Also, feel free to join us on Discord or Twitter.