We are thrilled to announce the release of Conduit Operator v0.0.2, designed to simplify the management and orchestration of Conduit instances within Kubernetes.

What Is the Conduit Operator?

The Conduit Operator extends the Kubernetes API, allowing users to manage Conduit instances as custom resources. These resources define how each Conduit pipeline is provisioned and managed throughout its lifecycle, giving you full control over your data flow while leveraging Kubernetes-native features like scaling, monitoring, and logging.

Conduit pipelines can be declared using YAML configuration, much like how any other Kubernetes resources are configured and deployed. This flexibility allows you to integrate your data streaming processes into existing DevOps workflows and infrastructure management tools with ease.

A Glimpse into Conduit Custom Resources

Conduit pipelines are represented as Kubernetes custom resources, where each pipeline runs as its own distinct Conduit instance. Below is a basic example of how a Conduit pipeline is defined:

apiVersion: operator.conduit.io/v1alpha
kind: Conduit
metadata:
  name: conduit-generator
spec:
  running: true
  name: generator.log
  description: generator pipeline
  connectors:
    - name: source-connector
      type: source
      plugin: builtin:generator
      settings:
        - name: format.type
          value: structured
        - name: format.options.id
          value: "int"
        - name: format.options.name
          value: "string"
        - name: format.options.company
          value: "string"
        - name: format.options.trial
          value: "bool"
        - name: recordCount
          value: "3"
    - name: destination-connector
      type: destination
      plugin: builtin:log

This configuration provides a declarative way to manage data pipelines, reducing the manual overhead typically required to build and manage streaming architectures.

Streamlining Connector Management

Using standalone connector with the Conduit Operator is simplified by allowing them to be hot loaded through GitHub repositories rather than included into the instance image.

# github.com/conduitio/conduit-connector-generator will be build and loaded
# by the operator.
- name: source-connector
  type: source
  plugin: conduitio/conduit-connector-generator
  pluginVersion: v0.8.0
  settings:
    ...

The operator automatically provisions and manages the necessary resources to run the connectors. These connectors, sourced from organizations like conduitio, meroxa, and conduitio-labs, provide out-of-the-box integrations with popular systems.

Schema Support for Enhanced Data Handling

As of Conduit v0.11.0, the platform supports schema registries, which allows connectors to encode and decode data using predefined schemas. This enables more robust data management and ensures compatibility between different data systems.

An example configuration for utilizing a schema registry looks like this:

apiVersion: operator.conduit.io/v1alpha
kind: Conduit
metadata:
  name: conduit-generator-schema-registry
spec:
  schemaRegistry:
    url: http://apicurio:8080/apis/ccompat/v7
    basicAuthUser:
      - value: <schemaUser>
    basicAuthPassword:
      - secretRef:
        key: schema-registry-password
        name: schema-registry-secret

This level of flexibility is essential for businesses dealing with large-scale data integrations, as it allows multiple Conduit instances to share a schema registry across different environments and scale pipelines independently.

Deploying Conduit Operator

Deployment of the Conduit Operator can be done via Helm, a popular Kubernetes package manager. By using Helm charts, you can easily manage deployments, scaling, and updates of Conduit instances within your Kubernetes clusters.

To deploy the operator, you can simply run:

helm repo add conduit https://conduitio.github.io/conduit-operator
helm install conduit-operator \
    conduit/conduit-operator --create-namespace -n conduit-operator

Monitoring and Scaling with Kubernetes

One of the key advantages of the Conduit Operator is its integration with Kubernetes-native features. For example, you can add annotations to your Conduit instances to automatically scrape metrics using Prometheus:

This is achieved by customizing the Helm value file when deploying the operator. Future work will allow for these annotations to be placed directly on the Conduit resource.

# Create values.yaml using these settings
controller:
  conduitMetadata:
    podAnnotations:
      prometheus.io/scrape: true
      prometheus.io/path: /metrics
      prometheus.io/port: 8080
      
# Install or upgrade the operator via helm
helm install conduit-operator \
    conduit/conduit-operator --create-namespace -n conduit-operator \
    -f values.yaml

This seamless integration enables robust monitoring and scaling options, ensuring your data pipelines are optimized for performance and reliability.

Why Use Conduit Platform?

While the Conduit Operator offers a robust solution for managing data pipelines within Kubernetes, the Conduit Platform takes this further by providing a low-code experience and additional enterprise features. With the Conduit Platform, you can easily build, monitor, and scale complex data pipelines with minimal manual effort.

Key advantages of using the Conduit Platform include:

• Low-Code Interface: Quickly configure and manage pipelines without extensive coding.
• Enterprise Features: Enhanced security, monitoring, and scaling options tailored for large-scale enterprise needs.
• Streamlined Workflows: Easily connect disparate data sources and sinks, optimizing data flow across your infrastructure.

Whether you're looking to deploy individual instances with Conduit Operator or scale enterprise-wide with the Conduit Platform, Meroxa provides the tools and flexibility to manage your data pipelines efficiently.

Conclusion

The Conduit Operator simplifies data pipeline management in Kubernetes environments, enabling you to easily manage data streams. For businesses looking to scale, integrate complex data systems, and optimize their operations, the Conduit Platform provides a powerful low-code solution that expands on the capabilities of the Conduit Operator.

Get started with Conduit Operator on GitHub and take your data pipeline management to the next level with the Conduit Platform for a low-code, enterprise-ready experience. Also check out our documentation.

Looking for managed platform solutions? Check out our Conduit Platform by requesting a demo. Let's build the future of data integration together!