Release 0.13 of Conduit brings to you our new Conduit CLI, designed to make configuring, managing, and running Conduit smoother than ever. Built with our open-source Ecdysis library, this CLI is a game-changer for users looking for efficiency and ease of use.
Why the Conduit CLI Matters
Before this update, managing Conduit pipelines and connectors often required a mix of API calls, configuration files, and going through documentation. The new Conduit CLI changes the game by offering a centralized command-line that transforms these tasks into a simple, accessible tool.
With Conduit CLI, you can now:
- Manage connectors, connector plugins, processors, processor plugins, and pipelines effortlessly
- List and describe Conduit components directly from the terminal
- Configure and Run Conduit components without leaving the CLI
- Easily initialize Conduit and get started
Built on Ecdysis: A Flexible Library for CLI Tools
The Conduit CLI is powered by Ecdysis, an open-source Go library designed to simplify CLI tool development. Ecdysis is built around spf13/cobra, acting as a wrapper to enhance its capabilities.
Ecdysis provides a structured approach to building command-line applications, with many features that include the following, among others:
- A robust command structure for defining and organizing commands efficiently.
- Automatic configuration parsing, reducing the need for manual setup.
- Flexible flag parsing, making it easy to customize command behavior.
By leveraging Ecdysis, Conduit CLI offers a consistent and extendable experience, making it easier for developers to interact with Conduit’s components using a well-architected CLI framework.
Getting Started with Conduit CLI
To check all the available commands, simply run:
$ conduit --helpThis will output a list of commands, as of the moment of writing, these include:
Available Commands:
config Shows the configuration to be used when running Conduit.
connector-plugins Manage Connector Plugins.
connectors Manage Conduit Connectors.
pipelines Initialize and manage pipelines.
processors Manage Processors.
run Run Conduit.
version Show the current version of Conduit.Each command is designed to give you control and observability over your data streaming pipelines. Let’s take a closer look at some of these key functionalities.
Initializing Conduit
The command conduit init creates the directories where you add your pipeline configuration files, connector binaries, and processor binaries. It also creates the file conduit.yaml that contains all the configuration parameters that Conduit supports.
$ conduit init
Created directory: processors
Created directory: connectors
Created directory: pipelines
Configuration file written to conduit.yaml
Conduit has been initialized!
To quickly create an example pipeline, run 'conduit pipelines init'.
To see how you can customize your first pipeline, run 'conduit pipelines init --help'.You can also use the init command to initialize a pipeline configuration file with your choice of source and destination, example:
$ conduit pipelines init file-to-pg --source file --destination postgresThis will initialize a pipeline configuration file, with all of the parameters for the source and destination connectors, by default the created file will be under the folder ./pipelines , and in this case it would look like:
version: "2.2"
pipelines:
- id: example-pipeline
status: running
name: "file-to-pg"
connectors:
- id: example-source
type: source
plugin: "file"
settings:
# Path is the file path used by the connector to read/write records.
# Type: string
# Required
path: ""
..
.. # more params
..
..
- id: example-destination
type: destination
plugin: "postgres"
settings:
# Key represents the column name for the key used to identify and
# update existing rows.
# Type: string
# Optional
key: ""
..
..
.. # more params
..
..
# Table is used as the target table into which records are inserted.
# Type: string
# Optional
table: '{{ index .Metadata "opencdc.collection" }}'
# URL is the connection string for the Postgres database.
# Type: string
# Required
url: ""
Managing Connector Plugins
One of the new additions to the Conduit CLI is the ability to list and describe available connector plugins.
Listing Connector Plugins
To list all available connector plugins, run:
$ conduit connector-plugins listThis command displays a table of all the built-in and standalone connector plugins available to Conduit:
+-------------------------------------+----------------------------------------+
| NAME | SUMMARY |
+-------------------------------------+----------------------------------------+
| builtin:file@v0.9.0 | A file source and destination plugin. |
| builtin:kafka@v0.11.1 | A Kafka source and destination plugin. |
| standalone:dynamodb@f9aeeee-dirty | A DynamoDB source plugin for Conduit. |
| standalone:grpc-client@v0.1.0 | A gRPC Source & Destination Client. |
+-------------------------------------+----------------------------------------+Describing a Specific Plugin
To get more details about a specific plugin, use the describe command followed by the plugin name. For example, to learn more about the PostgreSQL plugin:
$ conduit connector-plugins describe builtin:postgres@v0.10.1This provides a detailed breakdown of the plugin, including the author, version, description, summary, and the parameters for both the source and destination. Here’s an example of what you’ll see:
Name: builtin:postgres@v0.10.1
Summary: A PostgreSQL source and destination plugin for Conduit.
Author: Meroxa, Inc.
Version: v0.10.1
Source Parameters:
+--------+--------+----------------------------------+---------+-------------+
| NAME | TYPE | DESCRIPTION | DEFAULT | VALIDATIONS |
+--------+--------+----------------------------------+---------+-------------+
| url | string | Connection string for database. | "" | [required] |
| tables | string | List of tables to listen to. | "" | [required] |
+--------+--------+----------------------------------+---------+-------------+
Destination Parameters:
+------+--------+------------------+---------------------------------+----------+
| NAME | TYPE | DESCRIPTION | DEFAULT |VALIDATION|
+------+--------+------------------+---------------------------------+----------+
| url | string | Connection string| "" |[required]|
| table| string | Target table |{{.Metadata[opencdc.collection]}}| |
+------+--------+------------------+---------------------------------+----------+Running Conduit
To run Conduit directly from the CLI, simply run:
$ conduit runThis starts Conduit using the specified configurations.
Note: Most CLI commands require Conduit to be running for them to work properly, since they need access to the running components and their details.
Managing Pipelines, Connectors, and Processors
Beyond managing plugins, the Conduit CLI also provides access to pipelines, connectors, and processors. These follow a similar command structure:
Pipelines
- List all pipelines:
conduit pipelines list - Describe a pipeline:
conduit pipelines describe <pipeline-id>
Connectors
- List all connectors:
conduit connectors list - Describe a connector:
conduit connectors describe <connector-id>
Processors
- List all processors:
conduit processors list - Describe a processor:
conduit processors describe <processor-id>
These commands give you more observability over your conduit pipelines and their components.
Why You Should Try Conduit CLI
The new Conduit CLI is an important addition for developers and users working with Conduit. By offering a fast, intuitive, and simple way to manage Conduit components, the CLI will significantly improve productivity and reduce complexity.
Key Benefits:
✅ Easier management of Conduit components via the command line
✅ Clear visibility into available plugins and configurations
✅ Effortless setup with the initialization commands
✅ Faster debugging with detailed descriptions of connectors and pipelines
Get Started Today
The Conduit CLI is available now! If you haven’t already, install Conduit and give the CLI a try. For more details, check out our official documentation.
🚀 Run conduit --help and start exploring today!
As always, we welcome your feedback and contributions to help shape the future of Conduit. Get involved by starting a GitHub Discussion, opening an issue, or joining our Discord server and saying hello to the team behind Conduit! Follow us on Twitter, LinkedIn, and YouTube for more insights and updates!
