Managing real-time data pipelines across hundreds of different systems requires consistent, accurate, and up-to-date documentation. With Conduit 0.13, we’ve automated connector documentation using connector.yaml, ensuring seamless synchronization between code and documentation.
The Challenge: Keeping Connector Documentation in Sync
Conduit supports reading and writing data to hundreds of systems. As the number of connectors grows, maintaining consistent documentation becomes increasingly difficult. Traditionally, connector configurations were documented in README files within repositories, requiring manual updates whenever a parameter changed. Over time, this led to outdated information and increased developer friction.
Goals
1: Connector configuration is documented and always up-to-date
A connector’s configuration is usually documented in the README file in the connector’s repository. As the configurations change in code, it’s easy to forget about updating the README file, especially if changes are not too big (like changing a parameter’s default value, the description, for example.). We also need this process to be automated.
2: A central place with all connector information
Having a central place with all the connector information makes it easier to explore Conduit, and find the needed components for a pipeline that needs to be built and configured. This place is our website, where we already have a connectors list, and where we will be adding dedicated documentation pages for each connector.
3: Easy to use for developers
All the documentation needs to be in sync with the configuration code written by a connector's developer. Every connector has a description that can become quite lengthy and, in our experience, is very cumbersome to write in the code. Plus, there are no formatting options. Hence, our goal was to give developers an easy way to sync the documentation with code and easily describe what a connector is doing.
The solution
The source of truth for a connector’s configuration is in the code, in the configuration structs. That means that the process that updates a connector’s README file and our website needs to read the configuration code (eventually). However, the code is not enough. As mentioned above, connector descriptions are best placed outside the connector code.
That led us to a solution where a connector’s specification (name, description, configuration parameters, etc.) are written to a file that can easily be read by other tools, i.e. uses a widely used file format. That’s how connector.yaml was born.
What is connector.yaml?
connector.yaml is a file that contains information about a connector and its parameter validations. It’s central to all of our tooling that ensures the documentation is always up-to-date and can be collected into a single place.
connectors.yaml lives in the root of a connector's repository. The following is an example of the file connector'sconnector.yaml:
version: "1.0"
specification:
name: file
summary: A file source and destination plugin for Conduit.
description: |-
The file source allows you to listen to a local file and
detect any changes happening to it. Each change will create a new record. The
destination allows you to write record payloads to a destination file, each new record payload is appended to the file in a new line.
version: v0.10.0
author: Meroxa, Inc.
source:
parameters:
- name: path
description: Path is the file path used by the connector to read/write records.
type: string
default: ""
validations:
- type: required
value: ""
# other parameters A connector developer can then simply run conn-sdk-cli readmegen (as explained here), which will synchronize the README file with the configuration structs. Our documentation website uses the connector.yaml file to build a dedicated documentation page for a connector.
How is a connector.yaml populated?
The first part of a connector.yaml (name, summary, description, version, author) is filled out manually by the connector developer. connector.yaml is used in Markdown files (in the connector’s README and on our website), so you can use Markdown code here!
Our conn-sdk-cli tool updates the configuration parameters in connector.yaml automatically, as part of running go generate. Detailed instructions on how to do that can be found here.
Next steps
You’ll find more information about how to write a Conduit connector here. If you’d like to take a look at some real-world examples, feel free to explore our existing connectors. ⚡
Try conn-sdk-cli readmegen now and streamline your connector documentation!
💬 Join the Conduit community! Discuss with fellow developers on Discord or contribute via GitHub Discussions. Follow us on Twitter, LinkedIn, and YouTube for more insights and updates!
