Building My First Conduit Connector: A Google Drive Adventure

During Champagne Week—the magical time at Meroxa where we get to work on anything we want—I decided to dive into something totally new: building a connector for Conduit. I’d never written a line of Go before. I’d never run Conduit locally. So, naturally, I decided to do both at once.

Here’s the story of how I built a Google Drive destination connector for Conduit from scratch—the things I learned, the hiccups I hit, and why it was totally worth it.

🥂 Champagne Week = Choose Your Own Adventure

Champagne Week is when we get to explore ideas outside the day-to-day roadmap and chase whatever sparks our curiosity. I’d been meaning to learn more about Conduit’s internals, and connectors felt like a natural place to start.

Google Drive stood out as a fun target. It’s widely used, has a solid API, and writing a destination connector meant I could figure out how to reliably push data into Drive.

It was something that seemed both useful and interesting. On the useful side, Google Drive is a tool almost every team interacts with in some capacity—being able to automatically write data to Drive from a pipeline opens up a lot of lightweight automation use cases. Think: piping logs or daily reports straight into a shared folder, exporting transformed data for non-technical stakeholders, or syncing snapshots of datasets for backup or audit purposes.

At the same time, it was interesting from a technical perspective. I got to dive into Google’s APIs, work with OAuth and service accounts, and figure out how to map abstract Conduit records into files that make sense in the context of Drive. The blend of dealing with a real-world API and building something from scratch with Go made it feel like a fun puzzle, and I came away with a much better understanding of both the language and the Conduit platform.

🛠️ Getting Conduit Running Locally

Step one: get Conduit building and running locally. This was my first time doing it, so I wasn’t totally sure what to expect. Luckily, the process was smoother than I’d feared.

Here’s what I used:

• Go (latest stable version)
• make (used heavily in Conduit’s dev flow)
• VS Code + Go plugin

I leaned heavily on the Conduit docs—which, thankfully, are really well written. While I leaning on the example connector template to get started, I also poked around other existing connectors to see how they were structured, which was super helpful for figuring out best practices.

📁 What the Connector Does

The connector I built is a Google Drive destination connector, which means it takes records coming through a Conduit pipeline and uploads them into Google Drive.

In this case:

• Each record gets written as a file in a specified Drive folder.
• The file name can be derived from the record’s metadata.
• The file content comes from the record payload.

func (d *Destination) Write(ctx context.Context, r []opencdc.Record) (int, error) {
	// Log the number of records
	sdk.Logger(ctx).Trace().Int("records", len(r)).Msg("Starting file uploads...")

	// Initialize a counter to track the number of successfully uploaded records
	successfulUploads := 0

	// Loop through each record and upload it as a separate file
	for _, record := range r {
		if record.Operation != opencdc.OperationCreate {
			// Skip records that are not of type Create
			sdk.Logger(ctx).Trace().Msgf("Skipping record with operation: %s", record.Operation)
			successfulUploads++
			continue
		}
		fileData := record.Payload.After.Bytes()

		// Create a bytes buffer to hold the record data
		fileBuffer := bytes.NewBuffer(fileData)

		// Prepare the file metadata (include the folder ID in the Parents field)
		fileMetadata := &drive.File{
			Name:    fmt.Sprintf("%s.txt", record.Key.Bytes()), // Set the file name
			Parents: []string{d.config.DriveFolderID},          // Set the shared folder ID
		}

		// Upload the file directly from the bytes buffer
		uploadedFile, err := d.service.Files.Create(fileMetadata).Media(fileBuffer).Do()
		if err != nil {
			return successfulUploads, fmt.Errorf("unable to upload file: %w", err)
		}

		// Log the uploaded file's ID
		sdk.Logger(ctx).Trace().Msgf("File uploaded successfully! File ID: %s\n", uploadedFile.Id)

		// Increment the successful uploads counter
		successfulUploads++
	}

	return successfulUploads, nil
}

I kept it simple for a v1, focusing on JSON files and plain text, but there’s a lot of room to expand in the future. Right now, the connector only supports creating new files—each record is uploaded as a new file in the target Drive folder. It doesn’t handle updates or deletes yet, so there's no logic for overwriting existing files or removing them based on record changes. Down the line, it could support features like:

• Updating existing files
• Deleting files when a record indicates a delete event
• Supporting different MIME types and file formats (e.g. CSV, PDFs, images)
• Custom folder routing based on record content

For now, though, it’s a clean foundation to build on—simple, reliable, and focused.

🔧 That One Gotcha: make generate

One thing that tripped me up early: after changing any of the connector’s config variables (like folder_id, credentials, etc.), I needed to run:

make generate

Without this, Conduit wouldn’t pick up the new config fields, and I’d get confusing runtime errors or empty values. Once I realized this step was required, everything started working a lot more smoothly.

Pro tip: if something feels weird after changing config, run make generate. It'll probably fix it.

👩‍💻 First Time Writing Go

This was also my first time writing Go, and honestly? I loved it. The language is opinionated in a way that helps you write clean, readable code. Once I wrapped my head around struct tags, interfaces, and error handling, things fell into place quickly.

Compared to some of the dynamic languages I’m used to, Go felt rigid at first—but I came to appreciate that structure, especially for something like a connector where stability matters.

🔌 Connector Highlights

Here are a few things I implemented in the destination connector:

• Google Drive API integration using a service account
• Config validation, making sure required fields are present
• Simple file writing logic based on incoming record content

I tested everything by running Conduit locally, setting up a pipeline, and watching files appear in my Drive folder. Extremely satisfying.

💡 What I Learned

• Writing connectors isn’t as scary as it sounds. Conduit’s docs and examples make it approachable.
• Go is a great fit for this kind of work. The performance, structure, and ecosystem made sense quickly.
• Always make generate after a config change. Seriously. Just do it.

🎯 What’s Next?

This connector is a solid starting point, but there’s plenty of room to grow:

• Support for different file formats (CSV, binary, etc.)
• More flexible folder paths and naming schemes
• Handling large payloads and batching intelligently

And now that I’ve gotten my feet wet, I’m tempted to try building a source connector next.

🥳 Wrapping Up

Shipping my first connector—especially during a self-directed week like this—felt awesome. I got to learn Go, contribute something real to the Conduit ecosystem, and explore how connectors are built from the ground up.

If you’re even a little curious about writing your own connector, my advice is: go for it. It’s more doable than you think, and you’ll learn a ton along the way.