Real-Time Pipelines as Code with the Meroxa Terraform Provider

Making production-ready pipelines still requires a significant amount of time and effort. With the Meroxa CLI and the Meroxa Dashboard, your pipelines are streaming, real-time, and up and running in minutes, not months. Today, we’re adding a new way for you to build pipelines with versioning, speed, and consistency.

Introducing the Meroxa Terraform Provider. 🎉

The provider allows you to:

• Provision, modify and destroy various objects on the Meroxa platform as code.
• Easily share pipelines with your team.
• Manage pipelines next to infrastructure managed with Terraform.

If you’re new to Terraform, it is an open-source infrastructure as code software tool that provides a workflow and tooling to manage cloud infrastructure. Using the Terraform Provider, you can add your data pipeline resources to the list of items that Terraform can manage. For more information, check out the Terraform Getting Started Guide.

Getting Started

To get started with the Meroxa Terraform Provider, require it within yourTerraform File:

terraform {
  required_providers {
    meroxa = {
      version = "1.0"
      source = "meroxa.io/meroxa/meroxa"
    }
  }
}

Now, you can define your Meroxa resources within this Terraform project.

For example, here is an example pipeline that can assist with migration from PostgreSQL to Mongo. It sets up a pipeline keep both databases in sync in real-time:

# Require Provider 
terraform {
  required_providers {
    meroxa = {
      version = "0.1"
      source = "meroxa.io/meroxa/meroxa"
    }
  }
}

# Configure Provider
provider "meroxa" {
  access_token = var.access_token # optionally use MEROXA_ACCESS_TOKEN env var
}

# Define Pipeline
resource "meroxa_pipeline" "pipeline" {
  name = "sync-postgres-mongo"
}

# Configure Postgres Resource
resource "meroxa_resource" "postgres" {
  name = "my-postgres"
  type = "postgres"
  url = "POSTGRES_CONNECTION_URL"
}

# Configure MongoDB Resource
resource "meroxa_resource" "mongo" {
  name = "my-mongo"
  type = "mongodb"
  url = "MONGO_CONNECTION_URL"
}

# The PostgreSQL connector will capture CDC events for 
# every insert, update and delete operation from a Postgres table.
resource "meroxa_connector" "source" {
  name = "from-postgres"
  source_id = meroxa_resource.postgres.id
  input = "User"
  pipeline_id = meroxa_pipeline.pipeline.id
}

# The MongoDB connector will send data to a collection within MongoDB.
resource "meroxa_connector" "destination" {
  name = "to-mongo"
  destination_id = meroxa_resource.mongo.id
  input = meroxa_connector.source.streams[0].output[0]
  pipeline_id = meroxa_pipeline.pipeline.id
}

Once you’ve defined your pipeline, you can use theTerraform CLI to create, update, and destroy your Meroxa Resources.

Within the Meroxa Terraform Provider Documentation, you can view all the different configuration options for each resource type.

As always,

• If you need help, reach out tosupport@meroxa.io
• Join ourDiscord community.
• Follow us onTwitter andLinkedIn.

I can’t wait to see what you build 🚀