Upgrade your pipelines to data apps.

Build, test, and deploy real-time data apps in no time.

Get started >>>

Real-Time Pipelines as Code with the Meroxa Terraform Provider

By   Taron Foxworth

 6 Aug 2021

Making production-ready pipelines still requires a significant amount of time and effort. With the Meroxa CLI and the Meroxa Dashboard, your pipelines are streaming, real-time, and up and running in minutes, not months. Today, we’re adding a new way for you to build pipelines with versioning, speed, and consistency.

Introducing the Meroxa Terraform Provider. 🎉

The provider allows you to:

  • Provision, modify and destroy various objects on the Meroxa platform as code.
  • Easily share pipelines with your team.
  • Manage pipelines next to infrastructure managed with Terraform.


If you’re new to Terraform, it is an open-source infrastructure as code software tool that provides a workflow and tooling to manage cloud infrastructure. Using the Terraform Provider, you can add your data pipeline resources to the list of items that Terraform can manage. For more information, check out the Terraform Getting Started Guide.

Getting Started

To get started with the Meroxa Terraform Provider, require it within your Terraform File:

terraform {
required_providers {
meroxa = {
version = "1.0"
source = "meroxa.io/meroxa/meroxa"
}
}
}

Now, you can define your Meroxa resources within this Terraform project.

For example, here is an example pipeline that can assist with migration from PostgreSQL to Mongo. It sets up a pipeline keep both databases in sync in real-time:

# Require Provider terraform {
required_providers {
meroxa = {
version = "0.1"
source = "meroxa.io/meroxa/meroxa"
}
}
}
# Configure Providerprovider "meroxa" {
access_token = var.access_token # optionally use MEROXA_ACCESS_TOKEN env var
}
# Define Pipelineresource "meroxa_pipeline" "pipeline" {
name = "sync-postgres-mongo"
}
# Configure Postgres Resourceresource "meroxa_resource" "postgres" {
name = "my-postgres"
type = "postgres"
url = "POSTGRES_CONNECTION_URL"
}
# Configure MongoDB Resourceresource "meroxa_resource" "mongo" {
name = "my-mongo"
type = "mongodb"
url = "MONGO_CONNECTION_URL"
}
# The PostgreSQL connector will capture CDC events for
# every insert, update and delete operation from a Postgres table.
resource "meroxa_connector" "source" {
name = "from-postgres"
source_id = meroxa_resource.postgres.id
input = "User"
pipeline_id = meroxa_pipeline.pipeline.id
}
# The MongoDB connector will send data to a collection within MongoDB.resource "meroxa_connector" "destination" {
name = "to-mongo"
destination_id = meroxa_resource.mongo.id
input = meroxa_connector.source.streams[0].output[0]
pipeline_id = meroxa_pipeline.pipeline.id
}


Once you’ve defined your pipeline, you can use the Terraform CLI to create, update, and destroy your Meroxa Resources.

Within the Meroxa Terraform Provider Documentation, you can view all the different configuration options for each resource type.

As always,


I can’t wait to see what you build 🚀

         Meroxa

 Taron Foxworth

Taron Foxworth

I’m Fox. I talk about computers, writing software, and making content. I’m head of developer relations for @meroxadata.