At Meroxa, we’re empowering developers with Conduit OSS, a tool that simplifies real-time data engineering. With our latest release of new connectors, integrating with popular platforms is seamless, accelerating development and delivering real-time data insights. Here's a look at what's available now and what's coming next!
Released: New Conduit OSS Connectors
1. Salesforce Connector
Stream Salesforce object changes using the Salesforce Streaming API.
Configuration Example:
salesforce-source:
type: source
plugin: standalone:salesforce
settings:
auth.client_id: "<client_id>"
auth.client_secret: "<client_secret>"
auth.username: "<username>"
auth.password: "<password>"
api.version: "v52.0"
Use Case: Sync Salesforce opportunities to Snowflake for real-time sales insights.
2. Amazon Kinesis Connector
Integrate high-throughput streams to Kinesis for real-time analytics.
Configuration Example:
kinesis-destination:
type: destination
plugin: standalone:kinesis
settings:
aws.region: "us-east-1"
aws.access_key_id: "<access_key>"
aws.secret_access_key: "<secret_key>"
stream_name: "my-data-stream"
Use Case: Stream IoT data for real-time monitoring and alerts.
3. Amazon SQS Connector
Simplify message handling with Amazon Simple Queue Service (SQS).
Configuration Example:
sqs-source:
type: source
plugin: standalone:sqs
settings:
aws.region: "us-west-2"
aws.access_key_id: "<access_key>"
aws.secret_access_key: "<secret_key>"
queue_url: "https://sqs.us-west-2.amazonaws.com/123456789012/my-queue"
Use Case: Queue tasks for downstream services in a microservices architecture.
4. Elasticsearch Connector (Source)
Stream Elasticsearch index data into your pipelines for further analysis.
Configuration Example:
es-source:
type: source
plugin: standalone:elasticsearch
settings:
elasticsearch.url: "http://localhost:9200"
elasticsearch.index: "my-index"
Use Case: Extract logs for storage in a data lake or for further processing.
5. Amazon DynamoDB Connector (Source)
Leverage DynamoDB Streams for real-time updates, inserts, and deletes.
Configuration Example:
source:
type: source
plugin: standalone:dynamodb
settings:
aws.region: "us-east-1"
aws.access_key_id: "<access_key>"
aws.secret_access_key: "<secret_key>"
table_name: "my-table"
Use Case: Replicate data from DynamoDB to a relational database for reporting.
Coming Soon: Expanded Capabilities
1. MySQL Connector
Stream real-time MySQL changes with Change Data Capture (CDC).
Planned Configuration Example:
source:
type: source
plugin: standalone:mysql
settings:
mysql.host: "localhost"
mysql.user: "root"
mysql.password: "password"
mysql.database: "mydb"
Use Case: Synchronize MySQL data with cloud data warehouses for real-time analysis.
2. SFTP Connector
Automate file ingestion from SFTP servers into your pipelines.
Planned Configuration Example:
source:
type: source
plugin: standalone:sftp
settings:
sftp.host: "sftp.example.com"
sftp.username: "user"
sftp.password: "password"
file_pattern: "*.csv"
Use Case: Import nightly batch files for processing in data lakes or warehouses.
Why Developers Love Conduit OSS
- Developer-Centric Design: Pre-built connectors save time and effort.
- Real-Time Ready: Instantly access, stream, and process data.
- Scalability: Handles large data loads effortlessly.
- Flexibility: Easily configure sources and destinations for any workflow.
- Free and Transparent: As an open-source project, Conduit OSS is free to use, modify, and extend.
Get Started Today
Explore these connectors and start building:
- Salesforce Connector
- Amazon Kinesis Connector
- Amazon SQS Connector
- Elasticsearch Connector
- Amazon DynamoDB Connector
Join the Community
Collaborate with fellow developers and contribute to Conduit OSS on GitHub or Discord. Share your feedback and ideas to help us expand the ecosystem with new connectors and features.
Follow us on Twitter, LinkedIn, and YouTube for more insights and updates!
Have managed platform needs? Request a demo with one of our expert team members today!
