I'm trying to ingest data from Kafka in Avro format using RisingWave, but I'm encountering issues with the CREATE SOURCE command. Here's the Avro schema I'm using:
{
"type": "record",
"name": "account",
"namespace": "KafkaSeed.Models",
"fields": [
{
"name": "account_number",
"type": "string",
"aliases": [
"AccountNumber"
]
},
{
"name": "value",
"type": "double",
"aliases": [
"Value"
]
},
{
"name": "account_type",
"type": "string",
"aliases": [
"AccountType"
]
},
{
"name": "open_date",
"type": {
"type": "long",
"logicalType": "timestamp-micros"
},
"aliases": [
"OpenDate"
]
},
{
"name": "billing_start_date",
"type": {
"type": "long",
"logicalType": "timestamp-micros"
},
"aliases": [
"BillingStartDate"
]
},
{
"name": "managed",
"type": "boolean",
"aliases": [
"Managed"
]
},
{
"name": "rep_codes",
"type": [
"null",
{
"type": "array",
"items": "string"
}
],
"aliases": [
"RepCodes"
]
}
]
}I'm also having trouble with the routing between containers in Docker. How should I structure the CREATE SOURCE command and address the container routing issue?
Tim Gitchel
Asked on Mar 26, 2024
To correctly use CREATE SOURCE with Kafka and Avro in RisingWave, you need to ensure that the Avro schema is accessible to RisingWave and that the Kafka broker is reachable from the RisingWave instance. Here's an example of how you might structure the CREATE SOURCE command:
CREATE SOURCE IF NOT EXISTS account_source
WITH (
connector='kafka',
topic='accounts',
properties.bootstrap.server='redpanda-0:9092',
scan.startup.mode='earliest'
) FORMAT PLAIN ENCODE AVRO (
schema.registry = 'http://redpanda-0:8081',
message = 'KafkaSeed.Models.account'
);Regarding the container routing issue, if you're using Docker and have multiple containers that need to communicate, you should ensure that they are on the same Docker network or use Docker Compose to manage them. This will allow containers to communicate with each other using their service names as hostnames. Here's an example of how you might define a Docker Compose service for RisingWave:
services:
risingwave:
image: risingwave:latest
depends_on:
- redpanda
networks:
- mynetwork
networks:
mynetwork:
driver: bridgeMake sure that the redpanda service is also defined in the same Docker Compose file and is using the same network.