troubleshooting

Is there any working example somewhere of consuming a Kafka topic in AVRO format?

Jorge is looking for a working example of consuming a Kafka topic in AVRO format. He is facing issues with the AVRO schema and the protocol error related to the path scheme. The Kafka cluster is installed on an airgapped OpenShift cluster, using commercially supported Kafka from Red Hat (AMQ Streams) with a standard schema registry API from Confluent.

Jo

Jorge Luis Tudela Gonzalez de Riancho

Asked on Nov 24, 2022

  • Ensure that the AVRO schema is correctly configured and accessible.
  • Verify the protocol and path scheme being used for the AVRO schema location.
  • Check if the schema registry API response matches the expected schema definition.
  • Confirm that the Kafka cluster setup aligns with the requirements for consuming AVRO messages.
  • Consider using HTTPS for the AVRO schema location if supported.
  • Refer to documentation or examples for consuming Kafka topics in AVRO format.

Example:

// Sample code snippet for consuming Kafka topic with AVRO schema
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", KafkaAvroDeserializer.class.getName());
props.put("schema.registry.url", "http://schema-registry-url:8081");

KafkaConsumer<String, GenericRecord> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("avro-topic"));

while (true) {
    ConsumerRecords<String, GenericRecord> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, GenericRecord> record : records) {
        GenericRecord value = record.value();
        // Process the AVRO message
    }
}
Nov 25, 2022Edited by