I am trying to create a source with Kafka in SQL and encountering the error 'Protocol error: INCLUDE KEY clause must be set for FORMAT UPSERT ENCODE Json'. I want to set the email field as the key for upsert. How can I resolve this issue?
harshit dwivedi
Asked on Feb 12, 2024
To resolve the error 'Protocol error: INCLUDE KEY clause must be set for FORMAT UPSERT ENCODE Json' when creating a source with Kafka in SQL, you need to include the key using the INCLUDE KEY
clause.
Here is an example of how you can modify your SQL statement to include the key as email:
CREATE SOURCE IF NOT EXISTS global_contacts_source (
email VARCHAR,
first_name VARCHAR,
last_name VARCHAR,
state VARCHAR,
city VARCHAR,
country VARCHAR,
postal_code VARCHAR,
dob VARCHAR,
company VARCHAR
)
INCLUDE KEY AS email WITH (
connector = 'kafka',
topic = '*******************',
properties.bootstrap.server = '*******************',
scan.startup.mode = 'earliest',
properties.sasl.mechanism = 'SCRAM-SHA-512',
properties.security.protocol = 'SASL_SSL',
properties.sasl.username = '*******************',
properties.sasl.password = '*******************'
) FORMAT UPSERT ENCODE JSON
CREATE SOURCE
statement, you can specify the field to be used for upsert operations.