troubleshooting

How to resolve the error 'Protocol error: INCLUDE KEY clause must be set for FORMAT UPSERT ENCODE Json' when creating a source with Kafka in SQL?

I am trying to create a source with Kafka in SQL and encountering the error 'Protocol error: INCLUDE KEY clause must be set for FORMAT UPSERT ENCODE Json'. I want to set the email field as the key for upsert. How can I resolve this issue?

ha

harshit dwivedi

Asked on Feb 12, 2024

  1. To resolve the error 'Protocol error: INCLUDE KEY clause must be set for FORMAT UPSERT ENCODE Json' when creating a source with Kafka in SQL, you need to include the key using the INCLUDE KEY clause.

  2. Here is an example of how you can modify your SQL statement to include the key as email:

CREATE SOURCE IF NOT EXISTS global_contacts_source (
  email VARCHAR,
  first_name VARCHAR,
  last_name VARCHAR,
  state VARCHAR,
  city VARCHAR,
  country VARCHAR,
  postal_code VARCHAR,
  dob  VARCHAR,
  company  VARCHAR
)
INCLUDE KEY AS email WITH (
  connector = 'kafka',
  topic = '*******************',
  properties.bootstrap.server = '*******************',
  scan.startup.mode = 'earliest',
  properties.sasl.mechanism = 'SCRAM-SHA-512',
  properties.security.protocol = 'SASL_SSL',
  properties.sasl.username = '*******************',
  properties.sasl.password = '*******************'
) FORMAT UPSERT ENCODE JSON
  1. By including the key as email in the CREATE SOURCE statement, you can specify the field to be used for upsert operations.
Feb 12, 2024Edited by