I've recently been thinking about support for Struct type in Iceberg sink, since I'm testing if I can utilise RisingWave at work and such functionality is a necessity. When sinking struct data to iceberg catalog, an error is received due to a mismatch on the metadata
field in each Field
. The code just does a left == right
comparison. Is Struct support in Iceberg sink just a matter of lack of correct comparison or is there more context to that?
Szymon Orzechowski
Asked on Apr 29, 2024
The issue with Struct support in Iceberg sink is not solely due to a lack of correct comparison. While the error message indicates a mismatch in the metadata
field of each Field
, there may be additional complexities involved in supporting Struct types in Iceberg sink.
It's possible that the current implementation of Iceberg sink does not fully handle the serialization and deserialization of Struct types, leading to compatibility issues with RisingWave's Struct data.
To address this issue, a more comprehensive approach may be required to ensure proper handling of Struct types in Iceberg sink, including adjustments to the serialization logic and metadata comparison.
Creating an issue to track this specific use case and potential enhancements in the Iceberg sink for Struct support could help in resolving the compatibility issues with RisingWave's Struct data.