troubleshooting
Is it possible to decode protobuf as a transformation step or only from a source directly?
Neil is exploring the possibility of decoding protobuf as a transformation step instead of directly from a source, due to versioning concerns. Various team members provide insights and suggestions on the approach.
Ne
Neil
Asked on Dec 13, 2023
- Currently, the recommended way to parse protobuf messages is directly from a source.
- Storing raw binary in addition to parsing with different versions of protobuf files can be achieved by creating a new source every time with different protobuf definitions.
- Using a UDF for parsing protobuf messages is an alternative approach.
- A proposed
protobuf_decode(bytea, proto_url)function may not significantly differ from using a UDF, as it still requires a network hop to access the remoteproto_url.
Dec 18, 2023Edited by