troubleshooting
Is it possible to use a single Elasticsearch sink to write to multiple indexes based on a column value?
Florian Klein has a use case where they manually shard tenants using 1 index per tenant and wants to write to multiple indexes based on a column value. They are exploring options without provisioning a new sink for each tenant.
Fl
Florian Klein
Asked on Feb 09, 2024
- It is not possible to use a single Elasticsearch sink to write to multiple indexes based on a column value.
- You can create multiple sinks to write to different indexes or express the dispatch logic in SQL.
- Enabling
auto_create_indexin the ES cluster is a prerequisite for supporting this feature. - Creating a RisingWave sink for each new tenant when the corresponding ES index is created is a feasible approach.
- While the feature is technically doable, it may not align with product design principles.
- The concept of indexes in ES is similar to tables in traditional databases, and auto-creating indices for ES sink may introduce inconsistencies.
Mar 07, 2024Edited by