I am doing one bigger migration using enrich and ingest pipeline. I am getting more slowness when doing multiple reindex simultaneously using the ingest pipeline. So, how many reindex can run simultaneously?. So, Please help me.

I am doing one bigger migration using enrich and ingest pipeline. I am getting more slowness when doing multiple reindex simultaneously using the ingest pipeline. So, how many reindex can run simultaneously?. So, Please help me.

• elasticsearch
• nest
• elasticsearch-8

Share Improve this question Follow asked Nov 16, 2024 at 11:00 bala nbala n 3911 silver badge1212 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

There is no straight answer to your question, but I'll try to provide my thoughts on the matter.

In a nutshell, I would say, that you can put as many parallel reindex jobs as your cluster can handle or as many as the consequent performance hindrance is acceptable to you.

The cluster in general has a set number of threads that handle requests and a queue size that will put in waiting all those that cannot be handled straight away. If you overwhelm the cluster with the parallel reindex requests then documents will be dropped or reindexing will fail.

If you don't mind how long the reindex will take then you can throttle it by using

requests_per_second

This means that you can make reindex as slow and light as needed. The more you throttle the more time it'll be needed to complete. The less you throttle the less performant your cluster will be.