Bug 35086: Add chunk_size option to elasticsearch configuration
authorNick Clemens <nick@bywatersolutions.com>
Fri, 22 Dec 2023 20:11:34 +0000 (20:11 +0000)
committerLucas Gass <lucas@bywatersolutions.com>
Fri, 2 Feb 2024 16:25:22 +0000 (16:25 +0000)
commit19f442c18fd600f098b90fbde18479338d382d9a
treef02d1df85168fb3f633aabe519a89b22a58c31fe
parent486140a7582817c95b538806241f963d0b75e6c9
Bug 35086: Add chunk_size option to elasticsearch configuration

Whne performing batch operations we can send a large numebr of records for reindexing at once.
Currently this can create requetss that are too large for Elasticsearch to process. We need
to break these requests into chunks/

This patch adds a chunk_size configuration to the elasticsearch stanza in koha-conf.xml

If blank we default to 5000.

To test:
0 - Have Koha using Elasticsearch
1 - Create and download a report of all barcodes:
    SELECT barcode FROM items
2 - Batch modify these items
3 - Note a single ESindexing job is created
4 - Create and download a report of all authority ids:
    SELECT auth_header.authid FROM auth_header
5 - Setup a marc modification template, and batch modify all the authorities
6 - Again note a single ES backgorund job is created
7 - Apply patch
8 - Repeat the modifications above - you still get a single job
9 - Edit koha-conf.xml and add <chunk_size>250</chunk_size> to elasticsearch stanza
10 - Repeat modifications - you now get several background ES jobs
11 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 9951e230e43d08dc6a2fb635877d0db5c1e245ff)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 636920558caa114afef1c50256a5031f4be7b0da)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Koha/SearchEngine/Elasticsearch/Indexer.pm
etc/koha-conf.xml
t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t