Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue
authorNick Clemens <nick@bywatersolutions.com>
Mon, 9 Jan 2023 13:43:43 +0000 (13:43 +0000)
committerMartin Renvoize <martin.renvoize@ptfs-europe.com>
Wed, 3 May 2023 12:06:49 +0000 (13:06 +0100)
commit9a1de9b3ed4018f46d8ab7583e9064786a415328
tree3db69fc8a80655139f564a5ed7ec527c5b19dcf0
parent3f386d25731206d100f8f5c580f311460e76e6a6
Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue

Currently we generate large numebrs if single record reindex for circulation and other
actions. It can take a long time to process these as we need to load the ES settings for each.

This patch updates the Elasticsearch background jobs to throw records into a new queue
that can be processed by it's own worker and adds a dedicated worker that batches the jobs
every 1 second.

To test:
1 - Apply patches, set SearchEngine system preference to 'Elasticsearch'
2 - perl misc/search_tools/es_indexer_daemon.pl
3 - Leave the running in terminal and perform actions in staff interface:
    - Checking out a bib
    - Returning a bib
    - Editing a single bib
    - Editing a single item
    - Batch editing bibs
    - Batch editing items
4 - Confirm for each action that records are updated in search/search results
5 - Stop the script
6 - set SearchEngine system preference to 'Zebra'
7 - perl misc/search_tools/es_indexer_daemon.pl
8 - Script dies as Elasticsearch not enabled

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Bug 32594: (follow-up) Adjust logging per bug 32612

JD amended patch: tidy! There were tabs here...

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit aba2453ad65f8cb4cc9ea609dbf5bb3ed6f241db)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Koha/BackgroundJob/UpdateElasticIndex.pm
misc/workers/background_jobs_worker.pl
misc/workers/es_indexer_daemon.pl [new file with mode: 0755]