Commit graph

2825 commits

Author SHA1 Message Date
aba2453ad6
Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue
Currently we generate large numebrs if single record reindex for circulation and other
actions. It can take a long time to process these as we need to load the ES settings for each.

This patch updates the Elasticsearch background jobs to throw records into a new queue
that can be processed by it's own worker and adds a dedicated worker that batches the jobs
every 1 second.

To test:
1 - Apply patches, set SearchEngine system preference to 'Elasticsearch'
2 - perl misc/search_tools/es_indexer_daemon.pl
3 - Leave the running in terminal and perform actions in staff interface:
    - Checking out a bib
    - Returning a bib
    - Editing a single bib
    - Editing a single item
    - Batch editing bibs
    - Batch editing items
4 - Confirm for each action that records are updated in search/search results
5 - Stop the script
6 - set SearchEngine system preference to 'Zebra'
7 - perl misc/search_tools/es_indexer_daemon.pl
8 - Script dies as Elasticsearch not enabled

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>

Bug 32594: (follow-up) Adjust logging per bug 32612

JD amended patch: tidy! There were tabs here...

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-24 17:52:20 -03:00
4c51596a4b
Bug 32992: Move background_jobs_worker to misc/workers
On bug 32594 we are adding a new worker, dedicated to Elastic indexing.
We should have a common place for workers, and we agreed on misc/workers

To test:
1 - Apply patch
2 - reset_all in koha testing docker
3 - ps aux | grep background
4 - Confirm the workers are running, and running in the new directory
5 - Perform a batch item modification
6 - Ensure the job is processed by the worker

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-24 17:52:19 -03:00
d6433ff71e
Bug 32798: Update parameter passed to Koha::Biblio::Metadata->record
To test:
1 - Enable OAI sets, and define a set with mapping 952 y = BK
2 - perl misc/migration_tools/build_oai_sets.pl -v -i -r
3 - The script dies:
    Koha::Biblio::Metadata->record must be called on an instantiated object or like a class method with a record passed in parameter
4 - Apply patch
5 - Repeat
6 - Success!

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit: tcohen updated indentation
2023-02-21 10:23:34 -03:00
Magnus Enger
6a0cd4cc5e
Bug 32922: Remove space in shebang
Some of our scripts have a space in the "shebang" (first) line:

  #! /usr/bin/perl

This is not illegal, and it does work, but it is good to be
consistent, so this patch removes the space.

To test:
- Run: grep -rn --include=*.pl '#! /usr/' *
- See the list of files that have a space in the shebang
- Apply the patch
- Run the command again, there should be no output, meaning there
  are no more files with space in the shebang
- Have a look at the patch and check that it only changes the
  shebangs
- Sign off

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-20 09:44:06 -03:00
e8c232fcf7
Bug 30642: (QA follow-up) Do not rely on script names in modules, add unit test
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit: Kyle, stop impersonating John Doe
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-10 11:07:59 -03:00
aa3444505f
Bug 32356: Install installer translated files on update
If the installer files exist for a given language, the translate script
won't update it.
We should get a confirmation from Bernardo (author of bug 24262), but I
don't understand why it could be needed (side-effects?)

Test plan:
Installer several times the same language, drop the DB and run the
installer+onboarding process.
Check files installed by the installer (yaml for notice templates,
biblio frameworks) have inserted the data properly in DB.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-09 10:21:17 -03:00
d81bca8270
Bug 29100: (QA follow-up) Rename issue(s) keys to checkout(s)
Adapted test plan:

1) Apply this patch
2) For DUE and PREDUE notices, set the message body to the following:
   Title: [% checkout.title %]
3) For DUEDGST and PREDUEDGST notices, set the message body to the following:
   Titles:
   [% FOREACH c IN checkouts %]
    * [% c.title %][% END %]
4) Generate PREDUE and DUE notices for patrons including digests
5) Verify those notices contain the checkout titles

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit: tcohen renamed @issues => @checkouts as well
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-07 15:36:21 -03:00
fd7a612c6e
Bug 29100: Remove unnecessary loops
Remove loops that only operate one one result only

Signed-off-by: Felicity Brown <Felicity.Brown@montgomerycountymd.gov>
Signed-off-by: George Veranis <gveranis@dataly.gr>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-07 15:36:21 -03:00
2114c70d11
Bug 29100: Add checkouts data loop to predue notices script ( advance_notices.pl )
Predue / due notices are limited to using itemscontent to display checkouts data. It would be nice to make all the checkouts data available for those notices so that libraries could format that data more nicely if they wish.

1) Apply this patch
2) For DUE and PREDUE notices, set the message body to the following:
Title: [% issue.title %]
3) For DUEDGST and PREDUEDGST notices, set the message body to the following:
Titles:
[% FOREACH i IN issues %]
* [% i.title %]
[% END %]
4) Generate PREDUE and DUE notices for patrons including digests
5) Verify those notices contain the checkout titles

Signed-off-by: Felicity Brown <Felicity.Brown@montgomerycountymd.gov>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-07 15:36:21 -03:00
e5f251a709
Bug 15869: Change framework on overlay
This patch allows for selection of framework to use when overlaying
records - by default it is set to keep the initial framework

To test:
 1 - Create some records using one framework
 2 - Export the records
 3 - Edit the records to add fields not in original framework
 4 - Stage records using a rule that will find matches
 5 - Import
 6 - Note records contain new fields on display, but they are lost on edit
 7 - Apply patch
 8 - Stage records again
 9 - Select a framework that contains the new fields on import
10 - Import records
11 - Note records now use selected framework and are displayed/edited
correctly

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-07 10:32:22 -03:00
4734ee5d00
Bug 32793: import_patrons.pl typo in usage
Typo fix to prevent confusion in CLI usage

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-03 10:30:14 -03:00
2570062d25
Bug 32537: (QA follow-up): Tidy code
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-03 10:30:10 -03:00
b1867f8485
Bug 32537: (QA follow-up) Don't allow invalid no block values
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-03 10:30:09 -03:00
0880d80949
Bug 32537: Add no-block option to SIP cli emulator
This patch adds the option to pass a no-block option and use this
in checkout/checkin/renew messages

To test:
1 - perl misc/sip_cli_emulator.pl -a localhost -p 6001 -l CPL -su term1 -sp term1 -m checkin --item 39999000010831
2 - Confirm out like:
    SEND: 09N20221227    20183920221227    201839APCPL|AOCPL|AB39999000010831|ACterm1|BIN|
    (Note the N after 09)
3 - Apply patch
4 - Restart SIP, repeat 1
5 - Confirm still an N
6 - perl misc/sip_cli_emulator.pl -a localhost -p 6001 -l CPL -su term1 -sp term1 -m checkin --item 39999000010831 -n N
7 - Confirm still an N
8 - perl misc/sip_cli_emulator.pl -a localhost -p 6001 -l CPL -su term1 -sp term1 -m checkin --item 39999000010831 -n Y
9 - Confirm the send has a Y after 09

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-03 10:30:08 -03:00
3bf31ae4d4
Bug 32673: Remove misc/load_testing/ scripts
see comment 0 for more info

Signed-off-by: David Cook <dcook@prosentient.com.au>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-03 10:30:07 -03:00
89ff35b025
Bug 32612: (QA follow-up) Add missing interface for Logger
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-03 10:30:06 -03:00
3b1f2c57ea
Bug 32612: Add worker-error to log4perl conf
To test:
1 - Apply patch
2 - vim /etc/koha/sites/kohadev/log4perl.conf, Add lines below:
log4perl.logger.worker = WARN, WORKER
log4perl.appender.WORKER=Log::Log4perl::Appender::Screen
log4perl.appender.WORKER.stderr=1
log4perl.appender.WORKER.mode=append
log4perl.appender.WORKER.layout=PatternLayout
log4perl.appender.WORKER.layout.ConversionPattern=[%d] [%p] %m %l%n
log4perl.appender.WORKER.utf8=1
3 - Restart all
4 - Edit misc/background_jobs_worker.pl
-        my $job = Koha::BackgroundJobs->find($args->{job_id});
+        my $job;# = Koha::BackgroundJobs->find($args->{job_id});
5 - In another terminal: tail -f /var/log/koha/kohadev/koha-worker-error.log
6 - Force enqueue a job (that won't be found because of #4
  perl -e 'use Koha::BackgroundJob::BatchUpdateItem; my $bg = Koha::BackgroundJob::BatchUpdateItem->new(); $bg->enqueue({ record_ids=>['888888']});'
7 - Note error in log like:
  [2023/01/11 19:26:10] [WARN] No job found for id=2983 main:: /kohadevbox/koha/misc/background_jobs_worker.pl (111)

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-03 10:30:02 -03:00
08cc23626c
Bug 32393: (QA follow-up) Add explicit undef response in two catch blocks
Do not implicitly depend on last statement returning nothing.
Make it explicit. We want $args to be null here.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-01-27 15:10:01 -03:00
85c330d8f2
Bug 32393: Deal with the DB fallback part
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-01-27 15:10:01 -03:00
8df96262c1
Bug 32393: Split into 2 try catch blocks
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-01-27 15:10:00 -03:00
ca9a598183
Bug 32393: Prevent invalid job to block the job queue
I have faced a problem when testing an incorrect version of bug 32370.
The frame sent to the message broker was not a correct JSON encoded
string, and its decoding was obviously failing, exploding the worker
script.
Additionally, as we don't send a ack for this frame, the next pull will
result in processing the same message, and so in the same explosion.
No more messages can be processed!

This patch is logging the error and ack the message to the broker, in
order to not get stuck.

Test plan:
0. Dont' apply this patch
1. Enqueue a bad message
  a. Apply 32370
  b. Comment the following line in Koha::BackgroundJob::enqueue
    $self->set_encoded_json_field( { data => $job_args,    field => 'data' } );
  c. restart_all
  d. Use the batch item modification tool to enqueue a new job
=> Notice the error in the log
=> Note that the status of the job is "new"
=> Inspect rabbitmq queue:
% rabbitmq-plugins enable rabbitmq_management
% rabbitmqadmin get queue=koha_kohadev-long_tasks
You will notice there is a message in the "long_tasks" queue
2. Enqueue a good message
  a. Remove the change from 1.b
  b. restart_all
  c. Enqueue another job
=> Same error in the log
=> Both jobs are new
=> Inspect rabbitmq, there are 2 messages
3. Apply this patch
4. restart_all
=> Second (good) job is finished
=> rabbitmq long_tasks queue is empty

We cannot mark the first job as done, we have no idea which job it was!

QA: Note that this patch is dealing with another problem, not tested in
this test plan. If an exception is not correctly caught by the ->process
method of the job, we won't crash the worker. The job will be marked as
failed.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-01-27 15:10:00 -03:00
b3c5bd02e1
Bug 32573: Send ACK message to RabbitMQ before handling the job
Splitting off this functionality from bug 32558. This is the comment that started this discussion: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32558#c15

From the O'Reilly book Mobile and Web Messaging:

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-01-27 15:09:52 -03:00
790f00781a
Bug 32656: Script delete_records_via_leader.pl no longer deletes items
Bug 29788 inadvertantly replaced a call to safe_delete() with safe_to_delete()
such that any time the script should delete an item it only checks to see if
the item is delectable, after which deletion of the record fails because the
items were not deleted.

Test Plan:
1) Mark a record with items to be deleted via the record leader
2) Run delete_records_via_leader.pl -i -b -v
3) Note the script says it is deleting the items but then the record
   deletion fails. Note the items remain in the items table of the
   database.
4) Apply this patch
5) Repeat step 2
6) This time the items and record should be deleted!

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2023-01-20 13:53:09 +00:00
cd89c38321
Bug 32481: Limit prefetch size for background jobs worker
This patch adds a prefetch size of 1 to the background jobs worker,
so that it fetches 1 message at a time. Without this change,
the RabbitMQ connection timeout when too many messages for slow tasks
are fetched at the same time.

To test:
0. Apply patch
1. Run background worker
2. Rapidly enqueue multiple jobs that in total will take longer
than 30 minutes to process

Bug 32481: Use correct prefetch syntax for RabbitMQ

According to https://www.rabbitmq.com/stomp.html the header to
use for managing the prefetch is "prefetch-count".

You can verify the number of delivered and unacknowledged messages
on a channel on a connection by running "rabbitmqctl list_channels"
on the RabbitMQ host. This will tell you how many messages have been
delivered and are awaiting acknowledgement

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-01-11 20:46:01 -03:00
Koha translators
623fb377ad
Translation updates for Koha 22.11.00
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-25 15:20:05 -03:00
0b6837ea22
Add release notes for Koha 22.11.00
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-25 15:20:02 -03:00
404c88fe87 Bug 31969: Use filter_by_last_update
This script has a pattern to delete rows depending on a given
date/number of days, we should use the filter_by_last_update
Koha::Objects method.
No need for another method and tests, everything is already tested
there.

This patch also suggests to rename the reference to "background" and
"bg" with "jobs", which seems more appropriate and not an abbreviation

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-25 09:40:09 -03:00
Katrin Fischer
356e136b0f
Bug 32093: Make help and code match for bg-days in cleanup_database.pl
There as a mismatch between the parameter name in documentation (bg-days)
and what was checked in the code (bg-jobs). This makes both match.

To test:
* Apply patch
* Make sure you have some background_jobs older than one day
* Verify the help of the script documents the parameter bg-days
* Run the cleanup_database.pl job
  Example:  ./misc/cronjobs/cleanup_database.pl --bg-days  --confirm -v
* Verify there are no errors and the lines have been deleted as expected

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-17 14:55:12 -03:00
a9e0bcff16
Bug 27920: (QA follow-up) Fix POD in cli tool
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-09 14:37:27 -03:00
1afc3b741b
Bug 27920: (QA follow-up) Fix syntax error
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-09 14:37:20 -03:00
e4e453d5df
Bug 27920: Add ability to update patron expiration dates when importing patrons
Some libraries need to recalculate a patron's expiration date any time they are updated via a patron import from file.

Test Plan:
1) Apply this patch
2) prove t/db_dependent/Koha/Patrons/Import.t

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-09 14:37:07 -03:00
0232ee9cc4
Bug 32030: Remove maintenance file
This script is actually not used.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-09 09:48:21 -03:00
862de51915
Bug 32030: Wrong permissions on maintenance script
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 21:04:10 -03:00
de77f3a719
Bug 32030: I18N rewrite
The previous implementation was using vue-i18n inject localized strings
into the Vue app. However it was using json files, and we needed
additional overhead to convert from/to PO files.

I've also tried vue3-gettext that use PO, but the overhead to work with
our workflow was existent as well (see branch joubu/vue3-gettext).

vue-i18n-extract was using for extracting, and a specific misc script
(misc/translate_json.pl) was also used to generate the json file. They
can be removed.

Here we are simply reusing our existing workflow, and we will improve it
(ie. make it more vue-ish) later if we need it.

Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 09:44:38 -03:00
6be1de94e0
Bug 32030: I18N - Load de-DE, es-ES, fr-FR strings
Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 09:44:18 -03:00
487470b411
Bug 32030: maintenance holdingsIQ
Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 09:44:05 -03:00
151f6f3f0c
Bug 32030: add maintenance script for HoldingsIQ
Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 09:44:03 -03:00
74be73121e
Bug 32030: Update I18N
Note that we are adding an extra space for id and counter, otherwise
they got removed in favor of the "simple" string.
  { Agreement: Agreement }
replaced
  { Agreement: { id: ..., counter: ... } }

Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 09:44:02 -03:00
0c763b7dc4
Bug 32030: Improve I18N
* Remove ":" from the translation to prevent duplication string
* Add a script to auto-translate the strings in English

echo "{}" > koha-tmpl/intranet-tmpl/prog/js/vue/locales/en.json
npx vue-i18n-extract --vueFiles 'koha-tmpl/intranet-tmpl/prog/js/vue/**/*.?(js|vue)' \
                     --exclude koha-tmpl/intranet-tmpl/prog/js/vue/dist/main.js \
                     --languageFiles 'koha-tmpl/intranet-tmpl/prog/js/vue/locales/*.json' \
                     --add --remove
perl misc/translate_json.pl | sponge koha-tmpl/intranet-tmpl/prog/js/vue/locales/en.json

Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 09:44:01 -03:00
Joonas Kylmälä
8a63d78c4b
Bug 27259: Add HomeOrHoldingBranch checks where it was missing from
The TooMany() function and fine calculation functions were incorrectly
hard coded to use homebranch for fetching the circulation rules. Those
ignored completely the syspref HomeOrHoldingBranch where the user
might have set it to holdingbranch and therefore the fines and whether
patron has too many checkouts (TooMany()) were counted using the
unintended branch's rules. This problem only arises in the cases where
there are branch specific circulation rules defined.

Test plan:
1. Make sure following tests pass:
   $ prove t/db_dependent/Circulation/_CalculateAndUpdateFine.t
   $ prove t/db_dependent/Circulation/TooMany.t

Test plan for fines.pl:
1. Add branch specific fine rules for branches A and B. A having a
   fine of 1 per day and B having a fine of 0 per day.
2. Set sysprefs:
   CircControl = the library the items is from
   finesMode = Calculate and charge
   HomeOrHoldingBranch = holdingbranch
3. Create an item with home and holding branch of A
4. Checkout the item with a due date in the past (the past due date can be
   specified by clicking "Checkout settings" in the checkout page) and
   make sure the branch you are checking from is B.
5. Run perl /usr/share/koha/bin/cronjobs/fines.pl
6. Notice that fines have popped up now to the patron incorrectly
7. Apply patch
8. Pay fines, Check-in the item and check it out again
9. Run perl /usr/share/koha/bin/cronjobs/fines.pl
10. Notice that fine is now 0. This means that the branch
    B (holdingbranch of the checked-out item) specific rule is used.

Test plan for staticfines.pl:
1. Add branch specific fine rules for branches A and B. A having a
   fine of 1 per day and B having a fine of 0 per day.
2. Set sysprefs:
   CircControl = the library the items is from
   finesMode = Calculate and charge
   HomeOrHoldingBranch = holdingbranch
3. Create an item with homebranch A and holding branch of A
4. Checkout the item with a due date in the past (the past due date can be
   specified by clicking "Checkout settings" in the checkout page) and
   make sure the branch you are checking from is B.
5. Run perl staticfines.pl --library A --library B --category <PATRONS_CATEGORYCODE>
   and notice that now there is inccorectly fines
6. Apply patch
7. Pay fines, Check-in the item and check it out again
8. Run perl staticfines.pl --library A --library B --category <PATRONS_CATEGORYCODE>
    and notice the fines are now not generated

Rebased-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Petro Vashchuk <stalkernoid@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-04 19:04:18 -03:00
Katrin Fischer
469aff4c16
Bug 31854: (QA follow-up) Terminology: loans > checkouts
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-03 10:04:27 -03:00
Magnus Enger
e57ed68a02
Bug 31854: Document --not_borrowed_since and anonymization
To test:
- Run "perldoc misc/cronjobs/delete_patrons.pl"
- Note the absence of information about the potential conflict
  between --not_borrowed_since and anonymization
- Apply this patch
- Re-run the perldoc command and make sure the note about
  anonymization make sense

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-03 10:04:27 -03:00
Katrin Fischer
a4f2e6ae85
Bug 31995: (QA follow-up) Add 'system preference' to help text
We often get asked where something can be found in setting,
adding the hint that RealTimeHoldsQueue is a system preference
is hopefully helpful here.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-03 09:54:33 -03:00
1088e3fb79
Bug 31995: build_holds_queue.pl should check to see if the RealTimeHoldsQueue syspref is on
The real time hold queue and the build_holds_queue.pl jobs are not 100% compatible in that we should not be running the cron if the real time queue is enabled, this could lead to double server work. It would be good to have a check in build_holds_queue for the RealTimeHoldsQueue syspref and not run the job if the preference is enabled.

There might be times when we'd want to force a run of this job without changing the syspref. To that end we would also want a flag for this job so that system administrators could force the job from the command line if required, overriding this limitation.

Test Plan:
1) Apply this patch
2) Try run misc/cronjobs/holds/build_holds_queue.pl with the -h/--help and -m/--man options
3) Disable RealTimeHoldsQueue
4) Run with no options, should succeed
5) Enable RealTimeHoldsQueue
6) Run with no options, should display a message and not rebuild the
   holds queue
7) Run again with the -f/--force option, should succeed

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-03 09:54:32 -03:00
9aa21276b2
Bug 32012: runreport.pl should use binmode UTF-8
Like Bug 26832 added binmode UTF-8 to script misc/search_tools/export_elasticsearch_mappings.pl, this should be added to misc/cronjobs/runreport.pl.

Test plan :
1) Do not apply patch
2) Create a SQL report with :
   SELECT 'accentué',barcode FROM items limit 3
3) Note the id of this report, for example 1
4) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/without.csv
=> You see output with unknown character instead of é :
5) Run : file --mime-type /tmp/without.csv
=> You see : /tmp/without.csv: iso-8859-1
6) Apply patch
7) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/with.csv
=> You see correct output with é
8) Run : file --mime-type /tmp/without.csv
=> You see : /tmp/without.csv: utf-8

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-02 19:15:48 -03:00
Björn Nylén
78219e72f6
Bug 31969: (follow-up) Move deletion code to Koha::BackgroundJobs->purge for reuse.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-27 15:03:26 -03:00
Björn Nylén
6565328092
Bug 31969: Add background queue options to cleanup_database
This patch adds background queue options to cleanup_database.pl to allow
for purging completed background jobs.

--bg-jobs DAYS Purge all finished background jobs this many days old. Defaults to 1 if no DAYS provided.
--bg-type TYPE What type of background job to purge. Defaults to "update_elastic_index" if omitted
               Specifying "all" will purge all types. Repeatable.

To test:
 1 - Enable elastic search in Koha
 2 - perl misc/maintenance/touch_all_items.pl
 3 - Generate an number of diffrent types of bg-jobs (eg batch_hold_cancel,
     batch_biblio_record_deletion, batch_item_record_deletion)
 4 - Check db and note there are a bunch of diffrent jobs
 5 - Update to make them old
     UPDATE background_jobs SET ended_on = '2022-10-01 00:00:00', status='finished'
 6 - perl misc/cronjobs/cleanup_database.pl
 7 - Note bg-jobs entry shows in help
 8 - perl misc/cronjobs/cleanup_database.pl --bg-jobs 1 -v
 9 - Note that elasticqueue would have been cleared
10 - perl misc/cronjobs/cleanup_database.pl --bg-jobs 1 -v --confirm
11 - Note that number of entries deleted is reported
12 - Attempt to clear other job types, including "all" eg
     perl misc/cronjobs/cleanup_database.pl --bg-jobs 1 --bg-type batch_item_record_deletion  -v --confirm
     perl misc/cronjobs/cleanup_database.pl --bg-jobs 1 --bg-type all  -v --confirm
13 - Confirm in staff interface that jobs are gone:
     http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl
     (Uncheck 'Current jobs only')

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-27 15:03:21 -03:00
c14951baee Revert "Bug 15187: Index 880 in Zebra the same as Elasticsearch"
This reverts commit 85690a14f8.
2022-10-24 14:03:44 -03:00
ae986b992d
Bug 27265: (QA follow-up) Typo on L99 cron script
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-17 21:11:41 -03:00
da7bcc2252
Bug 27265: Enable message queue cron to accept multiple codes and types
It would be nice to be able to combine several types in a single run,
but exclude others, without having to have multiple cron lines

Test Plan:
1) Apply this patch
2) Run process_message_queue.pl with a single -c parameter
3) Note behavior is unchanged
4) Run process_message_queue.pl with multiple -c parameters
5) Note all the codes specified are processed
6) Repeat 2-5 with -t for type limits

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-17 21:11:40 -03:00
c184e88464
Bug 28555: (follow-up) Add to script documenation
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-17 21:09:03 -03:00
125ec8a207
Bug 28555: (follow-up) Make verbose count flags passed
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-17 21:09:03 -03:00
fe376c7d85
Bug 28555: Improve logging for overdue_notices.pl
Currenlty the output with -v from overdue_notices is a bit overwhelming and hard to parse

I do a few things here:
1 - Only show SQL if verbose > 1
2 - Add some separators to make it more readab;e
3 - Remove some redundant messages with branchcode and borrowernumber

To test:
1: perl misc/cronjobs/overdue_notices.pl -v
2: Notice you get a LOT of information all smushed together
3: Apply patch
4: perl misc/cronjobs/overdue_notices.pl -v
5: Less info, and easier to read
6: perl misc/cronjobs/overdue_notices.pl -v -v
7: All the info, but nicer to read

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-17 21:08:58 -03:00
Petro Vashchuk
b04e8185a3
Bug 31776: Fix a typo in cleanup_database.pl cronjob's help/usage
cleanup_database.pl cronjob has a typo in it's usage/help:
"preserve-logs" option should be "preserve-log" as it is everywhere
in the code.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-17 08:14:21 -03:00
543454f765
Bug 28950: Mark serial late when past grace period
This script had some oddities:
- There was a LEFT JOIN to subscription, but no error printed if no subscription
  I changed this to a JOIN as there is a constraint, so this shoudl not happen
- Publisheddate was checked after the SQL, moved into query
- We checked for the next issue, and marked late only if that issue was already expected
  This overruled grace period, which should be the mnarker for a serial being late
  The grace period should be extended if you wish to wait for next issue
- If no next published date, we reported an error on the planneddate
- Script without confirm had no reporting

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-11 10:42:26 -03:00
906d5b1a85
Bug 31203: Alter other cronjobs that currenlty use cronlogaction
Added command line ooption logging and completion logging where
cronlogaction was already imported. We should probably standardize all
cronjobs, but this is a start

One cron didn't log on confirm, likely we need to update all crons to log
if confirm, and possibly not log if running in test mode? Another bug as well

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-05 16:12:59 -03:00
7749e73526
Bug 31203: Add End action to cronjobs and log viewer
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-05 16:12:55 -03:00
e68d2beaf0
Bug 31203: POC advance_notices
This patch proposes a new standard for cronlogaction

We copy @ARGV into a space separated string, then pass this to
cronlogaction

At the end of the script we add a call with info "COMPLETED"

To test:
1 - Apply patch
2 - perl misc/cronjobs/advance_notices.pl -v -m 4 -n -c
3 - sudo koha-mysql kohadev
4 - SELECT * FROM action_logs WHERE interface="CRON";
5 - Note start and completion logged
6 - Note command line arguments included in info

If this looks good I will copy this pattern to the other cronjobs

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-05 16:12:51 -03:00
bf6bbc29a0
Bug 20457: Overdue and pre-overdue cronjobs not skipping phone notices
The overdue and pre-overdue cron scripts are not skipping the generation of phone notices. This causes many phone notices to be created that will always be left at 'pending' as the Talking Tech outbound script creates its own phone notices and puts them in the message queue.

Test Plan:
1) Enable Talking Tech
2) Enable predue and overdue notice phone transports for a patron
3) Generate overdues and predues, notice phone notices are generated
4) Apply the patch
5) Repeat steps 2-3
6) Note phone notices are not generated

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-03 14:13:18 -03:00
4c3d18091d
Bug 31540: Exclude expired holds from the reminder job
Some libraries don't want to auto-cancel holds, but we should not remind
a patron about a hold which has expired.

To test:
1 - Place a hold for a patron
2 - Set it waiting
3 - Run the holds reminder script in the future
    perl misc/cronjobs/holds/holds_reminder.pl -day 1 --date '2023-09-12' -v
4 - Note the holds would be reminded
5 - Set expirationdate for the hold less than today
    UPDATE reserves SET expirationdate = DATE_SUB(CURDATE(), INTERVAL 1 DAY);
6 - Run the remidner cron again
7 - No holds trigger!

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-10-03 08:39:06 -03:00
Andrew Isherwood
65a2fc2fa3
Bug 30484: Implement support for ILL request updates
This commit adds support for the concept of ILL request update notices.

- Adds a new Koha::Illrequest::SupplierUpdate class that is used to
encapsulate an update to a request, this update may come from a
supplier via a backend or from core ILL via, perhaps, a user action
- Adds a new Koha::Illrequest::SupplierUpdateProcessor base class that
can be subclassed in order to create a processor that can be passed an
update and act accordingly.
- Updates to Illrequest.pm to support the above classes and allow core
Koha to offer update processors
- A shell script to initiate a periodic process to check for updates
meeting given criteria and run the appropriate processors

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

https://bugs.koha-community.org/show_bug.cgi?id=28909
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-23 09:18:45 -03:00
980871b19f
Bug 7021: Remove superflous 'open'
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-22 09:31:28 -03:00
791702a363
Bug 7021: Terminology - usercode -> categorycode
Lets stick to standard terminology and use categorycode rather than
usercode here.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-22 09:31:28 -03:00
a80a96b933
Bug 31342: Add execution locking to process_message_queue.pl
This patch makes the script use an execution lock as provided by
Koha::Script. Previous attempt got too complex and contentious, and
given we have a simple way to achieve the same thing, I decided to
submit an alternative approach.

To test:
1. Apply this patch
2. Add the following content to misc/cronjobs/process_message_queue.pl
   on line 83:

  sleep 100;

3. Save the file
4. Have two terminals open
5. On the first one, run:
   $ kshell
  k$ misc/cronjobs/process_message_queue.pl
6. On the second one, run the same
=> SUCCESS: The second one dies telling it had to skip the run
7. Undo your changes:
   $ git checkout misc/cronjobs/process_message_queue.pl
8. Sign off :-D

Sponsored-by: ByWater Solutions
Signed-off-by: Liz Rea <liz@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:13:52 -03:00
95ebfa1f29
Bug 30755: (follow-up) Add a results hash to automatic renewals
It's slightly debatable whether auto_too_soon is an error or not. I think
it leans far enough towards error that the other patch is correct. The change, however,
will affect notices and means we cannot count auto_too_soon anymore

This patch adds a new 'results' hash to the info - this includes a count per error
(including auto_renew i.e. success) allowing for notices to decide how to display the info.

To test:
1 - Add to Auto renew digest notice:
[% IF results %]
There were [% results.auto_too_soon %] items that were too soon.
[% END %]
2 - Check out some items to a patron that will be too soon
3 - Check out one that is not too soon and will renew
4 - Ensure patron has auto renew digest selected and that AutoRenewalNotices is set to follow patron preferences
5 - Run the auto renewals cron
6 - Confirm the notice for the patron displays 1 successful renewal, no failures, and a correct count of too_soon

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:11:06 -03:00
c41fab2292
Bug 30755: Remove auto_too_soon from error count
This patch prevents auto_too_soon errors being counted towards auto
renew errors.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:11:06 -03:00
3d0015fe69
Bug 31299: Fix duplicate output in search_for_data_inconsistencies.pl
In search_for_data_inconsistencies.pl when there are several inconsistencies on same framework, the second output of items contains the first one.

Looks like it is since Bug 21466

Test plan using koha-testing-docker :
1) Create 2 inconsistencies on the same item via SQL :
   update items set itemlost = 9 where itemnumber=900;
   update items set notforloan = 8 where itemnumber=900;
2) Without patch
3) Run ./misc/maintenance/search_for_data_inconsistencies.pl
=> You see duplicate output :
== Wrong values linked to authorised values ==
	* The Framework *BKS* is using the authorised value's category *NOT_LOAN*, but the following items.notforloan do not have a value defined ({itemnumber => value }):
 {900 => 8}
	* The Framework *BKS* is using the authorised value's category *LOST*, but the following items.itemlost do not have a value defined ({itemnumber => value }):
 {900 => 8} {900 => 9}

4) Apply patch
5) Run ./misc/maintenance/search_for_data_inconsistencies.pl
=> Fixed :D
== Wrong values linked to authorised values ==
	* The Framework *BKS* is using the authorised value's category *LOST*, but the following items.itemlost do not have a value defined ({itemnumber => value }):
 {900 => 9}
	* The Framework *BKS* is using the authorised value's category *NOT_LOAN*, but the following items.notforloan do not have a value defined ({itemnumber => value }):
 {900 => 8}

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-16 16:46:41 -03:00
88f63e01d2
Bug 28553: (QA follow-up) default to auto renew email when cron switched used
This patch defaults to using "email" rather than "sms" notices
when AutoRenewalNotices is set to use the --send-notices cron switch

Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-16 10:33:32 -03:00
fef29151e0
Bug 28553: Patrons can be set to receive auto_renew notices as SMS, but Koha does not generate them
Following bug 18532, one can set a patron's messaging prefs to deliver an auto-renew notice via SMS.
However, the auto_renewals cron only knows how to generate email notices. We should not allow the selection of non-functional transport types.

Test Plan:
 1 - Set pref AutoRenewalNotices - 'according to patron messaging preferences'
 2 - Set circ rule with:
    Loan period: 5 days
    No renewal before: 5
    AutoRenewal: Yes
    Renewals allowed: 5
 3 - Set SMS Send driver to Email
 4 - Find a patron and set them to receive auto renewal notices via SMS and no other transport
 5 - Provide a proivder and sms number for the patron
 6 - Checkout an item to the patron, set due date to yesterday
 7 - Run auto renewals:
    perl misc/cronjobs/automatic_renewals.pl -v --send-notices -c
 8 - Item renewed, no message sent
 9 - Check in item, checkout again due yesterday
10 - Apply patch
11 - Browse to Tools->notices and slips
12 - Edit AUTO_RENEWALS and AUTO_RENEWALS_DIGEST to copy email template to sms and add a title
13 - perl misc/cronjobs/automatic_renewals.pl -v --send-notices -c
14 - Item renewed, confirm patron has a notice in their account
15 - Check the digest option, check in item, checkout again as due, repeat cronjob and note digest notic
16 - sign off

https://bugs.koha-community.org/show_bug.cgi?id=30355

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-16 10:30:40 -03:00
4481817701
Bug 31239: (QA follow-up) Fixing ternary formatting
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-16 10:27:20 -03:00
54711942f1
Bug 31239: Fix incorrect AV linked to a mapped biblio column
If an AV is linked to a MARC field mapped with a biblio column, the
search_for_data_inconsistencies.pl script might explode with

The method Koha::Biblioitem->title is not covered by tests!

Trace begun at /kohadevbox/koha/Koha/Object.pm line 875
Koha::Object::AUTOLOAD('Koha::Biblioitem=HASH(0x556b67fa7168)') called at misc/maintenance/search_for_data_inconsistencies.pl line 246

Test plan:
For a given framework, pick a biblio using it
Link 245$a with an authorised value category
Run the script
=> Notice that with this script applied you will see the warning
=> Without this patch you got the error

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-16 10:27:20 -03:00
202db89f83
Bug 31351: (QA follow-up) Use $self->json in Background modules
Making the disabling utf8 flag explicit instead of depending on
the default of the CPAN module.

Incorporating the change in background_jobs_worker too.

Test plan:
See next patches when we look at unit tests.
Restart koha-worker.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-07 13:59:03 -07:00
caebb20cbb
Bug 30042: Remove Date::Calc dependency in batch_anonymize.pl
This patch makes the script use the Koha::DateUtils tools instead.

To test:
1. Run:
   $ kshell
  k$ perl misc/cronjobs/batch_anonymise.pl --verbose --days 7
=> SUCCESS: You see (the date may vary):
Checkouts and holds before 2022-02-15 will be anonymised.
2. Apply this patch
3. Repeat 1
=> SUCCESS: Same output
4. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-07 13:23:31 -07:00
f6a08737c9
Bug 31356: Use hold expiration date instead of adding days to the waiting date
The Talking Tech outbound script currently just adds the
ReservesMaxPickUpDelay to the waiting date to calculate the
"expiration date", but it should use the actual hold expiration
date which may differ from this naive calculation based on
various system preference values.

Test Plan:
1) Create a holiday for tomorrow
2) Set ReservesMaxPickUpDelay to 5
3) Set "Days mode" to "Same week day"
4) Enable ExcludeHolidaysFromMaxPickUpDelay
5) Enable TalkingTechItivaPhoneNotification
6) Create a hold and fill the hold so it is waiting
7) Enable "Hold filled" phone notices for that patron
8) Create a 'phone' version of the HOLD notice
9) Run ./misc/cronjobs/thirdparty/TalkingTech_itiva_outbound.pl --type RESERVE -w 0
10) Note the output has the expiration date ( 15th colume ) 5 days from now which is *not* the hold's expiration date
11) Apply this patch
12) Repeat the command from step 9
13) Note the expiration date column now matches the holds actual expiration date!

Signed-off-by: Kyle Hall <kyle@bywatersolutions.com>

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-31 14:30:26 -03:00
381b79593e
Bug 28327: Unify CSV delimiter special behavior for tabulation
System preference 'CSVdelimiter' has a special case for tabulation.
Preference value contains string 'tabulation' but string '\t' must be used in CSV file.

This is OK in many places, for exemple Bug 17590.

This patch adds C4::Context->csv_delimiter to add a uniq metod dealing
with this behavior.
Also create Koha::Template::Plugin::Koha->CSVDelimiter for calls from
Toolkit Templates.

Test plan :
1) Set system preference 'CSVdelimiter' = 'tabs'.
2) Create CSV export in impacted pages
3) Check columns are separated by tabulation character and not string 'tabulation'
4) Check with another delimiter

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-19 15:53:12 -03:00
9afbe872b7
Bug 31282: Fix broken characters in patron_emailer.pl verbose mode
To test:
1. Have a report containing:
   SELECT tomascohen@theke.io;
2. Have a members notice containing ¡ and other non-ASCII characters.
3. Run (changing the report number and notice code accordingly):
   $ kshell
  k$ perl misc/cronjobs/patron_emailer.pl --report 4 \
           --notice BIRTHDAY --from tomascohen@theke.io --module members
=> FAIL: non-ASCII characters are broken
4. Apply this patch
5. Repeat 3
=> SUCCESS: Things print correctly!
6. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-19 09:56:13 -03:00
48bf9b1d91
Bug 30718: Use flatpickr's altInput
The idea rely on the KohaDates TT plugin for the date formatting. We
should not have any output_pref calls in pl or pm (there are some
exceptions, for ILSDI for instance).

Also flatpickr will deal with the places where dates are inputed. We
will pass the raw SQL value (what we call 'iso' in Koha::DateUtils), and
the controller will receive the same value, no need to additional
conversion.
Note that DBIC has the capability to auto-deflate DateTime objects,
which makes things way easier. We can either pass the value we receive
from the controller, or pass a DT object to our methods.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-19 08:26:31 -03:00
bd70e8989f
Bug 31312: Remove GetItemsInfo from misc/migration_tools/rebuild_zebra.pl
Hum... Item2Marc ok here?

Bug 27272 is going to remove C4::Items::GetItemsInfo in favour of Koha::Items->search.

Here we are going to deal with misc/migration_tools/rebuild_zebra.pl

Test plan:
I am not sure, what do we want to test here? Items' info indexed
correctly?

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-16 10:40:17 -03:00
Magnus Enger
3f1fe4f178
Bug 31325: Fix koha-preferences get
To reproduce:
$ perl koha/misc/admin/koha-preferences get MarcFlavour
This will output something like "HASH(0x55dd4d432840)".

To test:
Apply this patch and run the command again:
$ perl koha/misc/admin/koha-preferences get MarcFlavour
This should output something like "MARC21".

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-16 09:11:40 -03:00
Katrin Fischer
18b2389de4
Bug 31281: Use correct reply-to email when sending overdue mails
Overdue emails are either sent from the issuing or the home
library of an item. We never use the patron's home library,
so the reply-to address must explicitly be set in EnqueueLetter.

To test:
- Set up 2 branches (A and B) with different email addresses.
- Set up an SMTP server for each to use
- Set up an overdue notice trigger for the patron category you'll use
  First letter: 1 day delay, any notice
- Check out an item with home branch B to a patron from A
- Run the the script with:
  overdue_notices.pl -t --frombranch item-homebranch
  overdue_notices.pl -t --frombranch item-issuebranch
- Confirm for each setting that the correct email headers have been
  used. You can see the reply-to address and to-address in the
  message queue:
  SELECT * FROM message_queue;

Signed-off-by: Nason Bimbe <nason.bimbe@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-16 09:11:14 -03:00
Caroline Cyr La Rose
42b3169120
Bug 31155: Document --since option in help of borrowers-force-messaging-defaults.pl
This patch adds a section about the --since option in the help section
of borrowers-force-messaging-defaults.pl

To test:
0) Apply patch
1) run the script with the --help option, e.g.
./misc/maintenance/borrowers-force-messaging-defaults.pl --help
--> There should be an explanation of the --since option with examples
    for specific and relative dates
2) Optionally, run the script with different options
--> The behaviour shouldn't have changed

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-08 13:12:45 -03:00
Bernardo Gonzalez Kriegel
99790c6c38
Bug 31292: Untranslatable string in sample_notices.yaml
This patch removes a regex that discard lines in multiline YAML values
On close inspection, there is no need for it.

To test:
1) go to misc translator, update some language
    ./translate update fr-CA

2) check missing string

    egrep "You may pick up your article" po/fr-CA-installer.po

   from sample_notices.yaml

3) apply the patch, repeat 1)

4) repeat 2), verify the string is present in the translation file

5) translate the new string, install the language,
   verify string is present in the translated file
    ./translate install fr-CA

   check fr-CA/mandatory/sample_notices.yml

There are three new strings
msgid "%sDear %s %s,%s"
msgid "%s%s%sTitle: %s"
msgid "%sYou may pick up your article at %s.%sYou can download the scanned materials via the following url(s): %s.%s"

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-08 13:10:56 -03:00
e4d2ef0f46
Bug 31249: Use cronlogaction on update_patrons_category.pl
This patch does what the title says.

To test:
1. Run the script
2. Check the action logs
=> FAIL: Boo, no record of the running cronjob
3. Apply this patch
4. Repeat 1-2
=> SUCCESS: Yay!
5. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Liz Rea <liz@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-05 11:53:15 -03:00
Bernardo Gonzalez Kriegel
160d96e176
Bug 28707: Translate RECEIPT notice
There is a badly crafted regex used when extracting
strings in multiline fields in yaml files
The regex is my own, introduced in Bug 24262, sorry.

This patch correct it a little. Better eyes are welcome.

To test:
1) go to misc translator, update some language
    ./translate update fr-CA

2) check missing strings

    egrep "Total out|Operator ID|August 31" po/fr-CA-installer.po

   first two are from sample notices, third from sample creator data

3) apply the patch, repeat 1)

4) repeat 2), verify the strings are present in the translation file

5) translate some of the new strings, install the language,
   verify strings are present in the translated files
    ./translate install fr-CA
   check fr-CA/optional/sample_creator_data.yml and
   fr-CA/mandatory/sample_notices.yml

There are some 60+ new strings.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-01 09:06:03 -03:00
45de338715
Bug 29697: Use flag embed_items
Includes:
    Bug 29697: (follow-up) Use flag embed_items

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-22 15:24:11 -03:00
70d61d80fb
Bug 29697: Replace GetMarcBiblio occurrences with $biblio->metadata->record
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

JD Amended patch:
-# FIXME Special case here

-    print "Biblio not found\n,";
+    print "Biblio not found\n";

- my $biblio = Koha::Biblio->find($hostbiblionumber);
+ my $biblio = Koha::Biblios->find($hostbiblionumber);

Rebased-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-22 15:24:11 -03:00
eef6b0c022
Bug 31157: (QA follow-up) Update descriptions and wording of configuration options
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-22 15:14:43 -03:00
19fc6220fa
Bug 31157: Allow selection for overdue from address in the UI
This patch adds a new system preference to allow systems librarians the
option to pick wich address to use for overdues notices.

We default to 'cron' to allow existing uses of '--frombranch' on the
command line to continue to function but now allow end users to override
this option via the new OverdueNoticeFrom preference.

Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-22 15:14:42 -03:00
85690a14f8
Bug 15187: Index 880 in Zebra the same as Elasticsearch
This patch modifies the MARC21 export to Zebra, so that 880 fields
are rewritten as their linked fields, in the same way that we
already do with Elasticsearch, so that the alternate graphic
representation of fields are indexed accordingly. (ie 880 $6245-01
Chinese titles will be indexed into the title index using the 245 rules)

Test plan:
0. Apply patch
1. Turn on ICU indexing
1b. vi /etc/koha/zebradb/etc/default.idx
1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml
1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml
1e. Restart Zebra server
1f. Re-index Zebra
2. Add record with a 880 $6 245-01 $a 教牧書信 field.
3. Search for this record using a title index with the Chinese title
4. Note that the record is correctly retrieved

(Note: This test probably works better using author or series as they
 present as links on the detail page which makes the fix more obviously
 useful.)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-22 15:09:46 -03:00
David Nind
d614a0199b
Bug 31122: Terminology - Replace occurences of 'Notices & slips' with 'Notices and slips'
Replace occurences of 'Notices & slips' with 'Notices and slips'
(replacing '&' and '&amp' with 'and'), as per the terminology
guidelines.

See the terminology list:
https://wiki.koha-community.org/wiki/Terminology

Test plan:
1. Find occurrences of 'Notices & slips':
   - git grep 'Notices &amp; slips' -- :^misc/translator/po
   - git grep 'Notices & slips'
2. Review places in the staff interface where 'Messages & slips' is
   displayed:
   - Tools home page
   - Tools > Notices & slips: page title and breadcrumb
   - Other breadcrumbs:
     . Tools > Notices & slips > New notice > [select any module]
     . Tools > Notices & slips > [select Edit for any notice]
     . Tools > Notices & slips > [select Delete for any notice]
3. Review other occurences:
   . The cron job description for misc/cronjobs/holds/holds_reminder.pl:
     misc/cronjobs/holds/holds_reminder.pl -man (scroll down to the
     description)
   . The TalkingTech README file:
     vi misc/cronjobs/thirdparty/TalkingTech.README
4. Apply the patch.
5. Re-run the grep queries from step 1 - no occurences are now found.
6. Review places where 'Notices & slips' was found in steps 2 and 3 -
   these should all be replaced with 'Notices and slips' and should
   read correctly.
7. Sign off!

Alernative: review the diff for the patch and check that occurences of
'&amp' and '&' are replaced with 'and' and the updated text reads
correctly.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-08 15:08:23 -03:00
2d64b3ce8a
Bug 30275: Rename issues.renewals to issues.renewals_count
Rename the issues.renewals field to renewals_count to prevent a method
name collision with the new relation accessor introduced by this
patchset.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-05 09:45:55 -03:00
Katrin Fischer
30062ccc20
Bug 26311: (QA follow-up) Update script documentation
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-05 09:20:51 -03:00
50810e233d
Bug 26311: (follow-up) Fix undefined values in sprintf
Avoid warns 'Use of uninitialized value in sprintf' by using
'0' if lower age is undefined
'unlimited' if upper are is undefined

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-05 09:20:51 -03:00
03b32e7a27
Bug 26311: (follow-up) loop through the patron with date of birth and in categories having age limits
Also added category limits in message, for example (12-18)

Check patron :
- with no date of birth
- with invalid age in category having age required
- with invalid age in category having upper age limit
- with invalid age in category having age required and upper age limit

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-05 09:20:50 -03:00
Katrin Fischer
0151d5cd1f
Bug 26311: (QA follow-up) Rephrase output message
This is a suggestion for rephrasing the message slightly:

Before:
* Patron borrowernumber=15 in category 'J' has invalid age '56'

After:
* Patron borrowernumber=37 has an invalid age of 37 for their category 'K'

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-05 09:20:50 -03:00
cbdc424749
Bug 26311: Add patron invalid age to search_for_data_inconsistencies.pl
Patron categories may have age limits.
Add to script misc/maintenance/search_for_data_inconsistencies.pl the list of patrons which age is invalid regarding there category.

Test plan :
1) Create an adult patron category limited to 18-99 years
2) Create a patron in the category
3) Edit in database its date of birth so that he is 17 years old
4) Run misc/maintenance/search_for_data_inconsistencies.pl
=> You see the patron

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-05 09:20:50 -03:00
bde3a32277
Bug 30889: (follow-up) Record and use context in background_jobs
This patch records the current context to the background_jobs table at
enqueue time and then uses that record to set the context at process
time.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-06-30 12:03:18 -03:00
44b74010b1
Bug 30889: Set interface to 'intranet'
Is that correct?

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-06-30 12:03:18 -03:00
0f2425a243
Bug 30889: Set userenv for background jobs
We need to set the userenv when we process the jobs. It is useful for
stats (at least)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-06-30 12:03:13 -03:00
6c315c164d
Bug 31058: Fix import AutoUnsuspendHolds
This patch corrects missing import

To test:
1 - perl misc/cronjobs/holds/auto_unsuspend_holds.pl
2 - It dies
Undefined subroutine &main::AutoUnsuspendReserves called at misc/cronjobs/holds/auto_unsuspend_holds.pl line 38.
3 - Apply patch
4 - run again
5 - It succeeds!

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-06-28 09:40:11 -03:00