Bug 29544: (QA follow-up) Simplify code

I think this is a better approach for the same thing. Posting it just in
case it helps.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29544: Fix opac-issue-note.pl

We must check if logged in user is trying to modify one of their
checkouts

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29544: Ensure logged in user is allowed to modify checkout note

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29541: Prevent users from another group to access patron's images

We should respect group restrictions here.

Test plan:
Create a patron from another group of libraries and don't let them
access info from patrons outside of this group.
Access the following link and confirm that you can see the image only
for patrons from their group
  /cgi-bin/koha/members/patronimage.pl?borrowernumber=XX

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29541: Restrict access to patron's image to borrowers => * and circulate => *

The patron images is displayed on the 'circulation' and 'members'
modules.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29540: Raise flagsrequired in modrequest

Test plan:
Try modrequest with a user having only 'catalogue' perms and the following URLs:
[1] /cgi-bin/koha/reserve/modrequest.pl?reserve_id=XX&CancelBorrowerNumber=XX&CancelItemnumber=XX&biblionumber=XX
    Fill the XXs with correct identifiers for some item level hold.
[2] /cgi-bin/koha/reserve/modrequest_suspendall.pl?suspend=1&suspend_until=2021-12-01&borrowernumber=XX
    Fill the XX with borrowernumber for borrower that has pending holds.
You should see: Error: You do not have permission to access this page.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[AMENDED] More consensus for using reserveforothers than circulate_remaining.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 15285ae209f5a98ab2e77c730b0b70ff0b29c283)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28735: Self-checkout users can access opac-user.pl for sco user when not using AutoSelfCheckID

This patch makes the sandboxing of the selfcheckout more robust by
adding a "sco_user" session variable which is turned on when
logging into the self-checkout (either by AutoSelfCheckAllowed or manually).

If a user with this session variable turned on tries to access
other parts of the system (like the rest of the OPAC), it will
"kick out", so that the browser user will lose the authenticated session.

Test plan:
1) Apply the patch
2) koha-plack --restart kohadev
3) Go to http://localhost:8080/cgi-bin/koha/sco/sco-main.pl
4) Note that you are logged into the self-checkout
     So you see the login screen specific to the self-checkout.
     To log with the actual patron. It's a nested auth.
5) Go to http://localhost:8080/cgi-bin/koha/opac-main.pl
6) Note that you are not logged into the OPAC
7) Log into the staff interface and disable the
system preference AutoSelfCheckAllowed
8) Log out of the staff interface (this step is very important)
9) Go to http://localhost:8080/cgi-bin/koha/sco/sco-main.pl
10) Note that you are prompted to log into Koha
11) Login using the "koha" user (when using koha-testing-docker)
12) Note that you are logged into the self-checkout
13) Go to http://localhost:8080/cgi-bin/koha/opac-main.pl
14) Note that you are not logged into the OPAC
      Without the patch you would still be logged as "koha"
15) Go back to http://localhost:8080/cgi-bin/koha/sco/sco-main.pl
16) Note that you will need to log in again as you've lost your
session cookie
      Without the patch you will still be logged in the self-checkout
Voila!

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 6d022889a2fac79c9148dd5f20c36f926d66065c)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 26102: Prevent XSS when To.json is used: unimarc_field_4XX.tt

To test, edit a MARC framework to link a subfield to the
unimarc_field_4XX.tt. The process of triggering the plugin and selecting
a search result from the plugin popup should work correctly.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit dbd13593538b8dbba9dfe9ff200b1d472ec0595b)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 26102: Prevent XSS when To.json is used: subscription-add.tt

Test the process of adding a subscription, entering both a valid vendor
ID and a non-existent vendor ID. The non-existent vendor ID should
trigger a validation alert.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 583aad8e48790443a14ac4b7dfe85fa1bdeb91a2)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 26102: Prevent XSS when To.json is used: guarantor_search.tt

To test, edit a patron record and go through the process of adding a
guarantor. In the guarantor search results table the address should be
displayed correctly.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 11d0a05eb9f1a13c07f3c56d8e40dbbd1bc43938)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 26102: Prevent XSS when To.json is used: catalogue/results.tt

To test, perform a search in the catalogue and verify that search term
highlighting works correctly.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 0de86fd323545796d57d2e289c10a33970050716)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 26102: Prevent XSS when To.json is used: authorities/blinddetail-biblio-search.tt

Test the process of searching for and selecting an authority record for
use in the basic MARC editor.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 57a2a82c504815d5d8e95c20be43611d96abcf13)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 26102: Prevent XSS when To.json is used: authorities/authorities.tt

Check that mandatory tags and subfields are correctly required when
editing an authority record.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit d9ae296b23d6897070c6bb788387ab39e7da8f09)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 26102: Prevent XSS when To.json is used: admin/preferences.tt

Test that preference search term highlighting works correctly.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 5df95693f93e1ef95f74eb4a118319e84ed7703e)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29543: Set autocomplete off for SCO login fields

Cardnumber already had it set, adding for username and password

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 25856b460e3041c2a825c83d1abf0f48c77a9448)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29543: Add Mojo::JWT dependency

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit f07a666c2eb6338f1b450db9dcdc75cfb0d76601)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29543: (follow-up) Add a warning to SelfCheckoutByLogin

This updates the language to warn users of risk if using cardnumber for login and auto-self-check is enabled

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit b32dbaa9cfc43ddd9404a094a2d82c85936c0ba2)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29543: Prevent user to checkin or renew items they don't own

Checkin or renew must be restricted to the items they own.

Test plan:
Create an item with barcode bc_1
Check it in to user A
Login to SCO with user B
Get the token using the browser dev tool, from the cookie
Hit (replace $JWT)
    /cgi-bin/koha/sco/sco-main.pl?jwt=$JWT&op=renew&barcode=bc_1
    /cgi-bin/koha/sco/sco-main.pl?jwt=$JWT&op=returnbook&barcode=bc_1

You should see an error message

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 839b7c4a5c8bdba62776fdb74c5f2125622a9ff0)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29543: Enforce authentication for self-checkout

The self-checkout feature is assuming a patron is logged in if patronid
is passed. It also assumes that "We're in a controlled environment; we
trust the user", which is terribly wrong!

This patch is suggesting to generate a JSON Web Token (JWT) to store in
a cookie and only allow action (renew, check in/out) is the token is
valid. The token is only generated once the user has been authenticated
And is removed when the user finish the session/logout.

Test plan:
You must know exactly how the self-checkout feature works to test this patch.
The 4 following sysprefs must be tested:
 SelfCheckoutByLogin, AutoSelfCheckAllowed, AutoSelfCheckID, AutoSelfCheckPass
Confirm that you can renew, checkin for the items you own, and checkout new items.
Confirm that you are not allowed to access other account's info.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 77e21f30062dc23edb2c79f609d854d553e67f7c)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29543: Add JWT token handling

Mojo::JWT is installed already, it's not a new dependency.
We need a way to send the patron a token when it's correctly logged in,
and not assumed it's logged in only if patronid is passed

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit d978bf1506d761a6962d949f35b71f1740d0052a)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29543: Remove inputfocus variable

It's not used in template

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 7ea2d7cd68bef8d59807f221a23a680361d24b1a)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29543: Remove borrower variable

It's not needed, we have $patron

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 4c398daaf5df0522e8c302eb342c1ec30d6a17a9)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28698: Fix news for all displays in all locations

Some news are used for custom text with a specific language, for example "opacheader".
The bug is that in these locations, news for all (staff and opac) are displayed.
This is because GetNewsToDisplay uses SQL :
  opac_news.lang = '' OR opac_news.lang = ?

This patch is a quick fix.
This feature maybe needs a big revamping.

Test plan :
1) Remove any news
2) Create a news with empty 'Display location' and some content
3) Display OPAC interface
4) without patch you see in location of 'opacheader', 'opacnavright' ... the content
   of the news
5) With patch you see content of the news only in news table
6) Create a news for opacheader
7) Check it appears in correct language

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
(cherry picked from commit 33369d2522c23230e9f755466b19f9c032629365)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29437: (QA follow-up) Update TODO in Breeding.t

We now have some tests for BreedingSearch.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 0b428c36db956965bad008f419047aac0b7608d8)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29437: (QA follow-up) Remove Business::ISBN from addbooks

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 0b02e9ed4cd42d16a1fdd0673e77f15ed8a21c0f)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29437: Search reservoir for term as title, author, or variations of ISBN

The code in the script and the module attempt to determine whether a term is an isbn, or not. Rather
than try to do this, we can simply search it on the three fields: isbn, title, author

Additionally, we should search as any of the ISBN variations to broaden our matches

Note: Curently only an ISBN 10 is stored in import biblios, so for an ISBN13 that doesn't convert
the value will be blank - this is another bug

To test:
1 - Perform a cataloging search for a valid ISBN 13 with no ISBN10 counterpart:
    9798200834976
2 - 500 error
3 - Apply patch
4 - Repeat, no results
5 - Import some records
6 - Search by title/author/isbn
7 - Confirm searching works as expected

WNC amended to fix spelling

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
AMENDED: Useless call of ISBNs (plural) when you only pass one parameter.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit b422b7af2392440c5a4ca83b0740dceb262b9cd4)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29437: Unit tests

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 4c2fc8eac541a4eef624e7be88844cf058bad398)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29319: Use Business::ISBN to check ISBNs on addbooks.pl/cataloging search

To test:
1. Go to cataloging search and enter something like "7th Heaven".
2. Get an error when searching, Koha thinks you entered an ISBN
3. Apply patch
4. Try the same search, it should be a proper title search now
5. Find some stuff in the catalog with ISBN numbers in them.
6. The search should properly return ISBN13/ISBN10 searches, without with out the '-'.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 14c8d4c230ff96932ab87099f7594f311ffc4562)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: (follow-up) Fix code that I copied from too

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit c32cca53bdfe07a3ee27ebac36c7aae3d51c68d9)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: (follow-up) Improve error logging to prevent warns

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit cf29caf39fed2d81262735fe7893a1ac3a8d59b1)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: Unit test

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit fa56c8b069b3d8e3aa4651d39e98db5f45883de8)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: Handle the case of an exclamation point in parentheses

This expands the regex to handle this specific case

To test:
1 - Load record created for last patch
2 - Note analytics error
3 - Apply patch
4 - Restart and reload
5 - No more errorm also no Analytics link
6 - Add a 773$t to a record with title used before:
    Digger does it all (not really!)
7 - reload the initial record
8 - See 'Show analytics' link
9 - Click the link
10 - You should end up on the record you added the 773 to

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 7c74eb91248cbdc383e5b3d0d1c48c4ae5d570a3)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: Don't die on analytics searching error

This patch adds an eval around the call to search for analytic records
It pases a value to the template on the staff side, but logs the warning on
the opac

This seems similar to 'decoding_error' which is noted on staff side, but absent on OPAC

The eval follows the patter used during searching

To test:
1 - Add a title to catalog, with 245a:
    Digger does it all (not really!)
2 - Set searchEngine preference to: Elasticsearch
3 - The record does not load
4 - Apply patch
5 - The record loads, there is a note about analytics at the top fo the record
6 - View record in opac, no note
7 - Check logs on intranet and opac, searching error is logged

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit ab13f33eaee2599beb376366524d92afa79df034)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: (QA follow-up) Make clean_search_term public

With all the work that's gone into improving the internal
_clean_search_term method I feel we should expose it publically as it's
going to be more widely helpful

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 641e8e4096b8d55afb534d593743154807f77221)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: escape ES ranges if QueryAutoTruncate is enabled

if QueryAutoTruncate enabled we will have any special operators ruined
for example: "test [6 TO 7]" will be converted to "test* [6* TO* 7]"
so no reason to keep ranges when QueryAutoTruncate set to "enabled"

1) enable QueryAutoTruncate at your sysprefs.
2) perform a search using range, for example: "[1999 TO 2020]",
it shouldn't work the way it's supposed to.
3) apply the patch.
4) perform the same search with range, ensure that it works correctly.

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit e84759dfee9a4323a17146b88aba9d717ca0bb75)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: avoid messing up regexes in the search queries

This patch ensures that the behavior with
QueryRegexEscapeOptions set to values other than
"Escape" still will works as expected.

It does so by storing the contents of regexes
before escaping special characters and
then restores the contents of regexes back to how
it was before, ensuring that searching with regex is possible.

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 59c26ce5f35f3f18830090b048d0cd6c2a1eb6fc)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: escape exclamation signs in the query

Currently having exclamation sign at the end of the query makes ES
search fail, and when you try to search for a book that has exclamation
sign in the tittle (something like "Words! words") won't show results
correctly as it tries to negate everything that is after exclamation
sign, making it impossible to search for books that have in in the title

This patch escapes exclamation signs if it's at the end of the query or
has a space after it, resolving both of the issues listed above.

To reproduce:
1) with ES enabled, search for the book with title that contains
exclamation sight at the end, like "book!", this search should result
 in error.
2) do another search, but this time find/prepare beforehand book with a
title that has exclamation sign with a space after it,
e.g "exclamation! sign", it shouldn't find it as ES treats everything
after that exclamation sign as negation.
2) apply the patch.
3) perform searches from the steep one and two again.
Search from step one should no longer fail, while search from the step
two should find that book.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit cb156ac13224f03db8ce0bd1373335b7d4052437)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: escape brackets in the search query

This patch screens square and curly brackets which have no special
language meaning.

To reproduce:
1) using ES, search for the book with title that contains
square and/or curly brackets, like "book [second edition]", which will
result in error.
2) apply the patch.
3) search for that book again, ensure that it works now.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 952b5a6a469460e926905d6582688dd903988aad)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: screen unquoted semicolons and all followup colons

Currently searches like: "book:", ":book" and "host-item:test:n"
cause internal server errors.

This patch adds additional regexes that remove the colons at the start
and end of the query, and another regex that screens all follow-up
colons that go after the first colon to avoid errors when searching for
"host-item:test:n".

To reproduce:
1) using ES, search for the book with title that contains
semicolon at the start or at the end of the line, separated with spaces,
this should cause internal server error.
2) try doing the same with something like "host-item:test:n", it should
result in error as well.
3) apply the patch.
4) repeat steps 1-2, ensure that it works now.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 5b4b14e493fef9f8f84060b6b5d83fbdcccfc65e)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: add tests

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 405812b407f77de2d60c8f4534728399068789d8)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Update release notes for 20.11.13 release

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Increment version for 20.11.13 release

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Fix translations for Koha 20.11.13

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Translation updates for Koha 20.11.13

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 27801: Fix javascript price calculations

This patch ensures we're formatting the price values consistently for
the table total and the amount to pay input field.

Test plan
1) Add an item to charge at 0.10.
2) Add this same item 8 or 9 times (Do not use the 'quantity')
3) Note that the table total and the 'Amount paid' values do not match
4) Apply the patch and repeat the above steps.. the values should now
   match.

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Sally <sally.healey@cheshiresharedservices.gov.uk>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 7e72a7dde5a73ed6177b321d29fca0df13f08f2c)

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
(cherry picked from commit 9e16f33b37aaf107ff3293df0cd3da5f94563c81)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: REVERT ALL: due to string freeze

The follow-up bug 29284 introduces a new string.
Both patchsets will be backported for next release.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: REVERT ALL: due to string freeze

Will be backported for next release

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: (follow-up) Fix code that I copied from too

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit c32cca53bdfe07a3ee27ebac36c7aae3d51c68d9)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: (follow-up) Improve error logging to prevent warns

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit cf29caf39fed2d81262735fe7893a1ac3a8d59b1)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: Handle the case of an exclamation point in parentheses

This expands the regex to handle this specific case

To test:
1 - Load record created for last patch
2 - Note analytics error
3 - Apply patch
4 - Restart and reload
5 - No more errorm also no Analytics link
6 - Add a 773$t to a record with title used before:
    Digger does it all (not really!)
7 - reload the initial record
8 - See 'Show analytics' link
9 - Click the link
10 - You should end up on the record you added the 773 to

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 7c74eb91248cbdc383e5b3d0d1c48c4ae5d570a3)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: Unit test

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit fa56c8b069b3d8e3aa4651d39e98db5f45883de8)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29284: Don't die on analytics searching error

This patch adds an eval around the call to search for analytic records
It pases a value to the template on the staff side, but logs the warning on
the opac

This seems similar to 'decoding_error' which is noted on staff side, but absent on OPAC

The eval follows the patter used during searching

To test:
1 - Add a title to catalog, with 245a:
    Digger does it all (not really!)
2 - Set searchEngine preference to: Elasticsearch
3 - The record does not load
4 - Apply patch
5 - The record loads, there is a note about analytics at the top fo the record
6 - View record in opac, no note
7 - Check logs on intranet and opac, searching error is logged

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit ab13f33eaee2599beb376366524d92afa79df034)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: (QA follow-up) Make clean_search_term public

With all the work that's gone into improving the internal
_clean_search_term method I feel we should expose it publically as it's
going to be more widely helpful

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 641e8e4096b8d55afb534d593743154807f77221)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: escape ES ranges if QueryAutoTruncate is enabled

if QueryAutoTruncate enabled we will have any special operators ruined
for example: "test [6 TO 7]" will be converted to "test* [6* TO* 7]"
so no reason to keep ranges when QueryAutoTruncate set to "enabled"

1) enable QueryAutoTruncate at your sysprefs.
2) perform a search using range, for example: "[1999 TO 2020]",
it shouldn't work the way it's supposed to.
3) apply the patch.
4) perform the same search with range, ensure that it works correctly.

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit e84759dfee9a4323a17146b88aba9d717ca0bb75)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: avoid messing up regexes in the search queries

This patch ensures that the behavior with
QueryRegexEscapeOptions set to values other than
"Escape" still will works as expected.

It does so by storing the contents of regexes
before escaping special characters and
then restores the contents of regexes back to how
it was before, ensuring that searching with regex is possible.

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 59c26ce5f35f3f18830090b048d0cd6c2a1eb6fc)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: escape exclamation signs in the query

Currently having exclamation sign at the end of the query makes ES
search fail, and when you try to search for a book that has exclamation
sign in the tittle (something like "Words! words") won't show results
correctly as it tries to negate everything that is after exclamation
sign, making it impossible to search for books that have in in the title

This patch escapes exclamation signs if it's at the end of the query or
has a space after it, resolving both of the issues listed above.

To reproduce:
1) with ES enabled, search for the book with title that contains
exclamation sight at the end, like "book!", this search should result
 in error.
2) do another search, but this time find/prepare beforehand book with a
title that has exclamation sign with a space after it,
e.g "exclamation! sign", it shouldn't find it as ES treats everything
after that exclamation sign as negation.
2) apply the patch.
3) perform searches from the steep one and two again.
Search from step one should no longer fail, while search from the step
two should find that book.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit cb156ac13224f03db8ce0bd1373335b7d4052437)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: escape brackets in the search query

This patch screens square and curly brackets which have no special
language meaning.

To reproduce:
1) using ES, search for the book with title that contains
square and/or curly brackets, like "book [second edition]", which will
result in error.
2) apply the patch.
3) search for that book again, ensure that it works now.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 952b5a6a469460e926905d6582688dd903988aad)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: screen unquoted semicolons and all followup colons

Currently searches like: "book:", ":book" and "host-item:test:n"
cause internal server errors.

This patch adds additional regexes that remove the colons at the start
and end of the query, and another regex that screens all follow-up
colons that go after the first colon to avoid errors when searching for
"host-item:test:n".

To reproduce:
1) using ES, search for the book with title that contains
semicolon at the start or at the end of the line, separated with spaces,
this should cause internal server error.
2) try doing the same with something like "host-item:test:n", it should
result in error as well.
3) apply the patch.
4) repeat steps 1-2, ensure that it works now.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 5b4b14e493fef9f8f84060b6b5d83fbdcccfc65e)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 28316: add tests

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 405812b407f77de2d60c8f4534728399068789d8)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug ?????: Compile SCSS, forgotten in a previous release cycle

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29330: (QA follow-up) Change to message/rfc822

This content-type might be more appropriated to use as a temporary
label for the serialized email message with attachments.

WARNING: perl -cw tells you that the constant is redefined. This has
to do with an already existing module dependency loop of Letters.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 53127abc0377bc3f04364c4183baacef3d4c95c4)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29330: (QA follow-up) Use passed MIME type

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended: tidied.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 6c961026789ef1dfdd3ebd4ed18469fef7bd649b)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29330: Restore handling of serialized MIME messages in message_queue

This patch changes how multipart MIME messages are handled on the
message_queue table.

The email, with the required attachments, is now generated using
Koha::Email, and serialized using Koha::Email->as_string.

This bug also adds Koha::Email->new_from_string which is used to read
that data from the DB, and produce a Koha::Email object, that can be
further augmented/modified using regular Koha::Email methods.

This implementation should be considered a middle ground, with
backportability in mind. higher-level methods should encapsulate setting
the default headers and addresses, to clean the area a bit further.

Preparation:
- You need a valid SMTP configuration in koha-conf.xml. If you use Gmail
  you can generate an 'app password' and set things like this:

 <smtp_server>
    <host>smtp.gmail.com</host>
    <port>587</port>
    <timeout>5</timeout>
    <ssl_mode>STARTTLS</ssl_mode>
    <user_name>youraddress@gmail.com</user_name>
    <password>youpassword</password>
    <debug>1</debug>
 </smtp_server>
- Set KohaAdminAddress to your address.

To test:
1. Pick a patron. Make sure it doesn't have any email address (Acevedo?)
2. Set an overdue notice trigger for its category
3. Check something out, with due date in the past to force an overdue
4. Run:
   $ kshell
  k$ misc/cronjobs/overdue_notices.pl -v
  k$ exit
   $ koha-mysql kohadev
   > SELECT * FROM message_queue WHERE borrowernumber=the_borrowernumber;
=> SUCCESS: A notice has been created
5. Run:
   $ kshell
  k$ misc/cronjobs/process_message_queue.pl --verbose
=> SUCCESS: SMTP is ok => Email is sent
=> FAIL: Your inbox shows an email with weird content
6. Apply this patches
7. Run:
   $ koha-mysql kohadev
   > DELETE FROM message_queue;
8. Repeat 4 and 5
=> SUCCESS: You got an email with an attachment!
=> SUCCESS: The attachment contains an email that couldn't be delivered!
9. Try all the things that enqueue messages :-D
=> SUCCESS: No behavior change
10. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Edited the POD, restoring a few lines that describe the needed hash
keys of the attachments.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 54546da3f07380375470bc130f15a33830419ac8)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Bug 29330: Add Koha::Email->new_from_string

This patch introduces a new method to Koha::Email. This method allows us
to parse a MIME email to initialize the Koha::Email object. This is
particularly important when we are restoring emails from the DB. i.e.
from the *message_queue* table.

To test:
1. Apply this patch
2. Run:
   $ kshell
  k$ prove t/Koha/Email.t
=> SUCCESS: Tests pass! Koha::Email->new_from_string is the correct
counterpart for Koha::Email->as_string!
3. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 2d6a189e89cc81eff50cdd3a71f7eacc34287b3a)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Update release notes for 20.11.12 release

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Increment version for 20.11.12 release

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Fix translations for Koha 20.11.12

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Merge remote-tracking branch 'translate/20.11.12-translate-20211130' into 20.11.x

Translation updates for Koha 20.11.12

Bug 29564: Use List::MoreUtils so SIP U16/Xenial does not break

 - run prove t/00-load.t, see error
 - apply patch

    00:07:08.189 koha_1       | #   Failed test 'use C4::SIP::Sip::Configuration;'
    00:07:08.189 koha_1       | #   at t/00-load.t line 46.
    00:07:08.189 koha_1       | #     Tried to use 'C4::SIP::Sip::Configuration'.
    00:07:08.189 koha_1       | #     Error:  "uniq" is not exported by the List::Util module

 - run prove t/00-load.t, see tests pass :0)

https://bugs.koha-community.org/show_bug.cgi?id=29478

https://bugs.koha-community.org/show_bug.cgi?id=29564

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit fb2e8e3450963aa3bdfee4fa1ddd9837e4224732)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29524: (bug 28935 follow-up) Restore modification of some patron's attributes

On
  commit 5f37d8d2f496ce3c9fd6dfd5a2efa7a9fe435af3
  Bug 28935: No filtering on patron's data on member entry pages
we restricted the list of the columns from the borrowers table that can
be modified from the patron edit view.
We were too restrictive, the following 3 attributes can be edited from
this form: privacy_guarantor_fines, privacy_guarantor_checkouts,
checkprevcheckout and lang

Test plan:
Turn on the following prefs:
- AllowStaffToSetFinesVisibilityForGuarantor
- AllowStaffToSetCheckoutsVisibilityForGuarantor
- CheckPrevCheckout (set to 'unless overridden *')
- TranslateNotices
Edit a patron and see the 4 different options are now displayed.
Change their value, save, edit again
Confirm that the values have been saved

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 0ca1419b26be84f7670f60446e621cfd878f6580)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28803: (20.11.x) Add invalid email handling in _send_message_by_email

Quickfix version for 20.11.x because of missing dependancies.
Based on Bug 28996.

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28803: Add unit tests

This patch updates Letters.t to confirm that invalid email addresses in
the message_queue should not throw an exception when sending mail but
instead set the status to failed and pass error details to the end user.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

28870: (20.11.x) fix UT on SendQueuedMessages

Taken from Bug 28803 :
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=125345&action=diff#a/t/db_dependent/Letters.t_sec1

Bug 29341: DBIC schema changes

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29341: DBRev 20.11.11.002

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit c2430c6ee0d90753bc55700e01f159ec259f7b0d)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29341: Remove foreign keys on pseudonymized_transactions

Behave like the statistics table and don't remove the code even if the
branch or patron's category is removed.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit f4c61260f537fad0aac5179b2a5ae25b56070703)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29405: Fix date_renewed attribute format in patron spec

This patch just adds the 'date' format string to ensure we are
validating the input/output of the date_renewed field correctly.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit e165f57f248fbb5166834d5117caef4fddf3c3e3)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29452: remove unnecessary warns in the logs

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 8c8c7d77cfb220a7317739a4d84af48a466e569b)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29321: Remove a last without loop context

This reads better when converted to regular if.
Note that last within such a block is allowed in Perl, but it feels
better to use it only in a loop context.

Test plan:
Read the patch ;)
Bonus: Test if you enter the block by using pref OPACSuggestionMandatoryFields
when adding a suggestion from opac.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit: Added a space between 'if' and '(' :-D

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 0144bb310ee207a7fde2091c8ada9c1ec11ecb50)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 27145: Wrap delete and move_to_deleted in a transaction

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 90a4c9d878a523cbc161d302be7bb23e8d82cab5)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 27145: Rethrow all other exceptions

Bug 14708 introduced a try catch around $patron->delete in commit:
  "Bug 14708: (QA follow-up) Use try/catch blocks when calling"

However, in the catch block it only assumes the exception was from trying to
delete anonymous patron when it can be anything else as well, the code should
be modified so that it will handle the anonymous patron case and if it is
anything else we log the other exception.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit df47a2e195e0d415e190b7bc5bc13504ec8be0c0)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28627: Calculate unitprice if not set

On bug 23376 the developer was too clever by half. Instead of passing
values to the template we moved to simply passing the order object.

The calculations that populated the unitprice, however, were simply dropped.

This patch restores the behaviour of setting unitprice to the estimated cost
by default. We do this conditionally in the template

To test:
 1 - Create a basket and adding items to it. Set the 'Vendor price' to
    '20', do not add an 'Actual cost'
 2 - Close the basket
 3 - Select 'Receive shipment'
 4 - Set a value for vendor invoice
 5 - Receive the order you created
 6 - Observe 'Actual cost' is = 0.00
 7 - Apply patch
 8 - Refresh the order receival page, and confirm the 'Actual cost' =
    20.00
 9 - Cancel receipt - reopen basket - edit order - add an actual cost - close
10 - Receive the order again
11 - Confirm the actual price is used
12 - Repeat this whole plan, but ensure the vendor has a different value for
     Invoice prices: Include tax

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 4755db8a1dfc6fcf856d55369ea102b3571f1625)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29255: Built-in offline circulation broken with SQL error

This patch makes a minor correction to one of the queries used when
synchronizing transactions in the built-in offline circulation system.
This fixes the error:

DBD::mysql::st execute failed: Column 'branchcode' in field list is
ambiguous at /kohadevbox/koha/offline_circ/download.pl line 84

To test you must be using a browser which still supports
applicationCache (Firefox before version 81, Chrome before version 94).

- Apply the patch and restart services.
- Enable the AllowOfflineCirculation system preference.
- Go to Circulation -> Built-in offline circulation interface.
- Click "Synchronize."
- Click the "Download records" button.
- After the page refreshes you should see updated dates where it lists
  "last synced" information.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 6501ff67ac1b7942f424bed458314efb0132bd68)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29264: SIP config allows use of non-branchcode institution ids causes workers to die without responding

If is entirely possible to create an SIP institution whose ID does not match a valid branchcode in Koha's SIP config. In fact, Koha's example SIP config contains an example of this ( kohalibrary / kohalibrary2 ).

If a SIP login uses an institution with an id that doesn't match a valid branchcode, everything will appear to work, but the SIP worker will die anywhere that Koha gets the branch from the userenv and assumes it is valid.

The repercussions of this are that actions such as the checkout message simply die and do not return a response message to the requestor.

At the very least, we should output a warning to the SIP log.

I think we should strongly consider disallowing institution ids in the SIP config that do not match valid branchcodes. In this scenario, attempting to start the SIP server should result in a error message with the SIP server exiting immediately.

Test Plan:
1) Apply this patch
2) Make a sip login that uses an instution whose id is *not* a valid branchcode
3) Start the SIP server
4) Check sip.log, you should see a warning similar to the following:
[2021/10/18 12:18:29] [2068079] [ERROR] ERROR: Institution kohalibrary does does not match a branchcode. This can cause unexpected behavior. C4::SIP::Sip::siplog /kohadevbox/koha/C4/SIP/Sip.pm (220)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 8e1f0cfc8dd71c507af83f9f8033ba9c84d0b293)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29204: (QA follow-up) Fix variable name

The patch "Bug 29204: Fix reports/issues_stats.pl for MySQL 8"
accidentally changed the variable name from $strsth2 to $strsth

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 5d2f51f9ad9ec5f46cc7a5ef2264ad9c5a188199)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29204: Fix reports/issues_stats.pl for MySQL 8

Error was: Incorrect DATETIME value: ''

Test plan
Go to Reports -> Circulation
On Period select "Row" or "Column" and choose a date
Submit

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 9a4b7b1948ecba51c942bdbb6b202981a6cce144)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28870: non-FQDN addresses are valid

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 8883409e0acd4f703d765389128c8dffe3ff9052)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28870: Use Email::Address->parse

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit d35c9e5df7743382b023a732c584078d784767d2)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28870: Move email address validation to a specific class method

To ease testing and future changes if needed.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 6d5a0bd832d546377910d3bdbb2bf790c4d2f0c3)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28870: Remove traces of Email::Valid

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 171c8421aa8c2838479789ab76b977e4fed10f9c)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28870: Use Email::Address to validate email addresses

This patch adds a new dependency, Email::Address. It is used in
Koha::Email to replace the current use of Email::Valid, which proved to
be problematic when it comes to UTF-8 characters.

Email::Address provides suitable regexes that -when used- keep our
tests passing, but also deal better with UTF-8 data.

To test:
1. Apply the regression tests patch
2. Notice the only change is that it tweaks a couple addresses so they
   contain umlauts and also have the "Description <address>" format that
   is used when sending carts.
3. Run:
   $ kshell
  k$ prove t/Koha/Email.t
=> FAIL: Tests fail! Things die because Email::Valid doesn't like the
   from we passed.
4. Run:
   $ sudo apt install libemail-address-perl
5. Apply this patch
6. Repeat 3
=> SUCCESS: Tests pass!
7. Try what is described in comment 1
=> SUCCESS: Things are back to normal
8. Sign off :-D
9. Send cookies

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 5d612707f017899f8b2152985a4fafdadbd2159a)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28870: Regression tests

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit e6ff06d263f5728c400dfb3de8f9704f06499a4c)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29195: Compiled CSS

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29195: Fix highlighting on odd rows in circ-patron-search-results

This patch adds "!important" to the CSS rule governing the background
color of table cells when hovered. It only applies to tables with the
"selections-table" class.

Although "!important" is not usually recommended, in this case the
appliation is narrow enough that I think it outweighs the complicated
selector that would be required to make it work otherwise.

To test, apply the patch and rebuild the staff interface CSS
(https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client).

- Locate a record in the catalog and being the process of placing a
  hold.
- When you're asked to enter a card number or search by name, search
  using a name which will return multiple results.
- In the table of results, hovering the mouse over the table should
  highlight the row in yellow.
- Perform the same test when searching for a club by name.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit fb650607032c851471610873a3065b60d2aaffca)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 29416: (bug 28941 follow-up) Restore suggestion prefill

From commit 586bed1319592e05f5dc3acf64a1dba8cae69d6b
     Bug 28941: Filter suggestion inputs at the OPAC

But later in the script we send { suggestion => %$suggestion } to the
template and so erase the variable previously passed.

Test plan:
Login at the OPAC
Go to the bibliographic detail page
Suggest for purchase
=> The form must be prefilled!

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit fbd3d2d762cd2889d6207770ebe5f96f34b727c9)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 27708: (QA follow-up) Add a few tests for Koha::Edifact::Order

1. This adds a simple regression test to make sure order_line() method
executes correctly with basket create_items set to "ordering" and
"receiving".

2. A simple test for the filename method is added

To test:
 1) prove t/db_dependent/Koha/Edifact/Order.t

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Petro Vashchuk <stalkernoid@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 8c1ea4ec8ab6509c67538b1bf78cdf9e44d60c96)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 27708: Unify two item object creation blocks to be stored as hash

Previously for existing item data was stored as an object reference and
then treated as one, but for not yet existing item data was stored as
keys in hash reference in the same variable and later it was treated
like an object, hence why it crashed with "no method".
This patch unifies that variable in both cases filled as hash and
treated as such.

To reproduce:
    1) Go to "Administration->System preferences" and change
"AcqCreateItem" to "receiving an order."
    2) Now, go to "Acquisitions" and create a new Vendor,
or use an existing one.
    3) Next, go to "Administration->EDI Account" and add EDI account
(pick that Vendor that you created recently, or the one that you will
use for this test).
    4) Also in "Administration->Library EANs" add EAN if you didn't
have one previously.
    5) Go back to "Acquisitions" and add a new basket to your Vendor
that you will use for this test.
    6) Press "Create EDIFACT order" button. It should throw
"Can't call method "homebranch" on unblessed reference..."
software error.
    7) Apply the patch.
    8) Reload the page that threw software error previously
(or repeat steps 5, 6 if you need another basket),
it should go through now.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit e429519974bb7bd8125fa0194bb7a7a004c638ee)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28768: Fix borrowernumber from opac-readingrecord.tt

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit d5c7f3b3079f654dd1ec15da7f5300a6bd642a95)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Bug 28768: (QA follow-up) Remove unused variable

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 9f43e623c26386fbd3b7404a5cb2bef60bd5129b)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>