This set of patches makes it possible to protect patrons from being accidetally
deleted or merged with other patrons, from the UI and from (well behaved) cron
jobs. The following subroutines are affected:
- Koha::Patron::delete
- Koha::Patron::merge_with
- Koha::Patron::safe_to_delete
- C4::Members::GetBorrowersToExpunge
Please note:
- This does not intend to protect patrons from being edited, only from being
deleted
To test:
* Tests
- Run the affected tests:
prove t/db_dependent/Members.t
prove t/db_dependent/Koha/Patrons.t
* Editing protected status and manual deletion
- Add a new user, note the presence of the "Protected" field under "Library
management", but leave it at the default "No", for now.
- Note that "Protected" is displayed in the "Library use" section of the patron
details.
- Note that More > Delete is avaiable as an action when the patron is saved
- Edit the user and set "Protected" to "Yes"
- Note that More > Delete is now disabled, with a note that the patron is protected
* Batch patron deletion
- Go to Tools > Batch patron deletion and anonymization
- Check the box for "Verify you want to delete patrons"
- Choose the category of your protected patron for "whose patron category is"
and click "Next" to run the actual deletion
- Check that your protected patron was not deleted
* Merging patrons
- Make sure you have two patrons with similar names or the same category, so
you can find them with one search. One should be protected, one not.
- Search for the patrons, tick their boxes and click on "Merge selected patrons"
- Select one of the patrons as the "patron to keep".
. Click on "Merge patrons"
- "No valid patrons to merge were found" should be shown
- Repeat this with the other patron as the "patron to keep"
(A future enhancement could be to not allow a protected patron to be selected for
merging in the first place.)
* misc/cronjobs/delete_patrons.pl
- Make sure you have a protected patron, in a category with at least one more
patron.
- Run something like this (at least in ktd):
$ perl misc/cronjobs/delete_patrons.pl --category_code <code> -v --confirm
(Replace <code> with the actual categorycode.)
- Make sure the borrowernumber of the protected patron is not mentioned in the
output of the script.
- Check the protected patron was not deleted
- Check the non-protected patrons were deleted
* REST API (with ktd)
- Make sure you still have a protected patron, and note their borrowernumber
- Enable RESTBasicAuth and restart all the things
- Run these two commands from the command line on the host:
$ curl -u koha:koha --request GET "http://localhost:8081/api/v1/patrons/54"
$ curl -u koha:koha --request DELETE "http://localhost:8081/api/v1/patrons/54"
(Replace 54 with the actual borrowernumber of your protected patron.)
- The first curl command should give you the patron details. The second should
give this output:
{"error":"Protected patrons cannot be deleted","error_code":"is_protected"}
There could be more functions/scripts where patrons are deleted that I have not
thought about. Please report them on the bug if you find any!
Update 2023-10-19: Fix "More > Delete" on patron, so link can not be clicked.
Update 2023-10-19: Rebase
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
For some reasons, with Search::Elasticsearch 8.0 JSON::true is
interpreted as 1 and Elasticsearch will always respond that it found "at
least 1 record". Something like this:
{
total => {
value => 1,
relation => 'gte'
}
}
Replacing JSON::true by \1 works with every version of
Search::Elasticsearch I tested (6.80, 7.717, 8.0)
Also worth noting:
Search::Elasticsearch will stop being supported by Elastic
https://github.com/elastic/elasticsearch-perl/issues/220
We might need to find an alternative for future versions of
Elasticsearch
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Elasticsearch 8 removed the support of types. This patch adapts the
requests accordingly.
With this patch, Koha will still be compatible with Elasticsearch 7.x
but will no longer work with Elasticsearch 6.x
Since Elasticsearch 6.x is no longer maintained, this should not be a
problem.
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html
Tested with Search::Elasticsearch 6.80 and 7.717
Test plan:
1. Install Elasticsearch 8
(use docker image: docker.io/koha/elasticsearch-icu:8.x)
2. Change the elasticsearch server location in $KOHA_CONF
3. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
It should fail
4. Apply the patch
5. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
It should end with "Total XXX records indexed"
6. Try to search some biblios and verify that it works the same as
before
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 05cf3d9f6a7ddf163d2e144436a6ff8e5854d7e4)
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Tidy
$c->validation in API controllers
File permissions
.inc file filters
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
prove t/db_dependent/api/v1/erm_custom_reports.t
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Some report types include access_type and yop headers for each usage statistics
The hierarchy is as follows: yop - access_type - metric_type
I.e. for each yop there are report lines for access_type and then for each access_type there are also report lines for metric types
This patch adds helper functions to populate the report rows based on this hierarchy and the report data retrieved from the database
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Improve robustness of unique matching here, to make sure we always match for same title if the case,
Some report rows may correspond to the same title as the previous row but have an empty match field, or alternatively come with a filled match field (e.g. DOI or Print_ISSN in TR_J4. Because of this we only verify a uniqueness match field if both current row and previous have it non-empty, otherwise we keep checking the remaining uniqueness match fields.
Example of this use-case, COUNTER report:
title | publisher | platform | Proprietary_ID | Print_ISSN | DOI | YOP | usages
examplet | examplep | examplepl | 1 | 123 | | 2020 | usages
examplet | examplep | examplepl | 1 | 123 |someDOI | 2021 | usages
The above 2 rows is the same title, same publisher, same proprietary_id,
same Print_ISSN, etc. It just so happens that one was returned by SUSHI
with DOI and the other wasnt.
These 2 rows correspond to different usage statistics by YOP, not 2
different usage_titles
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Some COUNTER files have long data in these fields and were causing new
database entries to fail
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This is doing a few things, checking for sushi errors:
1) Abort background job and provide error message if SUSHI returned
{Severity} in response, this means SUSHI error, docs here:
https://app.swaggerhub.com/apis/COUNTER/counter-sushi_5_0_api/5.0.2#/SUSHI_error_model
2) Abort background job and provide error message if SUSHI provided
proper response, but contains Exceptions in Report_Header
3) Abort background job and provide error message if SUSHI returned 0
report items (e.g. everything is fine but there are no items for the
harvest period provided
4) Abort background job and provide error message if COUNTER file
created is larger than allowed by the database.
5) Provide error message if manual upload file size exceeds the max
allowed by the database
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Remove obsolete total_records code, this was a leftover from when this code was in UsageDataProvider.pm
Add a new job_callback 'added_usage_objects' to handle total objects added (titles, platforms, databases, items)
Update job report include file
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The cronjob always logs a dry run even on a live harvest and the end date parameter was not working correctly. This patch addresses both issues
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Currently a harvest fails completely if a single title fails to harvest. Rather than failing the entire job, this patch introduces try catch blocks to stop this and allow the job to continue. Failed imports are recorded and displayed in the background job UI on job completion
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Rather than fetching the counter files and embedding the counter logs, we now add a foreign key to the data provider in the counter logs table and fetch them directly.
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 9f384df2ad5f4c037b052e8914d7e4b3e5f26670)
(cherry picked from commit 02c9a57af9f27cf528a2eb924876037890cf5504)
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Currently there is no option to go back to using the registry API to create a provider once "Create manually" has been clicked
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We need to check whether the url ends in a trailing "/" and also whether it ends in "/reports" for a SUSHI harvest. Two new functions are added to manage this and prevent failed harvests.
report_type is also now lowercase in the url and the dates are truncated to YYYY-MM format
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Moves a lot of the logic for creating a counter file and parsing a SUSHI response to a new class to simplify logic
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The counter registry offers an API that provides SUSHI information for each provider. This will be incredibly useful for creating new providers as we can use the correct information from the registry for harvesting urls etc. This reduces the risk of user input errors when creating providers and gives greater reliability to the data required to successfully harvest from the SUSHI API
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
ErmSushiHarvester background job will now be initiated by either
enqueue_counter_file_processing_job or by enqueue_sushi_harvest_jobs
from Koha/ERM/UsageDataProvider.pm, the former if triggered by a manual
file upload, the latter if by the 'run now' button or by the cron script.
This commit also includes some rewording/refactoring, namely:
- COUNTER file validation now happens in the API, before enqueuing the job.
- Removal of no longer used POST /erm/counter_files endpoint
- Koha/ERM/UsageDataProvider.pm:
-- run method is now enqueue_sushi_harvest_jobs
-- new enqueue_counter_file_processing_job method
-- harvest method is now harvest_sushi
-- new set_background_job_callbacks method to set the background job callbacks
- REST/V1/ERM/UsageDataProviders
-- run method is now process_SUSHI_response
-- new process_COUNTER_file endpoint
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 9339eed9358301f7bf17934e2e13fc17205d9cd0)
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Currently the start and end dates in the summary tab are based on the earliest and latest harvest run, rather than the earliest and latest data harvested. This should be changed so that we can see the period of data harvested for each provider
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Remove the front end processing in favour of doing this in the backend and passing it in the API response
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In the reports builder you can select which columns you would like to display in a report. Not all columns are applicable to all data types so it needs limiting to only be selectable if that column is available for that data type
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Now that we are harvesting more data types, "titles" search no longer applies and should become a keyword search able to work for any data type
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Needed to display lists in the provider tabs in the UI
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
