This patch adds a new background job for pseduonymization
To test:
1 - Download the benchmarking script
2 - Ensure your koha has bcrypt_settings in koha-conf.xml
See bug 28911
3 - perl perf_check_pseudo.pl
4 - Note the slowdown after pseudonymization enabled
5 - Apply patches, restart all
6 - perl perf_check_pseudo.pl
7 - Note improvement
8 - Enable pseudonymization in sytem preferences
9 - Perform some checkouts and returns
10 - Verify the background jobs complete successfully
11 - Verify the pseudonymized_transactions table is updated correctly
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Tidy
$c->validation in API controllers
File permissions
.inc file filters
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This is doing a few things, checking for sushi errors:
1) Abort background job and provide error message if SUSHI returned
{Severity} in response, this means SUSHI error, docs here:
https://app.swaggerhub.com/apis/COUNTER/counter-sushi_5_0_api/5.0.2#/SUSHI_error_model
2) Abort background job and provide error message if SUSHI provided
proper response, but contains Exceptions in Report_Header
3) Abort background job and provide error message if SUSHI returned 0
report items (e.g. everything is fine but there are no items for the
harvest period provided
4) Abort background job and provide error message if COUNTER file
created is larger than allowed by the database.
5) Provide error message if manual upload file size exceeds the max
allowed by the database
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Remove obsolete total_records code, this was a leftover from when this code was in UsageDataProvider.pm
Add a new job_callback 'added_usage_objects' to handle total objects added (titles, platforms, databases, items)
Update job report include file
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Currently a harvest fails completely if a single title fails to harvest. Rather than failing the entire job, this patch introduces try catch blocks to stop this and allow the job to continue. Failed imports are recorded and displayed in the background job UI on job completion
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
ErmSushiHarvester background job will now be initiated by either
enqueue_counter_file_processing_job or by enqueue_sushi_harvest_jobs
from Koha/ERM/UsageDataProvider.pm, the former if triggered by a manual
file upload, the latter if by the 'run now' button or by the cron script.
This commit also includes some rewording/refactoring, namely:
- COUNTER file validation now happens in the API, before enqueuing the job.
- Removal of no longer used POST /erm/counter_files endpoint
- Koha/ERM/UsageDataProvider.pm:
-- run method is now enqueue_sushi_harvest_jobs
-- new enqueue_counter_file_processing_job method
-- harvest method is now harvest_sushi
-- new set_background_job_callbacks method to set the background job callbacks
- REST/V1/ERM/UsageDataProviders
-- run method is now process_SUSHI_response
-- new process_COUNTER_file endpoint
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This commit is a squash of the following:
SUSHI harvesting process in the data providers class:
* Builds the URL query and requests the SUSHI service endpoint
* Parses the JSON response and builds the csv COUNTER file and adds it to counter_files table
Usage statistics data processing:
* When a counter_files entry is stored, CounterFile.pm will:
* Parse the csv COUNTER file and
* Add a usage_titles entry for each unique title in the COUNTER file
* Add the title's respective erm_usage_mus (monthly usage) entries, repeating for each metric_type
* Add the title's respective erm_usage_yus (yearly usage) entries, repeating for each metric_type
Harvesting cronjob;
'Run now':
* API endpoint to start the harvesting process of a data provider
* Button in the data providers list to run the harvesting process for each data provider upon clicked
ERM SUSHI: Background job
Job progress is updated to total amount of usage titles after retrieving
the response from SUSHI;
Job warning and success messages are added accordingly
Redundant duplicate titles will not be added
Redundant duplicate monthly and yearly usage statistics will not be added
Data provider harvest background job harvests once per report_type
Enqueue one background job for each report_type in the usage data provider
Update the way we measure progress in the background job.
It now uses the COUNTER report body rows instead of SUSHI response results.
We're now incrementing and showing the number of skipped mus, skipped
yus, added mus and added yus
There's a bug in the way we calculate yus
Updates to background job progress bar - Depends on 34468
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch updates icons which are listed as having changed names in
this document:
https://fontawesome.com/docs/web/setup/upgrade/upgrade-from-v4
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
* h2 h2 directly following each other make no sense. Changed to h2 h3
* ids changed to IDs
* Linked the record ID to the catalog entry
Note: We also have this link in the batch record mod job details
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We don't do a lot of processing to the data, but we can still
make the display a bit nicer, to show how many records were
done, and show the record ids.
To test:
1 - Do some batch modifications on a system running ES
2 - Check background jobs to see ES jobs
3 - Note display
4 - Apply patch
5 - Confirm display is improved
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch replaces instances in Koha's templates where "attr('value')" is
used to get the value of a form field, using "val()" instead. This
ensures that we get the current value of the field, not just the
value of the field when the page first loaded.
To test, apply the patch and test the following pages:
- Tools -> Tags. Use the "Check lists" form in the right-hand sidebar
to test the approval status of some approved and rejected tags. It
should correctly respond with the tag's status.
- Cataloging -> Label creator.
- New -> Printer profile. Each offset and creep field should correctly
reflect the selected value in the "Units" dropdown. Changing the
dropdown should make the units change accordingly.
- New -> Label template. Perform the same tests as above.
- Tools -> Patron card creator.
- New -> Layout. If necessary, check the "Field 1" checkbox. The
units should be displayed correctly and change as you change the
"Units" dropdown.
- New -> Printer profile,
- New -> Card template: Perform the same tests as above.
- Lists -> List contents.
- Check that the "Add to" dropdown menu works for adding selected
titles to a cart or another list.
- Back record modification. Submit a batch of records for modification
and go to Administration -> Manage jobs.
- View the details of the job you just submitted.
- Confirm that the "Add modified records to the following list"
dropdown works correctly.
- Cataloging -> Find a bibliographic record -> Edit items.
- Confirm that clicking a cell in the table of existing items
triggers the "Edit item/Delete item" menu, and that these links
work correctly.
Works as described.
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We lost the display of the "Add staged files to basket" link when create
a new order from a new file
Test plan:
Add order to basket from a new file, select a file and import
See the detail of the job and notice the "Add staged files to basket",
click and confirm that it works as expected.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add the ability to create new titles and attach them to a package.
The MARC to KBART2 mapping is the following (based on
https://github.com/adambuttrick/marc_to_kbart/blob/master/convert.py):
publication_title = biblio.title
print_identifier = 020$a||020$z||022$a||022$y
online_identifier = 020$a||020$z||022$a||022$y
date_first_issue_online = 866$a (before '-')
date_last_issue_online = 866$a (after '-')
num_first_vol_online = 863$a (before '-')
num_last_vol_online = 863$a (after '-')
num_first_issue_online = ?
num_last_issue_online = ?
title_url = 856$u
first_author = biblio.first_author
embargo_info = ?
coverage_depth = title_url ? 'fulltext' : 'print'
notes = $852$z
publisher_name = 260$b
publication_type = ?
date_monograph_published_print = ?
date_monograph_published_online = ?
monograph_volume = ?
monograph_edition = ?
first_editor = ?
parent_publication_title_id = ?
preceeding_publication_title_id = ?
access_type = ?
Note that title is not created (and so the resource) if a title from
this package already has a link to this bibliographic record.
Is that correct, or should we create another resource?
Should the import screen also have "start date" and "end date" to set for the
resource?
QA note: Ideally we would like to fetch the list from the REST API but the routes
are not there yet.
Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The completed alert needs a condition on status.
The record_type variable needs a bit of 'context'.
Test plan:
Verify if a failed or new job does not have a Completed alert.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
Look at job results when reverting imported batch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
Run an stage and import.
Check import commit job.
Click to managed batch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
And a few minor fixes when they where causing issues for
translatability.
And rephrased a string about password reset to have it identical to
other strings with the same meaning.
Simplified via wrapping strings with <span> to split to huge
concatenated strings with a lot of %s everywhere.
== Test plan ==
This patch needs mainly proof reading. Still it's possible to do some
basic testing to demonstrate that adding a <span> in an IF doesn't
break anything.
Pick in one of the 110 modified templates a string that you know how to
display. Otherwise:
1. acquisitions => vendor => basket => add to basket =>
search "from existing record" => add order
2. Cancel the order
3. You see without issue "Bibliographic record will not be deleted"
4. administration => Patron categories
5. Try to delete a used and unused category
6. You see as expected
Category XXXX is in use. Deletion not possible!
and
Confirm deletion of category XXXX
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
To be nicer with translators.
Update the PO files for whichever languages will show how this is useful.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
To be nicer with translators.
Update the PO files for whichever languages will show how this is
useful.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
If some items have not been deleted, the detail view of the background job will display "1 item(s) could not be deleted: ", without the list of itemnumbers.
Also add a space character between itemnumbers.
Test plan:
1. Check an item out
2. Use the batch delete items tool to delete the item that is checked
out and another one.
3. On the confirmation view, use the dev tool of your browser to remove
the "disabled" property of the disabled checkbox and select both
items.
4. Submit
5. Go to the detail view of the background job and confirm that you see
the itemnumber of the item that has failed to be deleted
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Here we go!
Disclaimer: this patch is huge and does many things, but splitting it in
several chunks would be time consuming and painful to rebase. However it
adds many tests and isolate/refactor code to make it way more reusable.
This patchset will make the "batch item modification" and "batch item
deletion" features use the task queue (reminder: Since bug 28158, and so
21.05.00, we do no longer use the old "background job" functionality and
the user does not get any info about the progress of the job).
More than that, more of the code to build an item form and a list of
items is now isolated in module (.pm) and include files (.inc)
We are reusing the changes made by bug 27526 that simplifies the way we
edit/create items (no more unecessary serialization Koha > MARC > MARCXML
> XML > HTML)
New module:
* Koha::BackgroundJob::BatchDeleteItem
Subclass for process item deletion in batch
* Koha::BackgroundJob::BatchUpdateItem
Subclass for process item modification in batch
* Koha::Item::Attributes
We needed an object to represent item's attributes that are not
mapped with a koha field (aka "more subfields xml")
This module will help us to create the marcxml from a hashref and the
reverse.
* Koha::UI::Form::Builder::Item
The code that was used to build the add/edit item form is
centralised in this module. In conjunction with the
subfields_for_item BLOCK (from html_helpers.inc) it will be really
easy to reuse this code in other places where the item form is used
(acquisition and serials modules)
* Koha::UI::Table::Builder::Items
Same as previously for the table. We are now using this table from 3
different places (batch item mod, batch item del, backgroung job
detail view) and the code is only in one place.
To use with items_table_batchmod BLOCK (still from html_helpers.inc)
This patch is fixing some bugs about repeatable subfields and regex. A UI
change will reflect the limitation: if you want to apply a regex on a
subfield you cannot add several subfields for the same subfield code.
Test plan:
Prepare the ground:
- Make sure you are always using a bibliographic/item record using the framework
you are modifying!
- Add some subfields for items that are not mapped with a koha field
(note that you can use 'é' for more fun, don't try more funny
characters)
- Make some subfields (mapped and not mapped with a kohafield)
repeatable
- Add default values to some of your subfields
There are 4 main screens to test:
1. Add/edit item form
The behaviour should be the same before and after this patch.
See test plan from bug 27526.
Those 2 prefs must be tested:
* SubfieldsToAllowForRestrictedEditing
* SubfieldsToUseWhenPrefill
2. Batch modification
a. Fill some values, play with repeatable and regex.
Note that the behaviour in master was buggy, only the first value was modified by the regex:
* With subfield = "a | b"
1 value added with "new"
=> "new | b"
* With subfield = "a | b"
2 new fields "new1","new2"
=> "new2 | b"
Important note: For repeatable subfields, a regex will apply on the subfields in
the "concatenated form". To apply the regex on all the different subfields of a given
subfield code you must use the "g" modifier.
This could be improved later, but keep in mind that it's not a regression or behaviour
change.
b. Play with the "Populate fields with default values from default framework" checkbox
c. Use this tool to modify items and play with the different sysprefs that
interfer with it:
* NewItemsDefaultLocation
* SubfieldsToAllowForRestrictedBatchmod
* MaxItemsToDisplayForBatchMod
* MaxItemsToProcessForBatchMod
3. Batch deletion
a. Batch delete some items
b. Check items out and try to delete them
c. Use the "Delete records if no items remain" checkbox to delete
bibliographic records without remaining items.
d. Play with the following sysprefs and confirm that it works as
expected:
* MaxItemsToDisplayForBatchDel
e. Stress the tool: Go to the confirmation screen with items that can be
deleted, don't request the job to be processed right away, but check the
item out before.
4. Background job detail view
You must have seen it already if you are curious and tested the above.
When a new modification or deletion batch is requested, the confirmation
screen will tell you that the job has enqueued. A link to the progress
of the job can be followed.
On this screen you will be able to see the result of the job once it's
fully processed.
QA notes:
* There are some FIXME's that are not blocker in my opinion. Feel free to
discuss them if you have suggestions.
* Do we still need MaxItemsToProcessForBatchMod?
* Prior to this patchset we had a "Return to the cataloging module" link
if we went from the cataloguing module and that the biblio was deleted.
We cannot longer know if the biblio will be deleted but we could display
a "Go to the cataloging module" link on the "job has been enqueued"
screen regardless from where we were coming from.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Use additional_report to retrieve patron and biblio's info.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Same as the first patch, for authorities
Test plan:
Delete authority records using the batch record deletion tool
Confirm that the job is now delegated to the task queue and that
everything else is working as before
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
A new include file is created per background job to avoid
background_jobs.tt to grow too much
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>