Enable librarians in the staff client to control when unaccepted list
invites will be removed from the database.
Test plan:
1. Apply patches
2. Update database
sudo koha-shell <instance>
cd installer/data/mysql
./updatedatabase.pl
3. Confirm the PurgeListShareInvitesOlderThan syspref = 14
4. Run cleanup_database.pl with --list-invites parameter defined:
sudo koha-shell <instance>
cd misc/cronjobs
./cleanup_database.pl --list-invites 10 -v
5. Confirm the output contains:
"Purging unaccepted list share invites older than 10 days."
6. Set PurgeListShareInvitesOlderThan syspref = 15
7. Run cleanup_database.pl without the --list-invites parameter:
sudo koha-shell <instance>
cd misc/cronjobs
./cleanup_database.pl -v
8. Confirm the output contains:
"Purging unaccepted list share invites older than 15 days."
9. Empty PurgeListShareInvitesOlderThan syspref
10. Run cleanup_database without the --list-invites parameter:
sudo koha-shell <instance.
cd misc/cronjobs
./cleanup_database.pl
11. Observe you get the following output (along with documentation on all
parameters of the script):
"You did not specify any cleanup work for the script to do."
12. Run the cleanup_database with no days defined for the --list-invites
parameter:
sudo koha-shell <instance>
cd misc/cronjobs
./cleanup_database.pl --list-invites -v
13. Confirm the output contains:
"Purging unaccepted list share invites older than 14 days."
Sponsored-by: Catalyst IT, New Zealand
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This parameter is actually --log-module.
Test plan :
Run misc/cronjobs/cleanup_database.pl -h
Confirm you see --log-module
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Perltidied one line in cleanup_database.pl.
Bug 33837: Perl tidy block in Objects.t
WARN t/db_dependent/Koha/Objects.t
WARN tidiness
The file is less tidy than before (bad/messy lines before: 319, now: 320)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add --log-action parameter to cleanup_database.pl
Test plan:
1. Apply patch
2. Enable cataloguing and borrowers log
3. Make some changes to borrowers, create some borrowers, and edit some
biblio records
4. Change the action_logs.timestamp for all action_logs entries to 367
days ago
5. Run cleanup_database.pl with --logs 365 --log-module=MEMBERS
--log-action=CREATE --confirm
6. Confirm only the borrowers creation action_logs entries are removed
7. Run cleanup_database.pl with --logs 365
8. Confirm all action_logs entries are removed
Sponsored-By: Toi Ohomai Institute of Technology, New Zealand
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch now checks the status of messages and ignores any message
with a status of 'new'. It is also rebased to account for changes made
to cleanup_database.pl in bug 17350.
Test plan:
1) Ensure you have some EDI orders or even just some dummy messages in the edifact_messages table with a mixture of statuses including 'new'
2) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose (Change the number of days according to the data in your table)
3) The response should show a number of messages that would have been deleted
4) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose --confirm
5) The response should now show the same number of messages have been deleted
6) Check your edifact_messages table to confirm that the data has been deleted
7) Confirm that no messages marked 'new' have been deleted
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Changes a few occurrences of edifact in the output messages
to EDIFACT.
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch allows users to clear out old edifact_messages using the cleanup_database script. The number of days can either be set in the CLI or the default value of 365 can be used.
Test plan:
1) Ensure you have some EDI orders or even just some dummy messages in the edifact_messages table
2) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose (Change the number of days according to the data in your table)
3) The response should show a number of messages that would have been deleted
4) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose --confirm
5) The response should now show the same number of messages have been deleted
6) Check your edifact_messages table to confirm that the data has been deleted
Sponsored-by: PTFS Europe
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This amends the documentation of the cleanup_database.pl script
to include a hint for how the saved reports data is created.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
To test:
apply patch
1 - have or create a report
2 - run your report via the command line with --store-results. Do this twice.
3 - update the saved_reports table to set date_run for one of your two saved results set to a datetime more than 5 days ago
4 - perl misc/cronjobs/cleanup_database.pl --reports 5 --verbose --confirm
5 - Koha tells you its deleting saved reports data from more than 5 days ago
6 - confirm in the database and the staff interface that it's done so
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This script has a pattern to delete rows depending on a given
date/number of days, we should use the filter_by_last_update
Koha::Objects method.
No need for another method and tests, everything is already tested
there.
This patch also suggests to rename the reference to "background" and
"bg" with "jobs", which seems more appropriate and not an abbreviation
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
There as a mismatch between the parameter name in documentation (bg-days)
and what was checked in the code (bg-jobs). This makes both match.
To test:
* Apply patch
* Make sure you have some background_jobs older than one day
* Verify the help of the script documents the parameter bg-days
* Run the cleanup_database.pl job
Example: ./misc/cronjobs/cleanup_database.pl --bg-days --confirm -v
* Verify there are no errors and the lines have been deleted as expected
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds background queue options to cleanup_database.pl to allow
for purging completed background jobs.
--bg-jobs DAYS Purge all finished background jobs this many days old. Defaults to 1 if no DAYS provided.
--bg-type TYPE What type of background job to purge. Defaults to "update_elastic_index" if omitted
Specifying "all" will purge all types. Repeatable.
To test:
1 - Enable elastic search in Koha
2 - perl misc/maintenance/touch_all_items.pl
3 - Generate an number of diffrent types of bg-jobs (eg batch_hold_cancel,
batch_biblio_record_deletion, batch_item_record_deletion)
4 - Check db and note there are a bunch of diffrent jobs
5 - Update to make them old
UPDATE background_jobs SET ended_on = '2022-10-01 00:00:00', status='finished'
6 - perl misc/cronjobs/cleanup_database.pl
7 - Note bg-jobs entry shows in help
8 - perl misc/cronjobs/cleanup_database.pl --bg-jobs 1 -v
9 - Note that elasticqueue would have been cleared
10 - perl misc/cronjobs/cleanup_database.pl --bg-jobs 1 -v --confirm
11 - Note that number of entries deleted is reported
12 - Attempt to clear other job types, including "all" eg
perl misc/cronjobs/cleanup_database.pl --bg-jobs 1 --bg-type batch_item_record_deletion -v --confirm
perl misc/cronjobs/cleanup_database.pl --bg-jobs 1 --bg-type all -v --confirm
13 - Confirm in staff interface that jobs are gone:
http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl
(Uncheck 'Current jobs only')
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
cleanup_database.pl cronjob has a typo in it's usage/help:
"preserve-logs" option should be "preserve-log" as it is everywhere
in the code.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Added command line ooption logging and completion logging where
cronlogaction was already imported. We should probably standardize all
cronjobs, but this is a start
One cron didn't log on confirm, likely we need to update all crons to log
if confirm, and possibly not log if running in test mode? Another bug as well
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Silly mistake, 'delete if verbose' must be 'delete if confirm'
Test plan:
Try the cleanup_database.pl script to delete transfers and old issues.
Using --transfers --old-reserves and the --confirm flag the entries must be
removed
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We also add a test by inserting a simulated borrower modification
record in the same table which should not be deleted.
NOTE: This patch fixes the del-unv-selfreg parameter in the
cleanup db script. It did not even do what it promised :)
Test plan:
Verify if the crontab change is correct.
Run t/db_dependent/Members.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
The purpose of this script was to load the relevant Koha lib for the
different scripts (installation, cronjob, CLI, etc.)
However it is not used consistently and we prefer to rely on PERL5LIB.
From bug 28617 comment 6 from Galen:
"""
Time marches on, and one of the motivations for having kohalib.pl - making
it possible to install Koha without setting a single environment variable -
has been obviated by the vast improvements in the ease of installing Koha.
Consequently, I think kohalib.pl can go away.
"""
Test plan:
confirm that the changes make sense and that kohalib.pl can be removed
safely.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This adds two new parameters:
--log_module
--preserve_log
These can be repeated to include or exclude specific modules from the cleanup
To test:
0 - Apply patch
1 - Enable cataloging log and borrowers log
2 - Make some changes to borrowers and records
3 - run cleanup databse with --logs 0 --preserve_log=MEMBERS --preserve_log=CATALOGUING
4 - nothing is removed
5 - run cleanup databse with --logs 0 --log_module=MEMBERS
6 - the borrower logs are removed, the record changes remain
7 - run cleanup database with --logs 0
8 - record changes are removed (all other logs too)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Rebecca Coert <rcoert@arlingtonva.us>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
I think it is a bit more clear to use another variable for the
number of days from the preferences than overwriting the flag
variable.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tested by adding an authval for LOST.
Filling prefs ClaimReturnedLostValue, CleanUpDatabaseReturnClaims.
Claiming a return, resolving it. Setting date back via SQL.
Running the script with -v --return-claims, toggling --confirm.
The claim is counted and deleted with confirm.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add option to cleanup_database script to removed 'resolved' return claims from the database after a specified number of days.
Test Plan:
1) Apply this patch
2) Set the new syspref CleanUpDatabaseReturnClaims to a number of days
3) Run cleanup_database.pl
4) Note resolved claims older than the specified number of days are removed from the database
Bug 25429: Implement system preference, remove command line switch
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Rebecca Coert <rcoert@arlingtonva.us>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch remove a check for confirm when fetching the entries
for possible deletion from the zebraqueue
Without this we die when we try to access the entries for counting
To test:
1 - perl misc/cronjobs/cleanup_database.pl --sessions --zebraqueue 10 --list-invites --z3950 --logs 180 --mail 375 -v
2 - Dies
3 - Apply patch
4 - Repeat 1
5 - Success!
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch moves the SQL to a subroutine and uses the Label delete method to
remove the batches
We now return a count and do not delete if --confirm flag is not passed
Repeate previous test plan, confirm works as expected
JD amended patch: perltidy
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
For item label batches:
1- Create 3 item label batches of at least 2 items each
2- Perform the following updates via the database
3- In Batch 1, set all items' timestamps to 10+ days prior to today
4- In Batch 2, set 1 item's timestamp to 10+ days prior to today
5- In Batch 3, do not update any timestamps
6- Run cron with --labels 9
7- Confirm batch 1 is deleted, batches 2 and 3 are not
Repeat with card creator batches
8- Create 3 card creator batches of at least 2 items each
9- Perform the following updates via the database
10- In Batch 1, set all cards' timestamps to 10+ days prior to today
11- In Batch 2, set 1 card's timestamp to 10+ days prior to today
12- In Batch 3, do not update any timestamps
13- Run cron with --cards 9
14- Confirm batch 1 is deleted, batches 2 and 3 are noti
Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Signed-off-by: Deb Stephenson <dstephen@dubuque.lib.ia.us>
Signed-off-by: Abbey Holt <aholt@dubuque.lib.ia.us>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch ensures the script will lock patrons if the pref is set but no other actions
specified
It also changes the variable name to be a bit more explicit
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
See QA remarks.
Test plan:
Run t/db_dependent/Koha/Patrons.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Adding a search on locked patrons to the search_expired in cron script.
This prevents relocking.
Test plan:
Prove Koha/Patrons.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
misc/cronjobs/cleanup_database.pl provides some database cleanup.
This patch adds cleanup for messages table.
Test plan :
- Count messages : select count(*),date(message_date) from messages group by date(message_date);
- Run cleanup : misc/cronjobs/cleanup_database.pl -v --confirm --messages 365
- Recount messages
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The $count variable should be defined per block and set to 0 to avoid
confusion.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
How handy this will be, right?
This patch will change the behavior of this script as the --confirm flag
will now be required for effective changes.
Without --confirm and with --verbose you will see what could have done
the script in non dry-run mode.
Test plan:
Use different options of the script without the --confirm flag and have
a look at the output. It now tells you what would have been deleted if
the flag was passed.
Now use the --confirm flag and confirm that the changes are now
effective.
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Signed-off-by: Sonia Bouis <sonia.bouis@univ-lyon3.fr>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
anonymized_* tables have been added by bug 24151, this patch adds the
usual way to purge data from them.
The cleanup_database.pl script has been adjusted to take new parameters
that will help to delete pseudonymized data.
Test plan:
Call the scrip with the new parameter to remove pseudonymized data
* --pseudo-transactions DAYS will remove entries from pseudonymized_transactions older
than DAYS day
* --pseudo-transactions can be used without the parameter DAYS but with
-- pseudo-transactions-from and/or --pseudo-transactions-to instead, to provide a
range of date
You can use the patch from bug 24153 to make the tests easier, data will
not be deleted if the new --confirm flag is not passed.
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Signed-off-by: Sonia Bouis <sonia.bouis@univ-lyon3.fr>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Since these operations impact on the resultset, the counts should be
saved before.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
The warn must be done in the cronjob.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Followed this test plan (with two follow-ups applied):
[1] Prefs: UnsubscribeReflectionDelay=1, PatronAnonymizeDelay=2, PatronRemovalDelay=3, FailedLoginAttempts was undef
[2] Pick borrower and set expiry to NOW-2, and lock him (login_attempts=-1) Could be achieved too by settings FailedLoginAttempts and trying wrong passwords. Run cleanup job:
Locked 0 patrons
Anonymized 1 patrons
Deleted 0 patrons
[3] Pick borrower, set expiry to NOW-3. Run cleanup job:
Locked 0 patrons
Anonymized 0 patrons
Deleted 1 patrons
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
