Commit graph

123 commits

Author SHA1 Message Date
30816a7fc5
Bug 30349: Add missing double dash (--delete) to POD
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-22 10:35:24 +01:00
David Schmidt
06e7553dc9
Bug 30349: More double dashes
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-22 10:35:24 +01:00
David Schmidt
c84c614146
Bug 30349: Use double dash for multi character option names
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-22 10:35:23 +01:00
David Schmidt
ccf18acb97
Bug 30349: Tidy bulkmarcimport.pl to fix indentations
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-22 10:35:23 +01:00
David Gustafsson
6aa7118c13
Bug 25539: Strip items when adding new biblio to preserve previous behaviour of the script
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 18:03:40 +01:00
David Gustafsson
e1479daf11
Bug 25539: Remove AddBiblio option "defer_marc_save"
Items are no longer embedded in the MARCXML and because of this the MARC data
does not need to be saved once more after changing record items data.
The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only
place this option was utilized in order to resave MARC data after possibly changing
items data. There is also a bug bulkmarcimport.pl where the record data is re-saved
without stripping items if duplicate items are found and the dedup barcodes
option is enabled that is resolved by this change.

This change enables MARC Overlay rules to be enabled for bulkmarkcimport.pl as using
the defer_marc option would previously effectively bypass any defined rules.

To test
1) Apply patch 29440 (which this depends on), but no not yet apply this
   patch
2) Remove comments around "bulkmarcimport: _("bulkmarcimport.pl"),"
   in koha-tmpl/intranet-tmpl/prog/en/modules/admin/marc-overlay-rules.tt
   line 463
3) Enable MARC overlay rules and add a rule (with source => "*" or
   'bulkmarcimport') to protect some field.
4) Import a biblio using the bulkmarcimport script. Overlay rules will
   only be applied on updates so a match condition matching the
   record in Koha when bulkmarcimport.pl is run the next time for the same
   record needs to be supplied (for example -match "control-number,001")
5) Edit the imported biblio and change the value of the protected field
6) Run bulkmarcimport with the same parameters as in 4) and verify the
   the field that should have been protected has been overwritten
7) Revert changes for line 463 in marc-overlay-rules.tt (or will result in
   conflict applying the patch), apply the patch and repeat steps 4-5,
   verify that the field now was protected from being overwritten.

Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 18:03:40 +01:00
David Gustafsson
70de25b439
Bug 29440: Fix AddBiblio/ModBiblio options args
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:47 +01:00
David Gustafsson
188ea40db1
Bug 29440: Replace removed syspref BiblioAddsAuthorities with AutoLinkBiblios
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:46 +01:00
David Gustafsson
6b92516214
Bug 29440: Only log items has been inserted if actually have been so, and remove extra out of place logging of record insertion
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:46 +01:00
David Gustafsson
fd3efab876
Bug 29440: Rename $modify_biblio_marc_options to $mod_bilbio_options
This variable actually contains ModBibio options and
ModifyBiblioMarc accepts a subset of these

Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:45 +01:00
David Gustafsson
33d274bbb9
Bug 29440: Remove pointless call to _strip_item_fields
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:45 +01:00
David Gustafsson
2d50eb242d
Bug 29440: replace usage of removed GetMarcBiblio
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:44 +01:00
David Gustafsson
08908b5765
Bug 29440: defer biblio autolinking until records are indexed
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:44 +01:00
David Gustafsson
2dddbb4d14
Bug 29440: Fix QA-tool issues
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:43 +01:00
David Gustafsson
aa86825c33
Bug 29440: Make authorities import behavior consistent with biblios
The current behavior for when importing authorities, when match
option is enabled, is to insert as a new authority fetching from db
fails. This is both a bit confusing and not consistent with the biblio
case where an error is generated. Also clean up error and log messages
so both messages/errors and when messages/errors are issued are more
consistent for the biblio and authority case.

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:43 +01:00
David Gustafsson
261baeff85
Bug 29440: Refactor and clean up bulkmarcimport.pl
To test:
1) Import biblios using ./bulkmarcimport.pl -v -b -file biblios.mrc -insert -update -c=MARC21 -match "control-number,001" -n=1 -l "/tmp/import.log"
2) Verify in /tmp/import.log that one biblio has been inserted
3) Import again without the "-n" option, verify that one biblio has been
   updated and the rest inserted
4) In the staff interface search the catalog for some string appearing in
   the imported biblios to verify records have been indexed
5) View/edit some of the biblios the staff interface to very have been
   properly imported
6) Import authorities using ./bulkmarcimport.pl -v -m=MARCXML -a -file authorities.xml -insert -update -c=MARC21 -keepids="024a" -match "heading-main,100a" -l "/tmp/import.log" -yaml="/tmp/ids.yml"
7) Go to the authorities page and find the imported authority by for example search for a
   400 or 100 field value. Verify that the 001 field of the incoming record has been moved to 024a.
   Also check that the value of 024a appear as a key in /tmp/ids.yml.
8) Edit the authorities.xml file and change the value of field 400a
9) Import authorities again using the same options
10) Verify that the 400a field of the authority in Koha has not been
   updated (the import was skippped since revision did not change)
11) Edit authorities.xml, replace the value of field 005 in the file with field 005
    of the authority in Koha incremented by one
12) Import again with the same options
13) Verify that the 400a field of the authority in Koha now have been
    updated

Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-03-18 11:03:42 +01:00
David Gustafsson
7da4912f8e Bug 25616: Uppercase hard coded lower case boolean operators for Elasticsearch
Uppercase occurances of all (hopefully) lowercase "and"
used in ElasticSearch Query String Query contexts

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-02-24 14:35:36 -10:00
d7390be41b Bug 28617: (QA follow-up) Restore FindBin in two places
Resolving:
    Name "FindBin::Bin" used only once: possible typo

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2021-12-07 12:16:28 -10:00
238fabc4ab Bug 28617: Remove kohalib.pl and rely on PERL5LIB
The purpose of this script was to load the relevant Koha lib for the
different scripts (installation, cronjob, CLI, etc.)
However it is not used consistently and we prefer to rely on PERL5LIB.

From bug 28617 comment 6 from Galen:
"""
Time marches on, and one of the motivations for having kohalib.pl - making
it possible to install Koha without setting a single environment variable -
has been obviated by the vast improvements in the ease of installing Koha.

Consequently, I think kohalib.pl can go away.
"""

Test plan:
confirm that the changes make sense and that kohalib.pl can be removed
safely.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2021-12-07 12:16:28 -10:00
27f3d0af63 Bug 14957: (QA follow-up) Clarify 'context' param
This patch renames the (passed through) 'context' param for
'overlay_context'. I propose doing so, because in Koha-land 'context'
has a special meaning, related to C4::Context and it reads ambigous.

The patch itself is pretty trivial.

Tests should pass:
1. Run:
   $ kshell
  k$ prove t/db_dependent/Biblio/MarcOverlayRules.t
=> SUCCESS: Tests pass
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests still pass!
4. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Bug 14957: (follow-up) Clarify 'context' param

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-10-26 16:46:02 +02:00
David Gustafsson
139e6c30d6 Bug 14957: Merge rules system for merging of MARC records
Add a rule based system for merging MARC records to for example
prevent field data from being overwritten.

To test:
    1. Apply this patch.
    2. Log in to staff client.
    3. Enable new syspref MARCMergeRules.
    4. Click the new link "MARC merge rules" in the "Catalog"
       section of the Koha administration page.
    5. Create a new rule:
       Module: source, Filter: *, Tag: 245, Preset: Protect.
    6. Clicking "Edit" should allow you to edit corresponding rule.
    7. Clicking "Delete" should remove corresponding rule after confirmation.
    8. Selecting one or more rules followed by clicking "Delete
       selected" should remove all selected rules after confirmation.
    9. Try creating a rule with tag set to "**", the other options does
       not matter. Verify that saving this rule produces an error
       message complaining about invalid tag regular expression.
    10. Try creating a rule with tag set to "008" (or other control
        field) and set Appended: Append and Removed: Skip, the other
        options does not matter. Verify that saving this rule produces
        an error message complaining about invalid combination of actions
        for control field.
    11. With the 245 rule in step 5 in place, edit a bibliographic record,
        change 245a for example (which should be Title for MARC21) and save.
    12. Verify that the changes has not been saved.
    13. Create a new rule:
        Module: source, Filter: intranet, Tag: 245, Preset: Overwrite.
    14. Repeat step 12, and verify that the changes has now been saved.
    15. Run tests in t/db_dependent/Biblio/MarcMergeRules.t and very
        that all tests pass.

Sponsored-by: Halland County Library
Sponsored-by: Catalyst IT
Sponsored-by: Gothenburg University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Christian Stelzenmüller <christian.stelzenmueller@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-10-26 16:46:02 +02:00
e9ae9a95a4 Bug 17600: (follow-up) Fix missing C4::AuthoritiesMARC import
Oops

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-07-30 08:06:07 +02:00
728f3816e3 Bug 17600: Fix missing C4::AuthoritiesMARC import
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-07-29 15:34:25 +02:00
9d6d641d1f Bug 17600: Standardize our EXPORT_OK
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.

That way we will need to explicitely define the subroutine we want to
use from a module.

This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests

And a lot of other manual changes.

export.pl is a dirty script that can be found on bug 17600.

"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;

The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules

Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).

EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.

@EXPORT and @EXPORT_OK are the two main variables used during export operation.

@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.

@EXPORT_OK does export of symbols on demand basis.
"""

If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
  - use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-07-16 08:58:47 +02:00
3fe837bc74 Bug 28572: Remove C4::Debug
We are using Koha::Logger when it makes sense to keep the info,
otherwise we simply remove it

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 28572: Replace missing occurrence in misc/admin/koha-preferences

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-06-22 12:04:32 +02:00
aba8a17713 Bug 27673: (follow-up) Fix encoding issues - Dump
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-02-16 14:54:50 +01:00
a562101fdc Bug 27673: Fix encoding issues - Dump
Same as Load, but for Dump.

Test plan:
Edit ES mappings, replace withdrawn's label with "withdrawn ✔️ ❤️ ★"
Export the mappings
  perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
Reset mappings from the UI
=> Notice that the label is correct

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-02-16 14:54:50 +01:00
46f7239b08 Bug 27673: Replace YAML with YAML::XS
From tht YAML pod:

"""
This module has been released to CPAN as YAML::Old, and soon YAML.pm will be changed to just be a frontend interface module for all the various Perl YAML implementation modules, including YAML::Old.

If you want robust and fast YAML processing using the normal Dump/Load API, please consider switching to YAML::XS. It is by far the best Perl module for YAML at this time. It requires that you have a C compiler, since it is written in C.
"""

See also
https://gitlab.com/koha-community/qa-test-tools/-/merge_requests/35

Test plan:
Try some place where YAML::XS is not used and confirm that it works
correctly

QA note: This patch removes some uses of YAML that were not useful

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-02-16 14:54:50 +01:00
371a9d1c2a Bug 25306: (QA follow-up) Remove framework from bulkmarcimport
One last instance of ModBiblioMarc in bulkmarcimport still passed the
frameworkcode.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-01-12 16:13:50 +01:00
a96c73eb76 Bug 25306: Remove framework paramter from ModBiblioMarc
Tested with :

For a framework (not the default) :

Creation of a biblio record
Edition of this biblio record
Creation of an item of this record
Creation of an item of this record

./misc/batchRepairMissingBiblionumbers.pl OK

prove t/db_dependent/Biblio/ModBiblioMarc.t OK

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-01-12 16:13:50 +01:00
4a90c0b647 Bug 27245: Replace AutoCommit=0 with txn_begin in bulkmarcimport.pl
To prevent
DBD::mysql::db begin_work failed: Already in a transaction at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1588, <GEN45> line 1.

Test plan:
1 - Export more than 100 records
2 - Use bulkmarcimport (with -commit=10) to import them
    perl misc/migration_tools/bulkmarcimport.pl -b -file testbmi.xml -v -m=MARCXML --commit=10
3 - Modify a record to make the import fail (for instance having a too long
lccn)
4 - Use bulkmarcimport (with -commit=10) to import them
    perl misc/migration_tools/bulkmarcimport.pl -b -file testbmi.xml -v -m=MARCXML --commit=10
5 - Notice that the import stops but that the imported record are imported (apart from the last batch of 10)

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-01-08 15:17:22 +01:00
9622561636 Bug 26231: Remove incorrect use of AddAuthority() when 001 present
This patch removes the use of AddAuthority with a non-empty
authid argument, since that triggers an update rather than an insert.

In practice, the update also fails, but the error isn't raised,
as the database connection doesn't have RaiseError set.

Test plan:
1) Do not apply patch
2) Try to bulkmarcimport an authority file with a 001
3) Observe that the script reports success but no authority is added
4) Apply the path
5) Try to bulkmarcimport an authority file with a 001
6) Observe that the script reports success and the authority is added

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-09-28 10:09:52 +02:00
Andreas Roussos
642c29dc23 Bug 26451: fix typo in bulkmarcimport.pl
bulkmarcimport.pl displays a small typo when a MARC modification
template is being used: "modofication".

This patch fixes that.

Test plan: apply the patch, and confirm the script no longer
displays the typo.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-09-18 11:49:29 +02:00
e968af4377 Bug 21395: (QA follow-up) Remove some introduced issues
This patch removes some new error cases introduced during rebase

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-06-29 12:37:02 +02:00
Julian Maurice
b168f4a2e9 Bug 21395: Make perlcritic happy
This patch adds a .perlcriticrc (copied from qa-test-tools) and fixes
almost all perlcrictic violations according to this .perlcriticrc
The remaining violations are silenced out by appending a '## no critic'
to the offending lines. They can still be seen by using the --force
option of perlcritic
This patch also modify t/00-testcritic.t to check all Perl files using
the new .perlcriticrc.
I'm not sure if this test script is still useful as it is now equivalent
to `perlcritic --quiet .` and it looks like it is much slower
(approximatively 5 times slower on my machine)

Test plan:
1. Run `perlcritic --quiet .` from the root directory. It should output
   nothing
2. Run `perlcritic --quiet --force .`. It should output 7 errors (6
   StringyEval, 1 BarewordFileHandles)
3. Run `TEST_QA=1 prove t/00-testcritic.t`
4. Read the patch. Check that all changes make sense and do not
   introduce undesired behaviour

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-06-29 12:37:02 +02:00
Matthias Meusburger
15dd83268b
Bug 15214: bulkmarcimport should log if an authority has been updated
When importing with bulkmarcimport, authorities may or may not be updated based
on which authority is newer (005 are compared). This patch allows to keep track
in the result yaml file if an authority has been updated or not:

Signed-off-by: Jon Knight <J.P.Knight@lboro.ac.uk>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-03-24 10:54:46 +00:00
52e95fff34
Bug 24735: Remove QueryParser-related code
At the last development meeting we have voted to remove the
QueryParser-related code
https://wiki.koha-community.org/wiki/Development_IRC_meeting_19_February_2020

Hea tells us that it has not been adopted, and the code/bug tracker that
it is not really usable as it. As nobody is willing to work on it, we
decided to remove it instead.

Test plan:
 % prove t/db_dependent/Search.t
must return green

See commits from bug 9239 and confirm that the code is removed in this
patch.

Also play with the search on the UI and confirm that you do not see
obvious regressions

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-03-02 11:13:03 +00:00
263c013121
Bug 23377: (QA follow-up) Use OVERRIDE_SYSPREF
SYSPREF_OVERRIDE unfortunately is not supported ;)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-01-10 10:56:41 +00:00
Ian Walls
f036038318
Bug 23377: use ENV syspref override instead of complete disable
In an attempt to save time, bulkmarcimport temporarily sets CataloguingLog
and AuthoritiesLog to 0.  It does this by disabling syspref caching and saving
the changes to the database (then replacing the original values at completion).

Unfortunately, this disables other key sysprefs from being cached, and results in
a 50% increase in processing time for the script.

This patch instead utilizes the ENV variable override feature of sysprefs, which
preempts the cache in C4::Context->preference().

To test:
1. Perform a bulkmarcimport with a reasonable number of biblios (~1000 will do)
2. Note the time taken to complete
3. Apply patch
4. Revert the biblio load performed
5. Perform another bulkmarcimport with the same biblios and commandline options
6. Note the time taken to complete
7. Compare times.  The time from step 6 should be about 33% less than the time from step 2
8. Check Cataloguing and Authorities Logs to verify imported records were not logged
9. Profit!

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-01-10 10:56:32 +00:00
c121a53dd7
Bug 24123: Fix import of UTF-8 encoded MARC21 MARCXML using bulkmarcimport (elastic only)
If elastic is used as search engine, the bulkmarcimport.pl will not
handle correctly UTF-8 encoded MARCXML

Koha::SearchEngine::Search->new uses a require statement to load the correct Search module.
This is done l.257 of bulkmarcimport.pl:
  257 my $searcher = Koha::SearchEngine::Search->new

Koha::SearchEngine::Elasticsearch::Search will `use MARC::File::XML`, and so resets the arguments set before:
  216     $MARC::File::XML::_load_args{BinaryEncoding} = 'utf-8';

  220     $MARC::File::XML::_load_args{RecordFormat} = $recordformat;

An easy (but dirty) fix could be to move the declaration of my $searcher before in the script.
The tricky (but correct) fix would be to remove the long standing "ugly hack follows" comment.

This patch is the easy, and dirty, fix

Test plan:
Use the command line tool to import MARXCML records that contains unicode characters into Koha

Something like `misc/migration_tools/bulkmarcimport.pl -biblios -file record.marcxml -m=MARCXML`

Without this patch you will notice that unicode characters will not be displayed correctly

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-01-10 08:27:47 +00:00
68eeefa07e
Bug 22721: Remove frameworkcode parameter in GetMarcFromKohaField calls
Test plan:
Run tests, at least:
    t/db_dependent/Biblio.t
    t/db_dependent/Biblio/TransformHtmlToMarc.t
    t/db_dependent/Charset.t
    t/db_dependent/Circulation/GetTopIssues.t
    t/db_dependent/Filter_MARC_ViewPolicy.t
    t/db_dependent/ImportBatch.t
    t/db_dependent/Items.t
    t/db_dependent/Items/AutomaticItemModificationByAge.t
    t/db_dependent/Items/GetItemsForInventory.t
    t/db_dependent/Koha/Filter/EmbedItemsAvailability.t
    t/db_dependent/Serials.t
    t/db_dependent/XISBN.t
    t/db_dependent/FrameworkPlugin.t

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-07-15 11:28:08 +01:00
d2e189ca1c Bug 22600: Set 'commandline' interface appropriately
This patch change Koha::Cron to be a more generic Koha::Script class and
update all commanline driven scripts to use it.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-04-10 19:43:11 +00:00
Magnus Enger
b7b4694666 Bug 12488: Make bulkmarcimport.pl -d use DELETE instead of TRUNCATE
On MySQL >= 5.5 bulkmarcimport.pl with the -d (delete) switch gives
an error like "Cannot truncate a table referenced in a foreign
key constraint". This patch proposes to replace the offending
TRUNCATE with DELETE. Auto incerement counters are reset to 1 to
preserve the functionality from TRUNCATE.

To test:
- Make sure you havae a test database with some records and items
- Run bulkmarcimport.pl with the -d switch
- Observe the error described above
- Apply this patch
- Run bulkmarcimport.pl with the -d switch again
- Observe the lack of an error
- Verify that the newly imported records and items have biblionumber
  and itemnumbers starting with 1

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-02-15 18:42:46 +00:00
12b505f2b7 Bug 20811: (RM follow-up) fix check for matching
Notice should only be triggered if both settings are false

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-23 15:08:56 +00:00
15b865c6ee Bug 20811: (QA follow-up) Prevent calling AddItemBatchFromMarc and ModBiblioMarc without biblionumber
If you use -update but do not find matches (or did not want to match), you
should not call those routines. We should warn and skip this record.

Adding a warn at the start that the choice of options may not be smart.
Note that this needs further attention somewhere else. You could mix
-update with -insert for instance and still see some problems. (May depend
on items with unique barcode etc.)

Test plan:
Run -update without match or isbn.
Or run -update -isbn with a non-matching ISBN.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-23 15:08:56 +00:00
a49a7f08d6 Bug 20811: (QA follow-up) Make the isbn check work
The following code was never reached, since $isbn was not filled.
    if (!$biblionumber && $isbn_check && $isbn) {
        $sth_isbn->execute($isbn);
        ($biblionumber,$biblioitemnumber) = $sth_isbn->fetchrow;
    }
Solution: Fix the code with two $isbn declarations. Move the checkisbn
condition a level deeper.

Test plan:
Run misc/migration_tools/bulkmarcimport.pl -file bib726.utf8 --update -isbn
Since you do not match on biblionumber, the ISBN should match.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-23 15:08:56 +00:00
d15185025a Bug 20811: Fix wrong usage of ModBiblio
Since
  commit cefa7c21e2
  Bug 5635: bulkmarcimport new parameters & features

AddBiblio call has been replaced with ModBiblio, but the return values
are different. We should not replace the value of $biblionumber with
what returns this subroutine.

Test plan:
If you are familiar with bulkmarcimport.pl you should know what to test,
I am not.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-23 15:08:56 +00:00
Julian Maurice
79f5051ed7 Bug 11936: (QA follow-up) Consistent log message for item insert
Fix the same error in another place

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-19 13:55:54 -03:00
Magnus Enger
c2208e9ca7 Bug 11936: Consistent log message for item insert
If you use bulkmarcimport.pl to import records with items it looks
like the successfull insert of the record is reported multiple time,
but the second and subsequent "ok" is really related to importing
the item(s).

This patch changes the log message on successfully inserting an item
to match the log message given when inserting an item fails.

To test, the easy way:
- Look at lines 530 and 536 of bulkmarcimport.pl, and note that the
  "op" in those two lines are different
- Apply the patch
- Look at lines 530 and 536 again, and note that the "op" is now
  identical, and that this makes sense, since they are both related
  to the same operation, specifically inserting an item

To test, the hard way
- Have some records with items
- Import the records with bulkmarcimport.pl, and make sure to specify
  the -l option, to create a log of the actions taken
- Look at the log and verify it looks something like this:
  id;operation;status
  1;insert;ok
  1;insert;ok
  2;insert;ok
  2;insert;ok
- Apply this patch and import some more records with items. The log
  should now be similar to this:
  id;operation;status
  1;insert;ok
  1;insertitem;ok
  2;insert;ok
  2;insertitem;ok

Signed-off-by: Maksim Sen <maksim.sen@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-19 13:55:54 -03:00
6835760dcf Bug 18389: (QA follow-up) Fix tabs, note used template, info on verbose
Fixed whitespace for QA tools
Added a verbose note when template found
Only print 'Modifying MARC' if verbose

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-27 14:10:29 -03:00