When importing with bulkmarcimport, authorities may or may not be updated based
on which authority is newer (005 are compared). This patch allows to keep track
in the result yaml file if an authority has been updated or not:
Signed-off-by: Jon Knight <J.P.Knight@lboro.ac.uk>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Starting to replace the ModItem calls with Koha::Item->store
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch standardises the encoding name used in direct calls
to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'.
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
At the last development meeting we have voted to remove the
QueryParser-related code
https://wiki.koha-community.org/wiki/Development_IRC_meeting_19_February_2020
Hea tells us that it has not been adopted, and the code/bug tracker that
it is not really usable as it. As nobody is willing to work on it, we
decided to remove it instead.
Test plan:
% prove t/db_dependent/Search.t
must return green
See commits from bug 9239 and confirm that the code is removed in this
patch.
Also play with the search on the UI and confirm that you do not see
obvious regressions
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 9978 should have fixed them all, but some were missing.
We want all the license statements part of Koha to be identical, and
using the GPLv3 statement.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
We certainly faced 3 similar bugs due to this syntax: bug 23006, bug
22941 and bug 17526.
To prevent other issues related to this syntax this patch suggests to
replace them all in one go.
Test plan:
Confirm that the 2 syntaxes are similar
Eyeball the patch and confirm that there is no typo!
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
SYSPREF_OVERRIDE unfortunately is not supported ;)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In an attempt to save time, bulkmarcimport temporarily sets CataloguingLog
and AuthoritiesLog to 0. It does this by disabling syspref caching and saving
the changes to the database (then replacing the original values at completion).
Unfortunately, this disables other key sysprefs from being cached, and results in
a 50% increase in processing time for the script.
This patch instead utilizes the ENV variable override feature of sysprefs, which
preempts the cache in C4::Context->preference().
To test:
1. Perform a bulkmarcimport with a reasonable number of biblios (~1000 will do)
2. Note the time taken to complete
3. Apply patch
4. Revert the biblio load performed
5. Perform another bulkmarcimport with the same biblios and commandline options
6. Note the time taken to complete
7. Compare times. The time from step 6 should be about 33% less than the time from step 2
8. Check Cataloguing and Authorities Logs to verify imported records were not logged
9. Profit!
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
If elastic is used as search engine, the bulkmarcimport.pl will not
handle correctly UTF-8 encoded MARCXML
Koha::SearchEngine::Search->new uses a require statement to load the correct Search module.
This is done l.257 of bulkmarcimport.pl:
257 my $searcher = Koha::SearchEngine::Search->new
Koha::SearchEngine::Elasticsearch::Search will `use MARC::File::XML`, and so resets the arguments set before:
216 $MARC::File::XML::_load_args{BinaryEncoding} = 'utf-8';
220 $MARC::File::XML::_load_args{RecordFormat} = $recordformat;
An easy (but dirty) fix could be to move the declaration of my $searcher before in the script.
The tricky (but correct) fix would be to remove the long standing "ugly hack follows" comment.
This patch is the easy, and dirty, fix
Test plan:
Use the command line tool to import MARXCML records that contains unicode characters into Koha
Something like `misc/migration_tools/bulkmarcimport.pl -biblios -file record.marcxml -m=MARCXML`
Without this patch you will notice that unicode characters will not be displayed correctly
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
018 - [Reserved for other International Standard Numbers]
033 - Other System Persistent Record Identifier
183 - Coded Data Field: Type of Carrier
203 - Content Form and Media Type
231 - Digital File Characteristics
283 - Carrier Type
338 - Funding Information Note
Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
1. Take a look at files in misc/migration_tools/ifla/data. It
contains all data that will be inserted into Koha. Its content is
based on the previous patches.
2. Run the script misc/migration_tools/ifla/update.pl and verify
that it effectively added the new fields, subfields, authorised
values and authority types.
3. Run the script again and see that it doesn't update existing fields
4. Run with --force and verify that it update existing fields (you can
modify unimarc_ifla.yml to see changes)
5. Run with --force --po-file misc/migration_tools/ifla/language/fr.po
and verify that the labels are now in french
There is a POT file in misc/migration_tools/ifla/language/template.pot,
use it to create PO files for other languages.
Signed-off-by: Sonia BOUIS <sonia.bouis@univ-lyon3.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch change Koha::Cron to be a more generic Koha::Script class and
update all commanline driven scripts to use it.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
On MySQL >= 5.5 bulkmarcimport.pl with the -d (delete) switch gives
an error like "Cannot truncate a table referenced in a foreign
key constraint". This patch proposes to replace the offending
TRUNCATE with DELETE. Auto incerement counters are reset to 1 to
preserve the functionality from TRUNCATE.
To test:
- Make sure you havae a test database with some records and items
- Run bulkmarcimport.pl with the -d switch
- Observe the error described above
- Apply this patch
- Run bulkmarcimport.pl with the -d switch again
- Observe the lack of an error
- Verify that the newly imported records and items have biblionumber
and itemnumbers starting with 1
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch makes the code using Koha::Biblio::Metadata->marcflavour use
->schema instead for all interactions.
To test:
- Update the DB structure:
$ updatedatabase
- Update the schema files:
$ dbic
- Notice all the places in which biblio_metadata is used
$ cd kohaclone
$ git grep biblio_metadata
=> SUCCESS: They all use `schema` instead of marcflavour
- Notice all the places that use Koha::Biblio::Metadata:
$ git grep Koha::Biblio::Metadata
=> SUCCESS: They all use the schema attribute when they used to use
marcflavour
- Run all the modified tests and scripts
=> SUCCESS: We are all good
- Sign off :-D
Note: while this seems like a minor change, the places in which plain
SQL is used really require understanding the queries and how they are
used, because some query results might be passed to some other method
that in turn uses the marcflavour attribute. I of course took that into
account but errare humanum est :-D
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Zebra indexing script misc/migration_tools/rebuild_zebra.pl as a table arg to allow filtering.
When using table=items we should use DISTINCT(biblionumber) to avoid indexing several times the same biblio record when it has several items.
This patch adds DISTINCT(biblionumber) in all cases it does not harm if its already unique.
Test plan :
1) Be sur you have a biblio record with biblionumber 1 with 3 items
2) Run misc/migration_tools/rebuild_zebra.pl -v -b --table items --where="biblionumber=1"
3) Without patch you see "Records exported: 3", with patch only one
4) Check indexing works well
Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Zebra indexing script misc/migration_tools/rebuild_zebra.pl as a table arg to allow filtering.
It is missing biblio_metadata to allow filtering on MARCXML with ExtractValue.
Test plan :
1) Be sur you have a biblio record with biblionumber 1
2) Run misc/migration_tools/rebuild_zebra.pl -h
3) You see : --table specify a table (can be items, biblioitems, biblio, biblio_metadata) to retrieve biblionumber to index.
4) Run misc/migration_tools/rebuild_zebra.pl -v -b --table biblio_metadata --where="biblio_metadata.biblionumber=1"
5) Check you dont have SQL errors
Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
- Added missing GetHiddenItems parameter change case
Without this prove t had a failure.
- Always use mocks, not set_preference
- Tweaks so t/db_dependent/00-strict.t passes
There was a typo botcat vs borcat and borrowernumber was never
defined. Grabbing from userenv, like other code does.
- Tweak t/db_dependent/Items.t to fully test changes
This will test all the if structures fully in GetHiddenItemnumbers.
prove t/db_dependent/Items.t
- Tweak borrower category code
$borrower->{categorycode} on a Koha::Patron is not the
same as $borrower->categorycode. Fixed error.
- Search was returning URLS for wrong interface
There was one search context place wrong. Changed it to $is_opac
as the logic for setting $is_opac was modified correctly.
- Corrected issues with category code.
When a user isn't logged in, $borrower is undef and causes error
when determining category code. Added conditional check.
- Properly trigger all changes in C4/Search.pm
- Fix QA Test tool failures
C4/Search.pm had some tabs.
- Add some commenting to make sense of logic
- Refactor EmbedItemsInMarcBiblio parameters to hashref
- Trigger GetMarcBiblio's EmbedItemsInMarcBiblio call.
prove t/db_dependent/Items.t
- Add missing test to trigger Koha/BiblioUtils/Iterator change
- Add borrower category overrides
These files generally add borcat parameter to GetMarcBiblio.
Others might include correction of filtering of items
(opac-basket), or a comment as to why no changes were done
(opac-search).
In the case of opac-search, correcting the first FIXME will
likely correct the OpacHiddenItems issues on tags. As such,
that is beyond this bugs scope.
Some code had loop optimizations and fixes made, like a
'next unless $record' when the biblio shouldn't even be in
the list.
- Modify opac-ISBDdetail and opac-MARCdetail
Both files had similar logic. They were rearranged and
optimized, so that both files would have practically identical
initial blocks of code.
Optimizations were possible, because GetMarcBiblio
returns a filtered record, so that there is no double call
(once in the opac-### file and once in GetMarcBiblio) to
GetHiddenItemnumbers.
- Fix hiding in opac-tags
opac/opac-tags.pl was not properly hiding.
There is currently one known bug associated with tags left.
If you have two biblios tagged by different people with the
same tag, the opac-search will show the one you tagged that
is supposed to be hidden, because tag searches work differently
than regular searches. This is beyond the scope of this bug.
See the FIXME's in opac/opac-search.pl
- Trigger the C4::ILSDI::Services changes
prove t/db_dependent/ILSDI_Services.t
- Added missing 'my'
- Test C4/Labels/Label.pm changes
- Improve C4::Record::marcrecord2csv test cases
- Corrected opac-details searchResult call
- Fix breaking issues constraint in ITerator test
- Fix ILSDI_Services test when clubs with branch exist
- Rebased again!
- Rebased t/db_dependent/Items.t conflict.
The test plan is in comment #112 last I checked.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Remove:
- BIB_INDEX_MODE and AUTH_INDEX_MODE env var
- bib_index_mode and auth_index_mode options from scripts
- Warnings from about page, just kept one if zebra_bib_index_mode or
zebra_auth_index_mode still exist in config and are set to grs1
Test plan:
- Install Koha from src
- Install Koha from pkg
- Read the code, carefully!
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Rebased
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
If you use -update but do not find matches (or did not want to match), you
should not call those routines. We should warn and skip this record.
Adding a warn at the start that the choice of options may not be smart.
Note that this needs further attention somewhere else. You could mix
-update with -insert for instance and still see some problems. (May depend
on items with unique barcode etc.)
Test plan:
Run -update without match or isbn.
Or run -update -isbn with a non-matching ISBN.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
The following code was never reached, since $isbn was not filled.
if (!$biblionumber && $isbn_check && $isbn) {
$sth_isbn->execute($isbn);
($biblionumber,$biblioitemnumber) = $sth_isbn->fetchrow;
}
Solution: Fix the code with two $isbn declarations. Move the checkisbn
condition a level deeper.
Test plan:
Run misc/migration_tools/bulkmarcimport.pl -file bib726.utf8 --update -isbn
Since you do not match on biblionumber, the ISBN should match.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Since
commit cefa7c21e2
Bug 5635: bulkmarcimport new parameters & features
AddBiblio call has been replaced with ModBiblio, but the return values
are different. We should not replace the value of $biblionumber with
what returns this subroutine.
Test plan:
If you are familiar with bulkmarcimport.pl you should know what to test,
I am not.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
