The MARC preview of staged records fails when a staged record contains
items with UTF-8 characters (see attached screenshots). That's because
in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml()
does not explicitly specify 'UTF-8' as the optional encoding to use.
This patch fixes that.
Test plan (in a MARC21 instance):
0) Stage for import the test record attached to this bug report (make
sure that 'Check for embedded item record data?' is set to 'Yes'),
then go to Home > Tools > Manage staged MARC records and click on
the filename you've just staged.
1) In the 'Citation' column, click on the hyperlink to launch the MARC
preview -- notice how it fails to produce any output apart from the
modal heading.
2) Apply the patch, and restart Plack if necessary.
3) Re-launch the MARC preview: this time the output should include the
whole record, including the embedded item data in field 952.
4) Run the unit tests to ensure nothing was broken inadvertently:
$ prove t
$ prove xt
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
We certainly faced 3 similar bugs due to this syntax: bug 23006, bug
22941 and bug 17526.
To prevent other issues related to this syntax this patch suggests to
replace them all in one go.
Test plan:
Confirm that the 2 syntaxes are similar
Eyeball the patch and confirm that there is no typo!
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records.
Those imported items are stored as XML to be staged.
The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist.
You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a
This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml
Test plan :
- On a UNIMARC database
- Define a maching rule on title 200$a
- Select a record with items
- Export it using : Save as > MARC (Unicode/UTF-8)
- Delete all items
- Go to Tools > Stage MARC records for import
- Upload exported file
- Select title matching rule
- Select "Ingore incoming record" in "Action if matching record found :"
- Select Yes and "Always add items" in "Check for embedded item record data?"
- Click Stage for import
=> Without patch you get the error
=> With patch the import is staged
- Import into the catalog and check item is well recreated
Signed-off-by: Amandine Zocca <azocca@ville-montauban.fr>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
Try to search, preview and import authority from Z39.50, everything
should work as expected
Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
C4::Items::GetItemnumberFromBarcode calls can be replaced with
Koha::Items->find({ barcode => $barcode });
We should make sure the barcode existed in DB and so that ->find
returns an object. Note that most of the time we just wanted to know if
the barcode existed.
The changes are very simple, the only one that need attention is
the one in batchMod.pl. It is basically reusing what we did on
bug 21141.
Test plan:
Use the batch item modification/deletion tools to modify/delete items
from their barcode (using the textarea or a file)
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Fix for:
Field 'marcxml_old' doesn't have a default value
t/db_dependent/ImportBatch.t
Fixme - need to update existing DB?
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
TEST PLAN
---------
1) Run the following on a kohadevbox:
git checkout -b bug_19937 origin/master
sudo koha-shell -c bash kohadev
prove t/db_dependent/www/batch.t
cat /var/log/koha/kohadev/plack-error.log
The following errors are triggered at the end of the log file:
Use of uninitialized value in array element at
/home/vagrant/kohaclone/tools/batch_records_ajax.pl line 50.
Use of uninitialized value $results_per_page in numeric eq (==) at
/home/vagrant/kohaclone/tools/batch_records_ajax.pl line 53.
Use of uninitialized value in uc at
/home/vagrant/kohaclone/C4/ImportBatch.pm line 1120.
2) Run the following on a kohadevbox:
exit
git bz apply 19937
restart_all
sudo koha-shell -c bash kohadev
prove t/db_dependent/www/batch.t
cat /var/log/koha/kohadev/plack-error.log
The log file will end with the restarting of plack, not the
errors.
3) run koha qa test tools
Signed-off-by: Lee Jamison <ldjamison@marywood.edu>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended handling of $results_per_page.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
As requested by QA:
[1] Mock_config enable_plugins in the test.
[2] Fallback to MARC when format is empty. Remove die statement.
Added:
[3] Remove $marc. This variable got obsolete during development.
[4] Add test on $input_file and $plugin_class. Test $text before calling
Handler or processing $text. No need to split undef if somehow Handler
returned undef, etc. If the routine returns an empty arrayref,
stage-marc-import will do fine.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Bug 12412 added the use of to_marc plugins allowing arbitrary file formats
in stage-marc-import (as long as the plugins can handle them). The feature
was not very visible in the code, and when bug 10407 added the marcxml
format, it made some changes that broke the use of to_marc.
This patch restores the functionality by:
[1] Adding a sub RecordsFromMarcPlugin to ImportBatch.pm, specifically
addressing the conversion from arbitrary formats to MARC::Record.
The original to_marc interface is used: pass it the file contents,
and it returns a string consisting of a number of MARC blobs separated
by \x1D.
Consequently, the call of to_marc is removed from routine
BatchStageMarcRecords where it did not belong. The to_marc_plugin
parameter is removed and two calls are adjusted accordingly.
[2] Instead of a separate combo with plugins, the format combo contains
MARC, MARCXML and optionally some plugin formats.
[3] The code in stage-marc-import.pl now clearly shows the three main
format types: MARC, MARCXML or plugin based.
Note: This patch restores more or less the situation after bug 12412, but
I would actually recommend to have the to_marc plugins return MARC::Record
objects instead of large text strings. In the second example I added a
to_marc plugin that actually converts MARC record objects to string format,
while RecordsFromMarcPlugin reconverts them to MARC::Records.
Test plan:
See second patch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
C4::Biblio::GetBiblio can be replaced with Koha Biblio->find
Test plan:
Import batch, view issue history, search for items, see the image of a
bibliographic record, modify and delete records in a batch
Followed test plan, works as expected.
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Make it more explicit by adding join statements.
Test plan:
See next patch for adding a unit test.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
To test:
1 - Import a batch with some items
2 - Delete one of the imported items
3 - Browse to Tools->Staged MARC record management
4 - Click (Create label batch) for the batch you imported
5 - Recieve an error
6 - Apply patch
7 - Click (Create label batch)
8 - Batch is created with remaning items from the import
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
We will need a few additional parameters for merge later on. This patch
puts the original parameters in a parameter hash.
For the same reason DelAuthority gets a parameter hash here.
Note: We remove the second parameter from the DelAuthority call in
authorities/authorities-home.pl here. It was not used and could have
presented problems in the future.
Test plan:
[1] Run t/db_dependent/AuthoritiesMarc.t.
[2] Run t/db_dependent/Authorities/Merge.t.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Script tools/batch_delete_records.pl only checks the return value of
DelAuthority. The return value depends on DBI rows, which is not always
reliable. It may return -1 when it 'does not know'.
Testing $@ in tools/batch_delete_records.pl should actually be enough.
The return value was discarded in C4/ImportBatch.pm. Removing an unused
variable.
Test plan:
[1] Pick an authority record with a few linked biblios.
Delete this authority record via tools/batch_delete_records.pl.
Check if the linked biblio records are cleaned up.
[2] Bonus: Make a typo in the SQL statement of DelAuthority. Check
if batch_delete_records shows you the error message.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
If we import items that have a wrong branch code, the items will not
be imported but manage-marc-import reports them as imported. (A wrong
branch code probably occurs most, but other causes are possible too.)
The underlying cause is that AddItem does not look at the error
returned from _koha_new_item in Items.pm.
This patch deals with that omission in the most economical way. It adjusts
AddItem so that it returns undef if no item was added.
In ImportBatch.pm I check if an item was added and adjust the totals
accordingly instead of just always counting them.
Note: Several scripts like additem.pl use AddItemFromMarc to call
AddItem. They do not report an error, but fail silently. With this patch,
these scripts get undef and will still fail silently. (No change.)
Adding error checks around calls of AddItemFromMarc is outside the scope of
this report. Here we are taking care of correct imported item numbers.
Test plan:
[1] Verify that additem.pl still works by adding a new item.
[2] Run t/db_dependent/Items.t
[3] Add a new branchcode, say XXX.
[4] Pick a biblio record with a few items (n) and set one item to branch XXX.
[5] Export this biblio with items to a MARC file.
[6] Change the XXX item to the original branch and remove branch XXX.
[7] Import the MARC file. Verify that one item was not imported and that
the number of imported items reflects that (equals n-1).
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch makes the following changes:
[1] Based on the groundwork of the former patch, add call to
RecordsFromMARCXMLFile in stage-marc-import. Use format param.
[2] Add format to the template. Use file extension to determine.
If you use .xml or .marcxml as extension, MARCXML is selected.
[3] In stage-marc-import.tt mark UTF-8 encoding as UTF-8 not as utf8.
[4] BatchStageMarcRecords: do not call plugin if you have no records.
[5] RecordsFromISO2709File: also return errors in an array.
[6] In misc/stage_file.pl also use UTF-8. Handling of errors from [5].
Test plan:
[1] Import an empty file as MARC or MARCXML (with Tools/Stage..import).
[2] Import an non-empty file with invalid contents as MARC or MARCXML.
[3] Export a few records with Tools/Export as MARC and MARCXML.
[4] Import these two files. Check selected format versus file extension.
[5] Import a MARCXML file with misc/stage_file.pl.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Patch from Olli, manual rebase by Marcel (July 7, 2016).
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Needs follow-up. Test plan in the third patch.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Remove $dbh as argument to C4::Items::DelItemCheck
and C4::Items::ItemSafeToDelete, also change all
calls to these functions throughout the codebase.
Also remove remaining reference to 'do_not_commit' in
t/db_dependent/Items_DelItemCheck.t
Fixed doubled "$$" in C4/ImportBatch.pm
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
To test:
1) Go to Tools -> Staged MARC Management and clean a file. If you have no files to clean, go to 'Stage MARC for import' and upload one to clean following the necessary steps.
2) Confirm that once the file has been cleaned, the Action column now shows a Delete button. Confirm this button only shows for cleaned files.
3) Click the Delete button.
4) Confirm that clicking Cancel exits the pop-up message and does not delete the file.
5) Confirm that clicking OK refreshes the list of staged records and the one you just deleted is no longer on it (has been deleted). You can confirm this by checking for the file in mysql (SELECT * FROM import_batches WHERE import_batch_id = X;)
6) Run prove -v t/db_dependent/ImportBatch.t (have written unit tests for CleanBatch and DeleteBatch)
Sponsored-by: Catalyst IT
Signed-off-by: Liz Rea <liz@catalyst.net.nz>
Catalyst sign off, so needs another one but YAY this is great.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
subroutines should not take $dbh in parameter.
C4::Biblio::TransformMarcToKoha has it and does not use it.
Test plan:
Look at the patch and confirm that all occurrences of
TransformMarcToKoha have been modified.
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
perl -p -i -e 's/^.*set the version for version checking.*\n//' **/*.pm
+ manual adjustements
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Mainly a
perl -p -i -e 's/^.*3.07.00.049.*\n//' **/*.pm
Then some adjustements
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
perl -p -i -e 's/^(use vars .*)\$VERSION\s?(.*)/$1$2/' **/*.pm
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
If batch of records is used to overlay existing records, and one of
those records is deleted, any attempt to revert this batch will fail.
The unanimous consensus is that deleted records should stay deleted (
https://lists.katipo.co.nz/public/koha/2015-June/043048.html ).
Test Plan:
1) Import a batch of records
2) Import the same batch again, using ISBN as a matcher for overlay
3) Find a record that overlayed a previous record and delete it
( the match type will be "match applied" and there will be a link in
"match details" )
4) Attempt to revert the second batch ( that overlayed the first )
5) The progress will stop at some point and never complete
6) Apply this patch
7) Attempt to revert the second batch again
8) This time it should succeed!
Signed-off-by: Do, Tam T <tdo@albright.edu>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Many libraries would like to be able to import various types of files as
MARC records ( citations, csv files, etc ). We can add a new function to
the plugins system to allow that kind of behavior at a very custom
level.
Test Plan:
1) Ensure you have plugins enabled and configured correctly
2) Installed the attached version 2.00 of the Kitchen Sink plugin
3) Download the attached text file
4) Browse to "Stage MARC records for import"
5) Select the downloaded text file for staging
6) After uploading, you should see a new area "Transform file to MARC:",
select "Example Kitchen-Sink Plugin" from the pulldown menu
7) Click 'Stage for import"
8) Click 'Manage staged records"
9) You should now see two new MARC records!
Signed-off-by: Aleisha <aleishaamohia@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described - interesting new feature.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
http://bugs.koha-community.org/show_bug.cgi?id=9987
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
When records are imported into Koha, the items is stored into the
import_items table.
This marcxml in this table is never retrieved to display items.
Test plan:
1/ Import a records with items
2/ Before importing the batch into the catalog, you can see the marc
of the records, in the table below.
3/ Verify that the items is correctly displayed.
QA note: This patch does not provide test for new subroutines but the
module (C4::ImportBatch) is not tested at all and it will be time
consuming to provide them.
Signed-off-by: Nicole Engard <nengard@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Existing framework code is currently not retained when local record
gets replaced during batch import, or when the restore/reverse function
is being used.
This patch fixes aforementioned issues by correcting outdated GetBiblio()
calls in C4/ImportBatch.pm
To test:
1/ try to replicate the issue: import some MARC records with
"Tools -> Stage MARC records for import" etc., using test setup,
matching rules and so on such that some existing records will get
replaced with imported ones
2/ observe that records replaced during import now open in the editor
with 'Default' framework, even if they have some other framework
set up previously
3/ apply patch
4/ redo 1/, confirming that this problem is no longer replicable
5/ try use 'restore' function with some freshly imported
records, ensure that original framework code got retained
in the records which had their imports reverted
NOTE: I confirmed this change by
1) git grep "=\s*GetBiblio\s*("
-- this shows how GetBiblio is called elsewhere.
It differed! So then...
2) vi C4/Biblio.pm
/sub GetBiblio
-- find the right one, notice it only returns a HASH ref,
not an array.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
TODO:: Somebody who knows what these are about could write some documentation about it.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
- GetImportBiblios: prepare the query is useless here
- use Modern::Perl
- Remove useless indentation level in pl file and redirect to 404 if
required params are not given.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
New patch should apply correctly.
modified: C4/ImportBatch.pm
modified: koha-tmpl/intranet-tmpl/prog/en/modules/tools/manage-marc-import.tt
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Bug 11876 - Diff column is now displayed.
If a match is found, a 'View' link appears in the Diff column.
Old patch was not implementing datatables, and therefore was not compatible with this version of Koha.
modified: koha-tmpl/intranet-tmpl/prog/en/modules/tools/manage-marc-import.tt
modified: tools/batch_records_ajax.pl
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
These patches have some issues but I'm going to submit a follow-up which
I think will address them all and offer some improvements as well.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
When importing records, the ISBN is normalized and stored
into database (see C4::ImportBatch::_add_biblio_fields). But when
searching with ISBN into reservoir, it is not normalized
(see C4::Breeding::BreedingSearch). So search does not match.
This patch adds the normalisation to reservoir search. Also, it
replaces call private method _isbn_cleanup by GetNormalizedISBN,
the correct public method. Also allows the reservoir search
on ISBN with hyphens.
This is intended to fix only reservoir searches.
Revised Test plan
-----------------
1) Back up DB
2) Save copy of attached example somewhere findable
2) Home -> Tools -> Stage MARC records for import
3) Click Browse and select the example MARC file
4) Click Upload file
5) Tweak as desired then click Stage for import
6) Click Manage staged records
7) Click Import this batch into the catalog
8) Home -> Cataloging
9) In the Cataloging search text box type 978-0-691-14289-0 and
click Submit
-- ISBN13 with hypens not found in reservoir
10) In the Cataloging search text box type 9780691142890 and
click Submit
-- ISBN13 without hypens not found in reservoir
11) In the Cataloging search text box type 0-691-14289-0 and
click Submit
-- ISBN10 with hypens not found in reservoir
12) In the Cataloging search text box type 0691142890 and
click Submit
-- ISBN10 without hypens found in reservoir
13) Apply patch
14) Repeat steps 9-12, this time it is always found! :)
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
When the ability to stage authority records was added to Koha, sorting
record batches by citation ( i.e. title ) caused the addition of
"authorized_heading" to be added to the sort. When sorting by title
descending, this causes the order by clause to be "title,
authorized_heading DESC" which means sort by title ASC, then
authorized_heading DESC. This is incorrect and causes regular biblio
batches to always be sorted ascending.
Test plan:
1) Stage a batch of biblio records from a file
2) View the staged batch
3) Attempt to sort by title descending
4) Note it is still sorted by title ascending
5) Apply this patch
6) Note the sorting now works correctly
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Works as advertised. The code pertaining to sorting in routine
GetImportRecordsRange will probably not win beauty prizes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch addresses a number of issues with the main patch:
- regression on bug 2060 (i.e., displaying authority import batches
correctly)
- regression on bug 10170 (translation of import record states)A
- use of datatables.inc
- lack of clarity as to the licensing of tools/batch_records_ajax.pl
- insufficent sanitizing of input used to generate an SQL statement
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Some libraries would like to sort by columns for the records of an
import batch. This seems like a good use of Ajax DataTables.
Test plan:
1) Apply this patch
2) Import a record batch into Koha
a) Use some form of matching
b) Have some records that will match and some that won't
c) Have at least 30 records so you can test the pager
3) Verify the new table is functionally equivalent to the old static one
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Tests fine and looks good with the exception of the corrections I put in
a follow-up.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Bug 8015: Catch error in the SetUTF8Flag routine
The eval avoids the interface to run endlessly if an error occurred.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
The MARC Modification Templates system gives Koha users
the power to make alterations to MARC records automatically
while staging MARC records for import.
This tool is useful for altering MARC records from
various venders work with your MARC framework.
The system essentially allows one to create a basic script
using actions to Copy, Move, Add, Update and Delete fields.
Each action can also have an optional condition to check
the value or existance of another field.
The Copy & Move actions also support Regular Expressions,
which can be used to automatically modify field values during the
copy/move. An example would be to strip out the '$' character
in field 020$c.
Furthermore, the value for an update can include variables
that change each time the template is used. Currently,
the system supports two variables, __BRANCHCODE__ which
is replaced with the branchcode of the library currently
using the template, and __CURRENTDATE__ which is replaced
with the current date in ISO format ( YYYY-MM-DD ).
At its simplist, it can perform functions such as:
Copy field 092$a to 952$c
At its most complex it can run actions like:
Copy field 020$c to 020$c using RegEx s/\$// if 020$c equals RegEx m/^\$/
Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch adds the ability to overlay by either itemnumber,
or barcode. Itemnumbers take precendence over barcodes, which
allows us to batch update item barcodes with an overlay.
Test Plan:
1) Create a new record with 2 items, make sure to give it a unique ISBN
2) Download the record as MARCXML
3) Edit the MARC XML
a) Delete one of the two items
b) Change the barcode in the barcode field to something unused
4) Transform the xml file into marc with xml2marc
5) Browse to 'Stage MARC records for import'
6) Upload the binary marc file
7) Choose the following options:
Record matching rule: ISBN
Action if matching record found: Ignore incoming record
Action if no match is found: Ignore incoming record
Check for embedded item record data: Yes
How to process items: Replace items if matching bib was found
8) Click 'Stage for import' button
9) Verify a matching record was found, then click 'Manage staged records' link
10) Verify the rules are still set correctly
11) Click 'Import this batch into the catalog'
12) The import should tell you:
1 record was ignored
1 item was replaced
13) View the record details and verify the item's barcode was replaced
with your updated barcode value
14) Download the record as MARCXML
15) Edit the MARC XML
a) Delete one of the two items
b) Delete the itemnumber field for the remaining item
c) Alter the item's callnumber to a new value
16) Repeat steps 4 through 12
17) View the record details and verify the item's callnumber was replace
with your updated callnumber value
Signed-off-by: Henry Bankhead <hbankhead@losgatosca.gov>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
When staging biblios with items attached you previously had only two
options, add or don't add.
This patch adds a third option to replace an item record if a match is
found on itemnumber or barcode, else it adds the item.
Test Plan:
1) Stage a file of biblios with items attached.
2) Import the batch into the catalog.
3) Run the indexer so the matcher will match
4) Modify the item data for at least one bib in the file
5) Re-stage the file with the item matching option set to "Replace
items if matching bib was found"
6) Let the indexer run again
7) You should see updated item information after the overlay
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Henry Bankhead <hbankhead@losgatosca.gov>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
The patch for bug 9523 added a JOIN to the biblio table when identifying
the best match so that if a matched record had been deleted it would
not hold up the import process. Unfortunately, this broke all authority
matching, since of course authorities don't appear in the biblio table.
This patch adds a join to auth_header as well, and decides which to
check based on the record type.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Comment on third patch of this series.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Test Plan:
1) Stage a MARC record file that will have matches with existing records
2) Delete the bib from Koha that was matched on
3) Attempt to import the records into Koha, the import will hang
4) Apply the patch
5) Reload manage-marc-import.pl and attempt to import again, this time it should succeed.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Test Plan:
1) Catalog a new record with an ISBN
2) Add some items to the record
3) Download the record as MARCXML
4) Delete the itemnumbers from the 952 fields in the record,
Change the barcode fields to unused barcodes
5) Use xml2marc to save as a standard MARC file
6) Import the record using the 'Stage MARC for import' tool
Use the settings:
Record matching rule: ISBN
Action if matching record found: ignore
Action if no match found: ignore
Item processing: always_add
Check for embedded item record data?: Yes
How to process items: Always add items
7) Import, note the bib is ignored, and the items are not processed
8) Undo import into catalog
8) Apply this patch
9) Import this batch into the catalog
10) Note the items were processed and are now added to the matching
record
Signed-off-by: Mathieu Saby <mathieu.saby@univ-rennes2.fr>
Tested with UNIMARC record. I followed the test plan, just changing 952 by 995
Signed-off-by: Mason James <mtj@kohaaloha.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Only changing some documentation about GetXmlBiblio
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Added the word 'contain'
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Some Z39.50 server may use the MARC-8 encoding, which uses separated
diacritics. By forcing a normalization, all imported records will have
combined diacritics.
Records with separated diacritics might not show up in Zebra searches if
the search terms use accented characters.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
http://bugs.koha-community.org/show_bug.cgi?id=8610
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
checked it still works after the patch with UNIMARC and BNF server (that
provide utf-8 records)
A subroutine was not being imported by C4::ImportBatch (ironic, no?)
so this patch makes the call fully-qualified. This patch also cleans
up two warnings in C4::Auth that are raised when logged in as the
database user.
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
The staged MARC management script was not correctly informing
the decoder ring that we had UNIMARC authorities, and the decoder
ring was dutifully trying to turn the authority records into a
bibliographic box of cereal.
Expose authority import functionality to the command line import
scripts, and rename them from commit_biblios_file.pl and
stage_biblios_file.pl to commit_file.pl and stage_file.pl.
To test (note that these instructions assume you have a MARC21
installation and are using the provided sample file):
1. Find a file of authorities (a sample file with MARC21 authorities
is attached to bug 7475) and download it to your server
2. Stage the file using the following command (replace <filename> with
the name of the file you saved in step 1):
> misc/stage_file.pl --file <filename> --authorities
3. Note the batch number the script assigns to your batch
4. Commit the records using the following command (replace <batchnumber>
with the batch number you made note of in step 3):
> misc/commit_file.pl --batch-number <batchnumber>
5. Index the authorities Zebraqueue (or wait)
6. Confirm that the new authorities appear.
7. Create a matching rule with the following settings:
Code: AUTHTEST
Description: Personal name main entry
Match threshold: 999
Record type: Authority record
Search index: Heading-main
Score: 1000
Tag: 100
Subfields: a
Offset: 0
Length: 0
(note the ID of this matching rule)
8. Stage the authority file again, this time using the following
command:
> misc/stage_file.pl --file <filename> --authorities \
--match <matchingrule>
7. Revert the import with the following command:
> misc/commit_file.pl --batch-number <batchnumber> --revert
8. Index the authorities Zebraqueue (or wait)
9. Confirm that the records have been removed
10. Import an authority record with the Stage MARC/Manage staged MARC
tools in exactly the way you would for a bibliographic record,
but choose "Authority" instead of "Bibliographic" for the record
type.
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>
Testing plan delivers as it should.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
* Add the code necessary to handle authorities with matching rules and
import batches.
* Update all the scripts that use the matcher and import batch code
to use the new API.
* Add authority records to the matching rules interface in the staff
client.
http://bugs.koha-community.org/show_bug.cgi?id=2060
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
this patch checks the "on loan" and "reserved" status before deleting item, and
do noting in this case, so the record can't be deleted due to existing item.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
svc/import_bib:
* takes POST request with parameters in url and MARC XML as DATA
* pushes MARC XML to an impoort bach queue of type 'webservice'
* returns status and imported record XML
* is a drop-in replacement for svc/new_bib
misc/cronjobs/import_webservice_batch.pl:
* a cron job for processing impoort bach queues of type 'webservice'
* batches can also be processed through the UI
misc/bin/connexion_import_daemon.pl:
* a daemon that listens for OCLC Connexion requests and is compliant
with OCLC Gateway spec
* takes request with MARC XML
* takes import batch params from a config file and forwards the lot to
svc/import_bib
* returns status
ImportBatches:
* Added new import batch type of 'webservice'
* Changed interface to AddImportBatch() - now it takes a hashref
* Replaced batch_type = 'batch' with
batch_type IN ( 'batch', 'webservice' ) in some SELECTs
Signed-off-by: MJ Ray <mjr@phonecoop.coop>
On some record, the commit_biblio_file is creating wide
character
because as_xml is not used with correct parameter.
This patch fixes that.
To test on a UNIMARC Koha, stage attachment 7510 and
then import.
It hangs before the patch, it passes after.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Rather than having options for MARC21 and UNIMARC in the "Character encoding"
dropdown, the user should be able to select the appropriate character encoding.
The default retains the current behavior, which is to allow the system to guess
which character encoding is in use. However, it should be noticed that this is
almost always wrong for non-UTF8 records with non-ASCII characters. Specifying
a character set is much more reliable if you're not using UTF-8.
Rebased to use Template::Toolkit instead of HTML::Template::Pro.
Signed-off-by: Jared Camins-Esakov <jcamins@bywatersolutions.com>
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Fixed glitch where the first page of bibs in a batch (or the first
page of import batches) was displaying the entire list instead
of the correct number of records per page.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
- basket.pl: updating display, formatting dates,
- neworderempty: updating display, removing useless code, using ACQ framework if it exist. The ACQ framework will be used for creating items record during acquisitions. If it does not exist, default is used instead (which has many more informations, lot of them being irrelevant during acquisition, like the barcode)
- new order from imported batch: rewrite of the workflow. Now uses neworderempty and changing status of import_record to 'imported'
- s/copyrightdate/publicationyear/ as it's what libraries uses when ordering
- fixing some warnings
-
batches, it is now possible to 'clean' a batch by
removing all bib and item records staged in the batch. This
has the effect of helping to reduce database space used
by old import batches as well as removing staged records
from the cataloging reservoir search. Note that 'cleaning'
a batch affects only the copies of the records that were staged;
if the batch was committed, cleaning the batch does not
affect any bibs and items that were committed into the catalog.
Also note that once you clean a committed batch of records, it is
impossible to undo the previous commit operation.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Also fix issues with normalizing ISBNs and the default
normalizer in C4::Matcher.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The feature in question is described in bug 2952; to
summarize the enhancement, which the earlier patch
description did not do, the list of bibs in an
import record batch now includes a column linking
each import record to the bib that was actually created
or updated when the import batch was committed.
The improvements in this patch are:
* If bib in import batch has not been committed, it
has not been linked to a matching new or updated bib.
In that case, do not create a link to a guaranteed
404 (/cgi-bin/koha/catalogue/detail.pl?biblionumber=)
* When reverting an import batch, set matched_biblionumber
to NULL for affected records - otherwise, the Bib
column will include links to bibs that may no longer
exist.
* Fixed a minor HTML validation error.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
When recommiting a partially completed MARC
record batch, records that were already imported
(or had an error status) were being processed
again, leading to duplicate bibs. Corrected
so that these records are actually ignored.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Enhanced the ability of catalogers to specify how
bib and item records should be added, replaced, or
ignored during a staging import.
When an import batch of bib records is staged and commit,
the user can now explicitly specify what should occur
when an incoming bib record has, or does not have, a match
with a record already in the database. The options are:
if match found (overlay_action):
create_new (just add the incoming record)
replace (replace the matched record with the incoming one)
use_template (option not implemented)
ignore (do nothing with the incoming bib; however, the
items attached to it may still be processed
based on the item action)
if no match is found (nomatch_action):
create_new (just add the incoming record)
ignore (do nothing with the incoming bib; in this
case, any items attached to it will be
ignored since there will be nothing to
attach them to)
The following options for handling items embedded in the
bib record are now available:
always_add (add the items to the new or replaced bib)
add_only_if_match (add the items only if the incoming bib
matches an existing bib)
add_only_if_add (add the items only if the incoming bib
does *not* match an existing bib)
ignore (ignore the items entirely)
With these changes, it is now possible to support the following use cases:
[1] A library joining an existing Koha database wishes to add their
items to existing bib records if they match, but does not want
to overlay the bib records themselves.
[2] A library wants to load a file of records, but only handle
the new ones, not ones that are already in the database.
[3] A library wants to load a file of records, but only
handle the ones that match existing records (e.g., if
the records are coming back from an authority control vendor).
Documentation changes:
* See description above; also, screenshots of the 'stage MARC records
for import' and 'manage staged MARC records' should be updated.
Test cases:
* Added test cases to exercise staging and committing import batches.
UI changes:
* The pages for staging and managing import batches now have
controls for setting the overlay action, action if no match,
and item action separately.
* in the manage import batch tool, user is notified when they
change overlay action, no-match action, and item action
* HTML for manage import batch tool now uses fieldsets
Database changes (DB rev 076):
* added import_batches.item_action
* added import_batches.nomatch_action
* added 'ignore' as a valid value for import_batches.overlay_action
* added 'ignored' as a valid value for import_records.status
* added 'status' as a valid value for import_items.status
API changes:
* new accessor routines for C4::ImportBatch
GetImportBatchNoMatchAction
SetImportBatchNoMatchAction
GetImportBatchItemAction
SetImportBatchItemAction
* new internal functions for C4::ImportBatch to
determine how a given bib and item are to be
processed, based on overlay_action, nomatch_action,
and item_action:
_get_commit_action
_get_revert_action
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Added invocations of StripNonXmlChars to uses
of new_from_xml() that involve records
saved to Koha fields via MARC::Record->as_xml();
for batch jobs that work on MARC XML files
coming from external sources, StripNonXmlChars
should not necessarily be used, as it may
be better to reject a file or record if it
contains that kind of encoding error.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* IsStringUTF8ish - determine if scalar contains a string in UTF8
* MarcToUTF8Record - convert MARC blob or MARC::Record to UTF8
* SetMarcUnicodeFlag - set appropriate MARC21 or UNIMARC field to
indicate that record is in UTF-8.
Design points of this module include:
* No dependencies on other C4 modules, making it easier to add
more test cases
* All character conversion code in one place
* Single entry point for doing a character conversion on a
MARC record
* Capture of errors and warnings produced by Text::Iconv
and MARC::Charset
* Start of support for guessing the source character set of
a MARC record.
Several functions were moved from other scripts
or modules to C4::Charset:
* C4::Koha->FixEncoding (expanded and renamed
MarcToUTF8Record)
* C4::Koha->char_decode5426
* fMARC8ToUTF8 from bulkmarcimport.pl (renamed
_marc_marc8_to_utf8)
Several batch jobs were adjusted to use MarcToUTF8Record instead of
FixEncoding.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The problem occured during the population of
import_items.marcxml -- the MARC::Record object
created to store the item did not have the Leader/09
set to 'a', which means that MARC::File::XML
tried to transcode code the item from MARC-8 to UTF-8, which
breaks since the MARC data is already in UTF-8 at that point.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Batch import was not removing item fields (e.g., 952
or 995) from MARC records.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Introduced C4::Items module to separate items API
from biblio API. Details on changes will be
put in later commit messages.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
If a MARC batch was imported, then reverted, the 952s
from the saved copied of the bib were added back
when the bib was replaced, leading to duplication
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Enhancement to store the matching rule associated with an
import batch and to allow the current matching rule in
effect to be changed and the duplicate detection redone.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
As part of this, modified two routines in C4::ImportBatch
to support a callback for monitor progress of import
processing.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Revamps the import options on the tools menu to have two parts:
[1] Staging (load file into reservoir)
[2] Managing (review the list of staged batches, then
choose to commit or undo a given batch.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* Completely removed old marc_breeding table
* Started updated Tools import function to stage records
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
