MARC::Record and MARC::File::* modules sometimes use the position 09 of
the leader to detect encoding. A blank character means 'MARC-8' while an
'a' means 'UTF-8'.
In a UNIMARC authority this position is used to store the authority type
(see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]).
In this case, 'a' means 'Personal Name'.
The result is that the import will succeed for a Personal Name
authority, but it will fail for all other authority types.
Steps to reproduce:
0. Be sure to have a Koha UNIMARC instance.
1. Download the MARCXML for "Honoré de Balzac"
curl -o balzac.marcxml https://www.idref.fr/02670305X.xml
2. Verify that it's encoded in UTF-8
file balzac.marcxml
(should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode
text")
3. Go to Tools » Stage MARC for import and import balzac.marcxml with
the following settings:
Record type: Authority
Character encoding: UTF-8
Format: MARCXML
Do not touch the other settings
4. Once imported, go to the staged MARC management tool and find your
batch. Click on the authority title "Balzac Honoré de 1799-1850" to
show the MARC inside a modal window. There should be no encoding
issue.
5. Write down the imported record id (the number in column '#') and go
to the MARC authority editor. Replace all URL parameters by
'breedingid=THE_ID_YOU_WROTE_DOWN'
The URL should look like this:
/cgi-bin/koha/authorities/authorities.pl?breedingid=198
You should see no encoding issues. Do not save the record.
6. Import the batch into the catalog. Verify that the authority record
has no encoding issue.
7. Now download the MARCXML for "Athènes (Grèce)"
curl -o athènes.marcxml https://www.idref.fr/027290530.xml
8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you
should see encoding issues and that the position 9 of the leader was
rewritten from 'c' to 'a'. Strangely, importing this batch fix the
encoding issue, but we still lose the information in position 09 of
the leader
This patch makes use of the MARCXML representation of the record instead
of the ISO2709 representation, because, unlike
MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to
pass directly the encoding and the format, which prevents data to be
double encoded when position 09 of the leader is different that 'a'
Test plan:
- Follow the "steps to reproduce" above and verify that you have no
encoding issues.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 01d78e1ec7)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
1 - git grep ModAuthInBatch
2 - Confirm none of these occurences are calls
3 - Apply patch
4 - git grep ModAuthInBatch
5 - No occurences
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit d07e692823)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
To test:
1 - git grep ModBiblioInBatch
2 - Confirm these occurences are not calls
3 - Apply patch
4 - git grep ModBiblioInBatch
5 - No occurences
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 46fabf9bb5)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch fixes the broken commit_file.pl script.
It doesn't deal with commiting the import from the UI.
To test:
1. Pick a file for staging:
$ kshell
k$ misc/stage_file.pl --file TestDataImportKoha.mrc
=> SUCCESS: All good
2. Commit!
k$ misc/commit_file.pl --batch-number 1
=> FAIL: You see
DBIx::Class::Storage::DBI::_exec_txn_begin(): DBI Exception: DBD::mysql::db begin_work failed: Already in a transaction at /kohadevbox/koha/C4/Biblio.pm line 303
3. Apply this patch
4. Repeat 2
=> SUCCESS: Commit succeeds
5. Sign off :-D
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 29325: (QA follow-up) Remove unexisting parameters of BatchRevertRecords
There is no interval and callback as in BatchCommitRecords.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 29325: Call progress callback one last time to confirm comppletion
Previously after finishing the loop we were still in a transaction that never completed - we should report progress when done
one final time to commit the last records
To test:
1 - Stage a file with > 100 records
2 - Commit file
3 - Confirm batch is imported and no records left as staged
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 29325: Fix import from staff client
same test as before, but via the staff client
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 29325: Handle the transaction in BatchCommitRecords
Requiring the callback to commit was breaking reversion, and likely elsewhere
Let's simplify and say that the routine iteself will handle the txn and commit
TO test:
1 - Stage a file
2 - Import a file
3 - Revert a file
4 - Test staff client and command line
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 29325: (QA follow-up) Tidy up
tools/manage-marc-import.pl: sub commit_batch does no longer need $schema
tools/manage-marc-import.pl: sub revert_batch: calling BatchRevertRecords which has no interval and callback
misc/commit_file.pl: sub print_progress_and_commit does no longer commit, renamed
misc/commit_file.pl: sub print_progress does no longer have a schema parameter
misc/commit_file.pl: sub revert_batch reported deleted items as added
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Looking at the code this 'import_error' column is empty for the last 9 years. Not sure it makes much sense to have this single error now.
commit 1dba9c6409
Date: Wed Oct 10 14:21:22 2012 -0500
Bug 7131: teach MARC import how to overlay items
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 3c0c375d8e)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We are trying to insert "duplicate item barcode" into import_items.itemnumber (integer),
it fails with "Incorrect integer value: 'duplicate item barcode' for column 'itemnumber' at row 1"
To reproduce:
Export a biblio with an item
Import it
=> The item is not added, and there is no new row in import_items.
The error only appears in the log if you comment the close STDERR and
close STDOUT lines
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit c7a8a456c7)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Previously the sorting took 'chosen' into account and would move a selected match to the
top on next load - it is better to preserve the same sorting every time
When loading matches the 'cehcked' variable was not being cleared, so multiple matches were
being marked 'checked="checked"'. Fixing this ensures the correct record displays as chosen
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Ben Daeuber <bdaeuber@cityoffargo.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patchset adds the display of all matches found during import to the import management screen
A staff member with the permission to manage batches will be able to select for any individual record which match, or none, should be used during import
To test:
1 - Import a batch of records or export existing records from your catalog
2 - Import the file (again) and select a matching rule that will find matches
3 - Note that you now have radio buttons allowing you to select a record, or none
4 - Test scenarios:
I - When 'Action if matching record found' is 'Ignore'
a - Imported record ignored if match is selected
b - 'Action if no match found' followed if no match is selected (Ignore matches)
II - When 'Action if matching record found' is 'Replace'
a - The chosen record is the one overlayed (you can edit the chosen record before importing to confirm)
b - 'Action if no match found' followed if no match is selected (Ignore matches)
III - When 'Action if matching record found' is 'Add incoming record'
a - Record is added regardless of matches
5 - Confirm 'Diff' 'View' links work as expected
6 - Confirm that after records are imported the radio buttons to choose are disabled
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Bug 22785: API files
Signed-off-by: Ben Daeuber <bdaeuber@cityoffargo.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
The following authorities subroutines are used by the ImportBatch
script but are not accessible, because they aren't imported. This
caused MARC importing errors.
- GetAuthorityXML
- ModAuthority
- DelAuthority
These affected the BatchCommitRecords and BatchRevertRecords routines,
and it wasn't caught because there are no tests written for them.
To test:
1. Find an existing authority record, or import one to use.
2. Save this authority record (export/download).
3. Go to Admin -> Record matching rules. Create a new record matching
rule for authorities that matches on 001, Local-Number index.
4. Go to Tools -> Stage MARC for import. Upload the authority file you
just downloaded.
5. Change record type to authority.
6. Under 'Look for existing items in catalog?', set the record matching
rule to the rule you just made which matches on 001. If matching
record found, replace the existing one. If no match is found, ignore.
7. Stage the record. Once complete, a match should've been found.
8. Go to Staged MARC management.
9. Import the batch into the catalog. Notice it hangs and never
completes.
10. Apply the patch and restart services.
11. Repeat steps 4-9. This time importing should be successful.
Sponsored-by: Educational Services Australia SCIS
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch reuse the (awesome) Koha::Result::Boolean module to retrieve
the return of Koha::Item->safe_to_delete.
Test plan:
Try to delete an item that has previously been checked out and confirm
that you are still blocked.
Try using the cronjobs, the item and biblio detail pages, as well as the
batch delete item tool.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch renames the (passed through) 'context' param for
'overlay_context'. I propose doing so, because in Koha-land 'context'
has a special meaning, related to C4::Context and it reads ambigous.
The patch itself is pretty trivial.
Tests should pass:
1. Run:
$ kshell
k$ prove t/db_dependent/Biblio/MarcOverlayRules.t
=> SUCCESS: Tests pass
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests still pass!
4. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 14957: (follow-up) Clarify 'context' param
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add a rule based system for merging MARC records to for example
prevent field data from being overwritten.
To test:
1. Apply this patch.
2. Log in to staff client.
3. Enable new syspref MARCMergeRules.
4. Click the new link "MARC merge rules" in the "Catalog"
section of the Koha administration page.
5. Create a new rule:
Module: source, Filter: *, Tag: 245, Preset: Protect.
6. Clicking "Edit" should allow you to edit corresponding rule.
7. Clicking "Delete" should remove corresponding rule after confirmation.
8. Selecting one or more rules followed by clicking "Delete
selected" should remove all selected rules after confirmation.
9. Try creating a rule with tag set to "**", the other options does
not matter. Verify that saving this rule produces an error
message complaining about invalid tag regular expression.
10. Try creating a rule with tag set to "008" (or other control
field) and set Appended: Append and Removed: Skip, the other
options does not matter. Verify that saving this rule produces
an error message complaining about invalid combination of actions
for control field.
11. With the 245 rule in step 5 in place, edit a bibliographic record,
change 245a for example (which should be Title for MARC21) and save.
12. Verify that the changes has not been saved.
13. Create a new rule:
Module: source, Filter: intranet, Tag: 245, Preset: Overwrite.
14. Repeat step 12, and verify that the changes has now been saved.
15. Run tests in t/db_dependent/Biblio/MarcMergeRules.t and very
that all tests pass.
Sponsored-by: Halland County Library
Sponsored-by: Catalyst IT
Sponsored-by: Gothenburg University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Christian Stelzenmüller <christian.stelzenmueller@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended: POD line for $import_record_id.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
SetImportBatchStatus is not called with $batch_id
It has been caught by bug 25026, and www/search_utf8.t fails with
t/db_dependent/www/search_utf8.t .. 14/87 Error POSTing http://koha:8081/cgi-bin/koha/tools/manage-marc-import.pl: Internal Server Error at t/db_dependent/www/search_utf8.t line 240.
And, from logs:
manage-marc-import.pl: C4::ImportBatch::SetImportBatchStatus(): DBI Exception: DBD::mysql::st execute failed: Truncated incorrect DOUBLE value: 'importing' [for Statement "UPDATE import_batches SET import_status = ? WHERE import_batch_id = ?" with ParamValues: 0=undef, 1='importing'] at /kohadevbox/koh
a/C4/ImportBatch.pm line 579: /kohadevbox/koha/tools/manage-marc-import.pl, referer: http://koha:8081/cgi-bin/koha/tools/manage-marc-import.pl?import_batch_id=2
Test plan:
Read the changes and confirm it does make sense.
Import and revert a batch
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Prior to ug 23463 AddItemFromMarc where calling AddItem, that did not
take into account the itemnumber field.
Now that we are using Koha::Item, we need to remove the items.itemnumber
field from the MARC record
Test plan:
1 - find an existing bib in your system with just one item
2 - export that bib with the item attached
3 - delete the barcode from your item in Koha
4 - stage your exported marc file for reimport, match on biblionumber, set it to Always Add Items
5 - confirm that the bib matches and the incoming 952 is parsed
6 - click "Import this batch into the catalog"
=> Without this patch you get (in the logs, or hidden)
manage-marc-import.pl: DBD::mysql::st execute failed: Duplicate entry '23' for key 'PRIMARY' [for Statement "INSERT INTO `items` ( `barcode`, `biblioitemnumber`, `biblionumber`, `ccode`, `cn_sort`, `cn_source`, `damaged_on`, `dateaccessioned`, `datelastborrowed`, `datelastseen`, `holdingbranch`, `homebranch`, `itemcallnumber`, `itemlost_on`, `itemnumber`, `itype`, `location`, `more_subfields_xml`, `onloan`, `permanent_location`, `replacementpricedate`, `timestamp`, `withdrawn_on`) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, current_timestamp, ? )" with ParamValues: 0="BC_23", 1=8, 2=8, 3="REF", 4='CN__23', 5=undef, 6=undef, 7="2014-09-04", 8=undef, 9="2014-09-04", 10="FPL", 11="FPL", 12="CN_23", 13=undef, 14="23", 15="BK", 16="GEN", 17=undef, 18=undef, 19="GEN", 20="2014-09-04", 21=undef] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1836.
manage-marc-import.pl: DBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry '23' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 169
manage-marc-import.pl: {UNKNOWN}: Transaction aborted: Duplicate ID. Rollback failed: DBIx::Class::Storage::txn_rollback(): Refusing to roll back without a started transaction at /kohadevbox/koha/tools/manage-marc-import.pl line 253 at /kohadevbox/koha/tools/manage-marc-import.pl line 253
=> With this patch applied, the new item must be added to the existing bibliographic record
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the logic and the needed UI elements to be able to pre-load an import profile. It also displays which profile was used to stage an import in staged import manager.
To test:
1. Apply all patches
2. Updatedatabase
3. Go to Stage MARC records for Import tool in admin, and upload a file with MARC records.
CHECK => after uploading, there is a fieldset with the legend “Profile settings”
=> inside the fieldset there is a select labeled “Pre fill values with profile”. The only value it has is “Do not use profile”.
4. Change some settings, and set “profile 1” as profile name and click on “Add profile”
SUCCESS => The select now has the new profile selected
5. Change profile select to “Do not use profile”
SUCCESS => Default values are now displayed in the form
6. Reload the page and upload the file again
SUCCESS => the select still has the profile recently added
7. Select the profile, change some parameter in the form and set the profile name to “profile 2”, and click add profile
SUCCESS => there are two profiles now, and if you toggle between them, the parameter changes
8. Select profile 1, change one parameter and click on update profile
SUCCESS => if you toggle that profile with the other, the new parameter of the value is shown when you select profile 1
9. Select profile 2, change some parameter and click Add profile (leaving the name as profile 2)
SUCCESS => the page complains there is another profile with the same name, and asks if you want to replace it.
10. Click on accept
SUCCESS => profile 2 now has the new value in the parameter
11. Select profile 2 and change the name to profile 1
SUCCESS => the page complains there is another profile with that name, and asks if you want to replace it
12. Click on accept
SUCCESS => in profile select there is only one profile called profile 1 that has the values of profile 2
13. Select profile 1 and click remove profile
SUCCESS => there is no profile in profile select.
14. Create a profile and click on “Stage for import”
15. Go to Staged MARC management page
SUCCESS => Improt should have the name of the profile in profile column, and when you click on the file name, there should be the name of the profile in the details.
16. prove t/db_dependent/ImportBatch.t t/db_dependent/api/v1/import_batch_profiles.t
17. Sign off
Signed-off-by: Abbey Holt <aholt@dubuque.lib.ia.us>
Signed-off-by: Abbey Holt <aholt@dubuque.lib.ia.us>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds a .perlcriticrc (copied from qa-test-tools) and fixes
almost all perlcrictic violations according to this .perlcriticrc
The remaining violations are silenced out by appending a '## no critic'
to the offending lines. They can still be seen by using the --force
option of perlcritic
This patch also modify t/00-testcritic.t to check all Perl files using
the new .perlcriticrc.
I'm not sure if this test script is still useful as it is now equivalent
to `perlcritic --quiet .` and it looks like it is much slower
(approximatively 5 times slower on my machine)
Test plan:
1. Run `perlcritic --quiet .` from the root directory. It should output
nothing
2. Run `perlcritic --quiet --force .`. It should output 7 errors (6
StringyEval, 1 BarewordFileHandles)
3. Run `TEST_QA=1 prove t/00-testcritic.t`
4. Read the patch. Check that all changes make sense and do not
introduce undesired behaviour
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In an OO package, the logger initialization should happen in the
constructor. This is not an OO package and the initialization is
happening on loading it. This is a wrong behaviour and certainly breaks
in environments where initialization cannot happen (package building,
for example). There could be several options to solve this, as it is
used in a single sub on this package, I opted for initializing on that
sub.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
The MARC preview of staged records fails when a staged record contains
items with UTF-8 characters (see attached screenshots). That's because
in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml()
does not explicitly specify 'UTF-8' as the optional encoding to use.
This patch fixes that.
Test plan (in a MARC21 instance):
0) Stage for import the test record attached to this bug report (make
sure that 'Check for embedded item record data?' is set to 'Yes'),
then go to Home > Tools > Manage staged MARC records and click on
the filename you've just staged.
1) In the 'Citation' column, click on the hyperlink to launch the MARC
preview -- notice how it fails to produce any output apart from the
modal heading.
2) Apply the patch, and restart Plack if necessary.
3) Re-launch the MARC preview: this time the output should include the
whole record, including the embedded item data in field 952.
4) Run the unit tests to ensure nothing was broken inadvertently:
$ prove t
$ prove xt
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
We certainly faced 3 similar bugs due to this syntax: bug 23006, bug
22941 and bug 17526.
To prevent other issues related to this syntax this patch suggests to
replace them all in one go.
Test plan:
Confirm that the 2 syntaxes are similar
Eyeball the patch and confirm that there is no typo!
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records.
Those imported items are stored as XML to be staged.
The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist.
You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a
This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml
Test plan :
- On a UNIMARC database
- Define a maching rule on title 200$a
- Select a record with items
- Export it using : Save as > MARC (Unicode/UTF-8)
- Delete all items
- Go to Tools > Stage MARC records for import
- Upload exported file
- Select title matching rule
- Select "Ingore incoming record" in "Action if matching record found :"
- Select Yes and "Always add items" in "Check for embedded item record data?"
- Click Stage for import
=> Without patch you get the error
=> With patch the import is staged
- Import into the catalog and check item is well recreated
Signed-off-by: Amandine Zocca <azocca@ville-montauban.fr>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
Try to search, preview and import authority from Z39.50, everything
should work as expected
Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
C4::Items::GetItemnumberFromBarcode calls can be replaced with
Koha::Items->find({ barcode => $barcode });
We should make sure the barcode existed in DB and so that ->find
returns an object. Note that most of the time we just wanted to know if
the barcode existed.
The changes are very simple, the only one that need attention is
the one in batchMod.pl. It is basically reusing what we did on
bug 21141.
Test plan:
Use the batch item modification/deletion tools to modify/delete items
from their barcode (using the textarea or a file)
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Fix for:
Field 'marcxml_old' doesn't have a default value
t/db_dependent/ImportBatch.t
Fixme - need to update existing DB?
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
TEST PLAN
---------
1) Run the following on a kohadevbox:
git checkout -b bug_19937 origin/master
sudo koha-shell -c bash kohadev
prove t/db_dependent/www/batch.t
cat /var/log/koha/kohadev/plack-error.log
The following errors are triggered at the end of the log file:
Use of uninitialized value in array element at
/home/vagrant/kohaclone/tools/batch_records_ajax.pl line 50.
Use of uninitialized value $results_per_page in numeric eq (==) at
/home/vagrant/kohaclone/tools/batch_records_ajax.pl line 53.
Use of uninitialized value in uc at
/home/vagrant/kohaclone/C4/ImportBatch.pm line 1120.
2) Run the following on a kohadevbox:
exit
git bz apply 19937
restart_all
sudo koha-shell -c bash kohadev
prove t/db_dependent/www/batch.t
cat /var/log/koha/kohadev/plack-error.log
The log file will end with the restarting of plack, not the
errors.
3) run koha qa test tools
Signed-off-by: Lee Jamison <ldjamison@marywood.edu>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended handling of $results_per_page.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
As requested by QA:
[1] Mock_config enable_plugins in the test.
[2] Fallback to MARC when format is empty. Remove die statement.
Added:
[3] Remove $marc. This variable got obsolete during development.
[4] Add test on $input_file and $plugin_class. Test $text before calling
Handler or processing $text. No need to split undef if somehow Handler
returned undef, etc. If the routine returns an empty arrayref,
stage-marc-import will do fine.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Bug 12412 added the use of to_marc plugins allowing arbitrary file formats
in stage-marc-import (as long as the plugins can handle them). The feature
was not very visible in the code, and when bug 10407 added the marcxml
format, it made some changes that broke the use of to_marc.
This patch restores the functionality by:
[1] Adding a sub RecordsFromMarcPlugin to ImportBatch.pm, specifically
addressing the conversion from arbitrary formats to MARC::Record.
The original to_marc interface is used: pass it the file contents,
and it returns a string consisting of a number of MARC blobs separated
by \x1D.
Consequently, the call of to_marc is removed from routine
BatchStageMarcRecords where it did not belong. The to_marc_plugin
parameter is removed and two calls are adjusted accordingly.
[2] Instead of a separate combo with plugins, the format combo contains
MARC, MARCXML and optionally some plugin formats.
[3] The code in stage-marc-import.pl now clearly shows the three main
format types: MARC, MARCXML or plugin based.
Note: This patch restores more or less the situation after bug 12412, but
I would actually recommend to have the to_marc plugins return MARC::Record
objects instead of large text strings. In the second example I added a
to_marc plugin that actually converts MARC record objects to string format,
while RecordsFromMarcPlugin reconverts them to MARC::Records.
Test plan:
See second patch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
C4::Biblio::GetBiblio can be replaced with Koha Biblio->find
Test plan:
Import batch, view issue history, search for items, see the image of a
bibliographic record, modify and delete records in a batch
Followed test plan, works as expected.
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Make it more explicit by adding join statements.
Test plan:
See next patch for adding a unit test.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
To test:
1 - Import a batch with some items
2 - Delete one of the imported items
3 - Browse to Tools->Staged MARC record management
4 - Click (Create label batch) for the batch you imported
5 - Recieve an error
6 - Apply patch
7 - Click (Create label batch)
8 - Batch is created with remaning items from the import
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
We will need a few additional parameters for merge later on. This patch
puts the original parameters in a parameter hash.
For the same reason DelAuthority gets a parameter hash here.
Note: We remove the second parameter from the DelAuthority call in
authorities/authorities-home.pl here. It was not used and could have
presented problems in the future.
Test plan:
[1] Run t/db_dependent/AuthoritiesMarc.t.
[2] Run t/db_dependent/Authorities/Merge.t.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Script tools/batch_delete_records.pl only checks the return value of
DelAuthority. The return value depends on DBI rows, which is not always
reliable. It may return -1 when it 'does not know'.
Testing $@ in tools/batch_delete_records.pl should actually be enough.
The return value was discarded in C4/ImportBatch.pm. Removing an unused
variable.
Test plan:
[1] Pick an authority record with a few linked biblios.
Delete this authority record via tools/batch_delete_records.pl.
Check if the linked biblio records are cleaned up.
[2] Bonus: Make a typo in the SQL statement of DelAuthority. Check
if batch_delete_records shows you the error message.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
If we import items that have a wrong branch code, the items will not
be imported but manage-marc-import reports them as imported. (A wrong
branch code probably occurs most, but other causes are possible too.)
The underlying cause is that AddItem does not look at the error
returned from _koha_new_item in Items.pm.
This patch deals with that omission in the most economical way. It adjusts
AddItem so that it returns undef if no item was added.
In ImportBatch.pm I check if an item was added and adjust the totals
accordingly instead of just always counting them.
Note: Several scripts like additem.pl use AddItemFromMarc to call
AddItem. They do not report an error, but fail silently. With this patch,
these scripts get undef and will still fail silently. (No change.)
Adding error checks around calls of AddItemFromMarc is outside the scope of
this report. Here we are taking care of correct imported item numbers.
Test plan:
[1] Verify that additem.pl still works by adding a new item.
[2] Run t/db_dependent/Items.t
[3] Add a new branchcode, say XXX.
[4] Pick a biblio record with a few items (n) and set one item to branch XXX.
[5] Export this biblio with items to a MARC file.
[6] Change the XXX item to the original branch and remove branch XXX.
[7] Import the MARC file. Verify that one item was not imported and that
the number of imported items reflects that (equals n-1).
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch makes the following changes:
[1] Based on the groundwork of the former patch, add call to
RecordsFromMARCXMLFile in stage-marc-import. Use format param.
[2] Add format to the template. Use file extension to determine.
If you use .xml or .marcxml as extension, MARCXML is selected.
[3] In stage-marc-import.tt mark UTF-8 encoding as UTF-8 not as utf8.
[4] BatchStageMarcRecords: do not call plugin if you have no records.
[5] RecordsFromISO2709File: also return errors in an array.
[6] In misc/stage_file.pl also use UTF-8. Handling of errors from [5].
Test plan:
[1] Import an empty file as MARC or MARCXML (with Tools/Stage..import).
[2] Import an non-empty file with invalid contents as MARC or MARCXML.
[3] Export a few records with Tools/Export as MARC and MARCXML.
[4] Import these two files. Check selected format versus file extension.
[5] Import a MARCXML file with misc/stage_file.pl.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Patch from Olli, manual rebase by Marcel (July 7, 2016).
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Needs follow-up. Test plan in the third patch.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Remove $dbh as argument to C4::Items::DelItemCheck
and C4::Items::ItemSafeToDelete, also change all
calls to these functions throughout the codebase.
Also remove remaining reference to 'do_not_commit' in
t/db_dependent/Items_DelItemCheck.t
Fixed doubled "$$" in C4/ImportBatch.pm
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
To test:
1) Go to Tools -> Staged MARC Management and clean a file. If you have no files to clean, go to 'Stage MARC for import' and upload one to clean following the necessary steps.
2) Confirm that once the file has been cleaned, the Action column now shows a Delete button. Confirm this button only shows for cleaned files.
3) Click the Delete button.
4) Confirm that clicking Cancel exits the pop-up message and does not delete the file.
5) Confirm that clicking OK refreshes the list of staged records and the one you just deleted is no longer on it (has been deleted). You can confirm this by checking for the file in mysql (SELECT * FROM import_batches WHERE import_batch_id = X;)
6) Run prove -v t/db_dependent/ImportBatch.t (have written unit tests for CleanBatch and DeleteBatch)
Sponsored-by: Catalyst IT
Signed-off-by: Liz Rea <liz@catalyst.net.nz>
Catalyst sign off, so needs another one but YAY this is great.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
subroutines should not take $dbh in parameter.
C4::Biblio::TransformMarcToKoha has it and does not use it.
Test plan:
Look at the patch and confirm that all occurrences of
TransformMarcToKoha have been modified.
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
perl -p -i -e 's/^.*set the version for version checking.*\n//' **/*.pm
+ manual adjustements
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Mainly a
perl -p -i -e 's/^.*3.07.00.049.*\n//' **/*.pm
Then some adjustements
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
perl -p -i -e 's/^(use vars .*)\$VERSION\s?(.*)/$1$2/' **/*.pm
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
