This change was done in a transaction - it would either be set as imported
on success, or rolled back to staged on failure
There is no need for the intermediate status which is never committed
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
See comment1. Although we now fix the error on publishercode, it
is good to verify the result before pushing.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
To test:
1) Apply the patch
2) Visit C4::ImportBatch::RecordsFromMARCXMLFile
3) See that in the POD (mine was somewhere around line 1592) the line starting with '@PARAM1' now says '@PARAM1, String, absolute path to the MARCXML file.'
4) Sign off :)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When replacing existing records BatchCommitRecords will the table import_records will be updated three times for three different fields by three different queries. Not only is this inefficient ( especially for large batches ), it seems that this is causing the dreaded "Lock wait timeout exceeded; try restarting transaction" error on some mysql/mariadb configurations.
1) Test plan
2) Download a marc record from Koha
3) Modify the title of that same bib in Koha
4) Stage the downloaded record and overlay the existing record
5) Verify the title has reverted to the original title from the
downloaded record!
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch simply moves our indexing call after the transaction is committed so
that the job will exist in the DB when called.
To test:
1 - Have Koha using Elasticsearch
2 - Stage and import a file of records
3 - View the job in Admin->Manage jobs
4 - Note it is not finished
5 - Check log: /var/log/koha/kohadev/es-indexer-output.log
6 - Note: [WARN] No job found for id=###
7 - Apply patch
8 - Stage and import
9 - Note no error in log
10 - Note successful completion of indexing job
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Resolves:
C4::ImportBatch::_update_batch_record_counts(): DBI Exception: DBD::mysql::st execute failed: Deadlock found when trying to get lock; try restarting transaction at /usr/share/koha/C4/ImportBatch.pm line 392
See also bug 32558.
Test plan:
If you apply 32558 first, run multiple processes that stage a marc import.
Without this patch, you can trigger the deadlock.
With this patch, it works.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
C4::AuthoritiesMarc method GetAuthorizedHeading is not exported thus it is called in other modules :
> git grep GetAuthorizedHeading
C4/AuthoritiesMarc.pm:=head2 GetAuthorizedHeading
C4/AuthoritiesMarc.pm: $heading = &GetAuthorizedHeading({ record => $record, authid => $authid })
C4/AuthoritiesMarc.pm:sub GetAuthorizedHeading {
C4/Breeding.pm: $heading = C4::AuthoritiesMarc::GetAuthorizedHeading({ record => $marcrecord });
C4/ImportBatch.pm: $row->{'authorized_heading'} = C4::AuthoritiesMarc::GetAuthorizedHeading( { authid => $row->{'candidate_match_id'} } );
C4/ImportBatch.pm: my $authorized_heading = C4::AuthoritiesMarc::GetAuthorizedHeading({ record => $marc_record });
This patch adds it to be exported.
For example for use in Koha plugins.
Test plan :
Check import of authorities from a file is OK
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Resolve:
# Failed test 'Item's biblioitemnumber has not changed'
# at t/db_dependent/ImportBatch.t line 407.
#
# got: '4261'
# expected: '2371'
Do not compare $item1->biblionumber with $original_biblioitemnumber :)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.
When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files
To test:
1 - Be using the sample data in koha testing docker
2 - Stage the sample file on this report
3 - Match on KohaBiblio(999$c) / Item processing: replace
4 - Note the biblio match is "The complete novels / Austen, Jane"
5 - View the staged marc, note the barcode 39999000004090 in an item
6 - Search for that barcode
7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
8 - Import the file
9 - Check the db:
SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode
JD amended patch
- # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to
+ # We assume that when replacing items we do not want to move them - the onus is on the importer to
Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch allows for selection of framework to use when overlaying
records - by default it is set to keep the initial framework
To test:
1 - Create some records using one framework
2 - Export the records
3 - Edit the records to add fields not in original framework
4 - Stage records using a rule that will find matches
5 - Import
6 - Note records contain new fields on display, but they are lost on edit
7 - Apply patch
8 - Stage records again
9 - Select a framework that contains the new fields on import
10 - Import records
11 - Note records now use selected framework and are displayed/edited
correctly
Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When you submit a background jobs, and it fails, you do not expect
partial results in the database.
Note that when the Background feature would support a partially
completed status, things might change again.
Note that the >0 test was superfluous if you check for ^\d+$.
Test plan:
Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t
Note: This serves to verify that it still runs as expected.
The test plan of the following patch covers the new param.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch removes the now unused: _update_import_record_marc
Additionally, as items are already present in import biblio we no longer need to embed
them, so that parameter is removed and the option removed from the sub and pod and everywhere
it was used
In all cases, we were embedding, so we don't need a way to get without
Tests updated
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We are stripping the marc item tags at import - we save them when not importing items, but
strip when importing items
I think we can save on writes by leaving them in the record. This also allows comparison to what was staged
versus items created
To test:
1 - Stage a marc record with items, but do not look for items
2 - Confirm the item tags remain in staged marc
3 - Import the record
4 - Confirm items are nto in imported marc record
5 - Stage the record again, but look for items
6 - Confirm the item tags are stipped from imported record
7 - Import and confirm imported record has no item tags
8 - Apply patch and repeat 1-5
9 - Confirm item tags remain in record
10 - Import and confirm item tags not in imported marc
Signed-off-by: Andrew <andrewfh@dubcolib.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
AddBiblioToBatch and AddAuthToBatch were btoh passing a random number for the
parameter $update_counts
This simply removes that value as should have been done on 22532
To test:
1 - Stage a marc file
2 - Confirm it works befor and after patch
Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
MARC::Record and MARC::File::* modules sometimes use the position 09 of
the leader to detect encoding. A blank character means 'MARC-8' while an
'a' means 'UTF-8'.
In a UNIMARC authority this position is used to store the authority type
(see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]).
In this case, 'a' means 'Personal Name'.
The result is that the import will succeed for a Personal Name
authority, but it will fail for all other authority types.
Steps to reproduce:
0. Be sure to have a Koha UNIMARC instance.
1. Download the MARCXML for "Honoré de Balzac"
curl -o balzac.marcxml https://www.idref.fr/02670305X.xml
2. Verify that it's encoded in UTF-8
file balzac.marcxml
(should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode
text")
3. Go to Tools » Stage MARC for import and import balzac.marcxml with
the following settings:
Record type: Authority
Character encoding: UTF-8
Format: MARCXML
Do not touch the other settings
4. Once imported, go to the staged MARC management tool and find your
batch. Click on the authority title "Balzac Honoré de 1799-1850" to
show the MARC inside a modal window. There should be no encoding
issue.
5. Write down the imported record id (the number in column '#') and go
to the MARC authority editor. Replace all URL parameters by
'breedingid=THE_ID_YOU_WROTE_DOWN'
The URL should look like this:
/cgi-bin/koha/authorities/authorities.pl?breedingid=198
You should see no encoding issues. Do not save the record.
6. Import the batch into the catalog. Verify that the authority record
has no encoding issue.
7. Now download the MARCXML for "Athènes (Grèce)"
curl -o athènes.marcxml https://www.idref.fr/027290530.xml
8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you
should see encoding issues and that the position 9 of the leader was
rewritten from 'c' to 'a'. Strangely, importing this batch fix the
encoding issue, but we still lose the information in position 09 of
the leader
This patch makes use of the MARCXML representation of the record instead
of the ISO2709 representation, because, unlike
MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to
pass directly the encoding and the format, which prevents data to be
double encoded when position 09 of the leader is different that 'a'
Test plan:
- Follow the "steps to reproduce" above and verify that you have no
encoding issues.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch fixes the broken commit_file.pl script.
It doesn't deal with commiting the import from the UI.
To test:
1. Pick a file for staging:
$ kshell
k$ misc/stage_file.pl --file TestDataImportKoha.mrc
=> SUCCESS: All good
2. Commit!
k$ misc/commit_file.pl --batch-number 1
=> FAIL: You see
DBIx::Class::Storage::DBI::_exec_txn_begin(): DBI Exception: DBD::mysql::db begin_work failed: Already in a transaction at /kohadevbox/koha/C4/Biblio.pm line 303
3. Apply this patch
4. Repeat 2
=> SUCCESS: Commit succeeds
5. Sign off :-D
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 29325: (QA follow-up) Remove unexisting parameters of BatchRevertRecords
There is no interval and callback as in BatchCommitRecords.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 29325: Call progress callback one last time to confirm comppletion
Previously after finishing the loop we were still in a transaction that never completed - we should report progress when done
one final time to commit the last records
To test:
1 - Stage a file with > 100 records
2 - Commit file
3 - Confirm batch is imported and no records left as staged
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 29325: Fix import from staff client
same test as before, but via the staff client
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 29325: Handle the transaction in BatchCommitRecords
Requiring the callback to commit was breaking reversion, and likely elsewhere
Let's simplify and say that the routine iteself will handle the txn and commit
TO test:
1 - Stage a file
2 - Import a file
3 - Revert a file
4 - Test staff client and command line
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
BatchCommitItems is only being used within this module and isn't
mentioned in EXPORT_OK. This patch simply renames it to
_batchCommitItems to take the _ standard for private functions and also
adds a little hint to the POD of the function to clarify that the caller
must trigger a re-index.
JK: Amended patch to rename also the function in t/db_dependent/ImportBatch.t
and fix typo "commiting" => "commiting"
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When committing staged marc imports to the catalogue we will often be
importing a batch of records. We don't want to send one index request
per biblio affected, we want to index them all after the records have
been modified otherwise we will end up with multiple tasks per record
(when items are also affected).
Test plan:
1) Use the stage marc record tool to stage and commit a set of records and
confirm the behaviour remains correct.
2) If using Elastic, check that only one indexing job is queued to take
place resulting from the committed import.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This makes two simple changes:
- Limit TransformMarcToKoha to the fields we need
- Pass forward the biblioitemnumber when adding items to a new biblionumber
Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items
Reducing calls in item store to use a passed biblionumber was the largest gain.
To test:
1 - Import some records and items
2 - Verify values etc., revert
3 - Apply patch
4 - Import again
5 - Verify values etc. same as before
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch changes the call to use the fully qualified name and
adds import to C4::Breeding, Koha::MetaSearcher, and removes import
from Koha::MetadataRecord
Additionally it replaces missing fields from the update to using TransformMarcToKoha
Lastly, it reduces the fields used when saving the bredding record to the reservoir
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Missed these in the original commits
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
1 - git grep ModAuthInBatch
2 - Confirm none of these occurences are calls
3 - Apply patch
4 - git grep ModAuthInBatch
5 - No occurences
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
To test:
1 - git grep ModBiblioInBatch
2 - Confirm these occurences are not calls
3 - Apply patch
4 - git grep ModBiblioInBatch
5 - No occurences
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Looking at the code this 'import_error' column is empty for the last 9 years. Not sure it makes much sense to have this single error now.
commit 1dba9c6409
Date: Wed Oct 10 14:21:22 2012 -0500
Bug 7131: teach MARC import how to overlay items
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We are trying to insert "duplicate item barcode" into import_items.itemnumber (integer),
it fails with "Incorrect integer value: 'duplicate item barcode' for column 'itemnumber' at row 1"
To reproduce:
Export a biblio with an item
Import it
=> The item is not added, and there is no new row in import_items.
The error only appears in the log if you comment the close STDERR and
close STDOUT lines
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Previously the sorting took 'chosen' into account and would move a selected match to the
top on next load - it is better to preserve the same sorting every time
When loading matches the 'cehcked' variable was not being cleared, so multiple matches were
being marked 'checked="checked"'. Fixing this ensures the correct record displays as chosen
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Ben Daeuber <bdaeuber@cityoffargo.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patchset adds the display of all matches found during import to the import management screen
A staff member with the permission to manage batches will be able to select for any individual record which match, or none, should be used during import
To test:
1 - Import a batch of records or export existing records from your catalog
2 - Import the file (again) and select a matching rule that will find matches
3 - Note that you now have radio buttons allowing you to select a record, or none
4 - Test scenarios:
I - When 'Action if matching record found' is 'Ignore'
a - Imported record ignored if match is selected
b - 'Action if no match found' followed if no match is selected (Ignore matches)
II - When 'Action if matching record found' is 'Replace'
a - The chosen record is the one overlayed (you can edit the chosen record before importing to confirm)
b - 'Action if no match found' followed if no match is selected (Ignore matches)
III - When 'Action if matching record found' is 'Add incoming record'
a - Record is added regardless of matches
5 - Confirm 'Diff' 'View' links work as expected
6 - Confirm that after records are imported the radio buttons to choose are disabled
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Bug 22785: API files
Signed-off-by: Ben Daeuber <bdaeuber@cityoffargo.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
The following authorities subroutines are used by the ImportBatch
script but are not accessible, because they aren't imported. This
caused MARC importing errors.
- GetAuthorityXML
- ModAuthority
- DelAuthority
These affected the BatchCommitRecords and BatchRevertRecords routines,
and it wasn't caught because there are no tests written for them.
To test:
1. Find an existing authority record, or import one to use.
2. Save this authority record (export/download).
3. Go to Admin -> Record matching rules. Create a new record matching
rule for authorities that matches on 001, Local-Number index.
4. Go to Tools -> Stage MARC for import. Upload the authority file you
just downloaded.
5. Change record type to authority.
6. Under 'Look for existing items in catalog?', set the record matching
rule to the rule you just made which matches on 001. If matching
record found, replace the existing one. If no match is found, ignore.
7. Stage the record. Once complete, a match should've been found.
8. Go to Staged MARC management.
9. Import the batch into the catalog. Notice it hangs and never
completes.
10. Apply the patch and restart services.
11. Repeat steps 4-9. This time importing should be successful.
Sponsored-by: Educational Services Australia SCIS
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch reuse the (awesome) Koha::Result::Boolean module to retrieve
the return of Koha::Item->safe_to_delete.
Test plan:
Try to delete an item that has previously been checked out and confirm
that you are still blocked.
Try using the cronjobs, the item and biblio detail pages, as well as the
batch delete item tool.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch renames the (passed through) 'context' param for
'overlay_context'. I propose doing so, because in Koha-land 'context'
has a special meaning, related to C4::Context and it reads ambigous.
The patch itself is pretty trivial.
Tests should pass:
1. Run:
$ kshell
k$ prove t/db_dependent/Biblio/MarcOverlayRules.t
=> SUCCESS: Tests pass
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests still pass!
4. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 14957: (follow-up) Clarify 'context' param
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add a rule based system for merging MARC records to for example
prevent field data from being overwritten.
To test:
1. Apply this patch.
2. Log in to staff client.
3. Enable new syspref MARCMergeRules.
4. Click the new link "MARC merge rules" in the "Catalog"
section of the Koha administration page.
5. Create a new rule:
Module: source, Filter: *, Tag: 245, Preset: Protect.
6. Clicking "Edit" should allow you to edit corresponding rule.
7. Clicking "Delete" should remove corresponding rule after confirmation.
8. Selecting one or more rules followed by clicking "Delete
selected" should remove all selected rules after confirmation.
9. Try creating a rule with tag set to "**", the other options does
not matter. Verify that saving this rule produces an error
message complaining about invalid tag regular expression.
10. Try creating a rule with tag set to "008" (or other control
field) and set Appended: Append and Removed: Skip, the other
options does not matter. Verify that saving this rule produces
an error message complaining about invalid combination of actions
for control field.
11. With the 245 rule in step 5 in place, edit a bibliographic record,
change 245a for example (which should be Title for MARC21) and save.
12. Verify that the changes has not been saved.
13. Create a new rule:
Module: source, Filter: intranet, Tag: 245, Preset: Overwrite.
14. Repeat step 12, and verify that the changes has now been saved.
15. Run tests in t/db_dependent/Biblio/MarcMergeRules.t and very
that all tests pass.
Sponsored-by: Halland County Library
Sponsored-by: Catalyst IT
Sponsored-by: Gothenburg University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Christian Stelzenmüller <christian.stelzenmueller@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended: POD line for $import_record_id.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
SetImportBatchStatus is not called with $batch_id
It has been caught by bug 25026, and www/search_utf8.t fails with
t/db_dependent/www/search_utf8.t .. 14/87 Error POSTing http://koha:8081/cgi-bin/koha/tools/manage-marc-import.pl: Internal Server Error at t/db_dependent/www/search_utf8.t line 240.
And, from logs:
manage-marc-import.pl: C4::ImportBatch::SetImportBatchStatus(): DBI Exception: DBD::mysql::st execute failed: Truncated incorrect DOUBLE value: 'importing' [for Statement "UPDATE import_batches SET import_status = ? WHERE import_batch_id = ?" with ParamValues: 0=undef, 1='importing'] at /kohadevbox/koh
a/C4/ImportBatch.pm line 579: /kohadevbox/koha/tools/manage-marc-import.pl, referer: http://koha:8081/cgi-bin/koha/tools/manage-marc-import.pl?import_batch_id=2
Test plan:
Read the changes and confirm it does make sense.
Import and revert a batch
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Prior to ug 23463 AddItemFromMarc where calling AddItem, that did not
take into account the itemnumber field.
Now that we are using Koha::Item, we need to remove the items.itemnumber
field from the MARC record
Test plan:
1 - find an existing bib in your system with just one item
2 - export that bib with the item attached
3 - delete the barcode from your item in Koha
4 - stage your exported marc file for reimport, match on biblionumber, set it to Always Add Items
5 - confirm that the bib matches and the incoming 952 is parsed
6 - click "Import this batch into the catalog"
=> Without this patch you get (in the logs, or hidden)
manage-marc-import.pl: DBD::mysql::st execute failed: Duplicate entry '23' for key 'PRIMARY' [for Statement "INSERT INTO `items` ( `barcode`, `biblioitemnumber`, `biblionumber`, `ccode`, `cn_sort`, `cn_source`, `damaged_on`, `dateaccessioned`, `datelastborrowed`, `datelastseen`, `holdingbranch`, `homebranch`, `itemcallnumber`, `itemlost_on`, `itemnumber`, `itype`, `location`, `more_subfields_xml`, `onloan`, `permanent_location`, `replacementpricedate`, `timestamp`, `withdrawn_on`) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, current_timestamp, ? )" with ParamValues: 0="BC_23", 1=8, 2=8, 3="REF", 4='CN__23', 5=undef, 6=undef, 7="2014-09-04", 8=undef, 9="2014-09-04", 10="FPL", 11="FPL", 12="CN_23", 13=undef, 14="23", 15="BK", 16="GEN", 17=undef, 18=undef, 19="GEN", 20="2014-09-04", 21=undef] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1836.
manage-marc-import.pl: DBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry '23' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 169
manage-marc-import.pl: {UNKNOWN}: Transaction aborted: Duplicate ID. Rollback failed: DBIx::Class::Storage::txn_rollback(): Refusing to roll back without a started transaction at /kohadevbox/koha/tools/manage-marc-import.pl line 253 at /kohadevbox/koha/tools/manage-marc-import.pl line 253
=> With this patch applied, the new item must be added to the existing bibliographic record
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the logic and the needed UI elements to be able to pre-load an import profile. It also displays which profile was used to stage an import in staged import manager.
To test:
1. Apply all patches
2. Updatedatabase
3. Go to Stage MARC records for Import tool in admin, and upload a file with MARC records.
CHECK => after uploading, there is a fieldset with the legend “Profile settings”
=> inside the fieldset there is a select labeled “Pre fill values with profile”. The only value it has is “Do not use profile”.
4. Change some settings, and set “profile 1” as profile name and click on “Add profile”
SUCCESS => The select now has the new profile selected
5. Change profile select to “Do not use profile”
SUCCESS => Default values are now displayed in the form
6. Reload the page and upload the file again
SUCCESS => the select still has the profile recently added
7. Select the profile, change some parameter in the form and set the profile name to “profile 2”, and click add profile
SUCCESS => there are two profiles now, and if you toggle between them, the parameter changes
8. Select profile 1, change one parameter and click on update profile
SUCCESS => if you toggle that profile with the other, the new parameter of the value is shown when you select profile 1
9. Select profile 2, change some parameter and click Add profile (leaving the name as profile 2)
SUCCESS => the page complains there is another profile with the same name, and asks if you want to replace it.
10. Click on accept
SUCCESS => profile 2 now has the new value in the parameter
11. Select profile 2 and change the name to profile 1
SUCCESS => the page complains there is another profile with that name, and asks if you want to replace it
12. Click on accept
SUCCESS => in profile select there is only one profile called profile 1 that has the values of profile 2
13. Select profile 1 and click remove profile
SUCCESS => there is no profile in profile select.
14. Create a profile and click on “Stage for import”
15. Go to Staged MARC management page
SUCCESS => Improt should have the name of the profile in profile column, and when you click on the file name, there should be the name of the profile in the details.
16. prove t/db_dependent/ImportBatch.t t/db_dependent/api/v1/import_batch_profiles.t
17. Sign off
Signed-off-by: Abbey Holt <aholt@dubuque.lib.ia.us>
Signed-off-by: Abbey Holt <aholt@dubuque.lib.ia.us>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds a .perlcriticrc (copied from qa-test-tools) and fixes
almost all perlcrictic violations according to this .perlcriticrc
The remaining violations are silenced out by appending a '## no critic'
to the offending lines. They can still be seen by using the --force
option of perlcritic
This patch also modify t/00-testcritic.t to check all Perl files using
the new .perlcriticrc.
I'm not sure if this test script is still useful as it is now equivalent
to `perlcritic --quiet .` and it looks like it is much slower
(approximatively 5 times slower on my machine)
Test plan:
1. Run `perlcritic --quiet .` from the root directory. It should output
nothing
2. Run `perlcritic --quiet --force .`. It should output 7 errors (6
StringyEval, 1 BarewordFileHandles)
3. Run `TEST_QA=1 prove t/00-testcritic.t`
4. Read the patch. Check that all changes make sense and do not
introduce undesired behaviour
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In an OO package, the logger initialization should happen in the
constructor. This is not an OO package and the initialization is
happening on loading it. This is a wrong behaviour and certainly breaks
in environments where initialization cannot happen (package building,
for example). There could be several options to solve this, as it is
used in a single sub on this package, I opted for initializing on that
sub.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
The MARC preview of staged records fails when a staged record contains
items with UTF-8 characters (see attached screenshots). That's because
in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml()
does not explicitly specify 'UTF-8' as the optional encoding to use.
This patch fixes that.
Test plan (in a MARC21 instance):
0) Stage for import the test record attached to this bug report (make
sure that 'Check for embedded item record data?' is set to 'Yes'),
then go to Home > Tools > Manage staged MARC records and click on
the filename you've just staged.
1) In the 'Citation' column, click on the hyperlink to launch the MARC
preview -- notice how it fails to produce any output apart from the
modal heading.
2) Apply the patch, and restart Plack if necessary.
3) Re-launch the MARC preview: this time the output should include the
whole record, including the embedded item data in field 952.
4) Run the unit tests to ensure nothing was broken inadvertently:
$ prove t
$ prove xt
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
We certainly faced 3 similar bugs due to this syntax: bug 23006, bug
22941 and bug 17526.
To prevent other issues related to this syntax this patch suggests to
replace them all in one go.
Test plan:
Confirm that the 2 syntaxes are similar
Eyeball the patch and confirm that there is no typo!
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records.
Those imported items are stored as XML to be staged.
The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist.
You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a
This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml
Test plan :
- On a UNIMARC database
- Define a maching rule on title 200$a
- Select a record with items
- Export it using : Save as > MARC (Unicode/UTF-8)
- Delete all items
- Go to Tools > Stage MARC records for import
- Upload exported file
- Select title matching rule
- Select "Ingore incoming record" in "Action if matching record found :"
- Select Yes and "Always add items" in "Check for embedded item record data?"
- Click Stage for import
=> Without patch you get the error
=> With patch the import is staged
- Import into the catalog and check item is well recreated
Signed-off-by: Amandine Zocca <azocca@ville-montauban.fr>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
