Koha-community/Koha - Koha - Koha: The world's first free and open source library system

Commit Graph

Author	SHA1	Message	Date
Jonathan Druart	7d8b96803f	Bug 24545: Fix license statements Bug 9978 should have fixed them all, but some were missing. We want all the license statements part of Koha to be identical, and using the GPLv3 statement. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>	4 years ago
Martin Renvoize	d2e189ca1c	Bug 22600: Set 'commandline' interface appropriately This patch change Koha::Cron to be a more generic Koha::Script class and update all commanline driven scripts to use it. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>	5 years ago
Martin Renvoize	9ce71f34ee	Bug 19164: (QA follow-up) Minor whitespace cleanup Signed-off-by: Nick Clemens <nick@bywatersolutions.com>	5 years ago
Jon Knight	1513c97b59	Bug 19164: Add MARC modification template support to misc/stage-file.pl. https://bugs.koha-community.org/show_bug.cgi?id=19164 Bug 19164 - Allow MARC modification templates to be used in staged MARC imports Patch checks for duplicate MARC modificiation templates and if only one is found that matches the name given on the command line, submits the batch staging of the input files using that template. Test plan: 1) In the staff web client got to "Home › Tools › MARC modification templates" and create a MARC Modification Template called "Test Bug 19164". Set the action to be "Copy field 999$l to 998$l using RegEx s/NASH/FLASH/ if 999$l matches NASH" 2) Download the test MARC record from this bug into a file called marc1.mrc into your koha dev user's home directory. 3) Run misc/stage_file.pl with the command line: misc/stage_file.pl --file ~/marc1.mrc --add-items --marcmodtemplate "Test Bug 19164" --item-action replace --comment "testing MARC mod templates on batch staging" 4) In the staff client, go to Home › Tools › Manage staged MARC records and check that the file is visible and in status "Staged" 5) Click on the filename to bring up details of the staged batch, and then click on "Basic Christianity. Stott, John R. W. (0802811892)". Note that there is a 998$l field with the value "FLASH" that has been created using the MARC Modification Template. Signed-off-by: Mackey Johnstone <mackeyfj@gmail.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>	5 years ago
Marcel de Rooy	d24568b672	Bug 19049: Fix regression on stage-marc-import with to_marc plugin Bug 12412 added the use of to_marc plugins allowing arbitrary file formats in stage-marc-import (as long as the plugins can handle them). The feature was not very visible in the code, and when bug 10407 added the marcxml format, it made some changes that broke the use of to_marc. This patch restores the functionality by: [1] Adding a sub RecordsFromMarcPlugin to ImportBatch.pm, specifically addressing the conversion from arbitrary formats to MARC::Record. The original to_marc interface is used: pass it the file contents, and it returns a string consisting of a number of MARC blobs separated by \x1D. Consequently, the call of to_marc is removed from routine BatchStageMarcRecords where it did not belong. The to_marc_plugin parameter is removed and two calls are adjusted accordingly. [2] Instead of a separate combo with plugins, the format combo contains MARC, MARCXML and optionally some plugin formats. [3] The code in stage-marc-import.pl now clearly shows the three main format types: MARC, MARCXML or plugin based. Note: This patch restores more or less the situation after bug 12412, but I would actually recommend to have the to_marc plugins return MARC::Record objects instead of large text strings. In the second example I added a to_marc plugin that actually converts MARC record objects to string format, while RecordsFromMarcPlugin reconverts them to MARC::Records. Test plan: See second patch. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>	7 years ago
Marcel de Rooy	d0f850b51a	Bug 10407: [QA Follow-up] Hashref process_batch params As requested by Mark Tompsett. Hope this guarantees a signoff now.. Note: For consistency four additional parameters were needed to no longer use file level vars in this subroutine. Test plan: Import a file with stage_file.pl. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Imported a marc file and a marcxml file with stage_file.pl. Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>	8 years ago
Marcel de Rooy	3badc57c93	Bug 10407: Add marcxml import (follow-up) This patch makes the following changes: [1] Based on the groundwork of the former patch, add call to RecordsFromMARCXMLFile in stage-marc-import. Use format param. [2] Add format to the template. Use file extension to determine. If you use .xml or .marcxml as extension, MARCXML is selected. [3] In stage-marc-import.tt mark UTF-8 encoding as UTF-8 not as utf8. [4] BatchStageMarcRecords: do not call plugin if you have no records. [5] RecordsFromISO2709File: also return errors in an array. [6] In misc/stage_file.pl also use UTF-8. Handling of errors from [5]. Test plan: [1] Import an empty file as MARC or MARCXML (with Tools/Stage..import). [2] Import an non-empty file with invalid contents as MARC or MARCXML. [3] Export a few records with Tools/Export as MARC and MARCXML. [4] Import these two files. Check selected format versus file extension. [5] Import a MARCXML file with misc/stage_file.pl. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>	8 years ago
Olli-Antti Kivilahti	aa5bf4d3cc	Bug 10407: Allow MARCXML records to be imported via GUI (groundwork) Patch from Olli, manual rebase by Marcel (July 7, 2016). Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Needs follow-up. Test plan in the third patch. Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>	9 years ago
Joonas Kylmälä	b6880abc50	Bug 12289: (follow-up) fix the compilation error Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>	9 years ago
Martin Renvoize	b31eca9787	Bug 12289: Add --no-create options to stage_file.pl This patch adds the no-create options to the command line interface for bulk loading of records. To test 1/ Apply both patches 2/ Call the script with --no-create and your will find that new records are not created when no match is found. The default, of create_new remains the same as before. Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>	10 years ago
Kyle Hall	ca167b32b4	Bug 12412: Add ability for plugins to convert arbitrary files to MARC from record staging tool Many libraries would like to be able to import various types of files as MARC records ( citations, csv files, etc ). We can add a new function to the plugins system to allow that kind of behavior at a very custom level. Test Plan: 1) Ensure you have plugins enabled and configured correctly 2) Installed the attached version 2.00 of the Kitchen Sink plugin 3) Download the attached text file 4) Browse to "Stage MARC records for import" 5) Select the downloaded text file for staging 6) After uploading, you should see a new area "Transform file to MARC:", select "Example Kitchen-Sink Plugin" from the pulldown menu 7) Click 'Stage for import" 8) Click 'Manage staged records" 9) You should now see two new MARC records! Signed-off-by: Aleisha <aleishaamohia@hotmail.com> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Works as described - interesting new feature. Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>	10 years ago
Galen Charlton	7a1c343c91	Bug 8015: (follow-up) update stage_file.pl This patch fixes the call to BatchStageMarcRecords() made by the comand-line record importer. At some point in the future we could go further and teach that tool how to apply modification templates. Signed-off-by: Galen Charlton <gmc@esilibrary.com>	11 years ago
Galen Charlton	997a502c22	Bug 7131: (follow-up) update the command-line import tools misc/commit_file.pl now properly displays the number of replaced and ignored items. misc/stage_file.pl now has a new command-line switch --item-action: --item-action action to take if --add-items is specifed; choices are 'always_add', 'add_only_for_matches', 'add_only_for_new', 'ignore', or 'replace' Signed-off-by: Galen Charlton <gmc@esilibrary.com>	11 years ago
Jared Camins-Esakov	7ad5e203da	Bug 2060: Update command line MARC import scripts Expose authority import functionality to the command line import scripts, and rename them from commit_biblios_file.pl and stage_biblios_file.pl to commit_file.pl and stage_file.pl. To test (note that these instructions assume you have a MARC21 installation and are using the provided sample file): 1. Find a file of authorities (a sample file with MARC21 authorities is attached to bug 7475) and download it to your server 2. Stage the file using the following command (replace <filename> with the name of the file you saved in step 1): > misc/stage_file.pl --file <filename> --authorities 3. Note the batch number the script assigns to your batch 4. Commit the records using the following command (replace <batchnumber> with the batch number you made note of in step 3): > misc/commit_file.pl --batch-number <batchnumber> 5. Index the authorities Zebraqueue (or wait) 6. Confirm that the new authorities appear. 7. Create a matching rule with the following settings: Code: AUTHTEST Description: Personal name main entry Match threshold: 999 Record type: Authority record Search index: Heading-main Score: 1000 Tag: 100 Subfields: a Offset: 0 Length: 0 (note the ID of this matching rule) 8. Stage the authority file again, this time using the following command: > misc/stage_file.pl --file <filename> --authorities \ --match <matchingrule> 7. Revert the import with the following command: > misc/commit_file.pl --batch-number <batchnumber> --revert 8. Index the authorities Zebraqueue (or wait) 9. Confirm that the records have been removed 10. Import an authority record with the Stage MARC/Manage staged MARC tools in exactly the way you would for a bibliographic record, but choose "Authority" instead of "Bibliographic" for the record type. Signed-off-by: Elliott Davis <elliott@bywatersolutions.com> Testing plan delivers as it should. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> Rebased on latest master 11 September 2012	12 years ago
Katrin Fischer	1b49f54b88	Bug 7250: stage_biblios_file.pl is missing options for encoding - adds command line options for encoding, defaulting to utf8 - options match options availablen in the stage marc records form of the staff interface - activates warnings - adds copyright statement To test: Import records with diacritics using the stage_biblios_file.pl Records can be imported into the catalog using the staff interface or the commit_biblios_file.pl script. Signed-off-by: Ulrich Kleiber <ulrich.kleiber@bsz-bw.de> Successfully tested with default encoding utf8. Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>	13 years ago
Donovan Jones	5e0b850d49	Bug 2505 - Add commented use warnings where missing in the misc/ directory	14 years ago
Ryan Higgins	3c304cdb3a	Add no-replace option to stage_biblios_file.pl, allowing incoming bibs to not overwrite existing bibs. Signed-off-by: Joshua Ferraro <jmf@liblime.com>	16 years ago
Galen Charlton	438ed23337	staging import - enhance record overlay behavior Enhanced the ability of catalogers to specify how bib and item records should be added, replaced, or ignored during a staging import. When an import batch of bib records is staged and commit, the user can now explicitly specify what should occur when an incoming bib record has, or does not have, a match with a record already in the database. The options are: if match found (overlay_action): create_new (just add the incoming record) replace (replace the matched record with the incoming one) use_template (option not implemented) ignore (do nothing with the incoming bib; however, the items attached to it may still be processed based on the item action) if no match is found (nomatch_action): create_new (just add the incoming record) ignore (do nothing with the incoming bib; in this case, any items attached to it will be ignored since there will be nothing to attach them to) The following options for handling items embedded in the bib record are now available: always_add (add the items to the new or replaced bib) add_only_if_match (add the items only if the incoming bib matches an existing bib) add_only_if_add (add the items only if the incoming bib does not match an existing bib) ignore (ignore the items entirely) With these changes, it is now possible to support the following use cases: [1] A library joining an existing Koha database wishes to add their items to existing bib records if they match, but does not want to overlay the bib records themselves. [2] A library wants to load a file of records, but only handle the new ones, not ones that are already in the database. [3] A library wants to load a file of records, but only handle the ones that match existing records (e.g., if the records are coming back from an authority control vendor). Documentation changes: * See description above; also, screenshots of the 'stage MARC records for import' and 'manage staged MARC records' should be updated. Test cases: * Added test cases to exercise staging and committing import batches. UI changes: * The pages for staging and managing import batches now have controls for setting the overlay action, action if no match, and item action separately. * in the manage import batch tool, user is notified when they change overlay action, no-match action, and item action * HTML for manage import batch tool now uses fieldsets Database changes (DB rev 076): * added import_batches.item_action * added import_batches.nomatch_action * added 'ignore' as a valid value for import_batches.overlay_action * added 'ignored' as a valid value for import_records.status * added 'status' as a valid value for import_items.status API changes: * new accessor routines for C4::ImportBatch GetImportBatchNoMatchAction SetImportBatchNoMatchAction GetImportBatchItemAction SetImportBatchItemAction * new internal functions for C4::ImportBatch to determine how a given bib and item are to be processed, based on overlay_action, nomatch_action, and item_action: _get_commit_action _get_revert_action Signed-off-by: Joshua Ferraro <jmf@liblime.com>	16 years ago
Galen Charlton	b88682f1bb	bug 1372: count MARC records correctly Signed-off-by: Joshua Ferraro <jmf@liblime.com>	17 years ago
Ryan Higgins	58efea0215	Allow stage_biblios_file to take matcher id as parameter. Signed-off-by: Chris Cormack <crc@liblime.com> Signed-off-by: Joshua Ferraro <jmf@liblime.com>	17 years ago
Galen Charlton	991cdc31bc	speed boost: command-line stage and commit biblios Turned off autocommit and commit every 100 records. Signed-off-by: Chris Cormack <crc@liblime.com> Signed-off-by: Joshua Ferraro <jmf@liblime.com>	17 years ago
Galen Charlton	b8a58c4934	installer: command-line scripts improve finding C4 modules Command-line scripts now use a new SCRIPT_DIR/kohalib.pl to put installed location of Koha's Perl modules into @INC.	17 years ago
Galen Charlton	75bd4b2dac	bibliographic matching enhancements * serialization of matching rules to database * added two default matching rules for MARC21 * update import staging CGI to choose matching rule Signed-off-by: Chris Cormack <crc@liblime.com> Signed-off-by: Joshua Ferraro <jmf@liblime.com>	17 years ago
Galen Charlton	3fbd25602b	new batch job to stage a file of MARC biblios for import. As part of this, modified two routines in C4::ImportBatch to support a callback for monitor progress of import processing. Signed-off-by: Chris Cormack <crc@liblime.com> Signed-off-by: Joshua Ferraro <jmf@liblime.com>	17 years ago

14 Commits (c3f1ebec619977d60387ab6b182c2919f62ce689)