Commit graph

44 commits

Author SHA1 Message Date
Jenny Way
cad2db6016 Bug 20017: use Modern::Perl in Tools perl scripts
Test plan:
Check that the following files don't contain use strict; use warnings;
and have use Modern::Perl instead.
background-job-progress.pl
batchMod.pl
copy-holidays.pl
exceptionHolidays.pl
holidays.pl
import_borrowers.pl
koha-news.pl
letter.pl
manage-marc-import.pl
newHolidays.pl
overduerules.pl
quotes-upload.pl
quotes.pl
quotes/quotes-upload_ajax.pl
quotes/quotes_ajax.pl
scheduler.pl
stage-marc-import.pl
upload-cover-image.pl

Also the credits have been added to newHolidays.pl

Signed-off-by: Jon Knight <J.P.Knight@lboro.ac.uk>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-02-05 09:46:45 -03:00
af791d83e4 Bug 19163: Critical typo in stage-marc-import process
Template says ISO2709 but script wants MARC or MARCXML..
Slipped in with bug 19049.
Trivial but critical fix.

Test plan:
Import a MARC file.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I picked this fix because I think MARC is always ambiguous. ISO2709
clearly denotes the serialization format used.

Signed-off-by: Liz Rea <liz@catalyst.net.nz>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-08-23 10:22:32 -03:00
a423fcff42 Bug 19049: [QA Follow-up] Mock config, default format
As requested by QA:
[1] Mock_config enable_plugins in the test.
[2] Fallback to MARC when format is empty. Remove die statement.
Added:
[3] Remove $marc. This variable got obsolete during development.
[4] Add test on $input_file and $plugin_class. Test $text before calling
    Handler or processing $text. No need to split undef if somehow Handler
    returned undef, etc. If the routine returns an empty arrayref,
    stage-marc-import will do fine.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-08-15 12:17:42 -03:00
d24568b672 Bug 19049: Fix regression on stage-marc-import with to_marc plugin
Bug 12412 added the use of to_marc plugins allowing arbitrary file formats
in stage-marc-import (as long as the plugins can handle them). The feature
was not very visible in the code, and when bug 10407 added the marcxml
format, it made some changes that broke the use of to_marc.

This patch restores the functionality by:
[1] Adding a sub RecordsFromMarcPlugin to ImportBatch.pm, specifically
    addressing the conversion from arbitrary formats to MARC::Record.
    The original to_marc interface is used: pass it the file contents,
    and it returns a string consisting of a number of MARC blobs separated
    by \x1D.
    Consequently, the call of to_marc is removed from routine
    BatchStageMarcRecords where it did not belong. The to_marc_plugin
    parameter is removed and two calls are adjusted accordingly.
[2] Instead of a separate combo with plugins, the format combo contains
    MARC, MARCXML and optionally some plugin formats.
[3] The code in stage-marc-import.pl now clearly shows the three main
    format types: MARC, MARCXML or plugin based.

Note: This patch restores more or less the situation after bug 12412, but
I would actually recommend to have the to_marc plugins return MARC::Record
objects instead of large text strings. In the second example I added a
to_marc plugin that actually converts MARC record objects to string format,
while RecordsFromMarcPlugin reconverts them to MARC::Records.

Test plan:
See second patch.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-08-15 12:17:41 -03:00
158442eb9e Bug 17501: Remove Koha::Upload::get from Koha::Upload
The get routine actually returns records from uploaded_files. It should be
possible to replace its calls by direct calls of Koha::UploadedFiles.

This patch is the crux of this patch set. It deals with all scripts that
use Koha::Upload.

In the process we do:
[1] Add a file_handle method to Koha::UploadedFile. This was previously
    arranged via the fh parameter of get.
[2] Add a full_path method to UploadedFile. Previously returned in the
    path hash key of get. (Name is replaced by filename.)
[3] Add a search_term method too (implementing get({ term => .. }).
    This logic came from _lookup.
[4] Add a keep_file parameter to delete method. Only used in test now.

Test plan:
[1] Run t/db_dependent/Upload.t
[2] Go to Tools/Upload. Add an upload, download and delete.
[3] Add another public upload , search for it.
    Use the hashvalue to download via opac with URL:
        cgi-bin/koha/opac-retrieve-file.pl?id=[hashvalue]
[4] Go to Tools/Stage MARC for import. Import a marc file.
[5] Go to Tools/Upload local cover image. Import an image file.
    Enable OPACLocalCoverImages to see result.
[6] Test uploading a offline circulation file:
    Enable AllowOfflineCirculation, and create a koc file (plain text):
    Line1: Version=1.0\tA=1\tB=2
    Line2: 2016-11-23 16:00:00 345\treturn\t[barcode]
    Note: Replace tabs and barcode. The number of tabs is essential!
    Checkout the item with your barcode.
    Go to Circulation/Offline circulation file upload.
    Upload and click Apply directly.
    Checkout again. Repeat Offline circulation file upload.
    Now click Add to offline circulation queue.
[7] Connect the upload plugin to field 856$u.
    Enable HTML5MediaEnabled.
    Upload a webm file via the plugin. Click Choose to save the URL,
    and put 'video/webm' into 856$q. Save the biblio record.
    Check if you see the media tab with player on staff detail.
    (See also: Bug 17673 about empty OPACBaseURL.)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 14:20:05 +00:00
a3a1ec57aa Bug 16586: Koha Plugins: Limit results of GetPlugins by metadata
It would be helpful if we could find a plugin based on some metadata
tag as returned by the plugin. This extends the use of GetPlugins that
already supports searching on method.

GetPlugins is used in: admin/edi_accounts.pl, plugins/plugins-home.pl and
tools/stage-marc-import.pl. The changes in these three scripts are
minimal and just related to parameter passing.

Test t/db_dependent/Plugins.t includes another test for GetPlugins. In this
regard a metadata tag has been added to t/Koha/Plugins/Test.pm.

NOTE: This adjustment will also be used in a redesign for bug 15545.

Test plan:
Run t/db_dependent/Plugins.t.
Enable pref UseKohaPlugins and config var enable_plugins.
Go to plugins-home.pl. Verify that it still lists your plugins.
Bonus: Check edi_accounts or stage-marc-import.pl if you have a working
plugin for that.

Signed-off-by: Liz Rea <liz@catalyst.net.nz>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-09-09 12:13:39 +00:00
3badc57c93 Bug 10407: Add marcxml import (follow-up)
This patch makes the following changes:

[1] Based on the groundwork of the former patch, add call to
    RecordsFromMARCXMLFile in stage-marc-import. Use format param.
[2] Add format to the template. Use file extension to determine.
    If you use .xml or .marcxml as extension, MARCXML is selected.
[3] In stage-marc-import.tt mark UTF-8 encoding as UTF-8 not as utf8.
[4] BatchStageMarcRecords: do not call plugin if you have no records.
[5] RecordsFromISO2709File: also return errors in an array.
[6] In misc/stage_file.pl also use UTF-8. Handling of errors from [5].

Test plan:
[1] Import an empty file as MARC or MARCXML (with Tools/Stage..import).
[2] Import an non-empty file with invalid contents as MARC or MARCXML.
[3] Export a few records with Tools/Export as MARC and MARCXML.
[4] Import these two files. Check selected format versus file extension.
[5] Import a MARCXML file with misc/stage_file.pl.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-09-02 16:00:13 +00:00
3e541866cd Bug 10407: Resolve warnings from child process
After the staging job finishes, the child tries to print results to
STDOUT (which is closed). This creates warnings like:

stage-marc-import.pl: binmode() on closed filehandle STDOUT at C4/Templates.pm line 120.
stage-marc-import.pl: Filehandle STDOUT reopened as FH only for input at /usr/lib/perl5/Template/Provider.pm line 964.

Resolving it is simple: the child should exit after marking the job
as finished.

Test plan:
Import a marc file.
Observe that you do no longer have such warnings.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-09-02 16:00:13 +00:00
Olli-Antti Kivilahti
aa5bf4d3cc Bug 10407: Allow MARCXML records to be imported via GUI (groundwork)
Patch from Olli, manual rebase by Marcel (July 7, 2016).

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Needs follow-up. Test plan in the third patch.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-09-02 16:00:13 +00:00
02492bd6ea Bug 15005: Replace $ENV{SCRIPT_NAME} with the hardcoded script paths
Since I don't manage to make SCRIPT_NAME works directly with RequestHeader and/or
ReverseProxyPath, the easier way it to fix all the different occurrences.

Test plan:
On the import patrons page, confirm that the "Import" button does not redirect to a 'Not found' page.
You should stay on the import patrons page.

Note that if this change works, all others should work too.
Have a look at the diff of this patch and confirm there is no typo.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Tested in patron import and item batch modification

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-10-19 09:36:43 -03:00
83ea49a009 Bug 14321: Integrate Upload.pm into Koha
This patch makes the following changes to achieve that:
[1] Use Upload.pm in stage-marc-import.pl, upload-cover-image.pl,
    offline_circ/process_koc.pl and enqueue_koc.pl.
[2] A new file-upload.js replaces file-upload.inc in the associated template.
    We now use ajax to get progress figures instead of launching perl script
    upload-file-progress.
    The js changes now also allow for aborting a file upload.
[3] Adds a tools/upload script and template. It allows to upload multiple
    files at once.
[4] Makes upload-file return error messages in JSON. For a multiple upload,
    we could have some files with errors and others without errors.
    The upload is now marked as Failed only if there was no upload at all.
[5] The upload plugin is converted to use tools/upload with plugin param.
    Deleting an upload is now presented via the search results form.

NOTE: In editing the process_koc.tt I noticed that the form enqueuefile was
hidden and no longer used (with associated code in process_koc.pl). When a
file has been uploaded, I display the form again (with the Apply directly
button). The code still works.

NOTE: We fix an error in upload-file from one of the patches of bug 6874.
The userid of the Koha admin user is passed to haspermission, but we
should pick the userid from the session.

NOTE: Bug 14686 will add a specific permission for tools/upload.pl, and
will add the tools/upload script to the Tools menu.
For now, you need edit_catalogue to start upload.pl and you will
additionally need a permission like upload_local_cover_images
to successfully upload a new file.

Test plan:
[1] Upload a marc file in stage-marc-import. (This is temp storage.)
[2] Check new entry in table uploaded_files. Look for the file in your
    temporary directory (/tmp ?), subfolder koha_upload.
    Bonus: Remove permissions on this subfolder. Retry, check error and
    restore permissions again.
[3] Upload another (larger) file and abort the upload. Check table and
    directory again. You should have a partial file, but no record.
[4] Verify that Stage for import still works as expected.
[5] Test Upload local cover image. (Enable OPACLocalCoverImages.) You can
    test an individual image or a zip file including images and a file
    called datalink.txt (with lines biblionumber,filename).
[6] Test uploading a offline circulation file:
    Enable AllowOfflineCirculation, and create a koc file (plain text):
    Line1: Version=1.0\tA=1\tB=2
    Line2: 2015-08-06 08:00:00 345\treturn\t[barcode]
    Note: Replace tabs and barcode. The number of tabs is essential!
    Checkout the item with your barcode.
    Go to Offline circulation file upload. Upload and click Apply directly.
    Checkout again. Upload again, click Add to offline circulation queue.
[7] Upload three files via tools/upload.pl with a category and marked as
    public. Check the results in the table.
    Verify that you can download the file in OPAC without being logged in.
[8] Pick one new file and one of the files of step 7. Upload them in the
    same category. One upload should succeed. Check for reported error.
[9] Connect upload.pl to field 856$u.
    Goto Cataloguing editor.
    In an empty 856$u, click the tag editor. Upload a file and click Choose.
    Save the record. Open the record in the OPAC and click the link.
    Copy this link to your clipboard for next step.
[A] Go back to editor. Click the tag editor on the same 856 field.
    Choose for Delete.
    Open the link in your clipboard again. Error message?
[B] Check the process of upload, search, download and delete of an upload
    with some diacritical characters in the filename. (Bonus points for
    adding special chars in the category code.)
    Note: You can add categories via authorized values, UPLOAD key.

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Bug 14321: [QA Follow-up] Perltidy upload.pl

Run perltidy -pro=xt/perltidyrc on tools/upload.pl.
No other changes.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-09-25 12:02:52 -03:00
Jonathan Druart
3f3d0a6a32 Bug 9067: Hide item related strings when importing staged authority records
On importing authority records, the items related string should not be
displayed.

Test plan:
1/ Import an authority record
2/ Confirm the item related string are not displayed.
3/ Before this script, the record_type param was lost just after
importing the records. Now you should see
"Processing authority records" of "Processing bibliographic records" in
the "MARC staging results"
4/ Revert the import and check there is not item related strings.

No behavior should exist on importing bibliographic records.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-05-14 12:19:12 -03:00
115c4f348e Bug 14148: marc import broken if plugins are not enabled
Bug 12412 introduced a way to use plugins to generate MARC records
out of any arbitrary format. But failed to properly check for
a valid configuration (plugins_enabled entry on koha-conf.xml, and
even the UseKohaPlugins syspref).

This patch makes tools/stage-marc-import.pl test for the required
configuration before attempting to create the Koha::Plugins object.

To test:
1/ Make sure you don't have a plugins_enabled entry on your koha-conf.xml file
2/ On master, browse to tools/stage-marc-import.pl
=> FAIL: You get a nasty "Can't call method "GetPlugins"..." error
3/ Apply the patch and reload
=> SUCCESS: Stage MARC for import page renders correctly

Repeat with plugins_enabled == 0 and == 1. Also UseKohaPlugins enabled/disabled.
It should work as expected.

Regards
Tomas

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works as described, no koha-qa errors

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-05-06 12:44:46 -03:00
ca167b32b4 Bug 12412: Add ability for plugins to convert arbitrary files to MARC from record staging tool
Many libraries would like to be able to import various types of files as
MARC records ( citations, csv files, etc ). We can add a new function to
the plugins system to allow that kind of behavior at a very custom
level.

Test Plan:
1) Ensure you have plugins enabled and configured correctly
2) Installed the attached version 2.00 of the Kitchen Sink plugin
3) Download the attached text file
4) Browse to "Stage MARC records for import"
5) Select the downloaded text file for staging
6) After uploading, you should see a new area "Transform file to MARC:",
   select "Example Kitchen-Sink Plugin" from the pulldown menu
7) Click 'Stage for import"
8) Click 'Manage staged records"
9) You should now see two new MARC records!

Signed-off-by: Aleisha <aleishaamohia@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described - interesting new feature.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-05-04 15:33:51 -03:00
Jonathan Druart
a6c9bd0eb5 Bug 9978: Replace license header with the correct license (GPLv3+)
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>

http://bugs.koha-community.org/show_bug.cgi?id=9987

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-04-20 09:59:38 -03:00
6fbf0e679f Bug 13938 - Staging MARC records for import fails
It appears that bug 8970 has introduced a bug where a MARC file uploaded
for staging fails to import *unless* a matching rule is selected.

Test Plan:
1) Apply this patch
2) Stage and import a MARC record *without* choosing a matching rule
3) Note the staging works as expected

Signed-off-by: Cindy Ames <cmurdock@ccfls.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-04-08 09:42:39 -03:00
Jonathan Druart
e20270fec4 Bug 11944: use CGI( -utf8 ) everywhere
Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-01-13 13:07:21 -03:00
afd2418d73 Bug 11349: Change .tmpl -> .tt in scripts using templates
Since we switched to Template Toolkit we don't need to stick with the
sufix we used for HTML::Template::Pro.

This patch changes the occurences of '.tmpl' in favour of '.tt'.

To test:
- Apply the patch
- Install koha, and verify that every page can be accesed

Regards
To+

P.S. a followup will remove the glue code.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2014-07-17 11:05:49 -03:00
Robin Sheat
750a9ec97c Bug 8970 - fix MARC import under plack
There were database handles being shared between a parent and a child
process, which is a big no-no, and was leading to crazy crashes. Fine
under CGI, but not in a persistent environment. This causes the child to
make a new database handle to use. Also some small cleanups.

To test:
* In a plack environment,
* Tools -> stage MARC records for import
* Use a reasonable size file (but not too big as it all goes into RAM -
  I made one about 40MB.)
* Make sure that it works, and that the progress bars progress.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Tested with a 55M file, I reproduced the error and I confirm this patch
fixes it.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-05-25 23:51:12 +00:00
b38f0213fe Bug 8015: (follow-up) move from the Koha to the C4 namespace
Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2013-10-31 22:48:20 +00:00
622430cfb5 Bug 8015: Add MARC Modifications Templates
The MARC Modification Templates system gives Koha users
the power to make alterations to MARC records automatically
while staging MARC records for import.

This tool is useful for altering MARC records from
various venders work with your MARC framework.

The system essentially allows one to create a basic script
using actions to Copy, Move, Add, Update and Delete fields.

Each action can also have an optional condition to check
the value or existance of another field.

The Copy & Move actions also support Regular Expressions,
which can be used to automatically modify field values during the
copy/move. An example would be to strip out the '$' character
in field 020$c.

Furthermore, the value for an update can include variables
that change each time the template is used. Currently,
the system supports two variables, __BRANCHCODE__ which
is replaced with the branchcode of the library currently
using the template, and __CURRENTDATE__ which is replaced
with the current date in ISO format ( YYYY-MM-DD ).

At its simplist, it can perform functions such as:
Copy field 092$a to 952$c
At its most complex it can run actions like:
Copy field 020$c to 020$c using RegEx s/\$// if 020$c equals RegEx m/^\$/

Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2013-10-31 22:47:08 +00:00
Jared Camins-Esakov
7ad5e203da Bug 2060: Update command line MARC import scripts
Expose authority import functionality to the command line import
scripts, and rename them from commit_biblios_file.pl and
stage_biblios_file.pl to commit_file.pl and stage_file.pl.

To test (note that these instructions assume you have a MARC21
installation and are using the provided sample file):
1. Find a file of authorities (a sample file with MARC21 authorities
   is attached to bug 7475) and download it to your server
2. Stage the file using the following command (replace <filename> with
   the name of the file you saved in step 1):
   > misc/stage_file.pl --file <filename> --authorities
3. Note the batch number the script assigns to your batch
4. Commit the records using the following command (replace <batchnumber>
   with the batch number you made note of in step 3):
   > misc/commit_file.pl --batch-number <batchnumber>
5. Index the authorities Zebraqueue (or wait)
6. Confirm that the new authorities appear.
7. Create a matching rule with the following settings:
   Code: AUTHTEST
   Description: Personal name main entry
   Match threshold: 999
   Record type: Authority record
   Search index: Heading-main
   Score: 1000
   Tag: 100
   Subfields: a
   Offset: 0
   Length: 0
   (note the ID of this matching rule)
8. Stage the authority file again, this time using the following
   command:
   > misc/stage_file.pl --file <filename> --authorities \
     --match <matchingrule>
7. Revert the import with the following command:
   > misc/commit_file.pl --batch-number <batchnumber> --revert
8. Index the authorities Zebraqueue (or wait)
9. Confirm that the records have been removed
10. Import an authority record with the Stage MARC/Manage staged MARC
    tools in exactly the way you would for a bibliographic record,
    but choose "Authority" instead of "Bibliographic" for the record
    type.

Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>

Testing plan delivers as it should.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
2012-09-19 17:16:18 +02:00
Jared Camins-Esakov
6e71b80ca3 Bug 7475: Teach matching rules to handle authorities
* Add the code necessary to handle authorities with matching rules and
  import batches.
* Update all the scripts that use the matcher and import batch code
  to use the new API.
* Add authority records to the matching rules interface in the staff
  client.

http://bugs.koha-community.org/show_bug.cgi?id=2060
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
2012-09-19 17:15:56 +02:00
Chris Cormack
9573f444fc Merge remote-tracking branch 'kc/new/bug_5449' into kcmaster 2011-05-09 08:25:36 +12:00
Jared Camins-Esakov
415ea18007 Bug 5859: Allow user to set encoding for import
Rather than having options for MARC21 and UNIMARC in the "Character encoding"
dropdown, the user should be able to select the appropriate character encoding.
The default retains the current behavior, which is to allow the system to guess
which character encoding is in use. However, it should be noticed that this is
almost always wrong for non-UTF8 records with non-ASCII characters. Specifying
a character set is much more reliable if you're not using UTF-8.

Rebased to use Template::Toolkit instead of HTML::Template::Pro.

Signed-off-by: Jared Camins-Esakov <jcamins@bywatersolutions.com>
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
2011-04-19 13:05:48 +12:00
Julian Maurice
6e85ee6db0 Bug 5449: JSON malformed in Koha - Blocker with jQuery 1.4.x
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
2011-03-12 08:53:41 +13:00
ce5e2429db fixing various links to point to *.koha-community.org
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-10-21 22:08:24 -04:00
Donovan Jones
44201a54e1 Bug 2505 - Add commented use warnings where missing in the tools/ directory 2010-04-21 20:26:38 +12:00
Lars Wirzenius
49e0a8fc78 Fix FSF address in directory tools/
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-03-16 20:17:51 -04:00
Lars Wirzenius
9675fcb5f0 Fix wording of 'at your option' in GPL license statement
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-03-16 20:17:48 -04:00
Galen Charlton
7ce6822973 removed incorrect module import
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-07 20:09:22 -05:00
Nahuel ANGELINETTI
734cf78be6 Autoselect marcflavour in marc import
This patch is writted by Paul Poulain.
It auto-select the marcflavour from syspref in the stage marc import.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-03-04 10:57:22 -06:00
Galen Charlton
438ed23337 staging import - enhance record overlay behavior
Enhanced the ability of catalogers to specify how
bib and item records should be added, replaced, or
ignored during a staging import.

When an import batch of bib records is staged and commit,
the user can now explicitly specify what should occur
when an incoming bib record has, or does not have, a match
with a record already in the database.  The options are:

if match found (overlay_action):
  create_new (just add the incoming record)
  replace (replace the matched record with the incoming one)
  use_template (option not implemented)
  ignore (do nothing with the incoming bib; however, the
          items attached to it may still be processed
           based on the item action)

if no match is found (nomatch_action):
  create_new (just add the incoming record)
  ignore (do nothing with the incoming bib; in this
          case, any items attached to it will be
          ignored since there will be nothing to
          attach them to)

The following options for handling items embedded in the
bib record are now available:

  always_add (add the items to the new or replaced bib)
  add_only_if_match (add the items only if the incoming bib
                     matches an existing bib)
  add_only_if_add (add the items only if the incoming bib
                   does *not* match an existing bib)
  ignore (ignore the items entirely)

With these changes, it is now possible to support the following use cases:

[1] A library joining an existing Koha database wishes to add their
    items to existing bib records if they match, but does not want
    to overlay the bib records themselves.
[2] A library wants to load a file of records, but only handle
    the new ones, not ones that are already in the database.
[3] A library wants to load a file of records, but only
    handle the ones that match existing records (e.g., if
    the records are coming back from an authority control vendor).

Documentation changes:

* See description above; also, screenshots of the 'stage MARC records
for import' and 'manage staged MARC records' should be updated.

Test cases:

* Added test cases to exercise staging and committing import batches.

UI changes:

* The pages for staging and managing import batches now have
  controls for setting the overlay action, action if no match,
  and item action separately.
* in the manage import batch tool, user is notified when they
  change overlay action, no-match action, and item action
* HTML for manage import batch tool now uses fieldsets

Database changes (DB rev 076):

* added import_batches.item_action
* added import_batches.nomatch_action
* added 'ignore' as a valid value for import_batches.overlay_action
* added 'ignored' as a valid value for import_records.status
* added 'status' as a valid value for import_items.status

API changes:

* new accessor routines for C4::ImportBatch

    GetImportBatchNoMatchAction
    SetImportBatchNoMatchAction
    GetImportBatchItemAction
    SetImportBatchItemAction

* new internal functions for C4::ImportBatch to
  determine how a given bib and item are to be
  processed, based on overlay_action, nomatch_action,
  and item_action:

    _get_commit_action
    _get_revert_action

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-30 21:58:46 -05:00
Galen Charlton
fc6ccb1a61 granular permissions - updated Tools
Updated all scripts appearing on the tools page
to respect a granular permission defined for
each of them.

The tools menu and home page have been changed so
that only the specific tools that a user has
access are displayed.  This is simple, but depending on
the module and circumstance, it may be better to
display functions that the user has does not have
access to, but disable the links and do some sort
of visual styling to indicate that a function exists
but requires additional privileges to access.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-10 15:53:30 -05:00
Galen Charlton
b88682f1bb bug 1372: count MARC records correctly
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-08 20:22:43 -06:00
Galen Charlton
8f854c0867 speed boost: tools stage and commit bib records
Turned off autocommit; commit every 50 records.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-03 16:26:26 -06:00
Galen Charlton
74e51161d8 removed extraneous warns
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-28 21:44:27 -06:00
Galen Charlton
a869ed0435 MARC import: part 5 of large file support
Staging job now gets moved to background so that
it does not get killed if it takes longer
than the Apache timeout.

Added AJAX monitoring of job status.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-26 20:05:34 -06:00
Galen Charlton
cabb06c56c MARC import: part 4 of large file support
* Added C4::BackgroundJob to monitor tracking of
  background jobs
* start modifying stage-marc-import to use it

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-26 20:05:30 -06:00
Galen Charlton
e9f36cc227 MARC import: part 3 of large file support
* Introduced C4::UploadedFile to handle management and
  progress tracking of uploaded files.
* Modified stage-marc-import.pl to handle new upload
  mechanism

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-25 16:45:53 -06:00
Galen Charlton
b69facedc4 matching enhancements -- allow matching rule to be changed on the fly
Enhancement to store the matching rule associated with an
import batch and to allow the current matching rule in
effect to be changed and the duplicate detection redone.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-16 07:37:24 -06:00
Galen Charlton
75bd4b2dac bibliographic matching enhancements
* serialization of matching rules to database
* added two default matching rules for MARC21
* update import staging CGI to choose matching rule

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-16 07:37:21 -06:00
Galen Charlton
01c19a265a batch import rework: added suport for loading and dropping items
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-10-31 05:56:51 -05:00
Galen Charlton
853aa657ba batch import rework -- implement stage-commit-undo for batch import
Revamps the import options on the tools menu to have two parts:

[1] Staging (load file into reservoir)
[2] Managing (review the list of staged batches, then
    choose to commit or undo a given batch.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-10-29 16:47:59 -05:00
Renamed from tools/import.pl (Browse further)