Commit graph

35 commits

Author SHA1 Message Date
02492bd6ea Bug 15005: Replace $ENV{SCRIPT_NAME} with the hardcoded script paths
Since I don't manage to make SCRIPT_NAME works directly with RequestHeader and/or
ReverseProxyPath, the easier way it to fix all the different occurrences.

Test plan:
On the import patrons page, confirm that the "Import" button does not redirect to a 'Not found' page.
You should stay on the import patrons page.

Note that if this change works, all others should work too.
Have a look at the diff of this patch and confirm there is no typo.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Tested in patron import and item batch modification

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-10-19 09:36:43 -03:00
83ea49a009 Bug 14321: Integrate Upload.pm into Koha
This patch makes the following changes to achieve that:
[1] Use Upload.pm in stage-marc-import.pl, upload-cover-image.pl,
    offline_circ/process_koc.pl and enqueue_koc.pl.
[2] A new file-upload.js replaces file-upload.inc in the associated template.
    We now use ajax to get progress figures instead of launching perl script
    upload-file-progress.
    The js changes now also allow for aborting a file upload.
[3] Adds a tools/upload script and template. It allows to upload multiple
    files at once.
[4] Makes upload-file return error messages in JSON. For a multiple upload,
    we could have some files with errors and others without errors.
    The upload is now marked as Failed only if there was no upload at all.
[5] The upload plugin is converted to use tools/upload with plugin param.
    Deleting an upload is now presented via the search results form.

NOTE: In editing the process_koc.tt I noticed that the form enqueuefile was
hidden and no longer used (with associated code in process_koc.pl). When a
file has been uploaded, I display the form again (with the Apply directly
button). The code still works.

NOTE: We fix an error in upload-file from one of the patches of bug 6874.
The userid of the Koha admin user is passed to haspermission, but we
should pick the userid from the session.

NOTE: Bug 14686 will add a specific permission for tools/upload.pl, and
will add the tools/upload script to the Tools menu.
For now, you need edit_catalogue to start upload.pl and you will
additionally need a permission like upload_local_cover_images
to successfully upload a new file.

Test plan:
[1] Upload a marc file in stage-marc-import. (This is temp storage.)
[2] Check new entry in table uploaded_files. Look for the file in your
    temporary directory (/tmp ?), subfolder koha_upload.
    Bonus: Remove permissions on this subfolder. Retry, check error and
    restore permissions again.
[3] Upload another (larger) file and abort the upload. Check table and
    directory again. You should have a partial file, but no record.
[4] Verify that Stage for import still works as expected.
[5] Test Upload local cover image. (Enable OPACLocalCoverImages.) You can
    test an individual image or a zip file including images and a file
    called datalink.txt (with lines biblionumber,filename).
[6] Test uploading a offline circulation file:
    Enable AllowOfflineCirculation, and create a koc file (plain text):
    Line1: Version=1.0\tA=1\tB=2
    Line2: 2015-08-06 08:00:00 345\treturn\t[barcode]
    Note: Replace tabs and barcode. The number of tabs is essential!
    Checkout the item with your barcode.
    Go to Offline circulation file upload. Upload and click Apply directly.
    Checkout again. Upload again, click Add to offline circulation queue.
[7] Upload three files via tools/upload.pl with a category and marked as
    public. Check the results in the table.
    Verify that you can download the file in OPAC without being logged in.
[8] Pick one new file and one of the files of step 7. Upload them in the
    same category. One upload should succeed. Check for reported error.
[9] Connect upload.pl to field 856$u.
    Goto Cataloguing editor.
    In an empty 856$u, click the tag editor. Upload a file and click Choose.
    Save the record. Open the record in the OPAC and click the link.
    Copy this link to your clipboard for next step.
[A] Go back to editor. Click the tag editor on the same 856 field.
    Choose for Delete.
    Open the link in your clipboard again. Error message?
[B] Check the process of upload, search, download and delete of an upload
    with some diacritical characters in the filename. (Bonus points for
    adding special chars in the category code.)
    Note: You can add categories via authorized values, UPLOAD key.

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Bug 14321: [QA Follow-up] Perltidy upload.pl

Run perltidy -pro=xt/perltidyrc on tools/upload.pl.
No other changes.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-09-25 12:02:52 -03:00
Jonathan Druart
3f3d0a6a32 Bug 9067: Hide item related strings when importing staged authority records
On importing authority records, the items related string should not be
displayed.

Test plan:
1/ Import an authority record
2/ Confirm the item related string are not displayed.
3/ Before this script, the record_type param was lost just after
importing the records. Now you should see
"Processing authority records" of "Processing bibliographic records" in
the "MARC staging results"
4/ Revert the import and check there is not item related strings.

No behavior should exist on importing bibliographic records.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-05-14 12:19:12 -03:00
115c4f348e Bug 14148: marc import broken if plugins are not enabled
Bug 12412 introduced a way to use plugins to generate MARC records
out of any arbitrary format. But failed to properly check for
a valid configuration (plugins_enabled entry on koha-conf.xml, and
even the UseKohaPlugins syspref).

This patch makes tools/stage-marc-import.pl test for the required
configuration before attempting to create the Koha::Plugins object.

To test:
1/ Make sure you don't have a plugins_enabled entry on your koha-conf.xml file
2/ On master, browse to tools/stage-marc-import.pl
=> FAIL: You get a nasty "Can't call method "GetPlugins"..." error
3/ Apply the patch and reload
=> SUCCESS: Stage MARC for import page renders correctly

Repeat with plugins_enabled == 0 and == 1. Also UseKohaPlugins enabled/disabled.
It should work as expected.

Regards
Tomas

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works as described, no koha-qa errors

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-05-06 12:44:46 -03:00
ca167b32b4 Bug 12412: Add ability for plugins to convert arbitrary files to MARC from record staging tool
Many libraries would like to be able to import various types of files as
MARC records ( citations, csv files, etc ). We can add a new function to
the plugins system to allow that kind of behavior at a very custom
level.

Test Plan:
1) Ensure you have plugins enabled and configured correctly
2) Installed the attached version 2.00 of the Kitchen Sink plugin
3) Download the attached text file
4) Browse to "Stage MARC records for import"
5) Select the downloaded text file for staging
6) After uploading, you should see a new area "Transform file to MARC:",
   select "Example Kitchen-Sink Plugin" from the pulldown menu
7) Click 'Stage for import"
8) Click 'Manage staged records"
9) You should now see two new MARC records!

Signed-off-by: Aleisha <aleishaamohia@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described - interesting new feature.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-05-04 15:33:51 -03:00
Jonathan Druart
a6c9bd0eb5 Bug 9978: Replace license header with the correct license (GPLv3+)
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>

http://bugs.koha-community.org/show_bug.cgi?id=9987

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-04-20 09:59:38 -03:00
6fbf0e679f Bug 13938 - Staging MARC records for import fails
It appears that bug 8970 has introduced a bug where a MARC file uploaded
for staging fails to import *unless* a matching rule is selected.

Test Plan:
1) Apply this patch
2) Stage and import a MARC record *without* choosing a matching rule
3) Note the staging works as expected

Signed-off-by: Cindy Ames <cmurdock@ccfls.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-04-08 09:42:39 -03:00
Jonathan Druart
e20270fec4 Bug 11944: use CGI( -utf8 ) everywhere
Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-01-13 13:07:21 -03:00
afd2418d73 Bug 11349: Change .tmpl -> .tt in scripts using templates
Since we switched to Template Toolkit we don't need to stick with the
sufix we used for HTML::Template::Pro.

This patch changes the occurences of '.tmpl' in favour of '.tt'.

To test:
- Apply the patch
- Install koha, and verify that every page can be accesed

Regards
To+

P.S. a followup will remove the glue code.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2014-07-17 11:05:49 -03:00
Robin Sheat
750a9ec97c Bug 8970 - fix MARC import under plack
There were database handles being shared between a parent and a child
process, which is a big no-no, and was leading to crazy crashes. Fine
under CGI, but not in a persistent environment. This causes the child to
make a new database handle to use. Also some small cleanups.

To test:
* In a plack environment,
* Tools -> stage MARC records for import
* Use a reasonable size file (but not too big as it all goes into RAM -
  I made one about 40MB.)
* Make sure that it works, and that the progress bars progress.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Tested with a 55M file, I reproduced the error and I confirm this patch
fixes it.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-05-25 23:51:12 +00:00
b38f0213fe Bug 8015: (follow-up) move from the Koha to the C4 namespace
Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2013-10-31 22:48:20 +00:00
622430cfb5 Bug 8015: Add MARC Modifications Templates
The MARC Modification Templates system gives Koha users
the power to make alterations to MARC records automatically
while staging MARC records for import.

This tool is useful for altering MARC records from
various venders work with your MARC framework.

The system essentially allows one to create a basic script
using actions to Copy, Move, Add, Update and Delete fields.

Each action can also have an optional condition to check
the value or existance of another field.

The Copy & Move actions also support Regular Expressions,
which can be used to automatically modify field values during the
copy/move. An example would be to strip out the '$' character
in field 020$c.

Furthermore, the value for an update can include variables
that change each time the template is used. Currently,
the system supports two variables, __BRANCHCODE__ which
is replaced with the branchcode of the library currently
using the template, and __CURRENTDATE__ which is replaced
with the current date in ISO format ( YYYY-MM-DD ).

At its simplist, it can perform functions such as:
Copy field 092$a to 952$c
At its most complex it can run actions like:
Copy field 020$c to 020$c using RegEx s/\$// if 020$c equals RegEx m/^\$/

Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2013-10-31 22:47:08 +00:00
Jared Camins-Esakov
7ad5e203da Bug 2060: Update command line MARC import scripts
Expose authority import functionality to the command line import
scripts, and rename them from commit_biblios_file.pl and
stage_biblios_file.pl to commit_file.pl and stage_file.pl.

To test (note that these instructions assume you have a MARC21
installation and are using the provided sample file):
1. Find a file of authorities (a sample file with MARC21 authorities
   is attached to bug 7475) and download it to your server
2. Stage the file using the following command (replace <filename> with
   the name of the file you saved in step 1):
   > misc/stage_file.pl --file <filename> --authorities
3. Note the batch number the script assigns to your batch
4. Commit the records using the following command (replace <batchnumber>
   with the batch number you made note of in step 3):
   > misc/commit_file.pl --batch-number <batchnumber>
5. Index the authorities Zebraqueue (or wait)
6. Confirm that the new authorities appear.
7. Create a matching rule with the following settings:
   Code: AUTHTEST
   Description: Personal name main entry
   Match threshold: 999
   Record type: Authority record
   Search index: Heading-main
   Score: 1000
   Tag: 100
   Subfields: a
   Offset: 0
   Length: 0
   (note the ID of this matching rule)
8. Stage the authority file again, this time using the following
   command:
   > misc/stage_file.pl --file <filename> --authorities \
     --match <matchingrule>
7. Revert the import with the following command:
   > misc/commit_file.pl --batch-number <batchnumber> --revert
8. Index the authorities Zebraqueue (or wait)
9. Confirm that the records have been removed
10. Import an authority record with the Stage MARC/Manage staged MARC
    tools in exactly the way you would for a bibliographic record,
    but choose "Authority" instead of "Bibliographic" for the record
    type.

Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>

Testing plan delivers as it should.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
2012-09-19 17:16:18 +02:00
Jared Camins-Esakov
6e71b80ca3 Bug 7475: Teach matching rules to handle authorities
* Add the code necessary to handle authorities with matching rules and
  import batches.
* Update all the scripts that use the matcher and import batch code
  to use the new API.
* Add authority records to the matching rules interface in the staff
  client.

http://bugs.koha-community.org/show_bug.cgi?id=2060
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
2012-09-19 17:15:56 +02:00
Chris Cormack
9573f444fc Merge remote-tracking branch 'kc/new/bug_5449' into kcmaster 2011-05-09 08:25:36 +12:00
Jared Camins-Esakov
415ea18007 Bug 5859: Allow user to set encoding for import
Rather than having options for MARC21 and UNIMARC in the "Character encoding"
dropdown, the user should be able to select the appropriate character encoding.
The default retains the current behavior, which is to allow the system to guess
which character encoding is in use. However, it should be noticed that this is
almost always wrong for non-UTF8 records with non-ASCII characters. Specifying
a character set is much more reliable if you're not using UTF-8.

Rebased to use Template::Toolkit instead of HTML::Template::Pro.

Signed-off-by: Jared Camins-Esakov <jcamins@bywatersolutions.com>
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
2011-04-19 13:05:48 +12:00
Julian Maurice
6e85ee6db0 Bug 5449: JSON malformed in Koha - Blocker with jQuery 1.4.x
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
2011-03-12 08:53:41 +13:00
ce5e2429db fixing various links to point to *.koha-community.org
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-10-21 22:08:24 -04:00
Donovan Jones
44201a54e1 Bug 2505 - Add commented use warnings where missing in the tools/ directory 2010-04-21 20:26:38 +12:00
Lars Wirzenius
49e0a8fc78 Fix FSF address in directory tools/
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-03-16 20:17:51 -04:00
Lars Wirzenius
9675fcb5f0 Fix wording of 'at your option' in GPL license statement
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-03-16 20:17:48 -04:00
Galen Charlton
7ce6822973 removed incorrect module import
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-07 20:09:22 -05:00
Nahuel ANGELINETTI
734cf78be6 Autoselect marcflavour in marc import
This patch is writted by Paul Poulain.
It auto-select the marcflavour from syspref in the stage marc import.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-03-04 10:57:22 -06:00
Galen Charlton
438ed23337 staging import - enhance record overlay behavior
Enhanced the ability of catalogers to specify how
bib and item records should be added, replaced, or
ignored during a staging import.

When an import batch of bib records is staged and commit,
the user can now explicitly specify what should occur
when an incoming bib record has, or does not have, a match
with a record already in the database.  The options are:

if match found (overlay_action):
  create_new (just add the incoming record)
  replace (replace the matched record with the incoming one)
  use_template (option not implemented)
  ignore (do nothing with the incoming bib; however, the
          items attached to it may still be processed
           based on the item action)

if no match is found (nomatch_action):
  create_new (just add the incoming record)
  ignore (do nothing with the incoming bib; in this
          case, any items attached to it will be
          ignored since there will be nothing to
          attach them to)

The following options for handling items embedded in the
bib record are now available:

  always_add (add the items to the new or replaced bib)
  add_only_if_match (add the items only if the incoming bib
                     matches an existing bib)
  add_only_if_add (add the items only if the incoming bib
                   does *not* match an existing bib)
  ignore (ignore the items entirely)

With these changes, it is now possible to support the following use cases:

[1] A library joining an existing Koha database wishes to add their
    items to existing bib records if they match, but does not want
    to overlay the bib records themselves.
[2] A library wants to load a file of records, but only handle
    the new ones, not ones that are already in the database.
[3] A library wants to load a file of records, but only
    handle the ones that match existing records (e.g., if
    the records are coming back from an authority control vendor).

Documentation changes:

* See description above; also, screenshots of the 'stage MARC records
for import' and 'manage staged MARC records' should be updated.

Test cases:

* Added test cases to exercise staging and committing import batches.

UI changes:

* The pages for staging and managing import batches now have
  controls for setting the overlay action, action if no match,
  and item action separately.
* in the manage import batch tool, user is notified when they
  change overlay action, no-match action, and item action
* HTML for manage import batch tool now uses fieldsets

Database changes (DB rev 076):

* added import_batches.item_action
* added import_batches.nomatch_action
* added 'ignore' as a valid value for import_batches.overlay_action
* added 'ignored' as a valid value for import_records.status
* added 'status' as a valid value for import_items.status

API changes:

* new accessor routines for C4::ImportBatch

    GetImportBatchNoMatchAction
    SetImportBatchNoMatchAction
    GetImportBatchItemAction
    SetImportBatchItemAction

* new internal functions for C4::ImportBatch to
  determine how a given bib and item are to be
  processed, based on overlay_action, nomatch_action,
  and item_action:

    _get_commit_action
    _get_revert_action

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-30 21:58:46 -05:00
Galen Charlton
fc6ccb1a61 granular permissions - updated Tools
Updated all scripts appearing on the tools page
to respect a granular permission defined for
each of them.

The tools menu and home page have been changed so
that only the specific tools that a user has
access are displayed.  This is simple, but depending on
the module and circumstance, it may be better to
display functions that the user has does not have
access to, but disable the links and do some sort
of visual styling to indicate that a function exists
but requires additional privileges to access.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-10 15:53:30 -05:00
Galen Charlton
b88682f1bb bug 1372: count MARC records correctly
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-08 20:22:43 -06:00
Galen Charlton
8f854c0867 speed boost: tools stage and commit bib records
Turned off autocommit; commit every 50 records.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-03 16:26:26 -06:00
Galen Charlton
74e51161d8 removed extraneous warns
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-28 21:44:27 -06:00
Galen Charlton
a869ed0435 MARC import: part 5 of large file support
Staging job now gets moved to background so that
it does not get killed if it takes longer
than the Apache timeout.

Added AJAX monitoring of job status.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-26 20:05:34 -06:00
Galen Charlton
cabb06c56c MARC import: part 4 of large file support
* Added C4::BackgroundJob to monitor tracking of
  background jobs
* start modifying stage-marc-import to use it

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-26 20:05:30 -06:00
Galen Charlton
e9f36cc227 MARC import: part 3 of large file support
* Introduced C4::UploadedFile to handle management and
  progress tracking of uploaded files.
* Modified stage-marc-import.pl to handle new upload
  mechanism

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-25 16:45:53 -06:00
Galen Charlton
b69facedc4 matching enhancements -- allow matching rule to be changed on the fly
Enhancement to store the matching rule associated with an
import batch and to allow the current matching rule in
effect to be changed and the duplicate detection redone.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-16 07:37:24 -06:00
Galen Charlton
75bd4b2dac bibliographic matching enhancements
* serialization of matching rules to database
* added two default matching rules for MARC21
* update import staging CGI to choose matching rule

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-16 07:37:21 -06:00
Galen Charlton
01c19a265a batch import rework: added suport for loading and dropping items
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-10-31 05:56:51 -05:00
Galen Charlton
853aa657ba batch import rework -- implement stage-commit-undo for batch import
Revamps the import options on the tools menu to have two parts:

[1] Staging (load file into reservoir)
[2] Managing (review the list of staged batches, then
    choose to commit or undo a given batch.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-10-29 16:47:59 -05:00
Renamed from tools/import.pl (Browse further)