The get routine actually returns records from uploaded_files. It should be
possible to replace its calls by direct calls of Koha::UploadedFiles.
This patch is the crux of this patch set. It deals with all scripts that
use Koha::Upload.
In the process we do:
[1] Add a file_handle method to Koha::UploadedFile. This was previously
arranged via the fh parameter of get.
[2] Add a full_path method to UploadedFile. Previously returned in the
path hash key of get. (Name is replaced by filename.)
[3] Add a search_term method too (implementing get({ term => .. }).
This logic came from _lookup.
[4] Add a keep_file parameter to delete method. Only used in test now.
Test plan:
[1] Run t/db_dependent/Upload.t
[2] Go to Tools/Upload. Add an upload, download and delete.
[3] Add another public upload , search for it.
Use the hashvalue to download via opac with URL:
cgi-bin/koha/opac-retrieve-file.pl?id=[hashvalue]
[4] Go to Tools/Stage MARC for import. Import a marc file.
[5] Go to Tools/Upload local cover image. Import an image file.
Enable OPACLocalCoverImages to see result.
[6] Test uploading a offline circulation file:
Enable AllowOfflineCirculation, and create a koc file (plain text):
Line1: Version=1.0\tA=1\tB=2
Line2: 2016-11-23 16:00:00 345\treturn\t[barcode]
Note: Replace tabs and barcode. The number of tabs is essential!
Checkout the item with your barcode.
Go to Circulation/Offline circulation file upload.
Upload and click Apply directly.
Checkout again. Repeat Offline circulation file upload.
Now click Add to offline circulation queue.
[7] Connect the upload plugin to field 856$u.
Enable HTML5MediaEnabled.
Upload a webm file via the plugin. Click Choose to save the URL,
and put 'video/webm' into 856$q. Save the biblio record.
Check if you see the media tab with player on staff detail.
(See also: Bug 17673 about empty OPACBaseURL.)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
It would be helpful if we could find a plugin based on some metadata
tag as returned by the plugin. This extends the use of GetPlugins that
already supports searching on method.
GetPlugins is used in: admin/edi_accounts.pl, plugins/plugins-home.pl and
tools/stage-marc-import.pl. The changes in these three scripts are
minimal and just related to parameter passing.
Test t/db_dependent/Plugins.t includes another test for GetPlugins. In this
regard a metadata tag has been added to t/Koha/Plugins/Test.pm.
NOTE: This adjustment will also be used in a redesign for bug 15545.
Test plan:
Run t/db_dependent/Plugins.t.
Enable pref UseKohaPlugins and config var enable_plugins.
Go to plugins-home.pl. Verify that it still lists your plugins.
Bonus: Check edi_accounts or stage-marc-import.pl if you have a working
plugin for that.
Signed-off-by: Liz Rea <liz@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch makes the following changes:
[1] Based on the groundwork of the former patch, add call to
RecordsFromMARCXMLFile in stage-marc-import. Use format param.
[2] Add format to the template. Use file extension to determine.
If you use .xml or .marcxml as extension, MARCXML is selected.
[3] In stage-marc-import.tt mark UTF-8 encoding as UTF-8 not as utf8.
[4] BatchStageMarcRecords: do not call plugin if you have no records.
[5] RecordsFromISO2709File: also return errors in an array.
[6] In misc/stage_file.pl also use UTF-8. Handling of errors from [5].
Test plan:
[1] Import an empty file as MARC or MARCXML (with Tools/Stage..import).
[2] Import an non-empty file with invalid contents as MARC or MARCXML.
[3] Export a few records with Tools/Export as MARC and MARCXML.
[4] Import these two files. Check selected format versus file extension.
[5] Import a MARCXML file with misc/stage_file.pl.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
After the staging job finishes, the child tries to print results to
STDOUT (which is closed). This creates warnings like:
stage-marc-import.pl: binmode() on closed filehandle STDOUT at C4/Templates.pm line 120.
stage-marc-import.pl: Filehandle STDOUT reopened as FH only for input at /usr/lib/perl5/Template/Provider.pm line 964.
Resolving it is simple: the child should exit after marking the job
as finished.
Test plan:
Import a marc file.
Observe that you do no longer have such warnings.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Patch from Olli, manual rebase by Marcel (July 7, 2016).
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Needs follow-up. Test plan in the third patch.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Since I don't manage to make SCRIPT_NAME works directly with RequestHeader and/or
ReverseProxyPath, the easier way it to fix all the different occurrences.
Test plan:
On the import patrons page, confirm that the "Import" button does not redirect to a 'Not found' page.
You should stay on the import patrons page.
Note that if this change works, all others should work too.
Have a look at the diff of this patch and confirm there is no typo.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Tested in patron import and item batch modification
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch makes the following changes to achieve that:
[1] Use Upload.pm in stage-marc-import.pl, upload-cover-image.pl,
offline_circ/process_koc.pl and enqueue_koc.pl.
[2] A new file-upload.js replaces file-upload.inc in the associated template.
We now use ajax to get progress figures instead of launching perl script
upload-file-progress.
The js changes now also allow for aborting a file upload.
[3] Adds a tools/upload script and template. It allows to upload multiple
files at once.
[4] Makes upload-file return error messages in JSON. For a multiple upload,
we could have some files with errors and others without errors.
The upload is now marked as Failed only if there was no upload at all.
[5] The upload plugin is converted to use tools/upload with plugin param.
Deleting an upload is now presented via the search results form.
NOTE: In editing the process_koc.tt I noticed that the form enqueuefile was
hidden and no longer used (with associated code in process_koc.pl). When a
file has been uploaded, I display the form again (with the Apply directly
button). The code still works.
NOTE: We fix an error in upload-file from one of the patches of bug 6874.
The userid of the Koha admin user is passed to haspermission, but we
should pick the userid from the session.
NOTE: Bug 14686 will add a specific permission for tools/upload.pl, and
will add the tools/upload script to the Tools menu.
For now, you need edit_catalogue to start upload.pl and you will
additionally need a permission like upload_local_cover_images
to successfully upload a new file.
Test plan:
[1] Upload a marc file in stage-marc-import. (This is temp storage.)
[2] Check new entry in table uploaded_files. Look for the file in your
temporary directory (/tmp ?), subfolder koha_upload.
Bonus: Remove permissions on this subfolder. Retry, check error and
restore permissions again.
[3] Upload another (larger) file and abort the upload. Check table and
directory again. You should have a partial file, but no record.
[4] Verify that Stage for import still works as expected.
[5] Test Upload local cover image. (Enable OPACLocalCoverImages.) You can
test an individual image or a zip file including images and a file
called datalink.txt (with lines biblionumber,filename).
[6] Test uploading a offline circulation file:
Enable AllowOfflineCirculation, and create a koc file (plain text):
Line1: Version=1.0\tA=1\tB=2
Line2: 2015-08-06 08:00:00 345\treturn\t[barcode]
Note: Replace tabs and barcode. The number of tabs is essential!
Checkout the item with your barcode.
Go to Offline circulation file upload. Upload and click Apply directly.
Checkout again. Upload again, click Add to offline circulation queue.
[7] Upload three files via tools/upload.pl with a category and marked as
public. Check the results in the table.
Verify that you can download the file in OPAC without being logged in.
[8] Pick one new file and one of the files of step 7. Upload them in the
same category. One upload should succeed. Check for reported error.
[9] Connect upload.pl to field 856$u.
Goto Cataloguing editor.
In an empty 856$u, click the tag editor. Upload a file and click Choose.
Save the record. Open the record in the OPAC and click the link.
Copy this link to your clipboard for next step.
[A] Go back to editor. Click the tag editor on the same 856 field.
Choose for Delete.
Open the link in your clipboard again. Error message?
[B] Check the process of upload, search, download and delete of an upload
with some diacritical characters in the filename. (Bonus points for
adding special chars in the category code.)
Note: You can add categories via authorized values, UPLOAD key.
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Bug 14321: [QA Follow-up] Perltidy upload.pl
Run perltidy -pro=xt/perltidyrc on tools/upload.pl.
No other changes.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
On importing authority records, the items related string should not be
displayed.
Test plan:
1/ Import an authority record
2/ Confirm the item related string are not displayed.
3/ Before this script, the record_type param was lost just after
importing the records. Now you should see
"Processing authority records" of "Processing bibliographic records" in
the "MARC staging results"
4/ Revert the import and check there is not item related strings.
No behavior should exist on importing bibliographic records.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Bug 12412 introduced a way to use plugins to generate MARC records
out of any arbitrary format. But failed to properly check for
a valid configuration (plugins_enabled entry on koha-conf.xml, and
even the UseKohaPlugins syspref).
This patch makes tools/stage-marc-import.pl test for the required
configuration before attempting to create the Koha::Plugins object.
To test:
1/ Make sure you don't have a plugins_enabled entry on your koha-conf.xml file
2/ On master, browse to tools/stage-marc-import.pl
=> FAIL: You get a nasty "Can't call method "GetPlugins"..." error
3/ Apply the patch and reload
=> SUCCESS: Stage MARC for import page renders correctly
Repeat with plugins_enabled == 0 and == 1. Also UseKohaPlugins enabled/disabled.
It should work as expected.
Regards
Tomas
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works as described, no koha-qa errors
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Many libraries would like to be able to import various types of files as
MARC records ( citations, csv files, etc ). We can add a new function to
the plugins system to allow that kind of behavior at a very custom
level.
Test Plan:
1) Ensure you have plugins enabled and configured correctly
2) Installed the attached version 2.00 of the Kitchen Sink plugin
3) Download the attached text file
4) Browse to "Stage MARC records for import"
5) Select the downloaded text file for staging
6) After uploading, you should see a new area "Transform file to MARC:",
select "Example Kitchen-Sink Plugin" from the pulldown menu
7) Click 'Stage for import"
8) Click 'Manage staged records"
9) You should now see two new MARC records!
Signed-off-by: Aleisha <aleishaamohia@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described - interesting new feature.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
http://bugs.koha-community.org/show_bug.cgi?id=9987
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
It appears that bug 8970 has introduced a bug where a MARC file uploaded
for staging fails to import *unless* a matching rule is selected.
Test Plan:
1) Apply this patch
2) Stage and import a MARC record *without* choosing a matching rule
3) Note the staging works as expected
Signed-off-by: Cindy Ames <cmurdock@ccfls.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Since we switched to Template Toolkit we don't need to stick with the
sufix we used for HTML::Template::Pro.
This patch changes the occurences of '.tmpl' in favour of '.tt'.
To test:
- Apply the patch
- Install koha, and verify that every page can be accesed
Regards
To+
P.S. a followup will remove the glue code.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
There were database handles being shared between a parent and a child
process, which is a big no-no, and was leading to crazy crashes. Fine
under CGI, but not in a persistent environment. This causes the child to
make a new database handle to use. Also some small cleanups.
To test:
* In a plack environment,
* Tools -> stage MARC records for import
* Use a reasonable size file (but not too big as it all goes into RAM -
I made one about 40MB.)
* Make sure that it works, and that the progress bars progress.
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Tested with a 55M file, I reproduced the error and I confirm this patch
fixes it.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
The MARC Modification Templates system gives Koha users
the power to make alterations to MARC records automatically
while staging MARC records for import.
This tool is useful for altering MARC records from
various venders work with your MARC framework.
The system essentially allows one to create a basic script
using actions to Copy, Move, Add, Update and Delete fields.
Each action can also have an optional condition to check
the value or existance of another field.
The Copy & Move actions also support Regular Expressions,
which can be used to automatically modify field values during the
copy/move. An example would be to strip out the '$' character
in field 020$c.
Furthermore, the value for an update can include variables
that change each time the template is used. Currently,
the system supports two variables, __BRANCHCODE__ which
is replaced with the branchcode of the library currently
using the template, and __CURRENTDATE__ which is replaced
with the current date in ISO format ( YYYY-MM-DD ).
At its simplist, it can perform functions such as:
Copy field 092$a to 952$c
At its most complex it can run actions like:
Copy field 020$c to 020$c using RegEx s/\$// if 020$c equals RegEx m/^\$/
Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Expose authority import functionality to the command line import
scripts, and rename them from commit_biblios_file.pl and
stage_biblios_file.pl to commit_file.pl and stage_file.pl.
To test (note that these instructions assume you have a MARC21
installation and are using the provided sample file):
1. Find a file of authorities (a sample file with MARC21 authorities
is attached to bug 7475) and download it to your server
2. Stage the file using the following command (replace <filename> with
the name of the file you saved in step 1):
> misc/stage_file.pl --file <filename> --authorities
3. Note the batch number the script assigns to your batch
4. Commit the records using the following command (replace <batchnumber>
with the batch number you made note of in step 3):
> misc/commit_file.pl --batch-number <batchnumber>
5. Index the authorities Zebraqueue (or wait)
6. Confirm that the new authorities appear.
7. Create a matching rule with the following settings:
Code: AUTHTEST
Description: Personal name main entry
Match threshold: 999
Record type: Authority record
Search index: Heading-main
Score: 1000
Tag: 100
Subfields: a
Offset: 0
Length: 0
(note the ID of this matching rule)
8. Stage the authority file again, this time using the following
command:
> misc/stage_file.pl --file <filename> --authorities \
--match <matchingrule>
7. Revert the import with the following command:
> misc/commit_file.pl --batch-number <batchnumber> --revert
8. Index the authorities Zebraqueue (or wait)
9. Confirm that the records have been removed
10. Import an authority record with the Stage MARC/Manage staged MARC
tools in exactly the way you would for a bibliographic record,
but choose "Authority" instead of "Bibliographic" for the record
type.
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>
Testing plan delivers as it should.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
* Add the code necessary to handle authorities with matching rules and
import batches.
* Update all the scripts that use the matcher and import batch code
to use the new API.
* Add authority records to the matching rules interface in the staff
client.
http://bugs.koha-community.org/show_bug.cgi?id=2060
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
Rather than having options for MARC21 and UNIMARC in the "Character encoding"
dropdown, the user should be able to select the appropriate character encoding.
The default retains the current behavior, which is to allow the system to guess
which character encoding is in use. However, it should be noticed that this is
almost always wrong for non-UTF8 records with non-ASCII characters. Specifying
a character set is much more reliable if you're not using UTF-8.
Rebased to use Template::Toolkit instead of HTML::Template::Pro.
Signed-off-by: Jared Camins-Esakov <jcamins@bywatersolutions.com>
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
This patch is writted by Paul Poulain.
It auto-select the marcflavour from syspref in the stage marc import.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Enhanced the ability of catalogers to specify how
bib and item records should be added, replaced, or
ignored during a staging import.
When an import batch of bib records is staged and commit,
the user can now explicitly specify what should occur
when an incoming bib record has, or does not have, a match
with a record already in the database. The options are:
if match found (overlay_action):
create_new (just add the incoming record)
replace (replace the matched record with the incoming one)
use_template (option not implemented)
ignore (do nothing with the incoming bib; however, the
items attached to it may still be processed
based on the item action)
if no match is found (nomatch_action):
create_new (just add the incoming record)
ignore (do nothing with the incoming bib; in this
case, any items attached to it will be
ignored since there will be nothing to
attach them to)
The following options for handling items embedded in the
bib record are now available:
always_add (add the items to the new or replaced bib)
add_only_if_match (add the items only if the incoming bib
matches an existing bib)
add_only_if_add (add the items only if the incoming bib
does *not* match an existing bib)
ignore (ignore the items entirely)
With these changes, it is now possible to support the following use cases:
[1] A library joining an existing Koha database wishes to add their
items to existing bib records if they match, but does not want
to overlay the bib records themselves.
[2] A library wants to load a file of records, but only handle
the new ones, not ones that are already in the database.
[3] A library wants to load a file of records, but only
handle the ones that match existing records (e.g., if
the records are coming back from an authority control vendor).
Documentation changes:
* See description above; also, screenshots of the 'stage MARC records
for import' and 'manage staged MARC records' should be updated.
Test cases:
* Added test cases to exercise staging and committing import batches.
UI changes:
* The pages for staging and managing import batches now have
controls for setting the overlay action, action if no match,
and item action separately.
* in the manage import batch tool, user is notified when they
change overlay action, no-match action, and item action
* HTML for manage import batch tool now uses fieldsets
Database changes (DB rev 076):
* added import_batches.item_action
* added import_batches.nomatch_action
* added 'ignore' as a valid value for import_batches.overlay_action
* added 'ignored' as a valid value for import_records.status
* added 'status' as a valid value for import_items.status
API changes:
* new accessor routines for C4::ImportBatch
GetImportBatchNoMatchAction
SetImportBatchNoMatchAction
GetImportBatchItemAction
SetImportBatchItemAction
* new internal functions for C4::ImportBatch to
determine how a given bib and item are to be
processed, based on overlay_action, nomatch_action,
and item_action:
_get_commit_action
_get_revert_action
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Updated all scripts appearing on the tools page
to respect a granular permission defined for
each of them.
The tools menu and home page have been changed so
that only the specific tools that a user has
access are displayed. This is simple, but depending on
the module and circumstance, it may be better to
display functions that the user has does not have
access to, but disable the links and do some sort
of visual styling to indicate that a function exists
but requires additional privileges to access.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Staging job now gets moved to background so that
it does not get killed if it takes longer
than the Apache timeout.
Added AJAX monitoring of job status.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* Added C4::BackgroundJob to monitor tracking of
background jobs
* start modifying stage-marc-import to use it
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* Introduced C4::UploadedFile to handle management and
progress tracking of uploaded files.
* Modified stage-marc-import.pl to handle new upload
mechanism
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Enhancement to store the matching rule associated with an
import batch and to allow the current matching rule in
effect to be changed and the duplicate detection redone.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Revamps the import options on the tools menu to have two parts:
[1] Staging (load file into reservoir)
[2] Managing (review the list of staged batches, then
choose to commit or undo a given batch.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
