This patch adds a missed parameter (profile_id) in stage-marc-import.pl and also adds a missing import in the background job for staging marc files. This means that the profile used when importing can now be shown in the import batches table and also in batch details
Test plan:
1) Stage a record for import and make sure to save the import profile you use
2) Import a record using that profile
3) Navigate to Cataloging > Manage staged MARC records
4) In the Profile column, no value will be shown for the staged record
5) Click on the batch, the profile will be missing in this screen as well
6) Apply patch
7) Restart_all
8) Repeat steps 2-5, the profile name should be visible in both places
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Emmanuel Bétemps <e.betemps@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 38fc85ade9)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
We lost the display of the "Add staged files to basket" link when create
a new order from a new file
Test plan:
Add order to basket from a new file, select a file and import
See the detail of the job and notice the "Add staged files to basket",
click and confirm that it works as expected.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch modifies instances in the code which try to get cookies
using:
my %cookies = parse CGI::Cookie($cookie);
In some cases the relevant lines can be removed because they are unused.
In others it can be replaced with:
my %cookies = CGI::Cookie->fetch();
To test, apply the patch and restart_all. Test the following pages to
confirm they load without errors:
- Circulation -> Offline circulation file upload
- Circulation -> Offline circulation -> Add to queue
- Tools -> Batch item modification
- Tools -> Stage MARC for import
- Tools -> Staged MARC management
- Tools -> Batch patron modification
- Tools -> Upload local cover image
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
There is a "debug" parameter we are passing from the controller scripts
to C4::Auth::get_template_and_user, but it's not actually used!
Test plan:
Confirm the assumption
Review the changes from this patch
Generated with:
perl -p -i -e 's#\s*debug\s*=\>\s*(0|1),?\s*##gms' **/*.pl
git checkout misc/devel/update_dbix_class_files.pl # Wrong catch
+ Manual fix in acqui/neworderempty.pl
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Like it was done in Bug 18806.
Test plan :
Use stage-marc-import.pl with and without patch, with a matcher to find
duplicates
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We need to close STDERR when forking stage-marc-import.pl,
or else the CGI session with Apache httpd does not properly
finish. This leads to unexpected behaviour across different httpd
versions, operating systems, etc.
This patch closes the STDERR file handle when forking a child
process to do MARC imports, and it re-opens STDERR to a log file
in the logdir directory to catch any import errors.
Test plan:
1. Apply the patch
2. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl
3. Upload a MARC file with a large number of records (e.g. 30,000 records)
4. Open F12 dev tools
5. Click on "Network" tab
6. Clear all existing network logs
7. Click "Stage for import"
8. After ~30 seconds, the request to stage-marc-import.pl should return a 200 code
9. Immediately, calls to background-job-progress.pl should start, and the "Job progress"
bar should update at a maximum rate of every .5 seconds
(or more realistically 1-2 seconds)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the logic and the needed UI elements to be able to pre-load an import profile. It also displays which profile was used to stage an import in staged import manager.
To test:
1. Apply all patches
2. Updatedatabase
3. Go to Stage MARC records for Import tool in admin, and upload a file with MARC records.
CHECK => after uploading, there is a fieldset with the legend “Profile settings”
=> inside the fieldset there is a select labeled “Pre fill values with profile”. The only value it has is “Do not use profile”.
4. Change some settings, and set “profile 1” as profile name and click on “Add profile”
SUCCESS => The select now has the new profile selected
5. Change profile select to “Do not use profile”
SUCCESS => Default values are now displayed in the form
6. Reload the page and upload the file again
SUCCESS => the select still has the profile recently added
7. Select the profile, change some parameter in the form and set the profile name to “profile 2”, and click add profile
SUCCESS => there are two profiles now, and if you toggle between them, the parameter changes
8. Select profile 1, change one parameter and click on update profile
SUCCESS => if you toggle that profile with the other, the new parameter of the value is shown when you select profile 1
9. Select profile 2, change some parameter and click Add profile (leaving the name as profile 2)
SUCCESS => the page complains there is another profile with the same name, and asks if you want to replace it.
10. Click on accept
SUCCESS => profile 2 now has the new value in the parameter
11. Select profile 2 and change the name to profile 1
SUCCESS => the page complains there is another profile with that name, and asks if you want to replace it
12. Click on accept
SUCCESS => in profile select there is only one profile called profile 1 that has the values of profile 2
13. Select profile 1 and click remove profile
SUCCESS => there is no profile in profile select.
14. Create a profile and click on “Stage for import”
15. Go to Staged MARC management page
SUCCESS => Improt should have the name of the profile in profile column, and when you click on the file name, there should be the name of the profile in the details.
16. prove t/db_dependent/ImportBatch.t t/db_dependent/api/v1/import_batch_profiles.t
17. Sign off
Signed-off-by: Abbey Holt <aholt@dubuque.lib.ia.us>
Signed-off-by: Abbey Holt <aholt@dubuque.lib.ia.us>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
It defaults to 0 in get_template_and_user
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Owen Leonard 2018-03-16 10:47:47 UTC :
<<
I don't think the system preference adds any security. There are already multiple permissions required for working with plugins:
- Configure plugins
- Manage plugins ( install / uninstall )
- Use report plugins
- Use tool plugins
And even with those permissions your server must be configured to allow the use of plugins.
>>
Test plan :
1) Install kitchen sink plugin https://github.com/bywatersolutions/koha-plugin-kitchen-sink
2) Run misc/devel/install_plugins.pl
3) Set config enable_plugins=1
4) Check all parts of the plugin are working
5) Set config enable_plugins=0
6) Check all parts of the plugin are disabled
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a workflow for adding to a basket from a file. It
removes the need to stage the file before going to the basket and allows
you to stage the file and return to the basket
To test:
1 - Have basket in acquissitions
2 - Click add to basket
3 - Note new option to 'Add from new file'
4 - Click it
5 - You will be sent to the record import tool
6 - Stage your file
7 - There is a link after staging 'Add records to basket'
8 - Click it
9 - Your items should be added
Signed-off-by: Sonia <sonia.bouis@univ-lyon3.fr>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Test plan:
Check that the following files don't contain use strict; use warnings;
and have use Modern::Perl instead.
background-job-progress.pl
batchMod.pl
copy-holidays.pl
exceptionHolidays.pl
holidays.pl
import_borrowers.pl
koha-news.pl
letter.pl
manage-marc-import.pl
newHolidays.pl
overduerules.pl
quotes-upload.pl
quotes.pl
quotes/quotes-upload_ajax.pl
quotes/quotes_ajax.pl
scheduler.pl
stage-marc-import.pl
upload-cover-image.pl
Also the credits have been added to newHolidays.pl
Signed-off-by: Jon Knight <J.P.Knight@lboro.ac.uk>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Template says ISO2709 but script wants MARC or MARCXML..
Slipped in with bug 19049.
Trivial but critical fix.
Test plan:
Import a MARC file.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I picked this fix because I think MARC is always ambiguous. ISO2709
clearly denotes the serialization format used.
Signed-off-by: Liz Rea <liz@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
As requested by QA:
[1] Mock_config enable_plugins in the test.
[2] Fallback to MARC when format is empty. Remove die statement.
Added:
[3] Remove $marc. This variable got obsolete during development.
[4] Add test on $input_file and $plugin_class. Test $text before calling
Handler or processing $text. No need to split undef if somehow Handler
returned undef, etc. If the routine returns an empty arrayref,
stage-marc-import will do fine.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Bug 12412 added the use of to_marc plugins allowing arbitrary file formats
in stage-marc-import (as long as the plugins can handle them). The feature
was not very visible in the code, and when bug 10407 added the marcxml
format, it made some changes that broke the use of to_marc.
This patch restores the functionality by:
[1] Adding a sub RecordsFromMarcPlugin to ImportBatch.pm, specifically
addressing the conversion from arbitrary formats to MARC::Record.
The original to_marc interface is used: pass it the file contents,
and it returns a string consisting of a number of MARC blobs separated
by \x1D.
Consequently, the call of to_marc is removed from routine
BatchStageMarcRecords where it did not belong. The to_marc_plugin
parameter is removed and two calls are adjusted accordingly.
[2] Instead of a separate combo with plugins, the format combo contains
MARC, MARCXML and optionally some plugin formats.
[3] The code in stage-marc-import.pl now clearly shows the three main
format types: MARC, MARCXML or plugin based.
Note: This patch restores more or less the situation after bug 12412, but
I would actually recommend to have the to_marc plugins return MARC::Record
objects instead of large text strings. In the second example I added a
to_marc plugin that actually converts MARC record objects to string format,
while RecordsFromMarcPlugin reconverts them to MARC::Records.
Test plan:
See second patch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The get routine actually returns records from uploaded_files. It should be
possible to replace its calls by direct calls of Koha::UploadedFiles.
This patch is the crux of this patch set. It deals with all scripts that
use Koha::Upload.
In the process we do:
[1] Add a file_handle method to Koha::UploadedFile. This was previously
arranged via the fh parameter of get.
[2] Add a full_path method to UploadedFile. Previously returned in the
path hash key of get. (Name is replaced by filename.)
[3] Add a search_term method too (implementing get({ term => .. }).
This logic came from _lookup.
[4] Add a keep_file parameter to delete method. Only used in test now.
Test plan:
[1] Run t/db_dependent/Upload.t
[2] Go to Tools/Upload. Add an upload, download and delete.
[3] Add another public upload , search for it.
Use the hashvalue to download via opac with URL:
cgi-bin/koha/opac-retrieve-file.pl?id=[hashvalue]
[4] Go to Tools/Stage MARC for import. Import a marc file.
[5] Go to Tools/Upload local cover image. Import an image file.
Enable OPACLocalCoverImages to see result.
[6] Test uploading a offline circulation file:
Enable AllowOfflineCirculation, and create a koc file (plain text):
Line1: Version=1.0\tA=1\tB=2
Line2: 2016-11-23 16:00:00 345\treturn\t[barcode]
Note: Replace tabs and barcode. The number of tabs is essential!
Checkout the item with your barcode.
Go to Circulation/Offline circulation file upload.
Upload and click Apply directly.
Checkout again. Repeat Offline circulation file upload.
Now click Add to offline circulation queue.
[7] Connect the upload plugin to field 856$u.
Enable HTML5MediaEnabled.
Upload a webm file via the plugin. Click Choose to save the URL,
and put 'video/webm' into 856$q. Save the biblio record.
Check if you see the media tab with player on staff detail.
(See also: Bug 17673 about empty OPACBaseURL.)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
It would be helpful if we could find a plugin based on some metadata
tag as returned by the plugin. This extends the use of GetPlugins that
already supports searching on method.
GetPlugins is used in: admin/edi_accounts.pl, plugins/plugins-home.pl and
tools/stage-marc-import.pl. The changes in these three scripts are
minimal and just related to parameter passing.
Test t/db_dependent/Plugins.t includes another test for GetPlugins. In this
regard a metadata tag has been added to t/Koha/Plugins/Test.pm.
NOTE: This adjustment will also be used in a redesign for bug 15545.
Test plan:
Run t/db_dependent/Plugins.t.
Enable pref UseKohaPlugins and config var enable_plugins.
Go to plugins-home.pl. Verify that it still lists your plugins.
Bonus: Check edi_accounts or stage-marc-import.pl if you have a working
plugin for that.
Signed-off-by: Liz Rea <liz@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch makes the following changes:
[1] Based on the groundwork of the former patch, add call to
RecordsFromMARCXMLFile in stage-marc-import. Use format param.
[2] Add format to the template. Use file extension to determine.
If you use .xml or .marcxml as extension, MARCXML is selected.
[3] In stage-marc-import.tt mark UTF-8 encoding as UTF-8 not as utf8.
[4] BatchStageMarcRecords: do not call plugin if you have no records.
[5] RecordsFromISO2709File: also return errors in an array.
[6] In misc/stage_file.pl also use UTF-8. Handling of errors from [5].
Test plan:
[1] Import an empty file as MARC or MARCXML (with Tools/Stage..import).
[2] Import an non-empty file with invalid contents as MARC or MARCXML.
[3] Export a few records with Tools/Export as MARC and MARCXML.
[4] Import these two files. Check selected format versus file extension.
[5] Import a MARCXML file with misc/stage_file.pl.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
After the staging job finishes, the child tries to print results to
STDOUT (which is closed). This creates warnings like:
stage-marc-import.pl: binmode() on closed filehandle STDOUT at C4/Templates.pm line 120.
stage-marc-import.pl: Filehandle STDOUT reopened as FH only for input at /usr/lib/perl5/Template/Provider.pm line 964.
Resolving it is simple: the child should exit after marking the job
as finished.
Test plan:
Import a marc file.
Observe that you do no longer have such warnings.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Patch from Olli, manual rebase by Marcel (July 7, 2016).
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Needs follow-up. Test plan in the third patch.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Since I don't manage to make SCRIPT_NAME works directly with RequestHeader and/or
ReverseProxyPath, the easier way it to fix all the different occurrences.
Test plan:
On the import patrons page, confirm that the "Import" button does not redirect to a 'Not found' page.
You should stay on the import patrons page.
Note that if this change works, all others should work too.
Have a look at the diff of this patch and confirm there is no typo.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Tested in patron import and item batch modification
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch makes the following changes to achieve that:
[1] Use Upload.pm in stage-marc-import.pl, upload-cover-image.pl,
offline_circ/process_koc.pl and enqueue_koc.pl.
[2] A new file-upload.js replaces file-upload.inc in the associated template.
We now use ajax to get progress figures instead of launching perl script
upload-file-progress.
The js changes now also allow for aborting a file upload.
[3] Adds a tools/upload script and template. It allows to upload multiple
files at once.
[4] Makes upload-file return error messages in JSON. For a multiple upload,
we could have some files with errors and others without errors.
The upload is now marked as Failed only if there was no upload at all.
[5] The upload plugin is converted to use tools/upload with plugin param.
Deleting an upload is now presented via the search results form.
NOTE: In editing the process_koc.tt I noticed that the form enqueuefile was
hidden and no longer used (with associated code in process_koc.pl). When a
file has been uploaded, I display the form again (with the Apply directly
button). The code still works.
NOTE: We fix an error in upload-file from one of the patches of bug 6874.
The userid of the Koha admin user is passed to haspermission, but we
should pick the userid from the session.
NOTE: Bug 14686 will add a specific permission for tools/upload.pl, and
will add the tools/upload script to the Tools menu.
For now, you need edit_catalogue to start upload.pl and you will
additionally need a permission like upload_local_cover_images
to successfully upload a new file.
Test plan:
[1] Upload a marc file in stage-marc-import. (This is temp storage.)
[2] Check new entry in table uploaded_files. Look for the file in your
temporary directory (/tmp ?), subfolder koha_upload.
Bonus: Remove permissions on this subfolder. Retry, check error and
restore permissions again.
[3] Upload another (larger) file and abort the upload. Check table and
directory again. You should have a partial file, but no record.
[4] Verify that Stage for import still works as expected.
[5] Test Upload local cover image. (Enable OPACLocalCoverImages.) You can
test an individual image or a zip file including images and a file
called datalink.txt (with lines biblionumber,filename).
[6] Test uploading a offline circulation file:
Enable AllowOfflineCirculation, and create a koc file (plain text):
Line1: Version=1.0\tA=1\tB=2
Line2: 2015-08-06 08:00:00 345\treturn\t[barcode]
Note: Replace tabs and barcode. The number of tabs is essential!
Checkout the item with your barcode.
Go to Offline circulation file upload. Upload and click Apply directly.
Checkout again. Upload again, click Add to offline circulation queue.
[7] Upload three files via tools/upload.pl with a category and marked as
public. Check the results in the table.
Verify that you can download the file in OPAC without being logged in.
[8] Pick one new file and one of the files of step 7. Upload them in the
same category. One upload should succeed. Check for reported error.
[9] Connect upload.pl to field 856$u.
Goto Cataloguing editor.
In an empty 856$u, click the tag editor. Upload a file and click Choose.
Save the record. Open the record in the OPAC and click the link.
Copy this link to your clipboard for next step.
[A] Go back to editor. Click the tag editor on the same 856 field.
Choose for Delete.
Open the link in your clipboard again. Error message?
[B] Check the process of upload, search, download and delete of an upload
with some diacritical characters in the filename. (Bonus points for
adding special chars in the category code.)
Note: You can add categories via authorized values, UPLOAD key.
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Bug 14321: [QA Follow-up] Perltidy upload.pl
Run perltidy -pro=xt/perltidyrc on tools/upload.pl.
No other changes.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
On importing authority records, the items related string should not be
displayed.
Test plan:
1/ Import an authority record
2/ Confirm the item related string are not displayed.
3/ Before this script, the record_type param was lost just after
importing the records. Now you should see
"Processing authority records" of "Processing bibliographic records" in
the "MARC staging results"
4/ Revert the import and check there is not item related strings.
No behavior should exist on importing bibliographic records.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Bug 12412 introduced a way to use plugins to generate MARC records
out of any arbitrary format. But failed to properly check for
a valid configuration (plugins_enabled entry on koha-conf.xml, and
even the UseKohaPlugins syspref).
This patch makes tools/stage-marc-import.pl test for the required
configuration before attempting to create the Koha::Plugins object.
To test:
1/ Make sure you don't have a plugins_enabled entry on your koha-conf.xml file
2/ On master, browse to tools/stage-marc-import.pl
=> FAIL: You get a nasty "Can't call method "GetPlugins"..." error
3/ Apply the patch and reload
=> SUCCESS: Stage MARC for import page renders correctly
Repeat with plugins_enabled == 0 and == 1. Also UseKohaPlugins enabled/disabled.
It should work as expected.
Regards
Tomas
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works as described, no koha-qa errors
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Many libraries would like to be able to import various types of files as
MARC records ( citations, csv files, etc ). We can add a new function to
the plugins system to allow that kind of behavior at a very custom
level.
Test Plan:
1) Ensure you have plugins enabled and configured correctly
2) Installed the attached version 2.00 of the Kitchen Sink plugin
3) Download the attached text file
4) Browse to "Stage MARC records for import"
5) Select the downloaded text file for staging
6) After uploading, you should see a new area "Transform file to MARC:",
select "Example Kitchen-Sink Plugin" from the pulldown menu
7) Click 'Stage for import"
8) Click 'Manage staged records"
9) You should now see two new MARC records!
Signed-off-by: Aleisha <aleishaamohia@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described - interesting new feature.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
http://bugs.koha-community.org/show_bug.cgi?id=9987
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
It appears that bug 8970 has introduced a bug where a MARC file uploaded
for staging fails to import *unless* a matching rule is selected.
Test Plan:
1) Apply this patch
2) Stage and import a MARC record *without* choosing a matching rule
3) Note the staging works as expected
Signed-off-by: Cindy Ames <cmurdock@ccfls.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Since we switched to Template Toolkit we don't need to stick with the
sufix we used for HTML::Template::Pro.
This patch changes the occurences of '.tmpl' in favour of '.tt'.
To test:
- Apply the patch
- Install koha, and verify that every page can be accesed
Regards
To+
P.S. a followup will remove the glue code.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
There were database handles being shared between a parent and a child
process, which is a big no-no, and was leading to crazy crashes. Fine
under CGI, but not in a persistent environment. This causes the child to
make a new database handle to use. Also some small cleanups.
To test:
* In a plack environment,
* Tools -> stage MARC records for import
* Use a reasonable size file (but not too big as it all goes into RAM -
I made one about 40MB.)
* Make sure that it works, and that the progress bars progress.
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Tested with a 55M file, I reproduced the error and I confirm this patch
fixes it.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
The MARC Modification Templates system gives Koha users
the power to make alterations to MARC records automatically
while staging MARC records for import.
This tool is useful for altering MARC records from
various venders work with your MARC framework.
The system essentially allows one to create a basic script
using actions to Copy, Move, Add, Update and Delete fields.
Each action can also have an optional condition to check
the value or existance of another field.
The Copy & Move actions also support Regular Expressions,
which can be used to automatically modify field values during the
copy/move. An example would be to strip out the '$' character
in field 020$c.
Furthermore, the value for an update can include variables
that change each time the template is used. Currently,
the system supports two variables, __BRANCHCODE__ which
is replaced with the branchcode of the library currently
using the template, and __CURRENTDATE__ which is replaced
with the current date in ISO format ( YYYY-MM-DD ).
At its simplist, it can perform functions such as:
Copy field 092$a to 952$c
At its most complex it can run actions like:
Copy field 020$c to 020$c using RegEx s/\$// if 020$c equals RegEx m/^\$/
Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Expose authority import functionality to the command line import
scripts, and rename them from commit_biblios_file.pl and
stage_biblios_file.pl to commit_file.pl and stage_file.pl.
To test (note that these instructions assume you have a MARC21
installation and are using the provided sample file):
1. Find a file of authorities (a sample file with MARC21 authorities
is attached to bug 7475) and download it to your server
2. Stage the file using the following command (replace <filename> with
the name of the file you saved in step 1):
> misc/stage_file.pl --file <filename> --authorities
3. Note the batch number the script assigns to your batch
4. Commit the records using the following command (replace <batchnumber>
with the batch number you made note of in step 3):
> misc/commit_file.pl --batch-number <batchnumber>
5. Index the authorities Zebraqueue (or wait)
6. Confirm that the new authorities appear.
7. Create a matching rule with the following settings:
Code: AUTHTEST
Description: Personal name main entry
Match threshold: 999
Record type: Authority record
Search index: Heading-main
Score: 1000
Tag: 100
Subfields: a
Offset: 0
Length: 0
(note the ID of this matching rule)
8. Stage the authority file again, this time using the following
command:
> misc/stage_file.pl --file <filename> --authorities \
--match <matchingrule>
7. Revert the import with the following command:
> misc/commit_file.pl --batch-number <batchnumber> --revert
8. Index the authorities Zebraqueue (or wait)
9. Confirm that the records have been removed
10. Import an authority record with the Stage MARC/Manage staged MARC
tools in exactly the way you would for a bibliographic record,
but choose "Authority" instead of "Bibliographic" for the record
type.
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>
Testing plan delivers as it should.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
* Add the code necessary to handle authorities with matching rules and
import batches.
* Update all the scripts that use the matcher and import batch code
to use the new API.
* Add authority records to the matching rules interface in the staff
client.
http://bugs.koha-community.org/show_bug.cgi?id=2060
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
Rather than having options for MARC21 and UNIMARC in the "Character encoding"
dropdown, the user should be able to select the appropriate character encoding.
The default retains the current behavior, which is to allow the system to guess
which character encoding is in use. However, it should be noticed that this is
almost always wrong for non-UTF8 records with non-ASCII characters. Specifying
a character set is much more reliable if you're not using UTF-8.
Rebased to use Template::Toolkit instead of HTML::Template::Pro.
Signed-off-by: Jared Camins-Esakov <jcamins@bywatersolutions.com>
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
This patch is writted by Paul Poulain.
It auto-select the marcflavour from syspref in the stage marc import.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Enhanced the ability of catalogers to specify how
bib and item records should be added, replaced, or
ignored during a staging import.
When an import batch of bib records is staged and commit,
the user can now explicitly specify what should occur
when an incoming bib record has, or does not have, a match
with a record already in the database. The options are:
if match found (overlay_action):
create_new (just add the incoming record)
replace (replace the matched record with the incoming one)
use_template (option not implemented)
ignore (do nothing with the incoming bib; however, the
items attached to it may still be processed
based on the item action)
if no match is found (nomatch_action):
create_new (just add the incoming record)
ignore (do nothing with the incoming bib; in this
case, any items attached to it will be
ignored since there will be nothing to
attach them to)
The following options for handling items embedded in the
bib record are now available:
always_add (add the items to the new or replaced bib)
add_only_if_match (add the items only if the incoming bib
matches an existing bib)
add_only_if_add (add the items only if the incoming bib
does *not* match an existing bib)
ignore (ignore the items entirely)
With these changes, it is now possible to support the following use cases:
[1] A library joining an existing Koha database wishes to add their
items to existing bib records if they match, but does not want
to overlay the bib records themselves.
[2] A library wants to load a file of records, but only handle
the new ones, not ones that are already in the database.
[3] A library wants to load a file of records, but only
handle the ones that match existing records (e.g., if
the records are coming back from an authority control vendor).
Documentation changes:
* See description above; also, screenshots of the 'stage MARC records
for import' and 'manage staged MARC records' should be updated.
Test cases:
* Added test cases to exercise staging and committing import batches.
UI changes:
* The pages for staging and managing import batches now have
controls for setting the overlay action, action if no match,
and item action separately.
* in the manage import batch tool, user is notified when they
change overlay action, no-match action, and item action
* HTML for manage import batch tool now uses fieldsets
Database changes (DB rev 076):
* added import_batches.item_action
* added import_batches.nomatch_action
* added 'ignore' as a valid value for import_batches.overlay_action
* added 'ignored' as a valid value for import_records.status
* added 'status' as a valid value for import_items.status
API changes:
* new accessor routines for C4::ImportBatch
GetImportBatchNoMatchAction
SetImportBatchNoMatchAction
GetImportBatchItemAction
SetImportBatchItemAction
* new internal functions for C4::ImportBatch to
determine how a given bib and item are to be
processed, based on overlay_action, nomatch_action,
and item_action:
_get_commit_action
_get_revert_action
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Updated all scripts appearing on the tools page
to respect a granular permission defined for
each of them.
The tools menu and home page have been changed so
that only the specific tools that a user has
access are displayed. This is simple, but depending on
the module and circumstance, it may be better to
display functions that the user has does not have
access to, but disable the links and do some sort
of visual styling to indicate that a function exists
but requires additional privileges to access.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
