This patch removes the now unused: _update_import_record_marc
Additionally, as items are already present in import biblio we no longer need to embed
them, so that parameter is removed and the option removed from the sub and pod and everywhere
it was used
In all cases, we were embedding, so we don't need a way to get without
Tests updated
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
MARC::Record and MARC::File::* modules sometimes use the position 09 of
the leader to detect encoding. A blank character means 'MARC-8' while an
'a' means 'UTF-8'.
In a UNIMARC authority this position is used to store the authority type
(see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]).
In this case, 'a' means 'Personal Name'.
The result is that the import will succeed for a Personal Name
authority, but it will fail for all other authority types.
Steps to reproduce:
0. Be sure to have a Koha UNIMARC instance.
1. Download the MARCXML for "Honoré de Balzac"
curl -o balzac.marcxml https://www.idref.fr/02670305X.xml
2. Verify that it's encoded in UTF-8
file balzac.marcxml
(should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode
text")
3. Go to Tools » Stage MARC for import and import balzac.marcxml with
the following settings:
Record type: Authority
Character encoding: UTF-8
Format: MARCXML
Do not touch the other settings
4. Once imported, go to the staged MARC management tool and find your
batch. Click on the authority title "Balzac Honoré de 1799-1850" to
show the MARC inside a modal window. There should be no encoding
issue.
5. Write down the imported record id (the number in column '#') and go
to the MARC authority editor. Replace all URL parameters by
'breedingid=THE_ID_YOU_WROTE_DOWN'
The URL should look like this:
/cgi-bin/koha/authorities/authorities.pl?breedingid=198
You should see no encoding issues. Do not save the record.
6. Import the batch into the catalog. Verify that the authority record
has no encoding issue.
7. Now download the MARCXML for "Athènes (Grèce)"
curl -o athènes.marcxml https://www.idref.fr/027290530.xml
8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you
should see encoding issues and that the position 9 of the leader was
rewritten from 'c' to 'a'. Strangely, importing this batch fix the
encoding issue, but we still lose the information in position 09 of
the leader
This patch makes use of the MARCXML representation of the record instead
of the ISO2709 representation, because, unlike
MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to
pass directly the encoding and the format, which prevents data to be
double encoded when position 09 of the leader is different that 'a'
Test plan:
- Follow the "steps to reproduce" above and verify that you have no
encoding issues.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
BatchCommitItems is only being used within this module and isn't
mentioned in EXPORT_OK. This patch simply renames it to
_batchCommitItems to take the _ standard for private functions and also
adds a little hint to the POD of the function to clarify that the caller
must trigger a re-index.
JK: Amended patch to rename also the function in t/db_dependent/ImportBatch.t
and fix typo "commiting" => "commiting"
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit: minor typo fix
This patchset adds the display of all matches found during import to the import management screen
A staff member with the permission to manage batches will be able to select for any individual record which match, or none, should be used during import
To test:
1 - Import a batch of records or export existing records from your catalog
2 - Import the file (again) and select a matching rule that will find matches
3 - Note that you now have radio buttons allowing you to select a record, or none
4 - Test scenarios:
I - When 'Action if matching record found' is 'Ignore'
a - Imported record ignored if match is selected
b - 'Action if no match found' followed if no match is selected (Ignore matches)
II - When 'Action if matching record found' is 'Replace'
a - The chosen record is the one overlayed (you can edit the chosen record before importing to confirm)
b - 'Action if no match found' followed if no match is selected (Ignore matches)
III - When 'Action if matching record found' is 'Add incoming record'
a - Record is added regardless of matches
5 - Confirm 'Diff' 'View' links work as expected
6 - Confirm that after records are imported the radio buttons to choose are disabled
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Bug 22785: API files
Signed-off-by: Ben Daeuber <bdaeuber@cityoffargo.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Test plan:
Tests should return green
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended: POD line for $import_record_id.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Abbey Holt <aholt@dubuque.lib.ia.us>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
.pm must not have -x
.t must have -x
.pl must have -x
Test plan:
Apply only the first patch, run the tests and confirm that the failures
make sense
Apply this patch and confirm that the test now returns green
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Most of the time the tests are failing because the item is not created
correctly (missing biblio and/or biblioitem).
The usual error is:
t/db_dependent/selenium/regressions.t ..... 5/5 Can't call method "leader" on an undefined value at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch.pm line 534.
In this patch we are making sure $builder->build({ source => 'Item' })
is replace with $builder->build_sample_item
Test plan:
Turn on Elastic and confirm that all the tests pass!
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Owen Leonard 2018-03-16 10:47:47 UTC :
<<
I don't think the system preference adds any security. There are already multiple permissions required for working with plugins:
- Configure plugins
- Manage plugins ( install / uninstall )
- Use report plugins
- Use tool plugins
And even with those permissions your server must be configured to allow the use of plugins.
>>
Test plan :
1) Install kitchen sink plugin https://github.com/bywatersolutions/koha-plugin-kitchen-sink
2) Run misc/devel/install_plugins.pl
3) Set config enable_plugins=1
4) Check all parts of the plugin are working
5) Set config enable_plugins=0
6) Check all parts of the plugin are disabled
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch makes the tests enable the plugins that are required for
them. Otherwise unexpected failures can happen
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
As requested by QA:
[1] Mock_config enable_plugins in the test.
[2] Fallback to MARC when format is empty. Remove die statement.
Added:
[3] Remove $marc. This variable got obsolete during development.
[4] Add test on $input_file and $plugin_class. Test $text before calling
Handler or processing $text. No need to split undef if somehow Handler
returned undef, etc. If the routine returns an empty arrayref,
stage-marc-import will do fine.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We have to mock_config the pluginsdir before Plugins is loaded, and
we should pass an absolute path (not a relative one).
If you did not install the test to_marc plugin, this test would fail.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Verified patch is compatible with original KitchenSink to_marc plugin
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds a simple to_marc plugin in t/Koha/Plugin that is used
in the added subtest in ImportBatch.t.
Test plan:
[1] Run t/db_dependent/ImportBatch.t
[2] Copy the to_marc test plugin from t to your plugin directory.
Under Debian packages, you should do something like:
mkdir -p /var/lib/koha/master/plugins/Koha/Plugin/
cp [yourclone]/t/Koha/Plugin/MarcFieldValues.pm /var/lib/koha/master/plugins/Koha/Plugin/
[3] Check if you see this plugin on plugins/plugins-home.pl
[4] Create a text file with some fields like:
(Note: The plugin needs an empty line between both "records".)
100,a = Test Author 1
245,a = Title One
100,a = Author 2
245,a = Title Two
[5] Go to stage-marc-import.pl. Upload the created file. Select the plugin
in the format combo and proceed. Did you create two records ?
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test plan:
Run t/db_dependent/ImportBatch.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
To test:
1) Go to Tools -> Staged MARC Management and clean a file. If you have no files to clean, go to 'Stage MARC for import' and upload one to clean following the necessary steps.
2) Confirm that once the file has been cleaned, the Action column now shows a Delete button. Confirm this button only shows for cleaned files.
3) Click the Delete button.
4) Confirm that clicking Cancel exits the pop-up message and does not delete the file.
5) Confirm that clicking OK refreshes the list of staged records and the one you just deleted is no longer on it (has been deleted). You can confirm this by checking for the file in mysql (SELECT * FROM import_batches WHERE import_batch_id = X;)
6) Run prove -v t/db_dependent/ImportBatch.t (have written unit tests for CleanBatch and DeleteBatch)
Sponsored-by: Catalyst IT
Signed-off-by: Liz Rea <liz@catalyst.net.nz>
Catalyst sign off, so needs another one but YAY this is great.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Unit test added
http://bugs.koha-community.org/show_bug.cgi?id=11876
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
