Includes:
Bug 29697: (follow-up) Use flag embed_items
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
# Failed test 'Identify - earliestDatestamp in the right format'
# at t/db_dependent/OAI/Server.t line 182.
# Compared $data->{"Identify"}{"earliestDatestamp"}
# got : '2021-11-15T13:16:09Z'
# expect : '2014-05-07T13:36:23Z'
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch introduces a regression test for the fixed behavior.
I also changed a oneliner regex we had for stripping query parameters
because it was causing a warning with the tests base URL
(http://localhost) because it was generating an empty baseURL.
To test:
1. Apply the regression tests patch
2. Run:
$ kshell
k$ prove t/db_dependent/OAI/Server.t
=> FAIL: Tests fail, the response is incorrect!
3. Apply the bug's patch
4. Repeat 2
=> SUCCESS: Tests pass! Good response!
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
When the OAI-PMH provider was configured to include item information
in the returned records, deleting an item would cause the record to
be reported as deleted. The provider also did some useless checks when
determining the timestamp of a deleted record. These checks only had
a performance impact.
To test:
1) Create /var/lib/koha/kohadev/OAI.yaml with:
format:
marcxml:
metadataPrefix: marcxml
metadataNamespace: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim
schema: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd
include_items: 1
2) Set preference OAI-PMH to Enable
3) Set preference OAI-PMH:ConfFile to /var/lib/koha/kohadev/OAI.yaml
4) Delete an item from a record
5) View the page: http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml
6) Note the record is now listed as deleted
7) Run prove -v t/db_dependent/OAI/
Sponsored-by: The National Library of Finland
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
It has been missed on bug 17600.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Adjusted commit title.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds regression tests for the behavior.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Includes the following optimizations:
- Use next biblionumber instead of large offset in the queries.
- Use unions instead of subqueries
- Avoid fetching item timestamps when items are not included.
Test plan:
1. Without the patch, try harvesting a Koha database with (and without for good measure) `include_items: 1` in the OAI-PMH configuration file pointed to by preference OAI-PMH:ConfFile and take note of performance. For useful metrics the database must be large enough to not fit in InnoDB buffers or OS file cache.
2. Apply the patch.
3. Run tests: prove -v t/db_dependent/OAI
4. Try again the harvesting from step 1 and compare performance with step 1.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
From tht YAML pod:
"""
This module has been released to CPAN as YAML::Old, and soon YAML.pm will be changed to just be a frontend interface module for all the various Perl YAML implementation modules, including YAML::Old.
If you want robust and fast YAML processing using the normal Dump/Load API, please consider switching to YAML::XS. It is by far the best Perl module for YAML at this time. It requires that you have a C compiler, since it is written in C.
"""
See also
https://gitlab.com/koha-community/qa-test-tools/-/merge_requests/35
Test plan:
Try some place where YAML::XS is not used and confirm that it works
correctly
QA note: This patch removes some uses of YAML that were not useful
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test plan:
1. Run unit test:
sudo koha-shell <instancename>
prove t/db_dependent/OAI/Sets.t
2. All tests should pass
Sponsored-by: Catalyst IT
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
JD amended patch: Put tests inside a subtest, and perltidy
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Šťastný <martin.stastny@teamlibrary.cz>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test plan:
1. Check the OAI-PMH:AutoUpdateSets syspref is disabled
2. Run t/db_dependent/OAI/Sets.t tests and confirm tests pass
successfully
3. Enabled the OAI-PMH:AutoUpdateSets syspref
4. Repeat step 2 and confirm the tests fail
5. Apply this patch and restart plack: sudo koha-plack --restart
<instance>
6. Repeat step 2 and confirm the tests now pass
7. Disable the OAI-PMH:AutoUpdateSets syspref and repeat step 2 and
confirm tests still pass
Sponsored-By: Catalyst IT
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
.pm must not have -x
.t must have -x
.pl must have -x
Test plan:
Apply only the first patch, run the tests and confirm that the failures
make sense
Apply this patch and confirm that the test now returns green
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We should use Koha::DateUtils instead of Date::Time directly
This patch simplay replaces calls to now() with a call to dt_from_string()
which does effectively the same thing.
Probably reading the code and verifying changes is sufficient but...
To test:
1 - confirm the files all compile
2 - confirm all tests pass
3 - confirm Koha still works
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds two tests related to OAI-PMH sets. It creates a
dummy record with a repeated field, with different values. It then
tests if the record is caught by mappings that match the value in
the first field, and then if it is caught by mappings that match
the second field.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Currently, the rules used to create OAI sets are processed with the 'or'
boolean operator between each rule.
This patch allows to use 'or' or 'and' between the rules.
The evaluation of the rules is done according to the boolean operators
precedence: AND has a higher precedence than OR.
For example:
A and B or C and D
will be evaluated as follow:
(A and B) or (C and D)
Test plan:
- Apply the patch
- Apply the atomicupdate
- Prove t/db_dependant/OAI/AndSets.t
- Check that existing mappings still work
- Try modifying existing mappings
- Try creating new mappings
- Check that the boolean operators precedence is correctly taken into account
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch makes the code using Koha::Biblio::Metadata->marcflavour use
->schema instead for all interactions.
To test:
- Update the DB structure:
$ updatedatabase
- Update the schema files:
$ dbic
- Notice all the places in which biblio_metadata is used
$ cd kohaclone
$ git grep biblio_metadata
=> SUCCESS: They all use `schema` instead of marcflavour
- Notice all the places that use Koha::Biblio::Metadata:
$ git grep Koha::Biblio::Metadata
=> SUCCESS: They all use the schema attribute when they used to use
marcflavour
- Run all the modified tests and scripts
=> SUCCESS: We are all good
- Sign off :-D
Note: while this seems like a minor change, the places in which plain
SQL is used really require understanding the queries and how they are
used, because some query results might be passed to some other method
that in turn uses the marcflavour attribute. I of course took that into
account but errare humanum est :-D
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
TEST PLAN
---------
apply this test patch
prove t/db_dependent/OAI/Server.t
-- should fail
apply the other patch
prove t/db_dependent/OAI/Server.t
-- should pass
run koha qa test tools
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The problem is the calls to HTTP::OAI::Header, etc.
may reference gmtime which is continually changing by
the second. By forcing time to lock for all the tests,
except the last one, we can be assured things should
not fail.
TEST PLAN
---------
install libtest-mocktime-perl
apply the patch
restart_all
in kshell, prove t/db_dependent/OAI/Server.t
run koha qa test tools
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Since bug 17196, biblioitems.timestamp is not always updated after a
change in the MARC record.
Filtering should be based on biblio_metadata.timestamp instead.
Test plan:
1. prove t/db_dependent/OAI/Server.t
2. Verify that it SUCCEEDS
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Also fix tests for UNIMARC
Test plan:
1. prove t/db_dependent/OAI/Server.t
2. Verify that it FAILS
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Change parameters to a hashref.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Looks good to me.
Two calls in migration_tools/22_to_30 still in old style.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch replaces the TRUNCATE statement in ModOAISetsBiblios by a
DELETE statement. A truncate will cause an implicit commit and will
therefore commit the transaction started in the test script.
Also simplifying the module load in the test script.
Test plan:
Do not apply this patch and observe that biblio records are added to your
database by running t/db_dependent/OAI/Sets.t.
Apply this patch, run the test again and verify that it does no longer
add records to your biblio table.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This test may fail on slow servers, it compares the response date with
'now', but both can differ a bit.
https://jenkins.koha-community.org/job/Koha_Master_D8/198/consoleFull
Failed test 'ListMetadataFormats'
at t/db_dependent/OAI/Server.t line 150.
Structures begin differing at:
$got->{responseDate} = '2017-06-12T14:31:51Z'
$expected->{responseDate} = '2017-06-12T14:31:50Z'
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Fixed OAI-PMH Server tests to delete any existing issues before deleting biblios,
to delete oai_sets to avoid sets in the responses and to work with UNIMARC and
NORMARC as well as MARC21.
Signed-off-by: Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
- Fixed date handling to use UTC as specs require.
- Added support for second precision in time stamps.
- Added support for marc21 metadata prefix as recommended in the
guidelines (synonym for marcxml).
- Improved performance of database queries especially for large
collections.
- Unified functionality of ListRecords and ListIdentifiers to a common
base class.
- If items are included in the records, their timestamps are taken into
account everywhere so that whichever is the most recent (timestamp of
biblioitem or any of its items) is considered the record's timestamp.
- Fixed OAI.xslt to show correct record range.
- Incorporated extended tests from Bug 17493 and their tweaks from Bug 15108.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch makes t/db_dependent/OAI/Server.t test for the correct
schema and metadata namespace.
To test:
- Run
$ prove t/db_dependent/OAI/Server.t
=> FAIL: Tests fail due to differences in the output.
- Apply the patch
- Run
$ prove t/db_dependent/OAI/Server.t
=> SUCCESS: Tests now pass
- Sign off
Regards
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
It's possible to redirect the std output to a variable, it will avoid to
add this new dependency.
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
OK. + code factorisation with a function call.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Koha OAI server has been done in one unique .pl file because there
wasn't any object model or rules in the Koha project when it has been
coded. This patch modularized existing classes, putting each class in a
separate file in Koha::OAI::Server namespace. UT begining.
Add new dependency: Capture::Tiny
Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
OAI server moduralized succefully. Works for Debian Jessie and
Wheezy. Test pass successfully
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Testing C4::OAI::Sets in t/db_dependent/OAI/Sets.t
Adding &DelOAISetsBiblio in @EXPORT at the beginning
of OAI/Sets.pm, it was the only one missing
TEST PLAN
---------
1. Apply patch
2. prove -v t/db_dependent/OAI/Sets.t
-- All 145 tests should run successfully without any error or warning
3. Run koha qa test tools
-- No issues found.
Additionally, I comfirmed why DelOAISetsBiblio was added in
the list of exported functions.
$ grep "^sub " C4/OAI/Sets.pm | sort > check1
$ for i in `grep "\&[A-Z]" C4/OAI/Sets.pm | xargs `; do echo $i; done | sort > check2
Of course the outputs have a little extra around them, so
$ vi check1
$ vi check2
$ diff check1 check2
Lo and behold, it was that one function. Necessary for 100% coverage.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
