The goal is to be able to build a database handler (dbh) and to execute
queries without loading unnecessary stuff. This will be useful to reduce
memory usage of daemons that need to check the database
periodically
The patch provides a new method Koha::Database::dbh which returns a
database handler without loading the DBIx::Class schema. This method is
also used by DBIx::Class, so whether you use DBI or DBIx::Class, the
same method is used to initialize the connection.
The patch also moves some code in order to avoid loading C4::Context:
- C4::Context::timezone moves to Koha::Config
- C4::Context::db_scheme2dbi moves to Koha::Database
To measure memory usage I used the following commands:
* before the patch:
perl -MKoha::Database \
-E 'Koha::Database->schema->storage->dbh->do("select 1");' \
-E '$|=1; say $$; sleep 2' \
| while read pid; do ps -p $pid -o rss=; done
* after the patch:
perl -MKoha::Database \
-E 'Koha::Database->dbh->do("select 1");' \
-E '$|=1; say $$; sleep 2' \
| while read pid; do ps -p $pid -o rss=; done
It will give you the RSS (Resident Set Size) of the perl process in kB
What I get:
* before the patch: between 96.9MB and 97.2MB
* after the patch: between 17.8MB and 18.2MB
Note that if a timezone is configured (either from $KOHA_CONF or
TZ environment variable), Koha will load DateTime::Timezone to check if
it's valid, and it increases RSS to 36MB
Another interesting metric is the number of modules loaded:
* before the patch:
perl -MKoha::Database \
-E 'Koha::Database->schema->storage->dbh;' \
-E 'say scalar keys %INC'
Result: 567
* after the patch:
perl -MKoha::Database \
-E 'Koha::Database->dbh;' \
-E 'say scalar keys %INC'
Result: 51
Test plan:
1. Apply the patch & restart starman
2. Make sure Koha is still ok (ie. can access the database, does not
have encoding issues, ...)
3. Run the tests in t/Context.t, t/Koha/Config.t,
t/db_dependent/Koha/Database.t, t/timezones.t
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
There's no POD. This patch adds it. I wrote it as part of bug 28615 but
then it took another direction, so submitting here.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
So Debian 9's version of Test::MockModule doens't have ->redefine, and
Ubuntu 20.04's doesn't recognise qw(nostrict). So the only solution is
to just remove the keywords use completely and move back to using
->mock, as the rest of the codebase.
FIXME: using ->mock might be hiding some errors (like a method not being
defined/removed) and should be avoided. ->redefine will explode if the
method doesn't already exist, which is what we want, to catch this kind
of errors. That's why ->mock in strict mode is forbidden. We should try
packaging a newer Test::MockModule ourselves.
Tested on master-buster, master-stretch and master-focal.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In strict mode, ->mock is forbidden and ->redefine needs to be used
instead. I tested this on buster to see if it breaks something, but it
doesn't.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch introduces a new way to mock and test Koha::Logger.
As the POD says, it is used by calling
my $logger = t::lib::Mocks::Logger->new();
It then provides convenient methods for testing the logging itself per
log-level:
* warn_is
* warn_like
* debug_is
* debug_like
...
Methods for counting the logging activity and also for clearing the mock
buffer are provided as well. This is covered in the POD and also on the
follow-up, that makes use of this to fix Auth_with_shibboleth.t
To test:
1. Run:
$ kshell
k$ prove t/Auth_with_shibboleth.t
=> FAIL: Tests fail! It expects some warns but they are not returned by
the lib
2. Apply this patches
3. Repeat 1
=> SUCCESS: Tests pass! The tests now use the new lib, and they
correctly find the logging Auth_with_shibboleth.pm does on function
calls.
4. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
TestBuilder will generate an integer for the
Koha::Patron::Attribute::Type object, but if 1 is picked some tests are
failing randomly
At least t/db_dependent/Koha/Patrons.t and t/db_dependent/Koha/Patrons/Import.t
The expection "Missing mandatory extended attribute" is raised when the
patron is stored.
Test plan:
The following script should return 0 when the patch is applied:
"""
use t::lib::TestBuilder;
my $builder = t::lib::TestBuilder->new;
my $x = $builder->build_object(
{
class => 'Koha::Patron::Attribute::Types',
}
);
say $x->mandatory;
"""
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The TestBuilder::build_object function used any foreign keys to check
whether an object already exists or not. This brought incorrectly
results of unrelated objects because using any other keys other than
primary keys don't guarantee our results to point to one single
object. For example, as is put here in the unit test, if you created
two items with the same biblionumber and then tried to create a hold
using build_object() we were using the biblionumber to check whether
an item was linked to the hold already. Thus, we were checking whether
a random item was already linked to the hold instead of the one we
wanted either by passing it explicitly to build_object() or the one
build_object() created implicitly. This also resulted in following
warnings when there were more than one match:
DBIx::Class::Storage::DBI::select_single(): Query returned more than
one row. SQL that returns multiple rows is DEPRECATED for ->find and
->single at /kohadevbox/koha/t/lib/TestBuilder.pm line 235
To test:
$ prove t/db_dependent
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We added an upload of the source page and of the screenshot but both services are discontinued.
If we need them back we must host them ourselves.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch makes the build_sample_biblio method, correctly set the UTF-8
flag for the MARC::Record object.
Tests are added.
To test:
1. Apply the regression tests patch
2. Run:
$ kshell
k$ prove t/db_dependent/TestBuilder.t
FAIL: Tests fail! An unexpected encoding warning shows
3. Apply this patch
4. Repeat 2
=> SUCCESS: No warning! Tests pass!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds "1;" to the end of the following perl files:
Koha/Filter/MARC/EmbedItemsAvailability.pm
Koha/Filter/MARC/EmbedSeeFromHeadings.pm
Koha/Filter/MARC/Null.pm
Koha/Item/Search/Field.pm
Koha/SearchEngine.pm
misc/translator/VerboseWarnings.pm
t/lib/Koha/Plugin/Test.pm
This indicates the succesful execution of the initialization code.
Test plan:
Ensure that there are no other perl files that need "1;", but dont have
it.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Fix "submit is not a function error"
A submit button should not be named "submit", in this case, it's id.
https://stackoverflow.com/questions/833032/submit-is-not-a-function-error-in-javascript
Fix some uses of get_attribute()
Fix a fail by setting a global implicit_wait_timeout, default value is 0
in our lib. Other libs set it higher which helps to not have to manually
deal with part of the timing issues.
Fix: remove usage of click_when_visible() because it doesn't work with
elements not in the top of the page. Because they are off screen.
Fix: use $driver->quit() in error_handler to not forget an open Firefox.
With the current version, it fills /dev/shm and fails with around 5
Firefox opened.
Also use quit() it at the end of every script.
Fix: filling item fields, to fill only the displayed one (not those
with display:none)
== Test plan ==
1. Update selenium/standalone-firefox to the latest version [1]
2. prove t/db_dependent/selenium/authentication.t
3. It fails with: arguments[0].form.submit is not a function
4. Apply patch
5. Retest
6. Success
[1] In koha-testing-docker you can do it with
docker-compose.yml:
selenium:
- image: selenium/standalone-firefox:2.53.1-americium
+ image: selenium/standalone-firefox
Signed-off-by: Mason James <mtj@kohaaloha.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This is a follow-up bug for bug 26162
By finding the element before the click I hope to get the good element,
even if the page changed in the meanwhile.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1. Run the new unit test: t/db_dependent/Koha/Template/Plugin/Registers.t
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Setting items.materials to NULL at TestBuilder level will (certainly) prevent some
tests to fail if the pref is on
In t/db_dependent/selenium/basic_workflow.t we set 952$3 to an empty
string. The tests will pass even if the pref is turned on.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch tweaks the test plugin so it composes an exception based on
the fact that the hook was called with the item and item_id parameters
defined.
It then makes the tests expect a specific exception message with
information about this.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/Plugins/Biblio_and_Items_plugin_hooks.t
=> FAIL: Tests fail!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We tell TestBuilder to generate the categories with NULL (and rely on
the sysprefs)
Signed-off-by: Agustin Moyano <agustinmoyano@theke.io>
https://bugs.koha-community.org/show_bug.cgi?id=23826
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch replaces all calls to RefundLostItemFeeRules with
Koha::CirculationRules->get_lostreturn_policy and removes the module it
makes redundant.
Test plan
1/ Confirm that there are no longer any uses of RefundLostItemFeeRules
in the codebase
2/ Confirm circulation tests still all pass
3/ Confirm you can still set and unset the lost return rules
4/ Signoff
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The previous patch did not work as expected. We still got a
StaleElementReference exception.
But this time on
10:43:47 selenium_1 | Caused by: org.openqa.selenium.NoSuchElementException: Unable to locate element: {"method":"xpath","selector":"//*[@id=\"branchname\"]"}
Because we found the one that existed on the page, not the one sent back
in AJAX.
The idea of this patch is to search for the "Showing 1 to X of Y entries" info and wait for X == Y
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
See
https://stackoverflow.com/questions/12967541/how-to-avoid-staleelementreferenceexception-in-seleniumhttps://www.selenium.dev/exceptions/https://developer.mozilla.org/en-US/docs/Web/WebDriver/Errors/StaleElementReference
This patch will fix the following failure we get under D11:
18:47:07 selenium_1 | 09:47:07.478 WARN - Exception: Element not found in the cache - perhaps the page has changed since it was looked up
18:47:07 selenium_1 | For documentation on this error, please visit: http://seleniumhq.org/exceptions/stale_element_reference.html
18:47:07 selenium_1 | Build info: version: '2.53.1', revision: 'a36b8b1', time: '2016-06-30 17:37:03'
18:47:07 selenium_1 | System info: host: '78b9a07f51f2', ip: '192.168.16.2', os.name: 'Linux', os.arch: 'amd64', os.version: '4.19.0-9-amd64', java.version: '1.8.0_91'
18:47:07 selenium_1 | Driver info: driver.version: unknown
18:47:07 koha_1 |
18:47:07 koha_1 | STRACE: /usr/share/perl5/Try/Tiny.pm:123 in Selenium::Remote::Driver::catch {...}
18:47:07 koha_1 | /usr/local/share/perl/5.26.1/Selenium/Remote/Driver.pm:353 in Try::Tiny::try
18:47:07 koha_1 | (eval 1571):1 in Selenium::Remote::Driver::__ANON__
18:47:07 koha_1 | (eval 1573):2 in Selenium::Remote::Driver::__ANON__
18:47:07 koha_1 | (eval 1546):17 in Selenium::Remote::Driver::_execute_command
18:47:07 koha_1 | /usr/local/share/perl/5.26.1/Selenium/Remote/WebElement.pm:63 in Selenium::Remote::WebElement::_execute_command
18:47:07 koha_1 | /kohadevbox/koha/t/lib/Selenium.pm:184 in Selenium::Remote::WebElement::click
18:47:07 koha_1 | /kohadevbox/koha/t/lib/Selenium.pm:172 in t::lib::Selenium::click_when_visible
18:47:07 koha_1 | t/db_dependent/selenium/administration_tasks.t:131 in t::lib::Selenium::click
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch simplifies the payload as suggested on bug 25855. It also
keeps some specific params that cannot be deduced from the passed
checkout object, (e.g. if it is an onsite checkout).
Tests are cleared and added for this special exceptions.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds tests for AddReserve
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch makes the hook be passed the Koha::Checkout object instead
of a hand-crafted list of attributes.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch generalizes the hook so it can be used by other circulation
actions.
Tests are also simplified by mocking some of the (extensive) plugin
hooks.
To test:
1. Repeat the test plan on the original patch
=> SUCCESS: All good
2. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds tests fr a new circulation hook for plugins.
In this case the post_renewal_action hook,
The tests add the hook to the Test plugin, and verify that all the
required parameters are passed for the plugin hook to use them.
It relies on throwing an exception that is to be caught.
Sponsored-by: ByWater Solutions
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
I am really struggling finding the problem here.
Posting the source of the page may help.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
https://framapic.org/ is closing
"""
Framapic will be closing its doors on Tuesday, January 12, 2021. You will find a similar service on this page.
Uploading images is now disabled, but you can still retrieve your images on the My images page.
"""
https://framablog.org/2019/09/26/lets-de-frama-tify-the-internet/https://framablog.org/2020/03/03/10-bonnes-raisons-de-fermer-certains-services-framasoft-la-5e-est-un-peu-bizarre/
(French)
We can still use another service. However we may think about hosting the
service ourself!
Test plan:
Modify a selenium script to make it fail (for instance modify the path
for a find_element call)
Run it
Confirm that the screenshot has been uploaded correctly and that the
link works
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The API related tests still don't pass with the previous modifications.
They pass on D10 but fail on U18 and I did not manage to find where the
problems come from.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This is a try to fix failures in both
t/db_dependent/Template/Plugin/Branches.t
t/db_dependent/Koha/Biblio.t
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds tests the new RESTPublicAnonymousRequests syspref
doesn't apply to routes added by plugins.
It is up to the plugin developer to handle those conditions.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/REST/Plugin/PluginRoutes.t
=> FAIL: Notice the tests fail
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
# Failed test 'check_password hook tests'
# at t/db_dependent/Koha/Plugins/Patron.t line 83.
[The password was rejected by a plugin]# Looks like your test exited with 255 just after 5.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
We are done with AddItem, only need to log and index.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Remove a few last references to branchprinter, mostly from tests.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
As statistics does not have a PK we need to adjust the TestBuilder
tests.
Bug 18441 already exists for adding the PK.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
deletedborrowers does not have a PK, and adding it is out of the scope
of this patchset. Indeed we will have to handle possible duplication of
borrowernumber values, which does not seem trivial.
Having bug 20271 in mind, we will have to deal with this problematic
anyway later.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To make sure this kind of random failures will not appear in a future we
are going to fix it at TestBuilder level.
Test plan:
prove t/db_dependent/TestBuilder.t
and confirm it returns green
You could also only apply the tests against master, run them several
times and confirm that they fail most of the time.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
