Koha/t/db_dependent
Julian Maurice 4605304f62 Bug 28306: Allow to query database with minimal memory footprint
The goal is to be able to build a database handler (dbh) and to execute
queries without loading unnecessary stuff. This will be useful to reduce
memory usage of daemons that need to check the database
periodically

The patch provides a new method Koha::Database::dbh which returns a
database handler without loading the DBIx::Class schema. This method is
also used by DBIx::Class, so whether you use DBI or DBIx::Class, the
same method is used to initialize the connection.

The patch also moves some code in order to avoid loading C4::Context:
- C4::Context::timezone moves to Koha::Config
- C4::Context::db_scheme2dbi moves to Koha::Database

To measure memory usage I used the following commands:

* before the patch:
perl -MKoha::Database \
    -E 'Koha::Database->schema->storage->dbh->do("select 1");' \
    -E '$|=1; say $$; sleep 2' \
    | while read pid; do ps -p $pid -o rss=; done

* after the patch:
perl -MKoha::Database \
    -E 'Koha::Database->dbh->do("select 1");' \
    -E '$|=1; say $$; sleep 2' \
    | while read pid; do ps -p $pid -o rss=; done

It will give you the RSS (Resident Set Size) of the perl process in kB

What I get:
* before the patch: between 96.9MB and 97.2MB
* after the patch: between 17.8MB and 18.2MB

Note that if a timezone is configured (either from $KOHA_CONF or
TZ environment variable), Koha will load DateTime::Timezone to check if
it's valid, and it increases RSS to 36MB

Another interesting metric is the number of modules loaded:
* before the patch:
perl -MKoha::Database \
    -E 'Koha::Database->schema->storage->dbh;' \
    -E 'say scalar keys %INC'

Result: 567

* after the patch:
perl -MKoha::Database \
    -E 'Koha::Database->dbh;' \
    -E 'say scalar keys %INC'

Result: 51

Test plan:
1. Apply the patch & restart starman
2. Make sure Koha is still ok (ie. can access the database, does not
have encoding issues, ...)
3. Run the tests in t/Context.t, t/Koha/Config.t,
t/db_dependent/Koha/Database.t, t/timezones.t

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-08-31 10:27:52 +02:00
..
Acquisition Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
api/v1 Bug 22435: Update unit tests 2021-08-04 14:06:43 +02:00
Auth Bug 24387: Rename "News" with "Additional contents" 2021-08-16 11:55:55 +02:00
AuthoritiesMarc
Authority Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Biblio Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Circulation Bug 22435: Update unit tests 2021-08-04 14:06:43 +02:00
CourseReserves Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Creators Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
cronjobs
data Bug 27968: Fix framework import if isurl has an incorrect value 2021-04-23 10:03:30 +02:00
Exporter Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Holds Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Illrequest
Items Bug 22690: Refactor merging of records to improve performance (Elasticsearch) 2021-08-27 11:02:40 +02:00
Koha Bug 28306: Allow to query database with minimal memory footprint 2021-08-31 10:27:52 +02:00
Labels Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
LDAP
Letters Bug 24387: Fix Letters/TemplateToolkit.t 2021-08-16 15:18:10 +02:00
Members Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
misc/translator
OAI Bug 28741: Regression tests 2021-08-04 09:14:46 +02:00
Patron Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Record Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Reports Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Reserves Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Search Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
selenium Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Serials Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
SIP Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Template/Plugin Bug 24154: (follow-up) Make parameter and variable more explicit, add tests 2021-05-10 10:27:53 +02:00
Utils Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
www Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
00-strict.t Bug 28606: Remove $DEBUG and $ENV{DEBUG} 2021-06-24 11:53:44 +02:00
01-test_dbic.t
Accounts.t Bug 22435: (follow-up) Set CREATE offset for credits to positive 2021-08-04 14:06:43 +02:00
Acquisition.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Amazon.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ArticleRequests.t
AudioAlerts.t
Auth.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Auth_with_cas.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Auth_with_ldap.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
AuthorisedValues.t Bug 28554: Fix t/db_dependent/AuthorisedValues.t 2021-08-30 17:02:07 +02:00
AuthoritiesMarc.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
AuthUtils.t
BackgroundJob.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Barcodes.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Barcodes_ValueBuilder.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Biblio.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Breeding.t
Budgets.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Cache.t
Calendar.t
Charset.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
check_kohastructure.t
check_sysprefs.t
Circulation.t Bug 22435: (QA follow-up) More test updates 2021-08-04 14:06:43 +02:00
Clubs.t
Context.t Bug 28606: Remove $DEBUG and $ENV{DEBUG} 2021-06-24 11:53:44 +02:00
Contract.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
CourseReserves.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
db_structure.t
DecreaseLoanHighHolds.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Filter_MARC_ViewPolicy.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Fines.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
FrameworkPlugin.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Heading.t Bug 26852: Fix Heading.t 2021-08-11 17:01:51 +02:00
Heading_MARC21.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Hold.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Holds.t Bug 25619: (QA follow-up) Fix subtest description 2021-08-27 15:28:23 +02:00
HoldsQueue.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Holidays.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Illcomments.t
Illrequestattributes.t
Illrequests.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ILSDI_Services.t Bug 28630: Improve date comparisons in tests 2021-08-04 14:06:43 +02:00
ImportBatch.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ImportExportFramework.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Installer.t Bug 15067: (QA follow-up) Ensure update is idempotent 2021-08-04 14:06:43 +02:00
Items.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Koha.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Koha_Authority.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Koha_Database.t
Koha_Elasticsearch.t
Koha_ExternalContent_OverDrive.t
Koha_ExternalContent_RecordedBooks.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Koha_Misc_Files.t
Languages.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Letters.t Bug 28813: Update unit tests 2021-08-11 12:53:54 +02:00
LibraryGroups.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Linker_Default.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Linker_FirstMatch.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Log.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
MarcModificationTemplates.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Members.t Bug 17600: Fix t/db_dependent/Members.t 2021-07-26 19:34:59 +02:00
Message.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
MungeMarcPrice.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Overdues.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Passwordrecovery.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Patroncards.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
PatronLists.t
Patrons.t
Record.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
RecordProcessor_EmbedSeeFromHeadings.t
Reports.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Reserves.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
rollingloans.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
RotatingCollections.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Search.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Serials.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Serials_2.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Service.t
ShelfBrowser.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Sitemapper.t
Stats.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
StockRotationItems.t
StockRotationRotas.t
StockRotationStages.t
SuggestionEngine_ExplodedTerms.t
Suggestions.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
sysprefs.t
TablesSettings.t
Tags.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Templates.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
TestBuilder.t Bug 28479: Use primary keys to check object existence in TestBuilder 2021-06-21 12:07:28 +02:00
Upload.t
UsageStats.t Bug 28565: Refactor tests 2021-07-26 12:46:35 +02:00
Virtualshelves.t
XISBN.t Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
XSLT.t Bug 26302: (QA follow-up) Prefix prefs with OPAC 2021-08-30 17:04:31 +02:00
yaml.t
zebra_config.pl Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00