Bug 34587: SUSHI COUNTER Harvester
authorPedro Amorim <pedro.amorim@ptfs-europe.com>
Wed, 1 Mar 2023 17:38:20 +0000 (17:38 +0000)
committerTomas Cohen Arazi <tomascohen@theke.io>
Tue, 31 Oct 2023 19:45:58 +0000 (16:45 -0300)
commit2951edc697ea68fcd8c65d608db7a3a743e17563
tree7e1add793a448db55e8303421922c416b37bac67
parent1e50ed86b458cbb170f4f1d184809148f42b25d5
Bug 34587: SUSHI COUNTER Harvester

This commit is a squash of the following:

SUSHI harvesting process in the data providers class:
* Builds the URL query and requests the SUSHI service endpoint
* Parses the JSON response and builds the csv COUNTER file and adds it to counter_files table

Usage statistics data processing:
* When a counter_files entry is stored, CounterFile.pm will:
* Parse the csv COUNTER file and
* Add a usage_titles entry for each unique title in the COUNTER file
* Add the title's respective erm_usage_mus (monthly usage) entries, repeating for each metric_type
* Add the title's respective erm_usage_yus (yearly usage) entries, repeating for each metric_type

Harvesting cronjob;

'Run now':
* API endpoint to start the harvesting process of a data provider
* Button in the data providers list to run the harvesting process for each data provider upon clicked

ERM SUSHI: Background job

Job progress is updated to total amount of usage titles after retrieving
the response from SUSHI;
Job warning and success messages are added accordingly
Redundant duplicate titles will not be added
Redundant duplicate monthly and yearly usage statistics will not be added
Data provider harvest background job harvests once per report_type

Enqueue one background job for each report_type in the usage data provider

Update the way we measure progress in the background job.
It now uses the COUNTER report body rows instead of SUSHI response results.
We're now incrementing and showing the number of skipped mus, skipped
yus, added mus and added yus
There's a bug in the way we calculate yus
Updates to background job progress bar - Depends on 34468

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
35 files changed:
Koha/BackgroundJob.pm
Koha/BackgroundJob/ErmSushiHarvester.pm [new file with mode: 0644]
Koha/ERM/CounterFile.pm
Koha/ERM/UsageDataProvider.pm
Koha/ERM/UsageDatabase.pm [new file with mode: 0644]
Koha/ERM/UsageDatabases.pm [new file with mode: 0644]
Koha/ERM/UsageItem.pm [new file with mode: 0644]
Koha/ERM/UsageItems.pm [new file with mode: 0644]
Koha/ERM/UsagePlatform.pm [new file with mode: 0644]
Koha/ERM/UsagePlatforms.pm [new file with mode: 0644]
Koha/ERM/UsageTitle.pm
Koha/REST/V1/ERM/CounterFiles.pm
Koha/REST/V1/ERM/UsageDataProviders.pm
Koha/Schema/Result/ErmUsageDataProvider.pm
Koha/Schema/Result/ErmUsageDatabase.pm [new file with mode: 0644]
Koha/Schema/Result/ErmUsageItem.pm [new file with mode: 0644]
Koha/Schema/Result/ErmUsageMus.pm
Koha/Schema/Result/ErmUsagePlatform.pm [new file with mode: 0644]
Koha/Schema/Result/ErmUsageTitle.pm
Koha/Schema/Result/ErmUsageYus.pm
api/v1/swagger/definitions/erm_usage_mus.yaml
api/v1/swagger/definitions/erm_usage_title.yaml
api/v1/swagger/definitions/erm_usage_yus.yaml
api/v1/swagger/paths/erm_counter_files.yaml
api/v1/swagger/paths/erm_usage_data_providers.yaml
api/v1/swagger/paths/erm_usage_titles.yaml
api/v1/swagger/swagger.yaml
installer/data/mysql/atomicupdate/erm_usage_statistics_tables.pl
installer/data/mysql/kohastructure.sql
koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/erm_sushi_harvester.inc [new file with mode: 0644]
koha-tmpl/intranet-tmpl/prog/en/includes/str/erm_sushi_harvester.inc [new file with mode: 0644]
koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt
koha-tmpl/intranet-tmpl/prog/js/vue/components/ERM/UsageStatisticsDataProvidersList.vue
koha-tmpl/intranet-tmpl/prog/js/vue/fetch/erm-api-client.js
misc/cronjobs/erm_run_harvester.pl [new file with mode: 0755]