Koha/t/db_dependent
Jonathan Druart 63a1adff90 Bug 18066: Hea V2
This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
  hea-ws  - https://github.com/joubu/hea-ws/commits/v2
  hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.

More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.

Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).

With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).

The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
  1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
  2. Link the Privacy_Policy.md from the Hea interface
  3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.

You can find screenshot of this whole enhancement on bug 18066, comment 2.

What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
  * UsageStatsGeolocation
  * UsageStatsLibrariesInfo
  * UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations

How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.

TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?

Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
  CREATE DATABASE hea
  CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
  GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
  FLUSH PRIVILEGES;
3/ Fill the DB with some data
  mysql hea < hea-app/sql/schema.sql
  mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
  % cd hea-app
  % perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
  % perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
  DocumentRoot "/path/to/hea-ws"
  ServerName "hea.koha-community.org"
  <Directory "/">
    Options +ExecCGI
    Require all granted
    AddHandler cgi-script .pl
  </Directory>
</VirtualHost>

And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.

6/ Launch the hea-app
  % cd hea-app
  % edit README.md # to install the missing modules
  % cp environments/config.yml environments/development.yml
  % edit environments/development.yml # to fill the DB info
  % perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data

7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
  % perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified

B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2017-03-23 00:10:46 +00:00
..
Acquisition Bug 18115 - Fix use of Objects as hashref in acqui/addorderiso2709.pl - Bug 15503 followup 2017-03-22 18:58:54 +00:00
api/v1 Bug 17932: (followup) Fix /patrons endpoint 2017-02-17 15:33:00 +00:00
Auth
Authorities Bug 18070: Extend sub merge to remove fields for deleted authorities 2017-03-03 18:12:09 +00:00
Biblio Bug 15572: Add tests for TransformHtmlToMarc 2016-01-25 07:09:50 -07:00
Circulation Bug 18242: [SOLUTION 2]Add tests 2017-03-22 17:38:13 +00:00
Creators Bug 18044: Add a test 2017-02-07 17:46:00 +00:00
data Bug 12419: (QA followup) regression tests 2014-06-30 15:57:23 -03:00
Exporter Bug 17196: [QA Follow-up] Wrong module in Record.t 2017-01-13 13:49:29 +00:00
Holds Bug 17196: Move marcxml out of the biblioitems table 2017-01-13 13:49:26 +00:00
Items Bug 17251: Koha::AuthorisedValues - Remove GetKohaAuthorisedValuesMapping 2016-10-21 15:23:06 +00:00
Koha Bug 18169: Add test without before parameter for search_patrons_to_anonymise 2017-03-03 18:16:55 +00:00
Labels
LDAP
Letters Bug 17962: TT syntax for notices - Prove that ACQ_NOTIF_ON_RECEIV is compatible 2017-02-17 11:43:46 +00:00
Members Bug 18009 - IssueSlip.t test fails if launched between 00:00 and 00:59 2017-02-14 13:59:23 +00:00
OAI Bug 16172: fix OAI Server tests broken by bug 15946 2016-03-31 16:51:38 +00:00
Patron Bug 17486: [QA Follow-up] Changes as to set_userenv 2017-01-13 12:32:20 +00:00
Record Bug 17249: Remove GetKohaAuthorisedValuesFromField - add tests 2016-10-21 15:13:55 +00:00
Reports Bug 12787 [QA Followup] - Retain change to Reports_Guided.t 2016-04-22 00:54:36 +00:00
Reserves Bug 17766 - Patron notification does not work with multi item holds 2017-01-13 11:29:28 +00:00
Search BUG 8010: Correct a few syntax errors, and moved tests 2017-03-22 19:01:28 +00:00
selenium Bug 13691: Add changes for 3.22 2016-07-08 20:36:57 +00:00
Serials Bug 13726: Make Koha::Acq::Bookseller using Koha::Object 2016-12-30 11:54:32 +00:00
SIP Bug 15006: Centralize timeout logic and allow zero client timeout 2016-07-15 14:11:05 +00:00
Template/Plugin Bug 17796: Replace Koha::Issue[s] with Koha::Checkout[s] 2016-12-20 15:13:28 +00:00
Utils Bug 18094: Add tests to highlight the problem 2017-03-22 21:12:43 +00:00
www Bug 17540: (Bug 7216 followup) Fix tests for auth_values_input_www.t 2016-11-04 11:08:41 +00:00
XSLT_Handler
00-strict.t Bug 7844: (follow-up) don't test koha.psgi for ability to compile (yet) 2014-04-29 19:34:23 +00:00
Accounts.t Bug 17894 - Remove and replace WriteOffFee 2017-01-19 11:15:26 +00:00
Acquisition.t Bug 18256: Koha::Items - Remove GetItemsCount 2017-03-22 19:18:15 +00:00
AdditionalField.t Bug 13726: Make Koha::Acq::Bookseller using Koha::Object 2016-12-30 11:54:32 +00:00
Amazon.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
ArticleRequests.t Bug 14610 - Add unit tests 2016-10-26 12:15:15 +00:00
AudioAlerts.t Bug 11431: (QA followup) remove diags 2015-11-06 15:08:37 -03:00
Auth.t Bug 18275: Regression test 2017-03-20 13:00:01 +00:00
Auth_with_cas.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
Auth_with_ldap.t Bug 17615 - Fix unit tests 2017-01-20 13:37:34 +00:00
AuthorisedValues.t Bug 17642: Do not explode if no authorised value exist 2016-11-18 15:52:01 +00:00
AuthoritiesMarc.t Bug 18014: QA follow-up: Use ok() instead of is() for '>' comparison 2017-02-14 14:01:11 +00:00
BackgroundJob.t Bug 17426: Do not set AutoCommit in tests 2016-10-21 14:34:21 +00:00
Barcodes.t Bug 15878 - Updated unit tests for hbyymmincr barcodes 2016-05-23 17:42:10 +00:00
Barcodes_ValueBuilder.t Bug 15081: (QA followup) Fix issue with Barcodes_ValueBuilder.t 2015-11-04 12:32:57 -03:00
Biblio.t Bug 18284: [QA Follow-up] Add test descriptions 2017-03-22 10:26:05 +00:00
Bookseller.t Bug 13726: Make Koha::Acq::Bookseller using Koha::Object 2016-12-30 11:54:32 +00:00
Breeding.t Bug 12898: (regression test) test terms inside double quotes 2014-09-14 02:02:51 -03:00
Budgets.t Bug 13726: Make Koha::Acq::Bookseller using Koha::Object 2016-12-30 11:54:32 +00:00
Charset.t Bug 8218 : Add a maintenance script to sanitize biblio records 2014-11-11 15:38:36 -03:00
check_sysprefs.t Bug 17917: [AMENDED] Fix path issue in t/db_dependent/check_sysprefs.t 2017-01-20 13:39:37 +00:00
Circulation.t Bug 17941 add tests to cover the optimization of the nested loop 2017-03-03 16:50:27 +00:00
ClassSource.t Bug 16860: Remove the comment about the warning 2016-07-08 13:29:50 +00:00
ColumnsSettings.t Bug 16177: Fix tests for ColumnsSsettings.t 2016-04-01 19:08:23 +00:00
Context.t Bug 16813: Allow empty string for OPACBaseURL 2016-07-15 15:02:13 +00:00
Contract.t Bug 13726: Make Koha::Acq::Bookseller using Koha::Object 2016-12-30 11:54:32 +00:00
CourseReserves.t Bug 17716: (followup) Remove dep on existing data and tidy 2016-12-05 15:32:57 +00:00
db_structure.t Bug 16173: db_structure.t shouldn't have a fixed number of tests 2016-04-01 19:14:53 +00:00
DecreaseLoanHighHolds.t Bug 16387: Fix default shortened loan period time 2017-02-07 17:54:21 +00:00
default_search_class.pl Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
Filter_MARC_ViewPolicy.t Bug 11592: Add should_hide_marc method to filter 2016-09-08 11:54:16 +00:00
Fines.t Bug 13590: Add ability to charge fines at start of charge period 2015-10-22 14:51:24 -03:00
FrameworkPlugin.t Bug 14199: Unify marc orgcode plugins, not overwriting existing values 2016-02-26 14:04:28 +00:00
Heading_MARC21.t Merge remote-tracking branch 'origin/new/bug_7284' 2012-04-04 18:18:26 +02:00
Hold.t Bug 14642: (QA followup) Add tests for Koha::Hold->delete 2016-08-17 18:43:13 +00:00
Holds.t Bug 15758: Koha::Libraries - Ultimate duel for C4::Branch 2016-09-08 14:36:04 +00:00
HoldsQueue.t Bug 17196: Move marcxml out of the biblioitems table 2017-01-13 13:49:26 +00:00
Holidays.t Bug 16376: (regression tests) 2016-12-09 15:25:01 +00:00
ILSDI_Services.t Bug 17836: (ILSDI) Regression test 2017-01-20 13:44:57 +00:00
ImportBatch.t Bug 9259: Use is instead of is_deeply 2016-07-08 13:43:53 +00:00
Installer.t Bug 17234: Need to separate KEY and FOREIGN KEY checks 2017-01-12 12:43:10 +00:00
Items.t Bug 17512: Improve handling dates in C4::Items 2017-02-14 13:57:49 +00:00
Items_DelItemCheck.t Bug 14504: Changes missed while fixing patches 2016-08-26 12:07:27 +00:00
KitchenSinkPlugin.kpz Bug 17368: Update KitchenSinkPlugin.kpz 2016-10-21 14:48:53 +00:00
Koha.t Bug 17627: Move C4::Koha::GetItemTypesByCategory to Koha::ItemTypes 2017-02-17 15:31:32 +00:00
Koha_Authority.t Bug 15578: Insert authority records before processing tests 2016-01-23 18:28:49 +00:00
Koha_Database.t Bug 13645: Cache the DBIx connection 2015-02-20 14:37:08 -03:00
Koha_Elasticsearch.t Bug 17255 - Upgrade Elastic Search code to work with version 5.1 2017-02-07 17:17:54 +00:00
Koha_Elasticsearch_Indexer.t Bug 17372: Standardize Elasticsearch paths 2016-10-11 01:07:03 +00:00
Koha_Misc_Files.t Bug 3050: QA follow-up 2014-05-25 21:15:03 +00:00
Koha_SearchEngine_Elasticsearch_Search.t Bug 18068: ES - Fix location and (home|holding)branch facets 2017-02-17 11:34:41 +00:00
Languages.t Bug 16866: Catch warning t/db_dependent/Languages.t 2016-07-08 14:50:18 +00:00
Letters.t Bug 13726: Make Koha::Acq::Bookseller using Koha::Object 2016-12-30 11:54:32 +00:00
Linker_FirstMatch.t Bug 16868: Silence error t/db_dependent/Linker_FirstMatch.t 2016-08-04 20:04:24 +00:00
Log.t Bug 16829: Unit tests for logaction's 'interface parameter' 2016-08-17 18:01:50 +00:00
MarcModificationTemplates.t Bug 15669: Add a test for GetModificationTemplates 2016-03-03 22:11:44 +00:00
Members.t Bug 17578 [QA Followup] - Update number of tests 2016-12-16 13:14:50 +00:00
MungeMarcPrice.t Bug 15084: Replace C4::Budgets::GetCurrencies with Koha::Acquisition::Currencies->search 2016-03-03 20:39:01 +00:00
NewsChannels.t Bug 17960: Rename opac_news.new with opac_news.content 2017-02-06 17:42:12 +00:00
Overdues.t Bug 15758: Koha::Libraries - Remove GetBranches 2016-09-08 14:36:03 +00:00
Passwordrecovery.t Bug 18025: Fix test 2017-02-17 11:24:39 +00:00
PatronLists.t Bug 17486: [QA Follow-up] Changes as to set_userenv 2017-01-13 12:32:20 +00:00
Patrons.t Bug 17742: Use TestBuilder to create the library and patron category 2016-12-28 13:34:21 +00:00
Plugins.t Bug 15879: Add Tests 2017-01-11 14:03:00 +00:00
QueryParser.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
Record.t
RecordProcessor_EmbedSeeFromHeadings.t Bug 15380: Move Koha::Authority to Koha::MetadataRecord::Authority 2015-12-31 18:50:41 +00:00
RefundLostItemFeeRule.t Bug 14048: (QA followup) Default to 1 if no default rule 2016-07-07 10:37:01 -06:00
Reports.t Bug 5327: Shifting a db dependent test to the right place 2012-02-01 18:15:19 +01:00
Reserves.t Bug 18045: Fix Reserves.t tests - caching issues 2017-02-17 11:33:37 +00:00
rollingloans.t Bug 17578: GetMemberDetails - Remove GetMemberDetails 2016-12-16 13:12:44 +00:00
RotatingCollections.t Bug 15629: Koha::Libraries - Remove ModBranch 2016-02-24 03:55:06 +00:00
Search.t Bug 17917: [QA Follow-up] Reprove Search.t 2017-01-20 13:39:36 +00:00
Serials.t Bug 15030: Add tests 2017-02-07 17:43:49 +00:00
Serials_2.t Bug 7677: UT: Remove $dbh->rollback 2016-09-12 09:13:58 +00:00
Service.t Bug 5477 [SIGN-OFF] Fix test cases that require database access 2011-01-29 17:02:31 +13:00
ShelfBrowser.t
Sitemapper.t Bug 17920: t/db_dependent/Sitemapper.t fails because of permissions 2017-01-20 14:04:21 +00:00
Stats.t
SuggestionEngine_ExplodedTerms.t Bug 16869: Silence and catch warnings in t/db_dependent/SuggestionEngine_ExplodedTerms.t 2016-07-08 13:08:09 +00:00
Suggestions.t Bug 17099: C4::Koha - Remove GetSupportName and GetSupportList 2016-09-09 13:30:45 +00:00
sysprefs.t Bug 11998: Use Koha::Cache to cache sysprefs 2016-03-15 07:08:28 +00:00
Tags.t Bug 12916 - Missing Test to demonstrate warnings. 2014-10-30 00:31:26 -03:00
Templates.t Bug 18036: [QA Follow-up] Descriptions and mock_preference 2017-02-17 15:45:41 +00:00
TestBuilder.t Bug 17726: [QA Follow-up] Add test descriptions 2017-01-20 13:43:18 +00:00
Upload.t Bug 17501: [Follow-up] QA Requests 2017-01-20 14:20:07 +00:00
UsageStats.t Bug 18066: Hea V2 2017-03-23 00:10:46 +00:00
Virtualshelves.t Bug 16551: Display the name of lists to the search results at the OPAC 2016-09-25 15:43:07 +00:00
XISBN.t
XSLT_Handler.t Bug 13262: (QA followup) fix duplicate variable definition 2015-02-05 15:25:50 -03:00
zebra_config.pl