From 63a1adff9097493be0329aebab5fb3bf8a627b80 Mon Sep 17 00:00:00 2001 From: Jonathan Druart Date: Thu, 2 Feb 2017 16:53:58 +0100 Subject: [PATCH] Bug 18066: Hea V2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf8 Content-Transfer-Encoding: 8bit This patch is the Koha part of the Hea v2 project. You can find the (testing) code for the server at hea-ws - https://github.com/joubu/hea-ws/commits/v2 hea-app - https://github.com/joubu/hea-app/commits/v2 They contain the different pull requests made over the last 6 months. More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC The goal of this commit message is to provide an overview of what could be a new version of Hea. Prior to these changes, the Hea database was filled with 1 line per Koha installation. System preferences were filled by the libraries and a cronjob (share_usage_with_koha_community.pl) collected these values to send them to a webservice (hea-ws/upload.pl). With the need to collect more data we would want to collect data at the library level (branch) and not at the installation level. For instance the geolocation, the url or the country can be different from one library to another, even if managed from the same Koha installation. The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql). The hidden goal of this patch is to make Hea sexier and explain better to libraries how it can be useful to share their information with the Koha community. I guess the main problem is the lack of communication and explanations about what we are doing we these data. To fill this gap I'd like to (TODO) 1. Communicate on the ML about this new version of Hea (once it got pushed and backported) 2. Link the Privacy_Policy.md from the Hea interface 3. Get help from a native English speaker to add popup/help/info/whatever on "Home › Administration › Usage statistics", to clearly explain what happens (and what will not happen!) when an option or another is set. You can find screenshot of this whole enhancement on bug 18066, comment 2. What this patch does: - Create a new branches.geolocation DB field - Add 3 new sysprefs: * UsageStatsGeolocation * UsageStatsLibrariesInfo * UsageStatsPublicID - Integrate the Leaflet JS library to get a fancy map to pick geolocations How does it works: On the new administration page where statistics to share are configured, there are several new things. It is now possible to share information either per Koha installation or libraries. If UsageStatsLibrariesInfo is set, the info at library level (url, name, country, geolocation) will be sent to the Hea webservice. If it is not set, you can decide to fill UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry, UsageStatsGeolocation to share these information. Note that even if the data are retrieved at installation level, it's better to fill the prefs as well: On the Hea website the different libraries defined for a given Koha installation could be displayed on the same page. This page is a public page which will be attributed to every installation (with the pref UsageStatsPublicID). On this page all the info available publicly will be displayed. TODO later: - Add a button on the administration page to delete the info shared publicly. It will be easy to show that the info are no longer displayed on the public page. - Add an icon per Koha installation to get a better "public page" - Any suggestions? Test plan: We will need to test hea-ws, hea-app and the Koha-side code to test the whole enhancement. 1/ To start, clone the hea-ws and hea-app project and checkout the 'master' branch (*not* 'v2') 2/ Create the hea database and user CREATE DATABASE hea CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea'; GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost'; FLUSH PRIVILEGES; 3/ Fill the DB with some data mysql hea < hea-app/sql/schema.sql mysql hea < hea-app/sql/sql/mock-data.sql 4/ Checkout the 'v2' branch for both hea-ws and hea-app 5/ Execute the upgrade DB script % cd hea-app % perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info % perl sql/upgrade.pl Now the DB is using the v2 structure. That means we have 1 installation row per library previously defined. 1 library row has also been created. 5/ Configure hea-ws % echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts DocumentRoot "/path/to/hea-ws" ServerName "hea.koha-community.org" Options +ExecCGI Require all granted AddHandler cgi-script .pl And enable it with a2ensite, then restart apache. The copy the database.yml.sample to database.yml and edit it to fill the DB info. 6/ Launch the hea-app % cd hea-app % edit README.md # to install the missing modules % cp environments/config.yml environments/development.yml % edit environments/development.yml # to fill the DB info % perl bin/app.pl Then hit localhost:3000 You should see a local version of Hea with sample data 7/ Back to Koha side A. We will test that the webservice still works with previous version of Koha (without v2) a. Do not configure Hea % perl misc/cronjobs/share_usage_with_koha_community.pl -f -v Then hit localhost:3000 => Nothing added b. Configure Hea on admin/usage_statistics.pl perl misc/cronjobs/share_usage_with_koha_community.pl -f -v => New library added c. Modify the Hea configuration perl misc/cronjobs/share_usage_with_koha_community.pl -f -v => Info are modified B. Not we will test that it works with the new version (much more fun ;)) % git checkout hea-v2 # koha a. Configure Hea using /admin/usage_statistics.pl perl misc/cronjobs/share_usage_with_koha_community.pl -f -v => Check the result on localhost:3000 b. Share libraries's info perl misc/cronjobs/share_usage_with_koha_community.pl -f -v c. Continue to play a bit and share the info. Signed-off-by: Chris Cormack Signed-off-by: Julian Maurice Signed-off-by: Brendan A Gallagher --- C4/UsageStats.pm | 26 ++- Koha/Schema/Result/Branch.pm | 12 +- admin/usage_statistics.pl | 17 +- .../data/mysql/atomicupdate/bug_18066.perl | 21 +++ installer/data/mysql/sysprefs.sql | 3 + .../intranet-tmpl/prog/en/modules/about.tt | 3 + .../en/modules/admin/preferences/admin.pref | 11 ++ .../prog/en/modules/admin/usage_statistics.tt | 163 +++++++++++++++++- .../share_usage_with_koha_community.pl | 2 +- t/db_dependent/UsageStats.t | 15 +- 10 files changed, 257 insertions(+), 16 deletions(-) create mode 100644 installer/data/mysql/atomicupdate/bug_18066.perl diff --git a/C4/UsageStats.pm b/C4/UsageStats.pm index cf8b98eb9d..c71f99bc19 100644 --- a/C4/UsageStats.pm +++ b/C4/UsageStats.pm @@ -23,6 +23,8 @@ use POSIX qw(strftime); use LWP::UserAgent; use JSON; +use Koha::Libraries; + =head1 NAME C4::UsageStats @@ -56,14 +58,24 @@ sub NeedUpdate { } sub BuildReport { - my $report = { - library => { - id => C4::Context->preference('UsageStatsID') || 0, + my $report; + my @libraries; + if( C4::Context->preference('UsageStatsLibrariesInfo') ) { + my $libraries = Koha::Libraries->search; + while ( my $library = $libraries->next ) { + push @libraries, { name => $library->branchname, url => $library->branchurl, country => $library->branchcountry, geolocation => $library->geolocation, }; + } + } + $report = { + installation => { + koha_id => C4::Context->preference('UsageStatsID') || 0, name => C4::Context->preference('UsageStatsLibraryName') || q||, url => C4::Context->preference('UsageStatsLibraryUrl') || q||, type => C4::Context->preference('UsageStatsLibraryType') || q||, country => C4::Context->preference('UsageStatsCountry') || q||, + geolocation => C4::Context->preference('UsageStatsGeolocation') || q||, }, + libraries => \@libraries, }; # Get database volumetry. @@ -351,8 +363,12 @@ sub ReportToCommunity { Content => $json, ); my $content = decode_json( $res->decoded_content ); - C4::Context->set_preference( 'UsageStatsID', - $content->{library}{id} ); + if ( $content->{koha_id} ) { + C4::Context->set_preference( 'UsageStatsID', $content->{koha_id} ); + } + if ( $content->{id} ) { + C4::Context->set_preference( 'UsageStatsPublicID', $content->{id} ); + } } =head2 _count diff --git a/Koha/Schema/Result/Branch.pm b/Koha/Schema/Result/Branch.pm index ec44cded06..414b734209 100644 --- a/Koha/Schema/Result/Branch.pm +++ b/Koha/Schema/Result/Branch.pm @@ -128,6 +128,12 @@ __PACKAGE__->table("branches"); data_type: 'text' is_nullable: 1 +=head2 geolocation + + data_type: 'varchar' + is_nullable: 1 + size: 255 + =cut __PACKAGE__->add_columns( @@ -171,6 +177,8 @@ __PACKAGE__->add_columns( { data_type => "mediumtext", is_nullable => 1 }, "opac_info", { data_type => "text", is_nullable => 1 }, + "geolocation", + { data_type => "varchar", is_nullable => 1, size => 255 }, ); =head1 PRIMARY KEY @@ -543,8 +551,8 @@ Composing rels: L -> categorycode __PACKAGE__->many_to_many("categorycodes", "branchrelations", "categorycode"); -# Created by DBIx::Class::Schema::Loader v0.07042 @ 2016-10-24 13:56:21 -# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:/9YwsU+GXK+fzc6IX2Tj5g +# Created by DBIx::Class::Schema::Loader v0.07042 @ 2017-02-06 10:07:25 +# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:aq4mf5Dwbd8ps9ANLy8DuQ # You can replace this text with custom code or comments, and it will be preserved on regeneration diff --git a/admin/usage_statistics.pl b/admin/usage_statistics.pl index 4573321758..01e002eda0 100755 --- a/admin/usage_statistics.pl +++ b/admin/usage_statistics.pl @@ -21,6 +21,7 @@ use CGI qw ( -utf8 ); use C4::Auth; use C4::Output; use Koha::DateUtils qw( dt_from_string output_pref ); +use Koha::Libraries; my $query = new CGI; my ( $template, $loggedinuser, $cookie ) = get_template_and_user( @@ -42,12 +43,21 @@ if ( $op eq 'update' ) { my $UsageStatsLibraryName = $query->param('UsageStatsLibraryName'); my $UsageStatsLibraryType = $query->param('UsageStatsLibraryType'); my $UsageStatsLibraryUrl = $query->param('UsageStatsLibraryUrl'); + my $UsageStatsLibrariesInfo = $query->param('UsageStatsLibrariesInfo'); + my $UsageStatsGeolocation = $query->param('UsageStatsGeolocation'); C4::Context->set_preference('UsageStats', $UsageStats); C4::Context->set_preference('UsageStatsCountry', $UsageStatsCountry); C4::Context->set_preference('UsageStatsLibraryName', $UsageStatsLibraryName); C4::Context->set_preference('UsageStatsLibraryType', $UsageStatsLibraryType); C4::Context->set_preference('UsageStatsLibraryUrl', $UsageStatsLibraryUrl); - + C4::Context->set_preference('UsageStatsLibrariesInfo', $UsageStatsLibrariesInfo); + C4::Context->set_preference('UsageStatsGeolocation', $UsageStatsGeolocation); + my $libraries = Koha::Libraries->search; + while ( my $library = $libraries->next ) { + if ( my $latlng = $query->param('geolocation_' . $library->branchcode) ) { + $library->geolocation( $latlng )->store; + } + } } if ( C4::Context->preference('UsageStatsLastUpdateTime') ) { @@ -55,4 +65,9 @@ if ( C4::Context->preference('UsageStatsLastUpdateTime') ) { $template->param(UsageStatsLastUpdateTime => output_pref($dt) ); } +my $libraries = Koha::Libraries->search; +$template->param( + libraries => $libraries, +); + output_html_with_http_headers $query, $cookie, $template->output; diff --git a/installer/data/mysql/atomicupdate/bug_18066.perl b/installer/data/mysql/atomicupdate/bug_18066.perl new file mode 100644 index 0000000000..4ad2fc817b --- /dev/null +++ b/installer/data/mysql/atomicupdate/bug_18066.perl @@ -0,0 +1,21 @@ +$DBversion = 'XXX'; +if( CheckVersion( $DBversion ) ) { + unless( column_exists( 'branches', 'geolocation' ) ) { + $dbh->do(q| + ALTER TABLE branches ADD COLUMN geolocation VARCHAR(255) DEFAULT NULL after opac_info + |); + } + + $dbh->do(q| + INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type ) VALUES ('UsageStatsGeolocation', '', NULL, 'Geolocation of the main library', 'Free'); + |); + $dbh->do(q| + INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type ) VALUES ('UsageStatsLibrariesInfo', '', NULL, 'Share libraries information', 'YesNo'); + |); + $dbh->do(q| + INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type ) VALUES ('UsageStatsPublicID', '', NULL, 'Public ID for Hea website', 'Free'); + |); + + SetVersion( $DBversion ); + print "Upgrade to $DBversion done (Bug 18066 - Hea version 2)\n"; +} diff --git a/installer/data/mysql/sysprefs.sql b/installer/data/mysql/sysprefs.sql index 79229d7f95..eaf856a44a 100644 --- a/installer/data/mysql/sysprefs.sql +++ b/installer/data/mysql/sysprefs.sql @@ -541,10 +541,13 @@ INSERT INTO systempreferences ( `variable`, `value`, `options`, `explanation`, ` ('UsageStats', 0, NULL, 'Share anonymous usage data on the Hea Koha community website.', 'YesNo'), ('UsageStatsCountry', '', NULL, 'The country where your library is located, to be shown on the Hea Koha community website', 'Choice'), ('UsageStatsID', '', NULL, 'This preference is part of Koha but it should not be deleted or updated manually.', 'Free'), +('UsageStatsGeolocation', '', NULL, 'Geolocation of the main library.', 'Free'), ('UsageStatsLastUpdateTime', '', NULL, 'This preference is part of Koha but it should not be deleted or updated manually.', 'Free'), +('UsageStatsLibrariesInfo', '', NULL, 'Share libraries information', 'YesNo'), ('UsageStatsLibraryName', '', NULL, 'The library name to be shown on Hea Koha community website', 'Free'), ('UsageStatsLibraryType', '', 'public|school|academic|research|private|societyAssociation|corporate|government|religiousOrg|subscription', 'The library type to be shown on the Hea Koha community website', 'Choice'), ('UsageStatsLibraryUrl', '', NULL, 'The library URL to be shown on Hea Koha community website', 'Free'), +('UsageStatsPublicID', '', NULL, 'Public ID for Hea website', 'Free'), ('UseAuthoritiesForTracings','1','0','Use authority record numbers for subject tracings instead of heading strings.','YesNo'), ('UseBranchTransferLimits','0','','If ON, Koha will will use the rules defined in branch_transfer_limits to decide if an item transfer should be allowed.','YesNo'), ('UseControlNumber','0','','If ON, record control number (w subfields) and control number (001) are used for linking of bibliographic records.','YesNo'), diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/about.tt b/koha-tmpl/intranet-tmpl/prog/en/modules/about.tt index b2e73cfbc9..b3c43a3bd5 100644 --- a/koha-tmpl/intranet-tmpl/prog/en/modules/about.tt +++ b/koha-tmpl/intranet-tmpl/prog/en/modules/about.tt @@ -820,6 +820,9 @@

jquery.tablednd.js

The TableDnD plug-in for jQuery by Denis Howlett is licensed under the MIT License.

+ +

Leaflet

+

The Leaflet JavaScript library by Vladimir Agafonkinis licensed under the BSD License.

diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref index e9d95be216..ba491283f2 100644 --- a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref +++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref @@ -394,6 +394,17 @@ Administration: religiousOrg: "religious organization" subscription: "subscription" - Note that this value has no effect if the UsageStats system preference is set to "Don't share" + - + - pref: UsageStatsLibrariesInfo + choices: + yes: "Share" + no: "Do not Share" + - "libraries information (name, url, country)" + - Note that this value has no effect if the UsageStats system preference is set to "Don't share" + - + - Geolocation of the main library: + - pref: UsageStatsGeolocation + - Note that this value has no effect if the UsageStats system preference is set to "Don't share" Search Engine: - - "Use following search engine: " diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/usage_statistics.tt b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/usage_statistics.tt index 2850dfd60e..2e807b2067 100644 --- a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/usage_statistics.tt +++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/usage_statistics.tt @@ -5,6 +5,36 @@ Koha › Administration › Koha usage statistics [% INCLUDE 'doc-head-close.inc' %] [% INCLUDE 'calendar.inc' %] + + + + @@ -48,7 +78,7 @@

Modify the statistics you share with the Koha community

-
    +
    1. +
    2. @@ -159,8 +189,58 @@ Your data have never been shared [% END %]
      3. +
    3. + + + +
      4. +
    4. + + +
      5. + +
    5. +
      + Libraries +
        + [% FOR l IN libraries %] +
      1. + +
        +
        Country: [% l.branchcountry %]
        +
        Url: [% l.branchurl %]
        +
        + Geolocation: + +
        +
        +
        2. + [% END %] +
      +
      +
      6. +
    6. + + [% IF Koha.Preference('UsageStatsPublicID') %] + [% SET my_url = 'http://hea.koha-community.org/libraries/' _ Koha.Preference('UsageStatsPublicID') %] + [% my_url %] + [% ELSE %] + You do not have anything public yet. + [% END %] +
      7. +
    -
+
+
+
@@ -169,6 +249,83 @@
+ +
diff --git a/misc/cronjobs/share_usage_with_koha_community.pl b/misc/cronjobs/share_usage_with_koha_community.pl index 16de62aedc..9039558302 100755 --- a/misc/cronjobs/share_usage_with_koha_community.pl +++ b/misc/cronjobs/share_usage_with_koha_community.pl @@ -70,7 +70,7 @@ Only the total number is retrieved. In no case will private data be shared! In order to know which parts of Koha modules are used, this script will collect some system preference values. -If you want to tell us who you are, you can fill the UsageStatsLibraryName system preference with your library name, UsageStatsLibraryUrl, UsageStatsLibraryType and/or UsageStatsCountry. +If you want to tell us who you are, you can fill the UsageStatsLibraryName system preference with your library name, UsageStatsLibraryUrl, UsageStatsLibraryType and/or UsageStatsCountry, UsageStatsLibrariesInfo. All these data will be analyzed on the http://hea.koha-community.org Koha community website. diff --git a/t/db_dependent/UsageStats.t b/t/db_dependent/UsageStats.t index 7f5d543049..33af62dacd 100644 --- a/t/db_dependent/UsageStats.t +++ b/t/db_dependent/UsageStats.t @@ -15,11 +15,13 @@ # with Koha; if not, see . use Modern::Perl; -use Test::More tests => 57; +use Test::More tests => 59; use t::lib::Mocks qw(mock_preference); use POSIX qw(strftime); use Data::Dumper; +use Koha::Libraries; + BEGIN { use_ok('C4::UsageStats'); use_ok('C4::Context'); @@ -78,20 +80,22 @@ is( $update, 0, "Last update just be done, no update needed " ); #mock to 0 t::lib::Mocks::mock_preference( "UsageStatsID", 0 ); t::lib::Mocks::mock_preference( "UsageStatsLibraryName", 0 ); -t::lib::Mocks::mock_preference( "UsageStatsLibraryUrl", 0 ); +t::lib::Mocks::mock_preference( "UsageStatsLibrariesInfo", 0 ); t::lib::Mocks::mock_preference( "UsageStatsLibraryType", 0 ); t::lib::Mocks::mock_preference( "UsageStatsCountry", 0 ); +t::lib::Mocks::mock_preference( "UsageStatsLibraryUrl", 0 ); my $report = C4::UsageStats->BuildReport(); isa_ok( $report, 'HASH', '$report is a HASH' ); isa_ok( $report->{library}, 'HASH', '$report->{library} is a HASH' ); -is( scalar( keys %{$report->{library}} ), 5, "There are 5 fields in $report->{library}" ); +is( scalar( keys %{$report->{library}} ), 6, "There are 6 fields in $report->{library}" ); is( $report->{library}->{id}, 0, "UsageStatsID is good" ); is( $report->{library}->{name}, '', "UsageStatsLibraryName is good" ); is( $report->{library}->{url}, '', "UsageStatsLibraryUrl is good" ); is( $report->{library}->{type}, '', "UsageStatsLibraryType is good" ); is( $report->{library}->{country}, '', "UsageStatsCountry is good" ); +is( $report->{library}->{number_of_libraries}, undef, "UsageStatsLibrariesInfo is good" ); #mock with values t::lib::Mocks::mock_preference( "UsageStatsID", 1 ); @@ -99,17 +103,20 @@ t::lib::Mocks::mock_preference( "UsageStatsLibraryName", 'NAME' ); t::lib::Mocks::mock_preference( "UsageStatsLibraryUrl", 'URL' ); t::lib::Mocks::mock_preference( "UsageStatsLibraryType", 'TYPE' ); t::lib::Mocks::mock_preference( "UsageStatsCountry", 'COUNTRY' ); +t::lib::Mocks::mock_preference( "UsageStatsLibrariesInfo", 1 ); $report = C4::UsageStats->BuildReport(); isa_ok( $report, 'HASH', '$report is a HASH' ); isa_ok( $report->{library}, 'HASH', '$report->{library} is a HASH' ); -is( scalar( keys %{$report->{library}} ), 5, "There are 5 fields in $report->{library}" ); +is( scalar( keys %{$report->{library}} ), 6, "There are 6 fields in $report->{library}" ); is( $report->{library}->{id}, 1, "UsageStatsID is good" ); is( $report->{library}->{name}, 'NAME', "UsageStatsLibraryName is good" ); is( $report->{library}->{url}, 'URL', "UsageStatsLibraryUrl is good" ); is( $report->{library}->{type}, 'TYPE', "UsageStatsLibraryType is good" ); is( $report->{library}->{country}, 'COUNTRY', "UsageStatsCountry is good" ); +my $nb_of_libraries = Koha::Libraries->count; +is( $report->{library}->{number_of_libraries}, $nb_of_libraries, "UsageStatsLibrariesInfo is good" ); #Test report->volumetry --------------- #with original values -- 2.39.5