To test:
1 - Set SearchEngine syspref to Zebra
2 - Be a user with permission to manage search engine configuration (manage_search_engine_config)
3 - Confirm you do not see 'Search engine configuration' on Admin main page
4 - Apply patch
5 - Confirm you see it with '(Elasticsearch)' appended
6 - Be a user without above permission
7 - Confirm you cannot see the 'Search engine configuration'
8 - Confirm you cannot access directly:
/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.
How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.
Sponsorded-by: Gothenburg Univesity Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code
To test:
1 - Make sure you have records with differing languages (in the MARC21 008
field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch adds a new section 'Facet order' in the Biblio tab of the
'Search engine configuration' admin page of the Elastic mappings.
The idea is to let the librarians define the facet to display and order
them as their needs.
The ergonomic is not perfect and I am open to any suggestions.
Test plan:
Move up and down the field list to order the facets
Hide/show some facets
Rebuild index
At the OPAC and the staff interface you should see the changes on the
search result page.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Actual translation of ElasticSearch mappings page contains a huge block with a lot of %s.
It makes translation hard.
This patch uses new method from Bug 15395 to split into several translation blocks.
Test plan :
1) Apply patch "Bug 22080: [DO NOT PUSH] test code"
2) Apply this patch
3) Enable ElasticSearch in SearchEngine preference
4) Go to Administration > Search engine configuration
5) Check page is OK
6) Update the PO files of a language : perl translate update xx-YY
7) Write translation strings in misc/translator/po/xx-YY-messages.po
8) Install translation : perl translate install xx-YY
9) Check translated page
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Add persistent per index "index status" state to provide useful
user feedback when update of Elasticsearch server mappings fails
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Weight can be defined between 1 and 99.
only search fields mapped with biblios can be weighted
Test plan:
- apply this patch,
- update schema file (perl misc/devel/update_dbix_class_files.pl),
- go to Administration > Search engine configuration,
- in search fields tab, define weights for some fields and save,
- check weights has been saved,
- reset some weights (empty) and save,
- check weights has been reset
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
The CSS for <div class="error"> is obsolete and should not be used. This
patch removes the definition from the main CSS file and corrects
instances of its use in the templates to the standard <div class="dialog
alert">.
To test:
- In Acquisitions -> Late orders, locate an order from a vendor which
doesn't have an email address. Selecting that order and clicking
"Claim" should trigger an error dialog, "This vendor has no email." It
should be styled correctly.
- With AcqCreateItem set to "when placing an order," add to an existing
order using the "From a new (empty) record" option. Add two items with
identical barcodes and submit the form. A error should show at
the top of the page.
- With AcqCreateItem set to "when receiving an order," receive an order
and add two items with identical barcodes. Submitting the form should
trigger an error message at the top of the page.
With the remaining cases I don't know how to trigger the errors in
question, so a visual check of the changes may be required:
- Administration -> Funds -> "You are not authorized to modify this
fund"
- Administration -> Search engine configuration
(/admin/searchengine/elasticsearch/mappgings.pl) -> Various
modification errors.
- With the AutoEmailOpacUser preference set to "send," adding a patron
without an email address can trigger an error, "This member has no
email."
- With plugins enabled, and installed, there are error messages
displayed under various circumstances.
Signed-off-by: Andrew Isherwood <andrew.isherwood@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
To Test:
Check the following files that all all "biblio" or "biblo record" are
now writen in full as "bibliographic record".
modules/admin/authtypes.tt
modules/admin/biblio_framework.tt
modules/admin/preferences/authorities.pref
modules/admin/preferences/cataloguing.pref
modules/admin/preferences/circulation.pref
modules/admin/preferences/opac.pref
modules/admin/preferences/searching.pref
modules/admin/preferences/serials.pref
modules/admin/preferences/web_services.pref
modules/admin/searchengine/elasticsearch/mappings.tt
Corrected a single capitalization error during signoff
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch modifies several administration templates to use the
Bootstrap grid instead of YUI.
This patch also removes obsolete "text/javascript" attributes from
<script> tags and "text/css" attributes from <style> tags in the
modified templates.
To test, apply the patch and view the following pages, confirming that
they look correct at various browser widths:
- Navigate directly to /cgi-bin/koha/admin/printers.pl
- View and edit printers
- Administration -> Search engine configuration (or navigate directly
to /cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl
- Administration -> Circulation and fines rules
- Administration -> SMS cellular providers (enable SMSSendDriver or
navigate directly to /cgi-bin/koha/admin/sms_providers.pl)
- Administration -> Z39.50/SRU servers -> New SRU server -> Modify SRU
search fields mapping
- Administration -> System preferences -> Local use
- View and edit local use preferences
- Administration -> Transport cost matrix
- With UsageStats enabled, go to Administration -> Share your usage
statistics.
Signed-off-by: Claire Gravely <claire.gravely@bsz-bw.de>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
- Remove useless 'action' attribute in forms
- $op should never be equal to 'reset', so remove the test
- Move 'op' parameter inside <button>s to avoid having different HTML
tags for elements that have the same kind of behaviour
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.
To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Here we go, next step then.
As we did not fix the performance issue when autofiltering
the variables (see bug 20975), the only solution we have is to add the
filters explicitely.
This patch has been autogenerated (using add_html_filters.pl, see next
pathces) and add the html filter to all the variables displayed in the
template.
Exceptions are made (using the new 'raw' TT filter) to the variable we
already listed in the previous versions of this patch.
To test:
- Use t/db_dependent/Koha/Patrons.t to populate your DB with autogenerated
data which contain <script> tags
- Remove them from borrower_debarments.comments (there are allowed here)
update borrower_debarments set comment="html tags possible here";
- From the interface hit page and try to catch alert box.
If you find one it means you find a possible XSS.
To know where it comes from:
* note the exact URL where you found it
* note the alert box content
* Dump your DB and search for the string in the dump to identify its
location (for instance table.field)
Next:
* Ideally we would like to use the raw filter when it is not necessary
to HTML escape the variables (in big loop for instance)
* Provide a QA script to catch missing filters (we want html, uri, url
or raw, certainly others that I am forgetting now)
* Replace the html filters with uri when needed (!)
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.
To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 20073: Move Elasticsearch yaml files back to admin directory
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This reverts commit f489d2034b.
This commit breaks the install process when using debian packages.
Reverting as we are very close to the 18.05.00 release
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds a link to the Elasticsearch configuration page in the
administration sidebar menu when Elasticsearch is enabled.
This patch also corrects the title of the Elasticsearch page to match
the page heading. "Search engine configuration" instead of
"Elasticsearch mappings."
To test, apply the patch and set the SearchEngine system preference to
"Elasticsearch."
- View any page in the administration module which shows the
left-hand sidebar menu. The link to search engine configuration
should be there.
- View the search engine configuration page and confirm that the page
title is correct.
- Set the SearchEngine system preferenc to "Zebra."
- The search engine configuration link should no longer appear in the
administration sidebar menu.
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.
To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Having to write [% KOHA_VERSION %] for each url is bad because:
- It's easily forgettable when adding new <script> or <link>
- It prevents grep'ing for the full filename
- It violates the DRY principle
- If at some point we want to change the "force js and css reload"
mechanism, it will be tedious
This patch:
- adds a Template::Toolkit plugin that generates <script> and
<link> tags for JS and CSS files, and inserts automatically the Koha
version in the filename
- use the new plugin to remove all occurences of [% KOHA_VERSION %]
- remove the code that was adding KOHA_VERSION as a template variable
Test plan:
1. Apply patch
2. Go to several different pages in Koha (opac and intranet) while
checking your browser's dev tools (there should be no 404 for JS and
CSS files, and the Koha version should appear in filenames) and the
server logs (there should be no "File not found")
3. `git grep KOHA_VERSION` should return nothing
4. prove t/db_dependent/Koha/Template/Plugin/Asset.t
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch updates a warning message and error message in the Search
engine configuration for Elasticsearch.
Note: elasticsearch does not need to be active in order to test.
To test warning message:
1. go to: cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl
2. note the warning message
3. Apply patch
4. Note the change in warning message. Confirm it makes sense.
To test error message:
Apply patch and check the changed message in the template file mappings.tt and confirm it makes sense.
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
I ran:
$ git grep -l cat-search.inc | grep admin
This means I believe the outstanding ones are
(koha-tmpl/intranet-tmpl/prog/en/modules/):
- admin/auth_subfields_structure.tt
- admin/clone-rules.tt
- admin/marc_subfields_structure.tt
- admin/searchengine/elasticsearch/mappings.tt
One other was recommended by Katrin in comment #9:
- plugins/plugins-home.tt
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Typo found when translating in:
intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt:74
An error occurred when updateing mappings => updateing
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Also some close tag </selected> should be </select>
Follow previous patch to test
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Test plan
1) Have SearchEngine syspref set to Elastic
2) Look at source on admin/searchengine/elasticsearch/mappings.pl
-> without patch you'll see select with attribute
data-id="mapping_search_field_name" closing tag written badly as
"/select>"
-> with patch everything looks OK
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This new page (admin/searchengine/elasticsearch/mappings.pl) will permit
to manage the ES mappings.
For the biblios and authorities indexes, the different mappings can be
managed from this single page.
The interface let you add, remove and update mappings and search fields.
It's also possible to reorder the mappings, as the order can be important
in the indexation process. Note that the table can be displayed in a
different order that the one it was before saving, but the mappings are grouped
by search field and the order inside the search field is preserved.
Limitations:
- If something went wrong during the insertion/deletion/modification,
the users will loose all these changes.
TODO:
- Add a specific permission (?)
- Add some data checks client side (JS)
- Use checkboxes for facet and suggestible (lazy today...)
- Understand the difference between the 3 values that sortable can have
and improve the value for the options in the select box.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
