Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.
To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This followup
- changes some indexes in Queryparser configuration file
- supresses some clearly useless 6XX$9 in biblio-koha-indexdefs.xml and adds 2 new ones, probably useless (not sure of that)
- change the name of index Subject-geographical to Subject-name-geographical in ccl.properties (to match bib1.att)
the xsl file zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl was generated with the following command:
xsltproc zebradb/xsl/koha-indexdefs-to-zebra.xsl zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml > zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl
To test :
1) Apply the 3 patches
2) copy the modified files from the source directory to the directory where you store the config files for Zebra and Queryparser
The files modified by the 3 patches and that need to be copied are:
etc/zebradb/biblios/etc/bib1.att
etc/zebradb/ccl.properties
etc/searchengine/queryparser.yaml
etc/zebradb/ccl.properties
.../unimarc/biblios/biblio-koha-indexdefs.xml
.../unimarc/biblios/biblio-zebra-indexdefs.xsl
3) Rebuild Zebra
4) Create a record A with some values in critical fields, for example:
- the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2
- the string "subform" in 600$j
4) Create a record B with the string "subgeo" in 606$y
5) Create a record C with the string "subdate" in 606$z
WITHOUT QP activated in sysprefs ("Don't try to use QP"):
6) try to search "su:test9828". You should have no results
7) try to search "su-genre:subform". You should have 1 result : record A
8) try to search "su-geo:subgeo". You should have 1 result : record B
9) try to search "su-chrono:subdate". You should have 1 result : record C
10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant
WITH QP activated in sysprefs:
Same tests
Signed-off-by: Nick Clemens <nick@quecheelibrary.org>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Since nobody is currently working on the zebra layer introduced by bug
8233, Solr won't never work.
Some code has been introduced in 3.10 to prove several search engines
can cohabit into Koha but no help/fund has been found to go ahead.
It is useless to keep this code and to maintain an ambiguous situation.
I think the indexes configuration page could be restore later if someone
else introduces a new search engine into Koha.
Test plan:
Look at the code introduced by bug 8233 and verify all is removed.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
This patch makes the following changes to UNIMARC biblio indexing :
A. Changes to UNIMARC conf files
1. add comments to biblio-koha-indexdefs.xml
2. make biblio-koha-indexdefs.xml more compact by grouping some
declarations
Ex : 200$f and 200$g => one declaration for 200$fg
3. suppress unneeded declarations (indexing of some 4XX fields and 6XX
fields not in unimarc format)
4. unindex some (sub)fields unneeded by most users (318, 207,230,210a,
215, 4XXd)
5. change the way 308 field is indexed (no visible changes)
6. replace Title-host with Host-item -- see bug 11119
7. index 208 in Material-Type -- see bug 11119
8. index 100 pos 8-9 and 9-12 in pubdate:y and pubdate:n
9. index 100 pos 8-9 in pubdate:s instead of 210$d
10. Index all subfields of note 334 and 327 in note index
11. Index 304 and 327 in title index as well as note index
327 can contain a list of titles included in a work
304 can contain the title of the original work in case of a
translation
12. Index 314 in author index as well as note index
314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc.
author)
13. Index 328 note in Dissertation-information as well as note
14. Index 328$t in Title
B. Changes to ccl.properties :
1. add a new index Dissertation-information (1056)
2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes)
C. Changes to Search.pm
1. add Dissertation-information and suppress Title-host and UPC
D. Changes to QP config file queryparser.yaml
1. add Dissertation-information
2 fix EAN, pubdate and acqdate
Test plan :
If you cannot test in GRS1, test only in DOM, as GRS will be deprecated.
1. Apply the patch in a UNIMARC Koha running with DOM and ICU
2. copy src/etc/searchengine/queryparser.yaml into the main config
directory of QP
3. copy src/etc/zebradb/ccl.properties into the main config directory
of Zebra
4. copy src/etc/zebradb/marc_defs/unimarc/biblio/* into the main config
directory of Zebra
5. reindex biblios (rebuild_zebra.pl -r -b -x -v)
6. test note index : make some searches on 334$b or 327$b
7. test author index : make some searches on 314 field
8. test title index : make some searches on 304 and 327 field, make a
search on 328$t subfield
9. test dissertation-information index : make some searches on 328 field
10. In a record, put in the dates of 100 fields the values "1000" (1st
date) and "1001" (2d date) ; try to search a book written in year
1000, you should find the record ; idem for year 1001
11. make some searches and sort by date. It should work better as before,
especially if you have values like "c2009" or "impr. 2010" in 210
field
12. Regression test : make some searches on several indexes, like EAN,
etc. It should work as before
Test 10-12 with and without Queryparser activated.
Be careful: with Queryparser activated, the index names (title,
dissertation-information...) must be entered in lowercase only.
Of course, to test search and sort by dates, you need to have full
records, with dates in 100 field as well as 210 field.
Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch adds the Number-local-acquisition into QueryParser
configuration file.
Like in ccl.properties, "Number-local-acquisition" is the main index
name and "stocknumber" and "inv" are aliases.
Test plan :
Enable QueryParser :
- Enable UseQueryParser syspref
- Edit your koha-conf.xml
- Add to "config" node : <queryparser_config>[your path]/etc/searchengine/queryparser.yaml</queryparser_config>,
adapt [your patch] to your install configuration folder
- If needed copy from sources "etc/searchengine/queryparser.yaml" into
your install configuration folder
Test search :
- Add Number-local-acquisition on an existing subfield in records.abs.
For example on item barcode field
- Reindex Zebra database
- Choose a value of this field that will match some results. For
example : "0*" will match all barcodes beginning with zero
- In intranet, enter this URL : <your server>/cgi-bin/koha/catalogue/search.pl?idx=stocknumber&q=0*&sort_by=relevance
=> You get some results
- In intranet, enter this URL : <your server>/cgi-bin/koha/catalogue/search.pl?idx=inv&q=0*&sort_by=relevance
=> You get the same results
- In intranet, enter this URL : <your server>/cgi-bin/koha/catalogue/search.pl?idx=number-local-acquisition&q=0*&sort_by=relevance
=> You get the same results
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comments on case sensitivity of index names in QueryParser, see Bugzilla.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Corrects a double entry for language in yaml file.
Language should have been language-original.
Test plan:
Check that you have language-original in your zebra install.
Specifically, this index should cover MARC21 041$h.
Enable QueryParser and search for a record with this index.
Note that this patch does not enable searching on this
index without QueryParser. This is true for many more indexes
in record.abs that are not included in the getIndexes routine.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described - make sure you are testing with a current
indexing configuration.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch adds language-original to the list of search fields
recognized by QueryParser.
To test:
[1] After doing the tests in the main patch, copy the configuration
file etc/searchengine/queryparser.yaml into place, turn on the
UseQueryParser system preference, and verify that searching on
language-original still works.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch fixes a problem where if a staff member sets the
*defaultSortField/*defaultSortOrder system preferences to relevance
ascending while QueryParser is enabled, default keyword search
would break -- the query parser config did not declare relevance asc
as a possible "modifier".
Note that setting the sort order to relevance ascending does not
actually make catalog search return results with the least relevant
records showing up first; Zebra does not support such a mode. In other
words, relevance ascending acts exactly the same as relevance descending.
Test plan:
0/ Create some biblio with "history" in the title and
ensure that the QueryParser system preference is enabled.
1/ Define prefs defaultSortField = relevance and defaultSortOrder = asc
2/ Search "history" on the staff interface
3/ Note that no result is returned.
4/ Apply the patch
5/ Verify the queryparser config file in use takes the modification into
account (see the queryparser_config value in your $KOHA_CONF file).
6/ Relaunch the search and verify results are returned
Signed-off-by: Christopher Brannon <cbrannon@cdalibrary.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Remove a line of debug code from EG, provide better error handling
when presented with weird data in the authority linker, and correct
queryparser configuration to reflect the correct configuration for
Zebra.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
This patch also corrects the definition of the an= index, which was
missing exactness.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Since the most expressive query language supported by Zebra is PQF, this
patch adds a PQF driver for QueryParser which will translate QueryParser
queries into standard PQF (guided by mappings which have been written to
match Koha's existing Zebra configuration) which can then be sent to
Zebra. This driver, Koha::QueryParser::Driver::PQF(::*) extends the
OpenILS::QueryParser(::*) class(es), so as to preserve maximum
interoperability between the various users of the QueryParser driver.
Initially, search syntax is as follows:
* AND operator: &&
* OR operator: ||
* GROUPING operators: ( )
Fields can mostly be searched using the ccl prefixes they have now. The
exception is the various date limits which are searched with a syntax
like this: pubdate(2008)
For sorting, you can simply add #title-sort-az (etc.) to your query.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Elliott Davis <elliott@bywatersolions.com>
Test Passed successfully after installing missing dep for Test::Deep
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
