Extension to bug 36269: 1) when cataloguing old prints, the standardized form of
the place of publication is given in the field 752 and as such should also be indexed.
2) Also, normal UNIMARC fields for publication place should be indexed (210a, 214a).
Test plan
=========
Scenario A (MARC 21, field 752)
-------------------------------
0. Have an installation with Elasticsearch and MARC 21 configuration.
1. Activate subfields 752 $a, $d in the default framework (Visibility: Editor).
2. Edit an existing record with this framework, putting some rare words in the
752 $a and $d subfields.
3. Try to make a search with tis words (general keyword or by publication place,
i.e. pl:).
4. You hsold get no results.
5. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
6. Repeat p. 3. You should get your edited record.
Scenario B (UNIMARC)
--------------------
0. Have an installation with Elasticsearch and UNIMARC configuration.
1. Try to search for a publication place, either with pl: directly in the
search window, or in Advanced search, with Publisher location.
With ktd UNIMARC test date you can search for 'Ste Geneviève Cedex'.
2. You should get no results.
3. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
4. Repeat p. 1. You should get some results (with ktd test data set and
Ste Geneviève Cedex - record #1).
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new index for authorities, Other-control-number, and maps it to 035$a.
This will help when trying to match authority records when importing external records.
Test 1:
0. Make sure Elasticsearch is set as the search engine
1. Apply patch and restart
2. Import the attached record
2.1. Go to Cataloging > Stage records for import
2.2. Choose the file
2.3. Click Upload file
2.4. Choose Record type: Authority
2.5. Click Stage for import
(wait until the job is finished...)
2.6. Click View batch
2.7. Click Import this batch into the catalog
(wait until the job is finished...)
2.8. Click Manage imported batch
2.9. Click View next to the record
2.10. Note the auth id number
3. Examine the ES entry for the record (replace INDEX_NAME with the index name (found in koha-conf.xml and XX with the auth_id)
curl -XGET 'http://localhost:9200/INDEX_NAME_authorities/data/XX?_source_includes=other-control-number&pretty'
--> It should give you the value of 035$a
Test 2 (optional):
1. Steps 1 and 2 as above
2. Add a matching rule to match on 035$a for authority records
2.1. Go to Administration > Record matching rules
2.2. Click New record matching rule
2.3. Fill out the form
- Matching rule code: enter a code (for example AUTCONTROL)
- Description: enter a description (for example 035$a for authorities)
- Match threshold: 100
- Record type: Authority record
- Search index: Other-control-number
- Score: 100
- Tag: 035
- Subfields: a
2.4. Click Save
3. Import the same record again, checking for matches using the new rule
3.1. Go to Cataloging > Stage records for import
3.2. Choose the file
3.3. Click Upload file
3.4. Choose Record type: Authority
3.5. Choose Record matching rule: rule created above
2.6. Click Stage for import
(wait until the job is finished...)
--> It should say that 1 record was found using the rule
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
For MARC 21, canceled/invalid ISBNs (subfield 020 $z) are not being
indexed in Elasticsearch (search fields identifier-standard and isbn),
but it is for UNIMARC (subfield 010 $z). As a result, a user searching
for such an ISBN (e.g. with wrong checksum--and therefor put in 020 $z)
will not find the record in the catalogue.
A new isbn-all Elasticsearch search field 'isbn-all' has been created with
020az (MARC 21, and 010az for UNIMARC), 010z has been removed from isbn
search field for UNIMARC, and 020z has been added to identifier-standard
search field.
Test plan
=========
0. Have a test installation with Elasticsearch.
1. In ktd with its test data, make a biblio search for a cancelled ISBN
e.g. 9780007269854. You will get no results.
2. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
3. Repeat the test. You should get "The ice princess / Camilla Läckberg".
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We want the label from the DB displayed on the UI so we are adjusting
the yaml and the DB.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature allows to manage facets for Elasticsearch from the
administration page.
Prior to this the facet fields were hardcoded in the codebase.
Librarians can then add/remove facet fields and modify their label.
Test plan:
1. Create a new search field and set it a label different than its name.
2. Save
3. Go the bibliographic mapping tab
4. Add 1+ mapping for this search field (Searchable and facetable must be "Yes")
5. Add, reorder, remove new facets
6. Save and reindex your records
7. Search and notice the new facet list
QA: There are several wrong things in this area (ES + facets code,
everything, pm, pl, tt AND on this administration page). I have done my
best to clean the code as much as I could and keep the code cleaner
after than before. But there are still a lot to do.
There are still inconsistencies on this page (like we need to save to
see the changes applied to the other tables), but this is clearly out of
the scope of this bug report.
Another enhancement would be to move the facet list to a different DB
table, that could bring more flexibility:
* display or not (could be opac/staff/both/none)
* define the size per field (number of facet to display)
* order: move search_field.facet_order to this new table.
But, again, it's a lot more work.
More work is done in this area, please see related bug reports.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
- apply this patch,
- check that SearchEngine system preference is set to Elasticsearch,
- go to Admin > Search engine configuration,
- on the search fields tab, fill a new line at the bottom of the table
(name, label, type)
- click on the "Add" button and save,
- check that the new search field has been saved,
- also test field deletions,
- check that you can't delete already mapped fields.
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: David Schmidt <davewood@gmx.at>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently the index `pl` points to 008/15-17. It should
(additionally/instead?) point to 260a and/or 264a.
Test plan (for koha-testing-docker with ElasticSearch via `ktd --es7 up`)
Verify the old/broken behaviour:
* Go to Staff/Advanced Search
* Select "Publisher Location" and enter "cau", start search
* You will get some hits (~16), eg "Perl best practices / Damian Conway", which has 008 of "051222s2005 caua b 001 0 eng d" with "cau" on 15-17
* Edit this record (cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=5)
* Inspect 260$a, which should containt "Sebastopol, CA :"
* Go back to Advanced Search and search for "Publisher Location" = "Sebastopol"
* No hits!
Apply the patch!
* exit ktd and stop it (ktd --es7 down; ktd --es7 stop)
* start it again
* Go to Staff / Administration / Search Engine Config (Elasticsearch)
* Click on the Tab "Bibliographic records" and search/filter for "pl"
* you should see 3 entries for "pl", with Mapping values of "008_/15-17", "260a" and "264a"
* Go to Search,
* Select "Publisher Location" and enter "cau", start search
* same hits as befor
* Search again, but for "Sebastopol"
* Now you'll get 9 hits!!
Bonus: Test 264a
* Got to Admin / MARC bibl framework, select BKS -> MARC structure
* search for tag "264", edit subfields
* in tabs a, b, c: Check the "Editor" Checkbox (Visibility)
* Save changes
* find a book, eg again "Perl best practices" and edit it
* find field 264 and enter "Test" into 264a, Save
* Depending on your setup, you might have to manually re-index the book:
* enter ktd: ktd --shell
* reindex the one book (via --bn) or all (might also need a -d)
/usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl -v -b -d
* Go again to Advanced Search, Publisher Location = "Test"
* You should find the book
If your NOT running ktd, you might be able to just edit the ElasticSearch Mappings to add / change the mapping for "pl" to point to "260a".
One rather harsh way to do this (which I needed to do, because the ES Mappings UI did not work for me) is via this SQL:
update search_marc_map set marc_field='260a' where marc_field='008_/15-17';
Sponsored-by: Steiermärkische Landesbibliothek
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
opac-shelves - forms were stateless - changed to GET
sco-main - forms stateless - changed to GET
** untested because sco + auth is broken
smart-rules.tt - JS form template - added placeholder 'cud-' op
ill-requests - added cud- tp ops
** tested comments, but not checkout, simple changes
boraccount - removed repeated op - updated script
patroncards/edit-batch - add placeholder 'cud-' op
patroncards/manage - add placheholder 'cud-' op
elasticsearch/mappings - separate forms - this could use a style follow-up, but makes more sense separate I think
reports/dictionary - stateless - changed to GET
guided_reports_start - stateless - changed to GET
suggestion/suggestion - add placeholder 'cud-' op
inventory - filed bug 36305, needs more handling
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch finally adjust the default mappings to add a new field to
the elastic index with the title stripped of punctuation
This method optionally allows the library to place the filtered version in the same
search field, or a new search field. The default mappings will include the filtered version
in a keyword search, but not a targeted search
To test:
1 - Import some affected records via z3950, e.g.:
Carrie's war
1,000 Japanese words
2 - Search (using Elasticsearch) for the titles without including punctuation
Carries war
1000 Japanese words
3 - No results
4 - Reload mappings and reindex
perl misc/search_tools/rebuild_elasticsearch.pl -v -r
5 - Search again
6 - Success!
7 - Search title specifically:
ti:Carries war
8 - No results
9 - Adjust mappings.yaml to add second mapping for 245 to title index:
- facet: ''
marc_field: 245abp
marc_type: marc21
sort: 1
suggestible: 1
filter: punctuation
10 - Reload mappings and reindex
11 - Repeat 7
12 - Success
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds a new column to the 'Bibliographic records' tab in
Administration > Search engine configuration (Elasticsearch)
To test:
1 - Confirm the new 'filter' column shows
2 - Update an existing field to set filter to punctuation
3 - Confirm it can be saved
4 - Add a new field
5 - Confirm it saves correctly
6 - Unset filter for a field
7 - Confirm it saves
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
At some point the option for 'undef' was removed from te sort options
and was collapsed to yes/no
The dropdowns when adding a new field were missed, this patch corrects that.
While undef in a mappings file wil stil load, when saving we should not privde undef any longer
To test:
1 - Browse to bottom to add a new field on the 'Bibliographic records' tab in
Administration > Search engine configuration (Elasticsearch)
2 - Set sortable column to undef, set other columns and provide a valid field
3 - Click '+Add'
4 - Click 'Save'
5 - At top of page you receive an error:
An error occurred when updating mappings: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Column 'sort' cannot be null at /kohadevbox/koha/Koha/SearchField.pm line 37 .
6 - Apply patch, restart all
7 - Add a new mapping, your only choices are Yes/No
8 - Save mapping
9 - Confirm it saves correctly
Signed-off-by: Salah Ghedda <salah.ghedda@inLibro.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch simply remvoes sort from all elements that are not strictly
the main title
Note:
If multiple fields are set as sort, they are collapsed into a single entry in
the {field}__sort field in the ES index. The order will be determined by the order in
the marc record
To test:
1 - Apply patch
2 - perl misc/search_tools/rebuild_elasticsearch -r -v
3 - Search the catalog
4 - Sort by title
5 - Confirm records are correct
6 - Add a 240 (before the 245) with subfield a 'AAAAA'
7 - Confirm sorting is not affected
8 - View record details, click 'Elasticsearch record: Show'
9 - Find 'title__sort' and confirm it looks correct (does not include AAAAA)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In Elasticsearch fields config field_config.yaml, default type as a field 'ci_raw'. This is used for exact search.
This field is missing for type standard number 'stdno'.
Test plan :
1) In the staff interface, go to Administration, and search for SearchEngine
2) Make sure that the SearchEngine preference is set to Elasticsearch and save
3) Return to Administration and select "Search engine configuration"
4) Change the type of "Heading-Main" to "Std. Number" and save
5) Rebuild the index (e.g. "koha-elasticsearch --rebuild -d kohadev")
6) Go to the main staff page and select Authorities
7) Search for a heading (e.g. "A Dual-language book")
=> Result is found with or without patch
8) Click on the sliders and select "is exactly" for the operator and search
=> Result is found only with patch
9) Apply the patch
10) Rebuild the index (e.g. "koha-elasticsearch --rebuild -d kohadev")
11) Click on the sliders and select "is exactly" for the operator and search
=> Result is found only with patch
Signed-off-by: Kevin Carnes <kevin.carnes@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Before this patch we used two indexes for the thesaurus values, we can
simply index both needed fields into a single index and just form the
search correctly.
This patch also ensures we pass the 'thesaurus' vlaue for the heading
directly to the query builder - for zebra it goes through, and for ES
we convert it to the expected code.
This patch also moves the necessary mappings out of the user definable
mappings and hardcodes them. There is precedent for this with
'match-heading', it ensures matching works as expected
To test:
1 - Follow previous test plan in Zebra and ES
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Currently we only index a - but we can setup the system such that avxyz are searched
To test:
1 - define both a 655$a *and* 655$x value in a bib, save, reindex
2 - Set system preferences:
TraceSubjectSubdivisions: Include
TraceCompleteSubfields: Force
3 - View the record edited above in the opac
4 - Click on the subject heading
5 - No results found
6 - Copy zebra files:
sudo cp ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml \
/etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
sudo cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
/etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
7 - restart all and reindex
8 - Click on the subject heading in OPAC
9 - Sucess!
10 - Repeat with other fields (vyz)
11 - Repeat under ES, reindexing and resetting mappings
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds the following fields to the See-from index
- 450(abvxyz)
- 451(avxyz)
- 455(avxyz)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds the following fields to the Match-heading-see-from index
- 430(adfghklmnoprstvxyz)
- 448(avxyz)
- 450(abvxyz)
- 451(avxyz)
- 455(avxyz)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds see from and see also from terms for uniform title,
chronological term, topical term, geographic name, and genre/form term
to the Match index in Elasticsearch for MARC21.
Previously, only see from/see also from for personal names,
corporate names, and meeting names were indexed.
To test:
1. Without patch, import attached authority records
1.1. Download attached records
1.2. Go to Tools > Stage MARC records for import
1.3. Click 'Browse' and choose the downloaded file
1.4. Click 'Upload file'
1.5. Choose Record type = Authority
1.6. Click 'Stage for import'
1.7. From the job details, click 'View batch'
1.8. Click 'Import this batch into the catalog'
2. Without patch, search for see from and see also from tracings
2.1. Go to Authorities
2.2. In the 'Default' drop-down menu, choose 'Uniform title'
2.3. Choose the 'Search all headings' tab
2.4. Enter the search term 'Five hundred'
2.5. Click 'Submit' or press 'Enter'
--> No results
2.6. Redo the search for the following search terms
Authority type Search term Should be found in
Uniform title five hundred 430
Uniform title films préférés 530
Chronological term fifteenth 448
Chronological term middle ages 548
Topical term lalopathie 450
Topical term troubles communication 550
Geographic name cécropia 451
Geographic name canada francophone 551
Genre/Form term filmiques 455
Genre/Form term films 555
3. Apply patch
4. Delete index, reset mappings and reindex authorities (with command line, -a -d -r)
5. Redo the searches from step 2, there should now be results
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex
Sponsored-by: Lund University Library, Sweden
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add a new boolean ES field named 'available', which is true if at least
one item is available, which means the item is not on loan, not
"notforloan", not withdrawn, not lost and not damaged
A full indexation is required
Test plan:
1. Apply patch and run updatedatabase.pl
2. Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
3. Make sure you have some biblios whose items are all unavailable, some
biblios whose items are all available, and some biblios with at least
one item available and at least one item unavailable
4. Use the 'available' filter on both opac and intranet and make sure it
works as expected.
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch removes one of the two mappings for the 003 field to the
control-number-identifier index (for MARC21).
To test:
1) Apply patch
2) reindex with mappings reset
3) try to search for cni:code (for example cni:OSt)
--> it should return the desired results
4) try to search for control-number-identifier:code (for example
control-number-identifier:OSt)
--> it should return the desired results
5) Optionally, try the test plan in Bug 11175 to make sure it still
works
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Repeat previous tests with Elasticsearch engine
You will need to reindex and reset mappings to pickup the changes form the file
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Elasticsearch number of results is by default limited by setting "index.max-result-window", default value is 10000.
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-result-window
We use this setting:
44d6528b56/Koha/SearchEngine/Elasticsearch/Search.pm (L411)
I propose we add this setting in index config.
Test plan:
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes: misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check the settings of index (when using koha-testing-docker*):
curl 'es:9200/koha_kohadev_biblios/_settings?pretty&filter_path=**.max_result_window'
5) You should see:
"max_result_window" : "1000000"
* You also need to add this setting to the es section in koha-testing-docker's
docker-compose.yml (after the networks configuration):
ports:
- "9200:9300"
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When defining our sort fields in we defined all as 'numeric'
For other string containing numbers this is likely correct, however,
for callnumbers it is not. e.g. E45 should sort before E7
This patch adds a new 'callnumber' type and deifnes this for cn-sort and
adds to the field maping a sort without numeric set
To test:
0 - Be using ES with Koha
1 - On records with single item, add callnumbers:
VA65 E7 R63 1984
VA65 E7 T35 1990
VA65 E45 R67 1985
2 - Add public note 'shrimp' or something to make them easily searchable as a group
3 - Search for 'shrimp', sort by callnumber
4 - Note E45 comes last, it should come first
5 - Apply patch
6 - Reset ES mappings
7 - Reindex ES
8 - Repeat search
9 - Sorting should be correct when set to callnumber
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
A first step to "validate" the MARC mappings: Remove all whitespace, so
if a user enters "245a " (with a trailing whitespace, which can easily
happen when copy/pasting) we only store "245a" in the DB. This is
neccessary, because the ES indexer will throw an exception in an invalid
MARC mapping.
Test Plan:
* Go to /cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl
* Go to the Bibliographic Records Tab
* Enter "100 a b c " (notice the whitespaces!) in the first "mapping"
field
* Scroll down and save
* Go back to the Bibliographic Records Tab
* The spaces are still there
Now apply the patch
* Repeat the above steps
* After saving you should see "100abc" without any spaces in the
"mapping" field
Sponsored-by: Steiermärkische Landesbibliothek
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
The mappings must be editable even if ES is not turned on yet.
Using a separate array to store the errors as we are testing for $@ ||
@messages.
There is still something wrong that should be improve, but this patch
should be safe for backport.
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Andrew Nugged <nugged@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the cni/Control-number-identifier index to enable
searches to use the 003 field.
Test plan
1/ Apply patch
2/ Re-index using updated configurations
3/ Confirm cni:number searches yield the expected results
4/ Signoff
Split-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds 050 $a to the mapping for the lc-call-number index.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
I'm not sure how to search specifically for an LC call number.
You can confirm that 050 $a is displayed in the Search engine configuration page.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds fields 710$a and 711$a to the default author mappings for elasticsearch indexing.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for an author name found only in 710 using the Author index
in advanced search
4) Repeat search for an author name in 711
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch corrects a typo in the mappings.yaml file
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for a standard number found in 024$a using the Standard number index
in the advanced search
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds 710 to the default author-name-corporate index mappings for
elasticsearch.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for an author name found only in 710 using the Author (Corporate name) index
in advanced search
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds series added entries titles (800 $t, 810 $t, 811 $t, and 830 $a) in the
default title-series index mappings.
To test:
1) Apply patch
2) Reindex using rebuildelasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration and click "Reset mappings" and confirm
2b) Then reindex
3) Search for a series title found only in and added entry field
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
It'd be great if the Search Engine Configuration page would display
the various aliases (shortcuts) available : ti for title, sn for local-number, etc.
Patch changes Koha/SearchEngine/Elasticsearch/QueryBuilder.pm to move
hard-coded vars at the beging and adds a method to provide to %index_field_convert via a method.
Test plan :
1) Use Elasticsearch
2) Go to Administration > Search engine configuration (Elasticsearch)
3) Check you see new column 'Aliases' with for example ti for title.
4) Perform a search 'ti:<title>' and check you get results
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Search engine administration is very important,
we should log who/when it is changed.
I don't add a preference system to disable it,
like there is no for preference system logs.
Test plan :
1) Use searchengine Elasticsearch
2) Go to Administation > Search engine configuration (Elasticsearch)
3) Click on 'Reset Mappings' and accept
4) Edit some lines and save
5) Go to 'Tools' > 'Log viewer'
6) Select only 'Search engine' in Modules and submit
7) Select only 'Edit mappings' in Actions
8) Check you see a log
9) Select only 'Reset mappings' in Actions
10) Check you see a log
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
With Elasticsearch 6 (>6.4), we have a warning on index creation :
the default number of shards will change from [5] to [1] in 7.0.0
See https://github.com/elastic/elasticsearch/pull/30587
I propose to add number_of_shards in index config.
Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.
Test plan :
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes : misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check you dont have a warning about number of shards
5) Check the settings of index :
curl '<cluster>:9200/<myindex>_biblios/_settings?pretty&filter_path=**.number_of_*'
6) You should see :
"number_of_shards" : "5",
"number_of_replicas" : "1"
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
