The main entry form of corporate names (110), uniform titles (130), topical
terms (150), geographical names (151), and genre/form (155) are not indexed
with 'Match' search field in Elasticsearch standard mapping. As a result,
the respective records are not present on the result list when performing
an 'All headings' search for the authority records with the main heading form
(MARC 21).
Test plan
=========
0. Have a test installation with Elasticsearch.
1. In Authorities, make an 'All headings' search for a main entry
form from the corporate names, uniform titles, topical terms,
geographical names, or genre/form
--> e.g., in ktd: UK Archiving. You will get no results.
2. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -a kohadev
3. Repeat the test. You should see 'UK Archiving' on the result list.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Testing plan:
1. Start up KTD with Elasticsearch: ktd --es8 up
2. Apply the patch and restart everything (restart_all).
3. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)
4. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
5. Alternative to steps 3 and 4: reset_all
6. Update the visibility for 254$a in the default framework so that it is visible in the Editor (OPAC, Staff interface should already be selected).
7. Add a new record using the default framework and put a term in 254$a, such as 'Full score'.
Sponsored-by: Education Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
In Elasticsearch, the field 041 (subfields: a, d, e, i, j)
is not indexed with 'ln' search field. As a result, records cannot
be found when searching with languages present in 041 (but only
with the one from 008/35-37), and the languages are also missing
from the facet.
Subfields content (only relevant subfields):
$a - Language code of text/sound track or separate title
$d - Language code of sung or spoken text
$e - Language code of librettos
$i - Language code of intertitles
$j - Language code of subtitles
Test plan
=========
0. Have a test installation with Elasticsearch.
1. In ktd with its test data, make a biblio search for a language present
in 041 a/d/e/i/j but not in 008/35-37, e.g. for Japanese (with ln:jpn
or from Advance search). You will get no results.
2. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
3. Repeat the test. You should get some records and also you should see
the Japanese language in the Languages facet.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With Zebra, the general keyword search covered the entire record content,
even if the field was not explicitly indexed with a Zebra index (the Any index).
With Elasticsearch only the content of the fields explicitly indexed
can be found with general keyword search. This is OK, to some extent, but with
the current configuration it hides the content fo some important notes from the
searches, and librarians complained about that: 520 - Summary, etc.,
561 - Ownership and Custodial History, 563 - Binding Information.
This patch adds the content of 520, 561, and 563 fields to the 'note'
search field.
Test plan
=========
1. Have an installation with Elasticsearch.
2. Add some information to the 520, 561, and/or 563 fields (561 is
hidden in the default framework--one should enable it to be able to use
the 561 field). Save teh record.
3. Try to find the record with the general keyword search, with the words
you used. If they were specific enough, you will get no results.
4. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
5. Confirm that now you are able to find the record with the information
you entered.
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Extension to bug 36269: 1) when cataloguing old prints, the standardized form of
the place of publication is given in the field 752 and as such should also be indexed.
2) Also, normal UNIMARC fields for publication place should be indexed (210a, 214a).
Test plan
=========
Scenario A (MARC 21, field 752)
-------------------------------
0. Have an installation with Elasticsearch and MARC 21 configuration.
1. Activate subfields 752 $a, $d in the default framework (Visibility: Editor).
2. Edit an existing record with this framework, putting some rare words in the
752 $a and $d subfields.
3. Try to make a search with tis words (general keyword or by publication place,
i.e. pl:).
4. You hsold get no results.
5. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
6. Repeat p. 3. You should get your edited record.
Scenario B (UNIMARC)
--------------------
0. Have an installation with Elasticsearch and UNIMARC configuration.
1. Try to search for a publication place, either with pl: directly in the
search window, or in Advanced search, with Publisher location.
With ktd UNIMARC test date you can search for 'Ste Geneviève Cedex'.
2. You should get no results.
3. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
4. Repeat p. 1. You should get some results (with ktd test data set and
Ste Geneviève Cedex - record #1).
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new index for authorities, Other-control-number, and maps it to 035$a.
This will help when trying to match authority records when importing external records.
Test 1:
0. Make sure Elasticsearch is set as the search engine
1. Apply patch and restart
2. Import the attached record
2.1. Go to Cataloging > Stage records for import
2.2. Choose the file
2.3. Click Upload file
2.4. Choose Record type: Authority
2.5. Click Stage for import
(wait until the job is finished...)
2.6. Click View batch
2.7. Click Import this batch into the catalog
(wait until the job is finished...)
2.8. Click Manage imported batch
2.9. Click View next to the record
2.10. Note the auth id number
3. Examine the ES entry for the record (replace INDEX_NAME with the index name (found in koha-conf.xml and XX with the auth_id)
curl -XGET 'http://localhost:9200/INDEX_NAME_authorities/data/XX?_source_includes=other-control-number&pretty'
--> It should give you the value of 035$a
Test 2 (optional):
1. Steps 1 and 2 as above
2. Add a matching rule to match on 035$a for authority records
2.1. Go to Administration > Record matching rules
2.2. Click New record matching rule
2.3. Fill out the form
- Matching rule code: enter a code (for example AUTCONTROL)
- Description: enter a description (for example 035$a for authorities)
- Match threshold: 100
- Record type: Authority record
- Search index: Other-control-number
- Score: 100
- Tag: 035
- Subfields: a
2.4. Click Save
3. Import the same record again, checking for matches using the new rule
3.1. Go to Cataloging > Stage records for import
3.2. Choose the file
3.3. Click Upload file
3.4. Choose Record type: Authority
3.5. Choose Record matching rule: rule created above
2.6. Click Stage for import
(wait until the job is finished...)
--> It should say that 1 record was found using the rule
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
For MARC 21, canceled/invalid ISBNs (subfield 020 $z) are not being
indexed in Elasticsearch (search fields identifier-standard and isbn),
but it is for UNIMARC (subfield 010 $z). As a result, a user searching
for such an ISBN (e.g. with wrong checksum--and therefor put in 020 $z)
will not find the record in the catalogue.
A new isbn-all Elasticsearch search field 'isbn-all' has been created with
020az (MARC 21, and 010az for UNIMARC), 010z has been removed from isbn
search field for UNIMARC, and 020z has been added to identifier-standard
search field.
Test plan
=========
0. Have a test installation with Elasticsearch.
1. In ktd with its test data, make a biblio search for a cancelled ISBN
e.g. 9780007269854. You will get no results.
2. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
3. Repeat the test. You should get "The ice princess / Camilla Läckberg".
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We want the label from the DB displayed on the UI so we are adjusting
the yaml and the DB.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently the index `pl` points to 008/15-17. It should
(additionally/instead?) point to 260a and/or 264a.
Test plan (for koha-testing-docker with ElasticSearch via `ktd --es7 up`)
Verify the old/broken behaviour:
* Go to Staff/Advanced Search
* Select "Publisher Location" and enter "cau", start search
* You will get some hits (~16), eg "Perl best practices / Damian Conway", which has 008 of "051222s2005 caua b 001 0 eng d" with "cau" on 15-17
* Edit this record (cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=5)
* Inspect 260$a, which should containt "Sebastopol, CA :"
* Go back to Advanced Search and search for "Publisher Location" = "Sebastopol"
* No hits!
Apply the patch!
* exit ktd and stop it (ktd --es7 down; ktd --es7 stop)
* start it again
* Go to Staff / Administration / Search Engine Config (Elasticsearch)
* Click on the Tab "Bibliographic records" and search/filter for "pl"
* you should see 3 entries for "pl", with Mapping values of "008_/15-17", "260a" and "264a"
* Go to Search,
* Select "Publisher Location" and enter "cau", start search
* same hits as befor
* Search again, but for "Sebastopol"
* Now you'll get 9 hits!!
Bonus: Test 264a
* Got to Admin / MARC bibl framework, select BKS -> MARC structure
* search for tag "264", edit subfields
* in tabs a, b, c: Check the "Editor" Checkbox (Visibility)
* Save changes
* find a book, eg again "Perl best practices" and edit it
* find field 264 and enter "Test" into 264a, Save
* Depending on your setup, you might have to manually re-index the book:
* enter ktd: ktd --shell
* reindex the one book (via --bn) or all (might also need a -d)
/usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl -v -b -d
* Go again to Advanced Search, Publisher Location = "Test"
* You should find the book
If your NOT running ktd, you might be able to just edit the ElasticSearch Mappings to add / change the mapping for "pl" to point to "260a".
One rather harsh way to do this (which I needed to do, because the ES Mappings UI did not work for me) is via this SQL:
update search_marc_map set marc_field='260a' where marc_field='008_/15-17';
Sponsored-by: Steiermärkische Landesbibliothek
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch finally adjust the default mappings to add a new field to
the elastic index with the title stripped of punctuation
This method optionally allows the library to place the filtered version in the same
search field, or a new search field. The default mappings will include the filtered version
in a keyword search, but not a targeted search
To test:
1 - Import some affected records via z3950, e.g.:
Carrie's war
1,000 Japanese words
2 - Search (using Elasticsearch) for the titles without including punctuation
Carries war
1000 Japanese words
3 - No results
4 - Reload mappings and reindex
perl misc/search_tools/rebuild_elasticsearch.pl -v -r
5 - Search again
6 - Success!
7 - Search title specifically:
ti:Carries war
8 - No results
9 - Adjust mappings.yaml to add second mapping for 245 to title index:
- facet: ''
marc_field: 245abp
marc_type: marc21
sort: 1
suggestible: 1
filter: punctuation
10 - Reload mappings and reindex
11 - Repeat 7
12 - Success
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch simply remvoes sort from all elements that are not strictly
the main title
Note:
If multiple fields are set as sort, they are collapsed into a single entry in
the {field}__sort field in the ES index. The order will be determined by the order in
the marc record
To test:
1 - Apply patch
2 - perl misc/search_tools/rebuild_elasticsearch -r -v
3 - Search the catalog
4 - Sort by title
5 - Confirm records are correct
6 - Add a 240 (before the 245) with subfield a 'AAAAA'
7 - Confirm sorting is not affected
8 - View record details, click 'Elasticsearch record: Show'
9 - Find 'title__sort' and confirm it looks correct (does not include AAAAA)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Before this patch we used two indexes for the thesaurus values, we can
simply index both needed fields into a single index and just form the
search correctly.
This patch also ensures we pass the 'thesaurus' vlaue for the heading
directly to the query builder - for zebra it goes through, and for ES
we convert it to the expected code.
This patch also moves the necessary mappings out of the user definable
mappings and hardcodes them. There is precedent for this with
'match-heading', it ensures matching works as expected
To test:
1 - Follow previous test plan in Zebra and ES
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Currently we only index a - but we can setup the system such that avxyz are searched
To test:
1 - define both a 655$a *and* 655$x value in a bib, save, reindex
2 - Set system preferences:
TraceSubjectSubdivisions: Include
TraceCompleteSubfields: Force
3 - View the record edited above in the opac
4 - Click on the subject heading
5 - No results found
6 - Copy zebra files:
sudo cp ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml \
/etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
sudo cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
/etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
7 - restart all and reindex
8 - Click on the subject heading in OPAC
9 - Sucess!
10 - Repeat with other fields (vyz)
11 - Repeat under ES, reindexing and resetting mappings
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds the following fields to the See-from index
- 450(abvxyz)
- 451(avxyz)
- 455(avxyz)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds the following fields to the Match-heading-see-from index
- 430(adfghklmnoprstvxyz)
- 448(avxyz)
- 450(abvxyz)
- 451(avxyz)
- 455(avxyz)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds see from and see also from terms for uniform title,
chronological term, topical term, geographic name, and genre/form term
to the Match index in Elasticsearch for MARC21.
Previously, only see from/see also from for personal names,
corporate names, and meeting names were indexed.
To test:
1. Without patch, import attached authority records
1.1. Download attached records
1.2. Go to Tools > Stage MARC records for import
1.3. Click 'Browse' and choose the downloaded file
1.4. Click 'Upload file'
1.5. Choose Record type = Authority
1.6. Click 'Stage for import'
1.7. From the job details, click 'View batch'
1.8. Click 'Import this batch into the catalog'
2. Without patch, search for see from and see also from tracings
2.1. Go to Authorities
2.2. In the 'Default' drop-down menu, choose 'Uniform title'
2.3. Choose the 'Search all headings' tab
2.4. Enter the search term 'Five hundred'
2.5. Click 'Submit' or press 'Enter'
--> No results
2.6. Redo the search for the following search terms
Authority type Search term Should be found in
Uniform title five hundred 430
Uniform title films préférés 530
Chronological term fifteenth 448
Chronological term middle ages 548
Topical term lalopathie 450
Topical term troubles communication 550
Geographic name cécropia 451
Geographic name canada francophone 551
Genre/Form term filmiques 455
Genre/Form term films 555
3. Apply patch
4. Delete index, reset mappings and reindex authorities (with command line, -a -d -r)
5. Redo the searches from step 2, there should now be results
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex
Sponsored-by: Lund University Library, Sweden
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add a new boolean ES field named 'available', which is true if at least
one item is available, which means the item is not on loan, not
"notforloan", not withdrawn, not lost and not damaged
A full indexation is required
Test plan:
1. Apply patch and run updatedatabase.pl
2. Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
3. Make sure you have some biblios whose items are all unavailable, some
biblios whose items are all available, and some biblios with at least
one item available and at least one item unavailable
4. Use the 'available' filter on both opac and intranet and make sure it
works as expected.
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch removes one of the two mappings for the 003 field to the
control-number-identifier index (for MARC21).
To test:
1) Apply patch
2) reindex with mappings reset
3) try to search for cni:code (for example cni:OSt)
--> it should return the desired results
4) try to search for control-number-identifier:code (for example
control-number-identifier:OSt)
--> it should return the desired results
5) Optionally, try the test plan in Bug 11175 to make sure it still
works
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Repeat previous tests with Elasticsearch engine
You will need to reindex and reset mappings to pickup the changes form the file
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When defining our sort fields in we defined all as 'numeric'
For other string containing numbers this is likely correct, however,
for callnumbers it is not. e.g. E45 should sort before E7
This patch adds a new 'callnumber' type and deifnes this for cn-sort and
adds to the field maping a sort without numeric set
To test:
0 - Be using ES with Koha
1 - On records with single item, add callnumbers:
VA65 E7 R63 1984
VA65 E7 T35 1990
VA65 E45 R67 1985
2 - Add public note 'shrimp' or something to make them easily searchable as a group
3 - Search for 'shrimp', sort by callnumber
4 - Note E45 comes last, it should come first
5 - Apply patch
6 - Reset ES mappings
7 - Reindex ES
8 - Repeat search
9 - Sorting should be correct when set to callnumber
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Andrew Nugged <nugged@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the cni/Control-number-identifier index to enable
searches to use the 003 field.
Test plan
1/ Apply patch
2/ Re-index using updated configurations
3/ Confirm cni:number searches yield the expected results
4/ Signoff
Split-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds 050 $a to the mapping for the lc-call-number index.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
I'm not sure how to search specifically for an LC call number.
You can confirm that 050 $a is displayed in the Search engine configuration page.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds fields 710$a and 711$a to the default author mappings for elasticsearch indexing.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for an author name found only in 710 using the Author index
in advanced search
4) Repeat search for an author name in 711
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch corrects a typo in the mappings.yaml file
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for a standard number found in 024$a using the Standard number index
in the advanced search
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds 710 to the default author-name-corporate index mappings for
elasticsearch.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for an author name found only in 710 using the Author (Corporate name) index
in advanced search
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds series added entries titles (800 $t, 810 $t, 811 $t, and 830 $a) in the
default title-series index mappings.
To test:
1) Apply patch
2) Reindex using rebuildelasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration and click "Reset mappings" and confirm
2b) Then reindex
3) Search for a series title found only in and added entry field
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Apply patch
2 - ./installer/data/mysql/updatedatabase.pl
3 - Reset ES mapping: Administration->Search engine configuration , button at bottom of page
4 - 'issues' and 'title' mapping under 'search fields' should be mandatory and not editable
5 - On 'Bibliographic records' tab you should not be able to delete the single entry for issues
6 - You should be able to delete 'title' mappings, however, at the final one you should be stopped by javascript
7 - Bonus: force remove the last mapping from the page using developer tools - attempt to save and should be warned of missing mandatory mapping
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In admin/searchengine/elasticsearch/mappings.yaml the search field not-onloan-count is defined for MARC21 on 999x.
This should be for all the MARC flavours, like in Zebra config.
Test plan:
1) On a UNIMARC database
2) Reset Elasticsearch mappings
3) Check search engine config to see field 'not-onloan-count' on 999$x
4) Same on a NORMARC database
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.
In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality
To test:
1 - Start form a sample database with ElasticSearch working
2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
Waterworks
Waterworks - Costs
3 - Place a heading for the broader term in a record. e.g. Waterworks
In 650$a, without the cataloguing authority plugin. We don't want
the link created now.
You need syspref BiblioAddsAuthorities => allow
4 - Make sure linker is set to default
5 - Attempt to link the records
misc/link_bibs_to_authorities.pl
6 - Linking fails
7 - Apply patch
8 - refresh index settings (if using a custom file, remove 'match-heading')
You can reset mappings in the UI or run this:
misc/search_tools/rebuild_elasticsearch.pl -v -d -r
9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 25273: (follow-up)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We need to be consistent and provide the mappings.yaml file generated by
the script.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing.
Test plan:
1. Reset mappings and reindex biblios.
2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass.
3. Try that all of the following year range type work in publication date search and year limit in advanced search:
yyyy
yyyy-yyyy
-yyyy
yyyy-
Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Fixes handling of /0 selector in addition to several fixed field positions. Note that ff7-01-02 is in fact 00-01 in Zebra and that has been replicated here.
Test plan:
1. Before applying a patch, check from Elasticsearch (e.g. localhost:9200/koha_biblios/_search?q=_id:123) what is indexed in Elasticsearch for a record for date-entered-on-file, ff7-00, ff7-01, ff7-02 and lleader.
2. Apply the patch, update database and save the record again.
3. Verify that the contents of the forementioned fields is now correct in Elasticsearch.
4. Verify that tests pass: prove -v t/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code
To test:
1 - Make sure you have records with differing languages (in the MARC21 008
field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch adds a new section 'Facet order' in the Biblio tab of the
'Search engine configuration' admin page of the Elastic mappings.
The idea is to let the librarians define the facet to display and order
them as their needs.
The ergonomic is not perfect and I am open to any suggestions.
Test plan:
Move up and down the field list to order the facets
Hide/show some facets
Rebuild index
At the OPAC and the staff interface you should see the changes on the
search result page.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Test plan:
1. Reset mappings
2. Run perl misc/search_tools/rebuild_elastic_search.pl -d -b -v
3. Do a search and verify that the Places facet works
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.
Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
