In Bug 36578, some fields were added to the pl index, but somehow it
ended up in the other-control-number index. This aims to put fields
752ad (MARC21) and 210a/214a (UNIMARC) in the pl index.
Test plan:
----------
0. Have an installation with Elasticsearch and MARC 21 configuration.
1. Under Administration > Search engine configuration (Elasticsearch)
under the tab Bibliographic records check that you have 752ad in
other-control-number search field.
2. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
3. Under Administration > Search engine configuration (Elasticsearch)
under the tab Bibliographic records check that you have 752ad in pl
search field now.
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This enhancement adds field 370 to MARC21 default authority
index mappings.
To test:
1. Import the attached .mrc file
2. Go to Authorities
3. Search 'Main heading' for quebec
--> 1 result (normal)
4. Search 'Entire record' for quebec
--> 1 result
5. Go to the OPAC > Authority search
6. Redo steps 3 and 4
--> Same results
7. Apply patch
8. Reindex authorities (in KTD shell:
koha-elasticsearch --rebuild -a -v -r kohadev)
9. Redo step 3
--> Still 1 result (normal)
10. Redo step 4
--> 2 results
11. Redo step 5-6
--> Same results as steps 9-10
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This enhancement adds field 678 to the default MARC21 authority index mappings.
To test:
1. Import the attached .mrc file
2. Go to Authorities
3. Search "Entire record" for innu
--> No results
4. Go to the OPAC > Authority search
5. Search "in the complete record" for innu
--> No results
6. Apply patch
7. Reindex authorities (in KTD shell:
koha-elasticsearch --rebuild -a -v -r kohadev)
8. Redo step 3
--> 1 result
9. Redo steps 4-5
--> 1 result
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
- This index searches 022ayz, 023ayz (MARC21) and 011ayz (UNIMARC).
- 011y and 011z have been removed from issn search field for UNIMARC.
- 022yz and 023yz have also been added to the identifier-standard search field
Test plan:
1. Start up KTD with Elasticsearch: ktd --es8 up
2. Add a new biblio record putting the value '0046-2254' into the incorrect ISSN subfield (022y /023y for MARC21, or 011z for UNIMARC)
3. Do a biblio search for '0046-2254' and confirm no results are returned
4. Apply this patch and restart all services (restart_all)
5. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)
6. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
7. Alternative to steps 5 and 6: reset_all
8. Repeat the biblio search for '0046-2254' and confirm you get a result
Sponsored-by: Education Services Australia SCIS
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The main entry form of corporate names (110), uniform titles (130), topical
terms (150), geographical names (151), and genre/form (155) are not indexed
with 'Match' search field in Elasticsearch standard mapping. As a result,
the respective records are not present on the result list when performing
an 'All headings' search for the authority records with the main heading form
(MARC 21).
Test plan
=========
0. Have a test installation with Elasticsearch.
1. In Authorities, make an 'All headings' search for a main entry
form from the corporate names, uniform titles, topical terms,
geographical names, or genre/form
--> e.g., in ktd: UK Archiving. You will get no results.
2. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -a kohadev
3. Repeat the test. You should see 'UK Archiving' on the result list.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Testing plan:
1. Start up KTD with Elasticsearch: ktd --es8 up
2. Apply the patch and restart everything (restart_all).
3. Reset the mappings: Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (at the bottom of the page)
4. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
5. Alternative to steps 3 and 4: reset_all
6. Update the visibility for 254$a in the default framework so that it is visible in the Editor (OPAC, Staff interface should already be selected).
7. Add a new record using the default framework and put a term in 254$a, such as 'Full score'.
Sponsored-by: Education Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
See previous commit message, 'QA' part.
We do not support the "specials" categories.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature adds the ability to link an ES facet to an authorised
value category, in order to display a description instead of the code.
Test plan:
1. Go to the ES mapping admin page
2. Notice the new "Authorized value category" column in the facets
config table
3. Create a new facet that is linked to a search field containing values
from an authorised value category. Pick this category in the list
4. Save
5. Reindex
6. Notice that the descriptions are now displayed (staff and OPAC) in
the search result page.
QA: See the related bug report for possible improvements. It's not
easy at all to deal with the "specials" here, a lot of changes must be
done in how we fetch/cache the AVs.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
In Elasticsearch, the field 041 (subfields: a, d, e, i, j)
is not indexed with 'ln' search field. As a result, records cannot
be found when searching with languages present in 041 (but only
with the one from 008/35-37), and the languages are also missing
from the facet.
Subfields content (only relevant subfields):
$a - Language code of text/sound track or separate title
$d - Language code of sung or spoken text
$e - Language code of librettos
$i - Language code of intertitles
$j - Language code of subtitles
Test plan
=========
0. Have a test installation with Elasticsearch.
1. In ktd with its test data, make a biblio search for a language present
in 041 a/d/e/i/j but not in 008/35-37, e.g. for Japanese (with ln:jpn
or from Advance search). You will get no results.
2. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
3. Repeat the test. You should get some records and also you should see
the Japanese language in the Languages facet.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With Zebra, the general keyword search covered the entire record content,
even if the field was not explicitly indexed with a Zebra index (the Any index).
With Elasticsearch only the content of the fields explicitly indexed
can be found with general keyword search. This is OK, to some extent, but with
the current configuration it hides the content fo some important notes from the
searches, and librarians complained about that: 520 - Summary, etc.,
561 - Ownership and Custodial History, 563 - Binding Information.
This patch adds the content of 520, 561, and 563 fields to the 'note'
search field.
Test plan
=========
1. Have an installation with Elasticsearch.
2. Add some information to the 520, 561, and/or 563 fields (561 is
hidden in the default framework--one should enable it to be able to use
the 561 field). Save teh record.
3. Try to find the record with the general keyword search, with the words
you used. If they were specific enough, you will get no results.
4. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
5. Confirm that now you are able to find the record with the information
you entered.
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Extension to bug 36269: 1) when cataloguing old prints, the standardized form of
the place of publication is given in the field 752 and as such should also be indexed.
2) Also, normal UNIMARC fields for publication place should be indexed (210a, 214a).
Test plan
=========
Scenario A (MARC 21, field 752)
-------------------------------
0. Have an installation with Elasticsearch and MARC 21 configuration.
1. Activate subfields 752 $a, $d in the default framework (Visibility: Editor).
2. Edit an existing record with this framework, putting some rare words in the
752 $a and $d subfields.
3. Try to make a search with tis words (general keyword or by publication place,
i.e. pl:).
4. You hsold get no results.
5. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
6. Repeat p. 3. You should get your edited record.
Scenario B (UNIMARC)
--------------------
0. Have an installation with Elasticsearch and UNIMARC configuration.
1. Try to search for a publication place, either with pl: directly in the
search window, or in Advanced search, with Publisher location.
With ktd UNIMARC test date you can search for 'Ste Geneviève Cedex'.
2. You should get no results.
3. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
4. Repeat p. 1. You should get some results (with ktd test data set and
Ste Geneviève Cedex - record #1).
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new index for authorities, Other-control-number, and maps it to 035$a.
This will help when trying to match authority records when importing external records.
Test 1:
0. Make sure Elasticsearch is set as the search engine
1. Apply patch and restart
2. Import the attached record
2.1. Go to Cataloging > Stage records for import
2.2. Choose the file
2.3. Click Upload file
2.4. Choose Record type: Authority
2.5. Click Stage for import
(wait until the job is finished...)
2.6. Click View batch
2.7. Click Import this batch into the catalog
(wait until the job is finished...)
2.8. Click Manage imported batch
2.9. Click View next to the record
2.10. Note the auth id number
3. Examine the ES entry for the record (replace INDEX_NAME with the index name (found in koha-conf.xml and XX with the auth_id)
curl -XGET 'http://localhost:9200/INDEX_NAME_authorities/data/XX?_source_includes=other-control-number&pretty'
--> It should give you the value of 035$a
Test 2 (optional):
1. Steps 1 and 2 as above
2. Add a matching rule to match on 035$a for authority records
2.1. Go to Administration > Record matching rules
2.2. Click New record matching rule
2.3. Fill out the form
- Matching rule code: enter a code (for example AUTCONTROL)
- Description: enter a description (for example 035$a for authorities)
- Match threshold: 100
- Record type: Authority record
- Search index: Other-control-number
- Score: 100
- Tag: 035
- Subfields: a
2.4. Click Save
3. Import the same record again, checking for matches using the new rule
3.1. Go to Cataloging > Stage records for import
3.2. Choose the file
3.3. Click Upload file
3.4. Choose Record type: Authority
3.5. Choose Record matching rule: rule created above
2.6. Click Stage for import
(wait until the job is finished...)
--> It should say that 1 record was found using the rule
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
For MARC 21, canceled/invalid ISBNs (subfield 020 $z) are not being
indexed in Elasticsearch (search fields identifier-standard and isbn),
but it is for UNIMARC (subfield 010 $z). As a result, a user searching
for such an ISBN (e.g. with wrong checksum--and therefor put in 020 $z)
will not find the record in the catalogue.
A new isbn-all Elasticsearch search field 'isbn-all' has been created with
020az (MARC 21, and 010az for UNIMARC), 010z has been removed from isbn
search field for UNIMARC, and 020z has been added to identifier-standard
search field.
Test plan
=========
0. Have a test installation with Elasticsearch.
1. In ktd with its test data, make a biblio search for a cancelled ISBN
e.g. 9780007269854. You will get no results.
2. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -b kohadev
3. Repeat the test. You should get "The ice princess / Camilla Läckberg".
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We want the label from the DB displayed on the UI so we are adjusting
the yaml and the DB.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature allows to manage facets for Elasticsearch from the
administration page.
Prior to this the facet fields were hardcoded in the codebase.
Librarians can then add/remove facet fields and modify their label.
Test plan:
1. Create a new search field and set it a label different than its name.
2. Save
3. Go the bibliographic mapping tab
4. Add 1+ mapping for this search field (Searchable and facetable must be "Yes")
5. Add, reorder, remove new facets
6. Save and reindex your records
7. Search and notice the new facet list
QA: There are several wrong things in this area (ES + facets code,
everything, pm, pl, tt AND on this administration page). I have done my
best to clean the code as much as I could and keep the code cleaner
after than before. But there are still a lot to do.
There are still inconsistencies on this page (like we need to save to
see the changes applied to the other tables), but this is clearly out of
the scope of this bug report.
Another enhancement would be to move the facet list to a different DB
table, that could bring more flexibility:
* display or not (could be opac/staff/both/none)
* define the size per field (number of facet to display)
* order: move search_field.facet_order to this new table.
But, again, it's a lot more work.
More work is done in this area, please see related bug reports.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
- apply this patch,
- check that SearchEngine system preference is set to Elasticsearch,
- go to Admin > Search engine configuration,
- on the search fields tab, fill a new line at the bottom of the table
(name, label, type)
- click on the "Add" button and save,
- check that the new search field has been saved,
- also test field deletions,
- check that you can't delete already mapped fields.
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: David Schmidt <davewood@gmx.at>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently the index `pl` points to 008/15-17. It should
(additionally/instead?) point to 260a and/or 264a.
Test plan (for koha-testing-docker with ElasticSearch via `ktd --es7 up`)
Verify the old/broken behaviour:
* Go to Staff/Advanced Search
* Select "Publisher Location" and enter "cau", start search
* You will get some hits (~16), eg "Perl best practices / Damian Conway", which has 008 of "051222s2005 caua b 001 0 eng d" with "cau" on 15-17
* Edit this record (cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=5)
* Inspect 260$a, which should containt "Sebastopol, CA :"
* Go back to Advanced Search and search for "Publisher Location" = "Sebastopol"
* No hits!
Apply the patch!
* exit ktd and stop it (ktd --es7 down; ktd --es7 stop)
* start it again
* Go to Staff / Administration / Search Engine Config (Elasticsearch)
* Click on the Tab "Bibliographic records" and search/filter for "pl"
* you should see 3 entries for "pl", with Mapping values of "008_/15-17", "260a" and "264a"
* Go to Search,
* Select "Publisher Location" and enter "cau", start search
* same hits as befor
* Search again, but for "Sebastopol"
* Now you'll get 9 hits!!
Bonus: Test 264a
* Got to Admin / MARC bibl framework, select BKS -> MARC structure
* search for tag "264", edit subfields
* in tabs a, b, c: Check the "Editor" Checkbox (Visibility)
* Save changes
* find a book, eg again "Perl best practices" and edit it
* find field 264 and enter "Test" into 264a, Save
* Depending on your setup, you might have to manually re-index the book:
* enter ktd: ktd --shell
* reindex the one book (via --bn) or all (might also need a -d)
/usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl -v -b -d
* Go again to Advanced Search, Publisher Location = "Test"
* You should find the book
If your NOT running ktd, you might be able to just edit the ElasticSearch Mappings to add / change the mapping for "pl" to point to "260a".
One rather harsh way to do this (which I needed to do, because the ES Mappings UI did not work for me) is via this SQL:
update search_marc_map set marc_field='260a' where marc_field='008_/15-17';
Sponsored-by: Steiermärkische Landesbibliothek
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
opac-shelves - forms were stateless - changed to GET
sco-main - forms stateless - changed to GET
** untested because sco + auth is broken
smart-rules.tt - JS form template - added placeholder 'cud-' op
ill-requests - added cud- tp ops
** tested comments, but not checkout, simple changes
boraccount - removed repeated op - updated script
patroncards/edit-batch - add placeholder 'cud-' op
patroncards/manage - add placheholder 'cud-' op
elasticsearch/mappings - separate forms - this could use a style follow-up, but makes more sense separate I think
reports/dictionary - stateless - changed to GET
guided_reports_start - stateless - changed to GET
suggestion/suggestion - add placeholder 'cud-' op
inventory - filed bug 36305, needs more handling
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch finally adjust the default mappings to add a new field to
the elastic index with the title stripped of punctuation
This method optionally allows the library to place the filtered version in the same
search field, or a new search field. The default mappings will include the filtered version
in a keyword search, but not a targeted search
To test:
1 - Import some affected records via z3950, e.g.:
Carrie's war
1,000 Japanese words
2 - Search (using Elasticsearch) for the titles without including punctuation
Carries war
1000 Japanese words
3 - No results
4 - Reload mappings and reindex
perl misc/search_tools/rebuild_elasticsearch.pl -v -r
5 - Search again
6 - Success!
7 - Search title specifically:
ti:Carries war
8 - No results
9 - Adjust mappings.yaml to add second mapping for 245 to title index:
- facet: ''
marc_field: 245abp
marc_type: marc21
sort: 1
suggestible: 1
filter: punctuation
10 - Reload mappings and reindex
11 - Repeat 7
12 - Success
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds a new column to the 'Bibliographic records' tab in
Administration > Search engine configuration (Elasticsearch)
To test:
1 - Confirm the new 'filter' column shows
2 - Update an existing field to set filter to punctuation
3 - Confirm it can be saved
4 - Add a new field
5 - Confirm it saves correctly
6 - Unset filter for a field
7 - Confirm it saves
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
At some point the option for 'undef' was removed from te sort options
and was collapsed to yes/no
The dropdowns when adding a new field were missed, this patch corrects that.
While undef in a mappings file wil stil load, when saving we should not privde undef any longer
To test:
1 - Browse to bottom to add a new field on the 'Bibliographic records' tab in
Administration > Search engine configuration (Elasticsearch)
2 - Set sortable column to undef, set other columns and provide a valid field
3 - Click '+Add'
4 - Click 'Save'
5 - At top of page you receive an error:
An error occurred when updating mappings: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Column 'sort' cannot be null at /kohadevbox/koha/Koha/SearchField.pm line 37 .
6 - Apply patch, restart all
7 - Add a new mapping, your only choices are Yes/No
8 - Save mapping
9 - Confirm it saves correctly
Signed-off-by: Salah Ghedda <salah.ghedda@inLibro.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch simply remvoes sort from all elements that are not strictly
the main title
Note:
If multiple fields are set as sort, they are collapsed into a single entry in
the {field}__sort field in the ES index. The order will be determined by the order in
the marc record
To test:
1 - Apply patch
2 - perl misc/search_tools/rebuild_elasticsearch -r -v
3 - Search the catalog
4 - Sort by title
5 - Confirm records are correct
6 - Add a 240 (before the 245) with subfield a 'AAAAA'
7 - Confirm sorting is not affected
8 - View record details, click 'Elasticsearch record: Show'
9 - Find 'title__sort' and confirm it looks correct (does not include AAAAA)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In Elasticsearch fields config field_config.yaml, default type as a field 'ci_raw'. This is used for exact search.
This field is missing for type standard number 'stdno'.
Test plan :
1) In the staff interface, go to Administration, and search for SearchEngine
2) Make sure that the SearchEngine preference is set to Elasticsearch and save
3) Return to Administration and select "Search engine configuration"
4) Change the type of "Heading-Main" to "Std. Number" and save
5) Rebuild the index (e.g. "koha-elasticsearch --rebuild -d kohadev")
6) Go to the main staff page and select Authorities
7) Search for a heading (e.g. "A Dual-language book")
=> Result is found with or without patch
8) Click on the sliders and select "is exactly" for the operator and search
=> Result is found only with patch
9) Apply the patch
10) Rebuild the index (e.g. "koha-elasticsearch --rebuild -d kohadev")
11) Click on the sliders and select "is exactly" for the operator and search
=> Result is found only with patch
Signed-off-by: Kevin Carnes <kevin.carnes@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Before this patch we used two indexes for the thesaurus values, we can
simply index both needed fields into a single index and just form the
search correctly.
This patch also ensures we pass the 'thesaurus' vlaue for the heading
directly to the query builder - for zebra it goes through, and for ES
we convert it to the expected code.
This patch also moves the necessary mappings out of the user definable
mappings and hardcodes them. There is precedent for this with
'match-heading', it ensures matching works as expected
To test:
1 - Follow previous test plan in Zebra and ES
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Currently we only index a - but we can setup the system such that avxyz are searched
To test:
1 - define both a 655$a *and* 655$x value in a bib, save, reindex
2 - Set system preferences:
TraceSubjectSubdivisions: Include
TraceCompleteSubfields: Force
3 - View the record edited above in the opac
4 - Click on the subject heading
5 - No results found
6 - Copy zebra files:
sudo cp ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml \
/etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
sudo cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
/etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
7 - restart all and reindex
8 - Click on the subject heading in OPAC
9 - Sucess!
10 - Repeat with other fields (vyz)
11 - Repeat under ES, reindexing and resetting mappings
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds the following fields to the See-from index
- 450(abvxyz)
- 451(avxyz)
- 455(avxyz)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds the following fields to the Match-heading-see-from index
- 430(adfghklmnoprstvxyz)
- 448(avxyz)
- 450(abvxyz)
- 451(avxyz)
- 455(avxyz)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds see from and see also from terms for uniform title,
chronological term, topical term, geographic name, and genre/form term
to the Match index in Elasticsearch for MARC21.
Previously, only see from/see also from for personal names,
corporate names, and meeting names were indexed.
To test:
1. Without patch, import attached authority records
1.1. Download attached records
1.2. Go to Tools > Stage MARC records for import
1.3. Click 'Browse' and choose the downloaded file
1.4. Click 'Upload file'
1.5. Choose Record type = Authority
1.6. Click 'Stage for import'
1.7. From the job details, click 'View batch'
1.8. Click 'Import this batch into the catalog'
2. Without patch, search for see from and see also from tracings
2.1. Go to Authorities
2.2. In the 'Default' drop-down menu, choose 'Uniform title'
2.3. Choose the 'Search all headings' tab
2.4. Enter the search term 'Five hundred'
2.5. Click 'Submit' or press 'Enter'
--> No results
2.6. Redo the search for the following search terms
Authority type Search term Should be found in
Uniform title five hundred 430
Uniform title films préférés 530
Chronological term fifteenth 448
Chronological term middle ages 548
Topical term lalopathie 450
Topical term troubles communication 550
Geographic name cécropia 451
Geographic name canada francophone 551
Genre/Form term filmiques 455
Genre/Form term films 555
3. Apply patch
4. Delete index, reset mappings and reindex authorities (with command line, -a -d -r)
5. Redo the searches from step 2, there should now be results
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex
Sponsored-by: Lund University Library, Sweden
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add a new boolean ES field named 'available', which is true if at least
one item is available, which means the item is not on loan, not
"notforloan", not withdrawn, not lost and not damaged
A full indexation is required
Test plan:
1. Apply patch and run updatedatabase.pl
2. Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
3. Make sure you have some biblios whose items are all unavailable, some
biblios whose items are all available, and some biblios with at least
one item available and at least one item unavailable
4. Use the 'available' filter on both opac and intranet and make sure it
works as expected.
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch removes one of the two mappings for the 003 field to the
control-number-identifier index (for MARC21).
To test:
1) Apply patch
2) reindex with mappings reset
3) try to search for cni:code (for example cni:OSt)
--> it should return the desired results
4) try to search for control-number-identifier:code (for example
control-number-identifier:OSt)
--> it should return the desired results
5) Optionally, try the test plan in Bug 11175 to make sure it still
works
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Repeat previous tests with Elasticsearch engine
You will need to reindex and reset mappings to pickup the changes form the file
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Elasticsearch number of results is by default limited by setting "index.max-result-window", default value is 10000.
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-result-window
We use this setting:
44d6528b56/Koha/SearchEngine/Elasticsearch/Search.pm (L411)
I propose we add this setting in index config.
Test plan:
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes: misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check the settings of index (when using koha-testing-docker*):
curl 'es:9200/koha_kohadev_biblios/_settings?pretty&filter_path=**.max_result_window'
5) You should see:
"max_result_window" : "1000000"
* You also need to add this setting to the es section in koha-testing-docker's
docker-compose.yml (after the networks configuration):
ports:
- "9200:9300"
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When defining our sort fields in we defined all as 'numeric'
For other string containing numbers this is likely correct, however,
for callnumbers it is not. e.g. E45 should sort before E7
This patch adds a new 'callnumber' type and deifnes this for cn-sort and
adds to the field maping a sort without numeric set
To test:
0 - Be using ES with Koha
1 - On records with single item, add callnumbers:
VA65 E7 R63 1984
VA65 E7 T35 1990
VA65 E45 R67 1985
2 - Add public note 'shrimp' or something to make them easily searchable as a group
3 - Search for 'shrimp', sort by callnumber
4 - Note E45 comes last, it should come first
5 - Apply patch
6 - Reset ES mappings
7 - Reindex ES
8 - Repeat search
9 - Sorting should be correct when set to callnumber
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
A first step to "validate" the MARC mappings: Remove all whitespace, so
if a user enters "245a " (with a trailing whitespace, which can easily
happen when copy/pasting) we only store "245a" in the DB. This is
neccessary, because the ES indexer will throw an exception in an invalid
MARC mapping.
Test Plan:
* Go to /cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl
* Go to the Bibliographic Records Tab
* Enter "100 a b c " (notice the whitespaces!) in the first "mapping"
field
* Scroll down and save
* Go back to the Bibliographic Records Tab
* The spaces are still there
Now apply the patch
* Repeat the above steps
* After saving you should see "100abc" without any spaces in the
"mapping" field
Sponsored-by: Steiermärkische Landesbibliothek
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
The mappings must be editable even if ES is not turned on yet.
Using a separate array to store the errors as we are testing for $@ ||
@messages.
There is still something wrong that should be improve, but this patch
should be safe for backport.
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Andrew Nugged <nugged@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the cni/Control-number-identifier index to enable
searches to use the 003 field.
Test plan
1/ Apply patch
2/ Re-index using updated configurations
3/ Confirm cni:number searches yield the expected results
4/ Signoff
Split-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds 050 $a to the mapping for the lc-call-number index.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
I'm not sure how to search specifically for an LC call number.
You can confirm that 050 $a is displayed in the Search engine configuration page.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds fields 710$a and 711$a to the default author mappings for elasticsearch indexing.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for an author name found only in 710 using the Author index
in advanced search
4) Repeat search for an author name in 711
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch corrects a typo in the mappings.yaml file
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for a standard number found in 024$a using the Standard number index
in the advanced search
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds 710 to the default author-name-corporate index mappings for
elasticsearch.
To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r
If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings"
and confirm
2b) Then reindex
3) Search for an author name found only in 710 using the Author (Corporate name) index
in advanced search
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
