Commit graph

279 commits

Author SHA1 Message Date
5ef1b6710e Bug 18098: Add an index with the count of not onloan items
This patch adds a numeric index 'not-onloan-count' containing the value
of 999$x. This subfield is filled by 'rebuild_zebra.pl' by making use of
(bug's 18208) 'EmbedItemsAvailability' filter.

bib1.att and indexes definitions are updated accordingly.

To test:
- Apply the patch
- Pick the right biblio-zebra-indexdefs.xsl file for your setup and
  replace the one your Zebra uses [1]
- Replace your bib1.att
- Replace your ccl.properties
- Have at least one record with more than one item, checkout some
  item(s) from that record(s).
- Rebuild zebra's indexes:
  $ sudo koha-shell kohadev
 k$ cd kohaclone
 k$ misc/migration_tools/rebuild_zebra.pl -r -b -v -k
 (notice the dump directory is kept, you can try the XSLT yourself
  running:
    $ xsltproc \
       etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
       /tmp/the_dump_dir/biblios/exported_records | less
=> SUCCESS: There are records with the not-onloan-count index, and the
            value is correct!
- Check Zebra yourself:
  $ yaz-client unix:/var/run/koha/kohadev/bibliosocket
 Z> base biblios
 Z> find @attr 1=9013 @attr 2=5 @attr 4=109 0
=> SUCCESS: The search matches the amount of records with not-onloan
            items.
 Z> s 1+1
=> SUCCESS: Records with 999$x having a value higher than 0 are rendered
- Sign off :-D

Note: While this work is complete on its purpose, it is part of an
attempt to create a better way of filtering by availability.

Sponsored-by: ByWater Solutions

 [1] In kohadevbox this would be
/etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl

Edit: Added the missing XSLT changes for UNIMARC and NORMARC

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-05-08 09:21:41 -04:00
Zoe Schoeler
543dc8e036 Bug 17838 Availability limit broken until an item has been checked out.
TEST PLAN

1. Make sure you have no items checked out.
2. Run sudo koha-rebuild-zebra -f -v kohadev.
3. Go to search the catalog and search.
4. Check items availability and then click on limit to currently
available items.
5. This should return no results.
6. Apply patch and reload.
7. Results should show.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Attribute 14: " Specifies whether un-indexed fields should be ignored. A
zero value (default) throws a diagnostic when an un-indexed field is
specified. A non-zero value makes it return 0 hits."
From http://www.indexdata.com/zebra/doc/querymodel-zebra.html

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-14 14:05:40 +00:00
21d63c8fb0 Bug 17788: (MARC21) Add $9 fields to Koha-Auth-Number:w index
Looking at the default framework's fields that are linked to authority
records, there's a divergence with the Zebra index definitions.

This yields to authority usage count be incorrect for users searching
for authority records.

MariaDB [koha_kohadev]> SELECT tagfield,tagsubfield,authtypecode FROM
marc_subfield_structure WHERE authtypecode IS NOT NULL AND
authtypecode<>'' AND frameworkcode='' GROUP BY
tagfield,tagsubfield,authtypecode ;
+----------+-------------+--------------+
| tagfield | tagsubfield | authtypecode |
+----------+-------------+--------------+
| 100      | a           | PERSO_NAME   |
| 110      | a           | CORPO_NAME   |
| 111      | a           | MEETI_NAME   |
| 130      | a           | UNIF_TITLE   |
| 440      | a           | UNIF_TITLE   |
| 600      | a           | PERSO_NAME   |
| 610      | a           | CORPO_NAME   |
| 611      | a           | MEETI_NAME   |
| 630      | a           | UNIF_TITLE   |
| 648      | a           | CHRON_TERM   |
| 650      | a           | TOPIC_TERM   |
| 651      | a           | GEOGR_NAME   |
| 654      | a           | TOPIC_TERM   |
| 655      | a           | GENRE/FORM   |
| 656      | a           | TOPIC_TERM   |
| 657      | a           | TOPIC_TERM   |
| 658      | a           | TOPIC_TERM   |
| 662      | a           | GEOGR_NAME   |
| 690      | a           | TOPIC_TERM   |
| 691      | a           | GEOGR_NAME   |
| 696      | a           | PERSO_NAME   |
| 697      | a           | CORPO_NAME   |
| 698      | a           | MEETI_NAME   |
| 699      | a           | UNIF_TITLE   |
| 700      | a           | PERSO_NAME   |
| 710      | a           | CORPO_NAME   |
| 711      | a           | MEETI_NAME   |
| 730      | a           | UNIF_TITLE   |
| 796      | a           | PERSO_NAME   |
| 797      | a           | CORPO_NAME   |
| 798      | a           | MEETI_NAME   |
| 799      | a           | UNIF_TITLE   |
| 800      | a           | PERSO_NAME   |
| 810      | a           | CORPO_NAME   |
| 811      | a           | MEETI_NAME   |
| 830      | a           | UNIF_TITLE   |
| 896      | a           | PERSO_NAME   |
| 897      | a           | CORPO_NAME   |
| 898      | a           | MEETI_NAME   |
| 899      | a           | UNIF_TITLE   |
+----------+-------------+--------------+

This patch adds the missing ones to the authority number index as it is
done for the rest of the fields.

To test:
- Verify that
etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
contains intries pointing the $9 subfield of all the fields in the
'tagfield' column above, to the Koha-Auth-Number:w index.
- Sign off :-D

Signed-off-by: Hugo Agud <hagud@orex.es>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-14 13:58:23 +00:00
0eb5d8491e Bug 17871: Remove zebra::snippet to allow access to facets in YAZ 5.8.1+
This patch restores access to zebra facets (or zebra::snippet) with YAZ 5.8.1 or higher.

It was failing due to The <retrieval syntax="xml" name="zebra::*" /> entry in
retrieval-info-bib-dom.xml which IndexData said it wasn't even needed to
get that access.

Edit: I amended the commit message (tcohen)

Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I tested on kohadevbox and found no regression or behaviour change. I
will provide a followup for the packages.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-07 17:50:42 +00:00
cb3f899826 Bug 17547: (MARC21|NORMARC) have Chronological term field $9 indexed
This patch makes Zebra index the 648$9 link for chronological terms on
bibliographic records. This way an authority search on chronological terms
will show the right number in 'Used in X records' message.

To test:
- Have a record with a 648 field, linked to an authority record (i.e. with an authid on 648$9).
- Search for the record, notice it is indexed.
- Perform an authority search for the chronological term
=> FAIL: the term is linked to our record, but koha shows '0' count.
- Apply the patch
- Run:
  $ cd kohaclone
  $ xsltproc etc/zebra/xsl/koha-indexdefs-to-zebra.xsl \
       etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml \
     > etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
  $ git diff
=> SUCCESS: Notice the shipped etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
   is up-to-date
- Run:
  $ sudo cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
            /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
  $ sudo koha-restart-zebra kohadev
  $ sudo koha-rebuild-zebra -f -b -v kohadev
- Search for the record, notice it is indexed.
- Perform an authority search for the chronological term
=> SUCCESS: the term is linked to our record, usage count is 1
- Sign off :-D

I assume NORMARC is similar on this regard. Feel free to fail it if the NORMARC part of the
patch is wrong.

Sponsored-by: Universidad Nacional de Cordoba

Signed-off-by: Hugo Agud <hagud@orex.es>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-12-16 11:27:18 +00:00
bd13efd8ac Bug 16952: Czech language definitions for sorting in Zebra
Test plan:
1) Apply patch
2) Create new instance with parameter --zebralang cs
3) Insert some record with basic latin characters and some with "czech" characters (for example: "č" - should be sorted after "c", "š" - should be sorted after "s")
4) Try to search in katalog (staff and opac) and sort by other field then relevance - title or author for instance
5) Records should be sorted correctly by Czech rules
6) Look at code and confirm it is ok

Signed-off-by: radiuscz <radek.siman@centrum.cz>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
I did not test this patch, but trust in the author and signoffer

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-10-28 15:33:00 +00:00
f2196a2e4f Bug 17035 - Koha allows system-wide 'read' access to all Koha zebra databases, by default
to test bug...
 1/ make a random user
 2/ change to random user
 3/ access any zebra database with random user and no authentication
 4/ read zebra database

here is a transcript of the bug...
---------------------------
root@xen1:~# adduser bob
root@xen1:~# su -l bob

bob@xen1:~$ cd /var/lib/koha
bob@xen1:/var/lib/koha$ ls
topsecret

bob@xen1:/var/lib/koha$ yaz-client  unix:/var/run/koha/topsecret/bibliosocket
Connecting...OK.
Sent initrequest.
Connection accepted by v3 target.
ID     : 81
Name   : Zebra Information Server/GFS/YAZ
Version: 4.2.30 98864b44c654645bc16b2c54f822dc2e45a93031
Options: search present delSet triggerResourceCtrl scan sort extendedServices namedResultSets
Elapsed: 0.001002

Z> base biblios;

Z> find the
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 1130, setno 2
SearchResult-1: term=the cnt=1130
records returned: 0
Elapsed: 0.005518

Z> show
Sent presentRequest (1+1).
Records: 1
[biblios]Record type: USmarc
01824cam a2200397 a 4500
001 000045782309
003 AuCNLKIN
005 20111013213222.0
008 100707s2011    maua          001 0 e
...
---------------------------

5/ apply changes to a Koha instance's config files, that you plan to test

6/ restart zebra for instance
 # sudo koha-restart-zebra topsecret

7/ repeat steps 2 and 3, but receive a 'bad user/passwd ' error from zebra

bob@xen1:~$ yaz-client unix:/var/run/koha/topsecret/bibliosocket
Connecting...OK.
Sent initrequest.
Connection rejected by v3 target.
    1: code=1011 (Init/AC: Bad Userid and/or Password),

NOTE: this patch currently will only fixes newly created instances, it wont fix existing instances
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Good catch Mason

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-10-10 12:20:12 +00:00
5b4259be9c Bug 6499: [QA Follow-up] Trivial adjustments
Removes commented line from bib1.att.
Adjust OCLC-number to Other-control-number in comment of ccl properties.
No need to explicitly add 035$a and $z if you index 035 completely in
record.abs as well as biblio-koha-indexdefs.xml.
Rerun koha-indexdefs-to-zebra.xsl on index defs.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-08-09 10:13:11 +00:00
Barton Chittenden
84f51549c9 Bug 6499: Add Zebra index "Other-control-number" covering MARC21 035$a, 035$z and 035 (entire tag)
1) Apply patch
2) Make sure that you have a bib that has MARC21 035$a (and possibly also 035$z) populated.

pre 3) Replace all modified zebra files and restart zebra server

3) Rebuild zebra: misc/migration_tools/rebuild_zebra.pl -x -b -z
4) Add the following to the intranetuserjs syspref:

$(document).ready(function(){
    // Add Other Control Number to advanced search
    if (window.location.href.indexOf("catalogue/search.pl") > -1) {
        $(".advsearch").append('<option value="Other-control-number">Other Control Number</option>');
    }
});

5) Do an advanced search, select "Other Control Number" from the search menu, then add the Other Control Number in 035$a for the bib specified in step 1.

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works, no koha-qa errors

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-08-09 10:13:10 +00:00
ff6523765d Bug 15555: Index 024$a into Identifier-other:u url register when source $2 is uri
This patch indexes 024$a into the "phrase" index type, and the "url" index type,
if the 024$2 equals "uri".

TEST PLAN

1) Apply the patch.
1b) If you're using a gitified Koha or a git install,
you'll need to upgrade your instance or copy your zebradb files
over to /etc/koha/zebradb or your "kohadev" directory.
2) Add a 024$a with a URL like http://libris.kb.se/resource/bib/219553
 to a bibliographic record
3) Re-index Zebra
4) Type "id-other,st-urx,fuzzy=http://libris.kb.se/resource/bib/219553"
 into the "Search the catalog" box in the Staff Client and search
5) Note that you retrieve your record

NOTE: The fuzzy is required because Koha's query "parsing" functions change
http:// to http=// which won't correctly match the value in the "Identifier-other:u" index.
NOTE: Alternatively, you could do the following search instead:
"id-other,phr=http libris kb se resource bib 219553".
 It would work as well by using the "Identifier-other:p" index.

Advanced tester version:
4) In a terminal window, find the "koha-conf.xml" file in your "etc" directory.
5) Open "koha-conf.xml" and find <listen id="biblioserver">.
Copy the URI you find there. (e.g. unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket).
6) Type "yaz-client unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket"
7) After it connects, type "base biblios" and press enter
8) Type "format xml" and press enter
9) Type "elements zebra::index" and press enter
10) Type "f id-other,st-urx=http://libris.kb.se/resource/bib/219553" and press enter
11) Note that you should have at least one result
12) Type "show 1"
13) If you scroll through the results, you should find something like the following:

<index name="Identifier-other" type="w" seq="28">@^</index>
<index name="Identifier-other" type="w" seq="1"></index>
<index name="Identifier-other" type="w" seq="29">http</index>
<index name="Identifier-other" type="w" seq="30">libris</index>
<index name="Identifier-other" type="w" seq="31">kb</index>
<index name="Identifier-other" type="w" seq="32">se</index>
<index name="Identifier-other" type="w" seq="33">resource</index>
<index name="Identifier-other" type="w" seq="34">bib</index>
<index name="Identifier-other" type="w" seq="35">219553</index>
<index name="Identifier-other" type="p" seq="28">http libris kb se resource bib 219553</index>
<index name="Identifier-other" type="u" seq="36">http://libris.kb.se/resource/bib/219553</index>

Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Works as advertised the record is retrieved

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Some of the statements in the commit message do not work for me.
A search like "id-other,phr=http libris kb se resource bib 219553" does not
have results. Searching for "id-other,phr=libris.kb.se resource" does.
The steps in the advanced tester version do not work for me too.
I verified the following in yaz-client:
[1] Z> f @attr 1=9012 @attr 4=104 http://libris.kb.se/resource/bib/219553
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 1, setno 16
[2] First removed $2 and reindexed. Then searched again:
Z> f @attr 1=9012 @attr 4=104 http://libris.kb.se/resource/bib/219553
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 0, setno 1

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-04-29 13:19:28 +00:00
Barton Chittenden
2a8c68936d Bug 14277: (QA followup) Silent GRS-1 tests
changed ocurrences of 'lex' to 'lexile-number' in record.abs

Edits were made to the deprecated file record.abs *solely* to quiet
warnings in tests -- this makes sense until GRS-1 code is removed
from Koha.

Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
2016-04-07 10:35:18 -06:00
Barton Chittenden
690ab60da2 Bug 14277: add zebra indexes for lexile that respect 521 indicator 1.
Added the following indexes:

Interest-age-level | 591$a ind1=1
Interest-grade-level | 591$a ind1=2
lexile-number | 591$a ind1=8
Reading-grade-level | 591$a ind1=0

Moved 'lex' from a zebra index to a ccl alias to lexile-number.

Changed the handling of st-numeric in C4/Search.pm to allow for search ranges.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Works as advertised
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
2016-04-07 10:35:18 -06:00
Nicole C Engard
39f9b9ddb0 Bug 15694: Add aliases for date/time last modified
This patch will add indexes for Date/time-last-modified.

To test:

1. apply patch
2. reindex
3. search for dtlm:DATE and date-time-last-modified:DATE
4. confirm that you get results

Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Works as advertised

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
  I confirm Hector signing-off. A simple Zebra server restart suffice to get
  working the searches on date-time-last-modified and dtlm.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-11 21:56:50 +00:00
Zeno Tajoli
cc878aee77 Bug 14332: Skip articles in MARC21 using indicator n.2 of field 245
Only in MARC21 is possible to use ind2 of tag 245 to skip articles.
This patch is based on inserting a special template in
koha-indexdefs-to-zebra.xsl With this patch you must not insert index
Title:s in biblio-koha-indexdefs.xml, it is defined in
koha-indexdefs-to-zebra.xsl.  It is not the best setup, but I find very
difficult  to use  biblio-koha-indexdefs.xml.

To test it in a english MARC21 setup:

Insert same records with titles and correct values in ind2 of 245.
If you have articles not in the skiping list of sort-string-utf.chr (The|the|a|A|an|An)
you can see that the sort by articles use also articles.

Insert the patch
Rebuilt indexes from scratch

Now all articles of titles are skipped

TO TEST WITHOUT INDEXING:

1. Go to etc/zebradb/marc_defs/marc21/biblios directory.

2. Put the sample MARCXML file in this directory.

3. Transform the file into Zebra indexes:
   xsltproc biblio-zebra-indexdefs.xsl record.xml
   Observe that the Title:s index contains:
   01 Business and Technologies

4. Apply the patch.

5. Repeat:
   xsltproc biblio-zebra-indexdefs.xsl record.xml
   Observe that the Title:s index contains:
   Business and Technologies

Signed-off-by: Frederic Demians <f.demians@tamil.fr>

Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Verified working using yaz-client (as in
http://wiki.koha-community.org/wiki/Understanding_Zebra_indexing#Examine_Zebra_index,
though note that the `elem zebra::index` seems to be unneeded).

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-01-27 06:17:16 +00:00
Nicholas van Oudtshoorn
3d7bb7647d Bug 15405 - XML paths to zebra libraries is wrong for 64-bit installs on non-Debian linux Adjusts the modulePath to search in the well-known location for 64 bit systems. Required for at least Fedora.
Test plan:
Without this patch, doing a zebra reindex on a fedora-based install will cause errors like this:
15:10:47-01/05 zebraidx(16108) [warn] No such record type: dom./etc/koha/zebradb/biblios/etc/dom-config.xml
With this patch, reindexing should just work.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
I have tested this doesn't break on debian/ubuntu systems, someone with a non
debian system will need to test it on that

Signed-off-by: Bob Ewart bob-ewart@bobsown.com
It works on openSUSE Leap 42.1

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Just noting that the debian zebra files already contain much more paths here.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2015-12-30 14:35:45 +00:00
dd9a4ed6c6 Bug 15198: Make OpacSuppression work even if there are no records suppressed
This patch adds the Zebra special attribute 14 to ccl.properties and
opac-search.pl, so that we can turn on OpacSuppression and still return
results even if there are no records in Zebra for the Suppress index.

_TEST PLAN_

Before applying:

1) Make sure that you have no suppressed records indexed in Zebra
2) Turn on OpacSuppression system preference

3) Search using a keyword which should bring up records
4) Note that no records are returned in the results
5) Change UseQueryParser system preference to "Try"
6) Repeat steps 3-4

Apply the patch.

After applying:

7) Repeat step 3 (ie search using a keyword which should bring up records)
8) Confirm that records are appearing in the results!
9) Change UseQueryParser system preference to "Do not try"
10) Repeat step 3
11) Confirm that records are appearing in the results!

Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Works as advertised. No more, won't need to have at least one record with the
value "1" in the field mapped with this index. All records in OPAC returned.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-11-23 11:26:08 -03:00
b11eb03a4c Bug 14542: Transliterate rule for all single quote forms
Single quotes in common language (not in programming) are usually ', but
there is also the form known as &rsquo; in HTML.  See
https://fr.wikipedia.org/wiki/Apostrophe_%28typographie%29

This bug proposes to transliterate all forms into a space.

Test plan :
(I'll use the code &rsquo; instead of the unicode character)
- Without the patch
- Create a record with title : L&rsquo;avion d&rsquo;argile
- Index this record
- Search for "L&rsquo;avion d&rsquo;argile" => You find the record
- Search for "L'avion d'argile" => You do not find the record
- Apply patch
- Search for "L&rsquo;avion d&rsquo;argile" => You find the record
- Search for "L'avion d'argile" => You find the record
- Search for "L avion d argile" => You find the record

Signed-off-by: Frederic Demians <f.demians@tamil.fr>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-11-16 15:45:21 -03:00
Hector Eduardo Castro Avalos
de272e5b41 Bug 14198: RDA: Indexing 264 field (Zebra)
This patch add zebra indexes to RDA 264 field.
The new Provider index is added too.
QA comments corrected.

To test:
1) Download RDA records with 264 fields from this attachment <http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=36825>. Import the file and re-index/rebuild zebra. These records contain 260 and 264 fields per record.
2) Do a search with pb:Bethany two records will appear with title The guardian. Search with pl:Minneapolis too, the two records will appear.
3) Select one record of both records and delete the 260 field keeping the 264 field and save, rebuild your zebra.
4) Search again with pb:Bethany and just one record will appear. Thats mean 264 is not indexed.
5) Apply patches.
6) Rebuild your zebra but this time all biblio records.
7) Search again with pv:Bethany or Provider:Bethany, this time will appear the two records, 264 is indexed. Note that if you search again with pb only one record appear. This is because the suggestion of LOC.
10) Search with copydate:2013 only launch records with 260 fields and pv:2013 show both fields, i.e., 260 and 264.
11) Apply QA Test Tools

Sponsored-by: Universidad de El Salvador
Signed-off-by: Nick Clemens <nick@quecheelibrary.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-11-02 11:41:36 -03:00
Magnus Enger
9faada90bc Bug 14824: Fix sorting of Norwegian vowels
Currently, Norwegian vowels are not sorted correctly, even when
you have chosen "nb" as the ZEBRA_LANGUAGE during installation.

To test:
- Make sure you have three records with titles that begin with ÆØÅ
  respectively
- Do a search that turns up those three records and some others, and
  sort the results by title, bot ascending and descending.
- Verify that ÆØÅ is shown in some weird order.
- Edit your sort-string-utf.chr* so it is in line with the current
  patch. It should include these two lines:
  lowercase {0-9}{a-z}æøå
  uppercase {0-9}{A-Z}ÆØÅ
- Restart Zebra and reindex all the records, e.g.:
  $ sudo koha-restart-zebra <instancename>
  $ sudo koha-rebuild-zebra -f -v <instancename>
- Do the search again, make sure you order by title and check that
  ÆØÅ are sorted in the order of 1. Æ 2. Ø 3. Å, and after all other
  characters

* = If you are on a gitified install, you need to edit this file:
/etc/koha/zebradb/lang_defs/<your ZEBRA_LANGUAGE>/sort-string-utf.chr
NOT the file in your git clone (yeah, i wasted some time there...)

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-10-26 11:19:03 -03:00
9c3efeaab7 Bug 14106: (RM followup) sick of failing tests in Jessie
This patch adds the original fix for source installs too...

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-10-23 22:58:49 -03:00
aaf3ff3fec Bug 14154: 608$9 defined twice in UNIMARC biblio-koha-indexdefs.xml
In DOM config file :
etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml, the 608$9 is
defined a second time instead of 610$9.  Just a type I think.

Test plan :
- Apply patch
- Install a UNIMARC + DOM instance
- Define in a framework 610 using a thesaurus
- Create a new biblio
- Create a new authority (same type as the thesaurus defined above)
- Index : rebuild_zebra.pl -a -b -x -z
- Link the field 610 to the new authority
- Index : rebuild_zebra.pl -a -b -x -z
- In authorities search, search for the new authority
=> You see Use in 1 Records(s)

Signed-off-by: Frederic Demians <f.demians@tamil.fr>
  I confirm the typo.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-10-21 13:43:12 -03:00
5a9b8d9359 Bug 14861: Accession date comparison does not work in advanced search
_Test plan_

Prerequisites: Make sure that you have an item with a valid dateaccessioned,
and that the bib is indexed in zebra.

For the purposes of explanation, I'm going to use the date '2011-09-07'

1) Go to advanced search in the staff client and choose 'Acquisition date (yyyy-mm-dd)'
2) Enter 2011-09-07 (or the date of your choice).
3) Click the search button - you should get your item in the search results.
4) Return to the advanced search screen and select Acquisition date again.
5) Enter a start and end date in the text field separated by ' - '.

For example:

2011-09-01 - 2011-09-30

6) Click the search button -- this will return no results.
7) Apply the patch and copy etc/zebradb/ccl.properties to whatever directory is specified
by the koha-conf.xml referenced by $KOHA_CONF.
8) Try the search again -- this will return the expected results

Signed-off-by: Barton Chittenden <barton@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-10-19 11:52:44 -03:00
Barton Chittenden
cab481dbb2 Bug 14617: Add fields to ISBN and ISSN indexes: 020$z, 022$y, 022$z
1) Import MARC21 bibs containing

- ISBN in 020$z
- ISSN in 022$y
- ISSN in 022$z

2) Make sure that bibs are indexed

3) Search by ISBN and ISSN above -- bibs should not show in search.

4) Apply patch, re-index

5) Search again; ISBN in 020$z and ISSN in 022$y and 022$z should return
results.

Signed-off-by: kholten@switchinc.org
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-10-19 10:12:04 -03:00
5d46dbf3e9 Bug 14217: Add 'condition' attribute for DOM index definition
This patch introduces an extension to the current syntax for DOM index definition.
Specifically, it extends the 'index_subfields' tag to allow adding a 'condition'
attribute that is used as a condition ofr applying the specified index.

This (exotic) example is self-explanatory:

The previous syntax (which is keeped by this patch) took this snippet from biblio-koha-indexdefs.xml

  <index_subfields tag="100" subfields="acbd">
    <target_index>Encuadernador:w</target_index>
  </index_subfields>

and generated an XSLT snippet in the DOM indexing XSLT that looks like this:

    <xslo:for-each select="marc:subfield">
      <xslo:if test="contains('acbd', @code)">
        <z:index name="Encuadernador:w">
          <xslo:value-of select="."/>
        </z:index>
      </xslo:if>
    </xslo:for-each>

This patch introduces this syntax change (note the 'condition' attribute:

  <index_subfields tag="100" subfields="acbd" condition="@ind2='7'">
    <target_index>Encuadernador:w</target_index>
  </index_subfields>

which yields to this XSLT snippet in the DOM indexing XSLT:

    <xslo:if test="@ind2='7'">
      <xslo:for-each select="marc:subfield">
        <xslo:if test="contains('acbd', @code)">
          <z:index name="Encuadernador:w">
            <xslo:value-of select="."/>
          </z:index>
        </xslo:if>
      </xslo:for-each>
    </xslo:if>

To test:
- Verify that the shipped XSLT files are current regarding the shipped index definitions:
  $ for i in marc21 normarc unimarc; do
        xsltproc etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl \
              etc/zebradb/marc_defs/$i/biblios/biblio-koha-indexdefs.xml \
              > etc/zebradb/marc_defs/$i/biblios/biblio-zebra-indexdefs.xsl
    done
  $ git status
(repeat for authorities, skip normarc which doesn't have authorities)
- Apply the patch
- Re-run the previous commands
=> SUCCESS: no changes
- Add a condition to an index_subfields tag (for example, condition="@ind2='7'" in the Author's index
- Regenerate the specific XSLT
=> SUCCESS: doing a diff shows the only change is the code has been wrapped inside an xslo:if using the condition for the test
- Apply the generated xsl to a MARCXML record that has a field matching the condition like this:
  $ xsltproc .../biblio-zebra-indexdefs.xsl sample_record.xml
=> SUCCESS: There's an index on the result, containing the configured field/subfields, that matches the criteria.
- Sign off and feel really happy :-D

Note: the attached sample record includes a 100 field, with ind2=7 and $a=Tomasito

Edit: This patch was squashed once I figured it got too complex and Jonathan required a followup
to avoid code duplication.

This avoids code duplication, with the same results.

Sponsored-by: Orex Digital
Signed-off-by: Barton Chittenden <barton@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-09-25 11:53:24 -03:00
9eee8b90d7 Bug 14031: Itemnumber should be a numeric search in ccl.properties
This patch changes the "itemnumber" alias so that it acts like
"itemnumber,st-numeric". That is, it always does a numeric search.

_TEST PLAN_

The best way to test this patch is to apply the patch and then run
"make upgrade", I suspect. As this will refresh your "ccl.properties".

However, this patch is actually really small, so you can just apply
it manually to an existing "ccl.properties" if you rather save time.

Basically, you just need to do the following steps:

0) Do a search for "itemnumber:<insert real indexed itemnumber here>"
1) Note that you can't retrieve any results
2) Change your ccl.properties to say "itemnumber 1=8010 4=109"
3) Repeat the search for "itemnumber:<X>"
4) Note that you now retrieve your result

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Tested on a gitified package install. Made the change to
/etc/koha/zebradb/ccl.properties manually. After this change
I can successfully search for "itemnumber:1".

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
2015-08-21 10:05:55 -03:00
Katrin Fischer
e799e1cbc3 Bug 11620: Add dissertation-information index for MARC21 (502)
Bug 11202 introduced a new index 'dissertation-information' for
UNIMARC. This patch adds the index also for MARC21 installations.

http://www.loc.gov/marc/bibliographic/bd502.html

To test:
- Apply patch
- Copy files in etc/zebradb changed by this patch to your
  corresponding directory (koha-dev..)
- Make sure you have records with 502
- Reindex
- Verify you can search the field contents with
  dissertation-information= and
  diss=

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Can find by dissertation-information,
No errors

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
2015-07-20 10:31:06 -03:00
Mirko Tietgen
fbe25b1d8e Bug 14453: (followup) Fix shipped XSLT files
Make the shipped XSLTs for authorities (MARC21 and UNIMARC) the same as the generated version

Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-07-08 14:39:04 -03:00
2365537eea Bug 14453: kohaidx is missing for id in authority-koha-indexdefs.xml
In authority-koha-indexdefs.xml, all tags use the namespace "kohaidx" except the tag "id".

When re-generating authority-zebra-indexdefs.xsl, the line :
  <xslo:variable name="idfield" select="normalize-space(marc:controlfield[@tag='001'])"/>
is modified :
  <xslo:variable name="idfield" select="normalize-space()"/>
This is an error.

This patch adds kohaidx namespace to correct.

Test plan :
- Without patch
- go to etc/zebradb/marc_defs/marc21/authorities/
- run : xslproc xsltproc ../../../xsl/koha-indexdefs-to-zebra.xsl authority-koha-indexdefs.xml > authority-zebra-indexdefs.xsl
- read authority-zebra-indexdefs.xsl
=> the line has changed : <xslo:variable name="idfield" select="normalize-space()"/>
- Apply patch
- go to etc/zebradb/marc_defs/marc21/authorities/
- run : xslproc xsltproc ../../../xsl/koha-indexdefs-to-zebra.xsl authority-koha-indexdefs.xml > authority-zebra-indexdefs.xsl
- read authority-zebra-indexdefs.xsl
=> the line has not changed
(same for unimarc flavor)

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
As Mirko mentioned, the xslt's now generate the facet-processing templates in
the authority xslt's too. They are harmless because we don't define facets
for authority records. If we did, it would be harmless too.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-07-08 14:39:04 -03:00
Stefan Weil
f6aec46dda Bug 14383: etc/zebradb: Fix some typos in documentation and Bib-1 attribute set
All of them were found and fixed using codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2015-06-22 17:34:46 -03:00
f86743d893 Bug 14401: Zebra index configuration doesn't allow exact search for C.
2 lines in the Zebra configuration files prevent an exact search for C.,
while all other [A-Z]. searches work correctly.

After taking a look at the  /etc/zebradb/etc/word-phrase-utf.chr
those 2 lines cause the problem:

map (^c\.)          @
map (^C\.)          @

I propose to remove them.

To test:
- Catalog a record with an item with callnumber: C.
- Catalog a record with an item with callnumber: B.
- Try seaching for the second using callnum,ext:B. (exact field search)
  - Verify search works.
- Try searching for the other with callnum,ext:C.
  - Verify no result.
- Apply the patch - copy the zebra config file if necessary into the right spot
- Reindex
- Repeat searches - both should not bring up the correct record.

Signed-off-by: Indranil Das Gupta (L2C2 Technologies) <indradg@gmail.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
2015-06-22 11:26:13 -03:00
Jonathan Druart
b5e9691060 Bug 8992: Add 7..$3 to the Indentifier-standard index
Signed-off-by: valerie bertrand <valerie.bertrand@univ-lyon3.fr>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
2015-04-28 15:47:40 -03:00
9dbedfd854 Bug 13981 - Transliterate rule for oe and ae
NOTE : I use HTML codes for special characters to avoir encoding issues in patch file.

In ICU configuration, add a transliterate rule for
&oelig; = oe
&aelig; = ae

Test plan :
- Without patch
- Create a record R1 with title containing for example "c&oelig;ur"
- Create a record R2 with title containing for example "coeur"
- Index those records
- Search for "c&oelig;ur"
=> You only find R1
- Search for "coeur"
=> You only find R2
- Apply patch
- Restart zebra
- Index R1 and R2
- Search for "c&oelig;ur"
=> You find R1 and R2
- Search for "coeur"
=> You find R1 and R2
(Same test plan for ae)

------
Tested with all variants of Ae ae Oe oe. Search worked as expected.
Note: The words with special characters were not highlighted, but I think this can be done in an other bug.
Signed-off-by: Marc Veron <veron@veron.ch>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-04-20 10:03:41 -03:00
Nick Clemens
3e136b4f9f Bug 13800 - Diacritics not mapped
This patch adds a mapping for the lower case ð character to word-phrase-utf.chr ( Ð was already mapped to d)

To test:
1. Add a record with the  ð character (Arnaldur Indriðason is an example author)
2. Rebuild zebra
3. Search for your record using d instead of ð and verify it is not found
4. Apply patch and copy word-phrase-utf.chr to the appropriate folder
5. Restart and rebuild zebra
6. Search for your record using d instead of ð and verify it is found

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
works as expected

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-04-08 15:13:46 -03:00
Zeno Tajoli
c29a53ea20 Bug 12948: Use word indexing for language (MARC21)
This patch is for MARC21. To test:
1)Setup a site with
 MARC21
2)Insert 2 record, one lang A in 041 and 008 pos
 35-37 an other with lang A in 041 and lang B in 008 pos
 35-37
3)Index them
4)Search in advanced search with filter
 'languare' for lan A. You will see 2 records
5)Search in
 advanced search with filter 'languare' for lan B. You will
 see 0 records
6)Apply the patch
7)Full reindex
8)Search in advanced search
 with filter 'languare' for lan B. You will see 1 records

http://bugs.koha-community.org/show_bug.cgi?id=12948

Signed-off-by: Magnus Enger <magnus@enger.priv.no>
I have *not* actually tested this, but the changes are identical to the ones
done for NORMARC, which I have tested, so I think it is safe to sign off. If
anyone disagrees, please reset the bug to "Needs signoff".

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-02-20 11:51:59 -03:00
Zeno Tajoli
bf89e306a8 Bug 12948: Use word indexing for language (NORMARC)
This patch is for Normarc
Same test plan as patch for MARC21, except you need a setup with Normarc.

http://bugs.koha-community.org/show_bug.cgi?id=12948
Signed-off-by: Magnus Enger <magnus@enger.priv.no>

- Added a record with "bul" in 008pos35-37
- Verified that this did not turn up in an advanced search with language =
  Bulgarian
- Applied the patch
- I was testing on a gitified install, so I had to copy the patched index file
  to the right location with this command:

sudo cp etc/zebradb/marc_defs/normarc/biblios/biblio-zebra-indexdefs.xsl \
/etc/koha/zebradb/marc_defs/normarc/biblios/biblio-zebra-indexdefs.xsl

- Did a full reindex
- Verified that the record *did* turn up in an advanced search with language =
  Bulgarian
- Signing off! Thanks Zeno!

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-02-20 11:51:50 -03:00
a73c464f6e Bug 11927 - Add greek support to CHR (followup)
Small error in word-phrase-utf.chr.
It generates this logs :
17:03:25-21/01 zebraidx(10636) [warn] Map: 'ς' has no mapping
17:03:25-21/01 zebraidx(10636) [warn] duplicate entry for charmap from 'Σ'

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-01-22 18:22:04 -03:00
6f5f8b112f Bug 11927 - Small corrections on word-phrase-utf.chr
Small fixes :
more space characters : ¡¿
uppercase AE missing in equivalent
some trailling spaces

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-01-21 10:59:17 -03:00
5ac46633ef Bug 11927 - Add greek to word-phrase-utf.chr
Add greek support in word-phrase-utf.chr for searching in a Greek catalog (it can also contain latin records).

Developped in collaboration with Giannis Kourmoulis <ikourmou@lib.auth.gr>

Test plan :
- Install using CHR zebra indexing
- Index a greek catalog
- Look for results with mixed uppercase, lowercase and diacritics in title

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-01-21 10:59:11 -03:00
d0d150a0c5 Bug 11927 - Add greek chr lang_def file
Add the sort-string-utf.chr for sorting Greek catalog (it can also contain latin records).

Developped in collaboration with Giannis Kourmoulis <ikourmou@lib.auth.gr>

Test plan :
- Install using "gr" in "Primary language for Zebra indexing"
- Index a greek catalog
- Sort by title and check sorting is correct

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-01-21 10:59:04 -03:00
532b41934c Bug 13157: (QA followup) homebranch is 995$b on UNIMARC frameworks
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
2014-11-25 15:27:12 -03:00
9ebb6ba5d1 Bug 13157: UNIMARC holdingbranch facet is 995$c not 995$b
Fix a typo. Not test plan required, just a look at default UNIMARC framework.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
2014-11-25 15:27:05 -03:00
9220482cd3 Bug 13064 - Indexing problem with ICU on control characters
The ICU configuration files contains a rule to remove control characters :
  <transform rule="[:Control:] Any-Remove"/>
This rule is before tokenization.

The problem is that "[:Control:]" regex contains line feed, carriage return and tab. See http://www.regular-expressions.info/posixbrackets.html.
So when several lines are indexed, last word of line is joined with first line of next line. Thoses words are then not searchable.

For example :
  First line
  Second line
This will become "First lineSecond line", tokenized as "First", "lineSecond" and "line".

Test plan :
- Use ICU in Zebra configuration
- Choose an indexed field, like 300$a
- Create a new record
- Enter several lines in choosen field, like :
  First line
  Second line
- Index this record
=> Without patch the search on "Second" does not return the record
=> With patch the search on "Second" returns the record
- Same tests with tab and carriage return instead of line feed

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2014-11-14 12:03:12 -03:00
d4a7fa8580 Bug 13163: NORMARC DOM config missing <id> entry
This patch fixes the biblio-koha-indexdefs.xml for NORMARC, so
it includes the <id> element.

Because of how our DOM files work, the resulting biblio-zebra-indexdefs.xsl
for NORMARC picked the whole MARC record as ID, so every time the record
was edited, the id wouldn't match and a new record was created.

To test:
- Have a MARCXML record
- run:
  $ xsltproc etc/zebradb/marc_defs/normarc/biblios/biblio-zebra-indexdefs.xsl the_record | less
=> FAIL: verify the z:id property on the <z:record> line contains all subfields concatenated
- Apply the patch
- re-run the xsltproc line
=> SUCCESS: z:id contains the 999$c number
- Sign off :-D

Regards

Signed-off-by: Frederic Demians <f.demians@tamil.fr>

Known bug with DOM: Without <z:id> indexing biblionumber Zebra hasn't it record
unique ID, and so fails to identify existing records. Works as described. 999$c
is linked to biblionumber in default Normarc framework.

Signed-off-by: Magnus Enger <magnus@enger.priv.no>

I have applied the patch to my production server, and at least one customer has
confirmed that it fixes the problem with multiple copies of records in search
results.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes tests and QA script, fix matches what we have for the other MARC flavours.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2014-10-31 16:45:04 -03:00
c217b2c418 Revert "Bug 9828: More specific indexing of UNIMARC 6XX fields"
This reverts commit 0dd1ac40a0.
2014-10-28 12:02:34 -03:00
e43f012af6 Revert "ug 9828 : Add and fix comments in UNIMARC biblio-koha-indexdefs.xml"
This reverts commit 5bbe42932e.
2014-10-28 12:02:22 -03:00
b108a111f6 Revert "Bug 9828 : Followup for Queryparser and deletion of useless 6XX$9"
This reverts commit 49788987b2.
2014-10-28 12:02:09 -03:00
Mathieu Saby
49788987b2 Bug 9828 : Followup for Queryparser and deletion of useless 6XX$9
This followup
- changes some indexes in Queryparser configuration file
- supresses some clearly useless 6XX$9 in biblio-koha-indexdefs.xml and adds 2 new ones, probably useless (not sure of that)
- change the name of index Subject-geographical to Subject-name-geographical in ccl.properties (to match bib1.att)
the xsl file zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl was generated with the following command:
xsltproc zebradb/xsl/koha-indexdefs-to-zebra.xsl zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml > zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl

To test :
1) Apply the 3 patches
2) copy the modified files from the source directory to the directory where you store the config files for Zebra and Queryparser
The files modified by the 3 patches and that need to be copied are:
etc/zebradb/biblios/etc/bib1.att
etc/zebradb/ccl.properties
etc/searchengine/queryparser.yaml
etc/zebradb/ccl.properties
.../unimarc/biblios/biblio-koha-indexdefs.xml
.../unimarc/biblios/biblio-zebra-indexdefs.xsl
3) Rebuild Zebra
4) Create a record A with some values in critical fields, for example:
- the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2
- the string "subform" in 600$j
4) Create a record B with the string "subgeo" in 606$y
5) Create a record C with the string "subdate" in 606$z
WITHOUT QP activated in sysprefs ("Don't try to use QP"):
6) try to search "su:test9828". You should have no results
7) try to search "su-genre:subform". You should have 1 result : record A
8) try to search "su-geo:subgeo". You should have 1 result : record B
9) try to search "su-chrono:subdate". You should have 1 result : record C
10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant
WITH QP activated in sysprefs:
Same tests

Signed-off-by: Nick Clemens <nick@quecheelibrary.org>

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2014-10-27 12:46:47 -03:00
Mathieu Saby
5bbe42932e ug 9828 : Add and fix comments in UNIMARC biblio-koha-indexdefs.xml
Only cosmetic :
- the references to lines record.abs are now useless and outdated
- some comments added in record.abs could be usefull in biblio-koha-indexdefs.xml

No change expected, only comments

Signed-off-by: Nick Clemens <nick@quecheelibrary.org>

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2014-10-27 12:46:44 -03:00
Mathieu Saby
0dd1ac40a0 Bug 9828: More specific indexing of UNIMARC 6XX fields
[New commit on 18 Aug 2014 : rebased, and DOM indexing only]

Issues to fix :
Most of 6XX may contain a $2 that identifies the system used for indexing. It should not be indexed.
In French libraries, $2 contains "rameau". So searching books about the music composer "Rameau" retreive thousands of records!
For some 6XX fiels, other subfields should not be indexed, for example dates of persons and family, or adresses.
In Unimarc guide, 600$t,601$t,602$t are said to exist but to be "not used". I keep them indexed.

Additionnally, subject indexing could be improved by using specific indexes for each 6XX if possible :
In ccl.properties :
- su-to, su-geo and su-ut are defined as aliases of Subject.
- a specific index is defined, but not used in record.abs : Subject-name-personal, alias su-na
We can use these indexes and create new specific indexes by using existing bib1 attributes.

We could also index $j,$x,$y,$z subdivision in specific indexes.

This patch does the following changes :
1) For all 6XX : Not indexing $2 (LSCH, Rameau...), $3 and $5
2) Suppressing the indexing of some specific subfields, depending on the field:
600 : Personal name used as a subject // see Marc21 600
not indexing c (additional elements),f (dates),p (address/affiliation)
602 : Family name used as a subject // see Marc21 600 3X
not indexing f (dates)
616 : Trademark
not indexing c,f
3) For all 6XX : index $j,$x,$y,$z in several indexes in addition to the specfific index for their 6XX field:
4) Define in ccl.properties some specific indexes :
Subject-name-conference 1=1073 => alias su-conf
Subject-name-corporate 1=1074 => alias su-corp
Subject-genre-form 1=1075 => alias su-genre and su-form
Subject-geographical 1=1076 => alias su-geo
Subject-chronological 1=1077 => alias su-chrono
Subject-title 1=1078 => alias su-ut and su-ti
Subject-topical 1=1079 => alias su-to
5) Adding new aliases in Search.pm :
su-chrono, su-form, su-genre, su-corp, su-conf, su-ti
6) Using these new indexes in for
600 : Subject and Subject-Personal-Name ; all subfields except subdivisions in Personal-name
601 : Subject, Subject-name-conference and Subject-name-corporate and Subject-name-conf ; all subfields except subdivisions in Corporate-name and Conference-name
602 : same as 600 but could be improved later
604 : Subject and Subject-title ; $a in Subject-Personal-Name ; all subfields except subdivisions in Name-and-Title
605 : Subject and Subject-title
606 : Subject and Subject-topical
607 : Subject and Subject-geographical ; all subfields except subdivisions in Name-geographic
608 : Subject and Subject-genre-form

To test :

A. In a UNIMARC-DOM indexing environment
1) Apply the patch
2) Rebuild zebra
3) Create a record A with some values in critical fields, for example:
- the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2
- the string "subform" in 600$j
4) Create a record B with the string "subgeo" in 606$y
5) Create a record C with the string "subdate" in 606$z
6) try to search "su:test9828". You should have no results
7) try to search "su-genre:subform". You should have 1 result : record A
8) try to search "su-geo:subgeo". You should have 1 result : record B
9) try to search "su-chrono:subdate". You should have 1 result : record C
10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant

Indexing of subjects could maybe be improved later

Signed-off-by: Nick Clemens <nick@quecheelibrary.org>

All seems to work as expected, I am not super-familiar with UNIMARC but I wonder if in su-corp and su-conf the subdivisions might be useful (e.g. France-Gendarmie / Staatsbibliothek-Berlin)

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2014-10-27 12:46:42 -03:00
a657589b1f Bug 11362 - increase zebra AUTH register sizes, from 4G to 20G
To test...

- apply patch
- build and install a new Koha .deb from patched codebase
- create a new Koha instance
- add some authority records to instance
- do a full zebra reindex
- do an authorities search, and get some results

note: this patch does not fix existing Koha instances, just new ones
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2014-10-24 09:41:04 -03:00