This patch adds a regression test for the condition noted
in bug 9578, where attempting a sort of a Zebra search that
fails because of an invalid query crashes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch makes Koha <-> Zebra use MARCXML for the serialization when
using DOM, and USMARC for GRS-1.
* The following functions are modified to set the Zebra record syntax
according to the current sysprefs and configuration:
- C4::Context->Zconn
- C4::Context-_new_Zconn
* A new function 'new_record_from_zebra' is introduced, which checks the
context we are in, and creates the MARC::Record object using the right
constructor.
The following packages get touched to make use of the new function:
- C4::Search
- C4::AuthoritiesMarc
and the same happens to the UI scripts that make use of them (both in
the OPAC and STAFF interfaces).
* Calls to the unsafe ZOOM::Record->render()[1] method are removed.
Due to this last change the code for building facets was rewritten. And
for performance on the facets creation I pushed higher version
dependencies for MARC::File::XML and MARC::Record (we rely on
MARC::Field->as_string).
* Calls to MARC::Record->new_from_xml and MARC::Record->new_from_usmarc
are wrapped with eval for catching problems [2].
* As of bug 3087, UNIMARC uses the 'unimarc' record syntax. this case is
correctly handled.
* As of bug 7818 misc/migration_tools/rebuild_zebra.pl behaves like:
- bib_index_mode (defaults to 'grs1' if not specified)
- auth_index_mode (defaults to 'dom')
here we do exactly the same.
To test:
- prove t/db_dependent/Search.t should pass.
- Searching should remain functional.
- Indexing and searching for a big record should work (that's what the
unit tests do).
- Test an index scan search (on the staff interface):
Search > More options > Check "Scan indexes".
- Enable 'itemBarcodeFallbackSearch' and try to circulate any word, it
shouldn't break.
- Searching for a biblio in a new subscription shouldn't break.
- Running bulkmarcimport.pl shouldn't break.
- And so on... for the rest of the .pl files.
[1] http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#render()
[2] a record that cannot be parsed by MARC::Record is simply skipped (bug 10684)
Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This makes it easier to read. Also fixes the test count, and is rebased
against 11499 which is needed to correctly test the patches for this bug
(i.e. indexing mode needs to be correctly set in koha-conf.xml)
To test:
[1] Run prove -v t/db_dependent/Search.t
Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This adds direct tests of indexing and retrieving
bib records that are larger than the ISO2709 limit
of 99,999 octets.
To test: verify that prove -v t/db_dependent/Search.t passes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch fixes an issue where chosing 'None' as the sort order
for an authority search would result in zero hits if QueryParser is
eanbled.
This patch also adds some additional test cases.
To test:
[1] Enable QueryParser.
[2] Perform an authority search in the staff interface that
uses 'Heading A-Z' as the sort order and returns hits.
[3] Run the same search, but with the sort order set to 'None'.
No hits are returned.
[4] Apply the patch.
[5] Do step 3 again. This time, hits should be returned.
[6] Verify that prove -v t/db_dependent/Search.t passes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch duplicates the SearchAuthority tests, with QP enabled.
Searching shouldn't be affected by QP. Instructions for manually
reproducing this can be read in the first comment for the bug entry.
Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
t/db_dependent/Search.t creates its own temporary koha-conf.xml and
related files, and it fails to properly set indexing modes for
bibliographic and authority records.
This patch adds regression tests for that.
To test:
$ prove -v t/db_dependent/Search.t 2> /dev/null | grep 11499
The test should fail unless the main page is applied.
Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Under certain circumstances, namely where QueryParser is off,
QueryWeightFields is on, and QueryFuzzy and QueryAutoTruncate are
off, a search with a double-quote embedded in it can fail: for
example
web application
would return results where
web "application
does not.
This patch adds an automated regression test.
To test:
[1] Apply this patch and run prove -v t/db_dependent/Search.t;
two tests should fail.
[2] After applying the main patch that fixes the bug, the
prove -v t/db_dependent/Search.t should succeed.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch adds a regression test for Koha's ability
to gracefully handle records that Zebra will index but
which MARC::Record won't parse. In particular, MARC::File::USMARC
croaks on records that contain non-alphanumeric characters in the
tag name.
To test:
[1] After applying this test, run prove -v t/db_dependent/Search.t
The test script will die when it gets to the regression test.
[2] After applying the main patches that fix the bug, the test
script should pass.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
QueryParser was loading redundant search aliases, which in turn resulted
in certain queries involving two or more clauses joined by a Boolean
operator getting parsed in correctly.
To test:
[1] Enable QueryParser.
[2] Arrange your database so that some some bib records contain
the keyword 'history' while a smaller subset contain
both 'history' and 'earth'. (The exact words used don't
matter.)
[3] Run the following search from the OPAC global search bar:
kw:history && kw:earth
[4] Observe that the records that are returned include *all* bibs
in the database with the keyword 'history'. In other words, the
restriction that the records must also contain 'earth' is not
observed.
[5] Apply the patch and rerun the search.
[6] This time, the results that are returned should include only those
that contain both 'history' and 'earth'.
[7] Verify that prove -v t/QueryParser.t passes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes test plans, all tests and QA script.
Also checked the db_dependent/Search.t tests.
Tried some other searches, all seems to be working correctly.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
I reproduce the bug and I confirm this patch fixes it.
Without the patch, kw:history && kw:earth, I got:
Search biblios OK 4 1 1+0 RPN @attrset Bib-1 @attr 4=6 @attr 1=1016 history
With the path, I get:
Search biblios OK 2 1 1+0 RPN @attrset Bib-1 @and @attr 4=6 @attr 1=1016 history @attr 4=6 @attr 1=1016 earth
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch adds regression tests to verify that
Boolean searches using QueryParser function correctly.
This patch also ensures that QP is correctly initalized
when Search.t is run.
To test:
[1] Apply this patch and the following patch.
[2] Verify that prove -v t/QueryParser.t works
[3] Verify that prove -v t/db_dependent/Search.t works
[4] (optional) instead of applying both patches at the same
time, apply only the regression test patch and run the
tests listed in steps 2 and 3. The following tests
should fail:
t/db_dependent/Search.t (Wstat: 512 Tests: 198 Failed: 2)
Failed tests: 42, 71
Non-zero exit status: 2
t/QueryParser.t (Wstat: 256 Tests: 28 Failed: 1)
Failed test: 12
Non-zero exit status: 1
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch adds a regression test for doing a form-based catalog
search using an index that contains the string 'ns' in its name. In
this case, the index being tested is 'Code-institution'.
To test
-------
[1] Prove -v t/db_dependent/Search.t should have two failed tests;
one each for GRS-1 and DOM testing.
[2] After applying the main patch for this bug, the tests should
pass.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch enhances t/db_dependent/Search.t to test indexing and
searching of authority records. It supplies a sample authority file
for MARC21 (consisting of the record for William Shakespeare from the
Library of Congress) and a sample UNIMARC authority file supplied
by Henri-Damien Laurent.
It also adds tests for both MARC21 and UNIMARC authorities.
When the main patch for bug 7421 is applied, t/db_dependent/Search.t
should pass.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch adds a regression test that verifies that searching the
UNIMARC 'item' index for a value that is indexed by virtue of being
included in the fallback index definition for the 995 field works for
both GRS-1 and DOM.
The main patch will allow t/db_dependent/Search.t to pass.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch adds a few basic search and indexing tests for UNIMARC,
using sample data provied by Frédéric Demians. The new tests
by no means completely cover the possibilities, but should be
considered a starting point for future tests.
To test:
[1] Verify that prove -v t/db_dependent/Search.t passes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch refactors t/db_dependent/Search.t so that it can be
subsequently modified to add tests for indexing and searching
UNIMARC records without duplicating the setup code.
This patch also moves the MARC21 records from t/db_dependent/data/zebraexport
to t/db_dependent/data/marc21/zebraexport so that UNIMARC sample
data can be slotted in t/db_dependent/data/unimarc/zebraexport.
To test:
[1] Verify that prove -v t/db_dependent/Search.t passes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
There was some churn in previous patches about the desired
name of the music publisher number index, so this patch
adds tests to confirm that both the old and new names
work as CCL keywords.
To test:
[1] Verify that t/db_dependent/Search.t passes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch expands t/db_dependent/Search.t to run
the same tests using both the GRS-1 and DOM indexing
modes. It also adds hooks in zebra_config.pl to make
it easier to stage test cases for non-MARC21 Zebra
indexing.
Note that in DOM mode one of the tests is currently a
TODO, as relevance ranking for wegihted queries differs
between GRS-1 and DOM.
To test:
[1] Verify that prove -v t/db_dependent/Search.t passes.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch updates the wthdrawn field in items and deleteditems to be
withdrawn instead. No functional changes are made.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment: Save for translation files (that will be fixed on next release),
only occurrence of wthdrawn is on updatedatabase.pl
No koha-qa errors.
This touch many files, and I did not test everything,
but all seems normal. I think that any problem could
be fixed later.
Perhaps both entries in updatedatabase.pl could be joined
into one, but thats for QA.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This patch removes 'insecure' system preference.
Also removes remaining code that make use of
the preference. It's broken anyway.
Only remains a reference in POD of C4/Boolean.pm
To test:
1) If you like, enable 'insecure' syspref. Broken system.
WARN: be prepared to revert value in database.
2) Apply the patch
3) Run updatedatabase.pl
4) Check that Staff login proceeds as usual.
5) Check that 'insecure' syspref is no more.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Amended patch: Remove 2 occurrences of insecure (in comment only)
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Removed NoZebra vestiges. This comprises several code blocks that depend on the NoZebra syspref and NZ related functions/methods.
C4::Biblio->
GetNoZebraIndexes
_DelBiblioNoZebra
_AddBiblioNoZebra
C4::Search->
NZgetRecords
NZanalyse
NZoperatorAND
NZoperatorOR
NZoperatorNOT
NZorder
C4::Installer->
set_indexing_engine
Sponsored-by: Universidad Nacional de Córdoba
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
With the inclusion of this patch, all searches will (try) to use
QueryParser for handling queries for both the bibliographic and authority
databases if UseQueryParser is enabled. If QueryParser is unavailable,
UseQueryParser is disabled, or the search uses CCL indexes, the old
search code will be used.
To test:
1) Apply patch.
2) Run the unit test with `prove t/QueryParser.t`
3) Enable the UseQueryParser syspref.
4) Try searches that should return results in the following places:
* OPAC (simple search)
* OPAC (advanced search)
* OPAC (authorities)
* Staff client (header search)
* Staff client (advanced search)
* Staff client (cataloging search)
* Staff client (authorities)
* Staff client (importing a batch using a match point)
* Staff client (searching for an item for adding to a label)
* Staff client (acquisitions)
* Staff client (searching for a record to create a serial)
* ANYWHERE ELSE I HAVE FORGOTTEN
5) Disable the UseQueryParser syspref. Repeat at least some of the
searches you did above.
6) If all searches worked, sign off.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Elliott Davis <elliott@bywatersolions.com>
Searching still works as expected for variuos places.
QueryParser syspref seemed to be enabled by default
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
When working with hierarchical subject headings, it is sometimes helpful
to do a search for all records with a specific subject, plus
broader/narrower/related subjects. This patch adds support for these
"exploded" subject searches to Koha.
To test:
1) Make sure you have a bunch of hierarchical subjects. I created
geographical subjects for "Arizona," "United States," and "Phoenix,"
and linked them together using 551s, and made sure I had a half
dozen records linking to each (but not all to all three).
2) Do a search for su-br:Arizona (or choose "Subject and broader terms"
on the advanced search screen with "more options" displayed), and
check that you get the records with the subject "Arizona" and the
records with the subject "United States"
3) Do a search for su-na:Arizona (or choose "Subject and narrower terms"
on the advanced search screen with "more options" displayed), and
check that you get the records with the subject "Arizona" and the
records with the subject "Phoenix"
4) Do a search for su-rl:Arizona (or choose "Subject and related terms"
on the advanced search screen with "more options" displayed), and
check that you get the records with the subject "Arizona," the
records with the subject "United States," and the records with the
subject "Phoenix"
5) Ensure that other searches still work (keyword, subject, ccl,
whatever)
6) Sign off
Technical details:
This patch adds a shim in front of C4::Search::buildQuery in order to
preprocess the query and call the _handle_exploding_search callback.
This shim will allow us to gradually offload query parsing to a new
query parser module.
Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
The Zebra test uses Unix-domain sockets, which won't work if
Koha is being installed or tested from an NFS- or SSHFS-mounted
clone. Therefore, create (and clenaup) a temporary directory on
(hopefully) a normal filesytem to run the test.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This fixes failures of the test cases to run for me
on Perl 5.10.1 using prove -v t/db_dependent/Search.t
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch adds a unit test for C4::Search in t/db_dependent. In order
to test the functioning of the Zebra search, this patch actually includes
an entire Zebra sandbox, and pre-indexed files, which are stored in
t/db_dependent/data (the configuration files are generated on the fly).
This test depends on Test::Warn, Test::MockModule, and DBD::Mock.
To test:
1) Run the test. There should be no failures, and no warnings.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Patch changes the French string (with French stopwords) to an English string with English stopwords, as only English stopwords are installed in the default data.
This patch will allow Search.t to run on the database attached to this bug.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
