Commit graph

55 commits

Author SHA1 Message Date
Aleisha Amohia
33856898fd
Bug 26611: Make authority record matching use required match checks
This patch fixes record matching for authorities to correctly apply required match checks.

To test:

1) Create a record matching rule:

Threshold: 100
Record type: Authority record

Match point 1
Search index: subject-topical
Score: 100
Tag: 150
Subfields: a

Match check 1
Both source and target record check fields:
Tag: 040
Subfields: a

This rule says we want to match on the topical term heading, then confirm the match with 040$a.

2) Create two topical term authority records

Authority 1:
150$a: Test
040$a: A

Authority 2:
150$a: Test
040$a: B

3) Export authority 2 and save

4) Go to Cataloging -> Stage records for import

5) Upload your downloaded authority 2 file. Change the record type to Authority. Choose your new record matching rule.

6) Stage for import.

7) Confirm that with the record matching rule applied, both authorities 1 and 2 show as possible matches, even though only authority 2 has a matching 040$a.

8) Apply the patch and restart services

9) Change the matching rule to "Do not look for matching records" and apply. This is to essentially refresh the page.

10) Change the matching rule to your new matching rule and apply. Confirm only the matching authority 2 shows and is selected.

Sponsored-by: Waikato Institute of Technology
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-05-24 11:21:31 -03:00
4628d2e81c
Bug 33183: Error inserting matchpoint_components when creating record matching rules with MariaDB 10.6
The word "offset" is now a keyword. At the very least we need to escape the fieldname with backticks.
It would be nice to switch to Koha::Objects for this module eventually.

C4::Matcher::_store_matchpoint(): DBI Exception: DBD::mysql::st execute failed: You have an error in
your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax
to use near 'offset, length) VALUES ('27', '1', '999'...' at line 2 at /usr/share/koha/lib/C4/Matcher.pm
line 314

Test Plan:
1) Set up Koha with MariaDB 10.6
2) Attempt to create a matching rule
3) Note the error screen
4) Apply this patch
5) Restart all the things!
6) Attempt to create another matching rule
7) No errors this time

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-03-16 16:23:01 -03:00
eded6edacc
Bug 28290: Don't send subfields to 'as_string' if none to send
This fixes an error where we send a blank string to
Marc::Field->as_string

The subroutine fails as the regex is marked as incomplete as there is
nothing between the brackets

Also fixes a missing hidden input for 'item_action' when no items are
staged during import

 1 - Define a new matching rule with no subfields
    Match-threshold: 1000
    Record type: bibliographic record
    Search-index: Other-control-number
    Score: 1000
    Tag: 035
    Subfields:
    Offset:
    Length:
 2 - Attempt to stage a record with an 035 using this rule (i will
     attach example to bug)
 3 - Staging fails
 4 - Stage again with no matching rule
 5 - Manage the staged record, try to apply the matching rule
 6 - It fails, 500 error
 7 - Apply patch
 8 - Restart all the things
 9 - Apply rule again, it works!
10 - Stage the record again using the rule for matching
11 - It stages and matches!

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-07 13:50:21 -07:00
9d6d641d1f Bug 17600: Standardize our EXPORT_OK
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.

That way we will need to explicitely define the subroutine we want to
use from a module.

This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests

And a lot of other manual changes.

export.pl is a dirty script that can be found on bug 17600.

"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;

The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules

Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).

EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.

@EXPORT and @EXPORT_OK are the two main variables used during export operation.

@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.

@EXPORT_OK does export of symbols on demand basis.
"""

If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
  - use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-07-16 08:58:47 +02:00
baa2260503 Bug 26199: Include LDR for match check/match point
This patch adds the ability to specify the LDR instead of a numeric tag
when defining match checks in record matching rules. This is particularly
useful for checking LDR6 "Type of record" when importing records.

Test plan:
0. prove t/Matcher.t

1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1
2. Click "Save" and choose "MARCXML"
3. Copy bib-1.marcxml to bib-1-mod.marcxml
4. Change "01344cam a22003014i 4500" to "01344cmm a22003014i 4500"
5. Go to http://localhost:8081/cgi-bin/koha/admin/matching-rules.pl?op=edit_matching_rule&matcher_id=3
6. Click "Add match check"
7. Fill out the match checks:
Source:
Tag: LDR
Offset: 6
Length: 1
Target:
Tag: LDR
Offset: 6
Length: 1

8. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl
9. Choose "bib-1-mod.marcxml" and click "Upload file"
10. Choose "KohaBiblio (999$c)" for "Record matching rule"
11. Click "Stage for import"
12. Click "Manage staged records" on resulting page
13. Note the "Match type" says "No match"

14. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl
15. Choose "bib-1.marcxml" and click "Upload file"
16. Choose "KohaBiblio (999$c)" for "Record matching rule"
17. Click "Stage for import"
18. Click "Manage staged records" on resulting page
19. Note the Match type says "Match found"

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-03-09 17:20:47 +01:00
Julian Maurice
b168f4a2e9 Bug 21395: Make perlcritic happy
This patch adds a .perlcriticrc (copied from qa-test-tools) and fixes
almost all perlcrictic violations according to this .perlcriticrc
The remaining violations are silenced out by appending a '## no critic'
to the offending lines. They can still be seen by using the --force
option of perlcritic
This patch also modify t/00-testcritic.t to check all Perl files using
the new .perlcriticrc.
I'm not sure if this test script is still useful as it is now equivalent
to `perlcritic --quiet .` and it looks like it is much slower
(approximatively 5 times slower on my machine)

Test plan:
1. Run `perlcritic --quiet .` from the root directory. It should output
   nothing
2. Run `perlcritic --quiet --force .`. It should output 7 errors (6
   StringyEval, 1 BarewordFileHandles)
3. Run `TEST_QA=1 prove t/00-testcritic.t`
4. Read the patch. Check that all changes make sense and do not
   introduce undesired behaviour

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-06-29 12:37:02 +02:00
52e95fff34
Bug 24735: Remove QueryParser-related code
At the last development meeting we have voted to remove the
QueryParser-related code
https://wiki.koha-community.org/wiki/Development_IRC_meeting_19_February_2020

Hea tells us that it has not been adopted, and the code/bug tracker that
it is not really usable as it. As nobody is willing to work on it, we
decided to remove it instead.

Test plan:
 % prove t/db_dependent/Search.t
must return green

See commits from bug 9239 and confirm that the code is removed in this
patch.

Also play with the search on the UI and confirm that you do not see
obvious regressions

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-03-02 11:13:03 +00:00
Andreas Jonsson
4111f4f186
Bug 24348: Pass marc record to _passes_required_checks
Test plan:

* Add record matching rule:

 Matching rule code: test
 Description: 001 and 003
 Match threshold: 1000
 Record type: Bibliografic record

 Match points:
    Search index: Control-number
    Score: 1000
    Tag: 001
    Subfields:
    Offset:
    Length:
    Normalization rule: none

 Required match checks:

    Source (incoming) record check field

    Tag: 003
    Subfields:
    Offset:
    Length:
    Normalization rule: none

    Target (database) record check field

    Tag: 003
    Subfields:
    Offset:
    Length:
    Normalization rule: none

* Note the match rule identity number.
* Stage a marc-file for import, for instance this one ftp://ftp.libris.kb.se/pub/export2/X/marc/X.20200104.marc

sudo /usr/sbin/koha-shell -c "perl /home/vagrant/kohaclone/misc/stage_file.pl --match 4 --file 'X.20200104.marc' --format ISO2709 --comment 'test'" kohadev

* Note the batch number and commit the file using the batch number:

sudo /usr/sbin/koha-shell -c "perl /home/vagrant/kohaclone/misc/commit_file.pl --batch-number 1" kohadev

* Again, stage the same marc-file  for import:

sudo /usr/sbin/koha-shell -c "perl /home/vagrant/kohaclone/misc/stage_file.pl --match 4 --file 'X.20200104.marc' --format ISO2709 --comment 'test'" kohadev

* Note the number of records matched.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-01-28 14:54:29 +00:00
3d15819443
Bug 23324: Add an ISBN normalization routine
To test:
1 - Set SearchEngine to ElasticSearch
2 - Stage the sample file (import it if it doesn't already exist in your catalog and then stage again)
3 - Set matching rule to ISBN
4 - No matches found
5 - Apply patch
6 - Apply no matchign rule
7 - Change the ISBN matching rule to use ISBN normalizer
8 - Apply matching rule for ISBN
9 - It matches!

Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-08-16 12:26:48 +01:00
ce1b79d55d
Bug 23322: Correct case for boolean in ES search
I did not include tests only because this is a very small reasonable change and routine has no tests.
It does not affect behaviour, it only touches syntax lightly.

To test:
  0 - Set SearchEngine to ElasticSearch
  1 - Import the example record (a version is included in sample data in devbox/testing docker so you can skip)
  2 - Stage the attached example record
  3 - Match using 'ISBN' rule
  4 - No matches found
  5 - Apply patch
  6 - Restart all the things
  7 - Reapply matching with no rule
  8 - Reapply with ISBN matcing
  9 - It matches!
 10 - Set SearchEngine to Zebra
 11 - Reapply matching with no rule
 12 - Reapply with ISBN matching
 13 - Matching works as before in Zebra

Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-08-08 13:04:59 +01:00
68eeefa07e
Bug 22721: Remove frameworkcode parameter in GetMarcFromKohaField calls
Test plan:
Run tests, at least:
    t/db_dependent/Biblio.t
    t/db_dependent/Biblio/TransformHtmlToMarc.t
    t/db_dependent/Charset.t
    t/db_dependent/Circulation/GetTopIssues.t
    t/db_dependent/Filter_MARC_ViewPolicy.t
    t/db_dependent/ImportBatch.t
    t/db_dependent/Items.t
    t/db_dependent/Items/AutomaticItemModificationByAge.t
    t/db_dependent/Items/GetItemsForInventory.t
    t/db_dependent/Koha/Filter/EmbedItemsAvailability.t
    t/db_dependent/Serials.t
    t/db_dependent/XISBN.t
    t/db_dependent/FrameworkPlugin.t

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-07-15 11:28:08 +01:00
53545ba739 Bug 22046: Simplify and unify the code for get_matches
To test:
1 - Stage some records for import
2 - Manage the records
3 - Use several different matching rules and note the results
4 - Apply patch
5 - Try several matching rules again and note the results have not
changed
6 - Try under both search engines (Zebra and ES)

Signed-off-by: Liz Rea <wizzyrea@gmail.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-05-23 14:44:42 +00:00
Ere Maijala
be2bdfd511 Bug 19365: Fix several issues with the Elasticsearch code
Also optimize it so it's actually usable.

Test plan:

1. To test it properly you need biblio and authority data. You might get away with enabling AutoCreateAuthorities and BiblioAddsAuthorities so that authorities are created in the process. Another option would be to import authorities first e.g. from LoC.
2. Make sure the authority index has been properly created with "misc/search_tools/rebuild_elastic_search.pl -a -d"
3. Run "misc/link_bibs_to_authorities.pl -v -l" twice and observe the results.

Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:01 +00:00
32c4dbfd68 Bug 21606: [sql_modes] Fix matching rules insert
Fix:
Incorrect integer value: '' for column 'offset'
Incorrect integer value: '' for column 'score'

Test plan:
Add/edit matching rules

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-02 10:33:08 +00:00
Cori Lynn Arnold
e0ca753a2f Bug 18584: Removed white space in C4/Matcher.pm
Test Plan:
    1-go to C4/Matcher.pm
    2-verify there is no whitespace at line 25

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-10-03 17:50:17 +00:00
7c78e4c97e Bug 18131: Followup - Use data if biblionumber_tag < 10
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-11-26 12:59:51 -03:00
4ebb0bbb18 Bug 18131: ES - Fix matching staged records - Authority
Occurrences of C4::AuthoritiesMarc::SearchAuthorities must be replaced
by search_auth_compat.

You need to define the search index of matching rule with one of the
values defined in %koha_to_index_name (from
Koha::SearchEngine::Elasticsearch::QueryBuilder::build_authorities_query_compat)

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-11-26 12:59:51 -03:00
eed1c91c8f Bug 18131: ES - Fix matching staged records
The code in C4::Matches::get_matches is terrible and a bug has been introduced
by bug 12478 because of its way to handle uniqueness.

If search engine is elastic, simple_search_compat returns array ref of MARC::Record,
used as a string for the key of the matches hashref we get things like
"MARC::Record=HASH(0x8f76ab0)".
Yes, terrible...

The file is never staged and we get an internal server error:
    stage-marc-import.pl: Can't locate object method "fields" via package "MARC::Record=HASH(0x8f76ab0)" (perhaps you forgot to load "MARC::Record=HASH(0x8f76ab0)"?) at /home/vagrant/kohaclone/C4/Biblio.pm line 2691

To recreate the issue:
- Set SearchEngine == Elastic
- Create a matching rule on 999$c (you need to edit the existing one and specify
'Local-number' as search index, not 'local-number')
- Import a file with bibliographic records and use the matching rule you defined.

Test plan:
Import authority and bibliographic records with Zebra and Elastic using a matching rule.
Everything should work correctly.

Note: I found a bug when importing authorities using Elastic, see bug 17255 comment 38.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-11-26 12:59:51 -03:00
6906b848a7 Bug 17710 - C4::Matcher::get_matches and C4::ImportBatch::GetBestRecordMatch should use same logic
C4::ImportBatch::GetBestRecordMatch uses SQL to sort by score descending
then candidate_match_id descending. With C4::Matcher::get_matches, I
implement the same sort but use Perl code to do it, since we're sorting
search results.

It's a simple change, but it's in a big block of code, so I don't have
unit tests.

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-06-15 15:27:46 -03:00
50f4024055 Bug 15541 - Prevent normalization during matching/import process
This patch allows you to use the "qualifier,qualifier" syntax
in the Record Matching Rules "Search Index" when using the QueryParser.
While QueryParser doesn't support this syntax, it will now fallback
correctly to non-QueryParser functionality. Without the patch, your search
will just fail silently.

This patch also adds a "skip_normalize" option to C4::Search::SimpleSearch(),
and uses the option during C4::Matcher::get_matches. This prevents
the s/:/=/g and s/=/:/g normalization. This normalization is heavy-handed,
and while it is necessary sometimes to generate a valid CCL query or
QueryParser query, C4::Matcher::get_matches() already produces a valid
CCL query, so we don't need to do this normalization.

Additionally, this normalization causes problems when you use a
Zebra register which isn't normalized: namely the "u" register.
Strings are stored "as is", so http://localhost/koha/resource/1 is
stored as is during indexing. When you search, you need to pass
the exact same thing through the query to get a match. Using
http=//localhost/koha/resource/1 in your query will yield zero results.

_TEST PLAN_

0) Apply patch

1) Create a Record Matching Rule in Koha with the following details:
    Matching rule code: TEST
    Description: Test
    Match threshold: 100
    Record type: Bibliographic
    Match point 1:
        Search index: id-other,st-urx
        Score: 100
        Tag: 024
        Subfields: a
        Normalization rule: None
2) Create a record in Koha with an indexable URI
    2a) Default framework
    2b) 024 $a http://koha-community.org/test $2 uri
    2c) 040 $c test
    2d) 245 $a This is a test record
    2e) 942 $c Books
    2f) Save (save again if cautioned about missing fields as these should autofill)
3) Do a full re-index of Zebra
4) Download your record from Koha as a .mrc file (ie isomarc, binary marc, etc)
5) Go to "Stage MARC records for import"
    5a) Upload your .marc file.
    5b) Change your "Record matching rule" to "Test"
    5c) Click Stage for import
9) It should say "1 records with at least one match in catalog per matching rule "Test".

NOTE: For completeness, you can go through this process on a clean master branch,
and note that it will say '0 records with at least one match in catalog per matching rule "TEST"'

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-03-31 13:52:44 +00:00
68ced964fd Bug 14629 - Add aggressive ISSN matching feature equivalent to the aggressive ISBN matcher
This patch adds a syspref "AggressiveMatchOnISSN" allowing for a match
of ISSNs with or without hyphens. It uses Business::ISSN in order to
follow the use of Business::ISBN and allow for validation of ISSNs

To test:
1 - Find a record in your system with an ISSN (or add one)
2 - Stage a record containing the same ISSN but lacking a hyphen
3 - Matching on ISSN should find 0 matches
4 - Repeat with no hyphen ISSN in system and hyphen ISSN in import
5 - Matching should find 0
6 - Apply patch
7 - Update datbase and install Business::ISSN
8 - Leave AggressiveMatchOnISSN as don't and repeat original tests- no
change
9 - Set AggressiveMatchOnISSN as do and repeat original test
10 - You should find a match
11 - prove t/Koha.t - all tests pass

Sponsored by North Central Regional Library System (NCRL) www.ncrl.org

Signed-off-by: Chad Roseburg <croseburg@ncrl.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-10-28 11:58:14 +00:00
4579f1ad90 Bug 17318: Make C4::Matcher::_get_match_keys handle 'norms' param
The current implementation doesn't care about that parameter, and applies
a default normalization rule that seems counter-productive (in general) for
its aleged purpose.

This patch makes it handle the following values for 'norms':

    - upper_case
    - lower_case
    - remove_spaces
    - legacy_default
    - none

They make it call the relevant Koha::Utils::Normalize routines. 'legacy_default'
is used only for backwards compatibility, but could be removed if there's consensus.

To test:
- Run:
  $ prove t/Matcher.t
=> FAIL: most _get_match_keys tests fail
- Apply the patch
- Run:
  $ prove t/Matcher.t
=> SUCCESS: Tests pass!
- Sign off :-D

Sponsored-by: FIT

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-10-21 16:16:34 +00:00
Robin Sheat
b4392018bc Bug 12478: make things using SimpleSearch use the new version
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:07 +00:00
c564f30155 Bug 16169: Change prototype for C4::Biblio::TransformMarcToKoha
subroutines should not take $dbh in parameter.
C4::Biblio::TransformMarcToKoha has it and does not use it.

Test plan:
Look at the patch and confirm that all occurrences of
TransformMarcToKoha have been modified.

Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
2016-04-07 00:04:21 +00:00
fcbd81049f Bug 15745: C4::Matcher gets CCL parsing error if term contains ? (question mark)
Signed-off-by: Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi>

Also fixes ! and +
Rebased to master
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
It makes perfect sense and works as expected. This part of the code is too
under-tested so no point requiring a regression test for such a simple change.

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-04-01 19:12:51 +00:00
3cfedcf238 Bug 16011: $VERSION - Remove empty BEGIN block
perl -p -i -0 -e 's/BEGIN \{\n?\n?\}\n//' **/*.pm

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-24 17:20:41 +00:00
4c0e309677 Bug 16011: $VERSION - Remove use vars qw();
perl -p -i -e 's/use vars qw\(\s*\);\n//' **/*.pm

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-24 17:20:39 +00:00
798d38e4c7 Bug 16011: $VERSION - Remove comments
perl -p -i -e 's/^.*set the version for version checking.*\n//' **/*.pm

+ manual adjustements

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-24 17:20:29 +00:00
017699c345 Bug 16011: $VERSION - Remove the $VERSION init
Mainly a
  perl -p -i -e 's/^.*3.07.00.049.*\n//' **/*.pm
Then some adjustements

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-24 17:20:28 +00:00
3830d78d46 Bug 16011: $VERSION - remove use vars $VERSION
perl -p -i -e 's/^(use vars .*)\$VERSION\s?(.*)/$1$2/' **/*.pm

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-24 17:20:26 +00:00
Jonathan Druart
a6c9bd0eb5 Bug 9978: Replace license header with the correct license (GPLv3+)
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>

http://bugs.koha-community.org/show_bug.cgi?id=9987

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-04-20 09:59:38 -03:00
Galen Charlton
036f2a50e1 Bug 10500: (follow-up) disable AggressiveMatchOnISBN if UseQueryParser is on
The original patch did not correctly construction ISBN phrase
searches when QP is on.  Unfortunately, when attempting to fix that,
I discovered that there's a deep bug in QP that makes it generate
incorrect search queries when combining more than two atoms
in with the || operator.  Consequently, until that can be fixed,
this patch ensures that if UseQueryParser is on, AggressiveMatchOnISBN
has no effect.

To state it anther way, AggressiveMatchOnISBN works only when QP
is not in use.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-05-05 19:31:00 +00:00
Jonathan Druart
c1b782b634 Bug 10500: (follow-up) adjust for name of UNIMARC ISBN indexes
In UNIMARC, the isbn index is ISBN.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-05-05 18:05:42 +00:00
ed4b60fa1d Bug 10500: Improve ISBN matching when importing records
Test Plan:
1) Catalog a record with the ISBN "0394502884 (Random House)"
2) Export the record, edit it so the ISBN is now
   "0394502884 (UnRandomHouse)"
3) Using the record import tool, import this record with matching
   on ISBN.
4) You should not find a match
5) Apply this patch
6) Run updatedatabase.pl
7) Enable the new system preference AggressiveMatchOnISBN
8) Repeat step 3
9) The tool should now find a match

Signed-off-by: Tom McMurdo <thomas.mcmurdo@state.vp.us>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-05-05 18:04:55 +00:00
daf2ebc4f5 Bug 11096: support the retrieval of large MARCXML records
This patch makes Koha <-> Zebra use MARCXML for the serialization when
using DOM, and USMARC for GRS-1.

* The following functions are modified to set the Zebra record syntax
according to the current sysprefs and configuration:

- C4::Context->Zconn
- C4::Context-_new_Zconn

* A new function 'new_record_from_zebra' is introduced, which checks the
context we are in, and creates the MARC::Record object using the right
constructor.

The following packages get touched to make use of the new function:
- C4::Search
- C4::AuthoritiesMarc

and the same happens to the UI scripts that make use of them (both in
the OPAC and STAFF interfaces).

* Calls to the unsafe ZOOM::Record->render()[1] method are removed.

Due to this last change the code for building facets was rewritten. And
for performance on the facets creation I pushed higher version
dependencies for MARC::File::XML and MARC::Record (we rely on
MARC::Field->as_string).

* Calls to MARC::Record->new_from_xml and MARC::Record->new_from_usmarc
are wrapped with eval for catching problems [2].

* As of bug 3087, UNIMARC uses the 'unimarc' record syntax. this case is
  correctly handled.
* As of bug 7818 misc/migration_tools/rebuild_zebra.pl behaves like:

- bib_index_mode (defaults to 'grs1' if not specified)
- auth_index_mode (defaults to 'dom')

here we do exactly the same.

To test:
 - prove t/db_dependent/Search.t should pass.
 - Searching should remain functional.
 - Indexing and searching for a big record should work (that's what the
   unit tests do).
 - Test an index scan search (on the staff interface):
    Search > More options > Check "Scan indexes".
 - Enable 'itemBarcodeFallbackSearch' and try to circulate any word, it
   shouldn't break.
 - Searching for a biblio in a new subscription shouldn't break.
 - Running bulkmarcimport.pl shouldn't break.
 - And so on... for the rest of the .pl files.

[1] http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#render()
[2] a record that cannot be parsed by MARC::Record is simply skipped (bug 10684)

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-02-28 19:50:09 +00:00
Jared Camins-Esakov
2708db5277 Bug 10230: no need to use SimpleSearch for matching auths
When introducing QueryParser, I introduced a check for QueryParser at
too high a level, causing authority matching to try and use SimpleSearch
for authorities prematurely, when SearchAuthorities should be handling
it. This patch corrects the level of the check. This patch only moves
three lines, but thanks to the change in if level, it adjusts the
indentation quite a bit.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Comments on third patch of this series.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
2013-05-19 08:40:13 -04:00
Jared Camins-Esakov
144c7f4e4e Bug 9239: Allow the use of QueryParser for all queries
With the inclusion of this patch, all searches will (try) to use
QueryParser for handling queries for both the bibliographic and authority
databases if UseQueryParser is enabled. If QueryParser is unavailable,
UseQueryParser is disabled, or the search uses CCL indexes, the old
search code will be used.

To test:
1) Apply patch.
2) Run the unit test with `prove t/QueryParser.t`
3) Enable the UseQueryParser syspref.
4) Try searches that should return results in the following places:
   * OPAC (simple search)
   * OPAC (advanced search)
   * OPAC (authorities)
   * Staff client (header search)
   * Staff client (advanced search)
   * Staff client (cataloging search)
   * Staff client (authorities)
   * Staff client (importing a batch using a match point)
   * Staff client (searching for an item for adding to a label)
   * Staff client (acquisitions)
   * Staff client (searching for a record to create a serial)
   * ANYWHERE ELSE I HAVE FORGOTTEN
5) Disable the UseQueryParser syspref. Repeat at least some of the
   searches you did above.
6) If all searches worked, sign off.

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Elliott Davis <elliott@bywatersolions.com>
Searching still works as expected for variuos places.
QueryParser syspref seemed to be enabled by default

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
2013-03-16 21:32:32 -04:00
Jared Camins-Esakov
6e71b80ca3 Bug 7475: Teach matching rules to handle authorities
* Add the code necessary to handle authorities with matching rules and
  import batches.
* Update all the scripts that use the matcher and import batch code
  to use the new API.
* Add authority records to the matching rules interface in the staff
  client.

http://bugs.koha-community.org/show_bug.cgi?id=2060
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
2012-09-19 17:15:56 +02:00
Chris Cormack
509d673f10 Bug 7941 : Fix version numbers in modules
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
2012-06-11 17:29:38 +02:00
Srdjan
12ff7355bb bug_7613: OCLC Connexion gateway
svc/import_bib:
* takes POST request with parameters in url and MARC XML as DATA
* pushes MARC XML to an impoort bach queue of type 'webservice'
* returns status and imported record XML
* is a drop-in replacement for svc/new_bib

misc/cronjobs/import_webservice_batch.pl:
* a cron job for processing impoort bach queues of type 'webservice'
* batches can also be processed through the UI

misc/bin/connexion_import_daemon.pl:
* a daemon that listens for OCLC Connexion requests and is compliant
  with OCLC Gateway spec
* takes request with MARC XML
* takes import batch params from a config file and forwards the lot to
  svc/import_bib
* returns status

ImportBatches:
* Added new import batch type of 'webservice'
* Changed interface to AddImportBatch() - now it takes a hashref
* Replaced batch_type = 'batch' with
  batch_type IN ( 'batch', 'webservice' ) in some SELECTs

Signed-off-by: MJ Ray <mjr@phonecoop.coop>
2012-04-06 17:26:20 +02:00
Colin Campbell
d8b362e0f9 Bug 5415 Let calls of SimpleSearch utilize considtent interface
Remove some unnecessary checks when check of error is
sufficient. Make the order in some cases more logical
Should remove some possibilities of runtime warning noise.
Although some calls belong to the 'Nothing could
ever go wrong' school have added some warnings

Signed-off-by: Christophe Croullebois <christophe.croullebois@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
2011-04-08 13:52:57 +12:00
7c0e441d50 replace references to defunct info email address
Now links to Koha project website.

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-06-25 05:18:44 -04:00
Andrew Elwell
4370a19d75 Final batch of POD cleanups.
Should fix any remaining warnings with 'podchecker'

Signed-off-by: Andrew Elwell <Andrew.Elwell@gmail.com>
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-06-09 08:39:01 -04:00
Lars Wirzenius
7279f55b60 Fix FSF address in directory C4/
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-03-16 20:17:56 -04:00
Henri-Damien LAURENT
bf91bce7d7 Code Improvement : C4/Matcher.pm
substr dans act when passed only subfield offset
2009-10-05 07:48:52 +02:00
Galen Charlton
ed014edad7 bug 2505: enable warnings in C4::ImportBatch and C4::Matcher
Also fix issues with normalizing ISBNs and the default
normalizer in C4::Matcher.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-07 20:09:19 -05:00
Ryan Higgins
86ab0666a0 But 3117 : fix typo in C4::Matcher causing compilation failure.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-04-15 12:49:33 -05:00
J. David Bavousett
13cce1b3c5 Modifications to C4/Matcher for colon and ending punctuation
Recommended by Michele Maenpaa, this adds handling colon and end-punctuation stripping
in subroutine _normalize, and fixes length testing in subroutine _get_match_keys

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-04-03 19:10:49 -05:00
Andrew Moore
66c420c3d0 bug 1980: updateing calls to SimpleSearch to limit number of things returned
C4::Search::SimpleSearch was alredy patched to let you pass in the number of results you want back.
These instances were not using the new API. This patch makes all calls to SimpleSearch specify a limit.

I improved the documentation of SimpleSearch a bit to include the third returned value.

I believe there's a bug in C4::Output::pagination_bar, in that it doesn't deal well with URLs
with only one pair of parameter=value passed to it. I'm getting around this by passing in a second
pair that does nothing.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-07-10 09:11:24 -05:00
Chris Nighswonger
9b47b82a77 Porting SimpleSearch return changes to all code calling it.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-17 05:52:45 -05:00