Commit graph

248 commits

Author SHA1 Message Date
Ere Maijala
016bb29b60 Bug 21958: Fix bibliographic record field comparison with authority
This makes the comparison between bibliographic field and authority field more robust and per subfield. This makes the comparison not consider the same e.g. the following fields:

$a Test User
$a Test $b User

The actual issue cannot be as easily reproduced with the patches for bug 21826 applied, but here's a test plan anyway:

1. Make sure tests pass (especially t/db_dependent/AuthoritiesMarc*)
2. Make sure authority linking still works properly
3. Make sure authority and biblio frameworks allow subfield i
4. Make sure that even if you add subfield i to 700 in biblio, authority link is kept the same
5. Make sure that even if you add subfield i to the authority record, the authority link is kept the same

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-04-01 17:13:56 +02:00
ebe11f70e2 Bug 8976: Sort subfields for authority records
Same patch for the authority frameworks/records

Sponsored-by: Orex Digital

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-03-01 15:14:22 +01:00
517dcabdf7 Bug 25313: Add skip_merge to ModAuthority
Sometimes you know that the merge is useless, since the reporting
tag did not change.
Might be handy in batch processing of authority records.

Test plan:
Pick an authority with a few linked biblio recs (so that it will
immediately merge; lower than MergeLimit).
Open authority record and save it.
Verify that a linked biblio record was merged. Check time in 005.
You proved that this patch does not break the regular process.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-11-18 12:03:03 +01:00
Julian Maurice
96cc447045 Bug 25898: Prohibit indirect object notation
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-10-15 12:56:30 +02:00
88cb7f223d Bug 25265: Prevent double reindex of the same item in batchmod
When batch editing, 2 reindex calls are sent to ES/Zebra.
We can easily avoid that reusing the skip_modzebra_update (renamed skip_record_index)

Additionally we should only send one request for biblio, and we should
only do it if we succeed

As the whole batch mod is in a transaction it is possible to fail in which case
Zebra queue is reset, but ES indexes have already been set

In addition to the skip param this patchset moves Zebra and Elasticsearch calls to
Indexer modules and introduces a generic Koha::SearchEngine::Indexer so that we don't
need to check the engine when calling for index

The new index_records routine takes an array so that we can reduce the calls to
the ES server.

The index_records routine for Zebra loops over ModZebra to avoid affecting current behaviour

Test plan:

General tests, under both search engines:
  1 - Add a biblio and confirm it is searchable
  2 - Edit the biblio and confirm changes are searchable
  3 - Add an item, confirm it is searchable
  4 - Delete an item, confirm it is not searchable
  5 - Delete a biblio, confirm it is not searchable
  6 - Add an authority and confirm it is searchable
  7 - Delete an authority and confirm it is not searchable

Batch mod tests, under both search engines
  1 - Have a bib with several items, none marked 'not for loan'
  2 - Do a staff search that returns this biblio
  3 - Items show as available
  4 - Click on title to go to details page
  5 - Edit->Item in a batch
  6 - Set the not for loan status for all items
  7 - Repeat your search
  8 - Items show as not for loan
  9 - Test batch deleting items
    a - Test with a list of items, not deleting bibs
    b - Test with a list of items, deleting bibs if no items remain where all items are only item on a biblio:
     SELECT MAX(barcode) FROM items GROUP BY biblionumber HAVING COUNT(barcode) IN (1)
    c - Test with a list of items, deleting bibs if no items remain where some items are the only item on a biblio:
     SELECT MAX(barcode) FROM items GROUP BY biblionumber HAVING COUNT(barcode) IN (1,2)
 10 - Confirm records are update/deleted as appropriate

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-09-28 10:10:08 +02:00
Julian Maurice
b168f4a2e9 Bug 21395: Make perlcritic happy
This patch adds a .perlcriticrc (copied from qa-test-tools) and fixes
almost all perlcrictic violations according to this .perlcriticrc
The remaining violations are silenced out by appending a '## no critic'
to the offending lines. They can still be seen by using the --force
option of perlcritic
This patch also modify t/00-testcritic.t to check all Perl files using
the new .perlcriticrc.
I'm not sure if this test script is still useful as it is now equivalent
to `perlcritic --quiet .` and it looks like it is much slower
(approximatively 5 times slower on my machine)

Test plan:
1. Run `perlcritic --quiet .` from the root directory. It should output
   nothing
2. Run `perlcritic --quiet --force .`. It should output 7 errors (6
   StringyEval, 1 BarewordFileHandles)
3. Run `TEST_QA=1 prove t/00-testcritic.t`
4. Read the patch. Check that all changes make sense and do not
   introduce undesired behaviour

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2020-06-29 12:37:02 +02:00
278df89bb5
Bug 22437: (follow-up) Move deletion of merge requests to DelAuthority
Lines can be moved. Deletion can be done too if skip_merge is not set.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-05-12 11:02:05 +01:00
52e95fff34
Bug 24735: Remove QueryParser-related code
At the last development meeting we have voted to remove the
QueryParser-related code
https://wiki.koha-community.org/wiki/Development_IRC_meeting_19_February_2020

Hea tells us that it has not been adopted, and the code/bug tracker that
it is not really usable as it. As nobody is willing to work on it, we
decided to remove it instead.

Test plan:
 % prove t/db_dependent/Search.t
must return green

See commits from bug 9239 and confirm that the code is removed in this
patch.

Also play with the search on the UI and confirm that you do not see
obvious regressions

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-03-02 11:13:03 +00:00
78190371d0
Bug 24286: Escape forward slash in GENRE/FORM (and any authtype)
Note that bug 24267 will fix the issue during Z39 searching for new auth, but the duplicate check at saving will still fail

To test:
 0 - Set SearchEngine to Elasticsearch,  QueryRegexEscapeOptions  to 'Don't Escape'
 1 - Browse to Authorities
 2 - Select New->New from Z3950
 3 - Search for subject 'Short stories'
 4 - Auth search explodes
 5 - Apply patch
 6 - Repeat and confirm the search results return
 7 - Import and save the GENRE/FORM record for Short stories
 8 - Repeat the search and import the same record
 9 - Confirm that upon save you are notified of duplicate, don't save
10 - set  QueryRegexEscapeOptions to 'Escape' and confirm Z39 searching works and importing/save notifies of duplicate
11 - set SearchEngine to Zebra and confirm Z39 searching works and importing/save notifies of duplicate

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-01-23 09:05:08 +00:00
0eb9edbf22
Bug 23933: (bug 10132 follow-up) Fix commit_file.pl
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-11-08 12:55:03 +00:00
88b37330ab Bug 21036: Remove odd number of elements warnings from AuthoritiesMarc.pm
Like this one (16.11 line number):
Odd number of elements in anonymous hash at C4/AuthoritiesMarc.pm line 1070.

We need to add a scalar to some MARC::Field::subfield calls. In list context
an empty list returned affects the hash built around it.

Test plan:
Could reproduce this warning easily from OPAC authority search.
opac-authorities-home.pl calling BuildSummary.

Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-05-07 11:39:24 +00:00
3adf7708d5 Bug 21962: Fix authority search using 'all'
To test:
1 - Set search engine to Zebra
2 - Have some authorities and indexes up to date
3 - Search authorities with option 'search entire record'
4 - No results
5 - Apply patch
6 - Run unit tests, they pass!
7 - Repeat search
8 - Results!

Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-12-11 19:15:39 +00:00
Ere Maijala
a4c02b53cd Bug 19365: Fix search for duplicate authorities with Elasticsearch
Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:02 +00:00
541a90765c Bug 21581: (QA follow-up) Update comment according to code in SearchAuthorities
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-10-31 13:17:48 +00:00
6093e08443 Bug 21581: Pass through authorities indexes
To test:
 1 - Define a rule:
  Matching rule code: LCAUTH
  Match threshold: 100
  Record type: Authority record
  MATCH POINTS
  Search index: LC-card-number
  Score: 100
  Tag: 010
  Subfield: a
 2 - Have a record with an LC number in 010
 3 - Export it
 4 - Edit the record in Koha, move the value from 010 to another field
 5 - Import the original record
 6 - Use the matching rule above, you expect it won't match (nothing in 010 anymore)
 7 - It matches?!?!?!
 8 - Enable zebra logging in koha-conf.xml
  <zebra_loglevels>none,fatal,warn,request,info</zebra_loglevels>
 9 - tail -f /var/log/koha/kohadev/zebra-output.log
10 - On manage marc apply matchign with no rule, then reapply with rule
11 - View the output in the log - the index searched is 'Any'
12 - Apply patch
13 - Reapply matching
14 - Note we search the expected index and no match is found

NOTE: By applying bug 21579, I was able to view the difference, and
      see that it didn't match identically, because I had moved the
      value from 010 to 155. No log checking needed. clean/delete batch.
      Apply patch, no match. clean/delete batch.
      Edit authority back, matches.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-10-31 13:17:47 +00:00
49933cd449 Bug 14769: (Follow-up) Reduce number of preference parses
In sub merge the Koha::Authority object was already created, so it was kind
of silly to recreate it each time with new.
In Koha::Authority, the ControlledIndicators object can be saved in the
object too for reuse during its life time. Similarly, we prevent repeated
lookups for the reporting tag (via authority type).

Test plan:
[1] Run t/db_dependent/Authority/Merge.t
[2] Run t/db_dependent/Koha/Authorities.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Limited benchmarking shows me a reduction of 11% on Merge.t and a
reduction of 4% on Authorities.t when comparing with and without this
last patch. The larger impact on Merge.t could be expected.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-04-12 10:50:35 -03:00
420f8b01d4 Bug 14769: Put ControlledIndicators to work
This patch does:
[1] Adds Koha::Authority->controlled_indicators (with a test).
[2] Adds a call to controlled_indicators in AuthoritiesMarc::merge.
    Unit test Merge.t is extended too.
[3] Simplifies the code in authorities/blinddetail-biblio-search.pl by
    calling controlled_indicators.

Test plan:
[1] Run t/db_dependent/Koha/Authorities.t
[2] Run t/db_dependent/Authority/Merge.t
[3] Steps 3 to 7 for MARC21:
    Create a PERSO_NAME authority with 008/11=r and ind1=3
[4] Edit a biblio and add a 600 linked to the new authority.
[5] Verify that the biblio has ind1==3 and ind2==7 and $2==aat.
    (If $2 is not visible, check the metadata in biblio_metadata.)
[6] Edit the PERSO_NAME authority and change 008/11 to '|' (bar).
[7] Verify that merge updated your biblio record: $ind2==4 and $2 gone.
[8] UNIMARC: Follow the pattern from steps 3 to 7.
    Create authority, link it in a biblio, check indicators (they should
    be copied both). Edit authority, change indicators and verify the
    merge results in the biblio record.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tested PERSO_NAME and UNIF_TITLE.
For UNIF_TITLE the second authority indicator is copied to ind1 or ind2,
depending on the biblio tag involved.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-04-12 10:50:35 -03:00
Janusz Kaczmarek
70bb6fe0ae Bug 18458: Fix subfields ordering when Merging authority records
While doing a merge, if a subfield(s) precedes the controlled subfields
(like $i before $a in 7XX, which comes before $a -- rare but will
become more and more usual) after merging will be moved to the end.
This is not right.

The patch (with AuthorityMergeMode == loose) make merge consider the
subfields order: all subfields which originally were found before
the first controlled subfield (e.g. $i before $a in 7XX / MARC 21)
will remain in the front, the rest of not controlled subfields that
should remain in the field will come after the subfields copied
from authority rec.

As a bonus, $9 will be placed at the end.

To test:
0) Have AuthorityMergeMode == loose;
1) Have some field in bibliorecord, controlled by an authority, with
extra subfield(s) (i.e. not present in authority rec.) placed at the
beginning of the field;
2) Open (not necessarily edit) and save the connected authority;
3) See that the extra subfieds were moved to the end of the field
   (and $9 is in the front);
4) Apply the patch;
5) Reorder subfields in biblio field;
6) Open (not necessarily edit) and save the connected authority;
7) See that the order has been conserved, additionally $9 the last
   subfield in the field.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended:
Moved field creation to its original place. Changed $9 handling. Simplified the following add_subfields for loop. Edited comments.
Restored the append_fields_ordered call (see comment6).

With this patch, the Merge.t test now passes.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-12-22 13:15:36 -03:00
56500e8e67 Bug 19415: FindDuplicateAuthority is searching on biblioserver since 16.05
Commit b4392018bc [Bug 12478: make things using SimpleSearch use the new version] changed sub FindDuplicateAuthority and replaced this call (from 3.22.x) in 16.05 (pushed April 2016):
    my ($error, $results, $total_hits) = C4::Search::SimpleSearch( $query, 0, 1, [ "authorityserver" ] );

The new call does unfortunately not include the authorityserver:
    my ($error, $results, $total_hits) = $searcher->simple_search_compat( $query, 0, 1 );
Simple_search_compat redirects to C4/Search/SimpleSearch and SimpleSearch assumes a biblioserver if no server is passed.

This effectively makes FindDuplicateAuthority useless since we will no longer find duplicates and we could see an error like this in the log:
    16:51:42-04/10 zebrasrv(51) [request] Search biblios ERROR 114 1 1+0 RPN @attrset Bib-1 @and @attr 1=authtype PERSO_NAME @attr 1=Heading Moerenhout

The fix is obviously trivial.

Test plan:
[1] Go to Authorities.
[2] Add a new authority PERSO_NAME with 100a Moerenhout.
[3] Repeat step 2. Verify that you get the duplicate authority warning.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-09 13:46:41 -03:00
e722c0a29f Bug 18149: Move CountUsage calls to Koha namespace
After the introduction of Koha::Authorities->get_usage_count with bug
9988, we can now replace the remaining occurrences of CountUsage.

At the same time we remove CountUsageChildren. This was an empty sub.
The typo get_count_usage in a subtest title is adjusted.

Test plan:
[1] Run t/db_dependent/Koha/Authorities.t
[2] Perform a search on authorities-home.pl and verify that you see
    plausible numbers for 'used in xx records'.
[3] Click on Details for one authority. See the same number?
[4] Do the same as in 2/3 for Authority search on OPAC.
[5] Remember the authid and enter this in the record numbers box on
    tools/batch_delete_records.pl. Select Authorities and click
    Continue. The next form shows a column "Used in". Do you see
    the same count again?
[6] Git grep CountUsage.
    You should see just one hit in a comment that can be kept in
    Koha/Authorities.pm.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-09-19 11:47:32 -03:00
80c749df80 Bug 10132: Simplify code, call the method only once
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-09-19 10:39:41 -03:00
b36a0e1a59 Bug 10132: Add ability to set MARC Organization Code at library level
Test plan:
0. Apply patches
1. Update database
2. Go to administration -> libraries, try to update some library and
fill in some value into Marc Organization code field
3. Save this library and edit again - the code should be stored
correctly
4. Go to system preferences and fill in some value into MARCOrgCode
preference, note there is enhanced description mentioning the ability to
set organization code on library level
5. Set active library to the one with own org code stored
6. Go to cataloguing, create new empty record and click into field 003 -
there should be the code you filled for that library
7. Set active library to one withou marc org code
8. Go to cataloguing, create new empty record and click into field 003 -
there should be the code from system preference
9. Go to system preferences again and set AutoCreateAuthorities to
'generate' and BiblioAddsAuthorities to 'allow'
10. Go to cataloguing and create some biblio record, fill in any author
in to create its authority record, save the biblio
11. Go to authorities and find this created authority, go to details and
check the fields: 003, 040$a, 040$c, 670$a - there should be used right org code
12. prove t/db_dependent/AuthoritiesMarc.t t/db_dependent/Biblio.t t/db_dependent/Koha/Libraries.t

Signed-off-by: Hugo Agud <hagud@orex.es>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-09-19 09:47:27 -03:00
434162c175 Bug 17380: Graceful resolution of missing reporting tag in merge
Altough this patch deals with a mostly hypothetical case and this report
makes it practically impossible anymore to merge with records in the
Default framework (having no reporting tag), we can make the code of
sub merge still a bit more robust here.

If you would merge biblio records from one authtype to another and the new
framework would not have a reporting tag, before this patch the result would
be data loss. Merge would handle this request as a delete. This patch makes
merge handle it differently: instead of clearing the biblio records, it
keeps $a and $9 in order to make a future corrective merge possible.

Note: The additional condition on line 1468 for $tags_using_authtype
makes sure that we do not select all fields when the authtype should
unexpectedly be empty string (Default). This prevents crashing on
a "Control fields do not have subfields" error.

Test plan:
[1] Run t/db_dependent/Authorities/Merge.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-09-12 12:07:47 -03:00
Mark Tompsett
d5986c9b97 Bug 19040: Refactor GetMarcBiblio parameters
Change parameters to a hashref.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Looks good to me.
Two calls in migration_tools/22_to_30 still in old style.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-08-25 10:23:42 -03:00
9f47bd8825 Bug 18854: Protect few other occurrences of offset
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-07-14 12:02:04 -03:00
d49e8bd5e2 Bug 9988: Few subtle changes for postponed merge
The fails in the previous test showed that we need the first three
changes here. Some final polishing in points 4 to 6.

[1] Sub merge: Refine the condition for initializing $tags_new.
    A postponed 'modify'-merge (A to B) makes that $authtypefrom is not
    defined when running merge later. When crossing the type boundary, we
    need a new field too.
[2] Sub merge: Add condition for an empty @record_to array.
    This indicates also that a field should be removed, since we should
    otherwise only add a $9 subfield.
[3] Sub merge: Adjust initializing @record_from.
    This change is tested by verifying a cleared subfield in the test.
[4] DelAuthority: Adding a skipmerge parameter to allow the call from
    authorities/merge.pl to skip an unneeded merge.
    This also prevents that the 'delete-merge' would precede the
    'modify-merge' under a hypothetical race condition.
[5] DelAuthority: There is actually no need to call GetAuthority.
    The subfields of the old record are not relevant in this case.
[6] Added a few POD lines to merge.
[7] Removed a trailing space in a comment line in merge.

Test plan:
[1] Run t/db_dependent/Authorities/Merge.t.
    The last subtest should no longer fail now.
[2] See test plan of next patch.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-04-13 08:53:47 -04:00
04263912c7 Bug 9988: Check the merge limit in sub merge
At this point, we are replacing dontmerge functionality by the new
AuthorityMergeLimit logic. Instead of doing this check before calling
merge, we just call merge and check it there. In order to let the cron
job do the larger (postponed) merges, we add a parameter override_limit.

A subtest is added in Merge.t to test the 'postponed merge' feature. Since
merge now also calls get_usage_count, an additional mock is added. All
references to dontmerge are removed.

In merge two lines, initializing $dbh and $counteditbiblios, are moved.
The dontmerge test in DelAuthority and ModAuthority is removed. Since this
did not leave much in ModAuthority, I fixed the whitespace on the remaining
lines rightaway (yes, I know).
A minimal set of changes is applied to the cron script; it will get further
attention on a next patch.

Test plan:
[1] Run t/db_dependent/Authorities/Merge.t
[2] Set AuthorityMergeLimit to 2. Modify an authority with two linked
    biblios. Check that the merge was done immediately.
[3] Now modify an authority with more than 2 linked records.
    Verify that the merge was postponed; a record must be inserted in
    the need_merge_authorities table.
[4] Testing of the merge cron job is *postponed* to a next patch.
    Note: I tested a modification, but the script just needs more
    attention.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-04-13 08:53:46 -04:00
06647d29a7 Bug 9988: Remove the Zebra code from sub merge
Since we can now call linked_biblionumbers, we can now remove all Zebra
related code from merge. We also add a parameter biblionumbers; we use it
in the test now, but it may be handy too later in the maintenance script
when we want to trigger a merge for specific biblionumber(s). See bug
report 18071.

All mocks for ZOOM, Context::Zconn, Search::new_record_for_zebra in the
merge test can now be replaced by one mock for linked_biblionumbers. Note
that we test the biblionumbers parameter in the last subtest without that
mock.

Remove unused vars $countunmodifiedbiblio, $counterrors from merge.
Renamed zebrarecords to linkedrecords in the Merge test.

Test plan:
[1] Run t/db_dependent/Authorities/Merge.t
[2] Modify an authority record. Check the linked biblio records.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-04-13 08:53:46 -04:00
4c762ba69c Bug 9988: Merge should have a parameter hash
We will need a few additional parameters for merge later on. This patch
puts the original parameters in a parameter hash.
For the same reason DelAuthority gets a parameter hash here.

Note: We remove the second parameter from the DelAuthority call in
authorities/authorities-home.pl here. It was not used and could have
presented problems in the future.

Test plan:
[1] Run t/db_dependent/AuthoritiesMarc.t.
[2] Run t/db_dependent/Authorities/Merge.t.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-04-13 08:53:46 -04:00
fd8c2d4e35 Bug 18070: Correct error from podchecker on line 1376
Resolves:
 *** ERROR: Spurious =cut command at line 1376 in file C4/AuthoritiesMarc.pm

Test plan:
[1] Run podchecker on C4/AuthoritiesMarc.pm

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-03-03 18:12:10 +00:00
17e84750d9 Bug 18070: [Follow-up] What should DelAuthority return?
Script tools/batch_delete_records.pl only checks the return value of
DelAuthority. The return value depends on DBI rows, which is not always
reliable. It may return -1 when it 'does not know'.

Testing $@ in tools/batch_delete_records.pl should actually be enough.
The return value was discarded in C4/ImportBatch.pm. Removing an unused
variable.

Test plan:
[1] Pick an authority record with a few linked biblios.
    Delete this authority record via tools/batch_delete_records.pl.
    Check if the linked biblio records are cleaned up.
[2] Bonus: Make a typo in the SQL statement of DelAuthority. Check
    if batch_delete_records shows you the error message.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-03-03 18:12:09 +00:00
cb58b11f67 Bug 18070: Extend sub merge to remove fields for deleted authorities
In order to accomplish this, we need to add some additional checks in
the merge routine. The actual change to remove the field, is quite
small.

Furthermore, we need to add a merge call in DelAuthority and adjust
the merge cron job accordingly.

The change is well supported by additional tests, including a simulation
of postponed removal via cron, if dontmerge is enabled.

Note: Deleting an authority with linked biblios is tested on the next
patch.

Test plan:
[1] Run t/db_dependent/Authorities/Merge.t
[2] Delete an authority without linked biblios from the Authorities
    module. If the indexer is not fast enough, wait a bit and refresh to
    verify that the authority is gone on authorities-home.pl.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-03-03 18:12:09 +00:00
ad38b24308 Bug 18014: AddAuthority should respect AUTO_INCREMENT
Instead of using the MAX(authid)+1 logic, AddAuthority should just save
the record and get the new id. The authid column is an autoincrement.

This eliminates problems where a newly assigned authid also refers to a
previously deleted record. (And it will not cause problems when refining
the dontmerge functionality on report 9988.)

Note: ModAuthority also calls AddAuthority to update an existing record; in
that case we should not create a new record even if the record should not
be found any more (which should be exceptional).

This patch also simplifies handling of 001 in the authority record: in all
cases this field is updated now; no need to check its contents.

Test plan:
[1] Run t/db_dependent/AuthoritiesMarc.t
[2] Add a new authority record via the interface

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-14 14:01:11 +00:00
6ddd4b2f5f Bug 17913: [Follow-up] Fix duplicate $9s after merging in loose mode
We need to add $9 to the skip_subfields hash too. Formerly, it was
added to $exclude as well.
Thanks, Julian, for catching this one.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 13:55:13 +00:00
dd1c63230f Bug 17913: Run perltidy on the inner foreach loop
Kept the same number of lines.
You could verify with diff -w.

Test plan:
[1] Run t/db_dependent/Authorities/Merge.t
[2] As the last patch in this series, also test the interface:
    Set AuthorityMergeMode to loose. Set dontmerge to Do.
    Modify an authority record attached to multiple biblios.
    Edit a subfield, clear a subfield and add a subfield.
    Save. Wait a bit for the merge and Zebra update.
    Verify that the changes are merged properly into biblio records.
[3] Repeat step 2 with AuthorityMergeMode to strict.
    Remember that this affects the extra subfields in biblio records.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 13:55:12 +00:00
28b74224d3 Bug 17913: We always need some housekeeping
Remove some commented warnings
Remove the commented old code at the end of sub merge
Explicitly set merge mode in the first subtest
Move the return to loose mode from the second subtest to the third

Test plan:
Run t/db_dependent/Authorities/Merge.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 13:55:12 +00:00
ede08d725e Bug 17913: Remove possible duplicates in strict merge mode
Since strict mode does not allow additional subfields that would make
identical fields linked to the same authority different, there is no
need to keep them while merging.

We achieve this goal by simply:
[1] Count the number of same fields linked to mergefrom in strict mode to
    eliminate duplicates.
[2] Replaces the if-statement on auth_number by a next. (Tidy follows.)

Test plan:
Run t/db_dependent/Authorities/Merge.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 13:55:12 +00:00
681ae8430e Bug 17913: Do not keep a cleared subfield in loose merge mode
If you modify an authority and clear a specific subfield, you expect that
merge respects your edit and clears this subfield too in the biblio
records. It does in the new strict mode, but it does not yet in the
default loose mode.

This patch fixes that by adjusting the code around $exclude so that it
uses a new hash skip_subfields, built from the reporting tags from the old
and the new authority record.

This is supported again by some changes in the unit test.

Test plan:
Run t/db_dependent/Authorities/Merge.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 13:55:11 +00:00
8cde85451e Bug 17913: Fix the new field tag in merge when changing type
Originally aimed for 9988, adjusted for this report.

Old behavior was: pick the first tag. This is definitely wrong.
If you (would) merge 610 to 611, you don't want to get a 111.

This patch resolves the problem by determining the new tag in a small
helper routine _merge_newtag, and corrects the position of the new field
in the MARC record with append_fields_ordered. Too bad that MARC::Record
does not have such a function; it looks like insert_fields_ordered, but
it is different in case of multiple fields with the same tag.

Note: These two small helper functions are not tested separately, since they
should not be called outside of merge. They are implicitly tested by the
adjusted tests in Merge.t.

Note: In adding tests for this fix, I chose to simplify compare_field_count
(no need for the pass parameter), and replace the pass parameter of sub
compare_field_order by an exclude parameter, a hash of fields to exclude in
counting fields.

Test plan:
Run t/db_dependent/Authorities/Merge.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 13:55:11 +00:00
Frédérick
79c81920ab Bug 17913: Use AuthorityMergeMode pref in sub merge
Original fix from a patch on bug 11315.
Amended by Marcel de Rooy January 2017.

Test plan:
If you set mode to loose, the test will still pass.
If you set mode to strict, one test will fail. (Fixed later.)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 13:55:10 +00:00
ffe1f6555c Bug 17913: Use replace_with instead of insert_grouped_field
Original fix from a patch on bug 5572.
Amended by Marcel de Rooy January 2017.

Note: This does not yet resolve the field order when merging to another
auth type, but is a good start.

Test plan:
Run t/db_dependent/Authorities/Merge.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-20 13:55:10 +00:00
172ebfe520 Bug 16431: Use Koha::Cache to cache marc subfield structure
The marc subfield structure is currently cached using a global variable
of C4::Context. The infos are retrieved every time a new context is
created.
This patch suggests to use Koha::Cache instead.

To achieve this goal, a new subroutine is created
C4::Biblio::GetMarcSubfieldStructure, it will be called from code which
needs to get the marc subfield structure. GetMarcFromKohaField,
GetMarcSubfieldStructureFromKohaField, TransformKohaToMarc and
_get_inverted_marc_field_map are modified accordingly and the cache is cleared
when the table is updated (from the 3 pl scripts modified by this patch).

The caching done in C4::Context (marcfromkohafield) is removed.

Test plan:
Play with the marc subfield structure (in the administration module),
then add and edit records and make sure everything went fine.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Everything works as expected on my functional tests. I'm really happy to see the
patch introduces relevant tests for previously untested functions.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-07-08 12:44:58 +00:00
Jesse Weaver
37ea74669c Bug 12748: (QA followup) fix several tiny QA tools warnings
* admin/searchengine/elasticsearch/mappings.yaml:
    * Fix indentation so QA tools believe it's valid YAML
  * C4::AuthoritiesMarc: Remove tab character
  * C4::Search: Fix POD
  * Koha::BiblioUtils: remove "return undef"
  * Koha::Schema::Result::Search*: rename type to _type
  * Koha::SE::ES::Search:
    * Fix spelling/@_[0]
    * Add my to loop variable
    * Remove unused var $error
    * Add use Modern::Perl
    * Fix undefined variables
  * Koha::ES::Indexer: fix POD
  * Koha::SE::ES::QueryBuilder:
    * Don't define variables in conditionals
    * Fix POD
  * Koha::SE::Zebra::Search:
    * Fix debug warns/POD/barewords
    * Koha::SE::Zebra::Search: add use Modern::Perl
  * kohastructure.sql: remove old, unused elasticsearch_mappings table,
    add new tables
  * labels/label-item-search.pl: remove C4::Dates
  * opac/elasticsearch.pl: remove old file

Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:13 +00:00
Robin Sheat
b4392018bc Bug 12478: make things using SimpleSearch use the new version
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:07 +00:00
c564f30155 Bug 16169: Change prototype for C4::Biblio::TransformMarcToKoha
subroutines should not take $dbh in parameter.
C4::Biblio::TransformMarcToKoha has it and does not use it.

Test plan:
Look at the patch and confirm that all occurrences of
TransformMarcToKoha have been modified.

Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
2016-04-07 00:04:21 +00:00
2d961b5ad4 Bug 16056: Do not crash when searching for an authority if zebra's index is not up-to-date
This patch fixes a regression introduced by
commit 7e70202d34
  Bug 15381: Remove GetAuthType and GetAuthTypeCode

This first version will reintroduce the same behavior as before bug
15381: the record will be displayed even if it's no in the DB

Test plan:
Search for authorities
delete one
The zebra's index is not updated yet and the results will contain the
record you have deleted.

Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-04-01 19:04:38 +00:00
798d38e4c7 Bug 16011: $VERSION - Remove comments
perl -p -i -e 's/^.*set the version for version checking.*\n//' **/*.pm

+ manual adjustements

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-24 17:20:29 +00:00
017699c345 Bug 16011: $VERSION - Remove the $VERSION init
Mainly a
  perl -p -i -e 's/^.*3.07.00.049.*\n//' **/*.pm
Then some adjustements

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-24 17:20:28 +00:00
3830d78d46 Bug 16011: $VERSION - remove use vars $VERSION
perl -p -i -e 's/^(use vars .*)\$VERSION\s?(.*)/$1$2/' **/*.pm

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-03-24 17:20:26 +00:00
Julian Maurice
5ef5fb5617 Bug 15358: Fix authorities merge
Bug 8064 (Merge several biblio records) change some code used in both
biblios and authorities merge tool without updating the authorities
merge tool.
This patch fixes that.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
2016-02-03 23:03:33 +00:00