Commit graph

289 commits

Author SHA1 Message Date
d2e189ca1c Bug 22600: Set 'commandline' interface appropriately
This patch change Koha::Cron to be a more generic Koha::Script class and
update all commanline driven scripts to use it.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-04-10 19:43:11 +00:00
Magnus Enger
b7b4694666 Bug 12488: Make bulkmarcimport.pl -d use DELETE instead of TRUNCATE
On MySQL >= 5.5 bulkmarcimport.pl with the -d (delete) switch gives
an error like "Cannot truncate a table referenced in a foreign
key constraint". This patch proposes to replace the offending
TRUNCATE with DELETE. Auto incerement counters are reset to 1 to
preserve the functionality from TRUNCATE.

To test:
- Make sure you havae a test database with some records and items
- Run bulkmarcimport.pl with the -d switch
- Observe the error described above
- Apply this patch
- Run bulkmarcimport.pl with the -d switch again
- Observe the lack of an error
- Verify that the newly imported records and items have biblionumber
  and itemnumbers starting with 1

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-02-15 18:42:46 +00:00
fd30cac85a Bug 22155: Adapt uses of biblio_metadata.marcflavour to schema
This patch makes the code using Koha::Biblio::Metadata->marcflavour use
->schema instead for all interactions.

To test:
- Update the DB structure:
  $ updatedatabase
- Update the schema files:
  $ dbic
- Notice all the places in which biblio_metadata is used
  $ cd kohaclone
  $ git grep biblio_metadata
=> SUCCESS: They all use `schema` instead of marcflavour
- Notice all the places that use Koha::Biblio::Metadata:
  $ git grep Koha::Biblio::Metadata
=> SUCCESS: They all use the schema attribute when they used to use
marcflavour
- Run all the modified tests and scripts
=> SUCCESS: We are all good
- Sign off :-D

Note: while this seems like a minor change, the places in which plain
SQL is used really require understanding the queries and how they are
used, because some query results might be passed to some other method
that in turn uses the marcflavour attribute. I of course took that into
account but errare humanum est :-D

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-01-30 11:35:34 +00:00
Julian Maurice
875f3162c8 Bug 21908: (QA follow-up) Remove useless parentheses in query
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-12-11 19:19:47 +00:00
a1b08269a7 Bug 21908: Add DISTINCT biblionumber to rebuild_zebra.pl
Zebra indexing script misc/migration_tools/rebuild_zebra.pl as a table arg to allow filtering.
When using table=items we should use DISTINCT(biblionumber) to avoid indexing several times the same biblio record when it has several items.
This patch adds DISTINCT(biblionumber) in all cases it does not harm if its already unique.

Test plan :
1) Be sur you have a biblio record with biblionumber 1 with 3 items
2) Run misc/migration_tools/rebuild_zebra.pl -v -b --table items --where="biblionumber=1"
3) Without patch you see "Records exported: 3", with patch only one
4) Check indexing works well

Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-12-11 19:19:46 +00:00
c36597bc0c Bug 21908: Add biblio_metadata to rebuild_zebra.pl tables
Zebra indexing script misc/migration_tools/rebuild_zebra.pl as a table arg to allow filtering.
It is missing biblio_metadata to allow filtering on MARCXML with ExtractValue.

Test plan :
1) Be sur you have a biblio record with biblionumber 1
2) Run misc/migration_tools/rebuild_zebra.pl -h
3) You see : --table specify a table (can be items, biblioitems, biblio, biblio_metadata) to retrieve biblionumber to index.
4) Run misc/migration_tools/rebuild_zebra.pl -v -b --table biblio_metadata --where="biblio_metadata.biblionumber=1"
5) Check you dont have SQL errors

Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-12-11 19:19:46 +00:00
Mark Tompsett
e1b5fa657d Bug 14385: Squash of a lot of patches rebased
- Added missing GetHiddenItems parameter change case
     Without this prove t had a failure.

- Always use mocks, not set_preference

- Tweaks so t/db_dependent/00-strict.t passes
     There was a typo botcat vs borcat and borrowernumber was never
     defined. Grabbing from userenv, like other code does.

- Tweak t/db_dependent/Items.t to fully test changes
     This will test all the if structures fully in GetHiddenItemnumbers.
         prove t/db_dependent/Items.t

- Tweak borrower category code
     $borrower->{categorycode} on a Koha::Patron is not the
     same as $borrower->categorycode. Fixed error.

- Search was returning URLS for wrong interface
     There was one search context place wrong. Changed it to $is_opac
     as the logic for setting $is_opac was modified correctly.

- Corrected issues with category code.
     When a user isn't logged in, $borrower is undef and causes error
     when determining category code. Added conditional check.

- Properly trigger all changes in C4/Search.pm

- Fix QA Test tool failures
     C4/Search.pm had some tabs.

- Add some commenting to make sense of logic

- Refactor EmbedItemsInMarcBiblio parameters to hashref

- Trigger GetMarcBiblio's EmbedItemsInMarcBiblio call.
     prove t/db_dependent/Items.t

- Add missing test to trigger Koha/BiblioUtils/Iterator change

- Add borrower category overrides
     These files generally add borcat parameter to GetMarcBiblio.
     Others might include correction of filtering of items
     (opac-basket), or a comment as to why no changes were done
     (opac-search).

     In the case of opac-search, correcting the first FIXME will
     likely correct the OpacHiddenItems issues on tags. As such,
     that is beyond this bugs scope.

     Some code had loop optimizations and fixes made, like a
     'next unless $record' when the biblio shouldn't even be in
     the list.

- Modify opac-ISBDdetail and opac-MARCdetail
     Both files had similar logic. They were rearranged and
     optimized, so that both files would have practically identical
     initial blocks of code.

     Optimizations were possible, because GetMarcBiblio
     returns a filtered record, so that there is no double call
     (once in the opac-### file and once in GetMarcBiblio) to
     GetHiddenItemnumbers.

- Fix hiding in opac-tags
     opac/opac-tags.pl was not properly hiding.

     There is currently one known bug associated with tags left.
     If you have two biblios tagged by different people with the
     same tag, the opac-search will show the one you tagged that
     is supposed to be hidden, because tag searches work differently
     than regular searches. This is beyond the scope of this bug.
     See the FIXME's in opac/opac-search.pl

- Trigger the C4::ILSDI::Services changes
     prove t/db_dependent/ILSDI_Services.t

- Added missing 'my'

- Test C4/Labels/Label.pm changes

- Improve C4::Record::marcrecord2csv test cases

- Corrected opac-details searchResult call

- Fix breaking issues constraint in ITerator test

- Fix ILSDI_Services test when clubs with branch exist

- Rebased again!
- Rebased t/db_dependent/Items.t conflict.

The test plan is in comment #112 last I checked.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-02 10:33:12 +00:00
bee097b39b Bug 14302: Remove GRS1 specific code
Remove:
- BIB_INDEX_MODE and AUTH_INDEX_MODE env var
- bib_index_mode and auth_index_mode options from scripts
- Warnings from about page, just kept one if zebra_bib_index_mode or
zebra_auth_index_mode still exist in config and are set to grs1

Test plan:
- Install Koha from src
- Install Koha from pkg
- Read the code, carefully!

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Rebased

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-08-31 11:24:20 +00:00
12b505f2b7 Bug 20811: (RM follow-up) fix check for matching
Notice should only be triggered if both settings are false

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-23 15:08:56 +00:00
15b865c6ee Bug 20811: (QA follow-up) Prevent calling AddItemBatchFromMarc and ModBiblioMarc without biblionumber
If you use -update but do not find matches (or did not want to match), you
should not call those routines. We should warn and skip this record.

Adding a warn at the start that the choice of options may not be smart.
Note that this needs further attention somewhere else. You could mix
-update with -insert for instance and still see some problems. (May depend
on items with unique barcode etc.)

Test plan:
Run -update without match or isbn.
Or run -update -isbn with a non-matching ISBN.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-23 15:08:56 +00:00
a49a7f08d6 Bug 20811: (QA follow-up) Make the isbn check work
The following code was never reached, since $isbn was not filled.
    if (!$biblionumber && $isbn_check && $isbn) {
        $sth_isbn->execute($isbn);
        ($biblionumber,$biblioitemnumber) = $sth_isbn->fetchrow;
    }
Solution: Fix the code with two $isbn declarations. Move the checkisbn
condition a level deeper.

Test plan:
Run misc/migration_tools/bulkmarcimport.pl -file bib726.utf8 --update -isbn
Since you do not match on biblionumber, the ISBN should match.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-23 15:08:56 +00:00
d15185025a Bug 20811: Fix wrong usage of ModBiblio
Since
  commit cefa7c21e2
  Bug 5635: bulkmarcimport new parameters & features

AddBiblio call has been replaced with ModBiblio, but the return values
are different. We should not replace the value of $biblionumber with
what returns this subroutine.

Test plan:
If you are familiar with bulkmarcimport.pl you should know what to test,
I am not.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-23 15:08:56 +00:00
Julian Maurice
79f5051ed7 Bug 11936: (QA follow-up) Consistent log message for item insert
Fix the same error in another place

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-19 13:55:54 -03:00
Magnus Enger
c2208e9ca7 Bug 11936: Consistent log message for item insert
If you use bulkmarcimport.pl to import records with items it looks
like the successfull insert of the record is reported multiple time,
but the second and subsequent "ok" is really related to importing
the item(s).

This patch changes the log message on successfully inserting an item
to match the log message given when inserting an item fails.

To test, the easy way:
- Look at lines 530 and 536 of bulkmarcimport.pl, and note that the
  "op" in those two lines are different
- Apply the patch
- Look at lines 530 and 536 again, and note that the "op" is now
  identical, and that this makes sense, since they are both related
  to the same operation, specifically inserting an item

To test, the hard way
- Have some records with items
- Import the records with bulkmarcimport.pl, and make sure to specify
  the -l option, to create a log of the actions taken
- Look at the log and verify it looks something like this:
  id;operation;status
  1;insert;ok
  1;insert;ok
  2;insert;ok
  2;insert;ok
- Apply this patch and import some more records with items. The log
  should now be similar to this:
  id;operation;status
  1;insert;ok
  1;insertitem;ok
  2;insert;ok
  2;insertitem;ok

Signed-off-by: Maksim Sen <maksim.sen@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-19 13:55:54 -03:00
Olli-Antti Kivilahti
09e330aa24 Bug 13660: Exclude export phase and use existing exported MARCXML - rebuild_zebra_sliced.sh
When looking for a bad MARC Record using the rebuild_zebra_sliced.sh, it is
useful to skip the complete MARCXML exporting from Koha and reuse the exported
files for Zebra indexing.

This patch adds a new parameter:
    -x | --exclude-export Do not export Biblios from Koha, but use the existing
                          export-dir

Which depends on the:
     -d | --export-dir     Where rebuild_zebra.pl will export data
                           Default: $EXPORTDIR

 !---------!
! TEST PLAN !
 !---------!

1. Run
     "./rebuild_zebra_sliced.sh --length 1000"
   to export 1000 MARC Records
   and slice them to one big 1000-Record chunk.
2. Realize that you get an imaginary "stack smashing detected"-error crashing
   your indexing at some Record you dont know of and can't make out from the
   indexing logging.
3. Start looking for the bad Record by running:
     "./rebuild_zebra_sliced.sh --exlude-export --chunk-size 10"
   To skip Biblios export from Koha which takes ~2h and get straight into
   splitting your exported biblios to chunks of 10, and indexing them. You
   know which chunk fails so it is much easier to find the issue there.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-01-09 17:23:50 -03:00
6835760dcf Bug 18389: (QA follow-up) Fix tabs, note used template, info on verbose
Fixed whitespace for QA tools
Added a verbose note when template found
Only print 'Modifying MARC' if verbose

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-27 14:10:29 -03:00
Jon Knight
ab586abaee Bug 18690: Add error if there are multiple MARC templates with the same name
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-27 14:10:29 -03:00
Jon Knight
ff23e78f7b Bug 18690: Use MARC modification templates with bulkmarcimport.pl
When importing large numbers of MARC records from a legacy LMS to Koha
using bulkmarcimport.pl, it did not make use of the MARC modification
templates in the system (which can be useful for coversion of 852
fields to 952 fields for item holdings for example). This patch allows
MARC modification templates to be used with bulkmarcimport.pl.

To test:
1) Apply patch.
2) Set up a MARC modification template (in Home > Tools > MARC
modification templates) to make some changes to imported MARC
records (for example copy a subfield).
3) Take a test set of MARC records that have fields matching the
template and import them using the bulkmarcimport.pl tool. For example
if these MARC records are in testrecords.mrc and the MARC modification
template is called testtemplate use something like:

   perl misc/migration_tools/bulkmarcimport.pl -commit 1000 \\
     -file testrecords.mrc -marcmodtemplate testtemplate

4) Check the imported records in Koha to see that the required
modifications have been applied when the MARC records are imported.
5) Sign off.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-27 14:10:29 -03:00
10b5e1ee04 Bug 19122: (bug 18098 follow-up) Fix IncludeSeeFromInSearches behaviour
The IncludeSeeFromInSearches system preference is designed so that 'See from' headings from the authorities are included when you search in the catalog.
That means that you could find an author not only by the name printed on the book, but for example also by their pseudonym or a different spelling of their name.

It was added by bug 7417.

This regression has been introduced by
  commit 5ef1b6710e
  Bug 18098: Add an index with the count of not onloan items

-        } elsif ($record_type eq 'biblio' && C4::Context->preference('IncludeSeeFromInSearches')) {
-            my $normalizer = Koha::RecordProcessor->new( { filters => 'EmbedSeeFromHeadings' } );
[...]
+            push @filters, 'IncludeSeeFromInSearches'
+                if C4::Context->preference('IncludeSeeFromInSearches');

Test plan:
- Activate IncludeSeeFromInSearches
- Catalog an authority for a person
  - main heading in 100
  - see from headings in 400
- Catalog a bibliographic record and link it to the authority
- Make sure the record is indexed
- Verify that the record can be found searching for the main heading
- Verify that the record can be found searching for the see from headings

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Yet another reason to get rid of all this functions from this script.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-04 13:21:51 -03:00
2671eb2f93 Bug 18927: Use fully qualified subroutine names in C4::Items
rebuild_zebra.pl fails in some conditions (perl version?)
I do not recreate but it has been reported that reindex fails with:
  error retrieving biblio 94540 at /usr/share/koha/bin/migration_tools/rebuild_zebra.pl line 683, <DATA> line 751.

To fix it we can use fully qualified subroutine names for:
  GetMarcFromKohaField
  GetMarcBiblio
  GetBiblionumberFromItemnumber
  TransformKohaToMarc
  GetFrameworkCode

Test plan:
Confirm the rebuild_zebra script still works correctly after this patch

Signed-off-by: Lee Jamison <ldjamison@marywood.edu>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-09-06 12:55:00 -03:00
a89d60e88b Bug 19040: Update 2 occurrences of GetMarcBiblio in 22_to_30
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-08-25 10:51:24 -03:00
Mark Tompsett
d5986c9b97 Bug 19040: Refactor GetMarcBiblio parameters
Change parameters to a hashref.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Looks good to me.
Two calls in migration_tools/22_to_30 still in old style.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-08-25 10:23:42 -03:00
bd3b1c1a33 Bug 18262: Koha::Biblio - Remove GetBiblioData - part 1
Most of the time C4::Biblio::GetBiblioData is used to retrieve the title
and/or the author of a bibliographic record.

This patch replaces the easy occurrences of GetBiblioData, the ones
where the 2 joins are needed, but only data from biblio and biblioitems
table are.

Test plan:
It will be hard to test everything, I'd suggest a QAer to review this
patch and confirm that the difference occurrences of GetBiblioData have
been correctly replaced by calling Koha::Biblios->find or
$biblio->bibioitem

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-07-14 12:22:23 -03:00
ed03edddbd Bug 18277: Remove GetBiblionumberFromItemnumber - create_analytical_rel.pl
I have no idea how to test this patch, see bug 5528, or simply read the
code.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-07-10 13:03:37 -03:00
caa4cccfa0 Bug 16758: Use the default cache instance
I do not see a valid reason not to use the default one instead of the
syspref one.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-05-12 08:49:42 -04:00
Jacek Ablewicz
84dbc80074 Bug 16758 - Caching issues in scripts running in daemon mode
As L1 cache does not have expiration mechanism, scripts running
in daemon mode (rebuild_zebra.pl -daemon, sip server ?, ...) would
not be aware of any possible changes in the data being cached
in upstream L2 cache.

This patch adds ->flush_L1_caches() call in rebuild_zebra.pl
inside daemon mode loop.

To test:

1) apply patch
2) ensure that rebuild_zebra.pl -daemon is still working properly,
without any noticeable performance degradation
3) stop memcached daemon and try to run rebuild_zebra.pl -daemon
again: there should be a warning emitted stating that the script
is running in daemon mode but without recommended caching system

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-05-12 08:49:42 -04:00
5ef1b6710e Bug 18098: Add an index with the count of not onloan items
This patch adds a numeric index 'not-onloan-count' containing the value
of 999$x. This subfield is filled by 'rebuild_zebra.pl' by making use of
(bug's 18208) 'EmbedItemsAvailability' filter.

bib1.att and indexes definitions are updated accordingly.

To test:
- Apply the patch
- Pick the right biblio-zebra-indexdefs.xsl file for your setup and
  replace the one your Zebra uses [1]
- Replace your bib1.att
- Replace your ccl.properties
- Have at least one record with more than one item, checkout some
  item(s) from that record(s).
- Rebuild zebra's indexes:
  $ sudo koha-shell kohadev
 k$ cd kohaclone
 k$ misc/migration_tools/rebuild_zebra.pl -r -b -v -k
 (notice the dump directory is kept, you can try the XSLT yourself
  running:
    $ xsltproc \
       etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
       /tmp/the_dump_dir/biblios/exported_records | less
=> SUCCESS: There are records with the not-onloan-count index, and the
            value is correct!
- Check Zebra yourself:
  $ yaz-client unix:/var/run/koha/kohadev/bibliosocket
 Z> base biblios
 Z> find @attr 1=9013 @attr 2=5 @attr 4=109 0
=> SUCCESS: The search matches the amount of records with not-onloan
            items.
 Z> s 1+1
=> SUCCESS: Records with 999$x having a value higher than 0 are rendered
- Sign off :-D

Note: While this work is complete on its purpose, it is part of an
attempt to create a better way of filtering by availability.

Sponsored-by: ByWater Solutions

 [1] In kohadevbox this would be
/etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl

Edit: Added the missing XSLT changes for UNIMARC and NORMARC

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-05-08 09:21:41 -04:00
phette23
553ce38225 Bug 18432: code comments assume male gender
Comments throughout the Koha codebase assume that
all librarians or borrowers are male by using the
pronoun 'he' universally. This patch changes to
'he or she' / 'him or hers'.

Testing plan:

- ensuring modifying tests still pass:
	+ C4/SIP/t/06patron_enable.t
	+ t/db_dependent/Circulation.t
	+ t/db_dependent/Koha/Patrons.t
	+ t/db_dependent/Reserves.t

Sponsored-By: California College of the Arts

No code changes detected.
Signed-off-by: Marc Véron <veron@veron.ch>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-04-21 10:56:43 -04:00
dea93375dc Bug 9988: Refactor the cron script
The cron job is moved from migration tools to cronjobs. And renamed to
a plural form. The script is now based on Koha objects. It does no longer
include the code to merge one record. This can be done via the interface,
and will be added to a maintenance script on bug 18071. Should not be part
of this cron job.

Adding a cron_cleanup method to MergeRequests; this method is called from
the cron script to reset older entries still marked in progress and to
also remove old processed entries. Tested in a separate unit test.

Test plan:
[1] Run t/db_dependent/Authorities/MergeRequests.t
[2] Set AuthorityMergeLimit to 0. (All merges are postponed.)
[3] Modify an authority linked to a few records.
[4] Delete an authority linked to a few records with batch delete tool.
[5] And select two auth records with linked records.
    Merge these two records with authority/merge.pl.
    Note: Do not select Default. See also bug 17380.
[6] Check the need_merge_authorities table for inserted records.
[7] Run misc/cronjobs/merge_authorities.pl -b and inspect the linked
    records and the record status in need_merge_authorities.

Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-04-13 08:53:47 -04:00
04263912c7 Bug 9988: Check the merge limit in sub merge
At this point, we are replacing dontmerge functionality by the new
AuthorityMergeLimit logic. Instead of doing this check before calling
merge, we just call merge and check it there. In order to let the cron
job do the larger (postponed) merges, we add a parameter override_limit.

A subtest is added in Merge.t to test the 'postponed merge' feature. Since
merge now also calls get_usage_count, an additional mock is added. All
references to dontmerge are removed.

In merge two lines, initializing $dbh and $counteditbiblios, are moved.
The dontmerge test in DelAuthority and ModAuthority is removed. Since this
did not leave much in ModAuthority, I fixed the whitespace on the remaining
lines rightaway (yes, I know).
A minimal set of changes is applied to the cron script; it will get further
attention on a next patch.

Test plan:
[1] Run t/db_dependent/Authorities/Merge.t
[2] Set AuthorityMergeLimit to 2. Modify an authority with two linked
    biblios. Check that the merge was done immediately.
[3] Now modify an authority with more than 2 linked records.
    Verify that the merge was postponed; a record must be inserted in
    the need_merge_authorities table.
[4] Testing of the merge cron job is *postponed* to a next patch.
    Note: I tested a modification, but the script just needs more
    attention.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-04-13 08:53:46 -04:00
4c762ba69c Bug 9988: Merge should have a parameter hash
We will need a few additional parameters for merge later on. This patch
puts the original parameters in a parameter hash.
For the same reason DelAuthority gets a parameter hash here.

Note: We remove the second parameter from the DelAuthority call in
authorities/authorities-home.pl here. It was not used and could have
presented problems in the future.

Test plan:
[1] Run t/db_dependent/AuthoritiesMarc.t.
[2] Run t/db_dependent/Authorities/Merge.t.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-04-13 08:53:46 -04:00
cb58b11f67 Bug 18070: Extend sub merge to remove fields for deleted authorities
In order to accomplish this, we need to add some additional checks in
the merge routine. The actual change to remove the field, is quite
small.

Furthermore, we need to add a merge call in DelAuthority and adjust
the merge cron job accordingly.

The change is well supported by additional tests, including a simulation
of postponed removal via cron, if dontmerge is enabled.

Note: Deleting an authority with linked biblios is tested on the next
patch.

Test plan:
[1] Run t/db_dependent/Authorities/Merge.t
[2] Delete an authority without linked biblios from the Authorities
    module. If the indexer is not fast enough, wait a bit and refresh to
    verify that the authority is gone on authorities-home.pl.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-03-03 18:12:09 +00:00
b4c66d5cc4 Bug 17935: Adjust some POD lines, fix a few typos
This patch does the following:

[1] Move some POD lines from Cache to Caches.
[2] Correct C4::Plugins to Koha::Plugins in POD line of Koha::Plugins
[3] POD Koha/AuthorisedValue.pm: lib_opac moved to opac_description
[4] The POD in Koha/Patron.pm uses head2 and head3 inconsistently.
    Ran s/^=head2/=head3/ on those lines (7 substitutions on 7 lines)
[5] Correct a copied POD line from reports/issues_stats.pl in
    reports/reserve_stats.pl.
[6] Correct a test description in t/db_dependent/Koha/Authorities.t.
    You should never delete the library :)
[7] Correct typo shouild in a comment of rebuild_zebra.pl

Test plan:
[1] Read the patch. Does it make sense?
[2] Run perldoc Koha/Cache.pm and Koha/Caches.pm
[3] Run t/db_dependent/Koha/Authorities.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Cook <dcook@prosentient.com.au>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-14 14:12:50 +00:00
Magnus Enger
dbada67196 Bug 4126 - Exit bulkmarcimport if -a and -b given
Currently it is possible to spceify both --biblios and --authorities
as command line switches to bulkmarcimport.pl. This does not make sense
so we should exit early and explain that these switches are mutually
exclusive.

To test:
- Run one of these and check that there is no complaint about missing
  options:
  perl misc/migration_tools/bulkmarcimport.pl -a -b
  sudo koha-shell -c "perl misc/migration_tools/bulkmarcimport.pl -a -b"
  kohadev
- Observe that this displays the perldoc, but does not complain about
  mutually exclusive switches.
- Apply the patch
- Rerun the command(s) from earlier.
- Verify that the script is now halted and a small explanation given.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-14 14:00:02 +00:00
7a2cbfba1f Bug 17731: Remove noxml option from rebuild_zebra.pl
The removal of the noxml is a logical follow-up of bug 16506 (which
make xml the default).

Actually this option should have been removed by bug 10455 (it removes
the biblioitem.marc field).

Test plan:
Make sure the rebuild_zebra.pl script works as before.

Signed-off-by: Emma Smith <emma.nakamura.smith@gmail.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-19 13:05:08 +00:00
4fd41a4893 Bug 17196: Fix build_oai_sets.pl
The wrong value was retrieved.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-13 13:49:28 +00:00
12cf801945 Bug 17196: Update switch_marc21_series_info.pl
Not sure if this script is still used, could someone confirm?

Test plan:
If you know how to test it, please do

Signed-off-by: Mason James <mtj@kohaaloha.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-13 13:49:27 +00:00
c6e488f4af Bug 17196: Move marcxml out of the biblioitems table
Two discussions on koha-devel lead to the same conclusion:
biblioitems.marcxml should be moved out this table
- biblio and biblioitems
http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html
- biblioitems.marcxml & biblioitems.marc / HUGE performance issue !
http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html

There are several goals to do it:
- Performance
As Paul Poulain wrote, a simple query like
  SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear;
takes more than 10min on a DB with more than 1M bibliographic records
but only 3sec (!) on the same DB without the biblioitems.marcxml field
Note that priori to this patch set, the biblioitems.marcxml was not
retrieved systematically, but was, at least, in
C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders
- Flexibility
Storing the marcxml in a specific table would allow use to store several
kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats (marcflavour)
- Clean code
It would be a first step toward Koha::MetadataRecord for bibliographic
records (not done in this patch set).

Test plan:
- Update the DBIC Schema
- Add / Edit / Delete / Import / Export bibliographic records
- Add items
- Reindex records using ES
- Confirm that the following scripts still work:
    * misc/cronjobs/delete_records_via_leader.pl
    * misc/migration_tools/build_oai_sets.pl
- Look at the reading history at the OPAC (opac-readingrecord.pl)
- At the OPAC, click on a tag, you must see the result

Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108.

Signed-off-by: Mason James <mtj@kohaaloha.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-01-13 13:49:26 +00:00
bfcc7cad70 Bug 17376 - rebuild_zebra.pl in daemon mode no database access kills the process
When running rebuild_zebra.pl in daemon mode, a while loop runs the script for ever.
But if something crashes inside the rebuild process, the all daemon crashes.
For example when it can not access database.
This problem may be temporary so daemon should keep running.

This patch add eval around the rebuild process to allow a run to fail without killing the daemon.
Also moves the DB handler get inside daemon loop because it is broken is DB stoppes.

This is a big issue for indexer running in a systemd service.

Test plan :
- run rebuild_zebra.pl in daemon mode :
/home/koha/src/misc/migration_tools/rebuild_zebra.pl -daemon -z -a -b -x --sleep 30
- stop the database
- wait a minute
=> you see an error on database connexion
=> the daemon is still running
- restart the database
- test the indexer by creating a new record (wait for a minute)

Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-10-28 11:28:37 +00:00
6122b8fe6e Bug 16830: (followup) Remove weird character from warning in rebuild_zebra.pl
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-08-04 19:41:42 +00:00
6c65b64c84 Bug 16505: Make sure $as_xml will not be used later
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-05-23 17:29:23 +00:00
5dd1b1bb66 Bug 16506: Remove warning for UNIMARC installs
Use of uninitialized value in numeric eq (==)

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-05-23 17:29:12 +00:00
6c7a8c57e7 Bug 16506: (followup) Fix wrong option switch warning message
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-05-23 17:29:12 +00:00
7f22619001 Bug 16506: (Followup) remove warnings
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-05-23 17:29:12 +00:00
0b3b53e4f2 Bug 16506: Make rebuild_zebra.pl use XML as default
This patch deprecates the -x switch, making XML the default serialization format
used by rebuild_zebra.pl. It doesn't remove the option switch, but raises a warning
for the end user about the deprecation so they fix their cronjobs. Later we could remove it.

To test:
- Disable all indexing (daemon/cronjob)
- Create 2 records
- Edit one of them, delete the other one
- Verify they are queued for updates in zebraqueue
- sudo koha-mysql kohadev
  > SELECT * FROM zebraqueue WHERE done=0
...
| 265 |                265 | specialUpdate | biblioserver |    1 | 2016-05-13 14:23:45 |
| 266 |                  1 | recordDelete  | biblioserver |    1 | 2016-05-16 14:14:33 |
| 267 |                  2 | specialUpdate | biblioserver |    1 | 2016-05-16 14:15:06 |
+-----+--------------------+---------------+--------------+------+---------------------+
- Now go to koha-shell
  $ sudo koha-shell kohadev ; cd kohaclone
- Run:
  $ misc/migration_tools/rebuild_zebra.pl -k -b -z

  You will get something similar to this:
NOTHING cleaned : the export /tmp/jI0OeHy6Tn has been kept.
You can re-run this script with the -s  and -d /tmp/jI0OeHy6Tn parameters
if you just want to rebuild zebra after changing the record.abs
or another zebra config file
- Verify
  * less /tmp/jI0OeHy6Tn/del_biblio/exported_records
  * less /tmp/jI0OeHy6Tn/upd_biblio/exported_records
=> FAIL: They contain the records you added/modified/deleted but they are in
         USMARC format
- Apply the patch
- Mark your records for indexing (in koha-mysql kohadev)
  > UPDATE zebraqueue SET done=0 WHERE id > 264
- Run:
  $ misc/migration_tools/rebuild_zebra.pl -k -b -z

  You will get something similar to this:
<WARNINGS> [1]
NOTHING cleaned : the export /tmp/jI0OeHy6Tn has been kept.
You can re-run this script with the -s  and -d /tmp/jI0OeHy6Tn parameters
if you just want to rebuild zebra after changing the record.abs
or another zebra config file
- Verify
  * less /tmp/jI0OeHy6Tn/del_biblio/exported_records
  * less /tmp/jI0OeHy6Tn/upd_biblio/exported_records
=> SUCCESS: Data is correctly in XML format
- Run:
  $ misc/migration_tools/rebuild_zebra.pl -k -b -z -noxml

  You will get something similar to this:
<WARNINGS> [1]
NOTHING cleaned : the export /tmp/jI0OeHy6Tn has been kept.
You can re-run this script with the -s  and -d /tmp/jI0OeHy6Tn parameters
if you just want to rebuild zebra after changing the record.abs
or another zebra config file
- Verify
  * less /tmp/jI0OeHy6Tn/del_biblio/exported_records
  * less /tmp/jI0OeHy6Tn/upd_biblio/exported_records
=> SUCCESS: Data is correctly in USMARC format
- Sign off :-D

[1] Warnings covered by a followup

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
On top of Bug 16505
Work as described following test plan, usmarc default pre patch,
post patch xml default and usmarc on request.
No errors (all patchset)

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-05-23 17:29:12 +00:00
7a178fd262 Bug 16505: <collection> is missing the marc namespace and updates fail if -x is passed
Using rebuild_zerba.pl with the -x option switch, produces an incorrect output in
terms of what our XSLTs expect for indexing. This patch introduces the right namespace information
on the exported records so indexing succeeds.

To test:
- On current master, have some records on your db
- Run:
  $ sudo koha-shell kohadev
  $ cd kohaclone
  $ misc/migration_tools/rebuild_zebra.pl -r -b -k -x
=> you will get a message like this:

NOTHING cleaned : the export /tmp/NL5ufjUfpp has been kept.

- Run
  $ less /tmp/NL5ufjUfpp/biblio/exported_records
=> FAIL: The first line looks like this

<?xml version="1.0" encoding="UTF-8"?><collection><record

- Now run:
  $ xsltproc \
     /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
     /tmp/NL5ufjUfpp/biblio/exported_records
=> FAIL: No output
- Apply the patch
- Run:
  $ misc/migration_tools/rebuild_zebra.pl -r -b -k -x
- Take a look at the result file:
  $ less /tmp/asdiouqwiue/biblio/exported_records
=> SUCCESS: The start of the file looks like this:
<?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim">

- Run:
  $ xsltproc \
     /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
     /tmp/asdiouqwiue/biblio/exported_records
=> SUCCESS: There is actually indexing data :-D
- Sign off :-D

Edit: I changed qq{} for q{} as suggested by Jonathan.

Sponsored-by: American Numismatic Society

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works as described following test plan
No errors

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-05-23 17:04:04 +00:00
Robin Sheat
b4392018bc Bug 12478: make things using SimpleSearch use the new version
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:07 +00:00
ec3387185d Bug 16010: follow-up of 15381 - FIX merge_authorities migration script
Caused by commit 7e70202d34
    Bug 15381: Remove GetAuthType and GetAuthTypeCode

If you execute perl misc/migration_tools/merge_authority.pl -f 1 -t 2
you will get:
Can't locate object method "authtypecode" via package "1" (perhaps you forgot to load "1"?)
 at misc/migration_tools/merge_authority.pl line 58.

GetAuthority does not return a Koha::Authority but a MARC::Record:
there is no authtype code method!

Test plan:
perl misc/migration_tools/merge_authority.pl -f X -t Y
Should not return any error.
Note that if the authid X or Y does not exist, the script will die.

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
2016-03-21 18:28:46 +00:00
a60943bc1b Bug 15188: Do not remove unused auths if zebra is not reachable
Other conn errors should be checked (wrong user/pwd, etc.)

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-01-27 01:03:06 +00:00
Alex Arnaud
128e7f54b1 Bug 15188 - Fixes remove_unused_authorities.pl will delete all authorities if zebra is not running
Test plan:

1) Shut down zebra: koha-zebra-ctl.sh stop
2) Execute the script: remove_unused_authorities.pl (with -t if you don't want to really delete from your database).
3) Check that the script would have deleted all the authorities (eg: 31449 authorities parsed, 31449 deleted and 0 unchanged because used).
4) Apply patch
5) Run, notice that the script dies because zebra is unavailable

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2016-01-27 01:03:05 +00:00