Commit graph

152 commits

Author SHA1 Message Date
Robin Sheat
8de1ef7e94 Bug 5228 - make rebuild_zebra handle fixing the zebra dirs
If the zebra server directories don't exist, zebra will spit the dummy.
This makes rebuild_zebra.pl smart enough to create them if they're not
there. If that fails, it'll scream loudly so you know zebra isn't
reindexing.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
2010-12-13 21:59:49 +13:00
Chris Nighswonger
374cdb2678 Fixing up a regexp to stop a trivial warn 2010-11-04 14:24:46 -04:00
MJ Ray
65f8573b5d Display available error information during bulkmarcimport 2010-10-13 02:17:05 +01:00
Robin Sheat
57d11aee2c Bug 5077 - ensure rebuild_zebra will run somewhere it can read
This prevents it leaving files lying around in /tmp

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-10-06 08:00:17 -04:00
Lars Wirzenius
3e7e386148 Convert to UTF-8.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-05-06 17:58:24 -04:00
Donovan Jones
5e0b850d49 Bug 2505 - Add commented use warnings where missing in the misc/ directory 2010-04-21 20:26:44 +12:00
Lars Wirzenius
87d845969e Fix FSF address in directory misc/
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-03-16 20:17:54 -04:00
5916908e35 Bug 4125 - Reformat with perldoc bulkmarcimport.pl doc
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-02-06 08:06:07 -05:00
Paul Poulain
a6e1f838ae bulkmarcimport : removing warnings 2010-01-28 15:11:56 +01:00
Henri-Damien LAURENT
ce3adab2ee bulkmarcimport.pl Bug Fix matching biblios enhanced
matching biblios is now also getting biblioitemnumber so that Items management can be performed
2009-11-23 21:40:13 +01:00
Paul Poulain
7cc1115cba adding error details 2009-11-17 16:27:12 +01:00
Henri-Damien LAURENT
663eb1edd6 Adding some error proof on GetMarcRecord
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-09-30 11:29:24 +02:00
Henri-Damien LAURENT
7eca37db4f Authorities bulkmarcimport
Adding some new options to bulkmarcimport :
-k idtagsubfield in order to store the id of the file record into another field
-match tagsubfield,index
-a to import authorities
-l logfilename to store logs

Bug Fixing : C4/Charset.pm
Charset was incorrect for UNIMARC Authorities

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-09-30 11:22:21 +02:00
Ricardo Dias Marques
f8ff5879a5 Bug 3582: Missing usage information for -h / --help switch for rebuild_nozebra.pl
Fix for Bug 3582:  Missing usage information for -h / --help switch for rebuild_nozebra.pl

http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=3582

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-09-06 12:48:35 -04:00
Sébastien Hinderer
2a8df0bc2f Get rid of a few warnings in the bulkmarcimport script: C4/Biblio.pm, hunks #1, #2: a warning occurring in NoZebra configurations. C4/Biblio.pm hunk #3: warning occurring in Unimarc MARC flavour. misc/migration_tools/bulkmarcimport.pl hunk #1: warning occurring when no default format is specified on command-line with -m switch.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-09-06 09:54:11 -04:00
Colin Campbell
3199d032e5 Avoid numeric comparisons with leading zeroes
Numbers in perl with leading zeros are interpreted in octal
Ensure that comparisons are done using string operators
or where appropriate use the MARC::Field method

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-20 21:01:52 -04:00
Henri-Damien LAURENT
731b82f764 3519 : mergeauthority and authority edition were not synched
mergeauthority and ModAuthority were working on two separate directories.
So that no authority would ever be merged via cronjob or commandline script
when MergeAuthoritiesOnUpdate is disable

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-11 19:30:14 -04:00
3caec55fd1 removed redundant license statement
The standard license statement in the header is fine; please
don't confuse things by doing anything different.

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-01 08:17:52 -04:00
Paul Poulain
6b1df98ddf script to remove authorities without biblio attached
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-01 08:10:01 -04:00
459d732180 Bug 3301 - Speed up rebuild_zebra script
With this patch, rebuild_zebra can re-index a whole Koha DB
quickly:

  rebuild_zebra -r -b -nosanitize

Biblio (authority) records are dump directly in a file
from marcxml field without beeing transformed into
MARC::Record object and corrected.

DOCUMENTATION:

rebuild_zebra.pl new paramater:

-nosanitize  export biblio/authority records directly from DB marcxml
             field without sanitizing records. It speed up
             dump process but could fail if DB contains badly
             encoded records. Works now only with -x and -b

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-29 07:52:46 -05:00
Brian Harrington
6a2d9ffcf2 Bug 3313, bulkauthimport.pl skips MARC21 subdivision records.
This patch adds the MARC21 subdivsion record tags (18x) to the
block which recognizes and assigns authtypecodes to imported
authority records.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-08 17:03:03 -05:00
Galen Charlton
da51de184c bug 2926: fix staging import hang
Fixes a hang of the staging import tool when it
attempts to process a MARC21 record that claims
that it's UTF-8 when it is not.  The staging import
will now attempt to fix the character encoding of such
records.

Also added a FIXME to bulkmarcimport.pl, which because
of its use of MARC::Batch will skip over such records -
better than the original hang of the staging import, but
worse than the staging import's new ability to fix such
records.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-07 13:17:06 -05:00
Galen Charlton
3f4641bf30 bug 3201: missing090field.pl - skip bad bibs
Patch courtesy of G. Henry <henry@cmi.univ-mrs.fr>

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-07 13:17:01 -05:00
J. David Bavousett
a7d1ab0041 Changes to bulkmarcimport.pl
Adds three new switches:

-idmap <filename> - optional output file of
                    map of source record ID numbers
                    to Koha biblionumber
-x                - if idmap is supplied, MARC tag
                    to get source record ID from
-y                - if idmap is supplied, MARC subfield
                    to get source record ID from

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-04-03 19:18:29 -05:00
Henri-Damien LAURENT
911fddab4a merge_authority : Bug fixing
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-03-06 14:14:34 -06:00
Mason James
e9599f973c Fixes command-line 'number' arg in bulkauthimport.pl.
for HEAD and 3.0.x

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-03-04 10:43:49 -06:00
Brian Harrington
25cd35b3a1 bug 2924 fixed rebuild_zebra.pl to work when export is skipped
reindexing now occurs if there are $num_records_exported or if
$skip_export is set

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-03-04 08:28:22 -06:00
Galen Charlton
8f07521a2d bug 2955: fix remaining calls to GetMarcFromKohaField
This includes part of a patch from Henri-Damien Laurent
that could not be applied because Chris and Joe patches
happened to win the race.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-02-12 16:29:19 -06:00
Joe Atzberger
11b90be284 Cleanup and perltidy.
Add "use warnings", remove unused variables and unnecessary finish/disconnect
at the end.  This script could be improved to run only on tables that need to
be altered instead of touching all of them.  It should also probably contain
warnings to the effect that it does not rescue your DATA that was forced into
whatever encoding the table used previously.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-01-28 17:29:43 -06:00
Michael Hafen
086b3ccf9a bug in rebuild_zebra verbose logging - found another print I didn't want to see all the time
Add the phrase 'if ( $verbose_logging )' to the two print statements
concerning the skipping of biblio or authority records.

I recently had to split biblio and authority index updating in my cron
script ( had some really big records so had to add the -x switch which
should only be used on biblios accourding to the help ).  So I noticed
that rebuild_zebra.pl printed messages that it was skipping biblios or
authorities.

This patch is to conditionalize those prints based on the verbose
logging switch.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2008-12-11 09:23:28 -06:00
Michael Hafen
62a590a954 Reduce logging from rebuild_zebra.pl with a command line option
This reduces the output of the script and zebraidx, and creates a -v
command line switch which will increase the logging to their former
states.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2008-10-01 13:05:20 -05:00
Henri-Damien LAURENT
ca8d24546e Bug Fixing merge_authority.pl
merge works on the fly now.
But for an obscure reason, merge_authority.pl fails to update database when lanched on command line.
Adding one table to LOCK for noZebra UPDATE in Biblio.pm
You should remove C4::Search from merg_authority.pl

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-08-09 11:05:53 -05:00
Galen Charlton
df1f46f9da bug 2253: improve rebuild_zebra's handling of zebraqueue
Prior to this patch, rebuild_zebra.pl -z was effectively
hanging on to a lock on the zebraqueue table, preventing
other scripts from inserting new entries into the table.
This had the effect of causing circulation operations
to time out.

Refactored by having rebuld_zebra.pl pull the active
queue into memory, then mark entries done by zebraqueue.id.
Consequently, rebuild_zebra.pl should no longer
block adding new entries into zebraqueue.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-06-19 09:49:06 -05:00
Paul POULAIN
feae120738 BUGFIX : script to fix & fill onloan field in items table.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-05-12 09:24:43 -05:00
Galen Charlton
a78b115d35 kohabug 2076 - make biblioitems.marc longblob during upgrade
Change to match 3.0 definition of that column.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-05-11 05:37:18 -05:00
Paul POULAIN
8e1844d495 missing )
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-05-05 05:39:13 -05:00
Paul POULAIN
e7209ed02a UNIMARC specific rebuild items correctly
note 995 for items is hardcoded, so it's really for UNIMARC only. The script exit if you're not UNIMARCflavour

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-22 17:34:41 -05:00
Galen Charlton
3109d5820e rebuild_zebra.pl - add -y option
rebuild_zebra.pl will now mark all zebraqueue entries
of the affected record type(s) done when run in
normal mode to index all records (as opposed to running
it with -z to just process the zebraqueue).  This prevents
any running zebraqueue_daemon processes from attempting
to reindex the same records, redundantly.

The new -y swtich overrides this new behavior; in other words, if
running rebuild_zebra.pl without -z, you can specify
-y to *not* mark zebraqueue done.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-21 11:17:29 -05:00
004524584b Tweak bullmarcimport.pl
* Add a new parameter -o to begin importing input file after skiping
  n records.
* Enclose input file reading in an eval directive to avoid abording
  import if few records are corrupted: they are now skipped.
* Help formating.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-17 05:52:53 -05:00
Galen Charlton
e2c1f11715 fixed memory leak I introduced
Accidentally introducing a circular reference in a
MARC::Record object does not lead to goodness, particularly
if you export lots and lots of them.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-01 06:46:05 -05:00
Galen Charlton
4f001186b6 still more rebuild_zebra refactoring
Merged duplicate code for indexing bibs and
authorities into a single index_records() function.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:58:03 -05:00
Galen Charlton
a5576b8dfe IMPORTANT: added -z option to rebuild_zebra.pl
The -z option, when used in conjunction with -a and/or -b,
selects the records to reindex from the zebraqueue table.
Both record updates and record deletes are handled.

-z is cannot be used with -s or -r: the updated records
must always be freshly exported, and if zebraqueue
is to be processed, it's assumed that you don't want
to drop the Zebra index first.

This means that rebuild_zebra.pl -b -a -x can be
used as a cronjob to update the indexes periodically; it
is believed that this will offer much better indexing
performance on some setups as compared to zebraqueue_daemon.pl,
which uses Z39.50 extended services to send record updates
to Zebra.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:58:01 -05:00
Galen Charlton
57d128f727 rebuild_zebra: exit if both -a and -x specified
At moment using both -a (index authorities) and
-x (export records as MARC XML) is not allowed -
if the Zebra authority database is using the DOM
filter, zebraidx will not be able to process the
exported records correctly.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:57:44 -05:00
Galen Charlton
f0d5da7448 more rebuild_zebra.pl refactoring
1. Logic to fix up record IDs, UNIMARC 100 field,
   and record leader now in separate functions.
2. Removed (incorrect) logic to save corrected record
   in database.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:57:43 -05:00
Galen Charlton
f98c27a8bc refactor rebuild_zebra: new routine for invoking zebraidx
Created a routine for calling zebraidx, replacing
separate invocations for bibs and authorities.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:57:42 -05:00
Galen Charlton
ae8a76dacc rebuild_zebra.pl: removed disused $limit option
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:57:41 -05:00
Galen Charlton
b4f39e5c58 do not let MARC::Batch open MARC files
The version of MARC::Batch->new() distributed with version
2.0.0 of MARC::Record, if given a file name, will
open it using the ':utf8' layer.  This results in an
incorrect character conversion when processing records
in the MARC-8 character encoding.

To avoid this, batch jobs that use MARC::Batch now
open the file themselves, then pass the file handle
to MARC::Batch->new().

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-21 21:46:39 -05:00
Galen Charlton
ad0639e548 remove some unneeded use statements
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-21 21:46:29 -05:00
Galen Charlton
4e95689287 bulkmarcimport.pl: XML input option documented
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-03 13:01:00 -06:00
Galen Charlton
d49873cc2f bulkauthimport.pl - various improvements
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-03 13:00:59 -06:00