Commit graph

139 commits

Author SHA1 Message Date
Ricardo Dias Marques
f8ff5879a5 Bug 3582: Missing usage information for -h / --help switch for rebuild_nozebra.pl
Fix for Bug 3582:  Missing usage information for -h / --help switch for rebuild_nozebra.pl

http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=3582

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-09-06 12:48:35 -04:00
Sébastien Hinderer
2a8df0bc2f Get rid of a few warnings in the bulkmarcimport script: C4/Biblio.pm, hunks #1, #2: a warning occurring in NoZebra configurations. C4/Biblio.pm hunk #3: warning occurring in Unimarc MARC flavour. misc/migration_tools/bulkmarcimport.pl hunk #1: warning occurring when no default format is specified on command-line with -m switch.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-09-06 09:54:11 -04:00
Colin Campbell
3199d032e5 Avoid numeric comparisons with leading zeroes
Numbers in perl with leading zeros are interpreted in octal
Ensure that comparisons are done using string operators
or where appropriate use the MARC::Field method

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-20 21:01:52 -04:00
Henri-Damien LAURENT
731b82f764 3519 : mergeauthority and authority edition were not synched
mergeauthority and ModAuthority were working on two separate directories.
So that no authority would ever be merged via cronjob or commandline script
when MergeAuthoritiesOnUpdate is disable

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-11 19:30:14 -04:00
3caec55fd1 removed redundant license statement
The standard license statement in the header is fine; please
don't confuse things by doing anything different.

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-01 08:17:52 -04:00
Paul Poulain
6b1df98ddf script to remove authorities without biblio attached
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-01 08:10:01 -04:00
459d732180 Bug 3301 - Speed up rebuild_zebra script
With this patch, rebuild_zebra can re-index a whole Koha DB
quickly:

  rebuild_zebra -r -b -nosanitize

Biblio (authority) records are dump directly in a file
from marcxml field without beeing transformed into
MARC::Record object and corrected.

DOCUMENTATION:

rebuild_zebra.pl new paramater:

-nosanitize  export biblio/authority records directly from DB marcxml
             field without sanitizing records. It speed up
             dump process but could fail if DB contains badly
             encoded records. Works now only with -x and -b

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-29 07:52:46 -05:00
Brian Harrington
6a2d9ffcf2 Bug 3313, bulkauthimport.pl skips MARC21 subdivision records.
This patch adds the MARC21 subdivsion record tags (18x) to the
block which recognizes and assigns authtypecodes to imported
authority records.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-08 17:03:03 -05:00
Galen Charlton
da51de184c bug 2926: fix staging import hang
Fixes a hang of the staging import tool when it
attempts to process a MARC21 record that claims
that it's UTF-8 when it is not.  The staging import
will now attempt to fix the character encoding of such
records.

Also added a FIXME to bulkmarcimport.pl, which because
of its use of MARC::Batch will skip over such records -
better than the original hang of the staging import, but
worse than the staging import's new ability to fix such
records.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-07 13:17:06 -05:00
Galen Charlton
3f4641bf30 bug 3201: missing090field.pl - skip bad bibs
Patch courtesy of G. Henry <henry@cmi.univ-mrs.fr>

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-07 13:17:01 -05:00
J. David Bavousett
a7d1ab0041 Changes to bulkmarcimport.pl
Adds three new switches:

-idmap <filename> - optional output file of
                    map of source record ID numbers
                    to Koha biblionumber
-x                - if idmap is supplied, MARC tag
                    to get source record ID from
-y                - if idmap is supplied, MARC subfield
                    to get source record ID from

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-04-03 19:18:29 -05:00
Henri-Damien LAURENT
911fddab4a merge_authority : Bug fixing
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-03-06 14:14:34 -06:00
Mason James
e9599f973c Fixes command-line 'number' arg in bulkauthimport.pl.
for HEAD and 3.0.x

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-03-04 10:43:49 -06:00
Brian Harrington
25cd35b3a1 bug 2924 fixed rebuild_zebra.pl to work when export is skipped
reindexing now occurs if there are $num_records_exported or if
$skip_export is set

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-03-04 08:28:22 -06:00
Galen Charlton
8f07521a2d bug 2955: fix remaining calls to GetMarcFromKohaField
This includes part of a patch from Henri-Damien Laurent
that could not be applied because Chris and Joe patches
happened to win the race.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-02-12 16:29:19 -06:00
Joe Atzberger
11b90be284 Cleanup and perltidy.
Add "use warnings", remove unused variables and unnecessary finish/disconnect
at the end.  This script could be improved to run only on tables that need to
be altered instead of touching all of them.  It should also probably contain
warnings to the effect that it does not rescue your DATA that was forced into
whatever encoding the table used previously.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-01-28 17:29:43 -06:00
Michael Hafen
086b3ccf9a bug in rebuild_zebra verbose logging - found another print I didn't want to see all the time
Add the phrase 'if ( $verbose_logging )' to the two print statements
concerning the skipping of biblio or authority records.

I recently had to split biblio and authority index updating in my cron
script ( had some really big records so had to add the -x switch which
should only be used on biblios accourding to the help ).  So I noticed
that rebuild_zebra.pl printed messages that it was skipping biblios or
authorities.

This patch is to conditionalize those prints based on the verbose
logging switch.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2008-12-11 09:23:28 -06:00
Michael Hafen
62a590a954 Reduce logging from rebuild_zebra.pl with a command line option
This reduces the output of the script and zebraidx, and creates a -v
command line switch which will increase the logging to their former
states.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2008-10-01 13:05:20 -05:00
Henri-Damien LAURENT
ca8d24546e Bug Fixing merge_authority.pl
merge works on the fly now.
But for an obscure reason, merge_authority.pl fails to update database when lanched on command line.
Adding one table to LOCK for noZebra UPDATE in Biblio.pm
You should remove C4::Search from merg_authority.pl

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-08-09 11:05:53 -05:00
Galen Charlton
df1f46f9da bug 2253: improve rebuild_zebra's handling of zebraqueue
Prior to this patch, rebuild_zebra.pl -z was effectively
hanging on to a lock on the zebraqueue table, preventing
other scripts from inserting new entries into the table.
This had the effect of causing circulation operations
to time out.

Refactored by having rebuld_zebra.pl pull the active
queue into memory, then mark entries done by zebraqueue.id.
Consequently, rebuild_zebra.pl should no longer
block adding new entries into zebraqueue.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-06-19 09:49:06 -05:00
Paul POULAIN
feae120738 BUGFIX : script to fix & fill onloan field in items table.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-05-12 09:24:43 -05:00
Galen Charlton
a78b115d35 kohabug 2076 - make biblioitems.marc longblob during upgrade
Change to match 3.0 definition of that column.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-05-11 05:37:18 -05:00
Paul POULAIN
8e1844d495 missing )
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-05-05 05:39:13 -05:00
Paul POULAIN
e7209ed02a UNIMARC specific rebuild items correctly
note 995 for items is hardcoded, so it's really for UNIMARC only. The script exit if you're not UNIMARCflavour

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-22 17:34:41 -05:00
Galen Charlton
3109d5820e rebuild_zebra.pl - add -y option
rebuild_zebra.pl will now mark all zebraqueue entries
of the affected record type(s) done when run in
normal mode to index all records (as opposed to running
it with -z to just process the zebraqueue).  This prevents
any running zebraqueue_daemon processes from attempting
to reindex the same records, redundantly.

The new -y swtich overrides this new behavior; in other words, if
running rebuild_zebra.pl without -z, you can specify
-y to *not* mark zebraqueue done.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-21 11:17:29 -05:00
004524584b Tweak bullmarcimport.pl
* Add a new parameter -o to begin importing input file after skiping
  n records.
* Enclose input file reading in an eval directive to avoid abording
  import if few records are corrupted: they are now skipped.
* Help formating.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-17 05:52:53 -05:00
Galen Charlton
e2c1f11715 fixed memory leak I introduced
Accidentally introducing a circular reference in a
MARC::Record object does not lead to goodness, particularly
if you export lots and lots of them.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-01 06:46:05 -05:00
Galen Charlton
4f001186b6 still more rebuild_zebra refactoring
Merged duplicate code for indexing bibs and
authorities into a single index_records() function.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:58:03 -05:00
Galen Charlton
a5576b8dfe IMPORTANT: added -z option to rebuild_zebra.pl
The -z option, when used in conjunction with -a and/or -b,
selects the records to reindex from the zebraqueue table.
Both record updates and record deletes are handled.

-z is cannot be used with -s or -r: the updated records
must always be freshly exported, and if zebraqueue
is to be processed, it's assumed that you don't want
to drop the Zebra index first.

This means that rebuild_zebra.pl -b -a -x can be
used as a cronjob to update the indexes periodically; it
is believed that this will offer much better indexing
performance on some setups as compared to zebraqueue_daemon.pl,
which uses Z39.50 extended services to send record updates
to Zebra.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:58:01 -05:00
Galen Charlton
57d128f727 rebuild_zebra: exit if both -a and -x specified
At moment using both -a (index authorities) and
-x (export records as MARC XML) is not allowed -
if the Zebra authority database is using the DOM
filter, zebraidx will not be able to process the
exported records correctly.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:57:44 -05:00
Galen Charlton
f0d5da7448 more rebuild_zebra.pl refactoring
1. Logic to fix up record IDs, UNIMARC 100 field,
   and record leader now in separate functions.
2. Removed (incorrect) logic to save corrected record
   in database.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:57:43 -05:00
Galen Charlton
f98c27a8bc refactor rebuild_zebra: new routine for invoking zebraidx
Created a routine for calling zebraidx, replacing
separate invocations for bibs and authorities.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:57:42 -05:00
Galen Charlton
ae8a76dacc rebuild_zebra.pl: removed disused $limit option
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-25 07:57:41 -05:00
Galen Charlton
b4f39e5c58 do not let MARC::Batch open MARC files
The version of MARC::Batch->new() distributed with version
2.0.0 of MARC::Record, if given a file name, will
open it using the ':utf8' layer.  This results in an
incorrect character conversion when processing records
in the MARC-8 character encoding.

To avoid this, batch jobs that use MARC::Batch now
open the file themselves, then pass the file handle
to MARC::Batch->new().

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-21 21:46:39 -05:00
Galen Charlton
ad0639e548 remove some unneeded use statements
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-21 21:46:29 -05:00
Galen Charlton
4e95689287 bulkmarcimport.pl: XML input option documented
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-03 13:01:00 -06:00
Galen Charlton
d49873cc2f bulkauthimport.pl - various improvements
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-03 13:00:59 -06:00
Mason James
5057e74914 more \Q...\E wrapping on regexes, to handle occassionally problematic strings.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-28 07:58:45 -06:00
Mason James
d451f072f9 corrections to host-item, shelf_loc and collection-code indexes
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-28 07:58:43 -06:00
Mason James
be14507658 oops, removing un-needed $dbh->commit() calls
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-20 20:16:42 -06:00
Mason James
b57c146b26 setting $dbh->{AutoCommit} = 0, and adding a new --commit arg.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-20 20:16:41 -06:00
Paul POULAIN
0e2b065219 NoZebra : removing . and : before indexing
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-19 20:27:36 -06:00
Paul POULAIN
bcf36122a6 speeding a lot rebuild_nozebra by using autocommit OFF feature
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-19 20:27:24 -06:00
Mason James
bdd2afc747 a little speed tweak here, setting "SET FOREIGN_KEY_CHECKS = 0" *before* clearing bib/bi/items tables.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-18 22:07:09 -06:00
Ryan Higgins
71dd69d5ac add option to export and index xml to rebuild_zebra
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-15 08:25:46 -06:00
Mason James
3cb4ea7ecf added 440* and 490* 'series' indexes
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-11 16:14:54 -06:00
Galen Charlton
60a98d258a IMPORTANT - refactor MARC character set handling
* IsStringUTF8ish - determine if scalar contains a string in UTF8
* MarcToUTF8Record - convert MARC blob or MARC::Record to UTF8
* SetMarcUnicodeFlag - set appropriate MARC21 or UNIMARC field to
  indicate that record is in UTF-8.

Design points of this module include:

* No dependencies on other C4 modules, making it easier to add
  more test cases
* All character conversion code in one place
* Single entry point for doing a character conversion on a
  MARC record
* Capture of errors and warnings produced by Text::Iconv
  and MARC::Charset
* Start of support for guessing the source character set of
  a MARC record.

Several functions were moved from other scripts
or modules to C4::Charset:

* C4::Koha->FixEncoding (expanded and renamed
  MarcToUTF8Record)
* C4::Koha->char_decode5426
* fMARC8ToUTF8 from bulkmarcimport.pl (renamed
  _marc_marc8_to_utf8)

Several batch jobs were adjusted to use MarcToUTF8Record instead of
FixEncoding.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-03 07:23:56 -06:00
Daniel Bünzli
78f3e56e2c bulkauthimport fix
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-22 07:20:28 -06:00
Joshua Ferraro
2a37c19dac Rudimentary import of MARC21 authorities
Also adding support for ingesting format MARCXML in bulkmarcimport and bulkauthimport

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-04 21:30:17 -06:00
Joshua Ferraro
9c25d6368a improvements to INSTALL.debian, adding Symbols for currencies adding \n to make bulkmarcimport.pl prettier
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-03 21:28:37 -06:00