Commit graph

44 commits

Author SHA1 Message Date
Chris Nighswonger
374cdb2678 Fixing up a regexp to stop a trivial warn 2010-11-04 14:24:46 -04:00
MJ Ray
65f8573b5d Display available error information during bulkmarcimport 2010-10-13 02:17:05 +01:00
5916908e35 Bug 4125 - Reformat with perldoc bulkmarcimport.pl doc
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2010-02-06 08:06:07 -05:00
Paul Poulain
a6e1f838ae bulkmarcimport : removing warnings 2010-01-28 15:11:56 +01:00
Henri-Damien LAURENT
ce3adab2ee bulkmarcimport.pl Bug Fix matching biblios enhanced
matching biblios is now also getting biblioitemnumber so that Items management can be performed
2009-11-23 21:40:13 +01:00
Paul Poulain
7cc1115cba adding error details 2009-11-17 16:27:12 +01:00
Henri-Damien LAURENT
7eca37db4f Authorities bulkmarcimport
Adding some new options to bulkmarcimport :
-k idtagsubfield in order to store the id of the file record into another field
-match tagsubfield,index
-a to import authorities
-l logfilename to store logs

Bug Fixing : C4/Charset.pm
Charset was incorrect for UNIMARC Authorities

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-09-30 11:22:21 +02:00
Sébastien Hinderer
2a8df0bc2f Get rid of a few warnings in the bulkmarcimport script: C4/Biblio.pm, hunks #1, #2: a warning occurring in NoZebra configurations. C4/Biblio.pm hunk #3: warning occurring in Unimarc MARC flavour. misc/migration_tools/bulkmarcimport.pl hunk #1: warning occurring when no default format is specified on command-line with -m switch.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-09-06 09:54:11 -04:00
Colin Campbell
3199d032e5 Avoid numeric comparisons with leading zeroes
Numbers in perl with leading zeros are interpreted in octal
Ensure that comparisons are done using string operators
or where appropriate use the MARC::Field method

Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
2009-08-20 21:01:52 -04:00
Galen Charlton
da51de184c bug 2926: fix staging import hang
Fixes a hang of the staging import tool when it
attempts to process a MARC21 record that claims
that it's UTF-8 when it is not.  The staging import
will now attempt to fix the character encoding of such
records.

Also added a FIXME to bulkmarcimport.pl, which because
of its use of MARC::Batch will skip over such records -
better than the original hang of the staging import, but
worse than the staging import's new ability to fix such
records.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-06-07 13:17:06 -05:00
J. David Bavousett
a7d1ab0041 Changes to bulkmarcimport.pl
Adds three new switches:

-idmap <filename> - optional output file of
                    map of source record ID numbers
                    to Koha biblionumber
-x                - if idmap is supplied, MARC tag
                    to get source record ID from
-y                - if idmap is supplied, MARC subfield
                    to get source record ID from

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
2009-04-03 19:18:29 -05:00
004524584b Tweak bullmarcimport.pl
* Add a new parameter -o to begin importing input file after skiping
  n records.
* Enclose input file reading in an eval directive to avoid abording
  import if few records are corrupted: they are now skipped.
* Help formating.

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-04-17 05:52:53 -05:00
Galen Charlton
b4f39e5c58 do not let MARC::Batch open MARC files
The version of MARC::Batch->new() distributed with version
2.0.0 of MARC::Record, if given a file name, will
open it using the ':utf8' layer.  This results in an
incorrect character conversion when processing records
in the MARC-8 character encoding.

To avoid this, batch jobs that use MARC::Batch now
open the file themselves, then pass the file handle
to MARC::Batch->new().

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-21 21:46:39 -05:00
Galen Charlton
4e95689287 bulkmarcimport.pl: XML input option documented
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-03-03 13:01:00 -06:00
Mason James
bdd2afc747 a little speed tweak here, setting "SET FOREIGN_KEY_CHECKS = 0" *before* clearing bib/bi/items tables.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-18 22:07:09 -06:00
Galen Charlton
60a98d258a IMPORTANT - refactor MARC character set handling
* IsStringUTF8ish - determine if scalar contains a string in UTF8
* MarcToUTF8Record - convert MARC blob or MARC::Record to UTF8
* SetMarcUnicodeFlag - set appropriate MARC21 or UNIMARC field to
  indicate that record is in UTF-8.

Design points of this module include:

* No dependencies on other C4 modules, making it easier to add
  more test cases
* All character conversion code in one place
* Single entry point for doing a character conversion on a
  MARC record
* Capture of errors and warnings produced by Text::Iconv
  and MARC::Charset
* Start of support for guessing the source character set of
  a MARC record.

Several functions were moved from other scripts
or modules to C4::Charset:

* C4::Koha->FixEncoding (expanded and renamed
  MarcToUTF8Record)
* C4::Koha->char_decode5426
* fMARC8ToUTF8 from bulkmarcimport.pl (renamed
  _marc_marc8_to_utf8)

Several batch jobs were adjusted to use MarcToUTF8Record instead of
FixEncoding.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-02-03 07:23:56 -06:00
Joshua Ferraro
2a37c19dac Rudimentary import of MARC21 authorities
Also adding support for ingesting format MARCXML in bulkmarcimport and bulkauthimport

Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-04 21:30:17 -06:00
Joshua Ferraro
9c25d6368a improvements to INSTALL.debian, adding Symbols for currencies adding \n to make bulkmarcimport.pl prettier
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-03 21:28:37 -06:00
Galen Charlton
c2a0ed8077 item rework: replaced AddBiblioAndItems
Replace C4::Biblio::AddBiblioAndItems with two
things:

* An option to C4::Biblio::AddBiblio to defer writing
  biblioitems.marc and biblioitems.marcxml.  This
  option was created to give a significant
  speed boost to bulkmarcimport.pl, but is *not*
  recommended for general use.
* C4::Items::AddItemBatchFromMarc

This refactoring removes the need to have functions
in C4::Biblio and C4::Items that call each other's
private functions.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-03 16:26:16 -06:00
Galen Charlton
9d4d8897b2 item rework: various changes
* Move CheckItemPreSave to C4::Items (from C4::Biblio)
* Modified C4::Biblio::AddBiblioAndItems to use appropriate
   internal routines from C4::Items
* Moved GetItemnumberFromBarcode to C4::Items
* Removed duplicate C4::Biblio::_koha_new_items
* Removed disused C4::Biblio::MARCitemchange

Currently AddBiblioAndItems is a special routine that
uses private subs from both C4::Biblio and C4::Items.
This needs to be refactored.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-03 16:25:42 -06:00
Joshua Ferraro
5c23369af2 Fixing Database Definitions for Statuses *PARTIAL*
Prior to this fix, the status fields had three 'off' values, NULL, "",
and 0. I've reduced it to two in the db, removing the option for NULL, and
setting the default value to 0, however, we need to verify that we don't ever
write out as "" as this needlessly complicates the indexing process,
critical for searching or limiting by status (e.g., availability). Also,
queries that attempt to write a NULL value to one of these fields will fail
(based on my tests).

This patch includes the following changes:

* Updated the database definition for notforloan, damaged, itemlost, and
wthdrawn in kohastructure.sql to forbid NULL and default to 0; MySQL
can't forbid other values (such as empty ""), so this has to be handled
at the application layer and REQUIRES further patching.

* Fixed the 'limit by availability' query node in Search.pm to use a
much less confusing definition of 'available'

* Added code to set values to 0 where they are NULL or empty ( "" ) for
notforloan, damaged, itemlost or wthdrawn in both the MARC and the items
table:

  * Biblio.pm -> AddBiblioAndItems
  * catalogue/updateitem.pl
  * SEE NOTE BELOW, REQUIRES UPDATE TO THE REST OF KOHA'S ITEM MGT!

* Removed code in bulkmarcimport.pl that sets notforloan status depending
  on item-level or bib-level itemtype -- that flag is designed to be set
  only to override the notforloan setting for the item's (or bib's,
  depending on the syspref) assigned itemtype (it doesn't need to override
  to 'for loan', only to 'not for loan').

  added $dbh->do("truncate zebraqueue"); when operation is 'delete'

* I updated some notes in catalogue/updateitem.pl as to why ModItem can't be
used -- we don't have _a_ place where we can change the item and marc :/

  I've tested the following:

  bulkmarcimport.pl..........................MARC/items OK
  Staged Records Import......................NOT OK
  updateitem.pl (via moredetail.pl)..........MARC/items OK
  circulation.pl.............................NOT OK
  returns.pl.................................NOT OK
  addbiblio.pl...............................NOT OK
  additem.pl.................................NOT OK

Basically, there isn't a single place to apply this patch that will
update both item data and MARC data in one place ... a future patch
needs to address this issue.

Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2008-01-03 16:23:04 -06:00
Galen Charlton
3508933c66 bulkmarcimport: enable MARC-8 to UTF-8 conversion
Enabled automatic conversion of MARC-8 records to
UTF-8.  Record is converted if its Leader/09 contains
a blank and the -s (skip) option hasn't been supplied
on the command-line.  Any record that cannot be converted
to UTF-8 is skipped.

Also now use Unicode Normalization Form C (NFC) for
records converted from MARC-8.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-12-25 09:08:38 -06:00
Galen Charlton
d426a91d0e removed extraneous comments
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-12-25 09:08:35 -06:00
Galen Charlton
cb6cf680bc improved error detection in AddBiblioAndItems
Introduced new C4::Biblio function CheckItemPreSave,
which checks for duplicate barcodes and invalid
branch codes.  Not yet sure whether this function
needs to be exported or whether it will just be
used internally to C4::Bibli.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-12-25 09:08:34 -06:00
Galen Charlton
6b49df4c3f removed superfluous comments
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-12-25 09:08:31 -06:00
Galen Charlton
7d47666f7e bulk MARC record import - speed improved
Changes to improve speed of MARC bib and item
imports:

[1] Turn off autocommit and commit database
    transactions in larger batches.
[2] Introduce a new C4::Biblio function (AddBiblioAndItems)
    to combine AddBiblio and AddItems -- this is faster
    because we are not parsing the MARC XML of the biblio
    every time we add an item.
[3] Introduce FasterTransformMarcToKoha, which is much
    faster than TransformMarcToKoha.  The new version,
    which will replace the old one once it has been
    fully tested, scans through each field in the
    MARC record just once, instead of potentially
    dozens of times.
[4] Remove code in bulkmarcexport that moved the
    item tags to separate MARC::Record objects.

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-12-25 09:08:28 -06:00
Galen Charlton
b8a58c4934 installer: command-line scripts improve finding C4 modules
Command-line scripts now use a new SCRIPT_DIR/kohalib.pl
to put installed location of Koha's Perl modules
into @INC.
2007-12-17 09:13:54 -06:00
Galen Charlton
ad4e02f91d warn on attempts to add duplicate item barcodes during batch import
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-12-02 15:06:24 -06:00
Mason James
a51118833c wrapping AddBiblio(), and AddItem() in evals{} to protect import from failure due to bad records.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-11-11 18:44:13 -06:00
Paul POULAIN
1ac38782a1 #1474 : Bulkmarcimport croaks when Log is ON
set to 0 and restore at the end of the import

Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
2007-10-11 14:53:59 -05:00
Paul POULAIN
375d2f1158 (minor) updating doc & removing warn
Signed-off-by: Chris Cormack <crc@liblime.com>
2007-10-03 14:57:12 -05:00
Joshua Ferraro
b87d4924b9 commenting out set_service_options, but also removes commit op
Signed-off-by: Chris Cormack <crc@liblime.com>
2007-10-01 17:40:31 -05:00
Ryan Higgins
c44efe7b84 fix bad call to GetMarcFromKohaField in bulkmarcimport, and add -fk param, allowing disabling of fk constraints during import.
Signed-off-by: Chris Cormack <crc@liblime.com>
2007-09-30 21:16:50 -05:00
toins
4728830e34 it's faster to 'truncate' instead of using 'delete from'... 2007-06-08 09:41:14 +00:00
tipaul
a481fad4b7 Code cleaning :
== Biblio.pm cleaning (useless) ==
* some sub declaration dropped
* removed modbiblio sub
* removed moditem sub
* removed newitems. It was used only in finishrecieve. Replaced by a Koha2Marc+AddItem, that is better.
* removed MARCkoha2marcItem
* removed MARCdelsubfield declaration
* removed MARCkoha2marcBiblio

== Biblio.pm cleaning (naming conventions) ==
* MARCgettagslib renamed to GetMarcStructure
* MARCgetitems renamed to GetMarcItem
* MARCfind_frameworkcode renamed to GetFrameworkCode
* MARCmarc2koha renamed to TransformMarcToKoha
* MARChtml2marc renamed to TransformHtmlToMarc
* MARChtml2xml renamed to TranformeHtmlToXml
* zebraop renamed to ModZebra

== MARC=OFF ==
* removing MARC=OFF related scripts (in cataloguing directory)
* removed checkitems (function related to MARC=off feature, that is completly broken in head. If someone want to reintroduce it, hard work coming...)
* removed getitemsbybiblioitem (used only by MARC=OFF scripts, that is removed as well)
2007-03-29 13:30:31 +00:00
tipaul
a3999812e6 rel_3_0 moved to HEAD 2007-03-09 14:52:58 +00:00
thd
ad657e71eb For MARC 21, instead of deleting the whole subfield when a character does not
translate properly from MARC8 into UTF-8, only the problem characters are
deleted.
2006-09-01 17:11:53 +00:00
toins
eac83ccd45 Head & rel_2_2 merged 2006-07-04 15:02:42 +00:00
rangi
10b2315eb3 Fixing the problem that all items were getting biblioitem=1 set 2006-04-01 22:10:50 +00:00
kados
44b4d37b54 removed Zconns, no need for them anymore with new Context.pm setup 2006-02-27 01:06:30 +00:00
kados
fafe0896d6 minor bugfix with 'commit' option 2006-02-25 23:40:59 +00:00
kados
77abbe2caf A bulkmarcimport.pl that is based on the new Biblio.pm Zebra routines.
It now responds to:

-n : the number of records to import.
-commit : the number of records to wait before performing a 'commit' operation

ALSO: IMPORTANT: I took out the char_encoding as this should be handled by
MARC::File::XML now, unless I'm mistaken.
2006-02-25 21:53:48 +00:00
tipaul
c52e5b61dd synch'ing 2.2 and head 2005-08-04 14:10:52 +00:00
tipaul
51e204fa23 moving bulkmarcimport script to migration_tools directory 2005-01-03 15:25:50 +00:00
Renamed from misc/bulkmarcimport.pl (Browse further)