Commit graph

210 commits

Author SHA1 Message Date
kados
77abbe2caf A bulkmarcimport.pl that is based on the new Biblio.pm Zebra routines.
It now responds to:

-n : the number of records to import.
-commit : the number of records to wait before performing a 'commit' operation

ALSO: IMPORTANT: I took out the char_encoding as this should be handled by
MARC::File::XML now, unless I'm mistaken.
2006-02-25 21:53:48 +00:00
tipaul
f74823bf1b OK, this time it seems to work. The last blocking problem was... a space in
recordId: (bib1,Identifier-standard) just after the comma. Adam agreed it was a bug, and it should be solved soon. But now we are aware, we can avoid putting the space !

In this commit you have all what is needed to setup a working zebra DB in Unimarc :
* collection.abs is UNIMARC specific and must be rewritten for MARC21, in marc21 directory
* pdf.properties is to be copied unmodified in the marc21 directory (can also be put somewhere else)
* rebuild_zebra.pl is SLOW, but 1 step reindexing tool, using ZOOM
* rebuild_zebra_idx is FAST, but 2 step reindexing tool, and does not use zebra. run it, it will create all biblios XML files in /zebra/biblios directory, then zebraidx update biblios in your zebra directory
* zebra.cfg is the zebra config file ;-)
* test_cql2rpn.pl is a script that will query the database and show the results. Works for me, just change the query at the beginning to get answers you expect.

What has to be done :
* benchmarking : it seems the zebraidx update is faster than lightning (400biblios/sec : 10 000biblios in 25seconds), while ZOOM indexing is slow (something like 25biblios/second) More benchmarking could be done.
* completing collection.abs for UNIMARC. I'll take care of it.
* modifying Biblio.pm to use ZOOM instead of the "zebraidx through exec" running actually. I'll take care of it also.
* modify the search API & tools & screens. I'll let the ball to someone else (chris ?) for this. I agree SearchMarc.pm can be dropped and replaced by something else (maybe a new-and-clean Search.pm package)
2006-02-09 10:59:34 +00:00
tipaul
369ee65d94 new version of rebuild_zebra. Should work with Perl-ZOOM, but DOES NOT WORK for me.
I get  :
ZOOM error 10002 "Encoding failed" from diag-set 'ZOOM'

help expected from indexdata...
2006-01-10 17:03:32 +00:00
tipaul
d5938493d7 synch'ing head and rel_2_2 (from 2.2.5, including npl templates)
Seems not to break too many things, but i'm probably wrong here.
at least, new features/bugfixes from 2.2.5 are here (tested on some features on my head local copy)

- removing useless directories (koha-html and koha-plucene)
2006-01-06 16:39:37 +00:00
tipaul
dba37f38e7 This script can be use to rebuild the zebra DB. It stores all koha MARC records in iso2709, in the bilbios directory. After that, you just have to "zebraidx update biblios"
I tried on a 9900 DB, here are the results :

[paul@bureau migration_tools]$ ./rebuild_zebra.pl -c
9900
9903 MARC record done in 37.9104120731354 seconds

[paul@bureau zebra]$ zebraidx update biblios
<snip>
18:31:24-11/08 zebraidx(20348) [log] Iterations . . . 144575
18:31:24-11/08 zebraidx(20348) [log] Distinct words .  39891
18:31:24-11/08 zebraidx(20348) [log] Updates. . . . .     46
18:31:24-11/08 zebraidx(20348) [log] Deletions. . . .      2
18:31:24-11/08 zebraidx(20348) [log] Insertions . . .  39843
18:31:24-11/08 zebraidx(20348) [log] zebra_register_close p=0x8104cf8
18:31:25-11/08 zebraidx(20348) [log] Records:    9887 i/u/d 9881/6/0
18:31:25-11/08 zebraidx(20348) [log] user/system: 531/145
18:31:25-11/08 zebraidx(20348) [log] zebra_stop
18:31:25-11/08 zebraidx(20348) [log] zebraidx times: 11.33  5.31  1.45
2005-08-11 16:35:54 +00:00
tipaul
c52e5b61dd synch'ing 2.2 and head 2005-08-04 14:10:52 +00:00
tipaul
64cd740d2b synch'ing 2.2 and head 2005-05-04 08:58:30 +00:00
tipaul
93ff09d081 merging 2.2 branch with head. Sorry for not making it before, many many commits done here 2005-03-01 13:40:35 +00:00
tipaul
51e204fa23 moving bulkmarcimport script to migration_tools directory 2005-01-03 15:25:50 +00:00
tipaul
cd6f87a689 Auto-build LANG authorized values 2005-01-03 12:59:49 +00:00