This patch, permit to take care of similar entries, if there is some diacritics or not.
(cherry picked from commit 776c177e3debedaf08fec65fbf8111675ccc93e7)
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Allows temporary locations corresponding to 'in processing' and 'shelving'
so that newly-created items, and newly-returned items do not show
immediately as a available. Three new system preferences govern the usage
of these features.
NewItemsDefaultLocation. If system pref NewItemsDefaultLocation is set to a location code,
all newly catalogued items will be set to the location set in this preference.
Location code must be a valid LOC authorized value type.
InProcessingToShelvingCart. if the system pref InProcessingToShelvingCart is turned on,
any items run through returns.pl with a location code for 'PROC', will be modified to
have a new location code of 'CART'.
ReturnToShelvingCart. If the syspref ReturnToShelvingCart is turned on,
all items returned other than confirmed holds will have a new location code of 'CART'.
Any item issued is automatically taken of the shelving cart.
Adds a cron script shelf_to_cart.pl which should be run hourly.
Updates all items with a location of CART to the item's permanent location.
The original location code is stored in the new items column 'permanent_location'.
Original Author: PTFS Contractor <dbavousett@ptfs.com>
This work co-sponsored by
Middletown Township Public Library, Middletown, NJ USA and
East Brunswick Public Library, East Brunswick, NJ USA
Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Numbers in perl with leading zeros are interpreted in octal
Ensure that comparisons are done using string operators
or where appropriate use the MARC::Field method
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
mergeauthority and ModAuthority were working on two separate directories.
So that no authority would ever be merged via cronjob or commandline script
when MergeAuthoritiesOnUpdate is disable
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Fixed the holds queue job so that it correctly
ignores hold requests that are not yet scheduled
to be filled.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
This little script establishes a framework for database cleanup on some regular
schedule. Initial implementation provides for brute truncation of the sessions
table, and selective-by-age cleanup of the zebraqueue.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
The standard license statement in the header is fine; please
don't confuse things by doing anything different.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
It appears, in Ryan's patch, that he wants to throw a warning to the log if
the directory specified in --out is not present. (Further messages will
be given when the open-or-die occurs a few lines later.) However, it was
throwing the warning if --out was not specified at all, which is
undesirable. This patch modifies that bit to check for the presence of
whatever directory is going to be used, either --out, ENV{TMPDIR}, or /tmp.
As before, if the write to the directory fails for any reason--including
its' non-existence--that is handled later, but this message will help
inform the troubleshooter.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
With this patch, rebuild_zebra can re-index a whole Koha DB
quickly:
rebuild_zebra -r -b -nosanitize
Biblio (authority) records are dump directly in a file
from marcxml field without beeing transformed into
MARC::Record object and corrected.
DOCUMENTATION:
rebuild_zebra.pl new paramater:
-nosanitize export biblio/authority records directly from DB marcxml
field without sanitizing records. It speed up
dump process but could fail if DB contains badly
encoded records. Works now only with -x and -b
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This patch adds the MARC21 subdivsion record tags (18x) to the
block which recognizes and assigns authtypecodes to imported
authority records.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Fixes a hang of the staging import tool when it
attempts to process a MARC21 record that claims
that it's UTF-8 when it is not. The staging import
will now attempt to fix the character encoding of such
records.
Also added a FIXME to bulkmarcimport.pl, which because
of its use of MARC::Batch will skip over such records -
better than the original hang of the staging import, but
worse than the staging import's new ability to fix such
records.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The Offline Circ tool by Kyle Hall uses PHP's SQLite, which is SQLite
v2. Gnope, which Kyle links, ships with libSQLite v2.
Let's not count on libsqlite3 not being installed for perl. If it is
installed DBD::SQLite will use it, where DBD::SQLite2 will not.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
When marcxml cannot be parsed into a MARC::Record object, the biblio is
undisplayable and it obviously breaks many features in Koha. This script
can test to parse every marcxml, and alert on failures. Optionally, the
marcxml can be replaced from the marc field.
See extensive perldoc for details.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The problem is that we do not ensure that the issues table has valid
borrowernumber in each line. This is exacerbated by Getoverdues()
returning data sorted BY borrowernumber. So one NULL borrowernumber
in issues prevented ALL fines from being assessed. The actual error
from fines.pl cron log is:
No branchcode argument to new. Should be C4::Calendar->new(branchcode => $branchcode)
at /home/user/kohaclone/misc/cronjobs/fines.pl line 98
This patch deals only with getting fines to avoid crashing. It does
not fix the underlying data integrity problem.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This has bearing on bugs 2905, 2665, 2514 and other "wide character" crashes
related to diacritics and Unicode. This should help open the door for reliable
input of diacriticals via acquisitions.
MARC21_utf8_flag_fix.pl diagnoses and fixes existing problems with MARC data
affected by the bug.
Adding SetMarcUnicodeFlag to TransformKohaToMarc prevents the bug from corrupting
further data.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Acquisitions process seems to be adding records with incorrect
representation of the MARC encoding in leader/09. It should be
'a' meaning UTF-8, for all Koha's internalized records, but in
many cases it appears blank (for MARC-8). This script diagnoses
and repairs the value in the leader, depending on runtime options.
The symptom of this problem is that high-value UNICODE characters
in the record will cause Koha to crash whenever it tries to parse
the MARCXML, giving a "Wide character" fatal. While we work on
fixing the input, this script will fix the existing data.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
If series title was previously taken as collectiontitle
then you should launch
misc/maintenance/UNIMARC_fix_collectiontitle.pl
misc/batchRebuildBiblioTables.pl
[RM note: applies only to UNIMARC users - MARC21
users should not run the batch jobs.]
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
execute_query now refactored, returns reliable results, does
zero presentation-layer crap. Arguments reduced, client scripts
adapted to new API and performance improved. Text::CSV now used
to generate CSV output, ensuring portability, encoding and accuracy.
Replaced tools/runreport.pl with misc/cronjobs/runreport.pl:
~ security fixed
~ documentation improved
~ TODO: finish sendmail option.
Bug 3077 also fixed.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The old location of runreport.pl was under tools, leaving it exposed
to web requests. This is a security flaw since it does NOT check any
Auth and allows the user to request any Saved Report be run. This is
not a problem under misc/crontab/ and it suggests the more appropriate use.
Guided.pm is not fixed here (see bug 3066), but it is prepared to be fixed
and made compatible with runreport as detailed in the perldoc.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Adds three new switches:
-idmap <filename> - optional output file of
map of source record ID numbers
to Koha biblionumber
-x - if idmap is supplied, MARC tag
to get source record ID from
-y - if idmap is supplied, MARC subfield
to get source record ID from
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Pulled the GetMarcFromKohaField outside the loop, since we only need to
check it once. Same for $item_sth. Added safety checks for success of
delete_field and GetMarcItem, with warnings on failure.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Removed cronjob, which was written to work around
a bug in 2.2 that no longer applies and is specific
to a single library in any event.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This patch just fix the script which export to csv the overdue, and field the missing fields
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This does not fix all problems recorded in 2883 (see all the FIXME's), but
it does improve the script's basic feedback to an intelligible level.
It also adjusts the documentation and examples to correct bogus usage
instructions.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
* removed ersataz YAML::XS dependency
* use 'return' instead of 'return undef'
* minor language changes
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Improve URLs checker script in the way (half way) pointed out by Galen:
- A C4::URL::Checker class handle URL checking. This class is not yet
in a separate file in C4 directory. This class would be easily
extended to accomodate authorities URLs checking.
- Script output can now be formatted in CSV or HTML. HTML version
link directly to MARC biblio record editor.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Item type was not retrieved in a query, leading to a case
where an item could be selected by build_holds_queue.pl
to fill a hold request even where forbidden by the
library and item type-level policy.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This includes part of a patch from Henri-Damien Laurent
that could not be applied because Chris and Joe patches
happened to win the race.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
* use item branch instead of patron's branch to
look up the applicable hold policies - this makes
requesting in the OPAC consistent with the intranet.
* when generating pick list using build_holds_queue.pl, only match items
to patrons if request is allowed.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Add "use warnings", remove unused variables and unnecessary finish/disconnect
at the end. This script could be improved to run only on tables that need to
be altered instead of touching all of them. It should also probably contain
warnings to the effect that it does not rescue your DATA that was forced into
whatever encoding the table used previously.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This reverts commit def09f5a21.
As I emailed to the patches list Oct 06, 2008:
I suggest we need to revert Josh' commit def09f5a21.
The effect on the crontab example is to invalidate the lines being executed. The lines were apparently copied in from a cron source, not crontab, despite the header describing it NOT being an example for cron. It also runs longoverdue twice, instead of fines.pl.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The problem was that the TmplTokenizer create global symbols to store file
handlers so the objects aren't destroyed by the garbage collector.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
As rss.pl is not a CGI script, moved it to join the
other cronjobs. Full documentation of the script
is in misc/cronjobs/rss/rss.pl, but to summarize:
[1] rss.pl is run on the command line to produce
an RSS XML document. The output should be
placed in a directory accessible to the OPAC
(or staff) web interface so that users can download
the RSS feed. An example of usage:
misc/cronjobs/rss.pl lastAcquired.conf
Normally rss.pl should be run periodically (e.g., daily)
to keep the feed up-to-date.
[2] The configuration file (e.g., lastAcquired.conf) lists
* name of the template file to use
* path of output file
* SQL query
rss.pl runs the SQL query, then feeds the output of the
query through the template to produce the output file.
[3] The template file (e.g., lastAcquired.tmpl) uses
HTML::Template syntax like any of the HTML
templates for the web interface.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Add the phrase 'if ( $verbose_logging )' to the two print statements
concerning the skipping of biblio or authority records.
I recently had to split biblio and authority index updating in my cron
script ( had some really big records so had to add the -x switch which
should only be used on biblios accourding to the help ). So I noticed
that rebuild_zebra.pl printed messages that it was skipping biblios or
authorities.
This patch is to conditionalize those prints based on the verbose
logging switch.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Reduce the size of the SQLite database produced by
create_koc_db.pl by emiting only the columns
actually used by Kyle Hall's offline circ client.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This commit adds two options to the overdue_notices.pl script.
The notices are now generated in one of two modes: default or triggered.
In default mode, every time the script runs, a notice is generated if an item
is overdue within a notice date range. This mode is suitable for a weekly cron
(or manual run), but the notice triggers should be syncronized with the cron script such that
the script is run once per notice date range. Otherwise, duplicate messages will
be generated ( or no notice generated if a notice date range passes without the script being run).
Triggered mode is suitable for a nightly cron job: a notice is generated only on those
days that an overdue item enters a notice date range. This mode is susceptible to message
loss when the cron script fails to run, or when notice triggers are edited.
A second flag is added: --list-all, which causes items.content to list all currently overdue items.
Without this flag, only items that are overdue within the given notice's date range are listed on
that notice.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Restoring the functionality of these sysprefs as part of the
rework of build_holds_queue.pl:
* StaticHoldsQueueWeight is an ordered list of library codes.
The order of the libraries in this list determines the order
of libraries in which available items will be chosen to be
targeted to holds. Note that if this list does not include
a library, items from that library will never be targeted to
holds and staff will never be asked to retrieve items for
title-level holds from those libraries.
* RandomizeHoldsQueueWeight is a Boolean parameter, indicating
whether to randomize the libraries in the StaticHoldsQueueWeight
list.
If neither RandomizeHoldsQueueWeight nor StaticHoldsQueueWeight is
set, then the libraries will be used to target items to holds in
alphabetical order by library code.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This reduces the output of the script and zebraidx, and creates a -v
command line switch which will increase the logging to their former
states.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Most Perl scripts (as opposed to modules) do
not need to require Exporter.
No user-visible or documentation changes.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The offline circulation tool can use all of the patron information, not just their ID
numbers. This patch populates the offline circulation database with patron information.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
CalcFine returned values that mismatched expectations in fines.pl.
fines.pl refactored: added debugging, prevent needless recreation of
Calendar objects by storing them in hash by branch.
Still outstanding problems with fines, including the output of a field
that has no other references in Koha (so is always undef) and the
incorrect description of FinesMode.
Calendar exported "new" erroneously. I also cleaned up the queries to
avoid needlessly compiling additional statement handles.
Please test and consider application to 3.0 maintenance.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This is the script that generates the patron and holdings database for the offline circ module. I
touched up a few parts of it:
* fixed the database syntax to squash a bug or two
* extracted some methods for readability and reuse
* added some documentation
* added a --file and --force command line argument
* made it die if DBD::SQLite is not present. That module is required for this feature.
Here's an example crontab line that will generate the borrowers.db file each midnight.
0 0 * * * create_koc_db.pl --file /tmp/borrowers.db
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This script generates a sqlite database of patron and holdings data for the offline circulation tool.
It's a cronjob, so it should go into misc/cronjobs
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
SQL fix to properly ensure that if an item is on
the hold shelf, it will not be used for request
targeting, which applies only to items that are
not already on the hold shelf.
Prior to this fix, when checking out an item on the
hold shelf that fills patron A's hold request, it
was possible for that request to not be marked as
filled if another patron had an item-level request
on the item; the second patron's request was
incorrectly targeted by that item.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This batch job now automatically populates the
holds request targeting table hold_fill_targets.
This patch is essentially a revamp of the job,
and includes fixes for the following bugs:
2281 (holds queue report including unavailable items)
2331 (holds queue report not working with item-level
holds)
2332 (holds queue script should attempt to fill
many requests as possible).
Several functions in this batch job are candidates
for being moved to C4::Reserves:
GetBibsWithPendingHoldRequests()
GetPendingHoldRequestsForBib()
GetItemsAvailableToFillHoldRequestsForBib()
MapItemsToHoldRequests()
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This patch deprecates the NoReturnSetLost system preference, which, as it
turns out, was not implemented anyway. New longoverdue script allows one to
specify on the commandline system-wide delays for changing items to different
lost statuses, and optionally charge for the item.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
merge works on the fly now.
But for an obscure reason, merge_authority.pl fails to update database when lanched on command line.
Adding one table to LOCK for noZebra UPDATE in Biblio.pm
You should remove C4::Search from merg_authority.pl
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Correct bug in output of overdue notices that has
items.content displaying only the number of columns
in each items.content structure (currently, 4).
Patch inspired by patch submitted by Paul Poulain.
[LL bug 31]
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
This patch changes all $foo eq undef's to !defined($foo). It also makes misc/spellcheck_suggest/build_spellcheck_suggest.pl have proper syntax.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The lines for each item in the overdue notices were not separated by newlines. This cause them to
all be smashed together. I'm putting a newline between them.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
misc/maintenance/fix_accountlines_date.pl runs through the accountlines table and converts every date in the 'description' column from us or metric format (specified on the command line) to the format in the 'dateformat' syspref.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
These three programs have been replaced by misc/cronjobs/overdue_notices.pl:
overduenotices-30.pl
overduenotices-csv.pl
overduenotices.pl
This patch deletes them.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
This patch adds the misc/cronjobs/overdue_notices.pl script that is intended to replace
overduenotices.pl, overduenotices-30.pl and overduenotices-csv.pl. It adds messages to
the message_queue to be sent later (by process_message_queue.pl). It also marks borrowers
as debarred if their issues become too overdue.
It is intended to be run from cron nightly with usage something like:
0 2 * * * misc/cronjobs/overdue_notices.pl
C4::Members:
- improved documentation on ModMember
- made ModMember return a useful value (the return value of the database call)
- added a DebarMember method
- adding t/lib/KohaTest/Members/DebarMember.pm to test ModMember
misc/cronjobs/overdue_notices.pl
- designed to replace overduenotices.pl, overduenotices-30.pl, and overduenotice-csv
Changes to C4::Letters:
- EnqueueLetter now lets you pass in to_address and from_address which can override defaults
- _send_message_by_email pays attention to these defaults.
- now handles attachments with MIME::Lite
C4::Overdues
- added GetBranchcodesWithOverdueRules
- added t/lib/KohaTest/Overdues/GerBranchcodesWithOverdueRules.pm to test that.
circ/overdue.pl
- replaced call to obsolete overduenotices-csv.pl with call to overdue_notices.pl
KohaTest:
- added three helper methods: random_phone, random_email, random_ip
- these can be used to populate example records
- you can now pass an optional lengh to random_string
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
el-GR-i-staff-t-prog-v-3000000.po
es-ES-i-staff-t-prog-v-3000000.po
hy-Armn-i-staff-prog-v-3000000.po
ru-RU-i-staff-t-prog-v-3000000.po
tr-TR-i-staff-t-prog-v-3000000.po
uk-UA-i-staff-t-prog-v-3000000.po
zh-Hans-CN-i-staff-t-prog-v-3000000.po
These will require extensive modifs to apply correctly due to
syntax problems.
I've added methods to to C4::Letters to manage the database table
message_queue. This will let us keep track of messages sent
via email, sms, and rss to patrons. That way, we can show the history,
deal with failures, and reconstruct an RSS feed when needed.
misc/cronjobs/overduenotics.pl has been added. It prepares advance notices
and item due notices and stages messages to be sent in the message_queue
table.
C4::Overdues::Getoverdues now takes two optional arguments to tell it how
old of overdues to fetch.
Also, a C4::Circualtion::getUpcomingDueIssues method was added that
advance_notices.pl uses.
misc/cronjobs/process_message_queue.pl has been added. It sends the email
or SMS messages out of the message queue.
The C4::SMS module didn't work at all, and it has been rebuilt to use
an external perl module from CPAN, SMS::Send.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Prior to this patch, rebuild_zebra.pl -z was effectively
hanging on to a lock on the zebraqueue table, preventing
other scripts from inserting new entries into the table.
This had the effect of causing circulation operations
to time out.
Refactored by having rebuld_zebra.pl pull the active
queue into memory, then mark entries done by zebraqueue.id.
Consequently, rebuild_zebra.pl should no longer
block adding new entries into zebraqueue.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Created a new script, sync_items_to_marc_bib.pl,
to replace the item tags embedded in the MARC bib
records with fresh versions taken from the items table.
This script should be run as follows:
maintenance/sync_items_to_marc_bib.pl --run-update
Assuming that you're using Zebra, rebuild_zebra.pl -b -z
or rebuild_zebra.pl -b -r should be run after running
this script.
This script should be run if you have used
link_bibs_to_authorities.pl prior to the first
patch for bug 2258. It can also be used if there
is any reason to suspect that the embedded item tags
do not reflect the items table.
With this script I am creating a maintenance/ subdirectory of
misc/ to hold scripts that are meant to fix problems
in the database but are not (or should not be, anyway) necessary
for regular use.
Documentation: add to documentation for server side scripts
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
If a MARC bib is modified by this batch job,
do not duplicate the item tags embedded in
it (e.g., 952 for MARC21). When modifying
a bib record, any embedded item tags must
be removed before calling ModBiblio - perhaps
this should be moved to ModBiblio itself.
Also removed an error in the job's help text.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
During link_bibs_to_authorities.pl run, a bib was
observed that had both biblioitems.marc and biblioitems.marcxml
set to blank or NULL. Added check to verify that a
defined value is returned from GetMarcBiblio(); may follow
up with additional patches once it's determined how
the corrupt bib came to be loaded.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
[1] Increase sleep interval between checks of zebraqueue
from 0.01 seconds to 0.50.
[2] Batch up commits of changes to the zebraqueue table
[3] If the same record appears multiple times in the queue,
handle only once.
[4] Properly postpone failures to process record deletes to
avoid spinning.
[5] Correct how queue entries are marked done - avoid skipping
an authority record update, e.g., if it has the same
ID number as a bib that was updated.
[6] Added a FIXME about a possible later enhancement to
batch up updates so that Zebra isn't told to commit
after each record.
No documentation changes.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
from circulation dashboard, creates new sysprefs, assigns
the sysprefs to the proper tab in sysprefs (Circulation),
updatedatabase changes to do the previous, and fixing one
redundent limit in the query for build_holds_queue.pl
Note: still need to address item-level holds
the tmp_holdsqueue table. This is an alternative holds
targeting workflow that is more suitable for multi-location
libraries than the default holds picklist report.
Note to documentation writers: this summary should be
added to any holds documentation as an overview of
the avaialable methods for holds fulfillment.
This alternative holds workflow assumes an
expectation that the system should target a specific
item for a given hold request, attempt to fulfill the
hold with that item, and if unable to fulfill, select
an available item at another location to fulfill the
hold.
This is quite different than the default Koha behavior
which uses a 'broadcast' method of hold fulfillment.
How it works:
This script weights available locations for holds based
on options specified in two system preferences:
StaticHoldsQueueWeight
Allows the library to specify a list of library
location codes -- if used alone, it will rank the
list statically, selecting the top-ranking available
location to be added to the picklist.
RandomizeHoldsQueueWeight
If RandomizeHoldsQueueWeight and StaticHoldsQueueWeight
are set, the list of library codes in the
StaticHoldsQueueWeight syspref are randomized rather
than statically ranked. If RandomizeHoldsQueueWeight
alone is set, the list of all available library codes
is used to randomize the weight.
If neither syspref is set, the list is statically
ranked according to how they are pulled out of the system
database.
NOTE: This has not yet been tested with item-level holds
this script had quite serious issues :
- it would not use mindays and maxdays variables
- It would send latin1 where utf8 was expected
- It would send data without text delimiters (; was chosen if title contains ; it would have been a problem " used as delimiters now)
- It would write a file when it was not asked
Now stores the results in a string before printing it.
New option added to store result into a file : -o filename
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Enhanced the ability of catalogers to specify how
bib and item records should be added, replaced, or
ignored during a staging import.
When an import batch of bib records is staged and commit,
the user can now explicitly specify what should occur
when an incoming bib record has, or does not have, a match
with a record already in the database. The options are:
if match found (overlay_action):
create_new (just add the incoming record)
replace (replace the matched record with the incoming one)
use_template (option not implemented)
ignore (do nothing with the incoming bib; however, the
items attached to it may still be processed
based on the item action)
if no match is found (nomatch_action):
create_new (just add the incoming record)
ignore (do nothing with the incoming bib; in this
case, any items attached to it will be
ignored since there will be nothing to
attach them to)
The following options for handling items embedded in the
bib record are now available:
always_add (add the items to the new or replaced bib)
add_only_if_match (add the items only if the incoming bib
matches an existing bib)
add_only_if_add (add the items only if the incoming bib
does *not* match an existing bib)
ignore (ignore the items entirely)
With these changes, it is now possible to support the following use cases:
[1] A library joining an existing Koha database wishes to add their
items to existing bib records if they match, but does not want
to overlay the bib records themselves.
[2] A library wants to load a file of records, but only handle
the new ones, not ones that are already in the database.
[3] A library wants to load a file of records, but only
handle the ones that match existing records (e.g., if
the records are coming back from an authority control vendor).
Documentation changes:
* See description above; also, screenshots of the 'stage MARC records
for import' and 'manage staged MARC records' should be updated.
Test cases:
* Added test cases to exercise staging and committing import batches.
UI changes:
* The pages for staging and managing import batches now have
controls for setting the overlay action, action if no match,
and item action separately.
* in the manage import batch tool, user is notified when they
change overlay action, no-match action, and item action
* HTML for manage import batch tool now uses fieldsets
Database changes (DB rev 076):
* added import_batches.item_action
* added import_batches.nomatch_action
* added 'ignore' as a valid value for import_batches.overlay_action
* added 'ignored' as a valid value for import_records.status
* added 'status' as a valid value for import_items.status
API changes:
* new accessor routines for C4::ImportBatch
GetImportBatchNoMatchAction
SetImportBatchNoMatchAction
GetImportBatchItemAction
SetImportBatchItemAction
* new internal functions for C4::ImportBatch to
determine how a given bib and item are to be
processed, based on overlay_action, nomatch_action,
and item_action:
_get_commit_action
_get_revert_action
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
If running Zebra, try to locate zebrasrv and zebraidx
so that koha-zebra-ctl.sh can point to a functioning
zebrasrv.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
note 995 for items is hardcoded, so it's really for UNIMARC only. The script exit if you're not UNIMARCflavour
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The daemon control scripts (koha-zebra-ctl.sh, koha-zebraqueue-ctl.sh,
and koha-pazpar2-ctl.sh) are now copied and installed in a
runnable fashion for a 'dev'-mode install. By default
they are installed in the bin subdirectory of the runtime
directory.
Also:
* the control scripts now work if the EUID is other
than root (as would be expected for a 'dev' or 'single'
install).
* Split the SCRIPT_DIR installation target into
SCRIPT_DIR (scripts to copy regardless of install mode)
and SCRIPT_NONDEV_DIR (scripts to copy to SCRIPT_DIR
unless the install mode is 'dev').
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
rebuild_zebra.pl will now mark all zebraqueue entries
of the affected record type(s) done when run in
normal mode to index all records (as opposed to running
it with -z to just process the zebraqueue). This prevents
any running zebraqueue_daemon processes from attempting
to reindex the same records, redundantly.
The new -y swtich overrides this new behavior; in other words, if
running rebuild_zebra.pl without -z, you can specify
-y to *not* mark zebraqueue done.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* Add a new parameter -o to begin importing input file after skiping
n records.
* Enclose input file reading in an eval directive to avoid abording
import if few records are corrupted: they are now skipped.
* Help formating.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Add a explanation on DBD::mysql installation without test suite.
Add /misc/translator/install-code.pl script that creates templates
for specified language codes.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Accidentally introducing a circular reference in a
MARC::Record object does not lead to goodness, particularly
if you export lots and lots of them.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The -z option, when used in conjunction with -a and/or -b,
selects the records to reindex from the zebraqueue table.
Both record updates and record deletes are handled.
-z is cannot be used with -s or -r: the updated records
must always be freshly exported, and if zebraqueue
is to be processed, it's assumed that you don't want
to drop the Zebra index first.
This means that rebuild_zebra.pl -b -a -x can be
used as a cronjob to update the indexes periodically; it
is believed that this will offer much better indexing
performance on some setups as compared to zebraqueue_daemon.pl,
which uses Z39.50 extended services to send record updates
to Zebra.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
At moment using both -a (index authorities) and
-x (export records as MARC XML) is not allowed -
if the Zebra authority database is using the DOM
filter, zebraidx will not be able to process the
exported records correctly.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
1. Logic to fix up record IDs, UNIMARC 100 field,
and record leader now in separate functions.
2. Removed (incorrect) logic to save corrected record
in database.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The version of MARC::Batch->new() distributed with version
2.0.0 of MARC::Record, if given a file name, will
open it using the ':utf8' layer. This results in an
incorrect character conversion when processing records
in the MARC-8 character encoding.
To avoid this, batch jobs that use MARC::Batch now
open the file themselves, then pass the file handle
to MARC::Batch->new().
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The new tables have the same structure and constraints
as the tables they archive with the following exceptions:
* borrowernumber and biblionumber in old_reserves can be
NULL
* the FK constraints (e.g., for itemnumber) on old_reserves
set the child column to NULL if the parent row is deleted
instead of deleting the child row.
* there is no FK constraint on old_issues.branchcode, allowing
a branch to be deleted without changing archived requests.
Some miscellaneous cleanup was done as part of this patch:
* GetMemberIssuesAndFines (C4::Members) now uses bind variables
* fixed POD for GetMemberIssuesAndFines
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
INTRANET_TMPL_DIR
INTRANET_WWW_DIR
OPAC_TMPL_DIR
OPAC_WWW_DIR
KOHA_CONF_DIR
OPAC_TMPL_DIR
OPAC_WWW_DIR
PAZPAR2_CONF_DIR
ZEBRA_CONF_DIR
For each directory on the list, if an existing file
installed file is different from the version
coming from the new package, it is copied to a backup file
whose name will have the suffix "_koha_<old_version_number>"
or "_upgrade_backup", depending on whether
the --prev-install-log option was used or not
during 'perl Makefile.PL'.
The directories whose files were backed up were
chosen because they contain configuration and
HTML files that a non-programmer Koha user
is likely to change. Consequently, the backup
files produced by 'make upgrade' are not a substitute
for doing a complete backup of one's Koha
installation before upgrading.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* IsStringUTF8ish - determine if scalar contains a string in UTF8
* MarcToUTF8Record - convert MARC blob or MARC::Record to UTF8
* SetMarcUnicodeFlag - set appropriate MARC21 or UNIMARC field to
indicate that record is in UTF-8.
Design points of this module include:
* No dependencies on other C4 modules, making it easier to add
more test cases
* All character conversion code in one place
* Single entry point for doing a character conversion on a
MARC record
* Capture of errors and warnings produced by Text::Iconv
and MARC::Charset
* Start of support for guessing the source character set of
a MARC record.
Several functions were moved from other scripts
or modules to C4::Charset:
* C4::Koha->FixEncoding (expanded and renamed
MarcToUTF8Record)
* C4::Koha->char_decode5426
* fMARC8ToUTF8 from bulkmarcimport.pl (renamed
_marc_marc8_to_utf8)
Several batch jobs were adjusted to use MarcToUTF8Record instead of
FixEncoding.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Fixed syntax errors preventing compilation; however,
unsure whether this is a dead utility that should be
removed outright.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Needed to restore OpenSearch capabilities, and did the following while
I was at it:
* add support for unAPI: http://unapi.info/
* add basic support for COinS and OpenURL:
http://ocoins.info;
http://www.niso.org/committees/committee_ax.html
* ^^ Gives us Zotero Support!
* adding some XSLT stylesheets for handling additional transformations
NOTE: English and MARC21 specific unfortunately
* adding back opensearch/rss feed <link>s for autodiscovery
TODO: after the installation, to get the Zebra system running on an external
port it's necessary to hand-edit the configs. I'm looking into Virtual Hosts
which could solve that problem (run on both the socket and a port).
Need to add better error handling to the unapi and opensearch scripts
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Replace C4::Biblio::AddBiblioAndItems with two
things:
* An option to C4::Biblio::AddBiblio to defer writing
biblioitems.marc and biblioitems.marcxml. This
option was created to give a significant
speed boost to bulkmarcimport.pl, but is *not*
recommended for general use.
* C4::Items::AddItemBatchFromMarc
This refactoring removes the need to have functions
in C4::Biblio and C4::Items that call each other's
private functions.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* Move CheckItemPreSave to C4::Items (from C4::Biblio)
* Modified C4::Biblio::AddBiblioAndItems to use appropriate
internal routines from C4::Items
* Moved GetItemnumberFromBarcode to C4::Items
* Removed duplicate C4::Biblio::_koha_new_items
* Removed disused C4::Biblio::MARCitemchange
Currently AddBiblioAndItems is a special routine that
uses private subs from both C4::Biblio and C4::Items.
This needs to be refactored.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Prior to this fix, the status fields had three 'off' values, NULL, "",
and 0. I've reduced it to two in the db, removing the option for NULL, and
setting the default value to 0, however, we need to verify that we don't ever
write out as "" as this needlessly complicates the indexing process,
critical for searching or limiting by status (e.g., availability). Also,
queries that attempt to write a NULL value to one of these fields will fail
(based on my tests).
This patch includes the following changes:
* Updated the database definition for notforloan, damaged, itemlost, and
wthdrawn in kohastructure.sql to forbid NULL and default to 0; MySQL
can't forbid other values (such as empty ""), so this has to be handled
at the application layer and REQUIRES further patching.
* Fixed the 'limit by availability' query node in Search.pm to use a
much less confusing definition of 'available'
* Added code to set values to 0 where they are NULL or empty ( "" ) for
notforloan, damaged, itemlost or wthdrawn in both the MARC and the items
table:
* Biblio.pm -> AddBiblioAndItems
* catalogue/updateitem.pl
* SEE NOTE BELOW, REQUIRES UPDATE TO THE REST OF KOHA'S ITEM MGT!
* Removed code in bulkmarcimport.pl that sets notforloan status depending
on item-level or bib-level itemtype -- that flag is designed to be set
only to override the notforloan setting for the item's (or bib's,
depending on the syspref) assigned itemtype (it doesn't need to override
to 'for loan', only to 'not for loan').
added $dbh->do("truncate zebraqueue"); when operation is 'delete'
* I updated some notes in catalogue/updateitem.pl as to why ModItem can't be
used -- we don't have _a_ place where we can change the item and marc :/
I've tested the following:
bulkmarcimport.pl..........................MARC/items OK
Staged Records Import......................NOT OK
updateitem.pl (via moredetail.pl)..........MARC/items OK
circulation.pl.............................NOT OK
returns.pl.................................NOT OK
addbiblio.pl...............................NOT OK
additem.pl.................................NOT OK
Basically, there isn't a single place to apply this patch that will
update both item data and MARC data in one place ... a future patch
needs to address this issue.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
the unimarc stuff has been moved to marc_defs directory and the
lang specific is in lang_defs
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Enabled automatic conversion of MARC-8 records to
UTF-8. Record is converted if its Leader/09 contains
a blank and the -s (skip) option hasn't been supplied
on the command-line. Any record that cannot be converted
to UTF-8 is skipped.
Also now use Unicode Normalization Form C (NFC) for
records converted from MARC-8.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Introduced new C4::Biblio function CheckItemPreSave,
which checks for duplicate barcodes and invalid
branch codes. Not yet sure whether this function
needs to be exported or whether it will just be
used internally to C4::Bibli.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Changes to improve speed of MARC bib and item
imports:
[1] Turn off autocommit and commit database
transactions in larger batches.
[2] Introduce a new C4::Biblio function (AddBiblioAndItems)
to combine AddBiblio and AddItems -- this is faster
because we are not parsing the MARC XML of the biblio
every time we add an item.
[3] Introduce FasterTransformMarcToKoha, which is much
faster than TransformMarcToKoha. The new version,
which will replace the old one once it has been
fully tested, scans through each field in the
MARC record just once, instead of potentially
dozens of times.
[4] Remove code in bulkmarcexport that moved the
item tags to separate MARC::Record objects.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* adding zebra daemons for managing server and queue processes
* improvements to the README.debian file
* Fixes to Search.pm since last series of commits broke zebra-based
searching (again)
* moving some files to new misc/bin and misc/cronjobs
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
[1] Use File::Temp to create and manage
export directory if -d is not specified.
[2] Added usage message.
[3] Code that attempts to fix up Zebra
configuration files changed so that it
is invoked only if --munge-config option
is supplied; this code will ultimately
either be removed or moved to a separate
script -- the sorts of errors that it
tries to fix should no longer be appearing
in a standard install.
[4] Fixed Win32 portability problem when removing
temporary directory.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* rewrite-config.PL now puts in installed location
of koha-conf.xml in C4/Context.pm so that
correct config can be found even when
KOHA_CONF is not set. Note that setting KOHA_CONF
will still override path set by installer.
* changed references from koha.xml to koha-conf.xml
The correct field is 'items.wthdrawn', not 'items.withdrawn'.
Spelling may be corrected in post-3.0 version.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
There are only 2 UNIMARC specific files (.abs and .chr), they have been moved to etc/zebradb
The rebuild_zebra.pl takes all config file from this location now.
the misc/zebra/ can be removed (and will be soon)
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The authoritative data for the items is the items table, not the MARC data.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
this time it's when a biblio don't have biblionumber, has a 100$a field, and it's invalid.
1 biblio in my 300 000 DB (and it was biblio 294 359, of course !)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
the API of TransformMarcToKoha has changed.
+
For a reason I don't understand (& appear a conception bug), the biblioitemnumber is not shipped in the hash
and it's needed by the UPDATE. So it has to be reintroduced manually.
That's how it's done in Biblio.pm/Sub ModItem.
I think 3.2 will see the end of biblioitems (& it's replacement by something else)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
now, adding both on the fly when needed.
(had 2 biblios like that in a 290 000 DB, but was enought to have M::F::X complaining & diing !)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* now properly deletes deleted authorities from index
* now actually ignores duplicate work
* standardized on recordDelete instead of delete_record
for bibs and recordDelete for authorities
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
As part of this, modified two routines in C4::ImportBatch
to support a callback for monitor progress of import
processing.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
(authorities that don't have a 001 field containing authid)
also comment some code when exporting biblios (NOT tested, hdl,pls confirm this commit)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The zebraque_start scripts works, but it not friendly when you try to track what he did
as records are deleted after done.
This commit changes the behaviour : 2 columns are added done & timestamp.
done is set to 1 when a line have been done. And the timestamp contains the timestamp of the last action (either line creation if done=0 or operation if done=1)
should be helpfull to track problem.
the table will grow, but i'll add soon a DELETE FROM zebraqueue WHERE timestamp > 30 days or something like that
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
