See the script's documentation for more details
New parameters are:
- authtypes
- filter
- insert
- update
- all
Signed-off-by: Pascale Nalon <pascale.nalon@gmail.com>
This patch is live in Mines ParisTech since 2012-07-24.
Signing off
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
- Moved the sign-off from bugzilla to the commit message.
- All tests and QA script pass.
- Amended commit message to list new parameters.
- Verified this patch works on a UNIMARC installation.
- Verified normal import still works correct on a MARC21
installation.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Removed NoZebra vestiges. This comprises several code blocks that depend on the NoZebra syspref and NZ related functions/methods.
C4::Biblio->
GetNoZebraIndexes
_DelBiblioNoZebra
_AddBiblioNoZebra
C4::Search->
NZgetRecords
NZanalyse
NZoperatorAND
NZoperatorOR
NZoperatorNOT
NZorder
C4::Installer->
set_indexing_engine
Sponsored-by: Universidad Nacional de Córdoba
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
* Fix a long-standing bug in the linker that could crash the linker when
run against odd data.
* Sanitize input to SimpleSearch.
* Correctly handle CCL indexes with QP.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Remove a line of debug code from EG, provide better error handling
when presented with weird data in the authority linker, and correct
queryparser configuration to reflect the correct configuration for
Zebra.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
With the inclusion of this patch, all searches will (try) to use
QueryParser for handling queries for both the bibliographic and authority
databases if UseQueryParser is enabled. If QueryParser is unavailable,
UseQueryParser is disabled, or the search uses CCL indexes, the old
search code will be used.
To test:
1) Apply patch.
2) Run the unit test with `prove t/QueryParser.t`
3) Enable the UseQueryParser syspref.
4) Try searches that should return results in the following places:
* OPAC (simple search)
* OPAC (advanced search)
* OPAC (authorities)
* Staff client (header search)
* Staff client (advanced search)
* Staff client (cataloging search)
* Staff client (authorities)
* Staff client (importing a batch using a match point)
* Staff client (searching for an item for adding to a label)
* Staff client (acquisitions)
* Staff client (searching for a record to create a serial)
* ANYWHERE ELSE I HAVE FORGOTTEN
5) Disable the UseQueryParser syspref. Repeat at least some of the
searches you did above.
6) If all searches worked, sign off.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Elliott Davis <elliott@bywatersolions.com>
Searching still works as expected for variuos places.
QueryParser syspref seemed to be enabled by default
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
New Authority system preference created UNIMARCAuthorityField100 with default value "afrey50 ba0".
AuthoritiesMarc.pm uses the system preference instead of label "afrey50 ba0".
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described, all tests and QA script pass.
1) Check system preference is created correctly.
2) Change pref.
3) Catalog a new authority.
4) Change framework to make 100 show in frameworks or check in the
the database, that now 100 has the new defined value.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tested with Zebra, marc21, grs1.
Discovered that paging through auth search results does no longer work, but that is not related to these changes.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Tested with Zebra, marc21, dom.
All tests pass.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
When using authority records imported into Koha from elsewhere, you
can get an error like:
Can't use string ("HASH(0xbc6c{30)") as a HASH ref while "strict refs" in use at /usr/share/koha/lib/C4/AuthoritiesMarc.pm line 363.
in authorities/auth_finder.pl. This patch fixes that error.
To test:
1) You will need records imported from elsewhere.
2) Use the authority control plugin in a bib record to search for one of
those headings.
3) Observe you get a nasty error.
4) Apply patch.
5) Repeat step 2.
6) Observe the error is gone.
7) Sign off.
Signed-off-by: Magnus Enger <magnus@enger.priv.no>
Works as advertised. No warning about "defined(%hash) is deprecated"
under perl v5.10.1.
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
When doing a search with no terms for the default authority type under
Plack, you will get a segfault if it returns any records that do not
have any authority type. As far as I can tell, this cannot be reproduced
under CGI/Apache.
To test:
Confirm that you don't get an error if you click the Submit button in
the authorities sections in the staff client and OPAC without entering
a term or changing the authority type.
You may or may not be able to reproduce the problem prior to applying
the patch, but that is the only situation in which I managed to trigger
it.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
The GetAuthority shim does not check that authority retrieval was
successful, and therefore can call ->record on an undefined value.
This can be reproduced by trying to load an invalid authority record
using bulkmarcimport. There may be other ways to trigger it, but I'm
not sure what they are.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
another way to trigger this error is just forging your URL with a wrong
authid !
- Note field (3xx) are not displayed on search result page. We get
'HASH()' text.
- Parallel/Other forms (7xx) are not displayed, with language name in front of
heading, both on result and detail page.
- Note are not displayed.
- On result page, seealso form are displayed, but end with a superfluous '--'
- Style the result page, with condensed block, and space between them.
- Done both on OPAC/staff
To be applied on 3.8.x after bug 8523.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
This commit adds support for displaying authority hierarchies for all
flavours of MARC, not just UNIMARC. Display now uses the jQuery
jstree plugin, selected with the help of Owen Leonard, resulting in a
much faster experience for users.
Be aware that the jstree file uses tabs rather than 4-space indentation,
which I left as-is so as to make it easier to integrate upstream
releases in the future.
To test:
1) Enable the AuthDisplayHierarchy syspref
2) Create authority records with a hierarchy of see also fields
(in MARC21/NORMARC, you'll be using 5xx fields for this, with a
subfield $w=g for broader terms and subfield $w=h for narrower
terms)
3) View the authorities in the OPAC, noting the hierarchical view at
the top of the page.
This initial patch does not create bidirection linkages from
unidirectional links in MARC21 authorities. This means that when moving
up the authority hierarchy, lower levels will disappear. This is
intentional, as the first patch is intended merely to ensure that
AuthDisplayHierarchy functions the same for all marcflavours. A future
patch will add a cron job to generate the bidirectional linkages, once
we are sure that the hierarchy functionality for UNIMARC and
MARC21/NORMARC coexists peaceably.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Resolved conflicts in updatedatabase.pl, sysprefs.sql and in one of
the CSS files.
Test plan:
1) Run t/AuthoritiesMarc.t
New tests complete without any errors.
2) Make sure updatedatabase works correctly.
Update works nicely, new system preference is also added to syspref.sql
3) Make sure new terms are translatable.
Created new po files for de-DE and checked for new terms.
All translations appear correctly.
4) Make sure everything works with AuthDisplayHieararchy OFF
- Add authority
- Edit authority
- Delete authority
5) Test feature with AuthDisplayHieararchy ON
- Add authority
- Edit authority
- Delete authority
6) Add a couple of hierarchically linked authorities
Note: links have to be created in both directions
Example:
151 $aGermany
551 $a Baden-Württemberg $w h
151 $aBaden-Württemberg
551 $a Konstanz $w h
551 $a Germany $w g
151 $aKonstanz
551 $a Baden-Württemberg $w g
551 $a Fürstenberg $w h
551 $a Paradies $w h
151 $a Fürstenberg
551 $a Konstanz $w g
151 $a Paradies
551 $a Konstanz $w g
Tree shows up nicely above the authority record
- in staff
- in OPAC
- on the normal view tab
- on the MARC view tab
7) Checking the logs for warnings
- no Javascript errors or warnings
- no warnings or errors in log files
Expose authority import functionality to the command line import
scripts, and rename them from commit_biblios_file.pl and
stage_biblios_file.pl to commit_file.pl and stage_file.pl.
To test (note that these instructions assume you have a MARC21
installation and are using the provided sample file):
1. Find a file of authorities (a sample file with MARC21 authorities
is attached to bug 7475) and download it to your server
2. Stage the file using the following command (replace <filename> with
the name of the file you saved in step 1):
> misc/stage_file.pl --file <filename> --authorities
3. Note the batch number the script assigns to your batch
4. Commit the records using the following command (replace <batchnumber>
with the batch number you made note of in step 3):
> misc/commit_file.pl --batch-number <batchnumber>
5. Index the authorities Zebraqueue (or wait)
6. Confirm that the new authorities appear.
7. Create a matching rule with the following settings:
Code: AUTHTEST
Description: Personal name main entry
Match threshold: 999
Record type: Authority record
Search index: Heading-main
Score: 1000
Tag: 100
Subfields: a
Offset: 0
Length: 0
(note the ID of this matching rule)
8. Stage the authority file again, this time using the following
command:
> misc/stage_file.pl --file <filename> --authorities \
--match <matchingrule>
7. Revert the import with the following command:
> misc/commit_file.pl --batch-number <batchnumber> --revert
8. Index the authorities Zebraqueue (or wait)
9. Confirm that the records have been removed
10. Import an authority record with the Stage MARC/Manage staged MARC
tools in exactly the way you would for a bibliographic record,
but choose "Authority" instead of "Bibliographic" for the record
type.
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>
Testing plan delivers as it should.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
* Add the code necessary to handle authorities with matching rules and
import batches.
* Update all the scripts that use the matcher and import batch code
to use the new API.
* Add authority records to the matching rules interface in the staff
client.
http://bugs.koha-community.org/show_bug.cgi?id=2060
Signed-off-by: Elliott Davis <elliott@bywatersolutions.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 11 September 2012
The merge fails and the following errors appear in the log:
[Sun Sep 02 23:38:10 2012] [error] [client 75.149.175.233] ZOOM error 25
"Specified element set name not valid for specified database" (addinfo:
"F") from diag-set 'Bib-1', referer:
http://****/cgi-bin/koha/authorities/authorities.pl?authid=3010
[Sun Sep 02 23:38:11 2012] [error] [client 75.149.175.233] Premature end of
script headers: authorities.pl, referer:
http://****/cgi-bin/koha/authorities/authorities.pl?authid=3010
Test plan:
1) Set dontmerge to "Do"
2) Find an authority that is linked to a bib record.
3) Edit the main entry of the authority.
4) Save the authority record.
5) Notice that you got a nasty error and the bib record was not updated.
6) Apply patch.
7) Edit the main entry of the authority again (you can just set it back
to what it was to start with).
8) Save authority record.
9) Notice that there was no error, and the bib record was updated.
10) Sign off.
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Do not automatically populate $9 in bibliographic headings when the
$9 is set in the authorized heading field of the authority record.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Make see also links in both the OPAC and authority module search results
into hyperlinks and not just textual strings.
To test:
1. Do a search for an authority that will bring up a heading with a
see also reference in the staff client and the OPAC.
2. Confirm that the see also references listed in the search results
are now hyperlinks, which work.
Also quiets an unnecessary warning about an uninitialized value.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased 26 July 2012
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on master 1 August 2012
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on master 6 August 2012
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on master 11 September 2012
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Patch works nicely and is a great improvement.
Tests:
- Checked links for existing and non existing "see alsos" give correct
search results
- Verified links show up for all "see alsos" in a result list
- Verified links are properly linked with the correct names
- Checked logs don't show errors
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
This patch adds the Koha::Indexer::RecordNormalizer and
Koha::Indexer::MARC::RecordNormalizer::EmbedSeeFromHeadings packages
to enable the inclusion of alternate forms of headings in bibliographic
searches. When the new syspref IncludeSeeFromInSearches is turned on
(default is off) rebuild_zebra.pl will insert see from headings from
authority records into bibliographic records when indexing, so that a
search on an obsolete term will turn up relevant records.
To test:
1) Enable IncludeSeeFromInSearches
2) Add a heading that has an alternate form to a record (for example,
"Cooking" has the alternate form "Cookery," if you have authority
records from LC)
3) Index the zebraqueue (or reindex if you haven't indexed your system
yet)
4) Confirm that if you search for "Cookery" you get the record you
just modified
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on master 5 August 2012
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on master 11 September 2012
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Also checked:
- Verified database update works correctly
- Checked system preference and its description
- Checked staff/opac detail pages with feature on/off
- Checked staff/opac search facets
- Downloaded and tested records in various formats
- Tried different searches for 'see from' entries of authorities
- Ran all unit tests
No problems found.
One of the ideas behind authority records is that users who search for
one term should have related terms (according to the authority file)
suggested to them. At the moment, Koha doesn't do that. Adding an
authority searching step to regular searches and displaying any
suggestions in a "Did you mean" bar at the top of the results would be
very useful.
This commit adds a Koha::SuggestionEngine class which is in charge of
getting suggestions from individual suggestion engine plugins, which
much be in the Koha::SuggestionEngine::Plugin::* namespace, and extend
Koha::SuggestionEngine::Base. Suggestions are loaded asynchronously
using AJAX, and a link to a page with suggestions is provided for users
with Javascript turned off.
The AuthorityFile suggestion engine plugin looks up the specified search
terms in the authority file and then suggests searches for records using
matching authorities.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased 2 August 2012 and incorporated QA feedback
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Test plan:
- Verified database update added system preference correctly,
pref defaults to OFF
- Verified search results and detail pages in OPAC and staff
still worked the same as before
* for no results
* with results
- Activated system preference and tested various searches
* Searches from simple search
* Searches from advanced search
* Search links in records
- Deactivated Javascript - verified fallback works correctly
Notes:
- Suggested terms can include autorities with no linked records.
- When combining more than one search option using advanced search
this results in "no suggestions" more often. Feature works best
from simple search.
Overall great feature making use of authorities in a user friendly way!
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master 2012-09-10
Signed-off-by: wajasu <matted-34813@mypacks.net>
Adds a dropdown to allow the user to choose to search the 'mainentry,'
and 'match-heading'indexes, in addition to the 'any' index which is
the default.
To test (you will need authority records with see-from and
see-also headings):
1. Go to the browse subjects and authors section in the OPAC
2. Do a search for a fairly generic term that is used as a
see-also term in a lot of records. Note that your search
results include both the record that you wanted and all
the other records that refer to it.
3. Apply the patch.
4. Repeat the search from step 2. Notice there is no change
to the results.
5. Repeat the search from step 2, but choose "in main entry"
and observe that you are now only seeing authority records
with the search entered in their main entry.
6. Repeat the search from step 2, but choose "in any heading"
and observe that the results are the same as in step 2. This
is intentional, so that in the future notes in authorities
can be made searchable.
7. Search for an obsolete form of heading with the "in keyword"
option selected. Notice what results you get.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works nicely. Hiding the new options can be done with CSS:
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Also removes unnecessary variable declaration. Thanks to Jonathan Druart
for spotting it.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
UNIMARC display of authorities in the OPAC wasn't working due to
changes in the BuildSummary function since the code was originally
developed.
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Up until now, the only authority view in the OPAC was a
rather-unfriendly expanded MARC view. This patch adds a user-friendly
view similar to the biblio details view.
Specific features to be aware of:
* Right-to-left text in the MARC21 880 field will show up in the
appropriate location with the appropriate alignment and wrapping
* There is very little CSS styling. Any suggestions for how to make
the display more attractive would be gratefully received.
To test:
1) Do a search for an authority in the OPAC.
2) Choose an authority record to view.
3) Observe that the view is more user-friendly and polished.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works very nicely now. The old view is still available, but the
user is presented with a nicer non-MARC view first.
Although auth_types.summary contains a string describing the type of
authority in MARC21, it does not in UNIMARC. auth_types.authtypetext
contains the proper label in both marcflavours.
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Due to a peculiarity in the way SearchAuthorities worked, the authority
search results on the OPAC always listed the authority type that the
user searched for, rather than the authority type of the results.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works nicely, result list for OPAC search now shows the authority type
for each record.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased 20 June 2012.
Signed-off-by: Mirko Tietgen <5p4m@gmx.de>
Works as expected.
The HTML for authority search results was previously generated in
C4::AuthoritiesMarc::BuildSummary, which meant that it couldn't be
translated. This patch moves the HTML generation into the templates
by introducing a new authorities-search-results.inc include file for
both the OPAC and the Intranet which contains a Template::Toolkit BLOCK
for rendering the authority results. Fixes the authority autocomplete
by removing the untranslatable strings, and returning only data from
the database.
To test:
1. Apply patch.
2. Test authority searching in the authority module in the staff client
3. Test authority searching in the authority control plugin in the
cataloguing module (and the plugin for UNIMARC field 210$c, if you
can figure out how)
4. Test authority searching in the OPAC
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Tested with MARC21 data.
1) Applies cleanly on current HEAD.
2) Authority search in staff
Patch works wonderfully, only some small notes found while testing that have
not been changed by this patch:
ENH note: Search terms show up nowhere. So if I want to change the sorting, I have to
repeat typing in my search term. Even if the form does not keep the term, it
should be visible somewhere on the screen what I searched for.
ENH note: The pagination on top and at the bottom of the result list are formatted
differently. Maybe some missing CSS?
ENH note: Also the authority type is not shown at all in the result list.
3) Cataloguing and authority plugins
The autocomplete function works nicely.
ENH note: There is only one small enhancement
I could imagine. If I start my search from 100 it will limit the search to
'persons' but the autocomplete will also suggest other authorities. It would
be a bit cleaner, if the autocomplete could limit by the appropriate authority
type too. Very nice feature.
Plugins overall work nicely. Created links include the authority numbers and work
correctly.
4) Authority search in OPAC
Works nicely. Display is consistent, but translatability greatly improved.
ENH note: In staff we use 'Details' in OPAC we use 'View full heading' - I wonder
if maybe 'details' would be better understandable for users?
Note: Code reveals a system preference 'AuthDisplayHierarchy' that is
not available in the system preference editor. I talked to Jared and he
will work on this feature later on. For now it's no regression, as the
preference has never been visible.
Add the option of sorting authority search results by authid, and instruct the
FirstMatch and LastMatch linkers to use that sort order rather than the default
search order.
To test:
1. Install new Zebra authorities config
etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml,
etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl,
etc/zebradb/marc_defs/marc21/authorities/record.abs, and
etc/zebradb/marc_defs/unimarc/authorities/record.abs
2. Reindex authorities in Zebra
3. Set LinkerModule to FirstMatch or LastMatch
4. Add two identical authority records, and a bib record with a heading that
matches them
5. Run misc/link_bibs_to_authorities.pl on that record
6. Confirm that the authid that's been inserted into subfield $9 of that
heading is the first, if you selected FirstMatch, or last if you selected
LastMatch
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
I followed the test plan and checked that for "Last match" and "First match"
the correct authority was selected and linked to the record.
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Squashed patch incorporating all previous patches (there is no functional
change compared to the previous version of this patch, this patch merely
squashes the original patch and follow-up, and rebases on latest master).
=== TL;DR VERSION ===
*** Installation ***
1. Run installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt1
and installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt2
2. Make sure you copy the following files from kohaclone to koha-dev:
etc/zeradb/authorities/etc/bib1.att,
etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml,
etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl,
etc/zebradb/marc_defs/marc21/authorities/koha-indexdefs-to-zebra.xsl, and
etc/zebradb/marc_defs/unimarc/authorities/record.abs
3. Run misc/migration_tools/rebuild_zebra.pl -a -r
*** New sysprefs ***
* AutoCreateAuthorities
* CatalogModuleRelink
* LinkerModule
* LinkerOptions
* LinkerRelink
* LinkerKeepStale
*** Important notes ***
You must have rebuild_zebra processing the zebraqueue for bibs when testing this
patch.
=== DESCRIPTION ===
*** Cataloging module ***
* Added an additional box to the authority finder plugin for "Heading match,"
which consults not just the main entry but also See-from and See-also-from
headings.
* With this patch, the automatic authority linking will actually work properly
in the cataloging module. As Owen pointed out while testing the patch,
though, longtime users of Koha will not be expecting that. In keeping with
the principles of least surprise and maximum configurability, a new syspref,
CatalogModuleRelink makes it possible to disable authority relinking in the
cataloging module only (i.e. leaving it enabled for future runs of
link_bibs_to_authorities.pl). Note that though the default behavior matches
the current behavior of Koha, it does not match the intended behavior.
Libraries that want the intended behavior rather than the current behavior
will need to adjust the CatalogModuleRelink syspref.
*** misc/link_bibs_to_authorities.pl ***
Added the following options to the misc/link_bibs_to_authorities.pl script:
--auth-limit Only process those headings that match the authorities
matching the user-specified WHERE clause.
--bib-limit Only process those bib records that match the
user-specified WHERE clause.
--commit Commit the results to the database after every N records
are processed.
--link-report Display a report of all the headings that were processed.
Converted misc/link_bibs_to_authorities.pl to use POD.
Added a detailed report of headings that linked, did not link, and linked
in a "fuzzy" fashion (the exact semantics of fuzzy are up to the individual
linker modules) during the run.
*** C4::Linker ***
Implemented new C4::Linker functionality to make it possible to easily add
custom authority linker algorithms. Currently available linker options are:
* Default: retains the current behavior of only creating links when there is
an exact match to one and only one authority record; if the 'broader_headings'
option is enabled, it will try to link to headings to authority records for
broader headings by removing subfields from the end of the heading (NOTE:
test the results before enabling broader_headings in a production system
because its usefulness is very much dependent on individual sites' authority
files)
* First Match: based on Default, creates a link to the *first* authority
record that matches a given heading, even if there is more than one
authority record that matches
* Last Match: based on Default, creates a link to the *last* authority
record that matches a given heading, even if there is more than one record
that matches
The API for linker modules is very simple. All modules should implement the
following two functions:
<get_link ($field)> - return the authid for the authority that should be
linked to the provided MARC::Field object, and a boolean to indicate whether
the match is "fuzzy" (the semantics of "fuzzy" are up to the individual plugin).
In order to handle authority limits, get_link should always end with:
return $self->SUPER::_handle_auth_limit($authid), $fuzzy;
<flip_heading ($field)> - return a MARC::Field object with the heading flipped
to the preferred form. At present this routine is not used, and can be a stub.
Made the linking functionality use the SearchAuthorities in C4::AuthoritiesMarc
rather than SimpleSearch in C4::Search. Once C4::Search has been refactored,
SearchAuthorities should be rewritten to simply call into C4::Search. However,
at this time C4::Search cannot handle authority searching. Also fixed numerous
performance issues in SearchAuthorities and the Linker script:
* Correctly destroy ZOOM recordsets in SearchAuthorities when finished. If left
undestroyed, efficiency appears to approach O(log n^n)
* Add an optional $skipmetadata flag to SearchAuthorities that can be used to
avoid additional calls into Zebra when all that is wanted are authority
records and not statistics about their use
*** New sysprefs ***
* AutoCreateAuthorities - When this and BiblioAddsAuthorities are both turned
on, automatically create authority records for headings that don't have
any authority link when cataloging. When BiblioAddsAuthorities is on and
AutoCreateAuthorities is turned off, do not automatically generate authority
records, but allow the user to enter headings that don't match an existing
authority. When BiblioAddsAuthorities is off, this has no effect.
* CatalogModuleRelink - when turned on, the automatic linker will relink
headings when a record is saved in the cataloging module when LinkerRelink
is turned on, even if the headings were manually linked to a different
authority by the cataloger. When turned off (the default), the automatic
linker will not relink any headings that have already been linked when a
record is saved.
* LinkerModule - Chooses which linker module to use for matching headings
(current options are as described above in the section on linker options:
"Default," "FirstMatch," and "LastMatch")
* LinkerOptions - A pipe-separated list of options to set for the authority
linker (at the moment, the only option available is "broader_headings," which
is described below)
* LinkerRelink - When turned on, the linker will confirm the links for headings
that have previously been linked to an authority record when it runs. When
turned off, any heading with an existing link will be ignored.
* LinkerKeepStale - When turned on, the linker will never *delete* a link to an
authority record, though, depending on the value of LinkerRelink, it may
change the link.
*** Other changes ***
* Cleaned up authorities code by removing unused functions and adding
unimplemented functions and added some unit tests.
* This patch also modifies the authority indexing to remove trailing punctuation
from Match indexes.
* Replace the old BiblioAddAuthorities subroutines with calls into the new
C4::Linker routines.
* Add a simple implementation for C4::Heading::UNIMARC. (With thanks to F.
Demians, 2011.01.09) Correct C4::Heading::UNIMARC class loading. Create
biblio tag to authority types data structure at initialization rather than
querying DB.
* Ran perltidy on all changed code.
*** Linker Options ***
Enter "broader_headings" in LinkerOptions. With this option, the linker will
try to match the following heading as follows:
=600 10$aCamins-Esakov, Jared$xCoin collections$vCatalogs$vEarly works to
1800.
First: Camins-Esakov, Jared--Coin collections--Catalogs--Early works to 1800
Next: Camins-Esakov, Jared--Coin collections--Catalogs
Next: Camins-Esakov, Jared--Coin collections
Next: Camins-Esakov, Jared (matches! if a previous attempt had matched, it
would not have tried this)
This is probably relevant only to MARC21 and LCSH, but could potentially be of
great use to libraries that make heavy use of floating subdivisions.
=== TESTING PLAN ===
Note: all of these tests require that you have some authority records,
preferably for headings that actually appear in your bibliographic data. At
least one authority record must contain a "see from" reference (remember which
one contains this, as you'll need it for some of the tests). The number shown
in the "Used in" column in the authority module is populated using Zebra
searches of the bibliographic database, so you *must* have
rebuild_zebra.pl -b -z [-x] running in cron, or manually run it after running
the linker.
*** Testing the Heading match in the cataloging plugin ***
1. Create a new record, and open the cataloging plugin for an
authority-controlled field.
2. Search for an authority by entering the "see from" term in the Heading Match
box
3. Confirm that the appropriate heading shows up
4. Search for an authority by entering the preferred heading into the Main
entry or Main entry ($a only) box (i.e., repeat the procedure you usually
use for cataloging, whatever that may be)
5. Confirm that the appropriate heading shows up
*** Testing the cataloging interface ***
6. Turn off BiblioAddsAuthorities
7. Confirm that you cannot enter text directly in an authority-controlled field
8. Confirm that if you search for a heading using the authority control plugin
the heading is inserted (note, however, that this patch does not AND IS NOT
INTENDED TO fix the bugs in the authority plugin with duplicate subfields;
those are wholly out of scope- this check is for regressions)
9. Turn on BiblioAddsAuthorities and AutoCreateAuthorities
10. Confirm that you can enter text directly into an authority-controlled field,
and if you enter a heading that doesn't currently have an authority record,
an authority record stub is automatically created, and the heading you
entered linked
11. Confirm that if you enter a heading with only a subfield $a that fully
*matches* an existing heading (i.e. the existing heading has only
subfield $a populated), the authid for that heading is inserted into
subfield $9
12. Confirm that if you enter a heading with multiple subfields that *matches*
an existing heading, the authid for that heading is inserted into
subfield $9
13. Turn on BiblioAddsAuthorities and turn off AutoCreateAuthorities
14. Confirm that you can enter text directly into an authority-controlled field,
and if you enter a heading that doesn't currently have an authority record,
an authority record stub is *not* created
15. Confirm that if you enter a heading with only a subfield $a that *matches*
an existing heading, the authid for that heading is inserted into
subfield $9
16. Confirm that if you enter a heading with multiple subfields that *matches*
an existing heading, the authid for that heading is inserted into
subfield $9
17. Create a record and link an authority record to an authorized field using
the authority plugin.
18. Save the record. Ensure that the heading is linked to the appropriate
authority.
19. Open the record. Change the heading manually to something else, leaving
the link. Save the record.
20. Ensure that the heading remains linked to that same authority.
21. Change CatalogModuleRelink to "on."
22. Open the record. Use the authority plugin to link that heading to the
same authority record you did earlier.
23. Save the record. Ensure that the heading is linked to the appropriate
authority.
24. Open the record. Change the heading manually to something else, leaving
the link. Save the record.
25. Ensure that the heading is no longer linked to the old authority record.
*** Testing link_bibs_to_authorities.pl ***
26. Set LinkerModule to "Default," turn on LinkerRelink and
BiblioAddsAuthorities, and turn AutoCreateAuthorities and
LinkerKeepStale off
27. Edit one bib record so that an authority controlled field that has already
been linked (i.e. has data in $9) has a heading that does not match any
authority record in your database
28. Run misc/link_bibs_to_authorities.pl --link-report --verbose --test (you may
want to pipe the output into less or a file, as the result is quite a lot of
information)
29. Look over the report to see if the headings that you have authority records
for report being matched, that the heading you modified in step 2 is
reported as "unlinked," and confirm that no changes were actually made to
the database (to check this, look at the bib record you edited earlier, and
check that the authid in the field you edited hasn't changed)
30. Run misc/link_bibs_to_authorities.pl --link-report --verbose (you may want
to pipe the output into less or a file, as the result is quite a lot of
information)
31. Check that the heading you modified has been unlinked
32. Change the modified heading back to whatever it was, but don't use the
authority control plugin to populate $9
33. Run misc/link_bibs_to_authorities.pl --link-report --verbose
--bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
of the record you've been editing)
34. Confirm that the heading has been linked to the correct authority record
35. Turn LinkerKeepStale on
36. Change that heading to something else
37. Run misc/link_bibs_to_authorities.pl --link-report --verbose
--bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
of the record you've been editing)
38. Confirm that the $9 has not changed
39. Turn LinkerKeepStale off
40. Create two authorities with the same heading
41. Run misc/migration_tools/rebuild_zebra.pl -a -z
42. Enter that heading into the bibliographic record you are working with
43. Run misc/link_bibs_to_authorities.pl --link-report --verbose
--bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
of the record you've been editing)
44. Confirm that the heading has not been linked
45. Change LinkerModule to "FirstMatch"
46. Run misc/link_bibs_to_authorities.pl --link-report --verbose
--bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
of the record you've been editing)
47. Confirm that the heading has been linked to the first authority record it
matches
48. Change LinkerModule to "LastMatch"
49. Run misc/link_bibs_to_authorities.pl --link-report --verbose
--bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
of the record you've been editing)
50. Confirm that the heading has been linked to the second authority record it
matches
51. Run misc/link_bibs_to_authorities.pl --link-report --verbose
--auth-limit="authid=${AUTH}" (replacing ${AUTH} with an authid)
52. Confirm that only that heading is displayed in the report, and only those
bibs with that heading have been changed
If all those things worked, good news! You're ready to sign off on the patch
for bug 7284.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master and squashed follow-up, 16 February 2012
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master, 21 February 2012
Signed-off-by: schuster <dschust1@gmail.com>
The old default value for field 008 didn't have the correct length,
which could potentially cause data corruptions.
Uses the MARCAuthorityControlField008 preference added by BUG4839.
http://bugs.koha-community.org/show_bug.cgi?id=7072
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Pref MergeAuthoritiesOnUpdate does not exist; should be dontmerge
(AuthoritiesMarc.pm).
Instead of folder modified_authorities, now introducing a table for this
purpose: need_merge_authorities. This eliminates several permissions and
security issues. This change applies to AuthoritiesMarc.pm and
merge_authority.pl.
POD lines added for ModAuthority. Deprecated parameter $merge removed.
Test this patch by applying the db revision first from the second patch.
August 4, 2011: Rebased.
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Thanks Marcel. It works as advertised. Both modes are functionnal
(back):
- Immediate with dontmerge=0: After modifying an authority record, its
linked biblios are immediately modified. This isn't the case in 3.4.5.
- Delayed with dontmerge=1: After modifying an authority record, its
linked biblios are not modified. But an entry is added to
need_merge_authority new table and 'merge_authorities.pl -b' script
updates biblios.
Comment: need_merge_authority, like zebraqueue, should be cleared
from time to time.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
This removes the subfield codes shown for topical subject authorities only.
To test:
- Add different authority records and at least one for a topical subject.
- Use search in intranet and opac (turn on opacauthorities) to serch for your subjects.
- All headings will show without the subfield codes (at least $a in front), except the
topical subject.
This patch removes the subfield codes for the topical headings.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Works as advertised in opac and staff. (July 18, 2011)
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Remove some unnecessary checks when check of error is
sufficient. Make the order in some cases more logical
Should remove some possibilities of runtime warning noise.
Although some calls belong to the 'Nothing could
ever go wrong' school have added some warnings
Signed-off-by: Christophe Croullebois <christophe.croullebois@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Includes Heading-Main in authorities bib1 attribute set.
Also includes Any in authorities bib1 (was still missing).
Also removes temporary fix in AuthoritiesMarc.pm.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Rebased after pushed fix for #5924.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Fixing 100$a simple error: Sometimes, language is encoded fre50 and not
frey50.
This patch adds check for that problem
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Variable $summary contained <a> and <b> tag. They have been deleted.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Heading Main is defined in unimarc record.abs
re enabling that feature
Signed-off-by: fdurand <frederic.durand@univ-lyon2.fr>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
commit 58e18dfa4109fcdde3c71675ff07ed6aae86b258
Author: Henri-Damien LAURENT <henridamien.laurent@biblibre.com>
Date: Tue May 18 16:54:49 2010 +0200
Bug 5914 : followup BuildUnimarchierarchies fixing infinite loop
commit bbbcb7fcb0ff36eb8a072d9b0442996e71ebb247
Author: Henri-Damien LAURENT <henridamien.laurent@biblibre.com>
Date: Tue May 18 13:11:44 2010 +0200
Bug 5914 : BuildUnimarcHierarchies : Fixing infinite loop
When referenced authority in 5XX$3 would be the same as the current authid, the script would get into infinite loop
Conflicts solved: C4/AuthoritiesMarc.pm
commit 3581021997bc8965af807738c47d5fd27f2fd97f
Author: Henri-Damien LAURENT <henridamien.laurent@biblibre.com>
Date: Wed Apr 7 15:15:59 2010 +0200
Bug 5914 : BuildUnimarcHierarchies : Fixing infinite loop
When referenced authority in 5XX$3 would be the same as the current authid, the script would get into infinite loop
Signed-off-by: Stéphane Delaune <stephane.delaune@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Before this fix, any search with double-quotes would return an error
500.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
