Koha-community/Koha - Koha - Koha: The world's first free and open source library system

Commit Graph

Author	SHA1	Message	Date
Nick Clemens	d4783f244b	Bug 24269: Adjust C4::Heading to generate headings from auth tags To test: 1 - Be using Elasticsearch 2 - Check that field 150 is mapped to 'Match-heading' or add it (the subfields don't matter) 3 - Add a topic term authority record like "150 $aCats$vFiction" 4 - Add a 650 with $aCats and $vFiction and $e depicted to a bibliographic record 5 - Run the linker for the bib perl misc/link_bibs_to_authorities.pl --bib-limit "biblionumber=89" -v 6 - Confirm the record is not correctly linked to the record 7 - Apply patch 8 - Reindex authorities for ES perl misc/search_tools/rebuild_elasticsearch.pl -v -d -a 9 - Run the linker and confirm record is correctly linked perl misc/link_bibs_to_authorities.pl --bib-limit "biblionumber=89" -v Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>	4 years ago
Ere Maijala	7cb88d24a8	Bug 21826: Remove strict and warning from Heading.pm Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>	6 years ago
Ere Maijala	b2b7d376d4	Bug 21826: Use subfield map in LinkBibHeadingsToAuthorities Use only allowable subfields when creating authorities from bibliographic records. Sponsored-by: National Library of Finland Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>	6 years ago
Martin Renvoize	92c7c6339b	Bug 19365: (QA follow-up) Ensure we use a modern enough perl Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>	6 years ago
Ere Maijala	be2bdfd511	Bug 19365: Fix several issues with the Elasticsearch code Also optimize it so it's actually usable. Test plan: 1. To test it properly you need biblio and authority data. You might get away with enabling AutoCreateAuthorities and BiblioAddsAuthorities so that authorities are created in the process. Another option would be to import authorities first e.g. from LoC. 2. Make sure the authority index has been properly created with "misc/search_tools/rebuild_elastic_search.pl -a -d" 3. Run "misc/link_bibs_to_authorities.pl -v -l" twice and observe the results. Sponsored-by: National Library of Finland Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>	6 years ago
Jonathan Druart	017699c345	Bug 16011: $VERSION - Remove the $VERSION init Mainly a perl -p -i -e 's/^.3.07.00.049.\n//' */.pm Then some adjustements Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar> Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>	8 years ago
Jonathan Druart	a6c9bd0eb5	Bug 9978: Replace license header with the correct license (GPLv3+) Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de> http://bugs.koha-community.org/show_bug.cgi?id=9987 Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>	9 years ago
Chris Cormack	509d673f10	Bug 7941 : Fix version numbers in modules Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>	12 years ago
Jared Camins-Esakov	937480abe0	Bug 7617: Sort authority results by authid Add the option of sorting authority search results by authid, and instruct the FirstMatch and LastMatch linkers to use that sort order rather than the default search order. To test: 1. Install new Zebra authorities config etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml, etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl, etc/zebradb/marc_defs/marc21/authorities/record.abs, and etc/zebradb/marc_defs/unimarc/authorities/record.abs 2. Reindex authorities in Zebra 3. Set LinkerModule to FirstMatch or LastMatch 4. Add two identical authority records, and a bib record with a heading that matches them 5. Run misc/link_bibs_to_authorities.pl on that record 6. Confirm that the authid that's been inserted into subfield $9 of that heading is the first, if you selected FirstMatch, or last if you selected LastMatch Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> I followed the test plan and checked that for "Last match" and "First match" the correct authority was selected and linked to the record. Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>	12 years ago
Jared Camins-Esakov	5207699f98	signed off Bug 7284: Authority matching improvements Squashed patch incorporating all previous patches (there is no functional change compared to the previous version of this patch, this patch merely squashes the original patch and follow-up, and rebases on latest master). === TL;DR VERSION === * Installation * 1. Run installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt1 and installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt2 2. Make sure you copy the following files from kohaclone to koha-dev: etc/zeradb/authorities/etc/bib1.att, etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml, etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl, etc/zebradb/marc_defs/marc21/authorities/koha-indexdefs-to-zebra.xsl, and etc/zebradb/marc_defs/unimarc/authorities/record.abs 3. Run misc/migration_tools/rebuild_zebra.pl -a -r * New sysprefs * * AutoCreateAuthorities * CatalogModuleRelink * LinkerModule * LinkerOptions * LinkerRelink * LinkerKeepStale * Important notes * You must have rebuild_zebra processing the zebraqueue for bibs when testing this patch. === DESCRIPTION === * Cataloging module * * Added an additional box to the authority finder plugin for "Heading match," which consults not just the main entry but also See-from and See-also-from headings. * With this patch, the automatic authority linking will actually work properly in the cataloging module. As Owen pointed out while testing the patch, though, longtime users of Koha will not be expecting that. In keeping with the principles of least surprise and maximum configurability, a new syspref, CatalogModuleRelink makes it possible to disable authority relinking in the cataloging module only (i.e. leaving it enabled for future runs of link_bibs_to_authorities.pl). Note that though the default behavior matches the current behavior of Koha, it does not match the intended behavior. Libraries that want the intended behavior rather than the current behavior will need to adjust the CatalogModuleRelink syspref. * misc/link_bibs_to_authorities.pl * Added the following options to the misc/link_bibs_to_authorities.pl script: --auth-limit Only process those headings that match the authorities matching the user-specified WHERE clause. --bib-limit Only process those bib records that match the user-specified WHERE clause. --commit Commit the results to the database after every N records are processed. --link-report Display a report of all the headings that were processed. Converted misc/link_bibs_to_authorities.pl to use POD. Added a detailed report of headings that linked, did not link, and linked in a "fuzzy" fashion (the exact semantics of fuzzy are up to the individual linker modules) during the run. * C4::Linker * Implemented new C4::Linker functionality to make it possible to easily add custom authority linker algorithms. Currently available linker options are: * Default: retains the current behavior of only creating links when there is an exact match to one and only one authority record; if the 'broader_headings' option is enabled, it will try to link to headings to authority records for broader headings by removing subfields from the end of the heading (NOTE: test the results before enabling broader_headings in a production system because its usefulness is very much dependent on individual sites' authority files) * First Match: based on Default, creates a link to the first authority record that matches a given heading, even if there is more than one authority record that matches * Last Match: based on Default, creates a link to the last authority record that matches a given heading, even if there is more than one record that matches The API for linker modules is very simple. All modules should implement the following two functions: <get_link ($field)> - return the authid for the authority that should be linked to the provided MARC::Field object, and a boolean to indicate whether the match is "fuzzy" (the semantics of "fuzzy" are up to the individual plugin). In order to handle authority limits, get_link should always end with: return $self->SUPER::_handle_auth_limit($authid), $fuzzy; <flip_heading ($field)> - return a MARC::Field object with the heading flipped to the preferred form. At present this routine is not used, and can be a stub. Made the linking functionality use the SearchAuthorities in C4::AuthoritiesMarc rather than SimpleSearch in C4::Search. Once C4::Search has been refactored, SearchAuthorities should be rewritten to simply call into C4::Search. However, at this time C4::Search cannot handle authority searching. Also fixed numerous performance issues in SearchAuthorities and the Linker script: * Correctly destroy ZOOM recordsets in SearchAuthorities when finished. If left undestroyed, efficiency appears to approach O(log n^n) * Add an optional $skipmetadata flag to SearchAuthorities that can be used to avoid additional calls into Zebra when all that is wanted are authority records and not statistics about their use * New sysprefs * * AutoCreateAuthorities - When this and BiblioAddsAuthorities are both turned on, automatically create authority records for headings that don't have any authority link when cataloging. When BiblioAddsAuthorities is on and AutoCreateAuthorities is turned off, do not automatically generate authority records, but allow the user to enter headings that don't match an existing authority. When BiblioAddsAuthorities is off, this has no effect. * CatalogModuleRelink - when turned on, the automatic linker will relink headings when a record is saved in the cataloging module when LinkerRelink is turned on, even if the headings were manually linked to a different authority by the cataloger. When turned off (the default), the automatic linker will not relink any headings that have already been linked when a record is saved. * LinkerModule - Chooses which linker module to use for matching headings (current options are as described above in the section on linker options: "Default," "FirstMatch," and "LastMatch") * LinkerOptions - A pipe-separated list of options to set for the authority linker (at the moment, the only option available is "broader_headings," which is described below) * LinkerRelink - When turned on, the linker will confirm the links for headings that have previously been linked to an authority record when it runs. When turned off, any heading with an existing link will be ignored. * LinkerKeepStale - When turned on, the linker will never delete a link to an authority record, though, depending on the value of LinkerRelink, it may change the link. * Other changes * * Cleaned up authorities code by removing unused functions and adding unimplemented functions and added some unit tests. * This patch also modifies the authority indexing to remove trailing punctuation from Match indexes. * Replace the old BiblioAddAuthorities subroutines with calls into the new C4::Linker routines. * Add a simple implementation for C4::Heading::UNIMARC. (With thanks to F. Demians, 2011.01.09) Correct C4::Heading::UNIMARC class loading. Create biblio tag to authority types data structure at initialization rather than querying DB. * Ran perltidy on all changed code. * Linker Options * Enter "broader_headings" in LinkerOptions. With this option, the linker will try to match the following heading as follows: =600 10$aCamins-Esakov, Jared$xCoin collections$vCatalogs$vEarly works to 1800. First: Camins-Esakov, Jared--Coin collections--Catalogs--Early works to 1800 Next: Camins-Esakov, Jared--Coin collections--Catalogs Next: Camins-Esakov, Jared--Coin collections Next: Camins-Esakov, Jared (matches! if a previous attempt had matched, it would not have tried this) This is probably relevant only to MARC21 and LCSH, but could potentially be of great use to libraries that make heavy use of floating subdivisions. === TESTING PLAN === Note: all of these tests require that you have some authority records, preferably for headings that actually appear in your bibliographic data. At least one authority record must contain a "see from" reference (remember which one contains this, as you'll need it for some of the tests). The number shown in the "Used in" column in the authority module is populated using Zebra searches of the bibliographic database, so you must have rebuild_zebra.pl -b -z [-x] running in cron, or manually run it after running the linker. * Testing the Heading match in the cataloging plugin * 1. Create a new record, and open the cataloging plugin for an authority-controlled field. 2. Search for an authority by entering the "see from" term in the Heading Match box 3. Confirm that the appropriate heading shows up 4. Search for an authority by entering the preferred heading into the Main entry or Main entry ($a only) box (i.e., repeat the procedure you usually use for cataloging, whatever that may be) 5. Confirm that the appropriate heading shows up * Testing the cataloging interface * 6. Turn off BiblioAddsAuthorities 7. Confirm that you cannot enter text directly in an authority-controlled field 8. Confirm that if you search for a heading using the authority control plugin the heading is inserted (note, however, that this patch does not AND IS NOT INTENDED TO fix the bugs in the authority plugin with duplicate subfields; those are wholly out of scope- this check is for regressions) 9. Turn on BiblioAddsAuthorities and AutoCreateAuthorities 10. Confirm that you can enter text directly into an authority-controlled field, and if you enter a heading that doesn't currently have an authority record, an authority record stub is automatically created, and the heading you entered linked 11. Confirm that if you enter a heading with only a subfield $a that fully matches an existing heading (i.e. the existing heading has only subfield $a populated), the authid for that heading is inserted into subfield $9 12. Confirm that if you enter a heading with multiple subfields that matches an existing heading, the authid for that heading is inserted into subfield $9 13. Turn on BiblioAddsAuthorities and turn off AutoCreateAuthorities 14. Confirm that you can enter text directly into an authority-controlled field, and if you enter a heading that doesn't currently have an authority record, an authority record stub is not created 15. Confirm that if you enter a heading with only a subfield $a that matches an existing heading, the authid for that heading is inserted into subfield $9 16. Confirm that if you enter a heading with multiple subfields that matches an existing heading, the authid for that heading is inserted into subfield $9 17. Create a record and link an authority record to an authorized field using the authority plugin. 18. Save the record. Ensure that the heading is linked to the appropriate authority. 19. Open the record. Change the heading manually to something else, leaving the link. Save the record. 20. Ensure that the heading remains linked to that same authority. 21. Change CatalogModuleRelink to "on." 22. Open the record. Use the authority plugin to link that heading to the same authority record you did earlier. 23. Save the record. Ensure that the heading is linked to the appropriate authority. 24. Open the record. Change the heading manually to something else, leaving the link. Save the record. 25. Ensure that the heading is no longer linked to the old authority record. * Testing link_bibs_to_authorities.pl * 26. Set LinkerModule to "Default," turn on LinkerRelink and BiblioAddsAuthorities, and turn AutoCreateAuthorities and LinkerKeepStale off 27. Edit one bib record so that an authority controlled field that has already been linked (i.e. has data in $9) has a heading that does not match any authority record in your database 28. Run misc/link_bibs_to_authorities.pl --link-report --verbose --test (you may want to pipe the output into less or a file, as the result is quite a lot of information) 29. Look over the report to see if the headings that you have authority records for report being matched, that the heading you modified in step 2 is reported as "unlinked," and confirm that no changes were actually made to the database (to check this, look at the bib record you edited earlier, and check that the authid in the field you edited hasn't changed) 30. Run misc/link_bibs_to_authorities.pl --link-report --verbose (you may want to pipe the output into less or a file, as the result is quite a lot of information) 31. Check that the heading you modified has been unlinked 32. Change the modified heading back to whatever it was, but don't use the authority control plugin to populate $9 33. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing) 34. Confirm that the heading has been linked to the correct authority record 35. Turn LinkerKeepStale on 36. Change that heading to something else 37. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing) 38. Confirm that the $9 has not changed 39. Turn LinkerKeepStale off 40. Create two authorities with the same heading 41. Run misc/migration_tools/rebuild_zebra.pl -a -z 42. Enter that heading into the bibliographic record you are working with 43. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing) 44. Confirm that the heading has not been linked 45. Change LinkerModule to "FirstMatch" 46. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing) 47. Confirm that the heading has been linked to the first authority record it matches 48. Change LinkerModule to "LastMatch" 49. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing) 50. Confirm that the heading has been linked to the second authority record it matches 51. Run misc/link_bibs_to_authorities.pl --link-report --verbose --auth-limit="authid=${AUTH}" (replacing ${AUTH} with an authid) 52. Confirm that only that heading is displayed in the report, and only those bibs with that heading have been changed If all those things worked, good news! You're ready to sign off on the patch for bug 7284. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> Rebased on latest master and squashed follow-up, 16 February 2012 Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> Rebased on latest master, 21 February 2012 Signed-off-by: schuster <dschust1@gmail.com>	13 years ago
Paul Poulain	58c46b27af	Bug 6875 de-nesting C4::Heading C4::Search is needed only in authorities sub, moving it here and switching to require Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> Corrected call to SimpleSearch in both subs where it is used. Additional performance improvements are included in bug 7284. 28 Jan 2012	13 years ago
Colin Campbell	d8b362e0f9	Bug 5415 Let calls of SimpleSearch utilize considtent interface Remove some unnecessary checks when check of error is sufficient. Make the order in some cases more logical Should remove some possibilities of runtime warning noise. Although some calls belong to the 'Nothing could ever go wrong' school have added some warnings Signed-off-by: Christophe Croullebois <christophe.croullebois@biblibre.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>	13 years ago
Frédérick Capovilla	0f4edffc16	Bug 5410 : Adds "do-not-truncate" in the search queries of Heading.pm Signed-off-by: Stéphane Delaune <stephane.delaune@biblibre.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>	13 years ago
Galen Charlton	7c0e441d50	replace references to defunct info email address Now links to Koha project website. Signed-off-by: Galen Charlton <gmcharlt@gmail.com>	14 years ago
Andrew Elwell	aa9b4d92cd	POD Cleanups Signed-off-by: Andrew Elwell <Andrew.Elwell@gmail.com> Signed-off-by: Galen Charlton <gmcharlt@gmail.com>	14 years ago
Galen Charlton	29eadaa1f7	fix a common "developement" typo Signed-off-by: Galen Charlton <gmcharlt@gmail.com>	14 years ago
Galen Charlton	64daee5495	fix a common "developement" typo Signed-off-by: Galen Charlton <gmcharlt@gmail.com>	14 years ago
Donovan Jones	593a7d8e6a	Bug 2505 - Add commented use warnings where missing in *.pm	14 years ago
Lars Wirzenius	7279f55b60	Fix FSF address in directory C4/ Signed-off-by: Galen Charlton <gmcharlt@gmail.com>	14 years ago
Chris Nighswonger	9b47b82a77	Porting SimpleSearch return changes to all code calling it. Signed-off-by: Joshua Ferraro <jmf@liblime.com>	16 years ago
Andrew Moore	5c51edc653	bug 1890: fixing calls to C4::Search::SimpleSearch Signed-off-by: Galen Charlton <galen.charlton@liblime.com> Signed-off-by: Joshua Ferraro <jmf@liblime.com>	16 years ago
Galen Charlton	a0236d1460	improve authority heading matching (partial fix for 1905) Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Signed-off-by: Joshua Ferraro <jmf@liblime.com>	16 years ago
Galen Charlton	9af0017033	authorities: limit by heading type during matching Signed-off-by: Chris Cormack <crc@liblime.com> Signed-off-by: Joshua Ferraro <jmf@liblime.com>	17 years ago
Galen Charlton	d163382a27	authorities - new module - C4::Heading Objects of type C4::Heading represent headings and are (currently) initialized from bib MARC::Field objects. A C4::Heading has the ability via SimpleSearch to locate the authority records that are either authorizing the heading itself or contained more preferred terms for the heading via See links. Signed-off-by: Chris Cormack <crc@liblime.com> Signed-off-by: Joshua Ferraro <jmf@liblime.com>	17 years ago

23 Commits (67f695001a17556fc3b1796181fff9a1493cae94)