2 # Script to perform searching
3 # For documentation try 'perldoc /path/to/search'
7 # Copyright 2006 LibLime
9 # This file is part of Koha
11 # Koha is free software; you can redistribute it and/or modify it under the
12 # terms of the GNU General Public License as published by the Free Software
13 # Foundation; either version 2 of the License, or (at your option) any later
16 # Koha is distributed in the hope that it will be useful, but WITHOUT ANY
17 # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
18 # A PARTICULAR PURPOSE. See the GNU General Public License for more details.
20 # You should have received a copy of the GNU General Public License along with
21 # Koha; if not, write to the Free Software Foundation, Inc., 59 Temple Place,
22 # Suite 330, Boston, MA 02111-1307 USA
26 search - a search script for finding records in a Koha system (Version 3.0)
30 This script contains a new search API for Koha 3.0. It is designed to be
31 simple to use and configure, yet capable of performing feats like stemming,
32 field weighting, relevance ranking, support for multiple query language
33 formats (CCL, CQL, PQF), full or nearly full support for the
34 bib1 attribute set, extended attribute sets defined in Zebra profiles, access
35 to the full range of Z39.50 query options, federated searches on Z39.50
38 I believe the API as represented in this script is mostly sound, even if the
39 individual functions in Search.pm and Koha.pm need to be cleaned up. Of course,
40 you are free to disagree :-)
42 I will attempt to describe what is happening at each part of this script.
47 This script performs two functions:
51 =item 1. interacts with Koha to retrieve and display the results of a search
53 =item 2. loads the advanced search page
57 These two functions share many of the same variables and modules, so the first
58 task is to load what they have in common and determine which template to use.
59 Once determined, proceed to only load the variables and procedures necessary
62 =head2 THE ADVANCED SEARCH PAGE
64 If we're loading the advanced search page this script will call a number of
65 display* routines which populate objects that are sent to the template for
66 display of things like search indexes, languages, search limits, branches,
67 etc. These are not stored in the template for two reasons:
71 =item 1. Efficiency - we have more control over objects inside the script,
72 and it's possible to not duplicate things like indexes (if the search indexes
73 were stored in the template they would need to be repeated)
75 =item 2. Customization - if these elements were moved to the sql database it
76 would allow a simple librarian to determine which fields to display on the page
77 without editing any html (also how the fields should behave when being searched).
81 However, they create one problem : the strings aren't translated. I have an idea
82 for how to do this that I will purusue soon.
84 =head2 PERFORMING A SEARCH
86 If we're performing a search, this script performs three primary
91 =item 1. builds query strings (yes, plural)
93 =item 2. perform the search and return the results array
95 =item 3. build the HTML for output to the template
99 There are several additional secondary functions performed that I will
102 =head3 1. Building Query Strings
104 There are several types of queries needed in the process of search and retrieve:
108 =item 1 Koha query - the query that is passed to Zebra
110 This is the most complex query that needs to be built. The original design goal
111 was to use a custom CCL2PQF query parser to translate an incoming CCL query into
112 a multi-leaf query to pass to Zebra. It needs to be multi-leaf to allow field
113 weighting, koha-specific relevance ranking, and stemming. When I have a chance
114 I'll try to flesh out this section to better explain.
116 This query incorporates query profiles that aren't compatible with non-Zebra
117 Z39.50 targets to acomplish the field weighting and relevance ranking.
119 =item 2 Federated query - the query that is passed to other Z39.50 targets
121 This query is just the user's query expressed in CCL CQL, or PQF for passing to a
122 non-zebra Z39.50 target (one that doesn't support the extended profile that Zebra does).
124 =item 3 Search description - passed to the template / saved for future refinements of
127 This is a simple string that completely expresses the query in a way that can be parsed
128 by Koha for future refinements of the query or as a part of a history feature. It differs
129 from the human search description:
131 1. it does not contain commas or = signs
133 =item 4 Human search description - what the user sees in the search_desc area
135 This is a simple string nearly identical to the Search description, but more human
136 readable. It will contain = signs or commas, etc.
140 =head3 2. Perform the Search
142 This section takes the query strings and performs searches on the named servers, including
143 the Koha Zebra server, stores the results in a deeply nested object, builds 'faceted results',
144 and returns these objects.
148 The final major section of this script takes the objects collected thusfar and builds the
149 HTML for output to the template and user.
151 =head3 Additional Notes
157 use strict; # always use
159 ## STEP 1. Load things that are used in both search page and
160 # results page and decide which template to load, operations
167 use C4::Languages; # getAllLanguages
169 use POSIX qw(ceil floor);
170 use C4::Branch; # GetBranches
171 # create a new CGI object
172 # not sure undef_params option is working, need to test
173 use CGI qw('-no_undef_params');
176 my ($template,$borrowernumber,$cookie);
178 # decide which template to use
181 my @params = $cgi->param("limit");
182 if ((@params>=1) || ($cgi->param("q")) || ($cgi->param('multibranchlimit')) ) {
183 $template_name = 'catalogue/results.tmpl';
186 # use a UNIMARC-specific template if UNIMARC
187 if (C4::Context->preference("marcflavour") eq "UNIMARC" ) {
188 $template_name = 'catalogue/advsearch_unimarc.tmpl';
191 $template_name = 'catalogue/advsearch.tmpl';
193 $template_type = 'advsearch';
196 ($template, $borrowernumber, $cookie) = get_template_and_user({
197 template_name => $template_name,
200 authnotrequired => 0,
201 flagsrequired => { catalogue => 1 },
205 =head1 BUGS and FIXMEs
207 There are many, most are documented in the code. The one that
208 isn't fully documented, but referred to is the need for a full
214 # Deprecated, but preserved because it's interesting :-)
216 #my $uri = $cgi->url(-base => 1);
217 #my $relative_url = $cgi->url(-relative=>1);
218 #$uri.="/".$relative_url."?";
220 #my @cgi_params_list = $cgi->param();
221 #my $url_params = $cgi->Vars;
223 #for my $each_param_set (@cgi_params_list) {
224 # $uri.= join "", map "\&$each_param_set=".$_, split("\0",$url_params->{$each_param_set}) if $url_params->{$each_param_set};
226 #warn "New URI:$uri";
227 # Only re-write a URI if there are params or if it already hasn't been re-written
228 #unless (($cgi->param('r')) || (!$cgi->param()) ) {
229 # print $cgi->redirect( -uri=>$uri."&r=1",
230 # -cookie => $cookie);
235 my $branches = GetBranches();
237 #push @branch_loop, {value => "", branchname => "All Branches", };
238 for my $branch_hash (sort keys %$branches) {
239 push @branch_loop, {value => "branch:$branch_hash" , branchname => $branches->{$branch_hash}->{'branchname'}, };
242 my $categories = GetBranchCategories(undef,'searchdomain');
244 $template->param(branchloop => \@branch_loop, searchdomainloop => $categories);
247 my $itemtypes = GetItemTypes;
251 my $imgdir = getitemtypeimagesrc();
252 foreach my $thisitemtype ( sort {$itemtypes->{$a}->{'description'} cmp $itemtypes->{$b}->{'description'} } keys %$itemtypes ) {
253 my %row =( number=>$cnt++,
254 imageurl=> $itemtypes->{$thisitemtype}->{'imageurl'}?($imgdir."/".$itemtypes->{$thisitemtype}->{'imageurl'}):"",
255 code => $thisitemtype,
256 selected => $selected,
257 description => $itemtypes->{$thisitemtype}->{'description'},
260 $selected = 0 if ($selected) ;
261 push @itemtypesloop, \%row;
263 $template->param(itemtypeloop => \@itemtypesloop);
265 # # load the itypes (Called item types in the template -- just authorized values for searching)
266 # my ($itypecount,@itype_loop) = GetCcodes();
267 # $template->param(itypeloop=>\@itype_loop,);
269 # load the languages ( for switching from one template to another )
270 $template->param(languages_loop => getTranslatedLanguages('intranet','prog'));
272 # The following should only be loaded if we're bringing up the advanced search template
273 if ( $template_type eq 'advsearch' ) {
274 # load the servers (used for searching -- to do federated searching, etc.)
275 my $primary_servers_loop;# = displayPrimaryServers();
276 $template->param(outer_servers_loop => $primary_servers_loop,);
278 my $secondary_servers_loop;# = displaySecondaryServers();
279 $template->param(outer_sup_servers_loop => $secondary_servers_loop,);
281 # determine what to display next to the search boxes (ie, boolean option
282 # shouldn't appear on the first one, scan indexes should, adding a new
283 # box should only appear on the last, etc.
284 my @search_boxes_array;
285 my $search_boxes_count = C4::Context->preference("OPACAdvSearchInputCount") | 3; # FIXME: should be a syspref
286 for (my $i=1;$i<=$search_boxes_count;$i++) {
287 # if it's the first one, don't display boolean option, but show scan indexes
289 push @search_boxes_array,
295 # if it's the last one, show the 'add field' box
296 elsif ($i==$search_boxes_count) {
297 push @search_boxes_array,
304 push @search_boxes_array,
311 $template->param(uc(C4::Context->preference("marcflavour")) => 1,
312 search_boxes_loop => \@search_boxes_array);
314 # load the language limits (for search)
315 my $languages_limit_loop = getAllLanguages();
316 $template->param(search_languages_loop => $languages_limit_loop,);
318 $template->param(expanded_options => $cgi->param('expanded_options') | C4::Context->preference("expandedSearchOption"));
320 # load the sort_by options for the template
321 my $sort_by = $cgi->param('sort_by');
323 output_html_with_http_headers $cgi, $cookie, $template->output;
327 ### OK, if we're this far, we're performing an actual search
329 # Fetch the paramater list as a hash in scalar context:
330 # * returns paramater list as tied hash ref
331 # * we can edit the values by changing the key
332 # * multivalued CGI paramaters are returned as a packaged string separated by "\0" (null)
333 my $params = $cgi->Vars;
335 # Params that can have more than one value
336 # sort by is used to sort the query
338 @sort_by = split("\0",$params->{'sort_by'}) if $params->{'sort_by'};
340 # Use the servers defined, or just search our local catalog(default)
342 @servers = split("\0",$params->{'server'}) if $params->{'server'};
344 #FIXME: this should be handled using Context.pm
345 @servers = ("biblioserver");
346 # @servers = C4::Context->config("biblioserver");
349 # operators include boolean and proximity operators and are used
350 # to evaluate multiple operands
352 @operators = split("\0",$params->{'op'}) if $params->{'op'};
354 # indexes are query qualifiers, like 'title', 'author', etc. They
355 # can be simple or complex
357 if ($params->{'idx'}) {
358 @indexes = split("\0",$params->{'idx'});
360 $indexes[0] = 'kw,wrdl';
363 # an operand can be a single term, a phrase, or a complete ccl query
365 @operands = split("\0",$params->{'q'}) if $params->{'q'};
367 # limits are use to limit to results to a pre-defined category such as branch or language
369 @limits = split("\0",$params->{'limit'}) if $params->{'limit'};
371 if($params->{'multibranchlimit'}) {
372 push @limits, join(" or ", map { "branch: $_ "} @{GetBranchesInCategory($params->{'multibranchlimit'})}) ;
376 foreach my $limit(@limits) {
377 if ($limit =~/available/) {
381 $template->param(available => $available);
382 push @limits, map "yr:".$_, split("\0",$params->{'limit-yr'}) if $params->{'limit-yr'};
384 # Params that can only have one value
385 my $query = $params->{'q'};
386 my $scan = $params->{'scan'};
387 my $results_per_page = $params->{'count'} || 20;
388 my $page = $cgi->param('page') || 1;
389 my $offset = ($page-1)*$results_per_page;
391 my $expanded_facet = $params->{'expand'};
393 # Define some global variables
394 my $error; # used for error handling
395 my $search_desc; # the query expressed in terms that humans understand
396 my $koha_query; # the query expressed in terms that zoom understands with field weighting and stemming
398 my $query_type; # usually not needed, but can be used to trigger ccl, cql, or pqf queries if set
400 ## I. BUILD THE QUERY
401 ($error,$search_desc,$koha_query,$federated_query,$query_type) = buildQuery($query,\@operators,\@operands,\@indexes,\@limits);
403 ## II. DO THE SEARCH AND GET THE RESULTS
404 my $total; # the total results for the whole set
405 my $facets; # this object stores the faceted results that display on the left-hand of the results page
409 if (C4::Context->preference('NoZebra')) {
411 ($error, $results_hashref, $facets) = NZgetRecords($koha_query,$federated_query,\@sort_by,\@servers,$results_per_page,$offset,$expanded_facet,$branches,$query_type,$scan);
415 ($error, $results_hashref, $facets) = getRecords($koha_query,$federated_query,\@sort_by,\@servers,$results_per_page,$offset,$expanded_facet,$branches,$query_type,$scan);
419 $template->param(query_error => $error.$@);
421 output_html_with_http_headers $cgi, $cookie, $template->output;
424 my $op=$cgi->param("operation");
425 if ($op eq "bulkedit"){
426 my ($countchanged,$listunchanged)=
427 ModBiblios($results_hashref->{'biblioserver'}->{"RECORDS"},
428 $params->{"tagsubfield"},
429 $params->{"inputvalue"},
430 $params->{"targetvalue"},
433 $template->param(bulkeditresults=>1,
434 tagsubfield=>$params->{"tagsubfield"},
435 inputvalue=>$params->{"inputvalue"},
436 targetvalue=>$params->{"targetvalue"},
437 countchanged=>$countchanged,
438 countunchanged=>scalar(@$listunchanged),
439 listunchanged=>$listunchanged);
442 if (C4::Context->userenv->{'flags'}==1 ||(C4::Context->userenv->{'flags'} & ( 2**9 ) )){
443 #Edit Catalogue Permissions
444 $template->param(bulkedit => 1);
445 $template->param(tagsubfields=>GetManagedTagSubfields());
447 # At this point, each server has given us a result set
448 # now we build that set for template display
449 my @sup_results_array;
450 for (my $i=0;$i<=@servers;$i++) {
451 my $server = $servers[$i];
452 if ($server =~/biblioserver/) { # this is the local bibliographic server
453 $hits = $results_hashref->{$server}->{"hits"};
454 my $page = $cgi->param('page') || 0;
455 my @newresults = searchResults( $search_desc,$hits,$results_per_page,$offset,@{$results_hashref->{$server}->{"RECORDS"}});
456 $total = $total + $results_hashref->{$server}->{"hits"};
458 $template->param(total => $hits);
459 $template->param(searchdesc => ($query_type?"$query_type=":"")."$search_desc" );
460 $template->param(results_per_page => $results_per_page);
461 $template->param(SEARCH_RESULTS => \@newresults);
464 # my $pages = ceil($hits / $results_per_page);
465 # my $previous_page_offset = $offset - $results_per_page unless ($offset - $results_per_page <0);
466 # my $next_page_offset = $offset + $results_per_page;
467 # for (my $j=1; $j<=$pages;$j++) {
468 # my $this_offset = (($j*$results_per_page)-$results_per_page);
469 # my $this_page_number = $j;
470 # my $highlight = 1 if ($this_page_number == $current_page_number);
471 # if ($this_page_number <= $pages) {
472 # push @page_numbers, { offset => $this_offset, pg => $this_page_number, highlight => $highlight, sort_by => join " ",@sort_by };
475 # $template->param(PAGE_NUMBERS => \@page_numbers,
476 # previous_page_offset => $previous_page_offset,
477 # next_page_offset => $next_page_offset) unless $pages < 2;
478 my $link="/cgi-bin/koha/catalogue/search.pl?q=$search_desc&";
479 foreach my $sort (@sort_by){
480 $link.="&sort_by=".$sort."&";
483 pagination_bar => pagination_bar(
485 getnbpages($hits, $results_per_page),
491 } # end of the if local
493 # check if it's a z3950 or opensearch source
494 my $zed3950 = 0; # FIXME :: Hardcoded value.
496 my @inner_sup_results_array;
497 for my $sup_record ( @{$results_hashref->{$server}->{"RECORDS"}} ) {
498 my $marc_record_object = MARC::Record->new_from_usmarc($sup_record);
499 my $control_number = $marc_record_object->field('010')->subfield('a') if $marc_record_object->field('010');
500 $control_number =~ s/^ //g;
501 my $link = "http://catalog.loc.gov/cgi-bin/Pwebrecon.cgi?SAB1=".$control_number."&BOOL1=all+of+these&FLD1=LC+Control+Number+LCCN+%28K010%29+%28K010%29&GRP1=AND+with+next+set&SAB2=&BOOL2=all+of+these&FLD2=Keyword+Anywhere+%28GKEY%29+%28GKEY%29&PID=6211&SEQ=20060816121838&CNT=25&HIST=1";
502 my $title = $marc_record_object->title();
503 push @inner_sup_results_array, {
508 my $servername = $server;
509 push @sup_results_array, { servername => $servername, inner_sup_results_loop => \@inner_sup_results_array};
510 $template->param(outer_sup_results_loop => \@sup_results_array);
514 } #/end of the for loop
515 #$template->param(FEDERATED_RESULTS => \@results_array);
519 #classlist => $classlist,
521 searchdesc => ($query_type?"$query_type=":"")."$search_desc",
523 facets_loop => $facets,
525 search_error => $error,
527 ## Now let's find out if we have any supplemental data to show the user
528 # and in the meantime, save the current query for statistical purposes, etc.
529 my $koha_spsuggest; # a flag to tell if we've got suggestions coming from Koha
530 my @koha_spsuggest; # place we store the suggestions to be returned to the template as LOOP
531 my $phrases = $search_desc;
534 if ( C4::Context->preference("kohaspsuggest") ) {
535 my ($suggest_host, $suggest_dbname, $suggest_user, $suggest_pwd) = split(':', C4::Context->preference("kohaspsuggest"));
537 my $koha_spsuggest_dbh;
538 # FIXME: this needs to be moved to Context.pm
540 $koha_spsuggest_dbh=DBI->connect("DBI:mysql:$suggest_dbname:$suggest_host","$suggest_user","$suggest_pwd");
543 warn "can't connect to spsuggest db";
546 my $koha_spsuggest_insert = "INSERT INTO phrase_log(phr_phrase,phr_resultcount,phr_ip) VALUES(?,?,?)";
547 my $koha_spsuggest_query = "SELECT display FROM distincts WHERE strcmp(soundex(suggestion), soundex(?)) = 0 order by soundex(suggestion) limit 0,5";
548 my $koha_spsuggest_sth = $koha_spsuggest_dbh->prepare($koha_spsuggest_query);
549 $koha_spsuggest_sth->execute($phrases);
550 while (my $spsuggestion = $koha_spsuggest_sth->fetchrow_array) {
551 $spsuggestion =~ s/(:|\/)//g;
553 $line{spsuggestion} = $spsuggestion;
554 push @koha_spsuggest,\%line;
558 # Now save the current query
559 $koha_spsuggest_sth=$koha_spsuggest_dbh->prepare($koha_spsuggest_insert);
560 #$koha_spsuggest_sth->execute($phrases,$results_per_page,$ipaddress);
561 $koha_spsuggest_sth->finish;
563 $template->param( koha_spsuggest => $koha_spsuggest ) unless $hits;
564 $template->param( SPELL_SUGGEST => \@koha_spsuggest,
569 warn "Kohaspsuggest failure:".$@;
573 # VI. BUILD THE TEMPLATE
574 output_html_with_http_headers $cgi, $cookie, $template->output;