3 # Script to perform searching
4 # For documentation try 'perldoc /path/to/search'
8 # Copyright 2006 LibLime
10 # This file is part of Koha
12 # Koha is free software; you can redistribute it and/or modify it under the
13 # terms of the GNU General Public License as published by the Free Software
14 # Foundation; either version 2 of the License, or (at your option) any later
17 # Koha is distributed in the hope that it will be useful, but WITHOUT ANY
18 # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
19 # A PARTICULAR PURPOSE. See the GNU General Public License for more details.
21 # You should have received a copy of the GNU General Public License along with
22 # Koha; if not, write to the Free Software Foundation, Inc., 59 Temple Place,
23 # Suite 330, Boston, MA 02111-1307 USA
27 search - a search script for finding records in a Koha system (Version 2.4)
31 This script contains a demonstration of a new search API for Koha 2.4. It is
32 designed to be simple to use and configure, yet capable of performing feats
33 like stemming, field weighting, relevance ranking, support for multiple
34 query language formats (CCL, CQL, PQF), full or nearly full support for the
35 bib1 attribute set, extended attribute sets defined in Zebra profiles, access
36 to the full range of Z39.50 query options, federated searches on Z39.50
39 I believe the API as represented in this script is mostly sound, even if the
40 individual functions in Search.pm and Koha.pm need to be cleaned up. Of course,
41 you are free to disagree :-)
43 I will attempt to describe what is happening at each part of this script.
48 This script performs two functions:
52 =item 1. interacts with Koha to retrieve and display the results of a search
54 =item 2. loads the advanced search page
58 These two functions share many of the same variables and modules, so the first
59 task is to load what they have in common and determine which template to use.
60 Once determined, proceed to only load the variables and procedures necessary
63 =head2 THE ADVANCED SEARCH PAGE
65 If we're loading the advanced search page this script will call a number of
66 display* routines which populate objects that are sent to the template for
67 display of things like search indexes, languages, search limits, branches,
68 etc. These are not stored in the template for two reasons:
72 =item 1. Efficiency - we have more control over objects inside the script, and it's possible to not duplicate things like indexes (if the search indexes were stored in the template they would need to be repeated)
74 =item 2. Customization - if these elements were moved to the sql database it would allow a simple librarian to determine which fields to display on the page without editing any html (also how the fields should behave when being searched).
78 However, they create one problem : the strings aren't translated. I have an idea
79 for how to do this that I will purusue soon.
81 =head2 PERFORMING A SEARCH
83 If we're performing a search, this script performs three primary
88 =item 1. builds query strings (yes, plural)
90 =item 2. perform the search and return the results array
92 =item 3. build the HTML for output to the template
96 There are several additional secondary functions performed that I will
99 =head3 1. Building Query Strings
101 There are several types of queries needed in the process of search and retrieve:
105 =item 1 Koha query - the query that is passed to Zebra
107 This is the most complex query that needs to be built.The original design goal was to use a custom CCL2PQF query parser to translate an incoming CCL query into a multi-leaf query to pass to Zebra. It needs to be multi-leaf to allow field weighting, koha-specific relevance ranking, and stemming. When I have a chance I'll try to flesh out this section to better explain.
109 This query incorporates query profiles that aren't compatible with non-Zebra Z39.50 targets to acomplish the field weighting and relevance ranking.
111 =item 2 Federated query - the query that is passed to other Z39.50 targets
113 This query is just the user's query expressed in CCL CQL, or PQF for passing to a non-zebra Z39.50 target (one that doesn't support the extended profile that Zebra does).
115 =item 3 Search description - passed to the template / saved for future refinements of the query (by user)
117 This is a simple string that completely expresses the query in a way that can be parsed by Koha for future refinements of the query or as a part of a history feature. It differs from the human search description in several ways:
119 1. it does not contain commas or = signs
122 =item 4 Human search description - what the user sees in the search_desc area
124 This is a simple string nearly identical to the Search description, but more human readable. It will contain = signs or commas, etc.
128 =head3 2. Perform the Search
130 This section takes the query strings and performs searches on the named servers, including the Koha Zebra server, stores the results in a deeply nested object, builds 'faceted results', and returns these objects.
134 The final major section of this script takes the objects collected thusfar and builds the HTML for output to the template and user.
136 =head3 Additional Notes
142 use strict; # always use
144 #use warnings; # use only for development
146 ## STEP 1. Load things that are used in both search page and
147 # results page and decide which template to load, operations
154 use C4::Languages; # getAllLanguages
156 use C4::Branch; # GetBranches
157 use POSIX qw(ceil floor);
159 # create a new CGI object
160 # not sure undef_params option is working, need to test
161 use CGI qw('-no_undef_params');
164 my ( $template, $borrowernumber, $cookie );
166 # decide which template to use
168 my @params = $cgi->param("limit");
169 if ( ( @params > 1 ) || ( $cgi->param("q") ) ) {
171 $template_name = 'opac-results.tmpl';
175 $template_name = 'opac-advsearch.tmpl';
179 ( $template, $borrowernumber, $cookie ) = get_template_and_user(
181 template_name => $template_name,
184 authnotrequired => 1,
188 =head1 BUGS and FIXMEs
190 There are many, most are documented in the code. The one that
191 isn't fully documented, but referred to is the need for a full
197 # FIXME: URI re-writing should be tested more carefully and may better
198 # handled by mod_rewrite or something else. The code below almost works,
199 # but doesn't quite handle limits correctly when they are the only
200 # param passed -- I'll work on this soon -- JF
202 #my $uri = $cgi->url(-base => 1);
203 #my $relative_url = $cgi->url(-relative=>1);
204 #$uri.="/".$relative_url."?";
206 #my @cgi_params_list = $cgi->param();
207 #my $url_params = $cgi->Vars;
209 #for my $each_param_set (@cgi_params_list) {
210 # $uri.= join "", map "\&$each_param_set=".$_, split("\0",$url_params->{$each_param_set}) if $url_params->{$each_param_set};
212 #warn "New URI:$uri";
213 # Only re-write a URI if there are params or if it already hasn't been re-written
214 #unless (($cgi->param('r')) || (!$cgi->param()) ) {
215 # print $cgi->redirect( -uri=>$uri."&r=1",
216 # -cookie => $cookie);
221 my $branches = GetBranches();
223 for my $branch_hash ( keys %$branches ) {
226 value => "branch: $branch_hash",
227 branchname => $branches->{$branch_hash}->{'branchname'},
230 $template->param( branchloop => \@branch_loop, );
232 # load the itemtypes (Called Collection Codes in the template -- used for circ rules )
233 my $itemtypes = GetItemTypes;
237 my $imgdir = getitemtypeimagesrc();
238 foreach my $thisitemtype ( sort keys %$itemtypes ) {
241 imageurl => $imgdir . "/" . $itemtypes->{$thisitemtype}->{'imageurl'},
242 code => $thisitemtype,
243 selected => $selected,
244 description => $itemtypes->{$thisitemtype}->{'description'},
247 $selected = 0 if ($selected);
248 push @itemtypesloop, \%row;
250 $template->param( itemtypeloop => \@itemtypesloop );
252 # # load the itypes (Called item types in the template -- just authorized values for searching)
253 # my ($itypecount,@itype_loop) = GetCcodes();
254 # $template->param(itypeloop=>\@itype_loop,);
256 # load the languages ( for switching from one template to another )
257 # my @languages_options = displayLanguages($cgi);
258 # my $languages_count = @languages_options;
259 # if($languages_count > 1){
260 # $template->param(languages => \@languages_options);
263 # The following should only be loaded if we're bringing up the advanced search template
264 if ( $template_name eq "opac-advsearch.tmpl" ) {
266 # load the servers (used for searching -- to do federated searching, etc.)
267 my $primary_servers_loop; # = displayPrimaryServers();
268 $template->param( outer_servers_loop => $primary_servers_loop, );
270 my $secondary_servers_loop; # = displaySecondaryServers();
271 $template->param( outer_sup_servers_loop => $secondary_servers_loop, );
273 # load the limit types (icon-based limits in advanced search page)
274 my $outer_limit_types_loop = displayLimitTypes();
275 $template->param( outer_limit_types_loop => $outer_limit_types_loop, );
277 # load the search indexes (what a user can choose to search by)
278 my $indexes = displayIndexes();
280 # determine what to display next to the search boxes (ie, boolean option
281 # shouldn't appear on the first one, scan indexes should, adding a new
282 # box should only appear on the last, etc.
283 # FIXME: this stuff should be cleaned up a bit and the html should be turned
284 # into flags for the template -- I'll work on that soon -- JF
285 my @search_boxes_array;
286 my $search_boxes_count = 1; # should be a syspref
287 for ( my $i = 0 ; $i <= $search_boxes_count ; $i++ ) {
288 my $this_index = [@$indexes]; # clone the data, not just the reference
289 #@$this_index[$i]->{selected} = "selected";
291 push @search_boxes_array,
293 indexes => $this_index,
294 search_boxes_label => "<span class='labels'>Search for:</span>",
296 "<input type='checkbox' name='scan' id='scan' value='1'/><label for='scan'>Scan Indexes</label>",
300 elsif ( $i == $search_boxes_count ) {
301 push @search_boxes_array,
305 " <select name='op'><option value='and' selected='selected'>and</option><option value='or'>or</option><option value='not'>not</option></select>",
310 push @search_boxes_array,
314 " <select name='op'><option value='and' selected='selected'>and</option><option value='or'>or</option><option value='not'>not</option></select>",
319 uc( C4::Context->preference("marcflavour") ) => 1,
320 search_boxes_loop => \@search_boxes_array
323 # load the language limits (for search)
324 my $languages_limit_loop = getAllLanguages();
325 $template->param( search_languages_loop => $languages_limit_loop, );
327 # load the subtype limits
328 my $outer_subtype_limits_loop = displaySubtypesLimit();
329 $template->param( outer_subtype_limits_loop => $outer_subtype_limits_loop,
332 my $expanded_options;
333 if ( C4::Context->preference("expandedSearchOption") ) {
334 $expanded_options = C4::Context->preference("expandedSearchOption");
337 $expanded_options = $cgi->param('expanded_options');
340 $template->param( expanded_options => $expanded_options );
342 # load the sort_by options for the template
343 my $sort_by = $cgi->param('sort_by');
344 my $sort_by_loop = displaySortby($sort_by);
346 sort_by_loop => $sort_by_loop,
347 OpacCloud => C4::Context->preference("OpacCloud"),
348 OpacTopissue => C4::Context->preference("OpacTopissue"),
349 OpacAuthorities => C4::Context->preference("OpacAuthorities"),
352 output_html_with_http_headers $cgi, $cookie, $template->output;
356 ### OK, if we're this far, we're performing an actual search
358 # Fetch the paramater list as a hash in scalar context:
359 # * returns paramater list as tied hash ref
360 # * we can edit the values by changing the key
361 # * multivalued CGI paramaters are returned as a packaged string separated by "\0" (null)
362 my $params = $cgi->Vars;
364 # Params that can have more than one value
365 # sort by is used to sort the query
367 @sort_by = split( "\0", $params->{'sort_by'} ) if $params->{'sort_by'};
369 # load the sort_by options for the template
370 my $sort_by = $params->{'sort_by'};
371 my $sort_by_loop = displaySortby($sort_by);
372 $template->param( sort_by_loop => $sort_by_loop );
375 # Use the servers defined, or just search our local catalog(default)
377 @servers = split( "\0", $params->{'server'} ) if $params->{'server'};
380 #FIXME: this should be handled using Context.pm
381 @servers = ("biblioserver");
383 # @servers = C4::Context->config("biblioserver");
386 # operators include boolean and proximity operators and are used
387 # to evaluate multiple operands
389 @operators = split( "\0", $params->{'op'} ) if $params->{'op'};
391 # indexes are query qualifiers, like 'title', 'author', etc. They
392 # can be simple or complex
394 @indexes = split( "\0", $params->{'idx'} ) if $params->{'idx'};
396 # an operand can be a single term, a phrase, or a complete ccl query
398 @operands = split( "\0", $params->{'q'} ) if $params->{'q'};
400 # limits are use to limit to results to a pre-defined category such as branch or language
402 @limits = split( "\0", $params->{'limit'} ) if $params->{'limit'};
405 foreach my $limit (@limits) {
406 if ( $limit =~ /available/ ) {
410 $template->param( available => $available );
411 push @limits, map "yr:" . $_, split( "\0", $params->{'limit-yr'} )
412 if $params->{'limit-yr'};
414 # Params that can only have one value
415 my $query = $params->{'q'};
416 my $scan = $params->{'scan'};
417 my $results_per_page = $params->{'count'} || 20;
418 my $offset = $params->{'offset'} || 0;
420 my $expanded_facet = $params->{'expand'};
422 # Define some global variables
423 my $error; # used for error handling
424 my $search_desc; # the query expressed in terms that humans understand
426 ; # the query expressed in terms that zoom understands with field weighting and stemming
429 ; # usually not needed, but can be used to trigger ccl, cql, or pqf queries if set
431 ## I. BUILD THE QUERY
432 ( $error, $search_desc, $koha_query, $federated_query, $query_type ) =
433 buildQuery( $query, \@operators, \@operands, \@indexes, \@limits );
435 # warn "query : $koha_query";
436 ## II. DO THE SEARCH AND GET THE RESULTS
437 my $total; # the total results for the whole set
439 ; # this object stores the faceted results that display on the left-hand of the results page
443 if (C4::Context->preference('NoZebra')) {
445 ($error, $results_hashref, $facets) = NZgetRecords($koha_query,$federated_query,\@sort_by,\@servers,$results_per_page,$offset,$expanded_facet,$branches,$query_type,$scan);
449 ($error, $results_hashref, $facets) = getRecords($koha_query,$federated_query,\@sort_by,\@servers,$results_per_page,$offset,$expanded_facet,$branches,$query_type,$scan);
452 if ( $@ || $error ) {
453 $template->param( query_error => $error . $@ );
455 # warn "error: ".$error.$@;
456 output_html_with_http_headers $cgi, $cookie, $template->output;
460 # At this point, each server has given us a result set
461 # now we build that set for template display
462 my @sup_results_array;
463 for ( my $i = 0 ; $i <= @servers ; $i++ ) {
464 my $server = $servers[$i];
465 if ( $server =~ /biblioserver/ ) { # this is the local bibliographic server
466 $hits = $results_hashref->{$server}->{"hits"};
468 searchResults( $search_desc, $hits, $results_per_page, $offset,
469 @{ $results_hashref->{$server}->{"RECORDS"} } );
470 $total = $total + $results_hashref->{$server}->{"hits"};
472 $template->param( total => $hits );
473 $template->param( searchdesc => $search_desc );
474 $template->param( results_per_page => $results_per_page );
475 $template->param( SEARCH_RESULTS => \@newresults );
478 my $pages = ceil( $hits / $results_per_page );
479 my $current_page_number = 1;
480 $current_page_number = ( $offset / $results_per_page + 1 )
482 my $previous_page_offset = $offset - $results_per_page
483 unless ( $offset - $results_per_page < 0 );
484 my $next_page_offset = $offset + $results_per_page;
485 for ( $i = 1 ; $i <= $pages ; $i++ ) {
487 ( ( $i * $results_per_page ) - $results_per_page );
488 my $this_page_number = $i;
490 if ( $this_page_number == $current_page_number );
491 if ($this_page_number <= $pages) {
494 offset => $this_offset,
495 pg => $this_page_number,
496 highlight => $highlight,
503 PAGE_NUMBERS => \@page_numbers,
504 previous_page_offset => $previous_page_offset,
505 next_page_offset => $next_page_offset
508 } # end of the if local
511 # check if it's a z3950 or opensearch source
512 my $zed3950 = 0; # FIXME :: Hardcoded value.
514 my @inner_sup_results_array;
515 for my $sup_record ( @{ $results_hashref->{$server}->{"RECORDS"} } )
517 my $marc_record_object =
518 MARC::Record->new_from_usmarc($sup_record);
520 $marc_record_object->field('010')->subfield('a')
521 if $marc_record_object->field('010');
522 $control_number =~ s/^ //g;
524 "http://catalog.loc.gov/cgi-bin/Pwebrecon.cgi?SAB1="
526 . "&BOOL1=all+of+these&FLD1=LC+Control+Number+LCCN+%28K010%29+%28K010%29&GRP1=AND+with+next+set&SAB2=&BOOL2=all+of+these&FLD2=Keyword+Anywhere+%28GKEY%29+%28GKEY%29&PID=6211&SEQ=20060816121838&CNT=25&HIST=1";
527 my $title = $marc_record_object->title();
528 push @inner_sup_results_array,
534 my $servername = $server;
535 push @sup_results_array,
537 servername => $servername,
538 inner_sup_results_loop => \@inner_sup_results_array
540 $template->param( outer_sup_results_loop => \@sup_results_array );
544 } #/end of the for loop
546 #$template->param(FEDERATED_RESULTS => \@results_array);
547 # adding the $RequestOnOpac param
549 if (C4::Context->preference("RequestOnOpac")) {
552 # get site URL (for RSS link)
553 $cgi->url() =~ /(.*)\/(.*)/;
558 #classlist => $classlist,
560 searchdesc => $search_desc,
562 facets_loop => $facets,
564 . C4::Context->preference("BiblioDefaultView") => 1,
566 search_error => $error,
567 RequestOnOpac => $RequestOnOpac,
569 site_url => $site_url,
571 ## Now let's find out if we have any supplemental data to show the user
572 # and in the meantime, save the current query for statistical purposes, etc.
573 my $koha_spsuggest; # a flag to tell if we've got suggestions coming from Koha
575 ; # place we store the suggestions to be returned to the template as LOOP
576 my $phrases = $search_desc;
579 if ( C4::Context->preference("kohaspsuggest") ) {
581 my $koha_spsuggest_dbh;
583 # FIXME: this needs to be moved to Context.pm
585 $koha_spsuggest_dbh =
586 DBI->connect( "DBI:mysql:suggest:66.213.78.76",
587 "auth", "Free2cirC" );
590 warn "can't connect to spsuggest db";
593 my $koha_spsuggest_insert =
594 "INSERT INTO phrase_log(phr_phrase,phr_resultcount,phr_ip) VALUES(?,?,?)";
595 my $koha_spsuggest_query =
596 "SELECT display FROM distincts WHERE strcmp(soundex(suggestion), soundex(?)) = 0 order by soundex(suggestion) limit 0,5";
597 my $koha_spsuggest_sth =
598 $koha_spsuggest_dbh->prepare($koha_spsuggest_query);
599 $koha_spsuggest_sth->execute($phrases);
600 while ( my $spsuggestion = $koha_spsuggest_sth->fetchrow_array ) {
601 $spsuggestion =~ s/(:|\/)//g;
603 $line{spsuggestion} = $spsuggestion;
604 push @koha_spsuggest, \%line;
608 # Now save the current query
609 $koha_spsuggest_sth =
610 $koha_spsuggest_dbh->prepare($koha_spsuggest_insert);
612 #$koha_spsuggest_sth->execute($phrases,$results_per_page,$ipaddress);
613 $koha_spsuggest_sth->finish;
614 $template->param( koha_spsuggest => $koha_spsuggest ) unless $hits;
615 $template->param( SPELL_SUGGEST => \@koha_spsuggest, );
619 warn "Kohaspsuggest failure:" . $@;
623 # VI. BUILD THE TEMPLATE
624 output_html_with_http_headers $cgi, $cookie, $template->output;