The script dies with
Can't use an undefined value as a HASH reference at tools/export.pl
line 149.
if it is called from the command line.
This is introduced by bug 13040.
The C4::Context->userenv is not defined in this case.
Test plan:
Execute the script using the command line.
With the patch applies you should not get the error.
Signed-off-by: Joonas Kylmälä <j.kylmala@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Tools > Export data produces a file badly encoded, when marc/xml file
format is chosen. It works with CSV.
To test:
1. Tools > Export data
2. Choose a limited interval of biblionumber
3. Export in marc/xml/csv.
4. Check files exported at 3: csv file is ok. marc/xml files are
corrupted, ie badly encoded.
5. Apply the patch
6. Repeat steps 3&4, and confirm that all file formats are valid.
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Courret <scourret@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
To reproduce:
1/ Go on tools/export.pl
2/ Export some records and specify you don't want the fields 245 (the
whole field, do not specify a subfield).
3/ Export, BOOM
Software error:
Arguments must be MARC::Field object at /home/koha/src/tools/export.pl line 400.
Test plan:
Apply the patch and confirm the fields 245 are not exported and the
export works as expected.
Signed-off-by: Robin Sheat <robin@catalyst.net.nz>
Export fails to asplode with this patch
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
This allows the exporter (Tools -> Export) to have any combination of
branches selected, rather than it being all or only one.
Test Plan:
* Apply the patch
* Go to the exporter, see that instead of a dropdown you now have an
elegently laid out grid of branches you can select from
* Select some branches, run the export
* Note that only records with items in the selected branches are
returned.
* Repeat this with the item related options (as that code was refactored
slightly) and make sure everything is sane.
Sponsored-By: South Taranaki District Libraries
Signed-off-by: Thomas <tomsStudy@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
In the template of the export tools, the value 'marc' is set for
the 'format' variable.
It should be 'iso2709'.
There was a mess in the tools/export.pl script.
2 variables did the same thing: $format and $output_format
This patch fixes that replacing $format with $output_format
Test plan:
Try to export biblio in iso2709 and csv using the export tools and from
the checkout list.
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Before applying the patch, attemping an MARC export from tools/export.pl
resulted in an error. After the patch the export works correctly.
Exports from the checkouts page also work correctly.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes tests and QA script. Also tested authority export.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Test:
1. Edit record, add 100.000 chars text to 500a
2. xml export produce the record,
3. mrc export do not produce the record, warning on log export.pl:
Record length of 111000 is larger than the MARC spec allows (99999
bytes). at /usr/share/perl5/MARC/File/USMARC.pm line 314. record
(number 139489) is invalid and therefore not exported because its
reopening generates warnings above at...
Signed-off-by: Frederic Demians <f.demians@tamil.fr>
I confirm that exporting biblio records larger than 10000 characters in
ISO2709 produces invalid files. After applying this patch, the culprit
record (too large, but also other inconsistencies preventing record
parsing with MARC::File::USMARC) is not exported anymore. A warning is
produced in Koha Apache log file. Warnings to the user on WUI would be
better, but it isn't the case yet, so it isn't a regression.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
I agree that a visible warning/result message in the staff interface
would be nice, but this works as described.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Bug 11703 breaks the checkouts export feature.
To reproduce: Fill the ExportWithCsvProfile pref and go on the
circ/circulation.pl page. The export column appears, but not the export
button.
Test plan:
Go on the checkout list (circ/circulation.pl and members/moremember.pl)
and verify the export column and the export button appears.
If you click on the button, a file should be generated.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Test plan
1/ run ./tools/export.pl
2/ notice an error like cannot open 0: No such file or directory at ./tools/export.pl line 209.
3/ apply patch
4/ run again
5/ no errors
Signed-off-by: Chris <chris@bigballofwax.co.nz>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described, adding an additional check.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Since we switched to Template Toolkit we don't need to stick with the
sufix we used for HTML::Template::Pro.
This patch changes the occurences of '.tmpl' in favour of '.tt'.
To test:
- Apply the patch
- Install koha, and verify that every page can be accesed
Regards
To+
P.S. a followup will remove the glue code.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
The MARC export tool now includes the option to specify a file
containing record IDs to export. When run on the command line,
this is specified using the --id_list_file switch.
Note that the list of IDs acts as a filter on other criteria
for selecting records to import. For example, if you export
all bibs belong to a given library and also specify an ID file,
the bibs must both belong to the library and be in the ID file.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Work as described. No errors
Tested writing txt file with biblionumbers, loading in tools and exporting
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Test:
1. Edit a bib record, add 100.000 chars text to 500a
2. Go to Tools -> Export data (or run tools/export.pl on
the command line)
3. Verify that exporting as XML includes the record
4. Verify that exporting as MARC does not include the record,
and that you get the following warning:
export.pl: record (number 2025) length 102668 is larger than the MARC spec allows (99999 bytes) at...
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Work as described. No errors
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
The method of checking the logged in user for superlibrarian privileges
is obtuse ( $userenv && $userenv->{flags} % 2 != 1 ) to say the least.
The codebase is littered with these lines, with no explanation given. It
would be much better if we had one subroutine that returned a boolean
value to tell us if the logged in user is a superlibrarian or not.
Test Plan:
1) Apply this patch
2) Verify superlibrarian behavior remains unchanged
Signed-off-by: Joel Sasse <jsasse@plumcreeklibrary.net>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Comments on second patch.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Regex assumed all numeric characters were part of the tag number
This is obviously false as $9 would be a likely candidate
to be removed on export.
The code can be any printing ascii character other than space
according to LoC's website.
Also changed regexp to no longer allow a zero length tag number
which is nonsensical.
The old regex would accept shorter than 3 digit tags but these were
not stripped so I've removed that option, considering it a bug
not a feature
NB assumption that the code makes that a tag is always numeric is
incorrect but works in practice. Handling non-numeric tags is
a 'To be done'
Made code dependent on the regex succeeding. Picking up results
from a previous regex on failure can lead to weird hard to
identify bugs
In the course of testing Paola Rossi <paola.rossi@cineca.it>
spotted that the delete operation was flawed, only removing
the first occurence of the specified tag/tag subfield. Reworked
the delete loop to operate on all occurences
Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Small typo in query construction, fixed.
To test:
1) Try filtering a MARC export using "Start date"
2) Notice the lack of filtering
3) Apply patch
4) Try export again, records are filtered by dateaccessioned
Signed-off-by: Cedric Vita <cedric.vita@dracenie.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Test Plan:
1) Enable IndependantBranches
2) Apply this patch
3) Run updatedatabase.pl
4) Verify that the system preference still functions correctly
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
When exporting some biblio or authorities records with tools/export.pl (via web client or command-line) :
If choosing XML format, you get a concatenation of full XML records.
This patch uses MARC::File::XML to create a well formated file.
See http://search.cpan.org/~gmcharlt/MARC-XML-0.93/lib/MARC/File/XML.pm#close%28%29
Test plan:
- Go to Tools/Export data
- Enter numbers in from and to biblio number
(make sure that at least two records will
be exported).
- Select xml in file format
- Click "Export bibliographic records"
- Save file somewhere
- Look at downloaded file
=> File should look like :
<?xml version="1.0" encoding="UTF-8"?>
<collection
...
<record>
...
</record>
<record>
...
</record>
</collection>
=> "collection" is the root node and XML declaration exists only once
- Do the same for authorities export and command-line use of tools/export.pl
Second test plan:
- From the command line, run tools/export.pl, e.g.,
tools/export.pl --format=xml --filename=bibs.xml
- Verify that the output is valid XML, e.g.,
xmllint --noout bibs.xml # if the file is valid, no error messages will be displayed
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
- the dateformat value is send to all templates (from
C4::Auth::get_template_and_user)
- remove all assignment of dateformat in all .pl files
- the DHTMLcalendar_dateformat variable is unused
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Fixed conflicts:
- opac/sco/sco-main.pl
- reports/acquisitions_stats.pl
- tools/cleanborrowers.pl
All tests pass, perlcritic problems appeared in some files
before and after these patches were applied.
Checked sorting in following pages:
- acqui/addorderiso2709.tt - list of staged imports in acq
- acqui/histsearch.tt - sorting of dates in acq search result list
- acqui/invoices.tt - billing date in list of invoices in acq
- acqui/lateorders.tt - list of late orders in acq
- acqui/ordered.tt - ordered titles and estimated costs for a fund
- acqui/parcels.tt - receive shipment page
- acqui/spent.tt - received titles and actual costs for a fund
...
- serials-search.tt - subscription search result list
...
- opac/sco/sco-main.tt - due dates in list of checked out items
- reports/acquisitions-stats.tt - date searches, display of dates
- tools/cleanborrowers.tt
- tools.holidays.tt - different views of dates library is closed,
adding dates
Checked dates display according to system preference everywhere and
searching, entering dates etc. still worked as expected.
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Only changing some documentation about GetXmlBiblio
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Added the word 'contain'
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
When building the query for exports by call number the script
has the logic reversed. It tries to select records with call
numbers less than the starting call number and greater than
the ending call number. This should be reversed.
To test, test an export in an unpatched system. An export by
call number will return an empty file. After applying the patch
an export by call number should give valid results.
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Works as expected.
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
In the circulation page, you can now export (as csv or iso2709) a list
of items which are currently checked out by a borrower.
3 export types:
- iso2709 with items: Export the items list in iso2709 format with item
informations.
- iso2709 without items: Export the items list in iso2709 format without
item informations.
- CSV: Export the items list based on a csv profil.
2 new system preferences:
- DontExportFields: a list of fields not to be export
- CsvProfileForExport: The Csv profile name used for the csv export
Test plan:
- Fill the CsvProfileForExport syspref
- go on the borrower circulation page containing checkouts
- Select one or more items and export them to the 3 different formats.
- check if the result file is what you expected
- Test there is no regression with the export authority
- Test there is no regression using tools/export.pl with the command
line interface
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Valid download attempts were being denied thanks to an incorrect
regular expression. This patch fixes that, and makes it easier to
understand what's going on in the code.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
With plackup, @ARGV can contains elements so we can't check the size of
@ARGV to tell whether we are in command line mode or not.
Instead we check environment variable GATEWAY_INTERFACE.
Also fix the use of a global variable in a subroutine
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
export.pl [--format=format] [--date=date] [--dont_export_items]
[--deleted_barcodes] [--clean] --filename=outputfile
* format is either 'xml' or 'marc' (default)
* date should be entered as the 'dateformat' syspref is set
(dd/mm/yyyy for metric, yyyy-mm-dd for iso, mm/dd/yyyy for us)
* records exported are the ones that have been modified since 'date'
* if --deleted_barcodes is used, a list of barcodes of items deleted
since 'date' is produced (or from all deleted items if no date is
specified)
* --clean removes NSE/NSB
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Fixes the following things:
1. Sanitizes log output to prevent an attacker from using a specially
crafted POST to add extra lines to the log
2. Simplify a regular expression since "..file" cannot be used to
escape the current directory
3. Makes sure directories are consistent
4. Correct logic issues in misc/cronjobs/backup.sh
Thanks to Frere Sebastien Marie for catching these issues.
Signed-off-by: Robin Sheat <robin@catalyst.net.nz>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
This patch builds on work by Lars Wirzenius for the Koha packages.
To date, the only way for a Koha librarian to obtain a complete backup
of their system has been to log into the system via SSH (or FTP) to
download the mysqldump file. This patch makes it possible for
superlibrarians in properly configured systems to download night backups
via the staff client's Export tool.
Recognizing that this is functionality with potentially very grave
security implications, system administrators must manually enable these
features in the koha-conf.xml configuration file.
The following configuration settings have been added to the koha-conf.xml
file:
* backupdir => directory where backups should be stored.
* backup_db_via_tools => whether to allow superlibrarians to download
database backups via the Export tool. The default is disabled, and
there is no way -- by design -- to enable this option without manually
editing koha-conf.xml.
* backup_conf_via_tools => whether to allow superlibrarians to download
configuration backups via the Export tool (this may be applicable to
packages only). The default is disabled, and there is no way -- by
design -- to enable this option without manually editing koha-conf.xml.
This commit modifies the following scripts to make use of the new
backupdir configuration option:
* koha-dump and koha-run-backups in the Debian packages
* The sample backup script misc/cronjobs/backup.sh
Note that for security reasons, superlibrarians will not be allowed
to download files that are not owned by the web server's effective user.
This imposes a de facto dependency on ITK (for Apache) or running the
web server as the Koha user (as is done with Plack).
To test:
1. Apply patch.
2. Go to export page as a superlibrarian. Notice that no additional
export options appear because they have not been enabled.
3. Add <backupdir>$KOHADEV/var/spool</backup> to the <config> section
of your koha-conf.xml (note that you will need to adjust that so that
it is pointing at a logical directory).
4. Create the aforementioned directory.
5. Go to export page as a superlibrarian. Notice that no additional
export options appear because they have not been enabled.
6. Add <backup_db_via_tools>1</backup_db_via_tools> to the <config>
section of your koha-conf.xml
7. Go to the export page as a superlibrarian. Notice the new tab.
8. Go to the export page as a non-superlibrarian. Notice there is no
new tab.
9. Run: mysqldump -u koha -p koha | gzip > $BACKUPDIR/backup.sql.gz
(substituting appropriate user, password, and database name)
10. Go to the export page as a superlibrarian, and look at the "Export
database" tab. If you are running the web server as your Koha user,
and ran the above command as your Koha user, you should now see the
file listed as an option for download.
11. If you *did* see the file listed, change the ownership to something
else: sudo chown root:root $BACKUPDIR/backup.sql.gz
11a. Confirm that you no longer see the file listed when you look at the
"Export database" tab.
12. Change the ownership on the file to your web server (or Koha) user:
sudo chown www-data:www-data backup.sql.gz
13. Go to the export page as a superlibrarian, and look at the "Export
database" tab. You should now see backup.sql.gz listed.
14. Choose to download backup.sql.gz
15. Confirm that the downloaded file is what you were expecting.
If you are interested, you can repeat the above steps but replace
<backup_db_via_tools> with <backup_conf_via_tools>, and instead of
creating an sql file, create a tar file.
To test packaging: run koha-dump, confirm that it still creates a
usable backup.
------
This signoff contains two changes:
10-1. If no backup/conf files were present, then the message telling you
so doesn't appear and the download button does. Made them behave
correctly.
10-2. The test for a file existing required it to be owned by the
webserver UID. This change makes it so it only has to be readable.
Signed-off-by: Robin Sheat <robin@catalyst.net.nz>
Adds a separate tab on tools/export.pl for exporting authority records.
To test:
1. Try exporting authority records from the "Export data" tool
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Tried different exports and everything worked nicely:
- removing fields from export (example: 942)
- limiting to one authority type
- changing the given file name
- exporting as MARC or XML
When a Koha user has superlibrarian permissions, and if
IndependantBranches is enabled, in Tools > Export bibliographic and
holdings, this user can't export all biblio records. He will just export
the biblio from the branch he belongs to.
This patch fix that.
To test:
- Do a ALL records export as superlibrian on a multi-branches Koha instance
- Examine the resulting MARC file with the tool of your choice (MArcEdit,
yaz-marcdump, etc.)
- You will see that 952/995 contains exlusively items from your branch (as
Koha user)
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
use encoding(UTF-8) rather than utf-8 for stricter
encoding
Marking output as ':utf8' only flags the data as utf8
using :encoding(UTF-8) also checks it as valid utf-8
see binmode in perlfunc for more details
In accordance with the robustness principle input
filehandles have not been changed as code may make
the undocumented assumption that invalid utf-8 is present
in the imput
Fixes errors reported by t/00-testcritic.t
Where feasable some filehandles have been made lexical rather than
reusing global filehandle vars
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
This patch solves the situation that news is in another language than
the Koha interface AND makes that themelanguage routine is always called
the same way in order to prevent mixed display.
It fixes also a bug related to language preselection based on web
browser prefered language.
September 9: Adjusted with input of Frederic Demians.
Septembre 10: Avoid circular dependency, as pointed by Chris Cormack.
Templates related functions are moved from C4::Output to C4::Templates
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ian Walls <ian.walls@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Some conditions added AND items.biblionumber=biblio/biblionumber
where JOIN USING (biblionumber) already added that.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
If you entered low number and high number, you got only items that *exactly* matched either entry (if any).
If you enter only a low number, you got everying *lower* than that.
If you enter only a high number, you get everything *higher* than that.
This was a greater-than-less-than problem.
Signed-off-by: Nicole Engard <nengard@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
IndependantBranches set and user is kohaadmin
I think we can escape defaulting to the userenv Branch with
restricting the list to the user branch
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Before this patch, we used to test for flags == 1, which was wrong when patron had all privileges.
This patch just adds a %2 to check that patron has superlibrarian privilege, and maybe something else we don't care.
I think I fixed it everywhere except in acquisition, that will be addressed by BibLibre new acquisition module.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This patch adds default blank values for a couple of params the script might receive which were producing warns in the error log. Also wraps GetMarcBiblio in an eval.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This patch starts improving export.pl by adding warnings, skipping invalid records, and not trying to access fields which don't exist.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Most Perl scripts (as opposed to modules) do
not need to require Exporter.
No user-visible or documentation changes.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
