Tomas Cohen Arazi
40d3800cc4
Currently both the import_export_framework.pl script outputs data with
Perl's default encoding, ISO-8859. This patch properly sets
the binmode to UTF-8 when exporting SQL and CSV files using the PerlIO
layer (":encoding(UTF-8)") for STDOUT.
To test:
Export step test
- Use some ASCII character(s) with DIACRITICS in some field description
in a chosen framework.
- Export the framework at Administration > MARC frameworks
- Run this to check the file is ISO-8859 encoded:
$ file export_XXX.csv
export_XXX.csv: ISO-8859 text, with very long lines
(Note: try SQL and other output formats too. But not ODS)
- Apply the patch
- Export the framework again (change the name), and test encoding:
$ file export_XXX_2.csv
export_XXX_2.csv: UTF-8 Unicode text
Import step test
I assume you have two files, export_XXX.csv (ISO-8859 encoded) and
export_XXX_2.csv (XXX will depend on your framework's code)
- Reset your testing branch to master
- Import export_XXX.csv
- The string with non-ASCII chars is truncated at the first non-ASCII
char's position (Note: this is the current behaviour).
- Import export_XXX_2.csv
- The non-ASCII chars are broken, the logs show errors on non-UNICODE
chars. (Note: even thou UTF-8 is the expected encoding it is
treated as ISO-8859).
- Apply the patch
- Import the good (UTF-8 as expected) file and check everything worked
as expected.
No double encoding should occur with either combination of formats.
Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Magnus Enger <digitalutvikling@gmail.com>
I put some Norwegian and accented letters in a fremawork to test.
Before the patch, the exported CSV came out as ISO-8859, after the
patch it came out as UTF-8. ODS and XML (viewed in LibreOffice)
both looked good, before and after the patch.
Importing the ISO-8859 CSV cut off the strings at the first non-ASCII
char. Importing the UTF-8 CSV worked as epected.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as expected, passes tests and QA script.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
97 lines
3.5 KiB
Perl
Executable file
97 lines
3.5 KiB
Perl
Executable file
#!/usr/bin/perl
|
|
|
|
# Copyright 2010-2011 MASmedios.com y Ministerio de Cultura
|
|
#
|
|
# This file is part of Koha.
|
|
#
|
|
# Koha is free software; you can redistribute it and/or modify it under the
|
|
# terms of the GNU General Public License as published by the Free Software
|
|
# Foundation; either version 2 of the License, or (at your option) any later
|
|
# version.
|
|
#
|
|
# Koha is distributed in the hope that it will be useful, but WITHOUT ANY
|
|
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
|
|
# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
|
|
#
|
|
# You should have received a copy of the GNU General Public License along
|
|
# with Koha; if not, write to the Free Software Foundation, Inc.,
|
|
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
|
|
|
|
|
use strict;
|
|
use warnings;
|
|
use CGI;
|
|
use CGI::Cookie;
|
|
use C4::Context;
|
|
use C4::Auth qw/check_cookie_auth/;
|
|
use C4::ImportExportFramework;
|
|
|
|
my %cookies = CGI::Cookie->fetch();
|
|
my $authenticated = 0;
|
|
my ($auth_status, $sessionID);
|
|
if (exists $cookies{'CGISESSID'}) {
|
|
($auth_status, $sessionID) = check_cookie_auth(
|
|
$cookies{'CGISESSID'}->value,
|
|
{ parameters => 'parameters_remaining_permissions' },
|
|
);
|
|
}
|
|
if ($auth_status eq 'ok') {
|
|
$authenticated = 1;
|
|
}
|
|
|
|
my $input = new CGI;
|
|
|
|
unless ($authenticated) {
|
|
print $input->header(-type => 'text/plain', -status => '403 Forbidden');
|
|
exit 0;
|
|
}
|
|
|
|
my $frameworkcode = $input->param('frameworkcode') || '';
|
|
my $action = $input->param('action') || 'export';
|
|
|
|
## Exporting
|
|
if ($action eq 'export' && $input->request_method() eq 'GET') {
|
|
my $strXml = '';
|
|
my $format = $input->param('type_export_' . $frameworkcode);
|
|
ExportFramework($frameworkcode, \$strXml, $format);
|
|
|
|
if ($format eq 'csv') {
|
|
# CSV file
|
|
|
|
# Correctly set the encoding to output plain text in UTF-8
|
|
binmode(STDOUT,':encoding(UTF-8)');
|
|
print $input->header(-type => 'application/vnd.ms-excel', -attachment => 'export_' . $frameworkcode . '.csv');
|
|
print $strXml;
|
|
} elsif ($format eq 'excel') {
|
|
# Excel-xml file
|
|
print $input->header(-type => 'application/excel', -attachment => 'export_' . $frameworkcode . '.xml');
|
|
print $strXml;
|
|
} else {
|
|
# ODS file
|
|
my $strODS = '';
|
|
createODS($strXml, 'en', \$strODS);
|
|
print $input->header(-type => 'application/vnd.oasis.opendocument.spreadsheet', -attachment => 'export_' . $frameworkcode . '.ods');
|
|
print $strODS;
|
|
}
|
|
## Importing
|
|
} elsif ($input->request_method() eq 'POST') {
|
|
my $ok = -1;
|
|
my $fieldname = 'file_import_' . $frameworkcode;
|
|
my $filename = $input->param($fieldname);
|
|
# upload the input file
|
|
if ($filename && $filename =~ /\.(csv|ods|xml)$/i) {
|
|
my $extension = $1;
|
|
my $uploadFd = $input->upload($fieldname);
|
|
if ($uploadFd && !$input->cgi_error) {
|
|
my $tmpfilename = $input->tmpFileName($input->param($fieldname));
|
|
$filename = $tmpfilename . '.' . $extension; # rename the tmp file with the extension
|
|
$ok = ImportFramework($filename, $frameworkcode, 1) if (rename($tmpfilename, $filename));
|
|
}
|
|
}
|
|
if ($ok >= 0) { # If everything went ok go to the framework marc structure
|
|
print $input->redirect( -location => '/cgi-bin/koha/admin/marctagstructure.pl?frameworkcode=' . $frameworkcode);
|
|
} else {
|
|
# If something failed go to the list of frameworks and show message
|
|
print $input->redirect( -location => '/cgi-bin/koha/admin/biblio_framework.pl?error_import_export=' . $frameworkcode);
|
|
}
|
|
}
|