Commit graph

5 commits

Author SHA1 Message Date
Jonathan Druart
e20270fec4 Bug 11944: use CGI( -utf8 ) everywhere
Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
2015-01-13 13:07:21 -03:00
40d3800cc4 Bug 9114: Make frameworks import/export routines correctly use UTF-8
Currently both the import_export_framework.pl script outputs data with
Perl's default encoding, ISO-8859. This patch properly sets
the binmode to UTF-8 when exporting SQL and CSV files using the PerlIO
layer (":encoding(UTF-8)") for STDOUT.

To test:

Export step test
- Use some ASCII character(s) with DIACRITICS in some field description
  in a chosen framework.
- Export the framework at Administration > MARC frameworks
- Run this to check the file is ISO-8859 encoded:
 $ file export_XXX.csv
 export_XXX.csv: ISO-8859 text, with very long lines
 (Note: try SQL and other output formats too. But not ODS)
- Apply the patch
- Export the framework again (change the name), and test encoding:
 $ file export_XXX_2.csv
 export_XXX_2.csv: UTF-8 Unicode text

Import step test
I assume you have two files, export_XXX.csv (ISO-8859 encoded) and
export_XXX_2.csv (XXX will depend on your framework's code)
- Reset your testing branch to master
- Import export_XXX.csv
- The string with non-ASCII chars is truncated at the first non-ASCII
  char's position (Note: this is the current behaviour).
- Import export_XXX_2.csv
- The non-ASCII chars are broken, the logs show errors on non-UNICODE
  chars.  (Note: even thou UTF-8 is the expected encoding it is
  treated as ISO-8859).
- Apply the patch
- Import the good (UTF-8 as expected) file and check everything worked
  as expected.

No double encoding should occur with either combination of formats.

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Magnus Enger <digitalutvikling@gmail.com>
I put some Norwegian and accented letters in a fremawork to test.
Before the patch, the exported CSV came out as ISO-8859, after the
patch it came out as UTF-8. ODS and XML (viewed in LibreOffice)
both looked good, before and after the patch.

Importing the ISO-8859 CSV cut off the strings at the first non-ASCII
char. Importing the UTF-8 CSV worked as epected.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as expected, passes tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-03-14 14:37:31 +00:00
Galen Charlton
94e349ff6c Bug 11666: remove SQL as an option for MARC framework exports and imports
The SQL option for MARC framework imports was subject to a bug whereby
somebody could use it to gain access to arbitrary information in the
database by uploading an SQL file containing unexpected statements.

As it is difficult to securely sanitize SQL, this patch removes the
option to use SQL as an import or export format.

To test:

[1] Verify that SQL no longer appears as an import or export option
    for the MARC frameworks.
[2] Verify that exports and imports in CSV, Excel XML, and ODS formats
    still work.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Works as advertised. The UI doesn't offer exporting/importing in the SQL format.
Crafting the URL to export SQL fallbacks to a spreadsheet format (ODS).

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described, passes all tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-02-05 19:48:27 +00:00
Galen Charlton
03d4ed2468 Bug 11666: add permission check for MARC framework import/export
This patch makes the MARC framework import/export script require
that the staff user be logged in with appropriate permissions for
managing the MARC frameworks.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>

I can confirm the bug and the solution. After applying the patch
downloading the file without logging in first is no longer possible.
Also passes tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
2014-02-05 19:47:53 +00:00
Salvador Zaragoza Rubio
689bc19154 Bug 5478 - Automate MARC framework import/export
Module to Import/Export a Framework structure to CSV/Excel-xml/ODS/SQL in Intranet Administration - MARC Frameworks section.
    There are two new links: "Export" to export to a format; and "Import" to import from a file.
    The data exported/imported is the one stored in the MySQL tables marc_tag_structure, marc_subfield_structure.

    Exported works as follows:
    1) CSV: As this format only allows one worksheet, the data from the tables is splitted with a row with #-# cells or with the
    names of the fields of the next MySQL table. Each row has as much cells as fields has the MySQL table. The first row contains the
    field names, the remaining holds the data.
    2) Excel: Excel xml 2003 format. Each MySQL table has its own worksheet in the spreadsheet. Rows and cells data as CSV.
    3) ODS: OpenDocument Spreadsheet compressed format, creates a temporary directory to generate the files needed to create the zip file.
    Each MySQL table has its own worksheet in the spreadsheet. Rows and cells data as CSV.
    4) SQL: Text file, the first row for each table is a delete and the remaining are inserts.

    Importing reads the rows from the spreadsheet/text-file as follows:
    1) CSV: Each row inserts or updates the associated MySQL table for this framework. At the end of the importing for a MySQL table, deletes the rows in the database that don't possess a correspondence with the spreadsheet.
    2) Excel: Imports each worksheet to the associated MySQL table. Works as the CSV for each worksheet.
    3) ODS: Creates a temporary directory to decompress and read the content.xml. This file has the data needed to import.
    Works as the CSV for each worksheet.
    4) Executes the SQL file.
    If the file imported has a different frameworkcode that the framework importing, the framecode is changed along the process.

    The Csv format will be the default.
    It uses perl module Archive::Zip or zip/unzip system command to process ODS files.
    To parse the sql files when importing it uses SQL::Statement or homemade parsing.

Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
2011-04-01 21:17:33 +13:00