Also addresses QA test tools, perldoc in ImportExportFramework.pm
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch fixes the comments from Comment 29 and the import
functionality. You should now be able to import an exported file without
editing the file at all and the authority type code will be overwritten
in the file (same behaviour as biblio frameworks).
Sponsored-by: Catalyst IT
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Mazen Khallaf <mazen.i.khallaf@gamil.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch amends C4::ImportExportFramework to work for authority types as well as MARC frameworks.
New file: admin/import_export_authtype.pl
Update: Ensuring we are passing the right column to the right tables.
Update2: Making the error messages the same to be consistent with patch on Bug 15665
Update3: Fixing merge conflicts
Update4: Fixing merge conflicts and removing tabs
Update5: Getting rid of warns, making sure Import and Export of default
authority will work
Update6: Merge conflicts and making sure export of default auth type
works
Update7: Fixing merge conflicts and updating buttons to bootstrap3
To test:
1) Go to Admin -> Authority types
2) Confirm there are two new columns 'Export' and 'Import' in the table
3) Click 'Export' on an existing authority type and choose a file type, click 'Export'
4) Confirm that the authority type is exported as your chosen file type. Save the file
5) Create a new authority type
6) Import into your new authority type using the file you just exported
7) Confirm you are taken to auth_tag_structure.pl
8) Go back to Authority types
9) Export your new authority type. View the exported file and confirm
the authtypecode has been updated to match the code you set for the new
auth type
10) Click 'Import' next to any existing authority type and attempt to import a file that is not XML, CSV or ODS. Confirm that this fails and you are asked to import a file of the correct file type
11) Confirm Export and Import work for the Default authority type
12) Go to Admin -> MARC bibliographic framework
13) Confirm that the 'Export' and 'Import' functions still work here as well
Sponsored-by: Catalyst IT
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Mazen Khallaf <mazen.i.khallaf@gamil.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We are using Koha::Logger when it makes sense to keep the info,
otherwise we simply remove it
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 28572: Replace missing occurrence in misc/admin/koha-preferences
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
If isurl is set to undef in the DB, it will be exported as an empty
string, which is an invalid value for isurl (int)
Incorrect integer value: '' for column 'isurl'
Test plan:
Export framework structure in CSV and ODS, then reimport it and check that
all the subfields are important correctly
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
If a cell contains an empty string it was not taken into account.
Test plan:
Same as CSV but with ODS
Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The CSV import of the biblio frameworks is broken.
A change in LibreOffice resulted in CSV not formatted how the
C4::ImportExportFramework code is expected.
This code is quite broken, it expects all the CSV cells to be quoted.
The "add tests" patch shows the different between what we expected
before this patch, and what is the new version generated by LibreOffice.
Test plan:
Export a biblio framework in CSV
Open it with LibreOffice and save it undef a different name
Create a new biblio framework and use the new file to construct it
Compare the 2 frameworks and make sure they are identical
Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Remove a bit of the confusion by removing unless code (not related to
mysql)
Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
TEST PLAN
---------
1) Log in to staff client
2) Click Koha administration
3) Click MARC bibliographic framework
4) export default as xml
5) Read the error log
-- it's floody and export is default framework
6) export another framework as xml
7) Read the error log
-- it's floody and export is the correct framework
8) apply the patch
9) repeat steps 4-7
-- expected frameworks exported, and no flooding.
10) run koha qa test tools
Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Bug 9701 added 2 new columns to the table marc_tag_structure which broke
the import of biblio frameworks if the CSV file was generated with a
previous version.
Test plan:
- Export a biblio framework from 17.11
- Import it into 18.05+
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
https://bugs.koha-community.org/show_bug.cgi?id=21740
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Logs are full of:
Use of uninitialized value in substitution (s///) at
C4/ImportExportFramework.pm line 282.
Use of uninitialized value in concatenation (.) or string at
C4/ImportExportFramework.pm line 283.
Some field are NULL in DB, we need to init the value to an empty string
to avoid these warnings.
Note: This fix is not related to the original bug
Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Warnings avoided
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Mainly a
perl -p -i -e 's/^.*3.07.00.049.*\n//' **/*.pm
Then some adjustements
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
perl -p -i -e 's/^(use vars .*)\$VERSION\s?(.*)/$1$2/' **/*.pm
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
http://bugs.koha-community.org/show_bug.cgi?id=9987
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
The SQL option for MARC framework imports was subject to a bug whereby
somebody could use it to gain access to arbitrary information in the
database by uploading an SQL file containing unexpected statements.
As it is difficult to securely sanitize SQL, this patch removes the
option to use SQL as an import or export format.
To test:
[1] Verify that SQL no longer appears as an import or export option
for the MARC frameworks.
[2] Verify that exports and imports in CSV, Excel XML, and ODS formats
still work.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Works as advertised. The UI doesn't offer exporting/importing in the SQL format.
Crafting the URL to export SQL fallbacks to a spreadsheet format (ODS).
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described, passes all tests and QA script.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
RM note: Digest::MD5 is used in C4::ImportExportFramework as part
of an unnecessary reimplementation of functionality supplied by
File::Temp. See bug 10991 for a proposal to remove it.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bareword file handle opened at line 558, column 17. See pages 202,204 of PBP. (Severity: 5)
Two-argument "open" used at line 558, column 17. See page 207 of PBP. (Severity: 5)
http://bugs.koha-community.org/show_bug.cgi?id=6679
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Aleksa Vujicic <aleksa@catalyst.net.nz>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended to replace some copy-and-paste comments only with consent of MJR.
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Step to reproduce :
- export default framework as SQL
- create new framework
- Import the exported SQL file into this new framework.
Go to default framework :
It disappeared.
If some libraries want to share default frameworks, then it is a major bug.
This patch fixes the test in order to cope with 0 character framework name.
Signed-off-by: François Charbonnier <francois.charbonnier@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Function _export_table_csv modified to remove CR/LF in the data
from the database to built a correct csv file.
Function _import_table_csv modified to deal with CR/LF in the data
field. When a line is found with an unproper end (a data field has not
end quote), it's concatenated to a temporary buffer until the next
line ends ok and then it's proccessed.
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Ian Walls <ian.walls@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Module to Import/Export a Framework structure to CSV/Excel-xml/ODS/SQL in Intranet Administration - MARC Frameworks section.
There are two new links: "Export" to export to a format; and "Import" to import from a file.
The data exported/imported is the one stored in the MySQL tables marc_tag_structure, marc_subfield_structure.
Exported works as follows:
1) CSV: As this format only allows one worksheet, the data from the tables is splitted with a row with #-# cells or with the
names of the fields of the next MySQL table. Each row has as much cells as fields has the MySQL table. The first row contains the
field names, the remaining holds the data.
2) Excel: Excel xml 2003 format. Each MySQL table has its own worksheet in the spreadsheet. Rows and cells data as CSV.
3) ODS: OpenDocument Spreadsheet compressed format, creates a temporary directory to generate the files needed to create the zip file.
Each MySQL table has its own worksheet in the spreadsheet. Rows and cells data as CSV.
4) SQL: Text file, the first row for each table is a delete and the remaining are inserts.
Importing reads the rows from the spreadsheet/text-file as follows:
1) CSV: Each row inserts or updates the associated MySQL table for this framework. At the end of the importing for a MySQL table, deletes the rows in the database that don't possess a correspondence with the spreadsheet.
2) Excel: Imports each worksheet to the associated MySQL table. Works as the CSV for each worksheet.
3) ODS: Creates a temporary directory to decompress and read the content.xml. This file has the data needed to import.
Works as the CSV for each worksheet.
4) Executes the SQL file.
If the file imported has a different frameworkcode that the framework importing, the framecode is changed along the process.
The Csv format will be the default.
It uses perl module Archive::Zip or zip/unzip system command to process ODS files.
To parse the sql files when importing it uses SQL::Statement or homemade parsing.
Signed-off-by: Nicole C. Engard <nengard@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
