System preference 'CSVdelimiter' has a special case for tabulation.
Preference value contains string 'tabulation' but string '\t' must be used in CSV file.
This is OK in many places, for exemple Bug 17590.
This patch adds C4::Context->csv_delimiter to add a uniq metod dealing
with this behavior.
Also create Koha::Template::Plugin::Koha->CSVDelimiter for calls from
Toolkit Templates.
Test plan :
1) Set system preference 'CSVdelimiter' = 'tabs'.
2) Create CSV export in impacted pages
3) Check columns are separated by tabulation character and not string 'tabulation'
4) Check with another delimiter
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test Plan:
1. Go to Administration
2. Go to System Preferences
3. Find 'delimiter' (note the name)
4. Apply patch
5. Repeat step 1 and 2
6. Find 'CSVDelimiter' (note the name)
Signed-off-by: Sally <sally.healey@cheshiresharedservices.gov.uk>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Here we go, next step then.
As we did not fix the performance issue when autofiltering
the variables (see bug 20975), the only solution we have is to add the
filters explicitely.
This patch has been autogenerated (using add_html_filters.pl, see next
pathces) and add the html filter to all the variables displayed in the
template.
Exceptions are made (using the new 'raw' TT filter) to the variable we
already listed in the previous versions of this patch.
To test:
- Use t/db_dependent/Koha/Patrons.t to populate your DB with autogenerated
data which contain <script> tags
- Remove them from borrower_debarments.comments (there are allowed here)
update borrower_debarments set comment="html tags possible here";
- From the interface hit page and try to catch alert box.
If you find one it means you find a possible XSS.
To know where it comes from:
* note the exact URL where you found it
* note the alert box content
* Dump your DB and search for the string in the dump to identify its
location (for instance table.field)
Next:
* Ideally we would like to use the raw filter when it is not necessary
to HTML escape the variables (in big loop for instance)
* Provide a QA script to catch missing filters (we want html, uri, url
or raw, certainly others that I am forgetting now)
* Replace the html filters with uri when needed (!)
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Bonus fixes:
- Basketgroup: there was an unnecessary space.
" RRP tax excluded" replaced by "RRP tax excluded"
- Late orders:
- leading spaces before every line
- surrounding spaces for [% orders.size %]
Other changes:
- quoted "Total orders in late" to no worry about additional spaces
- Late orders: additional spaces which shouldn't be a problem because
all the fields are enclosed by quotes.
Test plan:
1. Set syspref "delimiter" to "#'s"
2. Change the language e.g. FR-fr
3. Create a vendor with minimal info
4. Create a basket with two items
5. Go to the basket. URL should be
/cgi-bin/koha/acqui/basket.pl?basketno=XXXXX
6. Close this basket
7. "Export as CSV"
8. You should see that the file contains commas "," This is the bug.
Leave the file open for future comparison to ensure that there are
no regressions.
9. Create a basket group with two baskets
(tick the close this basket group check box)
10. Go to the "Closed" tab to see your basket group
11. "Export as CSV"
12. You should see that the file contains commas "," This is the bug.
Leave the file open for future comparison to ensure that there are
no regressions.
13. Go to /cgi-bin/koha/acqui/lateorders.pl
14. Tick two orders. Of the same vendor, otherwise Koha won't let you tick
orders of different vendors.
15. "Export as CSV"
16. You should see that the file contains commas "," This is the bug.
Leave the file open for future comparison to ensure that there are
no regressions.
17. Apply this patch
18. Translation stuff
1. cd misc/translator
2. perl translate update YOUR_LANG
3. manually edit the po file, remove the fuzzy tags.
And translate the string. You can just copy the original and
replace some parts by gibberish.
4. perl translate install YOUR_LANG
19. Re-export the same CSVs and compare with the original version to check that
- delimiter syspref is honored
- the headers are translatable
- there is no whitespace regression (additional newlines or spaces)
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
I am going to try and explain the problem here in order to make it a
have-to-read for next changes.
There are several things to test when and most of the time we break
something when we try to fix something else.
You have to test different CSV export, not only one. There are not all
processed the same way. For instance acqui/csv/basket.tt does not
contain any strings to translate and the translate script will copy the
original file as it.
But reports/csv/orders_by_budget.tt will not! Indeed it contains "TOTAL"
which will be translated. The generated tt file for the translated
language will remove all the carriage returns! That's why we cannot rely
on TT for newlines (contrary to bug 16914 assumption).
There are two possible methods:
1/ Fix the translate script
=> Hum... nope
2/ Remove all carriage returns and make them explicit by including an
file that only contains 1 carriage return
This second method is implemented in this patch.
How it works: Use the PRE_CHOMP "[%-" and POST_CHOMP "-%]"
(http://www.template-toolkit.org/docs/manual/Config.html#section_PRE_CHOMP_POST_CHOMP)
to remove all the newlines that could be added by TT in the original
(en) files. Then include the new_line.inc to add a new line.
That way original and translated files will behave the same way.
The BLOCK in the csv_headers avoid to have the newlines added, in any
cases. For instance: by default we will have an empty line at the end of
the headers, but the translated headers will not have it.
Test plan:
At least 2 signoffs will be needed, please test carefully!
You will need to generate CSV with at least 2 entries!
Test with the non-translated interface (en) and with the language you
want. To test with a translated language you will *have to*:
- cd misc/translator
- perl translate update LANG
- manually edit the po file, remove the fuzzy tags, correctly translate
the string (do not forget the additional %s in the headers)
- perl translate install LANG
1/ Export basket and basketgroup as CSV
2/ Export late orders as CSV
3/ Export items (from the item search) as CSV
4/ Export the 2 reports "Orders by fund" and "Cash register" as CSV
Signed-off-by: Lee Jamison <ldjamison@marywood.edu>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This bug adds the "vendor note" for each order in the PDF for
basketgroups. The note is displayed only if it exists, just under the
bibliographic information.
I added a separation line "--------" between bibliographic information
and the note, so that it could be visible at 1st glance.
It also replaces the internal note with the vendor in the CSV for basket
and basketgroup. It is more logical and useful for libraries to export
the note made for vendor, as those files are destined to be sent to the
vendor.
Test plan :
- fill a basket with some orders, some with internal notes, some with
vendor notes
- export the basket in CSV : only the vendor notes should be present
- put the basket in a basketgroup
- export the basketgroup in CSV : only the vendor notes should be
present
- Select "English-2 pages" template for basketgroups in Sysprefs
- export the basket in PDF : the vendor notes should be present under
the bibliographic information
- Select "English-3 pages" template for basketgroups in Sysprefs
- export the basket in PDF : the vendor notes should be present under
the bibliographic information
Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
This followup answer QA remarks :
- neworderempty.pl updated so that the 2 new variables are passed
to the template
- modordernotes.tt fixed to make the translation easier
- in CSV headers, to make clear that no change are made for the moment,
rename "note" to "internal note"
Additionnaly, "Publisher code" was wrong in the csv headers. I changed
it to "Publisher" (the field in database is publishercode, but the
content is a real publisher name, not a code)
I did not change "Note:" in modordernotes.tt, because it is just under
a h1 tag which specifies the type of note the librarian is editing.
Test plan :
- edit an existing order, and try to change/add/delete the vendor note,
and the internal note. Check the changes are properly saved
- export a basket and a basketgroup in CSV. Check the columns headers
are "Publisher" and "Vendor note"
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Fixed some tabs. Passes QA script and tests.
Tested:
- add notes when creating an order
- edit notes modifying an order line
- edit notes using the links on the basket summary
- check basket CSV export
- close basket
- check basket group CSV export
- edit notes on order receive page using the links
- edit notes on receive
Note: Translatability of templates could be improved by a follow-up.
It's better not to divide up sentences with if/else structures.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
To reproduce:
- cd misc/translator
- ./translate update LANG
- ./translate install LANG
- go to the Koha mainpage and change the language.
- go to acqui/basketgroup.pl?booksellerid=XX and try to export a
basketgroup.
The headers is followed by the first basketgroup information. There is
no carriage return.
It looks like it is caused by a routine used by the translator script
(TmplTokenizer::string_canon).
To test this patch:
- apply it
- cd misc/translator
- ./translate -f update LANG
- translate headers in your po file
- ./translate -f install LANG
- go to acqui/basketgroup.pl?booksellerid=XX and try to export a
basketgroup.
- verify that the csv looks good now.
- same thing for basket.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Good idea and seems to work - just fixing a small glitch
with the first entry of the list in a follow-up.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>