it fixes xgettext and (maybe) friends
[12:22:29] Error: Command failed: misc/translator/xgettext.pl --charset=UTF-8 -s -o /tmp/koha-Jaa9rf/Koha-marc-NORMARC.pot -f /tmp/koha-Jaa9rf/files
/tmp/koha-Jaa9rf/Koha-marc-NORMARC.pot at misc/translator/xgettext.pl line 387.
Use of uninitialized value in subroutine entry at misc/translator/xgettext.pl line 388.
Argument ">:encoding(utf-8)" isn't numeric in subroutine entry at misc/translator/xgettext.pl line 388.
Argument "/tmp/koha-Jaa9rf/Koha-marc-NORMARC.pot" isn't numeric in subroutine entry at misc/translator/xgettext.pl line 388.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
misc/translator/translate was doing three different things:
- extract translatable strings
- create or update PO files
- install translated templates
This patch separates responsibilities by moving the string extraction
code into several 'xgettext-like' scripts and adds gulp tasks to
automate string extraction and PO files update
This has several benefits:
- gulp runs tasks in parallel, so it's a lot faster (updating all PO
files is at least 10 times faster with my 4-cores CPU)
- there is no need for $KOHA_CONF to be defined
LangInstaller.pm relied on $KOHA_CONF to get the different paths
needed. I'm not sure why, since string extraction and PO update should
work on source files, not installed files
- string extraction code can be more easily tested
This patch also brings a couple of fixes and improvements:
- TT string extraction (strings wrapped in [% t(...) %]) was done with
Template::Parser and PPI, which was extremely slow, and had some
problems (see bug 24797).
This is now done with Locale::XGettext::TT2 (new dependency) which is
a lot faster, and fixes bug 24797
- Fix header in 4 PO files
For backward compatibility, 'create' and 'update' commands of
misc/translator/translate can still be used and will execute the
corresponding gulp task
Test plan:
1. Run `yarn install` and install Locale::XGettext::TT2
2. Run `gulp po:update`
3. Verify the contents of updated PO files
4. Run `cd misc/translator && ./translate install <lang>`
5. Verify that all (templates, sysprefs, xslt, installer files) is
correctly translated
6. Run `gulp po:create --lang <lang>` and verify that it created all PO
files for that language
7. Run `prove t/misc/translator`
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Need to install yarn & gulp, no errors
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds aria-label to the allowlist for translations.
To test:
- Apply patches from here and bug 25244
- kshell
- cd misc/translator
- perl translate update de-DE
- Check for the aria-label in the diff
- Verify the strings appear in the po files now
- Translate them
- perl translate install de-DE
- Verify the translated strings appear in the template
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch removes some new error cases introduced during rebase
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds a .perlcriticrc (copied from qa-test-tools) and fixes
almost all perlcrictic violations according to this .perlcriticrc
The remaining violations are silenced out by appending a '## no critic'
to the offending lines. They can still be seen by using the --force
option of perlcritic
This patch also modify t/00-testcritic.t to check all Perl files using
the new .perlcriticrc.
I'm not sure if this test script is still useful as it is now equivalent
to `perlcritic --quiet .` and it looks like it is much slower
(approximatively 5 times slower on my machine)
Test plan:
1. Run `perlcritic --quiet .` from the root directory. It should output
nothing
2. Run `perlcritic --quiet --force .`. It should output 7 errors (6
StringyEval, 1 BarewordFileHandles)
3. Run `TEST_QA=1 prove t/00-testcritic.t`
4. Read the patch. Check that all changes make sense and do not
introduce undesired behaviour
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1) Create translation files for a new language
( cd misc/translator; ./translate create xx-YY)
A new language means one that isn't already in Koha, xx-YY=>something you
invent.
2) Verify double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check strange strings
3) Apply the patch
4) Create po files again
( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY)
5) Verify no more double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check normal string
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch fix the missing xml prolog in translated
files, XML or TT.
Is fixed teaching C4::TTParse not to ignore <?..?> constructs,
then teaching xgettext.pl to ignore those strings. Net result is
that they are copied in the translated file.
To test:
1) Update & install your preferred language,
(cd misc/translator/; perl translate update xx-YY; perl translate install xx-YY )
2) Compare the first lines (head -2) of:
koha-tmpl/opac-tmpl/bootstrap/en/xslt/compact.xsl
koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-opensearch.tt
koha-tmpl/intranet-tmpl/prog/en/xslt/plainMARC.xsl
and
koha-tmpl/opac-tmpl/bootstrap/xx-YY/xslt/compact.xsl
koha-tmpl/opac-tmpl/bootstrap/xx-YY/modules/opac-opensearch.tt
koha-tmpl/intranet-tmpl/prog/xx-YY/xslt/plainMARC.xsl
Check the missing prolog
3) Install this patch, repeat 1 and 2, now the prolog is present
on translated files.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
I am not sure what happened here, but adding strict for TmplTokenizer.pm
breaks a test for translation.
It has been caught by a failing test:
% prove xt/author/translatable-templates.t
will display lines "invalid multibyte sequence"
I did a diff between the generated files and found:
< msgid "Ã rbok"
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch implements the replacement of TTvariables
'[%...%]' with normal placeholders '%s' for text
inside HTML tags (eg. img, meta, input)
The replacement is done in two files:
* xgettext.pl, used on create/update, so TTvars are not
found on the text to translate (msgid)
* tmpl_proccess3.pl, used on install time, so the correct
translation is found (msgstr) and the right TTvar is put on
the translated text.
To test:
Before applying the patch
1) Update your favorite language
cd misc/traslator
perl translate update xx-YY
2) Build a list (ini.txt) of 'msgid' to do a comparison
msgcat --no-wrap --use-first po/xx-YY-*po -o - | egrep "^msgid" | sort > ini.txt
3) Apply the patch
4) Update your favorite language, again
cd misc/traslator
perl translate update xx-YY
5) Build a final list (end.txt) of 'msgid' to do a comparison
msgcat --no-wrap --use-first po/xx-YY-*po -o - | egrep "^msgid" | sort > end.txt
6) Compare both files, you must find (most) TTvars replaced by '%s'
diff ini.txt end.txt
for example:
< msgid "Translate item type [% itemtype.itemtype | html %]"
> msgid "Translate item type %s"
< msgid "Holds on this item: [% item_loo.item_holds | html %] / Total holds on this record: [% item_loo.holds | html -%]"
> msgid "Holds on this item: %s / Total holds on this record: %s"
< msgid "tag_anchor_[% innerloo.tag | html %]_[% innerloo.index | html %][% innerloo.random | html %]"
> msgid "tag_anchor_%s_%s%s"
< msgid "[% innerloo.tag_lib | html %] - Click to Expand this Tag"
> msgid "%s - Click to Expand this Tag"
7) Translate one of the examples, with more than one variable if
possible, for example this text in xx-YY-staff-prog.po
> msgid "Holds on this item: %s / Total holds on this record: %s"
Check it's not marked as 'fuzzy'
8) Apply the translation
perl translate install xx-YY
9) Check the translated string has all variables in the right order
edit koha-tmpl/intranet-tmpl/prog/es-ES/modules/tools/batchMod-edit.tt
around line '187', first '[% item_loo.item_holds | html %]', then
'[% item_loo.holds | html -%]' on this case,
or use another example
If approved this patch can be backported to current stable
versions.
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Conflicts:
misc/translator/text-extract.pl
misc/translator/text-extract2.pl
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Perl 5.26 (or earlier) introduced a security feature, where implicitly
including the program directory as a Perl library directory no longer
happens (perl -I. ).
This causes translate to fail because it cannot find the *.pm -files in
it's own directory.
This patch adds the familiar mantra
use lib $FindBin::Bin;
to the relevant scripts.
To test:
1. Install Ubuntu18.04 or something else with Perl 5.26
2. Install Koha (we use the dev-install)
3. cd $KOHA_PATH/misc/translator/
4. perl translate create fi-FI
5. Observe problems with missing modules.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Cannot recreate the issue right now but the changes make sense.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch gives a partial solution for this problem.
It ignores strings like "[% something %]", but not
"[% IF ( value ) %][% value %][% END %]"
We get 100+ strings less to translate
To test:
1) Update translation files for your preferred language
2) Apply the patch
3) Update again
4) Compare, you must find 100+ strings removed from
translation files
eg.
-msgid "[% SEARCH_RESULT.biblionumber |url %]"
-msgid "[% accepteddate | $KohaDates %]"
-msgid "[% amountoutstanding | format('%.2f') %]"
-msgid "[% authtypetext |html %]"
-msgid "[% barcode_llx |html %]"
-msgid "[% barcode_lly |html %]"
-msgid "[% biblio.quantity.length ? biblio.quantity : 1 %]"
-msgid "[% billingdate | $KohaDates %]"
-msgid "[% borname |html %]"
...
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Verified all strings removed from the po files were
pure TT.
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Since we switched to Template Toolkit we don't need to stick with the
sufix we used for HTML::Template::Pro.
This patch changes the occurences of '.tmpl' in favour of '.tt'.
To test:
- Apply the patch
- Install koha, and verify that every page can be accesed
Regards
To+
P.S. a followup will remove the glue code.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Using perl v5.18, the following is warned on updating/installing
templates:
Smartmatch is experimental at ./translate line 54.
Moreover, the following should exist with previous perl versions:
perlcritic xgettext
Subroutine prototypes used at line 29, column 1. See page 194 of PBP.
(Severity: 5)
Subroutine prototypes used at line 44, column 1. See page 194 of PBP.
(Severity: 5)
Subroutine prototypes used at line 60, column 1. See page 194 of PBP.
(Severity: 5)
Subroutine prototypes used at line 72, column 1. See page 194 of PBP.
(Severity: 5)
Subroutine prototypes used at line 89, column 1. See page 194 of PBP.
(Severity: 5)
Variable declared in conditional statement at line 105, column 13.
Declare variables outside of the condition. (Severity: 5)
Subroutine prototypes used at line 127, column 1. See page 194 of PBP.
(Severity: 5)
Subroutine prototypes used at line 136, column 1. See page 194 of PBP.
(Severity: 5)
Variable declared in conditional statement at line 190, column 7.
Declare variables outside of the condition. (Severity: 5)
Subroutine prototypes used at line 237, column 1. See page 194 of PBP.
(Severity: 5)
Bareword file handle opened at line 238, column 5. See pages 202,204 of
PBP. (Severity: 5)
Two-argument "open" used at line 238, column 5. See page 207 of PBP.
(Severity: 5)
Subroutine prototypes used at line 277, column 1. See page 194 of PBP.
(Severity: 5)
Subroutine prototypes used at line 311, column 1. See page 194 of PBP.
(Severity: 5)
Bareword file handle opened at line 350, column 9. See pages 202,204 of
PBP. (Severity: 5)
Two-argument "open" used at line 350, column 9. See page 207 of PBP.
(Severity: 5)
Bareword file handle opened at line 353, column 5. See pages 202,204 of
PBP. (Severity: 5)
Bareword file handle opened at line 359, column 5. See pages 202,204 of
PBP. (Severity: 5)
Two-argument "open" used at line 359, column 5. See page 207 of PBP.
(Severity: 5)
perl -wc xgettext.pl
\1 better written as $1 at TmplTokenizer.pm line 474.
main::token_negligible_p() called too early to check prototype at
xgettext.pl line 52.
xgettext.pl syntax OK
Test plan:
Apply this patch and verify the warnings/errors does not appear anymore
and no regression is found on creating/updating/installing templates.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Work as described, no koha-qa errros.
Tested with perl v5.18.2
No warnings, no problem detected on create/update/install i18n files
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
This patch removes several types of strings from the
PO files that cannot be usefully translated, including
ones that consist entirely of punctuation and/or HTML entities.
Test:
1) Update PO files of some lang, xx-YY-*po
cd misc/translator
perl translate update xx-YY
2) Do it again, just in case
3) rm po/xx-YY*po~
4) Extract all msgid's, sorted
cat po/xx-YY*po | egrep "^msgid" | sort | uniq > xx-YY-pre
5) Apply the patch
6) Repeat 1-3
7) Repeat 4 again, other file
cat po/xx-YY*po | egrep "^msgid" | sort | uniq > xx-YY-post
8) Do a diff, inspect results, only strings with %s and \s
diff xx-YY-pre xx-YY-post | less
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works as described, 380 strings less to 'translate'
No koha-qa errors.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Tested according to test plan, works as described.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
To test:
git checkout 3.6.x
cd misc/translator
./tmpl_process3.pl update -i ../../koha-tmpl/opac-tmpl/prog/en/ \
-s ./po/fr-FR-i-opac-t-prog-v-3006000.po -r
po/fr-FR-i-opac-t-prog-v-3006000.po contains broken diacritics for NORMARC
strings for example
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
fixes regression caused by bug 6752
use encoding(UTF-8) rather than utf-8 for stricter
encoding
Marking output as ':utf8' only flags the data as utf8
using :encoding(UTF-8) also checks it as valid utf-8
see binmode in perlfunc for more details
In accordance with the robustness principle input
filehandles have not been changed as code may make
the undocumented assumption that invalid utf-8 is present
in the imput
Fixes errors reported by t/00-testcritic.t
Where feasable some filehandles have been made lexical rather than
reusing global filehandle vars
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
This is a fairly hacky solution, a counter patch would be more than
welcome
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Updated, translated and installed German po files after applying this patch.
No problems found.
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
This test validate Template Toolkit (TT) Koha files.
For the time being an unique validation is done: Test if TT files
contain TT directive within HTML tag. For example:
<li[% IF
This kind of constuction MUST be avoided because it break Koha
translation process.
This patch transform also translation specific modules into C4 modules
in order to be able to use them in test case:
C4::TTPaser
C4::TmplToken
C4::TmplTokenType
This patch is a Perl adaptation of a Haskell script from Frère Sébastien
Marie.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Notes on testing:
- translate install de-DE - worked ok
- translate update de-DE > translate install de-DE - worked ok
- running the test xt/tt_valid.t - worked ok and pointed out lots of problems.
Found no problems.
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
In opac/staff templates .po files, we have comment lines contextualizing
extracted text in templates. Path to template are absolute. For example, we can
have:
#: /home/katrin/kohaclone/koha-tmpl/intranet-tmpl/prog/en/...
modules/cataloguing/addbiblio.tmpl:585
The first part of the pathname is useless. With this patch, we just keep
relative path to tempalte from Koha template main directory. The above example
becomes:
#: intranet-tmpl/prog/en/modules/cataloguing/addbiblio.tmpl:585
To be applied on [3.2]
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
xsl strings are now taken into account.
This is a first draft.
Some parameters of the functions will be taken into account
and <xsl> tags are not translated
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
- If a string is enclosed by a tag, remove that tag from the extracted string
- Generate automatic comments to provide more information for the translator
- A couple bug fixes
(Actually, xgettext.pl did not know how to handle them in the files listed
in the list of files.)
If the po file is empty (corrupted), $href->{'""'} will be undefined.
We just blindly dereferenced this null value without checking.
when minor whitespace changes occur in the original templates; it also
makes the strings much easier to read (e.g., instead of "foo\n\n\t\t bar",
xgettext.pl will now always generate "foo bar" and tmpl_process3.pl will
understand it to be the same as the original string).
Early termination of analysis if we encounter some strings, such as </h1>
or | or ||, in order to avoid extracting strings that are unnecessarily
long and which doesn't add any meaningful context.
either iso8859-1 or utf8, msgmerge(1) won't crap out. The code is ugly;
the conversion table is hard-coded, and in some place not very appropriate.
However, this does fix the case where a few strings containing French
characters can't be translated. As a side effect, tmpl_process3 can now
also be used for French or other languages using iso8859-1.
word order is too different than the word order of the target language to
yield meaningful translations.
The new scripts use a different translation file format (namely standard
gettext-style PO files).
This seems to reasonably work (e.g., producing an empty en_GB translation
then installing seems to not corrupt the "translated" files), but it likely
will still contain some bugs. There is also little documentation, but try
to run perldoc on the .p[lm] files to see what's there. There are also some
spurious warnings (both from bugs in the new scripts and from buggy third-
party Locale::PO module).
