Koha-community
/
Koha
11
2
Fork 1
Code Issues Releases 445 Wiki Activity
Browse Source

Bug 25067: Move PO file manipulation code into gulp tasks 

misc/translator/translate was doing three different things:
- extract translatable strings
- create or update PO files
- install translated templates

This patch separates responsibilities by moving the string extraction
code into several 'xgettext-like' scripts and adds gulp tasks to
automate string extraction and PO files update

This has several benefits:

- gulp runs tasks in parallel, so it's a lot faster (updating all PO
  files is at least 10 times faster with my 4-cores CPU)

- there is no need for $KOHA_CONF to be defined
  LangInstaller.pm relied on $KOHA_CONF to get the different paths
  needed. I'm not sure why, since string extraction and PO update should
  work on source files, not installed files

- string extraction code can be more easily tested

This patch also brings a couple of fixes and improvements:

- TT string extraction (strings wrapped in [% t(...) %]) was done with
  Template::Parser and PPI, which was extremely slow, and had some
  problems (see bug 24797).
  This is now done with Locale::XGettext::TT2 (new dependency) which is
  a lot faster, and fixes bug 24797

- Fix header in 4 PO files

For backward compatibility, 'create' and 'update' commands of
misc/translator/translate can still be used and will execute the
corresponding gulp task

Test plan:
1. Run `yarn install` and install Locale::XGettext::TT2
2. Run `gulp po:update`
3. Verify the contents of updated PO files
4. Run `cd misc/translator && ./translate install <lang>`
5. Verify that all (templates, sysprefs, xslt, installer files) is
   correctly translated
6. Run `gulp po:create --lang <lang>` and verify that it created all PO
   files for that language
7. Run `prove t/misc/translator`

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Need to install yarn & gulp, no errors

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
tags/v20.11.00
Julian Maurice 1 year ago
committed by Jonathan Druart <jonathan.druart@bugs.koha-community.org>
parent
d5d736909c
commit
3cfc2ec7bd
23 changed files with 1247 additions and 977 deletions
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. +1
    -1
      cpanfile
  2. +121
    -0
      docs/development/internationalization.md
  3. +301
    -1
      gulpfile.js
  4. +86
    -693
      misc/translator/LangInstaller.pm
  5. +9
    -1
      misc/translator/po/dz-pref.po
  6. +9
    -1
      misc/translator/po/gd-pref.po
  7. +9
    -1
      misc/translator/po/lv-pref.po
  8. +9
    -1
      misc/translator/po/te-pref.po
  9. +3
    -119
      misc/translator/tmpl_process3.pl
  10. +16
    -42
      misc/translator/translate
  11. +158
    -0
      misc/translator/xgettext-installer
  12. +151
    -0
      misc/translator/xgettext-pref
  13. +56
    -0
      misc/translator/xgettext-tt2
  14. +16
    -1
      misc/translator/xgettext.pl
  15. +3
    -0
      package.json
  16. +0
    -109
      t/LangInstaller.t
  17. +14
    -0
      t/misc/translator/sample.pref
  18. +1
    -1
      t/misc/translator/sample.tt
  19. +15
    -0
      t/misc/translator/sample.yml
  20. +32
    -0
      t/misc/translator/xgettext-installer.t
  21. +54
    -0
      t/misc/translator/xgettext-pref.t
  22. +74
    -0
      t/misc/translator/xgettext-tt2.t
  23. +109
    -6
      yarn.lock

+ 1
- 1
cpanfile View File

@@ -146,6 +146,7 @@ recommends 'Gravatar::URL', '1.03';
recommends 'HTTPD::Bench::ApacheBench', '0.73';
recommends 'LWP::Protocol::https', '5.836';
recommends 'Lingua::Ispell', '0.07';
recommends 'Locale::XGettext::TT2', '0.7';
recommends 'Module::Bundled::Files', '0.03';
recommends 'Module::Load::Conditional', '0.38';
recommends 'Module::Pluggable', '3.9';
@@ -157,7 +158,6 @@ recommends 'Net::SFTP::Foreign', '1.73';
recommends 'Net::Server', '0.97';
recommends 'Net::Z3950::SimpleServer', '1.15';
recommends 'PDF::FromHTML', '0.31';
recommends 'PPI', '1.215';
recommends 'Parallel::ForkManager', '0.75';
recommends 'Readonly', '0.01';
recommends 'Readonly::XS', '0.01';


+ 121
- 0
docs/development/internationalization.md View File

@@ -0,0 +1,121 @@
# Internationalization

This page documents how internationalization works in Koha.

## Making strings translatable

There are several ways of making a string translatable, depending on where it
is located

### In Template::Toolkit files (`*.tt`)

The simplest way to make a string translatable in a template is to do nothing.
Templates are parsed as HTML files and almost all text nodes are considered as
translatable strings. This also includes some attributes like `title` and
`placeholder`.

This method has some downsides: you don't have full control over what would
appear in PO files and you cannot use plural forms or context. In order to do
that you have to use `i18n.inc`

`i18n.inc` contains several macros that, when used, make a string translatable.
The first thing to do is to make these macros available by adding

[% PROCESS 'i18n.inc' %]

at the top of the template file. Then you can use those macros.

The simplest one is `t(msgid)`

[% t('This is a translatable string') %]

You can also use variable substitution with `tx(msgid, vars)`

[% tx('Hello, {name}', { name = 'World' }) %]

You can use plural forms with `tn(msgid, msgid_plural, count)`

[% tn('a child', 'several children', number_of_children) %]

You can add context, to help translators when a term is ambiguous, with
`tp(msgctxt, msgid)`

[% tp('verb', 'order') %]
[% tp('noun', 'order') %]

Or any combinations of the above

[% tnpx('bibliographic record', '{count} item', '{count} items', items_count, { count = items_count }) %]

### In JavaScript files (`*.js`)

Like in templates, you have several functions available. Just replace `t` by `__`.

__('This is a translatable string');
__npx('bibliographic record, '{count} item', '{count} items', items_count, { count: items_count });

### In Perl files (`*.pl`, `*.pm`)

You will have to add

use Koha::I18N;

at the top of the file, and then the same functions as above will be available.

__('This is a translatable string');
__npx('bibliographic record, '{count} item', '{count} items', $items_count, count => $items_count);

### In installer and preferences YAML files (`*.yml`)

Nothing special to do here. All strings will be automatically translatable.

## Manipulating PO files

Once strings have been made translatable in source files, they have to be
extracted into PO files and uploaded on https://translate.koha-community.org/
so they can be translated.

### Install gulp first

The next sections rely on gulp. If it's not installed, run the following
commands:

# as root
npm install gulp-cli -g

# as normal user, from the root of Koha repository
yarn

### Create PO files for a new language

If you want to add translations for a new language, you have to create the
missing PO files. You can do that by executing the following command:

# Replace xx-XX by your language tag
gulp po:create --lang xx-XX

New PO files will be available in `misc/translator/po`.

### Update PO files with new strings

When new features or bugfixes are added to Koha, new translatable strings can
be added, other can be removed or modified, and the PO file become out of sync.

To be able to translate the new or modified strings, you have to update PO
files. This can be done by executing the following command:

# Update PO files for all languages
gulp po:update

# or only one language
gulp po:update --lang xx-XX

### Only extract strings

Creating or updating PO files automatically extract strings, but if for some
reasons you want to only extract strings without touching PO files, you can run
the following command:

gulp po:extract

POT files will be available in `misc/translator`.

+ 301
- 1
gulpfile.js View File

@@ -1,13 +1,24 @@
/* eslint-env node */
/* eslint no-console:"off" */

const { dest, series, src, watch } = require('gulp');
const { dest, parallel, series, src, watch } = require('gulp');

const child_process = require('child_process');
const fs = require('fs');
const os = require('os');
const path = require('path');
const util = require('util');

const sass = require("gulp-sass");
const cssnano = require("gulp-cssnano");
const rtlcss = require('gulp-rtlcss');
const sourcemaps = require('gulp-sourcemaps');
const autoprefixer = require('gulp-autoprefixer');
const concatPo = require('gulp-concat-po');
const exec = require('gulp-exec');
const merge = require('merge-stream');
const through2 = require('through2');
const Vinyl = require('vinyl');
const args = require('minimist')(process.argv.slice(2));
const rename = require('gulp-rename');

@@ -62,8 +73,297 @@ function build() {
.pipe(dest(css_base));
}

const poTasks = {
'marc-MARC21': {
extract: po_extract_marc_marc21,
create: po_create_marc_marc21,
update: po_update_marc_marc21,
},
'marc-NORMARC': {
extract: po_extract_marc_normarc,
create: po_create_marc_normarc,
update: po_update_marc_normarc,
},
'marc-UNIMARC': {
extract: po_extract_marc_unimarc,
create: po_create_marc_unimarc,
update: po_update_marc_unimarc,
},
'staff-prog': {
extract: po_extract_staff,
create: po_create_staff,
update: po_update_staff,
},
'opac-bootstrap': {
extract: po_extract_opac,
create: po_create_opac,
update: po_update_opac,
},
'pref': {
extract: po_extract_pref,
create: po_create_pref,
update: po_update_pref,
},
'messages': {
extract: po_extract_messages,
create: po_create_messages,
update: po_update_messages,
},
'messages-js': {
extract: po_extract_messages_js,
create: po_create_messages_js,
update: po_update_messages_js,
},
'installer': {
extract: po_extract_installer,
create: po_create_installer,
update: po_update_installer,
},
'installer-MARC21': {
extract: po_extract_installer_marc21,
create: po_create_installer_marc21,
update: po_update_installer_marc21,
},
};

const poTypes = Object.keys(poTasks);

function po_extract_marc (type) {
return src(`koha-tmpl/*-tmpl/*/en/**/*${type}*`, { read: false, nocase: true })
.pipe(xgettext('misc/translator/xgettext.pl --charset=UTF-8 -s', `Koha-marc-${type}.pot`))
.pipe(dest('misc/translator'))
}

function po_extract_marc_marc21 () { return po_extract_marc('MARC21') }
function po_extract_marc_normarc () { return po_extract_marc('NORMARC') }
function po_extract_marc_unimarc () { return po_extract_marc('UNIMARC') }

function po_extract_staff () {
const globs = [
'koha-tmpl/intranet-tmpl/prog/en/**/*.tt',
'koha-tmpl/intranet-tmpl/prog/en/**/*.inc',
'koha-tmpl/intranet-tmpl/prog/en/xslt/*.xsl',
'koha-tmpl/intranet-tmpl/prog/en/columns.def',
'!koha-tmpl/intranet-tmpl/prog/en/**/*MARC21*',
'!koha-tmpl/intranet-tmpl/prog/en/**/*NORMARC*',
'!koha-tmpl/intranet-tmpl/prog/en/**/*UNIMARC*',
'!koha-tmpl/intranet-tmpl/prog/en/**/*marc21*',
'!koha-tmpl/intranet-tmpl/prog/en/**/*normarc*',
'!koha-tmpl/intranet-tmpl/prog/en/**/*unimarc*',
];

return src(globs, { read: false, nocase: true })
.pipe(xgettext('misc/translator/xgettext.pl --charset=UTF-8 -s', 'Koha-staff-prog.pot'))
.pipe(dest('misc/translator'))
}

function po_extract_opac () {
const globs = [
'koha-tmpl/opac-tmpl/bootstrap/en/**/*.tt',
'koha-tmpl/opac-tmpl/bootstrap/en/**/*.inc',
'koha-tmpl/opac-tmpl/bootstrap/en/xslt/*.xsl',
'!koha-tmpl/opac-tmpl/bootstrap/en/**/*MARC21*',
'!koha-tmpl/opac-tmpl/bootstrap/en/**/*NORMARC*',
'!koha-tmpl/opac-tmpl/bootstrap/en/**/*UNIMARC*',
'!koha-tmpl/opac-tmpl/bootstrap/en/**/*marc21*',
'!koha-tmpl/opac-tmpl/bootstrap/en/**/*normarc*',
'!koha-tmpl/opac-tmpl/bootstrap/en/**/*unimarc*',
];

return src(globs, { read: false, nocase: true })
.pipe(xgettext('misc/translator/xgettext.pl --charset=UTF-8 -s', 'Koha-opac-bootstrap.pot'))
.pipe(dest('misc/translator'))
}

const xgettext_options = '--from-code=UTF-8 --package-name Koha '
+ '--package-version= -k -k__ -k__x -k__n:1,2 -k__nx:1,2 -k__xn:1,2 '
+ '-k__p:1c,2 -k__px:1c,2 -k__np:1c,2,3 -k__npx:1c,2,3 -kN__ '
+ '-kN__n:1,2 -kN__p:1c,2 -kN__np:1c,2,3 --force-po';

function po_extract_messages_js () {
const globs = [
'koha-tmpl/intranet-tmpl/prog/js/**/*.js',
'koha-tmpl/opac-tmpl/bootstrap/js/**/*.js',
];

return src(globs, { read: false, nocase: true })
.pipe(xgettext(`xgettext -L JavaScript ${xgettext_options}`, 'Koha-messages-js.pot'))
.pipe(dest('misc/translator'))
}

function po_extract_messages () {
const perlStream = src(['**/*.pl', '**/*.pm'], { read: false, nocase: true })
.pipe(xgettext(`xgettext -L Perl ${xgettext_options}`, 'Koha-perl.pot'))

const ttStream = src([
'koha-tmpl/intranet-tmpl/prog/en/**/*.tt',
'koha-tmpl/intranet-tmpl/prog/en/**/*.inc',
'koha-tmpl/opac-tmpl/bootstrap/en/**/*.tt',
'koha-tmpl/opac-tmpl/bootstrap/en/**/*.inc',
], { read: false, nocase: true })
.pipe(xgettext('misc/translator/xgettext-tt2 --from-code=UTF-8', 'Koha-tt.pot'))

const headers = {
'Project-Id-Version': 'Koha',
'Content-Type': 'text/plain; charset=UTF-8',
};

return merge(perlStream, ttStream)
.pipe(concatPo('Koha-messages.pot', { headers }))
.pipe(dest('misc/translator'))
}

function po_extract_pref () {
return src('koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/*.pref', { read: false })
.pipe(xgettext('misc/translator/xgettext-pref', 'Koha-pref.pot'))
.pipe(dest('misc/translator'))
}

function po_extract_installer () {
const globs = [
'installer/data/mysql/en/mandatory/*.yml',
'installer/data/mysql/en/optional/*.yml',
];

return src(globs, { read: false, nocase: true })
.pipe(xgettext('misc/translator/xgettext-installer', 'Koha-installer.pot'))
.pipe(dest('misc/translator'))
}

function po_extract_installer_marc (type) {
const globs = `installer/data/mysql/en/marcflavour/${type}/**/*.yml`;

return src(globs, { read: false, nocase: true })
.pipe(xgettext('misc/translator/xgettext-installer', `Koha-installer-${type}.pot`))
.pipe(dest('misc/translator'))
}

function po_extract_installer_marc21 () { return po_extract_installer_marc('MARC21') }

function po_create_type (type) {
const access = util.promisify(fs.access);
const exec = util.promisify(child_process.exec);

const languages = getLanguages();
const promises = [];
for (const language of languages) {
const locale = language.split('-').filter(s => s.length !== 4).join('_');
const po = `misc/translator/po/${language}-${type}.po`;
const pot = `misc/translator/Koha-${type}.pot`;

const promise = access(po)
.catch(() => exec(`msginit -o ${po} -i ${pot} -l ${locale} --no-translator`))
promises.push(promise);
}

return Promise.all(promises);
}

function po_create_marc_marc21 () { return po_create_type('marc-MARC21') }
function po_create_marc_normarc () { return po_create_type('marc-NORMARC') }
function po_create_marc_unimarc () { return po_create_type('marc-UNIMARC') }
function po_create_staff () { return po_create_type('staff-prog') }
function po_create_opac () { return po_create_type('opac-bootstrap') }
function po_create_pref () { return po_create_type('pref') }
function po_create_messages () { return po_create_type('messages') }
function po_create_messages_js () { return po_create_type('messages-js') }
function po_create_installer () { return po_create_type('installer') }
function po_create_installer_marc21 () { return po_create_type('installer-MARC21') }

function po_update_type (type) {
const msgmerge_opts = '--backup=off --quiet --sort-output --update';
const cmd = `msgmerge ${msgmerge_opts} <%= file.path %> misc/translator/Koha-${type}.pot`;
const languages = getLanguages();
const globs = languages.map(language => `misc/translator/po/${language}-${type}.po`);

return src(globs)
.pipe(exec(cmd, { continueOnError: true }))
.pipe(exec.reporter({ err: false, stdout: false }))
}

function po_update_marc_marc21 () { return po_update_type('marc-MARC21') }
function po_update_marc_normarc () { return po_update_type('marc-NORMARC') }
function po_update_marc_unimarc () { return po_update_type('marc-UNIMARC') }
function po_update_staff () { return po_update_type('staff-prog') }
function po_update_opac () { return po_update_type('opac-bootstrap') }
function po_update_pref () { return po_update_type('pref') }
function po_update_messages () { return po_update_type('messages') }
function po_update_messages_js () { return po_update_type('messages-js') }
function po_update_installer () { return po_update_type('installer') }
function po_update_installer_marc21 () { return po_update_type('installer-MARC21') }

/**
* Gulp plugin that executes xgettext-like command `cmd` on all files given as
* input, and then outputs the result as a POT file named `filename`.
* `cmd` should accept -o and -f options
*/
function xgettext (cmd, filename) {
const filenames = [];

function transform (file, encoding, callback) {
filenames.push(path.relative(file.cwd, file.path));
callback();
}

function flush (callback) {
fs.mkdtemp(path.join(os.tmpdir(), 'koha-'), (err, folder) => {
const outputFilename = path.join(folder, filename);
const filesFilename = path.join(folder, 'files');
fs.writeFile(filesFilename, filenames.join(os.EOL), err => {
if (err) return callback(err);

const command = `${cmd} -o ${outputFilename} -f ${filesFilename}`;
child_process.exec(command, err => {
if (err) return callback(err);

fs.readFile(outputFilename, (err, data) => {
if (err) return callback(err);

const file = new Vinyl();
file.path = path.join(file.base, filename);
file.contents = data;
callback(null, file);
});
});
});
})
}

return through2.obj(transform, flush);
}

/**
* Return languages selected for PO-related tasks
*
* This can be either languages given on command-line with --lang option, or
* all the languages found in misc/translator/po otherwise
*/
function getLanguages () {
if (Array.isArray(args.lang)) {
return args.lang;
}

if (args.lang) {
return [args.lang];
}

const filenames = fs.readdirSync('misc/translator/po')
.filter(filename => filename.endsWith('.po'))
.filter(filename => !filename.startsWith('.'))

const re = new RegExp('-(' + poTypes.join('|') + ')\.po$');
languages = filenames.map(filename => filename.replace(re, ''))

return Array.from(new Set(languages));
}

exports.build = build;
exports.css = css;

exports['po:create'] = parallel(...poTypes.map(type => series(poTasks[type].extract, poTasks[type].create)));
exports['po:update'] = parallel(...poTypes.map(type => series(poTasks[type].extract, poTasks[type].update)));
exports['po:extract'] = parallel(...poTypes.map(type => poTasks[type].extract));

exports.default = function () {
watch(css_base + "/src/**/*.scss", series('css'));
}

+ 86
- 693
misc/translator/LangInstaller.pm View File

@@ -22,36 +22,15 @@ use Modern::Perl;
use C4::Context;
# WARNING: Any other tested YAML library fails to work properly in this
# script content
use YAML::Syck qw( Dump LoadFile DumpFile );
use YAML::Syck qw( LoadFile DumpFile );
use Locale::PO;
use FindBin qw( $Bin );
use File::Basename;
use File::Find;
use File::Path qw( make_path );
use File::Copy;
use File::Slurp;
use File::Spec;
use File::Temp qw( tempdir tempfile );
use Template::Parser;
use PPI;


$YAML::Syck::ImplicitTyping = 1;


# Default file header for .po syspref files
my $default_pref_po_header = Locale::PO->new(-msgid => '', -msgstr =>
"Project-Id-Version: PACKAGE VERSION\\n" .
"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\n" .
"Last-Translator: FULL NAME <EMAIL\@ADDRESS>\\n" .
"Language-Team: Koha Translate List <koha-translate\@lists.koha-community.org>\\n" .
"MIME-Version: 1.0\\n" .
"Content-Type: text/plain; charset=UTF-8\\n" .
"Content-Transfer-Encoding: 8bit\\n" .
"Plural-Forms: nplurals=2; plural=(n > 1);\\n"
);


sub set_lang {
my ($self, $lang) = @_;

@@ -60,7 +39,6 @@ sub set_lang {
"/prog/$lang/modules/admin/preferences";
}


sub new {
my ($class, $lang, $pref_only, $verbose) = @_;

@@ -75,32 +53,16 @@ sub new {
$self->{verbose} = $verbose;
$self->{process} = "$Bin/tmpl_process3.pl " . ($verbose ? '' : '-q');
$self->{path_po} = "$Bin/po";
$self->{po} = { '' => $default_pref_po_header };
$self->{po} = {};
$self->{domain} = 'Koha';
$self->{cp} = `which cp`;
$self->{msgmerge} = `which msgmerge`;
$self->{msgfmt} = `which msgfmt`;
$self->{msginit} = `which msginit`;
$self->{msgattrib} = `which msgattrib`;
$self->{xgettext} = `which xgettext`;
$self->{sed} = `which sed`;
$self->{po2json} = "$Bin/po2json";
$self->{gzip} = `which gzip`;
$self->{gunzip} = `which gunzip`;
chomp $self->{cp};
chomp $self->{msgmerge};
chomp $self->{msgfmt};
chomp $self->{msginit};
chomp $self->{msgattrib};
chomp $self->{xgettext};
chomp $self->{sed};
chomp $self->{gzip};
chomp $self->{gunzip};

unless ($self->{xgettext}) {
die "Missing 'xgettext' executable. Have you installed the gettext package?\n";
}

# Get all .pref file names
opendir my $fh, $self->{path_pref_en};
my @pref_files = grep { /\.pref$/ } readdir($fh);
@@ -175,7 +137,6 @@ sub new {
bless $self, $class;
}


sub po_filename {
my $self = shift;
my $suffix = shift;
@@ -186,162 +147,92 @@ sub po_filename {
return $trans_file;
}

sub get_trans_text {
my ($self, $msgid, $default) = @_;

sub po_append {
my ($self, $id, $comment) = @_;
my $po = $self->{po};
my $p = $po->{$id};
if ( $p ) {
$p->comment( $p->comment . "\n" . $comment );
}
else {
$po->{$id} = Locale::PO->new(
-comment => $comment,
-msgid => $id,
-msgstr => ''
);
}
}


sub add_prefs {
my ($self, $comment, $prefs) = @_;

for my $pref ( @$prefs ) {
my $pref_name = '';
for my $element ( @$pref ) {
if ( ref( $element) eq 'HASH' ) {
$pref_name = $element->{pref};
last;
}
}
for my $element ( @$pref ) {
if ( ref( $element) eq 'HASH' ) {
while ( my ($key, $value) = each(%$element) ) {
next unless $key eq 'choices' or $key eq 'multiple';
next unless ref($value) eq 'HASH';
for my $ckey ( keys %$value ) {
my $id = $self->{file} . "#$pref_name# " . $value->{$ckey};
$self->po_append( $id, $comment );
}
}
}
elsif ( $element ) {
$self->po_append( $self->{file} . "#$pref_name# $element", $comment );
}
my $po = $self->{po}->{Locale::PO->quote($msgid)};
if ($po) {
my $msgstr = Locale::PO->dequote($po->msgstr);
if ($msgstr and length($msgstr) > 0) {
return $msgstr;
}
}
}


sub get_trans_text {
my ($self, $id) = @_;

my $po = $self->{po}->{$id};
return unless $po;
return Locale::PO->dequote($po->msgstr);
return $default;
}

sub get_translated_tab_content {
my ($self, $file, $tab_content) = @_;

sub update_tab_prefs {
my ($self, $pref, $prefs) = @_;

for my $p ( @$prefs ) {
my $pref_name = '';
next unless $p;
for my $element ( @$p ) {
if ( ref( $element) eq 'HASH' ) {
$pref_name = $element->{pref};
last;
}
}
for my $i ( 0..@$p-1 ) {
my $element = $p->[$i];
if ( ref( $element) eq 'HASH' ) {
while ( my ($key, $value) = each(%$element) ) {
next unless $key eq 'choices' or $key eq 'multiple';
next unless ref($value) eq 'HASH';
for my $ckey ( keys %$value ) {
my $id = $self->{file} . "#$pref_name# " . $value->{$ckey};
my $text = $self->get_trans_text( $id );
$value->{$ckey} = $text if $text;
}
}
}
elsif ( $element ) {
my $id = $self->{file} . "#$pref_name# $element";
my $text = $self->get_trans_text( $id );
$p->[$i] = $text if $text;
}
}
if ( ref($tab_content) eq 'ARRAY' ) {
return $self->get_translated_prefs($file, $tab_content);
}
}

my $translated_tab_content = {
map {
my $section = $_;
my $sysprefs = $tab_content->{$section};
my $msgid = sprintf('%s %s', $file, $section);

sub get_po_from_prefs {
my $self = shift;
$self->get_trans_text($msgid, $section) => $self->get_translated_prefs($file, $sysprefs);
} keys %$tab_content
};

for my $file ( @{$self->{pref_files}} ) {
my $pref = LoadFile( $self->{path_pref_en} . "/$file" );
$self->{file} = $file;
# Entries for tab titles
$self->po_append( $self->{file}, $_ ) for keys %$pref;
while ( my ($tab, $tab_content) = each %$pref ) {
if ( ref($tab_content) eq 'ARRAY' ) {
$self->add_prefs( $tab, $tab_content );
next;
}
while ( my ($section, $sysprefs) = each %$tab_content ) {
my $comment = "$tab > $section";
$self->po_append( $self->{file} . " " . $section, $comment );
$self->add_prefs( $comment, $sysprefs );
}
}
}
return $translated_tab_content;
}

sub get_translated_prefs {
my ($self, $file, $sysprefs) = @_;

sub save_po {
my $self = shift;
my $translated_prefs = [
map {
my ($pref_elt) = grep { ref($_) eq 'HASH' && exists $_->{pref} } @$_;
my $pref_name = $pref_elt ? $pref_elt->{pref} : '';

my $translated_syspref = [
map {
$self->get_translated_pref($file, $pref_name, $_);
} @$_
];

# Create file header if it doesn't already exist
my $po = $self->{po};
$po->{''} ||= $default_pref_po_header;
$translated_syspref;
} @$sysprefs
];

# Write .po entries into a file put in Koha standard po directory
Locale::PO->save_file_fromhash( $self->po_filename("-pref.po"), $po );
say "Saved in file: ", $self->po_filename("-pref.po") if $self->{verbose};
return $translated_prefs;
}

sub get_translated_pref {
my ($self, $file, $pref_name, $syspref) = @_;

sub get_po_merged_with_en {
my $self = shift;

# Get po from current 'en' .pref files
$self->get_po_from_prefs();
my $po_current = $self->{po};
unless (ref($syspref)) {
$syspref //= '';
my $msgid = sprintf('%s#%s# %s', $file, $pref_name, $syspref);
return $self->get_trans_text($msgid, $syspref);
}

# Get po from previous generation
my $po_previous = Locale::PO->load_file_ashash( $self->po_filename("-pref.po") );
my $translated_pref = {
map {
my $key = $_;
my $value = $syspref->{$key};

for my $id ( keys %$po_current ) {
my $po = $po_previous->{Locale::PO->quote($id)};
next unless $po;
my $text = Locale::PO->dequote( $po->msgstr );
$po_current->{$id}->msgstr( $text );
}
}
my $translated_value = $value;
if (($key eq 'choices' || $key eq 'multiple') && ref($value) eq 'HASH') {
$translated_value = {
map {
my $msgid = sprintf('%s#%s# %s', $file, $pref_name, $value->{$_});
$_ => $self->get_trans_text($msgid, $value->{$_})
} keys %$value
}
}

$key => $translated_value
} keys %$syspref
};

sub update_prefs {
my $self = shift;
print "Update '", $self->{lang},
"' preferences .po file from 'en' .pref files\n" if $self->{verbose};
$self->get_po_merged_with_en();
$self->save_po();
return $translated_pref;
}


sub install_prefs {
my $self = shift;

@@ -350,45 +241,24 @@ sub install_prefs {
exit;
}

# Get the language .po file merged with last modified 'en' preferences
$self->get_po_merged_with_en();
$self->{po} = Locale::PO->load_file_ashash($self->po_filename("-pref.po"), 'utf8');

for my $file ( @{$self->{pref_files}} ) {
my $pref = LoadFile( $self->{path_pref_en} . "/$file" );
$self->{file} = $file;
# First, keys are replaced (tab titles)
$pref = do {
my %pref = map {
$self->get_trans_text( $self->{file} ) || $_ => $pref->{$_}
} keys %$pref;
\%pref;

my $translated_pref = {
map {
my $tab = $_;
my $tab_content = $pref->{$tab};

$self->get_trans_text($file, $tab) => $self->get_translated_tab_content($file, $tab_content);
} keys %$pref
};
while ( my ($tab, $tab_content) = each %$pref ) {
if ( ref($tab_content) eq 'ARRAY' ) {
$self->update_tab_prefs( $pref, $tab_content );
next;
}
while ( my ($section, $sysprefs) = each %$tab_content ) {
$self->update_tab_prefs( $pref, $sysprefs );
}
my $ntab = {};
for my $section ( keys %$tab_content ) {
my $id = $self->{file} . " $section";
my $text = $self->get_trans_text($id);
my $nsection = $text ? $text : $section;
if( exists $ntab->{$nsection} ) {
# When translations collide (see BZ 18634)
push @{$ntab->{$nsection}}, @{$tab_content->{$section}};
} else {
$ntab->{$nsection} = $tab_content->{$section};
}
}
$pref->{$tab} = $ntab;
}


my $file_trans = $self->{po_path_lang} . "/$file";
print "Write $file\n" if $self->{verbose};
open my $fh, ">", $file_trans;
print $fh Dump($pref);
DumpFile($file_trans, $translated_pref);
}
}

@@ -429,180 +299,6 @@ sub install_tmpl {
}
}


sub update_tmpl {
my ($self, $files) = @_;

say "Update templates" if $self->{verbose};
for my $trans ( @{$self->{interface}} ) {
my @files = @$files;
my @nomarc = ();
print
" Update templates '$trans->{name}'\n",
" From: $trans->{dir}/en/\n",
" To : $self->{path_po}/$self->{lang}$trans->{suffix}\n"
if $self->{verbose};

my $trans_dir = join("/en/ -i ",split(" ",$trans->{dir}))."/en/"; # multiple source dirs
# if processing MARC po file, only use corresponding files
my $marc = ( $trans->{name} =~ /MARC/ )?"-m \"$trans->{name}\"":""; # for MARC translations
# if not processing MARC po file, ignore all MARC files
@nomarc = ( 'marc21', 'unimarc', 'normarc' ) if ( $trans->{name} !~ /MARC/ ); # hardcoded MARC variants

system
"$self->{process} update " .
"-i $trans_dir " .
"-s $self->{path_po}/$self->{lang}$trans->{suffix} -r " .
"$marc " .
( @files ? ' -f ' . join ' -f ', @files : '') .
( @nomarc ? ' -n ' . join ' -n ', @nomarc : '');
}
}


sub create_prefs {
my $self = shift;

if ( -e $self->po_filename("-pref.po") ) {
say "Preferences .po file already exists. Delete it if you want to recreate it.";
return;
}
$self->get_po_from_prefs();
$self->save_po();
}

sub get_po_from_target {
my $self = shift;
my $target = shift;

my $po;
my $po_head = Locale::PO->new;
$po_head->{msgid} = "\"\"";
$po_head->{msgstr} = "".
"Project-Id-Version: Koha Project - Installation files\\n" .
"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\n" .
"Last-Translator: FULL NAME <EMAIL\@ADDRESS>\\n" .
"Language-Team: Koha Translation Team\\n" .
"Language: ".$self->{lang}."\\n" .
"MIME-Version: 1.0\\n" .
"Content-Type: text/plain; charset=UTF-8\\n" .
"Content-Transfer-Encoding: 8bit\\n";

my @dirs = @{ $target->{dirs} };
my $intradir = $self->{context}->config('intranetdir');
for my $dir ( @dirs ) { # each dir
opendir( my $dh, "$intradir/$dir" ) or die ("Can't open $intradir/$dir");
my @filelist = grep { $_ =~ m/\.yml/ } readdir($dh); # Just yaml files
close($dh);
for my $file ( @filelist ) { # each file
my $yaml = LoadFile( "$intradir/$dir/$file" );
my @tables = @{ $yaml->{'tables'} };
my $tablec;
for my $table ( @tables ) { # each table
$tablec++;
my $table_name = ( keys %$table )[0];
my @translatable = @{ $table->{$table_name}->{translatable} };
my @rows = @{ $table->{$table_name}->{rows} };
my @multiline = @{ $table->{$table_name}->{'multiline'} }; # to check multiline values
my $rowc;
for my $row ( @rows ) { # each row
$rowc++;
for my $field ( @translatable ) { # each field
if ( @multiline and grep { $_ eq $field } @multiline ) { # multiline fields, only notices ATM
my $mulc;
foreach my $line ( @{$row->{$field}} ) {
$mulc++;
next if ( $line =~ /^(\s*<.*?>\s*$|^\s*\[.*?\]\s*|\s*)$/ ); # discard pure html, TT, empty
$line =~ s/(<<.*?>>|\[\%.*?\%\]|<.*?>)/\%s/g; # put placeholders
next if ( $line =~ /^(\s|%s|-|[[:punct:]]|\(|\))*$/ or length($line) < 2 ); # discard non strings
if ( not $po->{ $line } ) {
my $msg = Locale::PO->new(
-msgid => $line, -msgstr => '',
-reference => "$dir/$file:$table_name:$tablec:row:$rowc:mul:$mulc" );
$po->{ $line } = $msg;
}
}
} else {
if ( defined $row->{$field} and length($row->{$field}) > 1 # discard null values and small strings
and not $po->{ $row->{$field} } ) {
my $msg = Locale::PO->new(
-msgid => $row->{$field}, -msgstr => '',
-reference => "$dir/$file:$table_name:$tablec:row:$rowc" );
$po->{ $row->{$field} } = $msg;
}
}
}
}
}
my $desccount;
for my $description ( @{ $yaml->{'description'} } ) {
$desccount++;
if ( length($description) > 1 and not $po->{ $description } ) {
my $msg = Locale::PO->new(
-msgid => $description, -msgstr => '',
-reference => "$dir/$file:description:$desccount" );
$po->{ $description } = $msg;
}
}
}
}
$po->{''} = $po_head if ( $po );

return $po;
}

sub create_installer {
my $self = shift;
return unless ( $self->{installer} );

say "Create installer translation files\n" if $self->{verbose};

my @targets = @{ $self->{installer} }; # each installer target (common,marc21,unimarc)

for my $target ( @targets ) {
if ( -e $self->po_filename( $target->{suffix} ) ) {
say "$self->{lang}$target->{suffix} file already exists. Delete it if you want to recreate it.";
return;
}
}

for my $target ( @targets ) {
my $po = get_po_from_target( $self, $target );
# create output file only if there is something to write
if ( $po ) {
my $po_file = $self->po_filename( $target->{suffix} );
Locale::PO->save_file_fromhash( $po_file, $po );
say "Saved in file: ", $po_file if $self->{verbose};
}
}
}

sub update_installer {
my $self = shift;
return unless ( $self->{installer} );

say "Update installer translation files\n" if $self->{verbose};

my @targets = @{ $self->{installer} }; # each installer target (common,marc21,unimarc)

for my $target ( @targets ) {
return unless ( -e $self->po_filename( $target->{suffix} ) );
my $po = get_po_from_target( $self, $target );
# update file only if there is something to update
if ( $po ) {
my ( $fh, $po_temp ) = tempfile();
binmode( $fh, ":encoding(UTF-8)" );
Locale::PO->save_file_fromhash( $po_temp, $po );
my $po_file = $self->po_filename( $target->{suffix} );
eval {
my $st = system($self->{msgmerge}." ".($self->{verbose}?'':'-q').
" -s $po_file $po_temp -o - | ".$self->{msgattrib}." --no-obsolete -o $po_file");
};
say "Updated file: ", $po_file if $self->{verbose};
}
}
}

sub translate_yaml {
my $self = shift;
my $target = shift;
@@ -716,35 +412,6 @@ sub install_installer {
}
}

sub create_tmpl {
my ($self, $files) = @_;

say "Create templates\n" if $self->{verbose};
for my $trans ( @{$self->{interface}} ) {
my @files = @$files;
my @nomarc = ();
print
" Create templates .po files for '$trans->{name}'\n",
" From: $trans->{dir}/en/\n",
" To : $self->{path_po}/$self->{lang}$trans->{suffix}\n"
if $self->{verbose};

my $trans_dir = join("/en/ -i ",split(" ",$trans->{dir}))."/en/"; # multiple source dirs
# if processing MARC po file, only use corresponding files
my $marc = ( $trans->{name} =~ /MARC/ )?"-m \"$trans->{name}\"":""; # for MARC translations
# if not processing MARC po file, ignore all MARC files
@nomarc = ( 'marc21', 'unimarc', 'normarc' ) if ( $trans->{name} !~ /MARC/ ); # hardcoded MARC variants

system
"$self->{process} create " .
"-i $trans_dir " .
"-s $self->{path_po}/$self->{lang}$trans->{suffix} -r " .
"$marc " .
( @files ? ' -f ' . join ' -f ', @files : '') .
( @nomarc ? ' -n ' . join ' -n ', @nomarc : '');
}
}

sub locale_name {
my $self = shift;

@@ -758,250 +425,6 @@ sub locale_name {
return $locale;
}

sub create_messages {
my $self = shift;

my $pot = "$Bin/$self->{domain}.pot";
my $po = "$self->{path_po}/$self->{lang}-messages.po";
my $js_pot = "$self->{domain}-js.pot";
my $js_po = "$self->{path_po}/$self->{lang}-messages-js.po";

unless ( -f $pot && -f $js_pot ) {
$self->extract_messages();
}

say "Create messages ($self->{lang})" if $self->{verbose};
my $locale = $self->locale_name();
system "$self->{msginit} -i $pot -o $po -l $locale --no-translator 2> /dev/null";
warn "Problems creating $pot ".$? if ( $? == -1 );
system "$self->{msginit} -i $js_pot -o $js_po -l $locale --no-translator 2> /dev/null";
warn "Problems creating $js_pot ".$? if ( $? == -1 );

# If msginit failed to correctly set Plural-Forms, set a default one
system "$self->{sed} --in-place "
. "--expression='s/Plural-Forms: nplurals=INTEGER; plural=EXPRESSION/Plural-Forms: nplurals=2; plural=(n != 1)/' "
. "$po $js_po";
}

sub update_messages {
my $self = shift;

my $pot = "$Bin/$self->{domain}.pot";
my $po = "$self->{path_po}/$self->{lang}-messages.po";
my $js_pot = "$self->{domain}-js.pot";
my $js_po = "$self->{path_po}/$self->{lang}-messages-js.po";

unless ( -f $pot && -f $js_pot ) {
$self->extract_messages();
}

if ( -f $po && -f $js_pot ) {
say "Update messages ($self->{lang})" if $self->{verbose};
system "$self->{msgmerge} --backup=off --quiet -U $po $pot";
system "$self->{msgmerge} --backup=off --quiet -U $js_po $js_pot";
} else {
$self->create_messages();
}
}

sub extract_messages_from_templates {
my ($self, $tempdir, $type, @files) = @_;

my $htdocs = $type eq 'intranet' ? 'intrahtdocs' : 'opachtdocs';
my $dir = $self->{context}->config($htdocs);
my @keywords = qw(t tx tn txn tnx tp tpx tnp tnpx);
my $parser = Template::Parser->new();

foreach my $file (@files) {
say "Extract messages from $file" if $self->{verbose};
my $template = read_file(File::Spec->catfile($dir, $file));

# No need to process a file that doesn't use the i18n.inc file.
next unless $template =~ /i18n\.inc/;

my $data = $parser->parse($template);
unless ($data) {
warn "Error at $file : " . $parser->error();
next;
}

my $destfile = $type eq 'intranet' ?
File::Spec->catfile($tempdir, 'koha-tmpl', 'intranet-tmpl', $file) :
File::Spec->catfile($tempdir, 'koha-tmpl', 'opac-tmpl', $file);

make_path(dirname($destfile));
open my $fh, '>', $destfile;

my @blocks = ($data->{BLOCK}, values %{ $data->{DEFBLOCKS} });
foreach my $block (@blocks) {
my $document = PPI::Document->new(\$block);

# [% t('foo') %] is compiled to
# $output .= $stash->get(['t', ['foo']]);
# We try to find all nodes corresponding to keyword (here 't')
my $nodes = $document->find(sub {
my ($topnode, $element) = @_;

# Filter out non-valid keywords
return 0 unless ($element->isa('PPI::Token::Quote::Single'));
return 0 unless (grep {$element->content eq qq{'$_'}} @keywords);

# keyword (e.g. 't') should be the first element of the arrayref
# passed to $stash->get()
return 0 if $element->sprevious_sibling;

return 0 unless $element->snext_sibling
&& $element->snext_sibling->snext_sibling
&& $element->snext_sibling->snext_sibling->isa('PPI::Structure::Constructor');

# Check that it's indeed a call to $stash->get()
my $statement = $element->statement->parent->statement->parent->statement;
return 0 unless grep { $_->isa('PPI::Token::Symbol') && $_->content eq '$stash' } $statement->children;
return 0 unless grep { $_->isa('PPI::Token::Operator') && $_->content eq '->' } $statement->children;
return 0 unless grep { $_->isa('PPI::Token::Word') && $_->content eq 'get' } $statement->children;

return 1;
});

next unless $nodes;

# Write the Perl equivalent of calls to t* functions family, so
# xgettext can extract the strings correctly
foreach my $node (@$nodes) {
my @args = map {
$_->significant && !$_->isa('PPI::Token::Operator') ? $_->content : ()
} $node->snext_sibling->snext_sibling->find_first('PPI::Statement')->children;

my $keyword = $node->content;
$keyword =~ s/^'t(.*)'$/__$1/;

# Only keep required args to have a clean output
my @required_args = shift @args;
push @required_args, shift @args if $keyword =~ /n/;
push @required_args, shift @args if $keyword =~ /p/;

say $fh "$keyword(" . join(', ', @required_args) . ");";
}

}

close $fh;
}

return $tempdir;
}

sub extract_messages {
my $self = shift;

say "Extract messages into POT file" if $self->{verbose};

my $intranetdir = $self->{context}->config('intranetdir');
my $opacdir = $self->{context}->config('opacdir');

# Find common ancestor directory
my @intranetdirs = File::Spec->splitdir($intranetdir);
my @opacdirs = File::Spec->splitdir($opacdir);
my @basedirs;
while (@intranetdirs and @opacdirs) {
my ($dir1, $dir2) = (shift @intranetdirs, shift @opacdirs);
last if $dir1 ne $dir2;
push @basedirs, $dir1;
}
my $basedir = File::Spec->catdir(@basedirs);

my @files_to_scan;
my @directories_to_scan = ('.');
my @blacklist = map { File::Spec->catdir(@intranetdirs, $_) } qw(blib koha-tmpl skel tmp t);
while (@directories_to_scan) {
my $dir = shift @directories_to_scan;
opendir DIR, File::Spec->catdir($basedir, $dir) or die "Unable to open $dir: $!";
foreach my $entry (readdir DIR) {
next if $entry =~ /^\./;
my $relentry = File::Spec->catfile($dir, $entry);
my $abspath = File::Spec->catfile($basedir, $relentry);
if (-d $abspath and not grep { $_ eq $relentry } @blacklist) {
push @directories_to_scan, $relentry;
} elsif (-f $abspath and $relentry =~ /\.(pl|pm)$/) {
push @files_to_scan, $relentry;
}
}
}

my $intrahtdocs = $self->{context}->config('intrahtdocs');
my $opachtdocs = $self->{context}->config('opachtdocs');

my @intranet_tt_files;
find(sub {
if ($File::Find::dir =~ m|/en/| && $_ =~ m/\.(tt|inc)$/) {
my $filename = $File::Find::name;
$filename =~ s|^$intrahtdocs/||;
push @intranet_tt_files, $filename;
}
}, $intrahtdocs);

my @opac_tt_files;
find(sub {
if ($File::Find::dir =~ m|/en/| && $_ =~ m/\.(tt|inc)$/) {
my $filename = $File::Find::name;
$filename =~ s|^$opachtdocs/||;
push @opac_tt_files, $filename;
}
}, $opachtdocs);

my $tempdir = tempdir('Koha-translate-XXXX', TMPDIR => 1, CLEANUP => 1);
$self->extract_messages_from_templates($tempdir, 'intranet', @intranet_tt_files);
$self->extract_messages_from_templates($tempdir, 'opac', @opac_tt_files);

@intranet_tt_files = map { File::Spec->catfile('koha-tmpl', 'intranet-tmpl', $_) } @intranet_tt_files;
@opac_tt_files = map { File::Spec->catfile('koha-tmpl', 'opac-tmpl', $_) } @opac_tt_files;
my @tt_files = grep { -e File::Spec->catfile($tempdir, $_) } @intranet_tt_files, @opac_tt_files;

push @files_to_scan, @tt_files;

my $xgettext_common_args = "--force-po --from-code=UTF-8 "
. "--package-name=Koha --package-version='' "
. "-k -k__ -k__x -k__n:1,2 -k__nx:1,2 -k__xn:1,2 -k__p:1c,2 "
. "-k__px:1c,2 -k__np:1c,2,3 -k__npx:1c,2,3 -kN__ -kN__n:1,2 "
. "-kN__p:1c,2 -kN__np:1c,2,3 ";
my $xgettext_cmd = "$self->{xgettext} -L Perl $xgettext_common_args "
. "-o $Bin/$self->{domain}.pot -D $tempdir -D $basedir";
$xgettext_cmd .= " $_" foreach (@files_to_scan);

if (system($xgettext_cmd) != 0) {
die "system call failed: $xgettext_cmd";
}

my @js_dirs = (
"$intrahtdocs/prog/js",
"$opachtdocs/bootstrap/js",
);

my @js_files;
find(sub {
if ($_ =~ m/\.js$/) {
my $filename = $File::Find::name;
$filename =~ s|^$intranetdir/||;
push @js_files, $filename;
}
}, @js_dirs);

$xgettext_cmd = "$self->{xgettext} -L JavaScript $xgettext_common_args "
. "-o $Bin/$self->{domain}-js.pot -D $intranetdir";
$xgettext_cmd .= " $_" foreach (@js_files);

if (system($xgettext_cmd) != 0) {
die "system call failed: $xgettext_cmd";
}

my $replace_charset_cmd = "$self->{sed} --in-place " .
"--expression='s/charset=CHARSET/charset=UTF-8/' " .
"$Bin/$self->{domain}.pot $Bin/$self->{domain}-js.pot";
if (system($replace_charset_cmd) != 0) {
die "system call failed: $replace_charset_cmd";
}
}

sub install_messages {
my ($self) = @_;

@@ -1012,8 +435,9 @@ sub install_messages {
my $js_pofile = "$self->{path_po}/$self->{lang}-messages-js.po";

unless ( -f $pofile && -f $js_pofile ) {
$self->create_messages();
die "PO files for language '$self->{lang}' do not exist";
}

say "Install messages ($locale)" if $self->{verbose};
make_path($modir);
system "$self->{msgfmt} -o $mofile $pofile";
@@ -1035,13 +459,6 @@ sub install_messages {
}
}

sub remove_pot {
my $self = shift;

unlink "$Bin/$self->{domain}.pot";
unlink "$Bin/$self->{domain}-js.pot";
}

sub compress {
my ($self, $files) = @_;
my @langs = $self->{lang} ? ($self->{lang}) : $self->get_all_langs();
@@ -1074,11 +491,15 @@ sub install {
my ($self, $files) = @_;
return unless $self->{lang};
$self->uncompress();
$self->install_tmpl($files) unless $self->{pref_only};
$self->install_prefs();
$self->install_messages();
$self->remove_pot();
$self->install_installer();

if ($self->{pref_only}) {
$self->install_prefs();
} else {
$self->install_tmpl($files);
$self->install_prefs();
$self->install_messages();
$self->install_installer();
}
}


@@ -1090,34 +511,6 @@ sub get_all_langs {
@files = map { $_ =~ s/-pref.(po|po.gz)$//r } @files;
}


sub update {
my ($self, $files) = @_;
my @langs = $self->{lang} ? ($self->{lang}) : $self->get_all_langs();
for my $lang ( @langs ) {
$self->set_lang( $lang );
$self->uncompress();
$self->update_tmpl($files) unless $self->{pref_only};
$self->update_prefs();
$self->update_messages();
$self->update_installer();
}
$self->remove_pot();
}


sub create {
my ($self, $files) = @_;
return unless $self->{lang};
$self->create_tmpl($files) unless $self->{pref_only};
$self->create_prefs();
$self->create_messages();
$self->remove_pot();
$self->create_installer();
}



1;




+ 9
- 1
misc/translator/po/dz-pref.po View File

@@ -1,5 +1,13 @@
msgid ""
msgstr "Project-Id-Version: PACKAGE VERSION\\nPO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\nLast-Translator: FULL NAME <EMAIL@ADDRESS>\\nLanguage-Team: Koha Translate List <koha-translate@lists.koha-community.org>\\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=UTF-8\\nContent-Transfer-Encoding: 8bit\\nPlural-Forms: nplurals=2; plural=(n > 1);\\n"
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: Koha Translate List <koha-translate@lists.koha-community.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

# Accounting
msgid "accounting.pref"


+ 9
- 1
misc/translator/po/gd-pref.po View File

@@ -1,5 +1,13 @@
msgid ""
msgstr "Project-Id-Version: PACKAGE VERSION\\nPO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\nLast-Translator: FULL NAME <EMAIL@ADDRESS>\\nLanguage-Team: Koha Translate List <koha-translate@lists.koha-community.org>\\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=UTF-8\\nContent-Transfer-Encoding: 8bit\\nPlural-Forms: nplurals=2; plural=(n > 1);\\n"
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: Koha Translate List <koha-translate@lists.koha-community.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

# Accounting
msgid "accounting.pref"


+ 9
- 1
misc/translator/po/lv-pref.po View File

@@ -1,5 +1,13 @@
msgid ""
msgstr "Project-Id-Version: PACKAGE VERSION\\nPO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\nLast-Translator: FULL NAME <EMAIL@ADDRESS>\\nLanguage-Team: Koha Translate List <koha-translate@lists.koha-community.org>\\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=UTF-8\\nContent-Transfer-Encoding: 8bit\\nPlural-Forms: nplurals=2; plural=(n > 1);\\n"
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: Koha Translate List <koha-translate@lists.koha-community.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

# Accounting
msgid "accounting.pref"


+ 9
- 1
misc/translator/po/te-pref.po View File

@@ -1,5 +1,13 @@
msgid ""
msgstr "Project-Id-Version: PACKAGE VERSION\\nPO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\nLast-Translator: FULL NAME <EMAIL@ADDRESS>\\nLanguage-Team: Koha Translate List <koha-translate@lists.koha-community.org>\\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=UTF-8\\nContent-Transfer-Encoding: 8bit\\nPlural-Forms: nplurals=2; plural=(n > 1);\\n"
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: Koha Translate List <koha-translate@lists.koha-community.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

# Accounting
msgid "accounting.pref"


+ 3
- 119
misc/translator/tmpl_process3.pl View File

@@ -204,11 +204,9 @@ sub usage {
my($exitcode) = @_;
my $h = $exitcode? *STDERR: *STDOUT;
print $h <<EOF;
Usage: $0 create [OPTION]
or: $0 update [OPTION]
or: $0 install [OPTION]
Usage: $0 install [OPTION]
or: $0 --help
Create or update PO files from templates, or install translated templates.
Install translated templates.

-i, --input=SOURCE Get or update strings from SOURCE directory(s).
On create or update can have multiple values.
@@ -230,7 +228,6 @@ Create or update PO files from templates, or install translated templates.
--help Display this help and exit
-q, --quiet no output to screen (except for errors)

The -o option is ignored for the "create" and "update" actions.
Try `perldoc $0` for perhaps more information.
EOF
exit($exitcode);
@@ -265,12 +262,6 @@ GetOptions(
VerboseWarnings::set_application_name($0);
VerboseWarnings::set_pedantic_mode($pedantic_p);

# keep the buggy Locale::PO quiet if it says stupid things
$SIG{__WARN__} = sub {
my($s) = @_;
print STDERR $s unless $s =~ /^Strange line in [^:]+: #~/s
};

my $action = shift or usage_error('You must specify an ACTION.');
usage_error('You must at least specify input and string list filenames.')
if !@in_dirs || !defined $str_file;
@@ -344,89 +335,9 @@ if (!defined $charset_out) {
$charset_out = TmplTokenizer::charset_canon('UTF-8');
warn "Warning: Charset Out defaulting to $charset_out\n" unless ( $quiet );
}
my $xgettext = './xgettext.pl'; # actual text extractor script
my $st;

if ($action eq 'create') {
# updates the list. As the list is empty, every entry will be added
if (!-s $str_file) {
warn "Removing empty file $str_file\n" unless ( $quiet );
unlink $str_file || die "$str_file: $!\n";
}
die "$str_file: Output file already exists\n" if -f $str_file;
my($tmph1, $tmpfile1) = tmpnam();
my($tmph2, $tmpfile2) = tmpnam();
close $tmph2; # We just want a name
# Generate the temporary file that acts as <MODULE>/POTFILES.in
for my $input (@in_files) {
print $tmph1 "$input\n";
}
close $tmph1;
warn "I $charset_in O $charset_out" unless ( $quiet );
# Generate the specified po file ($str_file)
$st = system ($xgettext, '-s', '-f', $tmpfile1, '-o', $tmpfile2,
(defined $charset_in? ('-I', $charset_in): ()),
(defined $charset_out? ('-O', $charset_out): ())
);
# Run msgmerge so that the pot file looks like a real pot file
# We need to help msgmerge a bit by pre-creating a dummy po file that has
# the headers and the "" msgid & msgstr. It will fill in the rest.
if ($st == 0) {
# Merge the temporary "pot file" with the specified po file ($str_file)
# FIXME: msgmerge(1) is a Unix dependency
# FIXME: need to check the return value
unless (-f $str_file) {
open(my $infh, '<', $tmpfile2);
open(my $outfh, '>', $str_file);
while (<$infh>) {
print $outfh $_;
last if /^\n/s;
}
close $infh;
close $outfh;
}
$st = system("msgmerge ".($quiet?'-q':'')." -s $str_file $tmpfile2 -o - | msgattrib --no-obsolete -o $str_file");
} else {
error_normal("Text extraction failed: $xgettext: $!\n", undef);
error_additional("Will not run msgmerge\n", undef);
}
unlink $tmpfile1 || warn_normal("$tmpfile1: unlink failed: $!\n", undef);
unlink $tmpfile2 || warn_normal("$tmpfile2: unlink failed: $!\n", undef);

} elsif ($action eq 'update') {
my($tmph1, $tmpfile1) = tmpnam();
my($tmph2, $tmpfile2) = tmpnam();
close $tmph2; # We just want a name
# Generate the temporary file that acts as <MODULE>/POTFILES.in
for my $input (@in_files) {
print $tmph1 "$input\n";
}
close $tmph1;
# Generate the temporary file that acts as <MODULE>/<LANG>.pot
$st = system($xgettext, '-s', '-f', $tmpfile1, '-o', $tmpfile2,
'--po-mode',
(defined $charset_in? ('-I', $charset_in): ()),
(defined $charset_out? ('-O', $charset_out): ()));
if ($st == 0) {
# Merge the temporary "pot file" with the specified po file ($str_file)
# FIXME: msgmerge(1) is a Unix dependency
# FIXME: need to check the return value
if ( @filenames ) {
my ($tmph3, $tmpfile3) = tmpnam();
$st = system("msgcat $str_file $tmpfile2 > $tmpfile3");
$st = system("msgmerge ".($quiet?'-q':'')." -s $str_file $tmpfile3 -o - | msgattrib --no-obsolete -o $str_file")
unless $st;
} else {
$st = system("msgmerge ".($quiet?'-q':'')." -s $str_file $tmpfile2 -o - | msgattrib --no-obsolete -o $str_file");
}
} else {
error_normal("Text extraction failed: $xgettext: $!\n", undef);
error_additional("Will not run msgmerge\n", undef);
}
unlink $tmpfile1 || warn_normal("$tmpfile1: unlink failed: $!\n", undef);
unlink $tmpfile2 || warn_normal("$tmpfile2: unlink failed: $!\n", undef);

} elsif ($action eq 'install') {
if ($action eq 'install') {
if(!defined($out_dir)) {
usage_error("You must specify an output directory when using the install method.");
}
@@ -554,14 +465,6 @@ translation, it can be suppressed with the %0.0s notation.
Using the PO format also means translators can add their
own comments in the translation files, if necessary.

=item -

Create, update, and install actions are all based on the
same scanner module. This ensures that update and install
have the same idea of what is a translatable string;
attribute names in tags, for example, will not be
accidentally translated.

=back

=head1 NOTES
@@ -569,22 +472,8 @@ accidentally translated.
Anchors are represented by an <AI<n>> notation.
The meaning of this non-standard notation might not be obvious.

The create action calls xgettext.pl to do the actual work;
the update action calls xgettext.pl, msgmerge(1) and msgattrib(1)
to do the actual work.

=head1 BUGS

xgettext.pl must be present in the current directory; both
msgmerge(1) and msgattrib(1) must also be present in the search path.
The script currently does not check carefully whether these
dependent commands are present.

Locale::PO(3) has a lot of bugs. It can neither parse nor
generate GNU PO files properly; a couple of workarounds have
been written in TmplTokenizer and more is likely to be needed
(e.g., to get rid of the "Strange line" warning for #~).

This script may not work in Windows.

There are probably some other bugs too, since this has not been
@@ -592,12 +481,7 @@ tested very much.

=head1 SEE ALSO

xgettext.pl,
TmplTokenizer.pm,
msgmerge(1),
Locale::PO(3),
translator_doc.txt

http://www.saas.nsw.edu.au/koha_wiki/index.php?page=DifficultTerms

=cut

+ 16
- 42
misc/translator/translate View File

@@ -54,14 +54,13 @@ usage() if $#ARGV != 1 && $#ARGV != 0;

my ($cmd, $lang) = @ARGV;
$cmd = lc $cmd;
if ( $cmd =~ /^(create|install|update|compress|uncompress)$/ ) {
if ( $cmd =~ /^(install|compress|uncompress)$/ ) {
my $installer = LangInstaller->new( $lang, $pref, $verbose );
if ( $cmd ne 'create' and $lang and not grep( {$_ eq $lang} @{ $installer->{langs} } ) ) {
if ( $lang and not grep( {$_ eq $lang} @{ $installer->{langs} } ) ) {
print "Unsupported language: $lang\n";
exit;
}
if ( $all ) {
usage() if $cmd eq 'create';
for my $lang ( @{$installer->{langs}} ) {
$installer->set_lang( $lang );
$installer->$cmd(\@files);
@@ -71,9 +70,19 @@ if ( $cmd =~ /^(create|install|update|compress|uncompress)$/ ) {
$installer->$cmd(\@files);
}

Koha::Caches->get_instance()->flush_all if $cmd ne 'update';
}
else {
Koha::Caches->get_instance()->flush_all;
} elsif ($cmd eq 'create' or $cmd eq 'update') {
my $command = "gulp po:$cmd";
$command .= " --silent" if (!$verbose);
$command .= " --lang $lang" if ($lang);

if ($verbose) {
print STDERR "Deprecation notice: PO creation and update are now gulp tasks. See docs/development/internationalization.md\n";
print STDERR "Running `$command`\n";
}

system($command);
} else {
usage();
}

@@ -85,12 +94,9 @@ translate - Handle templates and preferences translation

=head1 SYNOPSYS

translate create fr-FR
translate update fr-FR
translate install fr-FR
translate install fr-FR -f search -f memberentry
translate -p install fr-FR
translate install
translate compress [fr-FR]
translate uncompress [fr-FR]

@@ -98,7 +104,7 @@ translate - Handle templates and preferences translation

In Koha, three categories of information are translated based on standard GNU
.po files: opac templates pages, intranet templates and system preferences. The
script is a wrapper. It allows to quickly create/update/install .po files for a
script is a wrapper. It allows to quickly install .po files for a
given language or for all available languages.

=head1 USAGE
@@ -107,38 +113,6 @@ Use the -v or --verbose parameter to make translator more verbose.

=over

=item translate create F<lang>

Create 3 .po files in F</misc/translator/po> subdirectory: (1) from opac pages
templates, (2) intranet templates, and (3) from preferences. English 'en'
version of templates and preferences are used as references.

=over

=item F<lang>-opac-{theme}.po

Contains extracted text from english (en) OPAC templates found in
<KOHA_ROOT>/koha-tmpl/opac-tmpl/{theme}/en/ directory.

=item F<lang>-intranet.po

Contains extracted text from english (en) intranet templates found in
<KOHA_ROOT>/koha-tmpl/intranet-tmpl/prog/en/ directory.

=item F<lang>-pref.po

Contains extracted text from english (en) preferences. They are found in files
located in <KOHA_ROOT>/koha-tmpl/intranet-tmpl/prog/en/admin/preferences
directory.

=back

=item translate [-p] update F<lang>

Update .po files in F<po> directory, named F<lang>-*.po. Without F<lang>, all
available languages are updated. With -p option, only preferences .po file is
updated.

=item translate [-p|-f] install F<lang>

Use .po files to translate the english version of templates and preferences files


+ 158
- 0
misc/translator/xgettext-installer View File

@@ -0,0 +1,158 @@
#!/usr/bin/perl

# This file is part of Koha.
#
# Koha is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# Koha is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Koha; if not, see <http://www.gnu.org/licenses>.

=head1 NAME

xgettext-installer - extract translatable strings from installer YAML files

=head1 SYNOPSIS

xgettext-installer [OPTION] [INPUTFILE]...

=head1 OPTIONS

=over

=item B<-f, --files-from=FILE>

get list of input files from FILE

=item B<-o, --output=FILE>

write output to the specified file

=item B<-h, --help>

display this help and exit

=back

=cut

use Modern::Perl;

use Getopt::Long;
use Locale::PO;
use Pod::Usage;
use YAML::Syck qw(LoadFile);

$YAML::Syck::ImplicitTyping = 1;

my $output = 'messages.pot';
my $files_from;
my $help;

GetOptions(
'output=s' => \$output,
'files-from=s' => \$files_from,
'help' => \$help,
) or pod2usage(-verbose => 1, -exitval => 2);

if ($help) {
pod2usage(-verbose => 1, -exitval => 0);
}

my @files = @ARGV;
if ($files_from) {
open(my $fh, '<', $files_from) or die "Cannot open $files_from: $!";
push @files, <$fh>;
chomp @files;
close $fh;
}

my $pot = {
'' => Locale::PO->new(
-msgid => '',
-msgstr =>
"Project-Id-Version: Koha\n"
. "PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"
. "Last-Translator: FULL NAME <EMAIL\@ADDRESS>\n"
. "Language-Team: Koha Translate List <koha-translate\@lists.koha-community.org>\n"
. "MIME-Version: 1.0\n"
. "Content-Type: text/plain; charset=UTF-8\n"
. "Content-Transfer-Encoding: 8bit\n"
),
};

for my $file (@files) {
my $yaml = LoadFile($file);
my @tables = @{ $yaml->{'tables'} };

my $tablec = 0;
for my $table (@tables) {
$tablec++;

my $table_name = ( keys %$table )[0];
my @translatable = @{ $table->{$table_name}->{translatable} };
my @rows = @{ $table->{$table_name}->{rows} };
my @multiline = @{ $table->{$table_name}->{'multiline'} };

my $rowc = 0;
for my $row (@rows) {
$rowc++;

for my $field (@translatable) {
if ( @multiline and grep { $_ eq $field } @multiline ) {
# multiline fields, only notices ATM
my $mulc;
foreach my $line ( @{ $row->{$field} } ) {
$mulc++;

# discard pure html, TT, empty
next if ( $line =~ /^(\s*<.*?>\s*$|^\s*\[.*?\]\s*|\s*)$/ );

# put placeholders
$line =~ s/(<<.*?>>|\[\%.*?\%\]|<.*?>)/\%s/g;

# discard non strings
next if ( $line =~ /^(\s|%s|-|[[:punct:]]|\(|\))*$/ or length($line) < 2 );
if ( not $pot->{$line} ) {
my $msg = new Locale::PO(
-msgid => $line,
-msgstr => '',
-reference => "$file:$table_name:$tablec:row:$rowc:mul:$mulc"
);
$pot->{$line} = $msg;
}
}
} elsif (defined $row->{$field} && length($row->{$field}) > 1 && !$pot->{ $row->{$field} }) {
my $msg = new Locale::PO(
-msgid => $row->{$field},
-msgstr => '',
-reference => "$file:$table_name:$tablec:row:$rowc"
);
$pot->{ $row->{$field} } = $msg;
}
}
}
}

my $desccount = 0;
for my $description ( @{ $yaml->{'description'} } ) {
$desccount++;
if ( length($description) > 1 and not $pot->{$description} ) {
my $msg = new Locale::PO(
-msgid => $description,
-msgstr => '',
-reference => "$file:description:$desccount"
);
$pot->{$description} = $msg;
}
}
}

Locale::PO->save_file_fromhash($output, $pot);

+ 151
- 0
misc/translator/xgettext-pref View File

@@ -0,0 +1,151 @@
#!/usr/bin/perl

# This file is part of Koha.
#
# Koha is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# Koha is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Koha; if not, see <http://www.gnu.org/licenses>.

=head1 NAME

xgettext-pref - extract translatable strings from system preferences YAML files

=head1 SYNOPSIS

xgettext-pref [OPTION] [INPUTFILE]...

=head1 OPTIONS

=over

=item B<-f, --files-from=FILE>

get list of input files from FILE

=item B<-o, --output=FILE>

write output to the specified file

=item B<-h, --help>

display this help and exit

=back

=cut

use Modern::Perl;

use File::Basename;
use Getopt::Long;
use Locale::PO;
use Pod::Usage;
use YAML::Syck qw(LoadFile);

$YAML::Syck::ImplicitTyping = 1;

my $output = 'messages.pot';
my $files_from;
my $help;

GetOptions(
'output=s' => \$output,
'files-from=s' => \$files_from,
'help' => \$help,
) or pod2usage(-verbose => 1, -exitval => 2);

if ($help) {
pod2usage(-verbose => 1, -exitval => 0);
}

my @files = @ARGV;
if ($files_from) {
open(my $fh, '<', $files_from) or die "Cannot open $files_from: $!";
push @files, <$fh>;
chomp @files;
close $fh;
}

my $pot = {
'' => Locale::PO->new(
-msgid => '',
-msgstr => "Project-Id-Version: Koha\n"
. "PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"
. "Last-Translator: FULL NAME <EMAIL\@ADDRESS>\n"
. "Language-Team: Koha Translate List <koha-translate\@lists.koha-community.org>\n"
. "MIME-Version: 1.0\n"
. "Content-Type: text/plain; charset=UTF-8\n"
. "Content-Transfer-Encoding: 8bit\n"
),
};

for my $file (@files) {
my $pref = LoadFile($file);
while ( my ($tab, $tab_content) = each %$pref ) {
add_po(undef, basename($file));

if ( ref($tab_content) eq 'ARRAY' ) {
add_prefs( $file, $tab, $tab_content );
} else {
while ( my ($section, $sysprefs) = each %$tab_content ) {
my $context = "$tab > $section";
my $msgid = sprintf('%s %s', basename($file), $section);
add_po($tab, $msgid);
add_prefs( $file, $context, $sysprefs );
}
}
}
}

Locale::PO->save_file_fromhash($output, $pot);

sub add_prefs {
my ($file, $context, $prefs) = @_;

for my $pref (@$prefs) {
my $pref_name = '';
for my $element (@$pref) {
if ( ref($element) eq 'HASH' ) {
$pref_name = $element->{pref};
last;
}
}
for my $element (@$pref) {
if ( ref($element) eq 'HASH' ) {
while ( my ( $key, $value ) = each(%$element) ) {
next unless $key eq 'choices' or $key eq 'multiple';
next unless ref($value) eq 'HASH';
for my $ckey ( keys %$value ) {
my $msgid = sprintf('%s#%s# %s', basename($file), $pref_name, $value->{$ckey});
add_po( "$context > $pref_name", $msgid );
}
}
}
elsif ($element) {
my $msgid = sprintf('%s#%s# %s', basename($file), $pref_name, $element);
add_po( "$context > $pref_name", $msgid );
}
}
}
}

sub add_po {
my ($comment, $msgid ) = @_;

return unless $msgid;

$pot->{$msgid} = Locale::PO->new(
-comment => $comment,
-msgid => $msgid,
-msgstr => '',