PO-Files for improving language handling

1 Initial Problem

The language handling mechanism of ILIAS – i.e. the management of all translated interface texts – relies on a very outdated scheme: language variables are maintained in proprietary .lang files and imported into the database at deployment time, from where the application reads them at runtime. This approach is maintenance-intensive, is hardly compatible with common translation tooling, and provides no structured solution for texts that differ depending on quantity (singular/plural) or for language variants such as dialects.

Modern approaches now exist to solve exactly this problem. gettext, a well-established industry standard built around template, translation and compiled files (POT, PO, MO), offers a considerably more future-proof and maintainable foundation for the language service.

2 Conceptual Summary

The .lang files are replaced by standard gettext files (POT, PO, MO), one set per module, stored in the `lang/` folder of the component that owns the module. ILIAS reads them with the PHP library `gettext/gettext`; the native PHP function `gettext()` is not used (reasons in 2.2). `txt()` stays the only API for existing code, a new method `ntxt()` adds singular/plural handling.

The migration runs module by module. Local changes of an installation (GUI edits, customizing files) are no longer written into the database or into shipped files, but into a small overlay file per module and language in the client data directory. During the transition, the database tables are still written in parallel, so that not yet migrated modules, plugins and a rollback keep working. Once all modules are migrated, the database is removed as storage and runtime source for language variables.

The ILIAS GUI for editing language variables remains available with the same functions; it now reads the shipped files plus the overlay and writes to the overlay.

A pilot implementation covers two modules: `tos` (TermsOfService) and `poll` (Poll, first module with plural forms).

2.1 Advantages over the current scheme

2.2 Handling of singular and plural forms

One file per module and language, the key stays the key. Each module has a template (`poll.pot`) and one translation file per language (`poll_de.po`, `poll_pl.po`, …) in `components/ILIAS/Poll/lang/`. The `msgid` is the existing language key, exactly the string that is passed to `txt()` today. No context (`msgctxt`) is used.

A simple entry looks like this (German file `poll_de.po`):

In words: the text for the key `poll_add` in German is "Abstimmung anlegen". This is the same information as the former .lang line `poll#:#poll_add#:#Abstimmung anlegen`; the module is given by the file instead of being repeated in every line.

File header. Every PO file starts with a header that describes the file (German file `poll_de.po`, shortened):

Plural forms. The `Plural-Forms` line of the header states how many forms the language has and which number selects which form (the plural rule). German and English have two forms, Polish has three, Arabic six.

A quantity-dependent entry has one text per form (German, from the pilot module `poll`):

In words: for one vote the text is "Eine Stimmabgabe", for any other number "%s Stimmabgaben". The Polish file has three texts `msgstr[0]`–`msgstr[2]` for the same key, following the Polish rule. The calling code only passes the number:

The calling code does not need to know the plural rules of any language. `ntxt()` is the ILIAS counterpart to `ngettext()`.

Why not the native PHP `gettext()`?

The file formats stay standard PO/POT/MO, so translation tools work unchanged. Native gettext may be added later as an optional acceleration with fallback, only if a measured performance problem requires it.

2.3 Migration of existing language data

Module by module. A module is migrated by converting its .lang lines into a template and one PO file per language, and by registering the module's language folder in the component (`$contribute[LanguageFileDirectory::class]` in the component class). Not yet migrated modules keep working as before. Modules without a component of their own (e.g. `common`) move into the Language component.

Local changes of an installation. Shipped files are never written at runtime. GUI edits and values from customizing files are stored in an overlay (a PO/MO pair per module and language below the client data directory) that contains only the entries differing from the shipped state. At runtime the overlay is applied on top of the shipped translations. On update, a three-way comparison (local value, previously shipped value, newly shipped value) keeps local changes and takes over changes of the shipped text wherever the entry was not changed locally.

Exception, as today for the .lang files: in translation mode, the maintenance action "merge local changes into the global language file" writes the local changes of a language into the shipped PO files, so that translations made in the GUI can be committed. The previous file is copied to `lang/customizing/` first.

MO files are build artifacts. They are not versioned; `setup build` (run by `composer install`) compiles the shipped PO files reproducibly. After changing a shipped PO file, `setup build` is part of the deployment.

Database and rollback. During the transition, the tables `lng_data` and `lng_modules` are still written for migrated modules. They therefore always hold a current state, and a rollback means removing files or code (per module, per language or completely), never restoring data. The tables are switched off only after all modules are migrated and the remaining direct database accesses outside the Language component (Notifications, DataCollection) have been changed.

2.4 Uniqueness of language variables

Within a module, uniqueness is guaranteed per file: an identifier assigned twice is reported as an error during conversion instead of silently overwriting the other one.

Across modules, the rule "the last loaded module wins" stays in place. `txt()` still receives only the identifier and has no module parameter (see 2.5), and the loaded modules are merged into one list at runtime, as before. The migration does not change which text is displayed.

2.5 Consequences for developers

2.6 Documentation (language.md)

The existing language.md still fully describes the old .lang/database structure and is revised as part of this FR. Going forward, it is to describe, among other things:

2.7 Language variants

A language variant is a version of an existing language that differs only in some texts, e.g. an "easy language" version of German or Swiss German. It is not a new language, but a layer on top of its base language.

Files. A variant ships, per module, only the entries that differ from the base language, in a PO file next to the base file (e.g. `poll_de.po` plus a variant file such as `poll_de@easy.po`; the naming scheme is settled during implementation). The variant uses the plural rule of its base language.

Runtime. For a user with a variant, a module is read in three layers, each one replacing entries of the previous one:

If the variant has no text for a key, the text of the base language is shown – per key and automatically.

The language key stays the base language. Everywhere outside the Language component, a user with a variant still has the language `de`: `getLangKey()` returns `de`, and all settings, database columns and content languages keep using the two-letter key. Only the Language component knows the variant. This keeps the change inside the Language component: ILIAS identifies languages by two-letter keys in several hundred places and in many database columns, and regional keys such as `de_CH` would require changes across many components.

Selecting a variant.

Limits. A variant is only used for the interface texts of the current user. Mails to other users (`txtlng()`), the HTML `lang` attribute and content languages keep using the base language. Variants are not written to the database tables during the transition; being new, they need no rollback state. A language with its own ISO code remains a new language (a further set of PO files), not a variant.

3 User Interface Modifications

3.1 List of Affected Views

3.2 User Interface Details

No new views and no changed workflows. Small changes in the existing views:

3.3 New User Interface Concepts

No new UI concepts.

3.4 Accessibility Implications

No changes regarding accessibility.

4 Additional Information

4.1 Involved Authorities

4.2 Technical Aspects

The following are brief explanations of the terms gettext, POT file, PO file and MO file:

Reading and writing of these files is encapsulated in the Language component behind a narrow adapter around the library `gettext/gettext`, so a later change of library stays local. `txt()` and `ntxt()` read the compiled MO files plus the local overlay (see 2.2, 2.3).

4.3 Privacy

No changes.

4.4 Security

4.5 Contact

Person to be contacted in case of questions about the feature or for funding offers:  Auerbach, Jeanine [jeanine.auerbach]

4.6 Funding

Funding status and funding parties are listed in the block 'Status of Feature' in the right column of this page.

If you are interested to give funding for this feature, please get into contact with the person mentioned above as 'Contact'.

5 Discussion

6 Implementation

Feature has been implemented by {Please add related profile link of this person}

6.1 Description and Screenshots

{ Description of the final implementation and screenshots if possible. }

6.2 Test Cases

Test cases completed at {date} by {user}

6.3 Privacy

Information in privacy.md of component: updated at {date} by {user} | no change required

6.4 Approval

Approved at {date} by {user}.