File Format =========== The configuration files are in `TOML format `_. Below, we document the semantics of a well-formed TOML file, and how they are validated. Paths can include environment references using the syntax ``{reference}``. Undefined references are replaced with an empty string. .. todo:: Should variable expansion also apply to non-path values? Top-level Elements ------------------ ``basepath`` Any relative path in the configuration is resolved against this path. ``locales`` List of locale codes to enable for the described project. This is a list of BCP 47 locale identifiers, with the notable exception of ``ja-JP-mac``, which we temporarily need to keep for compatibility reasons. ``env`` Table of environment aliases, which allows to define short-hands to be used in the configuration file. Example: .. code-block:: toml [env] l = "{locale}/" ``build`` Table of build configuration entries. This is a good location to store locale lists for special builds. Example: .. code-block:: toml [build] exclude-multi-locale = [ "bn-BD", "zam", ] test-single-locale = [ "it", "he", "zh-TW", ] This could for example mean that from all the locales that are in a project (think Firefox for Android), some are excluded from a multi-locale build and only part of the single-locale builds. In addition, we could have a short list of locales that we always want to generate single-locale builds for to make sure they don't break? The builds to actually use and test would be the multi-locale build, though. The entries in ``build`` are project specific. Includes -------- The l10n configuration files can include external configuration files to make the whole configuration system modular. This is done by adding ``includes`` tables to the file. .. code-block:: toml [[includes]] path = "some/file/path.toml" The paths are resolved like this (``filedir`` being the path to the directory of the configuration file): ``filedir/{basepath}/path``. The global localization scope is defined by a root configuration file, which utilizes ``includes`` to define projects. Projects themselves can refer to further configuration files, to reuse the l10n configuration of shared code fragments. .. todo:: Define what makes a project. Can projects include projects? Excludes -------- Top-level configuration files can also exclude an external configuration file. This can be used to define localization projects where one excludes the other. When excluding a configuration with includes, all child configurations of that are also excluded. It's an error to use an exclude in a child configuration. If an included configuration (or the top-level configuration) explicitly includes an excluded sub-configuration file, the file is included, i.e., includes have precedence over excludes. There's a detailed example in :ref:`excludes_example`. Paths ----- This is the heart of the actual configuration, describing where to find the reference files, and where to put the localized files. In defining the localization paths, use the ``{locale}`` variable to denote the locale code. Use ``*`` to denote a single level wildcard, or ``**`` to denote all files in the given directory hierarchy. Example: .. code-block:: toml [[paths]] reference = "locales/en-US/my-file.ftl" l10n = "locales/{locale}/my-file.ftl" For bilingual file formats, it's acceptable to just specify the ``l10n`` path, though it's encouraged to define the ``reference`` to enable more tests. To limit a particular path pattern to a subset of locales, you can specify an optional ``locales`` entry: .. code-block:: toml [[paths]] reference = "locales/en-US/my-file.ftl" l10n = "locales/{locale}/my-file.ftl" locales = [ "de", "fr", ] The list of locales for a particular path is assumed to be a subset of the top-level ``locales`` entry. Additional locales will be ignored. The ``tests`` entry is optional and allows to specify additional tests to be run on this set of files. One example would be to run platform-specific tests, for example on Android, on a particular set of files. .. code-block:: toml [[paths]] reference = "mobile/android/base/locales/en-US/**" l10n = "{l}mobile/android/base/**" test = [ "android-dtd", ] .. todo:: Do we need a way to disable tests? Something locale specific? Filters ------- In some cases localizations can differ from the reference locale. For example it can be fine for some files to be missing, for particular entries to be missing in a file, or to have extra entries. Filters allow the configuration to specify what's acceptable. The filter table entries are defined as follows ``path`` A full file path, or a list of file paths. Wildcards (``*``, ``**``) and expansions are allowed. ``key`` A key name, or list of key names that are ignored. If you want to specify a set of keys, you can use regular expressions by prefixing the key entry with ``re:``. This entry is optional and, if missing, the filter will only apply to missing or additional files in the localization. If you want to ignore all entries in a file, specify a generic key, ``re:.``. ``action`` What to do when the filter matches. The value should be one of ``error`` Report the missing or obsolete string or file. This is the default behavior, but it can be useful to make decision on particular files or keys before taking more generic actions. ``ignore`` Don't report the missing or obsolete string or file. ``warning`` Report the missing string or file but, for example, don't take action when doing l10n-merge. This used to be called ``report`` in ``filter.py``. For any particular project, a string or a file of a localization has a status within that particular project, and a status within the global configuration. As the global l10n scope and the projects scopes are defined by aggregating individual configuration files, the status of a string or file is also determined by how the statuses of the string from each configuration aggregate. Evaluation ^^^^^^^^^^ In a single configuration file, the *filter* is evaluated as the action of the first rule with a matching ``path`` and ``key`` pattern. Both ``path`` and ``key`` can be arrays, which is just a shorthand for a filter rule for each combination. .. code-block:: toml [[filters]] path = [ "browser", "toolkit", ] action = "ignore" is a shorthand for .. code-block:: toml [[filters]] path = "browser" action = "ignore" [[filters]] path = "toolkit" action = "ignore" If none of the path and key combinations match, there is no resulting action. The status of a file or string in a project is ``ignore`` if any filter returns ``ignore``. The final status of a file or string is ``ignore`` if all project filters return ``ignore``. If all project filters return ``ignore`` or ``warning``, the status is ``warning``, otherwise ``error``. Thus, a string or file that's not covered by any filter has an ``error`` status. Here's pseudo code in Python that demonstrates the algorithm. .. code-block:: python def filter(config, path, key=None): """Determine the status of a file or string in a configuration file. """ for rule in config.filter_rules: if not rule.path.matches(path): continue if key is None and rule.key is None: return rule.action if key is not None and rule.key is not None: if rule.key.matches(key): return rule.action def project_status(project, path, key=None, global=True): """Determine the status of a file or string in a project with a set of configuration files. """ default_action = 'error' if global else 'ignore' return aggregate( (filter(config, path, key) for config in project.configs), 'ignore', default=default_action) def global_status(global_scope, path, key=None): """Determine the status of a file or string in the global context. """ return aggregate( (project_status(project, path, key) for project in global_scope.projects), 'error', default='error') def aggregate(actions_iterable, stop_action, default=None): """Helper function to aggregate actions over an iterable. """ actions = set() for action in actions_iterable: if action == stop_action: return stop_action actions.add(action) for aggregated_action in ('error', 'warning', 'ignore'): if aggregated_action in actions: return aggregated_action return default This evaluation of filters builds a system that allows for strings and files to be ignored for a particular project, but if any project needs the string, it needs to be translated. At the same time, files or strings not used anywhere are reported as obsolete, and can be removed. Meta Data --------- Meta data for a project is defined in the configuration repository only, and maintained by l10n-drivers. It's used to describe project management-related data about a project, like priorities, schedules, descriptions. Meta data can also be restricted to particular files or locales. *Example*: .. code-block:: toml [[project]] name = "Firefox" description = """The mozilla Desktop User Agent""" [[project.import]] repository = "https://hg.mozilla.org/mozilla-central" revision = "default" path = "browser/locales/l10n.toml" vcs = "mercurial" [[project.metadata]] priority = 2 flags = [ ] [[project.metadata.schedule]] when = 2017-02-29T00:00:00Z title = "Code freeze" [[project.metadata.schedule]] when = 2017-03-29T00:00:00Z title = "Go-Live to production" notify = 2017-03-22T00:00:00Z [[project.metadata]] locales = [ "de", ] priority = 1 [[project.metadata]] locales = [ "fr", ] path = "{l}browser/some/directory/file.properties" priority = 1