# Segmenter for gettext (.po) translation files

Inserts [ZWSP] between the segments of Chinese and Japanese text.

For Chinese, uses a high quality `zh_segmentation` model from Google: <https://tfhub.dev/google/zh_segmentation/1>.

For Japanese, uses [Sudachi](https://github.com/WorksApplications/sudachi.rs).

## Pre-requisites

1. Python. The easiest way to install Python on any Linux system is <https://github.com/asdf-vm/asdf>.
   On Windows you can use the [official installer](https://www.python.org/downloads/windows/).
   Note that the Python version must be [supported by tensorflow](https://www.tensorflow.org/install/pip#system-requirements) (this is usually not the latest Python version).

2. `gettext`. On Windows you can use [this installer](https://github.com/mlocati/gettext-iconv-windows/releases).

3. Python packages:

   ```shell
   pip install --upgrade -r tools/segmenter/requirements.txt
   ```

## Usage

To re-segment all the translation files:

```shell
tools/segmenter/segment_all.py
```

To re-segment the Chinese translation files:

```shell
tools/segmenter/segment_zh.py --input_path Translations/zh_CN.po
tools/segmenter/segment_zh.py --input_path Translations/zh_TW.po
```

To re-segment the Japanese translation files:

```shell
tools/segmenter/segment_ja.py --input_path Translations/ja.po
```

Additionaly, you can provide a different separator, such as `--separator='|'`, for debugging.

This tool performs a number of replacements to make sure interpolations are not affected etc.

You can also see the segmenter output for a given string like this:

```console
tools/segmenter/segment_zh.py --debug '返回到 {:d} 层'
```
```
返回｜到 {:d} 层
```

When inspecting the diffs, you can use `sed` to display the segments, e.g.:

```bash
git diff --color | sed "s/$(echo -ne '\u200B')/｜/g"
```

[ZWSP]: https://en.wikipedia.org/wiki/Zero-width_space