Jp itn 20240221 by BuyuanCui · Pull Request #141 · NVIDIA/NeMo-text-processing

BuyuanCui · 2024-02-21T21:03:28Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.
PR for Japanese itn instead of #101

Before your PR is "Ready for review"

Pre checks:

[ *] Have you signed your commits? Use git commit -s to sign.
[ *] Do all unittests finish successfully before sending PR?
1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
[ *] If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
[* ] Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
[ *] Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
[ *] Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
[ *] Remove import guards (try import: ... except: ...) if not already done.
If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
[ *] Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

[ *] New Feature
[ *] Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

Signed-off-by: Alex Cui <[email protected]>

for more information, see https://pre-commit.ci

+from nemo_text_processing.inverse_text_normalization.ja.graph_utils import (
+    INPUT_LOWER_CASED,
+    NEMO_SIGMA,
+    GraphFst,
+    delete_extra_space,
+    delete_space,
+    generator_main,
+    insert_space,
+)


+from nemo_text_processing.inverse_text_normalization.ja.taggers.decimal import DecimalFst
+from nemo_text_processing.inverse_text_normalization.ja.taggers.fraction import FractionFst
+from nemo_text_processing.inverse_text_normalization.ja.taggers.ordinal import OrdinalFst
+from nemo_text_processing.inverse_text_normalization.ja.taggers.preprocessor import PreProcessorFst


Signed-off-by: Alex Cui <[email protected]>

…-processing into jp_itn_20240221

for more information, see https://pre-commit.ci

Signed-off-by: Alex Cui <[email protected]>

…-processing into jp_itn_20240221

Signed-off-by: Alex Cui <[email protected]>

github-actions · 2024-03-09T01:45:20Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

github-actions · 2024-03-16T01:46:16Z

This PR was closed because it has been inactive for 7 days since being marked as stale.

Applying updates to the docker file Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Included Japanese Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Updates and fixing on the docker file location issue. Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Signed-off-by: Alex Cui <[email protected]>

github-actions · 2024-05-04T01:47:33Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

Signed-off-by: Alex Cui <[email protected]>

Signed-off-by: Buyuan(Alex) Cui <[email protected]>

for more information, see https://pre-commit.ci

BuyuanCui · 2024-07-10T00:51:19Z

The updated files from previous round of review.

Jenkinsfile (to add test for Japanese and date update)
tests/nemo_text_processing/ja/data_inverse_text_normalization/test_cases_fraction.txt (changed the text format, changed 1 3/4 to 1荷3/4 and 1と3/4)
tools/text_processing_deployment/pynini_export.py (added ITN Pocess fst for space issue)
nemo_text_processing/inverse_text_normalization/ja/verbalizers/post_processing.py (to resolve space issue)
updates later received but also already merged to the main including the files under tools/text_processing_deployment

Signed-off-by: Alex Cui <[email protected]>

Signed-off-by: Buyuan(Alex) Cui <[email protected]>

…-processing into jp_itn_20240221

Signed-off-by: Alex Cui <[email protected]>

BuyuanCui · 2024-07-16T15:50:43Z

Updating according to XInchao's comment.

Signed-off-by: Alex Cui <[email protected]>

BuyuanCui · 2024-07-16T20:22:47Z

updated file nemo_text_processing/inverse_text_normalization/ja/verbalizers/date.py at line 36 for comments
updated file nemo_text_processing/inverse_text_normalization/ja/taggers/date.py at line 37 and from line 89 to line 108

* adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja gramamrs Signed-off-by: Alex Cui <[email protected]> * adding ja tests Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * format fixs Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removing commented out tests Signed-off-by: Alex Cui <[email protected]> * updated for japanese Signed-off-by: Alex Cui <[email protected]> * added japanese for tests Signed-off-by: Alex Cui <[email protected]> * Update Dockerfile Applying updates to the docker file Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update export_grammars.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update pynini_export.py Included Japanese Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update sh_test.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update launch.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update test_sparrowhawk_inverse_text_normalization.sh Updates and fixing on the docker file location issue. Signed-off-by: Buyuan(Alex) Cui <[email protected]> * restyle Signed-off-by: Alex Cui <[email protected]> * updates Signed-off-by: Alex Cui <[email protected]> * updates Signed-off-by: Alex Cui <[email protected]> * configuration issue Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * copied from main; Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * typo Signed-off-by: Alex Cui <[email protected]> * typo Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issues Signed-off-by: Alex Cui <[email protected]> * updated fraction tagger and test cases Signed-off-by: Alex Cui <[email protected]> * changed data format for corrections Signed-off-by: Alex Cui <[email protected]> * added back all 6 tests Signed-off-by: Alex Cui <[email protected]> * added ja on line 168 Signed-off-by: Alex Cui <[email protected]> * updated the grammar to resolve empty sparrohawt test result Signed-off-by: Alex Cui <[email protected]> * updated for empty SH test result Signed-off-by: Alex Cui <[email protected]> * updates for japanese on dates and tests Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added ITNPostProcessingFst as None Signed-off-by: Alex Cui <[email protected]> * Update pynini_export.py added ITNPOstProcessorFst as None Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update ordinal.py remioviung unused imports Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update ordinal.py removing unnecessary comments Signed-off-by: Buyuan(Alex) Cui <[email protected]> * removing unsed import Signed-off-by: Alex Cui <[email protected]> * update for japanese itn Signed-off-by: Alex Cui <[email protected]> * fixing typo Signed-off-by: Alex Cui <[email protected]> * updates on era abbreviations Signed-off-by: Alex Cui <[email protected]> * updates on era abbreviations Signed-off-by: Alex Cui <[email protected]> * updates on the normalization comments Signed-off-by: Alex Cui <[email protected]> --------- Signed-off-by: Alex Cui <[email protected]> Signed-off-by: Buyuan(Alex) Cui <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Alex Cui <[email protected]>

* adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja gramamrs Signed-off-by: Alex Cui <[email protected]> * adding ja tests Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * format fixs Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removing commented out tests Signed-off-by: Alex Cui <[email protected]> * updated for japanese Signed-off-by: Alex Cui <[email protected]> * added japanese for tests Signed-off-by: Alex Cui <[email protected]> * Update Dockerfile Applying updates to the docker file Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update export_grammars.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update pynini_export.py Included Japanese Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update sh_test.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update launch.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update test_sparrowhawk_inverse_text_normalization.sh Updates and fixing on the docker file location issue. Signed-off-by: Buyuan(Alex) Cui <[email protected]> * restyle Signed-off-by: Alex Cui <[email protected]> * updates Signed-off-by: Alex Cui <[email protected]> * updates Signed-off-by: Alex Cui <[email protected]> * configuration issue Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * copied from main; Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * typo Signed-off-by: Alex Cui <[email protected]> * typo Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issues Signed-off-by: Alex Cui <[email protected]> * updated fraction tagger and test cases Signed-off-by: Alex Cui <[email protected]> * changed data format for corrections Signed-off-by: Alex Cui <[email protected]> * added back all 6 tests Signed-off-by: Alex Cui <[email protected]> * added ja on line 168 Signed-off-by: Alex Cui <[email protected]> * updated the grammar to resolve empty sparrohawt test result Signed-off-by: Alex Cui <[email protected]> * updated for empty SH test result Signed-off-by: Alex Cui <[email protected]> * updates for japanese on dates and tests Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added ITNPostProcessingFst as None Signed-off-by: Alex Cui <[email protected]> * Update pynini_export.py added ITNPOstProcessorFst as None Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update ordinal.py remioviung unused imports Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update ordinal.py removing unnecessary comments Signed-off-by: Buyuan(Alex) Cui <[email protected]> * removing unsed import Signed-off-by: Alex Cui <[email protected]> * update for japanese itn Signed-off-by: Alex Cui <[email protected]> * fixing typo Signed-off-by: Alex Cui <[email protected]> * updates on era abbreviations Signed-off-by: Alex Cui <[email protected]> * updates on era abbreviations Signed-off-by: Alex Cui <[email protected]> * updates on the normalization comments Signed-off-by: Alex Cui <[email protected]> --------- Signed-off-by: Alex Cui <[email protected]> Signed-off-by: Buyuan(Alex) Cui <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Namrata Gachchi <[email protected]>

* adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja Signed-off-by: Alex Cui <[email protected]> * adding ja gramamrs Signed-off-by: Alex Cui <[email protected]> * adding ja tests Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * format fixs Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removing commented out tests Signed-off-by: Alex Cui <[email protected]> * updated for japanese Signed-off-by: Alex Cui <[email protected]> * added japanese for tests Signed-off-by: Alex Cui <[email protected]> * Update Dockerfile Applying updates to the docker file Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update export_grammars.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update pynini_export.py Included Japanese Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update sh_test.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update launch.sh Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update test_sparrowhawk_inverse_text_normalization.sh Updates and fixing on the docker file location issue. Signed-off-by: Buyuan(Alex) Cui <[email protected]> * restyle Signed-off-by: Alex Cui <[email protected]> * updates Signed-off-by: Alex Cui <[email protected]> * updates Signed-off-by: Alex Cui <[email protected]> * configuration issue Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * copied from main; Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * typo Signed-off-by: Alex Cui <[email protected]> * typo Signed-off-by: Alex Cui <[email protected]> * copied from main Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issue Signed-off-by: Alex Cui <[email protected]> * space issues Signed-off-by: Alex Cui <[email protected]> * updated fraction tagger and test cases Signed-off-by: Alex Cui <[email protected]> * changed data format for corrections Signed-off-by: Alex Cui <[email protected]> * added back all 6 tests Signed-off-by: Alex Cui <[email protected]> * added ja on line 168 Signed-off-by: Alex Cui <[email protected]> * updated the grammar to resolve empty sparrohawt test result Signed-off-by: Alex Cui <[email protected]> * updated for empty SH test result Signed-off-by: Alex Cui <[email protected]> * updates for japanese on dates and tests Signed-off-by: Alex Cui <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added ITNPostProcessingFst as None Signed-off-by: Alex Cui <[email protected]> * Update pynini_export.py added ITNPOstProcessorFst as None Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update ordinal.py remioviung unused imports Signed-off-by: Buyuan(Alex) Cui <[email protected]> * Update ordinal.py removing unnecessary comments Signed-off-by: Buyuan(Alex) Cui <[email protected]> * removing unsed import Signed-off-by: Alex Cui <[email protected]> * update for japanese itn Signed-off-by: Alex Cui <[email protected]> * fixing typo Signed-off-by: Alex Cui <[email protected]> * updates on era abbreviations Signed-off-by: Alex Cui <[email protected]> * updates on era abbreviations Signed-off-by: Alex Cui <[email protected]> * updates on the normalization comments Signed-off-by: Alex Cui <[email protected]> --------- Signed-off-by: Alex Cui <[email protected]> Signed-off-by: Buyuan(Alex) Cui <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

BuyuanCui and others added 6 commits February 21, 2024 09:14

adding ja

317a178

Signed-off-by: Alex Cui <[email protected]>

adding ja

31dae98

Signed-off-by: Alex Cui <[email protected]>

adding ja

43a94d5

Signed-off-by: Alex Cui <[email protected]>

adding ja gramamrs

2bab707

Signed-off-by: Alex Cui <[email protected]>

adding ja tests

82555e9

Signed-off-by: Alex Cui <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

e6111f6

for more information, see https://pre-commit.ci

github-advanced-security AI found potential problems Feb 21, 2024

View reviewed changes

BuyuanCui and others added 7 commits February 21, 2024 13:33

format fixs

cb24883

Signed-off-by: Alex Cui <[email protected]>

Merge branch 'jp_itn_20240221' of https://github.com/NVIDIA/NeMo-text…

3931987

…-processing into jp_itn_20240221

[pre-commit.ci] auto fixes from pre-commit.com hooks

e1e10a6

for more information, see https://pre-commit.ci

removing commented out tests

eed0c27

Signed-off-by: Alex Cui <[email protected]>

Merge branch 'jp_itn_20240221' of https://github.com/NVIDIA/NeMo-text…

d80789b

…-processing into jp_itn_20240221

updated for japanese

6e26901

Signed-off-by: Alex Cui <[email protected]>

added japanese for tests

b610752

Signed-off-by: Alex Cui <[email protected]>

github-actions Bot added the Stale label Mar 9, 2024

github-actions Bot closed this Mar 16, 2024

BuyuanCui reopened this Apr 17, 2024

BuyuanCui added 6 commits April 16, 2024 18:55

Update Dockerfile

e3bba6f

Applying updates to the docker file Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Update export_grammars.sh

21e410f

Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Update pynini_export.py

b53e864

Included Japanese Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Update sh_test.sh

ce6b44b

Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Update launch.sh

d31d854

Updates from main Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Update test_sparrowhawk_inverse_text_normalization.sh

5914183

Updates and fixing on the docker file location issue. Signed-off-by: Buyuan(Alex) Cui <[email protected]>

github-actions Bot removed the Stale label Apr 18, 2024

restyle

c6e7a15

Signed-off-by: Alex Cui <[email protected]>

ekmb mentioned this pull request Apr 30, 2024

Jp itn #101

Closed

3 tasks

updates

c5e7c87

Signed-off-by: Alex Cui <[email protected]>

BuyuanCui and others added 3 commits July 9, 2024 15:12

updates for japanese on dates and tests

d654198

Signed-off-by: Alex Cui <[email protected]>

Merge branch 'main' into jp_itn_20240221

97b1bd9

Signed-off-by: Buyuan(Alex) Cui <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

10bb48c

for more information, see https://pre-commit.ci

github-advanced-security AI found potential problems Jul 9, 2024

View reviewed changes

Comment thread nemo_text_processing/inverse_text_normalization/ja/verbalizers/ordinal.py Fixed

github-actions Bot removed the Stale label Jul 10, 2024

BuyuanCui and others added 8 commits July 10, 2024 08:01

added ITNPostProcessingFst as None

4334750

Signed-off-by: Alex Cui <[email protected]>

Update pynini_export.py added ITNPOstProcessorFst as None

d63d7b3

Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Update ordinal.py remioviung unused imports

2df505e

Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Update ordinal.py removing unnecessary comments

3df7d0b

Signed-off-by: Buyuan(Alex) Cui <[email protected]>

Merge branch 'jp_itn_20240221' of https://github.com/NVIDIA/NeMo-text…

15274ef

…-processing into jp_itn_20240221

removing unsed import

f57a8ac

Signed-off-by: Alex Cui <[email protected]>

update for japanese itn

e16954c

Signed-off-by: Alex Cui <[email protected]>

fixing typo

21604ab

Signed-off-by: Alex Cui <[email protected]>

BuyuanCui added 3 commits July 16, 2024 13:15

updates on era abbreviations

5053f90

Signed-off-by: Alex Cui <[email protected]>

updates on era abbreviations

f6b2d44

Signed-off-by: Alex Cui <[email protected]>

updates on the normalization comments

17a38af

Signed-off-by: Alex Cui <[email protected]>

ekmb approved these changes Jul 19, 2024

View reviewed changes

ekmb merged commit 1d1374b into main Jul 19, 2024

ekmb deleted the jp_itn_20240221 branch July 19, 2024 00:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jp itn 20240221#141

Jp itn 20240221#141
ekmb merged 58 commits into
mainfrom
jp_itn_20240221

BuyuanCui commented Feb 21, 2024

Uh oh!

Check notice

Check notice

Uh oh!

Uh oh!

github-actions Bot commented Mar 9, 2024

Uh oh!

github-actions Bot commented Mar 16, 2024

Uh oh!

github-actions Bot commented May 4, 2024

Uh oh!

Uh oh!

BuyuanCui commented Jul 10, 2024

Uh oh!

BuyuanCui commented Jul 16, 2024

Uh oh!

BuyuanCui commented Jul 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

BuyuanCui commented Feb 21, 2024

What does this PR do ?

Before your PR is "Ready for review"

Uh oh!

Check notice

Check notice

Uh oh!

Uh oh!

github-actions Bot commented Mar 9, 2024

Uh oh!

github-actions Bot commented Mar 16, 2024

Uh oh!

github-actions Bot commented May 4, 2024

Uh oh!

Uh oh!

BuyuanCui commented Jul 10, 2024

Uh oh!

BuyuanCui commented Jul 16, 2024

Uh oh!

BuyuanCui commented Jul 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants