0. 说明

Google-ZYX指有VAE
Phoneme-HCSI指中文使用实验室G2P和韵律, 英文用KbGit和替换空格
DBMIX指双语/混语语料为标贝买的
为了和'春春'语料统一, 目前只使用中文10000和英文2000句

1. 代码调整

1.1. 拷贝之前的项目

使用Git: https://github.com/ruclion/Fantasy_Mix-Lingual_Tacotron_Version_2_Google-ZYX

项目为: /ceph/home/hujk17/Fantasy_Mix-Lingual_Tacotron_Version_4_Google-ZYX-Phoneme-HCSI-DBMIX

这个项目的Git: https://github.com/ruclion/Fantasy_Mix-Lingual_Tacotron_Version_4_Google-ZYX-Phoneme-HCSI-DBMIX

1.2. G2P_CN_HCSI得到中文Phoneme文本

1.2.1. 标贝中文的处理

主要是databaker_G2P.py, 在本项目下有一份: /ceph/home/hujk17/Fantasy_Mix-Lingual_Tacotron_Version_4_Google-ZYX-Phoneme-HCSI-DBMIX/G2P_CN_HCSI

这一份是不会变的

还有一份在: /ceph/home/hujk17/G2P_CN_HCSI

https://github.com/ruclion/G2P_CN_HCSI

这一份可能随着G2P的增加和语料处理的增加而改

将得到的train.txt改名字为DBMIX_CN_meta.csv.txt

1.2.2. 中文的pinyin和symbol和韵律

_来分割symbol, pinyin_G2P_2.py可以分解pinyin到symbols(声韵母)

韵律对于标贝的天然有实验室的代码

1.2.3. 声调embedding到symbol上

参看文献一, 2020阿里:

Instead of using a unified phone set across languages, we combine English and Mandarin phone sets together as a whole. For English utterances, we use 44 British English phoneme symbols plus 3 possible stress symbols. For Mandarin utterances, we use 62 Pinyin initials and finals plus 5 possible tones. The tone or stress symbols are attached to the corresponding phoneme symbols. We also use symbols to indicate in-utterance pauses and utterance ends.
[1] 理解: 中文音调1, 2, 3, 4, 5, 6. 英文重读7, 8, 9(0, 1, 2 + 7); 声母和辅音均使用韵母和元音的标记
[2] 理解: 中文音调1, 2, 3, 4, 5, 6. 英文重读7, 8, 9(0, 1, 2 + 7); 而没有音调的声母用10占位, 没有重音的辅音用11占位
[3] 还有一种理解: 0代表没有, 比如声母无音调, 英文无音调, 0代表没有, 辅音无重读, 中文无重读
但是我感觉[1]简单, 并且分开类别, 不同特征类别来描述不同特点的信息, 即使交叉也没问题. (川哥有一次分享了一篇BC的论文, 补上去TODO...)

参看文献二, 2019谷歌:

[未懂TODO...] Characters /Graphemes: Embeddings corresponding to each character or grapheme are the default inputs for end-to-end TTS models [2, 20, 23], requiring the model to implicitly learn how to pronounce input words (i.e. grapheme-to-phoneme conversion [26]) as part of the synthesis task. Extending a grapheme-based input vocabulary to a multilingual setting is straightforward, by simply concatenating grapheme sets in the training corpus for each language. This can grow quickly for languages with large alphabets, e.g. our Mandarin vocabulary contains over 4.5k tokens. We simply concatenate all graphemes appearing in the training corpus, leading to a total of 4,619 tokens. Equivalent graphemes are shared across languages. During inference all previously unseen characters are mapped to a special out-of-vocabulary (OOV) symbol.
Phonemes: Using phoneme inputs simplifies the TTS task, as the model no longer needs to learn complicated pronunciation rules for languages such as English. Similar to our grapheme-based model, equivalent phonemes are shared across languages. We concatenate all possible phoneme symbols, for a total of 88 tokens. To support Mandarin, we include tone information by learning phoneme-independent embeddings for each of the 4 possible tones, and broadcast each tone embedding to all phoneme embeddings inside the corresponding syllable. For English and Spanish, tone embeddings are replaced by stress embeddings which include primary and secondary stresses. A special symbol is used when there is no tone or stress.
[4] 理解: 中文音调1, 2, 3, 4. 英文重读5, 6(1, 2 + 4), 0代表轻声或者无重读; 音节共用一个标记, 保证音节内的集中统一; 使用IPA, 一共88个字符

参看文献三, 2020港中文:

English words and Chinese characters are transcribed as phonemes as input with stress and tonal information respectively.

目前采用理解[2], 原因是简单, 并且会实现, 也基本上有道理, 下面是不会的:

"I have $250 in my pocket.", # number -> spell-out
['AY1', ' ', 'HH', 'AE1', 'V', ' ', 'T', 'UW1', ' ', 'HH', 'AH1', 'N', 'D', 'R', 'AH0', 'D', ' ', 'F', 'IH1', 'F', 'T', 'IY0', ' ', 'D', 'AA1', 'L', 'ER0', 'Z', ' ', 'IH0', 'N', ' ', 'M', 'AY1', ' ', 'P', 'AA1', 'K', 'AH0', 'T', ' ', '.']
上述的辅音不知道如何划分音节, 需要懂一些专家知识

目前采用方案, 音调跟着前面的symbol, 重音跟着前面的symbol, 如果symbols后面没有数字, 则用10或者11

1.3. G2P_EN_Kb得到英文Phoneme文本

自带phoneme, 但不懂HCSI的规则, 所以之中英文句子, 接上G2P_EN_Kb

具体流程见代码

得到

1.4. preprocess.py

1.4.1. 不放心的

#M-AILABS (and other datasets) trim params (there parameters are usually correct for any data, but definitely must be tuned for specific speakers)

trim_fft_size = 512,

trim_hop_size = 128,

trim_top_db = 63,

这个用的春春的, 并不是DB的, 可能会有些问题, 但是目前没有管

1.4.2. 代码路径微调

略, 重新写好就行

1.4.3. 异常

有些wav有异常, 导致preprocess时候会莫名其妙结束, 改为全部的异常catch即可

1.4.4. 结果

1.4. preprocess.py

1.4.1. EN的phoneme文件形成symbols

路径: /ceph/home/hujk17/Fantasy_Mix-Lingual_Tacotron_Version_4_Google-ZYX-Phoneme-HCSI-DBMIX/G2P_EN_Kb/databaker_MIX_Phoneme/DBMIX_EN_symbolsList_symbol_split.csv.txt

1.4.2. CN的phoneme文件形成symbols

路径: /ceph/home/hujk17/Fantasy_Mix-Lingual_Tacotron_Version_4_Google-ZYX-Phoneme-HCSI-DBMIX/G2P_CN_HCSI/databaker_MIX_Phoneme/DBMIX_CN_symbolsList_symbol_split.csv.txt

1.4.3. 合并

合并的时候发现

英文的0, 1, 2忘了变成789, 后面没有重音的(辅音和韵律符号), 补上11
中文的后面没有音调的(声母和韵律符号), 补上10
这样一来在tacotron的text_to_sequence_MIX_Phoneme_Version就不需要区分中文和英文符号了
中文中有一些英文, 先不管. TODO...

TODO, 需要同步到Git上面

EN:

CN:

MIX:

training_data/train.txt重新做:

1.5. Feeder和Train和Tacotron改动

inputs->phoneme和tone_stress

挺麻烦的, 不过常规. 过程略

2. 训练

如何保证代码没错呢?

目前训练的还不错, 先不统计哪一步alignment就收敛了, 先往下走

3. 合成

When I found out about her death I was shocked, but not surprised, she said.
The latter serve as a worm aphrodisiac, getting the hermaphroditic worms to breed more often.
Artistic gymnastics, rhythmic gymnastics, trampoline, weightlifting, handball.
那些庄稼田园在果果眼里感觉太亲切了
她把鞋子拎在手上光着脚丫故意踩在水洼里
我为男主角感到有点遗憾
When I found out about her death 她把鞋子拎在手上 I was shocked, but not surprised, she said.
她把鞋子拎在手上 When I found out about her death 光着脚丫故意踩在水洼里

Fantasy Mix-Lingual Tacotron Version 4: Google-ZYX-Phoneme-HCSI-DBMIX相关推荐

不断开心的今天：Google Chrome Englive.cn iPhone Version。
我们iPhone版性能问题一直纠缠着我,现在上线的版本用iphone访问,页面加载时间高达14秒之久,即便我们已经完成了流量优化,页面间跳转均增量只有10K左右甚至更小,但是加载时间完全是巨无霸,并且 ...
移动硬盘改台式机硬盘_如何在台式机或移动设备上离线使用Google云端硬盘
移动硬盘改台式机硬盘 If there's any drawback to using cloud-based services for all your productivity and organ ...
android 更改软键盘_如何在Android的Google键盘上更改声音和振动
android 更改软键盘 Tactile feedback from a touch screen keyboard is crucial, in my opinion, but I don't l ...
谷歌 colab_如何在Google Colab上使用熊猫分析
谷歌 colab Recently, pandas have come up with an amazing open-source library called pandas-profiling. ...
如何了解Google Analytics（分析）和网站访问量
Google Analytics is a powerful web service that gives you insights into your website. But exactly ca ...
[gtest][002] A quick start to build the Google C++ Testing project
前言:上一篇谷歌的测试的框架,大致介绍了测试的框架的基本情况.这一节,讲述如何构建一个真正的自动化测试项目: 注意,文章最后一部分有一个视频,这个视频是一步一步构建谷歌测试案例,值得仔细研究. Uni ...
谷歌 console_使用Google Search Console有效增加网站流量的15条提示
谷歌 console Google Search Console is a powerful free tool created by Google to help website owners un ...
如何判断您是否拥有32位或64位版本的Google Chrome浏览器
Google Chrome is extremely popular with our readers, but did you know that they also have a 64-bit v ...
2021年在vue中使用 Google Map
目前在vue中使用google map有以下两种方法: 一.使用 JavaScript 创建地图实例该方法优点在于可配置项多,方便自定义,缺点是较复杂,且文档为英文,有一定学习难度. 在开始之前确保 ...

Fantasy Mix-Lingual Tacotron Version 4: Google-ZYX-Phoneme-HCSI-DBMIX

0. 说明

1. 代码调整

1.1. 拷贝之前的项目

1.2. G2P_CN_HCSI得到中文Phoneme文本

1.3. G2P_EN_Kb得到英文Phoneme文本

1.4. preprocess.py

1.4. preprocess.py

1.5. Feeder和Train和Tacotron改动

2. 训练

3. 合成

Fantasy Mix-Lingual Tacotron Version 4: Google-ZYX-Phoneme-HCSI-DBMIX相关推荐

最新文章

热门文章