TC-STAR Evaluation Run #2 for TTS will take place from Feb 24, 2006 to Mar 31, 2006. The complete schedule can be seen here.
Before the proper evaluation run, participants have access to development data composed of plain text, SSML and XML files, depending on the evaluation tasks. See below development data.
TTS evaluation will be run in 3 languages: English, Spanish and Chinese Mandarin.
Documentation: TC-STAR WP3/ECESS meeting, Berlin January 2006: "TTS Evaluation 2 - Status and Evaluation criteria" : presentation
TTS evaluation data can be downloaded here.
Back to Top
| Data
|
Language
|
Files
|
DTD
Input of text processing module (M1 module)
Input of prosody module (M2 module)
Input acoustic synthesis module (M3 module)
The specifications for the production of the input for Modules M2 and M3. |
UK English |
data/TTS/Run2/English.zip |
DTD
Input of prosody module (M2 module)
Input of acoustic module (M3 module)
Input of TTS component
The specifications for the production of the input for Modules M2 and phoneme segmentation. |
Spanish |
data/TTS/Run2/Spanish.zip |
DTD
Input of text processing module eval 1(M1 module)
Input of prosody module (M2 module)
Input of acoustic module (M3 module)
The specifications for the production of the input for Modules M2 and M3. |
Chinese Mandarin |
data/TTS/Run2/Chinese.zip |
| Data
|
Language + input/output format
|
| DTD |
UK, ES, CN |
| Evaluation of text processing module (M1) |
|
| Input and output of M1 |
UK |
| Input and output of M1 |
CN |
| Evaluation of prosody |
|
| Input of prosody module |
ES, EN (Siemens M1 output) |
| Output of prosody module |
ES: Output of UPC M2 output |
| Evaluation of acoustic module |
|
| Input of acoustic module |
ES: Output of UPC M2 output |
| Output of acoustic module |
.wav file |
| Evaluation of TTS system |
|
| Input of TTS system |
same as input of M1 module |
| Output of TTS system |
.wav file |
| Evaluation of VC |
|
| Input |
.wav file |
| Output |
.wav file |
| Evaluation of ES |
|
| Labelled speech (WP1 orthographic conventions, starting and ending times for each word) |
|
| End-to-end evaluation |
|
| Output of WP1 system |
|
Back to Top
You can find the specifications for the format of the input/output here. Please read carefully this document before running your system !
Modules |
Languages |
| M1 - Text processing evaluation |
|
| Text Normalisation (m1.1), End-Of-Sentence detection (m1.2) , POS tagging (m1.3), Grapheme-to-Phoneme conversion (m1.4) |
M1-EN |
Word Segmentation and POS tagging (m1.2-m1.3)
Grapheme-to-Phoneme conversion (m1.4) |
M1.2-M1.3-CN
M1.4-CN |
| M2 - Evaluation of Prosody Generation module |
M2-EN
M2-ES
M2-CN |
| M3 - Evaluation of Acoustic synthesis module |
|
| M3.1 - Acoustic Synthesis - SUS sentences |
M3.1-EN
M3.1-CN |
| M3.2 - Acoustic Synthesis - EPPS-EN data, 863-CN data |
M3.2-EN
M3.2-CN |
| S - Evaluation of TTS component |
|
| S1- System - EPPS-EN data, EPPS-ES data, 863-CN data |
S1-EN
S1-ES
S-CN |
| S2 - System - ASR + SLT outputs |
S2-EN
S2-ES |
| Evaluation of Voice conversion |
|
| IVC - Intra-lingual VC |
IVC-EN (updated with new UKL)
IVC-ES
IVC-CN |
| CVC - Crosslingual VC |
ES/UK |
| ES - Evaluation of Expresiv. STL |
ExSp-ES |
| E2E - Evaluation of End-to-end (ASR+SLT+TTS) system |
UK/ES |
Back to Top
!!!!! Information about the results can be found in the README (updated on 2006/04/06 -> VC1 and M31 sections)
!!!!! Information about the evaluation tasks, Language Resources, protocols, metrics and results can be found in the D16 - Evaluation report - TTS part - is available here. (updated on 2006/04/25 with results of 2006/04/06 for EN_VC1, EN_VC2, ES_VC1 and results of Z_S1, Z_S2)
Modules |
Results |
Raw Results |
| M1-Text processing evaluation |
M1-CN
EN_M1 |
Chinese (31/03/2006)
English (25/04/2006) Updates: Z_S1, Z_S2, M31: sclite files
Spanish (25/04/2006) Updates: Z_S1, Z_S2, M31: sclite files
Audio file list
Text file list |
M21-Evaluation of prosody using segmental information
M22-Judgment test using delexicalised utterances
M23-Judgment test using delexicalised utterances
M32-Judgment test
S1-TTS component (Z_S1 : test with AT&T system)
S2-TTS component (Z_S2 : test with AT&T system)
VC1-Comparison of speaker identities (Intralingual, crosslingual)
VC2- Evaluation of overall speech quality (Intralingual, crosslingual)
ES1-Judgment test
ES2-Functional test |
Chinese (31/03/2006)
English (25/04/2006) Updates: EN_VC1, EN_VC2
Spanish (25/04/2006) Updates: ES_VC1
|
M31 - Functional test (SUS tests)
|
CN sclite results (31/03/2006)
EN sclite results (31/03/2006) |
| S2 - Functional test (SUS tests) |
Test without AT&T system (S2)
EN sclite results (25/04/2006)
ES sclite results (25/04/2006)
Test with AT&T system (Z_S2)
ES sclite results (25/04/2006)
EN sclite results (25/04/2006) |
Back to Top
