TC-Star Evaluation Information (WP4)

Participation

This section only summarises the more complete SLT Evaluation Plan document to be found here.

TC-STAR Evaluation Run #1 for SLT took place from March 21, 2005 to April 8, 2005. The complete schedule can be seen here, but we can outline the important dates for SLT:

• Mid-February: ELDA disseminates SLT development data set through this web site
• March 11, 2005: ASR team prepare word graphs for SLT team
• March 18, 2005: End of SLT development phase
• March 21, 2005: Beginning of SLT run - ELDA sends source files to participants
• March 25, 2005: End of SLT run - translations are sent back to ELDA
• March 31, 2005: End of automatic scoring phase by ELDA - initial results
• April 1, 2005: Beginning of adjudication phase
• April 8, 2005: End of adjudication phase - results are definitive

Before the proper evaluation run, participants have access to training data and development data composed of parallel texts and transcriptions. See below training data and development data.

SLT evaluation was runned in 3 translation directions: English to Spanish, Spanish to English and Chinese Mandarin to English.

English to Spanish and Spanish to English were runned on recording transcriptions from the European Parliament Plenary Sessions (EPPS), while Chinese to English was runned from recording transcriptions of Voice of America.

External Participants

1. Declare your interest to the project's coordinator.
2. ELDA send you an End-Used Agreement to have access to the data. You must fill in, sign and return this agreement to ELDA.
3. Get access to training and development data:
   • EPPS training and development transcriptions can be downloaded (see section training data and development data)
   • Chinese training data can be any of those listed as training in section Resources.
   • Chinese development transcriptions can be downloaded (see section development data)
4. ELDA will contact you as any other participant for the evaluation run.
5. You must release the data after the end of the evaluation run, as said in the agreement.

Back to Top

SLT Participants

Direction	Input	Participants
Zh-->En	Text	IBM, IRST, RWTH, UKA
	Single-best ASR	IBM, IRST, RWTH, UKA
	Verbatim	IBM, IRST, RWTH, UKA
Es-->En	Text	IBM, IRST, RWTH, UKA, UPC
	Single-best ASR	IBM, IRST, RWTH, UKA
	Verbatim	IBM, IRST, RWTH, UKA, UPC
En-->Es	Text	IBM, RWTH, UKA, UPC
	Single-best ASR	IBM, RWTH, UKA
	Verbatim	IBM, RWTH, UKA, UPC

External Participants

Direction	Input	Participants
Zh-->En	Text	ATR
	Single-best ASR	ATR
	Verbatim	ATR
Es-->En	Text	UPV
	Single-best ASR	UPV
	Verbatim
En-->Es	Text	UPV
	Single-best ASR	UPV
	Verbatim

John Hopkins has not given its commitment yet.

Back to Top

SLT Resources

Language	Reference	Reference	Amount	IPR-owner	IPR-distrib	IPR-granted use	Comment
Training
Zh->En	FBIS data	LDC2003E14		LDC	research		LDC membership 03 required
	Hong Kong Laws Parallel Text	LDC2000T47		LDC	research		LDC membership 00 required
	Hong Kong News Parallel Text, sentence-aligned	LDC2003E25		LDC	research		LDC membership 03 required
	Hong Kong News Parallel Text	LDC2000T46		LDC	research		LDC membership 00 required
	Hong Kong Hansard Parallel Text, aligned doc level	LDC2000T50		LDC	research		LDC membership 00 required
	Hong Kong Hansard Parallel Text, aligned sentence level	LDC2000E49		LDC	research		LDC membership 00 required
	English Translation of Chinese Treebank	LDC2000E17		LDC	research		LDC membership 00 required
	Xinhua Chinese-English Parallel News Text v1.0 beta 2	LDC2000E18		LDC	research		LDC membership 00 required
	UN Chinese-English Parallel Text Version 2	LDC2004E12		LDC	research		LDC membership 04 required
	Hong Kong Hansard Parallel Text	LDC2004E09		LDC	research		LDC membership 04 required
	Chinese English Translation Lexicon version 3.0	LDC2002L27		LDC	research		LDC membership 02 required
	Sinorama Chinese-English Parallel Text	LDC2002E58		LDC	research		LDC membership 02 required
	Multiple-Translation Chinese Corpus (without VoA Mandarin transcripts)	LDC2002T01		LDC	research		LDC membership 02 required
	Multiple-Translation Chinese Part 2	LDC2003T17		LDC	research		LDC membership 03 required
	Chinese-English Name Entity Lists version 1.0 beta	LDC2003E01		LDC	research		LDC membership 03 required
	Multiple-Translation Chinese Corpus Part 3	LDC2003E04		LDC	research		LDC membership 03 required
	Chinese Treebank Version 4.0	LDC2004T05		LDC	research		LDC membership 04 required
	Chinese Treebank English Parallel Corpus	LDC2003E07		LDC	research		LDC membership 03 required
	Chinese News Translation Corpus Part 1	LDC2003E08		LDC	research		LDC membership 03 required
Es->En	EPPS Spanish verbatim transcriptions 3 May - 14 Oct 2004		40h transcribed	UPC	ELRA	research	Transcribed by UPC
	EPPS Spanish final text edition 3 May - 14 Oct 2004			EC	ELRA	research	English and Spanish parallel texts are aligned. Verbatim transcriptions are also aligned with FTE by RWTH.
	EPPS Spanish final text edition April 1996 to June 2004			EC	RWTH	research	Provided to TCSTAR by RWTH
En->Es	EPPS English verbatim transcriptions 3 May - 14 Oct 2004		40h transcribed	RWTH	ELRA	research	Transcribed by RWTH
	EPPS English final text edition 3 May - 14 Oct 2004			EC	ELRA	research	English and Spanish parallel texts are aligned. Verbatim transcriptions are also aligned with FTE by RWTH.
	EPPS English final text edition April 1996 to June 2004			EC	RWTH	research	Provided to TCSTAR by RWTH
Development
Zh->En	Mandarin TDT3 1 Dec 98 to 11 Dec 98	LDC2001T58	~25kword subset of ASR dev	LDC	research	LDC membership 01 required	selected by ELDA
	Ref. translations of Mandarin TDT3 1 Dec 98 to 11 Dec 98		2 ref translations	?	research	?	translated by ELDA
Es->En	EPPS Spanish resegmented transcriptions: 25-28 Oct 2004		~25 kword	ELRA	research	free in TCSTAR
	Ref. translations of EPPS Spanish resegmented transcriptions: 25-28 Oct 2004		2 ref translations	ELRA	research	nominal fee (ELRA)
En->Es	EPPS English resegmented transcriptions: 25-28 Oct 2004		~25 kword	ELRA	research	free in TCSTAR
	Ref. translations of EPPS English resegmented transcriptions: 25-28 Oct 2004		2 ref translations	ELRA	research	nominal fee (ELRA)
Test
Zh->En	Mandarin TDT3 14 Dec 98 to 22 Dec 98	LDC2001T58	~25kword subset of ASR test	LDC	research	LDC membership 01 required	selected by ELDA
	Ref. translations of Mandarin TDT3 14 Dec 98 to 22 Dec 98		2 ref translations	?	research	?	translated by ELDA
Es->En	EPPS Spanish resegmented transcriptions: 15-18 Nov 2004		~25 kword	ELRA	research	free in TCSTAR
	Ref. translations of EPPS Spanish resegmented transcriptions: 15-18 Nov 2004		2 ref translations	ELRA	research	nominal fee (ELRA)
En->Es	EPPS English resegmented transcriptions: 15-18 Nov 2004		~25 kword	ELRA	research	free in TCSTAR
	Ref. translations of EPPS English resegmented transcriptions: 15-18 Nov 2004		2 ref translations	ELRA	research	nominal fee (ELRA)

Back to Top

SLT Download area

Training Data

English and Spanish

You can use any of the training resources listed in the table above in addition to the EPPS training sets. To get these last sets on DVD, please contact Christian Gollan at RWTH.

Chinese

You can use any of the training resources listed in the table above excepted TDT3 audio files and transcriptions for the month of December 1998 (development and test sets will be built from these files).

Back to Top

Development Data

Verbatim transcriptions of EPPS are common with ASR evaluation. The difference is that only 2 files are used for SLT (instead of 3 for ASR) as only 25,000 words are needed.

Lastest release of the transcriptions is 27jan2005. All files have been converted to UTF-8.

About Final Text Edition transcriptions published by the EC: we provide the original HTML file (covering all the day), plus an HTML file cut from the original one and covering only the portion we are interested in, and the raw text version of this last file (saved in UTF-8). Finally, we provide another UTF-8 text file generated from the HTML excerpt following the translation guidelines of NIST, which they provide translation agencies with together with the source documents (please note that this is a format for human translators, and it is more relaxed than the NIST MT SGML format, which we will use in fine).

Data Set	Files
EPPS English Verbatim Transcriptions	Complete archive (tgz) - same as ASR dev data - use only: 20041026_1505_1700_EN_SAT 20041027_1210_1310_EN_SAT
EPPS English Final Text Edition	Complete archive (tgz) contains: original HTML file from EC excerpt in HTML for the pertinent time range raw text (UTF-8) excerpt for the time range
EPPS Spanish Verbatim Transcriptions	Complete archive (tgz) - same as ASR dev data - use only: 20041026_1505_1700_ES_SAT 20041027_1210_1310_ES_SAT
EPPS Spanish Final Text Edition	Complete archive (tgz) contains: original HTML file from EC excerpt in HTML for the pertinent time range raw text (UTF-8) excerpt for the time range
EPPS English->Spanish Reference Translations	English source set (NIST MT format): SRCSET containing 4 documents: 20041026_1505_1700_EN_SAT 20041027_1210_1310_EN_SAT 20041026_1505_1700_EN_FTE 20041027_1210_1310_EN_FTE Spanish reference translations (NIST MT format): REFSET containing 2 reference translations (different sysid) for each document (i.e. 8 documents): 20041026_1505_1700_EN_SAT / tcstar-run1-epps-dev-enes-ref1 20041026_1505_1700_EN_SAT / tcstar-run1-epps-dev-enes-ref2 20041027_1210_1310_EN_SAT / tcstar-run1-epps-dev-enes-ref1 20041027_1210_1310_EN_SAT / tcstar-run1-epps-dev-enes-ref2 20041026_1505_1700_EN_FTE / tcstar-run1-epps-dev-enes-ref1 20041026_1505_1700_EN_FTE / tcstar-run1-epps-dev-enes-ref2 20041027_1210_1310_EN_FTE / tcstar-run1-epps-dev-enes-ref1 20041027_1210_1310_EN_FTE / tcstar-run1-epps-dev-enes-ref2
EPPS Spanish->English Reference Translations	Spanish source set (NIST MT format): SRCSET containing 4 documents: 20041026_1505_1700_ES_SAT 20041027_1210_1310_ES_SAT 20041026_1505_1700_ES_FTE 20041027_1210_1310_ES_FTE English reference translations (NIST MT format): REFSET containing 2 reference translations (different sysid) for each document (i.e. 8 documents): 20041026_1505_1700_ES_SAT / tcstar-run1-epps-dev-enes-ref1 20041026_1505_1700_ES_SAT / tcstar-run1-epps-dev-enes-ref2 20041027_1210_1310_ES_SAT / tcstar-run1-epps-dev-enes-ref1 20041027_1210_1310_ES_SAT / tcstar-run1-epps-dev-enes-ref2 20041026_1505_1700_ES_FTE / tcstar-run1-epps-dev-enes-ref1 20041026_1505_1700_ES_FTE / tcstar-run1-epps-dev-enes-ref2 20041027_1210_1310_ES_FTE / tcstar-run1-epps-dev-enes-ref1 20041027_1210_1310_ES_FTE / tcstar-run1-epps-dev-enes-ref2
EPPS English-Spanish DevData	Complete archive (tgz)
VOA Chinese Transcriptions	Chinese source set (NIST MT format, GB2312 encoded): SRCSET containing 3 documents: 19981201_0700_0730_VOA_MAN 19981202_0900_0930_VOA_MAN 19981203_0700_0730_VOA_MAN English reference translations (NIST MT format, UTF-8 encoded): REFSET containing 2 reference translations (different sysid) for each document (i.e. 6 documents): 19981201_0700_0730_VOA_MAN / tcstar-run1-voa-dev-ref1 19981201_0700_0730_VOA_MAN / tcstar-run1-voa-dev-ref2 19981202_0900_0930_VOA_MAN / tcstar-run1-voa-dev-ref1 19981202_0900_0930_VOA_MAN / tcstar-run1-voa-dev-ref2 19981203_0700_0730_VOA_MAN / tcstar-run1-voa-dev-ref1 19981203_0700_0730_VOA_MAN / tcstar-run1-voa-dev-ref2
VOA Chinese Text Version	Chinese source set (NIST MT format, GB2312 encoded): SRCSET containing 3 documents: 19981201_0700_0730_VOA_MAN_text 19981202_0900_0930_VOA_MAN_text 19981203_0700_0730_VOA_MAN_text English reference translations (NIST MT format, UTF-8 encoded): REFSET containing 2 reference translations (different sysid) for each document (i.e. 6 documents): 19981201_0700_0730_VOA_MAN_text / tcstar-run1-voa-dev-ref1 19981201_0700_0730_VOA_MAN_text / tcstar-run1-voa-dev-ref2 19981202_0900_0930_VOA_MAN_text / tcstar-run1-voa-dev-ref1 19981202_0900_0930_VOA_MAN_text / tcstar-run1-voa-dev-ref2 19981203_0700_0730_VOA_MAN_text / tcstar-run1-voa-dev-ref1 19981203_0700_0730_VOA_MAN_text / tcstar-run1-voa-dev-ref2

The translation guidelines for the translation agencies are available here (MS Word document).

Word graphs/lattices are regularly posted on WP2's own web page.

Back to Top

Scoring Tools

The scoring tools proposed by ELDA are Perl scripts. You can download them in this zip containing:

• BLEU/NIST v11a
• mWER: multiple reference word error rate
• mPER: multiple reference position-independent word error rate
• mCER: multiple reference character error rate
• WNM: Weighted N-gram Model

Back to Top

Test Data

Source files for SLT evaluation are to be downloaded here.

Submission guidelines:

Submitted files should use the NIST MT format for TSTSET:

<TSTSET setid="..." srclang="..." trglang="...">

<DOC docid="..." sysid="...">

<SEG id="1"> TRANSLATED TEXT </SEG>

...

</DOC>

...

</TSTSET>

Submit one test set per file, i.e. one file for English verbatim transcripts, one file for English ASR output, one file for English FTE, etc.

The sysid attribute must identify the organisation, the condition, and the system. For instance, if the organisation ORG submits one primary condition and two secondary conditions for the English verbatim transcripts (one with the same system than for primary and the other with another system or system version), then it will send 3 files for this setid, with the following sysid:

• ORG-PRIMARY-system1
• ORG-SECONDARY-system1
• ORG-SECONDARY-system2

Source files

Data Set	Files
EPPS English Verbatim Transcriptions	English source set (NIST MT format, UTF-8 encoded, without punctuation): SRCSET containing 3 documents 20041116_1505_1800_EN_SAT_verbatim 20041117_0905_1240_EN_SAT_verbatim 20041118_1000_1225_EN_SAT_verbatim
EPPS English ASR Output (single-best)	English source set (NIST MT format, UTF-8 encoded, without punctuation, case-insensitive): SRCSET containing 3 documents 20041116_1505_1800_EN_SAT_asr 20041117_0905_1240_EN_SAT_asr 20041118_1000_1225_EN_SAT_asr For systems needing case-sensitivity, here is an alternative English source set produced by LIMSI (NIST MT format, UTF-8 encoded, without punctuation, case-sensitive): SRCSET containing 3 documents 20041116_1505_1800_EN_SAT_limsi 20041117_0905_1240_EN_SAT_limsi 20041118_1000_1225_EN_SAT_limsi
EPPS English Final Text Edition	English source set (NIST MT format, UTF-8 encoded): SRCSET containing 4 documents 20041115_1505_1735_EN_FTE 20041116_1505_1800_EN_FTE 20041117_0905_1240_EN_FTE 20041117_1500_1835_EN_FTE
EPPS Spanish Verbatim Transcriptions	Spanish source set (NIST MT format, UTF-8 encoded, without punctuation): SRCSET containing 5 documents 20041115_1705_1735_ES_SAT_verbatim 20041116_1505_1800_ES_SAT_verbatim 20041117_1500_1835_ES_SAT_verbatim 20041118_1000_1225_ES_SAT_verbatim 20041118_1500_1600_ES_SAT_verbatim
EPPS Spanish ASR Output (single-best)	Spanish source set (NIST MT format, UTF-8 encoded, without punctuation, case-insensitive): SRCSET containing 5 documents 20041115_1705_1735_ES_SAT_asr 20041116_1505_1800_ES_SAT_asr 20041117_1500_1835_ES_SAT_asr 20041118_1000_1225_ES_SAT_asr 20041118_1500_1600_ES_SAT_asr Warning: some segments may be empty for alignment reasons with verbatim transcripts. For systems needing case-sensitivy, here is an alternative Spanish source set produced by LIMSI (NIST MT format, UTF-8 encoded, without punctuation, case-sensitive): SRCSET containing 5 documents 20041115_1705_1735_ES_SAT_limsi 20041116_1505_1800_ES_SAT_limsi 20041117_1500_1835_ES_SAT_limsi 20041118_1000_1225_ES_SAT_limsi 20041118_1500_1600_ES_SAT_limsi
EPPS Spanish Final Text Edition	Spanish source set (NIST MT format, UTF-8 encoded): SRCSET containing 4 documents 20041115_1505_1735_ES_FTE 20041116_1505_1800_ES_FTE 20041117_0905_1240_ES_FTE 20041117_1500_1835_ES_FTE
VOA Chinese ASR Output (single best)	Chinese source set (NIST MT format, GB2312 encoded, without punctuation): SRCSET containing 3 documents: 19981214_0700_0730_VOA_MAN_asr 19981215_0900_0930_VOA_MAN_asr 19981216_0700_0730_VOA_MAN_asr Warning: some segments may be empty for alignment reasons with verbatim transcripts.
VOA Chinese Verbatim Transcriptions	Chinese source set (NIST MT format, GB2312 encoded, without punctuation): SRCSET containing 3 documents: 19981214_0700_0730_VOA_MAN_verbatim 19981215_0900_0930_VOA_MAN_verbatim 19981216_0700_0730_VOA_MAN_verbatim
VOA Chinese Text Version	Chinese source set (NIST MT format, GB2312 encoded): SRCSET containing 3 documents: 19981214_0700_0730_VOA_MAN_text 19981215_0900_0930_VOA_MAN_text 19981216_0700_0730_VOA_MAN_text

Reference Translations

Data Set	Files
EPPS English Verbatim Transcriptions	English-to-Spanish verbatim reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
EPPS English ASR Output (ROVERed)	English-to-Spanish ASR reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
EPPS English ASR Output (LIMSI)	English-to-Spanish ASR-limsi reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
EPPS English Final Text Edition	English-to-Spanish text reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
EPPS Spanish Verbatim Transcriptions	Spanish-to-English verbatim reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
EPPS Spanish ASR Output (ROVERed)	Spanish-to-English ASR reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
EPPS Spanish ASR Output (LIMSI)	Spanish-to-English ASR-limsi reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
EPPS Spanish Final Text Edition	Spanish-to-English text reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
VOA Chinese ASR Output (single best)	Chinese-to-English ASR reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
VOA Chinese Verbatim Transcriptions	Chinese-to-English verbatim reference translation set (NIST MT format, UTF-8 encoded, without punctuation): REFSET containing 2 reference translations for each of the 3 documents
VOA Chinese Text Version	Chinese-to-English text reference translation set (NIST MT format, UTF-8 encoded, with punctuation): REFSET containing 2 reference translations for each of the 3 documents

 

Output on test data

	En->Es	Es->En	Zh->En
ASR	IBM_PRIMARY IBM_SECONDARY_1 IBM_SECONDARY_2 RWTH_PRIMARY_1 RWTH_PRIMARY_2 RWTH_SECONDARY_1 RWTH_SECONDARY_2 UKA_PRIMARY UPV_PRIMARY	IBM_PRIMARY_1 IBM_PRIMARY_2 IBM_SECONDARY_1 IBM_SECONDARY_2 IRST_PRIMARY_1 IRST_PRIMARY_2 IRST_PRIMARY_3 RWTH_PRIMARY_1 RWTH_PRIMARY_2 RWTH_SECONDARY_1 RWTH_SECONDARY_2 UKA_PRIMARY UKA_SECONDARY UPV_SECONDARY	ATR_PRIMARY ATR_SECONDARY IBM_PRIMARY_1 IBM_PRIMARY_2 IBM_PRIMARY_3 IBM_PRIMARY_4 IRST_PRIMARY_1 IRST_PRIMARY_2 RWTH_PRIMARY_1 RWTH_PRIMARY_2 RWTH_PRIMARY_3 UKA_PRIMARY
text	IBM_PRIMARY_1 IBM_PRIMARY_2 IBM_SECONDARY_1 IBM_SECONDARY_2 RWTH_PRIMARY_1 RWTH_PRIMARY_2 RWTH_PRIMARY_3 RWTH_PRIMARY_4 UKA_PRIMARY UPC_PRIMARY UPV_PRIMARY	IBM_PRIMARY_1 IBM_PRIMARY_2 IBM_PRIMARY_3 IBM_SECONDARY_1 IBM_SECONDARY_2 IBM_SECONDARY3 IRST_PRIMARY_1 IRST_PRIMARY2 RWTH_PRIMARY_1 RWTH_PRIMARY_2 UKA_PRIMARY UPC_PRIMARY UPV_PRIMARY	ATR_PRIMARY ATR_SECONDARY IBM_PRIMARY_1 IBM_PRIMARY_2 IBM_PRIMARY_3 IBM_PRIMARY_4 IBM_PRIMARY_5 IRST_PRIMARY_1 IRST_PRIMARY_2 RWTH_PRIMARY_1 RWTH_PRIMARY_2 RWTH_PRIMARY_3 RWTH_PRIMARY_4 RWTH_PRIMARY_5 UKA_PRIMARY
Verbatim	IBM_PRIMARY IBM_SECONDARY_1 IBM_SECONDARY_2 RWTH_PRIMARY_1 RWTH_PRIMARY_2 UKA_PRIMARY UPC_PRIMARY	IBM_PRIMARY_1 IBM_PRIMARY_2 IBM_SECONDARY_1 IBM_SECONDARY_2 IRST_PRIMARY_1 IRST_PRIMARY_2 RWTH_PRIMARY_1 RWTH_PRIMARY_2 UKA_PRIMARY UPC_PRIMARY	ATR_PRIMARY ATR_SECONDARY IBM_PRIMARY_1 IBM_PRIMARY_2 IBM_PRIMARY_3 IBM_PRIMARY_4 IRST_PRIMARY_1 IRST_PRIMARY_2 RWTH_PRIMARY_1 RWTH_PRIMARY_2 RWTH_PRIMARY_3 UKA_PRIMARY

You can download all the output data in this zip file.

Back to Top