skip to: page content | links on this page | site navigation | footer (site information)

TC-Star Evaluation Information (WP4)

subglobal1 link | subglobal1 link | subglobal1 link | subglobal1 link | subglobal1 link | subglobal1 link | subglobal1 link
subglobal2 link | subglobal2 link | subglobal2 link | subglobal2 link | subglobal2 link | subglobal2 link | subglobal2 link
subglobal3 link | subglobal3 link | subglobal3 link | subglobal3 link | subglobal3 link | subglobal3 link | subglobal3 link
subglobal4 link | subglobal4 link | subglobal4 link | subglobal4 link | subglobal4 link | subglobal4 link | subglobal4 link
subglobal5 link | subglobal5 link | subglobal5 link | subglobal5 link | subglobal5 link | subglobal5 link | subglobal5 link
subglobal6 link | subglobal6 link | subglobal6 link | subglobal6 link | subglobal6 link | subglobal6 link | subglobal6 link
subglobal7 link | subglobal7 link | subglobal7 link | subglobal7 link | subglobal7 link | subglobal7 link | subglobal7 link
subglobal8 link | subglobal8 link | subglobal8 link | subglobal8 link | subglobal8 link | subglobal8 link | subglobal8 link

SLT Evaluation - Run #2

EN-ES Results Human EN-ES Results ES-EN Results ZH-EN Results Go back to SLT page

 

SLT Results for the English to Spanish direction

(in red the best primary system for each metric)

FTE Task

SYSTEM - Scoring
NIST cs*
BLEU cs*
NIST
BLEU
IBM
mWER cs*
mWER
mPER cs*
mPER
WNM / RECALL
WNM / F-measure
DFKI-PRIMARY-system1
8.6983
0.3632
9.029
0.3883
0.3633
48.0629
46.2763
36.3609
33.9754
0.422
0.432
IBM_PRIMARY_system1
9.8859
0.4754
9.938
0.4828
0.4756
41.2496
40.6272
31.4688
30.4763
0.4743
0.4891
IBM_PRIMARY_system2
9.911
0.4765
9.9626
0.484
0.4767
41.0814
40.469
31.415
30.4258
0.4734
0.489
IBM_PRIMARY_system3
9.9259
0.4771
9.9774
0.4845
0.4773
41.0006
40.395
31.3477
30.372
0.4733
0.4892
IRST-PRIMARY
10.2295
0.4981
10.3216
0.5078
0.49
39.3116
38.5108
30.2105
29.1001
0.4719
0.4931
IRST-SECONDARY-1
10.2261
0.4979
10.3099
0.5075
0.4912
39.1703
38.4368
30.0961
29.0362
0.4729
0.4937
ROVER
10.3786
0.5074
10.5313
0.5221
0.4996
38.1475
37.1583
29.2583
28.0033
0.4799
0.5013
RWTH_PRIMARY_1
10.1615
0.4944
10.2131
0.5013
0.4945
39.8096
39.1905
30.483
29.5611
0.4798
0.4951
RWTH_PRIMARY_2
10.13
0.4913
10.1831
0.4986
0.4858
39.7625
39.0996
30.732
29.7966
0.4718
0.4909
UED-PRIMARY
10.1132
0.495
10.2003
0.5051
0.4942
39.6918
38.9415
30.5065
29.477
0.478
0.4928
UKA-PRIMARY_1
9.5609
0.4404
9.7324
0.4543
0.4295
43.6082
42.296
33.6625
32.0744
0.4436
0.4642
UPC-PRIMARY-1
9.9995
0.482
10.0499
0.4893
0.4769
40.893
40.2907
31.489
30.5335
0.4602
0.4817
UPC-PRIMARY-2
9.8342
0.4684
9.8892
0.4758
0.4633
41.4683
40.8021
31.9129
30.9574
0.4538
0.4747
UPC-SECONDARY-1
10.0613
0.4885
10.1092
0.4956
0.4832
40.2133
39.6481
31.4621
30.5402
0.4628
0.4846
UW-PRIMARY-system1
10.0068
0.485
10.0964
0.4949
0.4805
40.3714
39.5942
30.9507
29.8942
0.4681
0.4888
UW-PRIMARY-system2
10.0277
0.4875
10.1147
0.4972
0.485
40.2873
39.5236
30.9742
29.9379
0.4709
0.4905
SYSTRAN**
8.566
0.3629
8.8964
0.3906
0.3631
47.7903
45.9465
37.3635
34.9612
0.4174
0.4248

*cs: case-sensitive
**translation with a commercial cd

SYSTEM - Ranking
NIST cs*
BLEU cs*
NIST
BLEU
IBM
mWER cs*
mWER
mPER cs*
mPER
WNM / RECALL
WNM / F-measure
DFKI-PRIMARY-system1
16
16
16
17
16
17
17
16
16
16
16
IBM_PRIMARY_system1
13
13
13
13
13
13
13
12
11
4
9
IBM_PRIMARY_system2
12
12
12
12
12
12
12
10
10
5
10
IBM_PRIMARY_system3
11
11
11
11
10
11
11
9
9
6
8
IRST-PRIMARY
2
2
2
2
5
3
3
3
3
8
4
IRST-SECONDARY-1
3
3
3
3
4
2
2
2
2
7
3
ROVER
1
1
1
1
1
1
1
1
1
1
1
RWTH_PRIMARY_1
4
5
4
5
2
6
6
4
5
2
2
RWTH_PRIMARY_2
5
6
6
6
6
5
5
6
6
9
6
UED-PRIMARY
6
4
5
4
3
4
4
5
4
3
5
UKA-PRIMARY_1
15
15
15
15
15
15
15
15
15
15
15
UPC-PRIMARY-1
10
10
10
10
11
10
10
13
12
13
13
UPC-PRIMARY-2
14
14
14
14
14
14
14
14
14
14
14
UPC-SECONDARY-1
7
7
8
8
8
7
9
11
13
12
12
UW-PRIMARY-system1
9
9
9
9
9
9
8
7
7
11
11
UW-PRIMARY-system2
8
8
7
7
7
8
7
8
8
10
7
SYSTRAN**
17
17
17
16
17
16
16
17
17
17
17

*cs: case-sensitive
**translation with a commercial cd

VERBATIM Task

SYSTEM - Scoring
NIST cs*
BLEU cs*
NIST
BLEU
IBM
mWER cs*
mWER
mPER cs*
mPER
WNM / RECALL
WNM / F-measure
IBM_PRIMARY_system1
9.6143
0.4512
9.713
0.4595
0.4512
43.5583
42.8007
32.601
31.7339
0.4611
0.4706
IBM_PRIMARY_system2
9.6571
0.4535
9.7557
0.4617
0.4535
43.2875
42.5299
32.4237
31.5501
0.4596
0.4705
IBM_PRIMARY_system3
9.6492
0.4525
9.7483
0.4608
0.4525
43.3165
42.559
32.456
31.5791
0.4596
0.4705
IRST-PRIMARY **
9.9064
0.4661
9.9931
0.4744
0.4633
42.185
41.5209
31.5082
30.7248
0.4595
0.4736
IRST-SECONDARY-1
9.5493
0.4485
9.9724
0.4749
0.4451
44.4512
41.45
33.8518
30.8119
0.458
0.4729
ROVER
10.0592
0.4753
10.1583
0.4843
0.4699
40.9245
40.1992
30.3864
29.5966
0.4622
0.4781
RWTH-PRIMARY_1
9.7103
0.4542
9.7738
0.4613
0.4542
43.1166
42.5428
32.0852
31.4469
0.4591
0.4712
RWTH-PRIMARY_2
9.7069
0.452
9.7715
0.4592
0.4513
43.2585
42.6621
32.2496
31.5985
0.4555
0.4683
RWTH-PRIMARY_3
9.6567
0.4463
9.7254
0.4539
0.4359
43.2456
42.6524
32.6913
32.0272
0.4456
0.4648
UKA-PRIMARY_1
9.0794
0.401
9.4203
0.4216
0.3959
47.6266
45.3668
36.1342
33.7454
0.4372
0.4497
UPC-PRIMARY
9.5009
0.4406
9.7384
0.4579
0.4347
44.6608
43.0006
33.681
31.9241
0.4458
0.4638
UW-PRIMARY-system1
9.2445
0.4257
9.6474
0.4524
0.4252
46.1501
43.3584
34.835
32.0111
0.4494
0.4618
UW-PRIMARY-system2
9.3633
0.4361
9.7681
0.4628
0.4356
45.5118
42.6943
34.4998
31.6533
0.4554
0.468
SYSTRAN***
8.0986
0.3297
8.4302
0.3551
0.3297
51.8593
50.054
39.7447
37.7234
0.4038
0.4022

*cs: case-sensitive
**submitted the 2006/03/29
***translation with a commercial cd

SYSTEM - Ranking
NIST cs*
BLEU cs*
NIST
BLEU
IBM
mWER cs*
mWER
mPER cs*
mPER
WNM / RECALL
WNM / F-measure
IBM_PRIMARY_system1
8
7
11
8
7
8
10
7
9
2
5
IBM_PRIMARY_system2
5
4
7
5
4
6
4
5
5
3
6
IBM_PRIMARY_system3
7
5
8
7
5
7
6
6
6
3
6
IRST-PRIMARY **
2
2
2
3
2
2
3
2
2
5
2
IRST-SECONDARY-1
9
8
3
2
8
9
2
10
3
7
3
ROVER
1
1
1
1
1
1
1
1
1
1
1
RWTH-PRIMARY_1
3
3
4
6
3
3
5
3
4
6
4
RWTH-PRIMARY_2
4
6
5
9
6
5
8
4
7
8
8
RWTH-PRIMARY_3
6
9
10
11
9
4
7
8
12
12
10
UKA-PRIMARY_1
13
13
13
13
13
13
13
13
13
13
13
UPC-PRIMARY
10
10
9
10
11
10
11
9
10
11
11
UW-PRIMARY-system1
12
12
12
12
12
12
12
12
11
10
12
UW-PRIMARY-system2
11
11
6
4
10
11
9
11
8
9
9
SYSTRAN***
14
14
14
14
14
14
14
14
14
14
14

*cs: case-sensitive
**submitted the 2006/03/29
***translation with a commercial cd

ASR Task

SYSTEM - Scoring
AS-WER
NIST cs*
BLEU cs*
NIST
BLEU
IBM
mWER cs*
mWER
mPER cs*
mPER
WNM / RECALL
WNM / F-measure
IBM_PRIMARY_1
51.0619
8.6244
0.3577
8.7974
0.3696
0.3567
52.0265
50.9413
38.7900
37.5661
0.4381
0.4503
IBM_PRIMARY_2
51.3545
8.6050
0.3562
8.7791
0.3681
0.3528
52.3098
51.2367
39.1216
37.8977
0.4373
0.4497
IBM_PRIMARY_3
51.3713
8.5967
0.3558
8.7702
0.3676
0.3522
52.3219
51.2518
39.1487
37.9339
0.4370
0.4494
IBM_PRIMARY_4
51.2740
8.5973
0.3564
8.7697
0.3682
0.3564
52.2465
51.1643
38.9437
37.7169
0.4388
0.4501
IRST-PRIMARY**
50.0198
8.7513
0.3597
8.8684
0.3683
0.3509
50.9533
50.1424
39.3085
38.3830
0.4356
0.4518
IRST-SECONDARY-1
50.1445
8.4789
0.3454
8.8247
0.3651
0.3369
52.6022
50.2962
41.0539
38.5730
0.4333
0.4500
RWTH-PRIMARY_1
50.0487
8.7151
0.3591
8.8661
0.3702
0.3502
50.5192
49.6300
38.6574
37.6053
0.4308
0.4480
RWTH-PRIMARY_2
50.3547
8.6542
0.3531
8.8034
0.3642
0.3447
50.7634
49.8922
39.0131
37.9640
0.4276
0.445
UKA-PRIMARY_1
56.3816
8.1047
0.3132
8.3857
0.3292
0.3058
55.4840
53.6874
43.1459
41.2347
0.4106
0.4258
UPC-PRIMARY
50.8726
8.5573
0.3476
8.7954
0.3610
0.3402
51.7883
50.3082
40.0139
38.3680
0.4248
0.4470
SYSTRAN***
61.8030
7.0302
0.2393
7.3558
0.2597
0.2386
62.1460
60.2981
47.8424
45.7564
0.3823
0.3794

*cs: case-sensitive
**submitted the 2006/03/29
***translation with a commercial cd

SYSTEM - Ranking
AS-WER
NIST cs*
BLEU cs*
NIST
BLEU
IBM
mWER cs*
mWER
mPER cs*
mPER
WNM / RECALL
WNM / F-measure
IBM_PRIMARY_1
3
4
3
5
2
1
5
6
2
1
2
2
IBM_PRIMARY_2
7
5
5
7
5
3
7
8
5
4
3
5
IBM_PRIMARY_3
9
7
6
8
6
4
8
9
6
5
4
6
IBM_PRIMARY_4
6
6
4
9
4
2
6
7
3
3
1
3
IRST-PRIMARY**
1
1
1
1
3
5
3
3
7
8
5
1
IRST-SECONDARY-1
4
9
9
3
7
9
9
4
9
9
6
4
RWTH-PRIMARY_1
2
2
2
2
1
6
1
1
1
2
7
7
RWTH-PRIMARY_2
8
3
7
4
8
7
2
2
4
6
8
9
UKA-PRIMARY_1
10
10
10
10
10
10
10
10
10
10
10
10
UPC-PRIMARY
5
8
8
6
9
8
4
5
8
7
9
8
SYSTRAN***
11
11
11
11
11
11
11
11
11
11
11
11

*cs: case-sensitive
**submitted the 2006/03/29
***translation with a commercial cd

Back to Top