Within its evaluation activities, ELDA has been contacted to evaluate the simultaneous speech-to-speech translation system of University of Karlsruhe (UKA). To do so, an end-to-end evaluation of UKA automatic simultaneous translation system, built with state-of-the-art components has been carried out recently. End-to-end evaluation shows if, and for which situations, such a system might be advantageous when compared to a human interpreter.

Using talks in English translated into Spanish, we’ve assessed both the recognition and translation components as well as for the overall system. The assessments are carried out by measuring the information preservation and the quality of the speech output. The information preservation is measured by using "comprehesion questionnaires" while the speech output is assessed by asking subjects to rate the quality of the output signal. In addition to subjective tests, automatic metrics such as BLEU or WER are also considered.

Even if the translation process remains the Achilles’ heel of the system, results showed that the system can keep at least half of the information, becoming potentially useful for final users.

More information are available in the following publications :

Olivier Hamon, Christian Fügen, Djamel Mostefa, Arranz Victoria, Muntsin Kolss and Khalid Choukri, "End-to-End Evaluation in Simultaneous Translation". In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), March 2009, Athens, Greece, pp. 345-353

Christian Fügen, Muntsin Kolss, Dietmar Bernreuther, Matthias Paulik, Sebastian Stüker, Stephan Vogel and Alex Waibel, "Open Domain Speech Recognition & Translation : Lectures and Speeches". In Proceedings of the ICASSP 2006, 31st IEEE International Conference on Acoustics, Speech, and Signal Processing, Toulouse, France, May 2006

Olivier Hamon, Djamel Mostefa, and Khalid Choukri, "End-to-end evaluation of a speech-to-speech translation system in TC-STAR". In Proceedings of the MT Summit XI, Copenhagen, Denmark, September 2007.

Contact: Khalid Choukri [javascript protected email address]