This site is an archival page for the project at the University of Pennsylvania. Current information about this project can be found on the website of Matt Huenerfauth at The City University of New York. http://eniac.cs.qc.cuny.edu/matt
The goal of this project is to develop new technologies that enable the machine translation of English text into animations of American Sign Language. This research will make more information and services available to the majority of Deaf Americans who face English literacy challenges. Because signed languages, like ASL, contain phenomena not seen in traditional written/spoken languages, they are particularly challenging to process using standard MT approaches. Exploring the computational linguistics of ASL can help us understand the limitations of current MT technologies and motivate the development of new ones.
This website contains background information about this topic, links to publications and presentations related to this project, a listing of the participating researchers, and important contact information.
Introduction and Motivation
People on the Project
Publications (up to 2007)
Funding
Although Deaf students in the U.S. and Canada are taught written English, their inability to hear spoken English results in most Deaf U.S. high school graduates (18 year olds) reading at a fourth-grade (10 year old) level (Holt, 1991). Unfortunately, many Deaf accessibility aids, like television closed captioning or teletype telephones, assume the user has strong English literacy skills. Many Deaf people with English reading difficulty are fluent in American Sign Language (ASL); so, an English-to-ASL automated machine translation (MT) system could make information and services accessible when English text captioning is too complex or an interpreter is unavailable.
English-to-ASL MT software could translate English text into an animation of a 3D virtual reality character performing ASL. In this way, a variety of English text sources could be made more accessible to the Deaf, including: closed captioning, teletype telephones, computer interfaces, Internet information and services, etc. An MT system also has educational applications; software for Deaf students learning English or for hearing students learning ASL could use this technology to translate novel English input sentences into an ASL animation.
American Sign Language has a different word order and linguistic structure than English; so, English-to-ASL translation is as complex as translation between pairs of spoken languages. In fact, ASL’s visual modality allows it to use linguistic phenomena not seen in any spoken language (Neidle et al., 2000; Liddell, 2003). The space around the signer can be used for many communicative purposes, including the production of constructions called "classifier predicates." These complex hand movements trace contours, identify locations, or draw paths through the space in front of the signer that topologically correspond to the shape, location, or movement of real world entities in a three-dimensional scene or event being discussed. When the ASL equivalent of an English sentence uses a classifier predicate, then the structure of two sentences is quite divergent. For example, the entire sentence "the car drove up the hill" may be expressed in ASL with a single hand movement tracing a hill-like 3D path through the air in front of the signer. Classifier predicates challenge traditional linguistic representations by their use of spatial metaphor and visualization (Liddell 2003) and thus cannot be generated by most MT software.
Matt Huenerfauth
Liming Zhao
Erdan Gu
Jan Allbeck
Mitch Marcus
Martha Palmer
Norman Badler
Recent publications for this project can be found on the website of Matt Huenerfauth at The City University of New York. http://eniac.cs.qc.cuny.edu/matt
Matt Huenerfauth, Liming Zhou, Erdan Gu and Jan Allbeck.
2007. Evaluating American Sign Language Generation Through the Participation
of Native ASL Signers. Ninth International ACM SIGACCESS Conference on
Computers and Accessibility, ASSETS-2007. Tempe, Arizona, USA. October 2007.
Best Paper Award.
Matt Huenerfauth, Liming Zhou, Erdan Gu and Jan Allbeck. 2007. Design and Evaluation of an American Sign Language Generator. 45th Annual Meeting of the Association for Computational Linguistics. Workshop on Embodied Language Processing. Prague, Check Republic. June 2007.
Matt Huenerfauth. 2006. Representing Coordination and Non-Coordination in American Sign Language Animations. Behaviour & Information Technology, Volume 25, Issue 4, Pages 285-295.
Matt Huenerfauth. 2005. Representing Coordination and
Non-Coordination in an American Sign Language Animation. The 7th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2005). Baltimore, MD, USA.
Best Paper Award.
Matt Huenerfauth. 2005. American Sign Language Spatial Representations for an Accessible User-Interface. 3rd International Conference on Universal Access in Human-Computer Interaction. Las Vegas, NV, USA.
Matt Huenerfauth. 2005. American Sign Language Generation: Multimodal NLG with Multiple Linguistic Channels. Student Research Workshop, The 43rd Annual Meeting of the Association for Computational Linguistics. Ann Arbor, MI, USA.
Matt Huenerfauth. 2005. American Sign Language Natural Language Generation and Machine Translation. ACM SIGACCESS Accessibility and Computing. New York: ACM Press. Issue 81 (January 2005).
Matt Huenerfauth. 2004. American Sign Language Natural
Language Generation and Machine Translation. The 6th International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS 2004), Doctoral
Consortium Presentation and Poster Session. Atlanta, Georgia, USA.
Selected as Best Doctoral Candidate and asked to give Closing Plenary address.
Matt Huenerfauth. 2004. Spatial and Planning Models of ASL Classifier Predicates for Machine Translation. The 10th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI 2004). Baltimore, MD, USA.
Matt Huenerfauth. 2004. Spatial Representation of Classifier Predicates for Machine Translation into American Sign Language. Workshop on the Representation and Processing of Signed Languages, 4th International Conference on Language Resources and Evaluation (LREC 2004), Lisbon, Portugal.
Matt Huenerfauth. 2004. A Multi-Path Architecture for Machine Translation of English Text into American Sign Language Animation. In the proceedings of the Student Workshop at Human Language Technolgies conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT-NAACL 2004), Boston, MA, USA.
Matt Huenerfauth. 2003. A Survey and Critique of American Sign Language Natural Language Generation and Machine Translation Systems. Technical Report MS-CIS-03-32, Computer and Information Science, University of Pennsylvania.
This work was supported by the National Science Foundation (Award #0520798, SGER: Generating Animations of American Sign Language Classifier Predicates, Universal Access Program, 2005.)
Software was also donated by Siemens UGS Tecnomatix and Autodesk.