Understanding the bi-directional relationships between human movement, natural languages, instructions, and communication

 

The goal of this project is to develop new technologies that enable the machine translation of English text into animations of American Sign Language.  This research will make more information and services available to the majority of Deaf Americans who face English literacy challenges.  Because signed languages, like ASL, contain phenomena not seen in traditional written/spoken languages, they are particularly challenging to process using standard machine translation approaches.  Exploring the computational linguistics of ASL can help us understand the limitations of current machine translation technologies and motivate the development of new ones.

This project also relates to the AVIS and RIVET projects, as we will be building an authoring tool for creating instructions from expert performances of the actual tasks. The experts will be motion captured, videotaped, and audio taped performing some complex assembly or disassembly task. Using these multi-modal inputs on a synchronized timeline, instructions can be authored semi-automatically. One of the principal challenges in acquiring such data is determining what is actually happening on and to the object assembly and its components. We hypothesize that we can ascertain what the expert is doing from line of regard (head direction), verbalizations, and hand motions, all inputs that we will be acquiring. The actions themselves will be formulated as Parameterized Actions and, as such, can benefit from partial matching inputs against an Actionary of known action templates.

Primary funding: NSF American Sign Language Natural Language Generation and Machine Translation (N. Badler and M. Marcus, PIs)

•  J. Allbeck and N. Badler. “Creating Embodied Agents with Cultural Context.” In R. Trappl and S. Payr (Eds.), Agent Culture: Designing virtual characters for a multi-cultural world . New York : Lawrence Erlbaum Associates, 2004.

• A. Bloomfield, Y. Deng, P. Rondot, J. Wampler, D. Harth, M. McManus and N. Badler. “A taxonomy and comparison of haptic actions for disassembly tasks.” IEEE Virtual Reality Conf., LA, CA, March 2003, pp. 225-231.