A virtual human simulation must represent human physical capabilities and limitations. For cognitive and intellectual domains, the computer must understand reasoning, decision-making, and communication. One connection between these two domains lies in the understanding and execution of instructions: commands or descriptions of physical activities or their consequences. Accordingly, the computational representation of information and processes sufficient for instruction understanding (from textual material into action) and text generation (from task performance or animation into instructions) is a challenging research topic.
This study examines the feasibility of bridging the computational gap between
the simulations of the physical world and the symbolic world of language as
instructions for human activities in those environments. Our scope is
somewhat further limited in this document to examining text generation rather
than understanding, though we have current and historical interest in the
instruction understanding problem as well [BWKE91,BPW93,WBE
95].
Textual instruction generation is well motivated by the desire to partially automate the production of Technical Orders (instruction manuals) for Air Force maintenance and repair activities. The desire to engage computational tools in this process is motivated by several unique characteristics of Technical Order (TO) production:
Increasing use of digital mockup and virtual prototyping tools is providing digital environments in which human tasks may be tried and tested. Graphical systems such as Jack [BPW93] and DEPTH are making design evaluation and human factors analysis feasible prior to system production and deployment. At this stage, it is desirable to debug maintenance tasks and correct errors in the design itself. Once the maintenance actions appear feasible (relative to the expected population of maintainers), the analyses should be preserved, and in fact a written record of the analysis is all that is required now (by storing a record in an LSAR database). An important observation is that additional useful information is potentially available that can aid in the generation of the requisite TO: namely, the animation ``commands'' that created the task analysis may be suitable as a framework for TO textual descriptions of that same task. This opportunity requires careful study, though, as the expression of human actions for animation or analysis is not the same as that used for TOs. While a desirable goal, current knowledge is simply not yet ready to deliver that kind of performance.
Our goal in this document to to examine the extent of the gap between animation and instruction generation, and make recommendations for research and development efforts that have the potential to reduce or eventually close this gap. Accordingly, this report focuses on two main themes. The first concerns the representational issues surrounding computer models of processes and human actions in simulated environments. The major goals of this part are to try to design a concept-rich structure that will capture characteristics of tasks in terms of sensing, acting, and decision-making, to justify the computational feasibility of the representation, and to show potential influences will be exerted on the task animation user interface. The second theme centers on Natural Language text generation. The major emphases are on methodologies of text generation, specialized requirements for TOs, the role of verbs and objects in existing TOs, and examples from the maintenance domain.