An important contribution of the Jack virtual human model is that it allows us to view and analyze human actions such as locomotion, reach, grasp and gesture. Tasks composed of such elementary actions may in turn be combined and performance observed. Simply stringing together sequences of tasks, however, is unrealistic since humans require time to visually attend and acquire information before or during execution of tasks. The action representation requires a framework of sensing processes in which visual search, hand-eye coordination and attention guides the sequencing and simulation of actions, especially motor tasks. Other sensing activities that we must represent are monitoring, probing the state of the world via global or internal variables, checking time on global or relative clocks and evaluating spatial/geometric predicates (relationships).
We contend that sensing activities or processes are themselves tasks
whose duration and execution are significant. Further, timing
information for processes such as attention and visual search is not
static. Such information will vary with cognitive load, expertise and
type of elementary action. Studies show that eye movement
latencies are longer when accompanied by certain spatially oriented
gestures like pointing[BKA
95].
The Jack system maintains a geometry of objects and relevant sites in a virtual world. Since access to these objects is negligible in terms of computer simulation time, some explicit representation of sensing or investigative process is particularly important. Further efforts are required to model the time durations introduced by sensing and attentive processes. Part of this problem is being addressed by explicit attentional models constructed within the OMAR [BBN94] system. Since it is likely that a comprehensive attentional timing model would be difficult to develop, we may first incorporate simplified reaction time computations to achieve a modest level of performance veracity.
The sensing processes themselves include a number of particular features:
To the extent made possible by the functional definitions, the future variable values of a process node may be queried by a sensing process. In the case of a formula for linear motion, for example, the present trajectory may be extrapolated to yield an estimate of possible future location. The internal object description includes fields for velocity and acceleration as well as position and orientation. It is speculated that the ability to successfully run processes in a predictive mode will improve agent/agent and agent/object interactions by changing ``pursuit'' situations into ``anticipation'' situations [Rei97].