### Summary

This paper presents a solution to computer puppetry, i.e. animating a computer generated character in real-time given motion capture input from an actor. Computer Puppetry is useful in producing things like daily cartoon shows where there is little time to produce the final animation and for displaying/transmitting a show in real-time, for example, to cover an event.

### Problem

Computer puppetry requires processing motion capture data in real-time in order to animate a computer generated character. However, (1) motion capture data useful for real-time applications is noisy, (2) processing must be done online, therefore one cannot "look in to the future" when processing data, (3) the animated character is not likely to have the same size or proportions as the actor requiring the motion data to be retargetted on the fly.

### Methods/Solution

An extended Kalman filter is used to de-noise the incoming data. The Kalman filter is useful here because it predicts future data and adapts as actual data is read in. The key factors here are that the filter must be fast enough to run in real-time and that it must be able to filter orientations, and thus account for the non-linearity in orientation space.

Due to the retargetting of the motion, both the positions of the end-effectors and the joint angles cannot be simultaneously preserved. It is therefore necessary to determine which is more important in the context of the environment. Basically, each end-effector is given an importance value based on its proximity to objects in the world. This is based on the observation that it is more important to preserve joint angles than end-effector positions unless an object is being interacted with. 

Finally, a fast IK solver is used to generate a pose that preserves end-effector positions when necessary and joint angles otherwise. This is done quickly by attempting to position the root node such that end-effector constraints are met (or nearly met). Then numerical optimization is performed on the root and upper body posture to ensure that end-effector constraints are met given their importance factors. Finally limb postures are solved for using an analytic solution. One thing to note here is that the most expensive portion of this process, the numerical optimization, has proven to be unnecessary much of the time thanks to the root node estimation.

### Contributions

1. An extended Kalman filter able to run in real-time and filter out high frequencies in orientation data via use of log and exponential maps linearize incremental orientation changes.

2. Method for computing the importance of preserving each end-effector position given positions and bounding volumes of objects in the world.

3. Fast IK solver for real-time use taking advantage of a powerful root node approximation method, and analytic solutions for a portion of the problem, only a small amount of numerical optimization is required.

### Questions/Comments

A few questions come to mind due to this paper. First, what applications this might have. Live performance comes to mind, but seems unlikely. It could be used for some sort of interactive application if done right. For producing things such as commercials, shorts, and features, it seems like doing offline processing is sufficient. I guess I'm curious where exactly computer puppetry is useful -- the examples in the paper weren't very convincing and I can't think of why the example of the children's show needed computer puppetry. It was just easier, perhaps?

I'm curious about the setup used for doing computer puppetry. One of the major points of this paper was preserving motion where the puppet should interact with the environment. I'm curious if the actor interacts with an environment that is simply duplicated in the computer generated world, or if the actor pretends to interact with objects, or if the objects in the real world are being tracked for display in the computer generated world.

This brings up an interesting problem. As an example, let's say the actor picks up a glass of water. Then the virtual glass of water must be moved with the real glass of water -- in real time. This seems to imply to me that objects are being tracked by the system. If the objects in the computer generated world are static, then they must either be pre-scripted, in which case synchronization becomes an issue, or the objects must be moved by an animator in real time which seems prohibitively difficult.