Based on our extensive MoCap and film production and post-production experience, we are developing advanced system that allows to mix all available technologies together.
Virtual camera
MoCap system can output reconstructed skeletal animation of many performers in real time (total latency of tenths of msec only). This is streamed to Motion Builder, rendered and displayed either on a back projection, or sent over wifi to the iPad tablet device (wifi adds additional latency, so total latency of a live action and the iPad's display is typically <0.2 sec). To our knowledge, this concept was pioneered by Peter Jackson's team in designing fighting scenes in Mines of Moria of Lord of the Rings: Fellowship of the ring, however our solution is far more advanced today.
The iPad is placed in a frame, that is being tracked as any other rigid body object, and may control position and rotation of a virtual camera.
- Because the image is streamed back to the iPad, it works as real window to the virtual scene - one can rotate it anywhere and it immediately displays the virtual scene in that direction on the display.
- This helps incredibly in mapping real studio to a virtual scene, localizing virtual walls and objects that don't exist in a real studio.
- Of course, trajectory of the iPad is being recorded and may be used for the production: one can record camera's trajectory with this tool.
- No annoying wires. Wireless image transfer makes everything easy to use and convenient without anoying cables lying on a floor.
Because iPad's application is custom designed and not just some remote desktop tool, the system may be fully controlled from the iPad remotely.
- it allows one to control zoom of a virtual camera with multitouch gestures.
- it supports syncing of all the recording devices and reference playback sources.
- recording may be started and stopped from it.
- switching between live camera, front, top and other views possible
- zooming in/out recorded with a MC data and applied to Motion Builder's camera afterwards
Augmented reality camera
An extension of a virtual camera with an overlaid image from a ordinary video camera in real time. This concept was used by John Cameron when filming Avatar movie extensively.
It allows a director to see virtual characters as "ghosts" over real camera image. The real camera is being tracked by the MoCap system and the virtual camera is calibrated to match real camera's lens characteristics closely. Outputs of both cameras are blended and displayed. The result is an image where image of a real MoCap performer is overlaid with an image of his or her CG ghost character correctly no matter how the camera moves around a place.
This is a ultimate system that helps mix and match blue screen performance with a virtual action.