Considerations for Driver MFT Implementations on Multi-pin Cameras

September 28, 2012

Abstract

This paper provides information about how IHVs and system OEMs can use Windows 8 to create video processing plug-ins in the form of an MFT.

This information applies to the following operating systems: Windows 8 References and resources discussed here are listed at the end of this paper. The current version of this paper is maintained on the Web at: Considerations for Driver MFT Implementations on Multi-pin Cameras

Disclaimer: This document is provided as-is. Information and views expressed in this document, including URL and other Internet website references, may change without notice. Some information relates to prereleased product which may be substantially modified before its commercially released. Microsoft makes no warranties, express or implied, with respect to the information provided here. You bear the risk of using it. Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. 2012 Microsoft. All rights reserved.

Considerations for Driver MFT Implementations on Multi-pin Cameras - 2

Document History
Date September 28, 2012 June 27, 2012 February 28, 2012 Change Updated to reflect rebranding Updated Windows 8 terminology First publication

Contents
Introduction ................................................................................................................... 3 One-Pin Webcam ........................................................................................................ 3 Three-Pin Camera ....................................................................................................... 5 Compressed Video ...................................................................................................... 6 Possible Mediation ...................................................................................................... 7 Communication between instances of the MFT0 ...................................................... 7 Resources ..................................................................................................................... 7

September 28, 2012 2012 Microsoft. All rights reserved.

Considerations for Driver MFT Implementations on Multi-pin Cameras - 3

Introduction
Windows 8 offers IHVs and system OEMs the ability to create video processing plugins in the form of a Media Foundation Transform (MFT). This special MFT is known as the driver MFT. It is also known as MFT0 to indicate it is the first MFT to operate in the source reader. A separate instance of MFT0 is attached to every pin on the capture source. For some system OEMs, the AVStream capture driver must support a preview pin, a capture pin, and a still pin. This means that there may be three instances of MFT0. The diagram below shows this architecture, with three copies of the IHV Plug-in MFT, one for each stream.

Typical scenarios for MFT0 may present challenges. Two popular functions for MFT0 are: Analyzing the video stream to provide feedback to the camera for improved capture (such as host-based auto focus and auto exposure). Adding video effects.

One-Pin Webcam
Historically, cameras have been exposed to Windows as a single capture pin. The following diagram represents the way a one-pin webcam works:

September 28, 2012 2012 Microsoft. All rights reserved.

Considerations for Driver MFT Implementations on Multi-pin Cameras - 4

App
Device app

Preview

Capture Engine Sink Writer

Network

Uncompressed Video Video w/ Effects Compressed Video Compressed Video w/o Effects Camera Control

Source Reader
MFT0 File

Redundant Camera Control User Control

DevProxy

preview

UVC Driver / MSH webcam

In this case, both camera control and video effects work as designed because the preview and stills are teed from the capture pin after camera control and video effects are applied. The result is that the saved file or the users chat buddy will see the same video effects that the user sees in their preview, and that there is only one instance of the camera control feature. If there is an associated Windows Store device app, the app is connected to the MFT0 on the capture pin, so the MFT0 obtains the control messages from the app (that is, the user).

September 28, 2012 2012 Microsoft. All rights reserved.

Considerations for Driver MFT Implementations on Multi-pin Cameras - 5

Three-Pin Camera
A three-pin camera may have as many as three instances of MFT0, depending on the application needs. The following diagram represents the way a three-pin camera works:
App
Device app

Preview

Capture Engine Sink Writer

Source Reader
MFT0 MFT0 MFT0 File

DevProxy

preview

capture

still

Vendor AVStream Miniport

This situation presents several challenges. First, in the case of a host-based autoexposure solution, which requires direct control of the camera sensor and ISP settings, three MFT0s may be trying to control the camera at the same time. This breaks the control system. Second, there are potentially three instances of video effects. Beyond incurring the cost of three times the calculations, the three instances of the MFT0 must now communicate in a way that ensures each video frame always has the same effect in the exact same state. Otherwise what the user sees will not be what is saved or shared. Additionally, there are two final compounding factors: Each instance of the MFT0 may be created or shut down at any time. The Windows Store device app is only connected to one instance of the MFT0.
September 28, 2012 2012 Microsoft. All rights reserved.

Considerations for Driver MFT Implementations on Multi-pin Cameras - 6

Compressed Video
A webcam or system OEM may choose to compress the video before it is presented on the capture pin (that is, on the webcam itself). Offloading the compression to a webcam allows lower powered PCs to save and share HD video. This compressed video stream generally cannot be analyzed to support camera control, nor can video effects be applied. This presents the challenge of making all instances of MFT0 (and the Windows Store device app, if it is there) aware that no effects will be applied to the capture stream. The following image represents compressed video:
D C A MoCam Preview

Capture Engine Sink Writer

Source Reader
MFT0 MFT0 MFT0 File

DevProxy

preview

capture H.264

still

Vendor AVStream Miniport

If a user applies a video effect to the preview stream, it cannot be applied to the capture stream or the still pin. Therefore, the user sees a video effect that is not applied to the saved or streamed video. If the preview stream is halted, the Windows Store device app will then try to connect to the capture stream. When a user is streaming compressed video, this does not allow for many features.

September 28, 2012 2012 Microsoft. All rights reserved.

Considerations for Driver MFT Implementations on Multi-pin Cameras - 7

Possible Mediation
Communication between instances of the MFT0
If the three instances of the MFT0 can communicate with each other, this may solve most issues. How these instances discover each other is up to the IHV. Once all MFT0s can communicate, the instance of the MFT0 connected to the Windows Store device app can let other instances know which effect is being applied and its current state. Also, the three instances can determine which instance applies camera control. Finally, the preview pin can determine if the capture pin is streaming encoded video. The preview pin can then disable video effects. Although communication between instances of the MFT0 solves the major user experience issues, three instances of video effects must still be running concurrently. Some video effects can use most available CPU resources, particularly when full screen video is being previewed and captured at the same time. These are serious issues. For performance reasons, each ISV should consider what processing may be performed once, such as face detection, and shared with all instances of MFT0.

Resources
Driver MFT Sample http://go.microsoft.com/fwlink/?LinkId=252881 Developing Windows Store device apps for Cameras http://msdn.microsoft.com/library/windows/hardware/hh454870 AVStream Overview http://go.microsoft.com/fwlink/?LinkID=228585 Windows Store device app for Camera Sample http://go.microsoft.com/fwlink/?LinkID=227865 Cameras: Windows Store device apps http://msdn.microsoft.com/en-us/library/windows/hardware/hh833796.aspx

September 28, 2012 2012 Microsoft. All rights reserved.