Escolar Documentos
Profissional Documentos
Cultura Documentos
Acceleration APIs
BY the Industry
FOR the Industry
Khronos Standards
3D Asset Handling
- 3D authoring asset interchange
- 3D asset transmission format
with compression
Visual Computing
- 3D Graphics
- Heterogeneous Parallel Computing
Acceleration in HTML5
Sensor Processing
- 3D in browser no Plug-in
- Heterogeneous computing for JavaScript
- Vision Acceleration
- Camera Control
- Sensor Fusion
Vision
Processing
Imagery
Enhanced sensor
and vision
capability deepens
the interaction
between real and
virtual worlds
Data
Graphics
Processing
Computational
Photography and
Videography
3D Scene/Object
Reconstruction
Augmented
Reality
Copyright Khronos Group 2014 - Page 5
Sensor Proliferation
Touch
Camera ISPs
Dedicated vision IP blocks
DSPs and DSP arrays
Programmable GPUs
Multi-core CPUs
19
Position
- GPS
- WiFi (fingerprint)
- Cellular trilateration
- NFC/Bluetooth Beacons
Accelerometer
Magnetometer
Gyroscope
Pressure / Temp / Humidity
Advanced
Sensors
X100
Power Efficiency
Wearables
X10
X1
Dedicated
Hardware
GPU
Compute
Multi-core
CPU
Computation Flexibility
Copyright Khronos Group 2014 - Page 7
Application
OpenCV open
source library
Other higher-level
CV libraries
Complementary to OpenCV
- Which is great for prototyping
Hardware vendor
implementations
Native
Camera
Control
OpenVX
Node
OpenVX
Node
OpenVX
Node
OpenVX
Node
Downstream
Application
Processing
OpenVX Specification
Evolution
OpenVX 1.0 defines
framework for
creating, managing and
executing graphs
Focused set of widely
used functions that are
readily accelerated
Implementers can add
functions as extensions
Copyright Khronos Group 2014 - Page 10
Stereo
Rectify with
Remap
Camera 2
Stereo
Rectify with
Remap
Compute Depth
Map
(User Node)
Detect and
track objects
(User Node)
Image
Pyramid
Object
coordinates
Compute
Optical
Flow
Delay
Tiling extension enables user nodes (extensions) to also optimally run in local memory
Governance
Conformance
Portability
Scope
Very wide
1000s of imaging and vision functions
Multiple camera APIs/interfaces
Efficiency
Memory-based architecture
Each operation reads and writes memory
Graph-based execution
Optimizable computation, data transfer
Use Case
Rapid experimentation
SLAM
VisionWorks
Framework
VisionWorks Primitives
Classifier
Corner
Detection
3rd Party
CUDA Libraries
Tegra K1
Copyright Khronos Group 2014 - Page 14
GPU
DSP
HW
CPU
CPU
JavaScript
binding for
initiation of
OpenCL C
kernels
Language for
image
processing and
computational
photography
MulticoreWare
open source
project on
Bitbucket
Embedded
array
language for
Haskell
Compiler
directives for
Fortran,
C and C++
PyOpenCL
Python
wrapper
around
OpenCL
Harlan
High level
language
for GPU
programming
SPIR
Standard Portable
Intermediate Representation
(extending LLVM for parallel computation)
Out of the Box Vision Framework - Operators and graph framework library
Can run on dedicated hardware no compiler needed
Easier performance portability to diverse hardware
Suited for low-power, always-on acceleration
Fixed set of operators but can be extended
Pre-processing
Bayer
Postprocessing
App
RGB
YUV
CMOS sensor
Color Filter Array
Lens
Optional mid
exposure
Long
exposure
HDR processing
HDR requires
- Precise control over camera parameters (exposure)
- Fast capture and processing of multiple images
- Note: with interlace HDR, only 1 image is needed
Copyright Khronos Group 2014 - Page 22
Made with
Image Signal
Processor (ISP)
EGLStreams
Image/Vision
Applications
OpenKCAM is FCAM-based
FCAM (2010) Stanford/Nokia, open source
Capture stream of camera images with precision control
- A pipeline that converts requests into image stream
- All parameters packed into the requests - no visible state
- Programmer has full control over sensor settings for each frame in stream
Control over focus and flash
- No hidden daemon running
Control ISP
- Can access supplemental
statistics from ISP if available
No global state
- State travels with image requests
- Every pipeline stage may have different state
- Enables fast, deterministic state changes
Copyright Khronos Group 2014 - Page 28
Camera API
Open source
project developed
by Nokia and
Stanford
HAL V3 adopts many
FCAM ideas and can use
EGL in its implementation
Group charter
approved
Specification
ratification
3Q14
Apr13
Jul13
Sample
implementation and
tests
1Q15
Sensor Types
Basic sensor data:
- Acceleration, Magnetic Field, Angular Rates
- Pressure, Ambient Light, Proximity, Temperature, Humidity, RGB light, UV light
- Heart rate, Blood Oxygen Level, Skin Hydration, Breathalyzer
Sensor fusion
- Orientation (Quaternion or Euler Angles), Gravity, Linear Acceleration
- Position
Context awareness
- Device Motion: general movement of the device: still, free-fall,
- Carry: how the device is being held by a user: in pocket, in hand,
- Posture: how the body holding the device is positioned: standing, sitting, step,
- Transport: about the environment around the device: in elevator, in car,
Sensor Discoverability
Sensor Code Portability
OS Sensor APIs
Middleware
Sensor
Sensor
Sensor
Sensor
Hub
Hub
Applications
API Interop
EGL provides efficient
transfer of data and events
between Khronos APIs
Context Robustness
- Defending against malicious code
EGLSync objects
- Improved OpenGL /OpenCL interop
Platform extensions
- Standardized interactions for multiple
OS e.g. Android and 64-bit platforms
Application Portability
EGL abstracts graphics context
management, surface and
buffer binding and rendering
synchronization
OS and Display
Platforms
Copyright Khronos Group 2014 - Page 39
MEMS
Sensors
Sensor
Fusion
Application
on CPUs, GPUs
and DSPs
Vision
Processing
Precision timestamps
on all sensor samples
Advanced Camera
Control and stream
generation
Audio
Rendering
Summary
Khronos is building a family of interoperating APIs for portable and
power-efficient vision processing
OpenVX 1.0 has been provisionally released and non-members are invited to
provide feedback on the forums
- http://www.khronos.org/message_boards/forumdisplay.php/110-OpenVX-General
OpenKCAM and StreamInput APIs are currently in design and complement and
integrate with OpenVX
Any company is welcome to join Khronos to influence the direction of mobile
and embedded vision processing!
- $15K annual membership fee for access to all Khronos API working groups
- Well-defined IP framework protects your IP and conformant implementations
www.khronos.org
- ntrevett@nvidia.com