[mediacapture-depth] Should we do depth-video synchronization in browser, in JS, in both, what's v1 and what's v2?

astojilj has just created a new issue for 
https://github.com/w3c/mediacapture-depth:

== Should we do depth-video synchronization in browser, in JS, in 
both, what's v1 and what's v2? ==
The API offers two possibilities to do synchronization:
1. inside browser code - by #138 Synchronization
2. In JS code - by #110 accessing camera calibrations (intrinsics and 
extrinsics)
Q1: Do we need both?
Q2: Do we need intrinsics&extrinsics for some other use case? Is 
deprojecting depth to 3D space an use case for v1?
Q3: Is manual calibration v1 use case?

If answer to Q2 or Q3 is "yes", then in we just need to implement 2. 
JS synchronization using #110 intrinsics and extrinsics (to support 
synchronization).

Comment 
[here](https://github.com/w3c/mediacapture-depth/issues/138#issuecomment-259838174)
 expresses why 1. inside browser code is problematic:
a) performance issue in depth->video
b) quality loss in video->depth and the often this sync is not 
possible/difficult. e.g. for RealSense depth->video is always 
supported.

Other concerns:
c) various distortion models for different cameras is difficult to 
code and maintain in multiple browsers.
- [Tango FOV equidistant (no distortion) pinhole 
model](https://developers.google.com/tango/overview/intrinsics-extrinsics#tangos_lens_distortion_models)
- [Tango polynomial - Tsai 
model](https://developers.google.com/tango/apis/c/reference/struct/tango-camera-intrinsics
) 
- [RealSense Modified Brown-Conrady Distortion
](https://github.com/IntelRealSense/librealsense/blob/master/doc/projection.md#distortion-models)
- [RealSense Inverse Brown-Conrady 
Distortion](https://github.com/IntelRealSense/librealsense/blob/master/doc/projection.md#distortion-models)

Most of the models are applied when deprojecting 2D depth to 3D space,
 RealSense Inverse Brown-Conrady distortion means that the distortion 
is compensated when projecting 3D to RGB video 2D space.



Please view or discuss this issue at 
https://github.com/w3c/mediacapture-depth/issues/145 using your GitHub
 account

Received on Tuesday, 15 November 2016 19:27:46 UTC