AVCaptureDeviceDiscoverySession & stream continuity across physical hinge transitions with Virtual Front Camera

Question

JulienKermarec OP

Created 3d

Replies 3

Boosts 0

Participants 2

This post is from the Activity iPhone Duo Photos & Camera Q&A.

Hello team,

When configuring an AVCaptureMultiCamSession or consuming the new Virtual Front Camera, what is the recommended architecture to avoid frame drops or pipeline rebuilds (AVCaptureSessionRuntimeError) during dynamic posture transitions (from folded to unfolded / half-open)?

Specifically:

Does the Virtual Front Camera handle the lens handover internally with continuous timestamps in CMSampleBuffer, or does the underlying device switch trigger a device disconnect/reconnect notification that requires re-attaching inputs?

How are the optical intrinsics (matrix_float3x3) and lens distortion coefficients updated through the hinge motion? Can we rely on continuous metadata attachments on each frame during the transition, particularly if running real-time CoreML / Vision processing?

Thanks!

Answer 1

Frameworks Engineer OP

Apple

3d

When using the Virtual Front Camera, you'll see frame drops when the app switches between the physical outer and inner front cameras.

Optical intrinsics and lens distortion coefficients should get updated as the Virtual Front Camera switches cameras.

Answer 2

JulienKermarec OP

3d

Thank you for clarifying!

Following up on the frame drop during the switch:

Does the session drop frames silently at the AVCaptureVideoDataOutput level, or will it explicitly fire a CMSampleBuffer with the .droppedFrameReason attachment (or post an AVCaptureSessionWasInterruptedNotification)?

What is the typical handover latency (in frames or ms) developers should account for to freeze/gracefully cross-fade the camera preview during posture transitions?

Answer 3

Frameworks Engineer OP

Apple

3d

There is no explicit dropped frame reason or session interruption when the Virtual Front Camera switches between cameras. When observing the timestamps of the CMSampleBuffers, you will notice a discontinuity in the timestamp of CMSampleBuffers sent out on AVCaptureVideoDataOutput.

The latency of this switch can vary based on how the AVCaptureSession is configured. We have optimized the switch time to be as minimal as possible, but certain configurations will require us to take longer to switch cameras.