Inter-media Synchronization

group PJMEDIA_AV_SYNC

Synchronize presentation time of multiple media in a session.

A call session may consist of multiple media, e.g: some audio and some video, which frequently have different delays when presented in the receiver side. This module synchronizes all media in the same session based on NTP timestamp & RTP timestamp info provided by the sender in RTCP SR.

Here are steps to use this module:

  1. Create AV sync using pjmedia_av_sync_create().

  2. Adds all media to be synchronized using pjmedia_av_sync_add_media().

  3. Call pjmedia_av_sync_update_ref() each time the media receiving an RTCP SR packet.

  4. Call pjmedia_av_sync_update_pts() each time the media returning a frame to be presented, e.g: via port.get_frame(). The function may request the media to adjust its delay.

  5. Call pjmedia_av_sync_del_media() when a media is removed from the session.

  6. Call pjmedia_av_sync_destroy() when the session is ended.

The primary synchronization logic is implemented within the pjmedia_av_sync_update_pts() function. This function will calculate the lag between the calling media to the earliest media and will provide a feedback to the calling media whether it is in synchronized state, late, or early so the media can respond accordingly. Initially this function will try to request slower media to speed up. If after a specific number of requests (i.e: configurable via PJMEDIA_AVSYNC_MAX_SPEEDUP_REQ_CNT) and the lag is still beyond a tolerable value (i.e: configurable via PJMEDIA_AVSYNC_MAX_TOLERABLE_LAG_MSEC), the function will issue slow down request to the fastest media.

Typedefs

typedef struct pjmedia_av_sync pjmedia_av_sync

Inter-media synchronizer, opaque.

typedef struct pjmedia_av_sync_media pjmedia_av_sync_media

Media synchronization handle, opaque.

Functions

void pjmedia_av_sync_setting_default(pjmedia_av_sync_setting *setting)

Get default settings for synchronizer.

Parameters:

setting – The synchronizer settings.

void pjmedia_av_sync_media_setting_default(pjmedia_av_sync_media_setting *setting)

Get default settings for media.

Parameters:

setting – The media settings.

pj_status_t pjmedia_av_sync_create(pj_pool_t *pool, const pjmedia_av_sync_setting *setting, pjmedia_av_sync **av_sync)

Create media synchronizer.

Parameters:

  • pool – The memory pool.

  • option – The synchronizer settings.

  • av_sync – The pointer to receive the media synchronizer.

Returns:

PJ_SUCCESS on success.

void pjmedia_av_sync_destroy(pjmedia_av_sync *av_sync)

Destroy media synchronizer.

Parameters:

av_sync – The media synchronizer.

pj_status_t pjmedia_av_sync_reset(pjmedia_av_sync *av_sync)

Reset synchronization states. Any existing media will NOT be removed, but their states will be reset.

Parameters:

av_sync – The media synchronizer.

Returns:

PJ_SUCCESS on success.

pj_status_t pjmedia_av_sync_add_media(pjmedia_av_sync *av_sync, const pjmedia_av_sync_media_setting *setting, pjmedia_av_sync_media **av_sync_media)

Add a media to synchronizer.

Parameters:

  • av_sync – The media synchronizer.

  • setting – The media settings.

  • av_sync_media – The pointer to receive the media synchronization handle.

Returns:

PJ_SUCCESS on success.

pj_status_t pjmedia_av_sync_del_media(pjmedia_av_sync *av_sync, pjmedia_av_sync_media *av_sync_media)

Remove a media from synchronizer.

Parameters:

  • av_sync – The media synchronizer.

  • av_sync_media – The media synchronization handle.

Returns:

PJ_SUCCESS on success.

pj_status_t pjmedia_av_sync_update_pts(pjmedia_av_sync_media *av_sync_media, const pj_timestamp *pts, pj_int32_t *adjust_delay)

Update synchronizer about the last presentation timestamp of the specified media. Normally this function is called each time the media produces a frame to be rendered (e.g: in port’s get_frame() method). Upon returning, the media may be requested to adjust its delay so it matches to the earliest or the latest media, i.e: by speeding up or slowing down.

Initially this function will try to request slower media to speed up. If after a specific number of requests (i.e: configurable via PJMEDIA_AVSYNC_MAX_SPEEDUP_REQ_CNT) and the lag is still beyond a tolerable value (i.e: configurable via PJMEDIA_AVSYNC_MAX_TOLERABLE_LAG_MSEC), the function will issue slow down request to the fastest media.

Parameters:

  • av_sync_media – The media synchronization handle.

  • pts – The presentation timestamp.

  • adjust_delay – Optional pointer to receive adjustment delay required, in milliseconds, to make this media synchronized to the fastest media. Possible output values are: 0 when no action is needed, possitive value when increasing delay is needed, or negative value when decreasing delay is needed.

Returns:

PJ_SUCCESS on success.

pj_status_t pjmedia_av_sync_update_ref(pjmedia_av_sync_media *av_sync_media, const pj_timestamp *ntp, const pj_timestamp *ts)

Update synchronizer about reference timestamps of the specified media. Normally this function is called each time the media receives RTCP SR packet.

Parameters:

  • av_sync_media – The media synchronization handle.

  • ntp – The NTP timestamp info from RTCP SR.

  • ts – The RTP timestamp info from RTCP SR.

Returns:

PJ_SUCCESS on success.

struct pjmedia_av_sync_setting
#include <av_sync.h>

Synchronizer settings.

Public Members

char *name

Name of the syncrhonizer

pj_bool_t is_streaming

Streaming mode. If set to PJ_TRUE, the delay adjustment values will be smoothened and marked up to prevent possible delay increase on all media.

struct pjmedia_av_sync_media_setting
#include <av_sync.h>

Media settings.

Public Members

char *name

Name of the media

pjmedia_type type

Media type.

unsigned clock_rate

Media clock rate or sampling rate.