Hi,

We are working on a iOS app which will connect to a remote device with
720P(1280x720) video streaming.
We follow the guide to make PJSIP accept 720P remote streaming.
But the video have serious latency even on iPhoneX, and the bottleneck
seems to be the I420ToARGB convert via libyuv.
I420ToARGB is too slow to make 720P remote video workable.
Is there any way to improve the performance?

Any suggestion for this situation would be welcome.
Thanks.

BR,

Kent

Kent,

If you are working on IOS. You might be able to do the conversion from 420toARGB in a VideoToolbox decompression session. Its very fast. Libyuv is CPU based.

This example uses the input in 420 and converts it to 32_ARGB.

CFMutableDictionaryRef destinationPixelBufferAttributes = CFDictionaryCreateMutable(

                                                                                    NULL, // CFAllocatorRef allocator

                                                                                    0,    // CFIndex capacity

                                                                                    &kCFTypeDictionaryKeyCallBacks,

                                                                                    &kCFTypeDictionaryValueCallBacks);



SInt32 destinationPixelType = kCVPixelFormatType_32ARGB;

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferPixelFormatTypeKey, CFNumberCreate(NULL, kCFNumberSInt32Type, &destinationPixelType));

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferWidthKey,  (__bridge CFNumberRef)@(1280));

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferHeightKey, (__bridge CFNumberRef)@(720));

OSStatus status =  VTDecompressionSessionCreate(NULL, _videoFormatDescr, NULL,

                                                destinationPixelBufferAttributes,

                                                &callBackRecord, &_decompressionSession);

You might need to open up darwindev.m and make the necessary changes. I believe pjsip does have a decompression session already created using video toolbox.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Thursday, February 14, 2019 at 7:39 AM
To: "pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org" <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: [pjsip] Handle 1280x720 remote video very slow in iOS App

Hi,

We are working on a iOS app which will connect to a remote device with 720P(1280x720) video streaming.
We follow the guide to make PJSIP accept 720P remote streaming.
But the video have serious latency even on iPhoneX, and the bottleneck seems to be the I420ToARGB convert via libyuv.
I420ToARGB is too slow to make 720P remote video workable.
Is there any way to improve the performance?

Any suggestion for this situation would be welcome.
Thanks.

BR,

Kent

Sure,

Let me know how it goes. Should be straightforward. Ive successfully done the conversion with video toolbox, however
modifying the pjsip code I have not, but Ive looked at the video toolbox darwindev impl pretty thoroughly. However, they are just wrapping the toolbox session into the hooks that the
C library needs to work with the pjsip arch. Since you are doing a decompression session make sure to log out any decode errors. The pjsip/qt/sdl camera demo works decently – I recommend checking that. It uses SDL to render which seems adequate (OSX/Win Only) I believe.

Send the sample buffers to an OpenGL ES session. There’s a couple implementations for IOS floating on the net somewhere.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Friday, February 22, 2019 at 5:40 PM
To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: Re: [pjsip] Handle 1280x720 remote video very slow in iOS App

Dear Austin:

Thanks for your reply.
I will try it. If necessary, please still support me in this issue.
Thanks.

Kent

Austin Kottke <Austin.Kottke@tollfreeforwarding.commailto:Austin.Kottke@tollfreeforwarding.com> 於 2019年2月15日 週五 上午2:29寫道：

Kent,

If you are working on IOS. You might be able to do the conversion from 420toARGB in a VideoToolbox decompression session. Its very fast. Libyuv is CPU based.

This example uses the input in 420 and converts it to 32_ARGB.

CFMutableDictionaryRef destinationPixelBufferAttributes = CFDictionaryCreateMutable(

                                                                                    NULL, // CFAllocatorRef allocator

                                                                                    0,    // CFIndex capacity

                                                                                    &kCFTypeDictionaryKeyCallBacks,

                                                                                    &kCFTypeDictionaryValueCallBacks);



SInt32 destinationPixelType = kCVPixelFormatType_32ARGB;

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferPixelFormatTypeKey, CFNumberCreate(NULL, kCFNumberSInt32Type, &destinationPixelType));

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferWidthKey,  (__bridge CFNumberRef)@(1280));

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferHeightKey, (__bridge CFNumberRef)@(720));

OSStatus status =  VTDecompressionSessionCreate(NULL, _videoFormatDescr, NULL,

                                                destinationPixelBufferAttributes,

                                                &callBackRecord, &_decompressionSession);

You might need to open up darwindev.m and make the necessary changes. I believe pjsip does have a decompression session already created using video toolbox.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Thursday, February 14, 2019 at 7:39 AM
To: "pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org" <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: [pjsip] Handle 1280x720 remote video very slow in iOS App

Hi,

We are working on a iOS app which will connect to a remote device with 720P(1280x720) video streaming.
We follow the guide to make PJSIP accept 720P remote streaming.
But the video have serious latency even on iPhoneX, and the bottleneck seems to be the I420ToARGB convert via libyuv.
I420ToARGB is too slow to make 720P remote video workable.
Is there any way to improve the performance?

Any suggestion for this situation would be welcome.
Thanks.

BR,

Kent

Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

Ensure your configuration (config_site) has

PJMEDIA_HAS_VID_TOOLBOX_CODEC=1

Otherwise the IOS video stream might not be using VideoToolBox. Recompile.

I don’t think (I could be wrong) that vid_port.c is used when VideoToolBox is enabled.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Monday, February 25, 2019 at 6:13 PM
To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: Re: [pjsip] Handle 1280x720 remote video very slow in iOS App

Dear Austin:

Thanks for your kind reply. I do appreciate for it.
We don't find a good point at darwin_dev.m, should I modify at darwin_stream_get_frame() ?

So far the following way is what we are going to do:
1.Disable original I422 to ARGB at convert_frame() of vid_port.c, make it pass original pjmedia_frame to ios_opengl_dev.m.
2.Convert I422 to ARGB at iosgl_stream_put_frame() of ios_opengl_dev.m.
3.Not only VideoToolbox, we are also considering if it is a good way to use this:

https://developer.apple.com/documentation/accelerate/1533095-vimageconvert_420yp8_cbcr8toargb?language=objc

Any further advise is welcome.
Thanks.

Kent

Austin Kottke <Austin.Kottke@tollfreeforwarding.commailto:Austin.Kottke@tollfreeforwarding.com> 於 2019年2月26日 週二 上午2:57寫道：
Sure,

Let me know how it goes. Should be straightforward. Ive successfully done the conversion with video toolbox, however
modifying the pjsip code I have not, but Ive looked at the video toolbox darwindev impl pretty thoroughly. However, they are just wrapping the toolbox session into the hooks that the
C library needs to work with the pjsip arch. Since you are doing a decompression session make sure to log out any decode errors. The pjsip/qt/sdl camera demo works decently – I recommend checking that. It uses SDL to render which seems adequate (OSX/Win Only) I believe.

Send the sample buffers to an OpenGL ES session. There’s a couple implementations for IOS floating on the net somewhere.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Friday, February 22, 2019 at 5:40 PM
To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: Re: [pjsip] Handle 1280x720 remote video very slow in iOS App

Dear Austin:

Thanks for your reply.
I will try it. If necessary, please still support me in this issue.
Thanks.

Kent

Austin Kottke <Austin.Kottke@tollfreeforwarding.commailto:Austin.Kottke@tollfreeforwarding.com> 於 2019年2月15日 週五 上午2:29寫道：

Kent,

If you are working on IOS. You might be able to do the conversion from 420toARGB in a VideoToolbox decompression session. Its very fast. Libyuv is CPU based.

This example uses the input in 420 and converts it to 32_ARGB.

CFMutableDictionaryRef destinationPixelBufferAttributes = CFDictionaryCreateMutable(

                                                                                    NULL, // CFAllocatorRef allocator

                                                                                    0,    // CFIndex capacity

                                                                                    &kCFTypeDictionaryKeyCallBacks,

                                                                                    &kCFTypeDictionaryValueCallBacks);



SInt32 destinationPixelType = kCVPixelFormatType_32ARGB;

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferPixelFormatTypeKey, CFNumberCreate(NULL, kCFNumberSInt32Type, &destinationPixelType));

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferWidthKey,  (__bridge CFNumberRef)@(1280));

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferHeightKey, (__bridge CFNumberRef)@(720));

OSStatus status =  VTDecompressionSessionCreate(NULL, _videoFormatDescr, NULL,

                                                destinationPixelBufferAttributes,

                                                &callBackRecord, &_decompressionSession);

You might need to open up darwindev.m and make the necessary changes. I believe pjsip does have a decompression session already created using video toolbox.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Thursday, February 14, 2019 at 7:39 AM
To: "pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org" <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: [pjsip] Handle 1280x720 remote video very slow in iOS App

Hi,

We are working on a iOS app which will connect to a remote device with 720P(1280x720) video streaming.
We follow the guide to make PJSIP accept 720P remote streaming.
But the video have serious latency even on iPhoneX, and the bottleneck seems to be the I420ToARGB convert via libyuv.
I420ToARGB is too slow to make 720P remote video workable.
Is there any way to improve the performance?

Any suggestion for this situation would be welcome.
Thanks.

BR,

Kent

Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

This looks good so far.

Put a few trace statements in darwin_dev and you should see the h264 Nal units being processed.

Darwin dev should also create the video toolbox session which you should be able to capture the
AvCaptureSession creation and see how the media flows in and out.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Monday, March 4, 2019 at 5:58 AM
To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: Re: [pjsip] Handle 1280x720 remote video very slow in iOS App

Dear Austin:

Thanks for your kind reply and remind.
It is my bad. darwin_dev.m is used already. darwin_stream_get_frame() is called,
But vid_port.c seems still be needed. We will make further study of it.

Besides, this is our config_site.h. It should be not miss anything,  right?

---======
#define PJ_CONFIG_IPHONE 1
#define PJ_HAS_IPV6 1

// disable background VoIP socket, use PushKit
#undef PJ_IPHONE_OS_HAS_MULTITASKING_SUPPORT
#define PJ_IPHONE_OS_HAS_MULTITASKING_SUPPORT 1

#define PJMEDIA_HAS_VIDEO 1
#define PJMEDIA_HAS_VID_TOOLBOX_CODEC 1
#include <pj/config_site_sample.h>

---======

BR. and Thanks.

Kent

Austin Kottke <Austin.Kottke@tollfreeforwarding.commailto:Austin.Kottke@tollfreeforwarding.com> 於 2019年2月26日 週二 上午11:35寫道：

Ensure your configuration (config_site) has

PJMEDIA_HAS_VID_TOOLBOX_CODEC=1

Otherwise the IOS video stream might not be using VideoToolBox. Recompile.

I don’t think (I could be wrong) that vid_port.c is used when VideoToolBox is enabled.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Monday, February 25, 2019 at 6:13 PM
To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: Re: [pjsip] Handle 1280x720 remote video very slow in iOS App

Dear Austin:

Thanks for your kind reply. I do appreciate for it.
We don't find a good point at darwin_dev.m, should I modify at darwin_stream_get_frame() ?

So far the following way is what we are going to do:
1.Disable original I422 to ARGB at convert_frame() of vid_port.c, make it pass original pjmedia_frame to ios_opengl_dev.m.
2.Convert I422 to ARGB at iosgl_stream_put_frame() of ios_opengl_dev.m.
3.Not only VideoToolbox, we are also considering if it is a good way to use this:

https://developer.apple.com/documentation/accelerate/1533095-vimageconvert_420yp8_cbcr8toargb?language=objc

Any further advise is welcome.
Thanks.

Kent

Austin Kottke <Austin.Kottke@tollfreeforwarding.commailto:Austin.Kottke@tollfreeforwarding.com> 於 2019年2月26日 週二 上午2:57寫道：
Sure,

Let me know how it goes. Should be straightforward. Ive successfully done the conversion with video toolbox, however
modifying the pjsip code I have not, but Ive looked at the video toolbox darwindev impl pretty thoroughly. However, they are just wrapping the toolbox session into the hooks that the
C library needs to work with the pjsip arch. Since you are doing a decompression session make sure to log out any decode errors. The pjsip/qt/sdl camera demo works decently – I recommend checking that. It uses SDL to render which seems adequate (OSX/Win Only) I believe.

Send the sample buffers to an OpenGL ES session. There’s a couple implementations for IOS floating on the net somewhere.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Friday, February 22, 2019 at 5:40 PM
To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: Re: [pjsip] Handle 1280x720 remote video very slow in iOS App

Dear Austin:

Thanks for your reply.
I will try it. If necessary, please still support me in this issue.
Thanks.

Kent

Austin Kottke <Austin.Kottke@tollfreeforwarding.commailto:Austin.Kottke@tollfreeforwarding.com> 於 2019年2月15日 週五 上午2:29寫道：

Kent,

If you are working on IOS. You might be able to do the conversion from 420toARGB in a VideoToolbox decompression session. Its very fast. Libyuv is CPU based.

This example uses the input in 420 and converts it to 32_ARGB.

CFMutableDictionaryRef destinationPixelBufferAttributes = CFDictionaryCreateMutable(

                                                                                    NULL, // CFAllocatorRef allocator

                                                                                    0,    // CFIndex capacity

                                                                                    &kCFTypeDictionaryKeyCallBacks,

                                                                                    &kCFTypeDictionaryValueCallBacks);



SInt32 destinationPixelType = kCVPixelFormatType_32ARGB;

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferPixelFormatTypeKey, CFNumberCreate(NULL, kCFNumberSInt32Type, &destinationPixelType));

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferWidthKey,  (__bridge CFNumberRef)@(1280));

CFDictionarySetValue(destinationPixelBufferAttributes, kCVPixelBufferHeightKey, (__bridge CFNumberRef)@(720));

OSStatus status =  VTDecompressionSessionCreate(NULL, _videoFormatDescr, NULL,

                                                destinationPixelBufferAttributes,

                                                &callBackRecord, &_decompressionSession);

You might need to open up darwindev.m and make the necessary changes. I believe pjsip does have a decompression session already created using video toolbox.

Austin

From: pjsip <pjsip-bounces@lists.pjsip.orgmailto:pjsip-bounces@lists.pjsip.org> on behalf of Kent Liu <chir0615@gmail.commailto:chir0615@gmail.com>
Reply-To: pjsip list <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Date: Thursday, February 14, 2019 at 7:39 AM
To: "pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org" <pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org>
Subject: [pjsip] Handle 1280x720 remote video very slow in iOS App

Hi,

We are working on a iOS app which will connect to a remote device with 720P(1280x720) video streaming.
We follow the guide to make PJSIP accept 720P remote streaming.
But the video have serious latency even on iPhoneX, and the bottleneck seems to be the I420ToARGB convert via libyuv.
I420ToARGB is too slow to make 720P remote video workable.
Is there any way to improve the performance?

Any suggestion for this situation would be welcome.
Thanks.

BR,

Kent

Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.orgmailto:pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org