patch: add reference counter to pjsip_inv_session to avoid race condition

AG
Alexei Gradinari
Mon, Aug 22, 2016 4:11 PM

Hello,

When a transport error occured on an INVITE session
the pjproject calls on_tsx_state_changed with new
state PJSIP_INV_STATE_DISCONNECTED and immediately
destroys the INVITE session.
At the same time this INVITE session could being
processed on another thread.
This thread could use the session's memory pools which
were already freed, so we get segfault.

This patch adds a reference counter and new functions:
pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref.
The INVITE session is destroyed only when the reference
counter has reached zero.

To avoid race condition an application should call
pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref.

Regards,
Alexei

Hello, When a transport error occured on an INVITE session the pjproject calls on_tsx_state_changed with new state PJSIP_INV_STATE_DISCONNECTED and immediately destroys the INVITE session. At the same time this INVITE session could being processed on another thread. This thread could use the session's memory pools which were already freed, so we get segfault. This patch adds a reference counter and new functions: pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref. The INVITE session is destroyed only when the reference counter has reached zero. To avoid race condition an application should call pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref. Regards, Alexei
M
Ming
Thu, Aug 25, 2016 3:49 AM

Hi Alexei,

Thanks for the report and the patch. Is dialog lock not sufficient for
this purpose? If not, can you please elaborate more, perhaps with a
sample scenario (and a brief stack trace, if necessary)?

FYI, we might have a similar problem in ticket #1902
(https://trac.pjsip.org/repos/ticket/1902).

Best regards,
Ming

On Tue, Aug 23, 2016 at 12:11 AM, Alexei Gradinari alex2grad@gmail.com wrote:

Hello,

When a transport error occured on an INVITE session
the pjproject calls on_tsx_state_changed with new
state PJSIP_INV_STATE_DISCONNECTED and immediately
destroys the INVITE session.
At the same time this INVITE session could being
processed on another thread.
This thread could use the session's memory pools which
were already freed, so we get segfault.

This patch adds a reference counter and new functions:
pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref.
The INVITE session is destroyed only when the reference
counter has reached zero.

To avoid race condition an application should call
pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref.

Regards,
Alexei


Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

Hi Alexei, Thanks for the report and the patch. Is dialog lock not sufficient for this purpose? If not, can you please elaborate more, perhaps with a sample scenario (and a brief stack trace, if necessary)? FYI, we might have a similar problem in ticket #1902 (https://trac.pjsip.org/repos/ticket/1902). Best regards, Ming On Tue, Aug 23, 2016 at 12:11 AM, Alexei Gradinari <alex2grad@gmail.com> wrote: > Hello, > > When a transport error occured on an INVITE session > the pjproject calls on_tsx_state_changed with new > state PJSIP_INV_STATE_DISCONNECTED and immediately > destroys the INVITE session. > At the same time this INVITE session could being > processed on another thread. > This thread could use the session's memory pools which > were already freed, so we get segfault. > > This patch adds a reference counter and new functions: > pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref. > The INVITE session is destroyed only when the reference > counter has reached zero. > > To avoid race condition an application should call > pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref. > > > Regards, > Alexei > _______________________________________________ > Visit our blog: http://blog.pjsip.org > > pjsip mailing list > pjsip@lists.pjsip.org > http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org >
AG
Alexei Gradinari
Fri, Aug 26, 2016 3:20 PM

Hello Ming,

The dialog lock is not sufficient.
Look at function inv_set_state in pjsip/src/pjsip-ua/sip_inv.c
starting with
if (inv->state == PJSIP_INV_STATE_DISCONNECTED &&
prev_state != PJSIP_INV_STATE_DISCONNECTED)
{

there isn't any dialog lock and immediately
destroying any data associated with pjsip_inv_session.

Look at function inv_on_state_incoming in pjsip/src/pjsip-ua/sip_inv.c

 case PJSIP_TSX_STATE_TERMINATED:
         /*.
         * This happens on transport error (e.g. failed to send
         * response)
         */
        inv_set_cause(inv, tsx->status_code, &tsx->status_text);
        inv_set_state(inv, PJSIP_INV_STATE_DISCONNECTED, e);

===
there isn't any dialog lock

The scenario:
Transport TLS
The incoming INVITE received, pjsip called an on_rx_request.
An application pushed the INVITE session's data to a task queue on the on_rx_request.
The application popped the task and started processing INVITE session on different thread.
The TCP/TLS connection has been disconnected (transport error),
as result PJSIP_TSX_STATE_TERMINATED.
The pjsip called inv_set_state with state PJSIP_INV_STATE_DISCONNECTED.
The pjsip called on_tsx_state_changed.
The application received on_tsx_state_changed, but on different thread then
this INVITE session was being processed.
The pjsip destroyed the INVITE session while application was still processing this session.

Attached the stack trace of the application thread which processed
the INVITE session.

Wednesday, August 24, 2016, 11:49:50 PM, you wrote:

Thanks for the report and the patch. Is dialog lock not sufficient for
this purpose? If not, can you please elaborate more, perhaps with a
sample scenario (and a brief stack trace, if necessary)?

FYI, we might have a similar problem in ticket #1902
(https://trac.pjsip.org/repos/ticket/1902).

Best regards,
Ming

On Tue, Aug 23, 2016 at 12:11 AM, Alexei Gradinari alex2grad@gmail.com wrote:

Hello,

When a transport error occured on an INVITE session
the pjproject calls on_tsx_state_changed with new
state PJSIP_INV_STATE_DISCONNECTED and immediately
destroys the INVITE session.
At the same time this INVITE session could being
processed on another thread.
This thread could use the session's memory pools which
were already freed, so we get segfault.

This patch adds a reference counter and new functions:
pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref.
The INVITE session is destroyed only when the reference
counter has reached zero.

To avoid race condition an application should call
pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref.

Hello Ming, The dialog lock is not sufficient. Look at function inv_set_state in pjsip/src/pjsip-ua/sip_inv.c starting with if (inv->state == PJSIP_INV_STATE_DISCONNECTED && prev_state != PJSIP_INV_STATE_DISCONNECTED) { there isn't any dialog lock and immediately destroying any data associated with pjsip_inv_session. Look at function inv_on_state_incoming in pjsip/src/pjsip-ua/sip_inv.c === case PJSIP_TSX_STATE_TERMINATED: /*. * This happens on transport error (e.g. failed to send * response) */ inv_set_cause(inv, tsx->status_code, &tsx->status_text); inv_set_state(inv, PJSIP_INV_STATE_DISCONNECTED, e); === there isn't any dialog lock The scenario: Transport TLS The incoming INVITE received, pjsip called an on_rx_request. An application pushed the INVITE session's data to a task queue on the on_rx_request. The application popped the task and started processing INVITE session on different thread. The TCP/TLS connection has been disconnected (transport error), as result PJSIP_TSX_STATE_TERMINATED. The pjsip called inv_set_state with state PJSIP_INV_STATE_DISCONNECTED. The pjsip called on_tsx_state_changed. The application received on_tsx_state_changed, but on different thread then this INVITE session was being processed. The pjsip destroyed the INVITE session while application was still processing this session. Attached the stack trace of the application thread which processed the INVITE session. Wednesday, August 24, 2016, 11:49:50 PM, you wrote: > Thanks for the report and the patch. Is dialog lock not sufficient for > this purpose? If not, can you please elaborate more, perhaps with a > sample scenario (and a brief stack trace, if necessary)? > FYI, we might have a similar problem in ticket #1902 > (https://trac.pjsip.org/repos/ticket/1902). > Best regards, > Ming > On Tue, Aug 23, 2016 at 12:11 AM, Alexei Gradinari <alex2grad@gmail.com> wrote: >> Hello, >> >> When a transport error occured on an INVITE session >> the pjproject calls on_tsx_state_changed with new >> state PJSIP_INV_STATE_DISCONNECTED and immediately >> destroys the INVITE session. >> At the same time this INVITE session could being >> processed on another thread. >> This thread could use the session's memory pools which >> were already freed, so we get segfault. >> >> This patch adds a reference counter and new functions: >> pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref. >> The INVITE session is destroyed only when the reference >> counter has reached zero. >> >> To avoid race condition an application should call >> pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref.
AG
Alexei Gradinari
Fri, Aug 26, 2016 9:48 PM

Hello Ming,

Attached the 2nd version of this patch.

Regards,
Alexei

Wednesday, August 24, 2016, 11:49:50 PM, you wrote:

Hi Alexei,

Thanks for the report and the patch. Is dialog lock not sufficient for
this purpose? If not, can you please elaborate more, perhaps with a
sample scenario (and a brief stack trace, if necessary)?

FYI, we might have a similar problem in ticket #1902
(https://trac.pjsip.org/repos/ticket/1902).

Best regards,
Ming

On Tue, Aug 23, 2016 at 12:11 AM, Alexei Gradinari alex2grad@gmail.com wrote:

Hello,

When a transport error occured on an INVITE session
the pjproject calls on_tsx_state_changed with new
state PJSIP_INV_STATE_DISCONNECTED and immediately
destroys the INVITE session.
At the same time this INVITE session could being
processed on another thread.
This thread could use the session's memory pools which
were already freed, so we get segfault.

This patch adds a reference counter and new functions:
pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref.
The INVITE session is destroyed only when the reference
counter has reached zero.

To avoid race condition an application should call
pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref.

Regards,
Alexei


Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org


Visit our blog: http://blog.pjsip.org

--
Best regards,
Alexei                            mailto:alex2grad@gmail.com

Hello Ming, Attached the 2nd version of this patch. Regards, Alexei Wednesday, August 24, 2016, 11:49:50 PM, you wrote: > Hi Alexei, > Thanks for the report and the patch. Is dialog lock not sufficient for > this purpose? If not, can you please elaborate more, perhaps with a > sample scenario (and a brief stack trace, if necessary)? > FYI, we might have a similar problem in ticket #1902 > (https://trac.pjsip.org/repos/ticket/1902). > Best regards, > Ming > On Tue, Aug 23, 2016 at 12:11 AM, Alexei Gradinari <alex2grad@gmail.com> wrote: >> Hello, >> >> When a transport error occured on an INVITE session >> the pjproject calls on_tsx_state_changed with new >> state PJSIP_INV_STATE_DISCONNECTED and immediately >> destroys the INVITE session. >> At the same time this INVITE session could being >> processed on another thread. >> This thread could use the session's memory pools which >> were already freed, so we get segfault. >> >> This patch adds a reference counter and new functions: >> pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref. >> The INVITE session is destroyed only when the reference >> counter has reached zero. >> >> To avoid race condition an application should call >> pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref. >> >> >> Regards, >> Alexei >> _______________________________________________ >> Visit our blog: http://blog.pjsip.org >> >> pjsip mailing list >> pjsip@lists.pjsip.org >> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org >> > _______________________________________________ > Visit our blog: http://blog.pjsip.org > pjsip mailing list > pjsip@lists.pjsip.org > http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org -- Best regards, Alexei mailto:alex2grad@gmail.com
RS
Riza Sulistyo
Tue, Aug 30, 2016 8:52 AM

Hi Alexei,

We have committed the patch with slight modification on #1959
https://trac.pjsip.org/repos/ticket/1959.
Thanks again for the patch.

Best Regards,

Riza

On Sat, Aug 27, 2016 at 4:48 AM, Alexei Gradinari alex2grad@gmail.com
wrote:

Hello Ming,

Attached the 2nd version of this patch.

Regards,
Alexei

Wednesday, August 24, 2016, 11:49:50 PM, you wrote:

Hi Alexei,

Thanks for the report and the patch. Is dialog lock not sufficient for
this purpose? If not, can you please elaborate more, perhaps with a
sample scenario (and a brief stack trace, if necessary)?

FYI, we might have a similar problem in ticket #1902
(https://trac.pjsip.org/repos/ticket/1902).

Best regards,
Ming

On Tue, Aug 23, 2016 at 12:11 AM, Alexei Gradinari alex2grad@gmail.com

wrote:

Hello,

When a transport error occured on an INVITE session
the pjproject calls on_tsx_state_changed with new
state PJSIP_INV_STATE_DISCONNECTED and immediately
destroys the INVITE session.
At the same time this INVITE session could being
processed on another thread.
This thread could use the session's memory pools which
were already freed, so we get segfault.

This patch adds a reference counter and new functions:
pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref.
The INVITE session is destroyed only when the reference
counter has reached zero.

To avoid race condition an application should call
pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref.

Regards,
Alexei


Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org


Visit our blog: http://blog.pjsip.org

--
Best regards,
Alexei                            mailto:alex2grad@gmail.com


Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

Hi Alexei, We have committed the patch with slight modification on #1959 <https://trac.pjsip.org/repos/ticket/1959>. Thanks again for the patch. Best Regards, Riza On Sat, Aug 27, 2016 at 4:48 AM, Alexei Gradinari <alex2grad@gmail.com> wrote: > Hello Ming, > > Attached the 2nd version of this patch. > > Regards, > Alexei > > Wednesday, August 24, 2016, 11:49:50 PM, you wrote: > > Hi Alexei, > > > Thanks for the report and the patch. Is dialog lock not sufficient for > > this purpose? If not, can you please elaborate more, perhaps with a > > sample scenario (and a brief stack trace, if necessary)? > > > FYI, we might have a similar problem in ticket #1902 > > (https://trac.pjsip.org/repos/ticket/1902). > > > Best regards, > > Ming > > > On Tue, Aug 23, 2016 at 12:11 AM, Alexei Gradinari <alex2grad@gmail.com> > wrote: > >> Hello, > >> > >> When a transport error occured on an INVITE session > >> the pjproject calls on_tsx_state_changed with new > >> state PJSIP_INV_STATE_DISCONNECTED and immediately > >> destroys the INVITE session. > >> At the same time this INVITE session could being > >> processed on another thread. > >> This thread could use the session's memory pools which > >> were already freed, so we get segfault. > >> > >> This patch adds a reference counter and new functions: > >> pjsip_inv_session_add_ref and pjsip_inv_session_dec_ref. > >> The INVITE session is destroyed only when the reference > >> counter has reached zero. > >> > >> To avoid race condition an application should call > >> pjsip_inv_session_add_ref/pjsip_inv_session_dec_ref. > >> > >> > >> Regards, > >> Alexei > >> _______________________________________________ > >> Visit our blog: http://blog.pjsip.org > >> > >> pjsip mailing list > >> pjsip@lists.pjsip.org > >> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org > >> > > > _______________________________________________ > > Visit our blog: http://blog.pjsip.org > > > pjsip mailing list > > pjsip@lists.pjsip.org > > http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org > > > > -- > Best regards, > Alexei mailto:alex2grad@gmail.com > _______________________________________________ > Visit our blog: http://blog.pjsip.org > > pjsip mailing list > pjsip@lists.pjsip.org > http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org > >