usrp-users@lists.ettus.com

Discussion and technical support related to USRP, UHD, RFNoC

View all threads

Testing single RFNoC block produces out-of-sequence (S) errors

RK
Rob Kossler
Tue, Jan 5, 2021 11:07 PM

Hi,
I created a custom pulse-detector block which attempts to pass/discard
samples based upon an instantaneous power estimate. Each set of
consecutively passed samples constitutes a "burst" (i.e. the EOB is set on
the TLAST corresponding to the final passed sample of the burst).  A burst
can have as few as 1 sample because my block does not prevent this.

The block does what I expect in the testbench and even in the FPGA binary
when using the same data as the testbench.  However, when using a long
input stream (~250MSamps from previous recording that I have handy) to test
this block running in the FPGA, I get out-of-sequence errors on receive. My
graph is tx_streamer => pulse_detector => rx_streamer (actually, my
pulse_detector is a 2x2 block so there are 2 tx_streamers and 2
rx_streamers, all of which are run in separate threads).

I am wondering if the Sequence errors have nothing to do with my block but
rather are related to the way Ubuntu handles incoming Ethernet data such
that perhaps it is running out of RX descriptors. I have tried slowing down
my Transmit and seen some improvement but without eliminating the errors
completely. So, I'm wondering what suggestions others might have.  My ideas
are:

  • invest some time to get DPDK working.  With this, perhaps Sequence errors
    cannot occur??
  • try slowing down my Tx even further?
  • create artificial data sets (rather than real collected data) such that I
    can control the length of the output pulses

Let me know if you have any suggestions.
Rob

Hi, I created a custom pulse-detector block which attempts to pass/discard samples based upon an instantaneous power estimate. Each set of consecutively passed samples constitutes a "burst" (i.e. the EOB is set on the TLAST corresponding to the final passed sample of the burst). A burst can have as few as 1 sample because my block does not prevent this. The block does what I expect in the testbench and even in the FPGA binary when using the same data as the testbench. However, when using a long input stream (~250MSamps from previous recording that I have handy) to test this block running in the FPGA, I get out-of-sequence errors on receive. My graph is tx_streamer => pulse_detector => rx_streamer (actually, my pulse_detector is a 2x2 block so there are 2 tx_streamers and 2 rx_streamers, all of which are run in separate threads). I am wondering if the Sequence errors have nothing to do with my block but rather are related to the way Ubuntu handles incoming Ethernet data such that perhaps it is running out of RX descriptors. I have tried slowing down my Transmit and seen some improvement but without eliminating the errors completely. So, I'm wondering what suggestions others might have. My ideas are: * invest some time to get DPDK working. With this, perhaps Sequence errors cannot occur?? * try slowing down my Tx even further? * create artificial data sets (rather than real collected data) such that I can control the length of the output pulses Let me know if you have any suggestions. Rob
JP
Jonathon Pendlum
Tue, Jan 12, 2021 2:49 AM

Hi Rob,

If you run netstat -s before and after running your flowgraph, do you see
"receive buffer errors" in the UDP section increasing? I imagine you've
tried this but just in case: have you increased the ring buffer size with
"sudo ethtool -G <interface> tx 4096 rx 4096"?

Jonathon

On Tue, Jan 5, 2021 at 6:08 PM Rob Kossler via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi,
I created a custom pulse-detector block which attempts to pass/discard
samples based upon an instantaneous power estimate. Each set of
consecutively passed samples constitutes a "burst" (i.e. the EOB is set on
the TLAST corresponding to the final passed sample of the burst).  A burst
can have as few as 1 sample because my block does not prevent this.

The block does what I expect in the testbench and even in the FPGA binary
when using the same data as the testbench.  However, when using a long
input stream (~250MSamps from previous recording that I have handy) to test
this block running in the FPGA, I get out-of-sequence errors on receive. My
graph is tx_streamer => pulse_detector => rx_streamer (actually, my
pulse_detector is a 2x2 block so there are 2 tx_streamers and 2
rx_streamers, all of which are run in separate threads).

I am wondering if the Sequence errors have nothing to do with my block but
rather are related to the way Ubuntu handles incoming Ethernet data such
that perhaps it is running out of RX descriptors. I have tried slowing down
my Transmit and seen some improvement but without eliminating the errors
completely. So, I'm wondering what suggestions others might have.  My ideas
are:

  • invest some time to get DPDK working.  With this, perhaps Sequence
    errors cannot occur??
  • try slowing down my Tx even further?
  • create artificial data sets (rather than real collected data) such that
    I can control the length of the output pulses

Let me know if you have any suggestions.
Rob


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hi Rob, If you run netstat -s before and after running your flowgraph, do you see "receive buffer errors" in the UDP section increasing? I imagine you've tried this but just in case: have you increased the ring buffer size with "sudo ethtool -G <interface> tx 4096 rx 4096"? Jonathon On Tue, Jan 5, 2021 at 6:08 PM Rob Kossler via USRP-users < usrp-users@lists.ettus.com> wrote: > Hi, > I created a custom pulse-detector block which attempts to pass/discard > samples based upon an instantaneous power estimate. Each set of > consecutively passed samples constitutes a "burst" (i.e. the EOB is set on > the TLAST corresponding to the final passed sample of the burst). A burst > can have as few as 1 sample because my block does not prevent this. > > The block does what I expect in the testbench and even in the FPGA binary > when using the same data as the testbench. However, when using a long > input stream (~250MSamps from previous recording that I have handy) to test > this block running in the FPGA, I get out-of-sequence errors on receive. My > graph is tx_streamer => pulse_detector => rx_streamer (actually, my > pulse_detector is a 2x2 block so there are 2 tx_streamers and 2 > rx_streamers, all of which are run in separate threads). > > I am wondering if the Sequence errors have nothing to do with my block but > rather are related to the way Ubuntu handles incoming Ethernet data such > that perhaps it is running out of RX descriptors. I have tried slowing down > my Transmit and seen some improvement but without eliminating the errors > completely. So, I'm wondering what suggestions others might have. My ideas > are: > * invest some time to get DPDK working. With this, perhaps Sequence > errors cannot occur?? > * try slowing down my Tx even further? > * create artificial data sets (rather than real collected data) such that > I can control the length of the output pulses > > Let me know if you have any suggestions. > Rob > _______________________________________________ > USRP-users mailing list > USRP-users@lists.ettus.com > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >
RK
Rob Kossler
Wed, Jan 13, 2021 3:12 AM

Hi Jonathon,
I had not tried the "netstat -s" but after your suggestion I did and the
"receive buffer errors" does increase after running my application.  I had
previously increased the ring buffer size to  the max 4096.
Rob

On Mon, Jan 11, 2021 at 9:50 PM Jonathon Pendlum jonathon.pendlum@ettus.com
wrote:

Hi Rob,

If you run netstat -s before and after running your flowgraph, do you see
"receive buffer errors" in the UDP section increasing? I imagine you've
tried this but just in case: have you increased the ring buffer size with
"sudo ethtool -G <interface> tx 4096 rx 4096"?

Jonathon

On Tue, Jan 5, 2021 at 6:08 PM Rob Kossler via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi,
I created a custom pulse-detector block which attempts to pass/discard
samples based upon an instantaneous power estimate. Each set of
consecutively passed samples constitutes a "burst" (i.e. the EOB is set on
the TLAST corresponding to the final passed sample of the burst).  A burst
can have as few as 1 sample because my block does not prevent this.

The block does what I expect in the testbench and even in the FPGA binary
when using the same data as the testbench.  However, when using a long
input stream (~250MSamps from previous recording that I have handy) to test
this block running in the FPGA, I get out-of-sequence errors on receive. My
graph is tx_streamer => pulse_detector => rx_streamer (actually, my
pulse_detector is a 2x2 block so there are 2 tx_streamers and 2
rx_streamers, all of which are run in separate threads).

I am wondering if the Sequence errors have nothing to do with my block
but rather are related to the way Ubuntu handles incoming Ethernet data
such that perhaps it is running out of RX descriptors. I have tried slowing
down my Transmit and seen some improvement but without eliminating the
errors completely. So, I'm wondering what suggestions others might have.
My ideas are:

  • invest some time to get DPDK working.  With this, perhaps Sequence
    errors cannot occur??
  • try slowing down my Tx even further?
  • create artificial data sets (rather than real collected data) such that
    I can control the length of the output pulses

Let me know if you have any suggestions.
Rob


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hi Jonathon, I had not tried the "netstat -s" but after your suggestion I did and the "receive buffer errors" does increase after running my application. I had previously increased the ring buffer size to the max 4096. Rob On Mon, Jan 11, 2021 at 9:50 PM Jonathon Pendlum <jonathon.pendlum@ettus.com> wrote: > Hi Rob, > > If you run netstat -s before and after running your flowgraph, do you see > "receive buffer errors" in the UDP section increasing? I imagine you've > tried this but just in case: have you increased the ring buffer size with > "sudo ethtool -G <interface> tx 4096 rx 4096"? > > Jonathon > > On Tue, Jan 5, 2021 at 6:08 PM Rob Kossler via USRP-users < > usrp-users@lists.ettus.com> wrote: > >> Hi, >> I created a custom pulse-detector block which attempts to pass/discard >> samples based upon an instantaneous power estimate. Each set of >> consecutively passed samples constitutes a "burst" (i.e. the EOB is set on >> the TLAST corresponding to the final passed sample of the burst). A burst >> can have as few as 1 sample because my block does not prevent this. >> >> The block does what I expect in the testbench and even in the FPGA binary >> when using the same data as the testbench. However, when using a long >> input stream (~250MSamps from previous recording that I have handy) to test >> this block running in the FPGA, I get out-of-sequence errors on receive. My >> graph is tx_streamer => pulse_detector => rx_streamer (actually, my >> pulse_detector is a 2x2 block so there are 2 tx_streamers and 2 >> rx_streamers, all of which are run in separate threads). >> >> I am wondering if the Sequence errors have nothing to do with my block >> but rather are related to the way Ubuntu handles incoming Ethernet data >> such that perhaps it is running out of RX descriptors. I have tried slowing >> down my Transmit and seen some improvement but without eliminating the >> errors completely. So, I'm wondering what suggestions others might have. >> My ideas are: >> * invest some time to get DPDK working. With this, perhaps Sequence >> errors cannot occur?? >> * try slowing down my Tx even further? >> * create artificial data sets (rather than real collected data) such that >> I can control the length of the output pulses >> >> Let me know if you have any suggestions. >> Rob >> _______________________________________________ >> USRP-users mailing list >> USRP-users@lists.ettus.com >> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >> >
JP
Jonathon Pendlum
Wed, Jan 13, 2021 4:30 AM

Hey Rob,

Since you are definitely dropping packets, I think trying DPDK is a good
next step.

Jonathon

On Tue, Jan 12, 2021 at 10:12 PM Rob Kossler rkossler@nd.edu wrote:

Hi Jonathon,
I had not tried the "netstat -s" but after your suggestion I did and the
"receive buffer errors" does increase after running my application.  I had
previously increased the ring buffer size to  the max 4096.
Rob

On Mon, Jan 11, 2021 at 9:50 PM Jonathon Pendlum <
jonathon.pendlum@ettus.com> wrote:

Hi Rob,

If you run netstat -s before and after running your flowgraph, do you see
"receive buffer errors" in the UDP section increasing? I imagine you've
tried this but just in case: have you increased the ring buffer size with
"sudo ethtool -G <interface> tx 4096 rx 4096"?

Jonathon

On Tue, Jan 5, 2021 at 6:08 PM Rob Kossler via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi,
I created a custom pulse-detector block which attempts to pass/discard
samples based upon an instantaneous power estimate. Each set of
consecutively passed samples constitutes a "burst" (i.e. the EOB is set on
the TLAST corresponding to the final passed sample of the burst).  A burst
can have as few as 1 sample because my block does not prevent this.

The block does what I expect in the testbench and even in the FPGA
binary when using the same data as the testbench.  However, when using a
long input stream (~250MSamps from previous recording that I have handy) to
test this block running in the FPGA, I get out-of-sequence errors on
receive. My graph is tx_streamer => pulse_detector => rx_streamer
(actually, my pulse_detector is a 2x2 block so there are 2 tx_streamers and
2 rx_streamers, all of which are run in separate threads).

I am wondering if the Sequence errors have nothing to do with my block
but rather are related to the way Ubuntu handles incoming Ethernet data
such that perhaps it is running out of RX descriptors. I have tried slowing
down my Transmit and seen some improvement but without eliminating the
errors completely. So, I'm wondering what suggestions others might have.
My ideas are:

  • invest some time to get DPDK working.  With this, perhaps Sequence
    errors cannot occur??
  • try slowing down my Tx even further?
  • create artificial data sets (rather than real collected data) such
    that I can control the length of the output pulses

Let me know if you have any suggestions.
Rob


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hey Rob, Since you are definitely dropping packets, I think trying DPDK is a good next step. Jonathon On Tue, Jan 12, 2021 at 10:12 PM Rob Kossler <rkossler@nd.edu> wrote: > Hi Jonathon, > I had not tried the "netstat -s" but after your suggestion I did and the > "receive buffer errors" does increase after running my application. I had > previously increased the ring buffer size to the max 4096. > Rob > > On Mon, Jan 11, 2021 at 9:50 PM Jonathon Pendlum < > jonathon.pendlum@ettus.com> wrote: > >> Hi Rob, >> >> If you run netstat -s before and after running your flowgraph, do you see >> "receive buffer errors" in the UDP section increasing? I imagine you've >> tried this but just in case: have you increased the ring buffer size with >> "sudo ethtool -G <interface> tx 4096 rx 4096"? >> >> Jonathon >> >> On Tue, Jan 5, 2021 at 6:08 PM Rob Kossler via USRP-users < >> usrp-users@lists.ettus.com> wrote: >> >>> Hi, >>> I created a custom pulse-detector block which attempts to pass/discard >>> samples based upon an instantaneous power estimate. Each set of >>> consecutively passed samples constitutes a "burst" (i.e. the EOB is set on >>> the TLAST corresponding to the final passed sample of the burst). A burst >>> can have as few as 1 sample because my block does not prevent this. >>> >>> The block does what I expect in the testbench and even in the FPGA >>> binary when using the same data as the testbench. However, when using a >>> long input stream (~250MSamps from previous recording that I have handy) to >>> test this block running in the FPGA, I get out-of-sequence errors on >>> receive. My graph is tx_streamer => pulse_detector => rx_streamer >>> (actually, my pulse_detector is a 2x2 block so there are 2 tx_streamers and >>> 2 rx_streamers, all of which are run in separate threads). >>> >>> I am wondering if the Sequence errors have nothing to do with my block >>> but rather are related to the way Ubuntu handles incoming Ethernet data >>> such that perhaps it is running out of RX descriptors. I have tried slowing >>> down my Transmit and seen some improvement but without eliminating the >>> errors completely. So, I'm wondering what suggestions others might have. >>> My ideas are: >>> * invest some time to get DPDK working. With this, perhaps Sequence >>> errors cannot occur?? >>> * try slowing down my Tx even further? >>> * create artificial data sets (rather than real collected data) such >>> that I can control the length of the output pulses >>> >>> Let me know if you have any suggestions. >>> Rob >>> _______________________________________________ >>> USRP-users mailing list >>> USRP-users@lists.ettus.com >>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>> >>