usrp-users@lists.ettus.com

Discussion and technical support related to USRP, UHD, RFNoC

View all threads

Backpressure over Ethernet

RK
Rob Kossler
Fri, Mar 24, 2023 2:39 PM

Hi,
I developed a custom RFNoC block and was testing it with a graph that was
essentially:
tx_streamer => myblock => rx_streamer

This works fine unless I send a large number of samples, in which case bad
stuff happens in terms of sequence errors, timeouts, etc (i.e., the typical
streaming at high speed woes).

In the case of a Radio that must stream output samples at a fixed rate, I
can easily understand the need for keeping up on the host side.  But, for
my case where I am just streaming through a custom block that does not care
about sample rate, it would be nice if backpressure could handle the host
rate variations.

In order to make things "work", I have inserted a "sleep" statement in my
transmit loop to purposely throttle the Tx rate so that the Rx could keep
up.  This works well enough but in order to get reliable streaming I am
forced to throttle more than I would like.  So, I am wondering if anyone
has ideas on a better way to handle this.

It seems that the host Rx gets overwhelmed and cannot backpressure my
block. I don't quite understand why because if there is no backpressure,
how do we get "Overflow" when the radio has no place to put incoming
samples "because of backpressure"?

Rob

Hi, I developed a custom RFNoC block and was testing it with a graph that was essentially: tx_streamer => myblock => rx_streamer This works fine unless I send a large number of samples, in which case bad stuff happens in terms of sequence errors, timeouts, etc (i.e., the typical streaming at high speed woes). In the case of a Radio that must stream output samples at a fixed rate, I can easily understand the need for keeping up on the host side. But, for my case where I am just streaming through a custom block that does not care about sample rate, it would be nice if backpressure could handle the host rate variations. In order to make things "work", I have inserted a "sleep" statement in my transmit loop to purposely throttle the Tx rate so that the Rx could keep up. This works well enough but in order to get reliable streaming I am forced to throttle more than I would like. So, I am wondering if anyone has ideas on a better way to handle this. It seems that the host Rx gets overwhelmed and cannot backpressure my block. I don't quite understand why because if there is no backpressure, how do we get "Overflow" when the radio has no place to put incoming samples "because of backpressure"? Rob
WF
Wade Fife
Sat, Mar 25, 2023 9:19 PM

It sounds like something isn't right. The streamers should automatically
backpressure. That's how it works with the radio, and it should be the same
for a custom block connected to stream endpoints.

We have seen cases where host ethernet interfaces can't keep up when you
start approaching the line rate of Ethernet (e.g., 10 Gbps) so they drop
packets, which show up as sequence errors on the RX streamer.

What kinds of rates are you getting across each interface? Is this over 10
GbE? Any chance the "ready" signal isn't being used properly in your block
to allow back-pressure? The block needs to deassert tready to tell the
stream endpoint to backpressure the streamer.

Wade

On Fri, Mar 24, 2023 at 9:41 AM Rob Kossler via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi,
I developed a custom RFNoC block and was testing it with a graph that was
essentially:
tx_streamer => myblock => rx_streamer

This works fine unless I send a large number of samples, in which case bad
stuff happens in terms of sequence errors, timeouts, etc (i.e., the typical
streaming at high speed woes).

In the case of a Radio that must stream output samples at a fixed rate, I
can easily understand the need for keeping up on the host side.  But, for
my case where I am just streaming through a custom block that does not care
about sample rate, it would be nice if backpressure could handle the host
rate variations.

In order to make things "work", I have inserted a "sleep" statement in my
transmit loop to purposely throttle the Tx rate so that the Rx could keep
up.  This works well enough but in order to get reliable streaming I am
forced to throttle more than I would like.  So, I am wondering if anyone
has ideas on a better way to handle this.

It seems that the host Rx gets overwhelmed and cannot backpressure my
block. I don't quite understand why because if there is no backpressure,
how do we get "Overflow" when the radio has no place to put incoming
samples "because of backpressure"?

Rob


USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com

It sounds like something isn't right. The streamers should automatically backpressure. That's how it works with the radio, and it should be the same for a custom block connected to stream endpoints. We have seen cases where host ethernet interfaces can't keep up when you start approaching the line rate of Ethernet (e.g., 10 Gbps) so they drop packets, which show up as sequence errors on the RX streamer. What kinds of rates are you getting across each interface? Is this over 10 GbE? Any chance the "ready" signal isn't being used properly in your block to allow back-pressure? The block needs to deassert tready to tell the stream endpoint to backpressure the streamer. Wade On Fri, Mar 24, 2023 at 9:41 AM Rob Kossler via USRP-users < usrp-users@lists.ettus.com> wrote: > Hi, > I developed a custom RFNoC block and was testing it with a graph that was > essentially: > tx_streamer => myblock => rx_streamer > > This works fine unless I send a large number of samples, in which case bad > stuff happens in terms of sequence errors, timeouts, etc (i.e., the typical > streaming at high speed woes). > > In the case of a Radio that must stream output samples at a fixed rate, I > can easily understand the need for keeping up on the host side. But, for > my case where I am just streaming through a custom block that does not care > about sample rate, it would be nice if backpressure could handle the host > rate variations. > > In order to make things "work", I have inserted a "sleep" statement in my > transmit loop to purposely throttle the Tx rate so that the Rx could keep > up. This works well enough but in order to get reliable streaming I am > forced to throttle more than I would like. So, I am wondering if anyone > has ideas on a better way to handle this. > > It seems that the host Rx gets overwhelmed and cannot backpressure my > block. I don't quite understand why because if there is no backpressure, > how do we get "Overflow" when the radio has no place to put incoming > samples "because of backpressure"? > > Rob > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >
RK
Rob Kossler
Mon, Mar 27, 2023 1:44 PM

Hi Wade,
I am using 10GbE.  It is likely an issue with host Ethernet handling (rx
descriptors or something like that) such that the OS ethernet handling gets
overwhelmed prior to UHD backpressuring the FPGA.  My OS is Ubuntu 22.04
LTS.

After lots of playing around, I am now getting semi-reliable behavior
running at about 7-8Gbps but it is curious that if I try to run my
application several times in a row, I often get bad behavior.  Instead, if
I run my application with a very short sample burst (say 4096 samples
total) following a typical execution with a long sample burst, this seems
to prevent bad behavior.  So, my theory is that running with the short
burst helps the host OS to clear / reset Ethernet buffers such that the
subsequent long burst can be successful.

Rob

On Sat, Mar 25, 2023 at 5:19 PM Wade Fife wade.fife@ettus.com wrote:

It sounds like something isn't right. The streamers should automatically
backpressure. That's how it works with the radio, and it should be the same
for a custom block connected to stream endpoints.

We have seen cases where host ethernet interfaces can't keep up when you
start approaching the line rate of Ethernet (e.g., 10 Gbps) so they drop
packets, which show up as sequence errors on the RX streamer.

What kinds of rates are you getting across each interface? Is this over 10
GbE? Any chance the "ready" signal isn't being used properly in your block
to allow back-pressure? The block needs to deassert tready to tell the
stream endpoint to backpressure the streamer.

Wade

On Fri, Mar 24, 2023 at 9:41 AM Rob Kossler via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi,
I developed a custom RFNoC block and was testing it with a graph that was
essentially:
tx_streamer => myblock => rx_streamer

This works fine unless I send a large number of samples, in which case
bad stuff happens in terms of sequence errors, timeouts, etc (i.e., the
typical streaming at high speed woes).

In the case of a Radio that must stream output samples at a fixed rate, I
can easily understand the need for keeping up on the host side.  But, for
my case where I am just streaming through a custom block that does not care
about sample rate, it would be nice if backpressure could handle the host
rate variations.

In order to make things "work", I have inserted a "sleep" statement in my
transmit loop to purposely throttle the Tx rate so that the Rx could keep
up.  This works well enough but in order to get reliable streaming I am
forced to throttle more than I would like.  So, I am wondering if anyone
has ideas on a better way to handle this.

It seems that the host Rx gets overwhelmed and cannot backpressure my
block. I don't quite understand why because if there is no backpressure,
how do we get "Overflow" when the radio has no place to put incoming
samples "because of backpressure"?

Rob


USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com

Hi Wade, I am using 10GbE. It is likely an issue with host Ethernet handling (rx descriptors or something like that) such that the OS ethernet handling gets overwhelmed prior to UHD backpressuring the FPGA. My OS is Ubuntu 22.04 LTS. After lots of playing around, I am now getting semi-reliable behavior running at about 7-8Gbps but it is curious that if I try to run my application several times in a row, I often get bad behavior. Instead, if I run my application with a very short sample burst (say 4096 samples total) following a typical execution with a long sample burst, this seems to prevent bad behavior. So, my theory is that running with the short burst helps the host OS to clear / reset Ethernet buffers such that the subsequent long burst can be successful. Rob On Sat, Mar 25, 2023 at 5:19 PM Wade Fife <wade.fife@ettus.com> wrote: > It sounds like something isn't right. The streamers should automatically > backpressure. That's how it works with the radio, and it should be the same > for a custom block connected to stream endpoints. > > We have seen cases where host ethernet interfaces can't keep up when you > start approaching the line rate of Ethernet (e.g., 10 Gbps) so they drop > packets, which show up as sequence errors on the RX streamer. > > What kinds of rates are you getting across each interface? Is this over 10 > GbE? Any chance the "ready" signal isn't being used properly in your block > to allow back-pressure? The block needs to deassert tready to tell the > stream endpoint to backpressure the streamer. > > Wade > > On Fri, Mar 24, 2023 at 9:41 AM Rob Kossler via USRP-users < > usrp-users@lists.ettus.com> wrote: > >> Hi, >> I developed a custom RFNoC block and was testing it with a graph that was >> essentially: >> tx_streamer => myblock => rx_streamer >> >> This works fine unless I send a large number of samples, in which case >> bad stuff happens in terms of sequence errors, timeouts, etc (i.e., the >> typical streaming at high speed woes). >> >> In the case of a Radio that must stream output samples at a fixed rate, I >> can easily understand the need for keeping up on the host side. But, for >> my case where I am just streaming through a custom block that does not care >> about sample rate, it would be nice if backpressure could handle the host >> rate variations. >> >> In order to make things "work", I have inserted a "sleep" statement in my >> transmit loop to purposely throttle the Tx rate so that the Rx could keep >> up. This works well enough but in order to get reliable streaming I am >> forced to throttle more than I would like. So, I am wondering if anyone >> has ideas on a better way to handle this. >> >> It seems that the host Rx gets overwhelmed and cannot backpressure my >> block. I don't quite understand why because if there is no backpressure, >> how do we get "Overflow" when the radio has no place to put incoming >> samples "because of backpressure"? >> >> Rob >> _______________________________________________ >> USRP-users mailing list -- usrp-users@lists.ettus.com >> To unsubscribe send an email to usrp-users-leave@lists.ettus.com >> >