usrp-users@lists.ettus.com

Discussion and technical support related to USRP, UHD, RFNoC

View all threads

remote streaming starts, but stops after a few packets received

KW
Kevin Williams
Tue, Jul 29, 2025 3:26 PM

This is the issue – my block processes inputs 1:1 without requiring to give back-pressure.

It decimates, sure, but that is 1:N.

I don’t understand how that radio block can get its fifo nearly full?

From: Brian Padalino bpadalino@gmail.com
Sent: Tuesday, 29 July 2025 17:12
To: Kevin Williams kevin.williams@vastech.co.za
Cc: nbalkanas@gmail.com; usrp-users@lists.ettus.com; Werner Bode werner.bode@vastech.co.za
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

On Tue, Jul 29, 2025 at 11:06 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

This is strange.

I now assert tready permanently for the radio regardless. (I before used to pass downstream tready up).

Now I still see the stream stalling, but after a few 10’000 packets of both usrp ports being used for tx (now 4x streams being transmitted).

It then drops to only one port, and then stalls – even though this is remote streaming.

How is it possible that the host causes the usrp to abandon a nic in remote streaming mode when the radio is always seeing tready?

Just before the stall I still see the usrp writing an overflow packet.

How is this possible?

It's because of the radio RX FSM here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/radio_rx_core.v#L565 https://url.za.m.mimecastprotect.com/s/yS8OCg5KEEcwEvmhNf8i42SUD?domain=github.com

When an error condition occurs, it reports the OVERFLOW then it waits for another stream command to start up again.

For what it's worth, I don't think you should blindly give it TREADY asserted when things can't process things downstream. The block should be designed such that you know you can consume all the samples in real time.

Brian

This is the issue – my block processes inputs 1:1 without requiring to give back-pressure. It decimates, sure, but that is 1:N. I don’t understand how that radio block can get its fifo nearly full? From: Brian Padalino <bpadalino@gmail.com> Sent: Tuesday, 29 July 2025 17:12 To: Kevin Williams <kevin.williams@vastech.co.za> Cc: nbalkanas@gmail.com; usrp-users@lists.ettus.com; Werner Bode <werner.bode@vastech.co.za> Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received On Tue, Jul 29, 2025 at 11:06 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: This is strange. I now assert tready permanently for the radio regardless. (I before used to pass downstream tready up). Now I still see the stream stalling, but after a few 10’000 packets of both usrp ports being used for tx (now 4x streams being transmitted). It then drops to only one port, and then stalls – even though this is remote streaming. How is it possible that the host causes the usrp to abandon a nic in remote streaming mode when the radio is always seeing tready? Just before the stall I still see the usrp writing an overflow packet. How is this possible? It's because of the radio RX FSM here: https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/radio_rx_core.v#L565 <https://url.za.m.mimecastprotect.com/s/yS8OCg5KEEcwEvmhNf8i42SUD?domain=github.com> When an error condition occurs, it reports the OVERFLOW then it waits for another stream command to start up again. For what it's worth, I don't think you should blindly give it TREADY asserted when things can't process things downstream. The block should be designed such that you know you can consume all the samples in real time. Brian
BP
Brian Padalino
Tue, Jul 29, 2025 6:11 PM

On Tue, Jul 29, 2025 at 11:26 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:

This is the issue – my block processes inputs 1:1 without requiring to
give back-pressure.

It decimates, sure, but that is 1:N.

I don’t understand how that radio block can get its fifo nearly full?

I'd recommend approaching this in 2 ways in parallel:

  • Build an FPGA image with an ILA that has all the ready/valid signals
    for each stage of your pipeline and trigger when ready is low for multiple
    clocks or when the overflow occurs and look back at what caused the backup
  • Simulate the full block design and randomly deassert the CHDR ready
    signal until you see a similar issue even if you know the cadence can't
    keep up with your data

I really prefer the simulation method over everything since you see all the
signals, but sometimes simulations can take forever to run and rebuilding
with an ILA can give you a little faster insight.

Good luck.

Brian

On Tue, Jul 29, 2025 at 11:26 AM Kevin Williams < kevin.williams@vastech.co.za> wrote: > This is the issue – my block processes inputs 1:1 without requiring to > give back-pressure. > > > > It decimates, sure, but that is 1:N. > > > > I don’t understand how that radio block can get its fifo nearly full? > I'd recommend approaching this in 2 ways in parallel: - Build an FPGA image with an ILA that has all the ready/valid signals for each stage of your pipeline and trigger when ready is low for multiple clocks or when the overflow occurs and look back at what caused the backup - Simulate the full block design and randomly deassert the CHDR ready signal until you see a similar issue even if you know the cadence can't keep up with your data I really prefer the simulation method over everything since you see all the signals, but sometimes simulations can take forever to run and rebuilding with an ILA can give you a little faster insight. Good luck. Brian
MB
Martin Braun
Wed, Jul 30, 2025 10:05 AM

It's a headscratcher, for sure. Something is holding up the radio. I'm
very sure that if you connect just the radio, or the radio+DDC to an SEP
and do remote streaming, it won't choke. But if it does, then I'd like to
know so we can look into it.

--M

On Tue, Jul 29, 2025 at 5:27 PM Kevin Williams kevin.williams@vastech.co.za
wrote:

This is the issue – my block processes inputs 1:1 without requiring to
give back-pressure.

It decimates, sure, but that is 1:N.

I don’t understand how that radio block can get its fifo nearly full?

From: Brian Padalino bpadalino@gmail.com
Sent: Tuesday, 29 July 2025 17:12
To: Kevin Williams kevin.williams@vastech.co.za
Cc: nbalkanas@gmail.com; usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received

On Tue, Jul 29, 2025 at 11:06 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:

This is strange.

I now assert tready permanently for the radio regardless. (I before used
to pass downstream tready up).

Now I still see the stream stalling, but after a few 10’000 packets of
both usrp ports being used for tx (now 4x streams being transmitted).

It then drops to only one port, and then stalls – even though this is
remote streaming.

How is it possible that the host causes the usrp to abandon a nic in
remote streaming mode when the radio is always seeing tready?

Just before the stall I still see the usrp writing an overflow packet.

How is this possible?

It's because of the radio RX FSM here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/radio_rx_core.v#L565
https://url.za.m.mimecastprotect.com/s/yS8OCg5KEEcwEvmhNf8i42SUD?domain=github.com

When an error condition occurs, it reports the OVERFLOW then it waits for
another stream command to start up again.

For what it's worth, I don't think you should blindly give it TREADY
asserted when things can't process things downstream. The block should be
designed such that you know you can consume all the samples in real time.

Brian


USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com

It's a headscratcher, for sure. *Something* is holding up the radio. I'm very sure that if you connect just the radio, or the radio+DDC to an SEP and do remote streaming, it won't choke. But if it does, then I'd like to know so we can look into it. --M On Tue, Jul 29, 2025 at 5:27 PM Kevin Williams <kevin.williams@vastech.co.za> wrote: > This is the issue – my block processes inputs 1:1 without requiring to > give back-pressure. > > > > It decimates, sure, but that is 1:N. > > > > I don’t understand how that radio block can get its fifo nearly full? > > > > *From:* Brian Padalino <bpadalino@gmail.com> > *Sent:* Tuesday, 29 July 2025 17:12 > *To:* Kevin Williams <kevin.williams@vastech.co.za> > *Cc:* nbalkanas@gmail.com; usrp-users@lists.ettus.com; Werner Bode < > werner.bode@vastech.co.za> > *Subject:* Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, > but stops after a few packets received > > > > On Tue, Jul 29, 2025 at 11:06 AM Kevin Williams < > kevin.williams@vastech.co.za> wrote: > > This is strange. > > > > I now assert tready permanently for the radio regardless. (I before used > to pass downstream tready up). > > > > Now I still see the stream stalling, but after a few 10’000 packets of > both usrp ports being used for tx (now 4x streams being transmitted). > > > > It then drops to only one port, and then stalls – even though this is > remote streaming. > > > > How is it possible that the host causes the usrp to abandon a nic in > remote streaming mode when the radio is always seeing tready? > > > > Just before the stall I still see the usrp writing an overflow packet. > > > > > > How is this possible? > > > > > > > > > > It's because of the radio RX FSM here: > > > > > https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/radio_rx_core.v#L565 > <https://url.za.m.mimecastprotect.com/s/yS8OCg5KEEcwEvmhNf8i42SUD?domain=github.com> > > > > When an error condition occurs, it reports the OVERFLOW then it waits for > another stream command to start up again. > > > > For what it's worth, I don't think you should blindly give it TREADY > asserted when things can't process things downstream. The block should be > designed such that you know you can consume all the samples in real time. > > > > Brian > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >
KW
Kevin Williams
Wed, Jul 30, 2025 10:25 AM

I added the Ettus DDC in-front of the splitter, so its radio->ddc->4 way splitter-> 4x my blocks, and it looks like things are working nicely if I set the early decimator to 2:1.

So it looks like I currently can get 4x 100 MHz UDP streams to my host. :)

I haven’t checked sequence numbers, but there are no indications of the overflow anymore.

Now for 8, albeit slower.

From: Martin Braun martin.braun@ettus.com
Sent: Wednesday, 30 July 2025 12:06
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

It's a headscratcher, for sure. Something is holding up the radio. I'm very sure that if you connect just the radio, or the radio+DDC to an SEP and do remote streaming, it won't choke. But if it does, then I'd like to know so we can look into it.

--M

On Tue, Jul 29, 2025 at 5:27 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

This is the issue – my block processes inputs 1:1 without requiring to give back-pressure.

It decimates, sure, but that is 1:N.

I don’t understand how that radio block can get its fifo nearly full?

From: Brian Padalino <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Sent: Tuesday, 29 July 2025 17:12
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: nbalkanas@gmail.com mailto:nbalkanas@gmail.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com ; Werner Bode <werner.bode@vastech.co.za mailto:werner.bode@vastech.co.za >
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

On Tue, Jul 29, 2025 at 11:06 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

This is strange.

I now assert tready permanently for the radio regardless. (I before used to pass downstream tready up).

Now I still see the stream stalling, but after a few 10’000 packets of both usrp ports being used for tx (now 4x streams being transmitted).

It then drops to only one port, and then stalls – even though this is remote streaming.

How is it possible that the host causes the usrp to abandon a nic in remote streaming mode when the radio is always seeing tready?

Just before the stall I still see the usrp writing an overflow packet.

How is this possible?

It's because of the radio RX FSM here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/radio_rx_core.v#L565 https://url.za.m.mimecastprotect.com/s/5lsaCj2MKKfGoWghWf9imjqds?domain=github.com

When an error condition occurs, it reports the OVERFLOW then it waits for another stream command to start up again.

For what it's worth, I don't think you should blindly give it TREADY asserted when things can't process things downstream. The block should be designed such that you know you can consume all the samples in real time.

Brian


USRP-users mailing list -- usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com mailto:usrp-users-leave@lists.ettus.com

I added the Ettus DDC in-front of the splitter, so its radio->ddc->4 way splitter-> 4x my blocks, and it looks like things are working nicely if I set the early decimator to 2:1. So it looks like I currently can get 4x 100 MHz UDP streams to my host. :) I haven’t checked sequence numbers, but there are no indications of the overflow anymore. Now for 8, albeit slower. From: Martin Braun <martin.braun@ettus.com> Sent: Wednesday, 30 July 2025 12:06 Cc: usrp-users@lists.ettus.com Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received It's a headscratcher, for sure. *Something* is holding up the radio. I'm very sure that if you connect just the radio, or the radio+DDC to an SEP and do remote streaming, it won't choke. But if it does, then I'd like to know so we can look into it. --M On Tue, Jul 29, 2025 at 5:27 PM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: This is the issue – my block processes inputs 1:1 without requiring to give back-pressure. It decimates, sure, but that is 1:N. I don’t understand how that radio block can get its fifo nearly full? From: Brian Padalino <bpadalino@gmail.com <mailto:bpadalino@gmail.com> > Sent: Tuesday, 29 July 2025 17:12 To: Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > Cc: nbalkanas@gmail.com <mailto:nbalkanas@gmail.com> ; usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> ; Werner Bode <werner.bode@vastech.co.za <mailto:werner.bode@vastech.co.za> > Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received On Tue, Jul 29, 2025 at 11:06 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: This is strange. I now assert tready permanently for the radio regardless. (I before used to pass downstream tready up). Now I still see the stream stalling, but after a few 10’000 packets of both usrp ports being used for tx (now 4x streams being transmitted). It then drops to only one port, and then stalls – even though this is remote streaming. How is it possible that the host causes the usrp to abandon a nic in remote streaming mode when the radio is always seeing tready? Just before the stall I still see the usrp writing an overflow packet. How is this possible? It's because of the radio RX FSM here: https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/blocks/rfnoc_block_radio/radio_rx_core.v#L565 <https://url.za.m.mimecastprotect.com/s/5lsaCj2MKKfGoWghWf9imjqds?domain=github.com> When an error condition occurs, it reports the OVERFLOW then it waits for another stream command to start up again. For what it's worth, I don't think you should blindly give it TREADY asserted when things can't process things downstream. The block should be designed such that you know you can consume all the samples in real time. Brian _______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> To unsubscribe send an email to usrp-users-leave@lists.ettus.com <mailto:usrp-users-leave@lists.ettus.com>