usrp-users@lists.ettus.com

Discussion and technical support related to USRP, UHD, RFNoC

View all threads

remote streaming starts, but stops after a few packets received

KW
Kevin Williams
Mon, Jul 28, 2025 2:14 PM

I did an experiment today with just this (Ettus blocks only):

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport: in0}

  • { srcblk: ep6,        srcport: out0,    dstblk: ddc0, dstport: in_0 }

  • { srcblk: ddc0,  srcport: out_0,    dstblk: ep6,      dstport: in0 }

Which did not work – the remote streaming stopped.

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?

From: Martin Braun martin.braun@ettus.com
Sent: Monday, 28 July 2025 15:57
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Ah, gotcha. Well.... I'm going to have to assume your block is stalling for some other reason then :/

Sorry, not that useful a response. Can you run an ILA to see if your block is not getting a tready on its output?

--M

On Fri, Jul 25, 2025 at 2:56 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

Hi Martin,

I didn’t mean to suggest the Ettus examples are not working – they do, and I get continuous streaming with the default image and even 2x remote streams with ddc0 and ddc1.

I am trying to figure out what my block is not doing correctly, and also trying to understand the mechanism that causes the usrp to stop remote streaming.

Its unclear if this is perhaps related to too-small endpoint buffers, etc.

Regards, Kevin

From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Friday, 25 July 2025 14:46
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: nbalkanas@gmail.com mailto:nbalkanas@gmail.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Kevin,

does rx_to_remote_udp.py work (formerly known as remote_rx.py)? (It's an example in host/examples/python).

--M

On Fri, Jul 25, 2025 at 1:20 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

HI,

It seems that explicitly adding “enable_fc=0” does not help.

I notice this control packet though, just a few packets before the streaming stops.

Is that perhaps related to the issue?

Regards, Kevin

From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Wednesday, 23 July 2025 16:57
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: nbalkanas@gmail.com mailto:nbalkanas@gmail.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Kevin,

are you specifying enable_fc=0 in the stream args or not? In the debug statement above, they're shown, but not in your last email. I'm asking because it looks like FC is kicking in and stopping the stream.

--M

On Wed, Jul 23, 2025 at 3:20 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

Hi Nikos,

They are: dest_addr=172.17.0.2,dest_mac_addr=02:42:ac:11:00:02,dest_port=1234,stream_mode=full_packet

Kind regards, Kevin

From: Nikos Balkanas <nbalkanas@gmail.com mailto:nbalkanas@gmail.com >
Sent: Wednesday, 23 July 2025 15:15
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: [EXTERNAL]Re: [USRP-users] remote streaming starts, but stops after a few packets received

Hi Kevin,

What are your stream_args?

Using UHD_STREAM_MODE_NUM_SAMPS_AND_DONE?

Not using RFNOC, but full_packet is not a UHD 4.6 mode:(

BR,

nikos

On Wed, Jul 23, 2025 at 3:14 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

Hi,

I have enabled remote streaming but according to wireshark I only receive 6x 4k packets.

Everything about them is correct – the destination ip, port, size, etc.

From the debug logs the stream is set up as:

[DEBUG] [RFNOC] Creating diverted RX stream with arguments: dest_addr=172.17.0.2,dest_mac_addr=02:42:ac:11:00:02,dest_port=1234,stream_mode=full_packet,__chdr_width=64,enable_remote_stream=1,enable_fc=0

[DEBUG] [X300::SFP0::TA_CTL] On transport adapter 0: Adding route from EPID 6 to destination 172.17.0.2:1234 https://url.za.m.mimecastprotect.com/s/uLPvCP1lEEINz0whzfMixbDH6?domain=172.17.0.2  (MAC Address: 02:42:ac:11:00:02), stream mode FULL_PACKET (0)

What could cause that streaming to stop (given there is no flow control)?

Many thanks, Kevin


USRP-users mailing list -- usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com mailto:usrp-users-leave@lists.ettus.com


USRP-users mailing list -- usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com mailto:usrp-users-leave@lists.ettus.com

I did an experiment today with just this (Ettus blocks only): connections: - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0} - { srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 } - { srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 } Which did not work – the remote streaming stopped. Changing the destination EP to a new one, ep7, worked again. From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP? From: Martin Braun <martin.braun@ettus.com> Sent: Monday, 28 July 2025 15:57 Cc: usrp-users@lists.ettus.com Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Ah, gotcha. Well.... I'm going to have to assume your block is stalling for some other reason then :/ Sorry, not that useful a response. Can you run an ILA to see if your block is not getting a tready on its output? --M On Fri, Jul 25, 2025 at 2:56 PM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: Hi Martin, I didn’t mean to suggest the Ettus examples are not working – they do, and I get continuous streaming with the default image and even 2x remote streams with ddc0 and ddc1. I am trying to figure out what my block is not doing correctly, and also trying to understand the mechanism that causes the usrp to stop remote streaming. Its unclear if this is perhaps related to too-small endpoint buffers, etc. Regards, Kevin From: Martin Braun <martin.braun@ettus.com <mailto:martin.braun@ettus.com> > Sent: Friday, 25 July 2025 14:46 To: Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > Cc: nbalkanas@gmail.com <mailto:nbalkanas@gmail.com> ; usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Kevin, does rx_to_remote_udp.py work (formerly known as remote_rx.py)? (It's an example in host/examples/python). --M On Fri, Jul 25, 2025 at 1:20 PM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: HI, It seems that explicitly adding “enable_fc=0” does not help. I notice this control packet though, just a few packets before the streaming stops. Is that perhaps related to the issue? Regards, Kevin From: Martin Braun <martin.braun@ettus.com <mailto:martin.braun@ettus.com> > Sent: Wednesday, 23 July 2025 16:57 To: Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > Cc: nbalkanas@gmail.com <mailto:nbalkanas@gmail.com> ; usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Kevin, are you specifying enable_fc=0 in the stream args or not? In the debug statement above, they're shown, but not in your last email. I'm asking because it looks like FC is kicking in and stopping the stream. --M On Wed, Jul 23, 2025 at 3:20 PM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: Hi Nikos, They are: dest_addr=172.17.0.2,dest_mac_addr=02:42:ac:11:00:02,dest_port=1234,stream_mode=full_packet Kind regards, Kevin From: Nikos Balkanas <nbalkanas@gmail.com <mailto:nbalkanas@gmail.com> > Sent: Wednesday, 23 July 2025 15:15 To: Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > Cc: usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> Subject: [EXTERNAL]Re: [USRP-users] remote streaming starts, but stops after a few packets received Hi Kevin, What are your stream_args? Using UHD_STREAM_MODE_NUM_SAMPS_AND_DONE? Not using RFNOC, but full_packet is not a UHD 4.6 mode:( BR, nikos On Wed, Jul 23, 2025 at 3:14 PM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: Hi, I have enabled remote streaming but according to wireshark I only receive 6x 4k packets. Everything about them is correct – the destination ip, port, size, etc. From the debug logs the stream is set up as: [DEBUG] [RFNOC] Creating diverted RX stream with arguments: dest_addr=172.17.0.2,dest_mac_addr=02:42:ac:11:00:02,dest_port=1234,stream_mode=full_packet,__chdr_width=64,enable_remote_stream=1,enable_fc=0 [DEBUG] [X300::SFP0::TA_CTL] On transport adapter 0: Adding route from EPID 6 to destination 172.17.0.2:1234 <https://url.za.m.mimecastprotect.com/s/uLPvCP1lEEINz0whzfMixbDH6?domain=172.17.0.2> (MAC Address: 02:42:ac:11:00:02), stream mode FULL_PACKET (0) What could cause that streaming to stop (given there is no flow control)? Many thanks, Kevin _______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> To unsubscribe send an email to usrp-users-leave@lists.ettus.com <mailto:usrp-users-leave@lists.ettus.com> _______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> To unsubscribe send an email to usrp-users-leave@lists.ettus.com <mailto:usrp-users-leave@lists.ettus.com>
BP
Brian Padalino
Mon, Jul 28, 2025 2:49 PM

On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:

I did an experiment today with just this (Ettus blocks only):

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport:
    in0}

  • { srcblk: ep6,        srcport: out0,    dstblk: ddc0, dstport: in_0 }

  • { srcblk: ddc0,  srcport: out_0,    dstblk: ep6,      dstport: in0 }

Which did not work – the remote streaming stopped.

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?

For what it's worth, I'm using remote streaming with a custom block and
it's working well.

In fact, the way remote streaming works (at least for an X440) is that the
Ethernet/UDP information is written here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671

The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.

So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.

If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.

Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.

Good luck.

Brian

On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams < kevin.williams@vastech.co.za> wrote: > I did an experiment today with just this (Ettus blocks only): > > > > connections: > > - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: > in0} > > - { srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 } > > - { srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 } > > > > Which did not work – the remote streaming stopped. > > > > Changing the destination EP to a new one, ep7, worked again. > > > > From the RFNoC 4 workshop slides I was under the impression blocks could > start and end on the same SEP? > For what it's worth, I'm using remote streaming with a custom block and it's working well. In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here: https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet. So if the streaming works when not doing remote streaming it might be something else since all data paths go through here. If you get the first few packets and it stops, is there any way you're providing `enable_fc` as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side. Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up. Good luck. Brian
KW
Kevin Williams
Tue, Jul 29, 2025 5:53 AM

Hi Brian,

I’ve got two observations:

  1. This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable:

No.        Time      Source  Destination        Protocol              Length  Info

1              0.000000000      10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

5343      49.277852197    10.22.128.3        10.23.128.1        UDP      60          49152 → 36716 Len=16

<5000-odd small udp and small rfnoc control & management packets. Setup I guess.>

7318      50.792688865    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<first seq=0 rfnoc data packet of the correct size given my tlast counter>

7319      50.792748665    Intel_e8:c3:4c    Broadcast            ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7320      50.792754229    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<a few 100 more correct data packets>

7775      50.795514072    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets>

7968      52.854255766    Intel_e8:c3:4c    Broadcast            ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7969      53.238261827    Intel_e8:c3:4e  NationalInst_35:aa:da    ARP        42          Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!)

7970      53.238475399    NationalInst_35:aa:da    Intel_e8:c3:4e  ARP        60          10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!)

<then the destination becomes unreachable?>

7971      53.878292746    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7972      53.878302721    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7973      53.878308143    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7974      53.878314734    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7975      53.878320545    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7976      53.878326301    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

<after that, just arp packets and the usrp broadcasting small udp packets>

8014      137.075344888  NationalInst_35:aa:da    Broadcast            ARP        60          ARP Announcement for 10.23.128.3

8015      137.075304321  NationalInst_35:aa:d9    Broadcast            ARP        60          ARP Announcement for 10.22.128.3

8016      140.701925975  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8017      140.701942078  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8018      142.361983307  10.23.128.1        10.23.128.255    UDP      50          59572 → 1534 Len=8

8019      150.005535184  10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

8020      150.005558707  10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

8021      152.097709946  NationalInst_35:aa:d9    Broadcast            ARP        60          ARP Announcement for 10.22.128.3

8022      152.097809876  NationalInst_35:aa:da    Broadcast            ARP        60          ARP Announcement for 10.23.128.3

8023      155.702401576  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8024      155.702431967  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8025      157.378508296  10.23.128.1        10.23.128.255    UDP      50          59572 → 1534 Len=8

  1. ILA results

With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming).

Regards, Kevin

From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

I did an experiment today with just this (Ettus blocks only):

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport: in0}

  • { srcblk: ep6,        srcport: out0,    dstblk: ddc0, dstport: in_0 }

  • { srcblk: ddc0,  srcport: out_0,    dstblk: ep6,      dstport: in0 }

Which did not work – the remote streaming stopped.

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?

For what it's worth, I'm using remote streaming with a custom block and it's working well.

In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 https://url.za.m.mimecastprotect.com/s/A3G4Cqj5nnSkYL0cZfqiEiSn-?domain=github.com

The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet.

So if the streaming works when not doing remote streaming it might be something else since all data paths go through here.

If you get the first few packets and it stops, is there any way you're providing enable_fc as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side.

Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up.

Good luck.

Brian

Hi Brian, I’ve got two observations: 1. This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable: No. Time Source Destination Protocol Length Info 1 0.000000000 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 5343 49.277852197 10.22.128.3 10.23.128.1 UDP 60 49152 → 36716 Len=16 <5000-odd small udp and small rfnoc control & management packets. Setup I guess.> 7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <first seq=0 rfnoc data packet of the correct size given my tlast counter> 7319 50.792748665 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <a few 100 more correct data packets> 7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets> 7968 52.854255766 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!) 7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!) <then the destination becomes unreachable?> 7971 53.878292746 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7972 53.878302721 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7973 53.878308143 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7974 53.878314734 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7975 53.878320545 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7976 53.878326301 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) <after that, just arp packets and the usrp broadcasting small udp packets> 8014 137.075344888 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3 8015 137.075304321 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3 8016 140.701925975 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8017 140.701942078 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8018 142.361983307 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8 8019 150.005535184 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 8020 150.005558707 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 8021 152.097709946 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3 8022 152.097809876 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3 8023 155.702401576 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8024 155.702431967 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8025 157.378508296 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8 2. ILA results With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming). Regards, Kevin From: Brian Padalino <bpadalino@gmail.com> Sent: Monday, 28 July 2025 16:49 To: Kevin Williams <kevin.williams@vastech.co.za> Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: I did an experiment today with just this (Ettus blocks only): connections: - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0} - { srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 } - { srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 } Which did not work – the remote streaming stopped. Changing the destination EP to a new one, ep7, worked again. From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP? For what it's worth, I'm using remote streaming with a custom block and it's working well. In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here: https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 <https://url.za.m.mimecastprotect.com/s/A3G4Cqj5nnSkYL0cZfqiEiSn-?domain=github.com> The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet. So if the streaming works when not doing remote streaming it might be something else since all data paths go through here. If you get the first few packets and it stops, is there any way you're providing `enable_fc` as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side. Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up. Good luck. Brian
KW
Kevin Williams
Tue, Jul 29, 2025 7:19 AM

Hi,

Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops.

What is this control write address (0x01000), and is it perhaps relevant?

From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; 'usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode werner.bode@vastech.co.za
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Hi Brian,

I’ve got two observations:

  1. This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable:

No.        Time      Source  Destination        Protocol              Length  Info

1              0.000000000      10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

5343      49.277852197    10.22.128.3        10.23.128.1        UDP      60          49152 → 36716 Len=16

<5000-odd small udp and small rfnoc control & management packets. Setup I guess.>

7318      50.792688865    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<first seq=0 rfnoc data packet of the correct size given my tlast counter>

7319      50.792748665    Intel_e8:c3:4c    Broadcast            ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7320      50.792754229    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<a few 100 more correct data packets>

7775      50.795514072    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets>

7968      52.854255766    Intel_e8:c3:4c    Broadcast            ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7969      53.238261827    Intel_e8:c3:4e  NationalInst_35:aa:da    ARP        42          Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!)

7970      53.238475399    NationalInst_35:aa:da    Intel_e8:c3:4e  ARP        60          10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!)

<then the destination becomes unreachable?>

7971      53.878292746    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7972      53.878302721    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7973      53.878308143    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7974      53.878314734    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7975      53.878320545    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7976      53.878326301    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

<after that, just arp packets and the usrp broadcasting small udp packets>

8014      137.075344888  NationalInst_35:aa:da    Broadcast            ARP        60          ARP Announcement for 10.23.128.3

8015      137.075304321  NationalInst_35:aa:d9    Broadcast            ARP        60          ARP Announcement for 10.22.128.3

8016      140.701925975  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8017      140.701942078  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8018      142.361983307  10.23.128.1        10.23.128.255    UDP      50          59572 → 1534 Len=8

8019      150.005535184  10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

8020      150.005558707  10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

8021      152.097709946  NationalInst_35:aa:d9    Broadcast            ARP        60          ARP Announcement for 10.22.128.3

8022      152.097809876  NationalInst_35:aa:da    Broadcast            ARP        60          ARP Announcement for 10.23.128.3

8023      155.702401576  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8024      155.702431967  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8025      157.378508296  10.23.128.1        10.23.128.255    UDP      50          59572 → 1534 Len=8

  1. ILA results

With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming).

Regards, Kevin

From: Brian Padalino <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: martin.braun@ettus.com mailto:martin.braun@ettus.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

I did an experiment today with just this (Ettus blocks only):

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport: in0}

  • { srcblk: ep6,        srcport: out0,    dstblk: ddc0, dstport: in_0 }

  • { srcblk: ddc0,  srcport: out_0,    dstblk: ep6,      dstport: in0 }

Which did not work – the remote streaming stopped.

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?

For what it's worth, I'm using remote streaming with a custom block and it's working well.

In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 https://url.za.m.mimecastprotect.com/s/A3G4Cqj5nnSkYL0cZfqiEiSn-?domain=github.com

The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet.

So if the streaming works when not doing remote streaming it might be something else since all data paths go through here.

If you get the first few packets and it stops, is there any way you're providing enable_fc as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side.

Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up.

Good luck.

Brian

Hi, Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops. What is this control write address (0x01000), and is it perhaps relevant? From: Kevin Williams Sent: Tuesday, 29 July 2025 07:53 To: 'bpadalino@gmail.com' <bpadalino@gmail.com> Cc: 'martin.braun@ettus.com' <martin.braun@ettus.com>; 'usrp-users@lists.ettus.com' <usrp-users@lists.ettus.com>; Werner Bode <werner.bode@vastech.co.za> Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Hi Brian, I’ve got two observations: 1. This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable: No. Time Source Destination Protocol Length Info 1 0.000000000 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 5343 49.277852197 10.22.128.3 10.23.128.1 UDP 60 49152 → 36716 Len=16 <5000-odd small udp and small rfnoc control & management packets. Setup I guess.> 7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <first seq=0 rfnoc data packet of the correct size given my tlast counter> 7319 50.792748665 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <a few 100 more correct data packets> 7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets> 7968 52.854255766 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!) 7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!) <then the destination becomes unreachable?> 7971 53.878292746 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7972 53.878302721 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7973 53.878308143 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7974 53.878314734 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7975 53.878320545 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7976 53.878326301 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) <after that, just arp packets and the usrp broadcasting small udp packets> 8014 137.075344888 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3 8015 137.075304321 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3 8016 140.701925975 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8017 140.701942078 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8018 142.361983307 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8 8019 150.005535184 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 8020 150.005558707 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 8021 152.097709946 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3 8022 152.097809876 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3 8023 155.702401576 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8024 155.702431967 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8025 157.378508296 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8 2. ILA results With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming). Regards, Kevin From: Brian Padalino <bpadalino@gmail.com <mailto:bpadalino@gmail.com> > Sent: Monday, 28 July 2025 16:49 To: Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > Cc: martin.braun@ettus.com <mailto:martin.braun@ettus.com> ; usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: I did an experiment today with just this (Ettus blocks only): connections: - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0} - { srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 } - { srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 } Which did not work – the remote streaming stopped. Changing the destination EP to a new one, ep7, worked again. From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP? For what it's worth, I'm using remote streaming with a custom block and it's working well. In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here: https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 <https://url.za.m.mimecastprotect.com/s/A3G4Cqj5nnSkYL0cZfqiEiSn-?domain=github.com> The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet. So if the streaming works when not doing remote streaming it might be something else since all data paths go through here. If you get the first few packets and it stops, is there any way you're providing `enable_fc` as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side. Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up. Good luck. Brian
MB
Martin Braun
Tue, Jul 29, 2025 7:44 AM

Kevin,

based on the src port, this looks like it's going from Device to Host, not
the other way around. This means it's an async message from an RFNoC block,
with address 0x1000. I can't tell for sure from this screenshot, but I
think this is coming from the radio, and 0x1000 is the "RX Error" address.
The data is incorrectly formatted (probably an issue of the CHDR dissector,
but I think it's telling us the data is "2" (if we read this in network
order).

Put these together, and we're looking at a simple overrun. Something in
your chain is holding up the radio after a few packets. Are you sure you're
not seeing an "O" anywhere in your output? You are using a radio block,
right?

--M

On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams kevin.williams@vastech.co.za
wrote:

Hi,

Another observation is the every time the streaming stalls, whether remote
streaming or normal rx_streamer operation, I see this packet from the host
to the x310 a few data packets before it stops.

What is this control write address (0x01000), and is it perhaps relevant?

From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; '
usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received

Hi Brian,

I’ve got two observations:

1. This is a summary of my custom block streaming where the data
packet stream ends with icmp packets about the destination becoming
unreachable:

No.        Time      Source  Destination        Protocol
Length  Info

1              0.000000000      10.23.128.1        10.23.128.255
UDP      50          1534 → 1534 Len=8

5343      49.277852197    10.22.128.3        10.23.128.1
UDP      60          49152 → 36716 Len=16

<5000-odd small udp and small rfnoc control & management packets. Setup I
guess.>

7318      50.792688865    10.22.128.3        10.22.128.1        RFNOC
4146      [Data]    ->  6

<first seq=0 rfnoc data packet of the correct size given my tlast counter>

7319      50.792748665    Intel_e8:c3:4c    Broadcast
ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7320      50.792754229    10.22.128.3        10.22.128.1        RFNOC
4146      [Data]    ->  6

<a few 100 more correct data packets>

7775      50.795514072    10.22.128.3        10.22.128.1        RFNOC
4146      [Data]    ->  6

<a string of more control and short 66 byte rfnoc packets, but no rfnoc
data packets>

7968      52.854255766    Intel_e8:c3:4c    Broadcast
ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7969      53.238261827    Intel_e8:c3:4e  NationalInst_35:aa:da
ARP        42          Who has 10.23.128.3? Tell 10.23.128.1 (duplicate
use of 10.23.128.1 detected!)

7970      53.238475399    NationalInst_35:aa:da    Intel_e8:c3:4e
ARP        60          10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use
of 10.23.128.1 detected!)

<then the destination becomes unreachable?>

7971      53.878292746    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7972      53.878302721    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7973      53.878308143    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7974      53.878314734    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7975      53.878320545    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7976      53.878326301    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

<after that, just arp packets and the usrp broadcasting small udp packets>

8014      137.075344888  NationalInst_35:aa:da    Broadcast
ARP        60          ARP Announcement for 10.23.128.3

8015      137.075304321  NationalInst_35:aa:d9    Broadcast
ARP        60          ARP Announcement for 10.22.128.3

8016      140.701925975  10.23.128.1        10.23.128.255    UDP
50          38981 → 1534 Len=8

8017      140.701942078  10.23.128.1        10.23.128.255    UDP
50          38981 → 1534 Len=8

8018      142.361983307  10.23.128.1        10.23.128.255    UDP
50          59572 → 1534 Len=8

8019      150.005535184  10.23.128.1        10.23.128.255    UDP
50          1534 → 1534 Len=8

8020      150.005558707  10.23.128.1        10.23.128.255    UDP
50          1534 → 1534 Len=8

8021      152.097709946  NationalInst_35:aa:d9    Broadcast
ARP        60          ARP Announcement for 10.22.128.3

8022      152.097809876  NationalInst_35:aa:da    Broadcast
ARP        60          ARP Announcement for 10.23.128.3

8023      155.702401576  10.23.128.1        10.23.128.255    UDP
50          38981 → 1534 Len=8

8024      155.702431967  10.23.128.1        10.23.128.255    UDP
50          38981 → 1534 Len=8

8025      157.378508296  10.23.128.1        10.23.128.255    UDP
50          59572 → 1534 Len=8

2. ILA results

With my block I see a continuously asserted TREADY, with TLAST’s at
exactly the correct sample counts, until streaming stops where I see TREADY
deasserted for 20-odd clocks, and then reasserted (without further
streaming).

Regards, Kevin

From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received

On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:

I did an experiment today with just this (Ettus blocks only):

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport:
    in0}

  • { srcblk: ep6,        srcport: out0,    dstblk: ddc0, dstport: in_0 }

  • { srcblk: ddc0,  srcport: out_0,    dstblk: ep6,      dstport: in0 }

Which did not work – the remote streaming stopped.

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?

For what it's worth, I'm using remote streaming with a custom block and
it's working well.

In fact, the way remote streaming works (at least for an X440) is that the
Ethernet/UDP information is written here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671
https://url.za.m.mimecastprotect.com/s/A3G4Cqj5nnSkYL0cZfqiEiSn-?domain=github.com

The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.

So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.

If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.

Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.

Good luck.

Brian

Kevin, based on the src port, this looks like it's going from Device to Host, not the other way around. This means it's an async message from an RFNoC block, with address 0x1000. I can't tell for sure from this screenshot, but I think this is coming from the radio, and 0x1000 is the "RX Error" address. The data is incorrectly formatted (probably an issue of the CHDR dissector, but I think it's telling us the data is "2" (if we read this in network order). Put these together, and we're looking at a simple overrun. Something in your chain is holding up the radio after a few packets. Are you sure you're not seeing an "O" anywhere in your output? You are using a radio block, right? --M On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <kevin.williams@vastech.co.za> wrote: > Hi, > > > > Another observation is the every time the streaming stalls, whether remote > streaming or normal rx_streamer operation, I see this packet from the host > to the x310 a few data packets before it stops. > > > > What is this control write address (0x01000), and is it perhaps relevant? > > > > > > *From:* Kevin Williams > *Sent:* Tuesday, 29 July 2025 07:53 > *To:* 'bpadalino@gmail.com' <bpadalino@gmail.com> > *Cc:* 'martin.braun@ettus.com' <martin.braun@ettus.com>; ' > usrp-users@lists.ettus.com' <usrp-users@lists.ettus.com>; Werner Bode < > werner.bode@vastech.co.za> > *Subject:* RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, > but stops after a few packets received > > > > Hi Brian, > > > > I’ve got two observations: > > > > 1. This is a summary of my custom block streaming where the data > packet stream ends with icmp packets about the destination becoming > unreachable: > > > > No. Time Source Destination Protocol > Length Info > > 1 0.000000000 10.23.128.1 10.23.128.255 > UDP 50 1534 → 1534 Len=8 > > 5343 49.277852197 10.22.128.3 10.23.128.1 > UDP 60 49152 → 36716 Len=16 > > <5000-odd small udp and small rfnoc control & management packets. Setup I > guess.> > > > > 7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC > 4146 [Data] -> 6 > > <first seq=0 rfnoc data packet of the correct size given my tlast counter> > > > > 7319 50.792748665 Intel_e8:c3:4c Broadcast > ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 > > 7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC > 4146 [Data] -> 6 > > <a few 100 more correct data packets> > > > > 7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC > 4146 [Data] -> 6 > > > > <a string of more control and short 66 byte rfnoc packets, but no rfnoc > data packets> > > > > 7968 52.854255766 Intel_e8:c3:4c Broadcast > ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 > > 7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da > ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate > use of 10.23.128.1 detected!) > > 7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e > ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use > of 10.23.128.1 detected!) > > <then the destination becomes unreachable?> > > > > 7971 53.878292746 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7972 53.878302721 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7973 53.878308143 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7974 53.878314734 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7975 53.878320545 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7976 53.878326301 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > > > <after that, just arp packets and the usrp broadcasting small udp packets> > > > > 8014 137.075344888 NationalInst_35:aa:da Broadcast > ARP 60 ARP Announcement for 10.23.128.3 > > 8015 137.075304321 NationalInst_35:aa:d9 Broadcast > ARP 60 ARP Announcement for 10.22.128.3 > > 8016 140.701925975 10.23.128.1 10.23.128.255 UDP > 50 38981 → 1534 Len=8 > > 8017 140.701942078 10.23.128.1 10.23.128.255 UDP > 50 38981 → 1534 Len=8 > > 8018 142.361983307 10.23.128.1 10.23.128.255 UDP > 50 59572 → 1534 Len=8 > > 8019 150.005535184 10.23.128.1 10.23.128.255 UDP > 50 1534 → 1534 Len=8 > > 8020 150.005558707 10.23.128.1 10.23.128.255 UDP > 50 1534 → 1534 Len=8 > > 8021 152.097709946 NationalInst_35:aa:d9 Broadcast > ARP 60 ARP Announcement for 10.22.128.3 > > 8022 152.097809876 NationalInst_35:aa:da Broadcast > ARP 60 ARP Announcement for 10.23.128.3 > > 8023 155.702401576 10.23.128.1 10.23.128.255 UDP > 50 38981 → 1534 Len=8 > > 8024 155.702431967 10.23.128.1 10.23.128.255 UDP > 50 38981 → 1534 Len=8 > > 8025 157.378508296 10.23.128.1 10.23.128.255 UDP > 50 59572 → 1534 Len=8 > > > > > > 2. ILA results > > > > With my block I see a continuously asserted TREADY, with TLAST’s at > exactly the correct sample counts, until streaming stops where I see TREADY > deasserted for 20-odd clocks, and then reasserted (without further > streaming). > > > > Regards, Kevin > > > > > > *From:* Brian Padalino <bpadalino@gmail.com> > *Sent:* Monday, 28 July 2025 16:49 > *To:* Kevin Williams <kevin.williams@vastech.co.za> > *Cc:* martin.braun@ettus.com; usrp-users@lists.ettus.com > *Subject:* Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, > but stops after a few packets received > > > > On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams < > kevin.williams@vastech.co.za> wrote: > > I did an experiment today with just this (Ettus blocks only): > > > > connections: > > - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: > in0} > > - { srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 } > > - { srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 } > > > > Which did not work – the remote streaming stopped. > > > > Changing the destination EP to a new one, ep7, worked again. > > > > From the RFNoC 4 workshop slides I was under the impression blocks could > start and end on the same SEP? > > > > For what it's worth, I'm using remote streaming with a custom block and > it's working well. > > > > In fact, the way remote streaming works (at least for an X440) is that the > Ethernet/UDP information is written here: > > > > > https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 > <https://url.za.m.mimecastprotect.com/s/A3G4Cqj5nnSkYL0cZfqiEiSn-?domain=github.com> > > > > The kv_map uses the destination EPID as the key for the ethernet > information which gets looked up for every packet. > > > > So if the streaming works when not doing remote streaming it might be > something else since all data paths go through here. > > > > If you get the first few packets and it stops, is there any way you're > providing `enable_fc` as an argument? That would enable flow control which > obviously wouldn't be good if you aren't doing any flow control processing > on your RX side. > > > > Lastly, I agree with Martin that you should probably add an ILA to your > block and the SEP interfaces to see where the AXI stream is getting stopped > up. > > > > Good luck. > > > > Brian >
MB
Martin Braun
Tue, Jul 29, 2025 7:45 AM

On Mon, Jul 28, 2025 at 4:14 PM Kevin Williams kevin.williams@vastech.co.za
wrote:

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?

They can, and even should, but how are you doing remote streaming in this
case? Are you connecting radio -> DDC -> remote streamer?

--M

On Mon, Jul 28, 2025 at 4:14 PM Kevin Williams <kevin.williams@vastech.co.za> wrote: > > > Changing the destination EP to a new one, ep7, worked again. > > > > From the RFNoC 4 workshop slides I was under the impression blocks could > start and end on the same SEP? > They can, and even should, but how are you doing remote streaming in this case? Are you connecting radio -> DDC -> remote streamer? --M
KW
Kevin Williams
Tue, Jul 29, 2025 7:52 AM

Hi Martin,

I do see a single “O”, but this is remote streaming so I didn’t think that should occur?

Yes, this is a radio -> my custom block dynamic connection.

Regards, Kevin

From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Kevin,

based on the src port, this looks like it's going from Device to Host, not the other way around. This means it's an async message from an RFNoC block, with address 0x1000. I can't tell for sure from this screenshot, but I think this is coming from the radio, and 0x1000 is the "RX Error" address. The data is incorrectly formatted (probably an issue of the CHDR dissector, but I think it's telling us the data is "2" (if we read this in network order).

Put these together, and we're looking at a simple overrun. Something in your chain is holding up the radio after a few packets. Are you sure you're not seeing an "O" anywhere in your output? You are using a radio block, right?

--M

On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

Hi,

Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops.

What is this control write address (0x01000), and is it perhaps relevant?

From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com mailto:bpadalino@gmail.com ' <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Cc: 'martin.braun@ettus.com mailto:martin.braun@ettus.com ' <martin.braun@ettus.com mailto:martin.braun@ettus.com >; 'usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com ' <usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com >; Werner Bode <werner.bode@vastech.co.za mailto:werner.bode@vastech.co.za >
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Hi Brian,

I’ve got two observations:

  1. This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable:

No.        Time      Source  Destination        Protocol              Length  Info

1              0.000000000      10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

5343      49.277852197    10.22.128.3        10.23.128.1        UDP      60          49152 → 36716 Len=16

<5000-odd small udp and small rfnoc control & management packets. Setup I guess.>

7318      50.792688865    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<first seq=0 rfnoc data packet of the correct size given my tlast counter>

7319      50.792748665    Intel_e8:c3:4c    Broadcast            ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7320      50.792754229    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<a few 100 more correct data packets>

7775      50.795514072    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets>

7968      52.854255766    Intel_e8:c3:4c    Broadcast            ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7969      53.238261827    Intel_e8:c3:4e  NationalInst_35:aa:da    ARP        42          Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!)

7970      53.238475399    NationalInst_35:aa:da    Intel_e8:c3:4e  ARP        60          10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!)

<then the destination becomes unreachable?>

7971      53.878292746    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7972      53.878302721    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7973      53.878308143    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7974      53.878314734    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7975      53.878320545    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7976      53.878326301    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

<after that, just arp packets and the usrp broadcasting small udp packets>

8014      137.075344888  NationalInst_35:aa:da    Broadcast            ARP        60          ARP Announcement for 10.23.128.3

8015      137.075304321  NationalInst_35:aa:d9    Broadcast            ARP        60          ARP Announcement for 10.22.128.3

8016      140.701925975  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8017      140.701942078  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8018      142.361983307  10.23.128.1        10.23.128.255    UDP      50          59572 → 1534 Len=8

8019      150.005535184  10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

8020      150.005558707  10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

8021      152.097709946  NationalInst_35:aa:d9    Broadcast            ARP        60          ARP Announcement for 10.22.128.3

8022      152.097809876  NationalInst_35:aa:da    Broadcast            ARP        60          ARP Announcement for 10.23.128.3

8023      155.702401576  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8024      155.702431967  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8025      157.378508296  10.23.128.1        10.23.128.255    UDP      50          59572 → 1534 Len=8

  1. ILA results

With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming).

Regards, Kevin

From: Brian Padalino <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: martin.braun@ettus.com mailto:martin.braun@ettus.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

I did an experiment today with just this (Ettus blocks only):

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport: in0}

  • { srcblk: ep6,        srcport: out0,    dstblk: ddc0, dstport: in_0 }

  • { srcblk: ddc0,  srcport: out_0,    dstblk: ep6,      dstport: in0 }

Which did not work – the remote streaming stopped.

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?

For what it's worth, I'm using remote streaming with a custom block and it's working well.

In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 https://url.za.m.mimecastprotect.com/s/irP4C98BNNiN4PosofKiqtmhw?domain=github.com

The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet.

So if the streaming works when not doing remote streaming it might be something else since all data paths go through here.

If you get the first few packets and it stops, is there any way you're providing enable_fc as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side.

Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up.

Good luck.

Brian

Hi Martin, I do see a single “O”, but this is remote streaming so I didn’t think that should occur? Yes, this is a radio -> my custom block dynamic connection. Regards, Kevin From: Martin Braun <martin.braun@ettus.com> Sent: Tuesday, 29 July 2025 09:44 Cc: usrp-users@lists.ettus.com Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Kevin, based on the src port, this looks like it's going from Device to Host, not the other way around. This means it's an async message from an RFNoC block, with address 0x1000. I can't tell for sure from this screenshot, but I think this is coming from the radio, and 0x1000 is the "RX Error" address. The data is incorrectly formatted (probably an issue of the CHDR dissector, but I think it's telling us the data is "2" (if we read this in network order). Put these together, and we're looking at a simple overrun. Something in your chain is holding up the radio after a few packets. Are you sure you're not seeing an "O" anywhere in your output? You are using a radio block, right? --M On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: Hi, Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops. What is this control write address (0x01000), and is it perhaps relevant? From: Kevin Williams Sent: Tuesday, 29 July 2025 07:53 To: 'bpadalino@gmail.com <mailto:bpadalino@gmail.com> ' <bpadalino@gmail.com <mailto:bpadalino@gmail.com> > Cc: 'martin.braun@ettus.com <mailto:martin.braun@ettus.com> ' <martin.braun@ettus.com <mailto:martin.braun@ettus.com> >; 'usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> ' <usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> >; Werner Bode <werner.bode@vastech.co.za <mailto:werner.bode@vastech.co.za> > Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Hi Brian, I’ve got two observations: 1. This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable: No. Time Source Destination Protocol Length Info 1 0.000000000 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 5343 49.277852197 10.22.128.3 10.23.128.1 UDP 60 49152 → 36716 Len=16 <5000-odd small udp and small rfnoc control & management packets. Setup I guess.> 7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <first seq=0 rfnoc data packet of the correct size given my tlast counter> 7319 50.792748665 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <a few 100 more correct data packets> 7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets> 7968 52.854255766 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!) 7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!) <then the destination becomes unreachable?> 7971 53.878292746 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7972 53.878302721 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7973 53.878308143 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7974 53.878314734 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7975 53.878320545 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7976 53.878326301 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) <after that, just arp packets and the usrp broadcasting small udp packets> 8014 137.075344888 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3 8015 137.075304321 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3 8016 140.701925975 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8017 140.701942078 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8018 142.361983307 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8 8019 150.005535184 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 8020 150.005558707 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 8021 152.097709946 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3 8022 152.097809876 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3 8023 155.702401576 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8024 155.702431967 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8025 157.378508296 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8 2. ILA results With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming). Regards, Kevin From: Brian Padalino <bpadalino@gmail.com <mailto:bpadalino@gmail.com> > Sent: Monday, 28 July 2025 16:49 To: Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > Cc: martin.braun@ettus.com <mailto:martin.braun@ettus.com> ; usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: I did an experiment today with just this (Ettus blocks only): connections: - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0} - { srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 } - { srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 } Which did not work – the remote streaming stopped. Changing the destination EP to a new one, ep7, worked again. From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP? For what it's worth, I'm using remote streaming with a custom block and it's working well. In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here: https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 <https://url.za.m.mimecastprotect.com/s/irP4C98BNNiN4PosofKiqtmhw?domain=github.com> The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet. So if the streaming works when not doing remote streaming it might be something else since all data paths go through here. If you get the first few packets and it stops, is there any way you're providing `enable_fc` as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side. Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up. Good luck. Brian
KW
Kevin Williams
Tue, Jul 29, 2025 8:00 AM

Yes, correct.

My host is 10.22.128.1 though, and the usrp (dest) is 10.22.128.3.

So from wireshark I read that last control packet as host->usrp?

These are the current routes:

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport: in0}

  • { srcblk: radio0,    srcport: out_1,    dstblk: device,  dstport: none}

  • { srcblk: radio1,    srcport: out_0,    dstblk: ep2,      dstport: in0}

  • { srcblk: radio1,    srcport: out_1,    dstblk: device,  dstport: none}

  • { srcblk: device,  srcport: none,  dstblk: radio0,    dstport: in_0 }

  • { srcblk: device,  srcport: none,  dstblk: radio0,    dstport: in_1 }

  • { srcblk: device,  srcport: none,  dstblk: radio1,    dstport: in_0 }

  • { srcblk: device,  srcport: none,  dstblk: radio1,    dstport: in_1 }

multiddc connections

  • { srcblk: ep6,        srcport: out0,    dstblk: multiddc0, dstport: in_0 }

  • { srcblk: multiddc0,  srcport: out_0,    dstblk: ep7,      dstport: in0 }

Changing ep7 to ep6 doesn’t work either although I believe that should be valid too.

From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:46
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

On Mon, Jul 28, 2025 at 4:14 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?

They can, and even should, but how are you doing remote streaming in this case? Are you connecting radio -> DDC -> remote streamer?

--M

Yes, correct. My host is 10.22.128.1 though, and the usrp (dest) is 10.22.128.3. So from wireshark I read that last control packet as host->usrp? These are the current routes: connections: - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0} - { srcblk: radio0, srcport: out_1, dstblk: _device_, dstport: _none_} - { srcblk: radio1, srcport: out_0, dstblk: ep2, dstport: in0} - { srcblk: radio1, srcport: out_1, dstblk: _device_, dstport: _none_} - { srcblk: _device_, srcport: _none_, dstblk: radio0, dstport: in_0 } - { srcblk: _device_, srcport: _none_, dstblk: radio0, dstport: in_1 } - { srcblk: _device_, srcport: _none_, dstblk: radio1, dstport: in_0 } - { srcblk: _device_, srcport: _none_, dstblk: radio1, dstport: in_1 } # multiddc connections - { srcblk: ep6, srcport: out0, dstblk: multiddc0, dstport: in_0 } - { srcblk: multiddc0, srcport: out_0, dstblk: ep7, dstport: in0 } Changing ep7 to ep6 doesn’t work either although I believe that should be valid too. From: Martin Braun <martin.braun@ettus.com> Sent: Tuesday, 29 July 2025 09:46 Cc: usrp-users@lists.ettus.com Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received On Mon, Jul 28, 2025 at 4:14 PM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: Changing the destination EP to a new one, ep7, worked again. From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP? They can, and even should, but how are you doing remote streaming in this case? Are you connecting radio -> DDC -> remote streamer? --M
MB
Martin Braun
Tue, Jul 29, 2025 8:01 AM

Normally flow control is the thing that will let the radio stall, but maybe
it's something else. From what I can see, there's two potential culprits:

  1. Your block is not permanently processing samples, but has some bubble
    cycles or something like that. 2) The SEP->SEP connection has an issue.

If you can, connect everything statically and see how that fares.

--M

On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams kevin.williams@vastech.co.za
wrote:

Hi Martin,

I do see a single “O”, but this is remote streaming so I didn’t think that
should occur?

Yes, this is a radio -> my custom block dynamic connection.

Regards, Kevin

From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received

Kevin,

based on the src port, this looks like it's going from Device to Host, not
the other way around. This means it's an async message from an RFNoC block,
with address 0x1000. I can't tell for sure from this screenshot, but I
think this is coming from the radio, and 0x1000 is the "RX Error" address.
The data is incorrectly formatted (probably an issue of the CHDR dissector,
but I think it's telling us the data is "2" (if we read this in network
order).

Put these together, and we're looking at a simple overrun. Something in
your chain is holding up the radio after a few packets. Are you sure you're
not seeing an "O" anywhere in your output? You are using a radio block,
right?

--M

On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:

Hi,

Another observation is the every time the streaming stalls, whether remote
streaming or normal rx_streamer operation, I see this packet from the host
to the x310 a few data packets before it stops.

What is this control write address (0x01000), and is it perhaps relevant?

From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; '
usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received

Hi Brian,

I’ve got two observations:

1. This is a summary of my custom block streaming where the data
packet stream ends with icmp packets about the destination becoming
unreachable:

No.        Time      Source  Destination        Protocol
Length  Info

1              0.000000000      10.23.128.1        10.23.128.255
UDP      50          1534 → 1534 Len=8

5343      49.277852197    10.22.128.3        10.23.128.1
UDP      60          49152 → 36716 Len=16

<5000-odd small udp and small rfnoc control & management packets. Setup I
guess.>

7318      50.792688865    10.22.128.3        10.22.128.1        RFNOC
4146      [Data]    ->  6

<first seq=0 rfnoc data packet of the correct size given my tlast counter>

7319      50.792748665    Intel_e8:c3:4c    Broadcast
ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7320      50.792754229    10.22.128.3        10.22.128.1        RFNOC
4146      [Data]    ->  6

<a few 100 more correct data packets>

7775      50.795514072    10.22.128.3        10.22.128.1        RFNOC
4146      [Data]    ->  6

<a string of more control and short 66 byte rfnoc packets, but no rfnoc
data packets>

7968      52.854255766    Intel_e8:c3:4c    Broadcast
ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7969      53.238261827    Intel_e8:c3:4e  NationalInst_35:aa:da
ARP        42          Who has 10.23.128.3? Tell 10.23.128.1 (duplicate
use of 10.23.128.1 detected!)

7970      53.238475399    NationalInst_35:aa:da    Intel_e8:c3:4e
ARP        60          10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use
of 10.23.128.1 detected!)

<then the destination becomes unreachable?>

7971      53.878292746    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7972      53.878302721    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7973      53.878308143    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7974      53.878314734    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7975      53.878320545    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

7976      53.878326301    10.23.128.1        10.22.128.3
ICMP    590        Destination unreachable (Host unreachable)

<after that, just arp packets and the usrp broadcasting small udp packets>

8014      137.075344888  NationalInst_35:aa:da    Broadcast
ARP        60          ARP Announcement for 10.23.128.3

8015      137.075304321  NationalInst_35:aa:d9    Broadcast
ARP        60          ARP Announcement for 10.22.128.3

8016      140.701925975  10.23.128.1        10.23.128.255    UDP
50          38981 → 1534 Len=8

8017      140.701942078  10.23.128.1        10.23.128.255    UDP
50          38981 → 1534 Len=8

8018      142.361983307  10.23.128.1        10.23.128.255    UDP
50          59572 → 1534 Len=8

8019      150.005535184  10.23.128.1        10.23.128.255    UDP
50          1534 → 1534 Len=8

8020      150.005558707  10.23.128.1        10.23.128.255    UDP
50          1534 → 1534 Len=8

8021      152.097709946  NationalInst_35:aa:d9    Broadcast
ARP        60          ARP Announcement for 10.22.128.3

8022      152.097809876  NationalInst_35:aa:da    Broadcast
ARP        60          ARP Announcement for 10.23.128.3

8023      155.702401576  10.23.128.1        10.23.128.255    UDP
50          38981 → 1534 Len=8

8024      155.702431967  10.23.128.1        10.23.128.255    UDP
50          38981 → 1534 Len=8

8025      157.378508296  10.23.128.1        10.23.128.255    UDP
50          59572 → 1534 Len=8

2. ILA results

With my block I see a continuously asserted TREADY, with TLAST’s at
exactly the correct sample counts, until streaming stops where I see TREADY
deasserted for 20-odd clocks, and then reasserted (without further
streaming).

Regards, Kevin

From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received

On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:

I did an experiment today with just this (Ettus blocks only):

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport:
    in0}

  • { srcblk: ep6,        srcport: out0,    dstblk: ddc0, dstport: in_0 }

  • { srcblk: ddc0,  srcport: out_0,    dstblk: ep6,      dstport: in0 }

Which did not work – the remote streaming stopped.

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?

For what it's worth, I'm using remote streaming with a custom block and
it's working well.

In fact, the way remote streaming works (at least for an X440) is that the
Ethernet/UDP information is written here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671
https://url.za.m.mimecastprotect.com/s/irP4C98BNNiN4PosofKiqtmhw?domain=github.com

The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.

So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.

If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.

Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.

Good luck.

Brian

Normally flow control is the thing that will let the radio stall, but maybe it's something else. From what I can see, there's two potential culprits: 1) Your block is not permanently processing samples, but has some bubble cycles or something like that. 2) The SEP->SEP connection has an issue. If you can, connect everything statically and see how that fares. --M On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <kevin.williams@vastech.co.za> wrote: > Hi Martin, > > > > I do see a single “O”, but this is remote streaming so I didn’t think that > should occur? > > > > Yes, this is a radio -> my custom block dynamic connection. > > > > Regards, Kevin > > > > *From:* Martin Braun <martin.braun@ettus.com> > *Sent:* Tuesday, 29 July 2025 09:44 > *Cc:* usrp-users@lists.ettus.com > *Subject:* [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but > stops after a few packets received > > > > Kevin, > > > > based on the src port, this looks like it's going from Device to Host, not > the other way around. This means it's an async message from an RFNoC block, > with address 0x1000. I can't tell for sure from this screenshot, but I > think this is coming from the radio, and 0x1000 is the "RX Error" address. > The data is incorrectly formatted (probably an issue of the CHDR dissector, > but I think it's telling us the data is "2" (if we read this in network > order). > > > > Put these together, and we're looking at a simple overrun. Something in > your chain is holding up the radio after a few packets. Are you sure you're > not seeing an "O" anywhere in your output? You are using a radio block, > right? > > > > --M > > > > On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams < > kevin.williams@vastech.co.za> wrote: > > Hi, > > > > Another observation is the every time the streaming stalls, whether remote > streaming or normal rx_streamer operation, I see this packet from the host > to the x310 a few data packets before it stops. > > > > What is this control write address (0x01000), and is it perhaps relevant? > > > > > > *From:* Kevin Williams > *Sent:* Tuesday, 29 July 2025 07:53 > *To:* 'bpadalino@gmail.com' <bpadalino@gmail.com> > *Cc:* 'martin.braun@ettus.com' <martin.braun@ettus.com>; ' > usrp-users@lists.ettus.com' <usrp-users@lists.ettus.com>; Werner Bode < > werner.bode@vastech.co.za> > *Subject:* RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, > but stops after a few packets received > > > > Hi Brian, > > > > I’ve got two observations: > > > > 1. This is a summary of my custom block streaming where the data > packet stream ends with icmp packets about the destination becoming > unreachable: > > > > No. Time Source Destination Protocol > Length Info > > 1 0.000000000 10.23.128.1 10.23.128.255 > UDP 50 1534 → 1534 Len=8 > > 5343 49.277852197 10.22.128.3 10.23.128.1 > UDP 60 49152 → 36716 Len=16 > > <5000-odd small udp and small rfnoc control & management packets. Setup I > guess.> > > > > 7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC > 4146 [Data] -> 6 > > <first seq=0 rfnoc data packet of the correct size given my tlast counter> > > > > 7319 50.792748665 Intel_e8:c3:4c Broadcast > ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 > > 7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC > 4146 [Data] -> 6 > > <a few 100 more correct data packets> > > > > 7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC > 4146 [Data] -> 6 > > > > <a string of more control and short 66 byte rfnoc packets, but no rfnoc > data packets> > > > > 7968 52.854255766 Intel_e8:c3:4c Broadcast > ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 > > 7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da > ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate > use of 10.23.128.1 detected!) > > 7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e > ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use > of 10.23.128.1 detected!) > > <then the destination becomes unreachable?> > > > > 7971 53.878292746 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7972 53.878302721 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7973 53.878308143 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7974 53.878314734 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7975 53.878320545 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > 7976 53.878326301 10.23.128.1 10.22.128.3 > ICMP 590 Destination unreachable (Host unreachable) > > > > <after that, just arp packets and the usrp broadcasting small udp packets> > > > > 8014 137.075344888 NationalInst_35:aa:da Broadcast > ARP 60 ARP Announcement for 10.23.128.3 > > 8015 137.075304321 NationalInst_35:aa:d9 Broadcast > ARP 60 ARP Announcement for 10.22.128.3 > > 8016 140.701925975 10.23.128.1 10.23.128.255 UDP > 50 38981 → 1534 Len=8 > > 8017 140.701942078 10.23.128.1 10.23.128.255 UDP > 50 38981 → 1534 Len=8 > > 8018 142.361983307 10.23.128.1 10.23.128.255 UDP > 50 59572 → 1534 Len=8 > > 8019 150.005535184 10.23.128.1 10.23.128.255 UDP > 50 1534 → 1534 Len=8 > > 8020 150.005558707 10.23.128.1 10.23.128.255 UDP > 50 1534 → 1534 Len=8 > > 8021 152.097709946 NationalInst_35:aa:d9 Broadcast > ARP 60 ARP Announcement for 10.22.128.3 > > 8022 152.097809876 NationalInst_35:aa:da Broadcast > ARP 60 ARP Announcement for 10.23.128.3 > > 8023 155.702401576 10.23.128.1 10.23.128.255 UDP > 50 38981 → 1534 Len=8 > > 8024 155.702431967 10.23.128.1 10.23.128.255 UDP > 50 38981 → 1534 Len=8 > > 8025 157.378508296 10.23.128.1 10.23.128.255 UDP > 50 59572 → 1534 Len=8 > > > > > > 2. ILA results > > > > With my block I see a continuously asserted TREADY, with TLAST’s at > exactly the correct sample counts, until streaming stops where I see TREADY > deasserted for 20-odd clocks, and then reasserted (without further > streaming). > > > > Regards, Kevin > > > > > > *From:* Brian Padalino <bpadalino@gmail.com> > *Sent:* Monday, 28 July 2025 16:49 > *To:* Kevin Williams <kevin.williams@vastech.co.za> > *Cc:* martin.braun@ettus.com; usrp-users@lists.ettus.com > *Subject:* Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, > but stops after a few packets received > > > > On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams < > kevin.williams@vastech.co.za> wrote: > > I did an experiment today with just this (Ettus blocks only): > > > > connections: > > - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: > in0} > > - { srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 } > > - { srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 } > > > > Which did not work – the remote streaming stopped. > > > > Changing the destination EP to a new one, ep7, worked again. > > > > From the RFNoC 4 workshop slides I was under the impression blocks could > start and end on the same SEP? > > > > For what it's worth, I'm using remote streaming with a custom block and > it's working well. > > > > In fact, the way remote streaming works (at least for an X440) is that the > Ethernet/UDP information is written here: > > > > > https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 > <https://url.za.m.mimecastprotect.com/s/irP4C98BNNiN4PosofKiqtmhw?domain=github.com> > > > > The kv_map uses the destination EPID as the key for the ethernet > information which gets looked up for every packet. > > > > So if the streaming works when not doing remote streaming it might be > something else since all data paths go through here. > > > > If you get the first few packets and it stops, is there any way you're > providing `enable_fc` as an argument? That would enable flow control which > obviously wouldn't be good if you aren't doing any flow control processing > on your RX side. > > > > Lastly, I agree with Martin that you should probably add an ILA to your > block and the SEP interfaces to see where the AXI stream is getting stopped > up. > > > > Good luck. > > > > Brian > >
KW
Kevin Williams
Tue, Jul 29, 2025 11:03 AM

The resolution is that the x310 has the rfnoc_chdr clock (which I used to clock my block) slower than the radio clock, whereas with my previous n300 that clock is faster..!

I need to create many output channels from my block now, so I think I will just ignore handshaking, and design on the basis of the radio streaming continuously.

From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Normally flow control is the thing that will let the radio stall, but maybe it's something else. From what I can see, there's two potential culprits: 1) Your block is not permanently processing samples, but has some bubble cycles or something like that. 2) The SEP->SEP connection has an issue.

If you can, connect everything statically and see how that fares.

--M

On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

Hi Martin,

I do see a single “O”, but this is remote streaming so I didn’t think that should occur?

Yes, this is a radio -> my custom block dynamic connection.

Regards, Kevin

From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Kevin,

based on the src port, this looks like it's going from Device to Host, not the other way around. This means it's an async message from an RFNoC block, with address 0x1000. I can't tell for sure from this screenshot, but I think this is coming from the radio, and 0x1000 is the "RX Error" address. The data is incorrectly formatted (probably an issue of the CHDR dissector, but I think it's telling us the data is "2" (if we read this in network order).

Put these together, and we're looking at a simple overrun. Something in your chain is holding up the radio after a few packets. Are you sure you're not seeing an "O" anywhere in your output? You are using a radio block, right?

--M

On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

Hi,

Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops.

What is this control write address (0x01000), and is it perhaps relevant?

From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com mailto:bpadalino@gmail.com ' <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Cc: 'martin.braun@ettus.com mailto:martin.braun@ettus.com ' <martin.braun@ettus.com mailto:martin.braun@ettus.com >; 'usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com ' <usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com >; Werner Bode <werner.bode@vastech.co.za mailto:werner.bode@vastech.co.za >
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

Hi Brian,

I’ve got two observations:

  1. This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable:

No.        Time      Source  Destination        Protocol              Length  Info

1              0.000000000      10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

5343      49.277852197    10.22.128.3        10.23.128.1        UDP      60          49152 → 36716 Len=16

<5000-odd small udp and small rfnoc control & management packets. Setup I guess.>

7318      50.792688865    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<first seq=0 rfnoc data packet of the correct size given my tlast counter>

7319      50.792748665    Intel_e8:c3:4c    Broadcast            ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7320      50.792754229    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<a few 100 more correct data packets>

7775      50.795514072    10.22.128.3        10.22.128.1        RFNOC  4146      [Data]    ->  6

<a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets>

7968      52.854255766    Intel_e8:c3:4c    Broadcast            ARP        42          Who has 10.22.128.1? Tell 10.23.128.1

7969      53.238261827    Intel_e8:c3:4e  NationalInst_35:aa:da    ARP        42          Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!)

7970      53.238475399    NationalInst_35:aa:da    Intel_e8:c3:4e  ARP        60          10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!)

<then the destination becomes unreachable?>

7971      53.878292746    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7972      53.878302721    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7973      53.878308143    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7974      53.878314734    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7975      53.878320545    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

7976      53.878326301    10.23.128.1        10.22.128.3        ICMP    590        Destination unreachable (Host unreachable)

<after that, just arp packets and the usrp broadcasting small udp packets>

8014      137.075344888  NationalInst_35:aa:da    Broadcast            ARP        60          ARP Announcement for 10.23.128.3

8015      137.075304321  NationalInst_35:aa:d9    Broadcast            ARP        60          ARP Announcement for 10.22.128.3

8016      140.701925975  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8017      140.701942078  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8018      142.361983307  10.23.128.1        10.23.128.255    UDP      50          59572 → 1534 Len=8

8019      150.005535184  10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

8020      150.005558707  10.23.128.1        10.23.128.255    UDP      50          1534 → 1534 Len=8

8021      152.097709946  NationalInst_35:aa:d9    Broadcast            ARP        60          ARP Announcement for 10.22.128.3

8022      152.097809876  NationalInst_35:aa:da    Broadcast            ARP        60          ARP Announcement for 10.23.128.3

8023      155.702401576  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8024      155.702431967  10.23.128.1        10.23.128.255    UDP      50          38981 → 1534 Len=8

8025      157.378508296  10.23.128.1        10.23.128.255    UDP      50          59572 → 1534 Len=8

  1. ILA results

With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming).

Regards, Kevin

From: Brian Padalino <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: martin.braun@ettus.com mailto:martin.braun@ettus.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received

On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:

I did an experiment today with just this (Ettus blocks only):

connections:

  • { srcblk: radio0,    srcport: out_0,    dstblk: ep0,      dstport: in0}

  • { srcblk: ep6,        srcport: out0,    dstblk: ddc0, dstport: in_0 }

  • { srcblk: ddc0,  srcport: out_0,    dstblk: ep6,      dstport: in0 }

Which did not work – the remote streaming stopped.

Changing the destination EP to a new one, ep7, worked again.

From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?

For what it's worth, I'm using remote streaming with a custom block and it's working well.

In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here:

https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 https://url.za.m.mimecastprotect.com/s/H9mfCKOByytM7xBsMfri5-4wa?domain=github.com

The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet.

So if the streaming works when not doing remote streaming it might be something else since all data paths go through here.

If you get the first few packets and it stops, is there any way you're providing enable_fc as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side.

Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up.

Good luck.

Brian

The resolution is that the x310 has the rfnoc_chdr clock (which I used to clock my block) slower than the radio clock, whereas with my previous n300 that clock is faster..! I need to create many output channels from my block now, so I think I will just ignore handshaking, and design on the basis of the radio streaming continuously. From: Martin Braun <martin.braun@ettus.com> Sent: Tuesday, 29 July 2025 10:01 Cc: usrp-users@lists.ettus.com Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Normally flow control is the thing that will let the radio stall, but maybe it's something else. From what I can see, there's two potential culprits: 1) Your block is not permanently processing samples, but has some bubble cycles or something like that. 2) The SEP->SEP connection has an issue. If you can, connect everything statically and see how that fares. --M On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: Hi Martin, I do see a single “O”, but this is remote streaming so I didn’t think that should occur? Yes, this is a radio -> my custom block dynamic connection. Regards, Kevin From: Martin Braun <martin.braun@ettus.com <mailto:martin.braun@ettus.com> > Sent: Tuesday, 29 July 2025 09:44 Cc: usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Kevin, based on the src port, this looks like it's going from Device to Host, not the other way around. This means it's an async message from an RFNoC block, with address 0x1000. I can't tell for sure from this screenshot, but I think this is coming from the radio, and 0x1000 is the "RX Error" address. The data is incorrectly formatted (probably an issue of the CHDR dissector, but I think it's telling us the data is "2" (if we read this in network order). Put these together, and we're looking at a simple overrun. Something in your chain is holding up the radio after a few packets. Are you sure you're not seeing an "O" anywhere in your output? You are using a radio block, right? --M On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: Hi, Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops. What is this control write address (0x01000), and is it perhaps relevant? From: Kevin Williams Sent: Tuesday, 29 July 2025 07:53 To: 'bpadalino@gmail.com <mailto:bpadalino@gmail.com> ' <bpadalino@gmail.com <mailto:bpadalino@gmail.com> > Cc: 'martin.braun@ettus.com <mailto:martin.braun@ettus.com> ' <martin.braun@ettus.com <mailto:martin.braun@ettus.com> >; 'usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> ' <usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> >; Werner Bode <werner.bode@vastech.co.za <mailto:werner.bode@vastech.co.za> > Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received Hi Brian, I’ve got two observations: 1. This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable: No. Time Source Destination Protocol Length Info 1 0.000000000 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 5343 49.277852197 10.22.128.3 10.23.128.1 UDP 60 49152 → 36716 Len=16 <5000-odd small udp and small rfnoc control & management packets. Setup I guess.> 7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <first seq=0 rfnoc data packet of the correct size given my tlast counter> 7319 50.792748665 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <a few 100 more correct data packets> 7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6 <a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets> 7968 52.854255766 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1 7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!) 7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!) <then the destination becomes unreachable?> 7971 53.878292746 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7972 53.878302721 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7973 53.878308143 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7974 53.878314734 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7975 53.878320545 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) 7976 53.878326301 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable) <after that, just arp packets and the usrp broadcasting small udp packets> 8014 137.075344888 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3 8015 137.075304321 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3 8016 140.701925975 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8017 140.701942078 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8018 142.361983307 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8 8019 150.005535184 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 8020 150.005558707 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8 8021 152.097709946 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3 8022 152.097809876 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3 8023 155.702401576 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8024 155.702431967 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8 8025 157.378508296 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8 2. ILA results With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming). Regards, Kevin From: Brian Padalino <bpadalino@gmail.com <mailto:bpadalino@gmail.com> > Sent: Monday, 28 July 2025 16:49 To: Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > Cc: martin.braun@ettus.com <mailto:martin.braun@ettus.com> ; usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com> Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za <mailto:kevin.williams@vastech.co.za> > wrote: I did an experiment today with just this (Ettus blocks only): connections: - { srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0} - { srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 } - { srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 } Which did not work – the remote streaming stopped. Changing the destination EP to a new one, ep7, worked again. From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP? For what it's worth, I'm using remote streaming with a custom block and it's working well. In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here: https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 <https://url.za.m.mimecastprotect.com/s/H9mfCKOByytM7xBsMfri5-4wa?domain=github.com> The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet. So if the streaming works when not doing remote streaming it might be something else since all data paths go through here. If you get the first few packets and it stops, is there any way you're providing `enable_fc` as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side. Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up. Good luck. Brian