usrp-users@lists.ettus.com
Discussion and technical support related to USRP, UHD, RFNoC
View all threadsremote streaming starts, but stops after a few packets received
Hi Kevin,
This seems like a network error.
(duplicate use of 10.23.128.1 detected!)
Seems like at some point smt (RFNOC?) is creating another 10.23.128.1 and
from that point on, it becomes unreachable:(
HTH
Nikos
On Tue, Jul 29, 2025 at 2:04 PM Kevin Williams kevin.williams@vastech.co.za
wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to
clock my block) slower than the radio clock, whereas with my previous n300
that clock is faster..!
I need to create many output channels from my block now, so I think I will
just ignore handshaking, and design on the basis of the radio streaming
continuously.
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Normally flow control is the thing that will let the radio stall, but
maybe it's something else. From what I can see, there's two potential
culprits: 1) Your block is not permanently processing samples, but has some
bubble cycles or something like that. 2) The SEP->SEP connection has an
issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think that
should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host, not
the other way around. This means it's an async message from an RFNoC block,
with address 0x1000. I can't tell for sure from this screenshot, but I
think this is coming from the radio, and 0x1000 is the "RX Error" address.
The data is incorrectly formatted (probably an issue of the CHDR dissector,
but I think it's telling us the data is "2" (if we read this in network
order).
Put these together, and we're looking at a simple overrun. Something in
your chain is holding up the radio after a few packets. Are you sure you're
not seeing an "O" anywhere in your output? You are using a radio block,
right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi,
Another observation is the every time the streaming stalls, whether remote
streaming or normal rx_streamer operation, I see this packet from the host
to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; '
usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
Hi Brian,
I’ve got two observations:
1. This is a summary of my custom block streaming where the data
packet stream ends with icmp packets about the destination becoming
unreachable:
No. Time Source Destination Protocol
Length Info
1 0.000000000 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1
UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I
guess.>
7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
7319 50.792748665 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc
data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da
ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate
use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e
ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use
of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255 UDP
50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255 UDP
50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255 UDP
50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255 UDP
50 59572 → 1534 Len=8
2. ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at
exactly the correct sample counts, until streaming stops where I see TREADY
deasserted for 20-odd clocks, and then reasserted (without further
streaming).
Regards, Kevin
From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport:
in0} -
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 }
-
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and
it's working well.
In fact, the way remote streaming works (at least for an X440) is that the
Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671
https://url.za.m.mimecastprotect.com/s/H9mfCKOByytM7xBsMfri5-4wa?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.
Good luck.
Brian
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
When the problem occurs check your arp cache on your host to verify it...
On Tue, Jul 29, 2025 at 3:38 PM Nikos Balkanas nbalkanas@gmail.com wrote:
Hi Kevin,
This seems like a network error.
(duplicate use of 10.23.128.1 detected!)
Seems like at some point smt (RFNOC?) is creating another 10.23.128.1 and
from that point on, it becomes unreachable:(
HTH
Nikos
On Tue, Jul 29, 2025 at 2:04 PM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to
clock my block) slower than the radio clock, whereas with my previous n300
that clock is faster..!
I need to create many output channels from my block now, so I think I
will just ignore handshaking, and design on the basis of the radio
streaming continuously.
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Normally flow control is the thing that will let the radio stall, but
maybe it's something else. From what I can see, there's two potential
culprits: 1) Your block is not permanently processing samples, but has some
bubble cycles or something like that. 2) The SEP->SEP connection has an
issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think
that should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host,
not the other way around. This means it's an async message from an RFNoC
block, with address 0x1000. I can't tell for sure from this screenshot, but
I think this is coming from the radio, and 0x1000 is the "RX Error"
address. The data is incorrectly formatted (probably an issue of the CHDR
dissector, but I think it's telling us the data is "2" (if we read this in
network order).
Put these together, and we're looking at a simple overrun. Something in
your chain is holding up the radio after a few packets. Are you sure you're
not seeing an "O" anywhere in your output? You are using a radio block,
right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi,
Another observation is the every time the streaming stalls, whether
remote streaming or normal rx_streamer operation, I see this packet from
the host to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; '
usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
Hi Brian,
I’ve got two observations:
1. This is a summary of my custom block streaming where the data
packet stream ends with icmp packets about the destination becoming
unreachable:
No. Time Source Destination Protocol
Length Info
1 0.000000000 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1
UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I
guess.>
7318 50.792688865 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<first seq=0 rfnoc data packet of the correct size given my tlast counter
7319 50.792748665 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc
data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da
ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate
use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e
ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use
of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255
UDP 50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255
UDP 50 59572 → 1534 Len=8
2. ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at
exactly the correct sample counts, until streaming stops where I see TREADY
deasserted for 20-odd clocks, and then reasserted (without further
streaming).
Regards, Kevin
From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport:
in0} -
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0
} -
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and
it's working well.
In fact, the way remote streaming works (at least for an X440) is that
the Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671
https://url.za.m.mimecastprotect.com/s/H9mfCKOByytM7xBsMfri5-4wa?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.
Good luck.
Brian
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Hi Nikos,
Indeed!! My host had that IP duplicated.
I think this was two issues – one, my block was clocked slower than the radio, and possibly two, rfnoc tried to use the second sfp port and at that stage the duplicate IP possibly became the issue.
I’m not sure what the rules are when that second port gets activated.
By clocking mine on the ce clock I didn’t see that overflow again, and thus possibly did not encounter the duplicate IP issue then either.
Anyway, fixed that now, as I am employing several more channels from my block.
Thanks!! Kevin
From: Nikos Balkanas nbalkanas@gmail.com
Sent: Tuesday, 29 July 2025 14:44
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com; Werner Bode werner.bode@vastech.co.za
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
When the problem occurs check your arp cache on your host to verify it...
On Tue, Jul 29, 2025 at 3:38 PM Nikos Balkanas <nbalkanas@gmail.com mailto:nbalkanas@gmail.com > wrote:
Hi Kevin,
This seems like a network error.
(duplicate use of 10.23.128.1 detected!)
Seems like at some point smt (RFNOC?) is creating another 10.23.128.1 and from that point on, it becomes unreachable:(
HTH
Nikos
On Tue, Jul 29, 2025 at 2:04 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to clock my block) slower than the radio clock, whereas with my previous n300 that clock is faster..!
I need to create many output channels from my block now, so I think I will just ignore handshaking, and design on the basis of the radio streaming continuously.
From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Normally flow control is the thing that will let the radio stall, but maybe it's something else. From what I can see, there's two potential culprits: 1) Your block is not permanently processing samples, but has some bubble cycles or something like that. 2) The SEP->SEP connection has an issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think that should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host, not the other way around. This means it's an async message from an RFNoC block, with address 0x1000. I can't tell for sure from this screenshot, but I think this is coming from the radio, and 0x1000 is the "RX Error" address. The data is incorrectly formatted (probably an issue of the CHDR dissector, but I think it's telling us the data is "2" (if we read this in network order).
Put these together, and we're looking at a simple overrun. Something in your chain is holding up the radio after a few packets. Are you sure you're not seeing an "O" anywhere in your output? You are using a radio block, right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
Hi,
Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com mailto:bpadalino@gmail.com ' <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Cc: 'martin.braun@ettus.com mailto:martin.braun@ettus.com ' <martin.braun@ettus.com mailto:martin.braun@ettus.com >; 'usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com ' <usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com >; Werner Bode <werner.bode@vastech.co.za mailto:werner.bode@vastech.co.za >
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Hi Brian,
I’ve got two observations:
- This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable:
No. Time Source Destination Protocol Length Info
1 0.000000000 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1 UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I guess.>
7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<first seq=0 rfnoc data packet of the correct size given my tlast counter>7319 50.792748665 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8
- ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming).
Regards, Kevin
From: Brian Padalino <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: martin.braun@ettus.com mailto:martin.braun@ettus.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0}
-
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 }
-
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and it's working well.
In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 https://url.za.m.mimecastprotect.com/s/q2ekCvg5vviLwR9tQf3iQubI-?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're providing enable_fc as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up.
Good luck.
Brian
USRP-users mailing list -- usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com mailto:usrp-users-leave@lists.ettus.com
Ah, yes -- that is, in fact, the entire purpose of the CE clock. If you're
doing sample processing, always use that clock. On all devices, we provide
that clock in a way that will let you do sample processing fast enough.
--M
On Tue, Jul 29, 2025 at 1:03 PM Kevin Williams kevin.williams@vastech.co.za
wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to
clock my block) slower than the radio clock, whereas with my previous n300
that clock is faster..!
I need to create many output channels from my block now, so I think I will
just ignore handshaking, and design on the basis of the radio streaming
continuously.
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Normally flow control is the thing that will let the radio stall, but
maybe it's something else. From what I can see, there's two potential
culprits: 1) Your block is not permanently processing samples, but has some
bubble cycles or something like that. 2) The SEP->SEP connection has an
issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think that
should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host, not
the other way around. This means it's an async message from an RFNoC block,
with address 0x1000. I can't tell for sure from this screenshot, but I
think this is coming from the radio, and 0x1000 is the "RX Error" address.
The data is incorrectly formatted (probably an issue of the CHDR dissector,
but I think it's telling us the data is "2" (if we read this in network
order).
Put these together, and we're looking at a simple overrun. Something in
your chain is holding up the radio after a few packets. Are you sure you're
not seeing an "O" anywhere in your output? You are using a radio block,
right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi,
Another observation is the every time the streaming stalls, whether remote
streaming or normal rx_streamer operation, I see this packet from the host
to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; '
usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
Hi Brian,
I’ve got two observations:
1. This is a summary of my custom block streaming where the data
packet stream ends with icmp packets about the destination becoming
unreachable:
No. Time Source Destination Protocol
Length Info
1 0.000000000 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1
UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I
guess.>
7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
7319 50.792748665 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc
data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da
ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate
use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e
ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use
of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255 UDP
50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255 UDP
50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255 UDP
50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255 UDP
50 59572 → 1534 Len=8
2. ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at
exactly the correct sample counts, until streaming stops where I see TREADY
deasserted for 20-odd clocks, and then reasserted (without further
streaming).
Regards, Kevin
From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport:
in0} -
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 }
-
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and
it's working well.
In fact, the way remote streaming works (at least for an X440) is that the
Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671
https://url.za.m.mimecastprotect.com/s/H9mfCKOByytM7xBsMfri5-4wa?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.
Good luck.
Brian
So you have 2 interfaces using the same IP...
I imagine you want them to load balance.
Try assigning both interfaces to the same virtual IP.
Then linux kernel will balance all in/out traffic from these interfaces:)
BR
Nikos
On Tue, Jul 29, 2025 at 4:26 PM Martin Braun martin.braun@ettus.com wrote:
Ah, yes -- that is, in fact, the entire purpose of the CE clock. If you're
doing sample processing, always use that clock. On all devices, we provide
that clock in a way that will let you do sample processing fast enough.
--M
On Tue, Jul 29, 2025 at 1:03 PM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to
clock my block) slower than the radio clock, whereas with my previous n300
that clock is faster..!
I need to create many output channels from my block now, so I think I
will just ignore handshaking, and design on the basis of the radio
streaming continuously.
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Normally flow control is the thing that will let the radio stall, but
maybe it's something else. From what I can see, there's two potential
culprits: 1) Your block is not permanently processing samples, but has some
bubble cycles or something like that. 2) The SEP->SEP connection has an
issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think
that should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host,
not the other way around. This means it's an async message from an RFNoC
block, with address 0x1000. I can't tell for sure from this screenshot, but
I think this is coming from the radio, and 0x1000 is the "RX Error"
address. The data is incorrectly formatted (probably an issue of the CHDR
dissector, but I think it's telling us the data is "2" (if we read this in
network order).
Put these together, and we're looking at a simple overrun. Something in
your chain is holding up the radio after a few packets. Are you sure you're
not seeing an "O" anywhere in your output? You are using a radio block,
right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi,
Another observation is the every time the streaming stalls, whether
remote streaming or normal rx_streamer operation, I see this packet from
the host to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; '
usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
Hi Brian,
I’ve got two observations:
1. This is a summary of my custom block streaming where the data
packet stream ends with icmp packets about the destination becoming
unreachable:
No. Time Source Destination Protocol
Length Info
1 0.000000000 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1
UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I
guess.>
7318 50.792688865 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<first seq=0 rfnoc data packet of the correct size given my tlast counter
7319 50.792748665 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc
data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da
ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate
use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e
ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use
of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255
UDP 50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255
UDP 50 59572 → 1534 Len=8
2. ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at
exactly the correct sample counts, until streaming stops where I see TREADY
deasserted for 20-odd clocks, and then reasserted (without further
streaming).
Regards, Kevin
From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport:
in0} -
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0
} -
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and
it's working well.
In fact, the way remote streaming works (at least for an X440) is that
the Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671
https://url.za.m.mimecastprotect.com/s/H9mfCKOByytM7xBsMfri5-4wa?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.
Good luck.
Brian
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
That’s an interesting concept, but I need the USRP to balance its output streams between the two 10 gbe nic’s.
I don’t think I can do that from the host side?
From: Nikos Balkanas nbalkanas@gmail.com
Sent: Tuesday, 29 July 2025 15:36
To: Martin Braun martin.braun@ettus.com
Cc: Kevin Williams kevin.williams@vastech.co.za; usrp-users@lists.ettus.com; Werner Bode werner.bode@vastech.co.za
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
So you have 2 interfaces using the same IP...
I imagine you want them to load balance.
Try assigning both interfaces to the same virtual IP.
Then linux kernel will balance all in/out traffic from these interfaces:)
BR
Nikos
On Tue, Jul 29, 2025 at 4:26 PM Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com > wrote:
Ah, yes -- that is, in fact, the entire purpose of the CE clock. If you're doing sample processing, always use that clock. On all devices, we provide that clock in a way that will let you do sample processing fast enough.
--M
On Tue, Jul 29, 2025 at 1:03 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to clock my block) slower than the radio clock, whereas with my previous n300 that clock is faster..!
I need to create many output channels from my block now, so I think I will just ignore handshaking, and design on the basis of the radio streaming continuously.
From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Normally flow control is the thing that will let the radio stall, but maybe it's something else. From what I can see, there's two potential culprits: 1) Your block is not permanently processing samples, but has some bubble cycles or something like that. 2) The SEP->SEP connection has an issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think that should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host, not the other way around. This means it's an async message from an RFNoC block, with address 0x1000. I can't tell for sure from this screenshot, but I think this is coming from the radio, and 0x1000 is the "RX Error" address. The data is incorrectly formatted (probably an issue of the CHDR dissector, but I think it's telling us the data is "2" (if we read this in network order).
Put these together, and we're looking at a simple overrun. Something in your chain is holding up the radio after a few packets. Are you sure you're not seeing an "O" anywhere in your output? You are using a radio block, right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
Hi,
Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com mailto:bpadalino@gmail.com ' <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Cc: 'martin.braun@ettus.com mailto:martin.braun@ettus.com ' <martin.braun@ettus.com mailto:martin.braun@ettus.com >; 'usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com ' <usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com >; Werner Bode <werner.bode@vastech.co.za mailto:werner.bode@vastech.co.za >
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Hi Brian,
I’ve got two observations:
- This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable:
No. Time Source Destination Protocol Length Info
1 0.000000000 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1 UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I guess.>
7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<first seq=0 rfnoc data packet of the correct size given my tlast counter>7319 50.792748665 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8
- ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming).
Regards, Kevin
From: Brian Padalino <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: martin.braun@ettus.com mailto:martin.braun@ettus.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0}
-
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 }
-
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and it's working well.
In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 https://url.za.m.mimecastprotect.com/s/g1I3CMjgAASkBYOhwfqi8MeC3?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're providing enable_fc as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up.
Good luck.
Brian
USRP-users mailing list -- usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com mailto:usrp-users-leave@lists.ettus.com
Ooh. No problem.
USRP will just see the single virtual host IP and send everything there
using both its cables.
Host will balance incoming data between its 2 NICS.
Host will balance sending data to USRP 2 cables.
I imagine that USRP can handle data arriving at its 2 ports simultaneously
(merging streams?)
So only 1side load balancing is needed.
Either on USRP or on Host. Not both:)
Using load balance on both would be wrong!
HTH
Nikos
On Tue, Jul 29, 2025 at 4:58 PM Kevin Williams kevin.williams@vastech.co.za
wrote:
That’s an interesting concept, but I need the USRP to balance its output
streams between the two 10 gbe nic’s.
I don’t think I can do that from the host side?
From: Nikos Balkanas nbalkanas@gmail.com
Sent: Tuesday, 29 July 2025 15:36
To: Martin Braun martin.braun@ettus.com
Cc: Kevin Williams kevin.williams@vastech.co.za;
usrp-users@lists.ettus.com; Werner Bode werner.bode@vastech.co.za
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
So you have 2 interfaces using the same IP...
I imagine you want them to load balance.
Try assigning both interfaces to the same virtual IP.
Then linux kernel will balance all in/out traffic from these interfaces:)
BR
Nikos
On Tue, Jul 29, 2025 at 4:26 PM Martin Braun martin.braun@ettus.com
wrote:
Ah, yes -- that is, in fact, the entire purpose of the CE clock. If you're
doing sample processing, always use that clock. On all devices, we provide
that clock in a way that will let you do sample processing fast enough.
--M
On Tue, Jul 29, 2025 at 1:03 PM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to
clock my block) slower than the radio clock, whereas with my previous n300
that clock is faster..!
I need to create many output channels from my block now, so I think I will
just ignore handshaking, and design on the basis of the radio streaming
continuously.
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Normally flow control is the thing that will let the radio stall, but
maybe it's something else. From what I can see, there's two potential
culprits: 1) Your block is not permanently processing samples, but has some
bubble cycles or something like that. 2) The SEP->SEP connection has an
issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think that
should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host, not
the other way around. This means it's an async message from an RFNoC block,
with address 0x1000. I can't tell for sure from this screenshot, but I
think this is coming from the radio, and 0x1000 is the "RX Error" address.
The data is incorrectly formatted (probably an issue of the CHDR dissector,
but I think it's telling us the data is "2" (if we read this in network
order).
Put these together, and we're looking at a simple overrun. Something in
your chain is holding up the radio after a few packets. Are you sure you're
not seeing an "O" anywhere in your output? You are using a radio block,
right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi,
Another observation is the every time the streaming stalls, whether remote
streaming or normal rx_streamer operation, I see this packet from the host
to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; '
usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
Hi Brian,
I’ve got two observations:
1. This is a summary of my custom block streaming where the data
packet stream ends with icmp packets about the destination becoming
unreachable:
No. Time Source Destination Protocol
Length Info
1 0.000000000 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1
UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I
guess.>
7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
7319 50.792748665 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC
4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc
data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da
ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate
use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e
ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use
of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255 UDP
50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255 UDP
50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255 UDP
50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255 UDP
50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255 UDP
50 59572 → 1534 Len=8
2. ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at
exactly the correct sample counts, until streaming stops where I see TREADY
deasserted for 20-odd clocks, and then reasserted (without further
streaming).
Regards, Kevin
From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport:
in0} -
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 }
-
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and
it's working well.
In fact, the way remote streaming works (at least for an X440) is that the
Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671
https://url.za.m.mimecastprotect.com/s/g1I3CMjgAASkBYOhwfqi8MeC3?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.
Good luck.
Brian
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Basically you are right. Both need to load balance.
Since your USRP already has 2 ports, it must already balance between them:)
Right now you only have 1/2 the balance.
To complete it you also need your host LB:)
HTH
Nikos
On Tue, Jul 29, 2025 at 5:48 PM Nikos Balkanas nbalkanas@gmail.com wrote:
Ooh. No problem.
USRP will just see the single virtual host IP and send everything there
using both its cables.
Host will balance incoming data between its 2 NICS.
Host will balance sending data to USRP 2 cables.
I imagine that USRP can handle data arriving at its 2 ports simultaneously
(merging streams?)
So only 1side load balancing is needed.
Either on USRP or on Host. Not both:)
Using load balance on both would be wrong!
HTH
Nikos
On Tue, Jul 29, 2025 at 4:58 PM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
That’s an interesting concept, but I need the USRP to balance its output
streams between the two 10 gbe nic’s.
I don’t think I can do that from the host side?
From: Nikos Balkanas nbalkanas@gmail.com
Sent: Tuesday, 29 July 2025 15:36
To: Martin Braun martin.braun@ettus.com
Cc: Kevin Williams kevin.williams@vastech.co.za;
usrp-users@lists.ettus.com; Werner Bode werner.bode@vastech.co.za
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
So you have 2 interfaces using the same IP...
I imagine you want them to load balance.
Try assigning both interfaces to the same virtual IP.
Then linux kernel will balance all in/out traffic from these interfaces:)
BR
Nikos
On Tue, Jul 29, 2025 at 4:26 PM Martin Braun martin.braun@ettus.com
wrote:
Ah, yes -- that is, in fact, the entire purpose of the CE clock. If
you're doing sample processing, always use that clock. On all devices, we
provide that clock in a way that will let you do sample processing fast
enough.
--M
On Tue, Jul 29, 2025 at 1:03 PM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to
clock my block) slower than the radio clock, whereas with my previous n300
that clock is faster..!
I need to create many output channels from my block now, so I think I
will just ignore handshaking, and design on the basis of the radio
streaming continuously.
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Normally flow control is the thing that will let the radio stall, but
maybe it's something else. From what I can see, there's two potential
culprits: 1) Your block is not permanently processing samples, but has some
bubble cycles or something like that. 2) The SEP->SEP connection has an
issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think
that should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun martin.braun@ettus.com
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but
stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host,
not the other way around. This means it's an async message from an RFNoC
block, with address 0x1000. I can't tell for sure from this screenshot, but
I think this is coming from the radio, and 0x1000 is the "RX Error"
address. The data is incorrectly formatted (probably an issue of the CHDR
dissector, but I think it's telling us the data is "2" (if we read this in
network order).
Put these together, and we're looking at a simple overrun. Something in
your chain is holding up the radio after a few packets. Are you sure you're
not seeing an "O" anywhere in your output? You are using a radio block,
right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
Hi,
Another observation is the every time the streaming stalls, whether
remote streaming or normal rx_streamer operation, I see this packet from
the host to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com' bpadalino@gmail.com
Cc: 'martin.braun@ettus.com' martin.braun@ettus.com; '
usrp-users@lists.ettus.com' usrp-users@lists.ettus.com; Werner Bode <
werner.bode@vastech.co.za>
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
Hi Brian,
I’ve got two observations:
1. This is a summary of my custom block streaming where the data
packet stream ends with icmp packets about the destination becoming
unreachable:
No. Time Source Destination Protocol
Length Info
1 0.000000000 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1
UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I
guess.>
7318 50.792688865 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<first seq=0 rfnoc data packet of the correct size given my tlast counter
7319 50.792748665 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1
RFNOC 4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc
data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast
ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da
ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate
use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e
ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use
of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3
ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255
UDP 50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255
UDP 50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast
ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast
ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255
UDP 50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255
UDP 50 59572 → 1534 Len=8
2. ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at
exactly the correct sample counts, until streaming stops where I see TREADY
deasserted for 20-odd clocks, and then reasserted (without further
streaming).
Regards, Kevin
From: Brian Padalino bpadalino@gmail.com
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts,
but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport:
in0} -
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0
} -
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could
start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and
it's working well.
In fact, the way remote streaming works (at least for an X440) is that
the Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671
https://url.za.m.mimecastprotect.com/s/g1I3CMjgAASkBYOhwfqi8MeC3?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet
information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be
something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're
providing enable_fc as an argument? That would enable flow control which
obviously wouldn't be good if you aren't doing any flow control processing
on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your
block and the SEP interfaces to see where the AXI stream is getting stopped
up.
Good luck.
Brian
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
This is strange.
I now assert tready permanently for the radio regardless. (I before used to pass downstream tready up).
Now I still see the stream stalling, but after a few 10’000 packets of both usrp ports being used for tx (now 4x streams being transmitted).
It then drops to only one port, and then stalls – even though this is remote streaming.
How is it possible that the host causes the usrp to abandon a nic in remote streaming mode when the radio is always seeing tready?
Just before the stall I still see the usrp writing an overflow packet.
How is this possible?
From: Nikos Balkanas nbalkanas@gmail.com
Sent: Tuesday, 29 July 2025 16:49
To: Kevin Williams kevin.williams@vastech.co.za
Cc: martin.braun@ettus.com; usrp-users@lists.ettus.com; Werner Bode werner.bode@vastech.co.za
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Ooh. No problem.
USRP will just see the single virtual host IP and send everything there using both its cables.
Host will balance incoming data between its 2 NICS.
Host will balance sending data to USRP 2 cables.
I imagine that USRP can handle data arriving at its 2 ports simultaneously (merging streams?)
So only 1side load balancing is needed.
Either on USRP or on Host. Not both:)
Using load balance on both would be wrong!
HTH
Nikos
On Tue, Jul 29, 2025 at 4:58 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
That’s an interesting concept, but I need the USRP to balance its output streams between the two 10 gbe nic’s.
I don’t think I can do that from the host side?
From: Nikos Balkanas <nbalkanas@gmail.com mailto:nbalkanas@gmail.com >
Sent: Tuesday, 29 July 2025 15:36
To: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Cc: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com ; Werner Bode <werner.bode@vastech.co.za mailto:werner.bode@vastech.co.za >
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
So you have 2 interfaces using the same IP...
I imagine you want them to load balance.
Try assigning both interfaces to the same virtual IP.
Then linux kernel will balance all in/out traffic from these interfaces:)
BR
Nikos
On Tue, Jul 29, 2025 at 4:26 PM Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com > wrote:
Ah, yes -- that is, in fact, the entire purpose of the CE clock. If you're doing sample processing, always use that clock. On all devices, we provide that clock in a way that will let you do sample processing fast enough.
--M
On Tue, Jul 29, 2025 at 1:03 PM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
The resolution is that the x310 has the rfnoc_chdr clock (which I used to clock my block) slower than the radio clock, whereas with my previous n300 that clock is faster..!
I need to create many output channels from my block now, so I think I will just ignore handshaking, and design on the basis of the radio streaming continuously.
From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Tuesday, 29 July 2025 10:01
Cc: usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Normally flow control is the thing that will let the radio stall, but maybe it's something else. From what I can see, there's two potential culprits: 1) Your block is not permanently processing samples, but has some bubble cycles or something like that. 2) The SEP->SEP connection has an issue.
If you can, connect everything statically and see how that fares.
--M
On Tue, Jul 29, 2025 at 9:52 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
Hi Martin,
I do see a single “O”, but this is remote streaming so I didn’t think that should occur?
Yes, this is a radio -> my custom block dynamic connection.
Regards, Kevin
From: Martin Braun <martin.braun@ettus.com mailto:martin.braun@ettus.com >
Sent: Tuesday, 29 July 2025 09:44
Cc: usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Kevin,
based on the src port, this looks like it's going from Device to Host, not the other way around. This means it's an async message from an RFNoC block, with address 0x1000. I can't tell for sure from this screenshot, but I think this is coming from the radio, and 0x1000 is the "RX Error" address. The data is incorrectly formatted (probably an issue of the CHDR dissector, but I think it's telling us the data is "2" (if we read this in network order).
Put these together, and we're looking at a simple overrun. Something in your chain is holding up the radio after a few packets. Are you sure you're not seeing an "O" anywhere in your output? You are using a radio block, right?
--M
On Tue, Jul 29, 2025 at 9:19 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
Hi,
Another observation is the every time the streaming stalls, whether remote streaming or normal rx_streamer operation, I see this packet from the host to the x310 a few data packets before it stops.
What is this control write address (0x01000), and is it perhaps relevant?
From: Kevin Williams
Sent: Tuesday, 29 July 2025 07:53
To: 'bpadalino@gmail.com mailto:bpadalino@gmail.com ' <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Cc: 'martin.braun@ettus.com mailto:martin.braun@ettus.com ' <martin.braun@ettus.com mailto:martin.braun@ettus.com >; 'usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com ' <usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com >; Werner Bode <werner.bode@vastech.co.za mailto:werner.bode@vastech.co.za >
Subject: RE: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
Hi Brian,
I’ve got two observations:
- This is a summary of my custom block streaming where the data packet stream ends with icmp packets about the destination becoming unreachable:
No. Time Source Destination Protocol Length Info
1 0.000000000 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
5343 49.277852197 10.22.128.3 10.23.128.1 UDP 60 49152 → 36716 Len=16
<5000-odd small udp and small rfnoc control & management packets. Setup I guess.>
7318 50.792688865 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<first seq=0 rfnoc data packet of the correct size given my tlast counter>7319 50.792748665 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7320 50.792754229 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<a few 100 more correct data packets>
7775 50.795514072 10.22.128.3 10.22.128.1 RFNOC 4146 [Data] -> 6
<a string of more control and short 66 byte rfnoc packets, but no rfnoc data packets>
7968 52.854255766 Intel_e8:c3:4c Broadcast ARP 42 Who has 10.22.128.1? Tell 10.23.128.1
7969 53.238261827 Intel_e8:c3:4e NationalInst_35:aa:da ARP 42 Who has 10.23.128.3? Tell 10.23.128.1 (duplicate use of 10.23.128.1 detected!)
7970 53.238475399 NationalInst_35:aa:da Intel_e8:c3:4e ARP 60 10.23.128.3 is at 00:80:2f:35:aa:da (duplicate use of 10.23.128.1 detected!)
<then the destination becomes unreachable?>
7971 53.878292746 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7972 53.878302721 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7973 53.878308143 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7974 53.878314734 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7975 53.878320545 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
7976 53.878326301 10.23.128.1 10.22.128.3 ICMP 590 Destination unreachable (Host unreachable)
<after that, just arp packets and the usrp broadcasting small udp packets>
8014 137.075344888 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3
8015 137.075304321 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3
8016 140.701925975 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8017 140.701942078 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8018 142.361983307 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8
8019 150.005535184 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
8020 150.005558707 10.23.128.1 10.23.128.255 UDP 50 1534 → 1534 Len=8
8021 152.097709946 NationalInst_35:aa:d9 Broadcast ARP 60 ARP Announcement for 10.22.128.3
8022 152.097809876 NationalInst_35:aa:da Broadcast ARP 60 ARP Announcement for 10.23.128.3
8023 155.702401576 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8024 155.702431967 10.23.128.1 10.23.128.255 UDP 50 38981 → 1534 Len=8
8025 157.378508296 10.23.128.1 10.23.128.255 UDP 50 59572 → 1534 Len=8
- ILA results
With my block I see a continuously asserted TREADY, with TLAST’s at exactly the correct sample counts, until streaming stops where I see TREADY deasserted for 20-odd clocks, and then reasserted (without further streaming).
Regards, Kevin
From: Brian Padalino <bpadalino@gmail.com mailto:bpadalino@gmail.com >
Sent: Monday, 28 July 2025 16:49
To: Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za >
Cc: martin.braun@ettus.com mailto:martin.braun@ettus.com ; usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Re: [EXTERNAL]Re: remote streaming starts, but stops after a few packets received
On Mon, Jul 28, 2025 at 10:15 AM Kevin Williams <kevin.williams@vastech.co.za mailto:kevin.williams@vastech.co.za > wrote:
I did an experiment today with just this (Ettus blocks only):
connections:
-
{ srcblk: radio0, srcport: out_0, dstblk: ep0, dstport: in0}
-
{ srcblk: ep6, srcport: out0, dstblk: ddc0, dstport: in_0 }
-
{ srcblk: ddc0, srcport: out_0, dstblk: ep6, dstport: in0 }
Which did not work – the remote streaming stopped.
Changing the destination EP to a new one, ep7, worked again.
From the RFNoC 4 workshop slides I was under the impression blocks could start and end on the same SEP?
For what it's worth, I'm using remote streaming with a custom block and it's working well.
In fact, the way remote streaming works (at least for an X440) is that the Ethernet/UDP information is written here:
https://github.com/EttusResearch/uhd/blob/40403b7c00154e4559c47bd6dde924f092992d45/fpga/usrp3/lib/rfnoc/xport_sv/chdr_xport_adapter.sv#L671 https://url.za.m.mimecastprotect.com/s/XdtuCvg5vviLwnNHQf3iQ6lcn?domain=github.com
The kv_map uses the destination EPID as the key for the ethernet information which gets looked up for every packet.
So if the streaming works when not doing remote streaming it might be something else since all data paths go through here.
If you get the first few packets and it stops, is there any way you're providing enable_fc as an argument? That would enable flow control which obviously wouldn't be good if you aren't doing any flow control processing on your RX side.
Lastly, I agree with Martin that you should probably add an ILA to your block and the SEP interfaces to see where the AXI stream is getting stopped up.
Good luck.
Brian
USRP-users mailing list -- usrp-users@lists.ettus.com mailto:usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com mailto:usrp-users-leave@lists.ettus.com
On Tue, Jul 29, 2025 at 11:06 AM Kevin Williams <
kevin.williams@vastech.co.za> wrote:
This is strange.
I now assert tready permanently for the radio regardless. (I before used
to pass downstream tready up).
Now I still see the stream stalling, but after a few 10’000 packets of
both usrp ports being used for tx (now 4x streams being transmitted).
It then drops to only one port, and then stalls – even though this is
remote streaming.
How is it possible that the host causes the usrp to abandon a nic in
remote streaming mode when the radio is always seeing tready?
Just before the stall I still see the usrp writing an overflow packet.
How is this possible?
It's because of the radio RX FSM here:
When an error condition occurs, it reports the OVERFLOW then it waits for
another stream command to start up again.
For what it's worth, I don't think you should blindly give it TREADY
asserted when things can't process things downstream. The block should be
designed such that you know you can consume all the samples in real time.
Brian
