usrp-users@lists.ettus.com
Discussion and technical support related to USRP, UHD, RFNoC
View all threadsX410 benchmark_rate underrun at 50MSps
I am running into underflow issues when running the benchmark_rate test on a x410. The command I am running is the following:
./benchmark_rate --args="addr=192.168.20.2,type=x4xx,send_frame_size=8000,recv_frame_size=8000" --tx_rate=50e6 --tx_channels=0 --tx_delay=1 --duration=60 --priority=high --ref=internal
I have a 100Gbit interface connected to the x410 using the below ethernet card.
ca:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
I'm running on Ubuntu 22.04.5 LTS with kernel 6.8.0-40-generic.
I've confirmed that the priority=high setting is working. "top" shows the priorities of all the threads have been elevated.
I've also updated a number of network settings.
sysctl -w net.core.wmem_max=250000000
sysctl -w net.core.rmem_max=250000000
sysctl -w net.core.wmem_default=250000000
sysctl -w net.core.rmem_default=250000000
ethtool -G ens8f0np0 tx 8192 rx 8192
The application outputs the following:
[INFO] [UHD] linux; GNU C++ version 8.4.0; Boost_108300; UHD_4.6.0.HEAD-0-g50fa3baa
[00:00:00.000277] Creating the usrp device with: addr=192.168.20.2,type=x4xx,send_frame_size=8000,recv_frame_size=8000...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.20.2,type=x4xx,product=x410,serial=3289B1C,name=ni-x4xx-3289 B1C,fpga=UC_200,claimed=False,addr=192.168.20.2,send_frame_size=8000,recv_frame_size=8000
[INFO] [MPM.PeriphManager] init() called with device args `fpga=UC_200,mgmt_addr=192.168.20.2,name=ni-x4xx-3289B1C,product=x410,recv_frame_s ize=8000,send_frame_size=8000,clock_source=internal,time_source=internal,initializing=True'.
Using Device: Single USRP:
Device: X400-Series Device
Mboard 0: x410
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: 0
RX Channel: 1
RX DSP: 1
RX Dboard: A
RX Subdev: 1
RX Channel: 2
RX DSP: 2
RX Dboard: B
RX Subdev: 0
RX Channel: 3
RX DSP: 3
RX Dboard: B
RX Subdev: 1
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: 0
TX Channel: 1
TX DSP: 1
TX Dboard: A
TX Subdev: 1
TX Channel: 2
TX DSP: 2
TX Dboard: B
TX Subdev: 0
TX Channel: 3
TX DSP: 3
TX Dboard: B
TX Subdev: 1
[00:00:02.516914784] Setting device timestamp to 0...
[WARNING] [0/DUC#0] The requested interpolation is odd; the user should expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is 49.152 MHz
[WARNING] [0/DUC#0] The requested interpolation is odd; the user should expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is 49.152 MHz
[WARNING] [0/DUC#1] The requested interpolation is odd; the user should expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is 49.152 MHz
[WARNING] [0/DUC#1] The requested interpolation is odd; the user should expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is 49.152 MHz
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Setting TX spp to 1992
[00:00:02.522008533] Testing transmit rate 49.152000 Msps on 1 channels
UUUUUUUUUUUUUUUUUUUUUUU[00:01:03.524365256] Benchmark complete.
Benchmark rate summary:
Num received samples: 0
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2948209800
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 23
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0
Done!
Does anyone have any other suggestions for what could be tried to avoid the underruns? With a 100Gbit/s NIC, I should have plenty of network throughput.
Hi Jeremy,
This is a good reference of things you can try:
https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks
Also, I wonder if RT_RUNTIME_SHARE could be causing problems? You can find
some info here on how to enable it:
https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Underruns_Every_Second_with_DPDK_.2B_Ubuntu
Another thing that comes to mind is enabling pause frames on the X410 as
described here:
https://files.ettus.com/manual/page_transport.html#transport_udp_linux
Wade
On Tue, Jan 21, 2025 at 7:40 AM Pallotta, Jeremy M (US) <
jeremy.pallotta@baesystems.us> wrote:
I am running into underflow issues when running the benchmark_rate test on
a x410. The command I am running is the following:
./benchmark_rate
--args="addr=192.168.20.2,type=x4xx,send_frame_size=8000,recv_frame_size=8000"
--tx_rate=50e6 --tx_channels=0 --tx_delay=1 --duration=60 --priority=high
--ref=internal
I have a 100Gbit interface connected to the x410 using the below ethernet
card.
ca:00.0 Ethernet controller: Mellanox Technologies MT2910 Family
[ConnectX-7]
I'm running on Ubuntu 22.04.5 LTS with kernel 6.8.0-40-generic.
I've confirmed that the priority=high setting is working. "top" shows the
priorities of all the threads have been elevated.
I've also updated a number of network settings.
sysctl -w net.core.wmem_max=250000000
sysctl -w net.core.rmem_max=250000000
sysctl -w net.core.wmem_default=250000000
sysctl -w net.core.rmem_default=250000000
ethtool -G ens8f0np0 tx 8192 rx 8192
The application outputs the following:
[INFO] [UHD] linux; GNU C++ version 8.4.0; Boost_108300;
UHD_4.6.0.HEAD-0-g50fa3baa
[00:00:00.000277] Creating the usrp device with:
addr=192.168.20.2,type=x4xx,send_frame_size=8000,recv_frame_size=8000...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args:
mgmt_addr=192.168.20.2,type=x4xx,product=x410,serial=3289B1C,name=ni-x4xx-3289
B1C,fpga=UC_200,claimed=False,addr=192.168.20.2,send_frame_size=8000,recv_frame_size=8000
[INFO] [MPM.PeriphManager] init() called with device args
`fpga=UC_200,mgmt_addr=192.168.20.2,name=ni-x4xx-3289B1C,product=x410,recv_frame_s
ize=8000,send_frame_size=8000,clock_source=internal,time_source=internal,initializing=True'.
Using Device: Single USRP:
Device: X400-Series Device
Mboard 0: x410
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: 0
RX Channel: 1
RX DSP: 1
RX Dboard: A
RX Subdev: 1
RX Channel: 2
RX DSP: 2
RX Dboard: B
RX Subdev: 0
RX Channel: 3
RX DSP: 3
RX Dboard: B
RX Subdev: 1
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: 0
TX Channel: 1
TX DSP: 1
TX Dboard: A
TX Subdev: 1
TX Channel: 2
TX DSP: 2
TX Dboard: B
TX Subdev: 0
TX Channel: 3
TX DSP: 3
TX Dboard: B
TX Subdev: 1
[00:00:02.516914784] Setting device timestamp to 0...
[WARNING] [0/DUC#0] The requested interpolation is odd; the user should
expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is
49.152 MHz
[WARNING] [0/DUC#0] The requested interpolation is odd; the user should
expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is
49.152 MHz
[WARNING] [0/DUC#1] The requested interpolation is odd; the user should
expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is
49.152 MHz
[WARNING] [0/DUC#1] The requested interpolation is odd; the user should
expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is
49.152 MHz
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Setting TX spp to 1992
[00:00:02.522008533] Testing transmit rate 49.152000 Msps on 1 channels
UUUUUUUUUUUUUUUUUUUUUUU[00:01:03.524365256] Benchmark complete.
Benchmark rate summary:
Num received samples: 0
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2948209800
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 23
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0
Done!
Does anyone have any other suggestions for what could be tried to avoid
the underruns? With a 100Gbit/s NIC, I should have plenty of network
throughput.
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Wade,
Thank you for the links. I was able to resolve the issue on my Ubuntu/X410 system by setting the CPU governor to “performance” mode.
for ((i=0;i<$(nproc --all);i++)); do sudo cpufreq-set -c $i -r -g performance; done
I have a similar “underrun” issue on a CentOS 7.6.1810 (kernel 3.10.0-957) connected to a USRP X310 using UHD 4.6.0. The network card is an Intel X710 with 10Gbit ethernet connection to USRP.
On this system, I’ve disabled hyperthreading, cpu scaling, and virtualization via the BIOS. I’ve confirmed that I can run the benchmark test with elevated thread priorities and nice.
I’ve configured the following network settings.
sysctl -w net.core.wmem_max=33554432
sysctl -w net.core.rmem_max=33554432
sysctl -w net.core.wmem_default=33554432
sysctl -w net.core.rmem_default=33554432
ethtool -G ens8f0np0 tx 4096 rx 4096
ethtool -A ens8f0np0 tx on
When I run the benchmark_rate test for 1 channel at 50MSps with priority=high, I get a consistent underrun at about 1Hz. This is similar to the issue described in the 2nd link you provided.
Finally, I’ve confirmed that I can run the benchmark_rate test successfully (no underruns) with UHD version 3.14.1.
Is there an issue with a newer UHD (4.6.0) and an older OS (CentOS 7.6)? Are there any optimizations that are perhaps different on the older CentOS-based system?
Jeremy
From: Wade Fife wade.fife@ettus.com
Sent: Tuesday, January 21, 2025 11:08 PM
To: Pallotta, Jeremy M (US) jeremy.pallotta@baesystems.us
Cc: usrp-users@lists.ettus.com
Subject: Re: [USRP-users] X410 benchmark_rate underrun at 50MSps
External Email Alert
This email has been sent from an account outside of the BAE Systems network.
Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar.
Hi Jeremy,
This is a good reference of things you can try:
https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks
Also, I wonder if RT_RUNTIME_SHARE could be causing problems? You can find some info here on how to enable it:
https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD#Underruns_Every_Second_with_DPDK_.2B_Ubuntu
Another thing that comes to mind is enabling pause frames on the X410 as described here:
https://files.ettus.com/manual/page_transport.html#transport_udp_linux
Wade
On Tue, Jan 21, 2025 at 7:40 AM Pallotta, Jeremy M (US) <jeremy.pallotta@baesystems.usmailto:jeremy.pallotta@baesystems.us> wrote:
I am running into underflow issues when running the benchmark_rate test on a x410. The command I am running is the following:
./benchmark_rate --args="addr=192.168.20.2,type=x4xx,send_frame_size=8000,recv_frame_size=8000" --tx_rate=50e6 --tx_channels=0 --tx_delay=1 --duration=60 --priority=high --ref=internal
I have a 100Gbit interface connected to the x410 using the below ethernet card.
ca:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
I'm running on Ubuntu 22.04.5 LTS with kernel 6.8.0-40-generic.
I've confirmed that the priority=high setting is working. "top" shows the priorities of all the threads have been elevated.
I've also updated a number of network settings.
sysctl -w net.core.wmem_max=250000000
sysctl -w net.core.rmem_max=250000000
sysctl -w net.core.wmem_default=250000000
sysctl -w net.core.rmem_default=250000000
ethtool -G ens8f0np0 tx 8192 rx 8192
The application outputs the following:
[INFO] [UHD] linux; GNU C++ version 8.4.0; Boost_108300; UHD_4.6.0.HEAD-0-g50fa3baa
[00:00:00.000277] Creating the usrp device with: addr=192.168.20.2,type=x4xx,send_frame_size=8000,recv_frame_size=8000...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.20.2,type=x4xx,product=x410,serial=3289B1C,name=ni-x4xx-3289 B1C,fpga=UC_200,claimed=False,addr=192.168.20.2,send_frame_size=8000,recv_frame_size=8000
[INFO] [MPM.PeriphManager] init() called with device args `fpga=UC_200,mgmt_addr=192.168.20.2,name=ni-x4xx-3289B1C,product=x410,recv_frame_s ize=8000,send_frame_size=8000,clock_source=internal,time_source=internal,initializing=True'.
Using Device: Single USRP:
Device: X400-Series Device
Mboard 0: x410
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: 0
RX Channel: 1
RX DSP: 1
RX Dboard: A
RX Subdev: 1
RX Channel: 2
RX DSP: 2
RX Dboard: B
RX Subdev: 0
RX Channel: 3
RX DSP: 3
RX Dboard: B
RX Subdev: 1
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: 0
TX Channel: 1
TX DSP: 1
TX Dboard: A
TX Subdev: 1
TX Channel: 2
TX DSP: 2
TX Dboard: B
TX Subdev: 0
TX Channel: 3
TX DSP: 3
TX Dboard: B
TX Subdev: 1
[00:00:02.516914784] Setting device timestamp to 0...
[WARNING] [0/DUC#0] The requested interpolation is odd; the user should expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is 49.152 MHz
[WARNING] [0/DUC#0] The requested interpolation is odd; the user should expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is 49.152 MHz
[WARNING] [0/DUC#1] The requested interpolation is odd; the user should expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is 49.152 MHz
[WARNING] [0/DUC#1] The requested interpolation is odd; the user should expect passband CIC rolloff.
Select an even interpolation to ensure that a halfband filter is enabled.
[WARNING] [MULTI_USRP] Could not set TX rate to 50.000 MHz. Actual rate is 49.152 MHz
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Setting TX spp to 1992
[00:00:02.522008533] Testing transmit rate 49.152000 Msps on 1 channels
UUUUUUUUUUUUUUUUUUUUUUU[00:01:03.524365256] Benchmark complete.
Benchmark rate summary:
Num received samples: 0
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 2948209800
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 23
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0
Done!
Does anyone have any other suggestions for what could be tried to avoid the underruns? With a 100Gbit/s NIC, I should have plenty of network throughput.
USRP-users mailing list -- usrp-users@lists.ettus.commailto:usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.commailto:usrp-users-leave@lists.ettus.com