Discussion and technical support related to USRP, UHD, RFNoC
View all threadsHello!
I want to get DPDK working with UHD 4.6 on our USRP X410. This might be a long post, but I want to make sure I provide as much information as possible. Below is a diagram of relevant network connections:
*---------------------* *---------------------*
| | NIC QSFP Port 0 | | QSFP28 Port 1 | |
| | ens1f0np0 <>---------------<> sfp1 | |
| | 192.168.20.1 | | 192.168.20.2 | |
| |-----------------| |-----------------| |
| H | NIC QSFP Port 1 | | QSFP28 Port 0 | X |
| O | ens1f1np1 <>---------------<> sfp0 | 4 |
| S | 192.168.10.1 | | 192.168.10.2 | 1 |
| T |-----------------| *-----------* |-----------------| 0 |
| | Ethernet Port 1 | |Router | | Ethernet Port | |
| | eno8403 <>-|192.168.1.1|-<> eth0 | |
| | 192.168.1.20 | | | | 192.168.1.19 | |
*---------------------* *-----------* *---------------------*
The Host computer has the following specs:
OS: Ubuntu 20.04.6 LTS
CPU: Intel(R) Xeon(R) Silver 4310
RAM: 64 GB
NIC: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
UHD: UHD_4.6.0.HEAD-0-g50fa3baa
DPDK: DPDK_19.11
I am using a uhd.conf with:
[use_dpdk=1]
dpdk_mtu=9000
dpdk_corelist=0,1,2,3
dpdk_num_mbufs=8192
dpdk_mbuf_cache_size=64
[dpdk_mac=MAC_ADDRESS]
dpdk_lcore = 1
dpdk_ipv4 = 192.168.20.1/24
[dpdk_mac=MAC_ADDRESS]
dpdk_lcore = 2
dpdk_ipv4 = 192.168.10.1/24
When I run “dpdk-devbind.py -s“, I get:
Network devices using kernel driver
---==
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8303 drv=tg3 unused=vfio-pci *Active*
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8403 drv=tg3 unused=vfio-pci *Active*
0000:31:00.0 'MT2892 Family [ConnectX-6 Dx] 101d' if=ens1f0np0 drv=mlx5_core unused=vfio-pci *Active*
0000:31:00.1 'MT2892 Family [ConnectX-6 Dx] 101d' if=ens1f1np1 drv=mlx5_core unused=vfio-pci *Active*
And the output of uhd_find_devices is:
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
--------------------------------------------------
-- UHD Device 0
--------------------------------------------------
Device Address:
serial: 326E872
addr: 192.168.20.2
claimed: False
fpga: CG_400
mgmt_addr: 192.168.1.19
mgmt_addr: 192.168.10.2
mgmt_addr: 192.168.20.2
name: ni-x4xx-326E872
product: x410
type: x4xx
This all makes me believe I at least have DPDK set up correctly. I followed the instructions here (https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config) for GRUB.
When I try to run any UHD-based script with DPDK, like ‘ uhd_find_devices --args "use_dpdk=1" ‘, I get errors like:
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
[ERROR] [DPDK] Error with EAL initialization
And the device is not found. After some searching, I found that the reason is DPDK doesn’t play nicely without elevated permissions. Going forward, assume I run everything as the root user. Running the same command as before then yields:
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:31:00.0 on NUMA socket 0
EAL: probe driver: 15b3:101d net_mlx5
EAL: PCI device 0000:31:00.1 on NUMA socket 0
EAL: probe driver: 15b3:101d net_mlx5
[ERROR] [DPDK] Could not find route to destination address 10.149.11.255
[ERROR] [X300] X300 Network discovery error RuntimeError: DPDK: Could not find route to destination address 10.149.11.255
[ERROR] [DPDK] Could not find route to destination address 192.168.1.255
[ERROR] [X300] X300 Network discovery error RuntimeError: DPDK: Could not find route to destination address 192.168.1.255
[ERROR] [DPDK] Could not find route to destination address 172.17.255.255
[ERROR] [X300] X300 Network discovery error RuntimeError: DPDK: Could not find route to destination address 172.17.255.255
--------------------------------------------------
-- UHD Device 0
--------------------------------------------------
Device Address:
serial: 326E872
addr: 192.168.1.19
claimed: False
fpga: CG_400
mgmt_addr: 192.168.1.19
name: ni-x4xx-326E872
product: x410
type: x4xx
Now the device is found, but with errors. When I further specify all the addresses with:
uhd_find_devices --args “addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1"
The errors are gone.
When I try running an example with DPDK though, like:
./benchmark_rate --rx_rate 491.52e6 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1" --duration 1
I get:
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:31:00.0 on NUMA socket 0
EAL: probe driver: 15b3:101d net_mlx5
EAL: PCI device 0000:31:00.1 on NUMA socket 0
EAL: probe driver: 15b3:101d net_mlx5
[00:00:00.001242] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.19,type=x4xx,product=x410,serial=326E872,name=ni-x4xx-326E872,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args `fpga=CG_400,mgmt_addr=192.168.1.19,name=ni-x4xx-326E872,product=x410,second_addr=192.168.20.2,use_dpdk=1,clock_source=internal,time_source=internal,initializing=True'.
[ERROR] [RFNOC::MGMT] EnvironmentError: IOError: Timed out getting recv buff for management transaction
[ERROR] [RFNOC::GRAPH] IO Error during GSM initialization. EnvironmentError: IOError: Timed out getting recv buff for management transaction
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: EnvironmentError: IOError: Timed out getting recv buff for management transaction
Error: RuntimeError: Failure to create rfnoc_graph.
What is the issue? How do I fix things? When I first set up DPDK, I also set up HugePages, although I don't exactly remember how. As I only have a basic understanding of Linux, I'm not sure I set that up correctly, but running:
cat /proc/meminfo | grep Huge
results in:
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 2048
HugePages_Free: 2045
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 4194304 kB
Hi zackkomo
You should check the HugePages settings.
cat /etc/default/grub
Thanks.
Hi Mikio,
Below are the contents of /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048 pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier"
GRUB_CMDLINE_LINUX=""
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true
# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"
# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
I’m assuming you want to check the GRUB_CMDLINE_LINUX_DEFAULT? Is hugepages=2048 correct for hugepages?
Thanks!
Zack
Hi Zack
your /etc/default/grub is Ok.
This problem may be caused by a NIC device driver. Can you upgrade the DPDK
version?
I use Ubuntu 22 and UHD 4.4 and DPDK 21.
Regards,
2024年3月14日(木) 23:55 zackkomo@utexas.edu:
Hi Mikio,
Below are the contents of /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=lsb_release -i -s 2> /dev/null || echo Debian
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on hugepages=2048 pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier"
GRUB_CMDLINE_LINUX=""
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
#GRUB_TERMINAL=console
#GRUB_GFXMODE=640x480
#GRUB_DISABLE_LINUX_UUID=true
#GRUB_DISABLE_RECOVERY="true"
#GRUB_INIT_TUNE="480 440 1"
I’m assuming you want to check the GRUB_CMDLINE_LINUX_DEFAULT? Is
hugepages=2048 correct for hugepages?
Thanks!
Zack
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
--
---=
福島 幹雄 (Mikio Fukushima)
株式会社ドルフィンシステム (Dolphin System Co.,Ltd)
〒170-0014 東京都豊島区池袋1-8-5
ミントロンビル5F
Mail: mikio@dolphinsystem.jp
URL : http://www.dolphinsystem.jp/
TEL/FAX : 03-6658-4949
---=