usrp-users@lists.ettus.com
Discussion and technical support related to USRP, UHD, RFNoC
View all threadsExtending duration of high-rate captures with the X410
Hello!
I am having trouble using a USRP X410 to capture some high-rate data. I would appreciate any insight or help. Below is a diagram of how our x410 is connected to our host device.
*---------------------* *---------------------*
| | NIC QSFP Port 0 | | QSFP28 Port 1 | |
| | ens1f0np0 <>---------------<> sfp1 | |
| | 192.168.20.1 | | 192.168.20.2 | |
| |-----------------| |-----------------| |
| H | NIC QSFP Port 1 | | QSFP28 Port 0 | X |
| O | ens1f1np1 <>---------------<> sfp0 | 4 |
| S | 192.168.10.1 | | 192.168.10.2 | 1 |
| T |-----------------| *-----------* |-----------------| 0 |
| | Ethernet Port 1 | |Router | | Ethernet Port | |
| | eno8403 <>-|192.168.1.1|-<> eth0 | |
| | 192.168.1.20 | | | | 192.168.1.19 | |
*---------------------* *-----------* *---------------------*
When I run $ uhd_find_devices
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
--------------------------------------------------
-- UHD Device 0
--------------------------------------------------
Device Address:
serial: [serial number]
addr: 192.168.20.2
claimed: False
fpga: CG_400
mgmt_addr: 192.168.1.19
mgmt_addr: 192.168.10.2
mgmt_addr: 192.168.20.2
name: [name]
product: x410
type: x4xx
I am using two images for the FPGA, each of which has different capabilities:
- UC_200, which on the wiki lists Port 0 as "unused" and can achieve up to 250 Msps per channel.
- CG_400, which lists both ports on the wiki as "100 GbE" and can achieve up to 500 Msps per channel.
I am using UHD's C++ API to instruct the x410. My capture process tries to call "recv" as soon as possible without waiting to write the previous buffer to a file. The psudocode is as follows:
- Initialize "circular" buffer of arrays. Each array is "spb" samples long, and there are N such buffers.
- Start a "writing" thread that writes data from a queue to a file.
- Set the number of samples I need (nsamps_requested), set the current number of samples to 0 (nsamps),
- set the current buffer index to 0 (buff_idx)
- set up rx_streamer to start 0.1 seconds in the future and issue STREAM_MODE_START_CONTINUOUS command.
While (nsamps <= nsamps_requested || not_error)
-Get a pointer to the buffer at buff_idx
-num_rx_samps <- recv(buffer...)
-check metadata returned object
-add the buffer to the queue, increment the buffer index
- stop streaming
When I use the process above, I can get very long captures using the UC_200 image, with 250 Msps, and by parsing the args "addr=192.168.20.2,mgmt_addr=192.168.1.19" when creating the multiusrp pointer.
When I switch to the CG_400 image, I can barely record a few seconds worth of data. I parse the args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19". I changed the logger level to get a better idea of what might be going on, below are some outputs for spb at 5000:
[DEBUG] [RFNOC::MGMT] Established a route from EPID=4 (SW) to EPID=3
[DEBUG] [RFNOC::MGMT] Throttling stream endpoint to 100% (0x0)
[DEBUG] [RFNOC::MGMT] Initiated RX stream setup for EPID=3
[DEBUG] [RFNOC::MGMT] Finished RX stream setup for EPID=3
[DEBUG] [0/Radio#0] spp value 2032 exceeds MTU of 8016! Coercing to 1988
....
[DEBUG] [RxStreamer#0] Received overrun message on port 0
After the last logger message, I catc this in my "check metadata returned object" step and stop the loop.
When this happens, I sometimes get 0 samples back, and sometimes get less than the samples per buffer I feed the recv command. I also found that while waiting for the issued command to take effect, instead of adding empty buffers to my queue, I check for ERROR_CODE_TIMEOUT and just continue the while loop.
I would imagine if I can handle half the rate at 250 Msps for minutes of continuous capture, going to 500 Msps I could get more than a couple seconds. Is this because the UC_200 image has 4GiB of DRAM and the CG_400 doesn't list any on the wiki? Is there something better I can be doing about the spb (I tried larger and smaller, with some luck around 50000 spb)? Am I properly configuring the x410 to use both QSFP28 ports for the CG_400 image?
Thanks!
Zack
Hi Zack,
I don't know the answers to your issues, but I'm wondering if you can just
use benchmark_rate rather than your custom code to evaluate performance. If
benchmark_rate can't run at the rate you want, it is unlikely that your
custom code will succeed.
Rob
On Mon, Mar 11, 2024 at 2:35 PM zackkomo@utexas.edu wrote:
Hello!
I am having trouble using a USRP X410 to capture some high-rate data. I
would appreciate any insight or help. Below is a diagram of how our x410 is
connected to our host device.
--------------------- ---------------------
| | NIC QSFP Port 0 | | QSFP28 Port 1 | |
| | ens1f0np0 <>---------------<> sfp1 | |
| | 192.168.20.1 | | 192.168.20.2 | |
| |-----------------| |-----------------| |
| H | NIC QSFP Port 1 | | QSFP28 Port 0 | X |
| O | ens1f1np1 <>---------------<> sfp0 | 4 |
| S | 192.168.10.1 | | 192.168.10.2 | 1 |
| T |-----------------| ----------- |-----------------| 0 |
| | Ethernet Port 1 | |Router | | Ethernet Port | |
| | eno8403 <>-|192.168.1.1|-<> eth0 | |
| | 192.168.1.20 | | | | 192.168.1.19 | |
--------------------- ----------- ---------------------
When I run $ uhd_find_devices
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
-- UHD Device 0
Device Address:
serial: [serial number]
addr: 192.168.20.2
claimed: False
fpga: CG_400
mgmt_addr: 192.168.1.19
mgmt_addr: 192.168.10.2
mgmt_addr: 192.168.20.2
name: [name]
product: x410
type: x4xx
I am using two images for the FPGA, each of which has different
capabilities:
-
UC_200, which on the wiki lists Port 0 as "unused" and can achieve up to
250 Msps per channel. -
CG_400, which lists both ports on the wiki as "100 GbE" and can achieve
up to 500 Msps per channel.
I am using UHD's C++ API to instruct the x410. My capture process tries to
call "recv" as soon as possible without waiting to write the previous
buffer to a file. The psudocode is as follows:
-
Initialize "circular" buffer of arrays. Each array is "spb" samples
long, and there are N such buffers. -
Start a "writing" thread that writes data from a queue to a file.
-
Set the number of samples I need (nsamps_requested), set the current
number of samples to 0 (nsamps), -
set the current buffer index to 0 (buff_idx)
-
set up rx_streamer to start 0.1 seconds in the future and issue
STREAM_MODE_START_CONTINUOUS command.
While (nsamps <= nsamps_requested || not_error)
-Get a pointer to the buffer at buff_idx
-num_rx_samps <- recv(buffer...)
-check metadata returned object
-add the buffer to the queue, increment the buffer index
- stop streaming
When I use the process above, I can get very long captures using the
UC_200 image, with 250 Msps, and by parsing the args
"addr=192.168.20.2,mgmt_addr=192.168.1.19" when creating the multiusrp
pointer.
When I switch to the CG_400 image, I can barely record a few seconds worth
of data. I parse the args
"addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19". I
changed the logger level to get a better idea of what might be going on,
below are some outputs for spb at 5000:
[DEBUG] [RFNOC::MGMT] Established a route from EPID=4 (SW) to EPID=3
[DEBUG] [RFNOC::MGMT] Throttling stream endpoint to 100% (0x0)
[DEBUG] [RFNOC::MGMT] Initiated RX stream setup for EPID=3
[DEBUG] [RFNOC::MGMT] Finished RX stream setup for EPID=3
[DEBUG] [0/Radio#0] spp value 2032 exceeds MTU of 8016! Coercing to 1988
....
[DEBUG] [RxStreamer#0] Received overrun message on port 0
After the last logger message, I catc this in my "check metadata returned
object" step and stop the loop.
When this happens, I sometimes get 0 samples back, and sometimes get less
than the samples per buffer I feed the recv command. I also found that
while waiting for the issued command to take effect, instead of adding
empty buffers to my queue, I check for ERROR_CODE_TIMEOUT and just continue
the while loop.
I would imagine if I can handle half the rate at 250 Msps for minutes of
continuous capture, going to 500 Msps I could get more than a couple
seconds. Is this because the UC_200 image has 4GiB of DRAM and the CG_400
doesn't list any on the wiki? Is there something better I can be doing
about the spb (I tried larger and smaller, with some luck around 50000
spb)? Am I properly configuring the x410 to use both QSFP28 ports for the
CG_400 image?
Thanks!
Zack
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Hello Rob!
I should have mentioned this in my original post but benchmark rate works well for me. Specifically, when I run:
./benchmark_rate --rx_rate 491.52e6 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19" --duration 600
I get
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
[00:00:00.000322] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.19,type=x4xx,product=x410,serial=326E872,name=ni-x4xx-326E872,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2
[INFO] [MPM.PeriphManager] init() called with device args `fpga=CG_400,mgmt_addr=192.168.1.19,name=ni-x4xx-326E872,product=x410,second_addr=192.168.20.2,clock_source=internal,time_source=internal,initializing=True'.
Using Device: Single USRP:
Device: X400-Series Device
Mboard 0: x410
RX Channel: 0
RX DSP: n/a
RX Dboard: A
RX Subdev: 0
RX Channel: 1
RX DSP: n/a
RX Dboard: A
RX Subdev: 1
RX Channel: 2
RX DSP: n/a
RX Dboard: B
RX Subdev: 0
RX Channel: 3
RX DSP: n/a
RX Dboard: B
RX Subdev: 1
TX Channel: 0
TX DSP: n/a
TX Dboard: A
TX Subdev: 0
TX Channel: 1
TX DSP: n/a
TX Dboard: A
TX Subdev: 1
TX Channel: 2
TX DSP: n/a
TX Dboard: B
TX Subdev: 0
TX Channel: 3
TX DSP: n/a
TX Dboard: B
TX Subdev: 1
[00:00:03.511001021] Setting device timestamp to 0...
[00:00:03.512894034] Testing receive rate 491.520000 Msps on 1 channels
[00:10:03.513454979] Benchmark complete.
Benchmark rate summary:
Num received samples: 294911842520
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 0
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0
Done!
When I use the example that saves to file:
./rx_samples_to_file --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19" --file [path] --duration 10 --rate 491.52e6 --freq 1575.42e6
I get :
Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.19,type=x4xx,product=x410,serial=326E872,name=ni-x4xx-326E872,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2
[INFO] [MPM.PeriphManager] init() called with device args `fpga=CG_400,mgmt_addr=192.168.1.19,name=ni-x4xx-326E872,product=x410,second_addr=192.168.20.2,clock_source=internal,time_source=internal,initializing=True'.
Using Device: Single USRP:
Device: X400-Series Device
Mboard 0: x410
RX Channel: 0
RX DSP: n/a
RX Dboard: A
RX Subdev: 0
RX Channel: 1
RX DSP: n/a
RX Dboard: A
RX Subdev: 1
RX Channel: 2
RX DSP: n/a
RX Dboard: B
RX Subdev: 0
RX Channel: 3
RX DSP: n/a
RX Dboard: B
RX Subdev: 1
TX Channel: 0
TX DSP: n/a
TX Dboard: A
TX Subdev: 0
TX Channel: 1
TX DSP: n/a
TX Dboard: A
TX Subdev: 1
TX Channel: 2
TX DSP: n/a
TX Dboard: B
TX Subdev: 0
TX Channel: 3
TX DSP: n/a
TX Dboard: B
TX Subdev: 1
Setting RX Rate: 491.520000 Msps...
Actual RX Rate: 491.520000 Msps...
Setting RX Freq: 1575.420000 MHz...
Setting RX LO Offset: 0.000000 MHz...
[WARNING] [MULTI_USRP] No DSP capabilities detected. Combining offset into target frequency of 1575.420MHz
Actual RX Freq: 1575.420000 MHz...
Locking LO on channel 0
Waiting for "lo_locked": ++++++++++ locked.
Press Ctrl + C to stop streaming...
Disk write test indicates that an overflow is likely to occur.
Your write medium must sustain a rate of 1966.080MB/s,
but write test returned write speed of 184.000MB/s.
The disk write rate is also affected by system load
and OS/disk caching capacity.
OGot an overflow indication. Please consider the following:
Your write medium must sustain a rate of 1966.080MB/s.
Dropped samples will not be written to the file.
Please modify this example for your purposes.
This message will not appear again.
OOOOOO^C
Done!
Which I abort since I get overruns.
So, if benchmark_rate can run successfully, maybe try rx_samples_to_file
with saving the dev/null or with saving to files in a RAM file system.
This will limit your capture depth to the size of the RAM. If this works,
then it is a challenging issue to save to permanent storage. Perhaps DPDK
will work better at the expense of more complex configuration.
On Tue, Mar 12, 2024 at 11:58 AM zackkomo@utexas.edu wrote:
Hello Rob!
I should have mentioned this in my original post but benchmark rate works
well for me. Specifically, when I run:
./benchmark_rate --rx_rate 491.52e6 --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19" --duration 600
I get
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
[00:00:00.000322] Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.19,type=x4xx,product=x410,serial=326E872,name=ni-x4xx-326E872,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2
[INFO] [MPM.PeriphManager] init() called with device args `fpga=CG_400,mgmt_addr=192.168.1.19,name=ni-x4xx-326E872,product=x410,second_addr=192.168.20.2,clock_source=internal,time_source=internal,initializing=True'.
Using Device: Single USRP:
Device: X400-Series Device
Mboard 0: x410
RX Channel: 0
RX DSP: n/a
RX Dboard: A
RX Subdev: 0
RX Channel: 1
RX DSP: n/a
RX Dboard: A
RX Subdev: 1
RX Channel: 2
RX DSP: n/a
RX Dboard: B
RX Subdev: 0
RX Channel: 3
RX DSP: n/a
RX Dboard: B
RX Subdev: 1
TX Channel: 0
TX DSP: n/a
TX Dboard: A
TX Subdev: 0
TX Channel: 1
TX DSP: n/a
TX Dboard: A
TX Subdev: 1
TX Channel: 2
TX DSP: n/a
TX Dboard: B
TX Subdev: 0
TX Channel: 3
TX DSP: n/a
TX Dboard: B
TX Subdev: 1
[00:00:03.511001021] Setting device timestamp to 0...
[00:00:03.512894034] Testing receive rate 491.520000 Msps on 1 channels
[00:10:03.513454979] Benchmark complete.
Benchmark rate summary:
Num received samples: 294911842520
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 0
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0
Done!
When I use the example that saves to file:
./rx_samples_to_file --args
"addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19" --file
[path] --duration 10 --rate 491.52e6 --freq 1575.42e6
I get :
Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.19,type=x4xx,product=x410,serial=326E872,name=ni-x4xx-326E872,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2
[INFO] [MPM.PeriphManager] init() called with device args `fpga=CG_400,mgmt_addr=192.168.1.19,name=ni-x4xx-326E872,product=x410,second_addr=192.168.20.2,clock_source=internal,time_source=internal,initializing=True'.
Using Device: Single USRP:
Device: X400-Series Device
Mboard 0: x410
RX Channel: 0
RX DSP: n/a
RX Dboard: A
RX Subdev: 0
RX Channel: 1
RX DSP: n/a
RX Dboard: A
RX Subdev: 1
RX Channel: 2
RX DSP: n/a
RX Dboard: B
RX Subdev: 0
RX Channel: 3
RX DSP: n/a
RX Dboard: B
RX Subdev: 1
TX Channel: 0
TX DSP: n/a
TX Dboard: A
TX Subdev: 0
TX Channel: 1
TX DSP: n/a
TX Dboard: A
TX Subdev: 1
TX Channel: 2
TX DSP: n/a
TX Dboard: B
TX Subdev: 0
TX Channel: 3
TX DSP: n/a
TX Dboard: B
TX Subdev: 1
Setting RX Rate: 491.520000 Msps...
Actual RX Rate: 491.520000 Msps...
Setting RX Freq: 1575.420000 MHz...
Setting RX LO Offset: 0.000000 MHz...
[WARNING] [MULTI_USRP] No DSP capabilities detected. Combining offset into target frequency of 1575.420MHz
Actual RX Freq: 1575.420000 MHz...
Locking LO on channel 0
Waiting for "lo_locked": ++++++++++ locked.
Press Ctrl + C to stop streaming...
Disk write test indicates that an overflow is likely to occur.
Your write medium must sustain a rate of 1966.080MB/s,
but write test returned write speed of 184.000MB/s.
The disk write rate is also affected by system load
and OS/disk caching capacity.
OGot an overflow indication. Please consider the following:
Your write medium must sustain a rate of 1966.080MB/s.
Dropped samples will not be written to the file.
Please modify this example for your purposes.
This message will not appear again.
OOOOOO^C
Done!
Which I abort since I get overruns.
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Hey Rob,
Saving to dev/null worked just fine, and didn’t even output the “Disk write test indicates that an overflow is likely to occur” warning.
In terms of saving to RAM, isn’t this essentially what my custom script does? I reserve all my buffers (which increases my RAM usage by a lot) and have a separate thread write from these to the filesystem.
Would anything be different if I used a ramdisk? I can try it out.
As for DPDK, I tried in the past to get it working but keep running into issues. I will post the output below, but it may warrant a separate post.
When running:
./rx_samples_to_file --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1" --file /dev/null --duration 10 --rate 491.52e6 --freq 1575.42e6
I get:
Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /run/user/1001/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: get_seg_fd(): open '/dev/hugepages/rtemap_0' failed: Permission denied
EAL: Couldn't get fd on hugepage file
EAL: get_seg_fd(): open '/dev/hugepages/rtemap_32768' failed: Permission denied
EAL: Couldn't get fd on hugepage file
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL initialization
EAL: FATAL: already called initialization.
EAL: already called initialization.
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL initialization
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.10.2
second_addr: 192.168.20.2
mgmt_addr: 192.168.1.19
use_dpdk: 1
When I run the same command as the root user, I get:
Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:31:00.0 on NUMA socket 0
EAL: probe driver: 15b3:101d net_mlx5
EAL: PCI device 0000:31:00.1 on NUMA socket 0
EAL: probe driver: 15b3:101d net_mlx5
[ERROR] [DPDK] Could not find route to destination address 192.168.10.2
[ERROR] [X300] X300 Network discovery error RuntimeError: DPDK: Could not find route to destination address 192.168.10.2
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.19,type=x4xx,product=x410,serial=326E872,name=ni-x4xx-326E872,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,use_dpdk=1
[WARNING] [MPM.RPCServer] A timeout event occured!
[INFO] [MPM.PeriphManager] init() called with device args `fpga=CG_400,mgmt_addr=192.168.1.19,name=ni-x4xx-326E872,product=x410,second_addr=192.168.20.2,use_dpdk=1,clock_source=internal,time_source=internal,initializing=True'.
[ERROR] [DPDK] Could not find route to destination address 192.168.10.2
[WARNING] [MPMD::XPORT::UDP] Error during MTU discovery on address 192.168.10.2: RuntimeError: DPDK: Could not find route to destination address 192.168.10.2
[ERROR] [RFNOC::MGMT] EnvironmentError: IOError: Timed out getting recv buff for management transaction
[ERROR] [RFNOC::GRAPH] IO Error during GSM initialization. EnvironmentError: IOError: Timed out getting recv buff for management transaction
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: EnvironmentError: IOError: Timed out getting recv buff for management transaction
Error: RuntimeError: Failure to create rfnoc_graph.
Hi Zack,
I would not count on equal performance between saving to a RAM file system
versus allocating your own buffers, but I don't know the reason. I think it
is worth a try to configure as large a RAM file system as you can and run
rx_samples_to_file (which will not create large RAM buffers). Plus, this
is a pretty easy test to run.
I can tell you that long ago I got pretty good performance by writing to
files in RAM (up to a predetermined fixed length) such that after a fixed
number of samples, I closed those files and opened new ones and continued
writing to the new ones. As a separate process, the closed files were
moved to persistent storage thus free-ing up RAM. This sounds a lot like
your circular buffer in RAM concept but I was never able to get the
circular buffer architecture working as well.
Rob
On Tue, Mar 12, 2024 at 1:00 PM zackkomo@utexas.edu wrote:
Hey Rob,
Saving to dev/null worked just fine, and didn’t even output the “Disk
write test indicates that an overflow is likely to occur” warning.
In terms of saving to RAM, isn’t this essentially what my custom script
does? I reserve all my buffers (which increases my RAM usage by a lot) and
have a separate thread write from these to the filesystem.
Would anything be different if I used a ramdisk? I can try it out.
As for DPDK, I tried in the past to get it working but keep running into
issues. I will post the output below, but it may warrant a separate post.
When running:
./rx_samples_to_file --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1" --file /dev/null --duration 10 --rate 491.52e6 --freq 1575.42e6
I get:
Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /run/user/1001/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: get_seg_fd(): open '/dev/hugepages/rtemap_0' failed: Permission denied
EAL: Couldn't get fd on hugepage file
EAL: get_seg_fd(): open '/dev/hugepages/rtemap_32768' failed: Permission denied
EAL: Couldn't get fd on hugepage file
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL initialization
EAL: FATAL: already called initialization.
EAL: already called initialization.
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL initialization
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.10.2
second_addr: 192.168.20.2
mgmt_addr: 192.168.1.19
use_dpdk: 1
When I run the same command as the root user, I get:
Creating the usrp device with: addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.6.0.HEAD-0-g50fa3baa
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:31:00.0 on NUMA socket 0
EAL: probe driver: 15b3:101d net_mlx5
EAL: PCI device 0000:31:00.1 on NUMA socket 0
EAL: probe driver: 15b3:101d net_mlx5
[ERROR] [DPDK] Could not find route to destination address 192.168.10.2
[ERROR] [X300] X300 Network discovery error RuntimeError: DPDK: Could not find route to destination address 192.168.10.2
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.19,type=x4xx,product=x410,serial=326E872,name=ni-x4xx-326E872,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,use_dpdk=1
[WARNING] [MPM.RPCServer] A timeout event occured!
[INFO] [MPM.PeriphManager] init() called with device args `fpga=CG_400,mgmt_addr=192.168.1.19,name=ni-x4xx-326E872,product=x410,second_addr=192.168.20.2,use_dpdk=1,clock_source=internal,time_source=internal,initializing=True'.
[ERROR] [DPDK] Could not find route to destination address 192.168.10.2
[WARNING] [MPMD::XPORT::UDP] Error during MTU discovery on address 192.168.10.2: RuntimeError: DPDK: Could not find route to destination address 192.168.10.2
[ERROR] [RFNOC::MGMT] EnvironmentError: IOError: Timed out getting recv buff for management transaction
[ERROR] [RFNOC::GRAPH] IO Error during GSM initialization. EnvironmentError: IOError: Timed out getting recv buff for management transaction
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: EnvironmentError: IOError: Timed out getting recv buff for management transaction
Error: RuntimeError: Failure to create rfnoc_graph.
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
On 12/03/2024 15:39, Rob Kossler via USRP-users wrote:
Hi Zack,
I would not count on equal performance between saving to a RAM file
system versus allocating your own buffers, but I don't know the
reason. I think it is worth a try to configure as large a RAM file
system as you can and run rx_samples_to_file (which will not create
large RAM buffers). Plus, this is a pretty easy test to run.
Writing to RAM via a RAM filesystem still incurs the machinations of the
OS kernel similar to what it would do for ANY filesystem,
so it cannot be as fast as simply moving bytes into a big buffer in
your application...
However, it's awfully convenient, since it has all the same semantics
(other than persistence) of a filesystem on persistent
media.
Hey Rob and Marcus,
Thanks for the responses! I have a basic understanding of linux, but am not very experienced. I tried the following to create the RAM filesystem:
sudo mount -t tmpfs -o size=8G tmpfs /mnt/tmpfs/
sudo mount -t ramfs -o size=8G ramfs /mnt/ramfs/
And ran the rx_samples_to_file, once with --file /mnt/tmpfs/test.bin, and once with --file /mnt/ramfs/test.bin, both times still getting o’s for overruns.
By my calculations, at ~500 M complex samples per second, each complex sample 4 bytes (defaulting to short for I and Q), that means just 1 second of capturing equates to 2 GB of data. My system has 64 GB of RAM. Am I creating the RAM filesystem correctly? Am I using it correctly?
On 12/03/2024 16:11, zackkomo@utexas.edu wrote:
Hey Rob and Marcus,
Thanks for the responses! I have a basic understanding of linux, but
am not very experienced. I tried the following to create the RAM
filesystem:
|sudo mount -t tmpfs -o size=8G tmpfs /mnt/tmpfs/|
|sudo mount -t ramfs -o size=8G ramfs /mnt/ramfs/ |
And ran the rx_samples_to_file, once with --file /mnt/tmpfs/test.bin,
and once with --file /mnt/ramfs/test.bin, both times still getting o’s
for overruns.
By my calculations, at ~500 M complex samples per second, each complex
sample 4 bytes (defaulting to short for I and Q), that means just 1
second of capturing equates to 2 GB of data. My system has 64 GB of
RAM. Am I creating the RAM filesystem correctly? Am I using it correctly?
USRP-users mailing list --usrp-users@lists.ettus.com
To unsubscribe send an email tousrp-users-leave@lists.ettus.com
Assuming that you did a "sudo mkdir of /mnt/ramfs" beforehand, this
should work.
Your mount command with tmpfs looks correct. Here is what mine is in my
/etc/fstab file (with 264GB avail RAM)
tmpfs /media/ramfolder/ tmpfs rw,nosuid,nodev,size=200G 0 0
You might want to try rx_samples_to_file with the --multi_streamer option.
I expect you will get better performance. Also, you can take your RAM FS
size higher from 8G to probably 60G if you want to try bigger recording
depths.
Rob
On Tue, Mar 12, 2024 at 5:13 PM Marcus D. Leech patchvonbraun@gmail.com
wrote:
On 12/03/2024 16:11, zackkomo@utexas.edu wrote:
Hey Rob and Marcus,
Thanks for the responses! I have a basic understanding of linux, but am
not very experienced. I tried the following to create the RAM filesystem:
sudo mount -t tmpfs -o size=8G tmpfs /mnt/tmpfs/
sudo mount -t ramfs -o size=8G ramfs /mnt/ramfs/
And ran the rx_samples_to_file, once with --file /mnt/tmpfs/test.bin, and
once with --file /mnt/ramfs/test.bin, both times still getting o’s for
overruns.
By my calculations, at ~500 M complex samples per second, each complex
sample 4 bytes (defaulting to short for I and Q), that means just 1 second
of capturing equates to 2 GB of data. My system has 64 GB of RAM. Am I
creating the RAM filesystem correctly? Am I using it correctly?
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Assuming that you did a "sudo mkdir of /mnt/ramfs" beforehand, this
should work.
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com