It is the switch that is internal to you hardware on USRP-2974.

You can see the switch below (from your documentation).  I’m sending from the SBC (Single board computer) shown in the diagram.

https://kb.ettus.com/File:2974_blk_dia_hiLevel_v01.png

Actually, you can probably expect that the ping -s with jumbo frames to fail, since the FPGA stack is very limited and most likely hasn’t implemented ICMP for jumbo frames.

However, the limitation of 8000 seems to be an artificial limit of the UHD code, as it I have been able to demonstrate it can send > 8000, but not at my requested value due to the way the driver is written.  You can see in the packet trace (if you do a tcpdump) is sends larger than 8000.

See “static const size_t DATA_FRAME_MAX_SIZE = 8000; // CHDR packet size in bytes” in x300_defaults.hpp and what appears to be some bad logic in determine_max_frame_size() (setting packet size to min(tx,rx)).
I hacked the code and have seen the Tx >= 8136, but don’t understand it enough yet.

You might also need to reach out to the FPGA team to actually see what the limitation is or is not.  Documentation says to set MTU to 9000, but UHD driver has some issues in implementation.

I hacked the code to set the DATA_FRAME_MAX_SIZE = 8500, half-way between 8000 and MTU 9000.

Then I made sure to set the send_frame_size=recv_frame_size=8700, which is equal, and over the MAX_SIZE.It still picks 8144.  You can see below.

I added a bunch of debug of the intermediate variables.

$~/uhd-4.6.0.0/host/build/examples$ ./tx_samples_c -a "type=x300,addr=192.168.40.2,send_frame_size=8700,recv_frame_size=8700"

Creating USRP with args "type=x300,addr=192.168.40.2,send_frame_size=8700,recv_frame_size=8700"...

[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_4.6.0.0-1-ga9f0b4c7

[INFO] [X300] X300 initialization sequence...

[INFO] [X300] user_frame_size.send_frame_size: 8700

[INFO] [X300] x300::DATA_FRAME_MAX_SIZE: 8500

[INFO] [X300] std::min(user_frame_size.send_frame_size, x300::DATA_FRAME_MAX_SIZE): 8500

[INFO] [X300] max_send_frame_size: 8500

[INFO] [X300] min_recv_frame_size: 8144

[INFO] [X300] min_send_frame_size: 8144

[INFO] [X300] Maximum frame size: 8144 bytes.

[WARNING] [X300] You requested a receive frame size of (8700) but your NIC's max frame size is (8144).Please verify your NIC's MTU setting using 'ip link' or set the recv_frame_size argument appropriately.UHD will use the auto-detected max frame size for this connection.

[WARNING] [X300] You requested a send frame size of (8700) but your NIC's max frame size is (8144).Please verify your NIC's MTU setting using 'ip link' or set the send_frame_size argument appropriately.UHD will use the auto-detected max frame size for this connection.

[WARNING] [X300] For the 192.168.40.2 connection, UHD recommends a send frame size of at least 8500 for best

performance, but your configuration will only allow 8144.This may negatively impact your maximum achievable sample rate.

Check the MTU on the interface and/or the send_frame_size argument.

[WARNING] [X300] For the 192.168.40.2 connection, UHD recommends a receive frame size of at least 8500 for best

performance, but your configuration will only allow 8144.This may negatively impact your maximum achievable sample rate.

Check the MTU on the interface and/or the recv_frame_size argument.

[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a

[INFO] [X300] Radio 1x clock: 200 MHz

Setting TX Rate: 200000000.000000...

Actual TX Rate: 200000000.000000...

Setting TX Gain: 0.000000 db...

Thanks for all the great info Marcus & Wade

Would you be able to provide some suggestions?  We need to keep precision timing for transmission.  Neither of the two proof of concepts (POC) below are meeting our needs.  Of course we are sending 2 packets due to lack of jumbo frames.

We have 2048 samples (4 bytes per sample) we would have liked to send per packet, or 2048/200E+6 = 10.24 us per packet or 97656 packets per second.  It’s a continuous stream.  This doesn’t include packet overhead.

The reason the Jumbo frame thing came up was because we were getting “L”s (missed timing) when using the time spec, and “U” when not using timing spec, most likely due to 2 packets per transmission instead of 1 jumbo frame.

I have provided POC of our test Tx program for 2 cases.  The first uses timing spec, which doesn’t keep up.  The second was just to see how it would perform if there wasn’t a time spec.

This thread runs as the highest priority in Linux, and we have tuned according to your documentation.

First Test Program 1), PPS is ~95500,  but we get “L”s constantly .

[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_

[INFO] [X300] X300 initialization sequence...

[INFO] [X300] Maximum frame size: 8000 bytes.

[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a

[INFO] [X300] Radio 1x clock: 200 MHz

Actual TX Rate: 200.000000 Msps

Actual TX Freq: 2400.000000 MHz

Actual TX Gain: 30.000000 dB

Actual TX Bandwidth: 16.000000 MHz

[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.

Using transmit_packets_timespec()

Start transmit_packets_timespec()

Packet size: 8192 bytes

--------------------------------------------

Transmitted packets: 30632386

Dropped packets: 9

PAUSE frames received: 0

PAUSE frames transmitted: 0

GPS lock lost events: 0

Network RX packets: 19733243

Network TX packets: 206298940

Network RX errors: 0

Network TX errors: 0

Network RX dropped: 0

Network TX dropped: 0

USRP TX overruns: 0

USRP TX underruns: 0

USRP sequence errors: 0

GPS time sync: 1136281795

GPS time sync errors: 0

Packets Per Second (PPS): 95428  

Second Test Program 2), PPS is ~98000, Buffer Size is 8192 (without USRP overhead), occasional “U”

[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_

[INFO] [X300] X300 initialization sequence...

[INFO] [X300] Maximum frame size: 8000 bytes.

[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a

[INFO] [X300] Radio 1x clock: 200 MHz

Actual TX Rate: 200.000000 Msps

Actual TX Freq: 2400.000000 MHz

Actual TX Gain: 30.000000 dB

Actual TX Bandwidth: 16.000000 MHz

[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.

Using transmit_packets_no_timespec()

Start transmit_packets_no_timespec()

Packet size: 8192 bytes

--------------------------------------------

Transmitted packets: 12844561

Dropped packets: 0

PAUSE frames received: 0

PAUSE frames transmitted: 0

GPS lock lost events: 0

Network RX packets: 5528620

Network TX packets: 77280212

Network RX errors: 0

Network TX errors: 0

Network RX dropped: 0

Network TX dropped: 0

USRP TX overruns: 0

USRP TX underruns: 0

USRP sequence errors: 0

GPS time sync: 1136280963

GPS time sync errors: 0

Packets Per Second (PPS): 98050

// 2 function to simulate our packet transmission
// First Test Program 1)
void transmit_packets_timespec(uhd::usrp::multi_usrp::sptr usrp, uhd::tx_streamer::sptr tx_stream, std::atomic<bool> &gps_locked, double rate) {
    uhd::tx_metadata_t md;
    md.start_of_burst = true;
    md.end_of_burst = false;
    md.has_time_spec = true;

    std::vector<std::complex<int16_t>> buffer(2048); // CRJ TEMP buffer(2048);

    std::cout << "Start transmit_packets_timespec()" << std::endl;
    std::cout << "Packet size: " << buffer.size() * sizeof(std::complex<int16_t>) << " bytes" << std::endl;

    // Get and print the maximum number of samples per packet
    size_t max_num_samps = tx_stream->get_max_num_samps();
    std::cout << "Max number of samples per packet: " << max_num_samps << std::endl;

    // Initialize timestamp
    md.time_spec = uhd::time_spec_t(usrp->get_time_now().get_full_secs() + 1.0); // Start 1 second in the future

    while (1 /*!stop_logging.load()*/) {
        //if (gps_locked.load()) {
            //  md.time_spec = uhd::time_spec_t(usrp->get_time_now().get_full_secs() + 1.0);
        //}

        size_t num_tx_samps = tx_stream->send(&buffer.front(), buffer.size(), md);

        if (num_tx_samps < buffer.size()) {
            dropped_packets++;
        } else {
            transmitted_packets++;
        }

        md.time_spec = md.time_spec + uhd::time_spec_t(buffer.size() / rate);

        md.start_of_burst = false;
        //std::this_thread::sleep_for(std::chrono::milliseconds(5));
    }

    // Mark end of burst
    md.end_of_burst = true;
    tx_stream->send("", 0, md);
}

// Second Test Program 2)
void transmit_packets_no_timespec(uhd::usrp::multi_usrp::sptr usrp, uhd::tx_streamer::sptr tx_stream, std::atomic<bool> &gps_locked, double rate) {
    uhd::tx_metadata_t md;
    md.start_of_burst = true;
    md.end_of_burst = false;
    md.has_time_spec = false; // No time specification

    std::vector<std::complex<int16_t>> buffer(2048);

    std::cout << "Start transmit_packets_no_timespec()" << std::endl;
    std::cout << "Packet size: " << buffer.size() * sizeof(std::complex<int16_t>) << " bytes" << std::endl;

    while (!stop_logging.load()) {
        size_t num_tx_samps = tx_stream->send(&buffer.front(), buffer.size(), md);

        if (num_tx_samps < buffer.size()) {
            dropped_packets++;
        } else {
            transmitted_packets++;
        }

        md.start_of_burst = false;
        // Adjust the sleep time to control the packet rate
        // std::this_thread::sleep_for(std::chrono::milliseconds(1));
    }

    // Mark end of burst
    md.end_of_burst = true;
    tx_stream->send("", 0, md);
}

Hi Wade,

I am now receiving “U” instead of “L”.  They don’t come out on the console that often, but it is “bad” if I see any.
In addition, I did add a priority to the thread, you can see below (takes on a value between 0.0-100.0).
I also tried setting the buffer to 10*1996.
None of these efforts helped.
Any suggestions on how I can get this to work without losing packets, or timing?  FYI usrp->set_tx_rate(200e6).
(Below is code, below that is the console output)

// Main function to simulate packet transmission
void transmit_packets_timespec(uhd::usrp::multi_usrp::sptr usrp, uhd::tx_streamer::sptr tx_stream, std::atomic<bool> &gps_locked, double rate) {
    uhd::tx_metadata_t md;
    md.start_of_burst = true;
    md.end_of_burst = false;
    md.has_time_spec = true;

    // Attempt to set the thread priority
    bool success = uhd::set_thread_priority_safe(0.90, true);
    if (success) {
        std::cout << "uhd::set_thread_priority_safe::Thread priority successfully set." << std::endl;
    } else {
        std::cout << "uhd::set_thread_priority_safe::Failed to set thread priority." << std::endl;
    }

    std::vector<std::complex<int16_t>> buffer(1996); 

    std::cout << "Start transmit_packets_timespec()" << std::endl;
    std::cout << "Packet size: " << buffer.size() * sizeof(std::complex<int16_t>) << " bytes" << std::endl;

    // Get and print the maximum number of samples per packet
    size_t max_num_samps = tx_stream->get_max_num_samps();
    std::cout << "Max number of samples per packet: " << max_num_samps << std::endl;

    // Initialize timestamp
    md.time_spec = uhd::time_spec_t(usrp->get_time_now().get_full_secs() + 1.0); // Start 1 second in the future

    while (true) {

        size_t num_tx_samps = tx_stream->send(&buffer.front(), buffer.size(), md);

        if (num_tx_samps < buffer.size()) {
            dropped_packets++;
        } else {
            transmitted_packets++;
        }

        //md.time_spec = md.time_spec + uhd::time_spec_t(buffer.size() / rate);
        md.has_time_spec = false;

        md.start_of_burst = false;
    }

    // Mark end of burst
    md.end_of_burst = true;
    tx_stream->send("", 0, md);
}

CONSOLE OUTPUT:

sudo ./tx_timed_samples_perf
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_4.6.0.0-1-ga9f0b4c7
[INFO] [X300] X300 initialization sequence...

[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Actual TX Rate: 200.000000 Msps
Actual TX Freq: 1223.000003 MHz
Actual TX Gain: 20.000000 dB
Actual TX Bandwidth: 200.000000 MHz
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Using transmit_packets_timespec()
uhd::set_thread_priority_safe::Thread priority successfully set.
Start transmit_packets_timespec()
Packet size: 7984 bytes
Max number of samples per packet: 1996
UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU

Hi,

I appreciate the response.  I previously made all of those changes (except bios, DPDK, and Sprectre).  In addition, I have MTU at 9000.  I just re-verified.

I’m still getting the U’s.

So we could have the same common code as a starting point, I modified your tx_timed_samples.cpp provided below so you can reproduce the problem.

Here are the changes:

• Sends continuously (instead of some a set number of samples)

• Data is sc16 and not fc32

• Data rate is 200e6 SPS instead of a much lower value

//
// Copyright 2010-2011,2014 Ettus Research LLC
// Copyright 2018 Ettus Research, a National Instruments Company
//
// SPDX-License-Identifier: GPL-3.0-or-later
//

#include <uhd/usrp/multi_usrp.hpp>
#include <uhd/utils/safe_main.hpp>
#include <uhd/utils/thread.hpp>
#include <boost/format.hpp>
#include <boost/program_options.hpp>
#include <boost/thread/thread.hpp>
#include <complex>
#include <iostream>

namespace po = boost::program_options;

int UHD_SAFE_MAIN(int argc, char* argv[])
{
    // variables to be set by po
    std::string args;
    std::string wire;
    double seconds_in_future;
    size_t total_num_samps;
    double rate;
    float ampl;

    // setup the program options
    po::options_description desc("Allowed options");
    // clang-format off
    desc.add_options()
        ("help", "help message")
//        ("args", po::value<std::string>(&args)->default_value(""), "single uhd device address args")
        ("args", po::value<std::string>(&args)->default_value("type=x300,addr=192.168.40.2"), "single uhd device address args")
        ("wire", po::value<std::string>(&wire)->default_value(""), "the over the wire type, sc16, sc8, etc")
        ("secs", po::value<double>(&seconds_in_future)->default_value(1.5), "number of seconds in the future to transmit")
        ("nsamps", po::value<size_t>(&total_num_samps)->default_value(10000), "total number of samples to transmit")
//        ("rate", po::value<double>(&rate)->default_value(100e6/16), "rate of outgoing samples")
        ("rate", po::value<double>(&rate)->default_value(200e6), "rate of outgoing samples")
        ("ampl", po::value<float>(&ampl)->default_value(float(0.3)), "amplitude of each sample")
        ("dilv", "specify to disable inner-loop verbose")
    ;
    // clang-format on
    po::variables_map vm;
    po::store(po::parse_command_line(argc, argv, desc), vm);
    po::notify(vm);

    // print the help message
    if (vm.count("help")) {
        std::cout << boost::format("UHD TX Timed Samples %s") % desc << std::endl;
        return ~0;
    }

    bool verbose = vm.count("dilv") == 0;

    // create a usrp device
    std::cout << std::endl;
    std::cout << boost::format("Creating the usrp device with: %s...") % args
              << std::endl;
    uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make(args);
    std::cout << boost::format("Using Device: %s") % usrp->get_pp_string() << std::endl;

    // set the tx sample rate
    std::cout << boost::format("Setting TX Rate: %f Msps...") % (rate / 1e6) << std::endl;
    usrp->set_tx_rate(rate);
    std::cout << boost::format("Actual TX Rate: %f Msps...") % (usrp->get_tx_rate() / 1e6)
              << std::endl
              << std::endl;

    std::cout << boost::format("Setting device timestamp to 0...") << std::endl;
    usrp->set_time_now(uhd::time_spec_t(0.0));

    // create a transmit streamer
//    uhd::stream_args_t stream_args("fc32", wire); // complex floats
    uhd::stream_args_t stream_args("sc16", "sc16");
    uhd::tx_streamer::sptr tx_stream = usrp->get_tx_stream(stream_args);

    // allocate buffer with data to send
//    std::vector<std::complex<float>> buff(tx_stream->get_max_num_samps(), std::complex<float>(ampl, ampl));
    typedef uint16_t sample_t;    // Either the I or Q portion of an IQ sample.
    std::vector<std::complex<sample_t>> buff(tx_stream->get_max_num_samps(), std::complex<sample_t>(ampl * 32767, ampl * 32767));

    // setup metadata for the first packet
    uhd::tx_metadata_t md;
    md.start_of_burst = false;
    md.end_of_burst   = false;
    md.has_time_spec  = true;
    md.time_spec      = uhd::time_spec_t(seconds_in_future);

    // the first call to send() will block this many seconds before sending:
    const double timeout =
        seconds_in_future + 0.1; // timeout (delay before transmit + padding)

#if 1
    while (true) {
        size_t num_tx_samps = tx_stream->send(&buff.front(), buff.size(), md);

        if (num_tx_samps < buff.size()) {
            std::cerr << "Send timeout..." << std::endl;
        }

        // do not use time spec for subsequent packets
        md.has_time_spec = false;
    }
#else
    size_t num_acc_samps = 0; // number of accumulated samples
    while (num_acc_samps < total_num_samps) {
        size_t samps_to_send = std::min(total_num_samps - num_acc_samps, buff.size());

        // send a single packet
        size_t num_tx_samps = tx_stream->send(&buff.front(), samps_to_send, md, timeout);

        // do not use time spec for subsequent packets
        md.has_time_spec = false;

        if (num_tx_samps < samps_to_send)
            std::cerr << "Send timeout..." << std::endl;
        if (verbose)
            std::cout << boost::format("Sent packet: %u samples") % num_tx_samps
                      << std::endl;

        num_acc_samps += num_tx_samps;
    }
#endif

    // send a mini EOB packet
    md.end_of_burst = true;
    tx_stream->send("", 0, md);

    std::cout << std::endl << "Waiting for async burst ACK... " << std::flush;
    uhd::async_metadata_t async_md;
    bool got_async_burst_ack = false;
    // loop through all messages for the ACK packet (may have underflow messages in queue)
    while (not got_async_burst_ack and tx_stream->recv_async_msg(async_md, timeout)) {
        got_async_burst_ack =
            (async_md.event_code == uhd::async_metadata_t::EVENT_CODE_BURST_ACK);
    }
    std::cout << (got_async_burst_ack ? "success" : "fail") << std::endl;

    // finished
    std::cout << std::endl << "Done!" << std::endl << std::endl;

    return EXIT_SUCCESS;
}

Hi,

Can you help me figure out what the problem is using the modified example (I provided in last response), a fast machine which has the tips and tricks implemented?

I directly connected our 64 core / 4 GPU / >250GI RAM workstation with 10G output directly to the X310 10G, which should easily keep up with 200Msps.

I ran the same program I provided above (I just modified your example program tx_timed_samples.cpp) and now it outputs S’s instead of U’s.

Here is the output:

cjohnson@demo:~/uhd_versions/uhd_4.4.0.0/host/build/examples$ ./tx_timed_samples

Creating the usrp device with: type=x300,addr=192.168.30.2...
[INFO] [UHD] linux; GNU C++ version 11.4.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Using Device: Single USRP:
  Device: X-Series Device
  Mboard 0: X310
  RX Channel: 0
    RX DSP: 0
    RX Dboard: A
    RX Subdev: UBX RX
  RX Channel: 1
    RX DSP: 1
    RX Dboard: B
    RX Subdev: UBX RX
  TX Channel: 0
    TX DSP: 0
    TX Dboard: A
    TX Subdev: UBX TX
  TX Channel: 1
    TX DSP: 1
    TX Dboard: B
    TX Subdev: UBX TX

Setting TX Rate: 200.000000 Msps...
Actual TX Rate: 200.000000 Msps...

Setting device timestamp to 0...
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Send timeout...
Send timeout...
Send timeout...
Send timeout...
Send timeout...
Send timeout...
Send timeout...
SSSSSSSS (... more S's)

I verified the interface is setup for 9000 MTU, and that I have made the adjustments indicated in https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks
except for Specture / bios changes).  This includes Ring Buffers for NIC and Network Buffers

(Below, same for CPU 0-63)
analyzing CPU 63:
  driver: intel_cpufreq
  CPUs which run at the same hardware frequency: 63
  CPUs which need to have their frequency coordinated by software: 63
  maximum transition latency: 20.0 us.
  hardware limits: 800 MHz - 3.50 GHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil
  current policy: frequency should be within 800 MHz and 3.50 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.50 GHz.

Hi, Did you give the modified host/examples/tx_timed_samples.cpp I provided above a try?  This maybe the best first path to go down, to make sure it is not a UHD thing.  You can just replace the current tx_timed_samples.cpp with this one (2 threads ago), then make in (for example) “~/uhd-4.4.0.0/host/build”.  Output is in “~/uhd-4.4.0.0/host/build/examples” as tx_timed_samples.

This is an actual server (64 core) running ubuntu 22.04

~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy