**Hi,** **Hopefully you can help me out with some clarification and help on a few questions.** **We are using the X310 all with timestamps as we try to get an understanding of the behavior.  We have created several scenarios to try out start of burst and end of burst and have noticed different behavior.**   **Scenario 1 is a “single burst” for a bunch of samples sent (for example 1996 samples of std::vector<std::complex<int16_t>>) where start_of_burst=end_of_burst=true via send(includes timeout).**   **Scenario 2 is a “continuous burst” where we start_of_burst=true for the first 1996 packet only with end_of_burst=false.  All other packets have end_of_burst=false** **\ 1) What is the difference between the "S" and "L"?  When we see "S" with the single-burst case, is that really the same as an "L"?** **We have overhead to do between sending samples, so we are trying to figure out the best way to do the overhead work and still send() on time at our desired rate (still using timestamps).** **We noticed through experimentation that the “system” (PC TCP/IP Stack <-> USRP buffering) could buffer a very large packet of 1996\*65=129,740 samples before we received flow-control, which means that we could send a bunch of samples with future timestamps to give us some time to do the “overhead”.   Of course 1996 is used because it is the largest UDP packet before fragmentation occurs with UDP on USRP.** **2) Is our observation about the 129K samples of buffering correct?  Is there something we can do to increase this?  For example, maybe some tweaks to the receive buffer sizes in the TCP/IP stack to make receive work reliably.** **3) What is the recommendation in terms of burst size (to allow us to do our “overhead”)** **Thanks**

Hi, I know it’s the holiday…so I understand. Trying to bring this back to the top of the queue. Best

I hope everyone had a good holiday break! Would you mind providing some guidance on the 3 questions? Thanks

On 05/01/2025 17:19, cjohnson@serranosystems.com wrote: > > I hope everyone had a good holiday break! > > Would you mind providing some guidance on the 3 questions? > > Thanks > > > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com I will be able to comment that 'S' on a transmit stream means that a packet got dropped on the network between the host and   the radio.  This shouldn't ever happen, but sometimes for a "fresh" UDP stream, the kernel driver will drop a UDP frame while   it completes ARP transactions.  But that should happen not very often, and there's likely a kernel parameter that can   disable this behavior.  But, for a fully-working network stack, my understanding is that 'S' should be very rare unless you're   offering a packet load that the "stack" just cannot cope with because it's running out of space to put your packets.

The X310 buffering depends on whether or not you are using the DRAMFIFO as part of the RFNoC flow. If so, then this buffer alone is 1GB, I think (250 MSamps). But, you can't use this if you are running two channels at 200 MS/s, because the DRAM bandwidth is not sufficient. If you are running the samples at 100 MS/s or less (into the DUC) or if you are only using one channel, it should be fine to use this. If you are not using the DRAMFIFO, then the amount of buffering is dependent on the FIFOs that are built into the Streaming Endpoints. You could check the yaml file that is used for the X310 FPGA build. Perhaps you could explain a bit more about your intended application. It's not clear to me if you are having issues with transmit streaming or if you are also having issues with receive streaming. Are you planning to use gnuradio? rfnoc (custom graph vs default multi_usrp)? C++ application? Rob On Tue, Dec 24, 2024 at 3:07 PM <cjohnson@serranosystems.com> wrote: > *Hi,* > > *Hopefully you can help me out with some clarification and help on a few > questions.* > > *We are using the X310 all with timestamps as we try to get an > understanding of the behavior. We have created several scenarios to try > out start of burst and end of burst and have noticed different behavior. * > > *Scenario 1 is a “single burst” for a bunch of samples sent (for example > 1996 samples of std::vector<std::complex<int16_t>>) where > start_of_burst=end_of_burst=true via send(includes timeout). * > > *Scenario 2 is a “continuous burst” where we start_of_burst=true for the > first 1996 packet only with end_of_burst=false. All other packets have > end_of_burst=false* > > > *1) What is the difference between the "S" and "L"? When we see "S" with > the single-burst case, is that really the same as an "L"?* > > *We have overhead to do between sending samples, so we are trying to > figure out the best way to do the overhead work and still send() on time at > our desired rate (still using timestamps).* > > *We noticed through experimentation that the “system” (PC TCP/IP Stack <-> > USRP buffering) could buffer a very large packet of 1996*65=129,740 samples > before we received flow-control, which means that we could send a bunch of > samples with future timestamps to give us some time to do the “overhead”. > Of course 1996 is used because it is the largest UDP packet before > fragmentation occurs with UDP on USRP.* > > *2) Is our observation about the 129K samples of buffering correct? Is > there something we can do to increase this? For example, maybe some tweaks > to the receive buffer sizes in the TCP/IP stack to make receive work > reliably.* > > *3) What is the recommendation in terms of burst size (to allow us to do > our “overhead”)* > > *Thanks* > > > > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >

Some more comments: - S means "sequence error", which means packets were lost somewhere between your host and the USRP. "L" means late, which just means that your packet didn't reach the radio in time. - If you are using a default X310 image, then the corresponding YAML is this: https://github.com/EttusResearch/uhd/blob/master/fpga/usrp3/top/x300/x310_XG_rfnoc_image_core.yml#L20 I've highlighted the line. The stream endpoint has a default buffer size of almost 500 kB. - Start-of-burst doesn't do anything on the X310 (but setting it doesn't hurt, either). - Buffering aside, you need to maintain an average software-send-rate that is on par with your sample rate. If you use timestamps to delay, then that will help you by filling up the buffer first, and that gives you a bit of elasticity (you can take a bit longer on occasion as long as on average, your SW processing rate is not below the hardware rate). - Maybe try a multi-threaded approach with a consumer-producer model to avoid IO waits to block your processing thread. - what is your rate? If it's small, maybe smaller packet sizes would be better (because you can produce packets faster). - If you have only a single channel, you can increase the buffer size (if you don't want to use the DRAM FIFO) by moving buffer from ep2 to ep0. --M On Mon, Jan 6, 2025 at 4:12 PM Rob Kossler via USRP-users < usrp-users@lists.ettus.com> wrote: > The X310 buffering depends on whether or not you are using the DRAMFIFO as > part of the RFNoC flow. If so, then this buffer alone is 1GB, I think (250 > MSamps). But, you can't use this if you are running two channels at 200 > MS/s, because the DRAM bandwidth is not sufficient. If you are running the > samples at 100 MS/s or less (into the DUC) or if you are only using one > channel, it should be fine to use this. If you are not using the DRAMFIFO, > then the amount of buffering is dependent on the FIFOs that are built into > the Streaming Endpoints. You could check the yaml file that is used for > the X310 FPGA build. > > Perhaps you could explain a bit more about your intended application. It's > not clear to me if you are having issues with transmit streaming or if you > are also having issues with receive streaming. Are you planning to use > gnuradio? rfnoc (custom graph vs default multi_usrp)? C++ application? > Rob > > On Tue, Dec 24, 2024 at 3:07 PM <cjohnson@serranosystems.com> wrote: > >> *Hi,* >> >> *Hopefully you can help me out with some clarification and help on a few >> questions.* >> >> *We are using the X310 all with timestamps as we try to get an >> understanding of the behavior. We have created several scenarios to try >> out start of burst and end of burst and have noticed different behavior. * >> >> *Scenario 1 is a “single burst” for a bunch of samples sent (for example >> 1996 samples of std::vector<std::complex<int16_t>>) where >> start_of_burst=end_of_burst=true via send(includes timeout). * >> >> *Scenario 2 is a “continuous burst” where we start_of_burst=true for the >> first 1996 packet only with end_of_burst=false. All other packets have >> end_of_burst=false* >> >> >> *1) What is the difference between the "S" and "L"? When we see "S" with >> the single-burst case, is that really the same as an "L"?* >> >> *We have overhead to do between sending samples, so we are trying to >> figure out the best way to do the overhead work and still send() on time at >> our desired rate (still using timestamps).* >> >> *We noticed through experimentation that the “system” (PC TCP/IP Stack >> <-> USRP buffering) could buffer a very large packet of 1996*65=129,740 >> samples before we received flow-control, which means that we could send a >> bunch of samples with future timestamps to give us some time to do the >> “overhead”. Of course 1996 is used because it is the largest UDP packet >> before fragmentation occurs with UDP on USRP.* >> >> *2) Is our observation about the 129K samples of buffering correct? Is >> there something we can do to increase this? For example, maybe some tweaks >> to the receive buffer sizes in the TCP/IP stack to make receive work >> reliably.* >> >> *3) What is the recommendation in terms of burst size (to allow us to do >> our “overhead”)* >> >> *Thanks* >> >> >> >> _______________________________________________ >> USRP-users mailing list -- usrp-users@lists.ettus.com >> To unsubscribe send an email to usrp-users-leave@lists.ettus.com >> > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >

There is another scenario where you can S's, which is when using bad network cables (this is most often a problem with cheap optical fibers). On Sun, Jan 5, 2025 at 11:57 PM Marcus D. Leech <patchvonbraun@gmail.com> wrote: > On 05/01/2025 17:19, cjohnson@serranosystems.com wrote: > > > > I hope everyone had a good holiday break! > > > > Would you mind providing some guidance on the 3 questions? > > > > Thanks > > > > > > _______________________________________________ > > USRP-users mailing list -- usrp-users@lists.ettus.com > > To unsubscribe send an email to usrp-users-leave@lists.ettus.com > I will be able to comment that 'S' on a transmit stream means that a > packet got dropped on the network between the host and > the radio. This shouldn't ever happen, but sometimes for a "fresh" > UDP stream, the kernel driver will drop a UDP frame while > it completes ARP transactions. But that should happen not very > often, and there's likely a kernel parameter that can > disable this behavior. But, for a fully-working network stack, my > understanding is that 'S' should be very rare unless you're > offering a packet load that the "stack" just cannot cope with because > it's running out of space to put your packets. > > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >

uhd::stream_args_t tx_stream_args("sc16", "sc16");

tx_stream_args.args\["enable_fc"\] = "1"; // Enable flow control

tx_stream_args.args\["stream_mode"\] = "full_packet"; // Required for handling flow control responses

tx_stream = usrp->get_tx_stream(tx_stream_args);

Hi Martin, Marcus & Rob, Thank you for your previous responses. Cy is working for me on this development. Let me jump in and explain what we are trying to accomplish. We are using an x310 USRP to transmit samples from a custom C++ application over a 10Gb Ethernet interface. The target sample rate is 200 MSps, with 16-bit I/Q samples. We have receive functions running running reliably over the 10Gb Ethernet interface at the 200 MSps rate. We would like to stream the transmit samples continuously, and we want the sample timing to be synchronized to GPS time (our x310 is fitted with the internal GPSDO). We understand that we may have occasional errors on the 10Gb Ethernet link, and that we will need to recover from them. We are trying to understand the best combination of burst size and packet size to accomplish this. We initially tried making the burst size and the packet size the same. So each packet has 1996 samples, and the metadata in each packet has start_of_burst, end_of_burst, and has_timestamp all set true. The time_spec in the metadata is updated by 1996 ticks for each packet, corresponding to the number of samples in each packet. We have flow control enabled on the transmit stream: uhd::stream_args_t tx_stream_args("sc16", "sc16"); tx_stream_args.args\["enable_fc"\] = "1"; // Enable flow control tx_stream_args.args\["stream_mode"\] = "full_packet"; // Required for handling flow control responses tx_stream = usrp->get_tx_stream(tx_stream_args); Similar to the tx_timed_samples example, we set the time_spec of the first packet to be 1 second in the future, to be sure that we have enough samples queued up in the USRP before it actually begins transmitting. This approach does not seem to work at all. We observe that after sending 64 packets, the call to tx_send() takes almost one second to return. So we believe the flow control is working, and there are 64 packets queue in the USRP. Very soon after that, we get an async message with EVENT_CODE_BURST_ACK. The time_spec in that message is one sample time after the end of the first packet. Very soon after that, we receive a number of async messages with EVENT_CODE_TIME_ERROR. The time_spec in each of these appears to be 2 samples after the end of the corresponding packet. 1\. Is is possible to send adjacent bursts? That is, with time_spec values that are exactly 1996 samples after the previous burst? Or is some gap required between the bursts? Based on some experiments with the tx_timed_samples example, I think there must be a gap of at least one sample between the bursts. 2\. Is there any limit to how long a burst can be? Currently, we are thinking to send the samples in one continuous burst. Then, if an error is detected, (e.g. EVENT_CODE_TIME_ERROR, EVENT_CODE_UNDERFLOW), we will end the burst and start a new burst. Only the first packet in the burst will include the time_spec parameter. Thanks for your help! Neal Riedel Serrano Systems, Inc.

Hi Neal, for continuous streaming, don't use EOB. Just always include timestamps. I'm not sure right now on adjacent bursts, you're probably right, but they're the wrong tool here anyway. There is, however, no limit on burst sizes. That means if your application is running for days, and let's assume we don't drop anything, you would have one very long burst. The USRP will hold a packet until the transmit-time has been reached. If you always add timestamps to the send() call, then every packet will have timestamps (if you call send() with more data than fits into one packet, then the streamer code will calculate timestamps for all outgoing packets). Like I said in the other thread, on X310, you have approx. 500 kB of buffer which corresponds to 64 packets of 8000 bytes each. If you lose a packet between the host and the USRP, then you will eventually see an "S" and a "U". And of course, you will have a gap in your transmission. But on the next packet, as long as it reaches the radio on time, you will be aligned again. EOBs will only get in the way, because the USRP will ACK every packet. So basically, only send an EOB when your application is going into shutdown, and you want to cleanly exit the transmit thread. Side note: You don't need to specify `enable_fc` (it's on by default), and you don't need to specify `full_packet` either (that's only relevant for Rx raw UDP streaming). --M On Wed, Jan 8, 2025 at 7:34 PM <nriedel@serranosystems.com> wrote: > Hi Martin, Marcus & Rob, > > Thank you for your previous responses. Cy is working for me on this > development. Let me jump in and explain what we are trying to accomplish. > > We are using an x310 USRP to transmit samples from a custom C++ > application over a 10Gb Ethernet interface. The target sample rate is 200 > MSps, with 16-bit I/Q samples. We have receive functions running running > reliably over the 10Gb Ethernet interface at the 200 MSps rate. > > We would like to stream the transmit samples continuously, and we want the > sample timing to be synchronized to GPS time (our x310 is fitted with the > internal GPSDO). > > We understand that we may have occasional errors on the 10Gb Ethernet > link, and that we will need to recover from them. We are trying to > understand the best combination of burst size and packet size to accomplish > this. > > We initially tried making the burst size and the packet size the same. So > each packet has 1996 samples, and the metadata in each packet has > start_of_burst, end_of_burst, and has_timestamp all set true. The time_spec > in the metadata is updated by 1996 ticks for each packet, corresponding to > the number of samples in each packet. We have flow control enabled on the > transmit stream: > > uhd::stream_args_t tx_stream_args("sc16", "sc16"); > > tx_stream_args.args["enable_fc"] = "1"; // Enable flow control > > tx_stream_args.args["stream_mode"] = "full_packet"; // Required for > handling flow control responses > > tx_stream = usrp->get_tx_stream(tx_stream_args); > > Similar to the tx_timed_samples example, we set the time_spec of the first > packet to be 1 second in the future, to be sure that we have enough samples > queued up in the USRP before it actually begins transmitting. > > This approach does not seem to work at all. We observe that after sending > 64 packets, the call to tx_send() takes almost one second to return. So we > believe the flow control is working, and there are 64 packets queue in the > USRP. > > Very soon after that, we get an async message with EVENT_CODE_BURST_ACK. > The time_spec in that message is one sample time after the end of the first > packet. Very soon after that, we receive a number of async messages with > EVENT_CODE_TIME_ERROR. The time_spec in each of these appears to be 2 > samples after the end of the corresponding packet. > > 1. Is is possible to send adjacent bursts? That is, with time_spec values > that are exactly 1996 samples after the previous burst? Or is some gap > required between the bursts? Based on some experiments with the > tx_timed_samples example, I think there must be a gap of at least one > sample between the bursts. > > 2. Is there any limit to how long a burst can be? Currently, we are > thinking to send the samples in one continuous burst. Then, if an error is > detected, (e.g. EVENT_CODE_TIME_ERROR, EVENT_CODE_UNDERFLOW), we will end > the burst and start a new burst. Only the first packet in the burst will > include the time_spec parameter. > > Thanks for your help! > > Neal Riedel > > Serrano Systems, Inc. > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >

Hi Neal, I want to modify what I previously said about DRAMFIFO. In recent UHD versions, the Replay block has taken on the functionality of what used to be the DRAMFIFO (i.e., the Replay block can act as a FIFO using the onboard RAM). There is apparently a tx_streamer stream arg "streamer = replay_buffered" that will cause the multi_usrp object to include the Replay block in the graph automatically and use it as a FIFO. I have not tried this, but there is some documentation here <https://uhd.readthedocs.io/en/latest/page_configuration.html#config_stream_args_args> . Rob On Wed, Jan 8, 2025 at 3:12 PM Martin Braun <martin.braun@ettus.com> wrote: > Hi Neal, > > for continuous streaming, don't use EOB. Just always include timestamps. > I'm not sure right now on adjacent bursts, you're probably right, but > they're the wrong tool here anyway. > > There is, however, no limit on burst sizes. That means if your application > is running for days, and let's assume we don't drop anything, you would > have one very long burst. > > The USRP will hold a packet until the transmit-time has been reached. If > you always add timestamps to the send() call, then every packet will have > timestamps (if you call send() with more data than fits into one packet, > then the streamer code will calculate timestamps for all outgoing packets). > Like I said in the other thread, on X310, you have approx. 500 kB of buffer > which corresponds to 64 packets of 8000 bytes each. > > If you lose a packet between the host and the USRP, then you will > eventually see an "S" and a "U". And of course, you will have a gap in your > transmission. But on the next packet, as long as it reaches the radio on > time, you will be aligned again. > > EOBs will only get in the way, because the USRP will ACK every packet. So > basically, only send an EOB when your application is going into shutdown, > and you want to cleanly exit the transmit thread. > > Side note: You don't need to specify `enable_fc` (it's on by default), and > you don't need to specify `full_packet` either (that's only relevant for Rx > raw UDP streaming). > > --M > > On Wed, Jan 8, 2025 at 7:34 PM <nriedel@serranosystems.com> wrote: > >> Hi Martin, Marcus & Rob, >> >> Thank you for your previous responses. Cy is working for me on this >> development. Let me jump in and explain what we are trying to accomplish. >> >> We are using an x310 USRP to transmit samples from a custom C++ >> application over a 10Gb Ethernet interface. The target sample rate is 200 >> MSps, with 16-bit I/Q samples. We have receive functions running running >> reliably over the 10Gb Ethernet interface at the 200 MSps rate. >> >> We would like to stream the transmit samples continuously, and we want >> the sample timing to be synchronized to GPS time (our x310 is fitted with >> the internal GPSDO). >> >> We understand that we may have occasional errors on the 10Gb Ethernet >> link, and that we will need to recover from them. We are trying to >> understand the best combination of burst size and packet size to accomplish >> this. >> >> We initially tried making the burst size and the packet size the same. So >> each packet has 1996 samples, and the metadata in each packet has >> start_of_burst, end_of_burst, and has_timestamp all set true. The time_spec >> in the metadata is updated by 1996 ticks for each packet, corresponding to >> the number of samples in each packet. We have flow control enabled on the >> transmit stream: >> >> uhd::stream_args_t tx_stream_args("sc16", "sc16"); >> >> tx_stream_args.args["enable_fc"] = "1"; // Enable flow control >> >> tx_stream_args.args["stream_mode"] = "full_packet"; // Required for >> handling flow control responses >> >> tx_stream = usrp->get_tx_stream(tx_stream_args); >> >> Similar to the tx_timed_samples example, we set the time_spec of the >> first packet to be 1 second in the future, to be sure that we have enough >> samples queued up in the USRP before it actually begins transmitting. >> >> This approach does not seem to work at all. We observe that after sending >> 64 packets, the call to tx_send() takes almost one second to return. So we >> believe the flow control is working, and there are 64 packets queue in the >> USRP. >> >> Very soon after that, we get an async message with EVENT_CODE_BURST_ACK. >> The time_spec in that message is one sample time after the end of the first >> packet. Very soon after that, we receive a number of async messages with >> EVENT_CODE_TIME_ERROR. The time_spec in each of these appears to be 2 >> samples after the end of the corresponding packet. >> >> 1. Is is possible to send adjacent bursts? That is, with time_spec values >> that are exactly 1996 samples after the previous burst? Or is some gap >> required between the bursts? Based on some experiments with the >> tx_timed_samples example, I think there must be a gap of at least one >> sample between the bursts. >> >> 2. Is there any limit to how long a burst can be? Currently, we are >> thinking to send the samples in one continuous burst. Then, if an error is >> detected, (e.g. EVENT_CODE_TIME_ERROR, EVENT_CODE_UNDERFLOW), we will end >> the burst and start a new burst. Only the first packet in the burst will >> include the time_spec parameter. >> >> Thanks for your help! >> >> Neal Riedel >> >> Serrano Systems, Inc. >> _______________________________________________ >> USRP-users mailing list -- usrp-users@lists.ettus.com >> To unsubscribe send an email to usrp-users-leave@lists.ettus.com >> > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >