Hi Tarik,

I didn't try to continously stream to storage of dual-channel full-rate
X3x0 recordings myself, but I do have some experience to share;
hopefully it helps more than it scares.

• Even NVMe SSDs aren't uniform in access latency and write speed. So,
  make sure your write rate is reliably above 1.6 GB/s

  • A marketing 2100MS/s write speed is what you'll get on average;
    you'd need that on "short-term minimum", with "short" being defined by
    how well your OS buffers writes in RAM.
  • A quick research on some "SSD user benchmarking" sites revealed the
    Samsung 970 PRO 1TB would indeed do more than 2.1 GB/s write speed –
    but only on strictly size-limited sequential writes, so you definitely
    don't want a journaling file system on the same device, or do
    anything with the SSD at the same time as storing samples. (Same site
    says "sustained write 1.5 GB/s", so, it really doesn't work out with
    but one of these)
  • Gut feeling: Get more CPU cores, probably more RAM for the
    buffering to compensate instantaneous write speed differences, and
    build a software RAID 0 out of at least two of these PCIe SSDs, or out
    of >> four SATA SSDs

• Storage subsystems usually aren't safe from preemption; you'll need
  to make sure that you have enough buffer to write things to while your
  storages system catches up after an interruption. Luckily, you can
  teach for example Linux how much storage write it is allowed to buffer
  into RAM before it starts blocking

• 2x200 MS/s vs Quad-Core: that's a lot of packets per second that
  you're trying to juggle with only four cores at 2.8 GHz, considering
  these cores handle the network packets, the unpacking of the samples
  from these, the file writing and the file system as well as storage
  interfacing.

• 16 GB RAM is certainly at the lower end of what I'd expect of a high-
  bandwidth storage system/workstation, but I don't think you'll need
  much more – after all, that's quite a lot of buffer (i.e. let's say 8s
  worth of samples, if you subtract all the RAM requirements of OS and
  applications that don't go into write buffering). Looking at the price
  of Dell's ECC RAM: you can probably live with some bit error
  probabilities in the 2e-11/bit/hr range (which I think is what's
  estimated for modern RAM) for noisy data without any problem, so ECC
  possibly isn't important here (but I don't know your operational
  requirements! So take with a grain of salt). The fact that you
  constantly overwrite with fresh values and thus most bits are probably
  shorter stored in RAM than one DRAM refresh cycle would be anyway does
  probably help physically, too.

Best regards,
Marcus

On Wed, 2018-06-27 at 08:30 +0000, Tarik Kazaz via USRP-users wrote:

Hi Marcus,

Thank you for your comments, although this scares me a bit.
Obviously storing samples at 1.6 GB/s will not be easy to achieve, or at least it is not straightforward.
I will wait a bit with further responses, I hope that someone else tried to do same.

Is it possible to use Dual 10Gbe connection towards two PCs (splitting single connection towards separate PCs)?

What I mean with this is to:
stream  IQ samples from one ADC0 (UBX0) to one PC at 0.8 GB/s and
stream IQ samples from another ADC1(UBX1) to another PC at 0.8 GB/s.

Maybe my idea is not valid, however might be solution.
Probably in this case there will be miss-alignment between samples stored at separate NVMe SSDs.

Kind Regards,

Tarik

-----Original Message-----
From: Marcus Müller [mailto:marcus.mueller@ettus.com]
Sent: woensdag 27 juni 2018 11:23
To: Tarik Kazaz; 'usrp-users@lists.ettus.com'
Subject: Re: [USRP-users] Streaming and storing signals of full BW of 2x UBX-160 cards to PC in file

Hi Tarik,

I didn't try to continously stream to storage of dual-channel full-rate
X3x0 recordings myself, but I do have some experience to share;
hopefully it helps more than it scares.

• Even NVMe SSDs aren't uniform in access latency and write speed. So,
  make sure your write rate is reliably above 1.6 GB/s

  • A marketing 2100MS/s write speed is what you'll get on average;
    you'd need that on "short-term minimum", with "short" being defined by
    how well your OS buffers writes in RAM.
  • A quick research on some "SSD user benchmarking" sites revealed the
    Samsung 970 PRO 1TB would indeed do more than 2.1 GB/s write speed –
    but only on strictly size-limited sequential writes, so you definitely
    don't want a journaling file system on the same device, or do
    anything with the SSD at the same time as storing samples. (Same site
    says "sustained write 1.5 GB/s", so, it really doesn't work out with
    but one of these)
  • Gut feeling: Get more CPU cores, probably more RAM for the
    buffering to compensate instantaneous write speed differences, and
    build a software RAID 0 out of at least two of these PCIe SSDs, or out
    of >> four SATA SSDs

• Storage subsystems usually aren't safe from preemption; you'll need
  to make sure that you have enough buffer to write things to while your
  storages system catches up after an interruption. Luckily, you can
  teach for example Linux how much storage write it is allowed to buffer
  into RAM before it starts blocking

• 2x200 MS/s vs Quad-Core: that's a lot of packets per second that
  you're trying to juggle with only four cores at 2.8 GHz, considering
  these cores handle the network packets, the unpacking of the samples
  from these, the file writing and the file system as well as storage
  interfacing.

• 16 GB RAM is certainly at the lower end of what I'd expect of a high-
  bandwidth storage system/workstation, but I don't think you'll need
  much more – after all, that's quite a lot of buffer (i.e. let's say 8s
  worth of samples, if you subtract all the RAM requirements of OS and
  applications that don't go into write buffering). Looking at the price
  of Dell's ECC RAM: you can probably live with some bit error
  probabilities in the 2e-11/bit/hr range (which I think is what's
  estimated for modern RAM) for noisy data without any problem, so ECC
  possibly isn't important here (but I don't know your operational
  requirements! So take with a grain of salt). The fact that you
  constantly overwrite with fresh values and thus most bits are probably
  shorter stored in RAM than one DRAM refresh cycle would be anyway does
  probably help physically, too.

Best regards,
Marcus

On Wed, 2018-06-27 at 08:30 +0000, Tarik Kazaz via USRP-users wrote:

About a year ago, we embarked on developing a similar application.

The TLDR is we never got to 2x200, but we did get 3x100, which to me is actually a bit better cuz 2x200 is really only 2x160 with the analog filtering, yet you have all those extra samples you have to take in for no benefit.

It is my conjecture that this was no fault of UHD, its rx processing is quite efficient for a single channel and single processor.  Do I wish I could control the thread affinity of UHD spawned threads?  Sure, but that would be an akward thing to expose in a platform independent and correct manner and few people ever need that (you are in the sub 1% zone, which is where we often play as well).

When it comes to streaming multiple 200 MSA/sec (800 MB/sec per stream) of data, the key is balancing resource utilization and the path the data takes from USRP device to disk.  EVERYTHING IS IMPORTANT HERE.

We used multiple PCIe SSDs (1.2 GB/sec sustained write, so essentially enough for a single 200 MSA/sec data stream with some margin) and essentially, we ran out of the ability to efficiently segment the data to dedicated lanes as you only get about 40 PCIe lanes in a single motherboard.

The chain you need to make sure is efficient is UHD device -> UHD receive -> app receive and buffer -> write to disk.  That is 4 threads per channel and even with affinity and priority etc, it is hard for the OS to handle that workload with the required latency to keep up...  Multiply this workload times the number of channels you want to receive and I think you start to get the problem.

We can chat further if you want offline if people find this too distracting :)

-----Original Message-----
From: USRP-users [mailto:usrp-users-bounces@lists.ettus.com] On Behalf Of Tarik Kazaz via USRP-users
Sent: Wednesday, June 27, 2018 7:41 AM
To: 'Marcus Müller' marcus.mueller@ettus.com; 'usrp-users@lists.ettus.com' usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Streaming and storing signals of full BW of 2x UBX-160 cards to PC in file

*** WARNING ***
EXTERNAL EMAIL -- This message originates from outside our organization.

Hi Marcus,

Thank you for your comments, although this scares me a bit.
Obviously storing samples at 1.6 GB/s will not be easy to achieve, or at least it is not straightforward.
I will wait a bit with further responses, I hope that someone else tried to do same.

Is it possible to use Dual 10Gbe connection towards two PCs (splitting single connection towards separate PCs)?

What I mean with this is to:
stream  IQ samples from one ADC0 (UBX0) to one PC at 0.8 GB/s and stream IQ samples from another ADC1(UBX1) to another PC at 0.8 GB/s.

Maybe my idea is not valid, however might be solution.
Probably in this case there will be miss-alignment between samples stored at separate NVMe SSDs.

Kind Regards,

Tarik

-----Original Message-----
From: Marcus Müller [mailto:marcus.mueller@ettus.com]
Sent: woensdag 27 juni 2018 11:23
To: Tarik Kazaz; 'usrp-users@lists.ettus.com'
Subject: Re: [USRP-users] Streaming and storing signals of full BW of 2x UBX-160 cards to PC in file

Hi Tarik,

I didn't try to continously stream to storage of dual-channel full-rate
X3x0 recordings myself, but I do have some experience to share; hopefully it helps more than it scares.

• Even NVMe SSDs aren't uniform in access latency and write speed. So, make sure your write rate is reliably above 1.6 GB/s

  • A marketing 2100MS/s write speed is what you'll get on average; you'd need that on "short-term minimum", with "short" being defined by how well your OS buffers writes in RAM.
  • A quick research on some "SSD user benchmarking" sites revealed the Samsung 970 PRO 1TB would indeed do more than 2.1 GB/s write speed – but only on strictly size-limited sequential writes, so you definitely don't want a journaling file system on the same device, or do
    anything with the SSD at the same time as storing samples. (Same site says "sustained write 1.5 GB/s", so, it really doesn't work out with but one of these)
  • Gut feeling: Get more CPU cores, probably more RAM for the buffering to compensate instantaneous write speed differences, and build a software RAID 0 out of at least two of these PCIe SSDs, or out of >> four SATA SSDs

• Storage subsystems usually aren't safe from preemption; you'll need to make sure that you have enough buffer to write things to while your storages system catches up after an interruption. Luckily, you can teach for example Linux how much storage write it is allowed to buffer into RAM before it starts blocking

• 2x200 MS/s vs Quad-Core: that's a lot of packets per second that you're trying to juggle with only four cores at 2.8 GHz, considering these cores handle the network packets, the unpacking of the samples from these, the file writing and the file system as well as storage interfacing.

• 16 GB RAM is certainly at the lower end of what I'd expect of a high- bandwidth storage system/workstation, but I don't think you'll need much more – after all, that's quite a lot of buffer (i.e. let's say 8s worth of samples, if you subtract all the RAM requirements of OS and applications that don't go into write buffering). Looking at the price of Dell's ECC RAM: you can probably live with some bit error probabilities in the 2e-11/bit/hr range (which I think is what's estimated for modern RAM) for noisy data without any problem, so ECC possibly isn't important here (but I don't know your operational requirements! So take with a grain of salt). The fact that you constantly overwrite with fresh values and thus most bits are probably shorter stored in RAM than one DRAM refresh cycle would be anyway does probably help physically, too.

Best regards,
Marcus

On Wed, 2018-06-27 at 08:30 +0000, Tarik Kazaz via USRP-users wrote:

USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com