Hi Vladimir,
I won't address all of your questions, but I will mention a few things
about our experience using X310s.

1. We use the 10Gb interface using a X520-DA2 board and SFP+ direct attach
   cable (5m cable from CablesOnDemand for $78).  This works quite well.  We
   rarely have any "communication issues".  We also use the same X520-DA2
   board with Intel E10GSFPSR SFP+ optical transceivers and LC/LC (OM3) fiber
   (~35m).  This also works quite well.
2. We use HP Workstations (Z230 SFF or Z440).  The Z230 has an i7-4770 cpu
   while the Z440 has a Xeon E5-1620 v3 cpu.  For both PCs, we have 32GB RAM
   installed.  This has been crucial for us because it allows us to set up a
   large RAM file system for streaming the RX data to "files" in RAM.
3. We can run the benchmark_rate utility and achieve 200 MS/s
   simultaneously on TX and RX.  Occasionally we will get a couple of
   underruns on the TX but it is not all that common.
4. When we try to stream data to files, we never have issues for RAM-based
   files.  But, on our Z440, we also have 4 SSDs (Samsung EVO 840) in a SW
   RAID 0 config which should be able to handle very large write speeds (~500
   MB/s per SSD), but we experience Overruns if we try to stream at rates
   higher than 100 MS/s.  (Note, all the rates I'm giving here are for single
   channel with the understanding that the rate should be divided by the
   number of channels).  We're still not sure how to fix this (via buffering
   or other).
5. We often use two X310s in a synchronized 4 channel RX configuration
   using the 2 SFP+ ports of the X520-DA2.  In this case if we try to record
   data to even RAM-based files, we cannot run at 200 MS/s on both X310s
   simultaneously.  This produces overruns indicating the PC is not keeping
   up.
6. Regarding PCIe, we haven't used this via UHD, but we have used it from
   LabVIEW.  We have the 8371 card, but we haven't tested it for high data
   rates.  I do know that the X310 PCIe interface is a Gen1 interface so it is
   only necessary to have a Gen1 host adapter card.  We have also successfully
   run the NI USRP RIO (i.e., X310) via a PCIe-over-fiber product from Samtec
   allowing us to use this interface with a remote device (again ~ 35m).
7. Regarding PCIe vs 10Gb ethernet, the latter is much more convenient with
   regard to being able to connect & disconnect without powering up/down the
   computer every time you want to change a cable or a device.  I have found
   both to be very reliable.  10Gb is also MUCH more convenient for running
   the device remotely over fiber.

Rob

On Thu, Jul 23, 2015 at 4:31 PM, Vladimir via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi Vladimir,

although he claimed not to answer your question, I think Rob did an
excellent job of providing the info you've asked for; thank you!

Hence, I'll keep this as short as possible
On 23.07.2015 22:31, Vladimir via USRP-users wrote:

That, or any direct attach SFP+ 10GE cable, or a pair of transceivers
for whatever electrical or optical medium you plan to use.

I must admit that this is news to me. But yes, PCIe is not the bus of
choice for maximum throughput. In fact, most PCIe 10Gigabit network
adapters, and especially the Intel devices, have drivers that are very
good of distributing work across multiple CPU cores and reducing
interrupt pressure -- it's in fact progressively hard for some customers
to get high rates (>ca. 150MS/s total) with the PCIe interface, mainly
because the NI driver for compatibility reasons is single threaded, and
a single core can only spin so fast.

I must admit I'm really not overly familiar with the NI side of PCIe /
PXI backend equipment. Generally, you only need to use /either/ 10GE
/or/ PCIe. Use PCIe if you want to combine things into large
LabVIEW-commanded clusters of USRPs  [1], or if you know that you need
the possibility to reduce USRP-host latency of the expense of more CPU
consumption. For the general user, I recommend 10GE.

You're quite right. So the point is that you can only use integer
factors of the master clock rate as sampling rates. For the default
Master Clock Rate of 200MHz, these are 200MS/s, 100MS/s, (66.666..MS/s),
50MS/s, ..., but there is also the 120MS/s master clock rate, which is
predominantly used by LabView, and can hence be used to get sampling
rates of 120, 60, (40), 30 MS/s. Also, we have an cellular-friendly
184.32 MHz master clock rate. Notice that I put some potential sampling
rates into parentheses; these are odd decimations of the master clock
rate, which hence don't exhibit the same quality of
flatness/anti-aliasing as the even decimations.

Uh, to be honest, I'm not quite sure about the "much better" part. "much
better" depends very much on your needs; if you couldn't use e.g. the
N210 for its frequency offset or drift being much too large, then you'll
want to use an external reference here, too. However, most customers are
quite happy with the oscillators we have on board -- simply because
frequency offset is a given fact in digital communications, and
typically, the oscillator outperforms things like mobile terminals very
significantly. This doesn't necessary apply to things like
high-performance interferometry, or very accurate signal detection.

No. Base Stations/eNodeBs define the clock of the cells they supply, so
you don't have to worry. The oscillators really aren't that bad -- they
just aren't perfect.

Well, you mentioned using UBX-160, but you /can/ use both daughterboard
slots with one daughterboard each, allowing you to do 2x2 MIMO.
A few typical frequently answered questions' answers:

• all RF interfaces are 50 Ohm matched
• the output of the UBX daughterboard can go up to /roughly/ 20dBm; for
  details, see [2]; safe input range is -infty to -15dBm.
• UHD itself doesn't need any special privileges and runs completely in
  userland. Only the network card or PCIe bridge card have kernel drivers.
• Ettus support will be on your side.

In my experience: 10GE. The problem here definitely is, as Rob
mentioned, that it's in no way "simple" to save 2x 100MS/s to disk.
There's been (and will always be, and that's a good and just thing) long
discussions on the needs, feasibility and example implementation support
of sustaining such hard drive write rates.
A quick calculation shows that, for 16bit short integer fixed point
complex recording:
datarate(100MS/s) = 16bit/number * 2numbers/complex 1complex/S *
100MS/s =3.2Gb/s for a single channel!
Now, you need to /guarantee/ these rates, meaning, that unless you can
come up with ridiculous amount of write buffering, the hard drive
minimum write rate must be above this value, and that doesn't even
account for the fact that data is typically written to file systems that
add even further overhead and latency. Now, I just came up with this
less than reliable source[2]; bad news is that the fastest hdd has an
average write rate of 193MB/s (ca 8200Mb/s = 1.6Gb/s) according to
their benchmark -- meaning that the minimum rate will be significantly
lower. So even a raid of 2 or 3 hard drives will not be sufficient, and
of four only if there weren't such things as seek latency. In short:
Writing that bulk of data to disk in realtime isn't easy anymore; but
RAM disks work fine, so be sure to have enough RAM.

Best regards,
Marcus Müller

[1]http://www.ni.com/white-paper/52382/en/
[2]
http://files.ettus.com/performance_data/ubx/UBX-without-UHD-corrections.pdf
[3] http://hdd.userbenchmark.com/

Hey,

On 24.07.2015 01:38, The Tilla via USRP-users wrote:

This is a particularly strong statement. Could you elaborate a bit more
on this? Do you (or anybody else) make any latency measurements over PCIe?

Just the fact that the NI driver isn't multi-threaded, which is a good
thing to know btw., doesn't necessarily mean it performs bad latency
wise. I guess this is partly also due to the fact that the original
hardware is often used in control engineering applications, and those
often have closed-loop algorithms that require low latency but not
necessarily high data rates.

Thanks
Andre

Hi guys,

I wish to thank all of you for valuable information, it's very helpful for us! Looking at all this, seems that we could start from 10 GBE link and then see if we need more latency or whatever else to try PCIe.

I forgot one question yesterday. From the point that has been discussed, how big is the difference btw X300 and X310? One of the diffs is that X310 has more memory - ~3.5 MB if I don't mistake, comparing to 2 MB in X300. Does it use this memory to do some sort of buffering when transferring data to/from host? Will it help to withstand any short hitches/delays that happen during streaming? For now, it's the only reason I could imagine to prefer X310 from this point of view.

Again, thank you very much for the info!
Vladimir

excellent job of providing the info you've asked for; thank you!

transceivers for whatever electrical or optical medium you plan to
use.

10Gigabit network adapters, and especially the Intel devices, have
drivers that are very good of distributing work across multiple CPU
cores and reducing interrupt pressure -- it's in fact progressively
hard for some customers to get high rates (>ca. 150MS/s total)
with the PCIe interface, mainly because the NI driver for
compatibility reasons is single threaded, and a single core can only
spin so fast.

/ PXI backend equipment. Generally, you only need to use either 10GE  or PCIe. Use PCIe if you want to combine things into
large LabVIEW-commanded clusters of USRPs  [1], or if you know that
you need the possibility to reduce USRP-host latency of the expense
of more CPU consumption. For the general user, I recommend 10GE.

factors of the master clock rate as sampling rates. For the default
Master Clock Rate of 200MHz, these are 200MS/s, 100MS/s,
(66.666..MS/s), 50MS/s, ..., but there is also the 120MS/s master
clock rate, which is predominantly used by LabView, and can hence be
used to get sampling rates of 120, 60, (40), 30 MS/s. Also, we have
an cellular-friendly 184.32 MHz master clock rate. Notice that I put
some potential sampling rates into parentheses; these are odd
decimations of the master clock rate, which hence don't exhibit the
same quality of flatness/anti-aliasing as the even decimations.

"much better" depends very much on your needs; if you couldn't use
e.g. the N210 for its frequency offset or drift being much too
large, then you'll want to use an external reference here, too.
However, most customers are quite happy with the oscillators we have
on board -- simply because frequency offset is a given fact in
digital communications, and typically, the oscillator outperforms
things like mobile terminals very significantly. This doesn't
necessary apply to things like high-performance interferometry, or
very accurate signal detection.

so you don't have to worry. The oscillators really aren't that bad
-- they just aren't perfect.

daughterboard slots with one daughterboard each, allowing you to do
2x2 MIMO.

in userland. Only the network card or PCIe bridge card have kernel
drivers.

mentioned, that it's in no way "simple" to save 2x 100MS/s to disk.
There's been (and will always be, and that's a good and just thing)
long discussions on the needs, feasibility and example
implementation support of sustaining such hard drive write rates.

complex recording:

100MS/s =3.2Gb/s for a single channel!

you can come up with ridiculous amount of write buffering, the hard
drive minimum write rate must be above this value, and that doesn't
even account for the fact that data is typically written to file
systems that add even further overhead and latency. Now, I just came
up with this less than reliable source[2]; bad news is that the
fastest hdd has an average write rate of 193MB/s (ca
8*200Mb/s = 1.6Gb/s) according to their benchmark -- meaning that
the minimum rate will be significantly lower. So even a raid of 2 or
3 hard drives will not be sufficient, and of four only if there
weren't such things as seek latency. In short: Writing that bulk of
data to disk in realtime isn't easy anymore; but RAM disks work
fine, so be sure to have enough RAM.

The difference is that the X310 has the bigger FPGA, which only makes a
difference if you want to do custom processing on it -- for example,
building a largish RF NoC application. Otherwise, X300 and X310 are
absolutely identical.

Best regards,
Marcus

On 24.07.2015 13:05, Vladimir via USRP-users wrote:

Marcus,

Yes, I understand this, but does it affect the connectivity with host, or its used only for signal processing algorithms? May be this is another question, but does X300 do any IO buffering at all, to provide accurate clocking of the samples when interfacing e.g. with ethernet? Is there some buffer there and how big is it?

Thanks,
Vladimir

makes a difference if you want to do custom processing on it -- for
example, building a largish RF NoC application. Otherwise, X300 and
X310 are absolutely identical.

  USRP-users wrote:

  helpful for us! Looking at all this, seems that we could start
  from 10 GBE link and then see if we need more latency or whatever
  else to try PCIe.

  discussed, how big is the difference btw X300 and X310? One of the
  diffs is that X310 has more memory - ~3.5 MB if I don't mistake,
  comparing to 2 MB in X300. Does it use this memory to do some sort
  of buffering when transferring data to/from host? Will it help to
  withstand any short hitches/delays that happen during streaming?
  For now, it's the only reason I could imagine to prefer X310 from
  this point of view.

    Marcus Müller via USRP-users <usrp-users@lists.ettus.com> :

            Rob did an excellent job of providing the info you've
            asked for; thank you!

            transceivers for whatever electrical or optical medium
            you plan to use.

            PCIe 10Gigabit network adapters, and especially the
            Intel devices, have drivers that are very good of
            distributing work across multiple CPU cores and reducing
            interrupt pressure -- it's in fact progressively hard
            for some customers to get high rates (>ca. 150MS/s
            total) with the PCIe interface, mainly because the NI
            driver for compatibility reasons is single threaded, and
            a single core can only spin so fast.

            side of PCIe / PXI backend equipment. Generally, you
            only need to use either 10GE  or PCIe. Use
            PCIe if you want to combine things into large
            LabVIEW-commanded clusters of USRPs  [1], or if you know
            that you need the possibility to reduce USRP-host
            latency of the expense of more CPU consumption. For the
            general user, I recommend 10GE.

            use integer factors of the master clock rate as sampling
            rates. For the default Master Clock Rate of 200MHz,
            these are 200MS/s, 100MS/s, (66.666..MS/s), 50MS/s, ...,
            but there is also the 120MS/s master clock rate, which
            is predominantly used by LabView, and can hence be used
            to get sampling rates of 120, 60, (40), 30 MS/s. Also,
            we have an cellular-friendly 184.32 MHz master clock
            rate. Notice that I put some potential sampling rates
            into parentheses; these are odd decimations of the
            master clock rate, which hence don't exhibit the same
            quality of flatness/anti-aliasing as the even
            decimations.

            better" part. "much better" depends very much on your
            needs; if you couldn't use e.g. the N210 for its
            frequency offset or drift being much too large, then
            you'll want to use an external reference here, too.
            However, most customers are quite happy with the
            oscillators we have on board -- simply because frequency
            offset is a given fact in digital communications, and
            typically, the oscillator outperforms things like mobile
            terminals very significantly. This doesn't necessary
            apply to things like high-performance interferometry, or
            very accurate signal detection.

            they supply, so you don't have to worry. The oscillators
            really aren't that bad -- they just aren't perfect.

            both daughterboard slots with one daughterboard each,
            allowing you to do 2x2 MIMO.

            to -15dBm.

            runs completely in userland. Only the network card or
            PCIe bridge card have kernel drivers.

            as Rob mentioned, that it's in no way "simple" to save
            2x 100MS/s to disk. There's been (and will always be,
            and that's a good and just thing) long discussions on
            the needs, feasibility and example implementation
            support of sustaining such hard drive write rates.

            fixed point complex recording:

            *1complex/S * 100MS/s =3.2Gb/s for a single channel!

            that unless you can come up with ridiculous amount of
            write buffering, the hard drive minimum write rate must
            be above this value, and that doesn't even account for
            the fact that data is typically written to file systems
            that add even further overhead and latency. Now, I just
            came up with this less than reliable source[2]; bad news
            is that the fastest hdd has an average write rate
            of 193MB/s (ca 8*200Mb/s = 1.6Gb/s) according to their
            benchmark -- meaning that the minimum rate will be
            significantly lower. So even a raid of 2 or 3 hard
            drives will not be sufficient, and of four only if there
            weren't such things as seek latency. In short: Writing
            that bulk of data to disk in realtime isn't easy
            anymore; but RAM disks work fine, so be sure to have
            enough RAM.

USRP-users mailing listUSRP-users@lists.ettus.com

Hi Vladimir,

no, from the computer's point of view, these devices are really identical.

There is some buffering on the X310 and X300, and as far as I know they
are absolutely identical (I'd have to read the FPGA source code [1]),
yes, but it's not large enough to buffer seconds of sample data -- that
would inherently lead to a lot of latency, and in almost all
applications, that would be unwanted. We actually do have two different
stock images for the FPGA, one using only SRAM (HGS), and one using DRAM
(XGS); the first is the default, and most customers use it.

Regarding accurate clocking: Both PCIe and 10GE are packet/burst-based,
so there's some kind of inevitable buffering; however, the USRP keeps
it's own reference oscillator-disciplined time, and gives you timestamps
with the RX samples, as well as allowing you to specify the exact time
when a sample should be sent, or a command (such as tuning, setting
gain) should be exectuted. That's why in many cases, the interface
latency doesn't matter at all -- you just make sure that the first bunch
of samples of your transmission burst is sent "early enough" to the
USRP, and add a timespec telling the USRP to hold these samples.

Setting the device time can be done e.g. by a GPSDO, or via the
interface on the edge of a PPS (pulse-per-second) signal (doesn't really
need to be a pulse per second if you really just want to set the device
time once -- a pull-up switch does work ;) ), or without external time
reference by just instructing the X3x0 to set a new time in its timing
registers.

All this is done via standard UHD API calls.

Best regards,
Marcus

[1] https://github.com/EttusResearch/fpga/tree/master/usrp3/top/x300
On 24.07.2015 13:19, Vladimir via USRP-users wrote:

Yes, there is an entire thread titles "X310 over PCIe Instability" started on February 27th.

I never got it stable enough to validate the latency improvements made on this page:

http://static1.1.sqspcdn.com/static/f/679473/23654458/1381240753367/grcon13_ettus_products.pdf?token=uxNR7AsfFTq%2FuYSVj2fywcDHq90%3D

It seemed to me that the interface is much more fragile to backups than Ethernet, but that is a bit of speculation.

Perhaps a different sample processing architecture with more application buffering would have had more success...

-----Original Message-----
From: USRP-users [mailto:usrp-users-bounces@lists.ettus.com] On Behalf Of Andre Puschmann via USRP-users
Sent: Friday, July 24, 2015 2:24 AM
To: usrp-users@lists.ettus.com
Subject: Re: [USRP-users] Hardware selection for X310-based system

Hey,

On 24.07.2015 01:38, The Tilla via USRP-users wrote:

This is a particularly strong statement. Could you elaborate a bit more on this? Do you (or anybody else) make any latency measurements over PCIe?

Just the fact that the NI driver isn't multi-threaded, which is a good thing to know btw., doesn't necessarily mean it performs bad latency wise. I guess this is partly also due to the fact that the original hardware is often used in control engineering applications, and those often have closed-loop algorithms that require low latency but not necessarily high data rates.

Thanks
Andre

USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hello,

Can anybody help me instructing how to obtain dBm values for usrp input stream? From what I have seen in online docs, I understood that now float values that uhd returns from the device are just sc16 ADC values scaled to some given range, is it right? That means they are proportional to amplitude values of the input signal, is it right? What would be the formula to rescale values for a given HF value? I understand that to measure power with some accuracy, theoretically I should make kind of calibration procedure for every HF in the range, but now we don't have to measure it accurate, just to have the estimation with moderate accuracy of what kind of signal level do we have, comparable to what regular spectrum analyzer shows. Or, is it already implemented in uhd and I just didn't find it?

Thank you!
Vladimir

You'd have to calibrate the instrument using a known signal generator.

On Sun, Jun 25, 2017 at 4:10 PM Vladimir via USRP-users <
usrp-users@lists.ettus.com> wrote:

--
Very Respectfully,

Dan CaJacob

Hi Vladimir,

the USRPs are not calibrated measurement equipment. The digital values
you see for real and imaginary part (in [-1,1]) of the complex numbers
you're getting from the USRP are proportional to the voltage seen by
the ADC, but that is also only proportional to the voltage seen at the
RX SMA connector. And these proportionality factors will depend on
frequency, gain, temperature, sample rate, …

So, it's really just a factor. But: that factor depends on very, very,
very many properties, and you'll need to calibrate with a known power
source, not only theoretically, but very practically!

So, get a source of known power, somehow get that power into the system
you want to calibrate (that often includes antennas, not only the
USRP!!), and then calibrate, at exactly the frequency, gain, sample
rate settings that you want to use. Make sure you're not driving the RX
chain into saturation!

Best regards,

Marcus

On 25.06.2017 22:09, Vladimir via USRP-users wrote:

Marcus,

Thank you for the clarification!

Vladimir

  values you see for real and imaginary part (in [-1,1]) of the
  complex numbers you're getting from the USRP are *proportional* to
  the voltage seen by the ADC, but that is also only proportional to
  the voltage seen at the RX SMA connector. And these
  proportionality factors will depend on frequency, gain,
  temperature, sample rate, …

  very, very many properties, and you'll need to calibrate with a
  known power source, not only theoretically, but very practically!

  system you want to calibrate (that often includes antennas, not
  only the USRP!!), and then calibrate, at *exactly* the frequency,
  gain, sample rate settings that you want to use. Make sure you're
  not driving the RX chain into saturation!

  USRP-users wrote:

  input stream? From what I have seen in online docs, I understood
  that now float values that uhd returns from the device are
  just sc16 ADC values scaled to some given range, is it right? That
  means they are proportional to amplitude values of the input
  signal, is it right? What would be the formula to rescale values
  for a given HF value? I understand that to measure power with some
  accuracy, theoretically I should make kind of calibration
  procedure for every HF in the range, but now we don't have to
  measure it accurate, just to have the estimation with moderate
  accuracy of what kind of signal level do we have, comparable to
  what regular spectrum analyzer shows. Or, is it already
  implemented in uhd and I just didn't find it?

USRP-users mailing list