Hello world,

I have an application that requires precise timing on N different E310 receivers at N locations.  Currently at each I have a burst receiver tracking nanosecond reception times of a periodic transmitter.

My problem is that once GPSDO lock is acquired and the flowgraph starts running, the clocks appear to be slipping in comparison to one another. The plots attached show time differences between two devices over 3 hours. One of the plots has the linear regression subtracted out. An earlier test looked similar with a .35 ns/s differential slip between receivers. This chart shows 103 ns/s drift. Ideally there would be zero differential offset + some random walk.

I have observed this phenomenon on several builds to varying degrees:

E310 Release 3 (2015 July)

E310 Fido Alpha (2015 October)

$uname -a; gnuradio-config-info -v;uhd_find_devices
Linux sdr04 3.14.2-xilinx #1 SMP PREEMPT Tue Sep 22 12:47:18 PDT 2015 armv7l GNU/Linux
3.7.8
linux; GNU C++ version 4.9.2; Boost_105700; UHD_003.009.000-0-unknown

I am using Michael West's (michael.west@ettus.com) guidance on GPSDO synchronization posted 2015-11-11:

The E310 does only have the internal clock reference, but the E310 does lock the reference clock to the selected PPS.  It does take several seconds for the reference to lock, so you need to poll the "ref_locked" sensor after changing the time source to the GPSDO until you see a lock.

I also recommend changing the time setting algorithm to:

1. Poll on usrp->get_time_last_pps() until a change is seen.
2. Sleep 200ms (allow NMEA string to propagate)
3. Use "usrp->set_time_next_pps(uhd::time_spec_t(usrp->get_mboard_sensor("gps_time").to_int()+1));" to set the time
4. Poll on usrp->get_time_last_pps() until a change is seen.
5. Sleep 200ms (allow NMEA string to propagate)
6. Verify that usrp->get_time_last_pps() and usrp->get_mboard_sensor("gps_time") return the same time.
   Examining the FPGA firmware we have come to the conclusion that the 'ref_locked' sensor is set true when the internal clock is within 40 nanoseconds of the GPSDO 10MHz. Is it possible that once this occurs the PLL is no longer running? There appears be be no additional synchronization after this first step.

We have considered tagging samples periodically w/$GPGGA strings and doing a live calibration of how many samples elapsed vs. expected, but this seems a bit hacky.

Thanks for looking.

Kyle Logue · The Aerospace Corporation · ETG-DCID · Senior MTS · 310.336.3038 · kyle.logue@aero.orgmailto:kyle.logue@aero.org

(I work with Kyle).

The original drift estimate was 35ns per second (not 0.35ns).

And here are some observations on the gpsdo firmware pps tracking:

There is a 40 MHz VCO (i.e. VCTCXO) that is tuned to match PPS's (or an external 10MHz that appears to not be an option on the E310).  That is multiplied up to a 200 MHz clock  (via Xilinx DCM/PLL) which appears to do much of the time stamping (or at least is used in keeping the VCO disciplined to the PPS).  Assuming we have 35 ns/sec drift between two 40'ish MHz clocks, then that means (for example) one clock ticks 40e6 cycles in 1 sec and one clock ticks 40e6 cycles in 1.000000035 seconds (or 1-35ns depending on pos drift or neg). So the other freq is +/- 1.4 Hz off.

Offset = 40e6 - 40e6/(1 +/- 35e-9) = +/- 1.4 Hz

Since the 40 MHz is used to derive the 200 MHz, then the 200 MHz will be about 7 Hz off (relative to the other unit).

Given a 1PPS (we can obviously only update our clocks once per second) and a 200 MHz clock, it can only detect offsets in multiples of 5ns (i.e. 1 clock cycle), plus there's some filter-type feedback (1st order loop) into a DAC that drives the VCO which adds some uncertainty. The comments in the FPGA code say they did some modeling and then hand tuned it since it didn't match the model.  It is a first-order filter, so if we have drifts due to temperature, there's a chance it won't be fully tracked and therefore each unit might have a different lag (due to VCO/loop gain variation) and have different effective frequencies.  However, this can't go on indefinitely since the temperature cannot always rise and I doubt the temp changes enough to cause just a long-term consistent offset.

I couldn't find the VCTCXO part to see how sensitive it is to the Voltage input to even estimate the loop gain.

Eric McDonald, PhD
Senior Engineering Specialist
Digital Communication Implementation Department
The Aerospace Corporation
310.336.0976
eric.j.mcdonald@aero.orgmailto:eric.j.mcdonald@aero.org

From: USRP-users [mailto:usrp-users-bounces@lists.ettus.com] On Behalf Of Kyle A Logue via USRP-users
Sent: Friday, December 11, 2015 4:29 PM
To: Ettus Mail List usrp-users@lists.ettus.com
Subject: [USRP-users] GPSDO Timing Sync Across Devices: Slippery Clock

Hello world,

I have an application that requires precise timing on N different E310 receivers at N locations.  Currently at each I have a burst receiver tracking nanosecond reception times of a periodic transmitter.

My problem is that once GPSDO lock is acquired and the flowgraph starts running, the clocks appear to be slipping in comparison to one another. The plots attached show time differences between two devices over 3 hours. One of the plots has the linear regression subtracted out. An earlier test looked similar with a .35 ns/s differential slip between receivers. This chart shows 103 ns/s drift. Ideally there would be zero differential offset + some random walk.

I have observed this phenomenon on several builds to varying degrees:

E310 Release 3 (2015 July)

E310 Fido Alpha (2015 October)

$uname -a; gnuradio-config-info -v;uhd_find_devices
Linux sdr04 3.14.2-xilinx #1 SMP PREEMPT Tue Sep 22 12:47:18 PDT 2015 armv7l GNU/Linux
3.7.8
linux; GNU C++ version 4.9.2; Boost_105700; UHD_003.009.000-0-unknown

I am using Michael West's (michael.west@ettus.commailto:michael.west@ettus.com) guidance on GPSDO synchronization posted 2015-11-11:
The E310 does only have the internal clock reference, but the E310 does lock the reference clock to the selected PPS.  It does take several seconds for the reference to lock, so you need to poll the "ref_locked" sensor after changing the time source to the GPSDO until you see a lock.
I also recommend changing the time setting algorithm to:

1. Poll on usrp->get_time_last_pps() until a change is seen.
2. Sleep 200ms (allow NMEA string to propagate)
3. Use "usrp->set_time_next_pps(uhd::time_spec_t(usrp->get_mboard_sensor("gps_time").to_int()+1));" to set the time
4. Poll on usrp->get_time_last_pps() until a change is seen.
5. Sleep 200ms (allow NMEA string to propagate)
6. Verify that usrp->get_time_last_pps() and usrp->get_mboard_sensor("gps_time") return the same time.
   Examining the FPGA firmware we have come to the conclusion that the 'ref_locked' sensor is set true when the internal clock is within 40 nanoseconds of the GPSDO 10MHz. Is it possible that once this occurs the PLL is no longer running? There appears be be no additional synchronization after this first step.

We have considered tagging samples periodically w/$GPGGA strings and doing a live calibration of how many samples elapsed vs. expected, but this seems a bit hacky.

Thanks for looking.

Kyle Logue * The Aerospace Corporation * ETG-DCID * Senior MTS * 310.336.3038 * kyle.logue@aero.orgmailto:kyle.logue@aero.org

Michael:

Subsequent testing has shown the following code to correctly syncronize (2) E310s using the GPSDO.

I believe my previous error was not including the 200ms sleep intervals:

def waitforGPS(usrp, snap=True):
'''
Align sample clock with GPSDO

Tested on USRP E310 Release 3 & Nov 2015 Alpha
TODO: Set clock_source? (if N210)
github.com/EttusResearch/uhd/blob/master/host/utils/query_gpsdo_sensors.cpp

Check GPSDO output with 'gpsmon' or 'cgps'
gpspipe -r | grep GPGGA
'''
### 0. Check for GPSDO Sensor
print('--','Checking for GPS Lock Sensor')
sensors = usrp.get_mboard_sensor_names()
if 'gps_locked' not in sensors:
    print('\n>>','GPS Sensor Not Found. GPSDO Installed?')
    exit(1)
### 1. Explicitly Set Time Source to GPSDO
print('--','Explicitly Set Time Source to GPSDO')
usrp.set_time_source('gpsdo',0)
### 2. Wait for GPS Lock (internal criterion = ?)
print('--','Waiting for GPS Lock Sensor...')
while usrp.get_mboard_sensor('gps_locked').value != 'true':
    time.sleep(1)
### 3. Wait for MB 10MHz REF Lock to GPSDO
print('--','Wait for MB 10MHz REF Lock to GPSDO...')
while usrp.get_mboard_sensor('ref_locked').value != 'true':
    time.sleep(1)
print('--','Extra Sleep for <40ns alignment in ppsloop.v')
time.sleep(3) # let the clock settle a little bit longer
### 4. Excplicitly set PPS Time
print('--','Explicity seting PPS Time...')
cur_pps = last_pps = usrp.get_time_last_pps().to_ticks(1.0)
while cur_pps == last_pps: # Wait for PPS Edge
    cur_pps = usrp.get_time_last_pps().to_ticks(1.0)
time.sleep(0.2) # allow NMEA string to propagate
usrp.set_time_next_pps(
    uhd.time_spec_t(usrp.get_mboard_sensor('gps_time').to_int() + 1));
cur_pps = last_pps = usrp.get_time_last_pps().to_ticks(1.0)
while cur_pps == last_pps: # Wait for PPS Edge
    cur_pps = usrp.get_time_last_pps().to_ticks(1.0)
time.sleep(0.2) # allow NMEA string to propagate
### 5. Check UHD, GPS, and Host time
print('--','Check UHD, GPS, and Host UTC Epoch')
gps_seconds = usrp.get_mboard_sensor('gps_time').to_int()
pps_seconds = usrp.get_time_last_pps().to_ticks(1.0)
dev_seconds = int(time.time())
print('  ','GPS  Time = {}'.format(gps_seconds),'(set by GPS)')
print('  ','USRP Time = {}'.format(pps_seconds),'(should be == GPS)')
print('  ','HOST Time = {}'.format(dev_seconds),'(set manually)')
if gps_seconds == pps_seconds:
    print('>> GPSDO Sync GOOD')
else:
    print('>> GPSDO Sync FAIL')
    exit(1)
### 6. Optionally set start time to a logical interval
if snap:
    interval = 15 # number of seconds
    slop = 5 # minimum closest start time
    dtime = usrp.get_time_now(0).get_real_secs()
    dttuple = datetime.datetime.utcfromtimestamp(dtime)
    starttuple = datetime.datetime(
        dttuple.year,
        dttuple.month,
        dttuple.day,
        dttuple.hour,
        dttuple.minute) + datetime.timedelta(
        seconds=((dttuple.second+slop) // 15)*15 + 15)
    startepoch = (starttuple - datetime.datetime(1970,1,1)).total_seconds()
    usrp.set_start_time(uhd.time_spec_t(startepoch))
    print('>>','Start time set for',starttuple.isoformat())

We still have an arbitrary time bias offset in the recievers, but we can tolerate that for now. In the image attached the bias was ~2300 nanoseconds consistently over an 18 hour recording.

I'm open to improvements, but this is working OK for now.

PS: I see that the new gnuradio release (3.9.2) mentions:

B2XX, E3XX, X3XX: Easier time-syncing features.

Is this something I should be looking forward to?

Thanks,

Kyle Logue

From: Michael West michael.west@ettus.com
Sent: Wednesday, December 16, 2015 17:04
To: Kyle A Logue
Cc: Ettus Mail List
Subject: Re: [USRP-users] GPSDO Timing Sync Across Devices: Slippery Clock

Hi Kyle,

The digital PLL in the E310 will claim a lock if the offset is within 40ns.  The PLL continuously runs and contains a PI control, so it should be more accurate over time.  The VCTCXO is very sensitive to temperature, so a rapid change in temperature can cause the PLL to unlock.  I doubt that is what you are experiencing, but you could run an experiment where you constantly monitor the ref_locked sensor and see if it ever unlocks.

But let's start with the basics.  Can you share the actual code you are using to initialize the E310?

Regards,
Michael

On Fri, Dec 11, 2015 at 4:29 PM, Kyle A Logue via USRP-users <usrp-users@lists.ettus.commailto:usrp-users@lists.ettus.com> wrote:

Hello world,

I have an application that requires precise timing on N different E310 receivers at N locations.  Currently at each I have a burst receiver tracking nanosecond reception times of a periodic transmitter.

My problem is that once GPSDO lock is acquired and the flowgraph starts running, the clocks appear to be slipping in comparison to one another. The plots attached show time differences between two devices over 3 hours. One of the plots has the linear regression subtracted out. An earlier test looked similar with a .35 ns/s differential slip between receivers. This chart shows 103 ns/s drift. Ideally there would be zero differential offset + some random walk.

I have observed this phenomenon on several builds to varying degrees:

E310 Release 3 (2015 July)

E310 Fido Alpha (2015 October)

$uname -a; gnuradio-config-info -v;uhd_find_devices
Linux sdr04 3.14.2-xilinx #1 SMP PREEMPT Tue Sep 22 12:47:18 PDT 2015 armv7l GNU/Linux
3.7.8
linux; GNU C++ version 4.9.2; Boost_105700; UHD_003.009.000-0-unknown

I am using Michael West's (michael.west@ettus.commailto:michael.west@ettus.com) guidance on GPSDO synchronization posted 2015-11-11:

The E310 does only have the internal clock reference, but the E310 does lock the reference clock to the selected PPS.  It does take several seconds for the reference to lock, so you need to poll the "ref_locked" sensor after changing the time source to the GPSDO until you see a lock.

I also recommend changing the time setting algorithm to:

1. Poll on usrp->get_time_last_pps() until a change is seen.
2. Sleep 200ms (allow NMEA string to propagate)
3. Use "usrp->set_time_next_pps(uhd::time_spec_t(usrp->get_mboard_sensor("gps_time").to_int()+1));" to set the time
4. Poll on usrp->get_time_last_pps() until a change is seen.
5. Sleep 200ms (allow NMEA string to propagate)
6. Verify that usrp->get_time_last_pps() and usrp->get_mboard_sensor("gps_time") return the same time.
   Examining the FPGA firmware we have come to the conclusion that the 'ref_locked' sensor is set true when the internal clock is within 40 nanoseconds of the GPSDO 10MHz. Is it possible that once this occurs the PLL is no longer running? There appears be be no additional synchronization after this first step.

We have considered tagging samples periodically w/$GPGGA strings and doing a live calibration of how many samples elapsed vs. expected, but this seems a bit hacky.

Thanks for looking.

Kyle Logue · The Aerospace Corporation · ETG-DCID · Senior MTS · 310.336.3038tel:310.336.3038 · kyle.logue@aero.orgmailto:kyle.logue@aero.org

USRP-users mailing list
USRP-users@lists.ettus.commailto:USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Our time calculations are made based on a subsample cross-correlation on two receivers of a known signal. I use the time_spec to determine the alleged sample time, then add in a subsample offset based on the cross-correlation peak shape.

From: Michael West michael.west@ettus.com
Sent: Tuesday, January 5, 2016 16:54
To: Kyle A Logue
Cc: Ettus Mail List
Subject: Re: [USRP-users] GPSDO Timing Sync Across Devices: Slippery Clock

Hi Kyle,

Glad you have it working.  The improvement in UHD 3.9.2 is to make sure all the channels on a single device have synchronized times without the user having to explicitly make that happen.  It won't help trying to synchronize multiple devices.

Correct me if I'm wrong, but it looks like you are measuring bias by looking at the time_specs in the RX metadata.  Are you doing the calculation down to the sample resolution or just on the buffer level?

Regards,
Michael

On Tue, Jan 5, 2016 at 4:19 PM, Kyle A Logue <kyle.a.logue@aero.orgmailto:kyle.a.logue@aero.org> wrote:

Michael:

Subsequent testing has shown the following code to correctly syncronize (2) E310s using the GPSDO.

I believe my previous error was not including the 200ms sleep intervals:

def waitforGPS(usrp, snap=True):
'''
Align sample clock with GPSDO

Tested on USRP E310 Release 3 & Nov 2015 Alpha
TODO: Set clock_source? (if N210)
github.com/EttusResearch/uhd/blob/master/host/utils/query_gpsdo_sensors.cpp<http://github.com/EttusResearch/uhd/blob/master/host/utils/query_gpsdo_sensors.cpp>

Check GPSDO output with 'gpsmon' or 'cgps'
gpspipe -r | grep GPGGA
'''
### 0. Check for GPSDO Sensor
print('--','Checking for GPS Lock Sensor')
sensors = usrp.get_mboard_sensor_names()
if 'gps_locked' not in sensors:
    print('\n>>','GPS Sensor Not Found. GPSDO Installed?')
    exit(1)
### 1. Explicitly Set Time Source to GPSDO
print('--','Explicitly Set Time Source to GPSDO')
usrp.set_time_source('gpsdo',0)
### 2. Wait for GPS Lock (internal criterion = ?)
print('--','Waiting for GPS Lock Sensor...')
while usrp.get_mboard_sensor('gps_locked').value != 'true':
    time.sleep(1)
### 3. Wait for MB 10MHz REF Lock to GPSDO
print('--','Wait for MB 10MHz REF Lock to GPSDO...')
while usrp.get_mboard_sensor('ref_locked').value != 'true':
    time.sleep(1)
print('--','Extra Sleep for <40ns alignment in ppsloop.v')
time.sleep(3) # let the clock settle a little bit longer
### 4. Excplicitly set PPS Time
print('--','Explicity seting PPS Time...')
cur_pps = last_pps = usrp.get_time_last_pps().to_ticks(1.0)
while cur_pps == last_pps: # Wait for PPS Edge
    cur_pps = usrp.get_time_last_pps().to_ticks(1.0)
time.sleep(0.2) # allow NMEA string to propagate
usrp.set_time_next_pps(
    uhd.time_spec_t(usrp.get_mboard_sensor('gps_time').to_int() + 1));
cur_pps = last_pps = usrp.get_time_last_pps().to_ticks(1.0)
while cur_pps == last_pps: # Wait for PPS Edge
    cur_pps = usrp.get_time_last_pps().to_ticks(1.0)
time.sleep(0.2) # allow NMEA string to propagate
### 5. Check UHD, GPS, and Host time
print('--','Check UHD, GPS, and Host UTC Epoch')
gps_seconds = usrp.get_mboard_sensor('gps_time').to_int()
pps_seconds = usrp.get_time_last_pps().to_ticks(1.0)
dev_seconds = int(time.time())
print('  ','GPS  Time = {}'.format(gps_seconds),'(set by GPS)')
print('  ','USRP Time = {}'.format(pps_seconds),'(should be == GPS)')
print('  ','HOST Time = {}'.format(dev_seconds),'(set manually)')
if gps_seconds == pps_seconds:
    print('>> GPSDO Sync GOOD')
else:
    print('>> GPSDO Sync FAIL')
    exit(1)
### 6. Optionally set start time to a logical interval
if snap:
    interval = 15 # number of seconds
    slop = 5 # minimum closest start time
    dtime = usrp.get_time_now(0).get_real_secs()
    dttuple = datetime.datetime.utcfromtimestamp(dtime)
    starttuple = datetime.datetime(
        dttuple.year,
        dttuple.month,
        dttuple.day,
        dttuple.hour,
        dttuple.minute) + datetime.timedelta(
        seconds=((dttuple.second+slop) // 15)*15 + 15)
    startepoch = (starttuple - datetime.datetime(1970,1,1)).total_seconds()
    usrp.set_start_time(uhd.time_spec_t(startepoch))
    print('>>','Start time set for',starttuple.isoformat())

We still have an arbitrary time bias offset in the recievers, but we can tolerate that for now. In the image attached the bias was ~2300 nanoseconds consistently over an 18 hour recording.

I'm open to improvements, but this is working OK for now.

PS: I see that the new gnuradio release (3.9.2) mentions:

B2XX, E3XX, X3XX: Easier time-syncing features.

Is this something I should be looking forward to?

Thanks,

Kyle Logue

From: Michael West <michael.west@ettus.commailto:michael.west@ettus.com>
Sent: Wednesday, December 16, 2015 17:04
To: Kyle A Logue
Cc: Ettus Mail List
Subject: Re: [USRP-users] GPSDO Timing Sync Across Devices: Slippery Clock

Hi Kyle,

The digital PLL in the E310 will claim a lock if the offset is within 40ns.  The PLL continuously runs and contains a PI control, so it should be more accurate over time.  The VCTCXO is very sensitive to temperature, so a rapid change in temperature can cause the PLL to unlock.  I doubt that is what you are experiencing, but you could run an experiment where you constantly monitor the ref_locked sensor and see if it ever unlocks.

But let's start with the basics.  Can you share the actual code you are using to initialize the E310?

Regards,
Michael

On Fri, Dec 11, 2015 at 4:29 PM, Kyle A Logue via USRP-users <usrp-users@lists.ettus.commailto:usrp-users@lists.ettus.com> wrote:

Hello world,

I have an application that requires precise timing on N different E310 receivers at N locations.  Currently at each I have a burst receiver tracking nanosecond reception times of a periodic transmitter.

My problem is that once GPSDO lock is acquired and the flowgraph starts running, the clocks appear to be slipping in comparison to one another. The plots attached show time differences between two devices over 3 hours. One of the plots has the linear regression subtracted out. An earlier test looked similar with a .35 ns/s differential slip between receivers. This chart shows 103 ns/s drift. Ideally there would be zero differential offset + some random walk.

I have observed this phenomenon on several builds to varying degrees:

E310 Release 3 (2015 July)

E310 Fido Alpha (2015 October)

$uname -a; gnuradio-config-info -v;uhd_find_devices
Linux sdr04 3.14.2-xilinx #1 SMP PREEMPT Tue Sep 22 12:47:18 PDT 2015 armv7l GNU/Linux
3.7.8
linux; GNU C++ version 4.9.2; Boost_105700; UHD_003.009.000-0-unknown

I am using Michael West's (michael.west@ettus.commailto:michael.west@ettus.com) guidance on GPSDO synchronization posted 2015-11-11:

The E310 does only have the internal clock reference, but the E310 does lock the reference clock to the selected PPS.  It does take several seconds for the reference to lock, so you need to poll the "ref_locked" sensor after changing the time source to the GPSDO until you see a lock.

I also recommend changing the time setting algorithm to:

1. Poll on usrp->get_time_last_pps() until a change is seen.
2. Sleep 200ms (allow NMEA string to propagate)
3. Use "usrp->set_time_next_pps(uhd::time_spec_t(usrp->get_mboard_sensor("gps_time").to_int()+1));" to set the time
4. Poll on usrp->get_time_last_pps() until a change is seen.
5. Sleep 200ms (allow NMEA string to propagate)
6. Verify that usrp->get_time_last_pps() and usrp->get_mboard_sensor("gps_time") return the same time.
   Examining the FPGA firmware we have come to the conclusion that the 'ref_locked' sensor is set true when the internal clock is within 40 nanoseconds of the GPSDO 10MHz. Is it possible that once this occurs the PLL is no longer running? There appears be be no additional synchronization after this first step.

We have considered tagging samples periodically w/$GPGGA strings and doing a live calibration of how many samples elapsed vs. expected, but this seems a bit hacky.

Thanks for looking.

Kyle Logue · The Aerospace Corporation · ETG-DCID · Senior MTS · 310.336.3038tel:310.336.3038 · kyle.logue@aero.orgmailto:kyle.logue@aero.org

USRP-users mailing list
USRP-users@lists.ettus.commailto:USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com