At that timescale, I'd expect aging to be lost in the noise.

How are you calibrating things?

--
These are my opinions.  I hate spam.

Hi Hal,
I don't think I understand your question.  So, I've attached a plot and you can tell me if that gives you anything to work with.  This uses my standard plotting script, so there are things you aren't interested in.  But, this is a plot of one unit from startup on the night of 10/29 through right now.

The blue band is the plot of the TIC in my GPSDO.  The TIC is affected by noise, which is filtered by the PID software, as well as an LPF in the EFC line.  So, the OCXO output is much more stable than the blue band would indicate.  The red trace (the DAC voltage in hex as read on the left) is what's important for this discussion.  Notice the steady decrease in the DAC voltage over time.  The orange trace is the temperature.  For scaling, use the right side numbers divided by 10 to get delta degrees F.  IOW, from 0 to 10 would be a 1 degree Fahrenheit temperature change.  Other items of interest on the plot are TDOP, number of sats seen, and number of sats used.

So, looking at the plot, it seems clear that time dominates the change in the DAC voltage.  But, there is a noticeable impact from temperature change.  That impact is not linear, except that small changes do seem to affect it in a linear manner.
As to how I'm calibrating this:  I've got several GPSDOs running.  One is being used as the 10MHz reference for the 5370A.  Another is being used as the 1PPS reference for the one I'm testing.  Since these are essentially identical units, though with different firmware, the impact of ionospheric change on the results is muted.  So, what the testing boils down to is bringing up a unit with the firmware to be tested, and allowing it to be well locked before disconnecting the antenna.  Generally I leave it overnight and reconnect the antenna some time in the morning.  So, this gives me two figures:  One is how far the unit drifts over some time period, as well as the rate of recovery once the antenna is restored. 

I'd include a Timelab plot except that I don't have the two units skewed enough in time to allow for 1uS of drift.  So, the time reported on Timelab would be misleading due to the 5370 slipping into "one sample every two seconds" mode as the phase difference exceeds the time skew.  For the next test I think I'll skew the DUT by 2uS so that I can get a clean plot.

Anyway, does any of this answer your question?  If not, let me know what's missing.

Bob -----------------------------------------------------------------
AE6RV.com

GFS GPSDO list:
groups.yahoo.com/neo/groups/GFS-GPSDOs/info

  From: Hal Murray <hmurray@megapathdsl.net>

To: Bob Stewart bob@evoria.net; Discussion of precise time and frequency measurement time-nuts@febo.com
Cc: hmurray@megapathdsl.net
Sent: Saturday, November 5, 2016 12:55 PM
Subject: Re: [time-nuts] Thermal impact on OCXO

At that timescale, I'd expect aging to be lost in the noise.

How are you calibrating things?

--
These are my opinions.  I hate spam.

I think that's a nice plot, it looks like you have stepped 160 LSB over 7
days or roughly 1 LSB per hour. With a 20bit dac you are trimming maybe 1
ppt/LSB to 4 ppt/LSB? In allan devation terms, the case of 1ppt/LSB, solely
due to drift, you're at 1E-12 at 3600*sqrt(2) = 5000 s, in the case of
4ppt/hour your at 1E-12 at 1280 s. Seems reasonable.

On Sat, Nov 5, 2016 at 2:47 PM, Bob Stewart bob@evoria.net wrote:

If your DAC spans the full EFC range than 1LSB is 1/2^20 ~ 1 PPM of the EFC
range, and the EFC tuning range is 8/10E6 ~ 1 PPM full scale, so 1 LSB is
~1PPT. So, if everything else is stable the DAC code reflects
changes solely due to the OCXO, which would be an aging of 24 PPT/day.

On Sat, Nov 5, 2016 at 9:57 PM, Bob Stewart bob@evoria.net wrote:

Well bob's comments & caution are accurate, everything drifts. In your
case, if the OCXO is rock solid then you would see a 160 PPM change on the
EFC line over 7 days which is a 1mV change on your 6 V full scale, which is
fairly easy to measure if you have a 6 1/2 digit DMM.

On Sat, Nov 5, 2016 at 10:43 PM, Bob Stewart bob@evoria.net wrote:

Hi

OCXO’s don’t age in a linear fashion. At least 90% of them don’t. If you dig into the FCS papers there are various
curves proposed as models. Mil-O-55310 has one of them as the “official” approach. All of them have the basic
issue of mistakenly fitting to to short a time constraint.

Bob

Here is a sample data point taken from
http://tycho.usno.navy.mil/ptti/1987papers/Vol%2019_16.pdf; the first that
showed up on a google search.

     Year   Aging [PPM]  dF/dt [PPT/Day]
        1       180.51       63.884
        2       196.65        31.93
        5          218       12.769
        9       231.69       7.0934
       10       234.15        6.384
       25        255.5       2.5535

If you have a set of coefficients you believe to be representative of your
OCXO, we can give those a go.

On Mon, Nov 7, 2016 at 7:08 AM, Bob Camp kb8tq@n1k.org wrote:

Typo, "Aging [PPM]" should read "Aging [PPB]".

On Mon, Nov 7, 2016 at 10:34 AM, Scott Stobbe scott.j.stobbe@gmail.com
wrote:

Hi Scott,
I wish I had some long term data, but I don't.  I had initially set out to build an accurate GPS frequency reference type of GPSDO.  So, aging wasn't an issue.  It's either on and locked or it's not.  So, I didn't worry about leaving a unit running for months and collecting aging data.

But then I had this crackpot idea of using the latches in a 7474 to stabilize the 1PPS from the receiver to the OCXO.  Tom had a bit of a misunderstanding about what I was doing and we had a bit of a discussion until he caught on to what I was doing and its limitations. 

But, somewhere along the line, I realized that the idea was sound but my implementation was poor.  The basic problem with using a timer in the dsPIC33 is that they use a PLL to generate the internal clock - even if you supply a clock.  That gives you a 1-count jitter in any output pulse you try to create.  So, I realized that I could use the latches in a 7474 to latch the OCXO to the output of a timer on the PIC.  With the PIC running at 40MHz, I have plenty of room for the jitter without the worry of a phase slip.  I think I've proved that that works, so now I have the possibility of using my system as a time server.  And that means I now have to deal with such arcane matters as holdover, aging, and generating the time from the OCXO.  The learning curve has been a bit steep.
Bob -----------------------------------------------------------------
AE6RV.com

GFS GPSDO list:
groups.yahoo.com/neo/groups/GFS-GPSDOs/info

  From: Scott Stobbe <scott.j.stobbe@gmail.com>

To: Bob Camp kb8tq@n1k.org
Cc: Bob Stewart bob@evoria.net; Discussion of precise time and frequency measurement time-nuts@febo.com
Sent: Monday, November 7, 2016 9:34 AM
Subject: Re: [time-nuts] Thermal impact on OCXO

Here is a sample data point taken from http://tycho.usno.navy.mil/ptti/1987papers/Vol%2019_16.pdf; the first that showed up on a google search.

         Year   Aging [PPM]  dF/dt [PPT/Day]
            1       180.51       63.884
            2       196.65        31.93
            5          218       12.769
            9       231.69       7.0934
           10       234.15        6.384
           25        255.5       2.5535
If you have a set of coefficients you believe to be representative of your OCXO, we can give those a go.
On Mon, Nov 7, 2016 at 7:08 AM, Bob Camp kb8tq@n1k.org wrote:

Hi

OCXO’s don’t age in a linear fashion. At least 90% of them don’t. If you dig into the FCS papers there are various
curves proposed as models. Mil-O-55310 has one of them as the “official” approach. All of them have the basic
issue of mistakenly fitting to to short a time constraint.

Bob

On Mon, November 7, 2016 11:20 am, Bob Stewart wrote:

Isn't that the entire point of an OCXO?

--
Chris Caudle

On Mon, November 7, 2016 11:41 am, Chris Caudle wrote:

Actually, that is not very precise language on my part.  The OCXO does not
increase the temperature, it increases the heat input to the oven to
counteract the decrease in ambient temperature, i.e. attempts to keep the
internal oven temperature constant.

--
Chris Caudle

Hi Bob,
When you say that OCXOs don't age in a linear fashion, what does that mean?  IOW, is this a case where it's almost linear over a week, and the non-linearity is only detectable when you consider the longer term?  If that's the case, then it seems reasonable to use the past 3 or so days of data collection to project the near-term behavior; where near-term is less than 3 days into the future.

Bob -----------------------------------------------------------------
AE6RV.com

GFS GPSDO list:
groups.yahoo.com/neo/groups/GFS-GPSDOs/info

  From: Bob Camp <kb8tq@n1k.org>

To: Bob Stewart bob@evoria.net; Discussion of precise time and frequency measurement time-nuts@febo.com
Cc: Scott Stobbe scott.j.stobbe@gmail.com
Sent: Monday, November 7, 2016 6:08 AM
Subject: Re: [time-nuts] Thermal impact on OCXO

Hi

OCXO’s don’t age in a linear fashion. At least 90% of them don’t. If you dig into the FCS papers there are various
curves proposed as models. Mil-O-55310 has one of them as the “official” approach. All of them have the basic
issue of mistakenly fitting to to short a time constraint.

Bob

Hi

Ok, You have a thermocouple junction at the + post on the DVM. You have one at the - post on the DVM.
You have a junction at the + connection to the board. You have a junction at the - connection to the board.
There are indeed more than that, but those four are pretty much a sure thing. The ones near the hot OCXO
are also worth looking at.

The metals involved are unknown, so we have to guess a bit. You can get some alarming junctions with
very normal test lead materials. Getting low thermal EMF connections requires special attention.
If your junctions have a 1 mV / K coefficient, then you need temperature data that is good to 0.001C at each
junction to work out what is going on at the junctions. That assumes you are after 1 uV on your “data” plot.
It’s not terribly hard to get a contact at one (or more than one) junction that messes you up.

Bob

Hi

…. and as it does so, it pulls more current (or less) which changes the voltage
on the ground pin which hits the EFC reference which impacts things to a
greater or lesser extent depending on the board layout.

Bob

Hi

I would suggest you read the papers on the subject. They are out there. Even over a few days
OCXO aging is likely to be non-linear ( => curved, not straight line). On an OCXO that is
warming up, your are dealing with retrace rather than aging. This can have an impact for a
few months after turn on. I have one unit in the basement that took 9 months to come out
of retrace after being in storage for “a while”. It was an eBay item so no idea just how long
the power off time was.

Bob

-accidentally omitted timenuts from my reply to bob-
Hi Bob,
The data is at the 10uV level.  As to whether I've plotted "data" or data, I can only report what I have.  It seems to be consistent over the long run.  I would expect to see something else if it were just noise.  If the 4 attachments were strongly thermocoupled, wouldn't the data have a stronger correlation to temperature, rather than just to temperature transients?

Bob -----------------------------------------------------------------
AE6RV.com

GFS GPSDO list:
groups.yahoo.com/neo/groups/GFS-GPSDOs/info

  From: Bob Camp <kb8tq@n1k.org>

To: Bob Stewart bob@evoria.net
Cc: Discussion of Precise Time and Frequency Measurement time-nuts@febo.com
Sent: Monday, November 7, 2016 11:54 AM
Subject: Re: [time-nuts] Thermal impact on OCXO

Hi

I would suggest you read the papers on the subject. They are out there. Even over a few days
OCXO aging is likely to be non-linear ( => curved, not straight line). On an OCXO that is
warming up, your are dealing with retrace rather than aging. This can have an impact for a
few months after turn on. I have one unit in the basement that took 9 months to come out
of retrace after being in storage for “a while”. It was an eBay item so no idea just how long
the power off time was.

Bob

Hi

Thermocouples in test leads are nasty because they operate on the temperature
delta rather than the absolute temperature. It’s the gradient you have to worry about.
Until you have run out a few dozen sensors around your bench (or test chamber) and
monitored them for a few days … you won’t believe just how little “point A” has in common
with “point B”.

Bob

Hi Bob,
OK, given that, what is your view of the plot I presented?  Let's remember that I plotted this because I was trying to discover whether the EFC circuitry has a time-related drift, or  whether the DAC value drift I plotted with the GPSDO locked to the GPS was caused by the OCXO.

In one of your posts today, you talked about a long-term retrace lasting some months.  I've been using the term "aging".  Has my choice of terms caused the discussion to take a turn that it wouldn't have taken if I had used the term "retrace"?  The fact is that I don't know whether it's aging or retrace.  I'm not really clear on how to tell the difference.  I'm especially handicapped by not having the mfg specs for the OCXOs I use.  It could be that I simply haven't had the test unit under power for long enough to let the OCXO settle out.  And for sure when I added the test lead to the EFC (breaking a trace in the process) there was enough off time to thoroughly insult the OCXO.

Bob -----------------------------------------------------------------
AE6RV.com

GFS GPSDO list:
groups.yahoo.com/neo/groups/GFS-GPSDOs/info

  From: Bob Camp <kb8tq@n1k.org>

To: Bob Stewart bob@evoria.net; Discussion of precise time and frequency measurement time-nuts@febo.com
Sent: Monday, November 7, 2016 1:01 PM
Subject: Re: [time-nuts] Thermal impact on OCXO

Hi

Thermocouples in test leads are nasty because they operate on the temperature
delta rather than the absolute temperature. It’s the gradient you have to worry about.
Until you have run out a few dozen sensors around your bench (or test chamber) and
monitored them for a few days … you won’t believe just how little “point A” has in common
with “point B”.

Bob

On Mon, 7 Nov 2016 17:20:58 +0000 (UTC)
Bob Stewart bob@evoria.net wrote:

These are most likely caused by temperature gradients due to temperature
changes. Once the temperature is (almost) constant, these differences
equalize and the effect goes away. Where that gradient is and what part
of the circuit it affects is not clear. It could be your circuit,
or it could be the measurement.

At least the measurment looks like that temperature dependence is low
enough that it shouldn't be much of an issue.

		Attila Kinali

--
Malek's Law:
Any simple idea will be worded in the most complicated way.

On Mon, 7 Nov 2016 17:47:19 +0000 (UTC)
Bob Stewart bob@evoria.net wrote:

Have a look at John Vig's tutorial. There are 2-3 slides on aging and retrace
with references to papers for details.

		Attila Kinali

--
Malek's Law:
Any simple idea will be worded in the most complicated way.

Hi

This is your design, and under your control. Looking at one plot here or there and trying to
guess what is going on is not a really good way to do this. The whole sub-thread we are on
is simply a result of a “if it’s not crystal aging it can’t be anything else” comment way way back.
My point here is that it can be something else and the list of things it can be is actually quite
long. The voltage measurement stuff is simply one example out of at least a dozen and probably
closer to a hundred possibles. Digging through all of those possible things is a lot of work. It’s work
that can only be done “hands on” with a set of units and test gear.  My first step would be to toss
the setup into an environmental chamber and do some 48 hour temperature sweeps.

Aging is a “long term” drift. It goes on for years. Warmup is a very short term effect. It goes on for hours
or days. Retrace is the process of re-settilitg to the previous long term aging line. It can take days / weeks /
months depending on the oscillator. If you were buying the OCXO’s as an OEM, the specs you
bought them to would be a function of what you asked for and how much you were willing to pay. It
is unlikely that as an OEM you would have requirements on  retrace and warmup, and aging. If you
did have a spec it would be under fairly specific conditions (on power for three months, measure
frequency, off power at XX C for 24 hours, on power again for 24 hours at YY C, measure delta
frequency. That would cover warmup, but really says nothing about aging or retrace.

Regardless of if an OCXO is spec’d for this or that issue or not, it experiences it. There are a lot of
other things that OCXO’s do that many or may not be spec’d. When using one and digging into it,
you have to deal with all the issues, regardless of if there is a spec or not. That assumes you want
to dig in at this level. A very reasonable question is - what level do you want to dig into? That is
why I bring up the good old “what is the goal?” question on a regular basis. Without a goal in mind,
you will keep digging ever deeper. The number of issues you need to look into grows exponentially
as you dig deeper. The time, learning, equipment, experiments, and number of test units each go
up rapidly. It is work, equipment, and experiments that only apply to your design. It is not a piggyback
process as you go deeper.

Yes, that’s a bit of a long winded reply … sorry about that.

Bob

Another nice plot! It looks like after 2am you see temperature swings of
1.5 degF roughly every 30 minutes? Correspondingly, the EFC line which is
nominally ~2.8vdc sees swings of +-50 uV?

On Monday, 7 November 2016, Bob Stewart bob@evoria.net wrote:

Deeper into the rabbit hole :)

I'm not sure what specific pic you are you using but most of them have at
least one timer that will run fully asynchronously and its timer input is
usually shared with one of the 32k osc pins. Which you can then use to wake
the prossesor from one of its sleep states. Do you end up hitting a clock
sync with the pwm block?

Good old 74 series to the rescue.

On Monday, 7 November 2016, Bob Stewart bob@evoria.net wrote:

On Mon, Nov 7, 2016 at 10:34 AM, Scott Stobbe scott.j.stobbe@gmail.com
wrote:

I thought I would come back to this sample data point and see what the
impact of using a 1st order estimate for the log function would entail.

The coefficients supplied in the paper are the following:
A1 = 0.0233;
A2 = 4.4583;
A3 = 0.0082;

F =  A1ln( A2x +1 ) + A3;  where x is time in days

 Fdot = (A1*A2)/(A2*x +1)

 Fdotdot = -(A1*A2^2)/(A2*x +1)^2

When x is large, the derivatives are approximately:

 Fdot ~= A1/x

 Fdotdot ~= -A1/x^2

It's worth noting that, just as it is visually apparent from the graph, the
aging becomes more linear as time progresses, the second, third, ...,
derivatives drop off faster than the first.

A first order taylor series of the aging would be,

 T1(x, xo) = A3 + A1*ln(A2*xo + 1) +  (A1*A2)(x - xo)/(A2*xo +1) + O(

(x-xo)^2 )

The remainder (error) term for a 1st order taylor series of F would be:
R(x) = Fdotdot(c) * ((x-xo)^2)/(2!);  where c is some value between x
and xo.

So, take for example, forward projecting the drift one day after the 365th
day using a first order model,
xo = 365

 Fdot(365) =  63.796 PPT/day, alternatively the approximate derivative

is: 63.836 PPT/day

 |R(366)| =  0.087339 PPT (more than likely, this is no where near 1

DAC LSB on the EFC line)

More than likely you wouldn't try to project 7 days out, but considering
only the generalized effects of aging, the error would be:

 |R(372)| = 4.282 PPT (So on the 7th day, a 1st order model starts to

degrade into a few DAC LSB)

In the case of forward projecting aging for one day, using a 1st order
model versus the full logarithmic model, would likely be a discrepancy of
less than one dac LSB.

Hi

If you already have data over a year (or multiple years) the fit is fairly easy.
If you try to do this with data from a few days or less, the whole fit process is
a bit crazy. You also have multiple time constants involved on most OCXO’s.
The result is that an earlier fit will have a shorter time constant (and will ultimately
die out). You may not be able to separate the 25 year curve from the 3 month
curve with only 3 months of data.

Bob