time-nuts@lists.febo.com

Discussion of precise time and frequency measurement

View all threads

Re: [time-nuts] FreeBSD, NetBSD, or Minix-III?

HM
Hal Murray
Mon, May 18, 2009 8:12 AM

I need to go back and read what you are trying to measure with your
clock. Is it internal to the computer or an external event ?

I was thinking of a FPGA on a PCI bus.  It has to be PCI rather than USB in
order to get reasonable timings.

I was going to put the Unix clock in the FPGA.  It's a pair of 32 bit words.
The high word is seconds since some magic date/time.  The low word is
nano-seconds within this second.

I was planning to drive that with a stable external clock, say a GPSDO.  So if things were setup properly, it should be a very good clock.

The typical Unix clock is implemented by looking at the TSC.  I think that's Intel's term for a register that counts each CPU cycle, Time Stamp Counter.  The CPU comes from an inexpensive crystal.  The frequency of that crystal varies (wildly) with temperature.  The temperature varies with activity.

The basic idea is that you maintain a reference time and TSC value.  To get the current time, you read the TSC, subtract off the reference TSC.  That gives you the number of CPU clock ticks since the reference time.  Convert that to nanoseconds, add on the reference time, and normalize.  You have to update the reference info periodically, in particular before that subtract will over/underflow.

phk points out that you can use the TSC type clock logic by reading a simple counter from the FPGA rather than the TSC register.  That gets you a stable clock without all the problems with reading 64 bits from the FPGA.

You could also replace the CPU crystal with a clock derived via a PLL from your stable 10 MHz clock.  Things get slightly messy because the CPU and PCI clocks usually have some spread spectrum to reduce EMI.  I don't have numbers.  I expect it will be small/tiny, but I'll bet you can measure it.

That just gets you a basic clock.  If you also feed a PPS signal into the FPGA, then you can get very accurate timings by grabbing a copy of the time register(s) on the leading edge of the PPS signal.  That has metastability opportunities, but if you can solve the basic metastability issues, the same trick will work here.

As phk points out, the idea of putting the whole Unix clock in the FPGA may be silly.  It seemed like a fun problem so I've been thinking about how to do it.

Another variation that might be interesting...  The implementations I described previously all assumed the clocks had round numbers so you could do things like add 30 or 40 ns to the counter each PCI clock tick.  Those round numbers are convenient.  You could also add fractions of a ns.  Since this stuff can be pipelined as much as necessary to meet timings, you can make the fractional part as wide as you want to get better accuracy.  This might be handy if your external clock was slightly off, say a Rubidium that wasn't calibrated.

--
These are my opinions, not necessarily my employer's.  I hate spam.

stanley_reynolds@yahoo.com said: > I need to go back and read what you are trying to measure with your > clock. Is it internal to the computer or an external event ? I was thinking of a FPGA on a PCI bus. It has to be PCI rather than USB in order to get reasonable timings. I was going to put the Unix clock in the FPGA. It's a pair of 32 bit words. The high word is seconds since some magic date/time. The low word is nano-seconds within this second. I was planning to drive that with a stable external clock, say a GPSDO. So if things were setup properly, it should be a very good clock. The typical Unix clock is implemented by looking at the TSC. I think that's Intel's term for a register that counts each CPU cycle, Time Stamp Counter. The CPU comes from an inexpensive crystal. The frequency of that crystal varies (wildly) with temperature. The temperature varies with activity. The basic idea is that you maintain a reference time and TSC value. To get the current time, you read the TSC, subtract off the reference TSC. That gives you the number of CPU clock ticks since the reference time. Convert that to nanoseconds, add on the reference time, and normalize. You have to update the reference info periodically, in particular before that subtract will over/underflow. phk points out that you can use the TSC type clock logic by reading a simple counter from the FPGA rather than the TSC register. That gets you a stable clock without all the problems with reading 64 bits from the FPGA. You could also replace the CPU crystal with a clock derived via a PLL from your stable 10 MHz clock. Things get slightly messy because the CPU and PCI clocks usually have some spread spectrum to reduce EMI. I don't have numbers. I expect it will be small/tiny, but I'll bet you can measure it. That just gets you a basic clock. If you also feed a PPS signal into the FPGA, then you can get very accurate timings by grabbing a copy of the time register(s) on the leading edge of the PPS signal. That has metastability opportunities, but if you can solve the basic metastability issues, the same trick will work here. As phk points out, the idea of putting the whole Unix clock in the FPGA may be silly. It seemed like a fun problem so I've been thinking about how to do it. Another variation that might be interesting... The implementations I described previously all assumed the clocks had round numbers so you could do things like add 30 or 40 ns to the counter each PCI clock tick. Those round numbers are convenient. You could also add fractions of a ns. Since this stuff can be pipelined as much as necessary to meet timings, you can make the fractional part as wide as you want to get better accuracy. This might be handy if your external clock was slightly off, say a Rubidium that wasn't calibrated. -- These are my opinions, not necessarily my employer's. I hate spam.
PK
Poul-Henning Kamp
Mon, May 18, 2009 9:20 AM

I was going to put the Unix clock in the FPGA.  It's a pair of 32 bit words.
The high word is seconds since some magic date/time.  The low word is
nano-seconds within this second.

Please, will you guys stop reinventing the octagonal wheel, and at least
look at the blueprint for the round wheel ?

http://phk.freebsd.dk/pubs/timecounter.pdf

If there is anything you don't understand or wants explained in that paper,
by all means ask!

--
Poul-Henning Kamp      | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG        | TCP/IP since RFC 956
FreeBSD committer      | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

In message <20090518081256.97F2BBCE6@ip-64-139-1-69.sjc.megapath.net>, Hal Murr ay writes: >I was going to put the Unix clock in the FPGA. It's a pair of 32 bit words. >The high word is seconds since some magic date/time. The low word is >nano-seconds within this second. Please, will you guys stop reinventing the octagonal wheel, and at least look at the blueprint for the round wheel ? http://phk.freebsd.dk/pubs/timecounter.pdf If there is anything you don't understand or wants explained in that paper, by all means ask! -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.
LJ
Lux, James P
Mon, May 18, 2009 1:18 PM

On 5/18/09 1:12 AM, "Hal Murray" hmurray@megapathdsl.net wrote:

I need to go back and read what you are trying to measure with your
clock. Is it internal to the computer or an external event ?

I was thinking of a FPGA on a PCI bus.  It has to be PCI rather than USB in
order to get reasonable timings.

I was going to put the Unix clock in the FPGA.  It's a pair of 32 bit words.
The high word is seconds since some magic date/time.  The low word is
nano-seconds within this second.

I would make that "magic time" the time when power was applied.  Do the
transformation from "hardware time" to "Unix time" in software (since it
requires just arithmetic.. An add (offset between the two time zeros) and a
multiply (for the scale factor between "unix time" and your hardware
counter)

If you have a solution where you "jam" the hardware counter to "set it", you
always have a question about the latency of that set operation.  It's pretty
easy in hardware (FPGA) to arrange an "atomic" read of an arbitrarily long
counter into a holding register, which software then reads.

I was planning to drive that with a stable external clock, say a GPSDO.  So if
things were setup properly, it should be a very good clock.

The typical Unix clock is implemented by looking at the TSC.  I think that's
Intel's term for a register that counts each CPU cycle, Time Stamp Counter.
The CPU comes from an inexpensive crystal.  The frequency of that crystal
varies (wildly) with temperature.  The temperature varies with activity.

I don't know about wildly.  Certainly in time-nuts land, a few tens of ppm
is wild variation, but in the overall scheme of things, it's still
reasonably good.

The basic idea is that you maintain a reference time and TSC value.  To get
the current time, you read the TSC, subtract off the reference TSC.  That
gives you the number of CPU clock ticks since the reference time.  Convert
that to nanoseconds, add on the reference time, and normalize.  You have to
update the reference info periodically, in particular before that subtract
will over/underflow.

On 5/18/09 1:12 AM, "Hal Murray" <hmurray@megapathdsl.net> wrote: > > > stanley_reynolds@yahoo.com said: >> I need to go back and read what you are trying to measure with your >> clock. Is it internal to the computer or an external event ? > > I was thinking of a FPGA on a PCI bus. It has to be PCI rather than USB in > order to get reasonable timings. > > I was going to put the Unix clock in the FPGA. It's a pair of 32 bit words. > The high word is seconds since some magic date/time. The low word is > nano-seconds within this second. > I would make that "magic time" the time when power was applied. Do the transformation from "hardware time" to "Unix time" in software (since it requires just arithmetic.. An add (offset between the two time zeros) and a multiply (for the scale factor between "unix time" and your hardware counter) If you have a solution where you "jam" the hardware counter to "set it", you always have a question about the latency of that set operation. It's pretty easy in hardware (FPGA) to arrange an "atomic" read of an arbitrarily long counter into a holding register, which software then reads. > > > I was planning to drive that with a stable external clock, say a GPSDO. So if > things were setup properly, it should be a very good clock. > > The typical Unix clock is implemented by looking at the TSC. I think that's > Intel's term for a register that counts each CPU cycle, Time Stamp Counter. > The CPU comes from an inexpensive crystal. The frequency of that crystal > varies (wildly) with temperature. The temperature varies with activity. I don't know about wildly. Certainly in time-nuts land, a few tens of ppm is wild variation, but in the overall scheme of things, it's still reasonably good. > > The basic idea is that you maintain a reference time and TSC value. To get > the current time, you read the TSC, subtract off the reference TSC. That > gives you the number of CPU clock ticks since the reference time. Convert > that to nanoseconds, add on the reference time, and normalize. You have to > update the reference info periodically, in particular before that subtract > will over/underflow. > >
LJ
Lux, James P
Mon, May 18, 2009 1:28 PM

On 5/18/09 6:18 AM, "Lux, James P" james.p.lux@jpl.nasa.gov wrote:

On 5/18/09 1:12 AM, "Hal Murray" hmurray@megapathdsl.net wrote:

I need to go back and read what you are trying to measure with your
clock. Is it internal to the computer or an external event ?

I was thinking of a FPGA on a PCI bus.  It has to be PCI rather than USB in
order to get reasonable timings.

I was going to put the Unix clock in the FPGA.  It's a pair of 32 bit words.
The high word is seconds since some magic date/time.  The low word is
nano-seconds within this second.

I would make that "magic time" the time when power was applied.  Do the
transformation from "hardware time" to "Unix time" in software (since it
requires just arithmetic.. An add (offset between the two time zeros) and a
multiply (for the scale factor between "unix time" and your hardware
counter)

I also wouldn't have the low order counter count nanoseconds, or even set it
up as seconds/subseconds.  Use one long counter that counts whatever
frequency you want to count.  An integer divide in software is quite fast
(unless you're working with something like a Z80).

There's no real advantage in having "hardware" count seconds.. It takes more
gates to count by arbitrary N than 2^M.  Also, say you set your counters up
to divide by 10E6..assuming you have a 10MHz source.  And then you find that
your source is really 10,000,001 Hz (I know... A 0.1 ppm error is anathema
in this crowd).. Now, if you want to convert your (estimated seconds,
estimated nanoseconds) into "real" time, you've got tricky arithmetic to do
(with all the roundoff and arithmetic issues to deal with).

Ticks = secondscounter*nominaltickrate + subsecondscounter;
Realseconds = floor(ticks/actualtickrate)
Realsubseconds = remainder/actualtickrate;

On 5/18/09 6:18 AM, "Lux, James P" <james.p.lux@jpl.nasa.gov> wrote: > > > > > On 5/18/09 1:12 AM, "Hal Murray" <hmurray@megapathdsl.net> wrote: > >> >> >> stanley_reynolds@yahoo.com said: >>> I need to go back and read what you are trying to measure with your >>> clock. Is it internal to the computer or an external event ? >> >> I was thinking of a FPGA on a PCI bus. It has to be PCI rather than USB in >> order to get reasonable timings. >> >> I was going to put the Unix clock in the FPGA. It's a pair of 32 bit words. >> The high word is seconds since some magic date/time. The low word is >> nano-seconds within this second. >> > > I would make that "magic time" the time when power was applied. Do the > transformation from "hardware time" to "Unix time" in software (since it > requires just arithmetic.. An add (offset between the two time zeros) and a > multiply (for the scale factor between "unix time" and your hardware > counter) I also wouldn't have the low order counter count nanoseconds, or even set it up as seconds/subseconds. Use one long counter that counts whatever frequency you want to count. An integer divide in software is quite fast (unless you're working with something like a Z80). There's no real advantage in having "hardware" count seconds.. It takes more gates to count by arbitrary N than 2^M. Also, say you set your counters up to divide by 10E6..assuming you have a 10MHz source. And then you find that your source is really 10,000,001 Hz (I know... A 0.1 ppm error is anathema in this crowd).. Now, if you want to convert your (estimated seconds, estimated nanoseconds) into "real" time, you've got tricky arithmetic to do (with all the roundoff and arithmetic issues to deal with). Ticks = secondscounter*nominaltickrate + subsecondscounter; Realseconds = floor(ticks/actualtickrate) Realsubseconds = remainder/actualtickrate;
PK
Poul-Henning Kamp
Mon, May 18, 2009 1:32 PM

In message C636B079.7D49%James.P.Lux@jpl.nasa.gov, "Lux, James P" writes:

An integer divide in software is quite fast
(unless you're working with something like a Z80).

You only need to divide when you want to change your estimate of the
counters range, for generating timestamps a multiplication will do.

There's no real advantage in having "hardware" count seconds.. It takes more
gates to count by arbitrary N than 2^M.

Not only that, it makes the calculation of timeintervals as differences
between two timestamps a royal mess:

#define timersub(tvp, uvp, vvp)                                       
do {                                                           
(vvp)->tv_sec = (tvp)->tv_sec - (uvp)->tv_sec;         
(vvp)->tv_usec = (tvp)->tv_usec - (uvp)->tv_usec;     
if ((vvp)->tv_usec < 0) {                             
(vvp)->tv_sec--;                               
(vvp)->tv_usec += 1000000;                     
}                                                     
} while (0)

--
Poul-Henning Kamp      | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG        | TCP/IP since RFC 956
FreeBSD committer      | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

In message <C636B079.7D49%James.P.Lux@jpl.nasa.gov>, "Lux, James P" writes: >An integer divide in software is quite fast >(unless you're working with something like a Z80). You only need to divide when you want to change your estimate of the counters range, for generating timestamps a multiplication will do. >There's no real advantage in having "hardware" count seconds.. It takes more >gates to count by arbitrary N than 2^M. Not only that, it makes the calculation of timeintervals as differences between two timestamps a royal mess: #define timersub(tvp, uvp, vvp) \ do { \ (vvp)->tv_sec = (tvp)->tv_sec - (uvp)->tv_sec; \ (vvp)->tv_usec = (tvp)->tv_usec - (uvp)->tv_usec; \ if ((vvp)->tv_usec < 0) { \ (vvp)->tv_sec--; \ (vvp)->tv_usec += 1000000; \ } \ } while (0) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.
MW
M. Warner Losh
Mon, May 18, 2009 3:24 PM

In message: C636B079.7D49%James.P.Lux@jpl.nasa.gov
"Lux, James P" james.p.lux@jpl.nasa.gov writes:
: I also wouldn't have the low order counter count nanoseconds, or even set it
: up as seconds/subseconds.

I'd echo this, since you are artificially limiting the clocks that are
input to having a period of an exact number of nanoseconds.  This
rounding could lead to systematic errors that would lead to a higher
noise in the measurements.

A simple counter is more flexible.  It allows for a number of
additional algorithms to be applied to the raw time measurements to
account for drift in the underlying oscillator.  Phk's timecounters
allow for this.  They assume a time source that is free-running.  It
can be measured against a known better source to improve its accuracy
(which is what ntpd does).  This allows one to correct over time for,
say, the relatively crappy internal oscillator found in most PCs.  The
nice thing about timecounters is they allow hardware as described in
this thread to replace the underlying system hardware.

Warner

In message: <C636B079.7D49%James.P.Lux@jpl.nasa.gov> "Lux, James P" <james.p.lux@jpl.nasa.gov> writes: : I also wouldn't have the low order counter count nanoseconds, or even set it : up as seconds/subseconds. I'd echo this, since you are artificially limiting the clocks that are input to having a period of an exact number of nanoseconds. This rounding could lead to systematic errors that would lead to a higher noise in the measurements. A simple counter is more flexible. It allows for a number of additional algorithms to be applied to the raw time measurements to account for drift in the underlying oscillator. Phk's timecounters allow for this. They assume a time source that is free-running. It can be measured against a known better source to improve its accuracy (which is what ntpd does). This allows one to correct over time for, say, the relatively crappy internal oscillator found in most PCs. The nice thing about timecounters is they allow hardware as described in this thread to replace the underlying system hardware. Warner