Real ntpd uses a drift file to track the local processor's frequency offset
and has a good estimate of processor clock drift after a day of tracking.

The Raspberry Pi processor clock, like any motherboards', will often be off
by off anywhere from +/-200ppm but the good news is that it usually varies
by less than +/-10ppm over a day and ntpd does a good job tracking this and
using the drift correction during no-signal periods.

+/-10ppm over 24 hours is about +/-1 second.

I would strongly discourage using a Raspberry Pi in any not-just-for-fun
application. I have had very poor luck keeping even brand-name hi-grade
microSD cards working for a year in 24x7 Raspberry Pi hardware. No-name or
store brand microSD cards seem to barely work at all when new.

Unless your server farms are completely isolated from the internet at
large, you should consider using a curated list of public NTP sources.

Tim N3QE

On Mon, Jul 22, 2019 at 10:05 AM wildylion via time-nuts <
time-nuts@lists.febo.com> wrote:

Yeah, of course I will NOT do anything home-grown for the datacenter.

But currently it uses 3 Stratum 2 NTP servers, one per DC, with them referencing a list of 4 close-by Stratum 1 sources.

ntpstat generally says that time is correct within ~50 ms, while jitter and offset generally don't exceed 1ms, the root dispersion is quite large.
Also these Stratum 2 NTPs are run from Cisco routers, which I doubt are very good at timekeeping.

When I implemented this scheme, I offered to have S2 on a set of x86 servers, but was overruled by management who said it'll be better if we run S2's off Cisco gear.

So what if we add a couple GPSDO's into the mix, using them as primary time sources alongside public Stratum1 NTP servers for sanity check? And of course moving the S2's to something more stable.

My own, homegrown S1 pool NTP is another matter entirely - I think just a tuned Raspi with a M8N will be enough?

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, 22 July 2019 г., 17:38, Taka Kamiya tkamiya9@yahoo.com wrote:

On Mon, Jul 22, 2019 at 5:03 PM wildylion via time-nuts <
time-nuts@lists.febo.com> wrote:

While one can run most Cisco routers as NTP servers, I would advise against
doing this as they don't have very good clocks, and it adds to the
management plane load.

Moving away from the Cisco routers would be my first step.

A Pi with a ublox is great fun, unfortunately the clock on the Pi is
"poor", and this is a limiting factor.
https://www.ntpsec.org/white-papers/stratum-1-microserver-howto/

Cheers

Arne

Tim Shoppa writes:

That assumes stable median temperature and predictable temperature
swings.

All rasPi I've bought so far (5 of them, all different models from three
different sources) were ~10ppm slow at RT and within -6ppm and 0ppm at
the apex temperature point of the crystal (around 60°C).  It is possible
to keep the temperature within about 0.2K of that point by using the CPU
itself as a heater and thermometer and the resulting frequency within
+-20ppb over a day when disciplined via pps-gpio.  The residual drift is
initially positive and gets smaller over time, mine are down to about
20~30ppb per week.

I've never tried to hold over since it's really hard to figure out when
to start holding and when to switch back.  A 24 hour holdover is pretty
severe, but I guess you could keep it within double-digit ms territory
without getting too fancy if you've got that right.

Instead, by having enough independent NTP servers (all with their own
antenna) the clients figure out by themselves when one is off by more
than about a millisecond and switch to another one.  Each server
monitors the other ones and if it finds itself off from that bunch too
much will drop down from stratum-1 to stratum-2.  When exactly that
happens depends a bit on how fast the drift rate is, but usually between
3ms and 5ms.

Indeed.  I'd opt for something like PCEngines' APU line since they have
accessible GPIO and PTP capable ethernet.

While I support that sentiment, after initial problems with off-brand
cards the ones I have now are all getting into their third year with no
signs of problems (knocks on wood).  In a datacenter you'd almost
certainly net boot the rasPi instead of giving them a microSD card.

The key to NTP reliability is copious redundancy, so you'll certainly
want local servers in addition to that, even if you have redundant
outside connectivity.

Regards,
Achim.

+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

wildylion via time-nuts writes:

You can't say, really.  It all depends on how the GPS gets lost, or what
the environment does around the time of loss and after.  If you have
multiple such NTP servers you'll quickly learn by comparing them to each
other that they sometimes do things you didn't think they should be
doing.  If you have just one, you'll probably never even know that.

Colocated spaces == racks where paying customers put their stuff?

You should really think about doing PTP instead on these systems,
preferrably over a dedicated network.

Nothing against the LeoNTP (which I'm unfamiliar with), but if time
actually is important (i.e. you need to guarantee some level of
performance rather than hope for it) you'll end up doing the majority
of the work that you'd otherwise buy with the certification.  That
almost certainly ends up being more costly than getting the certified
gear.

That really depends on what you are trying to achieve, but I'd say you
have yet to define what the problem is that you're trying to solve.  Do
you just want your 100 servers to have the same time within some defined
boundary, does that time need to track the official/legal time (at what
maximum offset) or do you need to have traceable time stamps all the way
back to the source (at what resolution and error)?  For each set of
answers entirely different architectures would be appropriate.

Regards,
Achim.

+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Dear colleagues, i've made some tests and wrote an article about Raspberry
Pi as a timeserver. I think we can trust on it and this board can keep a
good time from GNSS PPS. I'm open to suggestions. I hope this can help in
the decision making process. The link: rpi_article
https://www.researchgate.net/publication/334179026_Credit_card_size_computer_Stratum_1_NTP_Server

Also, in our tests as you can see in the article, for general purposes the
RPI has a good dealing with high loads. The difference is we've removed the
crystal oscillator and put a signal from a synthesizer, you can keep the
crystal and trust on pps signal too. I'm using a server at home with a
cheap module NEO-6M and also one with SAM-M8Q. For comparsions effect i've
put the NEO-6M to compare the ntp time of the rpi that this receiver is
connected to with the server from the article: you can see in the
thingspeak: thingspeak_channel
https://thingspeak.com/channels/691405/

If you want the source code for the 1 PPS generation thats it:
1pps_generation_src_code
https://github.com/LuizPauloDamaceno/rpi_pps_out/blob/master/ppsout.c

Best regards,

Luiz

Em seg, 22 de jul de 2019 às 11:05, wildylion via time-nuts <
time-nuts@lists.febo.com> escreveu:

On 7/22/19 09:46, Achim Gratz wrote:

I've also found that my Raspi's give the best results with board temp
right around 60°C. However, I found that the 'CPU as heater', while it
does help dramatically compared to nothing at all, it introduces a lot
of jitter on its own. Much better results by keeping them in an
insulated enclosure in a relatively stable location (for me, a closet!),
and 'tuning' the insulation as needed to balance venting against retention.

Something that just occurred to me, and I'm curious if I'm off in the
weeds here - while a GPS failure could take many forms, I wonder if
there's a good argument to be made to having at least one or two peers
who are geographically quite far from one's location - e.g. the other
side of the world - to at least ensure that some number of satellites
might be available when others might not. Thinking in terms of a
Carrington event scenario. Wouldn't the GNSS on the far side of the
earth possibly fare better in such an event - all other things being
equal? I guess it would depend on how long it took for the mass ejection
to pass the earth.

Yeah, I think I'm off in the weeds.

I can echo that sentiment re SD cards. I'm running Samsung EVO Plus and
Sandisk Ultra cards in my devices, never a problem. Considering that the
'sweet spot' for pricing on SD cards is now the 32gb ones, it provides a
great buffer for the controller to handle internal block allocation for
longer life.

--
Paul Theodoropoulos
www.anastrophe.com

On 22/07/2019 18.46, Achim Gratz wrote:

Regarding the massive offsets that sometimes come up, it is worth asking
when that 'truth' was learned, and with how old a kernel, if Linux was
involved.

Pre-2011, and if Debian was involved probably pre-2018, there was a
rounding error in the kernel, causing the clock code to miscount by a
lot (127 ppm) on most motherboards, and worse on boards with better
timer resolution.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.3-rc1&id=a386b5af8edda1c742ce9f77891e112eefffc005

After this, the offsets on all the machines I had became a lot more
realistic*. Putting together a crystal oscillator and pulling it so that
it is 200ppm off, and reliable for manufacturing, is actually hard.

As to reliable NTP: Over the past ~17 years I have learned that every
receiver will fail at some point, all receivers of a brand will fail at
the same time, and you need to plan for that.

/Kasper Pedersen

*There remains an error of typically 2 ppb after the patch, but I was
short on time.

Paul Theodoropoulos via time-nuts writes:

Well, I have jitter (PPS timestamps) within ~2µs and the occasional
"hair" that goes out to about 30µs.  The distribution is a little bit
wider on the three systems that are loaded with extra server tasks, but
not much.  The thing to note is that you need to boot with "nohz=off",
set a fixed frequency as high as possible and use an interval timer for
controlling the CPU load (I use all four cores as workers and adjust
them at 40Hz).

[I can post plots, but at ~500kiB I think I should rather ask for
permission first.]

Regards,
Achim.

+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

On 7/22/19 11:52, Achim Gratz wrote:

Yes, nohz off is necessary for sure. I use a  bunch of assorted tweaks
and am able to keep RMS time jitter under 300ns at 95th percentile in a
given 24 hour interval. However, it takes very, very, very little user
(me) activity on the device  itself to cause larger spikes into the
microsecond realm. I upgraded my devices from Raspbian Stretch to Buster
over the weekend, causing a spike to 3 microseconds offset. However,
offsets have settled in even lower since the upgrade.

I use,among other things, partrt to set up 'realtime' cpu partitioning,
and schedtool to nail ntpd to core 3 and gpsd to core 2 - but I suspect
any improvement I see probably falls into the psychosomatic realm. ;)

Disabling as many cronjobs and running services as possible provides by
far the most benefit to stability. Oh, and staying off the command line!

--
Paul Theodoropoulos
www.anastrophe.com

On 7/22/19 07:50, wildylion via time-nuts wrote:

Will you have access to a good location for the antenna? colo spaces
typically have a maze of cable runs, air handlers, etc above the space,
not to mention metal the cabinets themselves - it could be very
difficult to get a good gps signal.

--
Paul Theodoropoulos
www.anastrophe.com

Achim Gratz writes:

Permission granted, so here it is, data is for the last week plus today.

Keeping the CPU temp constant actually doesn't do much good, since the
crystal is at a lower temperature.  Lacking that measurement I use the
CPU load and a simple thermal resistance model to estimate the
environmental and crystal temperature.  The CPU temperature measurement
is also coarsely quantized (all data collection is triggered from the
PPS pulse), but it seems to be the result of some kind of delta-sigma
conversion, so it actually just needs a bit of filtering to come out
surprisingly detailed.

Note that the time values are asinh(t) scaled, which is a fancy way of
saying linear through zero and logarithmic elsewhere.

Measured CPU load and temperature (raw and filtered), modeled crystal
temperature and reconstructed environmental temperature:

PPS statistics (raw and filtered, overlaid with temperature):

NTP loop statistics:

NTP frequency error vs. predicted frequency error:

The spike two days ago was an update and the requisite reboot, the spike
yesterday was me putting another rasPi on top of the box this rasPi is
in and then putting an airgap inbetween the two, later the window was
open for the night.  This box does double duty as an ADS-B receiver, so
would pull around 30%…50% load on its own.

Regards,
Achim.

+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Hi,

On 7/23/19 12:24 AM, Glen English VK1XX wrote:

This would be IFCS (US) and EFTF (Europe).

We try to gather time-nuts beyond what we do already during conference.

Enrico I met one and a half week ago as I was in Besancon doing research.

Cheers,
Magnus

You may want to look at something an Endrun CDMA ntpserver for the
datacenter; don't need view of the sky. Available on the used market sub $1k

Standard solution for this scenario.

K5ROE Mike

On 7/22/19 10:50 AM, wildylion via time-nuts wrote:

Hi guys,

A little late to the party but hopefully someone finds this useful.

As the Raspberry Pi family of IoT devices are so popular and have been used in a host of different research projects to gather data (much of it including timing information), we were curious as to how stable the Raspberry Pi was and whether it was suitable for such work. To understand this better we characterised the main counter on the board (the STC) for all three generations of Pi available at the time. We presented the results at IMC2016 in "Time to measure the Pi" which is available here:

http://crin.eng.uts.edu.au/~darryl/Publications/Pi-hat_current.pdf

We also looked at how well it performed with NTP and also characterised the uBlox M8Q as a potential time source.

The Pi's (especially the latest generation at the time, the Pi 3) did surprisingly well and behaved similarly to what we previously observed in a "big box" machine.

In terms of its suitability as an NTP server, again it held up really well. That said until the Pi 4 (which I am still waiting to get my hands on), the network interface on the Pi goes through an internal USB hub and suffers from significant jitter. I've not sat down and studied it in depth, but initial tests showed occasional spikes well into the millisecond range.

Lastly although the Pi's are impressive given their price point and they can perform well for many use cases, there is a reason why Microsemi sells their solutions for 100x the price. At home or in the office, a Pi3 with a uBlox M8Q (with a good view of the sky) is likely to give you better results than using a random server from the Internet's NTP Pool, but I wouldn't want to bet on it for keeping critical infrastructure in sync (it's not going to do well in holdover situations for example).

Kind Regards,

Peter Membrey

----- Original Message -----
From: "Luiz Paulo Damaceno" luizpauloeletrico42@gmail.com
To: "wildylion" wildylion@protonmail.com, "Discussion of precise time and frequency measurement" time-nuts@lists.febo.com
Sent: Tuesday, 23 July, 2019 01:31:34
Subject: Re: [time-nuts] Noob question, NTP stratum 1.

Dear colleagues, i've made some tests and wrote an article about Raspberry
Pi as a timeserver. I think we can trust on it and this board can keep a
good time from GNSS PPS. I'm open to suggestions. I hope this can help in
the decision making process. The link: rpi_article
https://www.researchgate.net/publication/334179026_Credit_card_size_computer_Stratum_1_NTP_Server

Also, in our tests as you can see in the article, for general purposes the
RPI has a good dealing with high loads. The difference is we've removed the
crystal oscillator and put a signal from a synthesizer, you can keep the
crystal and trust on pps signal too. I'm using a server at home with a
cheap module NEO-6M and also one with SAM-M8Q. For comparsions effect i've
put the NEO-6M to compare the ntp time of the rpi that this receiver is
connected to with the server from the article: you can see in the
thingspeak: thingspeak_channel
https://thingspeak.com/channels/691405/

If you want the source code for the 1 PPS generation thats it:
1pps_generation_src_code
https://github.com/LuizPauloDamaceno/rpi_pps_out/blob/master/ppsout.c

Best regards,

Luiz

Em seg, 22 de jul de 2019 às 11:05, wildylion via time-nuts <
time-nuts@lists.febo.com> escreveu:

time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

US operators are planning to shut down their CDMA networks.  See for example
https://www.fiercewireless.com/wireless/verizon-to-shut-down-2g-cdma-1x-network-by-end-2019

I wouldn't be surprised to see this date get pushed back though.

On Mon, Jul 22, 2019 at 8:05 PM K5ROE Mike K5ROE@roetto.org wrote: