Project on precise timing over Ethernet
Dear nuts,
We have this ongoing project whose aim is to synchronize roughly one
thousand stations (typical distances around 10 km) to within 1 ns using
Ethernet:
http://www.ohwr.org/twiki/bin/view/OHR/WhiteRabbit/WhiteRabbit
The idea is basically to use Synchronous Ethernet and something PTP-like in
combination. Sync Ethernet means from a timing point of view there is
hierarchy: there is a master source and its frequency is used to encode data
in the uppermost Ethernet switch in the hierarchy. This frequency then
propagates everywhere because switches extract it off the incoming data on
their "uplink" port and use it to encode data on every "downlink" port.
Fiber delays are evaluated using a two-way scheme, and used in phase
shifters to produce accurate PPS pulses everywhere. Our first Ethernet
switch prototype proved the concept works, although we are far from a final
system covering all our specs, and now we are thinking about making this a
more serious project by applying for EU funding. In the proposal document
there is a very important chapter on "Impact" and I would be very interested
in reading about potential applications you might have for such a network.
Here is our list so far:
-
Synchronization of different parts of large experimental physics
facilities (the original purpose). The advantage wrt current approaches is
you could do both data and timing on the same fiber. I am not saying nobody
has done timing over Ethernet so far, we have done our homework and none of
the existing projects fulfilled our constraints concerning clock quality,
accuracy, number of nodes, distances, Ethernet compliance and
"transparency", latency, determinism, design openness, etc. -
Distributed data acquisition: you can have ADCs feeding data to rolling
buffers, and all connected to this White Rabbit (WR) network. When one of
them receives a trigger we want to produce the effect in the control room
that all of them were triggered at the same time. So the trigger pulse gets
a precise UTC time tag and then all the other ADCs are informed of it
(through the same network) and also requested to freeze their buffers and
return the appropriate subset of data (again through WR) for coherent
display on operational consoles. The 1 ns spec comes mainly from this type
of application, applied to analog signal acquisition in particle
accelerators. Other examples are distributed sound measurements, structural
monitoring (vibration), power monitoring (detection of partial discharge and
dielectric losses in high voltage power cables: by taking synchronized
measurements over a segment of cable, defects can be pinpointed and repaired
before complete failure of the transmission line occurs), distributed RF
Time Division Multiple Access (TDMA) and meteorological event measurements
(e.g. lightning). -
Backbone for wireless positioning networks. There are projects to build
wireless networks in factories where you can position a mobile wireless node
by measuring two-way delays wrt fixed wireless points, which need to be very
well synchronized among themselves.
I am very interested in hearing about any potential application you could
imagine for White Rabbit, and also of course about any remarks on the
project in general.
Cheers,
Javier
On 9/10/09 4:29 AM, "Javier Serrano" javier.serrano.pareja@gmail.com
wrote:
Dear nuts,
We have this ongoing project whose aim is to synchronize roughly one
thousand stations (typical distances around 10 km) to within 1 ns using
Ethernet:
http://www.ohwr.org/twiki/bin/view/OHR/WhiteRabbit/WhiteRabbit
Interesting stuff...
Have you looked at what the giant phased array radio telescopes are doing?
LOFAR and SKA both cover large areas, and have similar timing requirements.
Interesting. I've googled a bit and could not get precise details. Data in
LOFAR seems to arrive at correlators using UDP packets, but I don't know if
they get time-tagged in each station or in the correlator. No mention of
fiber-delay compensation anywhere. I'll try to contact somebody from LOFAR
and see what they are doing exactly. Thanks for the lead!
Javier
On Thu, Sep 10, 2009 at 4:34 PM, Lux, Jim (337C)
james.p.lux@jpl.nasa.govwrote:
Interesting stuff...
Have you looked at what the giant phased array radio telescopes are doing?
LOFAR and SKA both cover large areas, and have similar timing requirements.
Javier Serrano wrote:
Dear nuts,
We have this ongoing project whose aim is to synchronize roughly one
thousand stations (typical distances around 10 km) to within 1 ns using
Ethernet:
http://www.ohwr.org/twiki/bin/view/OHR/WhiteRabbit/WhiteRabbit
The idea is basically to use Synchronous Ethernet and something PTP-like in
combination. Sync Ethernet means from a timing point of view there is
hierarchy: there is a master source and its frequency is used to encode data
in the uppermost Ethernet switch in the hierarchy. This frequency then
propagates everywhere because switches extract it off the incoming data on
their "uplink" port and use it to encode data on every "downlink" port.
There is several types of "Synchronous Ethernet", but here is one where
the bit-clock is being steered just as in a SDH network, and infact this
variant is also supported in modern G.781 synchronisation.
The other form possible is by using counters and interchanging values.
Ah well.
Synchronous Ethernet that steers bitclock will be tainted by
inter-symbol interference, but it is not too bad for most purposes.
I have a thick WhiteRabbit document to read in detail, but other things
have come inbetween.
Cheers,
Magnus
Yes, exchanging data over a link will always harm its timing
accuracy/precision performance, irrespective of whether you steer clocks
(our case) or you exchange values or both. In fact, as I said to Jeremy in
another thread I tend to see clock steering (i.e. PLL) as a limit case of
interchanging values, where this exchange would happen at every single clock
tick. So you end up with lots more information, up to you to low-pass filter
to get rid of high frequency phase noise.
Cheers,
Javier
On Fri, Sep 11, 2009 at 12:30 AM, Magnus Danielson <
magnus@rubidium.dyndns.org> wrote:
Javier Serrano wrote:
Dear nuts,
We have this ongoing project whose aim is to synchronize roughly one
thousand stations (typical distances around 10 km) to within 1 ns using
Ethernet:
http://www.ohwr.org/twiki/bin/view/OHR/WhiteRabbit/WhiteRabbit
The idea is basically to use Synchronous Ethernet and something PTP-like
in
combination. Sync Ethernet means from a timing point of view there is
hierarchy: there is a master source and its frequency is used to encode
data
in the uppermost Ethernet switch in the hierarchy. This frequency then
propagates everywhere because switches extract it off the incoming data on
their "uplink" port and use it to encode data on every "downlink" port.
There is several types of "Synchronous Ethernet", but here is one where the
bit-clock is being steered just as in a SDH network, and infact this variant
is also supported in modern G.781 synchronisation.
The other form possible is by using counters and interchanging values. Ah
well.
Synchronous Ethernet that steers bitclock will be tainted by inter-symbol
interference, but it is not too bad for most purposes.
I have a thick WhiteRabbit document to read in detail, but other things
have come inbetween.
Cheers,
Magnus
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Javier Serrano wrote:
Yes, exchanging data over a link will always harm its timing
accuracy/precision performance, irrespective of whether you steer clocks
(our case) or you exchange values or both. In fact, as I said to Jeremy in
another thread I tend to see clock steering (i.e. PLL) as a limit case of
interchanging values, where this exchange would happen at every single clock
tick. So you end up with lots more information, up to you to low-pass filter
to get rid of high frequency phase noise.
Actually, you can see this as a Shannon information channel, analog or
digital. The produced information is the phase difference between the
master and slave oscillator, but it may be encoded digitally over other
clocks. However, since the information capacity is never properly
estimated the long term phase information quality can vary alot. The
systematic noise sources added is also almost never properly analyzed.
Ah well.
Cheers,
Magnus
Jim, I found a good reference for LOFAR clock and sync:
http://www.astron.nl/sites/astron.nl/files/cms/PDF/LOFAR_Rep_057_clock_sync.pdf
After a quick read, I think they would have been a perfect candidate to use
WR, pity it's a bit late. Their system uses quite a lot of cables to achieve
(worse than) what WR does with a single fiber, and there is no mention of
cabling delay compensation anywhere, which makes me think they must be doing
it by hand through some kind of calibration campaign at the beginning of the
run. We have done this in the past and it's both time-consuming and
error-prone, not to speak about thermal effects you just can't deal with in
this way. Anyway an interesting read. Thanks again for the idea.
Javier
On Thu, Sep 10, 2009 at 4:34 PM, Lux, Jim (337C)
james.p.lux@jpl.nasa.govwrote:
On 9/10/09 4:29 AM, "Javier Serrano" javier.serrano.pareja@gmail.com
wrote:
Dear nuts,
We have this ongoing project whose aim is to synchronize roughly one
thousand stations (typical distances around 10 km) to within 1 ns using
Ethernet:
http://www.ohwr.org/twiki/bin/view/OHR/WhiteRabbit/WhiteRabbit
Interesting stuff...
Have you looked at what the giant phased array radio telescopes are doing?
LOFAR and SKA both cover large areas, and have similar timing requirements.
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.