Here's some sample raw data for those interested in what I'm working with. Thanks a bunch for all the suggestions. The overall goal is to take this data, remove the DC bias, linear trend, and exponential transient baseline; then cleverly excise (or exclude from processing) areas with obvious artifacts (see Figure 15, IQ pair 4) for an example of an artifact. These data plots have had the mean removed, but no detrending done. The raw data does NOT have the mean removed. As with many real software development things, the basic algorithms are simple, it's handling real data with real glitches and anomalies that is important. So when I collect the data, before I load it into a more lengthy processing step, I want to do a quick evaluation and see if the overall data take was bad, or maybe I can process just part of the data (I'd rather have gaps than bad values). http://www.luxfamily.com/jimlux/Figure%209.pdf http://www.luxfamily.com/jimlux/Figure%2011.pdf http://www.luxfamily.com/jimlux/Figure%2013.pdf http://www.luxfamily.com/jimlux/Figure%2015.pdf Raw data for 4 channels http://www.luxfamily.com/jimlux/data.txt

Hi Jim, On 10/05/2013 07:44 PM, Jim Lux wrote: > Here's some sample raw data for those interested in what I'm working > with. Thanks a bunch for all the suggestions. > > The overall goal is to take this data, remove the DC bias, linear > trend, and exponential transient baseline; then cleverly excise (or > exclude from processing) areas with obvious artifacts (see Figure 15, > IQ pair 4) for an example of an artifact. > > These data plots have had the mean removed, but no detrending done. > The raw data does NOT have the mean removed. > > As with many real software development things, the basic algorithms > are simple, it's handling real data with real glitches and anomalies > that is important. So when I collect the data, before I load it into > a more lengthy processing step, I want to do a quick evaluation and > see if the overall data take was bad, or maybe I can process just part > of the data (I'd rather have gaps than bad values). > > > http://www.luxfamily.com/jimlux/Figure%209.pdf > http://www.luxfamily.com/jimlux/Figure%2011.pdf > http://www.luxfamily.com/jimlux/Figure%2013.pdf > http://www.luxfamily.com/jimlux/Figure%2015.pdf > > Raw data for 4 channels > http://www.luxfamily.com/jimlux/data.txt I loaded I1 and Q1 into TimeLab, and removing the quadratic as well as dropping the first samples, when comparing to the un-processed data there is only minor changes. Turns out that there is noise and a slope down as tau rages, and there isn't much drift effect on the ADEV. MADEV is much cleaner naturally. There is a sine additive noise, which best is removed by doing a notch filter. So, I would do this: 1) Drop first 10 samples 2) Notch filter for removing sine noise 3) Estimate quadratic with least square, and remove 4) MADEV Notching out sine noise is not something I've seen in the literature, but I have tried it and it works really well as long as one care about unity gain. It will also make the least square approximation not being fooled by the noise. It will also not obscure the noise processes that ADEV/MADEV do. The TDEV plot is probably giving you most info, and it's only about tau 300 s (assuming sample every s, so scale accordingly) where the drift processing kicks in and helps to sort things out. Cheers, Magnus