Real Time Clock accuracy

I have a real time clock attached to my RPi and I thought that it would be an interesting project to monitor its accuracy. The clock I am using is a Maxim DS1307… this type does not have any fancy compensation, but from what I read it should be around 10ppm.

So I built a program for monitoring the clock and comparing it with a reference. For a reference I am using the RPi system clock slaved to NTP… which should provide synchronisation to within a few milliseconds. The linux clock of course reports to quite a fine resolution (1 microsecond ticks)… but the DS1307 only reads out to the 1 second level which means on the face of it that a fairly long run would be needed to make an estimate of the clock accuracy.

Here is a table of expected deviation in seconds for clocks of various accuracies… as you can see we would have to wait for 10 days to get a result from a 10ppm clock with a significant offset. The problem being after 1 or 2 days the error might be 1 second, and the resolution of the measurement is +/-1 second… not great for measuring drift.

clockerrortheory1

So I thought a bit more about the problem of measuring. We actually have more info than this as we can measure the time on the RPi to a resolution of 1 microsecond and we can measure frequently. So while we run we can compare the time from the real time clock to that more accurate clock. (it is accurate as it is set from ntp and it has finer resolution). Of course the rounding implied in the RTC result compared to the system clock will give  a a false sense of accuracy.  So what I decided to do was to round both and compare to only 1 second resolution. You might ask how does this help? Well what we can do now is see the statistics of the comparison at 1 second resolution and use them to see the drift rate.

First we set the RTC to be the same as the system clock set by ntp. Then we read both and compare the seconds. Initially we will get an answer of 0, with the occasional -1 or 1 due to the rounding. Gradually as the RTC drifts the number of reads that say 1 will increase (assuming it is running slower than the system time.This gives us an estimate of how far apart they are. In effect we are looking at the ‘density’ of rounding transitions up or down over time to estimate the drift.

This looks like the following;

The y axis is the delta and the x axis is the run time… here I picked 256 samples of a part of the run about 1000 readings into the process.

clockerrortheory2

So now time for a  test run. I started the program and let it log to a file. Unfortunately after about 2 days the RPi crashed … but after all that was the point of looking at this method… to avoid the need for very long runs. Here is the picture of the just over 11,000 points taken every 15 seconds.

clockerrortheory3

Now we can see a clear trend of drift between the clocks. There is a fair amount of noise due to the rounding jitter so we’ll need some filtering a to derive a trend. I applied an average of the offset (last 64, 128,  256 and 512 samples), calculated the sliding average  and plotted a trend in excel to see if this technique would work. Here is the result:

clockerrortheory4

It looks like we have about 8ppm drift from this result. of course we could try estimate to more accuracy, but that would require more detailed examination of the filter length and noisiness of the signal to come up with something reliable… I did a quick check of the results and 1ppm seemed to be about the right level of resolution as with the different filters the slope was varying by about 0.5ppm. While running this test  the temperature variation was about +/- 2C and the ntp was locked to a stratum 2 clock.

So in conclusion, in 2 days we managed to get an estimate of clock drift using the trend line of the offsets even though the absolute drift was only 2 seconds and the resolution of the RTC is 1 second. Next we need to test this for a longer period of time, and build in the filters and calculations to remove the manual processing steps in excel. Plus look more into the ntp stats for the local clock drift compensation as we are measuring versus the system clock slaved to ntp.

Here is the code for the program.


#!/usr/bin/env python
#
# ClockTest Program
# original and driver by John C. Shovic, SwitchDoc Labs
# 07/10/2014
# Modified by hobbyistdave, Oct 2015
#
# imports

import sys
#sys.path.append ('./lib')              # path for local libraries
import time
import datetime
import requests   # HTTP for humans
                  # by Kenneth Reitz  docs.python-requests.org
import SDL_DS1307 # the RTC driver

# setup the info needed to send the results to thingspeak

thingspeakurl = "https://api.thingspeak.com/update"   # url for sending updates to thingspeak
apikey = "1234567890123456"     # thingspeak api address for this channel 
ioTInterval = 15                # interval for maximum writes to thingspeak in seconds (min interval is 15 seconds)
url = thingspeakurl

def writeToIoT(payload):        # function to write to IoT
    try :
        r = requests.get(url, params=payload)  # send to thingspeak
        print time.time(), "  ", r.text, "  ", ntpSecs ,"   ", rtcSecs ,"   ", deltaSecs ,"      ", runTime , " ", r.status_code  
        #r.text will be the data point number
        #r.status code should be 200 if the page posted correctly
    except requests.exceptions as e :
            print e # some simple error trapping to avoid the program timing out
    except requests.exceptions.ConnectionError as e1 :
            print e1 # some simple error trapping if the connection is lost
    return

# Main Program

print ""
print "ClockTest, hobbyistdave"
print ""
print "Program Started at: "+ time.strftime("%Y-%m-%d %H:%M:%S")

filename = time.strftime("%Y-%m-%d%H:%M:%SRTCTest") + ".txt"
startTime = datetime.datetime.utcnow()
#print startTime
print "Writing system time to RTC to make them equal at start of test"
ds1307 = SDL_DS1307.SDL_DS1307(1, 0x68)
ds1307.write_now()


try :    # wrapper to allow clean exit
    # now setup the command shell for printing the results
    print " "
    print "---------------------------------------------------------------------------"
    print " "
    
# Main Loop - sleeps 1 seconds, then reads and prints values of all clocks
# checks time since last update and writes to IoT

    startTime = datetime.datetime.now()
    print "startTime : ", startTime
    print " "

    print "---------------------------------------------------------------------------"
    print "  Time          Point    NTP     RTC   Delta    Runtime         Response   "
    print "  (s)                   (s)     (s)   (s)                                  "
    print "---------------------------------------------------------------------------"

    ioTTimeStamp = time.time()
    lastDelta=0   # initialise delta
    while True:

        ntpTime = datetime.datetime.now()
        rtcTime = ds1307.read_datetime()
        #deltaTime = ntpTime - rtcTime
        ntpSecs = datetime.datetime.now().strftime('%S')
        rtcSecs = rtcTime.strftime('%S')
        #print ntpTime, "  ", rtcTime, "  ", deltaTime
        deltaSecs =  int(ntpSecs)  - int(rtcSecs)
        if lastDelta < 30 and deltaSecs <-30 : # fix instances where the seconds wrap across a minute boundary deltaSecs = deltaSecs+60 # we dont expect time to drift apart by more than 30 seconds elif lastDelta > -30 and deltaSecs >30 :
            deltaSecs = deltaSecs-60
        lastDelta = deltaSecs  # remeber the last delta

        runTime = ntpTime - startTime

        payload = {'key' : apikey, 'field1' : ntpTime , 'field2' : rtcTime , 'field3' : deltaSecs , 'field4' : runTime  } # make the payload
        if time.time() >= (ioTTimeStamp + ioTInterval):  #check if enough time has passed before updating thingspeak
            writeToIoT(payload)
            ioTTimeStamp = time.time()
        else:
            time.sleep(0.001)

        time.sleep(1)

except KeyboardInterrupt:
    print " "
    print "Exiting program due to Keyboard Interrupt"
    print " "
    time.sleep (1)
finally:
    print "----------- The End----------------"



Mirroring command line output to a file

Quite often when you are running a program, you would like to capture its output into a file for later analysis. I want to do this for a real time clock accuracy monitoring project that I am in process of running. Of course you can write to the file directly from your program, but there is a neat trick to both write to the standard command output, capture to a file and monitor the output at the same time.

First you run your program with the output redirected into a file;


sudo python -u myprog.py > proglog.txt

The > is the redirect and the python -u switch means do not buffer. Without this you will not see your log file change for a while as the OS will buffer a significant amount of the output.  When you write directly to the command line the buffer is flushed every line, but not when you redirect. In the Debian wheezy used on the RPi the buffer is 4kbytes so if you redirect to a file you wont see output for a while if you don’t use the -u switch. This is a common problem and makes for a confusing time as you see the log file size stays at 0 while the buffer fills.

Then in another terminal window use;


tail -f proglog.txt

You will now see the last 10 lines of the file that is being created with the output. As the file is updated the ‘tail’ will ‘follow’ it continuously so you get the benefit of writing to the console and of logging to a file. You can increase the number of lines tailed with the -n switch followed by the number of lines you require.

Another way you can do this is to ‘tee’ the output so that the file is generated and the command line shows in one terminal.

sudo python -u myprog.py  | tee proglog.txt

or

sudo python -u myprog.py 2>&1 | tee proglog.txt 

The second version with 2>&1 sends both the standard output and the standard error to the file.

These files are great for post processing the data, for example by importing them into excel. If you want to append lines to the last file rather than create the file new each time, use >> rather than >.


sudo python -u myprog.py >> proglog.txt

Serial mouse fixes for external GPS on Win 7

One of the back burner projects I’ve been pursuing has been to participate in the global network of ADSB receivers for aircraft tracking. I’ll post more separately on that but as part of it I am running a GPS connected to the laptop where my RTL-SDR receiver connects. The GPS is attached via a USB to serial converter cable based on the Prolific chip or a clone. This seemed to run well, until I rebooted the laptop. At which time the utility I was using stopped being able to talk to the GPS, the com port disappeared and the mouse started acting up. A reboot later, the GPS locked onto the satellites  immediately as shown by its flashing 1PPS light but the mouse was jumping all over my screen even before I could login.  After various bits of research I found that this is a well known issue inherent in windows and is a legacy behaviour.  Windows looks at the serial ports including any virtual ones and if it sees any data it assumes that a serial mouse is connected. It then enumerates that mouse and installs a driver. To be more precise it installs a Microsoft Serial Ballpoint.  Of course the GPS is sending lots of NMEA data on its serial port in a  constant stream so windows thinks it has a mouse attached.

If you research the problem you find that in Win NT, 2000 and XP there was a boot.ini change that you could make and a command line that could be run to disable this. In Win7 the only solution is to edit the registry.

Here’s a nice tutorial that I found. I have only tried the Win7 fix.

Here is the entry you need to change.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\sermouse
The ‘start’ parameter needs to be changed to 4 from 3 to disable the mouse on startup.
Use the command ‘regedit‘ from a command line and before making a change backup the registry by exporting it.

Be warned though.. when you reboot the first time you must not have the GPS attached… if you do it resets the change to the registry and the NMEA messages are interpreted as mouse movement again.

Once you have rebooted once sucessfully, the registry change sticks and you can leave the GPS plugged in on future reboots. It is still enumerated as a serial ballpoint and displays a mouse icon in my case, but that part of the driver is off and the COM port is now open. The perfectionist in me is frustrated that the GPS and serial port adapter are misnamed, but at least it works.

serialmouse1                                     serialmouse2
I also have an FTDI UART adapter on this laptop that does not cause this problem… although that might be because the Arduino attached does not send anything until the laptop has booted.
It remains to be seen whether this ‘fix’ survives a windows update cycle… I’ll let you know.

Raspberry Pi and the ZX Spectrum

I only discovered the Raspberry Pi in late 2014 and was struck by the concept and also the history. The background of the founders really resonated. What happened in the late 70’s and early 80’s in the UK was that a whole generation of new engineers and computer scientists was inspired by easily accessible computers. This charge was led by the BBC with their educational programming, leading to the development of the BBC micro computer and its subsequent deployment into after school computer clubs. In parallel with this companies like Sinclair with their ZX81 and Spectrum made computing easily accessible to schoolkids at home. No longer was a single computer used for the whole class (I remember the single Research Machines 380Z we had at school), but instead individuals could use their own machine both at school and at home. This wave of experimenters did not have to be as hard core as previous generations and were enticed into a rewarding and fulfilling profession by their hobby. These students became the front end of the UK’s tech explosion and that heritage led to platforms like the ARM core that resides in so many smartphones. Indeed, Acorn the manufacturer of the BBC micro is the progenitor of ARM Holdings. The Raspberry Pi was intended to address the issue of technical skill among people entering Universities. In the 80’s and 90’s many of those entrants had been pretty experienced hobbyists, whereas in the 2000’s it appeared that they were more like computer users. So the Pi was intended to restart that trend.  What is interesting about the Raspberry Pi  though is that of the millions shipped it appears that most are in the hands of adult hobbyists… the schools part of the program is limited by both student interest and teaching resources whereas hobbyists have adopted it in droves.

The BBC micro computer I used was at school and I don’t have a photo, but in honour of the Pi here is a photo of my original ZX Spectrum. The 48K RAM version no less, costing 175 pounds back in 1982.

ZX Zpectrum

Spectrum1