Real Time Clock accuracy

I have a real time clock attached to my RPi and I thought that it would be an interesting project to monitor its accuracy. The clock I am using is a Maxim DS1307… this type does not have any fancy compensation, but from what I read it should be around 10ppm.

So I built a program for monitoring the clock and comparing it with a reference. For a reference I am using the RPi system clock slaved to NTP… which should provide synchronisation to within a few milliseconds. The linux clock of course reports to quite a fine resolution (1 microsecond ticks)… but the DS1307 only reads out to the 1 second level which means on the face of it that a fairly long run would be needed to make an estimate of the clock accuracy.

Here is a table of expected deviation in seconds for clocks of various accuracies… as you can see we would have to wait for 10 days to get a result from a 10ppm clock with a significant offset. The problem being after 1 or 2 days the error might be 1 second, and the resolution of the measurement is +/-1 second… not great for measuring drift.

clockerrortheory1

So I thought a bit more about the problem of measuring. We actually have more info than this as we can measure the time on the RPi to a resolution of 1 microsecond and we can measure frequently. So while we run we can compare the time from the real time clock to that more accurate clock. (it is accurate as it is set from ntp and it has finer resolution). Of course the rounding implied in the RTC result compared to the system clock will give  a a false sense of accuracy.  So what I decided to do was to round both and compare to only 1 second resolution. You might ask how does this help? Well what we can do now is see the statistics of the comparison at 1 second resolution and use them to see the drift rate.

First we set the RTC to be the same as the system clock set by ntp. Then we read both and compare the seconds. Initially we will get an answer of 0, with the occasional -1 or 1 due to the rounding. Gradually as the RTC drifts the number of reads that say 1 will increase (assuming it is running slower than the system time.This gives us an estimate of how far apart they are. In effect we are looking at the ‘density’ of rounding transitions up or down over time to estimate the drift.

This looks like the following;

The y axis is the delta and the x axis is the run time… here I picked 256 samples of a part of the run about 1000 readings into the process.

clockerrortheory2

So now time for a  test run. I started the program and let it log to a file. Unfortunately after about 2 days the RPi crashed … but after all that was the point of looking at this method… to avoid the need for very long runs. Here is the picture of the just over 11,000 points taken every 15 seconds.

clockerrortheory3

Now we can see a clear trend of drift between the clocks. There is a fair amount of noise due to the rounding jitter so we’ll need some filtering a to derive a trend. I applied an average of the offset (last 64, 128,  256 and 512 samples), calculated the sliding average  and plotted a trend in excel to see if this technique would work. Here is the result:

clockerrortheory4

It looks like we have about 8ppm drift from this result. of course we could try estimate to more accuracy, but that would require more detailed examination of the filter length and noisiness of the signal to come up with something reliable… I did a quick check of the results and 1ppm seemed to be about the right level of resolution as with the different filters the slope was varying by about 0.5ppm. While running this test  the temperature variation was about +/- 2C and the ntp was locked to a stratum 2 clock.

So in conclusion, in 2 days we managed to get an estimate of clock drift using the trend line of the offsets even though the absolute drift was only 2 seconds and the resolution of the RTC is 1 second. Next we need to test this for a longer period of time, and build in the filters and calculations to remove the manual processing steps in excel. Plus look more into the ntp stats for the local clock drift compensation as we are measuring versus the system clock slaved to ntp.

Here is the code for the program.


#!/usr/bin/env python
#
# ClockTest Program
# original and driver by John C. Shovic, SwitchDoc Labs
# 07/10/2014
# Modified by hobbyistdave, Oct 2015
#
# imports

import sys
#sys.path.append ('./lib')              # path for local libraries
import time
import datetime
import requests   # HTTP for humans
                  # by Kenneth Reitz  docs.python-requests.org
import SDL_DS1307 # the RTC driver

# setup the info needed to send the results to thingspeak

thingspeakurl = "https://api.thingspeak.com/update"   # url for sending updates to thingspeak
apikey = "1234567890123456"     # thingspeak api address for this channel 
ioTInterval = 15                # interval for maximum writes to thingspeak in seconds (min interval is 15 seconds)
url = thingspeakurl

def writeToIoT(payload):        # function to write to IoT
    try :
        r = requests.get(url, params=payload)  # send to thingspeak
        print time.time(), "  ", r.text, "  ", ntpSecs ,"   ", rtcSecs ,"   ", deltaSecs ,"      ", runTime , " ", r.status_code  
        #r.text will be the data point number
        #r.status code should be 200 if the page posted correctly
    except requests.exceptions as e :
            print e # some simple error trapping to avoid the program timing out
    except requests.exceptions.ConnectionError as e1 :
            print e1 # some simple error trapping if the connection is lost
    return

# Main Program

print ""
print "ClockTest, hobbyistdave"
print ""
print "Program Started at: "+ time.strftime("%Y-%m-%d %H:%M:%S")

filename = time.strftime("%Y-%m-%d%H:%M:%SRTCTest") + ".txt"
startTime = datetime.datetime.utcnow()
#print startTime
print "Writing system time to RTC to make them equal at start of test"
ds1307 = SDL_DS1307.SDL_DS1307(1, 0x68)
ds1307.write_now()


try :    # wrapper to allow clean exit
    # now setup the command shell for printing the results
    print " "
    print "---------------------------------------------------------------------------"
    print " "
    
# Main Loop - sleeps 1 seconds, then reads and prints values of all clocks
# checks time since last update and writes to IoT

    startTime = datetime.datetime.now()
    print "startTime : ", startTime
    print " "

    print "---------------------------------------------------------------------------"
    print "  Time          Point    NTP     RTC   Delta    Runtime         Response   "
    print "  (s)                   (s)     (s)   (s)                                  "
    print "---------------------------------------------------------------------------"

    ioTTimeStamp = time.time()
    lastDelta=0   # initialise delta
    while True:

        ntpTime = datetime.datetime.now()
        rtcTime = ds1307.read_datetime()
        #deltaTime = ntpTime - rtcTime
        ntpSecs = datetime.datetime.now().strftime('%S')
        rtcSecs = rtcTime.strftime('%S')
        #print ntpTime, "  ", rtcTime, "  ", deltaTime
        deltaSecs =  int(ntpSecs)  - int(rtcSecs)
        if lastDelta < 30 and deltaSecs <-30 : # fix instances where the seconds wrap across a minute boundary deltaSecs = deltaSecs+60 # we dont expect time to drift apart by more than 30 seconds elif lastDelta > -30 and deltaSecs >30 :
            deltaSecs = deltaSecs-60
        lastDelta = deltaSecs  # remeber the last delta

        runTime = ntpTime - startTime

        payload = {'key' : apikey, 'field1' : ntpTime , 'field2' : rtcTime , 'field3' : deltaSecs , 'field4' : runTime  } # make the payload
        if time.time() >= (ioTTimeStamp + ioTInterval):  #check if enough time has passed before updating thingspeak
            writeToIoT(payload)
            ioTTimeStamp = time.time()
        else:
            time.sleep(0.001)

        time.sleep(1)

except KeyboardInterrupt:
    print " "
    print "Exiting program due to Keyboard Interrupt"
    print " "
    time.sleep (1)
finally:
    print "----------- The End----------------"



Pressure, Humidity and Temperature

My weather project is coming along nicely. Now I have added the DHT22 humidity and temperature sensor. Once again I used the drivers from Adafruit. Here is a link to the information: Adafruit DHT tutorial

A few pointers… somehow I managed to install the modules as root, not under my login name (the penalty of not paying attention with lots of terminal windows open). Use ‘chown -R’ to change back the ownership. The DHT22 uses the Dallas Semiconductor one wire interface. On the Raspberry Pi this is accomplished using a GPIO pin and thus the program needs to be run as ‘sudo’ to command the pins. The one wire interface is not as robust as others, so its possible that you will get error messages (if the program crashes with an error of the wrong data type then what happened is that  a ’round’ was not returned). Hence I use the read_retry command from the Adafruit library. The DHT22 library returns a long , so I have used round() to truncate the answer for display on the console.

You should note that the one wire bus needs a pull up resistor on the bus. I am using a 4.7kohm pull up to Vcc. Only one pull up resistor is required. Note how the BMP180 and the DHT22 are powered from different rails.

DHT22+BMP180_2

A few other notes:

I have added code to enable a cleaner exit from the program (the wrapper ‘try’ loop)

To check the i2c devices use “sudo i2cdetect -y 1”.  This will show you the addresses of the various modules. Each needs to be unique. The BMP180 should show up at address 77. If you see UU it means that the kernel has a module running and talking to that address.

i2CdetectSCR

you can see here that I have another device on my I2C… it happens to be a DS1307 RTC.

Here are the new results. Looks like the DHT22 reported an erroneous value in there so eventually I will need to take a look at the driver and see if the robustness can be increased.

temp2humiditypressure

Here is the circuit. Note that the BMP180 is powered from 3.3V and can communicate with the RPi directly. I am using a bi-directional level converter (Fritzing part included in the files) to communicate with the DHT22 which I have powered from 5V.

pressuretemphumidity2

Fritzing files can be found at http://fritzing.org/projects/pressure-temperature-and-humidity

#!/usr/bin/python
# Original Author : Tony DiCola of Adafruit
# With additions by hobbyistdave  www.hobbyspot.org
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

# Can enable debug output by uncommenting:
#import logging
#logging.basicConfig(level=logging.DEBUG)

import Adafruit_BMP.BMP085 as BMP085   # the pressure and temnperature sensor
import Adafruit_DHT as DHT22           # the DHT22 hunidty and temperature sensor
import smbus
import time
import requests  # HTTPfor humans
                 # by Kenneth Reitz  docs.python-requests.org

# Default constructor will pick a default I2C bus.
#
# For the Raspberry Pi this means you should hook up to the only exposed I2C bus
# from the main GPIO header and the library will figure out the bus number based
# on the Pi's revision.
#
# For the Beaglebone Black the library will assume bus 1 by default, which is
# exposed with SCL = P9_19 and SDA = P9_20.
sensor = BMP085.BMP085()  # pressure and temperature
sensor2 = DHT22           # humidity and temperature
onewirepin = 4            # the pin for the one wire sensor
sensor1type = "BMP180"    # the type of sensor    
sensor2type = 22          # the type of sensor    

# Optionally you can override the bus number:
#sensor = BMP085.BMP085(busnum=2)

# You can also optionally change the BMP085 mode to one of BMP085_ULTRALOWPOWER, 
# BMP085_STANDARD, BMP085_HIGHRES, or BMP085_ULTRAHIGHRES.  See the BMP085
# datasheet for more details on the meanings of each mode (accuracy and power
# consumption are primarily the differences).  The default mode is STANDARD.

sensor = BMP085.BMP085(mode=BMP085.BMP085_ULTRAHIGHRES)

# setup the info needed to send the results to thingspeak

thingspeakurl = "https://api.thingspeak.com/update"   # url for sending updates to thingspeak
apikey = "1234567890123456"     # thingspeak api address for this channel 
TempField = "field1"            # temperature field
PressField = "field2"           # pressure field

# some debug info... here we print the BMP180 cal table
print "Calibration Coefficients read from sensor device "
print "-------------------------------------------------"
print "ac1 ", sensor.cal_AC1
print "ac2 ", sensor.cal_AC2
print "ac3 ", sensor.cal_AC3
print "ac4 ", sensor.cal_AC4 
print "ac5 ", sensor.cal_AC5
print "ac6 ", sensor.cal_AC6
print "b1  ", sensor.cal_B1
print "b2  ", sensor.cal_B2
print "mb  ", sensor.cal_MB
print "mc  ", sensor.cal_MC
print "md  ", sensor.cal_MD
print "--------------------------------------------------"
print " "

print "Initial data read to check sensors            "
print "-------------------------------------------------"
print "Time          ", time.ctime()
print "Epoch Seconds ", time.time()
print " "
temperature = sensor.read_temperature()  # Note: fixed an error in the Adafruit module where it incorrectly divides by 10 
pressure = sensor.read_pressure()

print "Data read from Sensor 1              "
print "---------------------------------------------------------"
print "Sensor type", sensor1type
print 'Temperature= {0:0.1f} *C  Pressure= {1:0.1f} mbar'.format(temperature,pressure/100)
print " "

print "Data read from Sensor 2              "
print "---------------------------------------------------------"
print "Sensor type", sensor2type
humidity2, temperature2 = DHT22.read_retry( sensor2type, onewirepin )
print 'Temperature= {0:0.1f} *C  Humidity= {1:0.1f} %'.format(temperature2,humidity2)
print " "

print "---------------------------------------------------------------------------"
print "  Time          Point    Temp     Temp2   Pressure     Humidity     Status "
print "(Epoch)           #       C         C        mbar          %         Code  "
print "---------------------------------------------------------------------------"
print " "
time.sleep(2)  # wait for the DHT sensor as it does not like to be accessed faster than every 2 seconds
# Now we enter the min code

try:           # wrapper to allow clean exit
    while True:

        temperature = sensor.read_temperature()
        pressure = sensor.read_pressure()
        humidity2, temperature2 = DHT22.read_retry( sensor2type, onewirepin )
        humidity2 = round(humidity2,3)
        temperature2 = round(temperature2,3)
        time.sleep(0.5)

        url = thingspeakurl
        payload = {'key' : apikey, 'field1' : temperature , 'field2' : pressure/ 100. , 'field3' : temperature2 , 'field4' : humidity2 } # make the payload
    
        try :
            r = requests.get(url, params=payload)  # send to thingspeak
            print time.time(), "  ", r.text, "   ", temperature ,"   ", temperature2 ,"   ", pressure / 100.,"      ", humidity2, "      ", r.status_code
            #r.text will be the data point number
            #r.status code should be 200 if the page posted correctly
        except requests.exceptions as e :
            print e # some simple error trapping to avoid the program timing out
        except requests.exceptions.ConnectionError as e1 :
            print e1 # some simple error trapping if the connection is lost
    
        time.sleep(15) # thingspeak is set to ignore requests that come faster than every 15 seconds

except KeyboardInterrupt:
    time.sleep (0.1)
    print " "
    print "Exiting program due to Keyboard Interrupt"
    print " "
except:
    time.sleep (0.1)
    print " "
    print "An error occurred"
    print " "

# end
 

The terminal output looks like this

presstemphumidSCR

Graphing sensor data from a Raspberry Pi

I’m interested in the Internet of Things and thought that the Raspberry Pi would be a good way to explore it. So as a getting starter project I attached a temperature and Humidity sensor to my RPi and set about exploring. The sensor I used is the Bosch BMP180 digital barometer and temperature sensor. This is the successor of the BMP085 and uses I2C to communicate with the host.  BMP180 Datasheet

I purchased it on a small board suitable for breadboarding.

BMP180

To communicate with the BMP180 I used ADAFruit’s BMP085 libraries. They have a fantastic tutorial on how to use it and how to connect the device. AdaFruit BMP085 info

I also found some great data on this device  posted by on his blog at http://blog.bitify.co.uk/

Here is the overall setup. In the picture you can see the RPi is feeding its I2C bus to a breadboard. I am using an external power unit (9V brick feeding regulators with 5V and 3.3V outputs).  In general I am using Arduino sesnor modules setup for 5V whereas the RPi uses 3.3V, so there is a bidirectional level shifter converting between the two voltage rails that you can see to the right by the power supply. On the left is the BMP180 temperature and barometer device… in the center is a real time clock (not being used here) and just off the picture a DHT22 one wire humidity and temperature sensor that will be used in future projects.

BMP180+RPi

Once the sensor was running the next task was to plot the results. At first I used GNUplot on the RPi but that was not a long term solution. After a bit of research I came upon thingspeak, an internet of things platform run by The Mathworks. This seemed ideal as it allows you to set up channels that you can write to as webpages, and it plots them in real time and makes them accessible on the internet. This seemed much easier than setting up a local web server. There are a couple of limitations that seem minor, basically each channel supports up to 8 fields and you can only send data every 15 seconds.

Now the task was to get the information sent to the thingspeak site. I played a bit with the native python libraries (UrlLib) for doing this but they make pretty unreadable code that is hard to debug. Then I stumbled on this fantastic library that made things readable.

http://docs.python-requests.org/en/latest/  by Kenneth Reitz who with his colleagues have done a great job of making this task simpler. With ‘requests’ sending the data to thingspeak became much easier to debug.

Here is the code that I put together. It uses the Adafruit BMP library coupled to “requests” to send to thingspeak.


#!/usr/bin/python
# Original Author : Tony DiCola of Adafruit
# With additions by hobbyistdave  www.hobbyspot.org
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

# Can enable debug output by uncommenting:
#import logging
#logging.basicConfig(level=logging.DEBUG)

import Adafruit_BMP.BMP085 as BMP085
import smbus
import time
import requests  # HTTP for humans by Kenneth Reitz  docs.python-requests.org

# Default constructor will pick a default I2C bus.
#
sensor = BMP085.BMP085()

# You can also optionally change the BMP085 mode to one of BMP085_ULTRALOWPOWER,
# BMP085_STANDARD, BMP085_HIGHRES, or BMP085_ULTRAHIGHRES.  See the BMP085
# datasheet for more details on the meanings of each mode (accuracy and power
# consumption are primarily the differences).  The default mode is STANDARD.

#
sensor = BMP085.BMP085(mode=BMP085.BMP085_ULTRAHIGHRES)

# setup the info needed to send the results to thingspeak

thingspeakurl = "https://api.thingspeak.com/update"   # url for sending updates to thingspeak
apikey = "1234567890123456"     # add your own thingspeak api address for your channel
TempField = "field1"            # temperature field
PressField = "field2"           # pressure field

temperature = sensor.read_temperature()  # Note: fixed an error in the Adafruit module where it incorrectly divides by 10
pressure = sensor.read_pressure()

print " "
print "---------------------------------------------------------"
print "Time          ", time.ctime()
print "Epoch Seconds ", time.time()
print "Temperature   ", temperature
print "Pressure      ", pressure/ 100.
print " "
print "---------------------------------------------------------"
print " "
print "Time            Point  Temp    Pressure        Status    "
print "---------------------------------------------------------"
print " "

while True:

temperature = sensor.read_temperature()
pressure = sensor.read_pressure()

time.sleep(1)

url = thingspeakurl   # the thingsspeak url
payload = {'key' : apikey, 'field1' : temperature , 'field2' : pressure/ 100. }  # set up  the payload

try :
r = requests.get(url, params=payload)   # send the data to thingspeak
print time.time(), "  ", r.text, "   ", temperature ,"   ", pressure / 100.,"      ", r.status_code # print locally
except requests.exceptions as e :
print e # some simple error trapping to avoid the program timing out
except requests.exceptions.ConnectionError as e1 :
print e1 # some simple error trapping if the connection is lost

time.sleep(15) # thingspeak is set to ignore requests that come faster than every 15 seconds

# end of code

Locally on the RPi you get a print out of the temperature and pressure, timestamp, data point number and return code from the server… In the code you can see the apikey (put your own one in there).  The thingspeak website has great instructions for setting their end up.

The key code block shows the elegance of the  “requests” api for handling the http:


url = thingspeakurl   # the thingsspeak url
payload = {'key' : apikey, 'field1' : temperature , 'field2' : pressure/ 100. }  # set up  the payload
r = requests.get(url, params=payload)   # send the data to thingspeak

And published to the cloud you get:

temppandpressure

Which is what I was aiming to achieve.

Raspberry Pi and the ZX Spectrum

I only discovered the Raspberry Pi in late 2014 and was struck by the concept and also the history. The background of the founders really resonated. What happened in the late 70’s and early 80’s in the UK was that a whole generation of new engineers and computer scientists was inspired by easily accessible computers. This charge was led by the BBC with their educational programming, leading to the development of the BBC micro computer and its subsequent deployment into after school computer clubs. In parallel with this companies like Sinclair with their ZX81 and Spectrum made computing easily accessible to schoolkids at home. No longer was a single computer used for the whole class (I remember the single Research Machines 380Z we had at school), but instead individuals could use their own machine both at school and at home. This wave of experimenters did not have to be as hard core as previous generations and were enticed into a rewarding and fulfilling profession by their hobby. These students became the front end of the UK’s tech explosion and that heritage led to platforms like the ARM core that resides in so many smartphones. Indeed, Acorn the manufacturer of the BBC micro is the progenitor of ARM Holdings. The Raspberry Pi was intended to address the issue of technical skill among people entering Universities. In the 80’s and 90’s many of those entrants had been pretty experienced hobbyists, whereas in the 2000’s it appeared that they were more like computer users. So the Pi was intended to restart that trend.  What is interesting about the Raspberry Pi  though is that of the millions shipped it appears that most are in the hands of adult hobbyists… the schools part of the program is limited by both student interest and teaching resources whereas hobbyists have adopted it in droves.

The BBC micro computer I used was at school and I don’t have a photo, but in honour of the Pi here is a photo of my original ZX Spectrum. The 48K RAM version no less, costing 175 pounds back in 1982.

ZX Zpectrum

Spectrum1