Oscilloscope

Once you are building electronics projects, the ability to take a look at what is going on in the real world is invaluable for troubleshooting. For this you really need an oscilloscope which are usually quite expensive. I found this kit that gives a low cost way of entering the world of electronics instrumentation.

oscilloscopekit

I bought the version with the surface mount chips presoldered as they are tiny and I was careful to sort the components before starting.

oscilloscopechip

I used my component tester to make sure that I was using the right components. Here it is working connected to the test signal.

oscilloscope1 (3)

You can power it from a small 9V battery making it very portable. While it only works up to 200kHz and the screen in tiny at 2.4 inches, that is plenty good enough for most projects and you can’t beat being able to see what is happening when you are building a project or writing some software that interacts with the real world.

Component Tester

When doing electronic projects, its critical to put in the right components. I found this fantastic component tester that solves that problem economically.

componenttester1 (2)

It can test Transistors, Diodes, LEDs, resistors, capacitors and Inductors.

Here are a few more modes.

 

The unit automatically detects the type of device under test. See how for the LED the forward voltage has been tested.

I purchased it from here: Banggood Component Tester

You can get kits and a few variations on the basic theme. I purchased a ready made one as the point was to be able to test components for building other kits. So far its been $22 well spent as I was able to avoid making mistakes when reading tiny 5 band resistor colour codes  like these that I had to use an eye loupe to read. In this case the red and brown were so close in colour I really needed the tester to be sure. Its much easier than a multi-meter. It has 3 ports for transistors. The only slight flaw i can see is sometime with an open port it thinks there might be a very large resistor present, but it still measures the device under test correctly. Also it does not measure under 30pF capacitors.

resistors1

I did some research and found these original sources that contain the schematics and code and theory of operation.

Markus Frejek

Karl-Heinz Kübbeler

Karl-Heinz Kübbeler continued

 

 

Real Time Clock accuracy

I have a real time clock attached to my RPi and I thought that it would be an interesting project to monitor its accuracy. The clock I am using is a Maxim DS1307… this type does not have any fancy compensation, but from what I read it should be around 10ppm.

So I built a program for monitoring the clock and comparing it with a reference. For a reference I am using the RPi system clock slaved to NTP… which should provide synchronisation to within a few milliseconds. The linux clock of course reports to quite a fine resolution (1 microsecond ticks)… but the DS1307 only reads out to the 1 second level which means on the face of it that a fairly long run would be needed to make an estimate of the clock accuracy.

Here is a table of expected deviation in seconds for clocks of various accuracies… as you can see we would have to wait for 10 days to get a result from a 10ppm clock with a significant offset. The problem being after 1 or 2 days the error might be 1 second, and the resolution of the measurement is +/-1 second… not great for measuring drift.

clockerrortheory1

So I thought a bit more about the problem of measuring. We actually have more info than this as we can measure the time on the RPi to a resolution of 1 microsecond and we can measure frequently. So while we run we can compare the time from the real time clock to that more accurate clock. (it is accurate as it is set from ntp and it has finer resolution). Of course the rounding implied in the RTC result compared to the system clock will give  a a false sense of accuracy.  So what I decided to do was to round both and compare to only 1 second resolution. You might ask how does this help? Well what we can do now is see the statistics of the comparison at 1 second resolution and use them to see the drift rate.

First we set the RTC to be the same as the system clock set by ntp. Then we read both and compare the seconds. Initially we will get an answer of 0, with the occasional -1 or 1 due to the rounding. Gradually as the RTC drifts the number of reads that say 1 will increase (assuming it is running slower than the system time.This gives us an estimate of how far apart they are. In effect we are looking at the ‘density’ of rounding transitions up or down over time to estimate the drift.

This looks like the following;

The y axis is the delta and the x axis is the run time… here I picked 256 samples of a part of the run about 1000 readings into the process.

clockerrortheory2

So now time for a  test run. I started the program and let it log to a file. Unfortunately after about 2 days the RPi crashed … but after all that was the point of looking at this method… to avoid the need for very long runs. Here is the picture of the just over 11,000 points taken every 15 seconds.

clockerrortheory3

Now we can see a clear trend of drift between the clocks. There is a fair amount of noise due to the rounding jitter so we’ll need some filtering a to derive a trend. I applied an average of the offset (last 64, 128,  256 and 512 samples), calculated the sliding average  and plotted a trend in excel to see if this technique would work. Here is the result:

clockerrortheory4

It looks like we have about 8ppm drift from this result. of course we could try estimate to more accuracy, but that would require more detailed examination of the filter length and noisiness of the signal to come up with something reliable… I did a quick check of the results and 1ppm seemed to be about the right level of resolution as with the different filters the slope was varying by about 0.5ppm. While running this test  the temperature variation was about +/- 2C and the ntp was locked to a stratum 2 clock.

So in conclusion, in 2 days we managed to get an estimate of clock drift using the trend line of the offsets even though the absolute drift was only 2 seconds and the resolution of the RTC is 1 second. Next we need to test this for a longer period of time, and build in the filters and calculations to remove the manual processing steps in excel. Plus look more into the ntp stats for the local clock drift compensation as we are measuring versus the system clock slaved to ntp.

Here is the code for the program.


#!/usr/bin/env python
#
# ClockTest Program
# original and driver by John C. Shovic, SwitchDoc Labs
# 07/10/2014
# Modified by hobbyistdave, Oct 2015
#
# imports

import sys
#sys.path.append ('./lib')              # path for local libraries
import time
import datetime
import requests   # HTTP for humans
                  # by Kenneth Reitz  docs.python-requests.org
import SDL_DS1307 # the RTC driver

# setup the info needed to send the results to thingspeak

thingspeakurl = "https://api.thingspeak.com/update"   # url for sending updates to thingspeak
apikey = "1234567890123456"     # thingspeak api address for this channel 
ioTInterval = 15                # interval for maximum writes to thingspeak in seconds (min interval is 15 seconds)
url = thingspeakurl

def writeToIoT(payload):        # function to write to IoT
    try :
        r = requests.get(url, params=payload)  # send to thingspeak
        print time.time(), "  ", r.text, "  ", ntpSecs ,"   ", rtcSecs ,"   ", deltaSecs ,"      ", runTime , " ", r.status_code  
        #r.text will be the data point number
        #r.status code should be 200 if the page posted correctly
    except requests.exceptions as e :
            print e # some simple error trapping to avoid the program timing out
    except requests.exceptions.ConnectionError as e1 :
            print e1 # some simple error trapping if the connection is lost
    return

# Main Program

print ""
print "ClockTest, hobbyistdave"
print ""
print "Program Started at: "+ time.strftime("%Y-%m-%d %H:%M:%S")

filename = time.strftime("%Y-%m-%d%H:%M:%SRTCTest") + ".txt"
startTime = datetime.datetime.utcnow()
#print startTime
print "Writing system time to RTC to make them equal at start of test"
ds1307 = SDL_DS1307.SDL_DS1307(1, 0x68)
ds1307.write_now()


try :    # wrapper to allow clean exit
    # now setup the command shell for printing the results
    print " "
    print "---------------------------------------------------------------------------"
    print " "
    
# Main Loop - sleeps 1 seconds, then reads and prints values of all clocks
# checks time since last update and writes to IoT

    startTime = datetime.datetime.now()
    print "startTime : ", startTime
    print " "

    print "---------------------------------------------------------------------------"
    print "  Time          Point    NTP     RTC   Delta    Runtime         Response   "
    print "  (s)                   (s)     (s)   (s)                                  "
    print "---------------------------------------------------------------------------"

    ioTTimeStamp = time.time()
    lastDelta=0   # initialise delta
    while True:

        ntpTime = datetime.datetime.now()
        rtcTime = ds1307.read_datetime()
        #deltaTime = ntpTime - rtcTime
        ntpSecs = datetime.datetime.now().strftime('%S')
        rtcSecs = rtcTime.strftime('%S')
        #print ntpTime, "  ", rtcTime, "  ", deltaTime
        deltaSecs =  int(ntpSecs)  - int(rtcSecs)
        if lastDelta < 30 and deltaSecs <-30 : # fix instances where the seconds wrap across a minute boundary deltaSecs = deltaSecs+60 # we dont expect time to drift apart by more than 30 seconds elif lastDelta > -30 and deltaSecs >30 :
            deltaSecs = deltaSecs-60
        lastDelta = deltaSecs  # remeber the last delta

        runTime = ntpTime - startTime

        payload = {'key' : apikey, 'field1' : ntpTime , 'field2' : rtcTime , 'field3' : deltaSecs , 'field4' : runTime  } # make the payload
        if time.time() >= (ioTTimeStamp + ioTInterval):  #check if enough time has passed before updating thingspeak
            writeToIoT(payload)
            ioTTimeStamp = time.time()
        else:
            time.sleep(0.001)

        time.sleep(1)

except KeyboardInterrupt:
    print " "
    print "Exiting program due to Keyboard Interrupt"
    print " "
    time.sleep (1)
finally:
    print "----------- The End----------------"



Graphing sensor data from a Raspberry Pi

I’m interested in the Internet of Things and thought that the Raspberry Pi would be a good way to explore it. So as a getting starter project I attached a temperature and Humidity sensor to my RPi and set about exploring. The sensor I used is the Bosch BMP180 digital barometer and temperature sensor. This is the successor of the BMP085 and uses I2C to communicate with the host.  BMP180 Datasheet

I purchased it on a small board suitable for breadboarding.

BMP180

To communicate with the BMP180 I used ADAFruit’s BMP085 libraries. They have a fantastic tutorial on how to use it and how to connect the device. AdaFruit BMP085 info

I also found some great data on this device  posted by on his blog at http://blog.bitify.co.uk/

Here is the overall setup. In the picture you can see the RPi is feeding its I2C bus to a breadboard. I am using an external power unit (9V brick feeding regulators with 5V and 3.3V outputs).  In general I am using Arduino sesnor modules setup for 5V whereas the RPi uses 3.3V, so there is a bidirectional level shifter converting between the two voltage rails that you can see to the right by the power supply. On the left is the BMP180 temperature and barometer device… in the center is a real time clock (not being used here) and just off the picture a DHT22 one wire humidity and temperature sensor that will be used in future projects.

BMP180+RPi

Once the sensor was running the next task was to plot the results. At first I used GNUplot on the RPi but that was not a long term solution. After a bit of research I came upon thingspeak, an internet of things platform run by The Mathworks. This seemed ideal as it allows you to set up channels that you can write to as webpages, and it plots them in real time and makes them accessible on the internet. This seemed much easier than setting up a local web server. There are a couple of limitations that seem minor, basically each channel supports up to 8 fields and you can only send data every 15 seconds.

Now the task was to get the information sent to the thingspeak site. I played a bit with the native python libraries (UrlLib) for doing this but they make pretty unreadable code that is hard to debug. Then I stumbled on this fantastic library that made things readable.

http://docs.python-requests.org/en/latest/  by Kenneth Reitz who with his colleagues have done a great job of making this task simpler. With ‘requests’ sending the data to thingspeak became much easier to debug.

Here is the code that I put together. It uses the Adafruit BMP library coupled to “requests” to send to thingspeak.


#!/usr/bin/python
# Original Author : Tony DiCola of Adafruit
# With additions by hobbyistdave  www.hobbyspot.org
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

# Can enable debug output by uncommenting:
#import logging
#logging.basicConfig(level=logging.DEBUG)

import Adafruit_BMP.BMP085 as BMP085
import smbus
import time
import requests  # HTTP for humans by Kenneth Reitz  docs.python-requests.org

# Default constructor will pick a default I2C bus.
#
sensor = BMP085.BMP085()

# You can also optionally change the BMP085 mode to one of BMP085_ULTRALOWPOWER,
# BMP085_STANDARD, BMP085_HIGHRES, or BMP085_ULTRAHIGHRES.  See the BMP085
# datasheet for more details on the meanings of each mode (accuracy and power
# consumption are primarily the differences).  The default mode is STANDARD.

#
sensor = BMP085.BMP085(mode=BMP085.BMP085_ULTRAHIGHRES)

# setup the info needed to send the results to thingspeak

thingspeakurl = "https://api.thingspeak.com/update"   # url for sending updates to thingspeak
apikey = "1234567890123456"     # add your own thingspeak api address for your channel
TempField = "field1"            # temperature field
PressField = "field2"           # pressure field

temperature = sensor.read_temperature()  # Note: fixed an error in the Adafruit module where it incorrectly divides by 10
pressure = sensor.read_pressure()

print " "
print "---------------------------------------------------------"
print "Time          ", time.ctime()
print "Epoch Seconds ", time.time()
print "Temperature   ", temperature
print "Pressure      ", pressure/ 100.
print " "
print "---------------------------------------------------------"
print " "
print "Time            Point  Temp    Pressure        Status    "
print "---------------------------------------------------------"
print " "

while True:

temperature = sensor.read_temperature()
pressure = sensor.read_pressure()

time.sleep(1)

url = thingspeakurl   # the thingsspeak url
payload = {'key' : apikey, 'field1' : temperature , 'field2' : pressure/ 100. }  # set up  the payload

try :
r = requests.get(url, params=payload)   # send the data to thingspeak
print time.time(), "  ", r.text, "   ", temperature ,"   ", pressure / 100.,"      ", r.status_code # print locally
except requests.exceptions as e :
print e # some simple error trapping to avoid the program timing out
except requests.exceptions.ConnectionError as e1 :
print e1 # some simple error trapping if the connection is lost

time.sleep(15) # thingspeak is set to ignore requests that come faster than every 15 seconds

# end of code

Locally on the RPi you get a print out of the temperature and pressure, timestamp, data point number and return code from the server… In the code you can see the apikey (put your own one in there).  The thingspeak website has great instructions for setting their end up.

The key code block shows the elegance of the  “requests” api for handling the http:


url = thingspeakurl   # the thingsspeak url
payload = {'key' : apikey, 'field1' : temperature , 'field2' : pressure/ 100. }  # set up  the payload
r = requests.get(url, params=payload)   # send the data to thingspeak

And published to the cloud you get:

temppandpressure

Which is what I was aiming to achieve.

CAD for electronic projects

As part of exploring the Raspberry Pi and the Arduino, I am interested in the aspect of those computers that interacts with the real world. This means adding sensors and actuators and the circuitry that  runs them and interfaces them to the microcomputers. Plus I wanted to document such projects in a good way that I can reproduce and share. So I have taken a  quick look at the CAD tools available to the amateur for designing electronic circuits.

I will admit that did not do an exhaustive search. I wanted a tool that not only allowed me to do designs but that also had a good community of people with a similar ambition to share ideas with and to get started. Libraries and part creation are some of the most important aspects as if you are doing anything interesting you will soon need to use a wide range of parts and also create your own parts.

At first I started with RS components DesignSpark as it seemed the most comprehensive with a large library of parts and modules and lots . This is a powerful CAD tool available for free to the hobbyist that contains full schematic capture, PCB layout, parts libraries and even simulation tools such as an inbuilt Spice.

I did start using DesignSpark and created a few schematic templates and explored some that other had done.  But the very first small project I started on required me to create new components (a specific opto-coupler that I wanted to use) and I found that this was not very intuitive in the tool and the learning curve a bit steep.

So I thought that I would look further and have now launched into using an open source tool called Fritzing.

This is a simpler tool although it too includes schematic capture, pcb layout, breadboard layout and part libraries. Once again this required me to add a component for the very first project, but the system was relatively simple. In Fritzing you can clone existing parts and use a vector graphics (SVG) editor to adjust them. I tried an on-line graphics tool but it was not precise enough in its dimension handling so now I am using Inkscape which is quite a powerful tool yet I found it reasonably simple to get started using.

While I am a fan of hobbies that require some investment of time to get going, if the process is too frustrating then people give up. So the combo of Fritzing and Inkscape seem good to get going.

Here is a screen shot of my first project.

FirstFritzing

Serial mouse fixes for external GPS on Win 7

One of the back burner projects I’ve been pursuing has been to participate in the global network of ADSB receivers for aircraft tracking. I’ll post more separately on that but as part of it I am running a GPS connected to the laptop where my RTL-SDR receiver connects. The GPS is attached via a USB to serial converter cable based on the Prolific chip or a clone. This seemed to run well, until I rebooted the laptop. At which time the utility I was using stopped being able to talk to the GPS, the com port disappeared and the mouse started acting up. A reboot later, the GPS locked onto the satellites  immediately as shown by its flashing 1PPS light but the mouse was jumping all over my screen even before I could login.  After various bits of research I found that this is a well known issue inherent in windows and is a legacy behaviour.  Windows looks at the serial ports including any virtual ones and if it sees any data it assumes that a serial mouse is connected. It then enumerates that mouse and installs a driver. To be more precise it installs a Microsoft Serial Ballpoint.  Of course the GPS is sending lots of NMEA data on its serial port in a  constant stream so windows thinks it has a mouse attached.

If you research the problem you find that in Win NT, 2000 and XP there was a boot.ini change that you could make and a command line that could be run to disable this. In Win7 the only solution is to edit the registry.

Here’s a nice tutorial that I found. I have only tried the Win7 fix.

Here is the entry you need to change.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\sermouse
The ‘start’ parameter needs to be changed to 4 from 3 to disable the mouse on startup.
Use the command ‘regedit‘ from a command line and before making a change backup the registry by exporting it.

Be warned though.. when you reboot the first time you must not have the GPS attached… if you do it resets the change to the registry and the NMEA messages are interpreted as mouse movement again.

Once you have rebooted once sucessfully, the registry change sticks and you can leave the GPS plugged in on future reboots. It is still enumerated as a serial ballpoint and displays a mouse icon in my case, but that part of the driver is off and the COM port is now open. The perfectionist in me is frustrated that the GPS and serial port adapter are misnamed, but at least it works.

serialmouse1                                     serialmouse2
I also have an FTDI UART adapter on this laptop that does not cause this problem… although that might be because the Arduino attached does not send anything until the laptop has booted.
It remains to be seen whether this ‘fix’ survives a windows update cycle… I’ll let you know.