Category Archives: Projects

Safe SCP and Delete

My current project involves creating many files on a Raspberry Pi, then immediately transferring them to a more traditional linux server with normal spinning disk hard drives and more system resources.

In order to reduce writes on the Rasberry Pi’s SD card, I intend to store these files on a ramdisk, then I need a safe way to copy the files to the remote server.  In the event a connection is unavailable, this script will detect that the connection failed, and move the file to a failback directory (on the SD card) instead of deleting it after the scp command completes.  If the file is moved to the failback directory, an accompanying json file is also created to store the necessary information.

Assumptions:

  • Script will be run from the source computer.
  • Both computers will be running current versions of OpenSSH.
  • Public key authentication is set up so that it does not require a password to ssh from the source to the destination.
  • scpis installed on the source, and located at /usr/bin/scp. (Verify using the command: which scp).

Variables required for the function:

  • source_file – This is the absolute location of the file to be transferred
    Example: /home/user/temp/transfer_this_file.jpg
  • destination_directory – This is the destination location as would be specified when using the scp command.
    Example: user@remotehost:/home/user/destination_directory/
  • failback_directory – Absolute location of the failback directory on the source computer.
    Example: /home/user/failback/

Python modules imported:

  • subprocess – Yes, I realize that os could be used instead of subprocess, but I’m already using subprocess for other functions in the same file.
  • os
  • uuid
  • json

The variable p is returned as to report on the status of the original transfer.  A zero is returned of the transfer was successful.  A one is returned if the transfer failed.

#! /usr/bin/python

import subprocess
import os
import uuid
import json

destination_directory = "user@remotehost:/home/user/incoming"
fail_directory = "user@notahost:/notadirectory"
fail_user = "bobert@remotehost:/home/ryan/temp"
fallback = "/home/pi/motion/fallback/"

def safe_scp(source_file, destination_directory, fallback_directory):
	# /usr/bin/scp -qB /source_filesystem/source_file.jpg user@host:/destination/
	cmd = "/usr/bin/scp -qB "+str(source_file)+" "+str(destination_directory)
	cmd = cmd.split()
	p = subprocess.Popen(cmd).wait()
	if (p == 1):
		# upload failed, move it to the fallback directory
		filename = os.path.split(source_file)[1]
		#
		# Using os.path to create an absoltue filename underneath the failback
		# directory for rename.
		fallback_file = os.path.abspath(os.path.join(fallback_directory, filename))
		os.rename(source_file, fallback_file)
		# generate filename with info to upload
		json_data = {
			"source_file" : fallback_file,
			"destination_directory" : destination_directory,
			"fallback_directory" : fallback_directory }
		# using uuid to ensure a truly unique filename
		json_filename = "failed_upload-"+str(uuid.uuid4())+".json"
		# creating the absolute filename under the failback directory
		json_filename = os.path.join(fallback_directory, json_filename)
		f = open(json_filename, "w")
		f.write(json.dumps(json_data))
		f.close()
	else:
		# delete source file in case of a successful transfer.
		os.remove(source_file)
	return p

# usage: I've created a bunch of blank text documents to test the transfer.

# simulating errors
print safe_scp("/home/pi/motion/1.txt", fail_directory, fallback)
#print safe_scp("/home/pi/motion/2.txt", fail_user, fallback)
# above line was commented out later in testing after a conflict with denyhosts
print safe_scp("/home/pi/motion/2.txt", fail_directory, fallback)

# normal operation
print safe_scp("/home/pi/motion/3.txt", destination_directory, fallback)
print safe_scp("/home/pi/motion/4.txt", destination_directory, fallback)
print safe_scp("/home/pi/motion/5.txt", destination_directory, fallback)
exit()

A quick note: Careful when playing around with intentionally failed authentication.  My computer is set up to block an ip after a number of failed login attempts.  I actually locked myself out from my Raspberry Pi when testing, and it took me longer than I care to admit to figure out what the exact problem was.  Hope this helps!

DIY Raspberry PI Camera Case

I’m currently working on a image analysis program using my Raspberry Pi and the Raspberry Pi camera.  Once that is in a complete state, and running full time, I plan to purchase a commercially produced mount.  In the meantime, I decided to make a little DIY Raspberry Pi mount.

Most hobby electronics ship in small electrostatic bags.  I keep every one of these I get.  I used one, trimmed it down, then cut a small hole for the camera to peek out.   This particular bag contained a Raspberry Pi.

Step 1 - Electrostatic bag

(forgive my crappy cameraphone focus, but I’m not going to take it apart again just to get more pictures.)

Next, I cut some cardboard to an approximate width of the camera board.  I used very thick cardboard and a boxcutter.

Step 2 - Cut some cardboard

 

Then, I taped one end of the electrostatic bag to to the cardboard so that I could wrap the bag around the cardboard.  Be sure to position it so the camera board will be centered on the cardboard.  I decided to use Scotch tape for this project as it would stick well to the bag.  I feared that duct tape would leave residue.

Step 3 - Tape the bag

Then, just wrap the bag around the cardboard and tape the other edge.

Step 4 - Wrap the bag

I used more tape, and just re-enforced the hole I had cut in. It may be difficult to see.

2013-08-28 11.36.30

I’m personally mounting my camera to look out the window, so I used more cardboard to sit in between the camera and the window.

2013-08-28 11.40.22Then, I just taped the whole thing to the window.  As you may have noticed, I added more cardboard below the camera to tilt the camera downward.

Tape to the windowThis will do just fine!  Again, I’ll be ordering a commercially produced mount  to replace this once my code is ready.  At that point, I’ll be mounting the camera higher on the window to avoid looking through the screen.  I like the mount currently for sale on Adafruit, but I’m also hoping for more variety in the coming weeks or months.

While I’m at it, I reviewed the Raspberry Pi here on Element 14’s community site: http://www.element14.com/community/roadTestReviews/1520

PiFace Digital Review

I was selected by Element14 to write a review for the PiFace Digital.

2013-05-20-21.04.50-768x1024

They were kind enough to ship me a free one, so I’m only going to link to the review on their page:
http://www.element14.com/community/roadTestReviews/1457

Here is the purchase link:
http://www.newark.com/piface/piface-digital/daugther-card-pi-face-expansion/dp/48W3976?COM=rasp-accessory-group

I’m currently working on a project to control and monitor the garage doors in my home.  I will document that project here once I’m able to work on it more, and get something working.

Aspell Custom Dictionary

I’ve been playing around a bit with google go and the aspell package.  It’s been working great, except I haven’t found a good way to tell aspell to exclude or include words as needed.

I was able to find ways to add words to your custom dictionary, but I did not find a good way to exclude the custom words.  In this particular project, I found that many small two letter combinations were marked as correct, but they were not defined as words according to the dictionary.  Perhaps they are abbreviations, but I did not want them marked as correct for this particular project.

Instead of messing with the existing aspell dictionary, I decided to create a new language dictionary in aspell.

The command “aspell dicts” will dump the existing dictionaries so you can see what already exists.  I chose rv_EN to use.  The dictionary files are kept in /usr/lib/aspell on my system.

First, I created the file rv_EN.multi, which contains only a single line: “add rv_EN.rws”.  The command “aspell dicts” will confirm that aspell can now see the en_RV dictionary.

Now, we will need to create the rv_EN.rws file that defines our dictionary.  This is essentially a three step process.

  1. Dump existing dictionary into a text file
    /usr/bin/aspell -d en dump master | aspell -l en expand > /home/ryan/cust_dict/words.txt
  2. Add or remove words as needed
    I created remove_bad.py for this
  3. Convert text file into custom.rws
    sudo aspell –lang=en create master /usr/lib/aspell/rv_EN.rws < /home/ryan/cust_dict/goodwords.txt

I’ve scripted this process, and have put all necessary files in /home/ryan/custom_dict/.  When running the scripts, I have three files:

  • exclude.txt – this contains a list of the words I would like to remove from the dictionary
  • remove_bad.py – This is a python script that generates a new word list.
  • update.sh – Shell script that will execute all commands.  It should be run as root as you will need root privs to write to /usr/lib/aspell/.

Here are my scripts:

#! /bin/bash
# update.sh - run this as root.
# This could be entered into cron, but I have not done so, as I just run
# the script manually after editing the excluded words text file.
#
# creates words.txt by dumping the english dictionary from aspell
# calls remove_bad.py, which generates goodwords.txt
#    goodwords.txt is all words in words.txt except for those listed in
#    exclude.txt
# creates /usr/lib/aspell/rv_EN.rws from goodwords.txt
# rv_EN is already configured to use custom.rws only

# export english dictionaries to words.txt
echo "Exporting words to text file."
/usr/bin/aspell -d en dump master | aspell -l en expand > /home/ryan/cust_dict/words.txt

# remove the bad words
/home/ryan/cust_dict/remove_bad.py

echo "Converting word list into dictionary file."
aspell --lang=en create master /usr/lib/aspell/rv_EN.rws < /home/ryan/cust_dict/goodwords.txt

echo "Cleaning up!"
rm /home/ryan/cust_dict/words.txt
rm /home/ryan/cust_dict/goodwords.txt

 

#! /usr/bin/python
# remove_bad.py - this script generates a text file containing a list of
# good words to include into the aspell dictionary.
# remove_bad.py is called by update.sh

# open up list of words to remove from aspell dictionary.
# File should contain one word per line.
f = open("/home/ryan/cust_dict/exclude.txt")
badw = f.readlines()
f.close()

# status message showing how many words are in exclude list
print len(badw), "words in the exclude list."

# opens up the text dump of existing dictionary
f = open("/home/ryan/cust_dict/words.txt")
lines = f.readlines()
f.close()

# number of words in original dictionary
print len(lines), "words in the original dictionary."

# create file of good words
f = open("/home/ryan/cust_dict/goodwords.txt","w")

# this will write the dictionary words into goodwords.txt
# if they do not exist on the exclude list
for line in lines:
if line not in badw:
f.write(line)

# a function could be added here to add words to goodwords.txt
# if desired.

f.close()
exit()

Yes, I realize that I could probably do this with just a shell script, but I just prefer coding the file operations with python.  Either, way, I have my new dictionary, so I just define that when I’m creating my speller in go:

speller, err := aspell.NewSpeller(map[string]string{"lang": "rv_EN",})
// code from the go-aspell documentation.

Reference links: golang.orggo-aspell on Github | aspell

Edit: I found that my custom localization of english wasn’t working as expected, so I created a new language/local title rv_EN.  I’ve updated my post to reflect my changes.

Still kicking, a quick update

I must apologize, I’ve left this blog a bit lifeless lately.  I haven’t been completely idle, but unfortunately, I can’t share intimate details on everything, but I can recap a few things I’ve dabbled in:

Xbox Live Tracker – Yes, my previous efforts were rendered inactive, and I’ve been putting out feelers to try and find a good alternate feed to breathe life back in the project.  I’ve got a few interesting ideas for adding new stats to track and fleshing out the page.  Over the last year, I stumbled upon a site that is currently working to supply a public feed.  That is XboxLeaders.com.  For a short time, I resurrected my Xbox live tracking script, they too decided to kill the public feed.  I’ve put requests out to gain access to the official Xbox API, but I doubt that is possible.  So, Xbox live tracker 2.0 is officially on the back burner.  Please feel free to link me to any other feeds, and I shall try to adapt to them.

Python Based IRC Bot – Yes, they exist, but I’ve also started writing one of my own.  I found that this gives me much more freedom in coding triggers.  I’ve been able to mate the Python bot with a MySQL database to keep track of data.  I’ve used it to query Google’s Weather API, an active shoutcast server, and keep track of triggers in the rooms.  Once I’ve perfected my script, I plan to share more information.

In other news, my personal life is becoming hectic, but I’m feeling the call to play again.  I’m hoping to play around with some home automation and microprocessors in the near future.  I guarantee nothing, but I’ll remain hopeful.

Also, I started a small referral link page.  Hopefully, I can get some free Dropbox gigs! (please?)  Don’t think of it as selfish spamming.  I’m recommending you try the service because I use it.  If it will help you in some way, the least you could do is help me out a bit by using the link.