Writing an SFTP Server in Python

Posted by Tim Freund Tue, 13 Jan 2009 05:47:00 GMT

There's nothing more relaxing than to sit down and write an implementation of a commonly available network service in your favorite scripting language. OK, so maybe writing an SFTP server in Python wasn't the most relaxing thing I could have done this evening, but the Paramiko library made it pretty darn easy, and I had a lot of fun because the work moved so quickly.

Did I mention how quickly the development went? 284 lines, including the license and a demo configuration file. And people wonder why I enjoy Python programming so much.

But Why?

Most code is written to scratch an itch, and this code was no different. We write a number of scheduled jobs at work that upload or download files from SFTP servers. We needed a way to test the jobs, but we didn't want to set up operating system accounts on our servers to run the tests.

My minimal implementation of an SFTP service allows us to test many job configurations with one SFTP service running as an unprivileged user. The user accounts are optionally chroot'ed into their "home" directory, so it appears as if they are connecting to a server that has the exact same file hierarchy that is present on the production file servers.

To play around with the code yourself, you can install Paramiko and clone the Simple SFTP Server Mercurial repository. If you don't have Mercurial installed, you can always download the script straight out of the repository.

The code works, but it is minimally functional: it only does what we needed it to do at work. No worries, though. It is licensed under the MIT license, so you are free to do with it what you please. Patches are accepted if you are feeling generous.

Posted in Open Source, Python | no comments

Evolution of an Android Application: Day 3

Posted by Tim Freund Wed, 07 Jan 2009 04:32:00 GMT

Where we last left off, Rhymote worked, but it was painfully slow. All of the network communication takes place in the UI thread, and that's a big problem. An application has 5 seconds to respond to a UI event, and if the click handler isn't done in that amount of time, the user will be asked if they'd like to terminate the application. The exact error message will look something like this:

Sorry!  Activity Rhymote
(in application Rhymote)
is not responding.

Even though Rhymote is only making one HTTP request per button press, that can put us over the 5 second limit. The HTTP requests return quickly in the emulator, but the phone is a totally different world.

A partial solution to the problem is threading. We want to start a new thread of execution separate from the UI to handle time intensive work. These time intensive tasks can then update the UI if necessary once they are done. If you are at all experienced with Java threading, you will already know how to create a Runnable object and pass that Runnable to a Thread. If you are new here, follow this cheat sheet:

package com.example.androidapp;

public class WebRequestRunnable implements Runnable {
    public void run(){
        // do resource intensive stuff here...
        // create an http request and read the results
    }
}

and

package com.example.androidapp;

public class Application {
    ...
    private OnClickListener mButtonListener = new OnClickListener() {
        public void onClick(View v) {
            WebRequestRunnable wrr = new WebRequestRunnable();
            Thread t = new Thread(wrr);
            t.start();
        }
    }
}

More information about this basic pattern can be found at the following Android resource pages:

• Developing Responsive Applications
• Handling Expensive Operations in the UI Thread

Although the application no longer hangs when responding to a button press, there is still a latency problem. The HTTP request that was taking 5+ seconds to execute in the UI thread is now taking 5+ seconds to execute in a second thread. Two potential issues come to mind:

• The Apache Commons HttpClient code is really slow
• I'm doing something foolish in my code

I learned long ago to never blame another programmer or library without some very solid evidence. When in doubt, it is probably my own fault. Next time we will attempt to find the most efficient way to deal with the required network requests.

Posted in Android, Open Source | no comments

Book Review: The Innovator's Dilemma

Posted by Tim Freund Tue, 06 Jan 2009 02:41:00 GMT

Is it possible to stay on top once you get there? That is the central question in The Innovator's Dilemma. Although I'm far from the top of anything, I read it anyway because so many other people have said it is an amazing book. Oh, and it can be had for a song on Amazon.com.

I read The Innovator's Dilemma primarily because it is mentioned by so many other entrepreneurial people as a great book. The fact that it can be had used for a fraction of the cover price didn't hurt, either.

I was able to storm through the book in a single day, not because it was captivating, but because it was a pretty good read and I was enduring a full day's worth of air travel. The first part of the book was more interesting than Sky Mall, but less interesting than much of the reading material around my house. There were some great nuggets of wisdom in the middle and back of the book that I may have never read if it weren't for my situation.

Christensen uses the hard drive industry as his primary example throughout the book. The industry has a large body of empirical evidence available for research, and, as a bonus for those of us who code, it is an industry that most software developers are already familiar.

Despite the evidence available in the hard drive industry, I was actually more interested in the other examples used in the book, especially the story of the mini steel mill industry.

The example of mini mill steel plants eating away at big steel's market from the bottom up is a great example of why it is OK to focus on small problems. Small problems solved well create opportunities for us to turn our attention to ever larger problems. If businesses can boom and blossom with that approach in a high capital high risk business like steel, then we software developers have no excuse not to consider small problems.

The Innovator's Dilemma isn't a book that I will keep on my shelf to read again and again, but it is one that I am very glad to have read through once.

Posted in Reading List | no comments

Evolution of an Android Application: Day 2

Posted by Tim Freund Fri, 19 Dec 2008 06:31:00 GMT

I'm still working on the Rhymote project.

I sat down with the intent of immediately putting the code on my phone, but it was just so ugly that I had to spend about 30 minutes cleaning things up. Let that be a lesson: late night programming can be fun, but it isn't always productive.

Configuring the G1 for Deployment

First I opened the Settings program and choose Applications.

I checked the box next to Unknown Sources

While I was still on the Applications screen of the Settings program, I selected Development and checked the box next to USB Debugging.

Posted in Android, Open Source | 3 comments

Evolution of an Android Application: Day 1

Posted by Tim Freund Wed, 17 Dec 2008 03:04:00 GMT

Programmers are supposed to be lazy right? Here is the scene for one of my lazy moments: my home office contains a desk and a futon in opposite corners. I often want to change music on my desktop PC while reading on the futon. Most people would just get up to change the music without much thought, but I'd much rather use my G1 as a remote control for Rhythmbox. This will be my first Android project, and it shouldn't be too much to handle for a newbie like myself.

I started the Rhymote project yesterday, and I am writing these posts to keep track of all the hurdles I encounter along the way. Although the project could end up doing all sorts of neat stuff, all I really want to do is a handful of operations:

• Play/Pause
• Next track
• Previous track
• Volume adjustments

I have a tendency to over-engineer solutions, so I'm going to explicitly avoid anything that smacks of over ambitious design. Quick and dirty is the name of the game. The code will grow up if it holds my interest for long enough.

Here's the gear I am working with:

• Ubuntu 8.10
• Rhythmbox 0.11.6
• Eclipse 3.4
• Android SDK + Eclipse Plug-in
• T-Mobile G1

All of the code that I write throughout the series can be found in this repository.

Start an Android Project

I write Java in Eclipse most of the day, so I felt pretty comfortable starting the Android portion of the Rhymote project. I followed along with the Hello Android tutorial, and it wasn't long before the string "Hello Android" was staring back at me from the emulator.

Create a User Interface

The first thing I needed was a UI, and building fat client user interfaces is foreign territory for me. Oh, what's that? Android uses XML to define screen layouts? Maybe this won't be so bad after all. Remember, I did say I am a Java developer by day.

Make no mistakes, my UI is ugly. Did I mention that I don't write much UI code at my day job?

Find the System Log

I spotted the commons-httpclient code in the API, and decided to code up a client in an OnClickListener for each of my buttons.

After deploying the code and watching the application crash, I was left scratching my head. Where can I see the logs? Fortunately the command adb logcat is the Android equivalent of tail -f, and I was able to pinpoint the problem with my HTTP client code.

Discover Permissions

The exception I found was a little strange:

E/AndroidRuntime( 6490): Caused by: java.net.SocketException: unknown error E/AndroidRuntime( 6490): at org.apache.harmony.luni.platform.OSNetworkSystem.createSocketImpl(Native Method) E/AndroidRuntime( 6490): at org.apache.harmony.luni.platform.OSNetworkSystem.createSocket(OSNetworkSystem.java:79) E/AndroidRuntime( 6490): at org.apache.harmony.luni.net.PlainSocketImpl2.create(PlainSocketImpl2.java:59) E/AndroidRuntime( 6490): at java.net.Socket.checkClosedAndCreate(Socket.java:763) E/AndroidRuntime( 6490): at java.net.Socket.connect(Socket.java:910) E/AndroidRuntime( 6490): at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117) E/AndroidRuntime( 6490): at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:129) E/AndroidRuntime( 6490): at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:164) E/AndroidRuntime( 6490): at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:119) E/AndroidRuntime( 6490): at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:348) E/AndroidRuntime( 6490): at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555) E/AndroidRuntime( 6490): at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:509) E/AndroidRuntime( 6490): at us.achievewith.Rhymote.Rhymote$1.onClick(Rhymote.java:60) E/AndroidRuntime( 6490): ... 23 more

I was expecting something along the lines of Connection Refused or Unknown Host, but unknown error was a little vague to be of much use. It was already late, and I foolishly struggled with the error for a while before letting Google Search work its magic, and that's when I learned about permissions.

Android applications don't just get free reign over the system without first asking for permissions to do most anything that interacts with the system at large or other systems. The following line cleared up the exception once it was added to the AndroidManifest.xml file:

<uses-permission android:name="android.permission.INTERNET"/>

One more run through the emulator, and I was able to start and stop my music. At this point it was 3:00 AM, and I decided that putting the code on the phone could wait.

Posted in Android, Open Source | 1 comment

Custom Error Pages with TurboGears 2

Posted by Tim Freund Tue, 21 Oct 2008 03:35:00 GMT

Your TurboGears 2 application is not perfect. Just for a second, let's pretend like it is perfect. Even with all of its perfection, your application will need to deal with bad incoming links and malformed data. Is it ready? It is only a matter of time before your users receive a 404 or 500 response from your most wonderful application. Whether the source of trouble is a bug in the code or a bad incoming link, why leave users lost in the dark? Custom error pages can put them back on track when something goes wrong, and they are very easy to implement. We will create one in the following few paragraphs.

We will use Turtle Goals to demonstrate the techniques in this tutorial. It is open source and fairly simple. Please feel free to follow along in the Turtle Goals source, or work along in your own TurboGears 2 project.

Look in your controllers directory. See that file named error.py? That's the key to a custom error page. The Routes package does a bit of work behind the scenes to send any request that generates an error through the ErrorController, so by customizing the ErrorController, we can customize the resulting 404 and 500 pages. The default document method produces a standard Pylons error page. It looks nice, but it probably doesn't look right compared to the rest of your project.

TurboGears projects default to the Genshi template engine, and that is the engine used by Turtle Goals. Let's create a new template, error.html, and save it in the templates directory.

So now all we need to do is add the expose decorator to the document method, and return a dictionary of appropriate values. Done, with enough time to check Reddit before your next meeting, right? Well, almost. Look closely at the ErrorController definition, and you will see that it is not a standard TurboGears controller. It extends a Pylons controller class, WSGIController, and that causes it to behave differently from our other controllers. At least it should extend WSGIController according to a post on the mailing list. Apparently there is a bug in the quickstart template, and you you may need to chage the ErrorController definition to extend WSGIController. I was easy to convince: as soon as I made the suggested change, my error page started working. Back to the point: the expose decorator will do you no good inside of the WSGIController. It is up to us to render the template to a string and return that string. Fortunately TurboGears provides a method to do just that:

from tg import render
rendered_template = render.render_genshi("error.html", {})

Of course there are also methods like render.render_mako and render.render_jinja if you prefer those other template engines. Here's the full listing of our modified error.py:

There's one other small matter to deal with: configuration. There are two relevant configuration values in your application: debug and full_stack. To use your custom 404 error but still get the interactive debug page for 500 errors, try the following:

debug = true
full_stack = true

The interactive debug page is inappropriate for production environments. When your application is deployed into production, use these settings instead:

debug = false
full_stack = true

If you have both set to false, you will get generic error pages that end users will run from, screaming. All done, for real this time. And you still have time to check Reddit, but you may want to check out these links instead:

• Pylons Error Document Details
• Google's 404 widget
• Wanted: Your 404 Error Pages
• 404 Error Pages: Reloaded

What is your strategy for designing and implementing custom 404 and 500 error pages?

Posted in Python, TurboGears, Tutorials | no comments