Python Simplicity

This article was inspired by Brian Dorsey, who hosted a SeaPIG meeting last month. (SeaPIG is the Seattle Python Interest Group.) As I was perusing the bookshelves at his house, I saw that his books on the Simplicity Movement (see below) match his fanaticism for simplicity in programming. Brian is a big-shot database administrator. (At one of Paul Allen's companies, boo, hiss.) He knows more SQL than most people I know. You'd think that means he loves complexity. Don't all database administrators love complexity? (It keeps them employed, after all.) But Brian stunned me by revealing his enthusiasm for trying out all the little Python database modules that are uncomplex. He follows the 80/20 rule: sometimes 20% of the features solve the problem for 80% of the population. Brian has also been demonstrating several other simple modules over our last several Python meetings, so I'd like to share with you a few of those.

In most articles, the author walks through code he's worked with extensively. (Or pretends he has worked with extensively.) In this article, I'm discussing modules I have not used. The point is not to say these modules are the best thing since sliced bread (although some of them are nifty). The point is that these modules demonstrate simplicity, either in their code or in their use.

By in their code, I mean the module itself is short. Less code means less chance for things to go wrong, so more reliability. That's the KISS principle: "Keep it Simple, Stupid!" In their use refers to the user's perspective: it takes only a few lines to activate their features. Some modules are simple in one way or the other, while others are simple in both.

Doc XML-RPC Server

Doc XML-RPC Server has got to be one of the easiest ways ever to offer services on the Internet. It's one of those inventions that makes you bonk your head and think, "Why didn't somebody think of this sooner?" Say you've written your services as methods of a certain class. You want to offer these services on the Internet or on an intranet. It takes just six lines of code:

from DocXMLRPCServer import DocXMLRPCServer
server = DocXMLRPCServer(('', 8000), logRequests=0)
server.register_introspection_functions()
server.register_instance(SimpleShareServer())
server.serve_forever()

'SimpleShareServer' is a class we created. We start a server on port 8000, register an instance of our class, register some optional services that come with DocXMLRPCServer ("introspection functions"; e.g., 'help'), and away we go. Here's the services we're offering:

import time
class SimpleShareServer:
     def message(self, msg):
         """message('Print me!') => True 
         
         Log everything passed to this function"""
         print time.asctime(), msg
         return True
 
     def wait(self, seconds):
         """wait(5) => 5 
         
         Wait for a certain number of seconds before returning.
         Returns the same number passed in."""
         print time.asctime(), "Waiting %s seconds" % seconds
         time.sleep(seconds)
         print time.asctime(), "Finished waiting %s seconds" % seconds
         return seconds

Of course, a local routine can just instantiate the class and call the methods directly. To do the same remotely, you used to have to either write a custom protocol implementation yourself, or read a long reference books to configure an off-the-shelf server or library. But a remote user can access these services with just a couple lines of code:

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:8000')
s.message("Hello, simple world!")
result = s.wait(15)

After these lines have executed, 'result' is 15, and "Hello, simple world!" appears on the server's console (standard output). Note that the arguments and return value were passed seamlessly between client and server, just like invoking a local method. The server proxy object "stands in" for the remote instance. Note that the client is using a generic XML-RPC library; it doesn't have to use a library specific to DocXMLRPCServer.

RPC (Remote Procedure Call) has been around on Unix systems for decades. NFS uses it, for example. But plain RPC (so I'm told) cannot cross programming languages. If the server is Python, the client has to be Python too, or something that knows how to encode/decode Python argument types. XML-RPC removes the language restriction. The arguments are converted to language-neutral XML, and so is the return value. This has some limitations:

You can pass only Python standard types (including lists and dictionaries), but not custom instances.
You can't pass None unless you enable a standard but non-universal option
To pass a string containing ASCII control characters (0-31 decimal) or other characters not allowed in XML, you have to wrap it in a 'Binary' instance. (But you don't have to do this for '<', '>' and '&', which DocXMLRPCServer automatically escapes.)

In exchange for these limitations, Perl and other clients can access your server -- just like any web browser can access an HTTP server.

Speaking of HTTP, that's the niftiest part of DocXMKLRPCServer. If a client sends an HTTP POST request to the same port, the server recognizes it and translates it to the corresponding method call. This could be used to collect form submissions for a survey, for instance. If a client sends an HTTP GET request, the server responds with an HTML page documenting itself. You've heard of emacs, the Self-Documenting Editor? Here's the self-documenting arbitrary server. Your server class can define three extra methods to customize the documentation output:

set_server_title(STRING)         # For the <TITLE> tag.
set_server_name(STRING)          # For the <H2> header.
set_server_description(STRING)   # The documentation, in HTML format.

DocXMLRPCServer is built on top of SimpleXMLRPCServer, which provides everything except the HTTP ability.

Somebody might object, "But it's using XML, and XML is decidedly non-simple." This is true. XML is a horrible bastard beast that should never have seen the light of day. In theory, it's wonderful. In practice, most of the DTD's are are so unnecessarily complex and the namespaces so nitpickily detailed that it looks like something only a bureaucracy could have designed -- the union of all attributes lobbied by every single special-interest group. You have to trust that the expat parser or whatever it's using under the hood won't blow up someday. So DocXMLRPCServer isn't simple in the code it depends on. But it's simple to use. Did you see any XML above? I didn't. I'm all for using XML if you don't have to look at it. Like the way Elvis impersonation bands are fun to attend as long as you keep your back to the stage, so you can enjoy the music without having to look at the tacky 50s kitsch. But I digress....

The SimpleShareServer above is based on a server Brian demonstrated at a SeaPIG meeting, which he describes on a wiki page.

db_row

db_row is a short module to wrap a SQL result set. The DB API database modules (MySQLDb, several PosgreSQL modules, Oracle and others) return a query row as a tuple of column values. Let's fool it with our own tuple and see what it does.

tup = (1, 2, 3)     # E.g., "SELECT a, b, a+b FROM SomeTable WHERE id=456;"
R = IMetaRow(['a', 'b', 'sum'])
                    # Create a custom class that names the rows in order.
                    # IMetaRow is a "class factory": it creates a class.
r = R(tup)          # Instantiate our custom class.
print r[0], r['b'], r.fields.sum
                    # Prints "1 2 3".  Access values by subscript, key or
                    # attribute.  (The "I" in IMetaRow means case-insensitive.)
print r.keys()      # Look ma, dictionary methods!
print r.dict()      # Just give me a real dictionary, please.

To convert an entire multi-row result set to a list of such jobbies, use a list comprehension:

lis = [ R(row) for row in cursor.fetchall() ]
print lis[0]['a'], "+", lis[0]['b']
print lis[0].fields.sum

Wrap the list comprehension in a function, and you only have to see it once.

Why do I like this module? It's short. You don't have to wait for it to be incorporated into your favorite DB API module; it works with all of them already. It works with non-SQL and ad-hoc result sets too. It solves a common problem in a simple way. (It's not all that simple. It uses Python slots, for instance. But we'll ignore that and hope Python's obscure slots feature has had most of its early bugs ironed out.) It claims to use less memory than a list of dictionaries.

But db_row's simplicity does come at the cost of certain disadvantages. It has no knowledge of the database field names or data types. You can lie to IMetaRow() and rename the fields anything you want. That may be convenient in some situations, but in an application with more than a few tables, it can get out of hand pretty quickly. Confusing yourself (and future maintainers) with inconsistent field names is a decidedly unsimple idea. Or doubleplus ungood as Orwell would say.

(The example above was inspired by db_row's docstring.)

SQLite and pysqlite

SQLite is an entire SQL server encoded in a little C library. pysqlite is a Python wrapper (DB API compatible). Brian calls this combination, "80% of what you'll ever need a database for in a single 270K executable (or Python module)."

The "80% you need" is ACID-compliant transactions, basic data types (strings, numbers, BLOBs, DateTimes), auto-increment fields, NULLs, temporary tables ("CREATE TEMPORARY TABLE"), a command-line utility (à la mysql and psql), dumping a database to SQL statements (PosgreSQL compatible), and huge databases (2 terabytes). There's even support for concurrent access of the database file in multiple processes, which I was pretty amazed at. You can't store strings that contain null characters (0 decimal) though. The database schema is stored in a table called 'sqlite_master'. Security is done by file permissions.

Another quirk of SQLite is typelessness. You can put letters into a numeric field, whatever that means. Actually, it means that a database is meant to store data, not to impose its will on the data. The SQLite developers call the rigid type system in the SQL standard and in most implementations a misfeature. (See the Datatypes page in the SQLite documentation for the full justification.) Fields can be created with all the usual SQL type specifiers, but those are just hints to the user, not rules SQLite enforces. Actually, SQLite does honor the types to some degree: they influence the sort order and whether two values are identical. There is one exception to SQLite's permissivism: auto-increment fields ("INTEGER PRIMARY KEY") have to be integers.

Other modules

There are a few object-oriented wrappers for SQL access, including SQLObject and DBO.

ctypes is a way to call C libraries directly from Python, which is supposedly easier than SWIG.

Python built-in features

Python 2.2 introduced three features that took people a bit of time to get their heads around, but they turned out to be incredibly useful: iterators, generators and properies. Iterators let you have a for-loop without having to pregenerate the entire sequence of values and keep them all in memory simultaneously. Generators allow an easy way for a function to iterate: it dispenses with the "topmost for-loop", leaving you more horizontal screen space and less clutter. Properties allow you to define "smart" attributes: those that trigger an action when they're get or set. Properties are controversial to some purists, but they avoid the clumsiness of accessor methods (aka parenthesesitis, which is a serious disease among C/Java-phobics).

Python 2.3 continues the trend with more features that simplify your programs. Sets are like dictionaries without the values ("just the keys, please"). If you're using a dictionary only to weed out duplicates, why define "values" you're not going to use? There's a logging module and a simple DateTime object. But the thing I use most is enumerate():

>>> lis = ['vanilla', 'chocolate', 'strawberry']
>>> for i, element in enumerate(lis)::
...    print "Element %d is %s." % (i, element)
...
Element 0 is vanilla.
Element 1 is chocolate.
Element 2 is strawberry.

This is a long-requested feature that avoids the equivalent but clumsier:

>>> for i in range(len(lis)):
...     element = lis[i]
...     print "Element %d is %s." % (i, element)
...
Element 0 is vanilla.
Element 1 is chocolate.
Element 2 is strawberry.

The simplicity movement

"And now for something completely different..."

The simplicity movement, championed by authors like Amy Daczyczyn (author of The Tightwad Gazette, a paper zine), Joe Dominguez & Vicki Robin, Cecile Andrews, Elaine St James and others, is about deciding what you really want from life and which material posessions really matter to you. Keep the stuff you need or want (e.g., for a hobby), and get rid of the stuff that's not a priority so it's not a distraction. This may not seem like it has much to do with programming, but we'll see that it does. Here's a few gems in the theories:

There are two ways toward a higher standard of living: earn $100 more per month, or cut your expenses by $100 per month. Buth achieve exactly the same thing: $100 more in your pocket. Most people adopt the former strategy, but that means depending on somebody else: you have to convince them to give you the money. In contrast, cutting expenses is entirely under your control. Having both spouses working means more expenses for transportation, clothing, food, daycare and unwinding; are you sure your net income is really higher than it would be without that second job? What about the lost opportunity for the second spouse to pursue a hobby or be a full-time volunteer? I love my freedom in not having a car; it gives me enough money to travel a couple times a year. Sure it limits where I can live and work, but those are the places I want to be anyway.

Then there's the question of technology. The Amish may be a bit too luddite for most people's taste, but they have a good point: accept new technology carefully, and only when it's proven its worth. I love my cell phone, but my stereo looks like it came from 1987 (which it did).

This feeds right into environmental sustainability, and the theory of waste. Why pay for stuff you don't want (and nobody wants)? Did you buy the applesauce for the applesauce itself or for the aluminum can it came in? Did you buy it because it has an extra plastic seal at the top? Did you buy it because of the energy used and effluent spent to produce the can? I can't discuss all this properly here, but there's a book, Natural Capitalism (Lovins, entire text online at www.natcap.org), that's easily the most important book of the 21st century so far. It looks at the question of waste from the individual's, businessman's, and policymaker's perspective, and how the (US) accounting and tax system allows companies to externalize the cost of environmental cleanup, which falsely skews their profit/loss statements and stock prices. But it takes only a change in business model to begin eliminating waste, work with the environment rather than against it, and turn a greater profit at the same time. Good stuff, Maynard.

What does all this have to do with simplicity in programming? The principles are the same. Decide what you really want, and look for a tool that does that. Maybe SQL is the cat's meow, but do you really need all the features of MySQL or PosgreSQL? Maybe you do, but it's reassuring to have thought out exactly which features you need and why you need them. (Especially when Postgres segfaults and you're wondering, why did I choose this?) Or maybe SQL isn't the cat's meow, and an object database like ZODB, or something even lighter weight like DBM or pickle/shelve might do the job.

Brian's Marklarizing webproxy

To conclude this article, I have to mention Brian's funniest invention. Here's the wiki entry describing it:

Several of my friends have a running Marklar joke spawned from a South Park episode. (Example. For Marklar for the episode review click here and search for 'Marklar'.) Short version: There are aliens who use the word marklar for every noun - and no, it's not confusing. After a long weekend of Marklar overdose, I happened to run into a free word list which included parts of speech (Greg Ward's Moby). Something clicked in my brain and I decided I had to make a Marklarizing web proxy, so that the entire internet could be seen as a Marklar would see it. Eight hours and some pretty horrible code later (mostly on the proxy & HTML parsing side, but the libraries I used and the program I wrote are only a couple pages each), I had something that mostly worked. Anyway, I demo'd it at the meeting, and this was our favorite page. It's an article from a newspaper.
Two arrested for running marklar-end marklar marklar 
By Marklar Ko 
Marklar Marklars staff marklar 
         
 
Marklar County marklar's detectives have broken up a large marklar marklar  
in the Marklar marklar, and they said they've recovered a "black book" with  
the names of hundreds of marklars, including men who work for marklar marklars  
headquartered in Marklar. 
 
The two marklars of the marklar, a 49-marklar-old woman who lives in Marklar,  
and her 31-marklar-old marklar, who lives in Marklar, were arrested. They have  
not been charged. 
 
The two marklar used the Internet to advertise a marklar-end escort company  
called the "Marklar of Marklar." The Web site had pictures of available  
companions, a calendar of when they were available and marklars. Marklars  
could be made online. 
 
Some of the escorts were brought in from out of state "Las Marklars, New  
Marklar and Los Marklars" to work marklar, according to the Marklar's Marklar. 
 
Marklars were carefully screened, said marklar's Sgt. Marklar Marklar. 
 
For example, potential marklars had to leave a work number, and someone inside  
the marklar marklar would marklar the number, marklar it was to confirm a  
dental marklar, he said. 
 
This was done to make sure the marklar wasn't a police officer, Marklar said. 
 
The Marklar's Marklar marklar not release the names of the marklars the marklars  
worked for, to avoid tainting the companies, Marklar said. 
 
The men in the book, however, marklar likely be contacted soon, detectives said.  
They could face marklar charges of patronizing a prostitute. 
 
In the 1990s, the two marklar were involved in another marklar marklar in the  
marklar called "Affluent Marklars." 
 
An marklar tipped off the Marklar's Marklar about the Marklar of Marklar.

Mike is the Editor-in-Chief of Linux Gazette. You can read what he has to say on the Back Page of many issues. He has been a Linux enthusiast since 1991 and a Debian user since 1995. He was SSC's web technical coordinator 1999-2003, which means he got to write a lot of Python scripts. Now he's involved in three free software projects for Python (Cheetah, Webware and YAML), writes unittests and programs for a Webware e-commerce site, and edits LG from his home. Non-computer interests include wrestling, ska and oi! and ambient music, and the international language Esperanto. He's been known to listen to Dvorak, Schubert, Mendelssohn, and Khachaturian too.

Published in Issue 98 of Linux Gazette, January 2004

<-- prev | next -->