Posted by Tim Freund
Tue, 21 Oct 2008 03:35:00 GMT
Your TurboGears 2 application is not perfect. Just for a second,
let's pretend like it is perfect. Even with all of its perfection,
your application will need to deal with bad incoming links and
malformed data. Is it ready? It is only a matter of time before your
users receive a 404 or 500 response from your most wonderful
application. Whether the source of trouble is a bug in the code or a
bad incoming link, why leave users lost in the dark? Custom error
pages can put them back on track when something goes wrong, and they
are very easy to implement. We will create one in the following few
paragraphs.
We will use Turtle Goals to demonstrate the techniques in this
tutorial. It is open source and fairly simple. Please feel free to
follow along in the Turtle Goals source, or work along in your
own TurboGears 2 project.
Look in your controllers directory. See that file named error.py?
That's the key to a custom error page. The Routes package does a
bit of work behind the scenes to send any request that generates an
error through the ErrorController, so by customizing the
ErrorController, we can customize the resulting 404 and 500 pages.
The default document method produces a standard Pylons error page.
It looks nice, but it probably doesn't look right compared to the rest
of your project.
TurboGears projects default to the Genshi template engine, and
that is the engine used by Turtle Goals. Let's create a new template,
error.html, and save it in the templates directory.
So now all we need to do is add the expose decorator to the
document method, and return a dictionary of appropriate values.
Done, with enough time to check Reddit before your next meeting,
right? Well, almost. Look closely at the ErrorController
definition, and you will see that it is not a standard TurboGears
controller. It extends a Pylons controller class, WSGIController,
and that causes it to behave differently from our other controllers.
At least it should extend WSGIController according to a post on the
mailing list. Apparently there is a bug in the quickstart
template, and you you may need to chage the ErrorController definition
to extend WSGIController. I was easy to convince: as soon as I made
the suggested change, my error page started working. Back to the
point: the expose decorator will do you no good inside of the
WSGIController. It is up to us to render the template to a string and
return that string. Fortunately TurboGears provides a method to do
just that:
from tg import render
rendered_template = render.render_genshi("error.html", {})
Of course there are also methods like render.render_mako and
render.render_jinja if you prefer those other template engines.
Here's the full listing of our modified error.py:
There's one other small matter to deal with: configuration. There are
two relevant configuration values in your application: debug and
full_stack. To use your custom 404 error but still get the
interactive debug page for 500 errors, try the following:
debug = true
full_stack = true
The interactive debug page is inappropriate for production
environments. When your application is deployed into production, use
these settings instead:
debug = false
full_stack = true
If you have both set to false, you will get generic error pages that
end users will run from, screaming. All done, for real this time. And
you still have time to check Reddit, but you may want to check out
these links instead:
What is your strategy for designing and implementing custom 404 and 500
error pages?
Posted in Python, TurboGears, Tutorials
Posted by Tim Freund
Wed, 24 Sep 2008 06:06:00 GMT
My mom always reminded me to cover my nose when I sneezed. Now I take
it a step further, and I cover my nose when I test.
Nose and the nosetest command are used to run unit tests for
Pylons and TurboGears 2 applications, as well as a multitude of other
Python applications and libraries. Although nose is great at running
tests and reporting back issues, it doesn't natively show developers
what isn't being tested. For that, we need a code coverage tool.
Just because nose doesn't handle code coverage reports natively
doesn't mean this will be a difficult task. Ned Batchelder's coverage package provides exactly those reports, and nose ships
with a plugin to enable it. To install coverage and invoke the
reports, you could do something like this:
$ easy_install coverage
$ nosetests --with-coverage
Name Stmts Exec Cover Missing
--------------------------------------------------------------------------
_strptime 228 149 65% 23, 80, 84-89, 155, 169-170, 175, 189, 237, 280-294, 303-304, 306, 314-323, 329, 332, 353-360, 366, 368, 374-388, 393-427, 431-432, 443-446
encodings.ascii 19 0 0% 9-42
ez_setup 103 11 10% 53-62, 80-104, 117-151, 156-190, 197-222, 226-229
fixture 10 9 90% 38
fixture.base 124 33 26% 10-19, 25-28, 48-49, 56, 60, 64, 98, 103-104, 122-217
fixture.dataset 301 225 74% 41, 51-52, 55-59, 65, 76-77, 80, 98-110, 141, 144, 225-229, 232-239, 243-248, 277, 285-289, 323-325, 447-450, 456-457, 461, 468, 479, 483, 529-530, 560-564, 571-574, 580, 631, 720-722, 725-737, 740-744, 752-753
... lots of lines cut to keep you from going blind ...
zope.interface.ro 22 22 100%
zope.interface.verify 45 29 64% 46, 51, 56-61, 66, 70, 75-79, 88, 93, 104, 107, 109, 111
--------------------------------------------------------------------------
TOTAL 17257 7646 44%
----------------------------------------------------------------------
Good Gravy, man! That's a big coverage report!?
As cool as it is to know that we can get coverage reports for the
entire Pylons/TG2 stack, how about we focus for just a minute on only
our project. That's the problem with kids today, no focus.
$ nosetests --with-coverage --cover-package=YOUR_PACKAGE_NAME
OK, that's better. Now the report only shows the coverage for code
inside of our package. But, seriously, do we really need to type those
arguments every time?
Of course not. Sitting there in the root of your Pylons or TurboGears
2 application is a file named setup.cfg. If I were a betting man, I'd
say you've never opened it. Ever. Let's knock the dust off and take
a look at it in any decent text editor. We're looking for a section
named [nosetests], and since you've never changed the file, it is
probably at line 8 and looks just like this:
[nosetests]
with-pylons=test.ini
We can add any additional options for nose in this section. Now is
your chance to spring into action. Add the following lines to
with-coverage=true
cover-package=YOUR_PACKAGE_NAME
Save setup.cfg and run your tests. You will see a code coverage
report at the end of the test run, and the last two columns will be
the most interesting. They show the percentage of the code that was
covered and the lines of code that were not covered, respectively.
Depending on the outcome of your coverage report, you may be feeling
rather smug right now. Stop it, we're not Rails developers, and there
is still work to do.
On the other hand, if your code coverage report leaves you feeling a
little ashamed of just how much code is uncovered, don't despair.
Just knowing what code coverage is and caring about the results of a
coverage report already puts you into a minority of all programmers.
Pick a block of untested code and write a test. All of the sudden,
your numbers are better and you'll start to leave those
fly-by-the-seat-of-their-pants programmers in the dust.
Remember that this setup.cfg trick will work with any python
application or library that is testable with nose. We happen to be
focusing on Pylons and TurboGears because that's what I've been busy
using lately. How do you use nose to help you write better code?
Posted in Python, TurboGears, Tutorials
Posted by Tim Freund
Mon, 10 Dec 2007 06:56:00 GMT
Earlier today I posted a note in the Amazon Developer Forums about FPyS, my attempt at a Python library for the Amazon Flexible Payment Service. It's rough, very rough, but it is a decent start at a fully featured FPS library. The library provides enough functionality to run FPeS, but not much beyond that.
For the curious, the demo application is a TurboGears application that allows people to select and purchase full sized images from a thumbnail gallery. The application is configured to process payments through the Amazon FPS Sandbox, so any completed transactions are drawn from imaginary credit cards. With that in mind, I'd like to welcome anybody to attempt a sale. The more the merrier, as it will cost us nothing and help to find bugs in the library.
Posted in Open Source, Python, TurboGears
Posted by Tim Freund
Mon, 06 Aug 2007 00:42:00 GMT
I learned about the importance of schema migrations the hard way. At my previous job, I helped a team upgrade a Java web application. The upgrade involved schema changes, and I had the forethought to script the upgrade and thoroughly test it on the development database. Even with that preparation, the night of the upgrade would teach me two important questions that new developers should always ask of their team. Does the development database schema match the production database schema? And do you know how to restore the database from backups should anything go wrong? The answers to both questions on that night were no and oh no. Sometime after midnight things started working.
A better way existed. I first learned of schema migrations a few months earlier when exposed to ActiveRecord::Migrations. After using them on several projects, I was itching to have the same capability in my Python and Java projects. The Pythonic answer came in the form of Migrate, a schema migration library for SQLAlchemy, and direct support for TurboGears was added with TGMigrate. Having migrate integrated with my projects greatly reduces my blood pressure on deployment days. Of course, I still make sure that my database backups are working.
Those interested in integrating Migrate with a TurboGears project might enjoy the screencast I just completed on the topic. If you use SQLAlchemy but avoid Migrate, I would be interested to hear what is holding you back.
Schema refactoring and migration was one of three topics at the last DotNext Kansas City Tech Coffee meeting. Notes on the schema migration talk were posted to Squidoo.
My apologies to those with small screens. I will record my next screencast in a smaller window. Your comments on today's screencast are appreciated.
Posted in Code, Python, TurboGears, Tutorials
Posted by Tim Freund
Thu, 14 Jun 2007 03:15:00 GMT
Identity
defines who we are. Identity is made up of all the little
distinguishing traits that differentiate one person from another.
We've all changed our identity throughout our lives. We change from
student to graduate, single to married, dogless to dogged, and more,
but that's not what we're talking about today. Identity is the
authentication and authorization framework for TurboGears, and it is
easy to extend.
At the core of the Identity framework is an IdentityProvider.
The Identity Provider interfaces with an authentication and
authorization repository to determine two things: are you who you
say you are, and do you belong where you are trying to go. The
framework comes with two providers, one each for SQLObject and
SqlAlchemy.
We will customize an IdentityProvider to authenticate
against an IMAP server in the few steps that follow. This would be
helpful for writing a web mail application, and the concept can be
applied to other authentication mechanisms as well, including LDAP,
Radius, and others.
Action Plan
- Quickstart a project
- Create an identity provider
- Define an entry point for the identity provider
- Configure the application for the new provider
- Finish the identity provider
- Test
- Relax
The code for this tutorial is available
from subversion
or as
a tar.gz
file. It is released under the MIT license. No TurboGears
installation? Install it like
so.
Step 1: Quickstart a Project
If you don't have an existing TurboGears project to experiment
with, now would be a great time to start one.
tim@iris ~/src $ tg-admin quickstart -s -i iddemo
...
tim@iris ~/src/ $ cd iddemo
tim@iris ~/src/iddemo $ tg-admin sql create
Note the -s and -i flags. This is a
project with support for SqlAlchemy and Identity.
Step 2: Create an Identity Provider
Any object can be an identity provider as long as it supplies the
following methods: validate_identity, validate_password,
load_identity, anonymous_identity, authenticated_identity, but
it isn't always necessary to write one from scratch. Extending an
existing provider often gets an application authenticating as required
without much trouble. We will extend
the SqlAlchemyIdentityProvider in this example to
authenticate against an IMAP server.
iddemo/identity.py
from turbogears.identity.saprovider import SqlAlchemyIdentityProvider
class ImapSqlAlchemyIdentityProvider(SqlAlchemyIdentityProvider):
pass
Step 3: Define an Entry Point
The Identity Framework uses an entry point
named turbogears.identity.provider to decide what class
to use when authenticating users. We are about to define a new option
for this entry point, but further reading on the subject of entry
points is recommended. Scroll to the bottom of this entry for a couple
of relevant links. It's OK, we have the time.
This entry point step won't be necessary in the
future, thanks to
this patch, but entry points are a powerful tool and worth
learning, regardless.
setup.py
setup(
name="iddemo",
... (more setup stuff) ...
entry_points="""
[turbogears.identity.provider]
imapsa = iddemo.identity:ImapSqlAlchemyIdentityProvider
""",
... (more setup stuff) ...
)
To let the setuptools system know about this new identity provider, run the following:
tim@iris ~/src/iddemo $ python setup.py develop
Step 4: Configure the Application
With our imapsa option defined for
the turbogears.identity.provider entry point, we can now
configure the application to call the new provider. There is a value
named identity.provider in app.cfg. We will replace the
existing value with imapsa. While app.cfg open is,
add the other three lines in the following example. They will be
explained in the next step.
iddemo/config/app.cfg
...
identity.provider='imapsa'
identity.imapprovider.imap_authoritative=True
identity.imapprovider.server="localhost"
identity.imapprovider.port=143
...
The application is now ready to run with the new identity provider.
Restart the application if it is currently running so that the
configuration change will take effect.
Step 5: Finish the Identity Provider
Now let's dig in and implement the new authentication behavior.
iddemo/identity.py
...
class ImapSqlAlchemyIdentityProvider(SqlAlchemyIdentityProvider):
def __init__(self):
SqlAlchemyIdentityProvider.__init__(self)
# These three lines get the configuration parameters we set in app.cfg
self.imap_authoritative = get("identity.imapprovider.imap_authoritative", False)
self.server = get("identity.imapprovider.server", "localhost")
self.port = get("identity.imapprovider.port", 143)
# These four lines make the user and visit classes available for
# later use
user_class_path = get("identity.saprovider.model.user", None)
self.user_class = load_class(user_class_path)
visit_class_path = get("identity.saprovider.model.visit", None)
self.visit_class = load_class(visit_class_path)
def validate_identity(self, user_name, password, visit_key):
if self.validate_password(None, user_name, password):
user = session.query(self.user_class).get_by(user_name=user_name)
if not user:
if self.imap_authoritative:
user = self.user_class()
user.user_name = user_name
user.save()
session.flush()
else:
return None
link = session.query(self.visit_class).get_by(visit_key=visit_key)
if not link:
link = self.visit_class(visit_key=visit_key, user_id=user.user_id)
session.save(link)
else:
link.user_id = user.user_id
session.flush()
return SqlAlchemyIdentity(visit_key, user)
return None
def validate_password(self, user, user_name, password):
rc = False
try:
imapcon = imaplib.IMAP4(self.server, self.port)
except:
log.error("Could not establish connection to server at %s:%d" % (self.server, self.port))
return rc
try:
if imapcon.login(user_name, password)[0] == 'OK':
rc = True
except:
# Probably threw an error for invalid username/password
log.info("Passwords don't match for user: %s", user_name)
imapcon.shutdown()
return rc
Take a look at each method to figure out what they accomplish.
__init__ invokes the constructor of the
SqlAlchemyIdentityProvider and then collects the configuration
parameters required for authentication.
validate_identity first
invokes validate_password. If the password validate
succeeds, the user is selected from the database. If the user does
not exist in the database, the provider will create the user when
the imap_authoritative option is set. Should you not
require this capability, you can remove this method entirely and rely
upon the implementation in SqlAlchemyIdentityProvider.
Finally, this method links the user with the
current visit_key.
validate_password handles all IMAP access. It is the
perfect method to override if all you need to do is change the
identity authentication mechanism.
Step 6: Test
Open controllers.py and change the identity decorator to require an
authenticated user when accessing the index page.
iddemo/controllers.py
class Root(controllers.RootController):
@expose(template="iddemo.templates.welcome")
@identity.require(identity.not_anonymous())
def index(self):
import time
flash("Your application is now running")
return dict(now=time.ctime())
The moment of truth has arrived. Point a browser at the
application. You should see a login page, and a valid IMAP
username/password should provide access to the application.
Step 7: Relax
We're done here. Go tell your friends and family how cool you are.
Any feedback is appreciated. Leave a comment here, post a response
on your own blog, or send an email to tim -at- achievewith -dot-
us. You can also usually find me lurking in #turbogears on
irc.freenode.net as timphnode.
For more information, follow these links:
Posted in Code, Python, TurboGears, Tutorials
Posted by Tim Freund
Thu, 22 Feb 2007 02:10:00 GMT
HTML is king of Web 2.0, and JavaScript is HTML's chief advisor and
errand boy. But the printed page still matters, especially when
working with Web 0.2 requirements and processes, and PDF documents
provide crisp and consistent print output across platforms.
Follow along with the rest of this document to see how to produce PDF
documents with the help of TurboGears, Cheetah, and ReportLab. We
will generate form letters in response to a job opening, and in Part
II we will print mailing labels for our letters.
The code for this tutorial is available from subversion or as a tar.gz file. It is released under the MIT license.
The Tools
-
ReportLab is a Python library that creates professional PDF documents
with minimal developer effort. It can be used in any Python
application. The <br/> tag doesn't seem to work with the
stable release, so try the snapshot instead.
-
Cheetah is a flexible template engine that can generate arbitrary
documents. It is supported by most Python web frameworks. TurboGears
support is provided by the TurboCheetah package.
-
TurboGears is a web framework that aims to make creating web apps
faster, easier, and more fun. It brings together a wide range of
powerful Python libraries into a coherent whole.
The Exercise
We will generate acceptance and rejection letters on behalf of Non
Compos Mentis Research, a mysterious scientific research organization
located somewhere underneath the the Midwestern United States. They
were seeking applicants for an "Evil Genius, Sr." position after their
long time employee, Dr. Phinius Fraggenblam, had an accident that lead
to early "termination." All of the interviews are complete, the Board
has made its decisions, and we have all of the applicant information
we need in a model class named pdfdemo.model.Applicant.
This demonstration breaks into three logical steps. We need to
-
Create a static PDF just to prove that we can
-
Insert data into the PDF
-
Serve the PDF to a web browser
Step 1: Generate a PDF from a Text File Using ReportLab
ReportLab allows programmers to specify every detail as they write
code to generate PDF documents. For the [pragmatic | impatient |
lazy] developer, ReportLab also provides an abstraction layer named
PLATYPUS, short for "Page Layout And Typography Using Scripts."
PLATYPUS provides a collection of "flowable" elements including
paragraphs and page breaks that can be inserted into templated
documents.
The Paragraph is a powerful Flowable
element. In addition to accepting a style definition from
reportlab.lib.styles, the Paragraph element
also understands a small set of intra-paragraph XML markup.
Two brief examples follow:
pdfdemo.letters.reject_plain
<para>Dear Sir/Madam:</para>
<para>After careful consideration, we regret to inform you that your application
was <font face="times" color="red">rejected</font>.</para>
<para>Please try again next year. <b>Or else</b>.</para>
<para>Best regards,</para>
<para>The Board<br/>
Non Compos Mentis Research</para>
pdfdemo.letters.accept_plain
<para>Dear Sir/Madam:</para>
<para>Your application rose to the top of the millions and millions that we
received. We are pleased to extend to you a position on our dynamic team.</para>
<para>We will come for you in the night, unannounced. Be prepared.</para>
<para><i>Congratulations!</i></para>
<para>The Board<br/>
Non Compos Mentis Research</para>
Given the above, we can generate a static PDF with two pages: one
acceptance letter, one rejection letter.
pdfdemo.printjobs.generate_plain_document()
def generate_plain_document():
"""Generates a PDF with two pages, an accept and reject letter.
Output is hardcoded to output.pdf in the current working directory
"""
output_file = open("output.pdf", "w")
document = SimpleDocTemplate(output_file)
components = []
sheets = styles.getSampleStyleSheet()
# increase font size to 14 points
sheets['Normal'].fontSize = 14
# increase line height to 16 points
sheets['Normal'].leading = 16
# increase space after a paragraph to 16 points.
sheets['Normal'].spaceAfter = 16
paragraphStyle = sheets['Normal']
for letter in (letters.accept_plain, letters.reject_plain):
# There are multiple paragraph elements (para) defined in our
# documents, and each Paragraph object can only accept one.
# Fortunately ElementTree is a requirement component of TurboGears,
# and it makes short work of splitting up the document by para element.
doc = ElementTree.XML(letter)
for para in doc.findall('para'):
components.append(Paragraph(ElementTree.tostring(para), paragraphStyle, None))
components.append(PageBreak())
document.build(components)
output_file.close()
Step 2: Use a Cheetah Template to Insert Custom Data into the PDF
Now there is the small matter of inserting custom
bits of data into the text before ReportLab does the voodoo that it does
so well. Note that ReportLab is generating the PDF from a text string.
Also note that Cheetah can generate arbitrary text strings from
templates and data with grace and aplomb. Since TurboGears is our
target platform, TurboCheetah is installed, and that makes things
even easier. Data available to a Cheetah template is passed as a
dictionary into the TurboCheetah render method. Variables are
rendered in Cheetah templates like so: ${my_variable.some_property}.
Below are examples of a Cheetah template, and a function invoking the
TurboCheetah.render method.
pdfdemo.letters.accept.tmpl
<doc>
<para>${applicant.prefix} ${applicant.full_name()}<br/>
${applicant.address1}<br/>
#if $applicant.address2 != None
${applicant.address2}<br/>
#end if
${applicant.city}, ${applicant.state} ${applicant.postal}</para>
<para>Dear ${applicant.prefix} ${applicant.full_name()}:</para>
<para>Your application rose to the top of the millions and millions that we
received. We are pleased to extend to you a position on our dynamic team.</para>
<para>We will come for you in the night, unannounced. Be prepared.</para>
<para><i>Congratulations!</i></para>
<para>The Board<br/>
Non Compos Mentis Research</para>
</doc>
pdfdemo.printjobs.generate_lettersv1()
def generate_lettersv1(applicants):
tcheetah = TurboCheetah()
pages = []
# render the templates
for person in applicants:
letter_data = {"applicant": person}
# convert status to a string because render doesn't
# appreciate unicode template names
template_name = "pdfdemo.letters.%s" % person.status.__str__().lower()
rendered = tcheetah.render(letter_data, template=template_name)
pages.append(rendered)
paragraphStyle = get_stylesheets()['Normal']
# assemble the PDF from the rendered templates
components = []
for page in pages:
page = ElementTree.XML(page)
for para in page.findall('para'):
components.append(Paragraph(ElementTree.tostring(para),
paragraphStyle,
None))
components.append(PageBreak())
# generate the PDF
output_file = open("output.pdf", "w")
document = SimpleDocTemplate(output_file)
document.build(components)
output_file.close()
Step 3: Render the PDF to a Browswer with TurboGears
The hard work is done, but the code still generates a file on the file
system, and we need to view the file in a browser. ReportLab can work
with any object that acts like a file, including a
StringIO Buffer. A new function,
generate_lettersv2, accepts a file like object rather
than opening a file itself. The controller then calls
generate_lettersv2 with a StringIO object as
the "file". This is a breeze.
pdfdemo.controllers.Root.letters()
@expose(content_type="application/pdf")
def letters(self, **kw):
letters_file = StringIO()
generate_lettersv2(Applicant.select(), letters_file)
pdf = letters_file.getvalue()
letters_file.close()
return(pdf)
Some really old browsers require the URL of a PDF document to end with
.pdf. This is accomplished by implementing a default
method for the controller.
pdfdemo.controllers.Root.default()
@expose()
def default(self, *path, **kw):
method_name = path[0]
if method_name.find('.') != -1:
method_name = method_name.split('.')[0]
method = getattr(self, method_name, None)
if method != None:
return(method(**kw))
raise cherrypy.NotFound
Get Up and Stretch. Grab a Soda.
This article has detailed the steps required to produce templated PDF
documents using TurboGears, Cheetah, and ReportLab. In the next
installment we will generate mailing labels so that the letters we
generated today are not lost due to poor handwriting skills. The Non
Compos Mentis organization does not tolerate mistakes.
Any feedback is appreciated, both on the technical content and the
silly back story. Leave a comment here, post a response on your own blog,
or send an email to tim -at- achievewith -dot- us. You can also
usually find me lurking in #turbogears on irc.freenode.net as
timphnode.
For more information, follow these links:
Posted in Code, Python, TurboGears, Tutorials
