Lucumr Cogitations

How not to do XML

2008-02-18T10:06:17Z

Imagine for the moment there was a PHP blog software that has the ability to dump the blog posts into some sort of extended RSS 2 feed and import from there later and probably from a different installation. That's nice, XML is a flexible format and RSS allows extensions via namespaces. Even better, there are XML parsers for all major programming languages and from python working with XML is especially cool because of lxml and element tree. But there is a problem with that... ...that XML, is not XML. It's called WordPress eXtended RSS (WXR) but it's not XML? And why in god's name did nobody notice so far? I mean, WordPress must have an importer for that. Why it's not XML? It has XML syntax, XML namespace declarations but what doesn't it have? A doctype. What's the problem? It's referencing HTML entities! So step one for parsing: inject an inline DTD that defines those entities. Great fun isn't it? Then it parses. I was happy and finished my work. That XML doesn't have HTML entities is something PHP developers probably don't know and their parser isn't resolving any entities during the parsing process. Or worse, their XML parser expands HTML entites. But it's worse! I loaded another dump that happened to have some broken HTML in comments (could happen, does happen, thanks broken trackback support). What happens next? THE XML DOESN'T PARSE ANY MORE! Why? Because comments are neither escaped nor marked as CDATA. I wonder why, especially because it's so much easier to handle embedded HTML/XHTML for dumping as cdata and not XML, especially if you are working with PHP. But WordPress was able to import that.... so I looked at their parser.... WORDPRESS PARSES THAT WXR FILE USING REGULAR EXPRESSIONS!!! Argharhgarhghargh. That's not XML what you are doing there, that's nothing. WordPress can't even parse it's own file if you bind the WordPress exporter namespace to a different prefix! WordPress can't handle it's own file if you replace their CDATA foobar against properly escaped stuff. Dammit! I can't even write a proper exporter using XML tools because what my XML tools generate is not compatible to WordPress. And what tops it all? Reading that in the #wordpress channel: ARGHARGHARGHARGH. and then the webpage says:

WordPress is a state-of-the-art semantic personal publishing platform with a focus on aesthetics, web standards, and usability.

Without further comments... I lost my faith into standards that moment. Wait a second, I lost it earlier. Still sad.

TextPress Development Blog Online

2008-02-16T14:38:34Z

TextPress has now a dedicated development blog. That means two things. For one development on TextPress is active again and we are looking for help on converters, missing extensions (especially ATOM publishing support) and more. That development blog is of course running on TextPress ;-)

GHRML — Haml for Genshi

2008-02-15T22:34:03Z

May I introduce: GHRML, the Genshi Human readable markup language. name not final of course, mika would kill me ^^. What's GHRML? First of all it's work in progress. But beside that it's a pretty cool clone of Haml for Python. But no, it's not another templating language, just a different representation for genshi markup templates. This way we can reuse all the genshi features like the directives or serializers. This gives you the full capabilities of the genshi templating language in a nice alternative syntax. So how does it look like? Here a small example: Most of the syntax is directly stolen from Haml. The only real change is that we don't need explicitly marked self closing elements because genshi knows that already in the serializer. Additionally you can use brackets for genshi directives like the for-loop above. And the parser works with variable indentation, if you want an indention level of four, just do so, the parser will recognize that. Sourcecode in the sandbox for those of you who want to try it, some updates later ;-)

Werkzeug 0.2 Released!

2008-02-13T23:01:12Z

Wohoo. Werkzeug 0.2 is out now. Werkzeug started as simple collection of various utilities for WSGI applications and has become one of the most advanced WSGI utility modules. It includes a powerful debugger, full featured request and response objects, HTTP utilities to handle entity tags, cache control headers, HTTP dates, cookie handling, file uploads, a powerful URL routing system and a bunch of community contributed addon modules. So, what's new in 0.2? Countless things and too many for this small list, but here the most important ones:

The path converter limitation is gone. rejoice!
In the contrib package there is now a secure cookie (basically a hashed a client side session storage)
Exceptions can now return response objects so that you can add headers etc.
You can now convert a response object to a different type of response objects on the fly (for example if you have your own response object subclass with special features but the response object returned by a function is a simple BaseResponse)
There are a bunch of extra features for response and request objects now (available as mixin classes) for HTTP header parsing and dumping
All the routing exceptions are now HTTPExceptions which simplifies dispatching a lot
werkzeug.script has a much simpler way of specifying boolean parameters
lazy_property is now called cached_property, update your code!
many cool small helper functions that deal with python modules and packages. There is find_modules which can return a generator for all the modules below a package and import_string which allows you to simply import objects from a string. No more __import__ hackery needed.
the usage of the map adapter is much easier now too and a lot more rest compliant. See the new documentation
dozens of small fixes and additions!

There is also a new website and documentation and the tutorial was translated to German. For 0.3 we hopefully have some more translations for the tutorial and a better documentation for the contrib modules which are currently just documented in docstrings. Grab it from the cheeseshop while it's hot.

Stuff you should buy

2008-02-13T17:47:03Z

The "Falco Symphonic" DVD. I know Falco is hyped currently here in Austria because that month ten years back he died (damn, I'm getting old) but the DVD is really worth it. For those of you who don't know the story behind that DVD here a small summary: 1994 Falco gave a concert with an orchestra. However the only record of that concert was a already (and bad mixed) tape from three cameras and a DAT tape directly from the mixer unit with the same quality. Because it was the only concert of that kind from Falco the band decided to record the instruments again and mix the original voice back into the new record. And the result is great. They even managed to get some records by fans so there are multiple angles for some scenes now. Great piece of music history and a brilliant production. Small hint: the CD features a different recording thatn the DVD, the DVD has eh live recording of the voice whereas the CD has a different arrangement of the instruments and Falco's voice from the studio recordings. Yes, it's not a real live version because the instruments are recorded in the studio afterwards but the impression is great. And in the studio the same musicians play and it's the only chance for us to get such a great recording in that quality so I think it's perfectly okay to listen to that ;-)

How To Kill a Game

2008-02-09T23:49:48Z

About four years ago I stumbled about a promising game called "Söldner --- Secret Wars" in a magazine. It was an online tactical shooter and featured a fully destroyable environment, more than one hundred weapons I think, many different vehicles, aircrafts and much more. It was hyped back then in the German gaming community, probably because it was developed by a German studio called "Wings". Long before the game was released the developers updated their online dairy about recent changes in the code, about what the game will feature and a lot more. Unfortunately they decided to release their game by Jowood, an Austrian publisher. Why was that bad? Back then Jowood once again had financial problems and they tried to solve it by convincing the sharedholders that their next game will be a major success. They set an totally unachievable release date and the developer team at wings had to live with that. It went that far that they had to design the box are themselves even though the game was still buggy. The game was released and was still buggy. I think GameStar even omitted the rating because they wanted to wait for the first patch. However the game sold pretty well the first two weeks despite the bad quality of the release version and the bad reviews. It even outsold the greatest competitor at that time (Joint Operations). I was one of the early adopters and I loved to play it. Owning a bad ISDN connection at that time it was terrible lagging, especially because their netcode was a catastrophe. Still, it was great fun and the destructible environment was great. Not only that, the game was somewhat revolutionary. It was the first online game I played with an (albeit bad configured) physics engine, it used python as scripting language, it had one big map that was generated from satellite images with detailed areas where the fights took place. You could buy yourself a jet and leave the map and visit all off Siberia. Not like Battlefield Vietnam where you hit the borders of the map with a jet after roughly 15 seconds. Dammit, it would still be fun and I bet it would have been bug-less by now. The community that appeared around that game was incredible. A wiki appeared, people tried to hack around on the code to get extra features into the game, unofficial mods appeared. There was even a project that wanted to create a mod for a medieval setting. Hell, it was great back then. But instead of fixing the bugs Jowood forced the developers to start an addon project... Shortly after the addon was finished the company developing that game was liquidated. The developers lost their job and the community continued developing the closed code. The group around "project zero" continued maintaining the game until that group broke apart and the gEasy team took over the work. The gEasy team was basically just the project zero team without the founder. They did an tremendous job but Jowood once again destroyed everything. They refused to pay their bills, then the gEasy team took down the master server and Ivan Ertlov appeared on the scene. He managed to resolve the problems between Jowood and gEasy team to some extend that the master server was up and running again. Who's that Ivan Ertlov? Very good question indeed and I personally don't know the full story but from what I've read on the forums he is called Johann Ertl and owns a company that sells guerrilla marketing and similar services. And he worked for Jowood and apparently still does, at least he is listed as community manager in the official forums. He also brought up the topic "Open Sourcing" the code after it was clear that there will be no Söldner 2. A sequel was actually under consideration and gEasy started working on that till the day they took down the master server because there was no payment by Jowood. But since some time there was no feedback any more and an open source version is somewhat unlikely. Frankly I don't know if Jowood payed or didn't. What Jowood did was destroying the game by forcing an early release, forcing an addon when the code was still unfinished, fired the developers. Even worse: they are telling their shareholders that everything is working perfectly and that they are releasing dozens of new games. Hell, they released Spellforce 1 and 2 which both are awesome games (just happen to have the worst copy protection ever designed), the Gothic games (the 3rd part was buggy like Söldner but probably also because Jowood forced an early release). Jowood however claims that they payed and that gEasy was lying. Who is running the portal now? Apparently Ivan's company. So why am I blogging about all that? Mainly because I think that topic hasn't gained a lot of attraction. I stopped playing that game after Wings was liquidated. For one because I switched to ubuntu and on the other hand because my Söldner plugin (an in-game Winamp controller) disappeared when the forum was updated to reflect the new ownership. All the old topics where either deleted or made unreadable when Wings was closed down. What nobody really notices is that the game had a tremendous German community. Hell, some of them even took over the development! If Jowood would have noticed that earlier they could have made that game the freaking best online shooter available at that time. But because of their small horizon they just thought about their next quarter and decided to do what shareholders want, not what players want. But not only Jowood is to blame but the root of all evil in that case. Krawall supported Jowood in the beginning to host their initial infrastructure needed for the game. Unfortunately the server software needed Direct X to run properly which caused a lot of trouble. One the one hand it was hard to get multiple servers running on one machine do to the way the server was designed, on the other hand you needed a windows server to host games. And because the initial code base was that buggy all the magazines flamed to game. Despite the good sales figures it magazines never wrote about the game and forced an early dead which harmed the community. A release six month later, a linux server from the beginning, no addon and good press coverage would have avoided all the problems this game was facing in the past. And it would have saved Wings, the game they were working on beside Söldner, and in the end of course Jowood which would have had a lot less bad press. Especially the gamers hate Jowood for their buggy products now and I doubt that they will be able to continue to ignore all the user feedback and push alpha versions as release versions for much longer. I'm especially interested what the Wings guys are doing now. I know that former Wings community manager and sound designed Marc Olbertz is working at Blizzard but that's about it. It's sad what happened there and that Austrian's only game publisher caused all that. Funky fact: The communication with the master server worked via jabber, the physics engine was ODE and the scripting language was Python. All open source technologies I never saw in a commercial game before :-) Disclaimer: the information on this page should be accurate but it's hard to say for sure because there is few information about this topic actually available. If you are able to understand German you can find some information in this thread on the Söldner forums: Söldner ein Abenteuer mit ungewissem Ausgang

New Werkzeug Website / Docs

2008-02-06T18:27:47Z

Disclaimer: Yes I know the colors are too bright. Everybody who saw that webpage told me that, so I will change that soon. Anyways. Because there is an upcoming 0.2 release I deployed the documentation for the 0.2 release already because most of the users are probably already working on the hg tip (at least that's what the situation looks like in #pocoo). The new website and documentation. Additionally if everything works well there will be a Werkzeug presentation on the Grazer Linuxtage April the 19th. The Werkzeug 0.2 release will be Feb 14th hopefully, until then I have to fight with Jabber, don't ask :-)

Until TextPress is ready…

2008-02-05T19:24:40Z

mitsuhiko@hammett:~$ cd lucumr/cogitations/
mitsuhiko@hammett:~/lucumr/cogitations$ rm xmlrpc.php 
mitsuhiko@hammett:~/lucumr/cogitations$

BarCamp Senza Confini 2008, Day 1

2008-02-02T20:32:54Z

And unfortunately also the only day for me because I can't be there tomorrow. Which is especially stupid because I'm very interested in the talk about broadband 2.0 by Alexander List. Well. It was my first barcamp so I presented a topic: mercurial. Good: There was at least one person in the audience that considers switching and I hope I whetted appetite to the others too, Bad: the talk was badly organized IMO, hope I will do better the next time. But now to the positive aspects of the day. I think the main reason why people go to barcamps are not the talks, but the possibility to meet people with similar interests and share experiences and knowledge. I meet a couple people there I didn't know before or only from IRC/Jabber/whatever and he have a couple of great discussions, especially about the currently problematic political situation here in Austria. And despite what I noticed in the past there are several groups operating here that try to fix the situation here. I'll hopefully blog a bit about it more soon :-)

Mercurial for Subversion Users

2008-01-28T18:59:11Z

More and more projects are switching over to mercurial or similar DVCS. Great as mercurial is, it's hard to get started if you are used to subversion because the concept behind Subversion (svn) and mercurial (hg) is fundamentally different. This article should help you understand how mercurial and similar systems work and how you can use it to contribute patches to the pocoo projects. If you compare Subversion to mercurial you won't find that many similarities beside the command arguments. Subversion works like FTP whereas mercurial is bittorrent. In Subversion the server is special: it keeps all the revision log and all the operations require a connection to this server. In mercurial I can take down the central repository if there is one an all developers will still be able to exchange changes. All the revision information is available to anyone and there is absolutely no difference between server and clients. This fundamental design decision means that there are dozens of separate branches of the code. hg makes it easy to merge and branch and it's developed exactly for that. In Subversion branching and merging is painful an often people just don't branch and don't commit there changes until the testsuite etc. passes again which of course results in huge changesets. But let's step right into it! The first thing in Subversion you do is either creating a repository on the server or checking it out on the client. In hg there is no difference between server and client so the process of creating a repository is available to everybody. Creating a repository is just as simple as typing "hg init name_of_the_repository". If that folder does not exist yet it will create an empty folder and initialize it as root of the repository, otherwise it will create the repository in the name of that folder. The process of checking out is a bit different from Subversion because it's effectively the same as creating a branch. Say you want to check out the current Pygments version to do some changes. The first thing you will do is looking for a way to access this repository. There are three very common ways to access it: filesystem, HTTP or SSH. Pygments is available as SSH and HTTP, but for non core developers only HTTP is available. Interestingly quite a few people have problems locating the checkout URL which is not very surprising because hgweb handles that. hgweb is the standard mercurial web interface which doesn't only provide a way to look at the changesets and tree but also handles patch exchange. In the case of Pygments this command should give you a fresh checkout in a few seconds into the new folder "pygments":

hg clone http://dev.pocoo.org/hg/pygments-main pygments

One thing you will notice is that it's incredible fast and even though the repository contains the whole history the checkout is pretty small. By the time I'm writing this blog post the pygments sourcecode including the unittests and example sourcecode without the revision history is 2.5MB. A complete mercurial checkout is only 5MB even though it includes 486 changesets. After you got your very own repository by cloning the pygments one you will notice that all the subversion-like commands ("hg ci", "hg add", "hg up", ...) work locally only. You check into your local version of the repository and hg up won't incorporate remote changes. One of the things that happen on hg clone is that mercurial will set the path to the repository you cloned from into the hgrc of the newly created repository. This file (".hg/hgrc") is used to store per-repository configuration like the path of remote repositories, the name used for checkins, plugins that are only enabled for this repository and more. Executing "hg pull" will automatically pull changes from this remote repository and put them into the current repository as second branch. To see what "hg pull" will pull from that remote repository you can execute "hg incoming" and it will print a list of changesets that are in the remote repository but not yet in the local one. After you have pulled you have to update the repository with "hg up" so that you can actually see the changes. If there were remote changes that require merging you have to "hg merge" them and "hg ci" the merge. Because this process is very common there are ways to simplify it. "hg pull && hg update" can be written as "hg pull -u". All the commands (pull, update, merge and checkin if required) can be handled in one go using "hg fe". This command however is part of a plugin which is disabled by default. If you want to use it you have to add the following lines into the repository hgrc or your personal one:

[extensions]
hgext.fetch=

The other important difference to subversion is how you push your changes back to the server. In open source projects usually only a small number of developers has access to the main repository and contributors create patches using "diff" or "svn diff" and mail it to one of the persons with commit rights or attach it to a ticket in the project's tracker. If you are a person with push privileges you can do "hg push" and it will push the changesets which are not yet on the server (you can look at them using "hg outgoing"). If you don't have push access you can create a bundle of changes and attach that to a ticket rather than a patch. A bundle stores multiple changesets in one file and it also preserves the correct author information and timestamps. Another way is mailing the changes to a different developer using the patchbomb extension (I won't cover that here, just google it up). Or you can let other people pull from your repository. Therefore you either have to configure your apache to server a hgweb instance or you just call "hg serve" and it will spawn a server on localhost:8000 everybody can pull from. Once the developer has decided to put your changes into the central repository and pushed them, your changes will appear there unaltered and with the same revision hashes. What will be different is the local number the changeset is given. If the revision was called deadbeef:42 locally it could be called deadbeef:52 on the server because different changesets were applied first. All the commands that interact with remote repositories ("hg pull", "hg push", "hg fe", ...) also take a different path than the default path from the hgrc as argument. This allows you to pull changes from repositories shared over the web. A cool example what mercurial allows you to do is our last ubuntuusers webteam meetup. There we used my notebook to store the central repository and everybody pushed the changes every once in a while to it. Additionally some people exchanged patches to not yet working features among each other so that the code on the central repo was seldom broken. When I left everybody had all the changes locally because they pulled and I could remove my notebook and everybody continued working on their way home. When we met again on IRC I copied my repo on the server and everybody pushed their local changes to it.

ubuntuusers Webteam Meeting

2008-01-28T15:26:03Z

The last four days I attended our first offline-webteam meeting and it was fun ;-) We spend a few days hacking on various parts of the ubuntuusers software and had a great time. While it's pointless to talk about what we did because hacking on broken code is not that much fun as such (unless you can laugh about PHP bugs and stuff like that) I think we can present some of the changes with the upcoming hardy release. Don't expect too much but that shows that we are doing something even if you think development on the German portal software is stalled. That's certainly not the case. But we are once again looking for web designers. The sourcecode is multilingual and the development takes place in both German and English for potential future translations to other languages than German, so you can participate even if you are not speaking German :-) If you are interested in participating and if you have advanced XHTML1/HTML4/CSS2 skills, we want you for ubuntuusers webteam! If you are interested mail me at mitsuhiko you know what comes next ubuntu dot com.

No…

2008-01-23T22:31:07Z

I will not blog about it, because it just sucks.

Python Code Introspection

2008-01-21T21:00:09Z

Georg blogged about sphinx recently and now we can all rejoice because sphinx can now be used with custom documentation (non c-python documentation) too. Unfortunately hand written documentation alone is hard to maintain. For Werkzeug 0.2 I want to switch to a combination of hand written documentation and source documentation generated from python sourcecode, powered by sphinx of course. The main problem is getting the information out of the sourcecode. Unlike compiled languages in Python a lot is dynamic and much of the functionality is only available at runtime. Unlike PHP objects can be modified later on, and unlike Ruby there are attributes and methods on classes so it's hard to hide private/protected stuff on a class. epydoc came up with some conventions (especially for rst users) but introspection is still too hard in python and will lead to unexpected results. A pretty common idom in python is to implement a class in one file and import it to somewhere else, where it will become importable from. One of the best example was Python's "re" module. Until 2.5 that module was a stub and the actual functions were imported from the "sre" module. This becomes difficult if not impossible to get right for automatically generated modules. Pydoc for example requires __all__ to be defined in that module, otherwise it will hide those objects like it hides every other import. Another example which is pain in the ass to get right by automatic introspection are function decorators and descriptors. Descriptors also break in pydoc half of the time and nobody seems to know when exactly it breaks. Function decorators are impossible to get right because even if you try to traverse the closure variables of all the wrapped functions up to the original function the result could be terrible wrong because one function in the middle is indeed variadic. Because it's that hard to get right I don't want to lose time on solving something that's unsolvable anyways. My current idea to make the documentation process a little less annoying I think about adding directives to sphinx that pull information from python objects into rst files in a semiautomatic way. Say you have a pretty complex module that implements a couple of functions and classes and who knows what. Additionally to the documentation from the docstrings you also want to group the objects by topics and add hand written documentation for every group. In the python code you document all your functions with nice rst docstrings with the epydoc conventions. Additionally you add group markers: You can do that with every function and object. To dump the documentation for a group to an rst document you would then use a little directive: This will dump all the objects flagged with that group in alphabetical order to the rst document. But of course this breaks for complex use cases you should be able to step back. Imagine you have a pretty complex object and you want to hide some of the methods / attributes, mark missing variables or make a member a descriptor rather than a method if the automatic discovery failed: But how exactly that should work, I don't have a good idea yet. But I want to find it quickly because right now I'm pretty unhappy because writing docs both in the sourcecode and in hand written documentation sucks ass. Update: Fixed Georg's name :-)

Error Reporting in WSGI Applications

2008-01-18T14:15:22Z

Thanks to an google alert on "pygments" I stumbled accross gluon/web2py, some sort of "enterprise ready framework". It's definitively not a framework I would use myself for countless reasons but there is one thing on the feature list which I found interesting. Apparently gluon files tickets for tracebacks in the database. While it's a terrible idea to put that data into the very same database all your application data goes (what happens if the DB is down?) it's a different approach to Django which sends mails on errors. Two days ago someone mentioned the paste WSGI middleware which sends emails on tracebacks, similar to the way Django does that. I'm not a fan of error reporting middlewares because I think that should go into the application. On an server error (caused by lost database connections, programmer errors etc.) you should not present the user a black-and-white error page or display an inline traceback like most PHP applications do. We can do better! First of all there is a module in the standard library everybody seems to forget about. It's called "logging" and does exactly that --- it logs errors. I don't know why so many people miss it or just don't use it, but it's really one of the good things in the python standard library. Why is it so good? It's extensible and configurable. The idea is that the application gets itself a logger and logs into that logger (debug messages, information messages, warnings, errors or tracebacks). Independently from the logging there are logger handlers which do something with the logged messages. For example you can tell the logger to log everything except of debug output into a rotated file and mail all errors to some mail addresses. In the Werkzeug wiki there is a wiki page about error handling in WSGI applications using the logger module. Total amount of code needed for a simple WSGI application with logging is about ten lines or so. And it's flexible enough to integrate in every WSGI application setup, no need to solve that in a middleware where you don't have access to your application's config or whatever. And the best thing about it is that you can configure the way errors are handled. Like I mentioned before gluon creates entries in the database for logged errors. I wrote a small logger handler that does the same but for an external trac. Whenever an application error occours the logger checks if there is already a ticket for that error, if not it creates one. For every new occurrence of that bug it will create a comment in that ticket. If you want to try it out yourself, I added the code to the sandbox repository. Update: I also created a trac hack for it.

New Stuff in Werkzeug (and the WSGI World)

2008-01-16T17:29:22Z

Unfortunately I'm very busy lately so there are few updates on Werkzeug and all the other libraries I personally contributed (and there are even some patches in my Mail queue I have to apply after reviewing) but that doesn't mean that there is no progress :-) There is actually quite a lot of new stuff between the 0.1 release and now. Werkzeug tries to implement stupid stuff you reimplement in every second application in a way that you can use it with minor modifications in your application. Because many of those features only come up if you have implemented them often in your own applications they didn't make it into 0.1. Thanks to all the early adopters we now have cool new stuff that was implemented because there was need for it. For example Werkzeug now has RFC-compliant Etag and Cache-Control parsing. You can also generate etags automatically for responses and make them conditional for some requests. The utility module was extended with many new stuff that fix limitations in the standard library or implement long missed functions like finding modules in a package (very useful if you want to automatically register controllers), importing modules by a string (useful if your URL map endpoints are (partial) import paths), generating URLs like trac does by calling an Href object, functions to fix URLs similar to the way Firefox fixes them, dumping and parsing HTTP dates, loading and dumping cookies with a simple function call (and support for http only) and probably a lot more before the actual 0.2 release. Another cool thing is (unrelated to Werkzeug) that more and more cool modules come up that make web development a charm. While babel is available for quite a while I haven't really used it until last weak and it's really great. Together with Werkzeug's routing system and the ability to do multiple inheritance in SQLAlchemy there are so cool ways to do web development of internationalized applications. I know the last sentence doesn't make sense without the context, so I guess I have to blog about that. sooo freaking cool. Form handling in Python is still a bit strange if you are not using django's newforms (I know that there is formencode but somehow it isn't what I'm looking for) but there is now WTForms which looks promising. With some more small changes it could become a very cool form handling system (for example I'm missing some default validators at the moment and I'm not completely sure how to pass choices to a select box in a per form instance basis). WTForms was derived from an application that is already in production so it solves already many problems nicely. It's a library I want to watch closely the next weeks. And I think that approach should become the way Werkzeug is developed in the future. Implement features a release earlier and mark them as "under consideration" if they are not yet used in production applications. If you adopt them early you can give feedback and we can improve it to the next Werkzeug release and streamline the API. What's to do until the next Werkzeug release? Georg is currently working on making the sphinx documentation builder independent from the CPython documentation so that other projects can use it too. I then want to semi-automatically build an API documentation for Werkzeug and combine them with hand written rst pages for the Werkzeug 0.2 documentation. I got some feedback for the Werkzeug docs and looks like they are a bit too chaotic and misleading. Especially getting started with Werkzeug is still too complicated so I hope we can address this with a new documentation that combines automatically generated documentation with tutorials. The documentation tool could probably be useful for other projects too, I guess Georg will drop some lines in his blog once it's ready. Updates regarding Jinja will be up shortly, there is currently a branch developed by Lakin Wecker to speed up Jinja template evaluation. And if you already know what GHRML/XAML will gonna be: I will try to get that running this weekend. That's it for the moment ;-)

How to fix Python

2008-01-10T16:23:01Z

For a long time I thought I'm the only person disliking the stdlib, but apparently there are more :-) Not just that every library has it's own code style, it also has different names, some are implemented insanely bad (Cookie.py anyone?) or Javaish (threading, unittest) or just plain stupid (codeop). I know we can't just get rid of that stdlib, wait a moment, why can't we? Python 3 is upcoming and we all have to adapt our libraries and applications anyways. It wouldn't be too hard actually. Python loads modules from zip files anyways. Just move all the stuff into a zip file, move it into "old" or somethign and old applications just have to alter their imports "from cgi import escape" to "from old.cgi import escape". Or the other way round, the old libraries stay where they are and the new stdlib goes into "py.*" or "std.*". Insane or a good idea? You be the judge.

Python Template Engine Comparison

2008-01-01T20:32:12Z

I was small-talking with zzzeek about some things when I told him that I'm using Jinja, Genshi and Mako depending on what I'm doing. He told me that it's unusual to switch tools like that but I don't think it's that unusual. All three template engines are totally different but have a one thing in common: All three are the "second generation" of template engines. Genshi is the formal successor of kid, Mako somewhat replaced Mygthy and Jinja was inspired by the django templates. All three of them are framework agnostic, use unicode internally and have a cool API you can use in WSGI applications without scratching your head. But what inspired those template engines and which template engine to chose for which situation? I often used PHP in the past to do simple header/footer inclusion. But what always drove me nuts was that I had to use mod_rewrite to get nice URLs or use a bunch of folders with index.php files or use files and folders and drop the extension in the apache config. While this is nice, this is now that portable and you can't have dynamic parts in the URL and once you want some more dynamic stuff such as RSS feeds etc. you notice that you made a mistake by choosing PHP. Some days ago I then started working on the website for TextPress (not yet online) and wanted to try something new: I wrote a tiny WSGI application (about 50 lines of code) that just uses werkzeug's routing system and uses template names as endpoints. These templates are then loaded with Mako, rendered and returned as responses. This is not possible in the same way with Jinja because you don't have python blocks and not so simple and straightforward with Genshi because you have to think about XML or use a rather limited text based template engine. Another very cool feature of Mako is that you can do dynamic inheritance which is not possible in Jinja. Mako is a great template engine if you know Python, if you need some logic in templates (and you know: logic in templates is not bad in every situation) and if you need the best performance. Without a doubt Mako is one of the fastest template engines for Python and the fastest template engine that does not use a C extension. Then there is Jinja which is also a text based template engine like Mako. However the focus is on a completely different level. When Mako is like PHP, Jinja is like Smarty (even though Mako is a million times better than PHP as template engine). When I stated working with Python as programming language for web applications I stumbled about django. I looked at the template engine and thought: WTF is that? The syntax seemed odd and the restrictions ridiculous. Later on I loved the syntax (and apparently others do to: the mini template engine by Ian Bicking (tempita if I recall correctly) and the Genshi text templates are using that syntax or a similar one too) but some of the restrictions seem still ridiculous. When I looked at all those Django templates I created over the time I noticed that I often moved calculations into template tags that could be function calls, that I did other calculations in in the view functions that did not belong there and even more important: that you could replace 95% of the custom template tags with function calls or function calls with an enclosed template block if the template engine had proper expressions. This lead to the development of what is now known as Jinja. The syntax, the fact that it's sandboxed and the designed friendliness is still very similar to Django, but unlike Django python like expressions are possible in Jinja. I'm using Jinja wherever I think web designers want to work on later on. For example as template engine for TextPress or other applications that should be styled by third party web designers. Genshi on the other hand is an XML template engine. As a result of that it's slower but also "context aware". It knows when it's processing a CDATA section, it knows when it's inside a tag or an attribute etc. This makes it possible to defend XSS in an automatic way. Per default Genshi inserts the text into the output stream as text and not as markup. That means all the HTML entities are automatically escaped for you. And because it's stream based you can rewrite streams during the rendering process. This makes it possible to fill form fields automatically, use XInclude for simple layout templates and a lot more. You can even translate your XML based templates into HTML4 on the fly. So you can use your XML tool chain internally and output HTML4 and use the best of both worlds. But because of this high flexibility Genshi also has some problems to fight: You need to have XML knowledge to use it. No problem if you are a programmer, but not that good if you are a web designer doing fancy layouts. You are also forced to use XML templates everywhere. It's true that Genshi has text templates too to fill the gaps, but they are not comparable with real text template engines and you are still operating on an XML stream, just that you don't see it. And lastly: this whole stream processing makes Genshi slow. Not so slow that you can't use it for big applications, but noticeably slower than Mako or Jinja. If you are using XML anyways in your application, Genshi is a very good idea. Also if you don't have template designers that don't know XML or if performance is not that much of a problem. Most of the time the bottleneck is the database anyways. I never had real problems with Genshi performance so far. I hope this post sums up why I'm using all three template engines and why I think we should be happy that we can chose between a couple of template engines :-) Why I'm not covering other template engines like Cheetah or SimpleTAL? Mostly because I looked at them, tried them out and never used them for something big. Mostly because Mako looks a lot nicer than Cheetah to me and SimpleTAL is far too much away from Python for me.

No, no, It’s just resting

2007-12-23T22:28:10Z

Currently on pocoo.org is a semi static webpage with a completely unmaintained semi static blog and completely outdated information. While pocoo is not dead yet (just resting) it's time to put something on the front page that represents the current state of the situation in a better way. Let's see if we can get that updated in the next two weeks. Oh, and TextPress is now up to date again, a bunch of bugs were fixed and we're much nearer to a release than some weeks ago, but unfortunately we're behind the roadmap once again :-)

Merry Christmas!

2007-12-22T17:46:02Z

Again we're coming close to December the 24th, a day many people will spend with their families or friends to celebrate. I want to use this Christmas to say thank you to a couple of persons that helped me a lot the last years, either directly or indirectly due to their tremendous work in the Open Source community. First of all I want to thank you Georg Brandl, who you thought me a lot about Python's internals. You had an answer to pretty all of my questions and you were the first person I worked with on a real Open Source project. Even though we still don't have Pocoo released it was great fun to work with you. Thank you Benjamin Wiegand, Christoph Hack and Christopher Grebs for all your code contributions and your work on Pocoo and the Pocoo libraries. A big thank you to the whole ubuntuusers team, also those of you who left the team in the past. We had many problems in the past but I looking back those three years I think most of the time we worked together in the past were plesant :-) Thank you Marek Kubica for all those nice discussions we've had. And thanks for your work on the German Python community and your patches to the Pocoo libraries. I guess the python forum and wiki wouldn't be the same without you. Of course a big thank you to Alexander Schremmer who keeps the pocoo server running. Learned a lot from you :-) Also a big thank you to Fritz Cizmarov who introduced me to ubuntu. Unfortunately he passed away two years ago but I'll never forget him. I also want to thank Mike Bayers for his work on SQLAlchemy which I use on a daily basis, all the guys over at edgewall for Genshi and Trac, the mercurial people for their ass-kicking DVCS, Martijn Faassen for lxml and all the people working on ubuntu. Keep up the good work and Merry Christmas to you all!

This Week in Pushin’ Bugs…

2007-12-17T12:23:50Z

This wonderful feature request for the German Keyboard Layouts. Open since quite a while and easy to fix (if someone knows how to do it). And if one finds some intuitive places for emdash, endash, ellipsis and some other often used typographical characters and gets that into the next LTS he/she gets a hug :-)

Werkzeug Identity Crisis?

2007-12-15T22:00:48Z

After the Werkzeug 0.1 release I got some feedback by Ian Bicking. Summarized it was about why Werkzeug pretty much does what webob does. Afterwards I thought about the issue and talked with some current Werkzeug users. And to sum it up: Werkzeug stays a standalone library without webob dependency and will stay with the current request/response object implementation. While I agree that there are things webob does what Werkzeug (currently?) does not provide such as byteranges I still think it's on the right path. We have now a solid feature set in Werkzeug itself such as form data parsing, file uploads, conditional requests, simple redirects, useful exceptions, a good routing system, a simple bridge to wsgiref for a development server, a small helper library for management scripts and much more and additionally an optional contrib package with dozens of small helpers such as client side sessions, utilities for application sessions, the iterio library, a stream limiter for Django applications etc. Even more important than having everything you need to get started with WSGI in one library the implementation is easy to understand, fast and written in a way that you will never hit a border. If you find a piece of code in Werkzeug that only solves 90% of your problem but provides absolute no way without side effects to get the full 100% solution that's a bug and should be fixed. The minimal configuration you need to get a full featured web app running in a production environment with Werkzeug is currently something like Werkzeug + Genshi + SQLAlchemy + flup. Makes a total of four dependencies. It's not like I think dependencies are problems but what I noticed so far is the more libraries a user has to install the higher the number of problems and the lower the number of happy users. This is one thing Django does right: install django and you're done. If you don't like that philosophy there is pylons with a whole string of dependencies ;-)

Secure Client Side Sessions

2007-12-15T17:19:56Z

Since a few changesets Werkzeug provides a module for client side sessions. While it's of course not yet perfect I think it's an interesting approach and we're currently replacing django's sessions with Werkzeug's secure cookie. How does it work? Basically the data is stored as pickled data inside the cookie. The pickled data is hashed and added to the cookie too. Whenever the data is loaded from the cookie a new hash is created and the two hashes are compared. If they match the data is unserialized, otherwise a new SecureCookie object is created with empty data. As a matter of fact there is no "session key" so if you want to use it with django you have to fake the "session key" by inserting it into the data. We use the following session middleware as replacement for the django session middleware: Additionally to the normal django session middleware this middleware can handle both persistent sessions and browser-session bound sessions. Per default a session ends at the end of the browser session, if you want to make it persistent you have to set "is_permanent_session" in the session data to True. Keep in mind that cookies have a size limit and that users will be able to look at that data (but not alter it). The example above requires the current werkzeug tip and not the 0.1 release version.

Django’s Problems and Why 2.0 is a Bad Idea

2007-12-12T11:00:58Z

I stumbled about this thread on django-developers which proposes calling the Django 1.0 release Django 2.0. One the one hand version numbers say nothing. Just take Wine or Trac as two examples that are already very stable but still below the magical 1.0 release. Open Source software often takes some time until a 1.0 is released and that's perfectly fine. However skipping a version number is purely a marketing trick IMO. Just think of Java which currently names 1.6 Java 6 whereas 1.4 still was Java 2. With django it looks like the plan is to keep up with Rails which went to 2.0 a few days ago. While I love to see that django kicks ass and it's moving toward a stable release I have a bad feeling naming it 2.0. Because there is currently a huge gap between rails and django unfortunately. Rails has gained really good integrated migrations, REST webservices, a debugger and many other things django is still lacking. Django makes an incredible good framework if you get your problem into the use case of django. But as soon as you break out of it and need something that goes beyond what's possible in django you wish you have chosen something else. The django ORM is far from optimal, the admin rocks but as soon as the number of users exceeds 10.000 users it's impossible to use it (chose yourself in a dropdown of 50.000 users …) or becomes utterly complex. Complex data models also look awkward in the admin or become too complicated to manage. And if you want to stick with the admin you cannot replace the user model. Now what do you do if you have a forum and want to count the posts? Use a UserProfile module? And how do you want to display a list of users sorted by their number of posts? Yes there are ways to hack around it but the more complex the application becomes the more you of django's strengths become obsolete. The application I'm working on right now now is only using two more contrib modules. The auth and the admin, and it looks like we have to drop them too, due to the limitations. All applications in that project hack around ORM limitations, we have an incredible number of recreated base middlewares, we have to monkey patch the request object to hack in subdomain support. I was talking with David Cramer from curse gaming about some of the issues and he told me that they have forked django at a given point and patched the ORM. The django template engine was replaced by Jinja (our application does the same) and they are caching the hell out of the application to scale it. Bryan McLemore from the curse team told me some time ago that some pages have up to 30 queries on a page. I don't want to say that django fails in what it's doing. But it's far from 2.0.

Werkzeug Example Applications Online

2007-12-11T22:30:00Z

I now installed the Werkzeug example applications on pocoo.org. So you don't have to pull all the code if you want to try them out :-) Here is the Wiki and the Planet. Have fun testing.

Werkzeug Debugger in Django

2007-12-10T17:30:24Z

(Thanks monkey patching). Ugly but works: using the werkzeug debugger with django. Now digg/reddit it ;-)

We don’t need namespaces…

2007-12-10T14:07:17Z

…because we use prefixes: Found in the Zend framework :-)

Werkzeug 0.1 Released

2007-12-09T18:58:22Z

Finally (sorry for the long delay) Werkzeug 0.1 is out. Here parts of the current featurelist:

Provides Request and Response objects for WSGI
Handles file uploads by using temporary files for incoming data.
Provides a middleware for static data for development purposes
Tiny wrapper around wsgiref for easier development (autoreload, optional multithreaded enviornment)
Unicode aware data processing. Just use unicode everywhere, werkzeug handles that for you.
Mini template engine. Sometimes string formattings just are not enough and real template engines are too big for that tiny task.
Context locals. Don't pass request/user/database connections and other objects around. Put them on a global context local object and werkzeug makes sure that everyting is cleaned up end delivered well.
Test utilities. Create fake WSGI environments and requests to test your application.
Interactive debugger. Application dies with an error? Hook the debugger in and inspect every frame.

Here a screenshot of the debugger in action: Small werkzeug example applications can be found in the trac. In the werkzeug.contrib package are also some pieces of code that can be useful for django developers. For example there is a stream wrapper that limits incoming form data to a given number of bytes. This is useful for django because django streams into the memory and not to the file system. Have fun and report bugs / feature wishes in the trac :-) Get it while it's hot from the Cheeseshop.

German Django Community

2007-12-08T19:51:47Z

Since yesterday a German community page for django exists: django-de.org. Good to see that people are working on that :-)

Listen to more…

2007-11-28T16:30:52Z

...Inina Gap. Which you probably haven't heard by now. It's not rock, metal or any other stuff I normally listen to, in fact it's completely different, more like electronic fusion. And disclaimer: I a) saw them live, and b) a friend of mine is playing bass in that band :-) Nonetheless: Check it out.

220…

2007-11-26T18:36:40Z

...dead. And that's a non lethal weapon? update: Georg points out that it's called a "less lethal" weapon.

Wiki Models with SQLAlchemy

2007-11-22T00:34:28Z

I was working on an example application for the upcoming Werkzeug release and decided to make a wiki for that purpose. (The main reason is that I found the Creoleparser on the pypi index and thought that it would integrate into a genshi powered wiki well) Now wikis have an interesting data structure. Basically you have pages and revisions, where a revision is bound to exactly one page. Simple to model and my database definition looks like that: Very simple schema and you can easily bind it to two classes: Works pretty well but as soon as you start putting your stuff into the template it feels unpythonic :-) You basically have to provide always two objects, the revision and the page. Why not combine that into one model? First I removed my database tables again and combined that into one table but then I found out that you can map joins to classes in SQLAlchemy. That and Python's ability to do multiple inheritance gives me the ability to combine both tables into one class: Very nice for the templates and also easy to query. It looks like a normal python class. I haven't tried if that also works for write access, but I doubt it. Because of that I have added an exception to the __init__ method to avoid creating pages and revisions via the RevisionedPage object. The full example can be found in the simplewiki sources.

Convert an internal iterator into an external in Python

2007-11-18T21:42:47Z

Example usage: