Polyglot Developer

Enlivepy A Different Approach to Html Templating in Python

2014-10-22T00:00:00+03:00

The Problem About Html Templating

Don’t worry - this blog entry won’t consist of me rambling about yet another templating engine that will change the world. Instead, I’ll be focusing on a practical aspect - the problems the current HTML templating libraries have and what can be possibly done about these issues.

Django and all other MVC/MVP frameworks do a great job at separating the Controller logic from the View logic. This way you can have all your complex operations in Controller and send back the data that needs to be shown or rendered to the end user. On the other end of this process you have a templating library - it takes Python objects and renders them.

Now, some of these templating libraries argue that having lots of logic in the representation layer is a bad thing (Django templating for example). Others, like Jinja2, are more lax and allow you to do lots of interesting things.

In both cases what we are actually doing is mixing data with logic. And while this might not seem like a problem with small applications, when you start operating on a greater scale, issues start to appear (a lot). Let me provide a few examples:

Front end developers (CSS/HTML developers) start having difficulties when editing html pages when something does not appear the way it should be appearing. They have to learn the templating engine your framework uses.
Sometimes HTML/CSS developers break the logic in html pages by replacing something by mistake.
Backend developers are not the best at editing both HTML and CSS (well, in most cases, at least!). I have had this happen to me quite a few times too. I receive the HTML from a designer team, after that I put it into the templating engine...and it has a radically changed appearance. When I compare the way it looks on the server and the original things are just looking completely different.
If HTML/CSS developers need to play or change something on the system, what they need is a working copy of the whole system. You need to give them ssh access to the staging server or something similar.

A possible solution to the problem

Recently I was playing around with different programming languages like Clojure, Go and Scala. I've found this Clojure library called enlive that approaches html templating in a slightly different manner. The general idea is you have the html of your site as static entity and you do some transformations on this html by generating a new html to be served on the server. The mentioned transformations are basic functions that take some nodes of html elements and return back new changed elements back.

This solves some of the painful problems I mentioned before hand. What’s different now is that:

We have a static html that does not include any logic in it. The designer team can edit it any time without breaking the logic. They can also have a fully working version of the representation without the need of having a running server around.
The backend team is happy - they don’t have to edit the html and combine it with a certain framework just to see it being radically different than the one they expected to have.
The backend team can use all of the programming language tricks without being afraid of polluting the representation with too many logic.

I liked this idea and created a clone of this library for Python programming language. It is called enlivepy. The purpose of my current blogpost is to show you a very simple template with a navigation bar. This will illustrate the concept in a pretty easy and educational manner and it will show you how it’s actually put into practice.

Getting Started

For the purpose of this blog entry I’ve created an example Django application. The mentioned application replaces the default Django templating with enlivepy html transformation library (I will explain more on this topic in another topic). To start with the application :

git clone https://github.com/makkalot/django-enlivepy-example.git
cd django-enlivepy-example/
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
cd example/
python manage.py syncdb
python manage.py runserver

If you don't want to wait to the end of the entry, you can check the expected result at :

http://127.0.0.1:8000/index/

A Basic Html Static Page

The first step will involve starting with a static html page that looks like (this)[https://github.com/makkalot/django-enlivepy-example/blob/master/example/templates/logonav/index.html]. You can open it in your browser to see that it is a working page without any logic in it. To put it brief, it is something like this (only the parts we're interested are shown):

<body>

    <!-- Page Content -->
    <div class="container">
        <div class="row">
            <div class="col-lg-12">
                <h1>Logo Nav by Start Bootstrap</h1>
                <p>Note: You may need to adjust some CSS based on your needs.</p>
            </div>
        </div>
    </div>
    <!-- /.container -->

    <!-- jQuery Version 1.11.0 -->
    <script src="static/js/jquery-1.11.0.js"></script>

    <!-- Bootstrap Core JavaScript -->
    <script src="static/js/bootstrap.min.js"></script>

</body>

The first task is to put some dynamic content inside the "container" which now has some static text. However, before we get there, we need to replace the static paths with those that will work on Django's STATIC_URL paths. For that purpose lets create a mixin :

class BaseMediaUrlTemplateMixin(object):

    def transform(self, nodes, *args, **kwargs):

        if not kwargs.get("STATIC_URL"):
            raise Exception("STATIC_URL not set")

        media_url = kwargs.get("STATIC_URL")
        #print "MEDIA : ",media_url

        #the next step is to run the at command
        at(nodes,
           "script", partial(self.trans_path, media_url, "script", "src"),
           "link",   partial(self.trans_path, media_url, "link", "href"))


    def trans_path(self, media_url, tag_name, attr_name, node):

        path = node.get(attr_name)
        if not path:
            return

        path = path.strip()
        if path.startswith("http") or not path:
            #do nothing in this case
            return

        fpath = ""
        if path.startswith(media_url[1:]):
            fpath = "".join(["/", path])
        else:
            fpath = os.path.join(media_url, path)

        attr_fn = set_attr(**{attr_name:fpath})
        return transform(node, tag_name, attr_fn)

The most important part of the code above is found in the transform method. This method takes the selected nodes so far (in this case - the whole page), some arguments which can be passed via view code (render_to_response), and selects the all script and link elements. Afterwards, it transforms their paths with paths that are compatible with Django's STATIC_URL variable. Therefore the transformed page's elements will look like this :

....
<script src="/static/js/jquery-1.11.0.js"></script>
<script src="/static/js/bootstrap.min.js"></script>
....

The Snippets

Enlive has something called a “Snippet” - a concept for reusable html code blocks. The idea here is to have some sort of transformation code that will select a proper part of the html page and then process it. Snippets come in many forms and have to be reusable. So for example a sidebar, navigation, forms, footers and others all count as a snippet. To illustrate this, I’ll create a snippet for the navigation. The static version of navigation looks like this :

<body>
    <!-- Navigation -->
    <nav class="navbar navbar-inverse navbar-fixed-top" role="navigation">
        <div class="container">
            <!-- Brand and toggle get grouped for better mobile display -->
            <div class="navbar-header">
                <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
                    <span class="sr-only">Toggle navigation</span>
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                </button>
                <a class="navbar-brand" href="#">
                    <img src="http://placehold.it/150x50&text=Logo" alt="">
                </a>
            </div>
            <!-- Collect the nav links, forms, and other content for toggling -->
            <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
                <ul class="nav navbar-nav">
                    <li>
                        <a href="#">About</a>
                    </li>
                </ul>
            </div>
            <!-- /.navbar-collapse -->
        </div>
        <!-- /.container -->
    </nav>

    <!-- jQuery Version 1.11.0 -->
    <script src="static/js/jquery-1.11.0.js"></script>

    <!-- Bootstrap Core JavaScript -->
    <script src="static/js/bootstrap.min.js"></script>

</body>

As you can see, there is only one link named "About". The purpose of the snippet is to get only the "nav" element, transform its contents with our dynamic navigation links and return only that portion back. Let’s creat this snippet like this:

class TodoNavSnippet(DjangoSnippet):

    template = "logonav/nav.html"
    selection = "nav.navbar"

    menu_items = ("Home", "About", "Services", "Contact")

    def transform(self, nodes, *args, **kwargs):
        navs_fn = clone_for(self.menu_items,
                            "li > a", lambda i: content(i),
                            "li > a", lambda i: set_attr(href="/"+i.lower()+"/"))
        at(nodes, "ul.nav > li", navs_fn)
        return nodes

The snippet above has a certain property called "selection". What it does is selecting only the nodes (with CSS selection) that constitute an interest for us. In my case, it selects "nav.navbar". In its transformation method it uses the clone_for transformer function, which duplicates the given node. Therefore, we're able to clone and produce a list for the menu_items attribute specified.

It is useful to note that template property finds in the same way Django templating system finds its files. What this means is that you can structure your html static pages as you would do in normal Django applications. Here’s the generated code for the snippet:

<nav class="navbar navbar-inverse navbar-fixed-top" role="navigation"><div class="container">
        <!-- Brand and toggle get grouped for better mobile display -->
        <div class="navbar-header">
            <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
                <span class="sr-only">Toggle navigation</span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
            </button>
            <a class="navbar-brand" href="#">
                <img src="http://placehold.it/150x50&amp;text=Logo" alt=""></a>
        </div>
        <!-- Collect the nav links, forms, and other content for toggling -->
        <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
            <ul class="nav navbar-nav"><li>
                    <a href="/home/">Home</a>
                </li>
            <li>
                    <a href="/about/">About</a>
                </li>
            <li>
                    <a href="/services/">Services</a>
                </li>
            <li>
                    <a href="/contact/">Contact</a>
                </li>

            </ul></div>
        <!-- /.navbar-collapse -->
    </div>
    <!-- /.container -->
</nav><!-- Page Content -->

The Template Concept

The next step is to replace the static content with dynamic content in the index page and add the transformer navigation node to it. For this purpose I’m going to use enlive's "template" concept. It is very similar to a snippet, but instead of returning a portion of the page, it will return the whole page back. In other words, the template is the part that combines the snippets together, similar to "base.html" in Django.

Once again, let me demonstrate:

class TodoIndex(BaseMediaUrlTemplateMixin, DjangoTemplate):

    template = "logonav/index.html"

    def transform(self, nodes, *args, **kwargs):
        #call the previous one first
        super(TodoIndex, self).transform(nodes, *args, **kwargs)
        #prepend the navigation snippet
        nav_snip = TodoNavSnippet()
        navbar = nav_snip(*args, **kwargs)

        at(nodes,
           "body", prepend(navbar),
           "div.col-lg-12 > h1", content("Index Todo List"),
           "div.col-lg-12 > p", None)

As you can see, what it does first is to call the BaseMediaUrlTemplateMixin class’s transform method which takes care of replacing the script and link tags as a start. Next, I prepend the already created NavigationSnippet to the “body” element. To finish all off, I replace the static content with the dynamic content.

In the example above I used the at function - it accepts multi selectors with multi transformer functions (you can check documentation for more information).

The transformed content will now look like this (navigation stripped):

<div class="container">
    <div class="row">
        <div class="col-lg-12">
            <h1>Index Todo List</h1>
        </div>
    </div>
</div>

Using Enlivepy in Django's Views

You can use enlivepy's templates in Django's views as you'd use the Django's default templates. In order to be able to do that, you have to first add the enlipy's template loader in settings:

TEMPLATE_LOADERS = (
    'enlivepy.django.loader.EnlivepyLoader',
)

Then you have to put your templates and snippets in a file called enlivetmpl.py in apps you want to have enlivepy enabled. Afterwards, you have to call enlivepy’s registry function autodiscover (it’s really the same pattern as Django admin) and put it in a place you know will be loaded as fast as possible (i.e init.py)

import enlivepy.django as enlivetmpl

#register the templates here
enlivetmpl.autodiscover()

Then at the end of your enlivetmpl.py file you have to register the templates you will be calling your views like :

register("todo_index", TodoIndex())

I want to bring to your attention the fact that you can name your templates however you wish.

And finally, in your views you can call your templates as you'd do in normal applications :

class TodoIndexView(TemplateView):

    template_name = "todo_index"

Summary

Enlivepy is still a pretty new and fresh concept I’ve been working on. Without any doubt there is a lot room of improvement. Examples that come off the top of my head include performance optimization or adding some missing parts from the original implementation.

I’ve started using it on my personal projects and it has been both effective and fun. However, it is definitely not ready for mainstream production usage. Hopefully after some more tinkering around with it, it maybe a very good solution to many of the problems found in traditional template-based programming.

So, in this post I did my best to explain how Enlivepy works and what current issues it aims to solve. I hope it’s been an interesting and educational read! I’d also love to hear your ways of tackling the problems you encounter with the current templating systems. How do you do it and is it effective?

Let me know in the comments below!

Django Authentication Workflow

2014-07-13T00:00:00+03:00

Django Authentication

The existing Django Documentation explains everything about the Django authentication process in a pretty good way. Directions include adding your back end, replacing the built-in User model and many more. However, despite this good coverage, I had certain problems with seeing the big picture of the overall process. At times when I needed to create a new authentication backend, I always had to glance through the Django source again.

In this blog entry I’ll focus on the Django source code and documentation -- to be more exact, I will share my findings during reading both the source code and the documentation.

Workflow as Pseudocode

If I play around with a typical Django app using all the defaults, the authentication process will look like this:

The user comes to django.contrib.auth.authenticate
In case this step is successful, the next stop is reaching django.contrib.auth.login
The final step is the user getting redirected to the successful login page.

But if I dive deeper and present the process in a detailed way, things will be structured like this:

django.contrib.auth.authenticate : Traverses all of the registered backends. The backend that is responsible for the user authentication gets assigned to the current user object. Then, the user object is returned back.

The code is something like this (stripped):

def authenticate(**credentials):
    """
    If the given credentials are valid, return a User object.
    """
    for backend in get_backends():
        try:
            user = backend.authenticate(**credentials)
        except TypeError:
            continue
        except PermissionDenied:
            return None
        if user is None:
            continue
        # Annotate the user object with the path of the backend.
        user.backend = "%s.%s" % (backend.__module__, backend.__class__.__name__)
        return user

The next step is to save the user and his corresponding backend into the session object. This can be done via the login function, present in the django.contrib.auth

django.contrib.auth.login :

def login(request, user):
    request.session[SESSION_KEY] = user.pk
    request.session[BACKEND_SESSION_KEY] = user.backend

Most of the code you see here is stripped in order to simplify the concept and have you understand it easier. After the user sends back a response to the whole auth/login process, the SessionMiddleware.process_response saves the changed session object into its database. This effectively means that next time the user accesses some of the views, the object will be loaded from this saved point.

Checking User in Session

But what happens if the user gains access to the predefined login_required views after the login process? In this case, we’re looking at a workflow structured like this:

SessionMiddleware loads the data of the current user in the request object from the session storage
AuthenticationMiddleware then loads the user from the session database and sets it as an attribute in the request object
And finally, the user has passed the auth check successfully and he can access the requested page

Again, if I break the process down and present it in a more in-depth manner, things will be structured like this:

django.contrib.sessions.middleware.SessionMiddleware : Loads the session info that is related to the current user :

class SessionMiddleware(object):
    def process_request(self, request):
        engine = import_module(settings.SESSION_ENGINE)
        session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME, None)
        request.session = engine.SessionStore(session_key)

The next step to be taken is:

django.contrib.auth.middleware.AuthenticationMiddleware : This part is responsible for loading the backend which was saved above (login function). It also prompts the backend 'get_user' method to load the user object. It is something like:

def get_user(request):
    from .models import AnonymousUser
    try:
        user_id = request.session[SESSION_KEY]
        backend_path = request.session[BACKEND_SESSION_KEY]
        assert backend_path in settings.AUTHENTICATION_BACKENDS
        backend = load_backend(backend_path)
        user = backend.get_user(user_id) or AnonymousUser()
    except (KeyError, AssertionError):
        user = AnonymousUser()
    return user

request.user = get_user(request)

I should note that if the user is not found either in the session store or the authentication backend, the AnonymousUser will be returned back. After undertaking such a step, the request object will now have a ‘user’ attribute assigned to it. In such a case you can easily check if a certain user is authenticated by typing:

if request.user.is_authenticated():
    #do some auth stuff

Actually, that is how login_required decorator is implemented. It checks for existing user.is_authenticated() condition on current user.

Summary

To summarize all of this, Django caches the user’s authentication backend when the login process is successful. After consecutive requests, this info gets extracted from the session store and is then used to load the user object into the current request.

Finally, the views that need to check if a user is authenticated in a valid way can do so by invoking the the is_authenticated method of the aforementioned user.