Serving Static Datatracker Files via a CDN

Intro

With release 6.4.0, the way that the static files used by the datatracker are

handled changes substantially. Static files were previously versioned under a

top-level ``static/`` directory, but this is not the case any more. External

files (such as for instance ``jquery.min.js``) are now placed under

``ietf/externals/static/`` and updated using a tool called bower_, while

datatracker-specific files (images, css, js, etc.) are located under

``ietf/static/ietf/`` and ``ietf/secr/static/secr/`` respectively.

The following sections provide more details about handling of internals,

externals, and how deployment is done.

Serving Static Files via CDN

If resources served over a CDN and/or with a high max-age don't have different

URLs for different versions, then any component upgrade which is accompanied

by a change in template functionality will have a long transition time

during which the new pages are served with old components, with possible

breakage. We want to avoid this.

The intention is that after a release has been checked out, but before it is

deployed, the standard django 'collectstatic' management command will be

run, resulting in all static files being collected from their working

directory location and placed in an appropiate location for serving via CDN.

This location will have the datatracker release version as part of its URL,

so that after the deployment of a new release, the CDN will be forced to fetch

the appropriate static files for that release.

An important part of this is to set up the ``STATIC_ROOT`` and ``STATIC_URL``

settings appropriately. In 6.4.0, the setting is as follows in production

mode::

The result is that all static files collected via the ``collectstatic``

command will be placed in a location served via CDN, with the release

version being part of the URL.

In development mode, ``STATIC_URL`` is set to ``/static/``, and Django's

``staticfiles`` infrastructure makes the static files available under that

local URL root.

Handling of External Javascript and CSS Components

In order to make it easy to keep track of and upgrade external components,

these are now handled by a tool called ``bower``, via a new management

command ``bower_install``. Each external component is listed in a file

``ietf/bower.json``. In order to update the version of a component listed in

``ietf/bower.json``, or add a new one, you should edit ``bower.json``, and

then run the management command::

(Not surprisingly, you need to have bower_ installed in order to use this

management command.)

That command will fetch the required version of each external component listed

in ``bower.json`` (actually, it will do this for *all* ``bower.json`` files

found in the ``static/`` directories of all ``INSTALLED_APPS`` and the

directories in ``settings.STATICFILES_DIRS``), saving them temporarily under

``.tmp/bower_components/``; it will then extract the relevant production

``js`` and ``css`` files and place them in an appropriately named directory

under ``ietf/externals/static/``. The latter location is taken from

``COMPONENT_ROOT`` in ``settings.py``.

Managing external components via bower has the additional benefit of

managing dependencies -- components that have dependencies will pull in

these, so that they also are placed under ``ietf/externals/static/``.

You still have to manually add the necessary stylesheet and/or javascript

references to your templates, though.

The ``bower_install`` command is not run automatically by ``bin/mkrelease``,

since it needs an updated ``bower.json`` in order to do anything interesting.

So when you're intending to update an external web asset to a newer version,

you need to edit the ``bower.json`` file, run ``manage.py bower_install``,

verify that the new version doesn't break things, and then commit the new

files under ``ietf/externals/static/`` and the updated ``bower.json``.

.. _bower: http://bower.io/

The ``ietf/externals/static/`` Directory

The directory ``ietf/externals/static/`` holds a number of subdirectories

which hold distribution files for external client-side components, collected

by ``bower_install`` as described above. Currently

(01 Aug 2015) this means ``js`` and ``css`` components and fonts.

These components each reside in their own subdirectory, which is named with

the component name::

The ``pt*`` fonts are an exception, in that there is no bower component

available for these fonts, so they have been put in place manually.

Handling of Internal Static Files

Previous to this release, internal static files were located under

``static/``, mixed together with the external components. They are now

located under ``ietf/static/ietf/`` and ``ietf/secr/static/secr``, and will be

collected for serving via CDN by the ``collectstatic`` command. Any static

files associated with a particular app will be handled the same way (which

means that all ``admin/`` static files automatically will be handled correctly, too).

Handling of Customised Bootstrap Files

We are using a customised version of Bootstrap_, which is handled specially,

by a SVN externals definition in ``ietf/static/ietf``. That pulls the content

of the ``bootstrap/dist/`` directory (which is generated by running ``grunt``

in the ``bootstrap/`` directory) into ``ietf/static/ietf/bootstrap``, from

where it is collected by ``collectstatic``.

Changes to Template Files

In order to make the template files refer to the correct versioned CDN URL

(as given by the STATIC_URL root) all references to static files in the

templates have been updated to use the ``static`` template tag when referring

to static files. This will automatically result in both serving static files

from the right place in development mode, and referring to the correct

versioned URL in production mode and the simpler ``/static/`` urls in

development mode.

.. _bootstrap: http://getbootstrap.com/

Deployment

During deployment, it is now necessary to run the management command::

before activating a new release. The deployment README file has been updated

accordingly, and now reads as follows::