These are some of the main elements of the stack used by Assembl. Some of these choices are not always the most mainstream, but were meanstream at the start of the project in mid-2012.

We use the Python language for backend development, because of its flexibility, dynamism and rich ecosystem. Because the project aims to explore new practices, we have given priority to ease of development over absolute performance.

The project was started on Python 2.7, and some components are still holding us back from the transition to Python 3.

This is a list of every backend dependency used directly in assembl code. The packages you would need to know about first are described in subsequent paragraphs.

This comes from :file:`requirements.txt`, but indirect dependencies are not listed.

Alembic

Database schema migration

anyjson

JSON parsing

Babel

localization and internationalization: extract strings from source files and compile translated files.

Beaker

Web sessions (uses Memcached)

beautifulsoup4

HTML parsing

celery

Tasks running in another process

colander

Data validation (not used)

cornice

RESTful APIs in Pyramid. Used for the :ref:`classical_api`

dogpile

Allows to cache and share the results of a long calculation. Was used for CIF caching.

enum34

Python3 enums backported to Python2

facebook_sdk

Facebook SDK, allows us to query facebook, post messages to it and retrieve comments.

feedparser

Used by :py:class:`assembl.models.feed_parsing.FeedSourceReader` to read Atom feeds.

gensim

Used by :py:mod:`assembl.nlp.cluster` to find clusters of similar posts.

google_api_python_client

Used in :py:class:`assembl.nlp.translation_service.GoogleTranslationService` to communicate with the google translation service and translate messages

imaplib2

Used to retrieve messages from an IMAP server, through the :py:mod:`assembl.processes.imap` celery process or the :py:class:`assembl.processes.imaplib2_source_reader.IMAPReader`.

iso639

Translate between ISO-639-1 and ISO-639-2 language codes.

isodate

Parse dates in ISO-8601 format to python datetime objects.

Jinja2

Our main templating mechanism.

jwzthreading

Used to deduce thread order of imported mail messages

kombu

Inter-process messaging. Uses Redis. Used by Celery and directly by the :py:mod:`assembl.processes.source_reader`

langdetect

Guess the original language of a message. Used in :py:mod:`assembl.nlp.translation_service`

lxml

Parse XML directly. Also used by beautifulsoup4.

nodeenv

Standardized NodeJS environment, coordinated with Virtualenv

premailer

flatten the CSS of HTML, for sending as email in notifications.

psycopg2

Connect to Postgres database

pygraphviz

Create graph images with Graphviz. Used in :py:func:`assembl.views.api2.discussion.as_mind_map`.

pyIsEmail

Validate whether an input value is a valid email.

PyJWT

JSON web tokens, allows to use Annotator from a bookmarklet.

Pyramid

Our Web framework.

pyramid_dogpile_cache

Use dogpile from Pyramid

pyramid_jinja2

Use Jinja2 from Pyramid

pyramid_mailer

Send emails from within a Pyramid transaction. The email will only be sent if the transaction succeeds. Used for invitations and password resets in :py:mod:`assembl.views.auth.views` until we refactor this with notifications.

pyramid_beaker

Use Beaker sessions from Pyramid

PyStemmer

The snowball stemmer will convert words to their stem, erasing flexions. Used in :py:mod:`assembl.nlp`

python_social_auth

Allows us to use social network identities for single-sign-on

pytz

Convert local time to UTC and back.

pyzmq

Use the ZeroMQ messaging bus from python. Used for the :py:mod:`assembl.processes.changes_router`.

sentry-sdk

Send error reports to Sentry.

RDFLib

Library to handle RDF data, used for Catalyst interoperability.

rdflib_jsonld

Formats RDF data as JSON-LD.

requests

Obtain data from a URL

scikit_learn

Machine learning algorithms, used in :py:mod:`assembl.nlp.cluster`

simplejson

Parse JSON data as Python objects and vice-versa

six

Abstract some of the differences between Python2 and Python3

sockjs_tornado

Serve a websocket connection using Tornado. Used by the :py:mod:`assembl.processes.changes_router`.

SQLAlchemy

The Object-Relational Mapper; allows us to use Python classes backed by database storage.

tornado

Another Web server, used by the :py:mod:`assembl.processes.changes_router`.

transaction

An abstraction for atomic transactions, most Pyramid web requests are wrapped in a transaction.

zope_interface

Allows to define interfaces (aka protocols, i.e. purely abstract classes), and retrieve a concrete class that fulfills this protocol by configuration. Used in the :py:mod:`assembl.lib.model_watcher`.

pytest

Our main testing framework. One feature we use a lot is pytest fixtures, in :py:mod:`assembl.tests.fixtures`.

Mocha

Our frontend testing framework

Chai

BDD and TDD assertions in front-end tests

coverage

Verify the coverage of back-end tests

mock

Mock objects and monkey-patch python modules.

flaky

Allows to define some tests as flaky, i.e. attempt them a few times before declaring failure.

WebTest

Encapsulate the Pyramid WSGI server for testing purposes.

ipython

Improved Python shell

pyramid_debugtoolbar

Debug toolbar in the web page

pyramid_debugtoolbar_ajax

Monitor AJAX calls in the debug toolbar

pyramid_ipython

Access Pyramid from ipython

sqltap

Monitor SQLAlchemy requests from the web application

uwsgitop

Monitor UWSGI

flower

Monitor Celery processes

PdbSublimeTextSupport

Shows debug code location in Sublime Text

waitress

A simple WSGI web server for development use. (The pyramid pserve command uses this.)

We have chosen Pyramid as the main Web application server. It is both quite complete, more so than some micro-frameworks; and quite configurable, less opinonated than some other frameworks. Notably, it allows either a classical, route-based URL configuration and a data-driven "traversal API"; Assembl uses both in hybrid app mode.

In particular, application web pages are defined along classical URL routes; and we have three APIs for data access.

1. :ref:`classical_api`, which allows a stable, well-optimized API.
2. The :ref:`magic_api`, which allows for an API that is always up-to-date with the data model
3. Linked Open Data (currently deactivated) in JSON-LD based on the traversal API

Sessions are handled by Beaker with a Memcached backing, and authentication with social services by Python Social Auth.

The data model is expressed as ORM objects using SQLAlchemy, which is the most popular ORM in Python. Migrations are done with Alembic. SQLAlchemy allows for very fine control over SQL queries, expressed in Python. The ORM allows many models of mapping class inheritance to database tables; we mostly use the joined table inheritance pattern. We use introspection extensively to mediate between the JSON representations in the API and the data model.

We also add metadata to the ORM model to map it to a RDF model (using RDFLib), which was historically done with Virtuoso's Linked Data Views, by our Virtuoso-python module. (Currently deactivated.)

The issue of data persistence in Assembl has a long history, which is still being written.

Because of recursive queries on the graph data structure of ideas, we were attracted by graph models. Also, future applications would benefit from a deductive database. Finally, considerations of interoperability with the broader scientific community make us favour Linked Open Data as a primary data publication model. All those considerations point in the direction of a Semantic database.

On the other hand, relational databases are more mature and robust, have better tooling, more developers are familiar with them, and most important good RDBMses can enforce data intergrity constraints. The first development team at Caravan choose to use Postgres in 2012, a traditional RDBMS, for all those reasons, and and because it has good support for transitive closure using Common Table Expressions.

Nonetheless, handling complex recursive queries in this first system introduced a fair amount of complexity, and the prospect of more complex queries to come made us attempt to reconcile this initial design with the semantic perspective using a hybrid database, OpenLink Virtuoso (open-source edition) in 2014, which allowed both a relational model and Linked Data Views over those models. The Virtuoso-SQLAlchemy driver was dependent on a forked version of PyODBC. However, we had issues with data integrity and data corruption, and we abandoned that solution in 2016.

Thanks to SQLAlchemy's abstraction layer, it was possible to port our codebase back to Postgres for data storage, and we abstracted the complexity of some of the queries by pushing part of the more complex calculations in the application layer (in :py:mod:`assembl.models.path_utils`.)

This was a setback for our Linked Open Data strategy, and does not solve the issue of deductive capacity, and we are now considering the option of moving towards a polyglot persistence model.

The Websocket mentioned previously is served by an independent process using sockjs_tornado (forked) and Tornado. That server receives changed models as JSON, transmitted through ZeroMQ. Changes are detected upon database commit by a SQLAlchemy event handler.

Some other operations are asynchronous and handled by processes. Those that are operations that run and terminate, such as email notifications, are modeled as Celery processes, using Kombu and Redis for communication. Long-standing processes, such as IMAP clients, use Kombu directly.

In the frontend, we have stuck to traditional Javascript (as opposed to various languages that compile to javascript) to maintain ease of debugging.

The frontend stack was originally developed in Backbone in 2012, because it allowed a transparent mapping of CRUD operations between the backend and frontend. Many of the more popular choices now (Angular, ReactJS, etc.) were not mature at that point, and later on their advantages were not deemed worth the cost of a transition. We have since added the Marionette layer above Backbone, which gives us a richer MVC model in the frontend.

Furthermore, we are currently embarking on a redesign of the entire front-end architecture! The initial phase of the development is nearly complete. Further information can be found below. See section New Frontend

This is a list of every frontend dependency used directly in the frontend code. Indirect dependencies are not listed. test libraries are described in the backend :ref:`backend_test_libraries` section.

Essentially, this comes from :file:`package.json` and :file:`bower.json`, as well as the content of js/app/lib

Underscore

The basis of backbone, but also used extensively in assembl code. It is a very useful Javascript utility library

Backbone

An unopinionated Model-Collection-View Javascript framework

Marionette

A thicker framework built on top of backbone.

Annotator

A fundamental dependency of Assembl, included in the git of Assembl in js/lib.

Actually generated from our own fork of annotator available at https://github.com/assembl/annotator

Backbone Subset

A backbone model to allow manipulating subsets of collections anywhere one can use a collection.

Backbone Modal

A backbone view extension to implement modal interface elements in a backbone or marionette frontend.

Ink

Allows generating responsive emails from templates. We intend to replace it with Foundation for Emails 2, by the same authors.

jquery-oembed-all

A jquery plugin to implement the oembed protocol completely client side. Allow embeedin g or previewing arbitrary URLs without hitting the backend.

Note: As of 2016-06-29, we use our own fork of jquery-oembed-all, with several new features

jquery.dotdotdot

A jquery plugin that allows implementing a "Read More" interface depending on the height of an element.

Note: As of 2016-06-29, we use our own fork of jquery.dotdotdot, to work around a bug. It should be possible to go back now that this is almost totally abstracted out in CKEditorField

jquery-autosize

A jquery plugin that allows textarea to automatically expand as the user types. Used in the message editor.

bootstrap_growl

Jquery plugin used to implement "Growl-like" notifications

CKEditor

For WYSIWYG editing in various parts of the interface

Hopscotch

A framework to build guided tours in one page applications

Bluebird

Used to implement promises, which we use extensively to deal with asynchronicity in the frontend

D3

Used to implement data visualisations

Jed

Used to implement gettext api in the frontend

Moment

Date calculation and logalized textual display in javascript

raven_js

Used to send client-side errors to Sentry in production (assembl v1)

sentry-javascript

Used to send client-side errors to Sentry in production (assembl v2)

sockjs_client

Used for websocket communication on the frontend

linkifyjs

Used to highlight hyperlinks in text-only content. Used in the messagelist

Bootstrap dropdown

Used in the messagelist header and attachment view to implement dropdown. Included in Assembl source code in js/lib/bootstrap-dropdown.js. Deprecated.

Bootstrap tooltip

Used everywhere to implement tooltips

Included in Assembl source code in js/lib/bootstrap-tooltip.js.

A set of widgets were developed later as part of the Catalyst project; because they were supposed to be generic to the Catalyst ecosystem and independent of Assembl, they were deliberately developed using a different codebase, namely Angular.

We are dealing with asynchronicity in the frontend at two levels:

First, we want to update the UI whenever the user takes an action, without blocking on backend requests. The solution has been to use Promises (aka futures) for most API calls, abstracted with the Bluebird library.

Second, since the Assembl data model is built collectively, we push all model changes from the backend to the frontend through a websocket using SockJS, so that Assembl participants get live updates of data.

The new front-end will be based on Facebook technologies, as we feel that this is the technology that the market is converging towards. We use React, alongside its ecosystem of tooling. Furthermore, we have made the decision to migrate away from RESTful APIs in favour of new GraphQL methodologies as much of our data can be labeled in graph form.

React:

The underbelly of our new front-end. We use component based approach in order to simplify the development and comprehension of views throughout Assembl.

React Router:

The defacto router for React. We use declarative approach to match routes to views, and to maintain a history stack.

React Bootstrap:

The Assembl Styleguide is based strongly on this package, as it converts typically Bootstrap related CSS/JS items into React-based components.

Redux:

The popular Flux implementation that is often coupled with React. Our state management library for components

Apollo:

A popular and future-thinking GraphQL library to manage calls to our graphQL server as well as managing the state of responses within components. It is a Provider-wrapper around Redux.

lodash:

The lightweight alternative to Underscore.

BabelJS:

The new codebase will utilize the latest in ECMAScript 6 standard. Babel will be used to transpile the modern code to currently compatible javascript versions used by popular browsers today.

Webpack:

The new front-end's asset management library will be Webpack instead of Gulp. Webpack offers more flexibility in managing the pipleline and the modern front-end stack is converging to this tool.

ESLint:

The new front-end will use AirBnb as it's Javascript coding standard. As a result, this linter is chosen in order for all developers to be on the same page in terms of development.

Flow:

The new front-end uses flow for static type checking. The linting rules used for flow types is eslint-plugin-flowtype with its recommended configuration.

There are not many changes in the backend for the new front-end, technology wise. The notable exception is the migration away from RESTful APIs in Pyramid towards GraphQL.

Graphene:

The GraphQL implementation for Python that is widely used.

Graphene SQLAlchemy:

A further wrapper for Graphene that will enable the use SQLAlchemy models to be used to generate a schema for GraphQL.

Assembl is unusual in that the template files are double-compiled. They are first compiled by Jinja2 and subsquently by Underscore templates.

We use SASS for CSS generation, using Libsass http://sass-lang.com/libsass and Bourbon

Bourbon

A mixin library for Sass, to avoid vendor prefixes and the like

Grid system:

TODO

Styleguide:

TODO

Fabric:

Used as main orchestrator of the build. To see available commands, type fab --list for the assembl virtualenv

Pip:

Used for python package management

Setuptools:

Used for python package building

npm:

User for Javascript package management

Bower:

Javascript package management (being phased out in favor of npm)

Gulp:

Used for Javascript and Sass code generation. Configured in :file:`assembl/gulpfile.js`

Assembl uses a gettext style translation pipeline. It goes through the pyramid machinery to extract the stransatable strings, including the strings from the frontend in fab -c assembl/configs/develop.rc make_messages

Python gettext https://docs.python.org/2/library/gettext.html ,

This is setup in setup.cfg and message-extraction.ini

The po files are subsequently converted to JSON usable from Jed in the frontend. This happens in po2json.py called by fab -c assembl/configs/develop.rc compile_messages

See :doc:`localization`

Administrative tasks are executed both remotely and locally through Fabric. The various processes that constitute the backend are kept running by Supervisord. Those two components are also the ones that are blocking the Python 3 transition.

Supervisord

Sentry

Piwik

Borg Backup

See :doc:`backups`

We use Nginx as a web server. Nginx is efficient, and could serve the static assets with automatic compression. Nginx talks to Pyramid through the uWSGI. We have used multiple processes rather than threads.