The IdeaLoom technology stack

These are some of the main elements of the stack used by IdeaLoom. Some of these choices are not always the most mainstream, but were meanstream at the start of the project in mid-2012.



We use the Python language for backend development, because of its flexibility, dynamism and rich ecosystem. Because the project aims to explore new practices, we have given priority to ease of development over absolute performance.

The project was started on Python 2.7, and some components are still holding us back from the transition to Python 3.

Packages and libraries we use directly

This is a list of every backend dependency used directly in IdeaLoom code. The packages you would need to know about first are described in subsequent paragraphs.

This comes from requirements.txt, but indirect dependencies are not listed.

Database schema migration
JSON parsing
localization and internationalization: extract strings from source files and compile translated files.
Web sessions (uses Memcached)
HTML parsing
Tasks running in another process
Data validation (not used)
RESTful APIs in Pyramid. Used for the Cornice RESTful services
Allows to cache and share the results of a long calculation. Was used for CIF caching.
Python3 enums backported to Python2
Facebook SDK, allows us to query facebook, post messages to it and retrieve comments.
Used by assembl.models.feed_parsing.FeedSourceReader to read Atom feeds.
Used by assembl.nlp.cluster to find clusters of similar posts.
Used in assembl.nlp.translation_service.GoogleTranslationService to communicate with the google translation service and translate messages
Used to retrieve messages from an IMAP server, through the assembl.tasks.imap celery process or the assembl.tasks.imaplib2_source_reader.IMAPReader.
Translate between ISO-639-1 and ISO-639-2 language codes.
Parse dates in ISO-8601 format to python datetime objects.
Our main templating mechanism.
Used to deduce thread order of imported mail messages
Inter-process messaging. Uses Redis. Used by Celery and directly by the assembl.tasks.source_reader
Guess the original language of a message. Used in assembl.nlp.translation_service
Parse XML directly. Also used by beautifulsoup4.
Standardized NodeJS environment, coordinated with Virtualenv
flatten the CSS of HTML, for sending as email in notifications.
Connect to Postgres database
Create graph images with Graphviz. Used in assembl.views.api2.discussion.as_mind_map().
Validate whether an input value is a valid email.
JSON web tokens, allows to use Annotator from a bookmarklet.
Our Web framework.
Use dogpile from Pyramid
Use Jinja2 from Pyramid
Send emails from within a Pyramid transaction. The email will only be sent if the transaction succeeds. Used for invitations and password resets in assembl.views.auth.views until we refactor this with notifications.
Use Beaker sessions from Pyramid
The snowball stemmer will convert words to their stem, erasing flexions. Used in assembl.nlp
Allows us to use social network identities for single-sign-on
Convert local time to UTC and back.
Use the ZeroMQ messaging bus from python. Used for the assembl.tasks.changes_router.
Send error reports to Sentry.
Library to handle RDF data, used for Catalyst interoperability.
Formats RDF data as JSON-LD.
Obtain data from a URL
Machine learning algorithms, used in assembl.nlp.cluster
Parse JSON data as Python objects and vice-versa
Abstract some of the differences between Python2 and Python3
Serve a websocket connection using Tornado. Used by the assembl.tasks.changes_router.
The Object-Relational Mapper; allows us to use Python classes backed by database storage.
Another Web server, used by the assembl.tasks.changes_router.
An abstraction for atomic transactions, most Pyramid web requests are wrapped in a transaction.
Allows to define interfaces (aka protocols, i.e. purely abstract classes), and retrieve a concrete class that fulfills this protocol by configuration. Used in the assembl.lib.model_watcher.


Our main testing framework. One feature we use a lot is pytest fixtures, in assembl.tests.fixtures. Front-end tests are run by the backend, thanks to splinter.
Our frontend testing framework
BDD and TDD assertions in front-end tests
Verify the coverage of back-end tests
Run the Mocha tests from pytest and collect test results.
Mock objects and monkey-patch python modules.
Drive the PhantomJS web browser through selenium for front-end tests.
Allows to define some tests as flaky, i.e. attempt them a few times before declaring failure.
Encapsulate the Pyramid WSGI server for testing purposes.


Improved Python shell
Debug toolbar in the web page
Monitor AJAX calls in the debug toolbar
Access Pyramid from ipython
Monitor SQLAlchemy requests from the web application
Monitor UWSGI
Monitor Celery tasks
Shows debug code location in Sublime Text
A simple WSGI web server for development use. (The pyramid pserve command uses this.)


We have chosen Pyramid as the main Web application server. It is both quite complete, more so than some micro-frameworks; and quite configurable, less opinonated than some other frameworks. Notably, it allows either a classical, route-based URL configuration and a data-driven “traversal API”; IdeaLoom uses both in hybrid app mode.

In particular, application web pages are defined along classical URL routes; and we have three APIs for data access.

  1. Cornice RESTful services, which allows a stable, well-optimized API.
  2. The Traversal API, which allows for an API that is always up-to-date with the data model
  3. Linked Open Data (currently deactivated) in JSON-LD based on the traversal API

Sessions are handled by Beaker with a Memcached backing, and authentication with social services by Python Social Auth.


The data model is expressed as ORM objects using SQLAlchemy, which is the most popular ORM in Python. Migrations are done with Alembic. SQLAlchemy allows for very fine control over SQL queries, expressed in Python. The ORM allows many models of mapping class inheritance to database tables; we mostly use the joined table inheritance pattern. We use introspection extensively to mediate between the JSON representations in the API and the data model.

We also add metadata to the ORM model to map it to a RDF model (using RDFLib), which was historically done with Virtuoso’s Linked Data Views, by our Virtuoso-python module. (Currently deactivated.)

Database layer

The issue of data persistence in IdeaLoom has a long history, which is still being written.

Because of recursive queries on the graph data structure of ideas, we were attracted by graph models. Also, future applications would benefit from a deductive database. Finally, considerations of interoperability with the broader scientific community make us favour Linked Open Data as a primary data publication model. All those considerations point in the direction of a Semantic database.

On the other hand, relational databases are more mature and robust, have better tooling, more developers are familiar with them, and most important good RDBMses can enforce data intergrity constraints. The first development team at Caravan choose to use Postgres in 2012, a traditional RDBMS, for all those reasons, and and because it has good support for transitive closure using Common Table Expressions.

Nonetheless, handling complex recursive queries in this first system introduced a fair amount of complexity, and the prospect of more complex queries to come made us attempt to reconcile this initial design with the semantic perspective using a hybrid database, OpenLink Virtuoso (open-source edition) in 2014, which allowed both a relational model and Linked Data Views over those models. The Virtuoso-SQLAlchemy driver was dependent on a forked version of PyODBC. However, we had issues with data integrity and data corruption, and we abandoned that solution in 2016.

Thanks to SQLAlchemy’s abstraction layer, it was possible to port our codebase back to Postgres for data storage, and we abstracted the complexity of some of the queries by pushing part of the more complex calculations in the application layer (in assembl.models.path_utils.)

This was a setback for our Linked Open Data strategy, and does not solve the issue of deductive capacity, and we are now considering the option of moving towards a polyglot persistence model.

Asynchronicity in the backend

The Websocket mentioned previously is served by an independent process using sockjs_tornado (forked) and Tornado. That server receives changed models as JSON, transmitted through ZeroMQ. Changes are detected upon database commit by a SQLAlchemy event handler.

Some other operations are asynchronous and handled by processes. Those that are operations that run and terminate, such as email notifications, are modeled as Celery tasks, using Kombu and Redis for communication. Long-standing tasks, such as IMAP clients, use Kombu directly.




In the frontend, we have stuck to traditional Javascript (as opposed to various languages that compile to javascript) to maintain ease of debugging.

The frontend stack was originally developed in Backbone in 2012, because it allowed a transparent mapping of CRUD operations between the backend and frontend. Many of the more popular choices now (Angular, ReactJS, etc.) were not mature at that point, and later on their advantages were not deemed worth the cost of a transition. We have since added the Marionette layer above Backbone, which gives us a richer MVC model in the frontend.

Packages and libraries we use directly

This is a list of every frontend dependency used directly in the frontend code. Indirect dependencies are not listed. test libraries are described in the backend Testing section.

Essentially, this comes from package.json and bower.json, as well as the content of js/app/lib

The basis of backbone, but also used extensively in IdeaLoom code. It is a very useful Javascript utility library
An unopinionated Model-Collection-View Javascript framework
A thicker framework built on top of backbone.

A fundamental dependency of IdeaLoom, included in the git of IdeaLoom in js/lib.

Actually generated from our own fork of annotator available at

Backbone Subset
A backbone model to allow manipulating subsets of collections anywhere one can use a collection.
Backbone Modal
A backbone view extension to implement modal interface elements in a backbone or marionette frontend.
Allows generating responsive emails from templates. We intend to replace it with Foundation for Emails 2, by the same authors.

A jquery plugin to implement the oembed protocol completely client side. Allow embeedin g or previewing arbitrary URLs without hitting the backend.

Note: As of 2016-06-29, we use our own fork of jquery-oembed-all, with several new features


A jquery plugin that allows implementing a “Read More” interface depending on the height of an element.

Note: As of 2016-06-29, we use our own fork of jquery.dotdotdot, to work around a bug. It should be possible to go back now that this is almost totally abstracted out in CKEditorField

A jquery plugin that allows textarea to automatically expand as the user types. Used in the message editor.
Jquery plugin used to implement “Growl-like” notifications
For WYSIWYG editing in various parts of the interface
A framework to build guided tours in one page applications
Used to implement promises, which we use extensively to deal with asynchronicity in the frontend
Used to implement data visualisations
Used to implement gettext api in the frontend
Date calculation and logalized textual display in javascript
Used to send client-side errors to Sentry in production
Used for websocket communication on the frontend
Used to highlight hyperlinks in text-only content. Used in the messagelist
Bootstrap dropdown
Used in the messagelist header and attachment view to implement dropdown. Included in IdeaLoom source code in js/lib/bootstrap-dropdown.js. Deprecated.
Bootstrap tooltip

Used everywhere to implement tooltips

Included in IdeaLoom source code in js/lib/bootstrap-tooltip.js.

Angular for widgets

A set of widgets were developed later as part of the Catalyst project; because they were supposed to be generic to the Catalyst ecosystem and independent of Assembl, they were deliberately developed using a different codebase, namely Angular.

Asynchronicity in the frontend

We are dealing with asynchronicity in the frontend at two levels:

First, we want to update the UI whenever the user takes an action, without blocking on backend requests. The solution has been to use Promises (aka futures) for most API calls, abstracted with the Bluebird library.

Second, since the IdeaLoom data model is built collectively, we push all model changes from the backend to the frontend through a websocket using SockJS, so that IdeaLoom participants get live updates of data.

Templating, HTML and CSS

IdeaLoom is unusual in that the template files are double-compiled. They are first compiled by Jinja2 and subsquently by Underscore templates.


We use SASS for CSS generation, using Libsass and Bourbon

A mixin library for Sass, to avoid vendor prefixes and the like

Grid system:




Build system

Used as main orchestrator of the build. To see available commands, type fab --list for the IdeaLoom virtualenv
Used for python package management
Used for python package building
User for Javascript package management
Javascript package management (being phased out in favor of npm)
Used for Javascript and Sass code generation. Configured in assembl/gulpfile.js


IdeaLoom uses a gettext style translation pipeline. It goes through the pyramid machinery to extract the stransatable strings, including the strings from the frontend in fab -c assembl/configs/develop.rc make_messages

Python gettext ,

This is setup in setup.cfg and message-extraction.ini

The po files are subsequently converted to JSON usable from Jed in the frontend. This happens in called by fab -c assembl/configs/develop.rc compile_messages

See Contributing to IdeaLoom translations


Administrative tasks are executed both remotely and locally through Fabric. The various processes that constitute the backend are kept running by Circus.




Borg Backup
See Setting-up backups for an IdeaLoom server

Web server

We use Nginx as a web server. Nginx is efficient, and could serve the static assets with automatic compression. Nginx talks to Pyramid through the uWSGI. We have used multiple processes rather than threads.