The IdeaLoom technology stack

These are some of the main elements of the stack used by IdeaLoom. Some of these choices are not always the most mainstream, but were meanstream at the start of the project in mid-2012.

Backend

Python

We use the Python language for backend development, because of its flexibility, dynamism and rich ecosystem. Because the project aims to explore new practices, we have given priority to ease of development over absolute performance.

The project was started on Python 2.7, and some components are still holding us back from the transition to Python 3.

Packages and libraries we use directly

This is a list of every backend dependency used directly in IdeaLoom code. The packages you would need to know about first are described in subsequent paragraphs.

This comes from requirements.txt, but indirect dependencies are not listed.

Alembic

Database schema migration

anyjson

JSON parsing

Babel

localization and internationalization: extract strings from source files and compile translated files.

Beaker

Web sessions (uses Memcached)

beautifulsoup4

HTML parsing

celery

Tasks running in another process

colander

Data validation (not used)

cornice

RESTful APIs in Pyramid. Used for the Cornice RESTful services

dogpile

Allows to cache and share the results of a long calculation. Was used for CIF caching.

enum34

Python3 enums backported to Python2

facebook_sdk

Facebook SDK, allows us to query facebook, post messages to it and retrieve comments.

feedparser

Used by assembl.models.feed_parsing.FeedSourceReader to read Atom feeds.

gensim

Used by assembl.nlp.cluster to find clusters of similar posts.

google_api_python_client

Used in assembl.nlp.translation_service.GoogleTranslationService to communicate with the google translation service and translate messages

imaplib2

Used to retrieve messages from an IMAP server, through the assembl.tasks.imap celery process or the assembl.tasks.imaplib2_source_reader.IMAPReader.

iso639

Translate between ISO-639-1 and ISO-639-2 language codes.

isodate

Parse dates in ISO-8601 format to python datetime objects.

Jinja2

Our main templating mechanism.

jwzthreading

Used to deduce thread order of imported mail messages

kombu

Inter-process messaging. Uses Redis. Used by Celery and directly by the assembl.tasks.source_reader

langdetect

Guess the original language of a message. Used in assembl.nlp.translation_service

lxml

Parse XML directly. Also used by beautifulsoup4.

nodeenv

Standardized NodeJS environment, coordinated with Virtualenv

premailer

flatten the CSS of HTML, for sending as email in notifications.

psycopg2

Connect to Postgres database

pygraphviz

Create graph images with Graphviz. Used in assembl.views.api2.discussion.as_mind_map().

pyIsEmail

Validate whether an input value is a valid email.

PyJWT

JSON web tokens, allows to use Annotator from a bookmarklet.

Pyramid

Our Web framework.

pyramid_dogpile_cache

Use dogpile from Pyramid

pyramid_jinja2

Use Jinja2 from Pyramid

pyramid_mailer

Send emails from within a Pyramid transaction. The email will only be sent if the transaction succeeds. Used for invitations and password resets in assembl.views.auth.views until we refactor this with notifications.

pyramid_beaker

Use Beaker sessions from Pyramid

PyStemmer

The snowball stemmer will convert words to their stem, erasing flexions. Used in assembl.nlp

python_social_auth

Allows us to use social network identities for single-sign-on

pytz

Convert local time to UTC and back.

pyzmq

Use the ZeroMQ messaging bus from python. Used for the assembl.tasks.changes_router.

raven_py

Send error reports to Sentry.

RDFLib

Library to handle RDF data, used for Catalyst interoperability.

rdflib_jsonld

Formats RDF data as JSON-LD.

requests

Obtain data from a URL

scikit_learn

Machine learning algorithms, used in assembl.nlp.cluster

simplejson

Parse JSON data as Python objects and vice-versa

six

Abstract some of the differences between Python2 and Python3

sockjs_tornado

Serve a websocket connection using Tornado. Used by the assembl.tasks.changes_router.

SQLAlchemy

The Object-Relational Mapper; allows us to use Python classes backed by database storage.

tornado

Another Web server, used by the assembl.tasks.changes_router.

transaction

An abstraction for atomic transactions, most Pyramid web requests are wrapped in a transaction.

zope_interface

Allows to define interfaces (aka protocols, i.e. purely abstract classes), and retrieve a concrete class that fulfills this protocol by configuration. Used in the assembl.lib.model_watcher.

Testing

pytest

Our main testing framework. One feature we use a lot is pytest fixtures, in assembl.tests.fixtures. Front-end tests are run by the backend, thanks to splinter.

Mocha

Our frontend testing framework

Chai

BDD and TDD assertions in front-end tests

coverage

Verify the coverage of back-end tests

jasmine_splinter_runner

Run the Mocha tests from pytest and collect test results.

mock

Mock objects and monkey-patch python modules.

splinter

Drive the PhantomJS web browser through selenium for front-end tests.

flaky

Allows to define some tests as flaky, i.e. attempt them a few times before declaring failure.

WebTest

Encapsulate the Pyramid WSGI server for testing purposes.

Debugging

ipython

Improved Python shell

pyramid_debugtoolbar

Debug toolbar in the web page

pyramid_debugtoolbar_ajax

Monitor AJAX calls in the debug toolbar

pyramid_ipython

Access Pyramid from ipython

sqltap

Monitor SQLAlchemy requests from the web application

uwsgitop

Monitor UWSGI

flower

Monitor Celery tasks

PdbSublimeTextSupport

Shows debug code location in Sublime Text

waitress

A simple WSGI web server for development use. (The pyramid pserve command uses this.)

Pyramid

We have chosen Pyramid as the main Web application server. It is both quite complete, more so than some micro-frameworks; and quite configurable, less opinonated than some other frameworks. Notably, it allows either a classical, route-based URL configuration and a data-driven “traversal API”; IdeaLoom uses both in hybrid app mode.

In particular, application web pages are defined along classical URL routes; and we have three APIs for data access.

1. Cornice RESTful services, which allows a stable, well-optimized API.

2. The Traversal API, which allows for an API that is always up-to-date with the data model

3. Linked Open Data (currently deactivated) in JSON-LD based on the traversal API

Sessions are handled by Beaker with a Memcached backing, and authentication with social services by Python Social Auth.

SQLAlchemy

The data model is expressed as ORM objects using SQLAlchemy, which is the most popular ORM in Python. Migrations are done with Alembic. SQLAlchemy allows for very fine control over SQL queries, expressed in Python. The ORM allows many models of mapping class inheritance to database tables; we mostly use the joined table inheritance pattern. We use introspection extensively to mediate between the JSON representations in the API and the data model.

We also add metadata to the ORM model to map it to a RDF model (using RDFLib), which was historically done with Virtuoso’s Linked Data Views, by our Virtuoso-python module. (Currently deactivated.)

Database layer

The issue of data persistence in IdeaLoom has a long history, which is still being written.

Because of recursive queries on the graph data structure of ideas, we were attracted by graph models. Also, future applications would benefit from a deductive database. Finally, considerations of interoperability with the broader scientific community make us favour Linked Open Data as a primary data publication model. All those considerations point in the direction of a Semantic database.

On the other hand, relational databases are more mature and robust, have better tooling, more developers are familiar with them, and most important good RDBMses can enforce data intergrity constraints. The first development team at Caravan choose to use Postgres in 2012, a traditional RDBMS, for all those reasons, and and because it has good support for transitive closure using Common Table Expressions.

Nonetheless, handling complex recursive queries in this first system introduced a fair amount of complexity, and the prospect of more complex queries to come made us attempt to reconcile this initial design with the semantic perspective using a hybrid database, OpenLink Virtuoso (open-source edition) in 2014, which allowed both a relational model and Linked Data Views over those models. The Virtuoso-SQLAlchemy driver was dependent on a forked version of PyODBC. However, we had issues with data integrity and data corruption, and we abandoned that solution in 2016.

Thanks to SQLAlchemy’s abstraction layer, it was possible to port our codebase back to Postgres for data storage, and we abstracted the complexity of some of the queries by pushing part of the more complex calculations in the application layer (in assembl.models.path_utils.)

This was a setback for our Linked Open Data strategy, and does not solve the issue of deductive capacity, and we are now considering the option of moving towards a polyglot persistence model.

Asynchronicity in the backend

The Websocket mentioned previously is served by an independent process using sockjs_tornado (forked) and Tornado. That server receives changed models as JSON, transmitted through ZeroMQ. Changes are detected upon database commit by a SQLAlchemy event handler.

Some other operations are asynchronous and handled by processes. Those that are operations that run and terminate, such as email notifications, are modeled as Celery tasks, using Kombu and Redis for communication. Long-standing tasks, such as IMAP clients, use Kombu directly.

Frontend

Overview

Javascript

In the frontend, we have stuck to traditional Javascript (as opposed to various languages that compile to javascript) to maintain ease of debugging.

The frontend stack was originally developed in Backbone in 2012, because it allowed a transparent mapping of CRUD operations between the backend and frontend. Many of the more popular choices now (Angular, ReactJS, etc.) were not mature at that point, and later on their advantages were not deemed worth the cost of a transition. We have since added the Marionette layer above Backbone, which gives us a richer MVC model in the frontend.

Packages and libraries we use directly

This is a list of every frontend dependency used directly in the frontend code. Indirect dependencies are not listed. test libraries are described in the backend Testing section.

Essentially, this comes from package.json and bower.json, as well as the content of js/app/lib

Underscore

The basis of backbone, but also used extensively in IdeaLoom code. It is a very useful Javascript utility library

Backbone

An unopinionated Model-Collection-View Javascript framework

Marionette

A thicker framework built on top of backbone.

Annotator

A fundamental dependency of IdeaLoom, included in the git of IdeaLoom in js/lib.

Actually generated from our own fork of annotator available at https://github.com/assembl/annotator

Backbone Subset

A backbone model to allow manipulating subsets of collections anywhere one can use a collection.

Backbone Modal

A backbone view extension to implement modal interface elements in a backbone or marionette frontend.

Ink

Allows generating responsive emails from templates. We intend to replace it with Foundation for Emails 2, by the same authors.

jquery-oembed-all

A jquery plugin to implement the oembed protocol completely client side. Allow embeedin g or previewing arbitrary URLs without hitting the backend.

Note: As of 2016-06-29, we use our own fork of jquery-oembed-all, with several new features

jquery.dotdotdot

A jquery plugin that allows implementing a “Read More” interface depending on the height of an element.

Note: As of 2016-06-29, we use our own fork of jquery.dotdotdot, to work around a bug. It should be possible to go back now that this is almost totally abstracted out in CKEditorField

jquery-autosize

A jquery plugin that allows textarea to automatically expand as the user types. Used in the message editor.

bootstrap_growl

Jquery plugin used to implement “Growl-like” notifications

CKEditor

For WYSIWYG editing in various parts of the interface

Hopscotch

A framework to build guided tours in one page applications

Bluebird

Used to implement promises, which we use extensively to deal with asynchronicity in the frontend

D3

Used to implement data visualisations

Jed

Used to implement gettext api in the frontend

Moment

Date calculation and logalized textual display in javascript

raven_js

Used to send client-side errors to Sentry in production

sockjs_client

Used for websocket communication on the frontend

linkifyjs

Used to highlight hyperlinks in text-only content. Used in the messagelist

Bootstrap dropdown

Used in the messagelist header and attachment view to implement dropdown. Included in IdeaLoom source code in js/lib/bootstrap-dropdown.js. Deprecated.

Bootstrap tooltip

Used everywhere to implement tooltips

Included in IdeaLoom source code in js/lib/bootstrap-tooltip.js.

Angular for widgets

A set of widgets were developed later as part of the Catalyst project; because they were supposed to be generic to the Catalyst ecosystem and independent of Assembl, they were deliberately developed using a different codebase, namely Angular.

Asynchronicity in the frontend

We are dealing with asynchronicity in the frontend at two levels:

First, we want to update the UI whenever the user takes an action, without blocking on backend requests. The solution has been to use Promises (aka futures) for most API calls, abstracted with the Bluebird library.

Second, since the IdeaLoom data model is built collectively, we push all model changes from the backend to the frontend through a websocket using SockJS, so that IdeaLoom participants get live updates of data.

Templating, HTML and CSS

IdeaLoom is unusual in that the template files are double-compiled. They are first compiled by Jinja2 and subsquently by Underscore templates.

SASS

We use SASS for CSS generation, using Libsass http://sass-lang.com/libsass and Bourbon

Bourbon

A mixin library for Sass, to avoid vendor prefixes and the like

Grid system:

TODO

Styleguide:

TODO

Build system

Fabric:

Used as main orchestrator of the build. To see available commands, type fab --list for the IdeaLoom virtualenv

Pip:

Used for python package management

Setuptools:

Used for python package building

npm:

User for Javascript package management

Bower:

Javascript package management (being phased out in favor of npm)

Gulp:

Used for Javascript and Sass code generation. Configured in assembl/gulpfile.js

Translation

IdeaLoom uses a gettext style translation pipeline. It goes through the pyramid machinery to extract the stransatable strings, including the strings from the frontend in fab -c assembl/configs/develop.rc make_messages

Python gettext https://docs.python.org/2/library/gettext.html ,

This is setup in setup.cfg and message-extraction.ini

The po files are subsequently converted to JSON usable from Jed in the frontend. This happens in po2json.py called by fab -c assembl/configs/develop.rc compile_messages

See Contributing to IdeaLoom translations

Operations

Administrative tasks are executed both remotely and locally through Fabric. The various processes that constitute the backend are kept running by Circus.

Circus

Sentry

Piwik

Borg Backup

See Setting-up backups for an IdeaLoom server

Web server

We use Nginx as a web server. Nginx is efficient, and could serve the static assets with automatic compression. Nginx talks to Pyramid through the uWSGI. We have used multiple processes rather than threads.