The IdeaLoom technology stack

These are some of the main elements of the stack used by IdeaLoom. Some of these choices are not always the most mainstream, but were meanstream at the start of the project in mid-2012.

Backend

Python

We use the Python language for backend development, because of its flexibility, dynamism and rich ecosystem. Because the project aims to explore new practices, we have given priority to ease of development over absolute performance.

The project was started on Python 2.7, and some components are still holding us back from the transition to Python 3.

Packages and libraries we use directly

This is a list of every backend dependency used directly in IdeaLoom code. The packages you would need to know about first are described in subsequent paragraphs.

This comes from requirements.txt, but indirect dependencies are not listed.

Alembic
Database schema migration
anyjson
JSON parsing
Babel
localization and internationalization: extract strings from source files and compile translated files.
Beaker
Web sessions (uses Memcached)
beautifulsoup4
HTML parsing
celery
Tasks running in another process
colander
Data validation (not used)
cornice
RESTful APIs in Pyramid. Used for the Cornice RESTful services
dogpile
Allows to cache and share the results of a long calculation. Was used for CIF caching.
enum34
Python3 enums backported to Python2
facebook_sdk
Facebook SDK, allows us to query facebook, post messages to it and retrieve comments.
feedparser
Used by assembl.models.feed_parsing.FeedSourceReader to read Atom feeds.
gensim
Used by assembl.nlp.cluster to find clusters of similar posts.
google_api_python_client
Used in assembl.nlp.translation_service.GoogleTranslationService to communicate with the google translation service and translate messages
imaplib2
Used to retrieve messages from an IMAP server, through the assembl.tasks.imap celery process or the assembl.tasks.imaplib2_source_reader.IMAPReader.
iso639
Translate between ISO-639-1 and ISO-639-2 language codes.
isodate
Parse dates in ISO-8601 format to python datetime objects.
Jinja2
Our main templating mechanism.
jwzthreading
Used to deduce thread order of imported mail messages
kombu
Inter-process messaging. Uses Redis. Used by Celery and directly by the assembl.tasks.source_reader
langdetect
Guess the original language of a message. Used in assembl.nlp.translation_service
lxml
Parse XML directly. Also used by beautifulsoup4.
nodeenv
Standardized NodeJS environment, coordinated with Virtualenv
premailer
flatten the CSS of HTML, for sending as email in notifications.
psycopg2
Connect to Postgres database
pygraphviz
Create graph images with Graphviz. Used in assembl.views.api2.discussion.as_mind_map().
pyIsEmail
Validate whether an input value is a valid email.
PyJWT
JSON web tokens, allows to use Annotator from a bookmarklet.
Pyramid
Our Web framework.
pyramid_dogpile_cache
Use dogpile from Pyramid
pyramid_jinja2
Use Jinja2 from Pyramid
pyramid_mailer
Send emails from within a Pyramid transaction. The email will only be sent if the transaction succeeds. Used for invitations and password resets in assembl.views.auth.views until we refactor this with notifications.
pyramid_beaker
Use Beaker sessions from Pyramid
PyStemmer
The snowball stemmer will convert words to their stem, erasing flexions. Used in assembl.nlp
python_social_auth
Allows us to use social network identities for single-sign-on
pytz
Convert local time to UTC and back.
pyzmq
Use the ZeroMQ messaging bus from python. Used for the assembl.tasks.changes_router.
raven_py
Send error reports to Sentry.
RDFLib
Library to handle RDF data, used for Catalyst interoperability.
rdflib_jsonld
Formats RDF data as JSON-LD.
requests
Obtain data from a URL
scikit_learn
Machine learning algorithms, used in assembl.nlp.cluster
simplejson
Parse JSON data as Python objects and vice-versa
six
Abstract some of the differences between Python2 and Python3
sockjs_tornado
Serve a websocket connection using Tornado. Used by the assembl.tasks.changes_router.
SQLAlchemy
The Object-Relational Mapper; allows us to use Python classes backed by database storage.
tornado
Another Web server, used by the assembl.tasks.changes_router.
transaction
An abstraction for atomic transactions, most Pyramid web requests are wrapped in a transaction.
zope_interface
Allows to define interfaces (aka protocols, i.e. purely abstract classes), and retrieve a concrete class that fulfills this protocol by configuration. Used in the assembl.lib.model_watcher.

Testing

pytest
Our main testing framework. One feature we use a lot is pytest fixtures, in assembl.tests.fixtures. Front-end tests are run by the backend, thanks to splinter.
Mocha
Our frontend testing framework
Chai
BDD and TDD assertions in front-end tests
coverage
Verify the coverage of back-end tests
jasmine_splinter_runner
Run the Mocha tests from pytest and collect test results.
mock
Mock objects and monkey-patch python modules.
splinter
Drive the PhantomJS web browser through selenium for front-end tests.
flaky
Allows to define some tests as flaky, i.e. attempt them a few times before declaring failure.
WebTest
Encapsulate the Pyramid WSGI server for testing purposes.

Debugging

ipython
Improved Python shell
pyramid_debugtoolbar
Debug toolbar in the web page
pyramid_debugtoolbar_ajax
Monitor AJAX calls in the debug toolbar
pyramid_ipython
Access Pyramid from ipython
sqltap
Monitor SQLAlchemy requests from the web application
uwsgitop
Monitor UWSGI
flower
Monitor Celery tasks
PdbSublimeTextSupport
Shows debug code location in Sublime Text
waitress
A simple WSGI web server for development use. (The pyramid pserve command uses this.)

Pyramid

We have chosen Pyramid as the main Web application server. It is both quite complete, more so than some micro-frameworks; and quite configurable, less opinonated than some other frameworks. Notably, it allows either a classical, route-based URL configuration and a data-driven “traversal API”; IdeaLoom uses both in hybrid app mode.

In particular, application web pages are defined along classical URL routes; and we have three APIs for data access.

  1. Cornice RESTful services, which allows a stable, well-optimized API.
  2. The Traversal API, which allows for an API that is always up-to-date with the data model
  3. Linked Open Data (currently deactivated) in JSON-LD based on the traversal API

Sessions are handled by Beaker with a Memcached backing, and authentication with social services by Python Social Auth.

SQLAlchemy

The data model is expressed as ORM objects using SQLAlchemy, which is the most popular ORM in Python. Migrations are done with Alembic. SQLAlchemy allows for very fine control over SQL queries, expressed in Python. The ORM allows many models of mapping class inheritance to database tables; we mostly use the joined table inheritance pattern. We use introspection extensively to mediate between the JSON representations in the API and the data model.

We also add metadata to the ORM model to map it to a RDF model (using RDFLib), which was historically done with Virtuoso’s Linked Data Views, by our Virtuoso-python module. (Currently deactivated.)

Database layer

The issue of data persistence in IdeaLoom has a long history, which is still being written.

Because of recursive queries on the graph data structure of ideas, we were attracted by graph models. Also, future applications would benefit from a deductive database. Finally, considerations of interoperability with the broader scientific community make us favour Linked Open Data as a primary data publication model. All those considerations point in the direction of a Semantic database.

On the other hand, relational databases are more mature and robust, have better tooling, more developers are familiar with them, and most important good RDBMses can enforce data intergrity constraints. The first development team at Caravan choose to use Postgres in 2012, a traditional RDBMS, for all those reasons, and and because it has good support for transitive closure using Common Table Expressions.

Nonetheless, handling complex recursive queries in this first system introduced a fair amount of complexity, and the prospect of more complex queries to come made us attempt to reconcile this initial design with the semantic perspective using a hybrid database, OpenLink Virtuoso (open-source edition) in 2014, which allowed both a relational model and Linked Data Views over those models. The Virtuoso-SQLAlchemy driver was dependent on a forked version of PyODBC. However, we had issues with data integrity and data corruption, and we abandoned that solution in 2016.

Thanks to SQLAlchemy’s abstraction layer, it was possible to port our codebase back to Postgres for data storage, and we abstracted the complexity of some of the queries by pushing part of the more complex calculations in the application layer (in assembl.models.path_utils.)

This was a setback for our Linked Open Data strategy, and does not solve the issue of deductive capacity, and we are now considering the option of moving towards a polyglot persistence model.

Asynchronicity in the backend

The Websocket mentioned previously is served by an independent process using sockjs_tornado (forked) and Tornado. That server receives changed models as JSON, transmitted through ZeroMQ. Changes are detected upon database commit by a SQLAlchemy event handler.

Some other operations are asynchronous and handled by processes. Those that are operations that run and terminate, such as email notifications, are modeled as Celery tasks, using Kombu and Redis for communication. Long-standing tasks, such as IMAP clients, use Kombu directly.

Frontend

Overview

Javascript

In the frontend, we have stuck to traditional Javascript (as opposed to various languages that compile to javascript) to maintain ease of debugging.

The frontend stack was originally developed in Backbone in 2012, because it allowed a transparent mapping of CRUD operations between the backend and frontend. Many of the more popular choices now (Angular, ReactJS, etc.) were not mature at that point, and later on their advantages were not deemed worth the cost of a transition. We have since added the Marionette layer above Backbone, which gives us a richer MVC model in the frontend.

Packages and libraries we use directly

This is a list of every frontend dependency used directly in the frontend code. Indirect dependencies are not listed. test libraries are described in the backend Testing section.

Essentially, this comes from package.json and bower.json, as well as the content of js/app/lib

Underscore
The basis of backbone, but also used extensively in IdeaLoom code. It is a very useful Javascript utility library
Backbone
An unopinionated Model-Collection-View Javascript framework
Marionette
A thicker framework built on top of backbone.
Annotator

A fundamental dependency of IdeaLoom, included in the git of IdeaLoom in js/lib.

Actually generated from our own fork of annotator available at https://github.com/assembl/annotator

Backbone Subset
A backbone model to allow manipulating subsets of collections anywhere one can use a collection.
Backbone Modal
A backbone view extension to implement modal interface elements in a backbone or marionette frontend.
Ink
Allows generating responsive emails from templates. We intend to replace it with Foundation for Emails 2, by the same authors.
jquery-oembed-all

A jquery plugin to implement the oembed protocol completely client side. Allow embeedin g or previewing arbitrary URLs without hitting the backend.

Note: As of 2016-06-29, we use our own fork of jquery-oembed-all, with several new features

jquery.dotdotdot

A jquery plugin that allows implementing a “Read More” interface depending on the height of an element.

Note: As of 2016-06-29, we use our own fork of jquery.dotdotdot, to work around a bug. It should be possible to go back now that this is almost totally abstracted out in CKEditorField

jquery-autosize
A jquery plugin that allows textarea to automatically expand as the user types. Used in the message editor.
bootstrap_growl
Jquery plugin used to implement “Growl-like” notifications
CKEditor
For WYSIWYG editing in various parts of the interface
Hopscotch
A framework to build guided tours in one page applications
Bluebird
Used to implement promises, which we use extensively to deal with asynchronicity in the frontend
D3
Used to implement data visualisations
Jed
Used to implement gettext api in the frontend
Moment
Date calculation and logalized textual display in javascript
raven_js
Used to send client-side errors to Sentry in production
sockjs_client
Used for websocket communication on the frontend
linkifyjs
Used to highlight hyperlinks in text-only content. Used in the messagelist
Bootstrap dropdown
Used in the messagelist header and attachment view to implement dropdown. Included in IdeaLoom source code in js/lib/bootstrap-dropdown.js. Deprecated.
Bootstrap tooltip

Used everywhere to implement tooltips

Included in IdeaLoom source code in js/lib/bootstrap-tooltip.js.

Angular for widgets

A set of widgets were developed later as part of the Catalyst project; because they were supposed to be generic to the Catalyst ecosystem and independent of Assembl, they were deliberately developed using a different codebase, namely Angular.

Asynchronicity in the frontend

We are dealing with asynchronicity in the frontend at two levels:

First, we want to update the UI whenever the user takes an action, without blocking on backend requests. The solution has been to use Promises (aka futures) for most API calls, abstracted with the Bluebird library.

Second, since the IdeaLoom data model is built collectively, we push all model changes from the backend to the frontend through a websocket using SockJS, so that IdeaLoom participants get live updates of data.

Templating, HTML and CSS

IdeaLoom is unusual in that the template files are double-compiled. They are first compiled by Jinja2 and subsquently by Underscore templates.

SASS

We use SASS for CSS generation, using Libsass http://sass-lang.com/libsass and Bourbon

Bourbon
A mixin library for Sass, to avoid vendor prefixes and the like

Grid system:

TODO

Styleguide:

TODO

Build system

Fabric:
Used as main orchestrator of the build. To see available commands, type fab --list for the IdeaLoom virtualenv
Pip:
Used for python package management
Setuptools:
Used for python package building
npm:
User for Javascript package management
Bower:
Javascript package management (being phased out in favor of npm)
Gulp:
Used for Javascript and Sass code generation. Configured in assembl/gulpfile.js

Translation

IdeaLoom uses a gettext style translation pipeline. It goes through the pyramid machinery to extract the stransatable strings, including the strings from the frontend in fab -c assembl/configs/develop.rc make_messages

Python gettext https://docs.python.org/2/library/gettext.html ,

This is setup in setup.cfg and message-extraction.ini

The po files are subsequently converted to JSON usable from Jed in the frontend. This happens in po2json.py called by fab -c assembl/configs/develop.rc compile_messages

See Contributing to IdeaLoom translations

Operations

Administrative tasks are executed both remotely and locally through Fabric. The various processes that constitute the backend are kept running by Circus.

Circus

Sentry

Piwik

Borg Backup
See Setting-up backups for an IdeaLoom server

Web server

We use Nginx as a web server. Nginx is efficient, and could serve the static assets with automatic compression. Nginx talks to Pyramid through the uWSGI. We have used multiple processes rather than threads.