The IdeaLoom technology stack¶
These are some of the main elements of the stack used by IdeaLoom. Some of these choices are not always the most mainstream, but were meanstream at the start of the project in mid-2012.
We use the Python language for backend development, because of its flexibility, dynamism and rich ecosystem. Because the project aims to explore new practices, we have given priority to ease of development over absolute performance.
The project was started on Python 2.7, and some components are still holding us back from the transition to Python 3.
Packages and libraries we use directly¶
This is a list of every backend dependency used directly in IdeaLoom code. The packages you would need to know about first are described in subsequent paragraphs.
This comes from
requirements.txt, but indirect dependencies are not listed.
Database schema migration
localization and internationalization: extract strings from source files and compile translated files.
Web sessions (uses Memcached)
Tasks running in another process
Data validation (not used)
Allows to cache and share the results of a long calculation. Was used for CIF caching.
Python3 enums backported to Python2
Facebook SDK, allows us to query facebook, post messages to it and retrieve comments.
assembl.models.feed_parsing.FeedSourceReaderto read Atom feeds.
assembl.nlp.clusterto find clusters of similar posts.
assembl.nlp.translation_service.GoogleTranslationServiceto communicate with the google translation service and translate messages
Used to retrieve messages from an IMAP server, through the
assembl.tasks.imapcelery process or the
Translate between ISO-639-1 and ISO-639-2 language codes.
Parse dates in ISO-8601 format to python datetime objects.
Our main templating mechanism.
Used to deduce thread order of imported mail messages
Guess the original language of a message. Used in
Parse XML directly. Also used by beautifulsoup4.
flatten the CSS of HTML, for sending as email in notifications.
Connect to Postgres database
Validate whether an input value is a valid email.
JSON web tokens, allows to use Annotator from a bookmarklet.
Our Web framework.
Send emails from within a Pyramid transaction. The email will only be sent if the transaction succeeds. Used for invitations and password resets in
assembl.views.auth.viewsuntil we refactor this with notifications.
The snowball stemmer will convert words to their stem, erasing flexions. Used in
Allows us to use social network identities for single-sign-on
Convert local time to UTC and back.
Send error reports to Sentry.
Library to handle RDF data, used for Catalyst interoperability.
Formats RDF data as JSON-LD.
Obtain data from a URL
Machine learning algorithms, used in
Parse JSON data as Python objects and vice-versa
Abstract some of the differences between Python2 and Python3
The Object-Relational Mapper; allows us to use Python classes backed by database storage.
Another Web server, used by the
An abstraction for atomic transactions, most Pyramid web requests are wrapped in a transaction.
Allows to define interfaces (aka protocols, i.e. purely abstract classes), and retrieve a concrete class that fulfills this protocol by configuration. Used in the
Our frontend testing framework
BDD and TDD assertions in front-end tests
Verify the coverage of back-end tests
Run the Mocha tests from pytest and collect test results.
Mock objects and monkey-patch python modules.
Allows to define some tests as flaky, i.e. attempt them a few times before declaring failure.
Improved Python shell
Debug toolbar in the web page
Monitor AJAX calls in the debug toolbar
Monitor SQLAlchemy requests from the web application
Monitor Celery tasks
Shows debug code location in Sublime Text
A simple WSGI web server for development use. (The pyramid
pservecommand uses this.)
We have chosen Pyramid as the main Web application server. It is both quite complete, more so than some micro-frameworks; and quite configurable, less opinonated than some other frameworks. Notably, it allows either a classical, route-based URL configuration and a data-driven “traversal API”; IdeaLoom uses both in hybrid app mode.
In particular, application web pages are defined along classical URL routes; and we have three APIs for data access.
Cornice RESTful services, which allows a stable, well-optimized API.
The Traversal API, which allows for an API that is always up-to-date with the data model
Linked Open Data (currently deactivated) in JSON-LD based on the traversal API
The data model is expressed as ORM objects using SQLAlchemy, which is the most popular ORM in Python. Migrations are done with Alembic. SQLAlchemy allows for very fine control over SQL queries, expressed in Python. The ORM allows many models of mapping class inheritance to database tables; we mostly use the joined table inheritance pattern. We use introspection extensively to mediate between the JSON representations in the API and the data model.
The issue of data persistence in IdeaLoom has a long history, which is still being written.
Because of recursive queries on the graph data structure of ideas, we were attracted by graph models. Also, future applications would benefit from a deductive database. Finally, considerations of interoperability with the broader scientific community make us favour Linked Open Data as a primary data publication model. All those considerations point in the direction of a Semantic database.
On the other hand, relational databases are more mature and robust, have better tooling, more developers are familiar with them, and most important good RDBMses can enforce data intergrity constraints. The first development team at Caravan choose to use Postgres in 2012, a traditional RDBMS, for all those reasons, and and because it has good support for transitive closure using Common Table Expressions.
Nonetheless, handling complex recursive queries in this first system introduced a fair amount of complexity, and the prospect of more complex queries to come made us attempt to reconcile this initial design with the semantic perspective using a hybrid database, OpenLink Virtuoso (open-source edition) in 2014, which allowed both a relational model and Linked Data Views over those models. The Virtuoso-SQLAlchemy driver was dependent on a forked version of PyODBC. However, we had issues with data integrity and data corruption, and we abandoned that solution in 2016.
Thanks to SQLAlchemy’s abstraction layer, it was possible to port our codebase
back to Postgres for data storage, and we abstracted the complexity of some of
the queries by pushing part of the more complex calculations in the application
This was a setback for our Linked Open Data strategy, and does not solve the issue of deductive capacity, and we are now considering the option of moving towards a polyglot persistence model.
Asynchronicity in the backend¶
The Websocket mentioned previously is served by an independent process using sockjs_tornado (forked) and Tornado. That server receives changed models as JSON, transmitted through ZeroMQ. Changes are detected upon database commit by a SQLAlchemy event handler.
Some other operations are asynchronous and handled by processes. Those that are operations that run and terminate, such as email notifications, are modeled as Celery tasks, using Kombu and Redis for communication. Long-standing tasks, such as IMAP clients, use Kombu directly.
The frontend stack was originally developed in Backbone in 2012, because it allowed a transparent mapping of CRUD operations between the backend and frontend. Many of the more popular choices now (Angular, ReactJS, etc.) were not mature at that point, and later on their advantages were not deemed worth the cost of a transition. We have since added the Marionette layer above Backbone, which gives us a richer MVC model in the frontend.
Packages and libraries we use directly¶
This is a list of every frontend dependency used directly in the frontend code. Indirect dependencies are not listed. test libraries are described in the backend Testing section.
Essentially, this comes from
well as the content of js/app/lib
A thicker framework built on top of backbone.
A fundamental dependency of IdeaLoom, included in the git of IdeaLoom in js/lib.
Actually generated from our own fork of annotator available at https://github.com/assembl/annotator
- Backbone Subset
A backbone model to allow manipulating subsets of collections anywhere one can use a collection.
- Backbone Modal
A backbone view extension to implement modal interface elements in a backbone or marionette frontend.
Allows generating responsive emails from templates. We intend to replace it with Foundation for Emails 2, by the same authors.
A jquery plugin to implement the oembed protocol completely client side. Allow embeedin g or previewing arbitrary URLs without hitting the backend.
Note: As of 2016-06-29, we use our own fork of jquery-oembed-all, with several new features
A jquery plugin that allows implementing a “Read More” interface depending on the height of an element.
Note: As of 2016-06-29, we use our own fork of jquery.dotdotdot, to work around a bug. It should be possible to go back now that this is almost totally abstracted out in CKEditorField
A jquery plugin that allows textarea to automatically expand as the user types. Used in the message editor.
Jquery plugin used to implement “Growl-like” notifications
For WYSIWYG editing in various parts of the interface
A framework to build guided tours in one page applications
Used to implement promises, which we use extensively to deal with asynchronicity in the frontend
Used to implement data visualisations
Used to implement gettext api in the frontend
Used to send client-side errors to Sentry in production
Used for websocket communication on the frontend
Used to highlight hyperlinks in text-only content. Used in the messagelist
- Bootstrap dropdown
Used in the messagelist header and attachment view to implement dropdown. Included in IdeaLoom source code in js/lib/bootstrap-dropdown.js. Deprecated.
- Bootstrap tooltip
Used everywhere to implement tooltips
Included in IdeaLoom source code in js/lib/bootstrap-tooltip.js.
Angular for widgets¶
A set of widgets were developed later as part of the Catalyst project; because they were supposed to be generic to the Catalyst ecosystem and independent of Assembl, they were deliberately developed using a different codebase, namely Angular.
Asynchronicity in the frontend¶
We are dealing with asynchronicity in the frontend at two levels:
First, we want to update the UI whenever the user takes an action, without blocking on backend requests. The solution has been to use Promises (aka futures) for most API calls, abstracted with the Bluebird library.
Second, since the IdeaLoom data model is built collectively, we push all model changes from the backend to the frontend through a websocket using SockJS, so that IdeaLoom participants get live updates of data.
Templating, HTML and CSS¶
Used as main orchestrator of the build. To see available commands, type
fab --listfor the IdeaLoom virtualenv
Used for python package management
Used for python package building
IdeaLoom uses a gettext style translation pipeline. It goes through the pyramid
machinery to extract the stransatable strings, including the strings from the
fab -c assembl/configs/develop.rc make_messages
Python gettext https://docs.python.org/2/library/gettext.html ,
This is setup in
The po files are subsequently converted to JSON usable from Jed in the
frontend. This happens in
po2json.py called by
fab -c assembl/configs/develop.rc compile_messages