Data interdependence in the frontend¶
IdeaLoom is characterized by highly interdependent data, and allowing for live updates incurs significant complexity. The most complex issues revolve around the Idea-Post relationship, where each idea is connected to a subset of all posts. Here is how this relationship is handled as of 2016/08.
In general, we have tended to load model collections in the frontend, in the
CollectionManager(). Those collections can be kept up-to-date with the WebSocket (see
assembl.lib.sqla.before_commit_listener()), and views can listen to backbone add/update events normally. However, various linking objects are not handled by the
CollectionManager(), and the information is put in the objects that are linked.
If the websocket is disconnected, we need to re-fetch the model collections; this is currently only implemented for the message structure collection. See
First, these are collections that the are maintained up-to-date through the WebSocket:
The collection of all extracts (loaded with the page)
The collection of all participants (loaded with the page)
The collection of all Ideas, each of which knows its parent and children, and the number of total and read posts. (loaded at application start.)
The collection of Post structure, i.e. their Id, type and parentId. (loaded when the message panel is first displayed.)
When the message panel is loaded, the full information for the posts that will be displayed is loaded, including the list of applicable
IdeaContentLinks for each post. We do not expect the frontend to be able to load all the post information in memory. New and modified messages are pushed, even if irrelevant in the frontend view; but those events are deemed rare enough that we can accept the added memory pressure. (We load the full structure so we can show threading structure and levels appropriately, even taking out-of-scope messages into account.)
When any of the following is created or modified, the backend broadcasts the data to the frontend:
Idea are added or modified, the frontend will update live, currently at the cost of some interface jerkiness.
Post are added, the frontend will generally not update live, but warn the user that new messages have come, and offer to reload the view. (This was done to avoid frequent loss of context, but could be improved.) Message counts are updated on the fly through sending ideas on the socket in
Similarly, adding or editing an
Extract will change message counts on the fly, but will not update the message panel. This is again done by sending ideas on the socket, in
The following links are handled indirectly, and link modification currently requires re-sending at least one of the objects linked.
Extract.) We do not keep a global collection, but we keep an ad-hoc per-Post collection (called
getIdeaContentLinkCollectionOnMessage().) As a consequence, new top posts will not be seen in the MessageList, but they will encourage a reload due to being new messages. (in
showPendingMessages()) (Again, related message count will update.) As for
assembl.models.idea_content_link.IdeaContentNegativeLink(thread breaks), they not yet handled.
assembl.models.action.ViewPostwill send back updated post count information for all affected
Idea(the socket is not involved: see
assembl.models.idea.IdeaLinkare expressed in ideas with the
numChildIdeaattributes. Changes cause all affected ideas to be sent to the socket. (see
assembl.models.idea_graph_view.SubGraphIdeaLinkAssociationare respectively expressed in the
ExplicitSubGraphViewobjects (such as the
Synthesis). Changes trigger sending the view on the socket: see
Our state is kept consistent by a WebSocket triggering Backbone global model events, which our views watch. This is fine for the main objects, but the links and counters require finer, value-level changes. It would be nice to be able to send those on the WebSocket. (It is already possible to send a Websocket message for a specific user.)
In the short run:
We need to leave the Idea-centric view and keep a separate collection of
IdeaLink. This will make for more focused view updates: the table of idea’s children collection could only watch the
IdeaLink collection. Similarly, we might want to keep a collection of
IdeaContentLink instead of only
In most cases, views watch generic changes in the idea and update globally; it would be nice to have more specific bindings.
For example, we update message counts in the table of ideas, but the underlying dependencies are not updated in the
We may want to not load the entire message structure someday, but then the backend would need another means to calculate the hierarchy (this may not be so difficult.)
we have chosen to not update the
MessageList() view to decrease changes, but that should be revisited. It would be nice to give an idea of where incoming messages would load (whether new messages or newly classified messages.) In the meantime, we should detect when a new message would not even appear in the current view, and not invite a reload.
We may want to separate changes to the data from changes to the various item counts (posts, actions, etc.) associated with an idea, so those could update independently. Some of those changes in derived data are costly to compute, and should be left to a separate (micro-)service. We could also store standing queries in those services. (as in Data Stream Management Systems or (better) Complex Event Processing.)