# Installing IdeaLoom¶

## Prerequisites¶

IdeaLoom has been tested on Ubuntu (esp. xenial and trusty), Debian (jessie and stretch) and Mac OS (10.9 onwards.) We recommend ubuntu xenial or debian stretch for production environments. In what follows, we’ll assume python3.5 is installed on linux environments, and python3.6 on mac environments, but either version is suitable.

• On Mac OS X 10.9.2: The system python is incompatible with the clang 5.1. You need to remove all occurences of -mno-fused-madd in /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py. Also renew (or delete) the corresponding .pyc, .pyo files.
• For production on linux using nginx/uwsgi you need the following ppa - Necessary for both saucy 13.10, raring 13.04, trusty 14.04 - Not needed for vivid 15.04 and later (incl. xenial.)
apt-add-repository ppa:chris-lea/uwsgi
apt-get install nginx uwsgi uwsgi-plugin-python3


### Homebrew on Mac¶

On mac, install Homebrew by following the instructions at http://brew.sh ; or simply paste the following in the terminal:

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go/install)"  Make sure /usr/local/bin is in your$PATH.

### Install fabric and a SSH server¶

You need fabric 1.5.1 and a SSH server installed. Fab should be able to ssh into the target server, even if it’s localhost. Fab will repeatedly ask for the login password, unless you set up ssh-agent on your host and ~/.ssh/authorized_keys on the target server.

Here is how to install the ssh server on Mac and on Ubuntu.

#### On Mac¶

Install the homebrew python and fabric:

brew install python fabric


MacOS has a SSH server installed. To activate it, go to System Preferences in the Apple Menu, then to the Sharing tab. Ensure the “Remote login” checkbox is active.

#### On Ubuntu¶

You can get all that you need to bootstrap with:

apt-get install fabric git openssh-server sudo


### Setting up a production user¶

On a production machine, you would want to create a user account dedicated to idealoom (we’ll call it idealoom_user in what follows). This user will not be sudoer, but will have some specific sudo permissions. We suggest defining a group (idealoom_group) for all idealoom users, in case there are multiple installs. This also assumes that the nginx web server runs in group www-data.

Run this as a sudoer user:

sudo apt-get install nginx uwsgi uwsgi-plugin-python3
sudo usermod -G www-data -G idealoom_group idealoom_user


Note: For a simple single-server setup, it is also possible to use the www-data user directly, and to put idealoom in /var/www.

On a development machine, we assume the idealoom server runs under your own user.

### Postgres¶

By default, postgres will not use passwords from postgres users who connect through the Unix socket domain (versus a network connection). So if you want to make your database to be safer and ask for password anyway, edit your /etc/postgresql/9.5/main/pg_hba.conf file and

# replace
local   all             all                                peer
# by
local   all             all                                md5


and then run

sudo service postgresql restart


## Installing the application¶

First, choose a directory for installation (which we will call application root). If you install from source, it may be the same as the git directory. These commands should be run as the idealoom_user if you created one. (You may use sudo -u idealoom_user -i to do so.)

### Installing from wheel¶

mkdir idealoom
cd idealoom
python3 -mvirtualenv -p /usr/bin/python3 venv
source ./venv/bin/activate
ln -s venv/lib/python3.5/site-packages/idealoom/fabfile.py .


The last step allows fabric to use the fabfile embedded in the package when running from the application root.

### Installing from source¶

git clone https://github.com/conversence/idealoom.git
cd idealoom


#### Ontology Submodule¶

The ontology module is a git submodule. As a result, after pulling in changes, update with the following:

git submodule update --init


## Setting initial parameters¶

You will create a local.rc file in the idealoom project root, which will be based on either base_env.rc (production) or develop.rc or mac.rc (development). The base environment is set in the _extends parameter. So in a basic production environment, it should contain at least:

_extends = base_env.rc
public_hostname = your_hostname


In a development environment, it might be as simple as _extends = develop.rc.

The rc file format is described in the IdeaLoom Configuration files document.

Here are a few more values you should set:

• idealoom_admin_email to your email
• _user to the username of the idealoom process

Some optional fabric commands require sudo privileges; you could do these commands as root, or designate a sudo-capable account as sudoer in the local.rc file. (Avoid making the idealoom_user a sudoer.) Fabric will then login as this sudo-capable user.

### Sentry¶

If you’re using Sentry to monitor, you need to set the following keys, as described in Sentry documentation.

• *sentry_host
• *sentry_key
• *sentry_secret
• *sentry_id

If you’re not using Sentry, you would want to include no_sentry.ini in the ini_files value chain, as described in IdeaLoom Configuration files.

### Postgres¶

You need to set a postgres user for the idealoom database. It is simplest if this postgres user has the same name as the unix idealoom_user account. This database user needs to have create database permissions. This user can be created with the fab check_and_create_database_user command, but this then requires the password of the postgres root account in the postgres_db_password configuration variable. (This account is usually postgres on linux, or the user’s account on mac.)

It is also a good idea to set a different password for the idealoom postgres account.

• *postgres_db_password: optional
• *db_user: usually idealoom_user
• *db_password: set to any value
• *db_database: optional
• *db_host: if different from localhost

### A note on vagrant¶

If you use vagrant, we have a few processes that expect to use socket files in %(here)s. Vagrant does not allow creating sockets in a shared folder; so if you insist on using vagrant, make sure to move sockets locations. Some are defined in circusd.conf.tmpl, and changes.socket is defined in the .ini files.

### Multiple environments¶

If you want to run multiple environments on your machine, some of the configuration parameters in each local.rc must have different values.

The variables that have to be different between instances are the following (for convenience they are marked with UNIQUE_PER_SERVER in base_env.rc and develop.rc):

public_port = 6543
changes_socket = ipc:///tmp/idealoom_changes/0
changes_websocket_port = 8085
redis_socket = 0
webpack_port = 8080
server:main__port = 6543


Most of these are ports, and it should be easy to find an unoccupied port; in the case of changes.socket, you simply need a different filename, and in the case of celery_task.*.broker, the final number has to be changed to another low integer.

The public_port field (located in app:idealoom section) is the actual port used by the UWSGI server which is rerouted through the reverse proxy served by nginx. For production context, use 80. There is also a port field in server:main section, which defaults to 6543. If not proxied by nginx or something, port needs to match public_port.

Also, set the uid field of the uwsgi section of your ini file to the username of the unix user you created above. For example: uid = idealoom_user If you have not added this user to the www-data group as advised previously (or to a group which is common with the ngnix user), then you also have to set the gid field to a common group name.

If you do not have an SSL certificate, then you have to set accept_secure_connection = false and require_secure_connection = false (because if you set accept_secure_connection = true, then the login page on IdeaLoom will try to show using https, which will not work).

The next command installs various components. It must be run as root on linux, or the _sudoer parameter must be set in the local.rc file:

fab -f assembl/fabfile.py -c local.rc install_single_server


You must omit the -f assembl/fabfile.py flag if you have installed from a wheel, as fabric will use the symbolic link. This holds for the next few commands.

Note: If on Mac, command fab -c assembl/configs/develop.rc install_single_server outputs “Low level socket error: connecting to host localhost on port 22: Unable to connect to port 22 on 127.0.0.1”, you have to go to System preferences > Sharing > check “Enable remote login”, and retry the command.

Again as the idealoom_user:

If you’re running from source:

fab -f assembl/fabfile.py -c local.rc bootstrap_from_checkout


If you’re running from wheel:

fab -c local.rc bootstrap_from_wheel


Note: If you get the following error: fabric.exceptions.NetworkError: Incompatible ssh server (no acceptable macs) Then you’ll need to reconfigure your ssh server

## Running¶

Note: postgres, openssl, memcached and redis must be running already.

source venv/bin/activate
circusd circusd.conf


Creating a user the first time you run IdeaLoom (so you have a superuser):

idealoom-add-user --email your_email@email.com --name "Your Name" --username desiredusername --password yourpassword local.ini


Note: Just running $venv/bin/circusd will NOT work, as celery will run command line tools, thus breaking out of the environment. You need to run source venv/bin/activate from the same terminal before running the above Note: If you do not want to source activate every time, you can hook it in your shell using something like Autoenv. Another option is to use VirtualenvWrapper and its Helper. At least one of us uses VirtualFish with auto-activation. On subsequent runs, just make sure circusd is running. ### In development¶ Then, start the development server with this command: env CIRCUSCTL_ENDPOINT=ipc://pwd/var/run/circus_endpoint circusctl start pserve  You can now type http://localhost:6543 in your browser and log in using the credentials you created. ## Final production tasks¶ ### Nginx connection (production)¶ idealoom-ini-files template -o {{idealoom.yourdomain.com}} local.rc nginx_default.jinja2  As root: put that {{idealoom.yourdomain.com}} file in /etc/nginx/sites_available. Activate this site, using: cd /etc/nginx/sites-enabled/ ln -s /etc/nginx/sites-available/{{idealoom.yourdomain.com}} .  Test that your configuration file works, by running: /usr/sbin/nginx -t  Restart nginx: /etc/init.d/nginx restart  ### Securing nginx¶ TODO ### Automating Idealoom startup¶ Copy the content of doc/sample_systemd_script/idealoom.service into /etc/systemd/system/idealoom.service, and modify fields IDEALOOM_PATH, User and Description. systemctl enable idealoom service idealoom restart  ### Mail setup¶ You may set up an external or internal SMTP server (TODO), an external IMAP server (TODO), and Piwik. The How to not be marked as spam document explains how to minimize the risk of email sent by idealoom being identified as spam. The Setting up Dovecot and VMM document explains how to set up an internal IMAP server. ### Backups¶ ## Updating an environment¶ cd ~/idealoom #Any git operations (ex: git pull) fab -c assembl/configs/develop.rc app_compile$venv/bin/circusctl start pserve webpack


You can monitor any of the processes, for example pserve, with these commands:

tail -f var/log/pserve.log
tail -f var/log/pserve.err.log


In production:

#(Instead of dev:*. You may have to stop dev:*)
\$venv/bin/circusctl start uwsgi


Updating an environment after switching branch locally (will regenerate css, all compiled files, update dependencies, database schema, etc.):

fab -c assembl/configs/develop.rc app_compile


Updating an environment to it’s specified branch, tag or revision:

cd ~/idealoom
fab -c assembl/configs/develop.rc app_fullupdate


### Schema migrations¶

alembic -c local.ini upgrade head

alembic -c local.ini revision -m "Your message"