Setting up a NORDUni instance

Work in progress

This guide is written for Ubuntu 10.04.

sudo apt-get install nginx git-core python-virtualenv postfix openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev

Make a virtual python environment.

cd /opt/python_environments/
sudo virtualenv --no-site-packages norduni

Install needed python modules.

sudo su
source python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder gunicorn

Get the NORDUni files:

cd /var/opt/
git clone git://git.nordu.net/norduni.git

Get the NERDS files:

mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

cd /var/opt/norduni/src/niweb/
ln -s /var/opt/python_environments/norduni/ env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni

Change the django settings.

# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py

Change at least the following settings.

NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = '/opt/norduni/dependencies/neo4jdb/'

Here you can end if you want to run a local installation of NOCLook.

Deploying NOCLook

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

urls.py

# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Create a gunicorn start file.

/opt/norduni/src/start_noclook.sh

#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
 --user=$USER --group=$GROUP --log-level=debug \
 --log-file=$LOGFILE 2>>$LOGFILE

Configure nginx.

/etc/nginx/sites-available/default

server {
    listen 80;
    root /opt/norduni/src/niweb;
    server_name ni.example.net;
    access_log /var/log/ni/noclook-access.log;
    error_log /var/log/ni/noclook-error.log;

    location /static/ {
        root   /opt/norduni/src/niweb/;
        autoindex on;
        access_log   off;
        expires      30d; 
    }

    location / {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8000/;
    }   
}

Install supervisord and set up the following start script.

easy_install supervisor
echo_supervisord_conf > /etc/supervisord.conf

/etc/init/supervisord.conf

description     "supervisord"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf

Add the noclook start script to the supervisor configuration.

[program:noclook]
directory = /opt/norduni/src/niweb/
user = user
command = /opt/norduni/src/start_noclook.sh
stdout_logfile = /var/log/ni/supervisor_logfile.log
stderr_logfile = /var/log/ni/supervisor_err_logfile.log

Set up a GIT repository for the JSON files produced by the NERDS producers and make a clone in /var/opt/. NORDUnet has a GIT repository called nistore
and it is cloned to /var/opt/nistore/.

Produce JSON files with the producers you want.

Consume the recently produces JSON files with the NORDUni consume scripts.

Neo4j >1.5 embedded with Python bindings

Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/)

pip install neo4j-embedded
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/jre/
pip install /path/to/jpype-version.zip

Setting up old NOCLook version.

sudo apt-get install apache2 libapache2-mod-wsgi git-core python-virtualenv postfix openjdk-6-jre-headless postgresql python-psycopg2 libpq-dev python-dev

Make a virtual python environment.

cd /opt/python_environments/
sudo virtualenv --no-site-packages norduni

Install needed python modules.

sudo su
source python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder gunicorn

Add the following python modules if you are running the Neo4j stand alone server:

pip install httplib2

Get the NORDUni files:

cd /var/opt/
git clone git://git.nordu.net/norduni.git
git checkout operation

Get the NERDS files:

mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

cd /var/opt/norduni/src/niweb/
ln -s /var/opt/python_environments/norduni/ env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni

Change the django settings.

# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py

Change at least the following settings.

NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = 'http://localhost:7474/db/data/'

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

urls.py

# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Create a file name niweb in /etc/apache2/conf.d/.

WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi

Add niweb path to niweb.wsgi

sys.path.append('/opt/norduni/src/niweb')

Add the following lines to /etc/apache2/sites-available/default-ssl

        Alias /admin_media /var/opt/norduni/src/niweb/static/admin/
        Alias /site_media /var/opt/norduni/src/niweb/static/
        <LocationMatch "^/(admin_media|site_media)">
           Options FollowSymLinks
           Order allow,deny
           allow from all
           SetHandler None
        </LocationMatch>

Neo4j 1.3 REST stand alone server

Download neo4j-1.3 from http://neo4j.org.
Extract the file to, for example, /var/opt/norduni/dependencies/.

cd /var/opt/norduni/dependencies/
tar xfz neo4j-1.3.tar.gz

Copy the startup script from /var/opt/norduni/src/scripts/neo4j/ to /etc/init.d/ and then make neo4j start on boot.

update-rc.d /etc/init.d/neo4j defaults

You need to shut down port 7474 from anything except localhost.

iptables -A INPUT -p tcp --dport 7474 -s localhost -j ACCEPT
iptables -A INPUT -p tcp --dport 7474 -j DROP

Save the rules and add the following line to your interface configuration so you will restore the rules on a reboot.

bash -c "iptables-save > /etc/iptables.rules"

vi /etc/network/interfaces

iface eth0 inet static
    pre-up iptables-restore < /etc/iptables.rules

Apache configuration for neo4j-embedded

If you want to run the latest version if NOCLook (using neo4j-embedded) you have to configure Apache to only use one process to run the Django code.

Install apache2-mpm-prefork and libapache2-mod-wsgi.