Versions Compared

Old Version 3

changes.mady.by.user Johan Lundberg

Saved on

compared with

New Version 4

changes.mady.by.user Johan Lundberg

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

titleWork in progress

 

Table of Contents

Setting up NOCLook

...

Code Block
$ sudo apt-get install python-setuptools git libpq-dev postgresql python-dev postfix nginx uwsgi
$ sudo easy_install pip
$ sudo pip install virtualenv
$ sudo adduser --disabled-password --home /var/opt/norduni ni

# sudo apt-get install git-core python-virtualenv openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev

We are using postgresql  but you can use any SQL database that Django supports. See Django database documentation for other supported SQL databases.

...

Neo4j database

Set password for database user and create a new databaseOracle java is recommended for Neo4j.

Code Block
$ sudo apt-u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni

NORDUni repository

add-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer

Download neo4j-community from http://neo4j.com/download/Get the NORDUni files.

Code Block
$ sudo -u ni -i
$ pwd
tar xvfz neo4j-community-2.1.7-unix.tar.gz
$ sudo mv neo4j-community-2.1.7 /var/opt/norduni.
$ git clone git://git.nordu.net/norduni.git

Python environment

Make a virtual python environment.

Code Block
$ virtualenv norduni_environment

Making a virtual environment is also just a suggestion but it makes it easier to keep your system clean.

Python requirements

Install required python modules.

Code Block
$ . norduni_environment/bin/activate
$ pip install -r norduni/requirements.txt

Django settings

Change the django settings.

Code Block
cd norduni/src/niweb/niweb/
cp generic_settings.py settings.py
vi settings.py

Change at least the following settings.

Code Block
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}

 

Neo4j >1.5 embedded with Python bindings

Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/)

Code Block
pip install neo4j-embedded
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
pip install /path/to/JPype-version.zip
Code Block
# Django Generic Login
(r'^accounts/login/$', 'django.contrib.auth.views.login'),

# Federated login
#(r'^accounts/', include('niweb.apps.fedlogin.urls')),
Code Block
python manage.py syncdb
python manage.py runserver 0.0.0.0:80

Now you should be able surf to your machines ip and see the NOCLook app started.

It is time to collect and insert some data.

Deploying NOCLook

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

Code Block
titleurls.py
# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Install nginx, postfix and gunicorn.

Code Block
sudo apt-get install nginx postfix
pip install gunicorn

Create a gunicorn start file.

...

title/opt/norduni/src/start_noclook.sh

...

sudo ln -s /var/opt/neo4j-community-2.1.7 /var/opt/neo4j-community
$ cd /var/opt/neo4j-community
$ sudo ./bin/neo4j-installer install

Set property keys to auto index in neo4j.

Code Block
$ sudo vi /var/opt/neo4j-community/conf/neo4j.properties

Add or update the following lines.
# Autoindexing

# Enable auto-indexing for nodes, default is false
node_auto_indexing=true

# The node property keys to be auto-indexed, if enabled
node_keys_indexable=name, description, ip_address, ip_addresses, as_number, hostname, hostnames, telenor_tn1_number, nordunet_id, version

# Enable auto-indexing for relationships, default is false
relationship_auto_indexing=true

# The relationship property keys to be auto-indexed, if enabled
relationship_keys_indexable=ip_address

Increase the number of files the neo4j user may concurrently access. A restart is required for the settings to take effect.

Code Block
$ sudo vi /etc/security/limits.conf

Add the lines below to limits.conf.
# User neo4j allowed concurrent files
neo4j   soft    nofile  40000
neo4j   hard    nofile  40000
Code Block
$ sudo vi /etc/pam.d/su

Uncomment the following line.
session    required   pam_limits.so

After the restart neo4-service should be running.

Code Block
$ sudo service neo4j-service status
Neo4j Server is running at pid 1475

Create full text index for nodes and relationships.

Code Block
$ curl -D - -H "Content-Type: application/json" --data '{"name" : "node_auto_index","config" : {"type" : "fulltext","provider" : "lucene"}}' -X POST http://localhost:7474/db/data/index/node/

HTTP/1.1 201 Created
*snip*

$ curl -D - -H "Content-Type: application/json" --data '{"name" : "relationship_auto_index","config" : {"type" : "fulltext","provider" : "lucene"}}' -X POST http://localhost:7474/db/data/index/relationship/

HTTP/1.1 201 Created
*snip*

Postgres database

Set password for database user and create a new database

Code Block
sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni

NORDUni repository

Get the NORDUni files.

Code Block
$ sudo -u ni -i
$ pwd
/var/opt/norduni
$ git clone git://git.nordu.net/norduni.git

Python environment

Make a virtual python environment.

Code Block
$ virtualenv norduni_environment

Making a virtual environment is also just a suggestion but it makes it easier to keep your system clean.

Python requirements

Install required python modules.

Code Block
$ . norduni_environment/bin/activate
$ pip install -r norduni/requirements.txt

Django settings

Change the django settings.

Code Block
$ cd norduni/src/niweb/niweb/
$ cp generic_settings.py settings.py
$ vi settings.py

Change at least the database settings.

Code Block
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}

Check if your settings are ok.

Code Block
$ python manage.py syncdb
$ python manage.py runserver 0.0.0.0:80

Now you should be able connect to your machines with your browser and see the NOCLook app index page.

Note
titleWork in progress
Below is not done yet.

Deploying NOCLook

Create a uwsgi configuration file.

Configure nginx.

Code Block
title/etc/nginx/sites-available/default
server {
    listen 80;
    root /opt/norduni/src/niweb;
    server_name ni.example.net;
    access_log /var/log/ni/noclook-access.log;
    error_log /var/log/ni/noclook-error.log;

    location /static/ {
        root   /opt/norduni/src/niweb/;
        autoindex on;
        access_log   off;
        expires      30d; 
    }

    location / {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8000/_addr;
    }   
}

Install supervisord and set up the following start script.

Code Block
easy_install supervisor
echo_supervisord_conf > /etc/supervisord.conf
Code Block
title/etc/init/supervisord.conf
description proxy_set_header X-Scheme $scheme;
      "supervisord"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf

Add the noclook start script to the supervisor configuration.

...

proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8000/;
    }   
}

 

Collecting and processing network data

...