Setting up a NORDUni instance

Work in progress

Setting up old NOCLook

This guide is written for Ubuntu 10.04.

sudo apt-get install nginx git-core python-virtualenv postfix openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev

Make a virtual python environment.

cd /opt/python_environments/
sudo virtualenv --no-site-packages norduni
source norduni/env/bin/acivate

Install needed python modules.

sudo su
source python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder gunicorn

Get the NORDUni files:

cd /var/opt/
git clone git://git.nordu.net/norduni.git

Get the NERDS files:

mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

cd /var/opt/norduni/src/niweb/
ln -s /var/opt/python_environments/norduni/ env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni

Change the django settings.

# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py

Change at least the following settings.

NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = '/opt/norduni/dependencies/neo4jdb/'

Neo4j >1.5 embedded with Python bindings

Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/)

pip install neo4j-embedded
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/jre/
pip install /path/to/jpype-version.zip

Here you can end if you want to run a development installation of NOCLook.

Deploying NOCLook

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

urls.py

# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Create a gunicorn start file.

/opt/norduni/src/start_noclook.sh

#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
 --user=$USER --group=$GROUP --log-level=debug \
 --log-file=$LOGFILE 2>>$LOGFILE

Configure nginx.

/etc/nginx/sites-available/default

server {
    listen 80;
    root /opt/norduni/src/niweb;
    server_name ni.example.net;
    access_log /var/log/ni/noclook-access.log;
    error_log /var/log/ni/noclook-error.log;

    location /static/ {
        root   /opt/norduni/src/niweb/;
        autoindex on;
        access_log   off;
        expires      30d; 
    }

    location / {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8000/;
    }   
}

Install supervisord and set up the following start script.

easy_install supervisor
echo_supervisord_conf > /etc/supervisord.conf

/etc/init/supervisord.conf

description     "supervisord"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf

Add the noclook start script to the supervisor configuration.

[program:noclook]
directory = /opt/norduni/src/niweb/
user = user
command = /opt/norduni/src/start_noclook.sh
stdout_logfile = /var/log/ni/supervisor_logfile.log
stderr_logfile = /var/log/ni/supervisor_err_logfile.log

Set up a GIT repository for the JSON files produced by the NERDS producers and make a clone in /var/opt/. NORDUnet has a GIT repository called nistore
and it is cloned to /var/opt/nistore/.

Consume the JSON files from the repository with the NOCLook consume scripts.

Collecting and processing network data

Juniper Configuration Producer

"host": {
    "juniper_conf": {
        "bgp_peerings": [
            {    
            "as_number": "", 
            "group": "", 
            "description": "", 
            "remote_address": "", 
            "local_address": "", 
            "type": ""
            },
        ], 
        "interfaces": [
            {
            "name": "", 
            "bundle": "", 
            "vlantagging": true/false, 
            "units": [
                {
                "address": [
                "", 
                ""
                ], 
                "description": "", 
                "unit": "", 
                "vlanid": ""
                }
            ], 
            "tunnels": [
            {
            "source": "", 
            "destination": ""
            }
            ], 
            "description": ""
            }, 
        ],
        "name": ""
        }, 
        "version": 1, 
        "name": ""        
    }

Alcatel ISIS Producer

"host": {
    "alcatel_isis": {
        "data": {
            "ip_address": "", 
            "link": "",
            "name": "",
            "osi_address": "", 
            "ots": "", 
            "type": ""
        }, 
        "name": "", 
        "neighbours": [
            {
                "metric": "", 
                "name": ""
            }, 
        ]
    }, 
    "name": "", 
    "version": 1
    }

NMAP Producer

"host" : {
    "." : {
        "os" : {
            "family" : "",
            "name" : ""
        }
    },
    "addrs" : [
        "127.0.0.1"
    ],
    "hostnames" : [
        "host.example.org"
    ],
    "name" : "host.example.org",
    "services" : {
        "ipv4": {
            "127.0.0.1": {
                "tcp": {
                    "1025": {
                        "product": "Microsoft Windows RPC", 
                        "confidence": "10", 
                        "name": "msrpc", 
                        "proto": "unknown"}, 
                    "1029": {
                        "product": "Microsoft Windows RPC over HTTP", 
                        "confidence": "10", 
                        "version": "1.0", 
                        "name": "ncacn_http", 
                        "proto": "unknown"}, 
                }
            }
        }
    },
    "version" : 1
}

CSV Site Producer/Consumer

https://github.com/fredrikt/nerds/blob/master/producers/csv_producer/csv_producer.py

The script produces JSON output in the NERDS format from the provided CSV file.

The csv file needs to start with the name of the node and then the node type.
After those two columns any other node property may follow.

Start your csv file with a line similar to the one below.
name;node_type;node_property1,node_property2;...;node_property15

NORDUnet site documentation example

Add later

Blank output from NORDUnet site documentation example

{
    "host": {
        "csv_producer": {
            "address": "", 
            "area": "", 
            "city": "", 
            "comment": "", 
            "country": "", 
            "floor": "", 
            "latitude": "", 
            "longitude": "", 
            "meta_type": "", 
            "name": "", 
            "node_type": "", 
            "owner_id": "", 
            "postcode": "", 
            "responsible_for": "", 
            "room": "", 
            "site_type": "", 
            "telenor_subscription_id": ""
        }, 
        "name": "", 
        "version": 1
    }
}

The consumer script should only be run once as it does not update the sites, only creates new.

The JSON file directory is then inserted in to the database using noclook_site_csv_consumer.py. http://git.nordu.net/?p=norduni.git;a=blob;f=src/scripts/noclook_site_csv_consumer.py
Change the path at the top of the script to be able to import norduni_client.py.

python noclook_site_csv_consumer.py -D /path/to/site_files/json

Setting up old NOCLook version

sudo apt-get install apache2 libapache2-mod-wsgi git-core python-virtualenv postfix openjdk-6-jre-headless postgresql python-psycopg2 libpq-dev python-dev

Make a virtual python environment.

cd /opt/python_environments/
sudo virtualenv --no-site-packages norduni
source norduni/env/bin/acivate

Install needed python modules.

sudo su
source python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder httplib2

Get the NORDUni files:

cd /var/opt/
git clone git://git.nordu.net/norduni.git
git checkout operation

Get the NERDS files:

mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

cd /var/opt/norduni/src/niweb/
ln -s /var/opt/python_environments/norduni/ env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni

Change the django settings.

# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py

Change at least the following settings.

NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = 'http://localhost:7474/db/data/'

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

urls.py

# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Create a file name niweb in /etc/apache2/conf.d/.

WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi

Add niweb path to niweb.wsgi

sys.path.append('/opt/norduni/src/niweb')

Add the following lines to /etc/apache2/sites-available/default-ssl

        Alias /admin_media /var/opt/norduni/src/niweb/static/admin/
        Alias /site_media /var/opt/norduni/src/niweb/static/
        <LocationMatch "^/(admin_media|site_media)">
           Options FollowSymLinks
           Order allow,deny
           allow from all
           SetHandler None
        </LocationMatch>

Neo4j 1.3 REST stand alone server

Download neo4j-1.3 from http://neo4j.org.
Extract the file to, for example, /var/opt/norduni/dependencies/.

cd /var/opt/norduni/dependencies/
tar xfz neo4j-1.3.tar.gz

Copy the startup script from /var/opt/norduni/src/scripts/neo4j/ to /etc/init.d/ and then make neo4j start on boot.

update-rc.d /etc/init.d/neo4j defaults

You need to shut down port 7474 from anything except localhost.

iptables -A INPUT -p tcp --dport 7474 -s localhost -j ACCEPT
iptables -A INPUT -p tcp --dport 7474 -j DROP

Save the rules and add the following line to your interface configuration so you will restore the rules on a reboot.

bash -c "iptables-save > /etc/iptables.rules"

vi /etc/network/interfaces

iface eth0 inet static
    pre-up iptables-restore < /etc/iptables.rules

Apache configuration for neo4j-embedded

If you want to run the latest version if NOCLook (using neo4j-embedded) you have to configure Apache to only use one process to run the Django code.

Install apache2-mpm-prefork and libapache2-mod-wsgi.