You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 47 Next »

Work in progress

Setting up old NOCLook

This guide is written for Ubuntu 10.04.

sudo apt-get install nginx git-core python-virtualenv postfix openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev

Make a virtual python environment.

cd /opt/python_environments/
sudo virtualenv --no-site-packages norduni
source norduni/env/bin/acivate

Install needed python modules.

sudo su
source python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder gunicorn

Get the NORDUni files:

cd /var/opt/
git clone git://git.nordu.net/norduni.git

Get the NERDS files:

mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

cd /var/opt/norduni/src/niweb/
ln -s /var/opt/python_environments/norduni/ env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni

Change the django settings.

# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py

Change at least the following settings.

NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = '/opt/norduni/dependencies/neo4jdb/'

Neo4j >1.5 embedded with Python bindings

Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/)

pip install neo4j-embedded
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/jre/
pip install /path/to/jpype-version.zip

Here you can end if you want to run a development installation of NOCLook.

Deploying NOCLook

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

urls.py
# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Create a gunicorn start file.

/opt/norduni/src/start_noclook.sh
#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
 --user=$USER --group=$GROUP --log-level=debug \
 --log-file=$LOGFILE 2>>$LOGFILE

Configure nginx.

/etc/nginx/sites-available/default
server {
    listen 80;
    root /opt/norduni/src/niweb;
    server_name ni.example.net;
    access_log /var/log/ni/noclook-access.log;
    error_log /var/log/ni/noclook-error.log;

    location /static/ {
        root   /opt/norduni/src/niweb/;
        autoindex on;
        access_log   off;
        expires      30d; 
    }

    location / {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8000/;
    }   
}

Install supervisord and set up the following start script.

easy_install supervisor
echo_supervisord_conf > /etc/supervisord.conf
/etc/init/supervisord.conf
description     "supervisord"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf

Add the noclook start script to the supervisor configuration.

[program:noclook]
directory = /opt/norduni/src/niweb/
user = user
command = /opt/norduni/src/start_noclook.sh
stdout_logfile = /var/log/ni/supervisor_logfile.log
stderr_logfile = /var/log/ni/supervisor_err_logfile.log

Set up a GIT repository for the JSON files produced by the NERDS producers and make a clone in /var/opt/. NORDUnet has a GIT repository called nistore
and it is cloned to /var/opt/nistore/.

Consume the JSON files from the repository with the NOCLook consume scripts.

Collecting network data

Juniper Configuration Producer

"host": {
    "juniper_conf": {
        "bgp_peerings": [
            {    
            "as_number": "", 
            "group": "", 
            "description": "", 
            "remote_address": "", 
            "local_address": "", 
            "type": ""
            },
        ], 
        "interfaces": [
            {
            "name": "", 
            "bundle": "", 
            "vlantagging": true/false, 
            "units": [
                {
                "address": [
                "", 
                ""
                ], 
                "description": "", 
                "unit": "", 
                "vlanid": ""
                }
            ], 
            "tunnels": [
            {
            "source": "", 
            "destination": ""
            }
            ], 
            "description": ""
            }, 
        ],
        "name": ""
        }, 
        "version": 1, 
        "name": ""        
    }

Alcatel ISIS Producer

"host": {
    "alcatel_isis": {
        "data": {
            "ip_address": "", 
            "link": "",
            "name": "",
            "osi_address": "", 
            "ots": "", 
            "type": ""
        }, 
        "name": "", 
        "neighbours": [
            {
                "metric": "", 
                "name": ""
            }, 
        ]
    }, 
    "name": "", 
    "version": 1
    }

NMAP Producer

"host" : {
    "." : {
        "os" : {
            "family" : "",
            "name" : ""
        }
    },
    "addrs" : [
        "127.0.0.1"
    ],
    "hostnames" : [
        "host.example.org"
    ],
    "name" : "host.example.org",
    "services" : {
        "ipv4": {
            "127.0.0.1": {
                "tcp": {
                    "1025": {
                        "product": "Microsoft Windows RPC", 
                        "confidence": "10", 
                        "name": "msrpc", 
                        "proto": "unknown"}, 
                    "1029": {
                        "product": "Microsoft Windows RPC over HTTP", 
                        "confidence": "10", 
                        "version": "1.0", 
                        "name": "ncacn_http", 
                        "proto": "unknown"}, 
                }
            }
        }
    },
    "version" : 1
}

CSV Producer

JSON Structure

Setting up old NOCLook version

sudo apt-get install apache2 libapache2-mod-wsgi git-core python-virtualenv postfix openjdk-6-jre-headless postgresql python-psycopg2 libpq-dev python-dev

Make a virtual python environment.

cd /opt/python_environments/
sudo virtualenv --no-site-packages norduni
source norduni/env/bin/acivate

Install needed python modules.

sudo su
source python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder httplib2

Get the NORDUni files:

cd /var/opt/
git clone git://git.nordu.net/norduni.git
git checkout operation

Get the NERDS files:

mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

cd /var/opt/norduni/src/niweb/
ln -s /var/opt/python_environments/norduni/ env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni

Change the django settings.

# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py

Change at least the following settings.

NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = 'http://localhost:7474/db/data/'

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

urls.py
# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Create a file name niweb in /etc/apache2/conf.d/.

WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi

Add niweb path to niweb.wsgi

sys.path.append('/opt/norduni/src/niweb')

Add the following lines to /etc/apache2/sites-available/default-ssl 

        Alias /admin_media /var/opt/norduni/src/niweb/static/admin/
        Alias /site_media /var/opt/norduni/src/niweb/static/
        <LocationMatch "^/(admin_media|site_media)">
           Options FollowSymLinks
           Order allow,deny
           allow from all
           SetHandler None
        </LocationMatch>

Neo4j 1.3 REST stand alone server

Download neo4j-1.3 from http://neo4j.org.
Extract the file to, for example, /var/opt/norduni/dependencies/.

cd /var/opt/norduni/dependencies/
tar xfz neo4j-1.3.tar.gz

Copy the startup script from /var/opt/norduni/src/scripts/neo4j/ to /etc/init.d/ and then make neo4j start on boot.

update-rc.d /etc/init.d/neo4j defaults

You need to shut down port 7474 from anything except localhost.

iptables -A INPUT -p tcp --dport 7474 -s localhost -j ACCEPT
iptables -A INPUT -p tcp --dport 7474 -j DROP

Save the rules and add the following line to your interface configuration so you will restore the rules on a reboot.

bash -c "iptables-save > /etc/iptables.rules"

vi /etc/network/interfaces

iface eth0 inet static
    pre-up iptables-restore < /etc/iptables.rules

Apache configuration for neo4j-embedded

If you want to run the latest version if NOCLook (using neo4j-embedded) you have to configure Apache to only use one process to run the Django code.

Install apache2-mpm-prefork and libapache2-mod-wsgi.

Create a file name niweb in /etc/apache2/conf.d/.

WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIDaemonProcess processes=1 threads=25 # For neo4j-embedded
WSGIApplicationGroup %{GLOBAL}           # For neo4j-embedded
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi

Add niweb path to niweb.wsgi

sys.path.append('/opt/norduni/src/niweb')

/etc/apache2.conf

<IfModule mpm_prefork_module>
    StartServers          1
    ServerLimit           1
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients            1
    MaxRequestsPerChild   0
</IfModule>

Apache Shibboleth configuration

        <LocationMatch "Shibboleth.sso">
           SetHandler None
        </LocationMatch>

        <Location "/accounts/login-federated/">
           AuthType shibboleth
           ShibRequireSession On
           require valid-user
           RequestHeader set X_REMOTE_USER %{eppn}e
           RequestHeader set EPPN %{eppn}e
           RequestHeader set GIVENNAME %{givenName}e
           RequestHeader set SN %{sn}e
           RequestHeader set MAIL %{mail}e
           RequestHeader set AFFILIATION %{affiliation}e   
           RequestHeader set ENTITLEMENT %{entitlement}e
        </Location>
  • No labels