Page History

Note

title	Work in progress

Deprecated. Please use Setting up a NORDUni 2.0 instance.

Table of Contents

Setting up NOCLook

This guide is written for Ubuntu 10.04.

Paths used in this guide is just a suggestion, as long as Python can find the files it needs to import it is all good.

Code Block
sudo apt-get install git-core python-virtualenv openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev

...

Make a virtual python environment.

Code Block
sudo su mkdir /opt/python_environments/ cd /opt/python_environments/ virtualenv --no-site-packages norduni

...

Install needed python modules.

Code Block


sudo su
source /opt/python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder django-tastypie xlwt south django-activity-stream django-jsonfield

...

Get the NORDUni files:

Code Block
cd /opt/ git clone git://git.nordu.net/norduni.git

Get the NERDS files:

Code Block
mkdir /opt/norduni/tools/ cd /opt/norduni/tools/ git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

Code Block
cd /opt/norduni/src/niweb/ ln -s /opt/python_environments/norduni env cd static ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

Code Block
sudo -u postgres psql postgres \password postgres Write password Write password again Ctrl+D sudo -u postgres createdb norduni

Change the django settings.

Code Block
# Django settings for niweb project cd /opt/norduni/src/niweb/ cp generic_settings.py settings.py vi settings.py

Change at least the following settings.

Code Block


NIWEB_ROOT = '/opt/norduni/src/niweb/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = '/opt/norduni/dependencies/neo4jdb/'

...

Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/^{Image Removed})

Code Block
pip install neo4j-embedded export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ pip install /path/to/JPype-version.zip

...

Change the following lines in urls.py:

Code Block
cd /opt/norduni/src/niweb/

Code Block
# Django Generic Login (r'^accounts/login/$', 'django.contrib.auth.views.login'), # Federated login #(r'^accounts/', include('niweb.apps.fedlogin.urls')),

Code Block
python manage.py syncdb python manage.py runserver 0.0.0.0:80

...

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

Code Block

title	urls.py


# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Install nginx, postfix and gunicorn.

Code Block
sudo apt-get install nginx postfix pip install gunicorn

...

Code Block

title	/opt/norduni/src/start_noclook.sh


#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
 --user=$USER --group=$GROUP --log-level=debug \
 --log-file=$LOGFILE 2>>$LOGFILE

...

Code Block

title	/etc/nginx/sites-available/default


server {
    listen 80;
    root /opt/norduni/src/niweb;
    server_name ni.example.net;
    access_log /var/log/ni/noclook-access.log;
    error_log /var/log/ni/noclook-error.log;

    location /static/ {
        root   /opt/norduni/src/niweb/;
        autoindex on;
        access_log   off;
        expires      30d; 
    }

    location / {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8000/;
    }   
}

Install supervisord and set up the following start script.

Code Block
easy_install supervisor echo_supervisord_conf > /etc/supervisord.conf

Code Block

title	/etc/init/supervisord.conf


description     "supervisord"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf

Add the noclook start script to the supervisor configuration.

Code Block


[program:noclook]
directory = /opt/norduni/src/niweb/
user = user
command = /opt/norduni/src/start_noclook.sh
stdout_logfile = /var/log/ni/supervisor_logfile.log
stderr_logfile = /var/log/ni/supervisor_err_logfile.log

...

To start have a look at the NERDS README then clone the NERDS project.

Code Block
cd /opt/norduni/ mkdir tools cd tools git clone https://github.com/fredrikt/nerds.git

...

Code Block

title	template.conf


[ssh]
user = view_account_user
password = not_so_secret_password

[sources]
remote = one.example.org two.example.org three.example.org
local = /var/conf/one.xml /var/conf/two.xml /var/conf/three.xml

Code Block

title	Blank output from the juniper_conf.py producer.


"host": {
    "juniper_conf": {
        "bgp_peerings": [
            {    
            "as_number": "", 
            "group": "", 
            "description": "", 
            "remote_address": "", 
            "local_address": "", 
            "type": ""
            },
        ], 
        "interfaces": [
            {
            "name": "", 
            "bundle": "", 
            "vlantagging": true/false, 
            "units": [
                {
                "address": [
                "", 
                ""
                ], 
                "description": "", 
                "unit": "", 
                "vlanid": ""
                }
            ], 
            "tunnels": [
            {
            "source": "", 
            "destination": ""
            }
            ], 
            "description": ""
            }, 
        ],
        "name": ""
        }, 
        "version": 1, 
        "name": ""        
    }

The JSON files can the be inserted using noclook_juniper_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Edit the template.conf file with the correct path to the Juniper NERDS files.

Code Block
[data] juniper_conf = /path/to/juniper/json nmap_services = alcatel_isis = noclook =

Then run:

Code Block
python noclook_juniper_consumer.py -C template.conf

...

Code Block

title	Optional mapping file


osi_address;name;other1;otherN
47002300000001000100010001002060280DB11D;NU-SHHM-ILA-01;info1;infoN

Code Block


"host": {
    "alcatel_isis": {
        "data": {
            "ip_address": "", 
            "link": "",
            "name": "",
            "osi_address": "", 
            "ots": "", 
            "type": ""
        }, 
        "name": "", 
        "neighbours": [
            {
                "metric": "", 
                "name": ""
            }, 
        ]
    }, 
    "name": "", 
    "version": 1
    }

The JSON files can be inserted with noclook_alcatel_consumer.py.
Edit the template.conf file with the correct path to the Alcatel ISIS NERDS files.
Change the path at the top of the script to be able to import norduni_client.py.

Code Block
[data] juniper_conf = nmap_services = alcatel_isis = /path/to/alcatel/json noclook =

Then run:

Code Block
python noclook_alcatel_consumer.py -C template.conf

nmap Producer/Consumer

Using the namp the nmap services producer you can scan a network or individual addresses. NORDUnet have a file
with networks that is used with the "-iL networks_file" option added to NERDS_NMAP_OPTIONS in the run.sh file.

You need to install two Perl modules if you don't already have them.

Code Block
cpan Nmap::Parser cpan JSON

python-nmap from https://github.com/johanlundberg/python-nmap if the pip version gives you trouble.

Then you can scan your localhost with:

Code Block
cd /opt/norduni/tools/nerds/producers/nmap_services ./run.sh -O . 127.0.0.1

You will find the JSON file in /opt/norduni/tools/nerds/producers/nmap_services/producers/json/.

Code Block


"host" : {
    "." : {
        "os" : {
            "family" : "",
            "name" : ""
        }
    },
    "addrs" : [
        "127.0.0.1"
    ],
    "hostnames" : [
        "host.example.org"
    ],
    "name" : "host.example.org",
    "services" : {
        "ipv4": {
            "127.0.0.1": {
                "tcp": {
                    "1025": {
                        "product": "Microsoft Windows RPC", 
                        "confidence": "10", 
                        "name": "msrpc", 
                        "proto": "unknown"}, 
                    "1029": {
                        "product": "Microsoft Windows RPC over HTTP", 
                        "confidence": "10", 
                        "version": "1.0", 
                        "name": "ncacn_http", 
                        "proto": "unknown"}, 
                }
            }
        }
    },
    "version" : 1
}

The JSON files can be inserted with noclook_nmap_consumer_py.py.
Edit the template.conf file with the correct path to the nmap services JSON files.
Change the path at the top of the script to be able to import norduni_client.py.

Code Block
[data] juniper_conf = nmap_services = /path/to/nmap/json alcatel_isis = noclook =

Then run:

Code Block
python noclook_nmap_consumer.py -C template.conf

...

Code Block

title	NORDUnet site documentation example


name;Host;site_type;address;area;postcode;city;country;floor;room;latitude;longitude;responsible_for;owner_id;telenor_subscription_id;comment

Code Block

title	Blank output from NORDUnet site documentation example


{
    "host": {
        "csv_producer": {
            "address": "", 
            "area": "", 
            "city": "", 
            "comment": "", 
            "country": "", 
            "floor": "", 
            "latitude": "", 
            "longitude": "", 
            "meta_type": "", 
            "name": "", 
            "node_type": "", 
            "owner_id": "", 
            "postcode": "", 
            "responsible_for": "", 
            "room": "", 
            "site_type": "", 
            "telenor_subscription_id": ""
        }, 
        "name": "", 
        "version": 1
    }
}

...

Change the path at the top of the script to be able to import norduni_client.py.

Then run:

Code Block
python noclook_site_csv_consumer.py -D /path/to/site_files/json

...

Change the path at the top of the script to be able to import norduni_client.py.

Code Block
[data] juniper_conf = /path/to/juniper/json nmap_services = /path/to/nmap/json alcatel_isis = /path/to/alcate/json noclook = #Used for loading backup.

Then run:

Code Block
python noclook_consumer.py -C template.conf -I

...

The database can be purged from all data using the P switch to the noclook_consumer.py script.

Code Block
python noclook_consumer.py -C template.conf -P

Setting up old NOCLook version

Code Block
sudo apt-get install apache2 libapache2-mod-wsgi git-core python-virtualenv postfix openjdk-6-jre-headless postgresql python-psycopg2 libpq-dev python-dev

Make a virtual python environment.

Code Block
cd /opt/python_environments/ sudo virtualenv --no-site-packages norduni source norduni/env/bin/acivate

Install needed python modules.

Code Block
sudo su source python_environments/norduni/bin/activate pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder httplib2

Get the NORDUni files:

Code Block
cd /var/opt/ git clone git://git.nordu.net/norduni.git git checkout operation

Get the NERDS files:

Code Block
mkdir /opt/norduni/tools/ cd /opt/norduni/tools/ git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

Code Block
cd /var/opt/norduni/src/niweb/ ln -s /var/opt/python_environments/norduni/ env cd static ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

Code Block
sudo -u postgres psql postgres \password postgres Write password Write password again Ctrl+D sudo -u postgres createdb norduni

Change the django settings.

Code Block
# Django settings for niweb project cd /opt/norduni/src/niweb/ cp generic_settings.py settings.py vi settings.py

Change at least the following settings.

Code Block


NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = 'http://localhost:7474/db/data/'

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

Code Block

title	urls.py


# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Create a file name niweb in /etc/apache2/conf.d/.

Code Block


WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi

Add niweb path to niweb.wsgi

Code Block
sys.path.append('/opt/norduni/src/niweb')

Add the following lines to /etc/apache2/sites-available/default-ssl

Code Block


        Alias /admin_media /var/opt/norduni/src/niweb/static/admin/
        Alias /site_media /var/opt/norduni/src/niweb/static/
        <LocationMatch "^/(admin_media|site_media)">
           Options FollowSymLinks
           Order allow,deny
           allow from all
           SetHandler None
        </LocationMatch>

Neo4j 1.3 REST stand alone server

Download neo4j-1.3 from http://neo4j.org^{Image Removed}.
Extract the file to, for example, /var/opt/norduni/dependencies/.

Code Block
cd /var/opt/norduni/dependencies/ tar xfz neo4j-1.3.tar.gz

Copy the startup script from /var/opt/norduni/src/scripts/neo4j/ to /etc/init.d/ and then make neo4j start on boot.

Code Block
update-rc.d /etc/init.d/neo4j defaults

You need to shut down port 7474 from anything except localhost.

Code Block
iptables -A INPUT -p tcp --dport 7474 -s localhost -j ACCEPT iptables -A INPUT -p tcp --dport 7474 -j DROP

Save the rules and add the following line to your interface configuration so you will restore the rules on a reboot.

Code Block
bash -c "iptables-save > /etc/iptables.rules" vi /etc/network/interfaces iface eth0 inet static pre-up iptables-restore < /etc/iptables.rules

...

Create a file name niweb in /etc/apache2/conf.d/.

Code Block


WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIDaemonProcess processes=1 threads=25 # For neo4j-embedded
WSGIApplicationGroup %{GLOBAL}           # For neo4j-embedded
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi

Add niweb path to niweb.wsgi

Code Block
sys.path.append('/opt/norduni/src/niweb')

/etc/apache2.conf

Code Block
<IfModule mpm_prefork_module> StartServers 1 ServerLimit 1 MinSpareServers 5 MaxSpareServers 10 MaxClients 1 MaxRequestsPerChild 0 </IfModule>

Apache Shibboleth configuration

Code Block


        <LocationMatch "Shibboleth.sso">
           SetHandler None
        </LocationMatch>

        <Location "/accounts/login-federated/">
           AuthType shibboleth
           ShibRequireSession On
           require valid-user
           RequestHeader set X_REMOTE_USER %{eppn}e
           RequestHeader set EPPN %{eppn}e
           RequestHeader set GIVENNAME %{givenName}e
           RequestHeader set SN %{sn}e
           RequestHeader set MAIL %{mail}e
           RequestHeader set AFFILIATION %{affiliation}e   
           RequestHeader set ENTITLEMENT %{entitlement}e
        </Location>

Using the Virtualbox image

Download the OVA file, NOCLook Test Ubuntu 12.04.ova.

Username: norduni
Password: norduni+123

To start the app:

Code Block
sudo su /opt/norduni/start_dev_noclook.sh

...

To add more hosts just go to /opt/norduni/tools/nerds/producers/nmap_services_py.

Code Block
./run json address to scan

After the data collection you insert the data using the consumer. (You have to stop the Django app first.)

Code Block
. /opt/norduni/src/niweb/env/bin/activate cd /opt/norduni/src/scripts python noclook_consumer.py -C test.conf -I

Child pages

Versions Compared

Old Version 77

New Version Current

Key

Setting up NOCLook

nmap Producer/Consumer

Setting up old NOCLook version

Neo4j 1.3 REST stand alone server

Apache Shibboleth configuration

Using the Virtualbox image