Page History

Note

title	Work in progress

Deprecated. Please use Setting up a NORDUni 2.0 instance.

Table of Contents

Setting up NOCLook

This guide is written for Ubuntu 10.04.

Paths used in this guide is just a suggestion, as long as Python can find the files it needs to import it is all good.

Code Block
sudo apt-get install git-core python-virtualenv openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev

...

Make a virtual python environment.

Code Block
sudo su mkdir /opt/python_environments/ cd /opt/python_environments/ sudo virtualenv --no-site-packages norduni source norduni/env/bin/acivate

Making a virtual environment is also just a suggestion but it makes it easier to keep your system clean.

Install needed python modules.

Code Block


sudo su
source /opt/python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder django-tastypie xlwt south django-activity-stream django-jsonfield

Here psycopg2 is optional but you need to install the Python database module of your SQL database choice. If you have trouble installing psycopg2.

Get the NORDUni files:

Code Block
cd /var/opt/ git clone git://git.nordu.net/norduni.git

Get the NERDS files:

Code Block
mkdir /opt/norduni/tools/ cd /opt/norduni/tools/ git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

Code Block
cd /var/opt/norduni/src/niweb/ ln -s /var/opt/python_environments/norduni/ env cd static ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

Code Block
sudo -u postgres psql postgres \password postgres Write password Write password again Ctrl+D sudo -u postgres createdb norduni

Change the django settings.

Code Block
# Django settings for niweb project cd /opt/norduni/src/niweb/ cp generic_settings.py settings.py vi settings.py

Change at least the following settings.

Code Block


NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = '/opt/norduni/dependencies/neo4jdb/'

...

Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/^{Image Removed})

Code Block
pip install neo4j-embedded export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/jre/ pip install /path/to/jpypeJPype-version.zip

Setting up a local/development NOCLook

Deploying NOCLook

...

Set up the SQL database using the Django project.

Change the following lines in urls.py:

Code Block
cd /opt/norduni/src/niweb/

...

Code Block

title	urls.py


# StaticDjango Generic serveLogin
    #(r'^site_media/(?P<path>.*)^accounts/login/$', 'django.contrib.auth.views.static.servelogin'),

# Federated   #    {'document_root': settings.STATIC_DEV_MEDIA}login
#(r'^accounts/', include('niweb.apps.fedlogin.urls')),

Install nginx, postfix and gunicorn.

Code Block
sudo apt-get install nginx postfix pip install gunicorn

Create a gunicorn start file.

...

python manage.py syncdb
python manage.py runserver 0.0.0.0:80

Now you should be able surf to your machines ip and see the NOCLook app started.

It is time to collect and insert some data.

Deploying NOCLook

Comment out the static media url in

...

/opt/norduni/src/

...


#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
 --user=$USER --group=$GROUP --log-level=debug \
 --log-file=$LOGFILE 2>>$LOGFILE

niweb/urls.py.

Code Block

title	urls.py

# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Install nginx, postfix and gunicorn.

Code Block
sudo apt-get install nginx postfix pip install gunicorn

Create a gunicorn start file.

Code Block

title	/opt/norduni/src/start_noclook.sh

#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=

Configure nginx.

Code Block

title	/etc/nginx/sites-available/default


server {
    listen 80;
    root /opt/norduni/src/niweb;
    server_name ni.example.net;
    access_log /var/log/ni/noclook-access.log;
    error_log /var/log/ni/noclook-error.log;

    location /static/ {
        root   
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
 --user=$USER --group=$GROUP --log-level=debug \
 --log-file=$LOGFILE 2>>$LOGFILE

Configure nginx.

Code Block

title	/etc/nginx/sites-available/default

server {
    listen 80/;
        autoindex on;
        access_log   off;
    root /opt/norduni/src/niweb;
   expires  server_name ni.example.net;
     30d; access_log /var/log/ni/noclook-access.log;
    }error_log /var/log/ni/noclook-error.log;

    location /static/ {
        proxy_pass_header Serverroot   /opt/norduni/src/niweb/;
        proxy_set_header Host $http_hostautoindex on;
        proxyaccess_redirectlog   off;
        proxy_set_header X-Real-IP $remote_addr;expires      30d; 
    }

    location / {
        proxy_setpass_header X-Scheme $schemeServer;
        proxy_connectset_timeout 10header Host $http_host;
        proxy_read_timeoutredirect 10off;
        proxy_pass http://localhost:8000/;
set_header X-Real-IP $remote_addr;
     }   
}
proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8000/;
    }   
}

Install supervisord and set up the following start script.

Code Block
easy_install supervisor echo_supervisord_conf > /etc/supervisord.conf

Code Block

title	/etc/init/supervisord.conf


description     "supervisord"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf

Add the noclook start script to the supervisor configuration.

Code Block


[program:noclook]
directory = /opt/norduni/src/niweb/
user = user
command = /opt/norduni/src/start_noclook.sh
stdout_logfile = /var/log/ni/supervisor_logfile.log
stderr_logfile = /var/log/ni/supervisor_err_logfile.log

Set up a GIT repository for the JSON files produced by the NERDS producers and make a clone in /var/opt/. NORDUnet has a GIT repository called nistore
and it is cloned to /var/opt/nistore/.

Consume the JSON files from the repository with the NOCLook consume scripts.

Collecting and processing Collecting and processing network data

To insert data you need to stop any python process that is using the Neo4j database. We hope to get the option to load more database instances in read-only mode in a near future then
this could be avoided.

Juniper Configuration Producer/Consumer

NORDUnet has a GIT repository called nistore and it is cloned to /opt/nistore/.

To start have a look at the NERDS README then clone the NERDS project.

Code Block
cd /opt/norduni/ mkdir tools cd tools git clone https://github.com/fredrikt/nerds.git

Juniper Configuration Producer/Consumer

The The Juniper configuration producer can load Juniper configuration directly from the router via SSH or Juniper configuration files in XML format from disk.

Code Block

title	template.conf


[ssh]
user = view_account_user
password = not_so_secret_password

[sources]
remote = one.example.org two.example.org three.example.org
local = /var/conf/one.xml /var/conf/two.xml /var/conf/three.xml

Code Block

title	Blank output from the juniper_conf.py producer.


"host": {
    "juniper_conf": {
        "bgp_peerings": [
            {    
            "as_number": "", 
            "group": "", 
            "description": "", 
            "remote_address": "", 
            "local_address": "", 
            "type": ""
            },
        ], 
        "interfaces": [
            {
            "name": "", 
            "bundle": "", 
            "vlantagging": true/false, 
            "units": [
                {
                "address": [
                "", 
                ""
                ], 
                "description": "", 
                "unit": "", 
                "vlanid": ""
                }
            ], 
            "tunnels": [
            {
            "source": "", 
            "destination": ""
            }
            ], 
            "description": ""
            }, 
        ],
        "name": ""
        }, 
        "version": 1, 
        "name": ""        
    }

The JSON files can the be inserted using noclook_juniper_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Edit the template.conf file with the correct path to the Juniper NERDS files.

Code Block
[data] juniper_conf = /path/to/juniper/json nmap_services = alcatel_isis = noclook =

Then run:

Code Block
python noclook_juniper_consumer.py -C template.conf

...

Code Block

title	Optional mapping file


To be added

osi_address;name;other1;otherN
47002300000001000100010001002060280DB11D;NU-SHHM-ILA-01;info1;infoN

Code Block


"host": {
    "alcatel_isis": {
        "data": {
            "ip_address": "", 
            "link": "",
            "name": "",
            "osi_address": "", 
            "ots": "", 
            "type": ""
        }, 
        "name": "", 
        "neighbours": [
            {
                "metric": "", 
                "name": ""
            }, 
        ]
    }, 
    "name": "", 
    "version": 1
    }

The JSON files can be inserted with noclook_alcatel_consumer.py.
Edit the template.conf file with the correct path to the Alcatel ISIS NERDS files.
Change the path at the top of the script to be able to import norduni_client.py.

Code Block
[data] juniper_conf = nmap_services = alcatel_isis = /path/to/alcatel/json noclook =

Then run:

Code Block
python noclook_alcatel_consumer.py -C template.conf

nmap Producer/Consumer

Using the namp the nmap services producer you can scan a network or individual addresses. NORDUnet have a file
with networks that is used with the "-iL networks_file" option added to NERDS_NMAP_OPTIONS in the run.sh file.

You need to install python-nmap from https://github.com/johanlundberg/python-nmap if the pip version gives you trouble.

Then you can scan your localhost with:

Code Block
cd /opt/norduni/tools/nerds/producers/nmap_services ./run.sh . 127.0.0.1

You will find the JSON file in /opt/norduni/tools/nerds/producers/nmap_services/producers/json/.

Code Block

"host" : {
    "."
"host" : {
    "." : {
        "os" : {
            "family" : "",
            "name" : ""
        }
    },
    "addrs" : [
        "127.0.0.1"
    ],
    "hostnames" : [
        "host.example.org"
    ],
    "name" : "host.example.org",
    "services" : {
        "ipv4": {
            "127.0.0.1": {
                "tcp": {
                    "1025": {
                        "product": "Microsoft Windows RPC", 
                        "confidence": "10", 
                        "name": "msrpc", 
                        "proto": "unknown"}, 
                    "1029": {
                        "product": "Microsoft Windows RPC over HTTP", 
                        "confidence": "10", 
                        "version": "1.0", 
                        "name": "ncacn_http", 
                        "proto": "unknown"}, 
                }
            }
        }
    },
    "version" : 1
}

The JSON files can be inserted with noclook_nmap_consumer_py.py.
Edit the template.conf file with the correct path to the nmap services JSON files.
Change the path at the top of the script to be able to import norduni_client.py.

Code Block
[data] juniper_conf = nmap_services = /path/to/nmap/json alcatel_isis = noclook =

Then run:

Code Block
python noclook_nmap_consumer.py -C template.conf

...

Code Block

title	NORDUnet site documentation example


Add later

name;Host;site_type;address;area;postcode;city;country;floor;room;latitude;longitude;responsible_for;owner_id;telenor_subscription_id;comment

Code Block

title	Blank output from NORDUnet site documentation example

Code Block

title	Blank output from NORDUnet site documentation example


{
    "host": {
        "csv_producer": {
            "address": "", 
            "area": "", 
            "city": "", 
            "comment": "", 
            "country": "", 
            "floor": "", 
            "latitude": "", 
            "longitude": "", 
            "meta_type": "", 
            "name": "", 
            "node_type": "", 
            "owner_id": "", 
            "postcode": "", 
            "responsible_for": "", 
            "room": "", 
            "site_type": "", 
            "telenor_subscription_id": ""
        }, 
        "name": "", 
        "version": 1
    }
}

...

Change the path at the top of the script to be able to import norduni_client.py.

Then run:

Code Block
python noclook_site_csv_consumer.py -D /path/to/site_files/json

...

Change the path at the top of the script to be able to import norduni_client.py.

Code Block
[data] juniper_conf = /path/to/juniper/json nmap_services = /path/to/nmap/json alcatel_isis = /path/to/alcate/json noclook = #Used for loading backup.

Then run:

Code Block
python noclook_consumer.py -C template.conf -I

...

The database can be purged from all data using the P switch to the noclook_consumer.py script.

Code Block
python noclook_consumer.py -C template.conf -P

Setting up old NOCLook version

Code Block
sudo apt-get install apache2 libapache2-mod-wsgi git-core python-virtualenv postfix openjdk-6-jre-headless postgresql python-psycopg2 libpq-dev python-dev

Make a virtual python environment.

Code Block
cd /opt/python_environments/ sudo virtualenv --no-site-packages norduni source norduni/env/bin/acivate

Install needed python modules.

Code Block
sudo su source python_environments/norduni/bin/activate pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder httplib2

Get the NORDUni files:

Code Block
cd /var/opt/ git clone git://git.nordu.net/norduni.git git checkout operation

Get the NERDS files:

Code Block
mkdir /opt/norduni/tools/ cd /opt/norduni/tools/ git clone https://github.com/fredrikt/nerds.git

Symlink admin media and the virtual python environment to niweb.

Code Block
cd /var/opt/norduni/src/niweb/ ln -s /var/opt/python_environments/norduni/ env cd static ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin

Set password for database user and create a new database

Code Block
sudo -u postgres psql postgres \password postgres Write password Write password again Ctrl+D sudo -u postgres createdb norduni

Change the django settings.

Code Block
# Django settings for niweb project cd /opt/norduni/src/niweb/ cp generic_settings.py settings.py vi settings.py

Change at least the following settings.

Code Block


NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'norduni',
        'USER': 'postgres',
        'PASSWORD': 'secret',
        'HOST': 'localhost'
    }
}
# Neo4j settings
NEO4J_RESOURCE_URI = 'http://localhost:7474/db/data/'

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

Code Block

title	urls.py


# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Create a file name niweb in /etc/apache2/conf.d/.

Code Block


WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi

Add niweb path to niweb.wsgi

Code Block
sys.path.append('/opt/norduni/src/niweb')

Add the following lines to /etc/apache2/sites-available/default-ssl

Code Block


        Alias /admin_media /var/opt/norduni/src/niweb/static/admin/
        Alias /site_media /var/opt/norduni/src/niweb/static/
        <LocationMatch "^/(admin_media|site_media)">
           Options FollowSymLinks
           Order allow,deny
           allow from all
           SetHandler None
        </LocationMatch>

Neo4j 1.3 REST stand alone server

Download neo4j-1.3 from http://neo4j.org^{Image Removed}.
Extract the file to, for example, /var/opt/norduni/dependencies/.

Code Block
cd /var/opt/norduni/dependencies/ tar xfz neo4j-1.3.tar.gz

Copy the startup script from /var/opt/norduni/src/scripts/neo4j/ to /etc/init.d/ and then make neo4j start on boot.

Code Block
update-rc.d /etc/init.d/neo4j defaults

You need to shut down port 7474 from anything except localhost.

Code Block
iptables -A INPUT -p tcp --dport 7474 -s localhost -j ACCEPT iptables -A INPUT -p tcp --dport 7474 -j DROP

Save the rules and add the following line to your interface configuration so you will restore the rules on a reboot.

Code Block
bash -c "iptables-save > /etc/iptables.rules" vi /etc/network/interfaces iface eth0 inet static pre-up iptables-restore < /etc/iptables.rules

...

Create a file name niweb in /etc/apache2/conf.d/.

Code Block


WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIDaemonProcess processes=1 threads=25 # For neo4j-embedded
WSGIApplicationGroup %{GLOBAL}           # For neo4j-embedded
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi

Add niweb path to niweb.wsgi

Code Block
sys.path.append('/opt/norduni/src/niweb')

/etc/apache2.conf

Code Block
<IfModule mpm_prefork_module> StartServers 1 ServerLimit 1 MinSpareServers 5 MaxSpareServers 10 MaxClients 1 MaxRequestsPerChild 0 </IfModule>

Apache Shibboleth configuration

Code Block


        <LocationMatch "Shibboleth.sso">
           SetHandler None
        </LocationMatch>

        <Location "/accounts/login-federated/">
           AuthType shibboleth
           ShibRequireSession On
           require valid-user
           RequestHeader set X_REMOTE_USER %{eppn}e
           RequestHeader set EPPN %{eppn}e
           RequestHeader set GIVENNAME %{givenName}e
           RequestHeader set SN %{sn}e
           RequestHeader set MAIL %{mail}e
           RequestHeader set AFFILIATION %{affiliation}e   
           RequestHeader set ENTITLEMENT %{entitlement}e
        </Location>

Using the Virtualbox image

Download the OVA file, NOCLook Test Ubuntu 12.04.ova.

Username: norduni
Password: norduni+123

To start the app:

Code Block
sudo su /opt/norduni/start_dev_noclook.sh

Now you should be able surf to your machines ip and see the NOCLook app started.
The NOCLook username is also norduni and the password is norduni+123.

To add more hosts just go to /opt/norduni/tools/nerds/producers/nmap_services_py.

Code Block
./run json address to scan

After the data collection you insert the data using the consumer. (You have to stop the Django app first.)

Code Block
. /opt/norduni/src/niweb/env/bin/activate cd /opt/norduni/src/scripts python noclook_consumer.py -C test.conf -I

Child pages

Versions Compared

Old Version 54

New Version Current

Key

Setting up NOCLook

Setting up a local/development NOCLook

Deploying NOCLook

Deploying NOCLook

Collecting and processing Collecting and processing network data

Juniper Configuration Producer/Consumer

Juniper Configuration Producer/Consumer

nmap Producer/Consumer

Setting up old NOCLook version

Neo4j 1.3 REST stand alone server

Apache Shibboleth configuration

Using the Virtualbox image