Page History

...

title	Work in progress

Table of Contents

Setting up NOCLook

...

Code Block

$ sudo apt-get install python-setuptools git libpq-dev postgresql python-dev postfix nginx-full uwsgi uwsgi-plugin-python libffi-dev
$ sudo easy_install pip
$ sudo pip install virtualenv
$ sudo adduser --disabled-password --home /var/opt/norduni ni

# sudo apt-get install git-core python-virtualenv openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev

We are using postgresql but you can use any SQL database that Django supports. See Django database documentation for other supported SQL databases.

...

Neo4j database

Set password for database user and create a new databaseOracle java is recommended for Neo4j.

Code Block
$ sudo apt-u postgres psql postgres \password postgres Write password Write password again Ctrl+D sudo -u postgres createdb norduni

NORDUni repository

Get the NORDUni files.

Code Block
add-repository ppa:webupd8team/java $ sudo apt-get update $ sudo apt-uget niinstall -i $ pwd /var/opt/norduni $ git clone git://git.nordu.net/norduni.git

Python environment

oracle-java8-installer

Download neo4j-community from http://neo4j.com/download/. NORDUnet and SUNET run 2.1.8. 2.3.2 has been tested and did not work as expectedMake a virtual python environment.

Code Block
$ tar virtualenv norduni_environment

Making a virtual environment is also just a suggestion but it makes it easier to keep your system clean.

Python requirements

Install required python modules.

Code Block
$ . norduni_environment/bin/activate $ pip install -r norduni/requirements.txt

Django settings

Change the django settings.

Code Block
cd norduni/src/niweb/niweb/ cp generic_settings.py settings.py vi settings.py

Change at least the following settings.

Code Block
# Database settings DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'norduni', 'USER': 'postgres', 'PASSWORD': 'secret', 'HOST': 'localhost' } }

Neo4j >1.5 embedded with Python bindings

Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/)

Code Block
pip install neo4j-embedded export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ pip install /path/to/JPype-version.zip

Code Block
# Django Generic Login (r'^accounts/login/$', 'django.contrib.auth.views.login'), # Federated login #(r'^accounts/', include('niweb.apps.fedlogin.urls')),

Code Block
python manage.py syncdb python manage.py runserver 0.0.0.0:80

Now you should be able surf to your machines ip and see the NOCLook app started.

It is time to collect and insert some data.

Deploying NOCLook

Comment out the static media url in /opt/norduni/src/niweb/urls.py.

Code Block

title	urls.py

# Static serve
    #(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
    #    {'document_root': settings.STATIC_DEV_MEDIA}),

Install nginx, postfix and gunicorn.

Code Block
sudo apt-get install nginx postfix pip install gunicorn

Create a gunicorn start file.

Code Block

title	/opt/norduni/src/start_noclook.sh

#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
 --user=$USER --group=$GROUP --log-level=debug \
 --log-file=$LOGFILE 2>>$LOGFILE

Configure nginx.

Code Block

title	/etc/nginx/sites-available/default

server {
    listen 80;
    root /opt/norduni/src/niweb;
    server_name ni.example.net;
    access_log /var/log/ni/noclook-access.log;
    error_log /var/log/ni/noclook-error.log;

    location /static/ {
        root   /opt/norduni/src/niweb/;
        autoindex on;
        access_log   off;
        expires      30d; 
    }

    location / {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8000/;
    }   
}

Install supervisord and set up the following start script.

Code Block
easy_install supervisor echo_supervisord_conf > /etc/supervisord.conf

Code Block

title	/etc/init/supervisord.conf

description     "supervisord"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf

Add the noclook start script to the supervisor configuration.

Code Block

[program:noclook]
directory = /opt/norduni/src/niweb/
user = user
command = /opt/norduni/src/start_noclook.sh
stdout_logfile = /var/log/ni/supervisor_logfile.log
stderr_logfile = /var/log/ni/supervisor_err_logfile.log

Collecting and processing network data

To insert data you need to stop any python process that is using the Neo4j database. We hope to get the option to load more database instances in read-only mode in a near future then this could be avoided.

NORDUnet has a GIT repository called nistore and it is cloned to /opt/nistore/.

To start have a look at the NERDS README then clone the NERDS project.

Code Block
cd /opt/norduni/ mkdir tools cd tools git clone https://github.com/fredrikt/nerds.git

Juniper Configuration Producer/Consumer

The Juniper configuration producer can load Juniper configuration directly from the router via SSH or Juniper configuration files in XML format from disk.

Code Block

title	template.conf

[ssh]
user = view_account_user
password = not_so_secret_password

[sources]
remote = one.example.org two.example.org three.example.org
local = /var/conf/one.xml /var/conf/two.xml /var/conf/three.xml

xvfz neo4j-community-2.1.8-unix.tar.gz
$ sudo mv neo4j-community-2.1.8 /var/opt/.
$ sudo ln -s /var/opt/neo4j-community-2.1.8 /var/opt/neo4j-community
$ cd /var/opt/neo4j-community
$ sudo ./bin/neo4j-installer install

Set property keys to auto index in neo4j.

Code Block

$ sudo vi /var/opt/neo4j-community/conf/neo4j.properties

Add or update the following lines.
# Autoindexing

# Enable auto-indexing for nodes, default is false
node_auto_indexing=true

# The node property keys to be auto-indexed, if enabled
node_keys_indexable=name, description, ip_address, ip_addresses, as_number, hostname, hostnames, telenor_tn1_number, nordunet_id, version

# Enable auto-indexing for relationships, default is false
relationship_auto_indexing=true

# The relationship property keys to be auto-indexed, if enabled
relationship_keys_indexable=ip_address

Increase the number of files the neo4j user may concurrently access. A restart is required for the settings to take effect.

Code Block
$ sudo vi /etc/security/limits.conf Add the lines below to limits.conf. # User neo4j allowed concurrent files neo4j soft nofile 40000 neo4j hard nofile 40000

Code Block
$ sudo vi /etc/pam.d/su Uncomment the following line. session required pam_limits.so

After the restart neo4-service should be running.

Code Block
$ sudo service neo4j-service status Neo4j Server is running at pid 1475

Create full text index for nodes and relationships.

Code Block

$ curl -D - -H "Content-Type: application/json" --data '{"name" : "node_auto_index","config" : {"type" : "fulltext","provider" : "lucene"}}' -X POST http://localhost:7474/db/data/index/node/

HTTP/1.1 201 Created
*snip*

$ curl -D - -H "Content-Type: application/json" --data '{"name" : "relationship_auto_index","config" : {"type" : "fulltext","provider" : "lucene"}}' -X POST http://localhost:7474/db/data/index/relationship/

HTTP/1.1 201 Created
*snip*

Postgres database

Set password for database user and create a new database

Code Block

$ sudo -u postgres psql postgres
template1=# CREATE USER ni with PASSWORD 'secret';
template1=# CREATE DATABASE norduni;
template1=# GRANT ALL PRIVILEGES ON DATABASE norduni to ni;
template1=# ALTER DATABASE norduni OWNER TO ni;				#  Allow user ni to drop and create for restoring
template1=# ALTER USER ni CREATEDB;							#  and development purposes
template1=# \q

NORDUni repository

Get the NORDUni files.

Code Block
$ sudo -u ni -i $ pwd /var/opt/norduni $ git clone git://git.nordu.net/norduni.git

Python environment

Make a virtual python environment.

Code Block
$ virtualenv norduni_environment

Making a virtual environment is also just a suggestion but it makes it easier to keep your system clean.

Python requirements

Install required python modules.

Code Block
$ . norduni_environment/bin/activate $ pip install -r norduni/requirements/prod.txt

Django settings

Change the django settings.

Code Block
$ cd norduni/src/niweb/ $ cp dotenv .env $ vi .env

The following settings need to be changed.

Code Block
REPORTS_TO= DB_PASSWORD= DEFAULT_FROM_EMAIL= EMAIL_HOST= SECRET_KEY=

Check if your settings are ok.

Code Block
$ python manage.py syncdb $ python manage.py migrate apps.noclook $ python manage.py migrate actstream $ python manage.py migrate tastypie $ python manage.py collectstatic $ python manage.py runserver

Now you should be able connect to the machine with your browser on http://localhost:8000 and see the NOCLook app index page.

Deploying NOCLook

uwsgi

Create a uwsgi configuration file.

Code Block

$ sudo vi /etc/uwsgi/apps-available/noclook.ini

The following configuration should be a good start.

[uwsgi]
# Django-related settings
plugins = python
protocol = uwsgi
# the base directory (full path)
chdir           = /var/opt/norduni/norduni/src/niweb/
# Django's wsgi file
wsgi-file       = /var/opt/norduni/norduni/src/niweb/niweb/wsgi.py
env             = DJANGO_SETTINGS_MODULE=niweb.settings.prod
# the virtualenv (full path)
home            = /var/opt/norduni/norduni_environment
# logging
daemonize       = /var/log/uwsgi/app/noclook.log
# process-related settings
# master
master          = true
# maximum number of worker processes
processes       = 5
#threads        = 2
max-requests    = 5000
# the socket (use the full path to be safe
socket          = 127.0.0.1:8001
# clear environment on exit
vacuum          = true

Link the configuration in to the correct directory.

Code Block
sudo ln -s /etc/uwsgi/apps-available/noclook.ini /etc/uwsgi/apps-enabled/noclook.ini

Make temp dir and log dir writable by the uwsgi user (www-data on ubuntu)

Code Block

sudo chown -R ni:www-data /tmp/django_cache
sudo chmod -R g+w /tmp/django_cache
 
sudo chown -R ni:www-data /var/opt/norduni/norduni/src/niweb/logs/
sudo chmod -R g+w /var/opt/norduni/norduni/src/niweb/logs/

nginx

Setup new dhparam file 2048 should suffice, but if you like you can go with 4096 instead:

Code Block
$ sudo openssl dhparam -out /etc/ssl/dhparams.pem 2048

Configure nginx.

Code Block

title	/etc/nginx/sites-available/default

$ sudo vi /etc/nginx/sites-available/default

The following configuration should be a good start.

upstream django {    
    server 127.0.0.1:8001; # for a web port socket
}

server {
    listen         80;
    listen         [::]:80;
    server_name    ni.nordu.net;
    rewrite        ^ https://$server_name$request_uri? permanent;
}

server {
    listen 443;
    listen [::]:443 default ipv6only=on; ## listen for ipv6
    ssl on;
	ssl_certificate /etc/ssl/ni_nordu_net.crt;
    ssl_certificate_key /etc/ssl/ni_nordu_net.key;

    # https://cipherli.st
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_session_cache shared:SSL:10m;
    ssl_ecdh_curve secp384r1;
    ssl_dhparam /etc/ssl/dhparams.pem;
    
    server_name ni.nordu.net;

    location /static/ {
        alias         /var/opt/norduni/norduni/src/niweb/niweb/static/;
        autoindex    on;
        access_log   off;
        expires      30d; 
    }

    location /

Code Block

title	Blank output from the juniper_conf.py producer.

"host": {
    "juniper_conf": {
        "bgp_peerings": [
            {    
            "as_number": "", 
            "group": "", 
            "description": "", 
            "remote_address": "", 
            "local_address": "", 
            "type": ""
            },
        ], 
        "interfaces": [
            {
            "name": "", 
            "bundle": "", 
            "vlantagging": true/false, 
            "units": [
                {
        include        "address": [/etc/nginx/uwsgi_params;
        proxy_set_header   Host      "", 
          $host;
      ""
  proxy_set_header   X-Real-IP           ], $remote_addr;
        proxy_set_header   X-Forwarded-For     "description": "",  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto     "unit": "", $scheme;
        proxy_redirect        "vlanid": ""
off;
        uwsgi_pass  django;
    }   
}

SAML SP

If you want to set up NOCLook as a SAML SP you need to install the following packages and Python modules.

Code Block
$ sudo apt-get install libffi-dev xmlsec1 $ sudo -u ni -i $ . norduni_environment/bin/activate $ pip install djangosaml2

You then need to uncomment the lines in settings.py that imports and sets up djangosaml2. You also have to create a pysaml2 configuration.
All this is best described in the documentation at https://pypi.python.org/pypi/djangosaml2.

Local saml metadata

To speed up login you can use local metadata. This metadata still needs to be updated and verified, and for that you can use https://github.com/NORDUnet/metadata-updater

You need to configure djangosaml2 to use local metadata, and you will have to add the meta-dataupdater to cron, preferably by running crontab -e as the ni user. Once an hour is reasonable, once a day can be ok, once a week might be tiresome when the cert expires.

Collecting and processing network data

To insert data you need to stop any python process that is using the Neo4j database. We hope to get the option to load more database instances in read-only mode in a near future then this could be avoided.

NORDUnet has a GIT repository called nistore and it is cloned to /var/opt/norduni/nistore/.

To start have a look at the NERDS README then clone the NERDS project.

Code Block
cd /var/opt/norduni/ mkdir tools cd tools git clone https://github.com/fredrikt/nerds.git

Juniper Configuration Producer/Consumer

The Juniper configuration producer can load Juniper configuration directly from the router via SSH or Juniper configuration files in XML format from disk.

Code Block

title	template.conf

[ssh]
user = view_account_user
password = not_so_secret_password

[sources]
remote = one.example.org two.example.org three.example.org
local = /var/conf/one.xml /var/conf/two.xml /var/conf/three.xml

Code Block

title	Blank output from the juniper_conf.py producer.

"host": {
    "juniper_conf": {
        "bgp_peerings": [
            {    
            "as_number": "", 
            "group": "", 
            "description": "", 
            "remote_address": "", 
            "local_address": "", 
            "type": ""
            },
        ], 
        "interfaces": [
            {
            "name": "", 
            "bundle": "", 
            "vlantagging": true/false, 
            "units": [
                {
                "address": [
                "", 
                ""
                ], 
                "description": "", 
                "unit": "", 
                "vlanid": ""
                }
            ], 
            "tunnels": [
       }
            ], 
            "tunnels": [
            {
            "source": "", 
            "destination": ""
            }
            ], 
            "description": ""
            }, 
        ],
        "name": ""
        }, 
        "version": 1, 
        "name": ""        
    }

...

Code Block

title	Blank output from NORDUnet site documentation example

{
    "host": {
        "csv_producer": {
            "address": "", 
            "area": "", 
        {
    "cityhost": "", {
            "comment"csv_producer": "", {
            "countryaddress": "", 
            "floorarea": "", 
            "latitudecity": "", 
            "longitudecomment": "", 
            "meta_typecountry": "", 
            "namefloor": "", 
            "node_typelatitude": "", 
            "owner_idlongitude": "", 
            "postcodemeta_type": "", 
            "responsible_forname": "", 
            "roomnode_type": "", 
            "siteowner_typeid": "", 
            "telenor_subscription_idpostcode": "", 
        }, 
        "nameresponsible_for": "", 
        "version": 1
    }
}

The consumer script should only be run once as it does not update the sites, only creates new.

The JSON file directory is then inserted in to the database using noclook_site_csv_consumer.py.

Change the path at the top of the script to be able to import norduni_client.py.

Then run:

Code Block
python noclook_site_csv_consumer.py -D /path/to/site_files/json

Daily database update

The producers are run with a cron job and the script noclook_consumer.py is used to run the three inserting/updating scripts (noclook_juniper_consumer.py, noclook_alcatel_consumer.py and noclook_nmap_consumer.py).

Change the path at the top of the script to be able to import norduni_client.py.

Code Block
[data] juniper_conf = /path/to/juniper/json nmap_services = /path/to/nmap/json alcatel_isis = /path/to/alcate/json noclook = #Used for loading backup.

Then run:

Code Block
python noclook_consumer.py -C template.conf -I

Setting up a local/development NOCLook

     "room": "", 
            "site_type": "", 
            "telenor_subscription_id": ""
        }, 
        "name": "", 
        "version": 1
    }
}

The consumer script should only be run once as it does not update the sites, only creates new.

The JSON file directory is then inserted in to the database using noclook_site_csv_consumer.py.

Change the path at the top of the script to be able to import norduni_client.py.

Then run:

Code Block
python noclook_site_csv_consumer.py -D /path/to/site_files/json

Daily database update

The producers are run with a cron job and the script noclook_consumer.py is used to run the three inserting/updating scripts (noclook_juniper_consumer.py, noclook_alcatel_consumer.py and noclook_nmap_consumer.py).

Change the path at the top of the script to be able to import norduni_client.py.

Code Block
[data] juniper_conf = /path/to/juniper/json nmap_services = /path/to/nmap/json alcatel_isis = /path/to/alcate/json noclook = #Used for loading backup.

Then run:

Code Block
python noclook_consumer.py -C template.conf -I

Setting up a local/development NOCLook

Code Block

# Clone a convenience repo
$ git clone https://github.com/NORDUnet/norduni-developer
$ cd norduni-developer
# Start dependencies
$ ./start.sh

# Clone NOCLook project repo
$ git clone https://git.nordu.net/norduni.git
$ cd norduni

Code Block

git clone https://git.nordu.net/norduni.git
git checkout neo4jdb-python
# Download neo4j docker image and start it
docker pull tpires/neo4j
docker run -d -v /path_to_repo/norduni/docker/neo4j.properties:/var/lib/neo4j/conf/neo4j.properties -v /opt/docker/neo4jdata:/var/lib/neo4j/data -p 7474:7474 tpires/neo4j
# Create the indexes with curl
curl -D - -H "Content-Type: application/json" --data '{"name" : "node_auto_index","config" : {"type" : "fulltext","provider" : "lucene"}}' -X POST http://localhost:7474/db/data/index/node/
curl -D - -H "Content-Type: application/json" --data '{"name" : "relationship_auto_index","config" : {"type" : "fulltext","provider" : "lucene"}}' -X POST http://localhost:7474/db/data/index/relationship/
# Create a virtualenv and activate it
$ virtualenv env
$ . env/bin/activate
# Install the python packages
pip$ install paver
pip install -r /path_to_repo/requirementsrequirements/dev.txt
# Create a settings.py from the template /path_to/repo/src/niweb/niweb
cp /path_to_repo/ file
$ cp src/niweb/niweb/generic_settings.py /path_to/repo/src/niwebdotenv src/niweb/settings.pydevenv
# Sync the db
python /path_to_repo/src/niweb/manage.py syncdb
$ python /path_to_repo/src/niweb/manage.py migrate apps.noclooksyncdb
$ python /path_to_repo/src/niweb/manage.py migrate actstream

# Run the app
$ python /path_to_repo/src/niweb/manage.py migrate tastypierunserver

Upgrading to newest versions

This is the general procedure for upgrading to newest version of norduni.

Code Block

# stash current local changes and update
$ git stash
$ git pull origin master
$ git stash apply
 
# Run the appmigrations
$ python /path_to_repo/src/niweb/manage.py runservermigrate
  
# OptionalPip postgres instead of sqlite3, don't forget to change database settings in settings.py.
# Download postgres docker image and start it
docker pull orchardup/postgresql
docker run -d -p 5432:5432 -e
 POSTGRESQL_USER=norduni -e POSTGRESQL_PASS=docker -e 
POSTGRESQL_DB=norduni -v 
/opt/docker/postgresql_data/:/var/lib/postgresql/ orchardup/postgresql
update requirements
$ pip install -U -r requirements/prod.txt
 
# Collect statics
$ python /path_to_repo/src/niweb/manage.py collectstatic
 
# Restart uwsgi
$ sudo services uwsgi restart

Child pages

Versions Compared

Old Version 3

New Version Current

Key

Setting up NOCLook

Neo4j database

NORDUni repository

Python environment

Python requirements

Django settings

Neo4j >1.5 embedded with Python bindings

Deploying NOCLook

Collecting and processing network data

Juniper Configuration Producer/Consumer

Postgres database

NORDUni repository

Python environment

Python requirements

Django settings

Deploying NOCLook

uwsgi

nginx

SAML SP

Local saml metadata

Collecting and processing network data

Juniper Configuration Producer/Consumer

Daily database update

Setting up a local/development NOCLook

Daily database update

Setting up a local/development NOCLook

Upgrading to newest versions