Setting up NOCLook
NOCLook is the main GUI component, a django webapp that allows users to access the data stored in NI, it is also often what people refer to when they say NI.
This guide is written for Ubuntu 18.04 and python3.
NEO4J database
The official neo4j installation guide for version 3.x is the reference for this part.
Install openjdk-java8
$ sudo apt-get update $ sudo apt-get install openjdk-8-jre $ java -version openjdk version "1.8.x"
NEO4J
$ wget -O - https://debian.neo4j.org/neotechnology.gpg.key | sudo apt-key add - $ echo 'deb http://debian.neo4j.org/repo stable/' | sudo tee -a /etc/apt/sources.list.d/neo4j.list $ sudo apt-get update $ sudo apt-get install neo4j=1:3.5.5
Configuration
$ sudo vim /etc/neo4j/neo4j.conf Add the following: #******************* # NI specific config #******************* # Autoindexing # Enable auto-indexing for nodes, default is false node_auto_indexing=true # The node property keys to be auto-indexed, if enabled node_keys_indexable=name, description, ip_address, ip_addresses, as_number, hostname, hostnames, telenor_tn1_number, nordunet_id, version # Enable auto-indexing for relationships, default is false relationship_auto_indexing=true # The relationship property keys to be auto-indexed, if enabled relationship_keys_indexable=ip_address
$ sudo rm /var/lib/neo4j/data/dbms/auth # Note the extra space before the command to avoid saving password in bash history $ sudo -u neo4j neo4j-admin set-initial-password your_awesome_password $ sudo service neo4j restart
Setup limits for neo4j
$ sudo vim /etc/security/limits.conf neo4j soft nofile 60000 neo4j hard nofile 60000
Postgres database
Set password for database user and create a new database
$ sudo apt-get install postgresql $ sudo -u postgres psql postgres template1=# CREATE USER ni with PASSWORD 'secret'; template1=# CREATE DATABASE norduni; template1=# GRANT ALL PRIVILEGES ON DATABASE norduni to ni; template1=# ALTER DATABASE norduni OWNER TO ni; # Allow user ni to drop and create for restoring template1=# ALTER USER ni CREATEDB; # and development purposes template1=# \q
NOCLook
Before installing NOCLook you need to install the required system libraries
$ sudo apt-get install git python3-pip libpq-dev $ sudo pip3 install -U pip $ sudo pip3 install virtualenv $ sudo adduser --disabled-password --home /var/opt/norduni ni
Now you are ready to install NOCLook, start by changing to the ni user.
$ sudo -u ni -i $ pwd /var/opt/norduni $ git clone https://code.nordu.net/norduni.git # Create virtual env $ virtualenv norduni_environment # Activate virtual env $ . norduni_environment/bin/activate # Install python dependencies $ pip install -r norduni/requirements/prod.txt
Configure NOCLook
$ cd /var/opt/norduni/norduni/src/niweb/ $ cp dotenv .env $ vi .env
You need to setup the following settings:
NEO4J_USERNAME=neo4j NEO4J_PASSWORD= REPORTS_TO= SECURITY_REPORTS_TO= DB_PASSWORD= ALLOWED_HOSTS=ni.yourdomain.tld localhost DEFAULT_FROM_EMAIL= EMAIL_HOST= SECRET_KEY= GOOGLE_MAPS_API_KEY=
The secret key should be at least 50 chars long consisting of the following characters: 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)'
You can use the following snippit to gennerate such a string locally:
# Missing ! since bash is sad when you use ! in anything, and pyton thinks \! is to be read as both \ and ! $ python -c "import random; print(''.join([random.SystemRandom().choice('abcdefghijklmnopqrstuvwxyz0123456789@#$%^&*(-_=+)') for i in range(50)]))"
Migrate databases and check config
# To make it easier for yourself set DJANGO_SETTINGS_MODULE=niweb.settings.prod in your bashprofile/bashrc $ cd /var/opt/norduni/norduni/src/niweb $ python manage.py migrate $ python manage.py collectstatic $ python manage.py runserver $ rm -r /tmp/django_cache
Create local superuser
$ python manage.py createsuperuser
Deploying NOCLook
Start by installing uwsgi and nginx.
$ sudo apt-get install nginx-full uwsgi uwsgi-plugin-python
UWSGI
$ sudo vi /etc/uwsgi/apps-available/noclook.ini The following configuration should be a good start. [uwsgi] # Django-related settings plugins = python protocol = uwsgi # the base directory (full path) chdir = /var/opt/norduni/norduni/src/niweb/ # Django's wsgi file wsgi-file = /var/opt/norduni/norduni/src/niweb/niweb/wsgi.py env = DJANGO_SETTINGS_MODULE=niweb.settings.prod # the virtualenv (full path) home = /var/opt/norduni/norduni_environment # logging daemonize = /var/log/uwsgi/app/noclook.log # process-related settings # master master = true # maximum number of worker processes processes = 5 #threads = 2 max-requests = 5000 # the socket (use the full path to be safe socket = 127.0.0.1:8001 # clear environment on exit vacuum = true # for now we run uwsgi in lazy-apps, due to neo4j session problems lazy-apps = true # less noisy uwsgi logs (especially with sentry) ignore-sigpipe = true ignore-write-errors = true disable-write-exception = true
Link the configuration in to the correct directory.
$ sudo ln -s /etc/uwsgi/apps-available/noclook.ini /etc/uwsgi/apps-enabled/noclook.ini
Make temp dir and log dir writable by the uwsgi user (www-data on ubuntu)
sudo chown -R ni:www-data /tmp/django_cache sudo chmod -R g+rw /tmp/django_cache sudo chown -R ni:www-data /var/opt/norduni/norduni/src/niweb/logs/ sudo chmod -R g+w /var/opt/norduni/norduni/src/niweb/logs/
Finally restart uwsgi
$ sudo service uwsgi restart
NGINX
$ sudo vi /etc/nginx/sites-available/default # The following configuration should be a good start. # Remember certificates or # sudo openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:4096 -keyout /etc/ssl/private/ni_nordu_net.key -out /etc/ssl/certs/ni_nordu_net.crt upstream django { server 127.0.0.1:8001; # for a web port socket } server { listen 80; listen [::]:80; server_name ni.nordu.net; return 301 https://$server_name$request_uri; } server { listen 443; listen [::]:443 default; ## listen for ipv6 server_name ni.nordu.net; ssl on; ssl_certificate /etc/ssl/certs/ni_nordu_net.crt; ssl_certificate_key /etc/ssl/private/ni_nordu_net.key; # https://mozilla.github.io/server-side-tls/ssl-config-generator/ modern ssl_session_timeout 1d; ssl_session_cache shared:SSL:50m; ssl_session_tickets off; ssl_protocols TLSv1.2; ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256'; ssl_prefer_server_ciphers on; location /static/ { alias /var/opt/norduni/norduni/src/niweb/niweb/static/; autoindex on; access_log off; expires 30d; } location / { include /etc/nginx/uwsgi_params; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; uwsgi_pass django; } }
Then restart nginx (still remember to setup ssl certificates)
$ sudo service nginx restart
SAML SP
If you want to set up NOCLook as a SAML SP you need to install the following packages and Python modules.
$ sudo apt-get install libffi-dev xmlsec1 $ sudo -u ni -i $ sudo vim /var/opt/norduni/norduni/src/niweb/.env # Add SAML_ENABLED=true # Set the login url to point to your IDP LOGIN_URL=/saml2/login/?idp=https://idp.nordu.net/idp/shibboleth # If you want to disable normal django login then add DJANGO_LOGIN_DISABLED=true
You also have to create a pysaml2 configuration.
All this is best described in the documentation at https://pypi.python.org/pypi/djangosaml2.
Local saml metadata
To speed up login you can use local metadata. This metadata still needs to be updated and verified, and for that you can use https://github.com/NORDUnet/metadata-updater
You need to configure djangosaml2 to use local metadata, and you will have to add the meta-dataupdater to cron, preferably by creating a cron.d file as that runs it as the ni user. Once an hour is reasonable, once a day can be ok, once a week might be tiresome when a cert is rolled.
Customising look and feel
It is now possible to change some of the NI styling.
In .env you can set the following:
# Displayed where it would normally say NORDUnet (e.g. on the front page) BRAND=NORDUnet # Displayed just above the NOCLook text in the logo LOGO_SUBTEXT=DEV # Change the logo color LOGO_COLOR='#e67528' # Change link colors LINK_COLOR='#e67528' LINK_HOVER='#71360a'
Upgrade from NI 3 to 3.3
There is not a lot that needs to be done when updating from 3 to 3.3, mostly you need to update your `.env` config.
After updating .env follow the General Upgrade NI guide, only be aware that you should nuke your `urls.py`.
Python2 if needed
Moving forward NI will be python 3 focused, but for now it should still work with python 2. One thing you need to do to have python2 support is install the requirements in `requirements/py2.txt` by running:
$ sudo -u ni -i # Activate virtual env $ . norduni_environment/bin/activate # Install python dependencies $ pip install -r norduni/requirements/py2.txt
General Upgrade NI
# In norduni directory as the NI user $ sudo -u ni -i $ cd norduni $ git stash $ git pull origin master $ git stash pop # You might get conflict e.g. in urls.py # Resolve it and run git reset $ rm -r src/niweb/norduniclient # Delete old virtualenv and create a new $ cd .. $ rm -r norduni_environment $ virtualenv norduni_environment $ . norduni_environment/bin/activate $ pip install -U pip $ pip install -U -r norduni/requirements/prod.txt # If running python 2.7 also run $ pip install -r norduni/requirements/py2.txt # update norduni/src/niweb/.env to have: NEO4J_USERNAME=neo4j NEO4J_PASSWORD=your_awesome_password NEO4J_RESOURCE_URI=bolt://localhost:7687 # If you have saml enabled you need to add the following to norduni/src/niweb/apps/saml2auth/config.py # Just beside the key_file and cert_file entries. 'encryption_keypairs': [{ 'key_file': path.join(BASEDIR, 'sp-key.pem'), # private part 'cert_file': path.join(BASEDIR, 'sp-cert.pem'), # public part }], # finally run migrate and collect statics $ python norduni/src/niweb/manage.py migrate --settings=niweb.settings.prod $ python norduni/src/niweb/manage.py collectstatic --settings=niweb.settings.prod
Then you can restart uwsgi.
$ rm -r /tmp/django_cache $ sudo service uwsgi restart
Troubleshooting
If you run into problems you can use `python src/niweb/manage.py -h` to see if there are errors.
- `AttributeError: 'NoneType' object has no attribute 'session'` is caused by wrong credentials (or missing configuration there of)
- `ImportError: cannot import name contextmanager` or `ImportError: cannot import name IntegrityError` - you need to delete `src/niweb/norduniclient`
- `ImportError: No module named django.core.exceptions` (you need to source your virtual environment)
- `ImportError: No module named neo4j.v1.exceptions` - Problems with virtualenv installed dependencies, delete your virtualenv and install again
- `neo4j.exceptions.AuthError: The client is unauthorized due to authentication failure.` - your password contains some characters that got mangled, e.g. # or @, remove
- `IOError: [Errno 13] Permission denied: '/tmp/django_cache/'` - the directory is probably owned by ni user, and not www-data, which uwsgi runs as. Remove the dir, and restart uwsgi
Example configs
.env
# Neo4j settings NEO4J_RESOURCE_URI=bolt://localhost:7687 NEO4J_MAX_DATA_AGE=24 NEO4J_MAX_POOL_SIZE=50 NEO4J_USERNAME=neo4j NEO4J_PASSWORD=<%= $neo4j_password %> # Report settings REPORTS_TO=<%= $reports_to %> # Optional CC and BCC and EXTRA_REPORT_TO REPORTS_CC=<%= $reports_cc %> REPORTS_BCC=<%= $reports_bcc %> EXTRA_REPORT_TO=<%= $extra_reports_to %> SECURITY_REPORTS_TO=<%= $security_reports_to %> # Database settings DB_ENGINE=django.db.backends.postgresql_psycopg2 DB_NAME=norduni DB_USER=ni DB_PASSWORD=<%= $postgres_password %> DB_HOST=localhost DB_PORT=5432 # Cache settings CACHE_BACKEND=django.core.cache.backends.filebased.FileBasedCache CACHE_LOCATION=/tmp/django_cache # General settings ALLOWED_HOSTS=<%= $server_name %> <%= $allowed_hosts %> localhost # If using sentry SENTRY_DSN=<%= $sentry_dsn %> DEFAULT_FROM_EMAIL=django@<%= $server_name %> EMAIL_HOST=localhost #EMAIL_HOST_PASSWORD= #EMAIL_HOST_USER= #EMAIL_PORT= # Secret key SECRET_KEY=<%= $secret_key %> # Google apis key GOOGLE_MAPS_API_KEY=<%= $google_api_key %> # Login # if $saml_enabled: LOGIN_URL=/saml2/login/?idp=https://idp.nordu.net/idp/shibboleth SAML_ENABLED=true # if $django_login_disabled DJANGO_LOGIN_DISABLED=true # Branding BRAND=SUNET LOGO_SUBTEXT=SUNET LOGO_COLOR='#e67528' LINK_COLOR='#e67528' LINK_HOVER='#71360a'