Page History

Wiki Markup
h5. Page index

...

Overview

NORDUnet Network inventory project is divided in to three parts. One part that gathers the data (producer), another part that uses the data (consumer) and a repository for the raw data.

...

Producer

For the producer we use the NERDS projects JSON format to save the data. How to actually collect data depends on what you want to collect data from.

...

Consumer

The only NORDUnet consumer today is NOCLook which is a web gui made with the Django web framework and Neo4j graph database as back end for the inventory data model. The data is added to the Neo4j database via the REST interface using norduni_client.py which builds on neo4j-rest-client.

Neo Tech has release official python bindings and they are used in the master branch of NOCLook. The speed improvement is great so they will be used in the operational NOCLook in a near future.

...

Repository

Git is used to handle the revisions of the collected data.

...

Data model

Image Removed
Hopefully these entities and relationships are enough.

Image Removed
The data that was inserted manually in the prototype.

...

Searching in NOCLook

In addition to the search box at the top of the page you can also use the URL field for searching.

Code Block



# [Overview|#overview]
# [Producer|#producer]
# [Consumer|#consumer]
# [Repository|#repo]
# [Data Model|#datamodel]
# [Searching in NOCLook|#search]
# [Indexed properties|#index]
# [Backup and Restore|#backuprestore]
# [Links|#links]
# [How to set up a NI suite|#setup]


{anchor:overview}
h3. Overview
NORDUnet Network inventory project is divided in to three parts. One part that gathers the data (producer), another part that uses the data (consumer) and a repository for the raw data.

{anchor:producer}
h3. Producer
For the producer we use the [NERDS projects|https://github.com/fredrikt/nerds] JSON format to save the data. How to actually collect data depends on what you want to collect data from.

{anchor:consumer}
h3. Consumer
The only NORDUnet consumer today is [NOCLook|http://git.nordu.net/?p=norduni.git;a=shortlog;h=refs/heads/operation] which is a web gui made with the Django web framework and Neo4j graph database as back end for the [inventory data model|https://portal.nordu.net/display/nordunetops/Network+Inventory+Project#NetworkInventoryProject-pdatamodel]. The data is added to the Neo4j database via the REST interface using [norduni_client.py|http://git.nordu.net/?p=norduni.git;a=history;f=src/niweb/norduni_client.py] which builds on [neo4j-rest-client|https://github.com/versae/neo4j-rest-client].

Neo Tech has release official [python bindings|http://pypi.python.org/pypi/neo4j-embedded] and they are used in the [master branch|http://git.nordu.net/?p=norduni.git;a=summary] of NOCLook. The speed improvement is great so they will be used in the operational NOCLook in a near future.

{anchor:repo}
h3. Repository
[Git|http://git-scm.com/] is used to handle the revisions of the collected data.

{anchor:datamodel}
h3. Data model

!NI_Data_Model.png|thumbnail!
Hopefully these entities and relationships are enough.

!NI_Prototype_Data.png|thumbnail!
The data that was inserted manually in the prototype.

{anchor:search}
h3. Searching in NOCLook
In addition to the search box at the top of the page you can also use the URL field for searching.

{code}
/findall/[value]/
ex. /findall/10/
/findall/[key]/[value]/
ex. /findall/link/10/
/findin/[node type]/[value]/
ex. /findin/optical-node/10/
/findin/[node type]/[key]/[value]/
ex. /findin/optical-node/link/10/
/getall/[node type]/
ex. /getall/optical-node/
{code}

The URL field version of the search only finds complete strings so you can't use wild cards but it is at lease case insensitive.

...



{anchor

...

Indexed properties

Neo4j provides Lucene indexes for nodes and relationships. A "search" index is created with the properties that will be searched when a user uses the search bar. Some optimization indexes was also created to speed up the consuming or presentation.

Node search index:
name (all nodes)
description (interface and unit nodes)
ip_addresses (unit and host nodes)
as_number (peering partner nodes)
hostname (host nodes)

Relationship search index:
ip_address (Uses and Depends_on relationships)

Node node_types index
node_type (all nodes)

All property values above is also indexed under the index key all in all indexes. For example in the search index you can query the all-key for both name, description, ip_addresses, as_number and hostname.

...

Links

:index}
h3. Indexed properties
Neo4j provides [Lucene indexes|http://lucene.apache.org/java/docs/index.html] for nodes and relationships. A "search" index is created with the properties that will be searched when a user uses the search bar. Some optimization indexes was also created to speed up the consuming or presentation.

*Node search index:*
name (all nodes)
description (interface and unit nodes)
ip_addresses (unit and host nodes)
as_number (peering partner nodes)
hostname (host nodes)

*Relationship search index:*
ip_address (Uses and Depends_on relationships)

*Node node_types index*
node_type (all nodes)

All property values above is also indexed under the index key *all* in all indexes. For example in the search index you can query the all-key for both name, description, ip_addresses, as_number and hostname.

{anchor:backuprestore}
h3. Backup and Restore
You can backup the Postgresql and Neo4j databases using these scripts:
{code}
# Backup the SQL database
su - postgres -c "pg_dumpall > /opt/nistore/producers/noclook/sql/postgres.sql"
# Run NOCLook producer for backup purpose
cd /opt/norduni/src/scripts/
rm /opt/nistore/producers/noclook/json/*
./noclook_producer.py -O /opt/nistore/producers/noclook/json/
# Push the changes to nistore master
/usr/local/sbin/ni-push.sh -r /opt/nistore/
{code} 

{code|title:restore.conf}
# All producers need to be listed here with a path data or empty
[data]
juniper_conf = 
nmap_services = 
alcatel_isis = 
noclook = /opt/nistore/producers/noclook/json/
{code}

To restore the data from a backup use psql and noclook_consumer.py.
{code}
sudo -u postgres psql -f /opt/nistore/producers/noclook/sql/postgres.sql postgres
noclook_consumer.py -C restore.conf -I
{code}

When using postgres you have to reset the autoincrementing handle_id property.
{code}
python manage.py sqlsequencereset noclook
{code}

Copy output from the previous command and paste it in the dbshell.
{code}
python manage.py dbshell
{code}

Now everything should be restored and ready to be updated as usual with noclook_consumer.py.

{anchor:links}
h3. Links
https://ni.nordu.net

...



http://nidev-consumer.nordu.net

...

 (bleeding edge)

...



Issue tracker:
https://project.nordu.net/browse/NORDUNIDEV

...

How to set up a NI suite

...



{anchor:setup}
h3. How to set up a NI suite
[Setting up a NORDUni instance]

Child pages

Versions Compared

Old Version 28

New Version 29

Key