Page History

# [Overview|#overview]
# [Producer|#producer]
# [Consumer|#consumer]
# [Repository|#repo]
# [Data Model|#datamodel]
# [Searching in NOCLook|#search]
# [Indexed properties|#index]
# [Backup and Restore|#backuprestore]
# [Links|#links]
# [How to set up a NI suite|#setup]


{anchor:overview}
h3. Overview
NORDUnet Network inventory project is divided in to three parts. One part that gathers the data (producer), another part that uses the data (consumer) and a repository for the raw data.

{anchor:producer}
h3. Producer
For the producer we use the [NERDS projects|https://github.com/fredrikt/nerds] JSON format to save the data. How to actually collect data depends on what you want to collect data from.

{anchor:consumer}
h3. Consumer
The only NORDUnet consumer today is [NOCLook|http://git.nordu.net/?p=norduni.git;a=shortlog;h=refs/heads/operation] which is a web gui made with the Django web framework and Neo4j graph database as back end for the [inventory data model|https://portal.nordu.net/display/nordunetops/Network+Inventory+Project#NetworkInventoryProject-pdatamodel]. The data is added to the Neo4j database via the REST interface using [norduni_client.py|http://git.nordu.net/?p=norduni.git;a=history;f=src/niweb/norduni_client.py] which builds on [neo4j-rest-client|https://github.com/versae/neo4j-rest-client].

Neo Tech has release official [python bindings|http://pypi.python.org/pypi/neo4j-embedded] and they are used in the [master branch|http://git.nordu.net/?p=norduni.git;a=summary] of NOCLook. The speed improvement is great so they will be used in the operational NOCLook in a near future.

{anchor:repo}
h3. Repository
[Git|http://git-scm.com/] is used to handle the revisions of the collected data.

{anchor:datamodel}
h3. Data model

!NI_Data_Model.png|thumbnail!
Hopefully these entities and relationships are enough.

!NI_Prototype_Data.png|thumbnail!
The data that was inserted manually in the prototype.

{anchor:search}
h3. Searching in NOCLook
In addition to the search box at the top of the page you can also use the URL field for searching.

{code}
/findall/[value]/
ex. /findall/10/
/findall/[key]/[value]/
ex. /findall/link/10/
/findin/[node type]/[value]/
ex. /findin/optical-node/10/
/findin/[node type]/[key]/[value]/
ex. /findin/optical-node/link/10/
/getall/[node type]/
ex. /getall/optical-node/
{code}

Only L2VPN services are set up in this way.

$ cat site.json
{
	"node_name": "DK-AAL",
	"node_type": "/api/v1/node_type/site/",
	"node_meta_type": "Location",
	"node": {
		"address": "Selma Lagerlöfsvej 300",
		"area": "Aalborg",
		"country": "Denmark",
		"country_code": "DK",
		"latitude": "57.011944",
		"longitude": "9.990560",
		"owner_id": "AAU",
		"postarea": "Aalborg",
		"postcode": "9220"
	}
}


$ curl -i -H "Authorization: ApiKey apiuser:secret" -H "Content-Type: application/json" -X POST --data @site.json

curl -k -H "Authorization: ApiKey apiuser:secret" -H "Content-Type: application/json" -X POST --data '{"node_name": "NU-SXXXXX1", "operational_state": "In service", "route_distinguisher": "rd1", "vrf_target": "vt1", "end_points": [{"device": "se-tug.nordu.net", "port":"l2vpn_test_port"}, {"device": "se-fre.nordu.net", "port":"l2vpn_test_port"}]}' https://nidev-consumer.nordu.net/api/v1/l2vpn/

Return:
{
    "created": "2013-02-20T16:45:41.228176", 
    "creator": "/api/v1/user/59/", 
    "description": null, 
    "handle_id": 11717, 
    "modified": "2013-02-20T16:45:41.228213", 
    "modifier": "/api/v1/user/59/", 
    "node": {
        "handle_id": 11717, 
        "name": "NU-SXXXXX1", 
        "node_type": "Service", 
        "operational_state": "In service", 
        "service_class": "MPLS", 
        "service_type": "L2VPN"
    }, 
    "node_id": 13707, 
    "node_meta_type": "logical", 
    "node_name": "NU-SXXXXX1", 
    "node_type": "/api/v1/node_type/service/", 
    "object_path": "/service/11717/", 
    "operational_state": "In service", 
    "relationships": [
        "/api/v1/relationship/47575/", 
        "/api/v1/relationship/47576/", 
        "/api/v1/relationship/47577/"
    ], 
    "resource_uri": "/api/v1/l2vpn/11717/", 
    "route_distinguisher": "rd1", 
    "service_type": "L2VPN", 
    "vrf_target": "vt1"
}

curl -k -H "Authorization: ApiKey apiuser:secret" -H "Content-Type: application/json" -X PUT --data '{"operational_state": "In service", "description": "Test REST API call l2vpn"}' https://nidev-consumer.nordu.net/api/v1/l2vpn/NU-SXXXXX1/

Return:
{
    "created": "2013-02-20T16:45:41.228176", 
    "creator": "/api/v1/user/59/", 
    "description": "Test REST API call l2vpn", 
    "handle_id": 11717, 
    "modified": "2013-02-20T16:51:16.077192", 
    "modifier": "/api/v1/user/59/", 
    "node": {
        "description": "Test REST API call l2vpn", 
        "handle_id": 11717, 
        "name": "NU-SXXXXX1", 
        "node_type": "Service", 
        "operational_state": "In service", 
        "service_class": "MPLS", 
        "service_type": "L2VPN"
    }, 
    "node_id": 13707, 
    "node_meta_type": "logical", 
    "node_name": "NU-SXXXXX1", 
    "node_type": "/api/v1/node_type/service/", 
    "object_path": "/service/11717/", 
    "operational_state": "In service", 
    "relationships": [
        "/api/v1/relationship/47577/", 
        "/api/v1/relationship/47576/", 
        "/api/v1/relationship/47575/"
    ], 
    "resource_uri": "/api/v1/l2vpn/11717/", 
    "route_distinguisher": "rt1", 
    "vrf_target": "vrf1"
}

# Backup the SQL database
pg_dump norduni > /var/opt/norduni

{anchor:index}
h3. Indexed properties
Neo4j provides [Lucene indexes|http://lucene.apache.org/java/docs/index.html] for nodes and relationships. A "search" index is created with the properties that will be searched when a user uses the search bar. Some optimization indexes was also created to speed up the consuming or presentation.

*Node search index:*
name (all nodes)
description (interface and unit nodes)
ip_addresses (unit and host nodes)
as_number (peering partner nodes)
hostname (host nodes)

*Relationship search index:*
ip_address (Uses and Depends_on relationships)

*Node node_types index*
node_type (all nodes)

All property values above is also indexed under the index key *all* in all indexes. For example in the search index you can query the all-key for both name, description, ip_addresses, as_number and hostname.

{anchor:backuprestore}
h3. Backup and Restore
You can backup the Postgresql and Neo4j databases using these scripts:
{code}
# Backup the SQL database
su - postgres -c "pg_dumpall > /opt/nistore/producers/noclook/sql/postgres.sql"
# Run NOCLook producer for backup purpose
cd /opt/norduni/src/scripts/
rm /opt/nistore/producers/noclook/json/*
./noclook_producer.py -O /opt/nistore/producers/noclook/json/
# Push the changes to nistore master
/usr/local/sbin/ni-push.sh -r /opt/nistore/
{code} 

{code|title:restore.conf}

# All producers need to be listed here with a path data or empty
[data]
juniper_conf = 
nmap_services_py = 
alcatel_isis = 
noclook = /opt/nistore/producers/noclook/json/
{code}

To restore the data from a backup use psql and noclook_consumer.py.
{code}
sudo -unagios_checkmk = 
cfengine_report =  

 postgres psql -f /opt/nistore/producers/noclook/sql/postgres.sql postgresnorduni
noclook_consumer.py -C restore.conf -I
{code}

$ sudo service uwsgi start|stop|restart

{anchor:setup}
h3. How to set up a NI suite
[Setting up a NORDUni instance]