...
Collecting and processing network data
To insert data you need to stop any python process that is using the Neo4j database. We hope to get the option to load more database instances in read-only mode in a near future then
this could be avoided.
Juniper Configuration Producer/Consumer
The Juniper configuration producer can load Juniper configuration directly from the router via SSH or Juniper configuration files in XML format from disk.
| Code Block | ||
|---|---|---|
| ||
[ssh]
user = view_account_user
password = not_so_secret_password
[sources]
remote = one.example.org two.example.org three.example.org
local = /var/conf/one.xml /var/conf/two.xml /var/conf/three.xml
|
| Code Block | ||
|---|---|---|
| ||
"host": { "juniper_conf": { "bgp_peerings": [ {"host": { "juniper_conf": { "bgp_peerings": [ { "as_number": "", "group": "", "description": "", "remote_address": "", "local_address": "", "type": "" }, ], "interfaces": [ { "name": "", "bundle": "", "vlantagging": true/false, "units": [ { "address": [ "", "" ], "description": "", "unit": "", "vlanid": "" } ], "tunnels": [ { "source": "", "destination": "" } ], "description": "" }, ], "name": "" }, "version": 1, "name": "" } |
Alcatel ISIS Producer
The JSON files can the be inserted using noclook_juniper_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Edit the template.conf file with the correct path to the Juniper NERDS files.
| Code Block |
|---|
[data]
juniper_conf = /path/to/juniper/json
nmap_services =
alcatel_isis =
noclook =
|
Then run:
| Code Block |
|---|
python noclook_juniper_consumer.py -C template.conf
|
Alcatel-Lucent ISIS Producer/Consumer
Using the output from the "show isis database detail" on a Cisco router
connected to the Alcatel-Lucent DCN network, nodes and their neighbors
will be grouped.
To get a more human readable result use the IOS command "clns" to map
the NSAP address to a hostname. eg. clns host hostname NSAP_address.
You can also provide a mapping CSV file. The mandatory columns are
osi_address and name. All following columns will be added to the JSON
output.
| Code Block | ||
|---|---|---|
| ||
To be added
|
| Code Block |
|---|
"host": {
"alcatel_isis": { |
| Code Block |
"host": { "alcatel_isis": { "data": { "ip_address": "", "link": "", "name": "", "osi_address": "", "data": { "ots "ip_address": "", "typelink": "", }, "name": "", "neighboursosi_address": "", [ { "ots": "", "metrictype": "", }, "name": "", "neighbours": [ }, ]{ }, "name": "", "versionmetric": 1 } |
NMAP Producer
| Code Block |
|---|
"host", : { "." : { "os" : { "family" "name": "", "name" : ""}, }] }, "addrsname": :"", [ "version": 1 } |
The JSON files can be inserted with noclook_alcatel_consumer.py.
Edit the template.conf file with the correct path to the Alcatel ISIS NERDS files.
Change the path at the top of the script to be able to import norduni_client.py.
| Code Block |
|---|
[data]
juniper_conf =
nmap_services =
alcatel_isis = /path/to/alcatel/json
noclook =
|
Then run:
| Code Block |
|---|
python noclook_alcatel_consumer.py -C template.conf
|
nmap Producer/Consumer
Using the namp services producer you can scan a network or individual addresses. NORDUnet have a file
with networks that is used with the "-iL networks_file" option added to NERDS_NMAP_OPTIONS in the run.sh file.
| Code Block |
|---|
"host" : { "."127.0.0.1" ], "hostnames" : [ "host.example.org" ], "name" : "host.example.org", "services" : { "ipv4os" : { "127.0.0.1": {family" : "", "name" : "tcp": { } }, "1025addrs" : {[ "127.0.0.1" ], "hostnames" : [ "product": "Microsoft Windows RPC", "host.example.org" ], "name" : "host.example.org", "services" : { "confidenceipv4": "10", { "name"127.0.0.1": "msrpc", { "prototcp": "unknown"}, { "10291025": { "product": "Microsoft Windows RPC over HTTP", "confidence": "10", "versionname": "1.0msrpc", "nameproto": "ncacn_httpunknown"}, "1029": { "proto": "unknown"} "product": "Microsoft Windows RPC over HTTP", } "confidence": "10", } } }, "version" : "1 } .0", "name": "ncacn_http", "proto": "unknown"}, } } } }, "version" : 1 } |
The JSON files can be inserted with noclook_nmap_consumer.py.
Edit the template.conf file with the correct path to the nmap services JSON files.
Change the path at the top of the script to be able to import norduni_client.py.
| Code Block |
|---|
[data]
juniper_conf =
nmap_services = /path/to/nmap/json
alcatel_isis =
noclook =
|
Then run:
| Code Block |
|---|
python noclook_nmap_consumer.py -C template.conf
|
CSV Site Producer/CSV Site Producer/Consumer
The script produces JSON output in the NERDS format from the provided CSV file.
...
| Code Block | ||
|---|---|---|
| ||
{
"host": {
"csv_producer": {
"address": "",
"area": "",
"host": {
"city": "csv_producer":, {
"addresscomment": "",
"areacountry": "",
"cityfloor": "",
"commentlatitude": "",
"countrylongitude": "",
"floormeta_type": "",
"latitudename": "",
"longitudenode_type": "",
"metaowner_typeid": "",
"namepostcode": "",
"noderesponsible_typefor": "",
"owner_idroom": "",
"postcodesite_type": "",
"responsibletelenor_subscription_forid": "",
},
"roomname": "",
"version": 1
"site_type": "",
"telenor_subscription_id": ""
},
"name": "",
"version": 1
}
}
|
The consumer script should only be run once as it does not update the sites, only creates new.
The JSON file directory is then inserted in to the database using noclook_site_csv_consumer.py.
}
}
|
The consumer script should only be run once as it does not update the sites, only creates new.
The JSON file directory is then inserted in to the database using noclook_site_csv_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Then run:
| Code Block |
|---|
python noclook_site_csv_consumer.py -D /path/to/site_files/json
|
Daily database update
The producers are run with a cron job and the script noclook_consumer.py is used to run the three inserting/updating scripts (noclook_juniper_consumer.py, noclook_alcatel_consumer.py and noclook_nmap_consumer.py).
Change the path at the top of the script to be able to import norduni_client.py.
| Code Block |
|---|
[data]
juniper_conf = /path/to/juniper/json
nmap_services = /path/to/nmap/json
alcatel_isis = /path/to/alcate/json
noclook = #Used for loading backup.
|
Then run:
| Code Block |
|---|
python noclook_consumer.py -C template.conf -I
|
Purging the database
The database can be purged from all data using the P switch to the noclook_consumer.py scriptChange the path at the top of the script to be able to import norduni_client.py.
| Code Block |
|---|
python noclook_site_csv_consumer.py -D /path/to/site_files/jsonC template.conf -P |
Setting up old NOCLook version
...