100921 NORDUnet Operational meeting

Table of content

Logistics

The date for the next NORDUnet operational meeting will be the 21st of September, between 10:00 and 16:00 CET.
The meeting will be done via video conference.

Contact details for the video conference system is:
IP: 193.11.3.156

Registered Participants

Stefan Liström (NORDUnet)
Fresia Pérez (NORDUnet/SUNET)
Jan Ferre (Forskningsnettet)
Pekka Savola (FUNET)
Juha Oinonen (FUNET)
Håvard Kusslid (UNINETT)

Draft agenda

Meeting minutes

Participants

Stefan Liström (NORDUnet)
Fresia Pérez (NORDUnet/SUNET)
Juha Oinonen (FUNET)
Håvard Kusslid (UNINETT)
Vegard Vesterheim (UNINETT)
Marius Ólafsson (RHnet)
Johan Lundberg (NORDUnet)

Introductions and setting the agenda

There was a "round the table" introduction to welcome the new NORDUnet NOC manager Fresia.

NREN Operational updates

UNINETT

Håvard talked about UNINETT implementing new procedures at the NOC for handing over services and taking in new services that the NOC will be responsible for.
UNINETT had a power outage earlier this year which got them to revise their procedures for the NOC during power outages. The NOC systems are now protected with diesel backup and the documentation is backed up each night to an "offline" repository.
The demand for Lambda services are increasing, atleast three new projects delivering Lambdas has been started since last meeting.
The sea cable project to Svalbard has been accepted and is now in the planning phase.
The CBF project between UNINETT and SUNET has also started. Preliminary it will be Olaf Schjelderup that will be the responsible for the planning in UNINETT and Börje Josefsson will be responsible for the planning in SUNET.

FUNET

Juha said that the reorganisation in FUNET is now more or less done, there are now two instead of three main organisational units.
FUNET NOC is trying to get a more clear distinction between their services, implementing service classes and monitoring according to those service classes.
The FUNET fiber optical network is now ready up to Sodankylä. The topology is more or less complete now. There are still some Lambdas that needs to be provisioned.
FUNET is investigating moving over to using some other distribution than Redhat for their service platform.

RHnet

Marius informed that the RHnet NOC is now handled by University of Iceland since two month ago.
The state of Iceland is investigating the possibility to reduce (combining) six of the universities on Iceland down to two universities.
RHnet now has their first 10G link in production in the Reykjavik fiber ring.

There was a discussion about merging universities and how to connect and carry university internal traffic in the NREN backbone.
In FUNET universities that merge usually continue operating in different physical locations and FUNET carry their internal traffic between the different locations using lambda connections.
UNINETT also have scattered universities and UNINETT carry those universities internal traffic between the different departments, mostly using their routed IP network.
Stefan mentioned that Stockholm university had recently merged with another University. Marius was interested in how that had been done and Fresia said that she can send Marius contact information for people that where part of that migration.

ACTION 1 on Fresia: to send Marius contact info at Stockholm university.

SUNET

Fresia informed about the redundancy project of the SUNET main site in Stockholm, a second site in Stockholm is being installed so that the main backbone site will have redundancy.
As SUNET is now offering more major services than a network service they have had to develop new routines for handling problems. For the network services there were already functioning communication ways from the end-user to the NOC. But for these new services the NOC have had to develop new routines. A system administrator role has been created to act as a connection between the end-users and the NOC. The model can be viewed in more detail on the SUNET website[1] (in Swedish). One part of this work has also been to start using external community tools[2] to get a more transparent interaction between the people operating services and the end-users.
Early spring this year SUNET started offering Adobe connect as a video conferencing and e-learning service and SUNET is now in the middle of setting up a central spam filter service for the universities.
There is also a project hosting a Adobe connect setup to UNINETT.
SUNET noticed a bug with the Ciena 4350 that made them crash after a certain uptime. This has caused alot of problems. There is a workaround but SUNET is trying to get a more permanent solution.

NORDUnet

The NOC have had to implement changes in their procedures due to the fact that they now also support other major services for SUNET than the network services.
NORDUnet is working on a project to federate login to several of their services. The first services to get federated login will be the NORDUnet portal and the NORDUnet issue tracking system. To start with the Kalmar federation will be supported.
A handover procedures for new services have been developed for the NOC.

There was a discussion about handover procedures both for completely new services and new service instances. As both UNINETT and NORDUnet are working on this at the moment we agreed to share what has been done on these topics so far and add it as a topic for further discussion on the agenda for the next meeting.

ACTION 2 on All: to send handover procedures (if there are such procedures) to the operational list for discussion.

Marius wondered about the current status of the NORDUnet cable going from London to Reykjavik as there have been several outages on that connection. Fresia promised to check the status and get back to Marius with more information.

ACTION 3 on Fresia: to check status on the Iceland cable (service) and get back to Marius with that information.

NORDUnet Network Inventory presentation

Johan made a presentation of the NORDUnet efforts to create a network inventory (NI) using a graph database[3] (non-relational database). The reason NORUDnet have chosen a graph database is because of the fact that NORDUnet found it was very complicated to create a common datastructure representing a hybrid network with active/passive physical hardware and logical connections using a standard relational database. Johan explained the datastructure in the graph database that consists or nodes and relationships. Each node and relationship can have extra attributes. The network inventory tool will use the Django framework[4] to visualise the information. The network inventory will be published on the NORDUnet GIT server [5].

All other NRENs where interesting in following the progress of the NI development. Johan will setup a test instance of the current prototype of the network inventory that has been developed.

ACTION 4 on Stefan: to include links in the MoM to the websites Johan referred to in his presentation.
ACTION 5 on Stefan: to publish pictures of the datamodel on the operations forum
ACTION 6 on Johan: to create a test instance of the current NI prototype so that other NRENs can test it.

Discussion about how to proceed with possible Network Inventory activity

There was a discussion about what methods each NREN use to store network related information.
FUNET keep alot of their information regarding the network and services in the equipment. They then utilize scripts to retrieve that information when needed. Documentation about passive hardware and fibers are kept separately.

UNINETT tried Telemator but they had problems depicting the their logical topology in that tool.
UNINETT tries to keep both active and passive hardware information documented in a separate tool. I.e. not only keeping certain information in the equipment.

SUNET keep their network information for both active and passive hardware in Telemator. Once the physical and logical topology is built in Telemator it is ok to use. It is however very timeconsuming to make changes or add new parts to the topology.

NORDUnet rely on both information in their active equipment and offline documentation for services and hardware.

The meeting decided to keep follow the NORDUnet effort creating a Network inventory.

TF-NOC update

Stefan gave a short update about the current work in TF-NOC. The goal of the first official TF-NOC meeting was to discuss how to organise the task force and which topics to focus on in the close future. The next step for the task force will be to define a template for NOC presentations. The idea behind this is to try to define what kind of information that is interesting to know about a NOC. Party to be able to compare relevant information when discussing the NOC topology. More information can be found on the TF-NOC homepage[6].

Federated service support discussion

There was a discussion about the NREN service list[7] and the federation options.
The meeting participants would appreciate a clarification what the federation options mean. Does federated login mean that only that users from that NREN can login to the service or are other NRENs included in the federation login. Does it only mean that users are able to login with their accounts or does it also indicate that users from other NRENs can use those services?

ACTION 7 on Stefan: to discuss the federation options with Jörgen to try to clarify their meaning.

There was also a discussion about the upcoming service hosting of the Adobe connect service that UNINETT will use. The conclusion was that there was not enough information available about the service to discuss specific operational issues about how the cooperation between UNINETT and NORDUnet would work.

TT-integration discussion

The initial information[8] about the use of ticket systems in the different organisations are still valid.

In average there are about 5-20 updated ticket for each organisation each day.
There are between 10-100 tickets open on average in our organisations.

In FUNET every NORDUnet ticket generates a new ticket in FUNET RT. Which then have to be handled, in most cases just closed as it is an update to a already existing ticket. Ticket integration efforts between our systems could help minimize this manual work in FUNET.

In UNINETT the NORDUnet tickets are automatically inserted and also automatically deleted depending on the date information in the trouble tickets sent out.

In RHnet the NORDUnet tickets does not generate much extra work efforts.

Future vision for TT system integration
Ideally nobody would need to do any manual work when tickets are generated in one organisation that affects other organisations. It would also be interesting to integrate monitoring systems with the trouble ticket systems to in certain cases be able to create, update and close tickets automatically.

Conclusion
The conclusion of the discussion was that it is not a major problem right now, that we do not have integrated trouble tickets systems. The situation is still manageable. It might however become more important in the future with regards to hybrid network and when we start federating services between the Nordic countries.

AOB and meeting wrap up

Vegard asked how the other countries had protected their NOC against power outages and all other countries where using diesel generators to provide backup power for their offices and the NOC environment.

Juha mentioned that FUNET would like to get information from NORDUnet about the agenda before APM meetings and an update of outcomes from that meeting afterwards. FUNET were also interested in getting more information about the GLIF efforts.

ACTION 8 on Stefan: to discuss with Jörgen how to increase the information dissemination to the Nordic NRENs regarding the GEANT APM meetings and the GLIF efforts.

Next meeting

Next meeting will preliminary be the 14-15 of February.

[1] http://www.sunet.se/Om-sunet/Policyfragor/Forvaltningsmodell.html
[2] http://community.sunet.se/sunet
[3] http://neo4j.org/
[4] http://www.djangoproject.com/
[5] http://git.nordu.net/?p=norduni.git
[6] http://www.terena.org/activities/tf-noc/
[7] https://portal.nordu.net/display/NORDUwiki/NREN+Services+map
[8] https://portal.nordu.net/display/opsforum/090603+NORDUnet+Operational+meeting