110309 NORDUnet Operational meeting

Table of content

Logistics

The date for the next NORDUnet operational meeting will be the 9th-10th of March, starting at 12:30 CET on Wednesday the 9th and end at 12:30 CET on Thursday the 10th.
The meeting will be hosted at the NORDUnet office in Copenhagen.

Hotel recommendations

Choice hotel
Right across the street from the NORDUnet office and very close to the airport.

The Square
If you prefer to stay in the center of Copenhagen, takes about 30-40 min with the metro to the NORDUnet office.

Registered Participants

Stefan Liström -SL (NORDUnet)
Fresia Pérez - FP (NORDUnet/SUNET)
Jan Ferre - JF (Forskningsnettet)
Juha Oinonen - JO (FUNET)
Vegard Vesterheim - VV (UNINETT)
David Simonsen - DS (WAYF)
Johan Lundberg -JL (participating via VC presenting NORDUnet Network inventory)
Fredrik Pettai - FP (participating via VC presenting NORDUnet IPv6 and DNSSec)
Börje Josefsson - BJ (Participating via VC presenting CBF update)

Draft agenda

Minutes of meeting

Wednesday 9th of March

Introductions and setting the agenda

The agenda was accepted

NREN Operational updates

• UNINETT - VV
  Because of bad weather conditions, two links to northern Norway were down simultaneously.
  But due to the third (Cross border fiber) link recently provisioned through SUNET, the traffic was saved for the northern part of Norway.
  UNINETT is involved in running fibreproject for connecting Longyearbyen to Nyålesund with (multiple) sea-cables. The application for permit to
  build have been delivered to Sysselmannen on Svalbard, possible delivery in 2012.
  Telephones in UNINETT are now sip based. One of the reasons for this is cost savings with SIP equipment compared to e.g. the old PBX.
  The switchboard application is developed inhouse. Hopefully this will also allow UNINETT to integrate the phones with other tools used by the NOC.
  Several customers are now connected with 10G
• FUNET - JO
  DWDM network is now fully extended and reach the northern parts of Finland.
  Merging of universities has resulted in more requests for point to point connections, mostly 1G lambdas.
  The deployment of the MX routers are more or less finalized.
  A large project currently worked on is the Kayaani datacenter. It will host the next supercomputer for CSC. It is scheduled to be operational in about a year. Cooling and power will be provided by water power.
• FSKnet - JF
  FSKnet are integrating layer two connectivity between universities.
  FSKnet are spending alot of time on provisioning lightpaths
  FSKnet have change chassis on core router to get a new service contract.
• SUNET - FP
  The SUNET sites redundancy project is now finalized, the SUNET core is now divided between two sites in Stockholm.
  The next large project will be to switch CPEs for the SUNET customers. The plan is to replace the Ciena 4350 equipment with Juniper MX80 equipment.
• NORDUnet - FP
  There have been major problems with one of the links between Greenland and Canada. The new estimated repair time is within three weeks of this meeting.
• RHnet
  RHnet is in the process of upgrading the routers in the POPs to 10G and IPv6 capabilities. We have now finished about a third of the POPs and are able to offer 10G and IPv6 native connectivity to the largest members (very little uptake so far, as most of them need to upgrade their equipment also).
  RHnet are currently upgrading L3/L2 equipment (C4809) to 10G capable C3560-12Ds).
  The operation support for RHnet is now contracted to the University of Iceland, but still minimally engaged yet. RHnet has discontinued
  the Usenet service, as well as central web-proxy services. These are now run by the members interested in these services.
  RHnet have had two periods of severe problems with our US connectivity (basically all traffic to/from RHnet to US destinations was degraded for many hours on two occasions). These were caused by faulty equipment/fiber on the NORDUnet circuit to St.Johns via the Greenland-Connect Sea-cable.
  These are NORDUnet circuts, so we were not directly involved in debugging this and basically this path is not in use now (the
  cable is down due to cable-break at sea between Greenland and Canada).

IPv6 and DNSSec status and plans

• NORDUnet and SUNET
  • IPv6
    IPv6 has been enabled in the core network for a long time.
    NORDUnet are adding IPv6 for all new hosts installed. Not necessarily (unfortunately) supported by the service installed on the host at this moment though.
  • DNSSec
    DNSSec nordu.net is not signed but resolvers do DNSec validation
    NORDUnet is planning to use Unbound instead of Bind for the resolver implementation, due to some inconveniences with DNSSec when using Bind.
• SUNET
  • IPv6
    Migrating virtual machines from zen to vm-ware and enable IPv6 on host at the same time.
    IPv6 implementation for SUNET customers:
    http://stats.sunet.se/ipv6/
  • DNSSec
    When it comes to DNSSec SUNET has the same status as NORDUnet
• FSKnet
  • IPv6
    Core routers have supported IPv6 for quite some time.
    DNS deliver IPv6 adresses but most of the universities do not use IPv6, one of the reasons are firewall problems at the universities.
  • DNSSec
    There is one nameserver using DNSSec in production but the main nameserver is running tinydns, which does work flawlessly with DNSsec.
• FUNET
  • IPv6
    Backbone has supported IPv6 for many years, challenge is to get connected organisations to adopt IPv6.
    FUNET looking into making a project plan or something generic that universities can use to increase adoption.
    FUNET are also pushing the people that have not started.
    Tracking which organisations that have implemented IPv6 is a good tool:
    http://www.csc.fi/funet/status/tools/funet_ipv6_status
    FUNET are working to ensure that all services support IPv6
    FUNET consider IPv6 when investigating new services
    There are still some issues with netflow ipv6 monitoring that have not been solved.
  • DNSSec
    Main interest last year for DNSSec was due to the fact that FUNET are providing the .fi rootname server.
    FUNET were asked to assist in implementing DNSSec on the .fi rootserver, this was done about six month ago.
    Most FUNET organisations use their own resolvers, FUNET are pushing for their own resolvers to be DNSSec compatible.
    When FUNET has been signed they will also start pushing connected organisations.
    HSM (hardware security module) have been tested and the tests are positive so far.
• UNINETT
  • IPv6
    The UNINETT core has been IPV6 enables for four years.
    New hosts are IPv6 enabled when installed.
    The SIP software Asterisk is a problem as it does not support IPv6 yet.
    Connected organisations have not been pushed very hard to deploy IPv6.
    List of connected organisations and their IPv6 deployment:https://ow.feide.no/gigacampus:ipv6status
  • DNSSec
    Not much work has been done on DNSSec in UNINETT yet, UNINETT have upgraded DNS servers in preparations of DNSSec. Some issues with new versions of bind on netbsd, but they have been resolved.
    Plans to deply DNSSec this year for .no rootserver.
• RHnet
  • IPv6
    As per above, IPv6 deployment is proceeding on schedule, pending availability of IPv6 capable equipment. Only the University of Iceland has received an IPv6 address space assignment from RHnet and are implementing IPv6 pending equipment upgrades also.
  • DNSSec
    DNSSec has not been discussed and there are no immediate plans to deploy.
• General discussion
  FP asked if the NRENs are part of the google "IPv6 program".
  NORDUnet and UNINETT are part of the program, FSKnet and FUNET will consider it.https://services.google.com/fb/forms/requestipv6/
  If all NRENs connected to NORDUnet are part of the program it will be easier for NORDUnet to push google to enable IPv6 on their services towards NORDUnet.
  FP asked about NREN support for IPv6 over Wireless services
  FUNET is not offering IPv6 connectivity over wireless
  UNINETT have done some experimentation with wireless IPv6
  FSKnet do not offer wireless as a service, but when they help setup wireless e.g. at conferences they do enable IPv6.

Cross-border fiber (CBF) activity report -BJ

BJ presented an update of the SUNET and UNINETT CBF project
SL asked how fiber breaks are handled on the fiber that cross the border.
The UNINETT and SUNET supplier have a good relationship, the UNINETT supplier even have access to the Swedish site (in Abisko).
When there is a fiber break UNINETT takes responsibility for the measurement and depending on the result of the measurement respective supplier will fix the problem. I.e. the UNINETT supplier will measure the fiber from the site in Norway and if the fiber break seems to be on the Norwegian side of the border they will handle it. But if the break seems to be on the Swedish side of the border they will report this to SUNET and the SUNET supplier will handle the fiber break.

There was also a discussion about the SUNET-FUNET CBF. Even though there is no critical need to have this in place it would still be beneficial to start the discussions as early in time as possible.
The meeting participants agreed that NORDUnet should facilitate such a meeting/discussion.

New AP on Stefan, check if a meeting with SUNET-FUNET is already planned and if not investigate the possibility of someone within NORDUnet to facilitate such a meeting.

NREN NOC tools and procedure development

• NORDUnet
  • Network Inventory development progress report and demo - JL
    JL presented the prof of concept version of the Network inventory (NI).
    The plan is to have a production version of the network inventory in Q3. The participants at the meeting were interested in getting more information when the production version would be operational. It was decided that Stefan will arrange a Demo when the network inventory is in the production stage.
    New AP on Stefan, arrange a Demo of the production version of the Network Inventory
    The current version of the NI can be accessed at the following address:
    nidev-consumer.nordu.net
    Contact Stefan for authentication information.
  • Single sign on and federated login migration - SL
    NORDUnet have now started connecting their services and applications to a Single Sign On setup and at the same time federating authentication towards their external services. Organisations participating in Kalmar2 or individuals with a Protect Network account can now use their own IdP account to get access to e.g. portal.nordu.net and project.nordu.net.
    The SSO backend is an integrated setup of an LDAP database, Cfengine, Kerberos, Active directory and the Atlassian product Crowd (which is used for web application). To also achieve federated access using Crowd NORDUnet has written a Shibboleth plugin module for Crowd.
• UNINETT
  Due to time restraint this topic was moved to Thursday instead.
  • Monitoring, issue tracking and documentation integration - VV
    UNINETT are planning to document alarms (and test for common problems) with links between monitoring to documentation
    A common problem is that a NOC get alarms that they do not necessarily need to act on right away. This creates a situation where it is hard to distinguish between which alarms to handle right away and which can wait for some other time. Which can lead to a situation where critical problems are not fixed as fast as they should.
    SL pointed out that it is important to only send the noc alarms that needs to be handled directly, otherwise the above situation will certainly appear very fast.
    In UNINETT the NOC manager sign the documentation directly in the documentation system when it is handed over to make sure which services are accepted by the NOC and monitored.
    The UNINETT documentation system keep information about services and dependencies for services, i.e. if a host goes down it show which services and customers are affected. The system consist of a database in the backend with a web frontend.
    What UNINETT are now investigating is if it is possible to automatically make tickets from their monitoring applications. There are currently some debate in UNINETT how to handle trouble tickets for the NOC and if they should implement a trouble ticket system.
    JF pointed out that one reason to use a trouble ticket sytem is to show that things are done.
    FP commented that the trouble ticket system is crucial for creating reports used both internally and towards external providers.
  • Routines for documentation - VV
    UNINETT are investigating how to improve their routines for documentation.
    Documentation should be checked atleast once a year to make sure it is up to date. This can be ensured by having one person responsible for each service and send out automatic reminders once a year to that person to check the documentation.
    One of the challenges is to get people to transfer knowledge e.g. when they quit or move to a new position within the company.
  • Inventory data - VV
    UNINETT would like to be able to keep more information about equipment and services and are investigating how to achieve this.
  • Ticket system with focus on ticket handover - VV
    UNINETT are investigating how to implement a ticket system were interested in how others handle ticket handovers
  • SL commented that one thing that works well for the SUNET NOC is keeping a special field for when a ticket is due. I.e. if a customer or supplier have agreed to give feedback during a certain day or a ticket has passed the general "waiting time" if nothing has happened in the case. That way the NOC can filter the tickets and only have to see/work on the tickets that are important right now. This also makes handover to the next shift easier as you e.g. easily can see which planned maintenance tickets will have to be handled during that day.
• Other NREN tool development reports - all
  UNINETT have also started investigating how to setup a customer portal. As the meeting was running out of time it was decided to continue discussing customer portals at a later operational forum meeting.

Thursday 10th of March

TF-NOC update - SL

Countries all around the world are now participating (signed up on the mailing list) in the TF-NOC activity.
During the NOC taxonomy discussion it was agreed that TF-NOC would investigate if it was possible to use the TMForum eTOM standard to investigate which functions are important for a NOC and how they are performed in different NOCs.
At the last meeting several NOCs presented their NOC. These presentations have been collected at the TF-NOC webpage.
The next major topic for the TF-NOC activity is to create a survey regarding Network operation centres. The next face-to-face discussion about that survey will be held at TNC 2011.
The next official TF-NOC meeting will be in Zurich the 28-29 June.

NOC and CERT interaction procedures - all

• SUNET
  SUNET has a separate CERT team which handle computer related incidents. The SUNET NOC has a set of procedures to follow depending on what kind of incident is reported to them and how it is reported. During a computer related incident the NOC always contact the CERT team.
• UNINETT
  The CERT in UNINETT is a separate group (about 5 persons) but they are part of the NOC rotation.
• FUNET
  FUNET has a separate CERT team with about four persons.
  The CERT team is not part of the NOC.
  Their main responsibility is serious denial of service attacks
  Cert communicate with security managers at campuses and NOC communicates with network managers
  FUNET are currently discussing how to create a clear responsibility and facilitate better communication between the NOC and the CERT team.
  New AP on Stefan, send the SUNET procedures to the operational forum email list as an example how such procedures could look.
• FSKnet
  In FSKnet the CERT team is also a separate group which is not part of the NOC.
  The CERT ask the NOC to create null routes when there are major issues.
  The CERT is typically scanning for problems and using honeypots to register problems.

Kalmar presentation - DS

David from WAYF presented how Kalmar2 is setup and how the different federations within Kalmar communicate between an IdP in each federation and service providers connecting services to the federation.

IdP/federation support discussion

The federations within the Nordic countries are separate groups from the NOC in each country.
As more and more services are enabled with federated authentication it will most likely become an issue to identify if a problem has to do with a federation/IdP or the actual service a user is trying to use. In other words it might not always be clear who has the responsibility to fix the problem, i.e. the NOC or the group responsible for the IdP.
To make it more clear if a problem is caused by an IdP or a service connected to an IdP there are certain test that can be done.
On the Kalmar webpage there is a link to a weathermap feature which show the state of federations within Kalmar.

AOB and next meeting

There were two suggested options for next meetings:

1. either we have the next meeting as a face-to-face at the NORDUnet conference and then have a video conference meeting after summer
2. or we have a video conference meeting sometime around the time of the NORDUnet conference and then a face-to-face meeting after summer
   New AP on Stefan, check possibility of having a face-to-face meeting at RHnet after summer.

Potential topics for later operational forum meetings:

• Customer portals
• Demonstrate integration in UNINETT documentation system