The MikroTik Security Guide and Networking with MikroTik: MTCNA Study Guide by Tyler Hart are available in paperback and Kindle!
I've worked with MPLS circuits for a long time, but always with provider hand-offs. This is most people's first and only real exposure to MPLS. The service provider gives the customer Ethernet connections and says, "This connection goes to Site A, this other connection goes to Site B, you have X amount of bandwidth, do whatever you want." It's magic, and the customer doesn't need to have any idea how it works on the backend. It makes networking remote sites much easier for the customer, and it's a lucrative value add for providers. Obviously there are a lot of other ways to network remote sites together. There is EoIP, GRE, IPSEC, GRE with IPSEC, and if you have Scrooge McDuck amounts of money you can run your own fiber too. With that being said, MPLS is extremely popular compared to other solutions because it's transparent for the customer, the customer doesn't have to administer the tunnels, and it's all fairly turnkey. I wanted to learn what was going on behind the curtain, what it actually takes to provide these tunnels, and so I did.
Before we go any further you should be familiar with some terms, and I suggest reading up on basic MPLS. Two terms to be familiar with are LSR and LER, or Label Switch Router and Label Edge Router respectively. An LSR is a router running MPLS that only performs label switching in the core; it doesn't add or remove labels at network ingress or egress. An LER is a router running MPLS that pushes (adds) or pops (removes) an MPLS label when a packet enters or exits the MPLS network. LSRs reside in the core, LERs reside at the edge.
This article describes how to set up a basic MPLS network in the core, supported by OSPF, and run VPLS tunnels over that core between customer sites. This lets you give the customer an Ethernet handoff on both sides of the tunnel, and basically tell them to pretend it's a Cat5 cable strung between sites.
Here is the topology that we're working with, with two customer devices attached to a Seattle and a Santa Fe provider router:
Mikrotik MPLS Topology
The customer wants to be able to connect to devices in Santa Fe from Seattle as if they were local devices. They don't want to see hops, routes, etc - just make it work.
IP addresses are already configured on Ethernet interfaces, I won't bore you with that. OSPF networks are all being advertised in the backbone for brevity. First we'll set up the core routers in Seattle, Santa Fe, and Atlanta, creating loopbacks, then getting OSPF up, then MPLS with LDP. VPLS will run over the top of all that. OSPF will give us some resiliency if a link fails, like between the Seattle and Santa Fe LSRs.
On Seattle LSR:
/interface bridge add comment="MPLS Loopback" name="MPLS Loopback" /routing ospf instance set [ find default=yes ] router-id=72.156.28.150 /routing ospf network add area=backbone network=72.156.28.0/30 add area=backbone network=72.156.28.8/30 add area=backbone network=72.156.28.150/32 add area=backbone network=72.156.29.0/24 /mpls interface set [ find default=yes ] interface=ether1 add interface=ether2 add interface=ether3 /mpls ldp set enabled=yes lsr-id=72.156.28.150 transport-address=72.156.28.150 /mpls ldp interface add interface=ether1 add interface=ether2 add interface=ether3 /mpls ldp neighbor add transport=72.156.28.151 add transport=72.156.28.152 add transport=72.156.29.120
On Santa Fe LSR:
/interface bridge add comment="MPLS Loopback" name="MPLS Loopback" /routing ospf instance set [ find default=yes ] router-id=72.156.28.151 /routing ospf network add area=backbone network=72.156.28.8/30 add area=backbone network=72.156.28.4/30 add area=backbone network=72.156.30.0/24 add area=backbone network=72.156.28.151/32 /mpls interface set [ find default=yes ] interface=ether2 add interface=ether3 add interface=ether1 /mpls ldp set enabled=yes lsr-id=72.156.28.151 transport-address=72.156.28.151 /mpls ldp interface add interface=ether1 add interface=ether2 add interface=ether3 /mpls ldp neighbor add transport=72.156.28.150 add transport=72.156.28.152 add transport=72.156.30.120
On Atlanta LSR:
/interface bridge add comment="MPLS Loopback" name="MPLS Loopback" /routing ospf instance set [ find default=yes ] router-id=72.156.28.152 /routing ospf network add area=backbone network=72.156.28.0/30 add area=backbone network=72.156.28.4/30 add area=backbone network=72.156.28.152/32 /mpls interface set [ find default=yes ] interface=ether1 add interface=ether3 /mpls ldp set enabled=yes lsr-id=72.156.28.152 transport-address=72.156.28.152 /mpls ldp interface add interface=ether1 add interface=ether3 /mpls ldp neighbor add transport=72.156.28.150 add transport=72.156.28.151
At this point we have OSPF running in the core, and MPLS running as well on the LSR routers. From this point on we'll focus on the LERs that actually connect to the customers. We'll add an additional bridge for VPLS traffic, configure OSPF and MPLS with LDP on each of the LERs, then we'll move on to building the VPLS tunnels.
On Seattle LER:
/interface bridge add comment="MPLS Loopback" name="MPLS Loopback" add comment="Customer #4306 Site 1" name="VPLS Customer 4306-1 Bridge" /routing ospf instance set [ find default=yes ] router-id=72.156.29.120 /routing ospf network add area=backbone network=72.156.29.0/24 add area=backbone network=72.156.29.120/32 /mpls interface set [ find default=yes ] interface=ether3 /mpls ldp set enabled=yes lsr-id=72.156.29.120 transport-address=72.156.29.120 /mpls ldp interface add interface=ether3 /mpls ldp neighbor add transport=72.156.28.150
On Santa Fe LER:
/interface bridge add comment="MPLS Loopback" name="MPLS Loopback" add comment="Customer #4306 Site 2" name="VPLS Customer 4306-2 Bridge" /routing ospf instance set [ find default=yes ] router-id=72.156.30.120 /routing ospf network add area=backbone network=72.156.30.0/24 add area=backbone network=72.156.30.120/32 /mpls interface set [ find default=yes ] interface=ether1 /mpls ldp set enabled=yes lsr-id=72.156.30.120 transport-address=72.156.30.120 /mpls ldp interface add interface=ether1 /mpls ldp neighbor add transport=72.156.28.151
At this point OSPF should be fully converged, and in the MPLS Bindings tab we should see some MPLS labels associated with destination addresses:
Mikrotik MPLS Local Bindings
We should also see some entries in the Forwarding Table too:
Mikrotik MPLS Forwarding Table
This is MPLS at work, associating routes with labels for quick lookup, which is what gives MPLS its trademark performance boost over regular end-to-end IP routing. We're ready now to add the VPLS tunnels and start moving some traffic transparently between sites. The extra bridge interfaces that we added on the two LERs will be used to bridge the VPLS virtual interfaces with physical Ethernet interfaces that we hand off to the customer.
On Seattle LER:
/interface vpls add comment="Customer 4306-2 VPLS" disabled=no l2mtu=1500 name="Customer 4306-2 VPLS" remote-peer=72.156.30.120 vpls-id=90:0 /interface bridge port add bridge="VPLS Customer 4306-1 Bridge" interface=ether1 add bridge="VPLS Customer 4306-1 Bridge" interface="Customer 4306-2 VPLS"
On the Santa Fe LER:
/interface vpls add comment="Customer 4306-1 VPLS" disabled=no l2mtu=1500 name="Customer 4306-1 VPLS" remote-peer=72.156.29.120 vpls-id=90:0
/interface bridge port add bridge="VPLS Customer 4306-2 Bridge" interface=ether3 add bridge="VPLS Customer 4306-2 Bridge" interface="Customer 4306-1 VPLS"
At this point we've created a Layer 2 connection between whatever is plugged into ether1 in Seattle and ether3 in Santa Fe. The customer could throw routers on those connections, or switches, or plug servers directly in. For demonstration purposes I put a virtual Ubuntu server on each of those physical interfaces, given them the IP addresses 10.2.2.1 and 10.2.2.2, and run iperf both directions to test bandwidth as shown below:
Mikrotik MPLS IPerf Testing
Bandwidth testing shows a consistent, fast connection. This whole network and servers are virtualized, so while it isn't running at gigabit wire speed it still performs well. One of the other requirements for this solution was that there be no hops between the two locations - this should all be transparent to the customer. Traceroute from 10.2.2.2 to 10.2.2.1 shows the following:
Mikrotik MPLS Traceroute
Exactly what we want to see, which is nothing. None of the provider routers in between, none of the hops. Next time we'll cover MPLS with QoS and all the other fancy features!
