Remote port-mirroring in Junos

20 03 2015

Information on remote port mirroring on Junos routers doesn’t seem to be very easy to come by for some reason – there is quite a lot of information about doing this on EX switches (a bit like RSPAN in Cisco’s IOS), which wasn’t what I needed.  Various other sources of information (such as Cluepon) say this can be done using a GRE tunnel, but that the capturing device needs to be a server that terminates the GRE tunnel – which all seemed a bit complicated.

I needed to remotely mirror a port on an MX to a second MX where a windows-based Wireshark was connected, so getting GRE working to a Windows host sounded like a non starter.

So I had to work it out myself – and hopefully this write-up will prove useful to someone else in the future.

An additional requirement for the customer was that the captured interface should be in a VRF, so the test-bed I set up below has production traffic flowing through a VRF, and the mirrored traffic in a GRE tunnel which is running in the global routing table.

Here’s the write-up I did for the customer:


Remote port-mirroring using GRE tunnel

 Objective:

Customer has an option-A interconnect with 10 VLANs on it.   Each unit is in a separate routing-instance.   Customer wants to port-mirror packets on some (but not all) to a remote destination.

Solution:

Customer ports were put into a routing instance to simulate separation from the global routing table (inet.0).

A GRE tunnel was established between the two MX loopbacks within the global routing table.

A firewall filter was created on MX1 putting all packets received on ge-1/0/0 into the tunnel.

A firewall filter was created on MX2 putting all packets received on the GRE tunnel out of ge-1/1/9 where the capturing device was located.

Bi-directional mirroring of ge-1/0/0 on MX1 was required, so the same filter was used both inbound and outbound.

Test-bed Network Diagram:

Remote Port Mirroring

Remote Port Mirroring

Note: One might consider that simple port-mirroring of ge-1/0/0 of MX2 might be all that is required here, but the objective was to perform remote port-mirroring using a GRE tunnel. Due to limited hardware availability at the time, this test-bed was created on just two routers. For a more clear illustration perhaps a third router should have been employed, whose sole purpose was to terminate the GRE tunnel(s) and mirror traffic on those tunnels to a capturing device.

Test-bed Configuration

Creating the GRE Tunnel

First, enable tunnelling:

imtech@MX80-1> show configuration chassis
fpc 0 {
   pic 0 {
       tunnel-services;
   }
}

This permits the creation of the GRE interface. Next create the GRE interface:

imtech@MX80-1> show configuration interfaces gr-0/0/0
unit 0 {
   tunnel {
       source 10.255.255.1;       <-- Loopback of MX1
       destination 10.255.255.2;  <-- Loopback of MX2
   }
   family inet {
       address 1.1.1.1/30;        <-- IP address within tunnel at MX1 end
   }
}

This won’t connect to anything yet, because the link to the second MX and routing to the tunnel endpoint (the lo0 interface on MX2) have not been set up yet. We do that next, using basic static routing for simplicity:

imtech@MX80-1> show configuration interfaces xe-0/0/1
mtu 9150;
description "GLOBAL ROUTING TABLE LINK TO MX2";
unit 0 {
   family inet {
       address 192.168.1.15/31;
   }
}


imtech@MX80-1> show configuration interfaces lo0
unit 0 {
   family inet {
       address 10.255.255.1/32;
   }
}

imtech@MX80-1> show configuration routing-options static
route 10.255.255.2/32 next-hop 192.168.1.14;

imtech@MX80-1>

Once the appropriate configuration is applied on MX2, the GRE tunnel comes up.

imtech@MX80-1> show interfaces gr-0/0/0
Physical interface: gr-0/0/0, Enabled, Physical link is Up
Interface index: 163, SNMP ifIndex: 543
Type: GRE, Link-level type: GRE, MTU: Unlimited, Speed: 100000mbps
Device flags   : Present Running
Interface flags: Point-To-Point SNMP-Traps
Input rate     : 0 bps (0 pps)
Output rate   : 37976 bps (20 pps)

Logical interface gr-0/0/0.0 (Index 336) (SNMP ifIndex 553)
   Flags: Up Point-To-Point SNMP-Traps 0x0 IP-Header 10.255.255.2:10.255.255.1:47:df:64:0000000000000000
   Encapsulation: GRE-NULL
   Copy-tos-to-outer-ip-header: Off
   Gre keepalives configured: Off, Gre keepalives adjacency state: down
   Input packets : 2
   Output packets: 77920705391
   Protocol inet, MTU: 9112
     Flags: Sendbcast-pkt-to-re
     Addresses, Flags: Is-Preferred Is-Primary
       Destination: 1.1.1.0/30, Local: 1.1.1.1, Broadcast: 1.1.1.3

imtech@MX80-1>

Creating the VRF for Customer Traffic

Interface ge-1/0/0 is configured with an IP address and plugged into the source of the traffic:

imtech@MX80-1> show configuration interfaces ge-1/0/0
description "LINK TO TESTER";
unit 0 {
   family inet {
       address 10.1.1.1/30;
   }
}

The link between the MXes is also configured in the VRF:

imtech@MX80-1> show configuration interfaces xe-0/0/0
description "LINK IN VRF TEST";
vlan-tagging;
mtu 9150;
unit 20 {
   vlan-id 20;
   family inet {
       address 10.0.0.1/30;
   }
}

OSPF is configured on the interfaces in the VRF to achieve IP reachability between all subnets involved:

imtech@MX80-1> show configuration routing-instances TEST
instance-type virtual-router;
interface xe-0/0/0.20;
interface ge-1/0/0.0;
protocols {
   ospf {
        area 0.0.0.0 {
           interface ge-1/0/0.0;
           interface xe-0/0/0.20;
       }
   }
}

Not shown here is the MX2 end, but the configuration is very similar.   On that side, the link to the tester is 10.2.2.0/30.   This subnet is learned successfully by MX1:

imtech@MX80-1> show route table TEST.inet.0
TEST.inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.0.0.0/30       *[Direct/0] 16:20:52
                   > via xe-0/0/0.20
10.0.0.1/32       *[Local/0] 16:32:59
                     Local via xe-0/0/0.20
10.1.1.0/30       *[Direct/0] 16:52:35
                   > via ge-1/0/0.0
10.1.1.1/32       *[Local/0] 16:52:35
                     Local via ge-1/0/0.0
10.2.2.0/30       *[OSPF/10] 16:03:33, metric 2
                   > to 10.0.0.2 via xe-0/0/0.20
224.0.0.5/32       *[OSPF/10] 16:33:03, metric 1
                     MultiRecv

Port-Mirroring Configuration

Remote port-mirroring has to be done in two places:

  1. Mirror packets from an interface on MX1 into a GRE tunnel
  2. Mirror packets in the tunnel arriving at MX2 to a port where a packet capture device is

Mirroring in Junos has three parts to it:

  1. Create a firewall filter to mirror packets
  2. Apply the filter to the interface you are interested in
  3. Set the mirroring destination in the ‘forwarding-options’ part of the config – there can be only one destination.

The configuration for this remote mirroring is in the following sections:

Configuring Port-Mirroring on MX1

First, create a firewall filter that has an action of ‘port-mirror’ and also ‘accept’.     This means that packets are mirrored, and also sent on to their final destination.   If the accept were omitted, mirroring packets would happen, but the packets would not be able to travel to their intended destination:

imtech@MX80-1> show configuration firewall
family inet {
   filter MIRROR {
       term t1 {
           then {
               port-mirror;
               accept;
           }
       }
   }
}

Next, apply the filter inbound, outbound or both to the interface where the interesting traffic is:

imtech@MX80-1> show configuration interfaces ge-1/0/0
description "LINK TO TESTER";
unit 0 {
   family inet {
       filter {
           input MIRROR;
           output MIRROR;
       }
       address 10.1.1.1/30;
   }
}

Finally, tell the router where all mirrored packets should go. In the case of MX 1, this is the GRE tunnel in the global routing table. We are taking packets seen on ge-1/0/0 within VRF ‘TEST’ and mirroring them into the GRE tunnel to MX2:

imtech@MX80-1> show configuration forwarding-options
port-mirroring {
   input {
       rate 1;
       run-length 1;               <-- Mirror every packet
   }
   family inet {
       output {
           interface gr-0/0/0.0;   <-- Destination of mirrored traffic
       }
   }
}

imtech@MX80-1>

Configuring Port-Mirroring on MX2

At the MX2 end, we have a sniffer attached on ge-1/1/9 ready to receive traffic.

The configuration is similar to that on MX:

A firewall filter is created that mirrors and accepts packets:

imtech@MX80-2> show configuration firewall
family inet {
   filter MIRROR {
       term t1 {
           then {
               port-mirror;
               accept;
           }
       }
   }
}

The filter is applied inbound on the GRE tunnel – remember that although I have logically drawn the tunnel ‘exit’ at MX2 in the diagram, the packets are actually being received into the tunnel interface from MX2’s perspective:

imtech@MX80-2> show configuration interfaces gr-0/0/0
unit 0 {
   tunnel {
       source 10.255.255.2;
       destination 10.255.255.1;
   }
   family inet {
       filter {
           input MIRROR;
       }
       address 1.1.1.2/30;
   }
}

Configure the destination for the mirrored packets as before – in this case it is ge-1/1/9. An IP address was implemented on this interface just so that the router could get an ARP entry for the tester:

 imtech@MX80-2> show configuration forwarding-options

port-mirroring {
   input {
       rate 1;
       run-length 1;
   }
   family inet {
       output {
           interface ge-1/1/9.0 {
               next-hop 2.1.1.2;
           }
       }
   }
}

Testing

Packet streams were set up to go between the tester ports at a rate of 10 packets per second in each direction.

The tester was set up with the following interfaces:

Tester-interface-setup

A bi-directional traffic stream was set up between the two ports:

Tester-flow-setup-1

With a flow-rate of 10 packets per second in each direction:

Tester-flow-setup-2

As can be seen, when traffic is started, the Tx and Rx traffic rates are the same, which is expected:

tester-flow-success

Using “monitor interface gr-0/0/0” it could be seen that packets were coming across the GRE tunnel at the expected rate of 20 packets per second:

gre-tunnel-stats

The ‘customer’ interface could be seen sending packets in both directions at a rate of 10PPS:

production-interface-stats

And the interface towards the capture device was seen sending 20PPS as expected:

mirroring-destination-port-stats

Finally, packets could be seen in the Wireshark capture file in both directions:

wireshark-capture

Advertisements

Actions

Information

4 responses

22 09 2015
Dmitry

other way for remote port-mirroring is using lt- interface as output interface and transport mirrored traffic over pweudowire.

22 09 2015
DataPlumber

Thanks Dmitry – I think we tried that and it didn’t work for us, but the customer was in a hurry so once we got the GRE solution working we didn’t try too hard on the pseudowire idea.

17 03 2016
Chintan Patel

Hi. Our mirrored traffic will be in the global table (ISP links). When I tried using the method above, it seems to create a routing loop, presumably due to the remote router taking that mirrored traffic coming across the tunnel and processing/forwarding it as regular traffic. Have you run into the issue?

6 04 2016
mukundh86

Chintan,

I believe the remote router has the filter with the “port-mirror” action and so the port-mirroring action will happen before the route-processing, So in essence, the packets should get forwarded out via the ge-1/1/9 interface.

However, I am testing this in lab as well.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: