Configuring the New Cisco ACE Card in Layer 3 Mode

19 12 2006

I’ve recently been configuring up a pair of Cisco ACE (Application Control Engine) blades for a customer to install into a Cat 6509. These things are pretty new and constitute the latest generation of their content-switching products. They’re so new in fact that there doesn’t appear to be a sample configuration to be had anywhere on Cisco’s website.

If you want some basic product overview stuff, have a look at this page.

What I wanted to do was to configure basic layer-3 load-balancing, with a public Virtual IP address (VIP) and a pair of servers at the back-end. If you’ve not used a service module in a Catalyst 6500 before, it is a bit odd to get your head around.

Basically, you create a VLAN that runs between the MSFC and the service module, put an IP address on the MSFC (an SVI), and hand control of the VLAN to the ACE module. The MSFC part of the config looks like this:

vlan 10
vlan 11
vlan 12

interface Vlan10
    description **Client side**
    ip address 172.16.1.1 255.255.255.0

svclc module 2 vlan-group 10
svclc vlan-group 10  10-12

So we’re making a VLAN for the side of the box where the clients are and giving it an IP address. After that, we create a VLAN-group (10) for the ACE module in slot 2, and assign VLANs 10, 11 and 12 to it.

Next, we get a session going to the ACE module. (Default login is admin with a password of admin). The first thing we need to do is the IP addressing on the VLAN interfaces:

interface vlan 10
    description **Client side**
    ip address 172.16.1.2 255.255.255.0
    no shutdown
interface vlan 12
    description **SERVER VLAN**
    ip address 192.168.1.1 255.255.255.0
    no shutdown

At this point, you should be able to ping the MSFC from the ACE. If you have your servers up and running in VLAN 12, you should be able to ping them too. They should be configured with the 192.168.1.1 as their default gateway.

Next, create the “rservers” or real servers. This is where we define each individual server and bring it into service:

rserver host SERVER1
    description **SERVER1**
    ip address 192.168.1.10
    inservice
rserver host SERVER2
    description **SERVER2**
    ip address 192.168.1.11
    inservice

And put the rservers into a server-farm:

serverfarm host SFARM1
    rserver SERVER1
        inservice
    rserver SERVER2
        inservice

Now we need to write some policy to match on traffic to a VIP (in this case, 172.16.1.100), and balance it across the two servers in our farm. We’re just doing the default round-robin method here for the time being. Once we’ve got this working, then we can get a bit more clever.

If you have experience using Cisco’s Modular QoS CLI (MQC) you will find some of the statements below familiar, but with some extra bits added to support the load-balancing.

There is a four-step process going on here – matching traffic at layers 3/4, matching traffic at layer 7, performing an action and applying the policy to an interface:

1. In the class-map below, we’re looking to match any protocol that hits our VIP.

class-map match-all MATCH_VIP
    match virtual-address 172.16.1.100 any

2. In our layer 7 policy map (L7_POLICY) we’re not actually doing any L7 load-balancing at all – we’re just stating which serverfarm to use (SFARM1) in the default class (class-default).
Class-default is a catch-all and it’s configured action is performed on everything not already acted upon in some other class. In this case, since there are no other classes referenced, that means everything. When we get a bit more adventurous, this is where our layer 7 actions (cookies etc.) would be done.

policy-map type loadbalance first-match L7_POLICY
    class class-default
       serverfarm SFARM1

3. In the L4_POLICY, we are binding together the VIP and the L7 load-balancing policy (which tells us which serverfarm to use)

policy-map multi-match L4_POLICY
    class MATCH_VIP
        loadbalance vip inservice
        loadbalance policy L7_POLICY

4. And we apply the policy to the incoming VLAN interface:

interface vlan 55
    service-policy input L4_POLICY

Finally – the ACE denies all traffic to the VIP by default (a security measure), so we need to write a standard ACL to permit the traffic on the VLAN interface:

access-list ALL line 10 extended permit ip any any

interface vlan 55
    access-group input ALL

And you should be good to go. Yes – I know the ACL above is very liberal. Since we were testing in the lab, it wasn’t really all that important to be secure – and anyway, if you’re running a datacentre with web-servers in it, you wouldn’t protect them with ACLs would you?

Useful commands to check this out are:

show rservers – to see if the servers are up
show serverfarm – to see if they are in the farm and getting hits
ping – do I really need to explain?
show service-policy vlan55 – to see the policy as applied to the VLAN
show access-list ALL – to see if the ACL is getting hits

Advertisements

Actions

Information

24 responses

3 01 2007
Richard B

Thanks for the tips. I’m getting ready to deploying a pair of ACEs in a new IDC. I’m still trying to get my head around why a L7 policy is needed; is that the only way to identify which serverfarm to use if you only care about L3/L4 (e.g., the VIP and port 80)? I come from a Cisco CSS background that has nothing in common with the ACE.

I’m also scouring Cisco’s site and the web — and not succeeding much — in tips for the ff: default gateway locations, bypass routing (recommended or not) with the ACE for non-load-balanced traffic, and multiple contexts to support production and staging environments.

3 01 2007
DataPlumber

I know what you mean about coming from a CSS background and there being nothing in common! I eventually got the hang of the interface, but it took a while… I think the CLI’s syntax is a bit confusing to be honest – I’m sure they could have thought it out a bit better.

Like you, I’d been looking for a sample config, and since I hadn’t found one I thought it was worth writing up. I’ve also got what I did for multiple contexts and redundant operation, so I will post that tomorrow – hopefully it can be some use.

As far as I can tell, the L7 rule needs to be there (even if it does nothing other than match everything using class-default) since that is the only place you can tie the server-farm in. You can’t do that in the L4 rule.

I didn’t do any bypass work – the customer wanted to put it firmly in the data path between servers and clients. Any solutions you manage to get for this would be interesting to see.

13 02 2007
ACE in layer 3 mode - diagram « The Data Plumber

[…] ACE for some time and only just got round to doing so!    The instructions and sample config are here if you’re […]

1 05 2007
Scott H.

Where did VLAN 55 come from in your example?
Do we need a svclc command in the 6500 for VLAN 55?
I was expecting it to be VLAN 10 since that is where the clients are coming in via.

10 09 2007
Mario

Will you post the full configuration , the one for the MSFC and the ACE with how to route the traffic from another router to the 6509 w/ACE

24 09 2007
Ashes

Hi,

Luckily found the site while looking for ACE sample configs.

Very illustrative.

Any sample configs on Layer 7 URL load balancing?

Thanks,

14 12 2007
corefile

Layer 3 loadbalancing???? that is not layer 3, that is called layer 4 load balancing

29 01 2008
PatG

Great stuff. Better than the Cisco documentation. I appreciate you putting this out there.

8 02 2008
GP

Is anyone using the ACE module for Firewall Loadbalancing?

8 02 2008
DataPlumber

Yeah – they are – one of my customers is using it that way, but I wasn’t involved in the configuration. I’ll see if I can dig out the notes I took about the things you needed to consider, and post them up.

18 02 2008
Nanthee

I am going to deploy a Primary Data Center with 6500 (Sup 720 ) switches , ACE modules , 10G modules , and 4948 (10G) switches. i have checked the cisco web sites for L3 Routed mode Sample configuration. that is not there…
So can any one come up with some brief of L3 Routed Designs and configuration… if so … it will be a great help…

Thanks,
Nanthee

8 04 2008
What u dont knw

Nanthee,

It’s better if you can ask it from your help desk. She might know.

17 11 2008
Dwight

Hi,

Thanks for this helpfull article. I’m also configuring a cat 6504 with ACE module. I made a vlan between the sup and the ACE module. I can ping from the ACE module to the sup interface but not the other way round. I cannot ping from the sup module. Do you have any idea?

17 11 2008
DataPlumber

Hi Dwight – I think the ACE card is ‘hardened’ by default, so you will need to create an access-list specifically to permit the pings. Otherwise the card discards traffic destined for itself – this is probably a defence against DoS attacks etc.

23 11 2008
Mohammad khalil

hey all
can i ping the interface vlan configured on the ACE module from the MSFC ??

26 11 2008
DataPlumber

Hi – you should be able to ping it if you have configured an access-list permitting ping on the ACE. By default the ACE permits no traffic to itself.

23 01 2009
John Q

Hi DataPlumber, I just configured the MSFC and ACE but I got 50% ping timeout from both side. Do you know any possible reason?

23 02 2009
Mary

Hello, very useful info, really helped me to understand what I am doing
my problem is that load balancing does not work as round-robin, it send more requests to one server than the other one. Any one know why?(weight:8 default)

23 02 2009
Mary

Hello, very useful info, really helped me to understand what I am doing
my problem is that load balancing does not work as round-robin, it send more requests to one server than the other one. Any one knows why?(weight:8 default)

28 03 2009
scott elmer

we have a pair of redundant ace moduals in cat 6509’s , our firewalls are juniper 208s, we replaced our juniper dx’s with the ace moduals, ever since we did this, our apahce servers are getting hammered, we found that keepalives aleviated some of the load to the servers, but then our firewalls were getting hammered, we have been working on trying to resolve this for months, we’ve even had cisco in here with the ace appliacances’ … has anyone else had this problem?/

30 01 2010
Jack

Great Tip … was very helpful . Thanks

13 07 2010
Loren

Hi DataPlumbmer,

My company just transfered me into group where I need to configure/maintain /troubleshoot a network of ACEs. The problem is that I don’t have a slightest understanding how these modules work. I am hoping that you could recommend me a good ACE book/content switching or even Cisco documentations so I can build my knowledge to support this network. After much research, I still can’t seem to find good documentation on the Cisco site. Thank you in advance and I am so glad I found your site.

20 07 2010
DataPlumber

Hi there –
Unfortunately I think I had the same issues as you did when I was originally doing the ACE work. It was quite a while ago now, and I seem to have forgotten a lot of stuff, but definitely remember the lack of documentation there was available. Are you dealing with the appliance or the Catalyst blade?
I seem to remember having to set my blades up in the lab and going through a lot of trial and error. If you’ve a spare device, maybe this is the best way forward. (And if you don’t, it’s definitely worth trying to get one to form a kind of ‘proof of concept’ testbed on which you can try stuff out – breaking a load-balancer usually causes some upset!). If you’ve a specific question, I’ll see what I can do to help out. If you’ve got a config you don’t understand, I could try to annotate it to show what it does maybe?

16 02 2016
CISCO ACE LOAD BALANCERS CONF:::: – skminhaj

[…] CONFIGURING THE NEW CISCO ACE CARD IN LAYER 3 MODE […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: