ACIKubernetes (K8)

ACI & Kubernetes – The Calico K8s CNI & MetalLB (Part Two)

When comparing the Cisco K8 CNI and Calico/MetalLB, I do tend to prefer the Calico/MetalLB implementation. Calico is more open in it flexibility to customise the solution. Calico runs in on-premise and cloud environments providing a nice level of consistency across hybrid deployments, we have been using the open source version in this deployment which is more that functional for a production environment (minus support of course), the enterprise version offers support but also a host of other management and visibility features, you can see these here.

If you missed Part 1 around build and configuration of this solution, its here.

With the Cisco CNI, if you read my write up about this, I do not like the way in which the CNI integration with ACI is forcing topologies (L3O w.PBR SG) / design within ACI, this should never be forced upon you, having said that I do like the ability to classify a deployment into a Tenant/Application/EPG automatically and have the endpoints dynamically updated (if pod IPs). This way I can apply contracts to specific deployments not just to the cluster as a whole.

Modelling EPG Security

When building the Calico solution we have modelled the L3Out EEPGs in a similar way, just that these are EEPGs instead on IEPGs. Each application (multicast or dashboard) is classified into an EEPG by its IP address (EEPGs are IP based classification) for either a pod IP or a service IP.

So two things we need to cover to get the same kind of dynamic EEPG membership that we get with the Cisco CNI.

1. Dynamic EEPG Endpoint Membership

To have a dynamic EEPG endpoint membership we need to use the K8 API to watch for changes to deployments (create, update, delete). Using the K8 API server to watch for changes, this can be implemented in a K8 deployment, using a number of different client libraries. The idea here is that code can maintain the EEPG which is related to a deployment and ensure that the subnets (list of pods and services) are kept up to date. With this in place we can apply the contracts in the way we see fit. Once contracts are in place the IP addresses are then maintained when scaling a application for example.

Create Deployments/Services

We need to understand if deployments with or without services have been created, if this is the case we then need to create an EEPG in a specified L3Out in ACI using the APIC REST API. Within the EEPG for the deployment, we would add subnets in the form of host addresses for the pod IP’s and service load balancer IP (if a service has been created).

Update Deployments/Services

Using the K8 API again to watch for changes for deployments and/or services, if we see a change to a deployment like the creation or deletion of a pod, we need to use the ACI APIC REST API to modify the EEPG subnets list and in the same way for a change to the service load balancer IP.

Delete Deployments/Services

Following on from the create and update, we of course delete the subnets from the EEPG or delete the EEPG entirely if the the deployment is deleted.

2. Service Graph PBR / NLB Integration

We have a number of options to provide load balancing, the following sections touch on a few of the many options.


Using MetalLB is the simplest option as we have seen in this deployment and the previous Dynamic EEPG Endpoint Membership workflow would work well with this. MetalLB is intended for this on-premise use case and keeps the application load balancing in the hands / or on the K8 side.

ACI PBR Service Graph

Using an ACI service graph would mean the Service Graph PBR (Redirect Policy) would be targeting the Pod IPs, if it targeted the MetalLB IP it would be pointless unless there were two deployments for availability and then ACI SG PBR could load balance between them and then MetalLB does the load balancing within the deployment.

For the situation where we have one deployment and ACI SG PBR across the pod IP’s, we need to use a headless deployment for the deployment service but still specify ‘externalTrafficPolicy: Local’ so we avoid kube-proxy moving traffic between nodes. its actually optional depending on how you distribute your pods. The key point here is that we would need to specify a Tenant and a Redirect Policy for the ‘watcher’ code to update with pod IP’s. I would only create/update the Redirect via the watcher code so the policy can be used in any service graph which is maintained outside of the ‘watcher’ code.

Also for a ACI PBR SG, we need to have an IP address which is used for the traffic to target (VIP), just like a normal load balancer, the problem with ACI is that this can be tricky and can be a bit of a fudge. So let see in the blog about this solution what our best options are.


If we use another external NLB, like F5, again we would want the server farm to be updated with the pod IP’s. On ACI we don’t actually need a SG for a NLB being inserted on its own unless we want reuse the consumer side many times, which then we can use a non PBR SG in ACI.

F5 have a solution on ACI with normal IEPGs. An app by F5 (F5 App Center – ACI App) is installed on the ACI APIC and its a stateful app which means its runs in its own container on the APIC and watched for changes in specified IEPGs (new/deletes endpoints) and updates a F5 server farm so keeping the F5 NLB service in sync with the hosts available, this way to scale an application, just spin up a few more host clones in the same EPG and F5 automatically adds them to the server farm.

F5 also have a solution for Kubernetes which is detailed here.

Wrap Up

I hope to find the time to write some code to cover the above examples as a POC and a bit of fun. I think there is a really good potential for a simple integration with ACI and K8 using Calico and with or without MetalLB. The Cisco CNI does too much and doesn’t allow any changes which makes it inflexible, the Calico/MetalLB solution has non of these issues and just wants a little bit of sync code put in place.

So I did find sometime to start something around what I have discussed would be the middle ground. You can take a look on GitHub.

Of course there are a lot more CNI’s to choose from of which you can find a list of some of them here.

Simon Birtles

I have been in the IT sector for over 20 years with a primary focus on solutions around networking architecture & design in Data Center and WAN. I have held two CCIEs (#20221) for over 12 years with many retired certifications with Cisco and Microsoft. I have worked in demanding and critical sectors such as finance, insurance, health care and government providing solutions for architecture, design and problem analysis. I have been coding for as long as I can remember in C/C++ and Python (for most things nowadays). Locations that I work without additional paperwork (incl. post Brexit) are the UK and the EU including Germany, Netherlands, Spain and Belgium.

2 thoughts on “ACI & Kubernetes – The Calico K8s CNI & MetalLB (Part Two)

  • Blake Lead

    I read thoroughly your articles about Cisco ACI and Kubernetes (both with Cisco CNI and Calico) and I find them an invaluable and rare piece of knowledge. Thank you very much for that!

    As I am new to the Cisco world and not too comfortable with network concepts, I have trouble understanding what part would be missing when considering migrating from ACI-CNI to Calico CNI.

    Is it right to say that the same level of informations would be available on the ACI console? (Pods IPs, health, etc.)
    It seems that with the conjunction of metallb, Calico would be equivalent to ACI-CNI. Am I missing something?
    And lastly, I understand you were working on a program that would sync ACI with Kubernetes. I thought metallb+Calico would suffice for that. What happens if a service is updated/deleted? Shouldn’t Calico handle the ACI routes update via BGP?

    • Hi Blake,
      With the Calico CNI you would not get the same level of information from the APIC in terms of Pod IPs, Health, etc as there is no integration between ACI and Calico. The Cisco CNI works in a slightly different way in that the Cisco CNI extends VXLAN from the ACI fabric into the k8s cluster with the CNI. Along with this, the Cisco CNI also communicates directly with the APIC from provide this metadata, in a similar1 way that the VMM VMWare vCenter integration works, in that the APIC “pulls” the VM data back from vCenter (via subscriptions to vCenter over HTTP). For the Calico CNI (and any other CNI for that matter i.e. Cilium) it is just the same as connecting an external router running BGP to the ACI fabric.

      I was working on an integration between k8s and ACI as a controller in k8s. The idea of this was only to update the ACI L3out EEPG subnets for new k8s loadbalancer services. The point of this was to add the host address of the new service to a specified EEPG for contract purposes, but this did not replace BGP, BGP still advertises the k8s subnets or host addresses as normal. The difference with the Cisco CNI was that the Cisco CNI did all this automatically using iEPGs (Internal EPGs). I did not finish this as I don’t have the time due to work commitments, but could be implemented in many other ways with DevOps tooling and using something like the external-dns controller with some DevOps tooling integration. This approach although valid is not my preferred way in that I would generally use a single EEPG for the loadbalancer host addresses (subnet) and control the traffic less through contracts but through CNI security policies and VRF firewalls – but this is highly dependent on your design.

      If you are using the enterprise versions of Calico or Cilium, you then get access to the premium observability tools which provide much more detailed data that the Cisco CNI and ACI but of course its extra cost.


Leave a Reply

Your email address will not be published. Required fields are marked *