When comparing the Cisco K8 CNI and Calico/MetalLB, I do tend to prefer the Calico/MetalLB implementation. Calico is more open in it flexibility to customise the solution. Calico runs in on-premise and cloud environments providing a nice level of consistency across hybrid deployments, we have been using the open source version in this deployment which is more that functional for a production environment (minus support of course), the enterprise version offers support but also a host of other management and visibility features, you can see these here.

If you missed Part 1 around build and configuration of this solution, its here.

With the Cisco CNI, if you read my write up about this, I do not like the way in which the CNI integration with ACI is forcing topologies (L3O w.PBR SG) / design within ACI, this should never be forced upon you, having said that I do like the ability to classify a deployment into a Tenant/Application/EPG automatically and have the endpoints dynamically updated (if pod IPs). This way I can apply contracts to specific deployments not just to the cluster as a whole.

Modelling EPG Security

When building the Calico solution we have modelled the L3Out EEPGs in a similar way, just that these are EEPGs instead on IEPGs. Each application (multicast or dashboard) is classified into an EEPG by its IP address (EEPGs are IP based classification) for either a pod IP or a service IP.

So two things we need to cover to get the same kind of dynamic EEPG membership that we get with the Cisco CNI.

1. Dynamic EEPG Endpoint Membership

To have a dynamic EEPG endpoint membership we need to use the K8 API to watch for changes to deployments (create, update, delete). Using the K8 API server to watch for changes, this can be implemented in a K8 deployment, using a number of different client libraries. The idea here is that code can maintain the EEPG which is related to a deployment and ensure that the subnets (list of pods and services) are kept up to date. With this in place we can apply the contracts in the way we see fit. Once contracts are in place the IP addresses are then maintained when scaling a application for example.

Create Deployments/Services

We need to understand if deployments with or without services have been created, if this is the case we then need to create an EEPG in a specified L3Out in ACI using the APIC REST API. Within the EEPG for the deployment, we would add subnets in the form of host addresses for the pod IP’s and service load balancer IP (if a service has been created).

Update Deployments/Services

Using the K8 API again to watch for changes for deployments and/or services, if we see a change to a deployment like the creation or deletion of a pod, we need to use the ACI APIC REST API to modify the EEPG subnets list and in the same way for a change to the service load balancer IP.

Delete Deployments/Services

Following on from the create and update, we of course delete the subnets from the EEPG or delete the EEPG entirely if the the deployment is deleted.

2. Service Graph PBR / NLB Integration

We have a number of options to provide load balancing, the following sections touch on a few of the many options.

MetalLB

Using MetalLB is the simplest option as we have seen in this deployment and the previous Dynamic EEPG Endpoint Membership workflow would work well with this. MetalLB is intended for this on-premise use case and keeps the application load balancing in the hands / or on the K8 side.

ACI PBR Service Graph

Using an ACI service graph would mean the Service Graph PBR (Redirect Policy) would be targeting the Pod IPs, if it targeted the MetalLB IP it would be pointless unless there were two deployments for availability and then ACI SG PBR could load balance between them and then MetalLB does the load balancing within the deployment.

For the situation where we have one deployment and ACI SG PBR across the pod IP’s, we need to use a headless deployment for the deployment service but still specify ‘externalTrafficPolicy: Local’ so we avoid kube-proxy moving traffic between nodes. its actually optional depending on how you distribute your pods. The key point here is that we would need to specify a Tenant and a Redirect Policy for the ‘watcher’ code to update with pod IP’s. I would only create/update the Redirect via the watcher code so the policy can be used in any service graph which is maintained outside of the ‘watcher’ code.

Also for a ACI PBR SG, we need to have an IP address which is used for the traffic to target (VIP), just like a normal load balancer, the problem with ACI is that this can be tricky and can be a bit of a fudge. So let see in the blog about this solution what our best options are.

NLB

If we use another external NLB, like F5, again we would want the server farm to be updated with the pod IP’s. On ACI we don’t actually need a SG for a NLB being inserted on its own unless we want reuse the consumer side many times, which then we can use a non PBR SG in ACI.

F5 have a solution on ACI with normal IEPGs. An app by F5 (F5 App Center – ACI App) is installed on the ACI APIC and its a stateful app which means its runs in its own container on the APIC and watched for changes in specified IEPGs (new/deletes endpoints) and updates a F5 server farm so keeping the F5 NLB service in sync with the hosts available, this way to scale an application, just spin up a few more host clones in the same EPG and F5 automatically adds them to the server farm.

F5 also have a solution for Kubernetes which is detailed here.

Wrap Up

I hope to find the time to write some code to cover the above examples as a POC and a bit of fun. I think there is a really good potential for a simple integration with ACI and K8 using Calico and with or without MetalLB. The Cisco CNI does too much and doesn’t allow any changes which makes it inflexible, the Calico/MetalLB solution has non of these issues and just wants a little bit of sync code put in place.

So I did find sometime to start something around what I have discussed would be the middle ground. You can take a look on GitHub.

Of course there are a lot more CNI’s to choose from of which you can find a list of some of them here.