Layer 3 External Routing or L3Out’s are a “bread and butter” requirement for designing and building Cisco ACI networks. This blog is a companion document to the blogs on different L3Out designs and configuration options and in some ways required fundamental knowledge and is here to refer to whilst reading the other blogs.

The goal of this blog is to understand the configuration options for L3Out (l3ext:Out) External Network Instance Profiles (l3ext:InstP) and the subnet (l3Ext:Subnet) scope options.

External Network Instance Profile – Subnet Control Options (l3extSubnet)

A L3Out receives network (subnet) route advertisements from outside the fabric using OSPF, EIGRP, BGP and even static routes (although defined internally on the L3Out Leafs). There is a difference or abstract logical separation that needs to be understood to help understand the options presented in the External Network Instance Profiles (External EPG) object within the L3Out. We need to split the L3Out function of receiving routes with the External Network Instance Profile, as stated, the L3Out receives (and sends) routes with the outside world.

The External Network Instance Profiles are special EPG’s associated with the L3Out which contain no endpoints by default, endpoints for this L3Out EPG are actually external networks (subnets). We may have stated the obvious here but the distinction is important because the L3Out will receive the routes from the outside world quite happily but do nothing with them until we put some policy (rules) in place.

The options are are about to discuss fall into a few categories and are best understood in this way. We define the categories as:

• Filtering Inbound & Outbound Route Exchanges
• Associating External Subnets With an External EPG
• Specifying Route Leak Options (Shared Service)
• Aggregate Subnet Options

Each of the following subsections discuss the options in theses categories, only the aggregate options are not technically part of the subnet configuration but nonetheless the aggregate options do effect how the APIC will process the options applied to the subnets defined in the scope of the L3Out. The managed object (MO) for the L3Out External EPG subnet is “l3Ext:Subnet” , the MO has an attribute “scope” which is the options we select for the L3Out External EPG (l3extInstP) subnets, these are the options we will be discussing first.

The MIM for the MO “l3ExtSubnet” attribute “scope” lists the possible values as:

import-rtctrl	Import Route Control Subnet	Route is for Import Route Control Policy
export-rtctrl	Export Route Control Subnet	Route is for Export Route Control Policy
import-security	External Subnets for the External EPG	Route is for Import Security Policy (Default)
shared-security	Shared Security Import Subnet	Route is for applying security on it in the Ctx/VRF into which it is shared
shared-rtctrl	Shared Route Control Subnet	Route is for leaking into consumers Ctx/VRF in case of Shared Service

Filtering Inbound & Outbound Route Exchanges

• Export Route Control Subnet (export-rtctrl)
  • Route is for Export Route Control Policy
  • Controls the exporting of external routes (external routes imported with another L3Out)
  • This option is only available if the L3Out has “Export Route Control Enforcement” enabled, by default it is enabled.
  • The option allows the advertisement of specific transit routes out of the fabric.
    • A transit route is a route that has been received into the fabric from outside by a L3Out and will be advertised outside through another L3Out.
  • This is for transit routes only, and it does not control the internal routes or default gateways configured on a bridge domain.
  • Controls which external networks are advertised out of the fabric, using route-maps and IP prefix-lists.
  • A ‘Route Control Profile’ may also be applied to an External Network Instance Profile (External EPG) to permit prefixes to be advertised outside through the L3Out
• Import Route Control Subnet (import-rtctrl)
  • Route is for Import Route Control Policy 
  • Controls the importing of external routes
  • Import route control is not only used to filter routes, it is also used to apply route-map match and set statements to received prefixes.
  • By default this option is disabled and all prefixes are allowed into the fabric.
  • This option is only available if the L3Out has “Import Route Control Enforcement” enabled, by default it is disabled.
  • Only valid with BGP and OSPF.

Associating External Subnets With an External EPG

Remember that all endpoints whether individual hosts or networks (subnets), internal or external must be a member of an EPG so contracts (permissions) can be granted for the endpoints in the different EPGs to communicate. Also take into account, there can be more than one External Network Instance Profile (External EPG) created in an L3Out so we need to know which external networks should be in which External EPGs so contracts can be applied.

• External Subnets for the External EPG (import-security)
  • Route is for Import Security Policy 
  • For all subnets received by the L3Out, this option declares (selects) which of these subnets should be associated with this External EPG l3extInstP.
  • Once declared subnets are associated with this External  EPG, the contracts applied to this External EPG then determine with which other EPGs the specified external subnet is allowed to communicate.
  • Specifies which external subnets (l3extSubnet) have contracts applied as part of a specific External EPG (l3extInstP)
  • For a subnet under the External EPG (l3extInstP) to be classified in an External EPG this option must be selected, the “l3extSubnet” scope will be “import-security” when selected.
  • For example, when traffic enters the ACI switch on the L3Out (L3extOut), a lookup occurs to determine which source IP addresses are associated with the External EPG (l3extInstP). This action is performed based on Longest Prefix Match (LPM) so that more specific subnets take precedence over more general subnets.

Specifying Route Leak Options (Shared Service)

The following two options relate to the leaking of subnets (routes) from the VRF in which the L3Out & External EPG are configured to another EPG (& VRF). This EPG could be an internal EPG or another External EPG. The first option is to permit inter-vrf leaking, the second option is to allow contracts to be applied to the subnets. The second option gives a little more filtering capability that usual contracts only, contracts state L4 filtering and not L3, this gives the L3 filtering control for contacts.

• Shared Route Control Subnet (shared-rtctrl)
  • Route is for leaking into consumers Ctx/VRF in case of Shared Service 
  • It is used to permit route leaking from the L3Out VRF to other VRF’s in the same or different tenants.
  • If the declared subnet in the External EPG is in the L3Out VRF and has been learned from outside the fabric it can be leaked to the other VRFs where contracts exist.
  • This is the only route leak option required out of this one and the one below, when we have a setup with a provider L3Out EPG to another Tenant/VRF EPG as a consumer via an exported contract from the L3Out Tenant.
• Shared Security Import Subnet (shared-security)
  • Route is for applying security on it in the Ctx/VRF into which it is shared 
  • This option applies the specified contract on the prefix within the VRF that has had the prefixed leaked\advertised into (leaking done by Shared Route Control Subnet).
  • If you want traffic to flow from one external EPG to another external EPG or from one external EPG to another internal EPG, the subnet must be marked with this control.
  • When this option is not selected and prefixes are leaked, there are default src and dst EPGs with a default filter applied to the prefix which is a drop filter.
    • Use the line card shell (vsh_lc) command (show system internal aclqos prefix) on the leaf switch, this will show the class ID applied to the prefix.
  • Using an example to see how this works, if we look at a leaf switch that has the VRF configure that receives an external prefix (leaked into) of 10.124.3.0/24. This route appears in this prefix table as the shared-rtctrl option (previously discussed and required) has been selected. The differences are shown in the two tables, the first being when this option ‘shared-security’ is selected, the prefix is shown a number of times but we are concerned about the entry with the Vrf-Vni of 2326538 because this is the VNI (called segment ID in the VRF) for the VRF having the route leaked into. In the first table this is the last entry with a class of 5480 which is the External EPG (l3extInstP) pcTag and communication works successfully. If we de-select this option and then look at the output of the same command (the second table), we see the prefix now has a class of 13, which is an APIC internal class ID, communication is now broken as this class ID ’13’ which is an internal EPG which has a deny contract/filter applied.

Option Selected
module-1# show system internal aclqos prefix
 
Vrf-Vni VRF-Id Table-Id          Addr                    Class  Shared Remote Complete
======= ====== ============ ============================ ====== ====== ====== ========
2359296 4      0x80000004     0::/0                        15     0      0      No
2359296 4      0x4            0.0.0.0/0                    15     0      0      No
2359296 4      0x4            10.124.3.0/24                49154  0      1      No
2129921 8      0x80000008     0::/0                        15     0      0      No
2129921 8      0x8            0.0.0.0/0                    15     0      0      No
2129921 8      0x8            10.124.3.0/24                5480   0      1      No
2129921 8      0x8            10.126.1.0/24                5480   0      1      No
2326538 21     0x80000015     0::/0                        15     0      0      No
2326538 21     0x15           0.0.0.0/0                    15     0      0      No
2326538 21     0x15           10.124.3.0/24                5480   1      1      No
module-1#
 
Option Deselected
module-1# show system internal aclqos prefix
 
Vrf-Vni VRF-Id Table-Id          Addr                    Class  Shared Remote Complete
======= ====== ============ ============================ ====== ====== ====== ========
2359296 4      0x80000004     0::/0                        15     0      0      No
2359296 4      0x4            0.0.0.0/0                    15     0      0      No
2359296 4      0x4            10.124.3.0/24                49154  0      1      No
2129921 8      0x80000008     0::/0                        15     0      0      No
2129921 8      0x8            0.0.0.0/0                    15     0      0      No
2129921 8      0x8            10.124.3.0/24                5480   0      1      No
2129921 8      0x8            10.126.1.0/24                5480   0      1      No
2326538 21     0x15           10.124.3.0/24                13     1      1      No
2326538 21     0x80000015     0::/0                        15     0      0      No
2326538 21     0x15           0.0.0.0/0                    15     0      0      No
module-1#

Aggregate Subnet Options

This option is not part of the subnet “l3ExtSubnet”  attribute “scope” classification, the “Aggregate Shared Routes” options are …

• Aggregate Shared Routes
  • Enable if the defined subnet is an aggregate if you want the routes to be advertised, by default the given subnet must match exactly the received subnet from external to be then injected into  fabric tenant VRFs. Enabling this allows more specifics of the aggregate received to be advertised into the fabric tenant VRFs
  • When aggregation is not set, the subnets are matched exactly. For example, if 11.1.0.0/16 is the subnet, then the policy will not apply to a 11.1.1.0/24 route but it will apply only if the route is 11.1.0.0/16. However, to avoid a tedious and error prone task of defining all the subnets one by one, aggregation can be used aggregate a set of subnets into one export, import or shared routes policy. At this time, only 0/0 subnets can be aggregated. When 0/0 is specified with aggregation, all the routes are imported, exported, or shared with (leaked to) a different VRF based on the selection option below:
    • Aggregate Export—Exports all routes of a subnet (0/0 subnets).
    • Aggregate Import—Imports all routes of a subnet (0/0 subnets).
    • Aggregate Shared Routes—When a route is learned in one VRF that needs to be leaked into another VRF, the routes can be leaked by matching the subnet exactly, or can be leaked in an aggregate way according to a subnet mask.
  • Note: For aggregate shared routes, multiple subnet masks can be used to determine which specific route groups are leaked between VRFs (for example, 10.1.0.0/16), or 0/0 can be used to share all subnet routes across multiple VRFS.