ACI

ACI pcTag & VNID Descriptions including CLI Commands

Simon Birtles

…

pcTag (sclass)

This tag has a few different names depending on where you are referencing. In the managed objects (MO) you will see in a REST call, the attribute is called ‘pcTag’. On the CLI it will mainly referred to as Class ID.

  • A numeric ID for an EPG or VRF (Context)
  • A shorthand for GroupPolicyClassTag: The classification tag used for policy enforcement and zoning.
  • Any managed object class that inherits from fv:ATg (Attachable Target Group) has a pcTag property, for example:
    • fvAEPg
    • l2extInstP
    • l3extInstP
    • mgmtInstP
    • mgmtOoB
  • pcTags can either be system, global or local.
    • System means an internal fabric tag. e.g. 13 is a drop EPG, 15 is used for L3Out 0/0
    • Global means the pcTag is globally unique across the fabric
    • Local signifies that the pcTag is locally significant for the scope (VRF/context).
  • pcTag Number Ranges
    • System Reserved pcTag – This pcTag is used for system internal rules (1-15).
    • Globally scoped pcTag – This pcTag is used for shared service (16-16385).
    • Locally scoped pcTag – This pcTag is locally used per VRF (range from 16386-65535).

When applying L3Out options the pcTag can be changed by the APIC e.g. from Local to Global, this change of pcTag
can cause a brief traffic disruption

Example

Using an EPG name, we query the APIC using moquery to obtain the pcTag for the EPG.

apic01# moquery -c fvAEPg -f 'fv.AEPg.name=="EPG_NET_SERVICES"' | grep pcTag
pcTag                : 10934

The pcTag has a value of 10934 which is in the global range (16-16385) which implies this EPG has contracts (VRF leaking) with EPGs in other VRFs. Without tracing all the contracts we can use the vsh CLI on a leaf switch we know has the EPG deployed and find out which VRF byt using the pcTag.

vsh_lc
module-1# show system internal epmc epg | egrep 10934|SCOPE
   SCOPE     CLASS ID       Type       Policy Complete  VLAN/BD/VRF list
    2424832       10934                            Yes
    2785280       10934                            Yes
    2916352       10934      fd-vlan|              Yes    24, 23
    2555904       10934                            Yes

The CLASS ID is the pcTag (sclass) of the EPG of course, the SCOPE is a VRF identifier and actually the VNID of the VRF (discussed in the next section). We can find out what these VRFs are on the APIC using moquery again.

 moquery -c fvCtx -f 'fv.Ctx.scope=="xxxxxxxxx"' | grep name

So for each of these VRF scopes (VNID) perform a moquery and grep for ‘dn’ which gives us the tenant and VRF.

apic-01# moquery -c fvCtx -f 'fv.Ctx.scope=="2916352"' | grep dn
dn                   : uni/tn-common/ctx-VRF_COMMON

apic-01# moquery -c fvCtx -f 'fv.Ctx.scope=="2785280"' | grep dn
dn                   : uni/tn-TEN_DEVOPS/ctx-VRF_DEVOPS

apic-01# moquery -c fvCtx -f 'fv.Ctx.scope=="2916352"' | grep dn
dn                   : uni/tn-common/ctx-VRF_COMMON

apic-01# moquery -c fvCtx -f 'fv.Ctx.scope=="2555904"' | grep dn
dn                   : uni/tn-TEN_MAIN/ctx-VRF_MAIN

VNI/VNID

A VNI (VNID) is the VXLAN network identifier, VXLAN ID’s are assigned to a VRF, Bridge Domain or certain EPG configurations with connected endpoints.

  • Applies To
    • VRF / fvCtx
    • Bridge Domain / fvBD
    • EPG (EPG and VXLAN X Connected EP)
    • EPG (VLAN X Connected EP)
  • VLAN tunneled traffic carries the VNID
    • Routed Traffic: VRF VNI
    • Switched Traffic: BD VNI
    • Assigned dynamically on fabric startup/restart

Example

Bridge Domain

Using the APIC CLI, moquery and a BD name, we can get the BD VNID called ‘seg’ in the MO class, you also can get this from the APIC UI in the BD ‘Advanced/Troubleshooting’ tab where it is called ‘segment’.

apic-01# moquery -c fvBD -f 'fv.BD.name=="BD_DEVOPS"'
Total Objects shown: 1

# fv.BD
name                     : BD_DEVOPS
OptimizeWanBandwidth     : no
annotation               :
arpFlood                 : no
bcastP                   : 225.0.163.16
childAction              :
configIssues             :
descr                    :
dn                       : uni/tn-TEN_DEVOPS/BD-BD_DEVOPS
epClear                  : no
epMoveDetectMode         :
extMngdBy                :
hostBasedRouting         : no
intersiteBumTrafficAllow : no
intersiteL2Stretch       : no
ipLearning               : yes
ipv6McastAllow           : no
lcOwn                    : local
limitIpLearnToSubnets    : yes
llAddr                   : ::
mac                      : 00:22:BD:F8:19:FF
mcastAllow               : no
modTs                    : 2020-12-21T07:44:12.377+02:00
monPolDn                 : uni/tn-common/monepg-default
mtu                      : inherit
multiDstPktAct           : bd-flood
nameAlias                :
ownerKey                 :
ownerTag                 :
pcTag                    : 16386
rn                       : BD-BD_DEVOPS
scope                    : 2785280
seg                      : 16547722
status                   :
type                     : regular
uid                      : 15374
unicastRoute             : yes
unkMacUcastAct           : proxy
unkMcastAct              : flood
v6unkMcastAct            : flood
vmac                     : not-applicable

We can look on the Spine nodes in BGP for the L2/L3 advertisements being sent. In the output we can see all the hosts in the given bridge domain being advertised as either a L2 entry (MAC only) or a L2/L3 entry (MAC/IP) in a EVPN address family. This is useful in multi-pod / GOLF (External EVPN) setups to ensure the hosts are being advertised between the pods if experiencing connectivity issues between hosts.

spine-01-101# show bgp l2vpn evpn vrf overlay-1 | grep -A1 -E 'Network|16547722'
   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 1:16777199    (L2VNI 1)
--
*>l[2]:[0]:[16547722]:[48]:[000c.2938.d5a2]:[0]:[0.0.0.0]/216
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.4155]:[0]:[0.0.0.0]/216
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.6f7e]:[0]:[0.0.0.0]/216
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.71f3]:[0]:[0.0.0.0]/216
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.72ab]:[0]:[0.0.0.0]/216
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.9ec5]:[0]:[0.0.0.0]/216
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.bd9f]:[0]:[0.0.0.0]/216
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.c9a9]:[0]:[0.0.0.0]/216
                      0.0.0.0                           100      32768 i
--
*>l[2]:[0]:[16547722]:[48]:[000c.2938.d5a2]:[32]:[10.243.176.19]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.4155]:[32]:[10.243.176.20]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.6f7e]:[32]:[10.243.176.33]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.71f3]:[32]:[10.243.176.21]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.72ab]:[32]:[10.243.176.10]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.72ab]:[32]:[10.243.176.253]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.9ec5]:[32]:[10.243.176.9]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.bd9f]:[32]:[10.243.176.100]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[16547722]:[48]:[0050.5688.c9a9]:[32]:[10.243.176.22]/272
                      0.0.0.0                           100      32768 i

VRF (Context)

To continue the example, in the output from the first moquery for the Bridge Domain ‘BD_DEVOPS’ above, an attribute called ‘scope’ is present. This scope is relative to the L3 domain (VRF), i.e. the scope of the bridge domain L3 network. Using moquery to find a VRF with a scope of ‘2785280’ we find this VRF is ‘VRF_DEVOPS’ with an attribute ‘seg’ (segment) which we know to be the VNID for the VRF. 

apic-01# moquery -c fvCtx -f 'fv.Ctx.scope=="2785280"'
Total Objects shown: 1

# fv.Ctx
name                 : VRF_DEVOPS
annotation           :
bdEnforcedEnable     : no
childAction          :
descr                :
dn                   : uni/tn-TEN_DEVOPS/ctx-VRF_DEVOPS
extMngdBy            :
ipDataPlaneLearning  : enabled
knwMcastAct          : permit
lcOwn                : local
modTs                : 2020-12-21T07:43:28.904+02:00
monPolDn             : uni/tn-common/monepg-default
nameAlias            :
ownerKey             :
ownerTag             :
pcEnfDir             : ingress
pcEnfDirUpdated      : yes
pcEnfPref            : enforced
pcTag                : 32770
rn                   : ctx-VRF_DEVOPS
scope                : 2785280
seg                  : 2785280
status               :
uid                  : 15374

Taking the ‘seg’ value of 2785280 (which is the same as scope which isn’t a real surprise in this simple case), we can look at the BGP EVPN routes to see what is being advertised in the VRF VXLAN ID. The output shows all the subnet gateways configured within this VRF.

spine-01-101# show bgp l2vpn evpn vrf overlay-1 | grep -A1 -E 'Network|2785280'
   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 1:16777199    (L2VNI 1)
--
*>l[2]:[0]:[2785280]:[48]:[0200.0000.0002]:[32]:[10.75.60.1]/272
                      0.0.0.0                           100      32768 i
*>l[2]:[0]:[2785280]:[48]:[0200.0000.0002]:[32]:[10.243.176.254]/272
                      0.0.0.0                           100      32768 i

Other Notable Items

FD_VLAN VNID: (Fabric Encapsulation VXLAN network identifier)

Flood domain VLAN.

The FD-VLAN is the forwarding VLAN used to forward traffic on the Broadcom ASIC. The FD_VLAN is directly linked to the ACCESS_ENC and is also referred to as the internal VLAN.

The FD_VLAN is used to represent the ACCESS_ENC instead of linking it directly to the BD_VLAN.

The FD_VLAN allows the BD_VLAN to link to different ACCESS_ENCs and treat all of them as if they were all in the same 802.1Q VLAN on a NX-OS switch.

When a broadcast packet comes into the leaf switch from the ACI fabric, the BD_VLAN can map to several FD_VLANs to allow the packet to be forwarded out different ports using different ACCESS_ENCs. The FD_VLAN is used to learn Layer 2 MAC addresses.

  • Used for special cases such as
    • when forwarding BPDUs
    • when using “Flood in Encapsulation”.
  • BPDUs are flooded throughout the fabric with a different VNID than the one associated with the bridge domain that the EPG belongs to.
    • This is to keep the scope of BPDU flooding separate from general multi-destination traffic in the bridge domain.
  • Assigned dynamically on fabric startup/restart

BGP Import Targets

  • VRF VNID used for [AS:VRF_VNID] route imports as shown in the previous output.

Protocol Route Maps

  • VRF VNID used as part of route map name e.g. exp-ctx-proto-xxxxxxx

PI-VLAN

  • Platform Independent (PI) VLAN
  • Leaf scope only

Maps EPG (VXLAN/VLAN) <=> PI-VLAN ID <=> VXLAN ID (Global Fabric Wide VXLAN ID)

Reference 

leaf-01-201# vsh_lc -c 'show system internal eltmc info vrf TEN_DEVOPS:VRF_DEVOPS'
VRF-TABLE: TEN_DEVOPS:VRF_DEVOPS
            vrf_type:         tenant   :::      context_id:             12
       overlay_index:              0   :::            vnid:        2785280
               scope:        2785280   :::          sclass:          32770
         v4_table_id:            0xc   :::     v6_table_id:     0x8000000c
          intf_count:              2   :::   intrn_vlan_id:              0
            VRF Intf:         Vlan36   ::: src_plcy_incomp:              1
            vnid_hex:       0x2a8000   :::  ingress_policy:            0x1
       vrf_intf_list: Vlan34,Vlan36,
          hw_vrf_idx:           4629   ::: nb_egr_outer_bd:              0
              seclbl:              9   ::: no_policy_apply:              0
     Telemetry label:              0   ::: Telemetry label Mask:              0
         vrf_bd_list: 42,36,34,
     sb_egr_outer_bd:              0   :::      sdk_vrf_id:             12
    Vrf Ext intf cnt:              0   :::     Vrf ft mask:              0
       Learn disable:              0   ::: ip_learn_disable:              0

          [SDK Info]:
            vrf_name:TEN_DEVOPS:VRF_DEVOPS
              vrf_id:             12   :::      hw_vrf_idx:           4629
            vrf_vnid:        2785280   :::        is_infra:              0
      tornbinfrahwbd:              0   :::  torsbinfrahwbd:              0
   ingressBdAclLabel:              0   ::: ingBdAclLblMask:              0
    egressBdAclLabel:              0   ::: egrBdAclLblMask:              0
            sg_label:              9   :::          sclass:          32770
       sp_incomplete:              1   :::      sclassprio:              3
           iplearnen:              0
          maclearnen:              0
       sclasslearnen:              0

[SDB INFO]:
v4 table
            vrf type:              1
              vrf id:             12
                vnid:        2785280
 internal infra vlan:              0
 external router mac:00:0c:0c:0c:0c:0c
v6 table
            vrf type:              1
              vrf id:             12
                vnid:        2785280
 internal infra vlan:              0
 external router mac:00:0c:0c:0c:0c:0c
::::

Simon Birtles

I have been in the IT sector for over 20 years with a primary focus on solutions around networking architecture & design in Data Center and WAN. I have held two CCIEs (#20221) for over 12 years with many retired certifications with Cisco and Microsoft. I have worked in demanding and critical sectors such as finance, insurance, health care and government providing solutions for architecture, design and problem analysis. I have been coding for as long as I can remember in C/C++ and Python (for most things nowadays). Locations that I work without additional paperwork (incl. post Brexit) are the UK and the EU including Germany, Netherlands, Spain and Belgium.