ACINexus

ACI ELAM Packet Capture – Examples

ACI ELAM examples with output.

ASIC Types Reference

Northstar: ns: – Gen1
Alpine: alp – Spines
Rocky: roc – F/FX/FXP
Tahoe: tah – GC/E/EX

Example 1

Find the source interface of a DNS packet from a client to a DNS server. The steps below should be performed on both switches where the ingress or egress is in a vPC over the vPC switch pair or any switch pair where traffic can transit, like an external border router with multiple ECMP connections spread over leaf switches.

Identify the Endpoint Physical Interface

dev-leaf-01-201# show endpoint ip 192.168.10.1
Legend:
 s - arp              H - vtep             V - vpc-attached     p - peer-aged
 R - peer-attached-rl B - bounce           S - static           M - span
 D - bounce-to-proxy  O - peer-attached    a - local-aged       m - svc-mgr
 L - local            E - shared-service
+-----------------------------------+---------------+-----------------+--------------+-------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface
      Domain                          VLAN            IP Address        IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
23                                        vlan-1201    0050.5688.712f LV                       po11
common:VRF_COMMON                         vlan-1201      192.168.10.1 LV                       po11
dev-leaf-01-201# show port-channel summary interface po11
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
        M - Not in use. Min-links not met
        F - Configuration failed
-------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
-------------------------------------------------------------------------------
11    Po11(SU)    Eth      LACP      Eth1/21(P)

Line Card Shell (VSH)

leaf-01-218# vsh_lc
module-1# show platform internal hal l2 port gpd
Legend:
-------
IfId:       Interface Id                                    IfName:      Interface Name
I P:        Is PC Mbr                                       IfId:           Interface Id
Uc PC Cfg:      UcPcCfg Idx                                 Uc PC MbrId:    Uc Pc Mbr Id
As:             Asic                                        AP:             Asic Port
Sl:             Slice                                       Sp:             Slice Port
Ss:             Slice SrcId                                 Ovec:           Ovector (slice | srcid)
L S:            Local Slot                                  Reprogram:
L3:             Is L3
    P:    PifTable                                          Xla Idx:        Xlate Idx
    RP:   Rw PifTable                                       Ovx Idx:        OXlate Idx
    IP:   If Profile Table                                  N L3:           Num. of L3 Ifs
    RS:   Rw SrcId Table                                    NI L3:          Num. of Infra L3 Ifs
    DP:   DPort Table                                       Vif Tid:        Vif Tid
    SP:   SrcPortState Table                                RwV Tid:        RwVif Tid
    RSP:  RwSrcPortstate Table                              Ing Lbl:        Ingress Acl Label
    UC:   UCPcCfg                                           Egr Lbl:        Egress Acl Label
    UM:   UCPcMbr                                           Reprogram:
PROF ID:        Lport Profile Id
    VS:   VifStateTable                                     HI:             LportProfile Hw Install
    RV:   Rw VifTable
Num. of Sandboxes: 1

Sandbox_ID: 0, BMP: 0x0
 Port Count: 49

==============================================================================================================================================
                       Uc   Uc                          |       Reprogram       |                                         | Rep |
                     I PC   Pc                        L |   R I R D    R  U U X | L Xla Ovx N  NI Vif    RwV    Ing  Egr  | V R | PROF H
IfId     Ifname      P Cfg  MbrID As AP Sl Sp Ss Ovec S | P P P S P Sp Sp C M L | 3 Idx Idx L3 L3 Tid    Tid    Lbl  Lbl  | S V | ID   I  smac
===============================================================================================================================================
1a000000 Eth1/1      0 f2   4a    0  15 0  14 28 28   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-1b7  -      c00  0      1 1   d8   0 0
1a001000 Eth1/2      0 43   6     0  16 0  15 2a 2a   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-1b4  -      0    0      0 0   7    0 0
1a002000 Eth1/3      0 f0   48    0  17 0  16 2c 2c   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-38d  -      0    0      0 0   ae   0 0
1a003000 Eth1/4      0 47   a     0  18 0  17 2e 2e   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-dd   -      0    0      0 0   b    0 0
1a004000 Eth1/5      0 49   c     0  11 0  10 20 20   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-15c  -      0    0      0 0   d    0 0
1a005000 Eth1/6      0 4b   e     0  12 0  11 22 22   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-366  -      0    0      0 0   f    0 0
1a006000 Eth1/7      0 4d   10    0  13 0  12 24 24   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-35   -      0    0      0 0   11   0 0
1a007000 Eth1/8      0 4f   12    0  14 0  13 26 26   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-20f  -      0    0      0 0   13   0 0
1a008000 Eth1/9      0 51   14    0  d  0  c  18 18   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-337  -      0    0      0 0   15   0 0
1a009000 Eth1/10     0 53   16    0  e  0  d  1a 1a   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-10d  -      0    0      0 0   17   0 0
1a00a000 Eth1/11     0 55   18    0  f  0  e  1c 1c   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-25e  -      0    0      0 0   19   0 0
1a00b000 Eth1/12     0 57   1a    0  10 0  f  1e 1e   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-64   -      0    0      0 0   1b   0 0
1a00c000 Eth1/13     0 59   1c    0  9  0  8  10 10   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-1e5  -      0    0      0 0   1d   0 0
1a00d000 Eth1/14     0 5b   1e    0  a  0  9  12 12   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-3df  -      0    0      0 0   1f   0 0
1a00e000 Eth1/15     0 5d   20    0  b  0  a  14 14   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-8c   -      0    0      0 0   21   0 0
1a00f000 Eth1/16     0 5f   22    0  c  0  b  16 16   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-2b6  -      0    0      0 0   23   0 0
1a010000 Eth1/17     1 0    64    0  5  0  4  8  8    1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  -      -      c00  0      1 0   0    0 0
1a011000 Eth1/18     0 63   26    0  6  0  5  a  a    1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-c6   -      0    0      0 0   27   0 0
1a012000 Eth1/19     0 157  44    0  7  0  6  c  c    1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-1ce  -      0    0      1 1   131  0 0
1a013000 Eth1/20     0 15b  30    0  8  0  7  e  e    1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-275  -      0    0      1 1   138  0 0
1a014000 Eth1/21     1 0    84    0  1  0  0  0  0    1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  -      -      c00  0      1 0   0    0 0
1a015000 Eth1/22     1 0    94    0  2  0  1  2  2    1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  -      -      c00  0      1 0   0    0 0
1a016000 Eth1/23     0 14a  46    0  3  0  2  4  4    1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-3d1  -      c00  0      0 0   123  0 0
1a017000 Eth1/24     0 150  36    0  4  0  3  6  6    1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-380  -      c00  0      0 0   129  0 0
1a018000 Eth1/25     0 142  4c    0  3d 1  14 28 a8   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-103  -      c00  0      1 1   11b  0 0
1a019000 Eth1/26     0 14e  2c    0  3e 1  15 2a aa   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-2b8  -      c00  0      1 1   127  0 0
1a01a000 Eth1/27     0 75   38    0  3f 1  16 2c ac   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-32c  -      0    0      0 0   39   0 0
1a01b000 Eth1/28     0 77   3a    0  40 1  17 2e ae   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-116  -      0    0      0 0   3b   0 0
1a01c000 Eth1/29     0 79   3c    0  39 1  10 20 a0   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-97   -      0    0      0 0   3d   0 0
1a01d000 Eth1/30     0 7b   3e    0  3a 1  11 22 a2   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-2ad  -      0    0      0 0   3f   0 0
1a01e000 Eth1/31     0 7d   40    0  3b 1  12 24 a4   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-1fe  -      0    0      0 0   41   0 0
1a01f000 Eth1/32     0 7f   42    0  3c 1  13 26 a6   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-3c4  -      0    0      0 0   43   0 0
1a020000 Eth1/33     0 129  4e    0  35 1  c  18 98   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-360  -      c00  0      1 0   fb   0 0
1a021000 Eth1/34     0 104  32    0  36 1  d  1a 9a   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-b4   -      c00  0      1 0   d5   0 0
1a022000 Eth1/35     0 12d  62    0  37 1  e  1c 9c   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-209  -      c00  0      1 0   ff   0 0
1a023000 Eth1/36     0 100  2e    0  38 1  f  1e 9e   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-1dd  -      c00  0      1 0   d6   0 0
1a024000 Eth1/37     0 12b  50    0  31 1  8  10 90   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-15a  -      c00  0      1 0   fd   0 0
1a025000 Eth1/38     0 fc   2a    0  32 1  9  12 92   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-36   -      c00  0      1 1   d7   0 0
1a026000 Eth1/39     1 0    74    0  33 1  a  14 94   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  -      -      c00  0      1 0   0    0 0
1a027000 Eth1/40     0 8f   52    0  34 1  b  16 96   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-3e   -      0    0      0 0   53   0 0
1a028000 Eth1/41     0 91   54    0  2d 1  4  8  88   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-106  -      0    0      0 0   55   0 0
1a029000 Eth1/42     0 93   56    0  2e 1  5  a  8a   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-33c  -      0    0      0 0   57   0 0
1a02a000 Eth1/43     0 192  24    0  2f 1  6  c  8c   1   0 0 0 0 0 0  0  0 0 0   1 66  0   4  0  D-38b  -      c00  0      1 0   1d8  0 0
1a02b000 Eth1/44     0 97   5a    0  30 1  7  e  8e   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-255  -      0    0      0 0   5b   0 0
1a02c000 Eth1/45     0 99   5c    0  29 1  0  0  80   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-3d4  -      0    0      0 0   5d   0 0
1a02d000 Eth1/46     0 110  4     0  2a 1  1  2  82   1   0 0 0 0 0 0  0  0 0 0   1 35  0   1  0  D-164  -      0    0      0 0   d1   0 0
1a02e000 Eth1/47     0 112  8     0  2b 1  2  4  84   1   0 0 0 0 0 0  0  0 0 0   1 37  0   2  0  D-35e  -      0    0      1 0   d3   0 0
1a02f000 Eth1/48     0 18b  28    0  2c 1  3  6  86   1   0 0 0 0 0 0  0  0 0 0   0 0   0   0  0  D-fd   -      0    0      0 0   1cc  0 0
1a035000 Eth1/54     0 6    2     0  4d 1  24 40 c0   1   0 0 0 0 0 0  0  0 0 0   1 4   2   2  2  D-324  -      200  0      0 0   3    0 0

Use Interface 'Ss' for specifying interface src-id
Output Select Lines Supported
0  Pktrw
1 Always used on ACI
5  Sideband
! EX Platform is tahoe
debug platform internal [ns|alp|roc|tah] elam asic 0

! Clear out any previous elam config
trigger reset

! Use the context sensitive help to get the correct codes for in-select
trigger init in-select ?
  10  Outerl4-innerl4-ieth
  13  Outer(l2|l3|l4)-inner(l2|l3|l4)-noieth
  14  Outer(l2(vntag)|l3|l4)-inner(l2|l3|l4)-ieth
  15  Outer(l2|l3|l4)-inner(l2|l3|l4)-ieth
  6   Outerl2-outerl3-outerl4
  7   Innerl2-innerl3-innerl4
  8   Outerl2-innerl2-ieth
  9   Outerl3-innerl3

trigger init in-select 6 out-select 1

Depending on where the capture is required the in-select code and setting the ‘set’ command outer|inner will be different. This is due to the composition of the frame/packet on switch ingress or egress and fabric side interface or host side interface.

! Options to use to filter the trigger
! set [outer|inner] [arp|ipv4|ipv6|l2|l4] [options]
! set [outer|inner] l2 [src_mac|dst_mac|...|]
! set [outer|inner] [ipv4] [src_ip|dst_ip|dscp|...|]
! set srcid {interface-src-id}

! Setting trigger for a SRC IP to DST IP & DST UDP Port 53 (DNS)
set outer ipv4 src_ip 192.168.2.4 dst_ip 192.168.10.1
set outer l4 l4-type 1 dst-port 53

! 'show' to see the configuration
module-1(DBG-elam-insel6)# show
ASIC       : 0
ASIC TYPE  : 6
Num Slices : 2
In-select  : 6
Out-select : 1
A_to_D     : 1
Slice      : All Slices
Port Src Id[Slice:0]: Any Port
Port Src Id[Slice:1]: Any Port
Outer L2
=========
L2 SNAP Valid         :0
L2 CNTag Valid        :0
L2 Qtag Vlan          :0
L2 Qtag Cos           :0
L2 Qtag de            :0
L2 Qtag Valid         :0
L2 Vntag Valid        :0
L2 Vntag source vif   :0
L2 Vntag dest vif     :0
L2 Vntag e bit        :0
L2 Vntag l bit        :0
L2 Vntag p bit        :0
L2 Src MAC            :0x000000000000
L2 Dest MAC           :0x000000000000
Outer L3
=========
L3 Type               :2
L3 Payload len        :0
L3 IPv6 Valid         :0
L3 Version            :0
L3 Header len         :0
L3 DSCP               :0x0
L3 ECN                :0x0
L3 Packet len         :0
L3 More Frags         :0
L3 Frag off           :0
L3 TTL                :0
L3 Next Proto         :0
L3 Checksum           :0x0
L3 Dest IP 0          :0x00000000
L3 Dest IP 1          :0x00000000
L3 Dest IP 2          :0x00000000
L3 Dest IP 3          :0xc0a80a01
L3 Src IP 0           :0x00000000
L3 Src IP 1           :0x00000000
L3 Src IP 2           :0x00000000
L3 Src IP 3           :0xc0a80204
Outer L4
=========
L4 Type               :1
L4 Source Port        :0
L4 Dest Port          :53
L4 Length             :0
L4 Checksum           :0x0
L4 Flags              :0x0
L4 Tn nonce vld       :0
L4 Tn lsb vld         :0
L4 Tn nonce info      :0x0
L4 Tn nonce sclass    :0x0000
L4 Tn nonce dre       :0
L4 Tn nonce dp        :0
L4 Tn nonce sp        :0
L4 Tn nonce e         :0
L4 Tn nonce dl        :0
L4 Tn nonce lb        :0
L4 Tn lsb info        :0x0
L4 Tn lsb metric      :0
L4 Tn lsb tag         :0
L4 Tn lsb m           :0
L4 vnid               :0x0

! Use command 'dec 0x..' to convert hex to decimal, for example the L3 Src IP 3 above is 0xc0a80204, so for each byte working backwards.

dec 0x04
4

dec 0x02
2

dec 0xa8
168

dec 0xc0
192

! So the IP address is 192.168.2.4 which is what we configured in the trigger filter.
! Activate the trigger
start 

! Use status repeatedly until a TRIGGERED response is given which indicates the trigger has seen the required traffic. 
module-1(DBG-elam-insel6)# status
 ELAM STATUS
===========
Asic 0 Slice 0 Status Armed

Asic 0 Slice 1 Status Armed


module-1(DBG-elam-insel6)# status
 ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered

The ELAM capture will only capture the first frame/packet and terminate capture when triggered. The status output will show ‘ARMED’ when active and will show ‘TRIGGERED’ when a frame/packet has matched.

Triggered

Once the trigger has a status of ‘Triggered’, the first frame that matched the trigger filter is saved and the trigger is then deactivated. The report command details the trigger and capture which is very detailed and very long, but using grep we pull out what we want.

! Full Report
report 

! Find the Source Port of the traffic - Where did the DNS traffic come from (client side)
module-1(DBG-elam-insel6)# report | grep src_port
        sug_lurw_vec.ihdr.ieth.hdr.src_port: 0x2F
    ieth.hdr.src_port: 0x2F

The traffic was sourced from port 0x2F, refer back to the earlier ‘ show platform internal hal l2 port gpd ‘ output but use the ‘AP’ (Asic Port) column and lookup 2F, this is port Eth1/43 which is correct as the client is outside of the fabric via a L3Out which is the connected port to the border router.

As of 4.2(1) there is an option for simplified ELAM output. Instead of using ‘report’ use ‘ereport’

This will dump two files to ‘/tmp/logs/’, find the logs you want by using grep as ‘ls | grep elam‘. You will see two files, one prefixed with ‘elam_…’ which is the standard output as above and one prefixed ‘pretty_elam….’ which as its name suggests is a more readable output.

Example 2

If we are looking for a ingress packet coming from a host port using L3 header filters, this will be a native frame without encapsulation (i.e. fabric VXLAN), so we would want to look at the outer header in the ‘set’ command and use in-select code of ‘6’ which provides the outer and inner L3 headers to capture on. Of course there is no L3 inner as this is a native frame from a host, so we use the ‘set outer’ command for the filter.

If we are looking to capture a packet on ingress into the switch from the fabric based on a endpoint MAC & IPv4 filter, the frame will have an outer VXLAN frame used in the fabric overlay and an inner packet which is the original native frame between endpoints. So we need to set the in-select code to ‘4’ and use the ‘set inner’ ‘l2’ & ‘ipv4’ filters to match what we are looking for.

You can probably see there are different ways of achieving the same thing by changing the in-select code and then using the relevant set outer|inner option, just keep in mind what the frame looks like from an layered encapsulation perspective and which layer you want to filter on, then select the relevant in-select code and filter inner|outer option.

References:

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2016/pdf/BRKACI-2102.pdf

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKDCN-3020.pdf

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/213848-nexus-9000-cloud-scale-asic-tahoe-nx-o.html

Simon Birtles

I have been in the IT sector for over 20 years with a primary focus on solutions around networking architecture & design in Data Center and WAN. I have held two CCIEs (#20221) for over 12 years with many retired certifications with Cisco and Microsoft. I have worked in demanding and critical sectors such as finance, insurance, health care and government providing solutions for architecture, design and problem analysis. I have been coding for as long as I can remember in C/C++ and Python (for most things nowadays). Locations that I work without additional paperwork (incl. post Brexit) are the UK and the EU including Germany, Netherlands, Spain and Belgium.