ACI ELAM Packet Capture – Examples
ACI ELAM examples with output.
ASIC Types Reference
Northstar: ns: – Gen1
Alpine: alp – Spines
Rocky: roc – F/FX/FXP
Tahoe: tah – GC/E/EX
Example 1
Find the source interface of a DNS packet from a client to a DNS server. The steps below should be performed on both switches where the ingress or egress is in a vPC over the vPC switch pair or any switch pair where traffic can transit, like an external border router with multiple ECMP connections spread over leaf switches.
Identify the Endpoint Physical Interface
dev-leaf-01-201# show endpoint ip 192.168.10.1
Legend:
s - arp H - vtep V - vpc-attached p - peer-aged
R - peer-attached-rl B - bounce S - static M - span
D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr
L - local E - shared-service
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
23 vlan-1201 0050.5688.712f LV po11
common:VRF_COMMON vlan-1201 192.168.10.1 LV po11
dev-leaf-01-201# show port-channel summary interface po11
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
F - Configuration failed
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
11 Po11(SU) Eth LACP Eth1/21(P)
Line Card Shell (VSH)
leaf-01-218# vsh_lc
module-1# show platform internal hal l2 port gpd
Legend:
-------
IfId: Interface Id IfName: Interface Name
I P: Is PC Mbr IfId: Interface Id
Uc PC Cfg: UcPcCfg Idx Uc PC MbrId: Uc Pc Mbr Id
As: Asic AP: Asic Port
Sl: Slice Sp: Slice Port
Ss: Slice SrcId Ovec: Ovector (slice | srcid)
L S: Local Slot Reprogram:
L3: Is L3
P: PifTable Xla Idx: Xlate Idx
RP: Rw PifTable Ovx Idx: OXlate Idx
IP: If Profile Table N L3: Num. of L3 Ifs
RS: Rw SrcId Table NI L3: Num. of Infra L3 Ifs
DP: DPort Table Vif Tid: Vif Tid
SP: SrcPortState Table RwV Tid: RwVif Tid
RSP: RwSrcPortstate Table Ing Lbl: Ingress Acl Label
UC: UCPcCfg Egr Lbl: Egress Acl Label
UM: UCPcMbr Reprogram:
PROF ID: Lport Profile Id
VS: VifStateTable HI: LportProfile Hw Install
RV: Rw VifTable
Num. of Sandboxes: 1
Sandbox_ID: 0, BMP: 0x0
Port Count: 49
==============================================================================================================================================
Uc Uc | Reprogram | | Rep |
I PC Pc L | R I R D R U U X | L Xla Ovx N NI Vif RwV Ing Egr | V R | PROF H
IfId Ifname P Cfg MbrID As AP Sl Sp Ss Ovec S | P P P S P Sp Sp C M L | 3 Idx Idx L3 L3 Tid Tid Lbl Lbl | S V | ID I smac
===============================================================================================================================================
1a000000 Eth1/1 0 f2 4a 0 15 0 14 28 28 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-1b7 - c00 0 1 1 d8 0 0
1a001000 Eth1/2 0 43 6 0 16 0 15 2a 2a 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-1b4 - 0 0 0 0 7 0 0
1a002000 Eth1/3 0 f0 48 0 17 0 16 2c 2c 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-38d - 0 0 0 0 ae 0 0
1a003000 Eth1/4 0 47 a 0 18 0 17 2e 2e 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-dd - 0 0 0 0 b 0 0
1a004000 Eth1/5 0 49 c 0 11 0 10 20 20 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-15c - 0 0 0 0 d 0 0
1a005000 Eth1/6 0 4b e 0 12 0 11 22 22 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-366 - 0 0 0 0 f 0 0
1a006000 Eth1/7 0 4d 10 0 13 0 12 24 24 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-35 - 0 0 0 0 11 0 0
1a007000 Eth1/8 0 4f 12 0 14 0 13 26 26 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-20f - 0 0 0 0 13 0 0
1a008000 Eth1/9 0 51 14 0 d 0 c 18 18 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-337 - 0 0 0 0 15 0 0
1a009000 Eth1/10 0 53 16 0 e 0 d 1a 1a 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-10d - 0 0 0 0 17 0 0
1a00a000 Eth1/11 0 55 18 0 f 0 e 1c 1c 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-25e - 0 0 0 0 19 0 0
1a00b000 Eth1/12 0 57 1a 0 10 0 f 1e 1e 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-64 - 0 0 0 0 1b 0 0
1a00c000 Eth1/13 0 59 1c 0 9 0 8 10 10 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-1e5 - 0 0 0 0 1d 0 0
1a00d000 Eth1/14 0 5b 1e 0 a 0 9 12 12 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-3df - 0 0 0 0 1f 0 0
1a00e000 Eth1/15 0 5d 20 0 b 0 a 14 14 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-8c - 0 0 0 0 21 0 0
1a00f000 Eth1/16 0 5f 22 0 c 0 b 16 16 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-2b6 - 0 0 0 0 23 0 0
1a010000 Eth1/17 1 0 64 0 5 0 4 8 8 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 - - c00 0 1 0 0 0 0
1a011000 Eth1/18 0 63 26 0 6 0 5 a a 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-c6 - 0 0 0 0 27 0 0
1a012000 Eth1/19 0 157 44 0 7 0 6 c c 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-1ce - 0 0 1 1 131 0 0
1a013000 Eth1/20 0 15b 30 0 8 0 7 e e 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-275 - 0 0 1 1 138 0 0
1a014000 Eth1/21 1 0 84 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 - - c00 0 1 0 0 0 0
1a015000 Eth1/22 1 0 94 0 2 0 1 2 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 - - c00 0 1 0 0 0 0
1a016000 Eth1/23 0 14a 46 0 3 0 2 4 4 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-3d1 - c00 0 0 0 123 0 0
1a017000 Eth1/24 0 150 36 0 4 0 3 6 6 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-380 - c00 0 0 0 129 0 0
1a018000 Eth1/25 0 142 4c 0 3d 1 14 28 a8 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-103 - c00 0 1 1 11b 0 0
1a019000 Eth1/26 0 14e 2c 0 3e 1 15 2a aa 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-2b8 - c00 0 1 1 127 0 0
1a01a000 Eth1/27 0 75 38 0 3f 1 16 2c ac 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-32c - 0 0 0 0 39 0 0
1a01b000 Eth1/28 0 77 3a 0 40 1 17 2e ae 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-116 - 0 0 0 0 3b 0 0
1a01c000 Eth1/29 0 79 3c 0 39 1 10 20 a0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-97 - 0 0 0 0 3d 0 0
1a01d000 Eth1/30 0 7b 3e 0 3a 1 11 22 a2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-2ad - 0 0 0 0 3f 0 0
1a01e000 Eth1/31 0 7d 40 0 3b 1 12 24 a4 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-1fe - 0 0 0 0 41 0 0
1a01f000 Eth1/32 0 7f 42 0 3c 1 13 26 a6 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-3c4 - 0 0 0 0 43 0 0
1a020000 Eth1/33 0 129 4e 0 35 1 c 18 98 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-360 - c00 0 1 0 fb 0 0
1a021000 Eth1/34 0 104 32 0 36 1 d 1a 9a 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-b4 - c00 0 1 0 d5 0 0
1a022000 Eth1/35 0 12d 62 0 37 1 e 1c 9c 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-209 - c00 0 1 0 ff 0 0
1a023000 Eth1/36 0 100 2e 0 38 1 f 1e 9e 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-1dd - c00 0 1 0 d6 0 0
1a024000 Eth1/37 0 12b 50 0 31 1 8 10 90 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-15a - c00 0 1 0 fd 0 0
1a025000 Eth1/38 0 fc 2a 0 32 1 9 12 92 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-36 - c00 0 1 1 d7 0 0
1a026000 Eth1/39 1 0 74 0 33 1 a 14 94 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 - - c00 0 1 0 0 0 0
1a027000 Eth1/40 0 8f 52 0 34 1 b 16 96 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-3e - 0 0 0 0 53 0 0
1a028000 Eth1/41 0 91 54 0 2d 1 4 8 88 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-106 - 0 0 0 0 55 0 0
1a029000 Eth1/42 0 93 56 0 2e 1 5 a 8a 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-33c - 0 0 0 0 57 0 0
1a02a000 Eth1/43 0 192 24 0 2f 1 6 c 8c 1 0 0 0 0 0 0 0 0 0 0 1 66 0 4 0 D-38b - c00 0 1 0 1d8 0 0
1a02b000 Eth1/44 0 97 5a 0 30 1 7 e 8e 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-255 - 0 0 0 0 5b 0 0
1a02c000 Eth1/45 0 99 5c 0 29 1 0 0 80 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-3d4 - 0 0 0 0 5d 0 0
1a02d000 Eth1/46 0 110 4 0 2a 1 1 2 82 1 0 0 0 0 0 0 0 0 0 0 1 35 0 1 0 D-164 - 0 0 0 0 d1 0 0
1a02e000 Eth1/47 0 112 8 0 2b 1 2 4 84 1 0 0 0 0 0 0 0 0 0 0 1 37 0 2 0 D-35e - 0 0 1 0 d3 0 0
1a02f000 Eth1/48 0 18b 28 0 2c 1 3 6 86 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D-fd - 0 0 0 0 1cc 0 0
1a035000 Eth1/54 0 6 2 0 4d 1 24 40 c0 1 0 0 0 0 0 0 0 0 0 0 1 4 2 2 2 D-324 - 200 0 0 0 3 0 0
Use Interface 'Ss' for specifying interface src-id
Output Select Lines Supported
0 Pktrw
1 Always used on ACI
5 Sideband! EX Platform is tahoe debug platform internal [ns|alp|roc|tah] elam asic 0 ! Clear out any previous elam config trigger reset ! Use the context sensitive help to get the correct codes for in-select trigger init in-select ? 10 Outerl4-innerl4-ieth 13 Outer(l2|l3|l4)-inner(l2|l3|l4)-noieth 14 Outer(l2(vntag)|l3|l4)-inner(l2|l3|l4)-ieth 15 Outer(l2|l3|l4)-inner(l2|l3|l4)-ieth 6 Outerl2-outerl3-outerl4 7 Innerl2-innerl3-innerl4 8 Outerl2-innerl2-ieth 9 Outerl3-innerl3 trigger init in-select 6 out-select 1
Depending on where the capture is required the in-select code and setting the ‘set’ command outer|inner will be different. This is due to the composition of the frame/packet on switch ingress or egress and fabric side interface or host side interface.
! Options to use to filter the trigger
! set [outer|inner] [arp|ipv4|ipv6|l2|l4] [options]
! set [outer|inner] l2 [src_mac|dst_mac|...|]
! set [outer|inner] [ipv4] [src_ip|dst_ip|dscp|...|]
! set srcid {interface-src-id}
! Setting trigger for a SRC IP to DST IP & DST UDP Port 53 (DNS)
set outer ipv4 src_ip 192.168.2.4 dst_ip 192.168.10.1
set outer l4 l4-type 1 dst-port 53
! 'show' to see the configuration
module-1(DBG-elam-insel6)# show
ASIC : 0
ASIC TYPE : 6
Num Slices : 2
In-select : 6
Out-select : 1
A_to_D : 1
Slice : All Slices
Port Src Id[Slice:0]: Any Port
Port Src Id[Slice:1]: Any Port
Outer L2
=========
L2 SNAP Valid :0
L2 CNTag Valid :0
L2 Qtag Vlan :0
L2 Qtag Cos :0
L2 Qtag de :0
L2 Qtag Valid :0
L2 Vntag Valid :0
L2 Vntag source vif :0
L2 Vntag dest vif :0
L2 Vntag e bit :0
L2 Vntag l bit :0
L2 Vntag p bit :0
L2 Src MAC :0x000000000000
L2 Dest MAC :0x000000000000
Outer L3
=========
L3 Type :2
L3 Payload len :0
L3 IPv6 Valid :0
L3 Version :0
L3 Header len :0
L3 DSCP :0x0
L3 ECN :0x0
L3 Packet len :0
L3 More Frags :0
L3 Frag off :0
L3 TTL :0
L3 Next Proto :0
L3 Checksum :0x0
L3 Dest IP 0 :0x00000000
L3 Dest IP 1 :0x00000000
L3 Dest IP 2 :0x00000000
L3 Dest IP 3 :0xc0a80a01
L3 Src IP 0 :0x00000000
L3 Src IP 1 :0x00000000
L3 Src IP 2 :0x00000000
L3 Src IP 3 :0xc0a80204
Outer L4
=========
L4 Type :1
L4 Source Port :0
L4 Dest Port :53
L4 Length :0
L4 Checksum :0x0
L4 Flags :0x0
L4 Tn nonce vld :0
L4 Tn lsb vld :0
L4 Tn nonce info :0x0
L4 Tn nonce sclass :0x0000
L4 Tn nonce dre :0
L4 Tn nonce dp :0
L4 Tn nonce sp :0
L4 Tn nonce e :0
L4 Tn nonce dl :0
L4 Tn nonce lb :0
L4 Tn lsb info :0x0
L4 Tn lsb metric :0
L4 Tn lsb tag :0
L4 Tn lsb m :0
L4 vnid :0x0
! Use command 'dec 0x..' to convert hex to decimal, for example the L3 Src IP 3 above is 0xc0a80204, so for each byte working backwards.
dec 0x04
4
dec 0x02
2
dec 0xa8
168
dec 0xc0
192
! So the IP address is 192.168.2.4 which is what we configured in the trigger filter.! Activate the trigger
start
! Use status repeatedly until a TRIGGERED response is given which indicates the trigger has seen the required traffic.
module-1(DBG-elam-insel6)# status
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Armed
module-1(DBG-elam-insel6)# status
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status TriggeredThe ELAM capture will only capture the first frame/packet and terminate capture when triggered. The status output will show ‘ARMED’ when active and will show ‘TRIGGERED’ when a frame/packet has matched.
Triggered
Once the trigger has a status of ‘Triggered’, the first frame that matched the trigger filter is saved and the trigger is then deactivated. The report command details the trigger and capture which is very detailed and very long, but using grep we pull out what we want.
! Full Report
report
! Find the Source Port of the traffic - Where did the DNS traffic come from (client side)
module-1(DBG-elam-insel6)# report | grep src_port
sug_lurw_vec.ihdr.ieth.hdr.src_port: 0x2F
ieth.hdr.src_port: 0x2F
The traffic was sourced from port 0x2F, refer back to the earlier ‘ show platform internal hal l2 port gpd ‘ output but use the ‘AP’ (Asic Port) column and lookup 2F, this is port Eth1/43 which is correct as the client is outside of the fabric via a L3Out which is the connected port to the border router.
As of 4.2(1) there is an option for simplified ELAM output. Instead of using ‘report’ use ‘ereport’
This will dump two files to ‘/tmp/logs/’, find the logs you want by using grep as ‘ls | grep elam‘. You will see two files, one prefixed with ‘elam_…’ which is the standard output as above and one prefixed ‘pretty_elam….’ which as its name suggests is a more readable output.
Example 2
If we are looking for a ingress packet coming from a host port using L3 header filters, this will be a native frame without encapsulation (i.e. fabric VXLAN), so we would want to look at the outer header in the ‘set’ command and use in-select code of ‘6’ which provides the outer and inner L3 headers to capture on. Of course there is no L3 inner as this is a native frame from a host, so we use the ‘set outer’ command for the filter.
If we are looking to capture a packet on ingress into the switch from the fabric based on a endpoint MAC & IPv4 filter, the frame will have an outer VXLAN frame used in the fabric overlay and an inner packet which is the original native frame between endpoints. So we need to set the in-select code to ‘4’ and use the ‘set inner’ ‘l2’ & ‘ipv4’ filters to match what we are looking for.
You can probably see there are different ways of achieving the same thing by changing the in-select code and then using the relevant set outer|inner option, just keep in mind what the frame looks like from an layered encapsulation perspective and which layer you want to filter on, then select the relevant in-select code and filter inner|outer option.
References:
https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2016/pdf/BRKACI-2102.pdf
https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKDCN-3020.pdf
