ACI Transit Overview
As we continue to deploy fabrics we always joke about these routing checkboxes shooting us in the foot. We play with different scenarios in the lab to ensure we understand how these pesky boxes work and what other options we have for future deployments. The scenario here was to use different OSPF areas connected to the same border leaf using ACI as the transit. This scenario brings up some certain challenges and hopefully my testing will help others understand it a little better as well.
Design:
We have two external routers coming into a border leaf on ACI, two L3Outs (required because of multiple areas), one is Area 0 (backbone) and one is Area 1. Here is the breakdown of routes on each router:
External Router 1 (Area 0):
- Loopback0: 2.2.2.2/32
- Loopback1: 4.4.4.4/32
- Transits: 192.168.0.0/29
External Router 2 (Area 1):
- Loopback0: 3.3.3.3/32
- Loopback1: 5.5.5.5/32
- Transits: 172.16.0.0/29
Using ACI as a transit we want routes from Area 0 to be imported into Area 1 and vice versa across the two L3Outs. We will skip the build of the L3Out portion but I want to focus on those pesky checkboxes. Whenever I build an L3Out my network EPG usually looks something like this:
By default, “External Subnets for the External EPG” is checked (this checkbox simply enforces policy on this L3out and contracts are applied to the specific subnet) and I am using 0.0.0.0/0 network as a catch all. Moving along with the defaults, I show full adjacency:
As well as a full routing table within ACI, receiving the networks above in the fabric as expected:
Note: Anything received from area 0 is shown as backbone and everything from area 1 is 0.0.0.1.
Now if look at the routing table for External Router 1 we see no routes across the fabric being received from External Router 2.
Let me check my OSPF Neighbors:
So, we have no OSPF routes but we have a neighbor relationship. Let’s go check External Router 2:
So, we are showing some OSPF routes but they are only the loopbacks of the ACI Fabric (Area 0), not what we are necessarily looking for. ACI blocks transit routes between different L3Outs unless permitted by policy via an OSPF area filter-list (to verify ssh to the border leaf and run “show route-map”). Let’s go look at the Network EPG checkboxes again and see if we can get routing to occur between OSPF Areas across the fabric.
As we showed earlier we are using a catch all 0.0.0.0/0 with “External Subnets for the External EPG”. If we select the “!” on the upper right portion of the screen we will receive the properties of this screen:
Based on what this screen states, “Export Route Control Subnet” – controls the export route direction and “Import Route Control Subnet” – controls the import direction. This sounds like what we need to get routing to traverse the fabric. Let’s go ahead and select them for Area 0 but before we can select import there is another configuration we need to apply before we can get import to not be grayed out. If we go back to the top of the L3Out in the navigation pane we need to select the “Route Control Enforcement:” import checkbox:
Now if we go back to the Network EPG we should have both options available to us:
Now let’s see if we have any changes within our routing table:
We still have no change in the table. Remember that we are using a catch all 0.0.0.0/0? This would require us to also select the aggregate export and import features on the subnet/network epg we have created for Area 0 and Area 1:
Time to verify:
This looks great. Now we will verify External Router 2:
If we want to take it a step further we can do network specific routes instead of a catch all:
Router 2:
Verify Router 2 is receiving 2.2.2.2/32 from Router 1:
Now we can send 3.3.3.3/32 from Router 2 into Router 1:
Router 1:
As I stated earlier these checkboxes are updating route-maps and prefix-lists within ACI. Prior to us selecting the import/export feature our route-maps had a deny all so no routes would traverse areas. Upon selecting these checkboxes, we can see the change:
Instead of giving you route-map blah I will try and breakdown the map for you just focusing on export route-map. Prior to import/export our route-map looked like this:
route-map exp-ctx-2392064-deny-external-tag, deny, sequence 1
Match clauses:
tag: 4294967295
Set clauses:
route-map exp-ctx-2392064-deny-external-tag, deny, sequence 9998
Match clauses:
ospf-area: backbone
Set clauses:
route-map exp-ctx-2392064-deny-external-tag, deny, sequence 9999
Match clauses:
ospf-area: 0.0.0.1
Set clauses:
route-map exp-ctx-2392064-deny-external-tag, permit, sequence 10000
Match clauses:
Set clauses:
route-map exp-ctx-2392064-set-external-tag, permit, sequence 2
Match clauses:
Set clauses:
tag 4294967295
route-map imp-ctx-bgp-st-interleak-2392064, deny, sequence 1
Match clauses:
tag: 4294967295
Set clauses:
route-map imp-ctx-bgp-st-interleak-2392064, permit, sequence 10000
Match clauses:
Set clauses:
You can see that we had “deny” for backbone and area 0.0.0.1 preventing us from using the fabric as a transit. After we selected import/export features our route-map is updated as such (just focusing on the export route-map):
route-map exp-ctx-2392064-deny-external-tag, deny, sequence 1
Match clauses:
tag: 4294967295
Set clauses:
route-map exp-ctx-2392064-deny-external-tag, permit, sequence 9801
Match clauses:
ip address prefix-lists: IPv4-ospf-rt2392064–0-dst-rtpfx
ipv6 address prefix-lists: IPv6-deny-all
ospf-area: backbone
Set clauses:
route-map exp-ctx-2392064-deny-external-tag, permit, sequence 9802
Match clauses:
ip address prefix-lists: IPv4-ospf-rt2392064–1-dst-rtpfx
ipv6 address prefix-lists: IPv6-deny-all
ospf-area: 0.0.0.1
Set clauses:
route-map exp-ctx-2392064-deny-external-tag, deny, sequence 9998
Match clauses:
ospf-area: backbone
Set clauses:
route-map exp-ctx-2392064-deny-external-tag, deny, sequence 9999
Match clauses:
ospf-area: 0.0.0.1
Set clauses:
route-map exp-ctx-2392064-deny-external-tag, permit, sequence 10000
Match clauses:
Set clauses:
route-map exp-ctx-2392064-set-external-tag, permit, sequence 2
Match clauses:
Set clauses:
tag 4294967295
route-map imp-ctx-bgp-st-interleak-2392064, deny, sequence 1
Match clauses:
tag: 4294967295
Set clauses:
route-map imp-ctx-bgp-st-interleak-2392064, permit, sequence 10000
Match clauses:
Set clauses:
Now our route-map has been updated with prefix-lists to allow our traffic across areas, we will look at the prefix-list itself:
Leaf-103# show ip prefix-list IPv4-ospf-rt2392064–0-dst-rtpfx
ip prefix-list IPv4-ospf-rt2392064–0-dst-rtpfx: 1 entries
seq 1 permit 0.0.0.0/0 le 32
The 0.0.0.0/0 catch all has been added and our routes can traverse the fabric. I suggest you also peak at the import route-maps and see what is happening under the hood there as well.