Calico + Macvlan Multi-CNI Data Forwarding Process

Background

CNIs are important components of a Kubernetes cluster. Typically, one CNI (e.g. Calico) is deployed and is responsible for the connectivity of the cluster network. In some cases, customers may use multiple types of CNIs in the cluster based on performance, security, etc., such as Macvlan CNIs of the Underlay type, and then there may be multiple Pods of different CNI types in a cluster, and different types of Pods are suitable for different scenarios:

Pod with a single Calico NIC: System components such as CoreDNS do not have the need for a fixed IP, nor do they need to communicate with north-south traffic, but only need to communicate with east-west traffic in the cluster.
Pods with a single Macvlan card: For applications with special requirements for performance and security, or for traditional up-cloud applications that require direct north-south traffic with the Pod IP.
Multi-Card Pod with Calico and Macvlan cards: A combination of both. Both need to access cluster north-south traffic with a fixed Pod IP and cluster east-west traffic (e.g., with a Calico Pod or Service).

In addition, when multiple CNI pods exist in a cluster, there are actually two different data forwarding schemes in the cluster: Underlay and Overlay. this can lead to a number of other issues:

Pods using the Underlay network cannot access the cluster's north-south traffic.
Pods using Underlay networks cannot communicate directly with Pods using Overlay networks in the cluster: Due to inconsistent forwarding paths, Overlay networks often need to go through nodes for secondary forwarding, whereas Underlay networks are generally forwarded directly through the underlying gateway. Therefore, when they access each other, packet loss may occur because the underlying switch does not synchronize the routes of the cluster subnet.
Using two network modes for a cluster may increase the complexity of use and operation, such as IP address management.

Spiderpool is a complete Underlay network solution that solves the interoperability problem when there are multiple CNIs in a cluster and reduces the IP address operation and maintenance burden. The following section describes the data forwarding process between them.

Quick start

Calico + Macvlan multi-network card quickstart can be found in get-stared-calico
For a single Macvlan card see get-started-macvlan.
Underlay CNI Access Service See underlay_cni_service

Serval Data forwarding Flow

Below are several typical communication scenarios:

Macvlan Accessing Macvlan Pods
Macvlan Pod Access to Macvlan Pod's Service
Macvlan Accessing Calico Pods
Macvlan Pod Accessing Calico Pod's Service
Calico+Macvlan Multi-NIC Pods Accessing Calico and Macvlan Pods
Calico+Macvlan Multi-NIC Pods Accessing Service
Accessing NodePort of Calico-Macvlan Multi-NIC Pods

Data Forwarding Process Between Macvlan Pods

Macvlan Pod Communication Flow

Macvlan Access to Pods in the Same Subnet

As shown in Figure 1 Label 2, Macvlan Pod1(172.16.1.3) directly accesses Macvlan Pod2(172.16.1.2) through the Macvlan Bridge mode mechanism. Whether on the same node or different nodes, gateway forwarding is not required.

Macvlan Access to Pods in Different Subnets

As shown in Figure 1 Label 1, Macvlan Pod1(172.16.1.2) accesses cross-segment and cross-node Macvlan Pod2(172.16.2.100) or cross-segment but same-node Macvlan Pod3(172.17.1.100). Due to different network segments, gateway forwarding is required.

Macvlan Pod Access to Local Node

As shown in Figure 1 Label 3: Due to the Macvlan bridge mechanism, Macvlan sub-interfaces cannot communicate directly with the Master interface. Therefore, we created a veth network card in the Macvlan Pod (e.g., 172.17.1.100) and set up routes in both Pod and node to solve the communication issue. The node(172.17.1.1) accesses Macvlan Pod(172.17.1.100) according to the route in node table500, forwarding from vethxxx to the Pod's network namespace.

    # pod network namespace
    ~# ip r 
    172.17.1.1 dev veth0 
    # host network namespace
    ~# ip r show table 500
    172.17.1.100 dev vethxxx

Macvlan Pod Access to Macvlan Pod's Service

Accessing ClusterIP (as shown by the black line in Figure 2):

Macvlan Pod(172.17.1.100) accessing ClusterIP(10.233.0.100) matches Pod internal routing and forwards packets through veth0 network card to the node
```
# pod network namespace
~# ip r 
10.233.0.0/18 dev veth0
```
Through the Service DNAT rules set by node Kube-proxy, ClusterIP(10.233.0.100) is resolved to target Macvlan Pod(172.17.1.200), and packets are forwarded to 172.17.1.200 through inter-node network. At this point, the source address has been SNAT'd to node IP: 172.17.1.1.
Upon reaching Node2(172.17.1.2), packets are forwarded to target Pod(172.17.1.200) through vethxx via the host's table500 routing table.
When the target Pod(172.17.1.200) sends response packets, its destination address is Node1's address: 172.17.1.1, so response packets are sent from veth0 through the node network back to Node1. Through Node1's Kube-proxy iptables rules, the source address is restored to ClusterIP(10.233.0.100) and destination address to Macvlan Pod(172.17.1.100)
Through the host's table500 routing table, response packets are forwarded via vethxx network card to source Pod(172.17.1.100), completing the access process.

Accessing NodePort Service:

as shown by the red line in Figure 2:

Focus on the scenario where NodePort Service is configured with ExternalTrafficPolicy=Local.

An external client: 1.1.1.1 accesses NodePort: 172.17.1.1:32456. Packets first reach node Node1(172.17.1.1) through external routing
Through the host's Kube-proxy configured iptables DNAT rules, the destination address is rewritten to Macvlan Pod(172.17.1.100), and through the host's table500 routing table, packets are forwarded via vethxx to the target Pod.
When the Pod sends response packets, seeing destination address 1.1.1.1, it matches the Pod's default route and forwards from eth0, causing communication failure. Spiderpool solves this by setting the following iptables rules and policy routing in the Pod's network namespace, ensuring NodePort traffic received by veth0 is still forwarded to the node via veth0.
```
# pod network namespace
iptables -i veth0 --set-xmark 0x1 ...
~# ip rule 
from all fwmark 0x1 lookup 100
~# ip r show table 100
default dev veth0
```
Through the node's network protocol stack, the packet's source address is changed to the node's IP(172.17.1.1), then through external cluster routing, packets are sent to the client(1.1.1.1), completing the access process.

Macvlan Access to Calico Pod

macvlan-calico-pod

There are three main scenarios:

Macvlan Pod accessing Calico Pod on the same node (as shown in Figure 3 Path 1):

Macvlan Pod(172.16.100.2) accessing Calico Pod(10.233.100.3/18), packets match Pod routing(10.233.64.0/18 via veth0), forwarding packets from veth0 network card to the node.
Packets match on the node: Calico's virtual route(10.233.100.3 dev calixxx) forwards packets through Calico virtual network card to Calico Pod(10.233.100.3).
Response packets are sent to the node through Calico virtual network card(calixxx), and since the destination address is Macvlan Pod(172.16.100.2), matching route(172.16.100.2 dev vethxxx table 500) forwards to Macvlan Pod, completing the access.

Macvlan Pod accessing Calico Pod across nodes (nodes in same subnet, as shown in Figure 3: 2->2.1):

Macvlan Pod(172.16.100.2) accessing Calico Pod(10.233.200.2/18), packets match Pod routing(10.233.64.0/18 via veth0), forwarding packets from veth0 network card to the node.
Through inter-node Calico routing, packets are forwarded to peer node(172.16.1.3). Since the destination address is 10.233.200.2, matching node's Calico route(10.233.100.2 dev calixxx) forwards packets through Calico virtual network card to Calico Pod(10.233.100.2).
Since the original source address is Macvlan Pod(172.16.100.2), Calico Pod will forward response packets first to its local node(172.16.1.3), then directly forward response packets to Macvlan Pod(172.16.100.2) without going through node forwarding, causing inconsistent packet forwarding paths. This may cause the kernel to mark the packet's conntrack state as invalid, leading to packet drop by a kube-proxy iptables rule:
```
  ~# iptables-save  -t filter | grep '--ctstate INVALID -j DROP'
  iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP
```

This rule was originally implemented to solve the issue raised in #Issue 74839, where some TCP packets exceeding window limits were marked as invalid conntrack state by the kernel, causing the entire TCP connection to reset. The k8s community addressed this by implementing this rule, but it may affect scenarios with inconsistent packet paths. Related community issues include: #Issue 117924, #Issue 94861, #Issue 177.

We helped resolve this issue through community collaboration, ultimately fixed in only drop invalid cstate packets if non liberal for kubernetes version v1.29. We need to ensure each node's sysctl parameter is set: sysctl -w net.netfilter.nf_conntrack_tcp_be_liberal=1, and restart Kube-proxy. This prevents kube-proxy from implementing the drop rule, avoiding impact on communication between single Macvlan pod and single Calico pod.

After execution, check if the drop rule still exists on the node. No output indicates normal operation. Otherwise, check if sysctl is correctly set and if kube-proxy has been restarted.

    ~# iptables-save -t filter | grep '--ctstate INVALID -j DROP'

Note: Must ensure k8s version is greater than v1.29. If your k8s version is below v1.29, this issue cannot be avoided.

Macvlan Pod Access to the Calico IP of a Calico+Macvlan Multi-NIC Pod

When a single Macvlan Pod accesses the Calico IP of a Calico+Macvlan multi-NIC Pod, the result depends on the return path selected inside the multi-NIC Pod.

For example, assume the Macvlan Pod is 172.100.0.201, and the multi-NIC Pod has:

Calico interface eth0: 10.233.106.188
Macvlan interface net1: 172.100.0.202/16
Main routing table keeps the direct Macvlan subnet route: 172.100.0.0/16 dev net1

The request packet reaches the multi-NIC Pod through the Calico path:

172.100.0.201 -> 10.233.106.188

However, the response packet matches the direct route in the Pod main table and is sent from net1:

10.233.106.188 -> 172.100.0.201 dev net1

The Macvlan Pod can receive the SYN-ACK and continue sending ACK and HTTP data. The packet loss happens later on the node that hosts the multi-NIC Pod: the ACK/HTTP packets return through the Calico tunnel and are dropped by Calico's workload ingress chain before they are delivered to the Pod. The connection on that node may stay in SYN_SENT [UNREPLIED], and the packet may match a rule similar to:

cali-tw-<workload> -m conntrack --ctstate INVALID -j DROP

This is different from the kube-proxy KUBE-FORWARD INVALID drop issue described above. Setting net.netfilter.nf_conntrack_tcp_be_liberal=1 and restarting kube-proxy removes the kube-proxy rule, but it does not remove Calico's workload endpoint INVALID check.

There are several ways to avoid this case:

Prefer accessing the multi-NIC Pod by its Macvlan IP when the client is in the same Macvlan subnet.
Prefer accessing the multi-NIC Pod through Service. ClusterIP access is handled through kube-proxy and was verified to work in this topology.
If this asymmetric direct Pod-IP path must be supported, set Calico Felix configuration disableConntrackInvalidCheck: true. This removes the Calico workload endpoint ctstate INVALID drop rule, but it is a cluster-level Calico behavior change and should be evaluated carefully before enabling it in production.

Example:

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  name: default
spec:
  disableConntrackInvalidCheck: true

Macvlan Pod Accessing Calico Pod's Service

macvlan-calico-service

Macvlan Pod(172.16.100.2) inserts a route to intercept service traffic, making Pod access to ClusterIP(10.233.0.100) forward packets through veth0 network card to node Node1(172.16.1.2).
```
～# ip r
10.233.0.0/18 via 10.7.168.71 dev eth0 src 10.233.100.2
```
After packets are forwarded to node Node1(172.16.1.2), the host network stack's kube-proxy converts clusterip to Calico Pod's IP: 10.233.200.2. Then through Calico's tunnel routing forwards to node Node2(172.16.1.3). Note that when request packets are sent from node3, their source address is SNAT'd to Node1's IP(172.16.1.2). This ensures response packets can return via the same path, avoiding inconsistent routing paths when Macvlan Pod accesses Calico Pod. After request packets are forwarded to host Node2, they are forwarded to Calico Pod(10.233.200.2) through the node's calixxx virtual network card.
Calico Pod(10.233.200.2) forwards response packets to node Node2. At this point, the response packet's destination address is node Node1(172.16.1.2), so it's forwarded to Node1 through node routing. Then through Kube-proxy, the destination address is rewritten to Macvlan Pod's IP: 172.16.100.2, and source address to ClusterIP(10.233.0.100). Then matching the host's route table500(172.16.100.2 dev vethxxx table 500), packets are sent to Macvlan Pod, completing the access.

Calico+Macvlan Multi-NIC Pod Accessing Calico and Macvlan Pods

calico-macvlan-pod

As shown in Figure 4: Node1 runs a Calico-Macvlan Pod (with both calico and macvlan network cards), while Node2 runs a Macvlan Pod (with only macvlan network card) and a Calico Pod (with only calico network card).

Calico + Macvlan Multi-NIC Pod Accessing Calico Pod:

as shown by red line in Figure 4:

Calico-Macvlan Pod has two network cards, Spiderpool coordinates routing between multiple cards, enabling Pod to access both Overlay and Underlay networks.

~# # all routing-table of eth0(calico)
~# ip route show table main
172.16.1.2 dev eth0
10.233.64.0/18 dev eth0
172.16.0.0/16 dev net1
default via 169.254.1.1 dev eth0
~# # all routing-table of net1(macvlan)
~# ip route show table 101
default via 172.16.0.1 dev net1
172.16.0.0/16 dev net1

When Calico-Macvlan Pod accesses Calico Pod(10.233.100.4), it matches Pod routing(10.233.64.0/18 dev eth0) and sends packets through eth0 to node Node1(172.16.1.1), with source address being Calico network card's IP(10.233.100.3). Since the destination address is 10.233.100.4, packets are forwarded through inter-node Calico network to node Node2(172.16.1.2), and finally to Calico Pod.
When Calico Pod sends response packets, the destination address is Calico-Macvlan Pod's calico network card IP(10.233.100.3). This process is similar to communication between Calico Pods, forwarding response packets through Calico node routing to Node1, then through Calico's virtual routing to Calico-Macvlan Pod, completing the access.

Calico + Macvlan Multi-NIC Pod Accessing Macvlan Pod

as shown by black line in Figure 4:

Calico-Macvlan Pod has two network cards, Spiderpool coordinates routing between multiple cards, enabling Pod to access both Overlay and Underlay networks simultaneously.

~# # all routing-table of eth0(calico)
~# ip route show table main
172.16.1.2 dev eth0
10.233.64.0/18 dev eth0
default via 169.254.1.1 dev eth0
172.16.0.0/16 dev net1
～# ip route show table 101
~# # all routing-table of net1(macvlan)
~# ip route show table 101
default via 172.16.0.1 dev net1
172.16.0.0/16 dev net1

When Calico-Macvlan Pod accesses Macvlan Pod(172.16.100.3), packets match routing(172.16.0.0/16 dev net1) and are sent through Pod's macvlan network card net1.
Packets are forwarded directly through Underlay network to Node2's Macvlan Pod(172.16.100.3)
When Macvlan Pod(172.16.100.3) sends response packets, since the destination address is 172.16.100.2, packets are sent back to Calico-Macvlan Pod through Underlay network, completing the access.

Calico+Macvlan Multi-NIC Pod Accessing Service

calico-macvlan-service

Node1 runs a Macvlan Pod (with only one Macvlan network card: 172.16.100.3) and a Calico Pod (with only one Calico network card: 10.233.100.4).

Calico-Macvlan Multi-NIC Pod Accessing ClusterIP (endpoint using Calico network)

As shown by the blue arrow in Figure 5:

According to the routing in the Pod, traffic accessing ClusterIP is forwarded through eth0 to node Node2. Then, the kube-proxy on the node resolves the target clusterip address to the Calico Pod's IP: 10.233.100.4, and forwards it through the node's routing to the target host Node1, finally forwarding it to the Calico Pod through the calixxx virtual network card.
Response packets are forwarded through eth0 to node Node1, with the destination address being the Calico-Macvlan Pod's Calico network card IP: 10.233.100.3. They are then forwarded through inter-node Calico routing to Node2. Subsequently, Node2's kube-proxy rewrites the source address to the clusterip address, and then sends it to the Calico-Macvlan Pod through the calixxx virtual network card, completing the access.

Calico-Macvlan Multi-NIC Pod Accessing ClusterIP (endpoint using Macvlan network)

As shown by the red arrow in Figure 5:

According to the routing in the Pod, traffic accessing ClusterIP is forwarded through eth0 to node Node2. Then, the kube-proxy on the node resolves the target clusterip address to the Macvlan Pod's IP: 172.16.100.3, and forwards it through inter-node routing to the target host Node1. Note that when packets are sent from Node2, their source address has already been SNAT'd to: 172.16.1.3.
Upon reaching Node1, packets are forwarded through the vethxxx virtual network card, ultimately reaching the Macvlan Pod.
When response packets are sent, since the destination address is Node2's IP, the packets are directly forwarded through eth0 to Node2. After passing through Node2's kube-proxy, the source address is rewritten to the clusterip address, and the destination address is changed to the Calico-Macvlan Pod's Calico IP (10.233.100.3). Finally, the packets are sent to the Calico-Macvlan Pod through the calixxx virtual network card, completing the access.

Accessing NodePort of Calico-Macvlan Multi-NIC Pod (as shown by the black arrow in `Figure 5`)

An external client (172.16.1.100) accesses NodePort Service (172.16.1.3:32567). The packets are routed externally to Node2 (172.16.1.3), and through the iptables DNAT rules set by Kube-proxy, the destination address is changed to: 10.233.100.3 (the Calico network card IP of the Pod), and then forwarded to the Pod through the calixxx virtual network card.
When the Pod sends response packets, the destination address is: 172.16.1.100. The response packets are sent directly through net1 (the IP of the Pod's Macvlan network card) to 172.16.1.100. The client receives the packets and finds that the five-tuple does not match, so it discards the packets.
To resolve this issue caused by asymmetric routing, Spiderpool tunes the policy routing in the Pod to ensure that the response packets to NodePort are sent out from eth0.
```
~# ip rule
...
from 10.233.100.3 lookup 100
~# ip route show table 100
default via 169.254.1.1 dev eth0
```
After tuning the policy routing, response packets are sent from eth0 to Node2, and through Kube-proxy, the source address is changed to Node2's IP (172.16.1.3), with the destination address being: 172.16.1.100. The response packets are routed externally to 172.16.1.100, completing the access.

Note: The issue of asymmetric routing when accessing NodePort only exists when the default route of the multi-NIC Pod is on eth0. If the default route is on net1, this issue does not occur.

Conclusion

The following table summarizes the verified connectivity and troubleshooting conclusions for Calico, Macvlan, and Calico+Macvlan multi-NIC Pods.

Scenario	Result	Failure point if not connected	Workaround or requirement
Calico Pod accesses Calico Pod IP	✅ Connected	N/A	No extra configuration.
Macvlan Pod accesses Macvlan Pod IP in the same subnet	✅ Connected	N/A	Use direct Macvlan forwarding.
Local node accesses Macvlan Pod IP, or Macvlan Pod accesses the local node	✅ Connected	N/A	Requires Spiderpool's veth route tuning between the Pod and the local node.
Macvlan Pod accesses Calico Pod IP	✅ Connected after the kube-proxy workaround on affected clusters	kube-proxy may install `KUBE-FORWARD` `ctstate INVALID -j DROP` and drop packets for asymmetric TCP paths.	Set `net.netfilter.nf_conntrack_tcp_be_liberal=1` on every node and restart kube-proxy. Kubernetes v1.29 or later is required for kube-proxy to skip installing the INVALID drop rule when this sysctl is enabled.
Macvlan Pod accesses Service for a Calico Pod	✅ Connected after the kube-proxy workaround on affected clusters	Same kube-proxy INVALID drop issue may affect asymmetric traffic.	Use the same kube-proxy workaround.
Calico Pod accesses Macvlan Pod IP	⚠️ Topology-dependent	Cross-node direct Pod-IP access can time out in kind when the request and response paths are not symmetric.	Prefer Service access for Calico-to-underlay traffic, or verify the exact node and underlay route topology.
Calico Pod accesses Service for a Macvlan Pod	⚠️ Topology-dependent for single-underlay backends	In kind, Service traffic to a cross-node single-underlay backend can be SNATed to the per-node veth peer address and reset on the backend node.	Verify the exact Service path before attributing the failure to route tuning.
Calico+Macvlan multi-NIC Pod accesses Calico Pod IP	✅ Connected	N/A	Use the Calico interface and Calico routes.
Calico+Macvlan multi-NIC Pod accesses Macvlan Pod IP	✅ Connected	N/A	Use the Macvlan interface and the direct Macvlan subnet route.
Macvlan Pod accesses the Macvlan IP of a Calico+Macvlan multi-NIC Pod	✅ Connected	N/A	Prefer this path when the client is in the same Macvlan subnet.
Macvlan Pod accesses Service for a Calico+Macvlan multi-NIC Pod	✅ Connected	N/A	Prefer Service access when the client should not depend on a specific Pod interface IP.
Macvlan Pod accesses the Calico IP of a Calico+Macvlan multi-NIC Pod in the same Macvlan subnet	⚠️ May fail	ACK/HTTP packets return to the multi-NIC Pod's node through the Calico tunnel, then Calico workload endpoint rules may drop them as `ctstate INVALID` before delivery to the Pod.	Prefer the multi-NIC Pod's Macvlan IP or Service. If direct access to the Calico IP is required, set Calico Felix `disableConntrackInvalidCheck: true` after evaluating the cluster-level impact.
Node or external client in the same underlay network accesses NodePort for a Calico+Macvlan Pod	✅ Connected with route tuning	Response packets may leave from the wrong Pod interface if policy routes are not tuned.	Requires Spiderpool's policy route tuning for NodePort return traffic.
Node or external client in the same underlay network accesses NodePort for a single Macvlan Pod	⚠️ Topology-dependent	The result depends on which node receives the NodePort traffic and where the backend Pod runs.	Verify the exact node and backend placement before attributing the failure to route tuning.

The following table summarizes the verified behavior after underlay policy route tables stopped copying synchronized Kubernetes or hijack routes:

Test item	Result	Notes
Single-underlay Pod route state	table 100 keeps only the veth default route. No `from <underlay-ip> lookup 100` rule is added in single-underlay mode.	This matches the single-underlay behavior: route copying into table 100 is separate from source-based underlay rule creation.
Single-underlay Pod accesses Calico Pod IP or Calico Service	✅ Connected	Verified from same-node source Pods with the kube-proxy INVALID workaround applied.
Calico or Calico+Macvlan Pod accesses single-underlay Pod IP	⚠️ Topology-dependent	Same-node access was connected. Cross-node direct Pod-IP access timed out in the kind topology. Prefer Service access.
Calico or Calico+Macvlan Pod accesses single-underlay Pod Service	⚠️ Topology-dependent	Packet capture showed the request was SNATed to the per-node veth peer address, and the backend node reset the SYN-ACK path.
Node or external underlay client accesses single-underlay Pod NodePort	⚠️ Topology-dependent	The result depends on the NodePort receiving node and backend placement. Do not attribute the observed behavior to route tuning without packet-path verification.
Calico+Macvlan Pod route state	table 100 keeps only net1 default and direct subnet routes. The main table still keeps the direct underlay subnet route.	Custom routes with a gateway are not kept in the main table by the direct-route preservation logic.
Macvlan Pod accesses Calico+Macvlan Pod Macvlan IP or Service	✅ Connected	Verified with the Calico+Macvlan Pod underlay NIC in `172.100.0.0/16`.
Macvlan Pod accesses Calico+Macvlan Pod Calico IP in the same Macvlan subnet	⚠️ May fail	The failure is caused by Calico workload endpoint `ctstate INVALID` handling.

Legend:

✅: Verified or expected to work with the documented Spiderpool route tuning.
⚠️: Works only with additional constraints or configuration.
Kube-proxy INVALID-drop workaround and Calico disableConntrackInvalidCheck: true solve different drop points. The former removes kube-proxy's KUBE-FORWARD INVALID drop rule; the latter removes Calico workload endpoint INVALID drop rules.
Underlay policy route tables do not receive synchronized Kubernetes or hijack routes. In single-underlay mode, the tested Pod-IP and Service paths did not show a connectivity difference caused by this route-table behavior alone; verify the full packet path before attributing Service or NodePort failures to it.