terway-troubleshooting

Terway Troubleshooting SOP

When to use this Skill

Use this Skill whenever the user:

• Reports "cni plugin not initialized" or similar CNI errors on nodes
• Reports Pod creation or deletion failures in a cluster using Terway as the CNI
• Suspects ENI/IPAM/resource issues related to Terway (centralized or non-centralized)
• Needs to interpret specific Kubernetes events or critical log messages from Terway components.

Always assume the cluster is running Kubernetes and Terway is the CNI plugin.

High-level troubleshooting flow

Follow this order to diagnose Terway issues efficiently:

1. Verify Terway Component Health

   • When CNI errors like "plugin not initialized" or "socket not found" occur, first check if Terway Pods are Running.
   • kubectl get pods -n kube-system -l app=terway-eniip -o wide
   • If a Pod is not Running, check its events and logs (terway-init and terway containers) using the patterns in Step 1.

2. Gather Necessary Context (As needed)

   • If the cause isn't obvious from Pod status, gather cluster and node configuration.
   • You can use the following scripts or run kubectl commands directly:
     • Cluster config: ./scripts/inspect-terway-cluster.sh
     • Node config: ./scripts/inspect-terway-node.sh <node-name>
     • Pod config: ./scripts/inspect-terway-pod.sh <namespace> <pod-name>

3. Use Kubernetes Events as the primary signal

   • For any problematic Pod, inspect its Events first: kubectl describe pod <pod> -n <ns>.
   • Map Terway-specific event reasons (e.g., AllocIPFailed, CniPodCreateError) to likely causes.

4. Inspect Terway IPAM / ENI controllers

   • Depending on IPAM type (crd vs default), check relevant CRDs (PodENI, Node) and their Events.

5. Deep Dive into Logs

   • Use logs only when Events are missing or point to an ambiguous failure.

Keep answers structured: first restate what has been checked, then propose next verification steps.

Step 1 – Terway and CNI initialization

1. If the user reports "cni plugin not initialized" or "dial unix ... eni.socket: no such file or directory":

   • Phenomenon: The CNI plugin cannot communicate with the Terway Daemon.
   • Immediate Action: Check the Terway Pod status: kubectl get pods -n kube-system -l app=terway-eniip -o wide.

2. Diagnose by Pod Status and Log Patterns:

   • If the Pod is in Init:Error: Inspect terway-init logs (kubectl logs <pod> -n kube-system -c terway-init).

     • exclusive eni mode changed:
       • Explanation: The label k8s.aliyun.com/exclusive-mode-eni-type was modified on an existing node. Exclusive mode only works for newly created nodes.
       • Fix: Revert the label or recreate the node.
     • get node ... error:
       • Explanation: Terway cannot reach the Kubernetes API to fetch node labels (network or RBAC issue).
     • unsupport kernel version, require >=5.10:
       • Explanation: The node's kernel is too old for the configured features.
     • failed process input:
       • Explanation: /etc/eni/eni_conf (from eni-config CM) is missing or invalid JSON.
     • mount failed:
       • Explanation: Failed to mount bpffs on /sys/fs/bpf (privilege or kernel support issue).
     • Init erdma driver failure:
       • Explanation: modprobe erdma failed on an ERDMA-enabled node.

   • If the Pod is in CrashLoopBackOff or Error: Inspect main container logs.

     • Daemon (terway) Patterns:
       • error restart device plugin after kubelet restart: Check permissions/mounts for /var/lib/kubelet/device-plugins.
       • unable to set feature gates: Invalid flag in --feature-gates.
       • error create trunk eni: OpenAPI failure during trunk ENI initialization (check Aliyun credentials/quota).
     • Controlplane (terway-controlplane) Patterns:
       • failed to create controller: Check RBAC permissions or CRD availability.
       • failed to setup webhooks: TLS certificate or WebhookConfiguration issues.

   • If the Pod is Running but the socket error persists:

     • Verify that the var/run/eni/ directory is correctly shared between the host and the container via hostPath volume.

3. Only after Terway is confirmed running on the node, proceed to Pod create/delete failures and Events.

Step 2 – Always start from Kubernetes Events

For any Pod with network-related failures:

1. Inspect Pod Events

   • Instruct the user to run kubectl describe pod <pod> -n <ns> and paste relevant Events.
   • Focus on Terway-related reasons (case-sensitive):
     • AllocIPFailed (Warning, Pod)
     • AllocIPSucceed (Normal, Pod)
     • VirtualModeChanged (Warning, Pod)
     • CniPodCreateError (Warning, Pod)
     • CniPodDeleteError (Warning, Pod)
     • CniCreateENIError (Warning, Pod)
     • CniPodENIDeleteErr (Warning, Pod)

2. Interpret common Pod event reasons

   • AllocIPFailed (Warning, Pod)
     • Message starts with cmdAdd: error alloc ip: Backend communication failure (daemon to controlplane or daemon internal).
     • Message: eth0 config is missing: Backend failed to return configuration for the primary interface.
     • Message contains OpenAPI errors:
       • InvalidVSwitchID.IPNotEnough / QuotaExceeded.PrivateIPAddress: VSwitch IP exhaustion.
       • ErrEniPerInstanceLimitExceeded: Node-level ENI quota reached.
   • AllocIPSucceed (Normal, Pod)
     • Message: Alloc IP %s took %s.
     • IP allocation succeeded; if the Pod still fails, the issue is likely after IP allocation (datapath setup, routes, iptables, etc.).
   • VirtualModeChanged (Warning, Pod)
     • Message: IPVLan seems unavailable, use Veth instead.
     • The node’s kernel version is likely below 4.19 or lacks required capabilities. IPVLan-based data plane acceleration cannot be enabled, but Pod creation is unaffected. Networking falls back to Veth mode safely.
   • CniPodCreateError (Warning, Pod)
     • From the controlplane Pod controller.
     • Message: error parse pod annotation: k8s.aliyun.com/pod-networks is malformed.
     • Message: podNetworking is empty: k8s.aliyun.com/pod-networking annotation is present but empty.
     • Message: error get podNetworking %s: The referenced PodNetworking CR is missing.
     • Message: can not found available vSwitch for zone %s: No available VSwitch in the current zone matching the selector.
   • CniPodDeleteError (Warning, Pod)
     • Failure in Pod delete cleanup.
   • CniCreateENIError / CniPodENIDeleteErr (Warning, Pod)
     • From the PodENI controller. Message contains specific OpenAPI errors or rollbackErr.

3. If no Terway-specific Events are present

   • Confirm that the Pod is scheduled to a node where Terway is running.
   • Then move to node-level and CRD-level Events.

Step 3 – Node and Node CR Events

Distinguish between:

• Kubernetes Node object (corev1.Node).
• Terway Node CRD (network.alibabacloud.com/v1beta1 Node) used in centralized IPAM.

1. On the Kubernetes Node (corev1.Node)

   • Important Terway-related event reasons:
     • AllocIPFailed (Warning, Node)
       • From local IPAM; indicates ENI/IP issues at node level.
     • ConfigError (Warning, Node)
       • From Terway node controllers when eni-config or node capabilities are invalid.
   • Use these to distinguish between misconfiguration vs. resource exhaustion.

2. On the Terway Node CRD (centralized IPAM)

   • When centralized IPAM is enabled, a Node CR under network.alibabacloud.com exists.
   • Terway emits events on this CR for ENI lifecycle and pool operations:
     • CreateENIFailed: Message: Failed to create ENI type=%s vsw=%s: %v. Check for OpenAPI errors like InvalidVSwitchID.IPNotEnough.
     • AttachENIFailed: Message: trunk eni id not found (agent not ready) or trunk eni is not allowed for eniOnly pod (scheduling/config mismatch).
     • DeleteENIFailed: Message: Failed to delete ENI %s: %v.
   • Node Conditions on Node CR:
     • SufficientIP: If False, reason is IPResInsufficient, meaning the node pool cannot be filled.

3. Link Node events to Pod failures

   • If Pods report AllocIPFailed or CniPodCreateError, check whether the corresponding Node / Node CR shows ENI/IPAM failures.
   • Use that correlation to explain whether the problem is capacity, config, or bug.

Step 4 – Centralized vs non-centralized IPAM behavior

When reasoning about Terway behavior, always clarify which IPAM mode is in use.

1. Detect mode from context

   • Centralized IPAM indicators:
     • Presence of Terway controlplane deployment.
     • CRDs like podenis.network.alibabacloud.com, nodes.network.alibabacloud.com, podnetworkings.network.alibabacloud.com.
     • Helm/config flag centralizedIPAM: true or controlplane config with CentralizedIPAM set.
   • Non-centralized/local IPAM indicators:
     • IPAM type in eni-config is default.
     • Node-local IPAM logic in the daemon is responsible for ENI/IP management.

2. If centralized IPAM

   • In addition to Pod and Node events, always consider:
     • PodENI CR (per-pod ENI and IP state): events like CreateENIFailed, AttachENIFailed, UpdatePodENIFailed.
     • Node CR: ENI pool and warmup behavior.
     • PodNetworking CR: Events SyncPodNetworkingSucceed/Failed when syncing vswitch lists.
   • For Pod failures:
     • Check Pod Events (Cni* reasons) → PodENI Events → Node CR Events → controlplane logs.

3. If non-centralized IPAM

   • Focus on:
     • Node Events (AllocIPFailed, ConfigError).
     • eni-config ConfigMap correctness (vswitches, security groups, ip_stack, trunk/erdma flags, etc.).
     • Terway daemon logs on the affected node.

Step 5 – Using logs only when Events are insufficient

1. **When to move to

Content truncated.