Ziti sidecar blocking local k8s traffic

I am sure I am doing something wrong. When I add the ziti-tunnel sidecar my other container in the pod can’t see one of my other microservices in the same namespace. Here is my sidecar definition with the dnsConfig.

      - image: netfoundry/ziti-tunnel:latest
        name: ziti-tunnel
        - name: NF_REG_NAME
          value: tunnel-sidecar
        - name: tunnel-sidecar-jwt
          mountPath: "/netfoundry/tunnel-sidecar.jwt"
          subPath: tunnel-sidecar.jwt
          readOnly: true
        - name: ziti-tunnel-persistent-storage
          mountPath: /netfoundry
            memory: 512M
            cpu: ".5"
            memory: 1Gi
            cpu: 1
              - NET_ADMIN
      dnsPolicy: "None"

Do I need to add my local k8s DNS in this list too?

Yes, that sounds right. You could make cluster DNS the secondary nameserver. Ziti DNS still needs to be first, and it will fail fast if the query doesn’t match an authorized service intercept. What IP is cluster DNS for you?

Is cluster DNS recursive? If not then you may still need a tertiary nameserver for non-local queries.

      dnsPolicy: "None"

So, I added the ClusterIP of my kube-dns service. I still can resolve my service.

Let me see if I can reproduce that. Meanwhile, please see if you can verify it’s a DNS problem. You might have a log message from the workload container that eliminates all doubt, or you might need to kubectl exec into that container, install a DNS resolver, and query the domain name of your other microservice in the same namespace.

First, let’s verify you can see both service/kube-dns as well as your other microservice in the list of services. Assuming the other microservice is in the default namespace, you should see an item here. If hello-netfoundry were your other microservice, then you could try resolving hello-netfoundry.default.svc in cluster DNS and you should get Sorry if this is painfully elementary, just want to make sure the bases are covered. :+1:

❯ kubectl --namespace=default --kubeconfig=${HOME}/Downloads/ZitiTest4.yaml get services                
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
hello-netfoundry   ClusterIP   <none>        80/TCP    14d
kubernetes         ClusterIP      <none>        443/TCP   14d

Now for cluster DNS itself. This should reveal the current cluster IP.

❯ kubectl --namespace=kube-system --kubeconfig=${HOME}/Downloads/ZitiTest4.yaml get service/kube-dns
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   <none>        53/UDP,53/TCP,9153/TCP   14d

Yes, if I create my service without the ziti-tunnel sidecar, then I can use dig to get the IP of my other service.

# dig zdp-gateway.arena.svc.cluster.local

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> zdp-gateway.arena.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17557
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

; EDNS: version: 0, flags:; udp: 4096
; COOKIE: dcb4b83448bbad17 (echoed)
;zdp-gateway.arena.svc.cluster.local. IN	A

zdp-gateway.arena.svc.cluster.local. 5 IN A

;; Query time: 1 msec
;; WHEN: Thu Jul 15 15:06:13 UTC 2021
;; MSG SIZE  rcvd: 127

I can curl the other service and get the expected response. When I deploy my service without the ztit-tunnel sidecar.

Hi Matt,
Is your cluster using kube-dns or coredns? It’s probably kube-dns unless you’ve migrated to coredns.

One theory is that dnsConfig isn’t being inherited in the expected way. I applied the sidecar demo deployment and found that /etc/hosts only received the first nameserver from dnsConfig with values like these.

      dnsPolicy: "None"
          -   # Ziti DNS
          -  # Cluster DNS
          -     # Recursive DNS
root@ziti-tunnel-sidecar-demo-8b5976c7b-5ffks:/# cat /etc/hosts
# Kubernetes-managed hosts file.       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters     ziti-tunnel-sidecar-demo-8b5976c7b-5ffks   # pod IP

I’ll keep investigating, and please let me know if you figure it out.

Hi @modonnell , just a thought. Can you exec into any other pod which is running on the cluster network and see if the /etc/resolv.conf entry matches the value you gave in the nameservers list ? Ideally, we should be able to resolve any service if we give the correct nameserver ip in the list.

1 Like

@modonnell Did you find a working DNS configuration for tproxy mode? One alternative approach is proxy mode. Here’s an example deployment using the proxy mode. The idea would be to tightly-couple the ziti-tunnel proxy configuration with the app’s requirements in the same pod manifest. This is more rigid than tproxy but doesn’t depend on DNS because the application always connects to a bound TCP port on the loopback interface.