Accessing DNS-Over-HTTPS On Android Through Ziti Fails

Support General Questions
1

Stemming from other threads, the scenario is this:

I have a DOH DNS server running behind ziti.

If I try to access this on LAN/Wifi with ziti enabled, Android says that it cannot connect for a brief moment with, "Private DNS server cannot be accessed."

If I try to do the same thing on mobile data, Android says that it cannot connect again, but never recovers.

This also kills DNS until "Private DNS" is set to automatic.

2

To what is private DNS set when it's not set to automatic?

Is your private nameserver answering queries via DoH or DoT?

Did you install a 3rd party app or root the device to enable DoH support in Android, or is it now natively supported?

3

It is native :slight_smile:
You can access it via Settings -> Network and Internet -> Private DNS

You can specify a "Private DNS provider hostname".
I created a ziti service to point to the internal IP address, and gave it that hostname.

4

I wonder if Android supports private DNS via a VPN and how its automatic private DNS nameserver discovery works. The Android tunneler could work with that if it's an open convention or standard.

5

From what I've seen thus far, Android's private DNS implements DoT, not DoH, so I suspect your device is failing to connect to 853/TCP (DoT) on the domain name you specified in private DNS settings, then falling back to plaintext DNS, if any is provided by the network profile.

Do you know for a fact Android's private DNS also supports DoH on 443/TCP? One way to test would be to listen on both for packets from the device, and give the device the address of the packet sniffer to see which ports it is spraying.

Edits:

  • I'd incorrectly stated DoT runs over 853/UDP by default, and updated my response to show the correct 853/TCP.

Updates:

  • I tried the packet sniffer and confirmed a Pixel running v15, when configured with explicit private DNS nameserver, will attempt to connect on 853/TCP and report a total loss of connectivity (no fallback to plaintext) if the required nameserver isn't available.
  • Android's "automatic" private DNS mode seems to work by probing the network's nameservers on 853/TCP. For example, if DHCP prescribes nameserver 10.11.12.13, and automatic is enabled (the default for Android), it will probe tcp:10.11.12.13:853 and fallback to plaintext on udp:10.11.12.13:53.
6

I misspoke, and meant to say DNS-over-TLS.

Thank you for the debugging!

I have been able to make it work in certain scenarios.

I am wondering if this can be done behind ziti, as it feels like it is in a self-defining loop.
Connecting to ziti requires DNS to resolve the controller's hostname.
Setting the DNS-over-TLS hostname that is behind ziti in Android requires ziti to be connected.

7

I don't see how it could work. :thinking: Paraphrasing your conclusions melded with mine: When I specify a private DNS nameserver, Android doesn't fall back to plaintext DNS if it's unavailable, so there's no internet access. This prevents the tunneler from dialing the controller, so it has no session and can't provide the private nameserver as a Ziti service.

We'd have to do something clever in ziti-edge-tunnel, like a new type of intercept config that binds a Ziti service to a tunneler's nameserver as a DoT forwarder. For example, create a Ziti service named "my-private-nameserver" and create a service with this new type of config instead of intercept.v1 to bind "my-private-nameserver to 853/TCP on the client tunneler's resolver address on the tun device.

This way, the VPN stays up and you don't need to configure Android's private DNS feature. It's set to "automatic" by default, so it will probe the network's nameservers, including the tunneler's resolver I assume, for DoT nameservers and use them if available.

8

With your clever solution, is this something I can test out now, or would this require changes to the codebase?

9

It's something that might work if I've guessed correctly how Android automatic private DNS works and someone builds it. So, there are three milestones to bring it into existence.

  1. Discover how Android automatic private DNS works. I suspect Android probes 853/TCP on network's nameserver IP addresses. If so, does this work with nameservers provided by a VPN? If so, how does it bootstrap cryptographic trust, via PKI like the web? Can we use this with our own DoT nameserver and certificate?
  2. In a test Ziti network, define a config type like android-private-dns.v1. Create a Ziti service with an instance of this new config type and a standard host.v1 config targeting the DoT nameserver.
  3. Enhance ziti-edge-tunnel to handle the new config type. It must bind the service to 853/TCP on its nameserver interface (default: 100.64.0.2).
10

Regarding your first point, I did a bit of research.
In automatic mode, it first queries whatever DNS server is provided by DHCP.
Following that, it switches to DNS-over-TLS, if it can.

I just tested the connection with Wireguard, and it does connect.
It fails temporarily, but then succeeds.

11

That's good news and suggests that it will probe nameservers provided by a VPN connection profile for private DNS capability.

Did you have to bootstrap trust between Android and your private DoT nameserver, i.e., is the DoT NS's certificate self signed, from LetsEncrypt, or something else?

12

Great!

I am testing this out with an Adguard Home instance, and it has a valid LetsEncrypt certificate.

I have verified functionality with this command:

kdig @192.168.1.xxx +tls-ca +tls-host=adguard.dot.domain.com example.com