Accessing DNS-Over-HTTPS On Android Through Ziti Fails

Stemming from other threads, the scenario is this:

I have a DOH DNS server running behind ziti.

If I try to access this on LAN/Wifi with ziti enabled, Android says that it cannot connect for a brief moment with, "Private DNS server cannot be accessed."

If I try to do the same thing on mobile data, Android says that it cannot connect again, but never recovers.

This also kills DNS until "Private DNS" is set to automatic.

To what is private DNS set when it's not set to automatic?

Is your private nameserver answering queries via DoH or DoT?

Did you install a 3rd party app or root the device to enable DoH support in Android, or is it now natively supported?

It is native :slight_smile:
You can access it via Settings -> Network and Internet -> Private DNS

You can specify a "Private DNS provider hostname".
I created a ziti service to point to the internal IP address, and gave it that hostname.

I wonder if Android supports private DNS via a VPN and how its automatic private DNS nameserver discovery works. The Android tunneler could work with that if it's an open convention or standard.

From what I've seen thus far, Android's private DNS implements DoT, not DoH, so I suspect your device is failing to connect to 853/TCP (DoT) on the domain name you specified in private DNS settings, then falling back to plaintext DNS, if any is provided by the network profile.

Do you know for a fact Android's private DNS also supports DoH on 443/TCP? One way to test would be to listen on both for packets from the device, and give the device the address of the packet sniffer to see which ports it is spraying.

Edits:

  • I'd incorrectly stated DoT runs over 853/UDP by default, and updated my response to show the correct 853/TCP.

Updates:

  • I tried the packet sniffer and confirmed a Pixel running v15, when configured with explicit private DNS nameserver, will attempt to connect on 853/TCP and report a total loss of connectivity (no fallback to plaintext) if the required nameserver isn't available.
  • Android's "automatic" private DNS mode seems to work by probing the network's nameservers on 853/TCP. For example, if DHCP prescribes nameserver 10.11.12.13, and automatic is enabled (the default for Android), it will probe tcp:10.11.12.13:853 and fallback to plaintext on udp:10.11.12.13:53.

I misspoke, and meant to say DNS-over-TLS.

Thank you for the debugging!

I have been able to make it work in certain scenarios.

I am wondering if this can be done behind ziti, as it feels like it is in a self-defining loop.
Connecting to ziti requires DNS to resolve the controller's hostname.
Setting the DNS-over-TLS hostname that is behind ziti in Android requires ziti to be connected.

I don't see how it could work. :thinking: Paraphrasing your conclusions melded with mine: When I specify a private DNS nameserver, Android doesn't fall back to plaintext DNS if it's unavailable, so there's no internet access. This prevents the tunneler from dialing the controller, so it has no session and can't provide the private nameserver as a Ziti service.

We'd have to do something clever in ziti-edge-tunnel, like a new type of intercept config that binds a Ziti service to a tunneler's nameserver as a DoT forwarder. For example, create a Ziti service named "my-private-nameserver" and create a service with this new type of config instead of intercept.v1 to bind "my-private-nameserver to 853/TCP on the client tunneler's resolver address on the tun device.

This way, the VPN stays up and you don't need to configure Android's private DNS feature. It's set to "automatic" by default, so it will probe the network's nameservers, including the tunneler's resolver I assume, for DoT nameservers and use them if available.

With your clever solution, is this something I can test out now, or would this require changes to the codebase?

It's something that might work if I've guessed correctly how Android automatic private DNS works and someone builds it. So, there are three milestones to bring it into existence.

  1. Discover how Android automatic private DNS works. I suspect Android probes 853/TCP on network's nameserver IP addresses. If so, does this work with nameservers provided by a VPN? If so, how does it bootstrap cryptographic trust, via PKI like the web? Can we use this with our own DoT nameserver and certificate?
  2. In a test Ziti network, define a config type like android-private-dns.v1. Create a Ziti service with an instance of this new config type and a standard host.v1 config targeting the DoT nameserver.
  3. Enhance ziti-edge-tunnel to handle the new config type. It must bind the service to 853/TCP on its nameserver interface (default: 100.64.0.2).

Regarding your first point, I did a bit of research.
In automatic mode, it first queries whatever DNS server is provided by DHCP.
Following that, it switches to DNS-over-TLS, if it can.

I just tested the connection with Wireguard, and it does connect.
It fails temporarily, but then succeeds.

That's good news and suggests that it will probe nameservers provided by a VPN connection profile for private DNS capability.

Did you have to bootstrap trust between Android and your private DoT nameserver, i.e., is the DoT NS's certificate self signed, from LetsEncrypt, or something else?

Great!

I am testing this out with an Adguard Home instance, and it has a valid LetsEncrypt certificate.

I have verified functionality with this command:

kdig @192.168.1.xxx +tls-ca +tls-host=adguard.dot.domain.com example.com

@qrkourier Has there been any development to enhance ziti-edge-tunnel with this new functionality?

Although I don't have a complete mental picture of what was needed, but there is a new feature in Ziti 1.7.0 that's potentially relevant.

Ziti's reverse proxy ports run mode works like port forwarding. The dialing tunneler binds/listens on a port, and any regular network traffic arriving on that port is piped into the associated Ziti service, just like a reverse proxy/network load balancer, meaning layer 4 (TCP or UDP).

The UDP support is new in 1.7.0. So, if this DNS use case happens to require UDP, then there might be a new way to use Ziti to publish a DNS service over UDP.

Again, I'm not remembering everything we talked about in this quick message, but I think your main goal was to publish special nameservers via Ziti that clients, like an Android device running Ziti Mobile Edge app as a VPN provider, would then automatically switch to preferring those special nameservers that are only avaialble via Ziti to authorized Ziti identities.

Is that the correct focus that's still relevant for you now, @thedarkula?

You have it pretty well remembered :slight_smile:

I have a DNS-over-TLS server that I want to be able to access via ziti, specifically with the "private DNS" setting in android's networking section.

This only needs TCP, but it is good to know that UDP is there as well, thank you :slight_smile:

I did just try to create a proxy.v1 config via the ziti console, and it threw this error:

Error Validating Config

    /port: must have required property 'port'
    /protocols: must have required property 'protocols'

I ticked TCP under protocols and specified a port number, along with giving the new config a name.
I also tried adding it via the JSON option instead of the form, but both fail.

Ah, right, it was Android. You might be trying to publish the remote nameserver in the Android device's local network. Is that right?

The console problem you encountered is being worked, and you can manage configs with the CLI or API in the meantime.