Ziti Tunneller complaining "not find a way to configure system resolver"

Tunneler Apps Ziti Edge Tunnel
1

Hi Team,

When i launched the tunneller with in openziti/quickstart docker container, i got this error in its logs:
root@04bbfaebdc52:/openziti# ./ziti-edge-tunnel run -i DemoZitiLinuxTunneller &
[1] 211
root@04bbfaebdc52:/openziti# [ 0.000] INFO tunnel-sdk:ziti_tunnel.c:53 ziti_tunneler_init() Ziti Tunneler SDK (v0.17.31)
[ 0.000] INFO tunnel-cbs:ziti_dns.c:147 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255
[ 0.049] INFO ziti-edge-tunnel:resolvers.c:72 init_libsystemd() Initializing libsystemd
[ 0.049] ERROR ziti-edge-tunnel:tun.c:174 find_dns_updater() could not find a way to configure system resolver. Ziti DNS functionality will be impaired
[ 0.049] ERROR ziti-edge-tunnel:utils.c:30 run_command_va() cmd{grep -q ā€˜^nameserver 100.64.0.2ā€™ /etc/resolv.conf} failed: 256/2/No such file or directory

Is there anything which could be done to solve this error ? And why it is complaining, i can see /etc/resolv.conf file present.

2

Hi Team,
I repeated my steps of launching ziti-edge-tunnel in

  1. ubuntu:focal docker container
  2. centos:latest docker container

I still see DNS issue and tun0 interface not getting added.

In Ubuntu system, resolvectl is not present. Also, as a matter of fact when i saw the code of to look into it, i found

find_dns_updater()
static struct dns_cmd dns_cmds[] = {
{
.path = ā€œ/usr/bin/resolvectlā€,
.update_fn = dns_update_resolvectl,
},
{
.path = ā€œ/usr/bin/systemd-resolveā€,
.update_fn = dns_update_systemd_resolve,
},
{
.path = ā€œ/usr/sbin/resolvconfā€,
.update_fn = dns_update_resolvconf
},
{0}
};

None of the path existks in ubuntu:focal docker image.

Then i thought to check the tunneler on centos:8 docker container, where systemd-resolve exist. So i thought this section of function would work
.path = ā€œ/usr/bin/systemd-resolveā€,
.update_fn = dns_update_systemd_resolve,
But to my surprise, instead of executing systemd-resolve command it tried to execute resolvectl command. Here is the log.

[root@ac79b8523582 home]# ./ziti-edge-tunnel run -i 1ALinuxTunneler2
[ 0.000] INFO tunnel-sdk:ziti_tunnel.c:53 ziti_tunneler_init() Ziti Tunneler SDK (v0.17.32)
[ 0.000] INFO tunnel-cbs:ziti_dns.c:147 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255
[ 0.028] INFO ziti-edge-tunnel:resolvers.c:72 init_libsystemd() Initializing libsystemd
[ 0.028] WARN ziti-edge-tunnel:resolvers.c:91 init_libsystemd() Failure during dynamic loading function: /lib64/libsystemd.so.0: undefined symbol: sd_bus_call_methodv
[ 0.036] INFO tunnel-cbs:ziti_tunnel_ctrl.c:768 load_ziti_async() attempting to load ziti instance from file[1ALinuxTunneler2]
[ 0.036] INFO tunnel-cbs:ziti_tunnel_ctrl.c:774 load_ziti_async() loading ziti instance from /home/1ALinuxTunneler2
[ 0.036] INFO ziti_log_set_level set log level: ziti_log_lvl=3 &ziti_log_lvl = 0x561507a8cc20
[ 0.036] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:631 load_id_cb() identity[1ALinuxTunneler2] loaded
[ 0.054] INFO ziti-sdk:ziti.c:393 ziti_init_async() ztx[0] Ziti C SDK version 0.27.2 @fc4d02e(HEAD) starting at (2022-04-29T10:57:59.829)
[ 0.054] INFO ziti-sdk:ziti.c:394 ziti_init_async() ztx[0] using uv_mbed[v0.14.4], tls[mbed TLS 3.1.0]
[ 0.054] INFO ziti-sdk:ziti.c:395 ziti_init_async() ztx[0] Loading from config[1ALinuxTunneler2] controller[https://174.129.210.139:443]
[ 0.054] INFO ziti-sdk:ziti_ctrl.c:375 ziti_ctrl_init() ctrl[174.129.210.139] ziti controller client initialized
[ 0.054] INFO ziti-sdk:ziti.c:779 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://174.129.210.139:443] api_session_status[0] api_session_expired[TRUE]
sd_bus_open_system: No such file or directory
[ 0.061] ERROR ziti-edge-tunnel:utils.c:30 run_command_va() cmd{resolvectl dns tun0 100.64.0.2} failed: 256/0/Success

sd_bus_open_system: No such file or directory
sd_bus_open_system: No such file or directory
[ 0.109] ERROR ziti-edge-tunnel:utils.c:30 run_command_va() cmd{resolvectl domain tun0 ā€˜ā€™} failed: 256/0/Success

[ 0.917] INFO ziti-sdk:ziti.c:1424 version_cb() ztx[0] connected to controller https://174.129.210.139:443 version v0.24.12(38fb70dce4b7 2022-03-01T20:32:23Z)
[ 1.399] ERROR ziti-sdk:ziti.c:1287 ziti_set_api_session() ztx[0] local clock is 340 seconds behind UTC (as reported by controller)
[ 1.399] INFO ziti-sdk:ziti.c:1314 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
[ 1.399] INFO tunnel-cbs:ziti_tunnel_ctrl.c:632 on_ziti_event() ziti_ctx[1ALinuxTunneler2] connected to controller

So clearly i see an issue here.

3

Hi Team,
I even tried running tunneler using docker images available in docker hub and got the following error.
ubuntu@ubuntuv:~/zitinw$ sudo docker run -t --network=host --cap-add=NET_ADMIN -v $(pwd)/zititun:/netfoundry -e NF_REG_NAME=1ALinuxTunneler3 netfoundry/ziti-tunnel:latest
WARN: identity configuration /netfoundry/1ALinuxTunneler3.json does not exist
INFO: /var/run/secrets/kubernetes.io/enrollment-token is not a directory
INFO: /enrollment-token is not a directory
INFO: looking for /netfoundry/1ALinuxTunneler3.jwt
INFO: enrolling with token from file ā€˜/netfoundry/1ALinuxTunneler3.jwtā€™ and value ā€˜eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbSI6Im90dCIsImV4cCI6MTY1MTQwNDIyNiwiaXNzIjoiaHR0cHM6Ly8xNzQuMTI5LjIxMC4xMzk6NDQzIiwianRpIjoiODEzNjZhZTUtYzNlZS00ZGQ3LTk0NzQtMDAwMjEzY2VhMDU5Iiwic3ViIjoiUnFMMDNmQi44WiJ9.0pjmuNKXlJdX10RLJwigb3G8gyjabmoiyM_vKu4iVgxRW6CEd_VudvlFJCdYDEaMIdpZimcAGzZmiZLM9p6v46jHV9TkG_Ff3Oe5KKqVRtpRv2pANjOLj453zA5S4O2wvvF3q8BUiM-SdFG3ZOrkj5wuMjlSNcSZOjgwINOESmwrWJBVHTKc1m59M1ODSnkLBBT6CC8sFtJPTB5nqosiqap5yfJDbv5KFeVCH3_uOqep2m-18H1nsmEDgiWhq1yDEcuggmf49kExw-98bGloh7fcq7_aYwRJ2pUN_z2SBXdMOA8_nOwWTpiqAAHT1JYCDY94De0Wr_F4k5rV0Ph_Owā€™
INFO generating 4096 bit RSA key
INFO enrolled successfully. identity file written to: /netfoundry/1ALinuxTunneler3.json
INFO: probing iptables
INFO: updating iptables alternative to iptables-legacy
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode
INFO: updating ip6tables alternative to ip6tables-legacy
update-alternatives: using /usr/sbin/ip6tables-legacy to provide /usr/sbin/ip6tables (ip6tables) in manual mode
WARN: not updating ebtables alternative to ebtables-legacy
WARN: not updating arptables alternative to arptables-legacy
running ziti-tunnel

  • ZITI_TUNNEL_PID=28
  • wait 28
  • ziti-tunnel -i /netfoundry/1ALinuxTunneler3.json run
    RESTY 2022/04/29 11:19:16 ERROR Get ā€œhttps://api.github.com/repos/openziti/ziti/releases/latestā€: context deadline exceeded (Client.Timeout exceeded while awaiting headers), Attempt 1
    [ 3.858] INFO ziti/ziti-tunnel/cmd/ziti-tunnel/subcmd.run: using tproxy interceptor
    [ 3.865] WARNING edge/tunnel/dns.flushDnsCaches: {error=[exec: ā€œsystemd-resolveā€: executable file not found in $PATH]} unable to find systemd-resolve in path, consider adding a dns flush to your restart process
    [ 3.865] INFO edge/tunnel/dns.NewDnsServer: starting dns serverā€¦
    [ 5.866] INFO edge/tunnel/dns.NewDnsServer: dns server running at 127.0.0.1:53
    [ 5.866] INFO edge/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
    [ 5.882] FATAL edge/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test: no such host

ziti-tunnel runs an internal DNS server which must be first in the hostā€™s
resolver configuration. On systems that use NetManager/dhclient, this can
be achieved by adding the following to /etc/dhcp/dhclient.conf:

prepend domain-name-servers 127.0.0.1:53;
  • alldone
  • [[ 28 =~ [1]+$ ]]
  • kill -INT 28
    /docker-entrypoint.sh: line 9: kill: (28) - No such process

To solve this, i tried the options to modify dhclient.conf file mentioned at Tunnelers | Ziti of the host ubuntu 18.04 system. but it didnā€™t workā€¦ It is failing at the testing phase.

To skip the testing phase, i had to add ā€œ19.65.28.94 ziti-tunnel.resolver.testā€ entry in /etc/hosts file in host system.

This time i got the out as

ubuntu@ubuntuv:~/zitinw$ sudo docker run -t --network=host --cap-add=NET_ADMIN -v $(pwd)/zititun:/netfoundry -e NF_REG_NAME=1ALinuxTunneler3 netfoundry/ziti-tunnel:latest
INFO: probing iptables
running ziti-tunnel

  • ZITI_TUNNEL_PID=13
  • wait 13
  • ziti-tunnel -i /netfoundry/1ALinuxTunneler3.json run
    [ 0.491] INFO ziti/ziti-tunnel/cmd/ziti-tunnel/subcmd.run: using tproxy interceptor
    [ 0.492] WARNING edge/tunnel/dns.flushDnsCaches: {error=[exec: ā€œsystemd-resolveā€: executable file not found in $PATH]} unable to find systemd-resolve in path, consider adding a dns flush to your restart process
    [ 0.492] INFO edge/tunnel/dns.NewDnsServer: starting dns serverā€¦
    [ 2.493] INFO edge/tunnel/dns.NewDnsServer: dns server running at 127.0.0.1:53
    [ 2.493] INFO edge/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
    [ 2.493] INFO edge/tunnel/dns.(*resolver).RemoveHostname: removing ziti-tunnel.resolver.test from resolver
    [ 3.324] INFO edge/tunnel/intercept.SetDnsInterceptIpRange: dns intercept IP range: 100.64.0.1 - 100.127.255.254

Is this fine ?

  1. 0-9 ā†©ļøŽ

4

Hi @sameersarkar-tcl , Is it your goal to terminate a Ziti service inside a Docker bridge network (ingress via Ziti), or do you wish to provide outgoing access for some container(s) to Ziti services that are hosted elsewhere?

For each of these use cases I would recommend a different way of using the Linux tunneler with Docker.

I will provide a couple of examples.

5

@sameersarkar-tcl There is a third use case you might have in mind, to provide access to the Docker Engine host to use Ziti services that are hosted on another device. That is, to ā€œinstallā€ the Linux tunneler with Docker instead of installing the executable natively on the Docker Engine host.

6

Here is an example for running the Linux tunneler in Docker to provide outgoing access to Ziti services as well as Ziti DNS. Both the Docker Engine and docker bridge networks using the Engineā€™s IP route table will have the same access to remote Ziti services and local Ziti DNS. You may wish to inspect the Compose file used in the example.

For this example to work:

  1. You must have a valid enrollment token (e.g. 1ALinuxTunneler3.jwt) or an enrolled identity file (e.g. 1ALinuxTunneler3.json) in the current directory.
  2. your Linux user must be a member of the privileged permission group for Docker (e.g. docker), or you may run Compose as root with sudo docker-compose as you did above.
$ wget https://github.com/openziti/ziti-tunnel-sdk-c/raw/main/docker/docker-compose.yml
$ NF_REG_NAME=1ALinuxTunneler3 docker-compose up ziti-tun

Note: A tunneler which is configured to provide outgoing access and DNS to Ziti services may also be configured to host / terminate additional services.

1 Like
7

Hi @qrkourier , I have primarily 2 usecases at the moment which i have to validate. I will mention it in here is a while.
My intention of sharing above issues in different scenarios above werenā€™t about the usecases i was trying to test, but to share my observations that the tunneller is having issues with OSā€™s systemd-resolve and resolvectl commands. Due to which it ainā€™t able to proceed further with certain functionalities. This is something i want to highlight and my focus is. Requesting your help in debugging this problem here and get a fix as early as possible.

Now coming to the usecases you asked, I am checking 2 usecases.

image
image845Ɨ481 36.6 KB

image
image831Ɨ472 38.3 KB

8

@sameersarkar-tcl You may wish to inspect the Compose file used in the example to understand how to make the systemd dbus socket and tun device available to ziti-edge-tunnel (the Linux tunneler) when it is running in a privileged container for the purpose of providing DNS and IP routes.

In use case 2ā€™s private network with a Docker network I can see the direction of the arrows are pointing from the Docker router tunneler to two different applications. Are those applications in host 1, host 2 also in a Docker network (bridge network?), or are those ā€œhostsā€ per se?

Will you share some details about how you are running the Docker router tunneler shown there in the private network for use case 2? Itā€™s clear that you want to publish the app 1, app 2 with Ziti, and I know there are some advantages to having a router near your application. If you decide itā€™s not necessary to have a router in the end then I can recommend a simplified tunneler configuration that you could use to publish / host / bind app 1, app 2 in the private network.

Here is a simplified example of using the old Linux tunneler to provide hosting only (no DNS, no IP route).

$ wget https://github.com/openziti/ziti/raw/main/ziti-tunnel/docker/docker-compose.yml
$ NF_REG_NAME=1ALinuxTunneler3 docker-compose up ziti-host

Note that this is a different file that is also named docker-compose.yml just like the previous example which was for a different use case involving DNS and IP routes.

9

I don't think we can help there, unfortunately. :frowning: What OS are you running? Is it Amazon Linux maybe? I might be able to try that out.

I literally just made a video for another user on another discourse thread for the "use case 2" you show above. It's exactly what you're diagram shows. In the video I use a linux virtual machine (running on my local windows) to access a totally private http server in aws. You can watch that, it might be useful. It's only 5m30s long:

Link to that post which has all the CLI commands is here. Where to begin with openziti? - #46 by TheLumberjack

Do let me know what linux variant you're using please? If we can, we'll see what happens when we try it

10

No they aren't running on docker at all. Those apps are deployed on old fashioned VMs'/Systems as binaries

1 Like
11

This is explained in https://www.youtube.com/watch?v=1GC7gt3rsrg&t=3s video made by Clint.

1 Like
12

I am using Ubuntu 18.04 OS running on a Virtual Box. This ubuntu is a desktop version of the OS.

13

I will definitely look into this video and get back

14

Strange that Ubuntu 18.04 was giving you troubles. Iā€™ve been using 20 to test and that was fine. My virtualbox environment has been unreliable lately. I donā€™t think Iā€™ll be able to test it until virtualbox updates.

I would expect Ubuntu to have resolved and work ā€˜fineā€™. I was able to use linux Mint just recently successfully. Thatā€™s a debian-based distro as well.

Not sure if we can help you much more - but yah, let us know how you get on and weā€™ll try.

15

Hi @TheLumberjack , @qrkourier ,
My problem statement is very simple.
The ziti-edge-tunneller binary is not running from within openziti/quickstart docker container.
It is giving the DNS problem as stated by me above and in Launched openziti network using docker compose, but client not able to fetch simple blue service - #14 by dovholuknf

But it looks like the DNS configuration issue is not causing my problem as clarified by @TheLumberjack in Launched openziti network using docker compose, but client not able to fetch simple blue service - #14 by dovholuknf

But something is.

16

@sameersarkar-tcl I think you want to emulate Clint's success with running ziti-tunnel host in a non-privileged container for the purpose of hosting aka terminating Ziti services. If that is accurate then you will need a setup similar to this example.

After downloading this Compose file ziti-host you may ensure you have the latest release by running docker-compose down --remove-orphans && docker-compose pull before re-running the above up command.

Importantly, this example is not a privileged container and does not use the Docker Engine Host's network and does not provide any DNS or IP routes aka "intercept" at all.

If my guess is not accurate then I think you must be trying to provide DNS and IP routes with ziti-edge-tunnel run. This can be accomplished with a privileged process and it is not particularly helpful to run this intercepting tunneler inside a container, although it is possible with the following example.

If indeed you do wish to enable Ziti DNS and IP routes for interception with ziti-edge-tunnel run then please be sure to use the latest release version.