Access to a service trough private router

Hey,

I have this setup consisting two VPS on two separate cloud providers :

  1. Host 1 → has a controller + the usual edge router. (Configured this manually and not using the quickstart)

  2. Host 2 → private edge router with tunneler enabled + a docker container running a http server which prints helloworld (on port 6980)

I’m trying to reach the http server on host 2 from host 1 without opening any port.

Here’s what I’ve done so far :

Host 1 -> Controller Configuration
---
v: 3

db:                     "/home/ubuntu/test_openziti/controller/controller.db"

identity:
  cert:                 "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/certs/overlay_network_intermediate_ca_client.cert"
  server_cert:          "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/certs/overlay_network_intermediate_ca_server.chain.pem"
  key:                  "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/keys/overlay_network_intermediate_ca_server.key"
  ca:                   "/home/ubuntu/test_openziti/pki/cas.pem"
 
network:

  routeTimeoutSeconds: 10
  createCircuitRetries: 2
  pendingLinkTimeoutSeconds: 10
  cycleSeconds: 15
  minRouterCost: 10
  routerConnectChurnLimit: 1m
  initialLinkLatency: 65s
  smart:
    rerouteFraction: 0.02
    rerouteCap: 4

ctrl:
  maxQueuedConnects: 1
  maxOutstandingConnects: 16
  connectTimeoutMs: 5000
  listener: tls:0.0.0.0:8440

healthChecks:
  boltCheck:
    interval: 30s
    timeout: 20s
    initialDelay: 30s

edge:
  api:
    activityUpdateInterval: 90s
    activityUpdateBatchSize: 250
    sessionTimeout: 30m
    address: 129.151.227.134:8441
  enrollment:
    signingCert:
      cert: "/home/ubuntu/test_openziti/pki/overlay_network_signing_intermediate_ca/certs/overlay_network_signing_intermediate_ca.cert"
      key:  "/home/ubuntu/test_openziti/pki/overlay_network_signing_intermediate_ca/keys/overlay_network_signing_intermediate_ca.key"
    edgeIdentity:
      duration: 180m
    edgeRouter:
      duration: 180m

web:
  - name: management_api
    bindPoints:
      - interface: 127.0.0.1:5309
        address: 127.0.0.1:5309
    identity:
      ca:          "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/certs/overlay_network_intermediate_ca.cert"
      key:         "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/keys/overlay_network_intermediate_ca_server.key"
      server_cert: "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/certs/overlay_network_intermediate_ca_server.chain.pem"
      cert:        "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/certs/overlay_network_intermediate_ca_client.cert"
    options:
      idleTimeout: 5000ms
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: fabric
        options: {}
      - binding: health-checks
        options: {}
      - binding: edge-management
        options: {}
  - name: client_api
    bindPoints:
      - interface: 0.0.0.0:8441
        address: 129.151.227.134:8441
    identity:
      ca:          "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/certs/overlay_network_intermediate_ca.cert"
      key:         "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/keys/overlay_network_intermediate_ca_server.key"
      server_cert: "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/certs/overlay_network_intermediate_ca_server.chain.pem"
      cert:        "/home/ubuntu/test_openziti/pki/overlay_network_intermediate_ca/certs/overlay_network_intermediate_ca_client.cert"
    options:
      idleTimeout: 5000ms
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: {}
Host 1 -> Public Edge Router Config
---
v: 3

identity:
  cert:                 "/home/ubuntu/test_openziti/pki/instance-20230806-1503_public_router/client.cert"
  server_cert:          "/home/ubuntu/test_openziti/pki/instance-20230806-1503_public_router/server.cert"
  key:                  "/home/ubuntu/test_openziti/pki/instance-20230806-1503_public_router/server.key"
  ca:                   "/home/ubuntu/test_openziti/pki/instance-20230806-1503_public_router/cas.cert"

ctrl:
  endpoint:             tls:129.151.227.134:8440

link:
  dialers:
    - binding: transport
  listeners:
    - binding:          transport
      bind:             tls:0.0.0.0:10080
      advertise:        tls:129.151.227.134:10080
      options:
        outQueueSize:   4

listeners:
  - binding: edge
    address: tls:0.0.0.0:8442
    options:
      advertise: 129.151.227.134:8442
      connectTimeoutMs: 5000
      getSessionTimeout: 60
  - binding: tunnel
    options:
      mode: host

edge:
  csr:
    country: US
    province: NC
    locality: Charlotte
    organization: NetFoundry
    organizationalUnit: Ziti
    sans:
      dns: ['129.151.227.134', 'localhost']
      ip: ['127.0.0.1', '129.151.227.134']

forwarder:
  latencyProbeInterval: 10
  xgressDialQueueLength: 1000
  xgressDialWorkerCount: 128
  linkDialQueueLength: 1000
  linkDialWorkerCount: 32
Host 2 -> Private Edge Router Config
---
v: 3

identity:
  cert:                 "/home/ubuntu/test_openziti/pki/instance-20230806-1758_private_router/client.cert"
  server_cert:          "/home/ubuntu/test_openziti/pki/instance-20230806-1758_private_router/server.cert"
  key:                  "/home/ubuntu/test_openziti/pki/instance-20230806-1758_private_router/server.key"
  ca:                   "/home/ubuntu/test_openziti/pki/instance-20230806-1758_private_router/cas.cert"

ctrl:
  endpoint:             tls:129.151.227.134:8440

link:
  dialers:
    - binding: transport

listeners:
  - binding: edge
    address: tls:0.0.0.0:8442
    options:
      advertise: 144.24.200.145:8442
      connectTimeoutMs: 5000
      getSessionTimeout: 60
  - binding: tunnel
    options:
      mode: host

edge:
  csr:
    country: US
    province: NC
    locality: Charlotte
    organization: NetFoundry
    organizationalUnit: Ziti
    sans:
      dns: ['144.24.200.145', 'localhost']
      ip: ['127.0.0.1', '129.151.227.134']

forwarder:
  latencyProbeInterval: 10
  xgressDialQueueLength: 1000
  xgressDialWorkerCount: 128
  linkDialQueueLength: 1000
  linkDialWorkerCount: 32
Edge Routers Listing
╭────────────┬───────────────────────────────────────┬────────┬───────────────┬──────┬────────────╮
│ ID         │ NAME                                  │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├────────────┼───────────────────────────────────────┼────────┼───────────────┼──────┼────────────┤
│ 1yYwOs-H6R │ instance-20230806-1758_private_router │ true   │ true          │    0 │ all        │
│ hPYwgiSqDR │ instance-20230806-1503_public_router  │ true   │ true          │    0 │ all        │
│            │                                       │        │               │      │ public     │
╰────────────┴───────────────────────────────────────┴────────┴───────────────┴──────┴────────────╯

Fabric Links Listing
╭───────────────────────┬───────────────────────────────────────┬──────────────────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID                    │ DIALER                                │ ACCEPTOR                             │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE     │ STATUS │ FULL COST │
├───────────────────────┼───────────────────────────────────────┼──────────────────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ nLdfvFGl1ibYaJfe0sU30 │ instance-20230806-1758_private_router │ instance-20230806-1503_public_router │           1 │       2.7ms │       2.7ms │ Connected │     up │         5 │
╰───────────────────────┴───────────────────────────────────────┴──────────────────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯

Edge Router Policies
╭────────────────────────┬───────────────────────────────┬────────────────────────────────────────┬────────────────────────────────────────╮
│ ID                     │ NAME                          │ EDGE ROUTER ROLES                      │ IDENTITY ROLES                         │
├────────────────────────┼───────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────┤
│ 78ERqdmXDSXTq1zUnceYIb │ all                           │ #all                                   │ #all                                   │
│ 7b1McM4q79XCVGCvHAqPAn │ all_endpoints_public_routers  │ #public                                │ #all                                   │
│ P2SM6fC7T              │ edge-router-P2SM6fC7T-system  │ @instance-20230806-1758_private_router │ @instance-20230806-1758_private_router │
│ g7Sp6hCXTg             │ edge-router-g7Sp6hCXTg-system │ @instance-20230806-1503_public_router  │ @instance-20230806-1503_public_router  │
╰────────────────────────┴───────────────────────────────┴────────────────────────────────────────┴────────────────────────────────────────╯

Service edge router policies
╭────────────────────────┬──────────────────────────┬───────────────┬───────────────────╮
│ ID                     │ NAME                     │ SERVICE ROLES │ EDGE ROUTER ROLES │
├────────────────────────┼──────────────────────────┼───────────────┼───────────────────┤
│ 1JZERkBxxniwabTj5NRzHy │ all_routers_all_services │ #all          │ #all              │
╰────────────────────────┴──────────────────────────┴───────────────┴───────────────────╯

Intercept Config
ziti edge create config hello.http.cfg.intercept intercept.v1 '{
    "addresses": ["hello.http.ziti"],
    "protocols": ["tcp"],
    "portRanges": [ {"low":80,"high":80} ]
}'
Host Config
ziti edge create config hello.http.cfg.host host.v1 '{
    "address": "127.0.0.1",
    "protocol": "tcp",
    "port": 6980
}'
Service
ziti edge create service hello.http \
    --configs hello.http.cfg.intercept,hello.http.cfg.host \
    --role-attributes all
Dial & Bind Service Policies
ziti edge create service-policy hello.http.dial Dial --identity-roles "#all" --service-roles "@hello.http"
ziti edge create service-policy hello.http.bind Bind --identity-roles "#all" --service-roles "@hello.http"

When I try to curl hello.http.ziti It seems like it cannot resolve the host, Im I doing something wrong ?

Woah - super cool use of collapsable sections! TIL! :slight_smile:

[details="This is a detail section"]

If that's the case, have you attempted to look at the logs for the side that should be doing the interception? Or have you tried to "dig/nslookup/Resolve-DNSName" on that machine?

That is most definitely a first step. I can see your intercept configuration is correct. I can see you have a dial policy linking #all to @hello.http...

Your steps all seem to be correct. You mention: " 1. Host 1 → has a controller + the usual edge router. (Configured this manually and not using the quickstart)". Are you trying to use the ziti router command to intercept traffic or are you running ziti edge tunnel?

I expect you're running ziti router and want it to intercept traffic? If that's the case, what does your "binding: tunnel" look like? Did you change that over to tproxy or is it still set to host?

  - binding: tunnel
    options:
      mode: host #tproxy|host
1 Like

Yup, I'm trying to use the router to intercept traffic.

Are we talking about the router on Host 1 or the one in Host 2 ? In any case, they both have host. Do I need to set it to tproxy on Host 1 in order to achieve what I'm looking forward?

On routers, host mode means "only let traffic exit from this node, do not intercept traffic". The tproxy mode will allow the router to not only offload traffic like with host mode, but also will intercept traffic too.

So "yes" you need to set it to tproxy and restart the process.

1 Like

Looks like my edge router’s going nuts when I change that to tproxy haha

Aug 07 21:45:39 instance-20230806-1503 sh[79174]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/dns/server.go:128","func":"github.com/openziti/edge/tunnel/dns.NewDnsServer","level":"fatal","msg":"system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test on 127.0.0.53:53: no such host\n\nziti-tunnel runs an internal DNS server which must be first in the host's\nresolver configuration. On systems that use NetManager/dhclient, this can\nbe achieved by adding the following to /etc/dhcp/dhclient.conf:\n\n    prepend domain-name-servers 127.0.0.1:53;\n\n","time":"2023-08-07T21:45:39.046Z"}

Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/router/xgress_edge_tunnel/tunneler.go:71","func":"github.com/openziti/edge/router/xgress_edge_tunnel.(*tunneler).Start","level":"info","mode":"tproxy","msg":"creating interceptor","time":"2023-08-07T21:45:41.239Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/intercept/tproxy/tproxy_linux.go:93","func":"github.com/openziti/edge/tunnel/intercept/tproxy.New","level":"info","msg":"udpIdleTimeout is less than 5s, using default value of 5m0s","time":"2023-08-07T21:45:41.239Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/intercept/tproxy/tproxy_linux.go:97","func":"github.com/openziti/edge/tunnel/intercept/tproxy.New","level":"info","msg":"udpCheckInterval is less than 1s, using default value of 30s","time":"2023-08-07T21:45:41.239Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/intercept/tproxy/tproxy_linux.go:100","func":"github.com/openziti/edge/tunnel/intercept/tproxy.New","level":"info","msg":"tproxy config: lanIf            =  []","time":"2023-08-07T21:45:41.239Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/intercept/tproxy/tproxy_linux.go:101","func":"github.com/openziti/edge/tunnel/intercept/tproxy.New","level":"info","msg":"tproxy config: diverter         =  []","time":"2023-08-07T21:45:41.239Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/intercept/tproxy/tproxy_linux.go:102","func":"github.com/openziti/edge/tunnel/intercept/tproxy.New","level":"info","msg":"tproxy config: udpIdleTimeout   =  [5m0s]","time":"2023-08-07T21:45:41.239Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/intercept/tproxy/tproxy_linux.go:103","func":"github.com/openziti/edge/tunnel/intercept/tproxy.New","level":"info","msg":"tproxy config: udpCheckInterval =  [30s]","time":"2023-08-07T21:45:41.239Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/router/xgress_edge/certchecker.go:118","func":"github.com/openziti/edge/router/xgress_edge.(*CertExpirationChecker).Run","level":"info","msg":"waiting 8615h55m15.759936317s to renew certificates","time":"2023-08-07T21:45:41.240Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/intercept/tproxy/tproxy_linux.go:265","func":"github.com/openziti/edge/tunnel/intercept/tproxy.(*interceptor).addIptablesChain","level":"info","msg":"added iptables 'mangle' link 'PREROUTING' --\u003e 'NF-INTERCEPT'","time":"2023-08-07T21:45:41.243Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/intercept/tproxy/tproxy_linux.go:136","func":"github.com/openziti/edge/tunnel/intercept/tproxy.New","level":"info","msg":"no lan interface specified with '-lanIf'. please ensure firewall accepts intercepted service addresses","time":"2023-08-07T21:45:41.243Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/dns/server.go:59","func":"github.com/openziti/edge/tunnel/dns.flushDnsCaches","level":"info","msg":"dns caches flushed","time":"2023-08-07T21:45:41.248Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/dns/server.go:86","func":"github.com/openziti/edge/tunnel/dns.NewDnsServer","level":"info","msg":"starting dns server...","time":"2023-08-07T21:45:41.248Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"endpoint":"tls:129.151.227.134:8440","file":"github.com/openziti/fabric@v0.23.45/router/env/ctrls.go:136","func":"github.com/openziti/fabric/router/env.(*networkControllers).connectToControllerWithBackoff.func3","level":"info","msg":"successfully connected to controller","time":"2023-08-07T21:45:41.343Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/router/handler_edge_ctrl/hello.go:77","func":"github.com/openziti/edge/router/handler_edge_ctrl.(*helloHandler).HandleReceive.func1","level":"info","msg":"received server hello, replying","time":"2023-08-07T21:45:41.344Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/router/handler_edge_ctrl/apiSessionAdded.go:200","func":"github.com/openziti/edge/router/handler_edge_ctrl.(*apiSessionAddedHandler).instantSync","level":"info","msg":"first api session syncId [cll1eiz8h00afwb8ns6lth2x5], starting","strategy":"instant","time":"2023-08-07T21:45:41.345Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/router/handler_edge_ctrl/apiSessionAdded.go:265","func":"github.com/openziti/edge/router/handler_edge_ctrl.(*apiSessionSyncTracker).Add","level":"info","msg":"received api session sync chunk 0, isLast=true","time":"2023-08-07T21:45:41.345Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"_channels":["link","linkListener"],"file":"github.com/openziti/fabric@v0.23.45/router/xlink_transport/listener.go:112","func":"github.com/openziti/fabric/router/xlink_transport.(*listener).BindChannel","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","linkProtocol":"tls","msg":"accepting link","routerId":"DZOhMSrwKp","time":"2023-08-07T21:45:41.633Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"_channels":["link","linkListener"],"channelType":1,"file":"github.com/openziti/fabric@v0.23.45/router/xlink_transport/listener.go:138","func":"github.com/openziti/fabric/router/xlink_transport.(*listener).bindSplitChannel","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","linkProtocol":"tls","msg":"accepted part of split conn","routerId":"DZOhMSrwKp","routerVersion":"v0.29.0","time":"2023-08-07T21:45:41.633Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/fabric@v0.23.45/router/handler_link/bind.go:165","func":"github.com/openziti/fabric/router/handler_link.(*bindHandler).verifyRouter","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","msg":"successfully verified router for link","routerId":"DZOhMSrwKp","time":"2023-08-07T21:45:41.634Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/fabric@v0.23.45/router/handler_link/bind.go:101","func":"github.com/openziti/fabric/router/handler_link.(*bindHandler).BindChannel","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","msg":"link destination support heartbeats","routerId":"DZOhMSrwKp","routerVersion":"v0.29.0","time":"2023-08-07T21:45:41.634Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"_channels":["link","linkListener"],"file":"github.com/openziti/fabric@v0.23.45/router/xlink_transport/listener.go:112","func":"github.com/openziti/fabric/router/xlink_transport.(*listener).BindChannel","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","linkProtocol":"tls","msg":"accepting link","routerId":"DZOhMSrwKp","time":"2023-08-07T21:45:41.672Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"_channels":["link","linkListener"],"channelType":2,"file":"github.com/openziti/fabric@v0.23.45/router/xlink_transport/listener.go:138","func":"github.com/openziti/fabric/router/xlink_transport.(*listener).bindSplitChannel","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","linkProtocol":"tls","msg":"accepted part of split conn","routerId":"DZOhMSrwKp","routerVersion":"v0.29.0","time":"2023-08-07T21:45:41.672Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/fabric@v0.23.45/router/handler_link/bind.go:165","func":"github.com/openziti/fabric/router/handler_link.(*bindHandler).verifyRouter","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","msg":"successfully verified router for link","routerId":"DZOhMSrwKp","time":"2023-08-07T21:45:41.673Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/fabric@v0.23.45/router/handler_link/bind.go:101","func":"github.com/openziti/fabric/router/handler_link.(*bindHandler).BindChannel","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","msg":"link destination support heartbeats","routerId":"DZOhMSrwKp","routerVersion":"v0.29.0","time":"2023-08-07T21:45:41.673Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/fabric@v0.23.45/router/accepter.go:19","func":"github.com/openziti/fabric/router.(*xlinkAccepter).Accept","level":"info","msg":"accepted new link [l/7dVGwSTUY3Up7Hce9F4J7x]","time":"2023-08-07T21:45:41.673Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"_channels":["link","linkListener"],"channelType":2,"file":"github.com/openziti/fabric@v0.23.45/router/xlink_transport/listener.go:160","func":"github.com/openziti/fabric/router/xlink_transport.(*listener).bindSplitChannel","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","linkProtocol":"tls","msg":"accepted link","routerId":"DZOhMSrwKp","routerVersion":"v0.29.0","time":"2023-08-07T21:45:41.673Z"}
Aug 07 21:45:41 instance-20230806-1503 sh[79186]: {"_channels":["link","linkListener"],"channelType":2,"file":"github.com/openziti/fabric@v0.23.45/router/xlink_transport/listener.go:163","func":"github.com/openziti/fabric/router/xlink_transport.(*listener).bindSplitChannel","level":"info","linkId":"7dVGwSTUY3Up7Hce9F4J7x","linkProtocol":"tls","msg":"link registered","routerId":"DZOhMSrwKp","routerVersion":"v0.29.0","time":"2023-08-07T21:45:41.673Z"}
Aug 07 21:45:42 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/router/handler_edge_ctrl/apiSessionAdded.go:124","func":"github.com/openziti/edge/router/handler_edge_ctrl.(*apiSessionAddedHandler).applySync","level":"info","msg":"finished sychronizing api sessions [count: 6, syncId: cll1eiz8h00afwb8ns6lth2x5, duration: 32.872µs]","time":"2023-08-07T21:45:42.346Z"}
Aug 07 21:45:43 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/dns/server.go:116","func":"github.com/openziti/edge/tunnel/dns.NewDnsServer","level":"info","msg":"dns server running at 127.0.0.1:53","time":"2023-08-07T21:45:43.249Z"}
Aug 07 21:45:43 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/dns/server.go:257","func":"github.com/openziti/edge/tunnel/dns.(*resolver).AddHostname","level":"info","msg":"adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver","time":"2023-08-07T21:45:43.249Z"}
Aug 07 21:45:43 instance-20230806-1503 sh[79186]: {"file":"github.com/openziti/edge@v0.24.364/tunnel/dns/server.go:128","func":"github.com/openziti/edge/tunnel/dns.NewDnsServer","level":"fatal","msg":"system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test on 127.0.0.53:53: no such host\n\nziti-tunnel runs an internal DNS server which must be first in the host's\nresolver configuration. On systems that use NetManager/dhclient, this can\nbe achieved by adding the following to /etc/dhcp/dhclient.conf:\n\n    prepend domain-name-servers 127.0.0.1:53;\n\n","time":"2023-08-07T21:45:43.251Z"}
Aug 07 21:45:43 instance-20230806-1503 systemd[1]: instance-20230806-1503_public_router_daemon.service: Main process exited, code=exited, status=1/FAILURE

Any Idea of what could be the issue ?

Maybe if we look at that message line wrapped it'll be easier to see. When using tproxy mode, the router will try to resolve a predefined intercept and verify it works. If it doesn't succeed, it logs this fatal message. Do you know if your system uses Net manager? Did you try to update dhclient.conf?

Unfortunately I'm going offline for a while, maybe another community member like @dariuszSki / @JamminSoleng / @qrkourier can help out if of that doesn't work

1 Like

When using a Ziti TPROXY mode of intercepting Ziti service traffice, you must configure the host’s DNS resolver to use Ziti DNS. The router will not start with tunnel binding tproxy enabled if the host’s DNS resolver is not configured, and will emit the error message that @TheLumberjack highlighted.

The ziti router command does not do this automatically. The precise steps to configure the host’s DNS vary depending on which flavor of Linux you’re running. If you’re unsure where to start, please investigate the status of your /etc/resolv.conf file or look up the instructions for your distro, e.g., Fedora or Ubuntu.

The correct configuration is for Ziti DNS (the nameserver provided by ziti router run listening on 127.0.0.1 at UDP port 53 unless you’ve configured it differently) to the primary nameserver for the host. Ziti DNS will answer affirmatively any queries that match your allowed Ziti services’ intercept addresses and answer negatively any other queries, so your host will continue trying to answer those non-matching queries with the next nameserver.

1 Like

Which distro are you running the router on?

distros that use systemd-resolved to manage DNS can be configured to use the ziti DNS server by adding a drop-in file:

$ systemctl is-active systemd-resolved
active
$ cat /etc/systemd/resolved.conf.d/ziti-tunnel.conf 
[Resolve]
DNS=127.0.0.1

Some distros manage DNS with network manager, which can be configured with a different drop-in:

$ systemctl is-active systemd-networkd
active
$ cat /etc/systemd/network/ziti-tunnel.network 
[Network]
DNS=127.0.0.1

Note that you’ll need to restart the corresponding service for the drop-in to take effect. e.g.:

$ sudo systemctl restart systemd-resolved.service
1 Like

Right, on the logs, at some point it said "127.0.0.53:53: no such host" that was the nameserver configured on /etc/resolv.conf by oci.

Applying this solved my problem, thanks alot!

The edge router is up and running now, but the curl still isn't working, the good news is that I'm having another error now, which is curl: (56) Recv failure: Connection reset by peer.

On the public edge router logs these two logs :

Aug 08 21:29:39 public sh[4015]: {"error":"invalid edge router for session","file":"github.com/openziti/edge@v0.24.364/router/xgress_edge_tunnel/fabric.go:237","func":"github.com/openziti/edge/router/xgress_edge_tunnel.(*fabricProvider).TunnelService","level":"warning","msg":"failed to dial fabric","service":"hello.http","time":"2023-08-08T21:29:39.307Z"}

Aug 08 21:29:39 public sh[4015]: {"error":"invalid edge router for session","file":"github.com/openziti/edge@v0.24.364/tunnel/tunnel.go:50","func":"github.com/openziti/edge/tunnel.DialAndRun","level":"error","msg":"tunnel failed","service":"hello.http","time":"2023-08-08T21:29:39.307Z"}

Public Edge router seems to be able to dial hello.http and private router is able to bind hello.http :

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
OKAY : public_public_router (2) -> hello.http (1) Common Routers: (1/1) Dial: Y Bind: N 

OKAY : private_private_router (2) -> hello.http (1) Common Routers: (1/1) Dial: N Bind: Y 

ERROR: Default Admin 
  - Identity does not have access to any services. Adjust service policies.

There's nothing on the private router logs, it looks like the service did not Bind to that router.

When I nslookup hello.http.ziti, it says ** server can't find hello.http.ziti: NXDOMAIN :

nslookup hello.http.ziti
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   hello.http.ziti
Address: 100.64.0.1
** server can't find hello.http.ziti: NXDOMAIN

Is there something else that I’m missing or does that means that I misconfigured the DNS part ?

nslookup does that. Use dig instead of nslookup to test for service addresses. If that fails, then the DNS config isn’t correct, but with nslookup, you can’t tell.

1 Like

I would think this should work. Or dig, or ping. I don’t understand why nslookup is complaining that it can’t find hello.http.ziti AND it seems to have found an address for it (100.64.0.1). nslookup and dig play some tricks and don’t necessarily respect the host’s resolver configuration. System utilities like ping do.

You can force nslooup and dig to use a specific DNS server. Assuming your router DNS server is using the default address 127.0.0.1:53, you could also try this:

nslookup hello.http.ziti 127.0.0.1
dig hello.http.ziti @127.0.0.1

edit: Oh, and if the lookup still fails when you’re pointing directly at the DNS server, make sure the router is actually aware of the service. One way to do this is with the ziti CLI policy advisor command:

ziti edge policy-advisor services hello.http

And make sure the row for your intercepting router’s identity contains “Dial: Y”

1 Like

Even with dig I had the same problem.
I re-created new VMs and did a fresh install and re-configured everything and now it's working. Guess I must've missed something somewhere but can't find what.

Anyway, thanks everyone for your help !

hi there, that thread was extremely helpful.
I'd like to add that I had to edit /etc/systemd/system/ziti-router.service.d/override.conf so that the router could bind to port 53.

# allow binding low ports, e.g., 443/tcp; required when ZITI_ROUTER_MODE=tproxy or ZITI_ROUTER_PORT <= 1024
AmbientCapabilities=CAP_NET_BIND_SERVICE

# allow adding IP routes and iptables rules; required when ZITI_ROUTER_MODE=tproxy
AmbientCapabilities=CAP_NET_ADMIN

I've got it working that way.
I set up a router edge policy with both @public-router and @private-router as edge router roles for @someuser as identity roles.
And I set up a service edge router policy with both @public-router and @private-router as edge router roles and @theservice.svc as service roles.

Why do I have to list both routers here? Shouldn't it be enough to just list the "terminating" router here?

Edge Router Policies dictate which users can connect to a given set of routers. With a router acting as a tunneler, you'll get an edge router policy for free from the system. You would still need to authorize other identities to use one or more routers.

Service Edge Router Policies are different. They control both the place traffic is allowed to get onto the overlay, as well as the final location the traffic is destined for (nothing in-between, traffic can traverse any router that allows data to traverse it as needed). This means you need to grant a service an "onboard" router as well as a "final" router. That's why you need to have both listed.

In general, I strongly urge most people/new users to use an #all/#all service edge router policy at first. It's been demonstrated to be exceedingly easy to get wrong. :slight_smile: Edge router policies are historically easier for people to understand.

Hope that helps

Thanks, that’s helpful indeed :+1: