According to the tutorial Use a Router as a Local Gateway | OpenZiti Create local-router and remote-router, and after all routers are started, curl www.zftest.com:5000 in the local-router environment to display the following content in the local-router log. There are no error logs in the controller, router, and remote-router，ziti version is 1.1.11
Oct 08 07:02:37 routerserver ziti-router[3021]: {"file":"github.com/openziti/ziti/tunnel/intercept/tproxy/tproxy_linux.go:311","func":"github.com/openziti/ziti/tunnel/intercept/tproxy.(*tProxy).acceptTCP","level":"info","msg":"received connection: 100.64.0.1:5000 --\u003e 100.64.0.0:42292","time":"2024-10-08T07:02:37.044Z"}
Oct 08 07:02:37 routerserver ziti-router[3021]: {"ctrlId":"www.ziti-test-ctrl.com","error":"context deadline exceeded","file":"github.com/openziti/ziti/router/xgress_edge_tunnel/fabric.go:269","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*fabricProvider).tunnelServiceV1","level":"warning","msg":"failed to dial fabric","service":"zftest","time":"2024-10-08T07:02:37.044Z"}
Oct 08 07:02:37 routerserver ziti-router[3021]: {"error":"context deadline exceeded","file":"github.com/openziti/ziti/tunnel/tunnel.go:51","func":"github.com/openziti/ziti/tunnel.DialAndRun","level":"error","msg":"tunnel failed","service":"zftest","time":"2024-10-08T07:02:37.045Z"}
ziti fabric list links
│ 3g6nWN5YqxERvphBvZb4o2 │ local-router │ www.ziti-test-router.com │ 1 │ 3.0ms │ 2.9ms │ Connected │ up │ 6 │
│ 5R4cuJXHyjDOg19fRsZz3l │ remote-router │ www.ziti-test-router.com │ 1 │ 3.0ms │ 3.0ms │ Connected │ up │ 6
ziti-test-router is public router, not open tunnel
Hi,
It looks like the route creation or at least attempt across fabric is not happening. If you increase the log level to at least debug, any luck with logs?
I have set the log level for both local-router and remote-router to - v
remote-router
[ 180.010] DEBUG ziti/router/forwarder.(*Scanner).scan: scanning [0] circuits
[ 180.117] DEBUG ziti/router/state.(*ManagerImpl). StartRouterModelSave.func1.(*RouterDataModel). Save.1: could not save router data model, no index
local-router
[ 101.018] INFO ziti/tunnel/intercept/tproxy.(*tProxy).acceptTCP: received connection: 100.64.0.1:5000 --> 100.64.0.0:60886
[ 101.018] WARNING ziti/router/xgress_edge_tunnel.(*fabricProvider).tunnelServiceV1: {ctrlId=[www.ziti-test-ctrl.com] error=[context deadline exceeded] service=[zftest]} failed to dial fabric
[ 101.019] ERROR ziti/tunnel.DialAndRun: {error=[context deadline exceeded] service=[zftest]} tunnel failed
[ 120.010] DEBUG ziti/router/forwarder.(*Scanner).scan: scanning [0] circuits
[ 120.093] DEBUG ziti/router/state.(*ManagerImpl).StartRouterModelSave.func1.(*RouterDataModel).Save.1: could not save router data model, no index
Can you capture the tcpdump on the local router on the main interface while trying to access the service? one interface on this router?
I will try to use tcpdump to capture routing data in the Ubuntu environment
I want the entire process to follow Use a Router as a Local Gateway | OpenZiti I went to configure it. The entire process is created through zitiu_router_auto-enroll. I don't know if it's because of version v1.1.11. The video is from two years ago, so is it outdated
Next, I made the following attempts
Figure 1 is accessible
The flipping of Figure 2 and Figure 2 is not working, as the router cannot be accessed by either the client or server, just like the errors in local-router and remote-router
The thing is that it seems to be timing out trying to create circuit across the fabric, i.e. fabric route in the public router and then should attempt to dial the end service endpoint on the remote router. The end to end circuit can not be created successfully if these events are not completed. Logs suggest that is the case. The dev team is working on the HA and some of that logic is evolving. The controller is involved in this process, but there seems to be no logs on the controller either.
I was just trying to see with wireshark if your local router is even talking the controller, since there are no logs indicating that either. Did you try to restart the local router or reboot the host? By the way, the process of creating the router did not change much if at all.
Indeed, the communication between the route and the controller was not captured. I think the most critical thing should be the route yml. The following one is generated by ziti_router_auto_enroll. Is there any problem?
In local-router, you can see successfully connected to controller. In api sessions, you can see local-router, but you can't see dial record in sessions because curl www.zftest.com:5000 has reported an error.
v: 3
identity:
cert: /opt/openziti/ziti-router/certs/cert.pem
server_cert: /opt/openziti/ziti-router/certs/server_cert.pem
key: /opt/openziti/ziti-router/certs/key.pem
ca: /opt/openziti/ziti-router/certs/ca.pem
ctrl:
endpoint: tls:www.ziti-test-ctrl.com:6262
link:
dialers:
- binding: transport
edge:
heartbeatIntervalSeconds: 60
csr:
country: US
province: NC
locality: Charlotte
organization: NetFoundry
organizationalUnit: Ziti
sans:
dns:
- localhost
ip:
- 172.16.222.190
- 127.0.0.1
listeners:
- binding: edge
address: tls:0.0.0.0:443
options:
advertise: 172.16.222.190:443
- binding: tunnel
options:
mode: tproxy
resolver: udp://172.16.222.190:53
lanIf: ens160
dnsSvcIpRange: 100.64.0.0/10
This looks ok to me, even though I don’t see that many controllers fqdns with www as a hostname.
Can you also share the controller’s config?
v: 3
db: "/opt/ziti/related/www.ziti-test-ctrl.com/db/ctrl.db"
identity:
cert: "/opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-intermediate/certs/www.ziti-test-ctrl.com-client.cert"
server_cert: "/opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-intermediate/certs/www.ziti-test-ctrl.com-server.chain.pem"
key: "/opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-intermediate/keys/www.ziti-test-ctrl.com-server.key"
ca: "/opt/ziti/related/www.ziti-test-ctrl.com/pki/cas.pem"
ctrl:
options:
advertiseAddress: tls:www.ziti-test-ctrl.com:6262
listener: tls:0.0.0.0:6262
healthChecks:
boltCheck:
interval: 30s
timeout: 20s
initialDelay: 30s
edge:
api:
sessionTimeout: 30m
address: "www.ziti-test-ctrl.com:1280"
enrollment:
signingCert:
cert: /opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-signing-intermediate/certs/www.ziti-test-ctrl.com-signing-intermediate.cert
key: /opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-signing-intermediate/keys/www.ziti-test-ctrl.com-signing-intermediate.key
edgeIdentity:
duration: 180m
edgeRouter:
duration: 180m
web:
- name: client-management
bindPoints:
- interface: "[::]:1280"
address: www.ziti-test-ctrl.com:1280
identity:
ca: "/opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-intermediate/certs/www.ziti-test-ctrl.com-intermediate.cert"
key: "/opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-intermediate/keys/www.ziti-test-ctrl.com-server.key"
server_cert: "/opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-intermediate/certs/www.ziti-test-ctrl.com-server.chain.pem"
cert: "/opt/ziti/related/www.ziti-test-ctrl.com/pki/www.ziti-test-ctrl.com-intermediate/certs/www.ziti-test-ctrl.com-client.cert"
options:
idleTimeout: 5000ms #http timeouts, new
readTimeout: 5000ms
writeTimeout: 100000ms
minTLSVersion: TLS1.2
maxTLSVersion: TLS1.3
apis:
- binding: edge-management
options: { }
- binding: edge-client
options: { }
- binding: fabric
options: { }
Can you enable debug on the controller and capture logs when trying to dial the service?
Current test scenarios
no any log about local-router，only server client log，172.16.222.43为egress-tunnel
[ 8.652] DEBUG ziti/controller/events.(*entityChangeEventDispatcher).flushLoop: cleaning up entity change events
[ 8.653] DEBUG ziti/controller/events.(*entityChangeEventDispatcher).processPreviousTxEvents: {txId=[36328]} cleaning up entity change events for tx
[ 12.829] DEBUG transport/v2/tls.(*sharedListener).getConfig [tls:0.0.0.0:1280]: {client=[172.16.222.43:58150]} client requesting protocols = [http/1.1]
[ 12.829] DEBUG transport/v2/tls.(*sharedListener).getConfig [tls:0.0.0.0:1280]: {client=[172.16.222.43:58150]} found handler for proto
[ 12.860] DEBUG transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[172.16.222.43:58150] client=[172.16.222.43:58150]} selected protocol = 'http/1.1'
[ 12.861] DEBUG ziti/controller/internal/routes.(*PublicQueryOptions).getFullQuery: query: [QueryOption Predicate: 'true', Sort: '', Paging: '[Paging Offset: '0', Limit: '25', ReturnAll: 'false']']
[ 17.656] DEBUG ziti/controller/handler_edge_ctrl.(*listTunnelServicesHandler).listServices: {router=[local-router]} service list requested, but no update available
Can you reproduce this test scenario?
I didn't get the chance today because of time. I'll try now.
Followed the same guide with Host OpenZiti Anywhere for the network part, and my setup is working as expected.
Public Router, Controller and ZAC on the same VM
Curl to reach service
Is tnet-ctrl-edge-router a tunnel router? Why do we need an additional tunnel routing? Can I take a look at your edge routers and policy sections
The quickstart configured the public router to have tunneler mode enable to host but not intercept services. I disabled the edge to make it just a relay node. Users have flexibility as to the router configuration. Here is the config
v: 3
identity:
cert: "/home/ziggy/.ziti/quickstart/dariusz-tnet-ctrl/dariusz-tnet-ctrl-edge-router.cert"
server_cert: "/home/ziggy/.ziti/quickstart/dariusz-tnet-ctrl/dariusz-tnet-ctrl-edge-router.server.chain.cert"
key: "/home/ziggy/.ziti/quickstart/dariusz-tnet-ctrl/dariusz-tnet-ctrl-edge-router.key"
ca: "/home/ziggy/.ziti/quickstart/dariusz-tnet-ctrl/dariusz-tnet-ctrl-edge-router.cas"
#alt_server_certs:
# - server_cert: ""
# server_key: ""
ctrl:
endpoint: tls:dariusz-tnet-ctrl:8440
link:
dialers:
- binding: transport
listeners:
- binding: transport
bind: tls:0.0.0.0:10080
advertise: tls:ctrl01.centralus.cloudapp.azure.com:10080
options:
outQueueSize: 4
#listeners:
# bindings of edge and tunnel requires an "edge" section below
#- binding: edge
# address: tls:0.0.0.0:8442
# options:
# advertise: ctrl01.centralus.cloudapp.azure.com:8442
# connectTimeoutMs: 5000
# getSessionTimeout: 60
#- binding: tunnel
# options:
# mode: host #tproxy|host
#edge:
# csr:
# country: US
# province: NC
# locality: Charlotte
# organization: NetFoundry
# organizationalUnit: Ziti
# sans:
# dns:
# - localhost
# - ctrl01.centralus.cloudapp.azure.com
# - dariusz-tnet-ctrl
# ip:
# - "127.0.0.1"
# - "::1"
# - "20.46.253.218"
#transport:
# ws:
# writeTimeout: 10
# readTimeout: 5
# idleTimeout: 120
# pongTimeout: 60
# pingInterval: 54
# handshakeTimeout: 10
# readBufferSize: 4096
# writeBufferSize: 4096
# enableCompression: true
forwarder:
latencyProbeInterval: 0
xgressDialQueueLength: 1000
xgressDialWorkerCount: 128
linkDialQueueLength: 1000
linkDialWorkerCount: 32
Service still works. Links:
/opt/openziti/ziti-router/ziti fabric list links
╭───────────────────────┬───────────────────────┬───────────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID │ DIALER │ ACCEPTOR │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE │ STATUS │ FULL COST │
├───────────────────────┼───────────────────────┼───────────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ BE85PJfC2s8Z5Yz3fiQNW │ dariusz-remote-router │ dariusz-tnet-ctrl-edge-router │ 1 │ 25.8ms │ 28.2ms │ Connected │ up │ 54 │
│ zyAwFMYJSmMrHZxSgwgjy │ dariusz-local-router │ dariusz-tnet-ctrl-edge-router │ 1 │ 42.0ms │ 40.6ms │ Connected │ up │ 82 │
╰───────────────────────┴───────────────────────┴───────────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
results: 1-2 of 2
Circuit for ssh:
ziti fabric list circuits
╭───────────┬───────────────────────────┬─────────┬────────────────────────┬─────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ID │ CLIENT │ SERVICE │ TERMINATOR │ CREATEDAT │ PATH │
├───────────┼───────────────────────────┼─────────┼────────────────────────┼─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONUC72A6X │ cm241s5e002ytv2dt6aopfyu3 │ ssh │ 25vAusYBL0UQTbzwPK0gbe │ 2024-10-11 11:57:34 │ r/dariusz-local-router -> l/zyAwFMYJSmMrHZxSgwgjy -> r/dariusz-tnet-ctrl-edge-router -> l/BE85PJfC2s8Z5Yz3fiQNW -> r/dariusz-remote-router │
╰───────────┴───────────────────────────┴─────────┴────────────────────────┴─────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
results: 1-1 of 1
Service Terminators
ziti fabric list terminators
╭────────────────────────┬─────────┬───────────────────────┬─────────┬────────────────────────┬──────────┬──────┬────────────┬──────────────┬────────────╮
│ ID │ SERVICE │ ROUTER │ BINDING │ ADDRESS │ INSTANCE │ COST │ PRECEDENCE │ DYNAMIC COST │ HOST ID │
├────────────────────────┼─────────┼───────────────────────┼─────────┼────────────────────────┼──────────┼──────┼────────────┼──────────────┼────────────┤
│ 25vAusYBL0UQTbzwPK0gbe │ ssh │ dariusz-remote-router │ tunnel │ 25vAusYBL0UQTbzwPK0gbe │ │ 0 │ default │ 2 │ GP.py8BUcB │
╰────────────────────────┴─────────┴───────────────────────┴─────────┴────────────────────────┴──────────┴──────┴────────────┴──────────────┴────────────╯
results: 1-1 of 1
SP
RP
SEP
