ohk , I'm able to ping but not curl
I think if you create an ec2 in aws install k3d , use public IP as NODE_IP and do all the stuffs listed in the guide in one machine (controller + router + tunneler) you must be able to produce this problem . I produced this for 3 times
Follow Deploy OpenZiti in Kubernetes with Ease Using k3d and Kubernetes Service | OpenZiti except that NODE_IP is the public IP of aws ec2 ubuntu , and then later you install tunnel in the same machine and try curl http://hello.ziti.internal and I did these all steps under "sudo su"
Since you are using EC2, does the Security Group allow IN for the TCP ports you used? I believe the blog post suggested 1280, 6262, 3022, 10080.
BTW, the NODE_IP example in the blog post is only used to construct a domain name (FQDN). You can use the EC2 instance's FQDN instead of an sslip.io name. I will try it in EC2 as well to see if there's any exception to the rule.
ya security rule I kept to allow all Traffic from everywhere
I'm trying it now live https://m.twitch.tv/qrkourier
sure thanks iam watching
I reproduced the problem on my EC2 instance. The controller, router, and client tunneler all report the "no terminators" error, but why the terminator was not created is unclear.
ubuntu@ip-172-31-12-201:~$ kubectl --namespace ziti logs --selector app.kubernetes.io/component=ziti-controller --tail=-1 | grep -E ERROR\|WARN
Defaulted container "ziti-controller" out of: ziti-controller, ziti-controller-init (init)
[71953.454] WARNING channel/v2.(*channelImpl).rxer [ch{Nh8CASMEq}->u{classic}->i{xVP7}]: dropped message. type [-33], sequence [-1], replyFor [-1]
[72047.213] ERROR ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError [ch{Nh8CASMEq}->u{classic}->i{xVP7}]: {token=[59dedbd9-9f8b-4f03-b9b2-4a1dadeb86cb] error=[service 4lM7cWz7Evumz5J8qgvuUG has no terminators] routerId=[Nh8CASMEq] operation=[create.circuit]} responded with error
ubuntu@ip-172-31-12-201:~$ kubectl --namespace ziti logs --selector app.kubernetes.io/component=ziti-router --tail=-1 | grep ERROR
[71317.742] ERROR ziti/router/state.(*ManagerImpl).LoadRouterModel: {error=[open /etc/ziti/config/ziti-router.yaml.json.gzip: no such file or directory]} could not load router model from file [/etc/ziti/config/ziti-router.yaml.json.gzip]
[71943.392] ERROR channel/v2.(*reconnectingImpl).Rx [u{reconnecting}->i{5on3}]: {error=[EOF]} rx error. closed peer and starting reconnection process
[71943.392] ERROR channel/v2.(*reconnectingDialer).Reconnect [u{reconnecting}->i{5on3} @tls:ec2-54-67-65-115.us-west-1.compute.amazonaws.com:6262]: unable to ping (use of closed network connection)
[71943.409] ERROR channel/v2.(*reconnectingDialer).Reconnect [u{reconnecting}->i{5on3} @tls:ec2-54-67-65-115.us-west-1.compute.amazonaws.com:6262]: reconnection attempt [#1] failed (EOF)
[71948.414] ERROR channel/v2.(*reconnectingDialer).Reconnect [u{reconnecting}->i{5on3} @tls:ec2-54-67-65-115.us-west-1.compute.amazonaws.com:6262]: reconnection attempt [#2] failed (EOF)
[71948.794] ERROR channel/v2.(*reconnectingImpl).Tx [u{reconnecting}->i{5on3}]: tx error (use of closed network connection). starting reconnection process
[71949.794] ERROR channel/v2.(*heartbeater).sendHeartbeat: {error=[timeout waiting for message to be written to wire: context deadline exceeded] channelId=[ch{ctrl}->u{reconnecting}->i{5on3}]} pulse failed to send heartbeat
ubuntu@ip-172-31-12-201:~$ kubectl --namespace ziti logs --selector app.kubernetes.io/component=ziti-router --tail=-1 | grep -E ERROR\|WARN
[71317.742] WARNING ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
[71317.742] ERROR ziti/router/state.(*ManagerImpl).LoadRouterModel: {error=[open /etc/ziti/config/ziti-router.yaml.json.gzip: no such file or directory]} could not load router model from file [/etc/ziti/config/ziti-router.yaml.json.gzip]
[71317.745] WARNING ziti/router/xlink_transport.loadListenerConfig: {addr=[tls:0.0.0.0:10080] error=[no network interface found for 0.0.0.0]} unable to get interface for address
[71943.392] ERROR channel/v2.(*reconnectingImpl).Rx [u{reconnecting}->i{5on3}]: {error=[EOF]} rx error. closed peer and starting reconnection process
[71943.392] ERROR channel/v2.(*reconnectingDialer).Reconnect [u{reconnecting}->i{5on3} @tls:ec2-54-67-65-115.us-west-1.compute.amazonaws.com:6262]: unable to ping (use of closed network connection)
[71943.409] ERROR channel/v2.(*reconnectingDialer).Reconnect [u{reconnecting}->i{5on3} @tls:ec2-54-67-65-115.us-west-1.compute.amazonaws.com:6262]: reconnection attempt [#1] failed (EOF)
[71948.414] ERROR channel/v2.(*reconnectingDialer).Reconnect [u{reconnecting}->i{5on3} @tls:ec2-54-67-65-115.us-west-1.compute.amazonaws.com:6262]: reconnection attempt [#2] failed (EOF)
[71948.794] ERROR channel/v2.(*reconnectingImpl).Tx [u{reconnecting}->i{5on3}]: tx error (use of closed network connection). starting reconnection process
[71949.794] ERROR channel/v2.(*heartbeater).sendHeartbeat: {error=[timeout waiting for message to be written to wire: context deadline exceeded] channelId=[ch{ctrl}->u{reconnecting}->i{5on3}]} pulse failed to send heartbeat
[72047.214] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{23dD}]: {error=[service 4lM7cWz7Evumz5J8qgvuUG has no terminators] token=[59dedbd9-9f8b-4f03-b9b2-4a1dadeb86cb] edgeSeq=[0] connId=[1] type=[EdgeConnectType] chSeq=[1]} failed to dial fabric
ubuntu@ip-172-31-12-201:~$ ziti tunnel proxy hello-service:8080 --identity ./hello-client.json --verbose --log-formatter pfxlog |& grep -E ERROR\|WARN
[72624.854] ERROR github.com/openziti/ziti/tunnel.DialAndRun: {service=[hello-service] error=[unable to dial service 'hello-service': dial failed: service 4lM7cWz7Evumz5J8qgvuUG has no terminators]} tunnel failed
@sadath-12 The problem was an incorrect helm command.
I added the missing value in the blog post: --set tunnel.mode="host".
The meaning of this input is to configure the router's tunneler. Tunnelers are proxies for Ziti services. In mode "host," the tunneler is only a reverse proxy.
You can upgrade your existing Helm release "ziti-router" with this new value.
helm get values "ziti-router" | tee ./ziti-router-values.yaml
Edit this file to add:
tunnel:
mode: host
Then upgrade your release with these values.
helm upgrade "ziti-router" openziti/ziti-router --values ./ziti-router-values.yaml
Finally, delete the old pod. A future chart improvement will do this automatically.
kubectl -n ziti delete pods --selector app.kubernetes.io/name=ziti-router
You can watch the new pod start running.
kubectl -n ziti get pods --watch
Then you should have a terminator.
ziti edge list terminators
And your service will work!
that was interesting session @qrkourier thanks ill have a look
thanks @qrkourier , the parameters of tunnel of not picked up by chart , but manually updating through file as you mentioned works.
A question:
I thought when running ziti edge create edge-router "router1" -t -o ./router1.jwt
the "-t" thing creates the tunnel proxy by itself since its mentioned at the guide
Correct. Both are necessary:
- administratively create the router with
-tor--tunneler-enabled(authorize this router to use any tunnel mode) - the router's config.yml must declare a tunnel mode
For Ziti routers in K8s, you would declare the tunnel mode as a Helm input value:
tunnel:
mode: host # tproxy, proxy, host, or none
@qrkourier when installing router via linux service how to cleanup in such a way if I install again it should ask for values in console as prompts
I think you're installing a Ziti router as a Linux service and want to change the installed router's configuration.
Modify the generated configuration file /var/lib/ziti-router/config.yml and run sudo systemctl restart ziti-router.service to load the changes.
For example, you can modify your config.yml like this to set the tunnel most to "host."
listeners:
- binding: tunnel
options:
mode: host
If you want to destroy the Linux router service's configuration you can follow the steps in the Uninstall section of the Linux router guide. Then, you will need to create a new router entity with the CLI or console and reinstall openziti-router.
@qrkourier I installed ziti-console in private eks via loadbalancer deployed in public subnets , sg is to allow everything .
When I access the URL i see a blank white page
I understand you see a white screen when you point your web browser to the load balancer you provisioned for the ziti-console service.
It sounds like your approach should work, so there must be a configuration or connectivity problem.
I will assume it is not connectivity because you mentioned it is public and unrestricted.
You will have to analyze the HTTP request with a command-line client like cURL or HTTPie.
This will help you to distinguish between an incorrect DNS name, closed port, TLS error, or HTTP error code.
Hi @qrkourier , I am not able to run the tunnel with trino pods please have a look at this weird error messages , but I tested the sidecar connection via dummy pod (apache container) and it works
I think you're following the sidecar proxy example and encountered a permission error from the ziti-tunnel container in the pod.
Did you see the special permissions in that example?
This one is always required: grant kernel capability NET_ADMIN to allow creating TPROXY rules in the pod's network namespace
Additionally, when you have granted the necessary capability, you must configure the pod to have the loopback IP as first DNS resolver. The second IP is the cluster resolver and can usually be found this way:
kubectl get services \
--namespace kube-system \
--selector=k8s-app=kube-dns \
--output go-template='{{range .items}}{{ .spec.clusterIP }}{{"\n"}}{{end}}'
ya it was securityContext related