What would openziti arch for this look like

[qrkourier]

Thanks for that information. That's the correct value for the controller's address. Let's also double-check the router configuration. I think you were able to enroll your tunneler successfully, and have only one Ziti router that was also successfully enrolled. Now your tunneler is failing to reach the edge listener provided by your router.

The edge listener is provided by the router on the router's advertised address, so the tunneler must be able to look up the router's address (FQDN) through the host's DNS resolver to find its current IP address. It must have an IP route to that address. Please ensure the router's address and port are not blocked by a firewall.

[sadath-12]

Checked , for simplicity I'm allowing All traffic .

Note : There is only one private router here , The IP address given to router here is the internal private IP of the machine residing in private subnet .

Just checking if we can still reach with just one private router

[qrkourier]

You can undoubtedly use Ziti with only private IP addresses. It will work if the tunneler, router, and controller can communicate.

Sometimes, we say "private router and "public" router. It's not a precise meaning, though. Typically, a "private" router has a private IP address, so it does not provide links to other routers, which it moot with only one router.

It will work if the tunneler can connect to the router's advertised address and the router and tunneler can connect to the controller's advertised address.

EDIT: say "tunneler" instead of "identity" to clarify

[sadath-12]

I'm also getting error

{"_context":"ch{edge}-\u003eu{classic}-\u003ei{zO50}","chSeq":2,"connId":0,"edgeSeq":0,"error":"service 5BxjZOIl576wSWUcl0PsM3 has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-06-04T15:41:51.136Z","token":"c0c6497b-afdd-476f-8b54-22674b793594","type":"EdgeConnectType"}

following the guide of k3ds deployment , last time did not see such errors

[sadath-12]

I even get

OKAY : router1 (1) -> hello-service (1) Common Routers: (1/1) Dial: N Bind: Y 

OKAY : hello-client (1) -> hello-service (1) Common Routers: (1/1) Dial: Y Bind: N

for the service and router

[qrkourier]

The error message indicates there are zero terminators for the dialed Ziti service.

error: service 5BxjZOIl576wSWUcl0PsM3 has no terminators

Let's check your Ziti service's policies.

ziti edge policy-advisor services -q 5BxjZOIl576wSWUcl0PsM3

The expected output is two lines beginning with OKAY. There should be an authorization each for bind and dial. It seems there is not yet an authorization for bind, which must be granted with a bind service policy.

You can create the bind service policy to grant permission for some identity to host (reverse proxy) the Ziti service.

For example, I will use your service's ID because it's available, but you may use the friendly name instead.

ziti edge create service-policy "my-service-bind-policy" Bind \
    --service-roles '@5BxjZOIl576wSWUcl0PsM3' --identity-roles '#my-service-hosts'

[qrkourier]

Thank you for that extra info. The router identity named "router1" has permission to host/bind the service.

I believe you have no terminators.

ziti edge list terminators

Let's also check your router policies. You can try relaxing the router policies if they do not already permit #all identities and #all services.

ziti edge list edge-router-policies;
ziti edge list service-edge-router-policies

[sadath-12]

have it like this

[sadath-12]

I actually followed this Kubernetes Service | OpenZiti .

I believe its referring to target service as terminator , I have deployed hello application as target along with ziti-service pointing to it

[qrkourier]

Is there an error message in the log for the router1 process deployed in your cluster? For some reason, it's failing to create the terminator.

[sadath-12]

Logs of the router1 pod

[qrkourier]

I have just followed the steps in the blog and first service tutorials and couldn't trigger the same problem.

There are four authorizations a Ziti service needs to function:

1. service (dial) policy allows a client identity (tunneler proxy or SDK)
2. service (bind) policy allows a hosting identity (tunneler reverse proxy or SDK)
3. service edge router policy allows the service to use one of the online, in-common routers
4. edge router policy allows the client and hosting identities to use one of the online, in-common routers

We've established these authorizations were granted, so it must be a connectivity or configuration problem for the router or controller, or both.

Let's check the controller log, too, for error messages, and dump the controller and router configuration YAML.

Will you post these as text instead of screenshots, if possible?

kubectl --namespace ziti logs --selector app.kubernetes.io/component=ziti-controller --tail=-1

kubectl --namespace ziti logs --selector app.kubernetes.io/component=ziti-router --tail=-1

kubectl --namespace ziti get configmap ziti-controller-config -o go-template='{{index .data "ziti-controller.yaml" }}'

kubectl --namespace ziti get configmap ziti-router-config -o go-template='{{index .data "ziti-router.yaml" }}'

You may find it helpful to filter the logs by level. Here's how I found only error level logs from the controller container. This is a common error to show the expected format. You need the jq command if not already installed.

kubectl logs \
--namespace=ziti \
--selector=app.kubernetes.io/component=ziti-controller \
--tail=-1 \
--container=ziti-controller \
| jq 'select(.level == "error")'

{
  "_context": "tls:0.0.0.0:1280",
  "error": "remote error: tls: bad certificate",
  "file": "github.com/openziti/transport/v2@v2.0.131/tls/listener.go:257",
  "func": "github.com/openziti/transport/v2/tls.(*sharedListener).processConn",
  "level": "error",
  "msg": "handshake failed",
  "remote": "10.42.0.1:36096",
  "time": "2024-06-04T16:14:56.700Z"
}

[sadath-12]

[sadath-12]

[sadath-12]

Let me know if I can be more clear thanks

[sadath-12]

I wish there are was more observability on the mesh to pin point errors

[qrkourier]

@sadath-12 It would help me a lot to have these as text, not images, especially since I need to parse the JSON to find interesting messages.

I suppose the terminal app might not be cooperating, making it difficult to capture the terminal scrollback on the clipboard.

When you run a command, you can redirect the output to a file with | tee ./log.txt or > ./config.yml , then you can paste the file's contents into the forum.

Here's one example.

kubectl get configmap ziti-router-config \
--output=go-template='{{index .data "ziti-router.yaml" }}' \
--namespace=ziti \
| tee ./ziti-router.yaml

This will create the YAML file you can then paste inside ```yaml fences in Discourse.

Regarding observability, I hope to find a clear error message. Did you find any interesting error level messages when you parsed the logs for the router and controller?

The handshake errors in the screenshot are common and can usually be ignored.

[sadath-12]

ok then let me try and find specific error messages and post it here

[sadath-12]

Note: All things are happening in same cluster right now which is k3d (just to figure out the issue)

[qrkourier]

From my computer, I can see the controller and router are responding on the expected ports with the correct protocols. This reduces the chance of a connectivity problem.