Access Multiple Private AKS Clusters in Azure using one ziti service

dariuszSki · September 7, 2022, 9:33pm

I want to leverage the addressable terminators to access 2 or more AKS clusters in different Azure regions using one ziti service. Is this doable today?
Service Architecture:
ZDE1 → fabric → Edge Router with T option Region1 → Private AKS API Server (dns name)
ZDE1 → fabric → Edge Router with T option Region2 → Private AKS API Server (dns name)
etc

If it is, can sample configs be provided? Preferably using the dns name to address each API server separately.

TheLumberjack · September 7, 2022, 9:43pm

Addressable terminators work best with ziti-aware client applications like zssh. If you’re not using a ziti-aware client, such as a tunneler, then you’ll lose the benefits of addressable terminators. Zssh has been purpose-built to understand and interact natively with an OpenZiti overlay. Tunnelers were specifically built to interact with the underlay and “non-ziti-aware” traffic…

That means that tunnelers are not currently able to do the sort of translations necessary to use addressable terminators. If you used addressable terminators, you’ll have to make two services for that to work in a tunneler, and at that point, you’ve lost the beauty of addressable terminators so it’s not worth it. It might be an interesting tunneler feature to add onto tunnelers to add “identities” to the internal DNS server, or something like that…

It looks like you’re trying to use some “api server” (http or other) so I doubt you’re trying to use ssh/zssh. If you were trying to use ssh, then you can use zssh instead and then you would get the benefit of addressable terminators.

I think there might be an interesting feature request in here though…

dariuszSki · September 7, 2022, 10:10pm

I may be wrong here. Correct me if I am, but I thought ZDE is a ziti-aware client. For VOIP, we had to use identity name as IP address to leverage the addressable terminator and dial by intercepted IP/PORT. I want to dial by intercepted DNS name, but I don’t want to name my hosted identity as the dialed DNS name. I want to add it as a tag or app data {key: value} if that make any sense ? Is this possible today?

"identity": {
  "description": "Associate the hosting terminator with the specified identity. '$tunneler_id.name' resolves to the name of the hosting tunneler's identity. '$tunneler_id.tag[tagName]' resolves to the value of the 'tagName' tag on the hosting tunneler's identity.",
   "type": "string"
 },

dariuszSki · September 7, 2022, 10:16pm

Robert said it is doable. I am trying to find out the config details.

something like this I think:

  "dialOptions": {
    "identity": "$dst_name"
  }

Intercepted Side

{
  "addresses": [
    "dariusz-kube01-dns-f14c920d.21dc697f-f076-4df2-8220-90210ffbe23c.privatelink.westus.azmk8s.io",
    "dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io"
  ],
  "protocols": [
    "tcp"
  ],
  "portRanges": [
    {
      "low": 443,
      "high": 443
    }
  ],
  "dialOptions": {
    "identity": "$dst_name"
  }
}

Not sure if correct?

TheLumberjack · September 7, 2022, 10:21pm

Ahhh… Yeah now that you mention it, I do remember something along these lines. I think it might be $tunneler_id.name that you’re looking for?

dariuszSki · September 7, 2022, 10:22pm

Hosted Side:

{
  "forwardProtocol": true,
  "allowedProtocols": [
    "tcp"
  ],
  "forwardAddress": true,
  "allowedAddresses": [
     "dariusz-kube01-dns-f14c920d.21dc697f-f076-4df2-8220-90210ffbe23c.privatelink.westus.azmk8s.io",
     "dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io"
  ],
  "forwardPort": true,
  "allowedPortRanges": [
    {
      "low": 443,
      "high": 443
    }
  ],
  "listenOptions": {
    "identity": "$tunneler_id.name"
  },
}

?

dariuszSki · September 7, 2022, 10:23pm

Trying to validate the configs if correct?

TheLumberjack · September 7, 2022, 10:25pm

looks like it is $dst_ip and $dst_hostname on the intercept side. I’m looking at the source in ziti_tunnel_cbs.c

github.com

openziti/ziti-tunnel-sdk-c/blob/main/lib/ziti-tunnel-cbs/ziti_tunnel_cbs.c#L244


      
                      app_data->dst_hostname = strdup(dst_hostname);
                  }
              }
          }
          if (client != NULL) {
              parse_socket_address(client, &app_data->src_protocol, &app_data->src_ip, &app_data->src_port);
          }
          if (source_ip != NULL && *source_ip != 0) {
              const ziti_identity *zid = ziti_get_identity(ziti_ctx);
              strncpy(source_addr, source_ip, sizeof(source_addr));
              string_replace(source_addr, sizeof(source_addr), "$tunneler_id.name", zid->name);
              string_replace(source_addr, sizeof(source_addr), "$dst_ip", app_data->dst_ip);
              string_replace(source_addr, sizeof(source_addr), "$dst_port", app_data->dst_port);
              string_replace(source_addr, sizeof(source_addr), "$src_ip", app_data->src_ip);
              string_replace(source_addr, sizeof(source_addr), "$src_port", app_data->src_port);
              app_data->source_addr = source_addr;
          }
          ssize_t json_len = tunneler_app_data_to_json_r(app_data, MODEL_JSON_COMPACT, buf, bufsz);
          
          
// value points to stack buffer
          app_data->source_addr = NULL;

and

github.com

openziti/ziti-tunnel-sdk-c/blob/main/lib/ziti-tunnel-cbs/ziti_tunnel_cbs.c#L344


      
                      if (app_data.dst_protocol != NULL) {
                          string_replace(resolved_dial_identity, sizeof(resolved_dial_identity), "$dst_protocol", app_data.dst_protocol);
                      }
                      if (app_data.dst_ip != NULL) {
                          string_replace(resolved_dial_identity, sizeof(resolved_dial_identity), "$dst_ip", app_data.dst_ip);
                      }
                      if (app_data.dst_port != NULL) {
                          string_replace(resolved_dial_identity, sizeof(resolved_dial_identity), "$dst_port", app_data.dst_port);
                      }
                      if (app_data.dst_hostname != NULL){
                          string_replace(resolved_dial_identity, sizeof(resolved_dial_identity), "$dst_hostname", app_data.dst_hostname);
                      }
                  free_tunneler_app_data(&app_data);
                  }
              }
              dial_opts.identity = resolved_dial_identity;
          }
          
          

          
dial_opts.app_data_sz = (size_t) json_len;
          dial_opts.app_data = app_data_json;

dariuszSki · September 7, 2022, 11:31pm

On the hosted side, if I use this (ER with T)

"listenOptions": {
              "identity": "$tunneler_id.name"
          }

It resolves to the identity name.

ziggy@oci-dariusz-privatedcmgt-nc:~$ ziti fabric list terminators "limit 20" | grep Azure-AKS-Clusters-APIs
│ 4eHlEsmDvm8XAYj1prUYWU │ Azure-AKS-Clusters-APIs │ nf-er-eastus2-02     │ tunnel    │ e62237e3-5267-4ced-8673-d8a0deb6d91b        │ nf-er-eastus2-02    │      │ default    │              │ cl7s3vco79derdu5a3x9p7o2y │
│ 8UcYwiPrPxrRz6n6bupJ2  │ Azure-AKS-Clusters-APIs │ nf-er-westus-01      │ tunnel    │ 8f35ce47-03cd-441c-b311-83f5d8542b6d        │ nf-er-westus-01     │      │ default    │              │ cl7s3vayn9deodu5acklj75tt │
ziggy@oci-dariusz-privatedcmgt-nc:~$

But if I want to use this

"listenOptions": {
              "identity": "$tunneler_id.tag[fqdn]"
          }

it does not resolve to the value of the tag.
Is this supported today?

"tags": {
    "fqdn": "dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io"
  },

TheLumberjack · September 8, 2022, 12:19am

I don’t see anywhere in the code where tags are processed like that. I’m gunshy now to say ‘no’ since I had forgotten about the other variables that it does support, but I didn’t see anything like that in the source code scrounging i did. @scareything will probably have to comment on this, if nobody else does.

dariuszSki · September 8, 2022, 12:27am

This works in my set up, i.e. WDE → ERwT.

ziggy@oci-dariusz-privatedcmgt-nc:~$ ziti edge list terminators "limit 40" | grep Azure-AKS-Clusters-APIs
│ 5PAl5A3pv3k7tBBqsggi7l │ Azure-AKS-Clusters-APIs │ dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io │ tunnel    │ 450813fe-cd87-432f-8d09-9d6b435125e7        │ dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io │    0 │ default    │            0 │
│ lj7nDZnKV0gk4hZhf9LKt  │ Azure-AKS-Clusters-APIs │ dariusz-kube01-dns-f14c920d.21dc697f-f076-4df2-8220-90210ffbe23c.privatelink.westus.azmk8s.io  │ tunnel    │ 6329a305-4318-43bc-8411-bdc122f86c7e        │ dariusz-kube01-dns-f14c920d.21dc697f-f076-4df2-8220-90210ffbe23c.privatelink.westus.azmk8s.io  │    0 │ default    │            0 │
ziggy@oci-dariusz-privatedcmgt-nc:~$

host.v1
{
    "allowedAddresses": [
        "dariusz-kube01-dns-f14c920d.21dc697f-f076-4df2-8220-90210ffbe23c.privatelink.westus.azmk8s.io",
        "dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io"
    ],
    "allowedPortRanges": [
        {
            "high": 443,
            "low": 443
        }
    ],
    "allowedProtocols": [
        "tcp"
    ],
    "forwardAddress": true,
    "forwardPort": true,
    "forwardProtocol": true,
    "listenOptions": {
        "identity": "$tunneler_id.name"
    }
}

intercept.v1
{
    "addresses": [
        "dariusz-kube01-dns-f14c920d.21dc697f-f076-4df2-8220-90210ffbe23c.privatelink.westus.azmk8s.io",
        "dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io"
    ],
    "dialOptions": {
        "identity": "$dst_hostname"
    },
    "portRanges": [
        {
            "high": 443,
            "low": 443
        }
    ],
    "protocols": [
        "tcp"
    ]
}

dariuszSki · September 8, 2022, 12:29am

I had to rename my ERs to the dns name though, not ideal. Also, had to restart them as well at least in my case. I rather to use tag or app-data to advertise it as listenOptions. It would be @plorenz, since it is ER on the hosting side.

dariuszSki · September 8, 2022, 12:44am

PublicMEC$ kubectl config use-context dariusz-kube01
Switched to context "dariusz-kube01".
PublicMEC$ kubectl get nodes
NAME                                STATUS   ROLES   AGE     VERSION
aks-agentpool-13536061-vmss000000   Ready    agent   5h22m   v1.22.6
PublicMEC$ kubectl config use-context dariusz-kube02
Switched to context "dariusz-kube02".
PublicMEC$ kubectl get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-agentpool-37721437-vmss000000   Ready    agent   12h   v1.22.6
PublicMEC$

[2022-09-08T00:38:48.520Z]    INFO tunnel-cbs:ziti_dns.c:465 format_resp() found record[100.64.0.66] for query[1:dariusz-kube01-dns-f14c920d.21dc697f-f076-4df2-8220-90210ffbe23c.privatelink.westus.azmk8s.io]
[2022-09-08T00:38:53.942Z]    INFO tunnel-cbs:ziti_dns.c:465 format_resp() found record[100.64.0.65] for query[1:dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io]

TheLumberjack · September 8, 2022, 1:02am

sweet! I learned a little something myself today!

dariuszSki · September 8, 2022, 1:19pm

Actually, appData works as well. I just needed to restart the routers for it to be resolved.

"appData": {
    "fqdn": "dariusz-kube02-dns-66779d65.984d9e6e-39e6-4a62-be6c-9563b41fe154.privatelink.eastus2.azmk8s.io"
  },

"listenOptions": {
    "identity": "$tunneler_id.appData[fqdn]"
  },

av-dev · September 8, 2022, 2:12pm

This looks like a great use case. Can I request a blog / detailed steps to do this? I am hoping this is transferable to other hosted Kubernetes like EKS.

dariuszSki · September 8, 2022, 2:54pm

Yes, it should work with EKS. I will write something up with more details.

Topic		Replies	Views
Domain, subdomain or/and subnet as dial options in the intercept config Ziti Edge Tunnel	9	457	October 27, 2022
Zssh service hosting Zitifications	15	779	April 13, 2022
Tips to integrate Ziti with a self hosted Git repo Building/Development	2	237	August 23, 2022
Service optimization question Ziti Edge Tunnel	14	341	December 3, 2022
Address multi-hosted service on specific host General Questions	21	59	September 12, 2024

Access Multiple Private AKS Clusters in Azure using one ziti service

Related topics