Hi all, question. How would you do the following?
When the edge client is connected, I’d like to forward dns requests to another server (ie internal domain controller). This would allow clients to do active directory stuff like joining domains etc just by having the ziti client connected. I was hoping I could set up a rule to say anything on port 53, send here. But can’t seem to get that to work. Any ideas?
Thanks
Actually - this is what I was actually wanting to point to: Conneting Remote Endpoints with a On-Prem AD - #6 by emoscardini
thank you @gooseleggs!, I had found an article similar on netfoundry. It seems to resolve with the SRV records, but things like gpupdate don’t work yet…will do some wiresharking
@cjpit, welcome to OpenZiti! And thanks for the assist @gooseleggs. Make sure you get an updated Wireshark. A while back (6mo ish I think) they updated it to work with WinTUN adapaters - you’ll need that to see any packets.
If we can help, let us know.
Well, I’m not sure what I’ve done now but keep getting refused for the dns requests…
It did work for a little bit, then I went to cleaning up some of my policies, and alas I can’t get it to work.
this is the wireshark output
1 0.000000 100.64.0.1 100.64.0.2 DNS 76 Standard query 0xc413 SRV _ldap._tcp.dc._msdcs.XXX.local
2 0.000291 100.64.0.2 100.64.0.1 DNS 87 Standard query response 0xc413 Refused SRV _ldap._tcp.dc._msdcs.XXX.local OPT
I’m just trying to get the dns to respond (works fine on the private router and resolves as expected)
intercept config:
{
"addresses":[
"*.XXX.local"],
"dialOptions":{
"identity":""
},
"portRanges":[
{
"high":53,
"low":53
}],
"protocols":[
"tcp",
"udp"],
"sourceIp":""
}
host config
{
"allowedAddresses":[
"*.XXX.local"],
"allowedPortRanges":[
{
"high":53,
"low":53
}],
"allowedProtocols":[
"tcp",
"udp"],
"allowedSourceAddresses":[
],
"forwardAddress":true,
"forwardPort":true,
"forwardProtocol":true
}
Now, If I just hit it to resolve something internally to the network, and not being used by any ziti tunnels, it resolves…
Resolve-DnsName abc.XXX.local
Name Type TTL Section IPAddress
---- ---- --- ------- ---------
abc.XXX.local A 60 Answer 100.64.0.17
but if I try and get the SRV record
Resolve-DnsName _ldap._tcp.dc._msdcs.XXX.local -Type SRV
Resolve-DnsName : _ldap._tcp.dc._msdcs.XXX.local : DNS operation refused
At line:1 char:1
+ Resolve-DnsName _ldap._tcp.dc._msdcs.XXX.local -Type SRV
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (_ldap._tcp.dc._msdcs.XXX.local:String) [Resolve-DnsName], Win32Exc
eption
+ FullyQualifiedErrorId : RCODE_REFUSED,Microsoft.DnsClient.Commands.ResolveDnsName
pretty weird?
Hi @cjpit - Did you add the DNS sever IP in the ER that is the terminator for the AD? Since you are familiar with cloudziti ( NetFoundry), refer the steps here - https://support.netfoundry.io/hc/en-us/articles/9850601290381-Join-Active-Directory-Domain-Controller-from-windows-machine-running-WDE
If the windows device that is trying to join the domain is behind a customer ER - https://support.netfoundry.io/hc/en-us/articles/9877526824589-How-to-Join-Active-Directory-Domain-Controller-via-Customer-Edge-Router
Thank you @surennaidu , yeah I had seen those articles, I think however I’m running into a bug or something weird on the windows tunneller client. Even if i have my edge router offline, it does the same thing on the client.
For example, with the *.XXX.local, basically any dns request will always return with a ziti ip inside , even if that doesnt even exist really. However the SRV request always is refused. That leads me to believe I’ve hit some type of bug on the client…
Ok, anything with an underscore or Type SRV fails…see below…
Resolve-DnsName anythingatall.XXX.local
Name Type TTL Section IPAddress
---- ---- --- ------- ---------
anythingatall.XXX.local A 60 Answer 100.64.0.38
Resolve-DnsName _anythingatall.XXX.local
Resolve-DnsName : _anythingatall.XXX.local : DNS operation refused
At line:1 char:1
+ Resolve-DnsName _anythingatall.XXX.local
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (_anythingatall.XXX.local:String) [Resolve-DnsName], Win32Exception
+ FullyQualifiedErrorId : RCODE_REFUSED,Microsoft.DnsClient.Commands.ResolveDnsName
Resolve-DnsName anythingatall.XXX.local -Type SRV
Resolve-DnsName : anythingatall.XXX.local : DNS operation refused
At line:1 char:1
+ Resolve-DnsName anythingatall.XXX.local -Type SRV
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (anythingatall.XXX.local:String) [Resolve-DnsName], Win32Exception
+ FullyQualifiedErrorId : RCODE_REFUSED,Microsoft.DnsClient.Commands.ResolveDnsName
Hi @cjpit - “-” is allowed but “_” is not allowed in DNS names. Were you able to get your windows machine join the AD?
The use of underscores in hostnames violates RFC 1123’s valid defined characters for a domain name.
here is a bit of debug info from the client when using the _underscore it doesn’t find the matching domain
[2022-11-29T06:47:39.343Z] DEBUG tunnel-sdk:tunnel_udp.c:252 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:51931] service[ziti:dns-resolver]
[2022-11-29T06:47:39.343Z] DEBUG tunnel-cbs:ziti_dns.c:238 on_dns_client() new DNS client
[2022-11-29T06:47:39.343Z] DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:51931] service[ziti:dns-resolver]
[2022-11-29T06:47:39.343Z] VERBOSE tunnel-sdk:tunnel_udp.c:100 on_udp_client_data() 40 bytes from 100.64.0.1:51931
[2022-11-29T06:47:39.343Z] DEBUG tunnel-cbs:ziti_dns.c:341 ziti_dns_lookup() matching domain[*.xxx.local] found for asdfasdfadsf.XXX.local
[2022-11-29T06:47:39.343Z] INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry asdfasdfadsf.xxx.local -> 100.64.0.41
[2022-11-29T06:47:39.343Z] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.41] for query[1:asdfasdfadsf.XXX.local]
[2022-11-29T06:47:39.344Z] DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:51931] service[ziti:dns-resolver]
[2022-11-29T06:47:39.344Z] DEBUG tunnel-sdk:tunnel_udp.c:112 tunneler_udp_close() closing ziti:dns-resolver session
[2022-11-29T06:47:39.344Z] DEBUG tunnel-sdk:tunnel_udp.c:252 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:50819] service[ziti:dns-resolver]
[2022-11-29T06:47:39.344Z] DEBUG tunnel-cbs:ziti_dns.c:238 on_dns_client() new DNS client
[2022-11-29T06:47:39.344Z] DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:50819] service[ziti:dns-resolver]
[2022-11-29T06:47:39.344Z] VERBOSE tunnel-sdk:tunnel_udp.c:100 on_udp_client_data() 40 bytes from 100.64.0.1:50819
[2022-11-29T06:47:39.344Z] DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:50819] service[ziti:dns-resolver]
[2022-11-29T06:47:39.344Z] DEBUG tunnel-sdk:tunnel_udp.c:112 tunneler_udp_close() closing ziti:dns-resolver session
[2022-11-29T06:47:44.847Z] DEBUG tunnel-sdk:tunnel_udp.c:252 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:51931] service[ziti:dns-resolver]
[2022-11-29T06:47:44.847Z] DEBUG tunnel-cbs:ziti_dns.c:238 on_dns_client() new DNS client
[2022-11-29T06:47:44.847Z] DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:51931] service[ziti:dns-resolver]
[2022-11-29T06:47:44.847Z] VERBOSE tunnel-sdk:tunnel_udp.c:100 on_udp_client_data() 41 bytes from 100.64.0.1:51931
[2022-11-29T06:47:44.847Z] WARN tunnel-cbs:ziti_dns.c:331 ziti_dns_lookup() invalid host lookup[_asdfasdfadsf.XXX.local]
[2022-11-29T06:47:44.847Z] DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:51931] service[ziti:dns-resolver]
[2022-11-29T06:47:44.847Z] DEBUG tunnel-sdk:tunnel_udp.c:112 tunneler_udp_close() closing ziti:dns-resolver session
[2022-11-29T06:47:44.847Z] DEBUG tunnel-sdk:tunnel_udp.c:252 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:50819] service[ziti:dns-resolver]
[2022-11-29T06:47:44.847Z] DEBUG tunnel-cbs:ziti_dns.c:238 on_dns_client() new DNS client
[2022-11-29T06:47:44.847Z] DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:50819] service[ziti:dns-resolver]
[2022-11-29T06:47:44.847Z] VERBOSE tunnel-sdk:tunnel_udp.c:100 on_udp_client_data() 41 bytes from 100.64.0.1:50819
[2022-11-29T06:47:44.847Z] WARN tunnel-cbs:ziti_dns.c:331 ziti_dns_lookup() invalid host lookup[_asdfasdfadsf.XXX.local]
[2022-11-29T06:47:44.847Z] DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:50819] service[ziti:dns-resolver]
[2022-11-29T06:47:44.847Z] DEBUG tunnel-sdk:tunnel_udp.c:112 tunneler_udp_close() closing ziti:dns-resolver session
@surennaidu that’s strange then and why it’s not working, how can it resolve the AD domain servers then?
The first step of joining a domain, the client goes looking for _ldap._tcp.dc._msdcs.XXX.local -Type SRV to find who the domain controllers are. It actually throws that error in the windows popup on joining the domain saying it can’t find that record.
@emoscardini notes this in Conneting Remote Endpoints with a On-Prem AD - #8 by emoscardini
He notes that this is passed through, but I don’t seem to see that happening?
Ok @cjpit - I see in your service config that you have allowed port 53. Is that the only port you have listed? Can you add the other ports listed in Conneting Remote Endpoints with a On-Prem AD - #8 by emoscardini or the range 1-65535 for testing?
@surennaidu Yeah I’ve tried that and have re-enabled with full open ports.
Here is the log for the Resolve-DnsName _ldap._tcp.dc._msdcs.XXX.local
[2022-11-29T07:30:51.106Z] DEBUG tunnel-sdk:tunnel_udp.c:252 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:49222] service[ziti:dns-resolver]
[2022-11-29T07:30:51.106Z] DEBUG tunnel-cbs:ziti_dns.c:238 on_dns_client() new DNS client
[2022-11-29T07:30:51.106Z] DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:49222] service[ziti:dns-resolver]
[2022-11-29T07:30:51.106Z] VERBOSE tunnel-sdk:tunnel_udp.c:100 on_udp_client_data() 63 bytes from 100.64.0.1:49222
[2022-11-29T07:30:51.106Z] DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:49222] service[ziti:dns-resolver]
[2022-11-29T07:30:51.106Z] DEBUG tunnel-sdk:tunnel_udp.c:112 tunneler_udp_close() closing ziti:dns-resolver session
@cjpit - And you confirm that
and
Hi @cjpit
Just reviewing this thread. Figured I’d show you what it looks like for a successful SRV lookup from the logs:
[2022-11-29T09:20:35.835Z] INFO tunnel-cbs:ziti_dns.c:641 proxy_domain_req() writing proxy resolve [{
"status":0,
"id":2075,
"recursive":0,
"question":[{
"name":"_ldap._tcp.dc._msdcs.ziti.contoso.io",
"type":33
}]
}]
[2022-11-29T09:20:35.896Z] INFO tunnel-cbs:ziti_dns.c:590 on_proxy_connect() proxy resolve connection established for domain[*.ziti.contoso.io]
[2022-11-29T09:20:35.896Z] INFO tunnel-cbs:ziti_dns.c:627 on_proxy_write() proxy resolve write: 126
[2022-11-29T09:20:35.912Z] INFO tunnel-cbs:ziti_dns.c:600 on_proxy_data() proxy resolve: {"id":2075,"status":0,"question":[{"name":"_ldap._tcp.dc._msdcs.ziti.contoso.io","type":33}],"answer":[{"name":"_ldap._tcp.dc._msdcs.ziti.contoso.io","type":33,"ttl":86400,"data":"advm.ziti.contoso.io.","port":389,"weight":100}]}
[2022-11-29T09:20:35.912Z] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[advm.ziti.contoso.io.] for query[33:_ldap._tcp.dc._msdcs.ziti.contoso.io]
Unfortunately if the above is not successful, nothing is logged. So you will not see any other information for this, even in verbose logging level. Maybe we should open a ticket for this, since it would be helpful to see the failure in the logs.
For troubleshooting I would recommend checking if other services are working first, something like are you able to RDP to DC? The reason this is important is the network needs to be functional for this entire process to work. Also, as I mentioned a few times in that original thread, the egressing identity(router/tunnel) needs to be able to actually resolve the name. If that egress is unable to find the record, you get the silent failure.
Hope this helps.
@emoscardini ah great, at least that makes sense as to what is happening with the logging. I have other services working fine through that tunnel, http and rdp on other XXX.local addresses. I wondered if they may have taken precendence in it, so I set up just a single identity with that one service. But doesn’t seem to make a differece. The private router can resolve no problems, eg
nslookup -Type=SRV _ldap._tcp.dc._msdcs.XXX.LOCAL
Server: 10.1.0.30
Address: 10.1.0.30#53
_ldap._tcp.dc._msdcs.XXX.LOCAL service = 0 100 389 dc001.xxx.local.
_ldap._tcp.dc._msdcs.XXX.LOCAL service = 0 100 389 dc003.xxx.local.
I note that @surennaidu was linking to the netfoundry setup, and I have been trying out the Naas version as well, but this is with the community/core version, so not sure if that makes a difference.
Note above that it doesn’t have 127.0.0.1 as the first resolving for the private edge router. It’s hitting the dns configured against the static ip. I think this is fine from what I read on other threads from what I can tell, and the fact that other services I have set up are using this and working fine.
Would you expect some type of logging to be visible at the private router? To me it doesn’t seem to get that far through the fabric, but I could be wrong?
The only things that would be different are some terminology and UI. The 'version' of OpenZiti is the same whether you host it yourself or let NetFoundry host/manage/maintain it for you. So no, that should not matter.
@emoscardini, @cjpit should be able to resolve the DNS name from the far end too right? Should we try resolving it from the client? Nevermind - I see some of that done up above, my bad
Hi @cjpit
Yes, you should be able to see if the traffic is reaching other side.
The egressing ER should log something like this:
[2022-11-29 11:24:43.434] INFO github.com/openziti/edge/tunnel/intercept.(*resolvConn).Write: {name=[_ldap._tcp.dc._msdcs.ziti.contoso.io] service=[ADService] type=[33]} resolving
You can also use something like tcpdump to check if the traffic is working:
sudo tcpdump port 53 and host 10.0.0.4
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:21:44.292433 IP erVM.33433 > 10.0.0.4.domain: 51401+ [1au] SRV? _ldap._tcp.dc._msdcs.ziti.contoso.io. (65)
11:21:44.293313 IP erVM.57436 > 10.0.0.4.domain: 6537+ [1au] PTR? 4.0.0.10.in-addr.arpa. (50)
11:21:44.294096 IP 10.0.0.4.domain > erVM.33433: 51401* 1/0/2 SRV advm.ziti.contoso.io.:389 0 100 (121)
11:21:44.366881 IP 10.0.0.4.domain > erVM.57436: 6537 NXDomain 0/1/1 (127)
We should at least be able to narrow this issue down to the ingress configuration if you don’t see either of the above on the egressing side.
I tried to using the configurations you posted, but the validation is failing since some of the fields require values(host config):
"error": {
"cause": {
"field": "allowedSourceAddresses",
"reason": "allowedSourceAddresses is invalid: allowedSourceAddresses: Array must have at least 1 items",
"value": "[]"
},
"code": "COULD_NOT_VALIDATE",
"message": "The supplied request contains an invalid document or no valid accept content were available, see cause",
"requestId": "6N89G5H1F"
},
"meta": {
"apiEnrollmentVersion": "0.0.1",
"apiVersion": "0.0.1"
}
}
If you don’t mind, can you post your current host & intercept configurations using the command:
ziti edge list configs -j 'name="the name of your configuration"'
or just list all of them like this:
ziti edge list configs -j
@emoscardini sure here they are, yeah sorry i was just getting the json from the web console…
"data": [
{
"_links": {
"self": {
"href": "./configs/7jFCIuWEQwGRJCa1u5Van3"
}
},
"createdAt": "2022-11-28T07:10:18.062Z",
"id": "7jFCIuWEQwGRJCa1u5Van3",
"tags": {},
"updatedAt": "2022-11-29T07:34:57.877Z",
"configType": {
"_links": {
"self": {
"href": "./config-types/NH5p4FpGR"
}
},
"entity": "config-types",
"id": "NH5p4FpGR",
"name": "host.v1"
},
"configTypeId": "NH5p4FpGR",
"data": {
"allowedAddresses": [
"*.XXX.local"
],
"allowedPortRanges": [
{
"high": 65535,
"low": 1
}
],
"allowedProtocols": [
"tcp",
"udp"
],
"forwardAddress": true,
"forwardPort": true,
"forwardProtocol": true
},
"name": "xxx.local.ad.server"
}
],
"meta": {
"filterableFields": [
"tags",
"isSystem",
"name",
"type",
"id",
"createdAt",
"updatedAt"
],
"pagination": {
"limit": 10,
"offset": 0,
"totalCount": 1
}
}
}
intercept
{
"data": [
{
"_links": {
"self": {
"href": "./configs/7AdSebZxOAWjeS0nftfOST"
}
},
"createdAt": "2022-11-28T07:06:42.838Z",
"id": "7AdSebZxOAWjeS0nftfOST",
"tags": {},
"updatedAt": "2022-11-29T07:34:42.158Z",
"configType": {
"_links": {
"self": {
"href": "./config-types/g7cIWbcGg"
}
},
"entity": "config-types",
"id": "g7cIWbcGg",
"name": "intercept.v1"
},
"configTypeId": "g7cIWbcGg",
"data": {
"addresses": [
"*.XXX.local"
],
"dialOptions": {
"identity": ""
},
"portRanges": [
{
"high": 65535,
"low": 1
}
],
"protocols": [
"tcp",
"udp"
],
"sourceIp": ""
},
"name": "xxx.local.ad.client"
}
],
"meta": {
"filterableFields": [
"name",
"type",
"id",
"createdAt",
"updatedAt",
"tags",
"isSystem"
],
"pagination": {
"limit": 10,
"offset": 0,
"totalCount": 1
}
}
}
so the requests are not hitting the private router at all, nothing through tcpdump etc… I must be missing something simple here…will keep looking