It looks like the ziti cli is just not on the path. If you docker cp the identity file outside of docker, do just remember that the identity file will refer to the controller to contact. When you're using docker, it can get tricky to make sure everything is addressable. So just be careful of those sorts of things. For example inside the docker container you might be able to issue: curl -sk https://ziti-edge-controller:1280 but when you're outside of docker ziti-edge-controller won't be resolvable unless you edit your hosts file or can control your own DNS...
I commented on the other post too but if you find yourself stuck still. I'll re-record this video with a walkthrough to make sure it's all still working
@TheLumberjack , I managed to get the data out but ran everything on my host Ubuntu (please excuse the messy output below):
> Task :run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by retrofit2.Platform (file:/home/nyck33/.gradle/caches/modules-2/files-2.1/com.squareup.retrofit2/retrofit/2.9.0/d8fdfbd5da952141a665a403348b74538efc05ff/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of retrofit2.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Result from database is: a:1
Result from database is: b:2
Result from database is: c:3
Result from database is: d:4
Result from database is: e:5
Result from database is: f:6
Result from database is: g:7
Result from database is: h:8
Result from database is: i:9
Result from database is: j:0
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 2s
3 actionable tasks: 1 executed, 2 up-to-date
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ nmap -p 5432 localhost
Command 'nmap' not found, but can be installed with:
sudo snap install nmap # version 7.94, or
sudo apt install nmap # version 7.94+git20230807.3be01efb1+dfsg-1
See 'snap info nmap' for additional versions.
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ sudo apt install nmap
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
fonts-lato libgmp-dev libgmpxx4ldbl ruby-net-telnet ruby-webrick ruby-xmlrpc ruby3.1-doc rubygems-integration
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
libblas3 liblinear4 libssh2-1 lua-lpeg nmap-common
Suggested packages:
liblinear-tools liblinear-dev ncat ndiff zenmap
The following NEW packages will be installed:
libblas3 liblinear4 libssh2-1 lua-lpeg nmap nmap-common
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 6,207 kB of archives.
After this operation, 27.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://jp.archive.ubuntu.com/ubuntu mantic/main amd64 libblas3 amd64 3.11.0-2build1 [237 kB]
Get:2 http://jp.archive.ubuntu.com/ubuntu mantic/universe amd64 liblinear4 amd64 2.3.0+dfsg-5 [41.4 kB]
Get:3 http://jp.archive.ubuntu.com/ubuntu mantic/main amd64 libssh2-1 amd64 1.11.0-2 [120 kB]
Get:4 http://jp.archive.ubuntu.com/ubuntu mantic/universe amd64 lua-lpeg amd64 1.0.2-2 [38.6 kB]
Get:5 http://jp.archive.ubuntu.com/ubuntu mantic/universe amd64 nmap-common all 7.94+git20230807.3be01efb1+dfsg-1 [4,094 kB]
Get:6 http://jp.archive.ubuntu.com/ubuntu mantic/universe amd64 nmap amd64 7.94+git20230807.3be01efb1+dfsg-1 [1,675 kB]
Fetched 6,207 kB in 3s (1,926 kB/s)
Selecting previously unselected package libblas3:amd64.
(Reading database ... 212877 files and directories currently installed.)
Preparing to unpack .../0-libblas3_3.11.0-2build1_amd64.deb ...
Unpacking libblas3:amd64 (3.11.0-2build1) ...
Selecting previously unselected package liblinear4:amd64.
Preparing to unpack .../1-liblinear4_2.3.0+dfsg-5_amd64.deb ...
Unpacking liblinear4:amd64 (2.3.0+dfsg-5) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../2-libssh2-1_1.11.0-2_amd64.deb ...
Unpacking libssh2-1:amd64 (1.11.0-2) ...
Selecting previously unselected package lua-lpeg:amd64.
Preparing to unpack .../3-lua-lpeg_1.0.2-2_amd64.deb ...
Unpacking lua-lpeg:amd64 (1.0.2-2) ...
Selecting previously unselected package nmap-common.
Preparing to unpack .../4-nmap-common_7.94+git20230807.3be01efb1+dfsg-1_all.deb ...
Unpacking nmap-common (7.94+git20230807.3be01efb1+dfsg-1) ...
Selecting previously unselected package nmap.
Preparing to unpack .../5-nmap_7.94+git20230807.3be01efb1+dfsg-1_amd64.deb ...
Unpacking nmap (7.94+git20230807.3be01efb1+dfsg-1) ...
Setting up lua-lpeg:amd64 (1.0.2-2) ...
Setting up libblas3:amd64 (3.11.0-2build1) ...
update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to provide /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in auto mode
Setting up nmap-common (7.94+git20230807.3be01efb1+dfsg-1) ...
Setting up libssh2-1:amd64 (1.11.0-2) ...
Setting up liblinear4:amd64 (2.3.0+dfsg-5) ...
Setting up nmap (7.94+git20230807.3be01efb1+dfsg-1) ...
Processing triggers for man-db (2.11.2-3) ...
Processing triggers for libc-bin (2.38-1ubuntu6) ...
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ sudo apt install nmap
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
nmap is already the newest version (7.94+git20230807.3be01efb1+dfsg-1).
The following packages were automatically installed and are no longer required:
fonts-lato libgmp-dev libgmpxx4ldbl ruby-net-telnet ruby-webrick ruby-xmlrpc ruby3.1-doc rubygems-integration
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ nmap -p 5432 localhost
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-11-06 21:25 JST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00013s latency).
PORT STATE SERVICE
5432/tcp open postgresql
Nmap done: 1 IP address (1 host up) scanned in 0.02 seconds
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9a70790a797c openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes 0.0.0.0:3022->3022/tcp, 0.0.0.0:10080->10080/tcp pg-ziti-edge-router-1
be541c7bbdd3 openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes pg-ziti-fabric-router-br-1
6cbfc2b59753 openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes 0.0.0.0:3023->3023/tcp, 0.0.0.0:10081->10081/tcp pg-ziti-edge-router-wss-1
a2262ce18201 openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes pg-ziti-private-red-1
22a0ddd8b1cc openziti/zac "docker-entrypoint.s…" 48 minutes ago Up 47 minutes 1408/tcp, 0.0.0.0:8443->8443/tcp pg-ziti-console-1
e618a3e13b82 openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes pg-ziti-private-blue-1
b371f4c36ca1 openziti/quickstart:latest "/var/openziti/scrip…" 48 minutes ago Up 48 minutes (healthy) 0.0.0.0:1280->1280/tcp, 0.0.0.0:6262->6262/tcp pg-ziti-controller-1
fcd0b12dc16b postgres "docker-entrypoint.s…" 48 minutes ago Up 48 minutes 5432/tcp pg-postgres-db-1
2da832a8c0e2 openziti/hello-world "/bin/sh -c 'echo \"h…" 48 minutes ago Up 48 minutes (healthy) 0.0.0.0:80->8000/tcp pg-web-test-blue-1
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ psql -h pg-postgres-db-1 -U postgres
psql: error: could not translate host name "pg-postgres-db-1" to address: Temporary failure in name resolution
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ psql -U localhost -U postgres
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: Peer authentication failed for user "postgres"
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ psql -h localhost -p 5432 -U postgres -d simpledb -c "SELECT * FROM simpletable LIMIT 10;"
Password for user postgres:
I am guessing I have to run the Docker network on a VM or another machine for the postgres to be inaccessible. Please confirm at your next convenience.
Actually, if you're getting data back data by running that docker compose file and you ran the java program outside of docker... YOU DID IT!!!
Notice the #ports section is commented out?
That means the docker environment is not exposed in any way, shape or form.
You can check this by running docker ps and looking at the ports. It should look like mine:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a88176d6eff2 openziti/zac "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 1408/tcp, 0.0.0.0:8443->8443/tcp pg-ziti-console-1
fc3680c2ef43 openziti/quickstart:latest "/bin/bash /var/open…" 4 minutes ago Up 4 minutes 0.0.0.0:3022->3022/tcp, 0.0.0.0:10080->10080/tcp pg-ziti-edge-router-1
44cf095de28e openziti/quickstart:latest "/var/openziti/scrip…" 4 minutes ago Exited (0) 4 minutes ago pg-ziti-controller-init-container-1
f5b1e5339e42 postgres "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 5432/tcp pg-postgres-db-1
8a189b96e5a6 openziti/quickstart:latest "/var/openziti/scrip…" 4 minutes ago Up 4 minutes (healthy) 0.0.0.0:1280->1280/tcp, 0.0.0.0:6262->6262/tcp pg-ziti-controller-1
Even though port 5432 is listed, it's not exposed. Is that how yours looks? I would think it is!
You can test further by trying to login to postgres on port localhost:5432. Nothing is listening! Further proof by looking at ss -lntp or netstat -nap. Port 5432 won't be listening at all.
I'm going to put a new PR up with some small changes as well as make a new video soon. I'll post back here when that's up
If you can login to the database that's surprising. Are you running the java example from inside docker? That would explain why you're able to login using "localhost".
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9a70790a797c openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes 0.0.0.0:3022->3022/tcp, 0.0.0.0:10080->10080/tcp pg-ziti-edge-router-1
be541c7bbdd3 openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes pg-ziti-fabric-router-br-1
6cbfc2b59753 openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes 0.0.0.0:3023->3023/tcp, 0.0.0.0:10081->10081/tcp pg-ziti-edge-router-wss-1
a2262ce18201 openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes pg-ziti-private-red-1
22a0ddd8b1cc openziti/zac "docker-entrypoint.s…" 48 minutes ago Up 47 minutes 1408/tcp, 0.0.0.0:8443->8443/tcp pg-ziti-console-1
e618a3e13b82 openziti/quickstart:latest "/bin/bash /var/open…" 48 minutes ago Up 47 minutes pg-ziti-private-blue-1
b371f4c36ca1 openziti/quickstart:latest "/var/openziti/scrip…" 48 minutes ago Up 48 minutes (healthy) 0.0.0.0:1280->1280/tcp, 0.0.0.0:6262->6262/tcp pg-ziti-controller-1
fcd0b12dc16b postgres "docker-entrypoint.s…" 48 minutes ago Up 48 minutes 5432/tcp pg-postgres-db-1
2da832a8c0e2 openziti/hello-world "/bin/sh -c 'echo \"h…" 48 minutes ago Up 48 minutes (healthy) 0.0.0.0:80->8000/tcp pg-web-test-blue-1
I ran the java client on its own, not in Docker. I was also surprised to be able to login to Postgres so I was wondering if that was due to running everythinng on my Ubuntu Host rather than the setup you had with Java client on Windows host and Ubuntu VM.
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ netstat -nap | grep 5432
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN -
unix 2 [ ACC ] STREAM LISTENING 22126 - /var/run/postgresql/.s.PGSQL.5432
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ ss -lntp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:10081 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:10080 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:8443 0.0.0.0:*
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:3023 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:3022 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:1280 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:80 0.0.0.0:*
LISTEN 0 4096 127.0.0.1:631 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:6262 0.0.0.0:*
LISTEN 0 244 127.0.0.1:5432 0.0.0.0:*
LISTEN 0 50 [::ffff:127.0.0.1]:45841 *:* users:(("java",pid=41001,fd=237))
LISTEN 0 50 [::ffff:127.0.0.1]:35175 *:* users:(("java",pid=5926,fd=248))
LISTEN 0 4096 [::1]:631 [::]:*
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ nmap -p 5432 localhost
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-11-07 08:53 JST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00012s latency).
PORT STATE SERVICE
5432/tcp open postgresql
Nmap done: 1 IP address (1 host up) scanned in 0.02 seconds
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ psql -h localhost -p 5432 -U postgres -d simpledb "select * from simpletable limit 10;"
psql: warning: extra command-line argument "select * from simpletable limit 10;" ignored
Password for user postgres:
psql: error: connection to server at "localhost" (127.0.0.1), port 5432 failed: fe_sendauth: no password supplied
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ nc 127.0.0.1 5432 -v
Connection to 127.0.0.1 5432 port [tcp/postgresql] succeeded!
^C
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6f6fe2efbb39 openziti/quickstart:latest "/bin/bash /var/open…" 21 minutes ago Up 21 minutes 0.0.0.0:3023->3023/tcp, 0.0.0.0:10081->10081/tcp pg-ziti-edge-router-wss-1
93057fc2aed3 openziti/quickstart:latest "/bin/bash /var/open…" 21 minutes ago Up 21 minutes 0.0.0.0:3022->3022/tcp, 0.0.0.0:10080->10080/tcp pg-ziti-edge-router-1
4f0d02e4ace4 openziti/zac "docker-entrypoint.s…" 21 minutes ago Up 21 minutes 1408/tcp, 0.0.0.0:8443->8443/tcp pg-ziti-console-1
6f8930a620c7 openziti/quickstart:latest "/bin/bash /var/open…" 21 minutes ago Up 21 minutes pg-ziti-fabric-router-br-1
bb0795b992ca openziti/quickstart:latest "/bin/bash /var/open…" 21 minutes ago Up 21 minutes pg-ziti-private-blue-1
cca16577105f openziti/quickstart:latest "/bin/bash /var/open…" 21 minutes ago Up 21 minutes pg-ziti-private-red-1
5a6afcd31ed0 openziti/quickstart:latest "/var/openziti/scrip…" 21 minutes ago Up 21 minutes (healthy) 0.0.0.0:1280->1280/tcp, 0.0.0.0:6262->6262/tcp pg-ziti-controller-1
04dc84b9a92f postgres "docker-entrypoint.s…" 21 minutes ago Up 21 minutes 5432/tcp pg-postgres-db-1
d0c32527621f openziti/hello-world "/bin/sh -c 'echo \"h…" 21 minutes ago Up 21 minutes (healthy) 0.0.0.0:80->8000/tcp pg-web-test-blue-1
(base) nyck33@nyck33-tt:/media/nyck33/1TB-backup/cybersec/openziti/ziti-sdk-jvm/samples/jdbc-postgres$
Also if I were to nitpick, the path to the java identity created in the controller is container-name:/persistent/java-identity.json and the names of the containers use -'s only and not a mix of _'s and -'s.
Looks like you do have a postgres instance running on 5432 for sure. I think you should look for another postgres process or maybe turn docker off and see if it's somehow exposing ports that it shouldn't be. I've never seen that behavior before.
I'm actually redoing that right now, so that's a thing I'll be happy to fix. I'll change them to use hyphens (not underscores).
I hope this message finds you well. Upon further investigation, I've discovered that the issue I was encountering stemmed from my local Postgres service running concurrently, which inadvertently intercepted the connections I was attempting to establish elsewhere.
I apologize for any inconvenience my oversight may have caused and appreciate the time you’ve dedicated to assisting me. It was certainly a learning moment for me, and I’m grateful for your patience.
Awesome. Glad to hear that was what it was and you found it. I'm pretty certain you did successfully connect to the totally private postgres running inside docker though! Glad you got it all sorted
At 6:45 in the video: https://youtu.be/7ON6Ybs1s3A?t=399
it is mentioned that the "port is inconsequential, only used to identify the service" meaning although the port is not exposed, when someone for example runs nc localhost 5432 they know they cannot reach it but they would notice from the error message that something is indeed running there?
Also in the new video you create an identity for postgres but in the 2022 version ofsamples/jdbc-postgres example in the ziti-sdk-jvm repo is that done in the run-controller.sh script? I can't find it.
Inside the controller for 2022 version:
I remembered to shut down my local postgres this time and ran it successfully, the 2022 version.
base) nyck33@nyck33-tt:~/Documents/cybersec/openziti/ziti-sdk-jvm-zerotrust-postgres/samples/jdbc-postgres$ ./gradlew run
> Task :run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by retrofit2.Platform (file:/home/nyck33/.gradle/caches/modules-2/files-2.1/com.squareup.retrofit2/retrofit/2.9.0/d8fdfbd5da952141a665a403348b74538efc05ff/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of retrofit2.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Result from database is: a:1
Result from database is: b:2
Result from database is: c:3
Result from database is: d:4
Result from database is: e:5
Result from database is: f:6
Result from database is: g:7
Result from database is: h:8
Result from database is: i:9
Result from database is: j:0
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 1s
3 actionable tasks: 1 executed, 2 up-to-date
I am however left scratching my head as to where the postgres db's identity was created and enrolled as I outlined above.
It wouldn't matter what the error message says. Netstat, ss, nmap would all confirm there's no port listening even if the message said there was. At that point he port is only consequential to the ziti-native application. Ziti uses the fictitious host name and port to locate the actual ziti service to dial, that's it.
I believe all the videos around postgres we have will create an identity for the Java client and use a router for the offload from the overlay back to the underlay. When the router's are created, they are created with "tunneler-mode" enabled so that they are able to perform that task. That's why you won't find anywhere the identity is created. In the docker-compose examples, they'll effectively run ziti edge create edge-router -t (-t, --tunneler-enabled Can this edge router be used as a tunneler). Without that flag, no identity is created for the router, and it can't do that offloading (as there would be no identity).
This: docker cp prep-db.sql pg-postgres-db-1:prep-db.sql is just a bug. I put a PR up to remove that from the cheatsheet but it's not merged yet.
Further proof by looking at 