ahh,, that is it
Weird… as the docker container is running… all up
mmmmmm. That seems like a bug then… Do this - leave ALL your stuff in place just like you have it, delete the bind service policy, and recreate it…
ziti edge delete service-policy postgres-bind-policy
… wait for like 60s or until you see the router notice that it no longer has the service… You will see things like “error creating terminator” in the logs…
recreate the policy - wait for the router to notice the service
ziti edge create service-policy postgres-bind-policy Bind --identity-roles '@ziti-private-blue' --service-roles '#private-postgres-services'
you should see:
INFO edge/router/xgress_edge_tunnel.(*fabricProvider).establishTerminator: {service=[private-postgres] address=[5117ebbd-ca73-47f1-8031-8df2c7be18c2] terminatorId=[31OtEwzqLaWCbjDlajtS6O] routerId=[&{0xc0003f4600 -4LYBMothS map[]}]} created terminator
then ziti edge list terminators will have a terminator shown.
Once you see the terminator - then try the java app again
Sorry about this… the terminator came up…
But when I rerun the app… same error…
…
Exception in thread “main” org.postgresql.util.PSQLException: Something unusual has occurred to cause the driver to fail. Please report this exception.
at org.postgresql.Driver.connect(Driver.java:282)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
at jdbc.postgres.App.main(App.java:37)
Caused by: java.nio.channels.UnresolvedAddressException
at java.base/sun.nio.ch.Net.checkAddress(Net.java:131)
at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:306)
at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210)
at org.openziti.net.nio.NetUtilsKt$connectSuspend$3.invokeSuspend(NetUtils.kt:72)
at org.openziti.net.nio.NetUtilsKt
well dangit… you can see the terminator now though, right? 
Let’s see the logs again from the controller and the private blue router… Run the app, then run:
controller:
docker logs -n 10 docker_ziti-controller_1
ziti-private-blue:
docker logs -n 10 docker_ziti-private-blue_1
Your terminator is correct now. Well that’s progress at least. Let’s see the logs now after you run the app please.
…
docker logs -n 10 docker_ziti-controller_1
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get “http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/docker_ziti-controller_1/json”: dial unix /var/run/docker.sock: connect: permission denied
[opc@instance-20220518-1244 jdbc-postgres]$ sudo docker logs -n 10 docker_ziti-controller_1
[ 380.582] INFO fabric/controller/network.(*Network).assemble: {srcRouterId=[OdC.XtNrw] dstRouterId=[.nj1Ilqrm] linkId=[6WT0PfRbnO7O6ELdbJCB2g]} sending link dial
[ 380.585] INFO fabric/controller/handler_ctrl.(*faultHandler).handleFault [ch{OdC.XtNrw}->u{classic}->i{6Mj5}]: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} link fault
[ 380.586] INFO fabric/controller/network.(*Network).handleLinkChanged: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} changed link
[ 380.586] INFO fabric/controller/handler_ctrl.(*routerLinkHandler).HandleLinks [ch{OdC.XtNrw}->u{classic}->i{6Mj5}]: {linkId=[18xsTlc4KKYGdhzZyO2XaY] destRouterId=[.nj1Ilqrm] routerId=[OdC.XtNrw]} router reported link added
[ 440.587] INFO fabric/controller/network.(*Network).clean: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} removing failed link
[ 754.577] INFO : http: TLS handshake error from 172.24.0.4:44898: remote error: tls: bad certificate
[ 777.596] INFO : http: TLS handshake error from 172.24.0.4:44910: remote error: tls: bad certificate
[3421.006] INFO : http: TLS handshake error from 172.24.0.4:45168: remote error: tls: bad certificate
[4340.775] INFO edge/controller/handler_edge_ctrl.(*createTunnelTerminatorHandler).CreateTerminator: {serviceId=[3IVX1rfgH] service=[private-postgres] terminator=[DpG7] routerId=[OdC.XtNrw]} created terminator
[4542.498] INFO : http: TLS handshake error from 172.24.0.4:45284: remote error: tls: bad certificate
…
sudo docker logs -n 10 docker_ziti-private-blue_1
[ 293.913] WARNING fabric/router/handler_link.(*heartbeatCallback).CheckHeartBeat: {channelId=[ch{l/7azULBa3yCH6NkNgan2OaC}->u{classic}->i{52PL}]} heartbeat not received in time, link may be unhealthy
[ 294.386] INFO edge/router/handler_edge_ctrl.(*apiSessionAddedHandler).applySync: finished sychronizing api sessions [count: 15, syncId: cl4gz8qo9008y8al78qsfcx4t, duration: 2.002229ms]
[ 294.886] WARNING fabric/router/handler_link.(*heartbeatCallback).CheckHeartBeat: {channelId=[ch{l/7azULBa3yCH6NkNgan2OaC}->u{classic}->i{LqXW}]} heartbeat not received in time, link may be unhealthy
[ 294.904] WARNING fabric/router/handler_link.(*heartbeatCallback).CheckHeartBeat: {channelId=[ch{l/7azULBa3yCH6NkNgan2OaC}->u{classic}->i{52PL}]} heartbeat not received in time, link may be unhealthy
[ 353.392] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {address=[tls:ziti-edge-router-wss:10080] existingLinkId=[18xsTlc4KKYGdhzZyO2XaY] linkProtocol=[tls] routerVersion=[v0.25.10] linkId=[6WT0PfRbnO7O6ELdbJCB2g] routerId=[.nj1Ilqrm]} existing link found
[4313.483] INFO edge/tunnel/intercept.(*ServiceListener).HandleServicesChange: {service=[private-postgres]} adding service
[4313.501] INFO edge/tunnel/intercept.(*ServiceListener).addService: Hosting newly available service private-postgres
[4313.506] INFO edge/router/xgress_edge_tunnel.(*fabricProvider).establishTerminatorWithRetry.func1: {service=[private-postgres]} attempting to establish terminator
[4313.585] INFO edge/router/xgress_edge_tunnel.(*fabricProvider).establishTerminator: {routerId=[&{0xc0000ac200 OdC.XtNrw map}] service=[private-postgres] address=[6caa0c9d-33f4-4058-bec0-89d12a473e9b] sessionId=[cl4h1mwpl03ey8al7429cbzf2]} received new session
[4313.588] INFO edge/router/xgress_edge_tunnel.(*fabricProvider).establishTerminator: {address=[6caa0c9d-33f4-4058-bec0-89d12a473e9b] terminatorId=[DpG7] routerId=[&{0xc0000ac200 OdC.XtNrw map}] service=[private-postgres]} created terminator
sorry.. need to run the app
Oh no… I wonder, when you did the docker down did you use:
docker-compose down -v
The way the doc tells you to? If so - I’m thinking you didn’t copy the json file out of the container and the identity file is old? Could that be the situation? inside the docker container is there a “java-identity.json” and is that file different than the one you use for the app?
sudo docker logs -n 10 docker_ziti-controller_1
…
[ 380.582] INFO fabric/controller/network.(*Network).assemble: {srcRouterId=[OdC.XtNrw] dstRouterId=[.nj1Ilqrm] linkId=[6WT0PfRbnO7O6ELdbJCB2g]} sending link dial
[ 380.585] INFO fabric/controller/handler_ctrl.(*faultHandler).handleFault [ch{OdC.XtNrw}->u{classic}->i{6Mj5}]: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} link fault
[ 380.586] INFO fabric/controller/network.(*Network).handleLinkChanged: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} changed link
[ 380.586] INFO fabric/controller/handler_ctrl.(*routerLinkHandler).HandleLinks [ch{OdC.XtNrw}->u{classic}->i{6Mj5}]: {linkId=[18xsTlc4KKYGdhzZyO2XaY] destRouterId=[.nj1Ilqrm] routerId=[OdC.XtNrw]} router reported link added
[ 440.587] INFO fabric/controller/network.(*Network).clean: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} removing failed link
[ 754.577] INFO : http: TLS handshake error from 172.24.0.4:44898: remote error: tls: bad certificate
[ 777.596] INFO : http: TLS handshake error from 172.24.0.4:44910: remote error: tls: bad certificate
[3421.006] INFO : http: TLS handshake error from 172.24.0.4:45168: remote error: tls: bad certificate
[4340.775] INFO edge/controller/handler_edge_ctrl.(*createTunnelTerminatorHandler).CreateTerminator: {serviceId=[3IVX1rfgH] service=[private-postgres] terminator=[DpG7] routerId=[OdC.XtNrw]} created terminator
[4542.498] INFO : http: TLS handshake error from 172.24.0.4:45284: remote error: tls: bad certificate
I realised that a little while back.. this time I am sure its 100% correct.. as I have not shutdown the container.. since I copied the json from the container to the shared drive
sudo docker logs -n 10 docker_ziti-controller_1
…
[ 380.582] INFO fabric/controller/network.(*Network).assemble: {srcRouterId=[OdC.XtNrw] dstRouterId=[.nj1Ilqrm] linkId=[6WT0PfRbnO7O6ELdbJCB2g]} sending link dial
[ 380.585] INFO fabric/controller/handler_ctrl.(*faultHandler).handleFault [ch{OdC.XtNrw}->u{classic}->i{6Mj5}]: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} link fault
[ 380.586] INFO fabric/controller/network.(*Network).handleLinkChanged: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} changed link
[ 380.586] INFO fabric/controller/handler_ctrl.(*routerLinkHandler).HandleLinks [ch{OdC.XtNrw}->u{classic}->i{6Mj5}]: {linkId=[18xsTlc4KKYGdhzZyO2XaY] destRouterId=[.nj1Ilqrm] routerId=[OdC.XtNrw]} router reported link added
[ 440.587] INFO fabric/controller/network.(*Network).clean: {linkId=[6WT0PfRbnO7O6ELdbJCB2g]} removing failed link
[ 754.577] INFO : http: TLS handshake error from 172.24.0.4:44898: remote error: tls: bad certificate
[ 777.596] INFO : http: TLS handshake error from 172.24.0.4:44910: remote error: tls: bad certificate
[3421.006] INFO : http: TLS handshake error from 172.24.0.4:45168: remote error: tls: bad certificate
[4340.775] INFO edge/controller/handler_edge_ctrl.(*createTunnelTerminatorHandler).CreateTerminator: {serviceId=[3IVX1rfgH] service=[private-postgres] terminator=[DpG7] routerId=[OdC.XtNrw]} created terminator
[4542.498] INFO : http: TLS handshake error from 172.24.0.4:45284: remote error: tls: bad certificate
[opc@instance-20220518-1244 jdbc-postgres]$ sudo docker logs -n 10 docker_ziti-private-blue_1
[ 293.913] WARNING fabric/router/handler_link.(*heartbeatCallback).CheckHeartBeat: {channelId=[ch{l/7azULBa3yCH6NkNgan2OaC}->u{classic}->i{52PL}]} heartbeat not received in time, link may be unhealthy
[ 294.386] INFO edge/router/handler_edge_ctrl.(*apiSessionAddedHandler).applySync: finished sychronizing api sessions [count: 15, syncId: cl4gz8qo9008y8al78qsfcx4t, duration: 2.002229ms]
[ 294.886] WARNING fabric/router/handler_link.(*heartbeatCallback).CheckHeartBeat: {channelId=[ch{l/7azULBa3yCH6NkNgan2OaC}->u{classic}->i{LqXW}]} heartbeat not received in time, link may be unhealthy
[ 294.904] WARNING fabric/router/handler_link.(*heartbeatCallback).CheckHeartBeat: {channelId=[ch{l/7azULBa3yCH6NkNgan2OaC}->u{classic}->i{52PL}]} heartbeat not received in time, link may be unhealthy
[ 353.392] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {address=[tls:ziti-edge-router-wss:10080] existingLinkId=[18xsTlc4KKYGdhzZyO2XaY] linkProtocol=[tls] routerVersion=[v0.25.10] linkId=[6WT0PfRbnO7O6ELdbJCB2g] routerId=[.nj1Ilqrm]} existing link found
[4313.483] INFO edge/tunnel/intercept.(*ServiceListener).HandleServicesChange: {service=[private-postgres]} adding service
[4313.501] INFO edge/tunnel/intercept.(*ServiceListener).addService: Hosting newly available service private-postgres
[4313.506] INFO edge/router/xgress_edge_tunnel.(*fabricProvider).establishTerminatorWithRetry.func1: {service=[private-postgres]} attempting to establish terminator
[4313.585] INFO edge/router/xgress_edge_tunnel.(*fabricProvider).establishTerminator: {routerId=[&{0xc0000ac200 OdC.XtNrw map}] service=[private-postgres] address=[6caa0c9d-33f4-4058-bec0-89d12a473e9b] sessionId=[cl4h1mwpl03ey8al7429cbzf2]} received new session
[4313.588] INFO edge/router/xgress_edge_tunnel.(*fabricProvider).establishTerminator: {address=[6caa0c9d-33f4-4058-bec0-89d12a473e9b] terminatorId=[DpG7] routerId=[&{0xc0000ac200 OdC.XtNrw map}] service=[private-postgres]} created terminator
PS.. I am wondering if this is an issue with Oracle Linux 8..
and additional security controls it has
I think I know where the audit logs are... I will see if I can find them
I replicated the exact sample in Oracle OCP and it worked fine. Here’s what I did.
Oracle linux/OCP setup
ssh to ocp instance
install java:
sudo yum install java -y
java -version
java version "11.0.15.1" 2022-04-22 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.15.1+2-LTS-10)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.15.1+2-LTS-10, mixed mode)
install git:
sudo yum install git -y
install docker using Install Docker Engine on CentOS | Docker Documentation
sudo yum install -y yum-utils
sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
sudo systemctl start docker
sudo docker run hello-world
make the current user able to run docker without sudo:
sudo groupadd docker
sudo usermod -aG docker $USER
exit/logout of shell, then ssh back to the environment
verify docker runs without sudo:
docker run hello-world
install docker-compose:
sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
Do the stuff from the cheatsheet.md
read and follow - ziti-sdk-jvm/cheatsheet.md at main · openziti/ziti-sdk-jvm · GitHub
here’s a condensed version of what I did exactly:
make a new folder for docker compose, put the compose file and env file in that folder:
mkdir /tmp/docker
curl https://raw.githubusercontent.com/openziti/ziti/release-next/quickstart/docker/docker-compose.yml > /tmp/docker/docker-compose.yml
curl https://raw.githubusercontent.com/openziti/ziti/release-next/quickstart/docker/.env > /tmp/docker/.env
add postgres to compose file:
vi /tmp/docker/docker-compose.yml
# add postgres:
postgres-db:
image: postgres
#ports:
# - 5432:5432
networks:
- zitiblue
volumes:
- ./data/db:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
bring up docker:
cd /tmp/docker/
# if you see an error like:
# ERROR: no such image: :: invalid reference format
# you are not in the proper folder - cd to /tmp/docker
#
docker-compose -f /tmp/docker/docker-compose.yml -p pg up
add hosts entry (only do this one time):
echo "127.0.0.1 ziti-edge-controller" | sudo tee -a /etc/hosts
echo "127.0.0.1 ziti-edge-router" | sudo tee -a /etc/hosts
Ziti Setup
Copy and paste the commands exactly from the cheatsheet directly, do not transcribe/edit them.
Run the Sample
clone ziti-sdk-jvm:
git clone https://github.com/openziti/ziti-sdk-jvm.git
cd to the sample:
cd ziti-sdk-jvm/samples/jdbc-postgres
Results
[opc@clintozapr09b jdbc-postgres]$ git checkout gradlew
Updated 1 path from the index
[opc@clintozapr09b jdbc-postgres]$ git pull
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 5 (delta 4), reused 3 (delta 3), pack-reused 0
Unpacking objects: 100% (5/5), 1.01 KiB | 1.01 MiB/s, done.
From https://github.com/openziti/ziti-sdk-jvm
d10a02a..ff491f1 main -> origin/main
Updating d10a02a..ff491f1
Fast-forward
samples/jdbc-postgres/gradlew | 0
1 file changed, 0 insertions(+), 0 deletions(-)
mode change 100644 => 100755 samples/jdbc-postgres/gradlew
[opc@clintozapr09b jdbc-postgres]$ ./gradlew run --args="/tmp/java-identity.json"
> Task :run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by retrofit2.Platform (file:/home/opc/.gradle/caches/modules-2/files-2.1/com.squareup.retrofit2/retrofit/2.9.0/d8fdfbd5da952141a665a403348b74538efc05ff/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of retrofit2.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Result from database is: a:1
Result from database is: b:2
Result from database is: c:3
Result from database is: d:4
Result from database is: e:5
Result from database is: f:6
Result from database is: g:7
Result from database is: h:8
Result from database is: i:9
Result from database is: j:0
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 4s
3 actionable tasks: 1 executed, 2 up-to-date
[opc@clintozapr09b jdbc-postgres]$
At this point, if it doesn’t work - I have to punt. I don’t think there’s more I can do to troubleshoot. Try a new environment maybe?
1 Like
Yes
Thanks so much...
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Result from database is: a:1
Result from database is: b:2
Result from database is: c:3
Result from database is: d:4
Result from database is: e:5
Result from database is: f:6
Result from database is: g:7
Result from database is: h:8
Result from database is: i:9
Result from database is: j:0
One thing that may have caused the problems was that I found a few old docker containers..
so I ran the following to delete them all and start from ground zero
docker system prune
docker ps -a
also.. the problem could also have been with permissions
make current user able to run docker without sudo
sudo groupadd docker
sudo usermod -aG docker $USER
This could have been something related... but don't know for sure
Anyway.. super excited now.. a massive learning experience. 
After a bit of investigation… I found the cause of all my problems
I accidentally deleted the edge-router-policy
That would do it
I realised this after a bit more testing… as the problem came back again…
Then I realised that maybe I deleted too many things between the tunneler demo… versus the jdbc demo.
and presto… I now have it all working end to end