Odd issue with FTPS over OpenZiti Overlay

I noticed this with the latest version on their website as well, I downloaded the one without the sponsor stuff too, I think it's a false positive as they just released that one.

I don't think it does, it shows me certs (I had posted it above, minus the actual certs and with modified domain names for privacy). I can try with another FTP client to see if it does it there. Here is the log from FileZilla, when the issue occurs:

Note: I know about the passive unrouteable address (it says 127.0.0.1... but FileZilla is smart enough to try the actual resolved IP, which is the Open Ziti IP in this case)

I have been trying to reconcile this but i think the cap you sent is not quite complete.

What I want to see is on the client side, i want to see the FTP Request: AUTH TLS and on the far side I want to see the Request: AUTH TLS exit.

I need to see that from both client side and server side.

Can you send the full pcaps to me too? Filter out the port/ip and export just the FTP sessoin (hopefully you kjnow how to do that?)

I can see in that client cap the TLS Response: 234 Auth TLS successful...

Specifically I want to capture the whole flow, from the client side and server side. SYN/SYN,ACK/FTP Request AUTH TLS etc...

That make sense?

I've filtered the pcaps based on IP and sent them to you in a zip file via a DM, the ports/protocols involve multiple (passive data port, ftp, tcp) so I felt it best to leave that to you if you'd like to filter it further.

Thanks for the pcaps. I've looked at them and there's definitely "something strange" going on. I can't quite understand how and where things go wrong. I wish we/I could reproduce this so I wouldn't require your help. Thanks again for helping figure out this very strange issue...

Does this ever happen on the very first attempt or is it always after making a few requests? When it goes wrong, do you close the client side app and open it again or do you just keep making requests with the same client? Does it happen immediately or does it seem to take a while?

Can you start the captures, turn on the client, reproduce the problem and then stop the captures? Let's capture all the packets from very start if we can?

I hope that will be the last capture/test I ask you to do for a while... thx again

From Gemini AI and my own inspection, I think it's that the FTP control session succeeds, enters passive mode (as most NAT clients do), ProFTPD sends a passive port (that is defined in a specified range - all ports allowed through the Ziti service), FileZilla uses the Ziti IP of the service to establish a connection on the port the server gives it, it succeeds/acknowledges and then the session gets dropped out immediately (hard to tell... but it seems related to that malformed packet or missing data). I have confirmed the same issue on my laptop running FileZilla, Windows 10, ZDEW as well.

Yes it does, it can also happen on subsequent requests (e.g. if I re-login or I browse to a directory which re-opens a connection). It seems to be more likely to happen, if I have no connected/reconnected/made an action for a bit (I haven't timed it... but let's say 10+ minutes).

I keep it open during the tests.

Happens very quickly (within a second or two), there's no "timeout" that I can tell.

Will do!

DM'd you the packet captures... this time I opened FileZilla AFTER starting the captures. I also tested on my W10 laptop to see if it was PC specific, and it's not - it happens the same way so it seems this is re-producible. Are there any information you'd need to re-produce this on your side? I'm happy to keep testing for you, but it sounds like you'd like to see it happen in a duplicated environment.

Yeah it's just a lot easier if we can debug it as we aren't reliant on the async back/forth of a support forum.

I have installed filezilla on a VM, but I'll need to configure it exactly how you have in order to replicate the problem. Can you detail for me what you did to configure it? Anything you could do to help me configure it as you have is great. I've never used it before, so knowing exactly how you're using it would also be useful. Anything that would allow me to replicate (or not replicate) the problem.

FileZilla:
Click "Site Manager" at top-left. These are my settings - but the "Normal" encryption type does the same thing if your server requires/supports FTPS.


All the other tabs are default.

I decided to try another FTP client as well, called WinSCP and received a similar error (different wording):

Server sent passive reply with unroutable address 127.0.0.1, using host address instead.
TLS connect: error in SSLv3/TLS write client hello
The server is using unsupported protocol. Your WinSCP session is configured to use TLSv1.2 through TLSv1.3. It can be configured to use TLSv1.0 through TLSv1.3. Though, avoid using old insecure protocols, whenever possible.
Can't establish TLS connection.
Could not retrieve directory listing.

ProFTPd:
Below are the ftp server config files (the main ones, anyways - controlling the server and TLS). Apologies for the formatting... tried to convert to windows new-lines for display here but it didn't work perfectly.

proftpd.conf:

ServerName "FTP"
ServerIdenton "FTP Server ready."
ServerAdmin root@localhost
DefaultServer on
DefaultRoot ~ !adm

Include /etc/proftpd/tls.conf

<IfModule mod_vroot.c>
VRootEngine on
VRootAlias/etc/security/pam_env.conf etc/security/pam_env.conf
</IfModule>

AuthPAMConfig proftpd
AuthOrdermod_auth_pam.c* mod_auth_unix.c
UseReverseDNS off
User proftpd
Group nogroup
MaxInstances 20
UseSendfile off
LogFormatdefault "%h %l %u %t \"%r\" %s %b"
LogFormatauth    "%v [%P] %h %t \"%r\" %s"
ListOptions -a
RequireValidShell off
PassivePorts 12000 12100
#MasqueradeAddress              10.10.10.10
DebugLevel                      1

<Global>
Umask002
AllowOverwrite yes
<Limit ALL SITE_CHMOD>
AllowAll
</Limit>
</Global>

tls.conf:

<IfModule mod_dso.c>
    # If mod_tls was built as a shared/DSO module, load it
    LoadModule mod_tls.c
</IfModule>
<IfModule mod_tls.c>
TLSEngine                               on
TLSLog                                  /var/log/proftpd/tls.log
# this is an example of protocols, proftp works witl all, but use only the most secure ones like TLSv1.1 and TLSv1.2
TLSCipherSuite                          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256
TLSProtocol                             TLSv1.2 TLSv1.3
TLSServerCipherPreference               on
#
# Server SSL certificate. You can generate a self-signed certificate using
# a command like:
#
# openssl req -x509 -newkey rsa:1024 \
#          -keyout /etc/ssl/private/proftpd.key -out /etc/ssl/certs/proftpd.crt \
#          -nodes -days 365
#
# The proftpd.key file must be readable by root only. The other file can be
# readable by anyone.
#
# chmod 0600 /etc/ssl/private/proftpd.key
# chmod 0640 /etc/ssl/private/proftpd.key
#
TLSRSACertificateFile                   /usr/local/hestia/ssl/certificate.crt
TLSRSACertificateKeyFile                /usr/local/hestia/ssl/certificate.key
#
# CA the server trusts...
#TLSCACertificateFile                    /etc/ssl/certs/CA.pem
# ...or avoid CA cert and be verbose
#TLSOptions                      NoCertRequest EnableDiags
# ... or the same with relaxed session use for some clients (e.g. FireFtp)
#TLSOptions                      NoCertRequest EnableDiags NoSessionReuseRequired
#
#
# Per default drop connection if client tries to start a renegotiate
# This is a fix for CVE-2009-3555 but could break some clients.
#
#TLSOptions                      AllowClientRenegotiations
#
TLSOptions                      NoSessionReuseRequired 
#AllowClientRenegotiations
# Authenticate clients that want to use FTP over TLS?
#
#TLSVerifyClient                         off
#
# Are clients required to use FTP over TLS when talking to this server?
#
TLSRequired                             off
#
# Allow SSL/TLS renegotiations when the client requests them, but
# do not force the renegotations.  Some clients do not support
# SSL/TLS renegotiations; when mod_tls forces a renegotiation, these
# clients will close the data connection, or there will be a timeout
# on an idle data connection.
#
TLSRenegotiate                          none
#required off

</IfModule>

Thanks. I'll give this a try after Ziti TV today (in about 1.5 hrs). I'll see if I can replicate it. If not, would you be open to a debug session I faciliate? (via zoom) I kinda want to understand what's goin on here. Appreciate the support so far, cheers.

Yes that would be ok. I'm available this afternoon around 2-4pm EST

It appears to be specific to my VPS with Servarica - I suspect it may be a performance issue, as I've set it up on my local server and everything was much faster, and works flawlessly with the same OS/versions/configuration. Odd how a TLS issue would stem from a connection that seems to still connect at a reasonable speed, but it would explain the randomness of it succeeding or failing.

My next steps are to open a ticket with my server provider and, worst case, try another VPS provider. Thank-you Open Ziti team, for all your assistance!

1 Like

Just a follow-up here. Opened a ticket with the VPS provider - we did some tests - turns out it was the host the VPS was on, they migrated me to another and I no longer get the error message. All good now!

2 Likes

Wow. Amazing support from the vps provider. Crazy issue. Thanks for following up again.

1 Like