Use of http instead of https for the Desktop tunneller

[markamind]

I need to prepare for a question that I may receive when giving a demo next week.

The demo will be to showcase how the Desktop Tunneller works.. where I will access a website over the ziti network.

As can be see in this screen shot, the desktop browser is accessing the website over http instead of https.

The concern would be that if it's going over http, it's not secure.

For example, say this page was a login screen with username and password..

would it mean that you can view the traffic from the website as clear text

as opposed to it being encrypted if the site was being accessed over ssl

While this may appear the case from an outsiders perspective.. my understanding is that this concern is addressed in how the tunneller works.. by intercepting such traffic..

Once intercepted by the Edge Tunneller.. it is encrypted with the client certificate. In this sense, it's more secure than over https..... because this facilities mutual TLS.

Does this make sense? Is there a better way of answering this question?

I would also like to know how Browzer changes this, as it avoids the need for a Desktop tunneller. Does this mean that traffic to Browzer is still over http.. or is it over https instead?

I look forward to your further comments about this.

[PhilipGriffiths]

I will give my thoughts, i am sure others may have better opinions:

The app may use HTTP but as soon as Ziti intercepts it its better than HTTPS – HTTPS is based on TLS, we are using m-TLS as outer encryption with E2EE as a further layer (E2EE can be switched off). In the picture below, the green line is HTTP.

Here are the different ways in which this is applied:

• App embedded SDK – Ziti encryption happens in app process/memory space so at no point is it HTTP on device
• Tunneler – encryption happens on host OS network, thus only ‘unsecure’ in a tiny window. I believe the only way a malicious actor could exploit this would be to be on the compromised device and do MITM before hitting ZT.
• BrowZer – encryption happens in the browser so I believe the malicious actor would need to hack into the browser to try to compromise.

i.e., HTTP may not be secure but OpenZiti massively increases the secure very easily making it more secure than HTTPS.

[markamind]

Thanks @PhilipGriffiths for these details. It helps fill in a few gaps.

One question that remains relates is how the interceptor works… and integrates with the browser.

I am assuming that when you run the Ziti Edge application… it starts a listener service… that allows the intercept to happen.

However… how does this interact with the browser… to then manage the traffic… so that when you enter the http address in the browser… it knows how to respond to what the Ziti Edge app returns.

This would be very helpful.

Scott

[TheLumberjack]

@markamind

The concern would be that if it’s going over http, it’s not secure.

it is absolutely a concern for the local, host OS if you are using a tunneller. all bytes on the host network are going to be 'in the clear'. If you were to run a packet sniffing program such as wireshark, you would see credentials in plain text. After the bytes are intercepted by that tunneller, then you get the full e2e encryption, and zero trust overlay protections. However, if you wrote a mobile app of your own and were NOT using a tunneller. Those bytes would be e2e encrypted by the OpenZiti sdk you adopted, all the way to the other side.

If you allow those bytes to be sent over the remote network, out/away from the OpenZiti overlay... Anything on that remote network which was able to get between the edge sdk on the 'far' side and the final destination of the HTTP bytes would ALSO be able to read the packets in plain text.... UNLESS of course, you use OpenZiti in the HTTP server - and then you're back to full e2e encrypted communications using an insecure protocol. Philips picture above shows that scenario.

I hope those words are clear enough. I didn't want to write too, too many words and make it harder to understand.

You can solve this by delivering HTTPS from the remote server. Then, even local/remote packet sniffing techniques won't work.

Browzer changes this by using TLS for everything locally, without the need to install a tunneling app. I wonder if @curt would agree but Browzer kind of is a tunneling app in my head. The main differences are that it's focused on http/https/web traffic exclusively, and requires no installation (and is not quite ready for prod yet as we know, )

[TheLumberjack]

Here’s a slide I use in some presentations. The “Zero Trust” areas show you where HTTP would be protected.

Top represents “edge router” to “edge router” traffic. All traffic on the host device and local network would use http in the clear

Middle represents “tunneller” to “tunneller” traffic. All traffic on the local networks is safe for http, but “host” traffic on both sides is in the clear

Bottom slide is what @PhilipGriffiths is showing. All traffic on local networks AND host networks is not trusted and safe for http, protected by OpenZiti as it is “app to app”.

The two sides are independent. You can mix and match any style you wish. For example you can mix app embedded zero trust and ‘network access’ shown below if you wish. Trusting the ‘server network’ fully, but not the ‘client host/local network’ at all.

Maybe that helps?

[markamind]

@TheLumberjack

You can solve this by delivering HTTPS from the remote server. Then, even local/remote packet sniffing techniques won’t work.

thanks for the clarification.. can you provide more details about this

why does this happen if the target server is over SSL... despite the browser showing that its over http?

The scenario that I want to showcase is a user updating their user account details.. where the target is protected via SSL

When I put a reverse proxy in front of this, I can redirect the SSL traffic to an end point that is accessed via an edge device.

I am pretty sure I am going to be asked some tough questions about this...

Do you have any recommendations on what I need to read up on to better understand the handover process..

how does the intercept know how to take control over the behaviour of the browser to protect the data?

[curt]

It can be argued that browZer shares characteristics with both the tunneling style of interception and app-embedding/sdk style of interception.

It may sometimes feel like a tunneler because the web app developer didn’t need to instrument their code (i.e. we did it automatically for them - touchless Zitification).

It can also feel like sdk/embedding because browZer doesn’t intercept every network operation on the host machine like a tunneler does. Instead, we only intercept network primitives (fetch/XHR/WebSocket) within the scope of the single Zitified web app (i.e. other browser tabs are ignored by browZer).

As far as TLS/mTLS/end-to-end-encryuption is concerned, the drawing below is one I use to help illustrate:

[TheLumberjack]

I’ll work up a reply. I might add it to the docs for this one. it’ll take a bit to formulate some thoughts

[TheLumberjack]

Here’s an image that hopefully helps. When you use http from your browser, http traffic will enter your host network as plain text. At that point, any packet sniffers installed on the machine, which are sufficiently privileged to monitor the network, can absolutely read the plain text packets from the network. After the packets hit the local network, the OS will route the packets to the proper location. When you’re using a tunneling app, the OS will direct the packets to the tunneling app (ziti). At this point, ziti takes over and you have secure, e2e encryption to ‘the other side’. If the other side is also a tunneling app (or your proxy for example) the packets are unencrypted again. When you send that traffic forward over local network, or as shown, on the host network, the packets are once again plain text.

When you use https - the packets which enter the network will NOT be in plain text. The packets the browser puts onto the local network will be encrypted by TLS (SSL is old news, TLS is ‘the new SSL’ fwiw ). The encryption, the TLS, will have been negotiated by the browser and the https server (which would have been done over open ziti). When the browser now puts packets onto the local OS network they are encrypted and safe/secure from packet capture. The same is true for the far side of the OpenZiti tunnel. When you send packets to the final, https server, destination - they are all protected by TLS and not susceptible to packet capture.

It is true that often times “TLS Termination” is something that load balancers are setup to do. Nowadays, I would not recommend that sort of setup. I would recommend using https (TLS) all the way, and not ‘trust’ the local os nor the remote network. What Ziti provides here that’s maybe the most attractive is the darkness (and all the other zero trust principles it already provides). An OpenZiti protected HTTPS server does not need any open, listening ports on the open internet. Now in your setup - you still have both, but that’s what I would think would be the next logical step, turning off the externally listening port.

As to what to read up on - I’d say it’s really the same as any reverse proxy. Looking for things like TLS/SSL termination at load balancer or TLS termination at reverse proxy or man in the middle reverse proxy might yield some good articles to read.

[TheLumberjack]

At the end of the day - it’s not possible to totally protect a server that is ONLY configure to send HTTP traffic. You can make it much stronger with OpenZiti though, MUCH stronger.

I really think what you’re looking for is to provide TLS from your http server at the far side. I think you own https://www.markamind.com/ would be my guess… I see that it has a legit cert:

I would recommend you make a “golang.zitified.markamind.com” cert, and teach the example golang http server to present https. THEN you will be doing TLS from the browser on the ‘client’ side. You’d go to “https://golang.zitified.markamind.com”, you’d get the padlock in the browser, it’d all look safe and secure AND you don’t need a listening port on the internet. That’s a PRETTY COOL DEMO imo already!!!

That make sense? I think might be what you’re missing?

[markamind]

Thanks so much @TheLumberjack @curt @PhilipGriffiths .. as I believe I now understand what is happening.. lots of moving parts here.

which also helps me improve my solution
and think of lots more question to ask.. but for later on.

To illustrate my learning, I will aim to bring this back to my situation..

When I visit my website directly.. I receive a padlock, that highlights that traffic is encrypted with the server certificate .. ie TLS

This however changes when I want to access my website over a zitified intercept address using a tunneller

The first step is to setup a reverse proxy where

1 origin server is my website
2 target server is a router intercept address

In this scenario, the intercept address is

`http://golanghttp.zitified:2000

Assuming that I have the tunneller app working (along with an enrolled identity on my desktop), when I enter the http address of the intercept in my browser.. the tunneller app knows what to do with this..

It says.. hey... I need to dial a ziti service.. which then calls the identity associated with the service (which is configured in the Ziti controller). Once the link is established.. the controller sends the request to the server identity.. which in my case.. is the reverse proxy.

the http request is clear text until it reaches the tunneller
from the tunneller onwards, all of the traffic is encrypted because its operating over a ziti network

The reverse proxy then dials the origin server to collect the data.. and then returns back the request.. passing all of the data back to the Edge tunneller

during this process.. all of the traffic is encrypted.. as my website is encrypted over TLS.. and the traffic up to this point has been over the ziti network

After the Edge tunneller.. the data is offloaded onto the host OS (my desktop) as clear text.. which is then picked up by the browser and rendered as clear text.

In this specific scenario, the configuration is a ZTHA.. which is a Ziti Host Access point.

The concern in this scenario is that other applications on my machine could be sniffing http traffic.

this would include filling out a login form to send data back to the origin server to authenticate the user.

If this is the case, then what is the point of all of this?

If I go back to the example of visiting my website directly over the browser.. the traffic is encrypted with the server certificate as data is moved back and forth between the browser and server.

the issue in this case is that

1. the server can be attacked because it requires open ports to allow remote access
2. the client credentials have not been validates before allowing the user to connect to the network.. to avoid "man in the middle" attacks

Hence, while server based certificates provides a strong layer of encryption that is the basis of securing the internet today.. its not a full solution to protect from cyber attacks.

This is why I like to describe Zero Trust Networks in terms of Yin and Yang.

To have highest level of protection, you need both server and client certificates

If you only have server side certificates, because its the current standard approach used today.. you will be constantly chasing your tail .. playing catchup for the next security vulnerability... and leaving yourself more exposed than is otherwise possible

Yes.. you can add on lots of additional technical capabilities and packet filtering tools to add more "server side" protection.. though.. this is all about treating the symptoms of inadequate protection.. rather than offering a solution that addresses the cause.

The cause of cyber attacks is largely facilitated by allowing unidentified traffic to connect to your network and hosts.. which why client side certificates are so important.

This is the fundamental basis of Zero Trust Networks..
or at least as I understand it in simple terms

In my scenario above, the client side certificates are enabled through the use of the Edge tunneller. This allows access to the reverse proxy server.. which is not accessible over a public DNS... as its a dark host that operates over ziti network

However, the weakness of this configuration is that the encryption ends at the tunneller.. rather than the browser.

[markamind]

So... how can you improve upon this?

Well .. after all of all the great insights everyone has provided.. I see a few different directions.. depending upon what is possible and what makes sense.

I have tried to offer a full range of options.. please add to them if you have further ideas.. or if they can be further improved upon.

1. BrowZer

I am very keen to use this.. as it is going to be the long term solution.

As I understand, it works natively within the browser to authenticate the user.. before being provided with access to the ziti network

It would be great to learn more about this, especially how to works with and manages client based certificates to identify the user.

2. Turn user account management functionality into an app

This would provide the means to integrate into the Ziti SDK for native end to end encryption.

However, there are some scenarios where this may not be possible.. and will still require a webapp.

3. Personalised Subdomains

After reflecting on the comments from @TheLumberjack, personalised subdomains could work.. though can it scale?.. it will require a bit more effort to setup... so is the effort worth it?.. maybe if it could be automated.

so what would it look like?

All users would log into the "user account" section of your website using their on subdomain.. One example is

' http://scott.markamind.com

This subdomain would have a server certificate to encrypt the data between the users machine and the server.

Ok.. but where is the client certificate?

Great question.. the personalised subdomain would act as a reverse proxy.. that is accessible over a public DNS

To provide the client certificate, I would enrol the identity for "Scott" on this instance of the reverse proxy. This would happen as a part of the user onboarding process.

Then.. as I go about updating my account information on my personalised subdomain.. I am accessing the backend server over the ziti network.

Now, my backend application server can be locked down and only provide connection with clients that that have been validated with a client certificate.

This is not necessarily a complete solution... as the reverse proxy can still be compromised to gain access to the client certificate.

If this did happen, the controls within the Ziti controller can be used to mitigate the scope of the attack.

Some options include

1. Multi factor authentication
2. Regular posture checking
3. Instant access revocation

When you compare this to a standard server certificate scenario, the advantage is the size of the surface attack.

Rather than exposing your entire app server to the public DNS.. it is limited to the individualised reverse proxy server per user.

As I understand, you can do something like this with a load balancer.. and put in application firewalls and other bits and pieces..to offer protection...

but then again.. such a solution cannot easily support client based certificates.. as I understand.. otherwise.. it would already be the standard today

However, this scenario does not compare to making your application server only accessible over a ziti network.

In my mind, I believe it worthwhile exploring how much can be automated to make it a viable solution... as there is a lot of ways it can be extended.

For instance.. rather than provide users with a reverse proxy.. what if they brought their own? This then changes the game.. as it allows the reverse proxy to be used by other user account services for other companies.

4. Ziti Browser

This is quite a radical idea..and way outside of my current capability.

In short, you take an open source browser like Firefox / Chromium etc.. and modify it so that it can only operate over a ziti network.

This would sort of like be like the Edge Tunneller.. where you enrol users into the network.. but rather than the encryption ending at the tunneller host, its now done natively in the browser.

Let me know if you have other suggestions / ideas.

Scott

[curt]

Hi Scott. Above you said:

As I understand, it works natively within the browser to authenticate the user… before being provided with access to the ziti network. It would be great to learn more about this, especially how it works with and manages client based certificates to identify the user.

In the coming weeks, I will continue rolling out more and more information detailing exactly how browZer users are authenticated onto a Ziti network, how certs are dynamically acquired to facilitate the mTLS connection(s) between the browser and the edge router(s), etc

Meanwhile, if you haven't already seen it, I recommend the following video which I published last week. It touches upon some of the things you are curious about.

In particular, leveraging an external IdP (Auth0) and integrating that with Ziti's new Auth Policies and External JWT Signer mechanisms, all within the context of the browZer environment.

Enjoy...

Touchless Web App Zitification - 4/28 browZer readout

[markamind]

Thanks for the update. This is really exciting.

[markamind]

@TheLumberjack

You’d go to “https://golang.zitified.markamind.com”, you’d get the padlock in the browser, it’d all look safe and secure AND you don’t need a listening port on the internet. That’s a PRETTY COOL DEMO imo

After a week of hustle and toil.. I worked out how to render the content from a ziti intercept.. to another server.. that has SSL enabled. In this case.. its a local host running on my laptop.

This demonstrates the use of 2 reverse proxy's

1. The first reverse proxy makes the content addressable over a ziti url..

however.. this is served as http when using a tunneller and accessing the ziti address in a browser
http://golanghttp.zitified:2000

2. The second reverse proxy serves the content from the ziti url.. to a localhost running over TLS.. now providing encryption all of the way to the browser

[markamind]

There are a few problems that I now need to work through..

#1. performance.. as its a bit slow
#2. understanding the following errors that come back from the second reverse proxy

Let me know if you have any tips... this could be a simple configuration change that may be required re the TLS timeout etc

22/05/14 20:50:39 proxy_url: localhost
2022/05/14 20:50:39 origin: http://golanghttp.zitified:2000
2022/05/14 20:50:39 proxy_url: localhost
ERRO[0227] rx error (read tcp 192.168.20.2:49300->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{N9QO}"
ERRO[0227] rx error (read tcp 192.168.20.2:49304->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{jYMO}"
WARN[0228] error initiating channel with hello _context="tls:168.138.10.79:8442" error="read tcp 192.168.20.2:49308->168.138.10.79:8442: read: connection reset by peer"
INFO[0228] defaulting to version 2
WARN[0228] Retrying dial with protocol version 2 _context="tls:168.138.10.79:8442"
WARN[0228] error initiating channel with hello _context="tls:168.138.10.79:8442" error="read tcp 192.168.20.2:49313->168.138.10.79:8442: read: connection reset by peer"
INFO[0228] defaulting to version 2
WARN[0228] Retrying dial with protocol version 2 _context="tls:168.138.10.79:8442"
WARN[0228] error initiating channel with hello _context="tls:168.138.10.79:8442" error=EOF
INFO[0228] defaulting to version 2
WARN[0228] Retrying dial with protocol version 2 _context="tls:168.138.10.79:8442"
ERRO[0228] rx error (read tcp 192.168.20.2:49319->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{jXrJ}"
WARN[0228] error initiating channel with hello _context="tls:168.138.10.79:8442" error="read tcp 192.168.20.2:49316->168.138.10.79:8442: read: connection reset by peer"
INFO[0228] defaulting to version 2
WARN[0228] Retrying dial with protocol version 2 _context="tls:168.138.10.79:8442"
WARN[0229] error initiating channel with hello _context="tls:168.138.10.79:8442" error="read tcp 192.168.20.2:49320->168.138.10.79:8442: read: connection reset by peer"
INFO[0229] defaulting to version 2
WARN[0229] Retrying dial with protocol version 2 _context="tls:168.138.10.79:8442"
WARN[0229] error initiating channel with hello _context="tls:168.138.10.79:8442" error=EOF
INFO[0229] defaulting to version 2
WARN[0229] Retrying dial with protocol version 2 _context="tls:168.138.10.79:8442"
ERRO[0229] rx error (underlay closed) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{NGJK}"
ERRO[0229] rx error (read tcp 192.168.20.2:49329->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{j4ek}"
WARN[0229] error initiating channel with hello _context="tls:168.138.10.79:8442" error=EOF
INFO[0229] defaulting to version 2
WARN[0229] Retrying dial with protocol version 2 _context="tls:168.138.10.79:8442"
2022/05/14 20:50:43 origin: http://golanghttp.zitified:2000
2022/05/14 20:50:43 proxy_url: localhost
ERRO[0229] rx error (read tcp 192.168.20.2:49341->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{NPrY}"
ERRO[0229] rx error (read tcp 192.168.20.2:49345->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{y7xE}"
ERRO[0230] rx error (read tcp 192.168.20.2:49353->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{jKKd}"
ERRO[0230] rx error (read tcp 192.168.20.2:49339->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{yZRp}"
ERRO[0230] rx error (read tcp 192.168.20.2:49357->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{N9AY}"
ERRO[0230] rx error (read tcp 192.168.20.2:49354->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{DkkZ}"
2022/05/14 20:50:43 origin: http://golanghttp.zitified:2000
2022/05/14 20:50:43 proxy_url: localhost
ERRO[0230] rx error (read tcp 192.168.20.2:49360->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{yxRm}"
ERRO[0230] rx error (read tcp 192.168.20.2:49364->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{j2rJ}"
ERRO[0230] rx error (read tcp 192.168.20.2:49365->168.138.10.79:8442: use of closed network connection) _context="ch{ziti-sdk[router=tls://168.138.10.79:8442]}->u{classic}->i{N3E6}"
2022/05/14 20:50:44 origin: http://golanghttp.zitified:2000
2022/05/14 20:50:44 proxy_url: localhost
2022/05/14 20:50:44 origin: http://golanghttp.zitified:20

[TheLumberjack]

What does “a bit slow” mean? My guess, looking at these errors and that description is that you might be somehow closing a connection and therefore causing the overlay to have to reestablish/retry connections?

I’m making that guess based on the ”use of closed network connection" error. That makes me think you’re somehow using a connection which is no longer usable because it’s closed. Unfortunately understanding why and where it closed is probably a fair bit of debugging. I would however start debugging in your proxy. This functionality is hardened pretty well on the client side (ziti tunneler).

I’d be interested in hearing if you have the same experience using the ziti-edge-tunnel at the far end. It can’t do the tls termination you’re currently doing but it might help you diagnose the issue.

Also, if you want to look at go code which performs this function you can look at the ziti-tunnel project. It predates the ziti-edge-tunnel and it’s pretty similar. It also won’t do tls termination like you are though, but it might serve as inspiration for your project?

I’m not sure if that helps or not.

[markamind]

Great tips… I will put in some more tracing on the second reverse proxy to investigate.

I was quite excited to achieve this… as its a full implementation of the golang SDK on both sides of the ziti network

[markamind]

Quick update… I think I have found the source of the performance issues… which relates to the Goloang reverse proxy. I need to write some extra code to pool the connections.

Designing a Reverse Proxy: Why Golang performs better than Java(Spring Boot),...