I want to implement a system where i establish connections to many CDE (cloud development environments) from the main server (operating as a proxy) for end users via ziti. The question I have, is it better to create a new identity for each cde and then create a custom server -> cde policy/service or is it better to do a server -> single edge router an then handle access control with application logic
Hi @sam-ulrich1, welcome to the community and to OpenZiti!
I'm unsure if I understand the question 
. In general, if this was me, I think I would make one identity for the server, and grant that server access to services using policies. I feel like maybe each CDE runs on its own network, hence this part of your question: "create a new identity for each cde"? Each offload point will definitely need their own identity. Provisioning an edge router with tunneling mode enabled will make an identity for you (when you provision the router) as opposed to making an identity and deploy the ziti-edge-tunnel for example. Both are good options, one way the identity is a bit more implicit (the edge router way) and one way it's explicit (ziti-edge-tunnel) is all.
The users will connect to the CDE over Ziti as well I'd assume? I am just having a hard time seeing the overall topology and it makes it hard for me to give a good answer. If you could give me a couple of examples or more details, I think I could give you a better answer. I'm not sure if this helps so far. 
@TheLumberjack Thanks for the time. You mostly understand it. The main question is the latency cost (to the application experience not the network) for dynamically creating and destroying identities, services and policies in an online environment. If the latency cost is low and the overhead of hundreds of identities, services and policies is not a problem then i think I'd prefer to create a new identity and dedicated service/policy for each CDE. If the latency is high then I'd prefer to have just a tunnel inside the k8s namespace that exposes the network to the main server. Here are 2 diagrams to visualize the options.
If you are interested, you can experience this yourself right now if you want. This site https://appetizer.openziti.io/ will provision an identity for you and assign that identity the rights to access a particular set of services that I've already deployed. Or you can just watch this gif, but feeling it live is maybe better:
The overhead of creating an identity and assigning it to policies and whatnot is totally negligible imo. If you supply the same name over and over, this server will delete and recreate that identity over and over. So you can try it/feel it... (and if you're inclined, give the appetizer demo a go and provide feedback good or bad? ) it's something I'm working on "literally now" .... heheh
I assume each CDE is a pod based on the K8S/Namespace usage... So if each pod had a sidecar, and you assign a service/identity dynamically like this, and the "server side" (the K8s pod) already had the service assigned to that identity and service, it'll be available "as soon as you make the client identity" - exactly like the way this demo works.
To recap this demo, it:
- asks for input from the user
- user clicks the button
- sends a get request to a url with a query param
- parses the param, finds the name of the identity to generate
- deletes the identity if it exists, then adds it back using attributes to an already existing service policy (bootstrapped during server startup)
- user can then access the service
Again, I'm not sure I understand but I think so? Hth. Also -- SUPER COOL PROJECT! I'm interested in how it shakes out!!!
Thanks man! This is perfect. You can checkout the project @ GitHub - Gage-Technologies/gigo.dev: Gigo is an end-to-end platform for learning to code and developing your skills. Gigo won't drop you off after some basic syntax, buckle up to build real applications with the latest tech in the most streamlined way possible.
We currently have a forked version of Coder v2 @ GitHub - coder/coder: Provision remote development environments via Terraform which uses a tailscale system and DERP server but it's flaky (Coder's implementation is better apparently) but I'm tired of dealing with the DERP and the half OS tailscale stuff. Ziti is more aligned with the principle of what we want which is a true API configurable tunnel mesh
2 Likes
That demo is sweet btw!
2 Likes
new GitHub 
for you ! Let us know if we can help out more.
One more question. Our server nodes are stateless so while they normally shutdown cleanly, occasionally we may lose on without a clean shutdown. Same thing with the CDEs (they normal shut down cleaner though). Is there anyway to configure a timeout on an identity so the ziti controller auto cleans it up after x amount of inactivity. I assume a bunch of orphans isn't good for the system
Not at this time. That's something you'd have to automate on your own. I haven't added that actual code to the appetizer demo yet, but I plan to do that sort of thing eventually for the appetizer/demo site. Right now they accumulate. It's not meant to be robust and stable (obviously, it's a demo) but it' d be nice to have rate limiting, automated cleanup and whatnot on it for hygiene's sake.
If I get around to adding some sample type code, I'll share it here/post back.
Thanks man. I can just build in a sub routine to handle it. Just wanted to see if there is an idiomatic way to handle it. When possible I prefer to offload logic to dependencies
1 Like
@andrew.martinez - any thoughts around that spark for you? As the idea bounced around in my head, this almost feels like "ephemeral" access. You might have some ideas around the idea for short-term-authorized workloads like this. Might be an interesting idea in here for people looking to do that kinda thing... Maybe?
One other thing actually, currently our system basically exposes the entire pods local net to the server then all of our access is managed by application logic. I'd like to preserve that kind of idea since provisioning ports on the fly is a bit annoying (but I'd happily do that if necessary). Is there a way that a service can expose literally all of the ports on the target identity? Since all of the access is proxies by our system I don't see any security issues. Additionally, the CDE needs to operate as a full on machine in the cloud so offering access to any port as it becomes available is helpful
Ephemeral Identities or Short Lives Identities isn't something I have heard us consider, nor has it been something I have thought deeply about. I also can't recall anyone else asking for them.
Without a clear understanding of how other people would use/need this kind of functionality my initial reaction is to let this be out of scope for OpenZiti.
I did sit here for a bout 15 minutes brainstorming different ways something like this could be implemented:
- Time to live
- Dead man's switch
- Spawning sub identities off a main identity (for policy reasons)
It does somewhat "depend". When you're offloading from OpenZiti, you can do whatever you want. Usually the ports are more interesting to tunnelers and to the "intercepting" side.
I don't know what the plan is to onboard to the OpenZiti fabric (from the client itself or from the server), but regardless... I'll assume "somewhere" you're using a tunneler and you're not using application embedded zero trust (though honestly if it were me, i'd just bake ziti right into the server). Anyway, assuming you'll have an intercepting tunneler you can specify port ranges in the intercept. So you would do something like:
ziti edge create config private-postgres-intercept.v1 intercept.v1 '{"protocols":["tcp"],"addresses":["zitified-postgres"], "portRanges":[{"low":5432, "high":5432}]}'
Notice the port ranges. You can set that to 1-65535, specify two sets of ranges (1-1000, 2000-3000) or whatever. Then on the far side you instruct that offloading tunneler to offload the port that was intercepted. So whatever port came in is whatever port goes out:
ziti edge create config "ubussh.host.v1" host.v1 '{"protocol":"tcp", "address":"localhost","forwardPort": true, "allowedPortRanges":[{"low":1, "high":16384}] }'
Here notice two things. First I have "forwardPort" set to true, and that forces me to declare the ports that are allowed. Here, ports 1-16384 would be fine, but port 50000 would not be allowed to egress.
I'm building ziti directly into the server and the agent so I won't be using a tunneler. I think I've figured it out through the API though. Once the whole thing is done I'll link the code
Oh - excellent! Much easier that way for sure. If you run into any snags, do let us know!
So the big thing I'm facing trying to integrate ziti is that I'm not sure the best way to do port-range forwarding. My thoughts are:
-
Create a service for each agent that exposes the full port range. My problem here is that it's not entirely clear to me how to perform the forwarding at the point of the agent. Are you aware of any projects I could look at? My problem is I don't know how to inform the agent when the server forwards a connection to the service what local port the agent should be forwarding to.
-
Create a service by the server on the fly for each port the end user wants to access. Then, on the agent watch for new services and expose them on the fly. This one makes more sense to me w.r.t. correlating an incoming connection to a local port but I don't know what the user latency would look like. This seems to be more like how the tunneler you provide works just natively built into my application. I've used the tunneler for other things but never dynamic forwarding for short-lived (15m-1h) connections. My previous uses were always permanent forwarding
I'm open to other ideas though!
The user's browser will connect directly, via underaly IP (not openziti) to the server in your diagram, right? Then that server will understand what CDE that user wants to get to and figure out what 'service' to dial, which is where the user's traffic to send traffic to for that particular user, is that right?
So, if I understand, the server needs to know how to send traffic to the 'correct' gigo identity/agent. From the gigo agent, that agent needs to know what port to send traffic to? Do I have that overall design correct?
The easiest answer, if it's a 1:1 would be just use the same port over and over. The richer way to do this is to make your own config type and config per "type" of server. If it's a java server, use port 90, golang use port 100 that sort of thing. I think that'd be pretty easy and straightforward. You can stuff all that information into one single config type if you like, or you can have multiple config types and then logic based on which config types that identity has access to.
Does that make sense? You're basically doing the same sort of thing the tunnelers do. You're processing traffic at the agent and deciding where to offload it. OpenZiti comes with 5 built in config types like that "host.v1" I referenced before. Then each service gets a "config" of "config type host.v1". You can make your own config types though and do that same sorta thing. I think that'd be pretty easy/straightforward