Upgrade from standalone to HA setup

Support General Questions
1

I was following ziti/doc/ha/upgrading.md at release-next · openziti/ziti · GitHub to move my current controller to HA setup as per recommendation. My controller certificate does contain spiffe uri but still controller start up failed.

{"error":"invalid controller certificate, no controller SPIFFE ID in cert","file":"github.com/openziti/ziti/controller/raft/raft.go:654","func":"github.com/openziti/ziti/controller/raft.(*Controller).validateCert","level":"fatal","msg":"controller cert must have Subject Alternative Name URI of form spiffe://\u003ctrust domain\u003e/controller/\u003ccontroller id\u003e","time":"2024-08-01T23:38:31.851Z"}

From the error message it looks like it expects the spiffe format in certain pattern.

spiffe://<my-trust-domain>/controller/<controller-id>

Is that correct? And if yes, may I know why is that restriction?

We use spiffe uri in all of our internal CA issued certificates and those do have a pattern and mean something internally. Could you please advise?

Thank you.

2

We use spiffe ids so that controllers can identify each other and extract the controller ID. For that to work we need a known pattern. We could theoretically use the whole SPIFFE ID as the controller ID. That could make other things awkward though, since whole URI is generally going to be much longer.

Since the cert is for the controller (and presumably is exclusive to that controller) is there an issue with having the SPIFFE id conform to the expected format? Alternatively, you can provide multiple SPIFFE IDs. Can you just have both ids in the cert?

Let us know your thoughts and we can discuss.

Cheers,
Paul

1 Like
3

Unfortunately the spiffe id is implemented in a certain format for us which maps to a unique identity across the company in a multi-tenant environment so it cant deviate from the format. The internal CA server issuing certificate mandates that format and there are way too many use cases / implementations which extract the identity from the spiffe id relying on that format. So its safe to say I dont have the liberty to change the format. :frowning:

Regarding having multiple Spiffe ids, Spiffe spec doesnt allow more than one atleast for x509 svid - spiffe/standards/X509-SVID.md at main · spiffe/spiffe · GitHub

From what i can see, the most flexible option would be to allow a list of spiffe ids in controller raft ( or other ) configuration as an optional construct.

Please let me know if i can provide any other information.

4

Also I was thinking to assign the same identity to all the controllers ( i.e same spiffe id ) and treat multiple controller deployments as “instances” of that identity. Would that be a problem?

Everything in our authorization system works based on the identity. So if controllers have to have diff spiffe ids then i will have to specify N entries for N controller instances but thats doable.

5

That would definitely cause problems. We use the controller ids internally for a variety of things, and if there are duplicates then things will break.

We're brainstorming some ways to support your use case, I'll let you know what ideas we have.

cheers,
Paul

1 Like
6

that part i can still manage ( assigning unique identities to diff controllers ) but if you can provide some option without having to change spiffe id format, that would be great. Thank you. :pray:t4:

7

This is quite the pickle.

In OpenZiti we are moving towards adopting SPIFFE IDs in all our components in order to make it easier to identify certificate ownership and direct manage of PKIs. The concept of OpenZiti having its own SPIFFE trust domain has a nascent form in both standalone controllers and HA controllers. Additionally this concept is slated to spread to routers and SDK certificates.

In OpenZiti it is currently assumed that the server and client certificates used in OpenZiti will be part of the OpenZiti trust domain. The SPIFFE spec doesn't allow for a single certificate to be a member of multiple trust domains.

It sounds like in your situation, that your organization has decided that all certificates are part of the organizations trust domain(s). This inherently makes it impossible to ever work with any other software that uses SPIFFE IDs where the path implementation conveys information to the implementor. This seems like a severely limiting design choice that destroys interoperability with other SPIFFE implementors.

Working around this problem would add complexity to the OpenZiti implementation and additionally could have security implications as we are moving towards more fluid enrollment mechanisms. We considered the following:

  1. Allowing regular expression matching on SPIFFE ids
  2. Introducing a specific ziti:// SPIFFE ID format

Option 1 sounds straight forward until you consider that pattern matching could fail for a number of reasons and cause two different certificates to appear to be the same but have different paths. This opens a number of attack vectors and opportunities for poor configurations that are attack prone. Additionally during troubleshooting, it adds a layer of complexity, needed to dereference the id from the SPIFFE ID + configuration. It would be a hurdle for level 1 and 2 support.

Option 2 moves of away from the SPIFFE specification and reduces OpenZiti's overall interoperability while still adding complexity for troubleshooting as what is used is dependent on the issuer.

If you have further ideas I am open to hearing them. However, I do not see a solution that doesn't have downsides that are untenable.

1 Like
8

@andrew.martinez would you mind elaborating on "This inherently makes it impossible to ever work with any other software that uses SPIFFE IDs where the path implementation conveys information to the implementor. This seems like a severely limiting design choice that destroys interoperability with other SPIFFE implementors."
As far as I understand, there is no hard-and-fast rule in spiffe spec which says you have to use diff trust domains across an organization and that choice is left to implementors.

I am not sure I understand why openziti relies on having its own trust domain? If the primary objective of spiffe is to have a unique identity, why should it matter if its part of a same trust domain or 2 diff trust domains? I feel its a fair assumption that any organization who is already using spiffe is going to have some structure in their spiffe ids.

would love to hear your thoughts.

1 Like
9

would you mind elaborating on "This inherently makes it impossible to ever work with any other software that uses SPIFFE IDs where the path implementation conveys information to the implementor'

The SPIFFE spec does not specify what a SPIFFE URI's path represents. The spec leaves it up to the implementor. However, it does state that the implementor may elect to use it to describe hierarchal information (SPIFFE-ID 2.2) and then provides some examples.

When an organization declares that only its SPIFFE ID format will be used within a certificate, it also implies that no other software can use the path information to store hierarchy information as it cannot guarantee the path formatting. Essentially, one implementor is saying only their format(s) is allowed, and any other format is invalid.

One could say that an agreement between the implementor's format and software could be made (i.e., "make it configurable"). However, there is no specification for that, meaning it would be a one-off implementation or a generic parsing solution. A generic parsing solution raises issues regarding declaration/parsing and poor configuration, adding complexity and attack vectors.

As an example, in OpenZiti, we are planning on moving towards network-specific SPIFFE IDs with hierarchal path information:

  • Controllers: spiffe://<trust domain>/controller/<id>
  • Routers: spiffe://<trust domain>/router/<id>
  • Identity: spiffe://<trust domain>/identity/<id>
  • API Session Certificate: spiffe://<trust domain>/identity/<id>/api-session/<id>/certificate/<id>
  • ...among others

If someone says, "OpenZiti has no agency over their SPIFFE IDs," then OpenZiti cannot implement the features that seek to leverage SPIFFE IDs where the path conveys hierarchy information.

SPIFFE IDs can be used opaquely between implementors (i.e., using the URI as a single value). This is only spec assured usage when working between SPIFFE ID implementors.

As far as I understand, there is no hard-and-fast rule in spiffe spec which says you have to use diff trust domains across an organization and that choice is left to implementors.

There isn't and that isn't the issue. The trust domain could be the same. The issue is declaring eminent domain over the path formatting.

I am not sure I understand why openziti relies on having its own trust domain? If the primary objective of spiffe is to have a unique identity, why should it matter if its part of a same trust domain or 2 diff trust domains?

OpenZiti doesn't have to have its own trust domain. The only requirement is that OpenZiti requires it to be freely able to use its own path format when specifying SPIFFE IDs. If the trust domain OpenZiti is being instructed to use has its own required path formatting, then there is an issue.

I feel its a fair assumption that any organization who is already using spiffe is going to have some structure in their spiffe ids.

I agree.

3 Likes
10

Thank you for the explanation. Now I understand, where you are coming from. I will have to go back to the drawing board and see how we can find a middle ground between our current implementation and use cases like above.
Going back to your options from the earlier response, would it be possible for Openziti's spiffe implementation to accept a format / prefix as a way to understand current spiffe format?

e.g. say my certificates are issued with spiffe format - spiffe://<trust-domain>/ns/<namespace>/sa/<sa>
Can the configuration input to Openziti be spiffe://<trust-domain>/ns/*/sa/* as a format and the actual spiffe uri in the certificate will be spiffe://<trust-domain>/ns/<namespace>/sa/<sa>/controller/<controller-id>

11

I was honestly thinking that same thing. A default of openziti but can be configured. That would at least provide a way to separate usages in mixed environments.

Another option is using sub-domain in the trust domain: mytd and openziti.mytd for example.

1 Like