Hello again. It's boring today. I decided to investigate this problem in more detail.
I found out that the main difference between laptop and PC is the revision version (TPM specification):
Laptop (works):
PhysicalPresenceVersionInfo : 1.3
SpecVersion : 2.0, 0, 1.59
Desktop (doesn't work):
PhysicalPresenceVersionInfo : 1.3
SpecVersion : 2.0, 0, 1.38
I started by looking at the WER results. I have 2 message types in the event log.
Application level
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="TPM" Guid="{1b6b0772-251b-4d42-917d-faca166bc059}" />
<EventID>17</EventID>
<Version>0</Version>
<Level>4</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2025-07-15T19:45:57.8846998Z" />
<EventRecordID>14974</EventRecordID>
<Correlation />
<Execution ProcessID="4" ThreadID="412" />
<Channel>System</Channel>
<Computer>DESKTOP-NUBVVAU</Computer>
<Security UserID="S-1-5-18" />
</System>
- <EventData>
<Data Name="driverFile">0x1e</Data>
<Data Name="lineNumber">926</Data>
<Data Name="TpmCommandOrdinal">349</Data>
<Data Name="TpmResponseCode">469</Data>
</EventData>
</Event>
TpmResponseCode is the one to pay attention to here. We can decompose the obtained value according to this specification, and get the cause of the problem: TPM_RC_FMT1 | TPM_RC_P | TPM_RC_SIZE | TPM_RC_1. So here I see the same error that I saw in the Ziti log.
NCrypt level
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Microsoft-Windows-Crypto-NCrypt" Guid="{e8ed09dc-100c-45e2-9fc8-b53399ec1f70}" />
<EventID>1</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>1</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000001</Keywords>
<TimeCreated SystemTime="2025-07-16T19:33:32.7842087Z" />
<EventRecordID>22526</EventRecordID>
<Correlation />
<Execution ProcessID="18672" ThreadID="5472" />
<Channel>Microsoft-Windows-Crypto-NCrypt/Operational</Channel>
<Computer>DESKTOP-NUBVVAU</Computer>
<Security UserID="S-1-5-18" />
</System>
- <EventData>
<Data Name="OperationType">6</Data>
<Data Name="ProviderName">Microsoft Platform Crypto Provider</Data>
<Data Name="KeyName">ziti://<IDENTITY_HASH>@<ZITI_CONTROLLER>:8441</Data>
<Data Name="KeyType" />
<Data Name="AlgorithmName" />
<Data Name="Status">0x80280095</Data>
<Data Name="ProcessName">ziti-edge-tunnel.exe</Data>
</EventData>
</Event>
Here the TPM returns 0x80280095 which corresponds to TPM_20_E_SIZE (TPM 2.0: Structure is wrong size).
I have read on the internet what this problem may be related to, and most often it happens when a larger or smaller size is passed with the selected hashing algorithm. As far as I know, OpenZiti uses mTLS for communication between client and server. And mTLS solutions at least previously signed the key as ECDSA P-256 + SHA-384 as one of the possible methods. I.e., the hash curve is used as 256 bits and the digest as 384 bits. Some TPM firmware may reject such requests, as in my case.
I decided to see if there would indeed be such a problem bypassing OpenZiti.
Reproducing the issue
First, I created a req.inf configuration file to create the key:
[Version]
Signature = "$Windows NT$"
[NewRequest]
Subject = "CN=TMP-TEST"
KeyAlgorithm = ECDSA_P256
KeyLength = 256
KeySpec = 1
Exportable = FALSE
MachineKeySet = TRUE
ProviderName = Microsoft Platform Crypto Provider
RequestType = Cert
[Extensions]
2.5.29.37 = "{text}"
_continue_ = "1.3.6.1.5.5.7.3.2"
I specified a 256-bit key in the configuration. Then I tried signing with a 256-bit digest and a 384-bit digest and then looked at the results.
I then imported the key into TPM:
PS C:\Users\wusik\Desktop> certreq -new req.inf tmp-tpm.cer
Installed Certificate:
Serial Number: 51ca4ab4a31bada743732304ec50efaf
Subject: CN=TMP-TEST
NotBefore: 7/16/2025 2:23 PM
NotAfter: 7/16/2026 2:43 PM
Thumbprint: 3aa36e010ae486417403a59df8a65c530696aa1a
Microsoft Platform Crypto Provider
tq-ad366111-8a8d-44b8-8381-89c3e6e3585c
CertReq: Certificate Created and Installed
PS C:\Users\wusik\Desktop> $prov = New-Object System.Security.Cryptography.CngProvider "Microsoft Platform Crypto Provider"
PS C:\Users\wusik\Desktop> $key = [System.Security.Cryptography.CngKey]::Open(
"tq-ad366111-8a8d-44b8-8381-89c3e6e3585c",
$prov,
[System.Security.Cryptography.CngKeyOpenOptions]::MachineKey
)
PS C:\Users\wusik\Desktop> $ecdsa = [System.Security.Cryptography.ECDsaCng]::new($key)
PS C:\Users\wusik\Desktop> $data = [Text.Encoding]::ASCII.GetBytes("hello TPM")
PS C:\Users\wusik\Desktop> $h256 = [System.Security.Cryptography.SHA256]::Create().ComputeHash($data)
PS C:\Users\wusik\Desktop> $h384 = [System.Security.Cryptography.SHA384]::Create().ComputeHash($data)
Laptop:
PS C:\Users\wusik\Desktop> try { [BitConverter]::ToString($ecdsa.SignHash($h256)) ; "OK" }
>> catch { "Error: 0x{0:X8}" -f $_.Exception.HResult }
F1-96-B2-C8-7B-39-14-E7-80-9A-4C-7F-B3-30-E4-FA-09-11-1B-03-C7-36-A8-CC-FA-31-9D-E2-C1-41-B0-2B-C9-7B-9B-94-79-07-FB-42-2C-65-E7-B2-B6-35-41-89-A9-CD-00-1B-A2-E5-57-42-5F-1A-32-6A-45-EF-00-15
OK
PS C:\Users\wusik\Desktop> try { [BitConverter]::ToString($ecdsa.SignHash($h384)) ; "OK" }
>> catch { "Error: 0x{0:X8}" -f $_.Exception.HResult }
BE-62-EE-7B-19-AF-B0-5D-2B-05-A8-30-F5-28-A4-B0-60-FD-0A-81-85-7C-92-DD-5F-24-73-B9-C3-55-B2-D0-E9-42-13-86-E0-95-DE-76-5E-D2-D9-C4-07-D7-7E-16-07-E8-A2-8F-EB-E7-0B-9F-8D-7D-B1-35-E9-43-F1-50
OK
PS C:\Users\wusik\Desktop>
PC:
PS E:\Desktop> try { [BitConverter]::ToString($ecdsa.SignHash($h256)) ; "OK" }
>> catch { "Error: 0x{0:X8}" -f $_.Exception.HResult }
5E-30-16-C0-98-9F-2A-ED-2D-A2-EC-1A-42-6E-58-6B-4F-D6-11-E4-3C-49-B5-14-45-23-EF-D4-18-44-3B-DF-C1-96-FF-9A-68-34-63-DD-99-24-00-61-91-E8-7B-6C-F3-23-FC-37-44-21-E3-44-4D-28-8E-13-28-FF-9D-A9
OK
PS E:\Desktop> try { [BitConverter]::ToString($ecdsa.SignHash($h384)) ; "OK" }
>> catch { "Error: 0x{0:X8}" -f $_.Exception.HResult }
Error: 0x80131501
Don't worry about this error code. It's the .NET Runtime result. Now I can see event after this command:
NCrypt event
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Microsoft-Windows-Crypto-NCrypt" Guid="{e8ed09dc-100c-45e2-9fc8-b53399ec1f70}" />
<EventID>1</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>1</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000001</Keywords>
<TimeCreated SystemTime="2025-07-16T19:33:24.0839784Z" />
<EventRecordID>22524</EventRecordID>
<Correlation ActivityID="{7a0c9ed7-f5ce-0004-3085-107acef5db01}" />
<Execution ProcessID="19340" ThreadID="22420" />
<Channel>Microsoft-Windows-Crypto-NCrypt/Operational</Channel>
<Computer>DESKTOP-NUBVVAU</Computer>
<Security UserID="S-1-5-21-3428362137-4034877142-4161271481-1001" />
</System>
- <EventData>
<Data Name="OperationType">6</Data>
<Data Name="ProviderName">Microsoft Platform Crypto Provider</Data>
<Data Name="KeyName">tq-67063e4b-4c60-42de-98cf-ec0ccb48a9df</Data>
<Data Name="KeyType" />
<Data Name="AlgorithmName" />
<Data Name="Status">0x80280095</Data>
<Data Name="ProcessName">powershell.exe</Data>
</EventData>
</Event>
And here I see a completely identical error.
Next, I looked at the list of active keys in the TPM, and the OpenZiti key is not there. Apparently, the process is interrupted at the stage of transfer to TPM.
First, I looked at what algorithm JWT is signed with.
wusik@DESKTOP-NUBVVAU:Downloads cut -d. -f1 user.wj-pc.jwt | base64 -d
{"alg":"RS256","kid":"1f55069d3d9d79806104cfa70007e4fd1c2b3bbf","typ":"JWT"}base64: invalid input
wusik@DESKTOP-NUBVVAU:Downloads
Then I checked the created identity on the basis of which further interaction between ZDWE and TPM takes place. OpenZiti stores active connections in the C:Windows/System32/config/systemprofile/AppData/Roaming/NetFoundry folder. Using certutil, I have obtained the given certificate. I probably shouldn't share the full commands here because of the confidential information in the key. I will provide the data that I got in the end.
Key read result
Certificate X509:
Version: 3
Serial Number: 03bf82
Signature Algorithm::
ObjectID of algorithm: 1.2.840.113549.1.1.11 sha256RSA
Algorithm parameters:
05 00
Issuer:
CN=s1-dev-signing-intermediate
OU=ADV-DEV
O=NetFoundry
L=Charlotte
C=US
Name hash (sha1): e0c65aa60df60883d1eeac9ffbbd5d32bb40f593
Name hash (md5): 9c5fbf997999ff673581c8eb20d253e1
NotBefore: 16.07.2025 22:50
NotAfter: 16.07.2026 22:51
Subject:
CN=GFYYku3MP
O=OpenZiti
Name hash (sha1): 11ac3fcb8b8bf517d169d1509dec87316b0ef374
Name hash (md5): 50acf702cbb1dc06ab4eef1d5aa32915
Public Key Algorithm:
ObjectID of algorithm: 1.2.840.10045.2.1 → ECC
Algorithm parameters:
06 05 2b 81 04 00 22 → 1.3.132.0.34 ECDH_P384
Public Key Length: 384 bits
Public Key: UnusedBits = 0
<BYTES>
Certificate Extensions: 3
2.5.29.15: Flags = 1 (Critical), Length = 4
Key Usage:
Digital Signature, Key Encipherment, Data Encipherment (b0)
2.5.29.37: Flags = 0, Length = c
Enhanced Key Usage:
Client Authentication (1.3.6.1.5.5.7.3.2)
2.5.29.35: Flags = 0, Length = 18
Authority Key Identifier:
KeyID = 1facf6094daabb8792a436b70c94c75132a05d4c
Signature Algorithm:
ObjectID of algorithm: 1.2.840.113549.1.1.11 sha256RSA
Algorithm parameters:
05 00
Signature: UnusedBits=0
<BYTES>
Not a root certificate
KeyID hash (rfc-sha1): 04edfad648b1e9f434da230a11de7046ed385f94
KeyID hash (sha1): 6e362aaeea48908c42aafc8fb5ee2acc850e71ce
KeyID hash (bcrypt-sha1): 26f1618ab921510535df4e2383cdb57e9c424126
KeyID hash (bcrypt-sha256): 56892301fe43e71c4c6b08f745b369014a840f86da7aaf3bb3a793de7d42d231
KeyID hash (md5): cf72d95b98ceafc842c6c39f3a7820c8
KeyID hash (sha256): 396ca693d13076e3537f68b46ee6056f8eb7e42e373a9fb744c135bd88895c40
KeyID hash (pin-sha256): tiGa7zip1OfGx5mTLyCExmg1BacwvDbx6oBHF5cqnYo=
KeyID hash (pin-sha256-hex): b6219aef38a9d4e7c6c799932f2084c6683505a730bc36f1ea804717972a9d8a
Certificate hash (md5): 109968b2a74c283c6eaf1a6c5f55de9e
Certificate hash (sha1): 9a0ea92d514aa7c505d1455082253563a6e604f1
Certificate hash (sha256): e2b3f56f227efa90a2c092102977120f2571b3ff52527e7224f50a5ffe73c359
Signature hash: 8a7b7a20f325ebfd2165e2fcafbbd6592ec119dd35864783041a89ca99218f8c
And here you can see that the JWT is signed with a 256-bit key, and the certificate in the JSON file has a 384-bit digest. There is a mismatch between the expected data and the algorithm, and TPM returns an invalid size error.