Users locks randomly on the controller

Hi everyone!
Happy new year!

I've been experiencing a very anoying issue about creating admin users that locked after a while.

From the last week I've created those users as always, but now they freeze and are unable to login again after been logged from the API and ZAC. Default Admin also has been locked!

But some users can still login like nothing happens and I don't know what to think about.

The SDK I am using for my deployment is v0.31.0 and dockerized ZAC latest

Please, give me a hand



Hi @MoonCraver. That sounds like something I've never personally experienced. I've never seen the admin user get locked. In the past, we had a very annoying bug when running in docker that the password would get changed in the .env file, but that wouldn't change the actual admin user password.

I don't know of any functionality in ziti that "locks" a user. Are you able to reproduce the issue reliably? Are you automating this process? Is it possible your automation is doing that same sort of thing our docker continer used to (overriding/recreating the users)? To be honest, that's the only thing that stands out to me. As far as I know, there's no functionality that locks a user, but I might just not know it exists.

Have you looked through the controller logs to see if something interesting is logged? Can you explain more about how you're using "the SDK for deployment"? Any other details might help, if this doesn't...

HI @TheLumberjack . I admit that is the strangest thing I ever experienced as well.

I dont't see any irregular trace about the users right now, 'cause I've restarted the controller service.

In this case this controller is on a developers environment with full service. Identities are working fine but the only anomally detected was the admin users started to get 401 before login for the first time when using let's encrypt certificates one week ago.

Also returning to the nodejs deployment with the Ziti PKI didn't work, sadly.

SDK for deployment it's the way I name the ziti binary file from the releases :smiley:

I'll be checking tomorrow, with a work colleage, debugging the trace for the issue.

I'm not quite sure what you mean. Are you logging in using ZAC itself and you're finding the login doesn't succeed or are you using the ziti CLI? You're not replacing the PKI at all are you or are you possibly deploying more than one controller?

If you can reproduce the issue and have some clear steps to reproduce, I'm happy to try it myself and reproduce on my side too. I log into more than one controller and sometimes that can cause problems if I'm not paying attention to the controller being logged into, but I expect you just have a single controller? Any extra info you have to share, that'd be helpful. I'll look for whatever you discover tomorrow.

Both of them. For policy operations or checking on services, I'm login to ZAC, but for API operations like creating Admin users or edge-routers I'd prefer to use CLI. I only have one ZAC for all of my controllers

For the nodejs deployment I've followed the guide from using the ziti pki deployed in the quickstart, but for the docker deployment I've put let's encrypt certificates for a change.

And just the beginning of my headache with this change I made.

Login to my other controllers have no problems at all, only for the oldest one I created 3 years ago.

You mention you're making 'admin' users, is there any chance one of those admins is changing their password but changing your admin user mistakenly? I've never seen a controller reset/lock a user. Everytime I've had login problems it's been something I've done somehow.

I feel like without steps to reproduce this bug, it might be a mystery. :frowning:

Hi @TheLumberjack,

I think the db corrupted and is blocking the users who have admin permission. Today, one of the has been operating normally till go unauthorized without reason.

There's a way to check on the db directly and see the registries? Or, at least modify registries for admin users. We almost lost all admins in this controller.

There are a couple of tools that might help. Note that the DB is locked when opened, so only one process can access it at a time. In other words, the controller can't be running if you want to poke at the DB file with another tool.

  1. ziti ops db explore will let you poke around in the DB.
  2. ziti ops db add-debug-admin will add or reset an admin user. It was added in v0.30.5.

What OS are running the controller on? Let us know if find evidence of corruption and/or have a way to reproduce it.


Hi guys. Hope I found the error crashing the admin users. I'm testing right now, but was a name for an admin registry. The db didn't like to work with his ID :smile:

Actually, if everithing works fine, will be solved tomorrow.

I'll reach you if there's any news

HI there!

I've finally discovered about the error Admin users enters in "disabled": true; after 3 to 4 hours. I don't know if theres a new function about this, but will be usefull to add a new functionallity to ops, like changing this parameter from true to false and viceversa without login.


That is really odd. I've never seen an identity set to disabled. Do you happen to have any automated processes that do this?

@andrew.martinez -- is there any automated process within the controller that might set set disabled: true? Can you comment here please?

The only process I know of is if the authentication policy is set to lock accounts after a certain number of invalid authentication attempts. The default is zero which is to never lock. Someone would have to either change the default authentication policy or assign a new one.

If they did, there might be an external automated process that's causing invalid authentications.