I have 2 public routers and 2 private routers. From private routers I have commented out listeners, so those connect to public ones. This I understand.
There is that groups settings and it resolved the problem. Take it some time to understand that I needed to add that groups settings to both dialers and listeners
As the docs says
groups - (optional, [default]) - Both dialers and listeners can now specify a set of groups. If no groups are specified, the dialer or listener will be placed in the default group. Dialers will only attempt to dial listeners who have at least one group in common with them.
Now I have following config on internal routers and it works.