I actually got this resolved.
There is that groups settings and it resolved the problem. Take it some time to understand that I needed to add that groups settings to both dialers and listeners 
As the docs says
groups- (optional, [default]) - Both dialers and listeners can now specify a set of groups. If no groups are specified, the dialer or listener will be placed in thedefaultgroup. Dialers will only attempt to dial listeners who have at least one group in common with them.
Now I have following config on internal routers and it works.
