Is it possible to specify frontend's robots.txt?
User-agent: *
Disallow: /
Is it possible to specify frontend's robots.txt?
User-agent: *
Disallow: /
EDIT: besides a frontend config, you supply a robots.txt at the root of your proxy or web shares
No, not currently. This option for public HTTP and private HTTP or TCP shares would eliminate the need for another proxy to handle those requests for /robots.txt
, and that's the only way I can think of doing it at the moment.
I would probably use that! I'm doubtful the public frontend should prescribe the response for all shares, since zrok is designed to support many unrelated users. It could, however, define a default for shares that don't specify a robots.txt.