maxinstuff , 6 days ago The real problem is robots.txt is an honour system in the first place - It's never been a defence against bad (or even simply poor faith) actors.
The real problem is robots.txt is an honour system in the first place - It's never been a defence against bad (or even simply poor faith) actors.