Call for Participation in Update robots.txt standards Community Group
Posted on:The Update robots.txt standards Community Group has been launched: Robots.txt is currently based on opting out of what you do not want your website to be a part of. This is hard to maintain (almost a full time job right … Continue reading