Please Explantion about: /etc/csf/csf.rignore

Post Reply
nadav123
Junior Member
Posts: 1
Joined: 23 Mar 2021, 11:07

Please Explantion about: /etc/csf/csf.rignore

Post by nadav123 »

hey, i see a post of someone his cron got a block and crawler bots as well.
viewtopic.php?p=20673&sid=bbe730ccbd82c ... 746#p20673

I notice sometimes google bot got a block with more other ones... what we can do about that?

I look on csf.ringore
but I didn't understand.
is not get IPs

so what i actually need to do if the crawler have like that:
crawl-66-249-64-130.googlebot.com

and if i need to comment out all of this:
# For example, the following are all valid entries:
# www.configserver.com
# .configserver.com
# .configserver.co.uk
# .googlebot.com
# .crawl.yahoo.net
# .search.msn.com

The file is not detailed, you dont know if is block them or give them a pass.
you didn't specified the word "comment out" so is can be something also.... no?

Please help and direction what is right and if i comment it out, is actually good or bad? i didn't get the explanation on the file
is not detailed right and people can do mistakes.
Sergio
Junior Member
Posts: 1685
Joined: 12 Dec 2006, 14:56

Re: Please Explantion about: /etc/csf/csf.rignore

Post by Sergio »

Hi.
if you have:
so what i actually need to do if the crawler have like that:
crawl-66-249-64-130.googlebot.com
Thenm on the csf.rignore file you should add:
.googlebot.com

The following is the reason why you have to write it like that:
google bots have a lot of different IPs and servers. All the servers are defined per example:
crawl-###-###-###-###.googlebot.com
where the "###" represents octects on the rage of 0 to 255 each
So, if you want to ignore all the google bots you have to do it like ".googlebot.com", and that will ignore any google bot servers.

Regards,
Sergio
Post Reply