An IP from China got caught by lfd trying to access a cpanel account. However, China is country blocked (CC_DENY), and the IP is in the blocked range, as confirmed by csf (csf -g output below). Can someone explain how the IP managed to access cpanel even though it is in iptables? Thanks.
csf -g 203.2.xxx.xxx
Table Chain num pkts bytes target prot opt in out source destination
No matches found for 203.2.xxx.xxx in iptables
IPSET: Set:chain_DENY Match:203.2.xxx.xxx Setting: File:/etc/csf/csf.deny
IPSET: Set:cc_cn Match:203.2.xxx.xxx Setting:CC_DENY Country:CN
ip6tables:
Table Chain num pkts bytes target prot opt in out source destination
No matches found for 203.2.xxx.xxx in ip6tables
csf.deny: 203.2.xxx.xxx # lfd: (cpanel) Failed cPanel login from 203.2.xxx.xxx (CN/China/-): 5 in the last 3600 secs
Did you ever resolve this? I just recently switched on ipset with geoblocking using allow filter for a handful of countries. I increased the LF_IPSET_MAXELEM to 80000 to accomodate the US geo-ip list. I also enabled use conntrack at the same time but made no changes to any block settings. Prior to making these changes china ips were blocked properly. Since enabling ipset logs are showing lfd is banning china ips for failed login attempts. Am I missing a setting or something?
CSF config mentions:
To use this option you must have a fully functioning installation of ipset
installed either via rpm or source from http://ipset.netfilter.org/
I believe I meet this requirement. How would I confirm? Centos 6.10, ipset 6.11, iptables 1.4.7, latest kernel 2.6.32-754.12.1.el6.x86_64 which I thought supports conntrack but I don't believe iptables included CT helper support until 1.4.8 which is only for kernel 3+. Help is appreciated.
Edit: just wanted to add, the china ip that got through for me was also in 203.2.xxx.xxx subnet. csf -g 203.2.xxx.xxx shows the ip is not present in the block list, but that's expected since I'm using cc_allow_filter and not the deny list. Also, messages.log shows other countries being blocked successfully. So not sure why this ip slipped through. Could it be a server memory issue? I'm using almost all of the 2GB I have allocated for my VPS. If I hit the ceiling can I expect connections to start slipping through?
Last edited by kdub on 22 Apr 2019, 19:15, edited 1 time in total.
No this is still ongoing. They are pretty infrequent, though we recently had around 6 in a few days. All except one have been from the 203.2.xxx.xxx subnet, the other was from 140.143.xxx.xxx (also China). All are attempts to log in to a cpanel account that no longer exists on the server. The strange thing is that once they get picked up by lfd and a rule specific to that IP is added, further login attempts get stopped. There are other blocked entries in messages.log from China.
We have LF_IPSET_MAXELEM at the default of 65536, but I don't think that is an issue. There are 7261 entries in the current cc_cn ipset.
I have done a fair amount of reading, but I am out of my depth and have drawn a blank.
Thanks phih for the quick reply. I also noticed after lfd adds the ip to the blocklist it is successfully blocked. In fact, the logs show the same 203.2.x.x ip hammered away with multiple connection attempts within the same second for about 45 minutes before it finally stopped. So I'm thinking maybe the limited resources of my VPS (2 core, 2GB) aren't enough to keep up with ipset enabled? For instance, if I hit the 2GB limit I'm assuming it drops to swap which would probably be too slow to catch all the connection attempts. Although I would think ipset would be faster than not using it as before. Who knows. I'm going to temporarily enable logging of all established connections in iptables and look at the log to see if other connections are getting through the CC block.
Last edited by kdub on 22 Apr 2019, 20:14, edited 1 time in total.
The attempts we've seen were generally at the rate of about 1 every 1-2 seconds. We have an 8 core / 8GB VPS. Some of the attempts were at times when the server was not particularly busy, but maybe you are correct that it is a resource thing. Let us know if you find out any more.
I haven't been able to investigate too much. But looking at apache error logs I see Chinese IP addresses hammering away looking for exploits and files not currently on the server. LFD has also recently blocked Bulgarian and Vietnamese IP addresses, neither of which are in my the cc_allow_filter. Interestingly, messages.log continues to show there are IP addresses from China, Bulgaria, and Vietnam being blocked. But for some reason others pass right through.
kdub wrote: ↑28 Apr 2019, 20:38
I haven't been able to investigate too much. But looking at apache error logs I see Chinese IP addresses hammering away looking for exploits and files not currently on the server. LFD has also recently blocked Bulgarian and Vietnamese IP addresses, neither of which are in my the cc_allow_filter. Interestingly, messages.log continues to show there are IP addresses from China, Bulgaria, and Vietnam being blocked. But for some reason others pass right through.
Did you ever resolve this? I simply recently switched on ipset with geoblocking the use of permit clear out for a handful of countries. I expanded the LF_IPSET_MAXELEM to 80000 to accomodate the us geo-ip list. I also enabled use conntrack on the identical time but made no modifications to any block settings. previous to making those adjustments china ips had been blocked properly. given that permitting ipset logs are displaying lfd is banning china ips for failed login tries. Am I lacking a putting or something?
@mopa5000 Unfortunately, not. I continue to see IP addresses from outside the cc_allow_filter countries in the apache logs. I also continue to receive LFD notifications for failed login attempts from outside IPs. I tried disabling faststart in csf thinking maybe flushing everything would help, it had no effect. I also reconfigured the system to free up more ram, it had no effect either.
It looks like csf is still adhering to the port rules and blocking access to ports not listed in the TCP_IN and UDP_IN. I think that's why IP addresses from outside the cc_allow_filter list are still appearing in the log as being blocked, but only for the disallowed ports.
I have no idea why this doesn't work. IPs I manually add or added by lfd to the deny_list get inserted into ipset and work fine. I know the order of the iptable rules matters so maybe the order that csf is inserting them is causing the cc_allow_filter to be ignored.
I have mod_security installed, so it catches IPs looking for exploits fairly well. I just recompiled apache 2.4 with mod_evasive which allows blacklisting of IPs that make excessive requests like in a DDoS. I'm trying it out to see how well it catches the IPs that are filling up my apache logs looking for files that don't exist. So far it seems to be working. My only concern is that it will start blocking legitimate high request IPs. I'm still playing with the configuration.
Anyway, I've pretty much given up relying on ipset with cc_allow_filter. I'll be migrating to Centos7 in about 6 months, I'm hoping that resolves the issue. I sure would like to know why it doesn't work though.
Hey guys, I'm having the same issues - simply trying to block all access to the server except for US,CA, so in CC_ALLOW I have it set to: US,CA (+using LF_IPSET) - but I still get attacked from outside those countries. What is the point of using CC_ALLOW if it doesn't work?
chadreitsma wrote: ↑05 Jan 2020, 21:34
Hey guys, I'm having the same issues - simply trying to block all access to the server except for US,CA, so in CC_ALLOW I have it set to: US,CA (+using LF_IPSET) - but I still get attacked from outside those countries. What is the point of using CC_ALLOW if it doesn't work?
I still haven't solved this on my CentOS 6 box. It's not a memory limitation issue as there is at least 1GB free even after populating the hash tables. As far as I can tell it's something to do with a kernel incompatibility even though the documentation says it should support the minimum version of IPSET and iptables required. My CentOS 7 & 8 boxes work fine with the same configuration. Since you have only a couple allow country codes (vs a handful of deny), I imagine the impact of filtering without IPSET should be minimal. Since my server also runs an authoritative nameserver, I allowed global traffic to port 53 only and allowed only a few countries to everything else.
I wish I had an answer for you. With CentOS 6 at EOL I suspect no one really cares at this point. As an aside, I find it super annoying that CentOS doesn't provide an upgrade path for newer versions. I've already started a very slow migration to another box and it's definitely a hassle to set everything up from scratch again.