Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Wordpress VPS load issue due to attacks
New on LowEndTalk? Please Register and read our Community Rules.

Wordpress VPS load issue due to attacks

TinkuTinku Member
edited June 2016 in Help

I am facing this weird issue on my VPS where i get 100s of http calls at the same time which create load on my VPS because of the php and mysql on use so my VPS company take down my VPS regularly. I tried CSF, many wordpress plugins to stop brute force attacks, xml-rpc attacks and almost everything but no success i can't stop these attacks.

Just now i had 800 connections to my VPS and it went down again. And this got nothing to do with ddos because i switched to many vps providers with ddos protection but the issue remain same.

Any idea how can i fix it?

«1

Comments

  • blackblack Member

    Do you have caching on your wordpress website? What about limiting requests per minute? Nginx has http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

  • TinkuTinku Member

    @black said:
    Do you have caching on your wordpress website? What about limiting requests per minute? Nginx has http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

    no wordpress cache plugin active currently but i do use cloudflare and also memcached/zend opcache with nginx/php7/mariadb

    Is this limiting requests option builtin in nginx or i have to add it by module?

  • jarjar Provider

    Block xmlrpc requests entirely.

    Founder @ MXroute

  • TinkuTinku Member

    @jarland said:
    Block xmlrpc requests entirely.

    by giving 404 when accessing domain.com/xmlrpc.php?

  • jarjar Provider

    Tinku said: by giving 404 when accessing domain.com/xmlrpc.php?

    That's one way. Basically make the web server respond, not the PHP interpreter. By either blocking it or 404ing it, as long as your 404 page isn't generated by Wordpress (then it doesn't help much as PHP still draws the page).

    Thanked by 1netomx

    Founder @ MXroute

  • rokokrokok Member

    All uptodate plugin? Have you try debug and check PHP7 compatibility with your plugin ( hehe I just upgrade my wp site to php7 yesterday ).

    CEO of PT. Rokok Kopi Internet Tidur Tbk.

  • TinkuTinku Member

    @jarland said:

    Tinku said: by giving 404 when accessing domain.com/xmlrpc.php?

    That's one way. Basically make the web server respond, not the PHP interpreter. By either blocking it or 404ing it, as long as your 404 page isn't generated by Wordpress (then it doesn't help much as PHP still draws the page).

    ok cool tip i will try it

  • TinkuTinku Member

    @rokok said:
    All uptodate plugin? Have you try debug and check PHP7 compatibility with your plugin ( hehe I just upgrade my wp site to php7 yesterday ).

    nah i had this issue even with php 5.6, 5.5 and 5.4 so it's because of version and yes all plugins, theme and wordpress are up to date.

  • edanedan Member
    edited June 2016

    Simple rule via htaccess http://pastebin.com/BmtfuYaP (block xmlrpc.php)

  • Block xmlrpc requests from ALL, other than Automattic (makers of WP) IP blocks. That should keep Jetpack working. If you use their mobile app, you can try having another virtual host with unblocked access to xmlrpc.php...

    My comments are mine and mine alone, and do not reflect the opinion of my business and operations. Ex-wellknown shitposter.

  • what control panel do you use on your VPS?

  • Tinku said: no wordpress cache plugin active currently but i do use cloudflare and also memcached/zend opcache with nginx/php7/mariadb

    Cloudflare and memcached/zend opcache is likely no cache at all. Install and use supercache or w3totalcache.

    Thanked by 1jar
  • FalzoFalzo Member

    if its not xmlrpc.php watch out for some plugins making ajax calls like messaging/chat-plugins on the frontüage looking for updates on short intervals or something like that.

    awstats or even plain logs may help to find which page or php-script is the one to get requested that often...

    Thanked by 1jar

    UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month | Netcup VPS/rootDS - 5€ off: 36nc15279180197 (ref)

  • KodisKodis Member without signature

    Use cloudflare

  • Kodis said: Use cloudflare

    Read topic.

  • sinsin Member

    Generate/preload static pages with Supercache or W3TC or setup Fastcgi microcache so php-fpm isn't called as much.

    location = /xmlrpc.php { deny all; }

  • TheLinuxBugTheLinuxBug Member
    edited June 2016

    @Tinku

    The things I would check my self would be:

    1. Assuming you have a reasonable amount of ram make sure MySQL(MariaDB) is doing joins and selects in memory instead of on disk (use /dev/shm for tmpdir)

    2. Setup fastcgi caching in nginx for the site, this will reduce load overall on php.

    3. Review and make sure you are not using any plugins which cause a rediculous amount of database requests, if you do this will quickly kill your server under load. While some plugins look cool, a lot are poorly written and poorly optimized. I would usually login to mysql on command line and run 'show full processlist' while this issue is happening to see what type of load and queries are being run at the time where you see the slow down.

    WordPress stuff can often be very bulky and poorly optimized and if you don't know how to recognize this, it can often lead to overloading your server like your explaining. A lot of times you will need a little more skill than just dropping in a plugin and hoping it works as you like, it is always best to review and optimize things your self. If you continue to see these issues and can't find a solution, please feel to reach out and maybe I can take a look for you and at least point you in the right direction.

    my 2 cents.

    Cheers!

    Have an Allwinner H3 device? Check out H3Droid! | Lichee Pi Zero - The 6$ SBC | #SYSarm - Get It! | Armbian | Atomic Pi Mirror
    22+ Years IT Experience in Linux/Windows Hosting, Administration and Development Services

  • try tuning Apache (if you're using it) for high traffic and CSF + mod_security to filter unwanted traffic, of course DDoS mitigation helps too. this setup seem to work for my wordpress sites, hopefully it'll continue........

  • Hey, I have run into this - have ended up installing "brute force login protection" on each WordPress instance, which blocks it at the .htaccess level, rather than the WordPress level. It means that it doesn't have to load WordPress every time there is a bunch of failed logins to tell them they're denied - it relies on Apache instead, which cuts the server load considerably.

    Also, add the free Comodo ModSecurity rules to your server: https://forums.comodo.com/free-modsecurity-rules-comodo-web-application-firewall/comodo-as-a-modsecurity-vendor-in-cpanel-t110147.0.html

  • AbdussamadAbdussamad Member
    edited June 2016

    Most likely these requests are hitting your wp-login.php and xmlrpc.php pages. You can 403 xmlrpc. For wp-login there are a bunch of ways you can deal with this. Easiest way IMO would be to use basic http authentication for requests to wp-login.php. You can add any password you like in .htpasswd. Bots are not setup to deal with this so they won't even attempt to login.

    If you are using apache 2.4 or later there's even an authbasicfake option that makes this dead easy:

    <Location "/wp-login.php">
       AuthBasicFake wut lolcat
    </Location>
    

    http://httpd.apache.org/docs/2.4/mod/mod_auth_basic.html

  • rokokrokok Member

    You guys really not read respond comments lad, op use nginx

    CEO of PT. Rokok Kopi Internet Tidur Tbk.

  • If you don't mind, can you please share few hundreds of access log file lines here, so we can check it try to help you because without access logs files, its very difficult to make any comment.

    Signature removed

  • SplitIceSplitIce Member, Provider
    edited June 2016

    Bruteforces can take down sites hosted on small servers, expecially if you havent optimized your hosting (i.e opcode caching etc).

    Keep in mind nginx is not a silver but, its higher performance - but its unlikely to be OPs only problem (the difference between an optimised Apache and nginx isnt particularly large).

    I strongly recommend getting good Layer 7 DDoS protection, while its not usually what they are meant for it usually (in our case definitely) filters out bruteforce attacks and XMLRPC (outgoing) requests.

    Also at "100 calls at the same time" thats particularly fast, are you sure its just not a DDoS attack?

    X4B - DDoS Protection: Affordable Anycast DDoS mitigation with PoPs in Europe, Asia, North and South America.
    Latest Offer: Brazil Launch 2020 Offer
  • TinkuTinku Member
    edited June 2016

    I never thought i will get this much help and these many replies to help me fix my problem. Thank you guys i really appreciate the great help and suggestions i got here :)

    Now lets come to the topic i have a better night today after so many weeks for the first time my site didn't go down because of high load (although the load touched 5-6 for a few minutes a couple of times around same time but that is nothing i used to have 20-50+ load average for almost an hour regularly during that attack. I hope this is because of the the steps i have taken going by the many suggestions here and i hope it wasn't because that hacker kid decided to sleep tonight :)

    Here are things i did yesterday on my VPS and i am also mentioning other things i already enabled few days ago.

    Things i did yesterday.

    Blocked access to all wp*.php files and xmlrpc.php from external ips in nginx.
    Renamed wp-login and wp-admin so that only i can access it.
    Installed and setup W3TotalCache and i am using memcached cache for page/minify/sql/object cache.
    Added in Nginx conf to block sql injection / block http post attack.
    Added in Nginx cnf conn_limit_per_ip.

    Things i did before yesterday that are still active.

    CloudFlare on with medium security.
    Installed CSF
    Upgraded to PHP7 with FCGI and MariaDB 10
    Installed ZendOpcache

    @SplitIce i tried many vps services that claimed to offer layer 7 ddos protection but still no success. Now i am at OVH

    @praveenk I will post the results here.

    @codingconcepts i did this trick of blocking to access all php files and wp-admin lets see if this works.

    @TheLinuxBug Followed your suggestion and made the changes so now MariaDB is doing joins and selects in memory instead of disk @ /dev/shm

    @sin @khuongcomputer Started using W3TC and you were right memcache and cf were doing nothing now i see more difference in site loading thn i had before when i had cf and memcache / zend op

  • In my case, installing Disable XML-RPC plugin helped me stop the issue.

    My list of reliable providers :
    Ramnode : HostHatch : Serverica : CloudCone : InceptionHosting : AlphaVps : LittleCreekHosting : Lunanode : Few more under testing!

  • SplitIceSplitIce Member, Provider

    @Tinku The DDoS Protection included with most VPS services is just network protection from the likes of OVH, Voxility etc. Usually this does not feature Layer 7 protection, or if it does it needs to be specifically configured.

    X4B - DDoS Protection: Affordable Anycast DDoS mitigation with PoPs in Europe, Asia, North and South America.
    Latest Offer: Brazil Launch 2020 Offer
  • Tinku said: i tried many vps services that claimed to offer layer 7 ddos protection but still no success. Now i am at OVH

    There are rather less services which can offer full-featured layer 7 ddos protection. If you need real protection - can offer you a managed solution with guarantees / SLA.

    Freelance System Administrator, available for hire. Primary tasks i do concentrated on: PHP, MySQL, Postgres, Nginx, DDoS-protection, application security, high-performance solutions, high-availability / clustering.

  • TinkuTinku Member

    @Tinku so these php load type attacks also comes under ddos category? layer 7? One of the host claimed to provide layer 7 accused me that issue is with my wordpress installation not their layer 7 protection.

    @Profforg I really appreciate it but i think managed services will go out of my budget.

    @fastwebhost I actually renamed both wp-login and wp-admin so now only i have access to it.

  • Tinku said: @Profforg I really appreciate it but i think managed services will go out of my budget.

    Perhaps, depends on the budget. Feel free to pm me so i can tell you exact numbers. You can jump from one host to another without stop searching for a cheap solution, there are a lot providers offering DDoS protection or similar, but near none of them can protect from clever attacks or the opposite, too simple to filter them but still hurtful for website attacks, or provide zero false-positive result. My solution is stop-by.

    Freelance System Administrator, available for hire. Primary tasks i do concentrated on: PHP, MySQL, Postgres, Nginx, DDoS-protection, application security, high-performance solutions, high-availability / clustering.

  • Securing Wordpress site is like mission impossible sometimes.

    Thanked by 1tux
  • simple solution, don't use wordpress

    Thanked by 1tux
  • @pedagang said:
    simple solution, don't use wordpress

    Simple and effective solution :D

    Thanked by 2pedagang tux
  • TinkuTinku Member

    And i guess that attacking kick is back WTH not able to connect to my VPS to see whats going on. Reboot isn't helping and ping timeout no matter how quickly i try to connect to VPS after the reboot.

  • TinkuTinku Member

    So i tried to connect to my VPS through the VNS in panel and all i am seeing is this i don't know whats going on looks like flood?

    Screenshot

  • blackblack Member

    Looks like a SYN flood. Maybe it's a good idea to pick up a DDoS protected host and GRE tunnel your site traffic. Maybe try https://athenalayer.com/pricing.html free tier? I hear some people use them and they've been good but I haven't tried it myself. You should also ask your VPS host for an IP change once you put your website behind a reverse proxy / GRE tunnel.

  • TinkuTinku Member

    @black said:
    Looks like a SYN flood. Maybe it's a good idea to pick up a DDoS protected host and GRE tunnel your site traffic. Maybe try https://athenalayer.com/pricing.html free tier? I hear some people use them and they've been good but I haven't tried it myself. You should also ask your VPS host for an IP change once you put your website behind a reverse proxy / GRE tunnel.

    OVH don't stop syn flood? Is it possible to add extra filter for this through CSF? I will look at athenalayer thanks for providing the link.

  • blackblack Member

    OVH should but if it's still able to take down your services then you need to seek other solutions.

  • blackblack Member

    Wait... those source IPs are directly from cloudflare. You shouldn't block those SYN packets.

  • TinkuTinku Member

    @black said:
    Wait... those source IPs are directly from cloudflare. You shouldn't block those SYN packets.

    Yea i am using cloudflare+ovh+csf+cache but attackers are still having a regular fun day :(

  • blackblack Member

    If you block SYN packets from CF then legit users will not be able to use your service.

    It's better to see what real IP addresses are attacking your website and use CF's firewall to drop those IPs. You might have to contact CF directly or use CF_CONNECTING_IP in the HTTP headers to get that information.

  • FlamesRunnerFlamesRunner Member
    edited June 2016

    @black

    AthenaLayer?

    Ehhh, it's run by Nick Lim and he wasn't exactly the greatest person when it came to flashing his signature wherever he could.

    Thanked by 1inthecloudblog

    wget https://s.flamz.pw/dl/bench.sh && bash bench.sh

    curl https://s.flamz.pw/analytics/bench/stats.php

  • blackblack Member

    FlamesRunner said: AthenaLayer?

    Ehhh, it's run by Nick Lim and he's not exactly the greatest person when it came to flashing his signature wherever he could.

    Ah ok. I never used his services so I'm not sure.

  • You need to block these requests at the firewall level. Have a look at your logs or just google the IP ranges that you should be blocking. If you are on CentOS, install csf. If you aren't on CentOS, you'll just need some iptables scripts in place to start blocking. It's less resource intensive to block at the iptables level. Also, when the script kiddies start seeing 5xx errors from their scripts, they'll stop hitting you. Good luck!

  • TinkuTinku Member
    edited June 2016

    ok guys i finally found one of his trick and fixed it!

    I installed a real time access log viewer to monitor traffic every second instead of opening big access log file every time and there i found these user agents attacking my site from hundred and thousands of different ip addresses.

    All those usera gents had one thing in common they were some kind of fake wordpress pingbacks from different ips so what i did is that i created a simple condition in my nginx conf file to detect these useragents and return them 403. My VPS load was touching 20-50 during the attack and as soon as i applied the condition and restarted nginx load came down to normal and now those bots are still attacking but getting 403 in return and no more load on the VPS.

    Now waiting for his next move!

  • TinkuTinku Member
    edited June 2016

    Using a php script to read access log for this attacks i came to know that in last 30 minutes i received around 10,000 ping requests to my site.

    What i don't understand is that i already blocked access to my xmlrpc file so how come the load was still creating when the page wasn't even accessible? The load only came down once i returned 403 for those requests.

  • matteobmatteob Member, Host Rep

    @Tinku said:

    is a little synflood, install synproxy and you will be safe http://www.seflow.net/2/index.php/en/blog/synproxy-module-protect-yourself-by-syn-flood

    Matteo Berlonghi - SeFlow s.n.c.
    SeFlow.Net - Affordable DDoS Protected Services. SeFlow Secure Network 20+ IXP connected with Default DDoS Protection.

  • @Tinku said:
    ok guys i finally found one of his trick and fixed it!

    I installed a real time access log viewer to monitor traffic every second instead of opening big access log file every time and there i found these user agents attacking my site from hundred and thousands of different ip addresses.

    All those usera gents had one thing in common they were some kind of fake wordpress pingbacks from different ips so what i did is that i created a simple condition in my nginx conf file to detect these useragents and return them 403. My VPS load was touching 20-50 during the attack and as soon as i applied the condition and restarted nginx load came down to normal and now those bots are still attacking but getting 403 in return and no more load on the VPS.

    Now waiting for his next move!

    Which log viewer ?

  • TinkuTinku Member

    @Junkless said:
    Which log viewer ?

    Pimp My Log
    pimpmylog.com

    It's really cool with features to read nginx access log, error log and php error log with features like every x seconds refresh.

    Thanked by 2Junkless sin
  • TinkuTinku Member

    @matteob said:

    @Tinku said:

    is a little synflood, install synproxy and you will be safe http://www.seflow.net/2/index.php/en/blog/synproxy-module-protect-yourself-by-syn-flood

    Is it going to work fine with Cloudflare? because cloudflare hides real ip of users.

  • matteobmatteob Member, Host Rep

    @Tinku said:

    cloudflare should protect you if you have business plan. If you have free plans cloudflare is useless on tcp attacks, so you need to switch to a ddos protected provider or upgrade cf to business plan.

    Matteo Berlonghi - SeFlow s.n.c.
    SeFlow.Net - Affordable DDoS Protected Services. SeFlow Secure Network 20+ IXP connected with Default DDoS Protection.

Sign In or Register to comment.