Posts: 169
Threads: 0
Joined: Sep 2016
Reputation:
7
How do I solve this weird server problem?
07-21-2018, 10:36 AM
Top -n 1 -b > some-filename.out
-n is number of iterations top runs before it exits
-b is batch
Not sure of a way to get it capture snapshot over 5 minutes outside of just increasing the -n value. You can always "man top" to get the docs.
Any new/updated plugins?
Never cross streams.
Posts: 504
Threads: 0
Joined: Jun 2016
Reputation:
5
How do I solve this weird server problem?
07-21-2018, 11:14 AM
My first guess is that somebody found an exploit and is running some sort of malware on your server. Given the load, I don't think it is a miner, but it could be one that is careful to not saturate the server. I suggest looking at network traffic, as in what kinds of outgoing connections are being made and to where.
Posts: 169
Threads: 0
Joined: Sep 2016
Reputation:
7
How do I solve this weird server problem?
07-21-2018, 10:12 PM
Running those top commands during the high latency times? They memory and CPU load looks light.
Also, it looks like the forum, rok, and reaxxion are all running on the one server? The forum doesnt seem to be experiencing any problems. Are you seeing the same issues on your WP panel for reaxxion?
Never cross streams.
Posts: 2
Threads: 0
Joined: Dec 2017
How do I solve this weird server problem?
07-22-2018, 01:40 AM
Quote: (07-21-2018 09:24 AM)Roosh Wrote:
I used to know a top command where you can take a snapshot over a period of time (e.g. 5 minutes), but I forgot how to do that.
Any ideas?
'sar' maybe?
Posts: 504
Threads: 0
Joined: Jun 2016
Reputation:
5
How do I solve this weird server problem?
07-22-2018, 10:08 AM
This thread reminded me that I had not set up logwatch on a couple of my servers. That has been remedied. It's a somewhat useful tool.
Posts: 2,309
Threads: 0
Joined: Nov 2014
Reputation:
48
How do I solve this weird server problem?
07-22-2018, 10:11 AM
Quote: (07-22-2018 08:36 AM)Roosh Wrote:
I ran netstat twice, once without Cloudflare protection and one with.
With the protection, there were 264 TCP connections. I turned protection off, and then the count jumped to 443 TCP connections. So when the protection is disabled, the connections almost double. And of course those connections come from Cloudflare IPs. Is this a big clue?
Okay, I'm not officially a sys admin, I just play one on TV ;-)
Looking at your netstat and top output I think you're looking at an external threat. And yes the fact that enabling Cloudfare protection lowers the number of open connections suggests that the server is being targeted. It's not a DOS attack per se - since it's php-cgi that's blowing up it reminds me of this:
http://seclists.org/fulldisclosure/2014/May/21
Some suggestions for mitigation:
https://www.nightlionsecurity.com/blog/n...nx-apache/
By the way, this is few years old though, have you properly patched your server?
Also, it sounds like you are dealing with this all by yourself. Do you know/have a good sys admin?
FWIW, if you don't know much about top, netstat, ps, etc. and Linux server administration in general then you are ill equipped to dealing with a live attack. Do yourself a favor and spring the money for a decent sys admin. You are obviously a target and will remain going forward. Be prepared.
*******************************************************************
"The sheep pretend the wolf will never come, but the sheepdog lives for that day."
– Lt. Col. Dave Grossman
Posts: 19,387
Threads: 0
Joined: Aug 2008
Reputation:
414
How do I solve this weird server problem?
07-23-2018, 08:36 PM
Problem persists.
Without protection, here's the number of active connections:
# netstat -nap | grep 80 | grep EST | wc -l
58
And then I turned the protection back on:
# netstat -nap | grep 80 | grep EST | wc -l
4
Posts: 13
Threads: 0
Joined: May 2017
Reputation:
0
How do I solve this weird server problem?
09-28-2018, 04:11 PM
I am a unix sysadmin and have a few suggestions:
Check to see if there are issues with swap space as root
free -h
If that is good, run the ps -auwx command and see if there is anything running that is unusual. Any unusual user or process is a red flag.If you find, delete the id then cleanup the server.
Change all your passwords.
Analyse your log files ( apache,mysql,php) . Look at the IP addresses and see if there are any patterns or unusual activity. if so,
If you can, patch these system when you can.
Posts: 600
Threads: 0
Joined: Apr 2015
Reputation:
27
How do I solve this weird server problem?
09-28-2018, 08:49 PM
Roosh here's your root cause. You're most likely getting DDOSed.
Advanced botnets can actually "masquerade" as real browsers. They achieve this via malware installed on unsuspecting victims' machines, by running Chrome/Firefox (or any supporting browser) in something called "headless" mode. Headless browsers work just like full fledged browsers using the same rendering engine, but instead of rendering the page on screen they do it in memory, in the background. Since it's essentially the same as the real thing and it works exactly in the same way (loading the page and all resources and running any javascript on the DOM the same way a real browser would) we use it for things like automating GUI testing. But it can also be used for mischief.
Cloudflare will do a pretty good job of mitigating threats like this but it won't be a 100% solution. The spiked metric "browser challenged" means exactly this; it means a browser navigated to your page and managed to get past the initial checks, but then Cloudflare ran some more advanced heuristics (essentially "challenging" the browser) to determine if the visit was legit but since there was no actual human being behind the headless browser, it failed this challenge and got cut off.
Memory usage being high could be easily related to the server receiving way too many requests. Then the server process crashing would be due to running out of memory. MySQL most likely isn't your root cause here -- increased request count means more MySQL loads means higher CPU usage.
Seeing one UDP IN block every 30 seconds isn't something you should worry about. There are hackers out there that probe whole IP ranges all the way to see if they can find any machines with known vulnerabilities. If you were getting targeted you'd see thousands per second.
One thing you should definitely do is to examine the output of "top -bcn1 -w512" here 'b' is batch mode, 'n1' is take 1 sample, 'c' shows the full command line for each process and 'w512' makes the output 512 chars wide so the whole command will wrap around and fit the screen. The full command line for each process is important since it will be easy to tell if there's any fucky process going on. You can post the output, or preferably, post in private or send me over PM (so you dont expose anything about your server filesystem structure publicly) and I'll take a look.
Another thing you should definitely do is to check access and error log files, these can be anywhere depending on your distro but try "/var/log/apache2/access.log" and "/var/log/apache2/error.log". If they're not there, maybe try under "/var/log/httpd/" or google your distro name and check where the logs are. The error log is especially important in your case. Check these to see if you can see any sketchy patterns.
Apart from that you can look at "last" command output to see last logins. "w" command to see who has a shell open at any given time. See if there are any logins that you don't recognize.
Same with syslog. It can be at different paths depending on your distro. Seems yours log at "/var/log/messages" which makes me think you're running something like RHEL/CentOS/Fedora.
I don't know what forum/blog software you run (apart from Wordpress) but in the unlikely event that you've been breached, the hacker probably put backdoors there to gain easy access next time around. I think WP has a file integrity checker, you should run it. For an all-around WP security solution I usually install a plugin called "Sucuri" for my setups.
Last but not least, keep the server up to date. Best of luck..
Edit: I saw that you're running php as the users 'rok' and 'rooshv' are you using these for terminal login by any chance? If yes, that's an ENORMOUSLY huge gaping security hole! (10x as big if they also have sudo access) You should definitely disable their shells and passwords, limit their access and create another limited account to use for SSH'ing into the host. If that's the case let me know and I can help you with that too.
“Our great danger is not that we aim too high and fail, but that we aim too low and succeed.” ― Rollo Tomassi
Posts: 600
Threads: 0
Joined: Apr 2015
Reputation:
27
How do I solve this weird server problem?
09-28-2018, 09:29 PM
Quote: (07-22-2018 08:36 AM)Roosh Wrote:
With the protection, there were 264 TCP connections. I turned protection off, and then the count jumped to 443 TCP connections. So when the protection is disabled, the connections almost double. And of course those connections come from Cloudflare IPs. Is this a big clue?
Yeah along with the netstat it's a big clue, when you increase protection the heuristic checks to tell apart the fake browsers get tighter, so you see a drop in the amount of connections that actually make it through.
“Our great danger is not that we aim too high and fail, but that we aim too low and succeed.” ― Rollo Tomassi
![[Image: attachment.jpg39554]](https://rooshvforum.network/attachments/attachment.jpg39554)
![[Image: attachment.jpg39555]](https://rooshvforum.network/attachments/attachment.jpg39555)
