[Roosh]

Here's the ROK server specs:

Dual 5520 Quad-Core Xeon 2.26GHz
12GB RAM

Here's the current apache settings:



Obviously we don't have enough juice for the traffic we're seeing, but I'm wondering if there is something else I can try. I did change max clients from 250 to what it is now, which allows more users but still leaves server at a stand still for most people.

[ryanf]

Is apache crashing, or is it possibly the database from the sheer volume of queries?

[ryanf]

Actually, I'm seeing super high times on loading static assets (jquery, css, etc). So it's likely apache. Are apache and mysql on the same machine?

[Roosh]

(11-19-2013 07:46 PM)ryanf Wrote:  

Actually, I'm seeing super high times on loading static assets (jquery, css, etc). So it's likely apache. Are apache and mysql on the same machine?

Yes same machine. When I open the site back up, all 600 apache clients get filled right away. Here's the current apache stats:

Total accesses: 453846 - Total Traffic: 6.7 GB
CPU Usage: u1193.73 s190.12 cu3321.35 cs0 - 247% CPU load
238 requests/sec - 3.6 MB/second - 15.4 kB/request
599 requests currently being processed, 0 idle workers

[thatGuy]

A CDN, or even something more like CloudFlare, will give you a bigger bang for the effort put in than tuning Apache. Also, high-performance websites usually go with nginx (and memcached, and varnish), even if they just put it in front of Apache.

[ryanf]

Yeah, so it's the sheer volume of requests. For every html page load (that is a single apache request), it's making an additional 30-ish http requests (each image is 1 request, each css file is 1 request, each js file is one request, etc). So each visitor is making 30 requests. If you have 20 visitors per second, that puts you at the 600 client limit. And when it reaches the max number of clients, it has to queue them up while the others get served. And the queue will grow until it starts dropping connections, hence why we are getting denials.

So, there are a few options (in order of how good they are to implement in terms of $ and time)

1. Use a CDN for static assets. This is a *great* option, as CDNs are more tuned for static file access (and are inherently more scalable).
2. Add additional server(s) - have them load balanced. This will distribute the load across multiple machines, and remove the "single point of failure"
3. Implement more caching - I believe WP has caching plugins, although I'm not familiar with how they work
4. Increase the server size - probably expensive and more downtime. Not a great idea (at least not now).

[ryanf]

(11-19-2013 07:57 PM)thatGuy Wrote:  

A CDN, or even something more like CloudFlare, will give you a bigger bang for the effort put in than tuning Apache. Also, high-performance websites usually go with nginx, even if they just put it in front of Apache.

Nginx is a great server, it deals with heavy volume better, although it can be finicky as well, and somewhat difficult to debug.

[Jersey]

Yeah, trying to maneuver in the face of this traffic is going to be rough no matter what you do. I'd agree with most of the commenters above - move the static assets off to a different server and/or enable a CDN.

You can set up an s3 bucket through Amazon pretty quickly. (You can copy them over via http://www.crossftp.com/amazon-s3-client.htm) Then, rename all the files in your template to use that root URL. That will at least offload static template traffic and probably take 30-60 mins.

Are you using local versions of jQuery or other JS libraries? There are free CDN versions of them from Google. Still, that's just a drop in the bucket.

The key to this is triage. Do what you can that's quick, but more importantly - as simple as possible. Truth be told - the simplest thing to do might be to move your images directory to a new location. 404s will return a lot faster than anything else. It won't reduce the load, but may buy you a little bit at the expense of an uglier site.

[Jersey]

Whoa - just saw that you were trying to serve 7,000 comments off that post? That's going to be a huge DB bottleneck (that may present itself as a PHP/Apache problem, when in reality, they are waiting on the DB). Sites much bigger than this (with 25-50 servers) can't handle that load without timing out.

Turn off comments on that post and start a new one for follow ups. Also, caching or a CDN works wonders when all of your traffic is to one URL, so it may be hard to start up, but it's perfect for this situation.

Also, it's too late now, but server monitoring services like NewRelic do a great job of helping you figure out exactly what is wrong at times like this, for future reference.

[ryanf]

ROK is disqus commenting, correct?

[Enigma]

(11-19-2013 08:43 PM)ryanf Wrote:  

ROK is disqus commenting, correct?

Yep.

[ryanf]

(11-19-2013 08:59 PM)Enigma Wrote:  

(11-19-2013 08:43 PM)ryanf Wrote:  

ROK is disqus commenting, correct?

Yep.

From what I know, Disqus comments are stored on the disqus server, so that shouldn't be what's causing the issue. It's the volume of requests.

[Anteros]

Roosh, see here:

https://httpd.apache.org/docs/2.2/mod/mp...erverlimit

It seems that this should be set equal or greater than the MaxClients directive:
FTA:"With the prefork MPM, use this directive only if you need to set MaxClients higher than 256 (default). Do not set the value of this directive any higher than what you might want to set MaxClients to."

Also, from: https://httpd.apache.org/docs/2.2/mod/mp...maxclients
"The MaxClients directive sets the limit on the number of simultaneous requests that will be served."

it seems that you don't need to simultaneously serve 600 at once slowly but maybe 200 at once a bit more quickly.

NOTE: I an not an Apache expert but just trying to assist.

[ryanf]

Of course, if you just want to get it back usable, we could deny requests from people have a specific referrer (for example, coming from jezebel[dot]com). Is that something you want to try?

[Ensam]

Check out http://www.stochasticgeometry.ie/2010/03...lashdotte/ for some tips on how to rapidly retune Apache to handle the load spike.

[Tuthmosis]

The collective power and knowledge in this forum is, when you stop and think about it, astounding.

[skulled]

As a couple of others have pointed out, Apache is quite inefficient and slow compared to something like Nginx, and hogs up more resources on the server.

Something that you could try to alleviate problems in the short term would be the following:

Install Nginx on the server, and put it in front of Apache as a reverse proxy server. What this allows you to do is let Nginx handle every incoming request and only pass things to Apache if and when required.

This will enable you to serve off all static assets and resources directly from Nginx without the request having to touch Apache. And Nginx is really fast and efficient at doing this.

Another more involved setup is to put Varnish in between Nginx and Apache in the above setup to add another level of caching. Varnish can cache a lot of stuff from blocks of content, to entire pages. So in such a setup, Apache will see only the absolute minimum of requests and therefore never end up in a bloated resource hogging state.

The other easier setup would be to if server administration is not your cup of tea, would be to sign up on something like CloudFlare which acts as CDN and can cache most of your pages and serve them directly without the request ever having to hit your server.

If you have any particular questions or doubts on how to some of the above steps, give me a shout and I can explain more.

[paninaro]

I think the best quick fix here is to install CloudFlare, as others have suggested. I'm sure some of the haters will get their boyfriends/hangers-on to implement a DOS (denial of service) attack soon enough, so CF will protect against that too.

Basically they cache and optimize the content, and pull it from your server to their server, then serve the request from their servers, which are placed at locations around the world. But while they do it, they also combat DOS attacks, and anyway those attacks end up being handled by their servers not yours. It's basically like putting a security guard in front of your server who controls who gets access to content.

As for Apache tuning, I work mostly with IIS tuning (which is a different model due to shared processes) but one item that stands out is that keep-alives are off. I believe that the benefits of keeping this on, on a page with so many items (30+ images, scripts, etc) is that the browser can use its existing idle connection to the server to pull the next item, instead of making a new connection. That's a big performance improvement because connection set-up time is slow.

The drawback is that you have lots of open connections, but most Unix systems can handle that well. The thing I'd worry about is the overhead to keep killing off the keep-alive connections as they idle out, but I'm not familiar enough with Apache to comment on how well it does this.

[Blaster]

Somehow missed this thread earlier, but I also agree with the responses so far. Very likely that apache is burning up CPU trying to (inefficiently) serve static assets. An nginx reverse-proxy is probably the cheapest option. (memcached / varnish for caching might help too but nginx's event-driven static content server is the big one)

I searched for some docs on setting up nginx as a reverse proxy for apache and came across this:

Using Nginx as a reverse caching proxy for Wordpress: results (Overview)
Using Nginx as a reverse caching proxy for Wordpress: the setup (Configuration details)
The nginx config file

(H/T http://bostinno.streetwise.co/channels/n...wordpress/)

[WesternCancer]

(11-19-2013 10:34 PM)Tuthmosis Wrote:  

The collective power and knowledge in this forum is, when you stop and think about it, astounding.

Definitely man, and with more and more members meeting each other we're getting a large real-life network in place as well.

[Yul Brynner]

A CDN and protecting against a DDOS attack would be my priorities. CloudFlare is free but Roosh if you wanna go for CloudFlare Pro I'd be happy to chip in $10 for the first month.
Should I just go put $10 in the ROK tip jar instead?

[VincentVinturi]

Good suggestions above.

Also, the WP Super Cache wordpress plugin integrates with Amazon S3 and CloudFront.

Here's a step-by-step tutorial for the layperson:

http://nycgadgetgirl.com/2012/04/cdn-wp-...press.html

[Roosh]

Thanks for the help so far guys. Here has been my triage solution:

-Use CDN for static files
-Turn on Keep Alive with a timeout of 2 seconds
-Reduce max clients to 350
-Reduce timeout to 10 seconds
-Ask people via our Twitter feed not to share our site anymore.

As of 4am EST, the site is doing fine with 1,000 simultaneous users.

Installing NGINX was planned, but I don't want to do any major upgrades until this viral episode passes.

[Caligula]

Nginx, good Russian software. Their site is cool, no bullshit - http://nginx.org/

[HD668B]

(11-20-2013 06:07 AM)Caligula Wrote:  

Nginx, good Russian software. Their site is cool, no bullshit - http://nginx.org/

No idea why you are still on Apache. Its extremely slow compared to nginx and it gets raped by LiteSpeed. The latter is paid, but the fastest right now on the market.