Extreme Slowness?

[Naiah]

Page created in 27.946 seconds with 20 queries.

February 09, 2013, 04:58:12 PM



Page created in 43.096 seconds with 28 queries.
February 09, 2013, 08:02:59 PM


Page created in 57.514 seconds with 28 queries.
February 09, 2013, 11:56:09 PM

[LunarSage]

Page created in 9.165 seconds with 16 queries.
February 09, 2013, 05:46:34 PM

[Dovel]

Page created in 19.378 seconds with 16 queries.
February 09, 2013, 6:11 PM

[Vekseid]

So yeah.

There's nothing more I can do with the hardware we have. Can ask staff on Elliquiy and Blue Moon to break up threads over 10k posts, and for people in general to use threads instead of pms where willing, but beyond this, we need new hardware. The bottleneck isn't even RAM anymore. We're actually hitting thread limits.

[Naiah]

So this is why E just crashed? And why its REALLY slow right now?

Would deleting old game threads perhaps help? Or set up an archive for threads that havent been posted in for over a year?

[Lorelei Adelaide]

I am glad it wasn't just me that was experiencing this slowness!

[Vekseid]

I rebooted the server to clear up memory fragmentation, which seems to have helped somewhat, but it doesn't prevent actual lockups.

The server doesn't load things that aren't used. There's no benefit to archiving/deleting threads that isn't gained by simply splitting the large ones that people occasionally browse, and by asking people to stay out of them until they are split.

[Naiah]

We barely have any 10k+ threads anymore, does the site really crash if 1 or 2 look into the old locked ones?

[Vekseid]

The way it works is - the server is capable of actively processing 16 things at once. It may seem to do more, because what happens is threads 10,000-10,015 will get CPU time for a few milliseconds, then threads 10,016-10,031 will get time (although most threads on the server are 'asleep' at any given moment, so numbers won't be sequential, and some things have higher priority than others).

Private messages aside (far worse on Blue Moon than on Elliquiy, but both systems are pretty bad), the schema for handling threads is actually fairly decent, and I've done a lot of work making sure that they are fairly well optimized on both Blue Moon and Elliquiy proper. All I had to do was fix some indexes for messages.

So what happens is, when you make a post, it stores that post alongside a few indexes - basically directory lookups, like a phone book.
- The topic/thread ID you see in the URL bar is the most obvious one
- The post time is another important one (stored in seconds since 1970).
- Your user id is yet another important one. Had to spend some time specifically fixing 'view this user's posts' because yes it gets hairy.
- The post ID itself.

Looking at an index will return the data set associated with the value specified.
- Obviously, looking at a single second in time isn't going to return a lot of posts. Rarely more than the number returned by looking at a specific post ID.
- This leaves the data sets returned by users (people looking through post histories) and large threads.

The smaller this index gets pulled, sorted, and its data sent, the faster it is no longer taking up an thread. The sooner it isn't taking up a thread, the better off we are.

Right now, thread thrashing is being caused by the following events, in order of seriousness
1) Blue Moon's private messaging system. By far the worst offender.
2) Elliquiy's large threads. Occasionally bad enough that the site locks up without Blue Moon's help.
3) Elliquiy's PM system.
4) Blue Moon's large threads
5) Post histories of very active users.

Once more than sixteen of these get queued at once, if their results are not fully in the memory pool, the server goes kaput.

I've known this was occasionally an issue for awhile, actually, and altogether it threw a fairly nasty wrench into my new CMS plans because I didn't properly account for the size and scope of some people's conversations in my database schema. How do you cleanly solve this without adding more hardware? It's not an easy problem.

In any case it has reached a point where it's simply become untenable for us to remain with our current configuration. We need a new home.

[Koren]

I dont know if it works this way so sorry if this is a stupid question but would cleaning out inboxs help with that somewhat at all?

[Blitzy]

Would it be of any help to stop some of the forum games for a little while? And for us to stop posting pictures, etc, in the Finders and Seekers threads?

[Naiah]

Maybe lock threads at 5000, instead of letting them run all the way to 10k?

[Lord Drake]

As far as I understand from what Veks said, one of the big issues is people going through other people (or maybe their own) post histories. We are starting to have carnitas around here and I understand that a single person's post history is a HELLUVA thread.

I don't know if there is a way to solve this. Maybe having post histories show only the last month or two of activity?

[Naiah]

From what I understand, E needs a new home, so what would that cost and how do we go about financing it?

[Cecilia]

From what I understand, E needs a new home, so what would that cost and how do we go about financing it?

A new server is the best solution.  One way to help with financing E in general is, for anyone with the ability to, to subscribe and ask their friends to do the same.  We've had some fundraising efforts in the past--like the recent volunteer auction, and there are some other things people are trying to figure out as well.  We're willing to consider all sorts of fundraising activities, so if you have an idea pm a staff member or take a look at this thread and add your ideas.

[Vekseid]

Maybe lock threads at 5000, instead of letting them run all the way to 10k?

Every thread the server is thrashing on has been one of the ~30k ones.

Going from 30k to 10k gives a performance increase of roughly an order of magnitude (in terms of how long it takes the server to finish that part of that request), so I think that's good enough for now.

I dont know if it works this way so sorry if this is a stupid question but would cleaning out inboxs help with that somewhat at all?

In order for a message to be deleted, it must be deleted by both the sender and all of its recipients.

I'm really loathe to ask people to delete stuff in general, or to cease communication. Just is yucky.

[Izu]

And how much would a new server cost us? I mean... in the last auction there were still so many volunteers that we had people for about 4-5 auctions to be scheduled in the future. Maybe we can have those auctions sooner?

[Naiah]

Knowing what a new server would cost would help in figuring out what we need to do to get those funds.


This might sound wrong, but pleeease dont take it the wrong way..

But it sort of irks me that Bluemoon is sharing the server with us. I mean people here on E are donating money to help maintain this server we have now and it seems a bit unfair that a whole other site can get a free ride off of what we donate here. Isnt it possible to split the two? Set Bluemoon up on something else?

[Saerrael]

Does post history load in one go? If it does, is it possible to cut the loading into parts?
Just a thought that wandered my mind.

[Vekseid]

Knowing what a new server would cost would help in figuring out what we need to do to get those funds.


This might sound wrong, but pleeease dont take it the wrong way..

But it sort of irks me that Bluemoon is sharing the server with us. I mean people here on E are donating money to help maintain this server we have now and it seems a bit unfair that a whole other site can get a free ride off of what we donate here. Isnt it possible to split the two? Set Bluemoon up on something else?

Could easily drop $10k on a 1u server. More realistically though, I'm looking at $6-8k. I did have some money saved up but the hospital trip ate into that, and I still don't have the bill for the Ambulance or ER admission yet. The hospital stay was fortunately less than I feared.

Getting a separate server pair for Blue Moon might be possible... but it's also another $300/month to colocate, on top of the cost of buying a second server with only slightly lower specs. With a 16-core machine, splitting the databases between two SSDs would be prudent anyhow. That way Blue Moon and other sites shouldn't be able to impact Elliquiy much.

In any case, I'd certainly bring Blue Moon in on fundraising for a new server. They are a quarter of the server's load, after all, and some members do make donations (though they don't have any subscribers).

Does post history load in one go? If it does, is it possible to cut the loading into parts?
Just a thought that wandered my mind.

Sort of, and not as-is, really. For the new software I've been thinking of what amounts to an auto-'archival' system, which would be fairly transparent to the user, and it would work a bit like that, only loading up to four thousand or so paragraphs at a time.

[Naiah]

With the donations we make here on E, dont we have enough to have our own server then? Let each site fund their own?

[Nessy]

I thought Vek paid for them both which means it would be the same pot of money even if you allocated the funds separately.

[Naiah]

I meant more so we dont share a server, seeing as BM can actually make this site crash and be uber slow.

[Kye]

I believe that at least part of the problem with having different servers is the cost.  My understanding is that the monthly costs are covered by donations and subscriptions across both sites, and there isn't all that much left over if anything at the end of the month. Two separate servers would cost more per month, not just up front. Veks can of course correct me if I'm wrong, this is just my educated guess.

[Naiah]

My understanding is that there are more than just E and BM and that E gathers most of the donations? I do think that if we gather the most, then we do deserve not to suffer thanks to too many PMs on other sites going out (as an example)