Randomly slow/quick to load

[Pete]

First off, this isn't a post about me bashing UFOPaedia.org, so bear with me until the end of this topic

 

It turns out that after monitoring the new server for a while that it's actually UFOPaedia.org that was causing the old server to crash randomly and this new server to stall occasionally.

 

Looking at the processes on the server that are running when things start to go slow, it's the wiki software at UFOPaedia.org that randomly chews through half the available RAM and processor at times, which is quite something as I increased the RAM to 1GB (more than enough for a web server usually).

 

I can't blame the site for being popular, but it has come to light just how inefficient the wiki software itself is, something that is documented across the internet with peope trying different things to make it run more smoothly. Changing software isn't an option as that would ruin things - no two wiki software packages are alike and it would require everyone who's contributed there to learn slightly diffrent markup, plus there would be no guarantee that different software would run any better.

 

I've done a couple of things over the past few weeks in an attempt to make it run more smoothly. Firstly, I turned on a caching method that serves up static pages to non-logged-in users and bots so it doesn't query the database (runs really quick when you're not logged in ). Secondly, I told Google not to send 50 Google bots to crawl the site every second (I think this is what has been causing recent slowdowns this week).

 

Unfortunately, as the site gets more popular over time, the only thing that will make it run more smoothly is more CPU and RAM which means throwing more money at it.

 

I think it's running alright at the moment, but just so you know there are hurdles to overcome in the not too distant future.

 

Is it worth putting a PayPal donation button on the wiki in the sidebar at the bottom, as well as maybe a page where donators get their name in lights as thanks for their contributions?

[Matri]

Well, unless you go with a locked Wiki like the TVTropes site (where only approved, registered users are allowed to change anything) you'll have to contend with the increased load due to every Tom, Dick & Harry wanting to leave their mark and vandalise and show everyone how big their e-dick is (Spoiler, it doesn't and will never exist, idiots).

[NKF]

Except for the spammers, bots and occasional bursts of updates, I didn't think there was a heck of a lot of activity. Is there a lot of processing done each time any visitor just views a page?

 

edit:

 

Looked at the wiki stats and crikey - 10,233,708 visits. Doesn't say from when, but that's a lot of page views. Wonder how many are real people though?

 

- NKF

[Pete]

I think the problem is that they've coded the main index.php file so it does a lot of work. This is the case on most sites, including SC's CMS and forums, but from what I understand even with caching turned on on the wiki it does loads a fair amount of stuff it doesn't need to on every page load.

 

It is far quicker when you're not logged in now with file caching turned on, but that just highlights the problem more clearly - the only thing that's visibly different on a page when you're logged in is your name at the top of the screen. That should really only be one database query.

 

The other issue with the file cache follows on from being logged in - if you're logged in it ignores the cache so every page you view is pulled from the database. This is supposedly because the cached version might not always be bang up to date on an insanely busy wiki, but that just sounds like a bad caching system.

 

Anyhow, it's probably all more technical than that, but I don't see why when you edit a page that it can't update the cached version, however I must admit SC's CMS doesn't do this either, preferring to delete the whole cache just to be sure - at least the wiki doesn't go to those extremes.

 

From what I understand of the file cache, the files are updated every so often so multiple minor edits to the same pages don't kill the server, so having that switched on is good for search engine bots and the likes as it saves querying the database.

 

The Google Analytics stats we have for the site should show accurate, non-bot figures in terms of visitors, however there are many search engines out there so getting dozens of search engine spiders hit the site at once probably doesn't help.

 

I did find some documentation on making it run much more smoothly but I need to find a spare few hours to test it out with breaking anything, so for now I'm hoping the tweaks I've made will keep things running smoothly for the next week or so until I get time to look at it again.

[Bomb Bloke]

I don't know if this helps you any, but I suspect some of the weird caching design has to do with "template" pages. These articles are included within others, so if you edit one template, all pages which make use of that template change as well.

 

To my understanding, the wiki does go through and update the cache for the edited page and all affected pages straight after you make the edit, but this can "potentially" take some time (depending on how many pages need to be updated by that edit). This would be a decent reason not to rely on the cache to provide current versions of certain pages, especially on larger wikis.

 

You can view the length of the "job queue" which handles that here. As a test, I tried editing this template (used with these pages) - it took roughly a minute or two to process, and the time it took to refresh the stats page went up quite a bit while it was underway.

 

(This was actually a heck of a lot slower then I expected. I was assuming the entire queue would've been processed before I could even reload the stats page - or within half a minute, tops).

 

I don't know whether the "what links here" info is cached or not. There's no real reason for it to be, but if it is, that'd crank up the load a fair bit too - for any page edit.

[NKF]

Templates are such a handy device, but if they're causing such a big strain on the server, then perhaps we'll need to either find an alternative to them or find some way to reduce the number of them that are used. Some of the lesser ones can actually be done away with.

 

- NKF

[Bomb Bloke]

Frankly I'd prefer to have a slow wiki then one without templates - the only real alternatives would be to manually go through each page that would otherwise use a template and manually tweak them (which would take longer for users and up the server load further, albeit spread out over time), or just do away with the navigation benefits altogether (forget that!).

 

It's not like we edit them that often anyways, so I doubt they are the sole cause of the speed issues - as I said, I'd expected the job queue to be processed much faster then it was. Though we could switch them to protected status (meaning only sysops or above would be able to alter them).