Fixing Things

My hallway

My hallway

You might be wondering what happened to this website last week. It, along with the rest of my life, was broken. The web site, at least, now seems to be fixed.

As I mentioned previously, I’m now living in a hotel while reconstruction is happening at home. Tuesday morning of last week, I packed some clothes, the laptop, and my collection of ancient erotic poetry into the car. I then let the workers in to destroy the place while I went to work. I didn’t get to the hotel until about 9 PM that evening. I tried out the internet and discovered that the connection was idiosyncratic. When I went to BoF, I got a server error. That can’t be right! It must be a crazy quirk of the hotel’s internet connection. I didn’t have any more time to futz with it, so I went to bed.

By Wednesday night, it was still down, and I strongly suspected it wasn’t merely a problem with the hotel’s internet connection. Unfortunately, in my rush to pack, I hadn’t had time to properly set up my password database on the laptop. I had the database; I just couldn’t get into it. That meant I also couldn’t get into the Dreamhost back end to see what was going on. Nor could I go home to access my desktop machine. It was a bit inaccessible (see above photograph).

By Thursday night, I had managed to get into my password database. That enabled me to get into the Dreamhost “web panel”. Everything appeared normal there, so I went to the Support area to file an issue.

In there, I discovered a notice from Dreamhost! It’s time-stamped Tuesday morning, less than an hour after I moved out of my condo. The notice said:

Greetings to you, the lucky finder of this golden ticket…

Whoops, wrong notice. It said:

Your site bay-of-fundie.com has been causing quite a bit of server load, enough that other people on your shared server are complaining about it. Sadly I had to disable your script to bring the load back to normal.

I looked at the log files and discovered that about half of my traffic was from Google and Yahoo image searches. Historically, that has been about a third of my traffic, so that’s up a little bit. Of those searches, about two thirds were people looking for Duggar family images. Historically, it has been about half. This means that the day the news broke that Jim Bob was pulling another rabbit out of the cat, everybody hopped onto the internet looking for pictures. Just because I wrote several snarky articles a few years ago, I’m somehow considered the go-to guy.

OK, people! You want Duggar pictures? I’ll give you a Duggar picture:

Michelle Duggar's vagina

Michelle Duggar’s vagina (actual size)

What’s peculiar, though, is that overall traffic wasn’t up by very much. More of it was Duggar, and most of that was images, but apparently it wasn’t horrendously more than normal. If I read the notice correctly, they didn’t shut me down for excessive bandwidth; they shut me down for hogging the CPU (they kept talking about the “load” I was putting on the server).

They gave me the impression than an unoptimized WordPress installation can cause an excessive server load, and they gave me some links to articles that would help me. I spent Thursday and Friday night trying to figure out how to optimize WordPress.

By Friday night, I still hadn’t figured out what the hell I was doing, but I discovered how they turned my site off. So I merely undid what they did, and voila! I’m back up! I still needed to speed up WordPress, though. By Sunday, I decided that the method least likely to break some other part of WordPress was to install DB Cache Reloaded. That only caches the database, so it doesn’t conflict with other features.

There is one more optimization that I need to do. I’ve known for a while that a few people out there have been hotlinking my images. That’s just plain rude. It slows the server down, making page loads longer. Those people are stealing your time! When I was looking at the logs, I discovered that there’s a lot more of it going on than I realized. It looks like at least 20% of my bandwidth is being stolen. On a shared server, I don’t know what real impact that has on your load times. I do know that it makes me look like more of a resource hog. There are a couple of things I can do to block this, which I hope to implement within the week.

I don’t know if just installing the database cache and blocking hotlinking will be enough to keep this from happening again. We’ll just have to see.

spacer

Update (09/09/09, 9:09 AM)

The site was down again last night, this time with a server error. I received this message from Dreamhost support:

I am sorry to hear of this trouble but I think I have found the cause. Unfortunately, your “bof” user’s php5.cgi process is being killed by our procwatch daemon, and this is the source of the errors you are receiving.

Procwatch is a daemon that runs constantly on shared servers to monitor the usage of RAM/CPU and execution time so that no single user can use an inappropriately high percentage of the shared resources and impact the overall health of the server or the server’s ability to serve all users’ pages.

When it finds a process that is using too great an amount of resources, it kills that process. If it does this to a running script, the resultant error the server shows is “premature end of script headers” – the script can’t complete.

This is happening regularly with your site. Below please see excerpts from that log:

2009-09-09 03:06:56 procwatch2 INFO: Process(pid=29920, name=’php5.cgi’,
uid=ellyjonez(3477227), tty=None, cpu=0.2, rss=79412.0, vsize=133272.0):
kill for total RAM (79412.00 self, 157184.00 total exceeds limit)
2009-09-09 03:07:27 procwatch2 INFO: Process(pid=1338, name=’php5.cgi’,
uid=ellyjonez(3477227), tty=None, cpu=4.8, rss=80176.0, vsize=134096.0):
kill for total RAM (80176.00 self, 215744.00 total exceeds limit)
2009-09-09 03:07:38 procwatch2 INFO: Process(pid=3669, name=’php5.cgi’,
uid=ellyjonez(3477227), tty=None, cpu=0.1, rss=78948.0, vsize=132556.0):
kill for total RAM (78948.00 self, 157840.00 total exceeds limit)
2009-09-09 03:07:48 procwatch2 INFO: Process(pid=3108, name=’php5.cgi’,
uid=ellyjonez(3477227), tty=None, cpu=6.6, rss=82944.0, vsize=136708.0):
kill for total RAM (82944.00 self, 162840.00 total exceeds limit)
2009-09-09 03:20:47 procwatch2 INFO: Process(pid=5310, name=’php5.cgi’,
uid=tentimeywimey13(3468816), tty=None, cpu=4.2, rss=45356.0,
vsize=99792.0): kill for total RAM (45356.00 self, 184424.00 total
exceeds limit)

In short, what this is saying is that the combined PHP scripts run by the user that runs that site (“bof”) is simply using too much memory for a shared server. Note this could be a single site, or more than one site – with PHP it is hard to tell as each PHP process is only identified as php5.cgi. This can be telling us that a single request added to other running requests, or a single large request alone, is exceeding acceptable memory limits.

There are only two ways to resolve this:

1) Decrease the overall memory required for your scripts to process, create, and serve your site.

2) Move the site to a VPS (DreamHost PS), or dedicated server where you can reserve sufficient RAM for your own processes to use without impacting other users on the server.

The first approach can prove difficult depending upon your scripts, your content, and your expertise, but if you can do that your site can continue to run on a shared server without errors. The second alternative is simpler, but involves additional cost for the PS services, and sufficient RAM to avoid these errors.

I am sorry if this news is discouraging, but I felt it was important that you have the full benefit of knowing what is occurring here so you can make the appropriate decision to meet your needs for this site.

All I can think of is that one of the WordPress plugins has run amuck. I’ll try to fix this in the next couple of days. I doubt I’ll post any new articles until I get this figured out.

22 Responses to “Fixing Things”

  1. Parrotlover77 Says:

    Don’t you subscribe to a business level account? Disabling your site due to excessive load with that short notice would be very bad for business. How in the hell could anybody implement a workaround in that period of time? In addition, you’d think they’d at least let you pay more money to upgrade your server load.

    Does your contract indicate anything about what they do for excessive CPU load? This whole thing stinks pretty bad. I can’t say I’m too impressed with how your new host handled this situation. You are being punished for being popular.

  2. Ron Britton Says:

    I got the impression it isn’t a popularity thing. It was using the entire CPU. Google Analytics doesn’t show much of a traffic spike.

    I had a business account at 1&1. Here they don’t have different levels. I can get a private server if I want.

    A lot of WordPress plugins are poorly written. I’ve had trouble with those before. If something went goofy and hogged the CPU, it was probably one of those.

  3. Allanimal Says:

    There are a couple of things I can do to block this (hotlinking), which I hope to implement within the week.

    I say goaste ’em.
    (my spelling checker says that isn’t spelled right, but I am too scared to google it to find out)

  4. Ron Britton Says:

    Allanimal:

    I don’t want to be too disgusting. When I catch them, I usually substitute this image, which I found somewhere:

    Raping turtles for fun and profit

    That’s labor intensive, because I have to discover the incoming link, resize this image to match the one they’re stealing, then rename the stolen image and this one, so they get this one instead.

    There are settings or scripts that will just block requests from foreign servers. I’m going to use one of those.

  5. Sarah Says:

    Ha, I’ve also been getting a lot of Duggar-related search results in my stats. I wrote about them during the last pregnancy, but I refuse to give those mentally ill godbotherers any more virtual ink.

  6. Magnus Bergmark Says:

    I’d guess the recent comments sidebar can take quite a dent in your caches. Every time someone comments, all cached pages needs to be cleared instead of just the cached page for that particular article, rendering your cache almost useless if your comments/hit ratio is large enough.

    Also, there was a WordPress worm out in the wild a few days ago. If you got infected, this could also be a source for problems with load. I hope you are using the latest version of WP (but even that isn’t 100% safe yet!).

    If you want to find out the cause of the high load, perhaps you could set up some simple logging in your app. When the app starts for each request, log the date and time to a file. When it ends, log to the same file again, marking the request OK.

    If you check this file, you should be able to figure out which requests that got killed, and if you have a date and time of the request you could check your apache logs (should be under “logs” in your home dir, AFAIR) to see what caused it.

    Of course, this might already be present in the apache logs (error_log, for example). I don’t know.

    Good luck!

  7. Magnus Bergmark Says:

    Almost forgotโ€ฆ Have you checked if there is much traffic to your feeds. Give 1200 monkeys a RSS reader set to “Refresh every 2 minutes” and you got yourself (in average) 10 requests / second.

    Hopefully, your RSS feed is cached to the max.

    Be wary with the hotlinking script and RSS, though. It will block RSS readers, but that might also be a neccessary evil. ๐Ÿ™‚

  8. Brian Says:

    A facebook friend recently used the quote of the year while talking about the Duggars: “A vagina is not a clown car!”

  9. Ron Britton Says:

    Brian:

    A facebook friend recently used the quote of the year while talking about the Duggars: “A vagina is not a clown car!”

    That actually comes from a famous image that has been floating around for a while. It’s not original to me, but I think I was one of the first on the internet to stick it into a blog post. I’m not going to reproduce it here. Those people have contaminated this site enough. If you must see it, you can go to the original post.

  10. OtherRob Says:

    So what’s wrong with your house, anyway? Asbestos? Evil, godless, pagan ritual gone awry? Nosy people want to know. ๐Ÿ™‚

  11. Ron Britton Says:

    Sarah:

    Ha, Iโ€™ve also been getting a lot of Duggar-related search results in my stats. I wrote about them during the last pregnancy, but I refuse to give those mentally ill godbotherers any more virtual ink.

    I agree. I was afraid to even mention them in this post, because the comments will probably be taken over by people arguing about them. Those comments will be deleted. I only mentioned them at all, because it was looking like they somehow contributed to the site problems. I now think that isn’t true. I think it’s entirely a WordPress problem, independent of how much traffic I get.

  12. Ron Britton Says:

    OtherRob:

    In order for me to have a godless pagan ritual go awry, I must first have a godless pagan ritual. Those look like a lot of fun. I could really get into a midnight orgy in the woods. (hint to all you buxom pagans out there!)

    I merely got soaked by the upstairs neighbor, which caused mold, which made me sick. The moldy walls and asbestos ceiling meant they had to treat it as a toxic waste site.

  13. OtherRob Says:

    I gotta say, godless pagan ritual does sound much more interesting and fun than asbestos and mold. ๐Ÿ˜‰

    Hope you’re feeling better.

  14. Jeff Eyges Says:

    Obviously Fungar, the god of mold, is trying to get your attention.

  15. Sarah Says:

    I can’t post at all on my WordPress blog, and I don’t get insane amounts of traffic, so something’s definitely up with the site. Oh noes, I’ll have to vent my spleen on one of my other dozen blogs!

  16. OtherRob Says:

    Hey, Ron, we just found out that we have mold. Yay…. Our roof leaked with the massive amounts of rain we had and now we’ve got mold. I suspect there’s an old leak that wasn’t disclosed when we bought the house and the mold has been there for a while. No asbestos though. Any advice? (I’ll be calling people tomorrow.)

  17. Ron Britton Says:

    OtherRob:

    Hey, Ron, we just found out that we have mold.

    The first thing you need to do is stop the source of the leak. The second thing you need to do is dry the place out.

    Then you need to find everything that was water damaged and have it removed. In my case, that was part of the ceiling and a couple of walls. All the damaged sheetrock got pulled off the walls and disposed of. They then sprayed the studs with bleach (but then I dried off with a towel!).

    When you think all of the bad stuff has been removed, before you can go any further, you need to get your air tested for mold. If the results still show a large difference between outside and inside, that means they didn’t get it all. You need to find out where the rest of it is. Remove that and retest, ad infinitum. If you have any house left at the end of this process, then you can rebuild.

    (And if you ever need help choosing a toilet, I’m now an expert on that, too!)

  18. Parrotlover77 Says:

    I have a leak via the chimney somehow. Been battling that for months. The bright side is that it was caught early. So when it rains now I just put a fan on the subfloor to dry it out. It’s already been bleached. No mold. But I did get a sweet ass check from the insurance company. The actual repair costs from my contractor is half of what the insurance contractor was charging. Insurance company said they didn’t care what I did with the difference.

  19. Parrotlover77 Says:

    I should clarify that I’m not leaving it like this. lol. We just have to keep trying things until we find the leak… And it takes many weeks between rain showers to see if it worked…

  20. OtherRob Says:

    We’ve got calls out to a roofing guy, a mold guy, and the insurance adjuster. Just waiting to hear back. They’re all *very* busy right. In fact, the first roof guy we called wasn’t taking any new business.

    (I’ve put in a new toilet before so I think I’m covered there, but thanks for the offer, Ron. ๐Ÿ™‚ Though I did have to do it without any toilet forums…)

  21. Another Steve Says:

    They then sprayed the studs with bleach (but then I dried off with a towel!).

    I’ll bet you thought you got away with that. Well let me tell you bucko I’m watching your shenanigans.

  22. Brian Says:

    Not one to be left out, the other day my 7 yr. old son started the upstairs bath without telling anyone and then forgot about it. I’m hearing rain and I’m wondering where its coming from. I look outside. No rain. Downstairs bathroom? Good. Oh, shit. There’s water pouring down the wall below the bathroom. I wish it was a few years later when this will be funny.