...more recent posts
I changed the code slightly for the textarea boxes on all the /post pages. This is in response to long continuous strings of text (usually URLs) not wrapping for Dave in Firefox on Windows.
Did that solve it for you Dave?
Did that fuck it up for anyone else?
Well I had a little fuck up. I erased a bunch of new comment indicators in the database. Everybody with a user id over 50 has had all new comments marked as read in the database. These are the only people with elevated powers and a user id over 50 (not just 'remember me' signed in people):
joester
sally mckay
Kristin
selma
crystal
L.M.
So those people definitely lost any indication of new comments. Sorry about that. If you had some very old ones they might still show up but they'd have to be pretty old (over a month.)
To be clear, the comments haven't been erased. Just the red new comment markers on the front page.
Probably a lot of the other accounts over 50 don't actually read the site or use the new comments, but there are probably some at least on Sally's page. I'll write you an email Sally if you don't see this. Sorry.
It would appear that the server is under attack. And not from the googlebot or some other out of control crawler. This isn't http traffic. I don't know what it is. It is a single incoming spike in traffic that has happened like clockwork once an hour since yesterday. My guess, from a day of frantic googling, is that we are under a denial of service attack from a bot net. It's a pretty weak attack (incoming traffic only spikes to 2 mb/s, and it's just one spike an hour, not constant,) but evidently this sort of thing happens (presumably it's intermittent because the bot net is rotating through attacks on many different hosts.)
Of course, it still might not be anything malicious, but the traffic graphs are really weird, and my intuition is that it is malicious. This does seem strange though because we are really not hosting any controversial content.
I have mailed hurricane (my host) yesterday, but hadn't heard back yet so I just called. They had me mail the info to a different address where they said it would be looked at.
I'll let you know what I find. So far it is clearly not impacting the server (everything seems pretty snappy,) but the traffic level is very worrisome.
Holy cow we are being demolished by the googlebot! It is going crazy and absolutely pegging our bandwidth with bizarre page requests like:
/treehouse/date/2000/systemnews/index.php3/mrwilson/rachael/ mrwilson/arboretum.php3/dave/systemnews/index.php3/
WTF?
Unfortunately, instead of generating a 404 (since that obviously is not a real page) this requests generates the entire year 2000 archive for the /treehouse page (a huge transfer!) The system just sees /treehouse/date/2000/ and can't make anything of the rest of it, so serves the whole year.
Ouch.
I have temporarily disabled entire year archives. We'll see if that helps. Damn. I should have caught this a few days ago, but I wasn't paying close enough attention. Might cost me. Literally.
This is just a general update to let people know what fun the future holds around here.
I am very close to buying a new server. Okay, sure, I did buy that Pentium II Penguin server years ago (before we moved to he.net) and then never used it. And it is sitting right here on the floor under my desk mocking me as I prepare to drop some cash on yet another shiny toy. But I don't think it will dissuade me in the end.
Basically, or this is what I am telling myself at least, I am waiting for Apple to release Tiger Server (OS X 10.4). This is rumored to be announced on April 1 and delivered around the middle of the month. As soon as that happens my plan is to buy.
Then there will be some time where I have the server here and am configuring it and beginning to implement some of my new software plans. Say roughly the month of May. Then I will put it in its new home (I think 75 Broad St. here in Manhattan,) and continue to test and tweak. Probably I'll move other domains (the restaurants, etc...) over to the new box first and leave dmtree at hurricane until I am pretty sure of the new setup. Possibly we will move over sometime around the end of June.
That is a pretty speculative calendar, but that's what I am thinking. Our storage space will be immense. 1.2 TB in total (3 x 400 GB) although some of that I am going to use for redundancy so it won't all be available. Still, I imagine using at least one full drive (400 GB) for music. You can all thank Dave when that goes live. And then we still will have hundreds of gigabytes each for photos and video.
Bandwidth will increase to 1 mb/s which will be 4 times what we have now and roughly 8 to 10 times what we use now (not counting that one unusual spike last month which still didn't top .25 mb/s.) So space and bandwidth should be okay.
I am going to make lots of software changes, but I have no plans to make any low level changes to the code here, so those of you who remember scary tales of data loss from system upgrades of yore should probably worry slightly less than you might be inclined to. The main thrust of my changes will be to the large media file uploading subsystem and to the traffic logging subsystem.
Other goodies enabled by Tiger will be IMAP for email accounts (something we don't have now,) a Jabber IM server, as well as streaming audio and video servers. My first goal is to con(vince) Dave into having his own radio show.
And I would love to at least experiment with some live streaming video. A video camera attached to a laptop with an EV-DO card streaming H.264 over the internet to the server which then rebroadcasts to any connected viewers actually seems doable. Not sure what we would use wireless remote streaming video powers for, but this is NYC, and interesting stuff does sometimes happen.
Sorry it took so long to block all the 'andrewsaluk.com' referrers. I think they should be stopped now.
We've done well against comment spam, but now I need to turn my attention to all the referrer spam. Our friend Mr. Saluk is only the worst offender. In the process I want to rebuild the entire logging system so that we can have a much better look at what is happening in terms of site traffic.
I've added accesskey access to the navigation bars on all pages here. That means you can use control+ a key (or alt+ a key) to access the navigation bar items. So, for instance, pressing control+p is like clicking on [post]. Control+e is [editpage]. Control+u is [upload].
Let me know if it doesn't work in your browser. Pretty big time saver if you get use to it.
Just woke up to find the site down again. I am still not sure why mySQL is having this intermittent problem. It ran so well for years. The only thing I can think is that maybe as our hard drive becomes very close to full we are causing problems in some cache system. But that is pretty speculative.
Does anybody have any sense of when it went down? I think I saw it up around midnight.
Not that this would be of interest to anyone, but I made a web interface for the output of a mysql show status command. Despite the two recent outages, and despite the fact that none of our numbers here would be impressive to anyone who really knows anything, I am nonetheless constantly amazed at how much information the database serves.
Here is a short explanation of the different variables.
Another outage today. Worse this time, as all sites on the server were inaccessible. I'm sure my few customers are very pleased they hosted with me!
I don't know exactly what was wrong. But I strongly suspect this was the same MySQL problem. Somewhere there is a script that is (or scripts that are) connecting the MySQL, but then not releasing the connection when they are done. And with enough requests this eventually results in too many connections to the database.
Probably those requests would time out eventually and everything would start working on it's own, but rebooting the server made this happen right away.
Hopefully no one was mid post when this happened.
I am going to try to get to the bottom of this. I will keep you all posted.