git tear-hair

I use git. It's fast, it's convenient, it gives you github, and mostly it works. But several times now I've managed to bollix a git repository, been faced with impenetrable messages, and been unable to continue using git unless I sort them out. This latest time I had enough stubbornness to figure out how to un-bollix my repository, and I wanted to record it for posterity (specifically, for people whose problem-solving technique involves Googling for the specific error message). The error I received was:

error: object file .git/objects/86/5d2dffe9a3d72917934ed9693c7167efb6d8d5 is empty
fatal: loose object 865d2dffe9a3d72917934ed9693c7167efb6d8d5 (stored in .git/objects/86/5d2dffe9a3d72917934ed9693c7167efb6d8d5) is corrupt

Read on for how it happened, my understanding of what it means, and how I fixed it. Or skip this even-more-technical-than-usual post; I'll try to post something with corals or kittens soon.
What went wrong was, I was using my laptop on battery (outdoors, to try to reach dodgy Central American wifi), and the battery was running out. Like, imminently. So I did a quick git commit, let it finish, and asked the laptop to turn off. All appeared well. But when I turned the laptop back on again, I received the above error on essentially any git operation.

What I think happened was that while the laptop was trying to power down, perhaps when the userland CPU power manager was shut down, the battery died and shut it down hard. Unfortunately, the kernel still had not written all the file contents to disk, and the objects that were supposed to be stored in git were left as empty files. Specifically, the commit had been logged, the branch had been updated to include the commit, but neither the tree nor any of the file contents objects had been written to disk (or rather, the files existed but had zero length). This left git extremely unhappy.

The solution, ultimately, was to blow away the commit and reconstruct it as best I could. I still had the working copies of the files (somewhat updated, because — see "dodgy Central American wifi" — I didn't wait to solve the git problem before working on my project), and I could find the commit message and previous commit SHA in the text file .git/logs/HEAD. So not much was lost by pretending the commit never happened, going back to the previous tree, and checking in the working copies again. (Of course I kept a copy of the whole tree and its .git directory...)

To do this, first I ran git fsck repeatedly, and every time it complained about a zero-length loose object I just used rm -f to get rid of it. Then the only problem was that HEAD pointed to a nonexistent commit. Actually, .git/HEAD is a sort of symbolic link pointing to (in my case; it's just a text file) .git/refs/heads/master, which itself points to a nonexistent commit. So I extracted the SHA of the previous good commit and ran git update-ref refs/heads/master SHA. Lo and behold! git fsck is clean, I can now commit again.

And yes, backups are good; if I'd been pushing to github all along I could have just pulled down a new clone; while I would still have lost this commit, I would not have had to learn git plumbing commands. But see "dodgy Central American wifi" above. Speaking of which, I hope this post saves okay.