Now, I think I’ll tell you a tale how I really lost all content of this blog and changed my server template from FreeBSD to Debian.
I first created zxstudio.org in 2010, using then-current FreeBSD 8 template, used by my hosting provider, FirstVDS. I hadn’t much experience with FreeBSD then, but I thought I don’t really need much in terms of configuration, and FreeBSD was the cheapest option, as far as I remember. How different from Linux could it really be?!
Well, it is different enough. You probably know, that most Linux distributions have binary packages manager, which allows you to install new software from repositories. Goold old FreeBSD 8 had no such option, until version 8.4, which came out later. Instead, you had ports. Port is a piece of software that was adapted (or created) for use with FreeBSD, but you have to build it from sources yourself. There are some programs that make this easier, like downloading and building all ports needed to build this port you need, but in the end, you still have to run C++ compiler on your machine.
This doesn’t present a problem, usually, but when all you have is a shared web server with 512Mb of RAM, you may not have enough memory to compile some of more complex C++ libraries and programs.
One bright Saturday evening, I set my eyes upon the task of installing Redmine task tracker for a project I’m not going to talk about just yet. Usually, installing open source web-based solutions isn’t a big problem: you just download an archive, extract it into your web site’s folder, maybe edit some configuration files or run a script. This, at least, was my experience with installation of such software prior to Redmine.
This time, however, proved to be very different.
For one thing, Redmine has quite a lot dependencies. So, instead of just downloading current version and plonking it into my www folder, I decided to use FreeBSD’s ports collection to install everything at once. Sure enough, there was a port of Redmine, and “portinstall redmine” provided me with about 80Mb of packages to download and build. I left it to its devices for a while, expecting to find a more-or-less working instance of Redmine when I got back. Instead, there was a log of errors. Some required modules, it transpired, simply could not be built with my puny 512Mb of RAM!
Not one to despair, I went googling for a solution. Pretty soon, I discovered that since FreeBSD 8.4, this OS can also use pre-built binary packages from developers’ repositories. I, however, was using FreeBSD 8.0, since that was what my ISP provided me with four years ago. So I banged “portsnap fetch && portsnap update && portupgrade -a” into terminal, to get my system up to date. As you can guess, that didn’t work quite as I expected (I still was expecting good things by then). But the first hurdle was not my site’s lack of RAM: it was the fact that the very first upgraded package simply refused to be built. It has no unresolved dependencies, but its build script was throwing an error. It was not a very descriptive error: just some syntax problems in Python script that was used in build process. Some googling brought me the answer: I needed to upgrade Python to version 3.x (my installation of FreeBSD came with 2.6). This was actually a quite reasonable request, but why that package did not have Python3 as a dependence, I will never know.
This difficulty resolved, I pushed forward with my upgrade plan. Pretty soon, I once again was hampered by “out of memory” error during compilation of one module or another. It didn’t stop me. The system was already not-quite-8.0-but-certainly-not-8.4, so it was broken, unless I was ready to go to the end. So I stopped httpd and mysqld to free some RAM. Again, it helped with some packages, but still others could not be compiled.
At this point, I thought my system to be beyond repair and it was getting pretty late. So, I decided to reinstall it to reset it to default state. It was, of course, absolutely necessary to back up the site and blog before doing it. The www portion of site provided no problems. I logged on with my FTP credentials and copied everything to my PC. There wasn’t any complex structure set up, so no problems arose. But I could not reach /var/db/ folder to back up MySQL database where blog messages were stored, because it needed root access.
So, I logged in as root with SSH, and simply copied contents of /var/db/ to site’s folder, so it would be accessible through FTP, and the tried to back it up to PC. “Access denied” was all I got for my troubles, since I forgot to change permissions to database files, and still needed root access to read them. This is where things got muddy. I guess I was too tired to think and act clearly, and here’s the lessons number one for you for today: DO NOT reinstall OS on your website when it’s already past midnight. I don’t really remember what I did, but at some point, when I tried to copy db files to PC, I got no errors, and, satisfied, proceeded to reinstall OS.
Surprise, surprise, when I tried to restore my backups, I discovered that only some log files got backed up! MySQL database was gone, and all post for the past four years with it, including some by my friend, whom I promised to provide a relatively safe and stable platform to post progress of his project.
If you think this was the end of my tale, you’d be wrong. The next day, I had a fresh FreeBSD 8.0 installation, clean blog, and still no Redmine. I was feeling down, but not out, so, since I have already destroyed my blog beyond any repair, I decided to go ahead and finally move my sever from the old FreeBSD hosting plan, which caused my so much pain, to a shiny new Linux one, with Debian 7, no less! I knew, that in Linux, I would not have to deal with pesky ports I need to build on my machine, but could just use binary packages from official repositories via wonderful apt-get.
So, I destroyed the old server (there was no upgrade option) and set up the new one. Then, I went to console and type “apt-get install redmine” into it.
Boy, didn’t that worked out well… Oh, the installation of Redmine and all its dependencies went without a hitch. No problems with lack of recent Python version or RAM here. But then, Redmine configuration script popped up. It was very nice. It asked me, which database should it use. What is the name of root user, so it could create the new database, and Redmine-specifics user. And them it failed. Because it never once thought to ask me for PASSWORD for root db user! I spent about an hour trying to get configuration script to work, but in the end, I had to give up on it: the bug, apparently, was reported to Redmine developers, but no fix was available, as far as I could see. So, I went about configuring things manually and setting up the database.
After a while, it seemed that I might yet succeed. Guess what happened.
Redmine isn’t your daddy’s web-based task tracker. It isn’t written in some old fuddy-duddy programming language like PHP or Perl, no sir! Instead, it uses moderately mature and very popular modern language called Ruby. That does not seem like a problem, until you consider the fact, that Ruby-based web software, including Redmine, somehow cannot be run by default installation of the most popular web server, Apache, which I happen to use. To get it working, you need to install an additional module called “Phusion Passenger”. That I did. However, no matter what I tried, I simply could not get Apache to recognize Ruby files as something other than text. After banging my head against this for about an hour, I’ve had enough. I purged all traces of Redmine from my system. I was defeated. This defeat cost me about 4 or 6 hours of my time, and the whole content of my blog, but I had to cut my losses.
In the end, I’ve installed MediaWiki (I did not really needed a task tracker just yet, but I thought it would be nice to have one ready when we will have tasks). MediaWiki is written in PHP and its installation took my about an hour, most of which I misspent by trying to get it to run from /usr/share/mediawiki instead of just placing the whole thing into my site’s folder, after which things worked instantly.
So, the second and the third lessons from this fiasco, I guess, are those: learn to cut your losses. When you need to wreck your whole OS just to install a piece of software, maybe you don’t need this software so badly. Also, Redmine sucks. Or maybe Ruby-on-Rails sucks, I’m not sure who’s to blame here, but read its installation guide. It shouldn’t be this complex. I can install WordPress or MediaWiki with far less fuss and bother. Maybe I just went wrong somewhere, and it isn’t so bad, but, well, that was my experience with Redmine, and it was not just bad, it was UGLY.
That’s all I have to say on the matter.
Uhm, the real lesson here is that you need to:
1) Learn to make proper backups
2) Check your backups after you make them!!!
By they way, you’re lucky your server never crashed and lost all of it’s data before, that can happen, and does happen, a proper server admin doesn’t even blink an eye if it does. He just restores the backup and goes on with life. Sorry to say but your lessons learned are wrong!