vmguest corruption due to full nfs

Last tuesday tragedy hit. I felt as if the world had opened up and swallowed me whole. I thought I had lost 8 projects that were stored on the corrupted vm.
As rule we store our project data on a linux machine's /home path, which was supposedly stored on the vmguest.


It's strange how the mind works... I calmed myself down by desiging a more thorough backup system with rsync and nfs. Then I calmed myself down even more by redesigning how our switches could be more server-friendly, by optimizing for the number of available gigabit ports on the switches.


When I finally chilled the fuck out, I tried rebooting the vmguest twice more. I finally came to the realization that the vmguest had crossed the river styx for vmguests. Then I recalled that I had an rsync job that I wrote a while ago, that pushed away backups of /home over rsync. I checked the backup, apparently the last push was on last friday. Suddenly, I realized, there must be some vmdeity watching over me. I had already seperated /home from the vmguest. A while ago, I made this vmguest by cloning the real machine with this incantation:

dd if=/dev/sda of=/mnt/{external-drive}/image.raw

I converted the raw image to a qemu's qcow2 format and ran the image. Everything went well except for the cloned /home which was corrupted and unmountable. So I copied the /home from the real machine to the vmhost and mounted it as an nfs drive on the vmguest as /home. This ultimately saved my ass.

My project data was completely intact, only my linux kernel/userspace system was ruined. I reincarnated the system by installing an ubuntu system and installing pip, with a better functioning body, with a mounted nfs /home.
I was able to fix up about 60% of the system after working 16 hours that day. My minions helped to fix the rest of my fuckup for me.

What caused it? After I did some basic post-mortem checks of the system, I came up with the following:

Running vmguest images from an nfs or smb or any remote diskmount is a very very very bad idea, when the host running the remote diskmount's diskspace has a high probablity of hitting 100% some time soon.

In my case, I made the stupid mistake of scheduling a backup dump on the same remote diskmount. I had underestimated the available diskspace. So when the diskmount hit 100%, my vmguests locked hard and got corrupted.

The end

Don't run a voip server on a virtual host

On GNU/Linux with KVM/qemu virtualization do the following:

• Run an Active Directory (AD) vmguest for 50 people or less (samba4 or something else);
• Run everything actually on a vmguest...;

Except for mission-critical services you want to guarantee low-latency transmission of data... like voip. You run on bare-fucking metal. Oh and make  backups of the configurations. Start there and everything will fall into place.

After weeks of trying to eliminate parameters that may contribute to the latency of voip services in our network, I have to face up to the music.

It's not the 60Mbits/s down or the 6Mbits/s up connection; it's not the traffic control (linux); it's not the internal saturation of our gigabit switches; it's not the voip telephones; it's not the voip-carrier (external server); the bottleneck is the vmguest being housed with another bunch of other vmguests on the same vmhost.

This is quite easy to check, just start pinging your voip vmguest, when the voip connection seems stuttery, flaky or whenever you suspect udp packets are being dropped randomly. You'll get extremely long ping times or even negative ping times.

The vmhost is being hammered at rushhour, peak-time; so all the critical vmguests are being hammered. I think I knew this somewhere; but I buried it.

I learned my lesson. Always be brutally honest to yourself, about technical matters.

Update 05-01-2011:

3CX is shite. Go with asterix. Make sure you have a secondary outgoing line for voip. If your voip provider works with 3cx, run away for dear life.

I don't get social media

I don't get tweeting; or any other form of social media. I guess I like my privacy. I don't like dirty laundry hanging about. I don't want future employers to see my drunken nekkid self on some random website, ready to be used against me. (note to future employers: I abstain from being drunk and naked, unless you want to employ me as a drunk and nekkid programmer for tons and tons of cash; this is negotiable.)

Putting stuff out there about yourself feels like mental onanism to me. Putting stuff out there for no other reason than putting it out there, to boost your own self-esteem or some misguided desire to belong to a group, is stupid. I dislike blind conformity.

I guess I can understand why some people do it; especially those whose livelihoods depend on them being some type of social media personality; they have to be constantly out there, to sell there wares; be it good or crap, be it figuratively or physical. We have a strong need to eat, be sheltered, use plumbing etc. at regular intervals.

Running a blog is kinda like using social media, right? So why the hell am I publishing my fecal matter on the web, standing on this pedestal? Partly, the reason I'm doing this is because I have a strong desire to inform, teach, enthuse people about stuff that grab my attention. I like the idea that someone might take some bit of info from this and build something beautiful.

Also, I like seeing my own stuff out there. I can live another day without it, but it's nice to vent to a complete stranger. Onanism isn't all that bad. Self-release the tension building up.

For me it's all about using you, the reader, as a sounding board, to get some kind of connection. I probably won't live to see the singularity, when our humanity is pummeled into bits, bytes and words and human existence will cease to be of flesh and blood; when we're mind-melded up to the wazoo. This will have to do in the meantime.

Wine, winetricks and naughty fonts

When using wine and winetricks, winetrick might complain about samyak and oriya fonts; being naughty non-conformistic naughty fonts and so on.

Apparently, these are included in a bunch of linux distros.

In slackware 13.1 they are included in the package: "ttf-indic-fonts-0.4.7.4-noarch-...".

Just grep oriya in /var/log/packages/ if you don't believe me:

grep -nr oriya /var/log/packages ; xlsfonts 2>/dev/null | egrep -i "samyak|oriya"

Once you've blown the package away, you will find that winetricks will keep on complaining, about those fonts. This is due to "xlsfonts"; this binary is a part of Xorg to show all of your fonts. Run it, to see it for yourself.

I turned off this check in winetricks, by commenting off the code, but somehow the installation of .NET 1 to 3 kept crashing on me. Evidently that check is there for a good reason.

A quick google search reveals that xlsfonts reads its input from /etc/fonts. (edit: This is false. I still have no idea where xorg gets the idea samyak is on my system.)

I crafted a silly little sed command to try to remove it like this:

grep -nr riya . | sed 's@\(Lohit Oriya\)@<!--\1-->@'

Then I committed the changes to the files. Notice the parameter "-i".

sed -i 's@\(Lohit Oriya\)@<!--\1-->@' /etc/fonts/conf.avail/65-nonlatin.conf


Alas, that was a mistake. Since, samyak was found not oriya. Anyway, no harm done. Just uncommented some xml file. This can be easily undone, by firing up an editor and removing it by hand.

That'll teach me to want to use .NET instead of mono.

Argh. Dead end. How do I remove samyak from the system completely?!

Update 20-07-2010:  This is still broken in wine 1.2 with the winetricks from winehq.org. It still prematurely finds the offending samyak. Although .NET 1 through 3 can all install now. You can uncomment the lines in winetrick though to prevent the nagscreen; but you're on your own from there on.

Slackware 13.0 x86_64 cross-compiling to 32-bit

I'm a programmer.

I like programming.

I want my computer with slackware 13.0 x86_64 to churn out 32-bit and 64-bit linux binaries; and I want to run wine.

I want to learn to compile to multiple targets using HaXe. Those targets are flash, android, php, javascript, C/C++/SDL (iPhone). So, download it; unpack it (tar xf); install it as root.

So the first thing I had to do was to refit my x86_64 compiler to cross-compile to 32-bit.

The first thing you find on the intarweb is this: http://connie.slackware.com/~alien/multilib/. Don't skim it. You'll thank me. Oh, you also owe a thanks to Eric Hameleers (aka AlienBob) and Fred Emmott (aka fred, of slamd64 fame). Go on irc.freenode.net and thank them. Thanks to them you can run wine on slackware 13.0 x86_64 and thus access win32 applications e.g. Flash Develop (note: must get .NET 2 to work on wine; used winetricks but failed horribly, removed some fonts, gave up, subject of another blogpost ).

Do everything described in the previous link.

When compiling with slackbuilds from slackbuild.org (SBo), you need for 32-bit applicationsto run on your machine. It would be wise to run the slackbuild like this from the shell as root:


$ ARCH=i686 sh gc.SlackBuild


ARCH=i686 will force the variable inside the slackbuild script to take on this value: "i686" which activates a few parameters for gcc to compile to this architecture, which is the canonical x86 architecture. By the way, my example "gc" is Boehm's garbage collector. It is required for neko to run properly.

Next thing, learn to use "ldd". Type `which ldd` to find your ldd binary, it checks whether the dependencies of a certain binary you're running can find the libraries they depend on.

You might find out that the slackbuild for gc on SBo fails, with the following message:


checking for C compiler default output file name...
configure: error: C compiler cannot create executables
See `config.log' for more details.

Fortunately, someone only forgot to put "-m32" in this line:

...
elif [ "$ARCH" = "i686" ]; then
SLKCFLAGS="-O2 -march=i686 -mtune=i686" # <-- this line!

So add it there.

SLKCFLAGS="-O2 -march=i686 -mtune=i686 -m32"


Run it again, then you're good to go!

Good luck.

Lame first post...

Hi, intarweb.

This is my 2nd foray in the blogsphere. Forgive me for the lameness of this first post for it is a test.

Actually, this is my twentieth or thirtieth post; and third or fourth or fifth blog.

This is another one of my creations http://jstool.blogspot.com/ don't go here unless you want to read about drag and droppable objects into containers in crappy javascript and some whinging about jQuery, mootools and ext.

Another glimpse into the stuff that used to hold my attention: http://erlang-cms.blogspot.com. It's about erlang and the blog I wanted to build using erlyweb. Which is more or less a dead framework now. I drank the cool-aid, it's hard to believe I resisted using django for this. Erlang programmers, please fix strings, lol , rofl, tnx; binaries are cute, but they're so 20th century.

My primary goal is to give back the knowledge soaked up after being years and years on irc freenode and reading technical blogs.