Pages

Wednesday, December 15, 2010

Wat ik geleerd heb van het kopen van een pak

Introductie

Blijkbaar is het een ongeschreven regel dat men voor een solicitatiegesprek voor een management of leidinggevende positie in het algemeen, met een pak moet aankomen. Ik ga ervanuit dat de lezer, net zoals ik, graag herenpakken wil (gaan) dragen, dus ik schrijf deze blogpost niet vanuit het perspectief van de vrouw. Maar sommige principes die ik beschrijf zijn ook wel van toepassing op vrouwenpakken, denk ik.

Gij zult een corporate kostuum dragen voor een belangrijk gesprek. Ongeacht of die mooie trui waarmee jij wil verschijnen en een goede indruk wil wekken honderden euro's gekost heeft. Ongeacht of dat speciaal shirtje van je, je eerder geluk heeft mogen brengen, waardoor jij dat shirt al in jaren niet meer hebt gewassen... doe gewoon een pak aan. Zoals dat sentiment mooi verwoord wordt in de tv-serie, HIMYM (How I met your mother):







Basisstappen

Nee? Nog nooit een pak gekocht? Geen ramp. Let op de volgende zaken:
  • Bepaal ten eerste wat je maximaal wil gaan betalen, kortom bepaal je budget, als de euro geen zware inflatie ondergaat, dan zal 450 euro voldoen, incl. schoenen, overhemd en das);
  • Kijke, kijke, niet kope, zoek een aantal winkels op in de grote stad, wellicht ook in de buurt van jouw omgeving;
  • Bepaal bij de eerste zaak die je bezoek welke maat je hebt, door dat aan iemand te vragen, dit scheelt tijd namelijk, voor je volgende bezoeken;
  • Vraag om hulp, wees niet te trots om hulp te vragen, wees assertief, maar niet onbeschoft, jij bent de klant en de klant is koning, maar vraag expliciet aan de verkoper of hij zijn professionele advies wil geven, wanneer je merkt dat er te vaak ja wordt geknikt;
  • Bekijk verschillende pakken, let op bepaalde aspecten wat je wel of niet leuk vindt;
  • Vraag de hulpgevende winkelbediende (of hoe dat tegenwoordig heet), hoe lang verkoper werkzaam is in de kostuumbranche (herenmode branch);
  • Toets de kennis van de verkoper, bijv. vraag wat voor effect horizontale, diagonale, verticale lijnen heeft op de drager van een pak:
    • Horizontaal maakt dik (als je slankt bent);
    • Verticaal maakt lang;
    • Diagonaal voor het versterken van het contrast t.o.v. de reeds aanwezige lijnen. Je ziet doorgaands nooit pakken met diagonale strepen, dit wekt de indruk (vermoedelijk) dat je wankel bent, en scheef staat.
  • Als je een zaak binnenloopt, bepaal alvorens voor jezelf met welk doel je die pak gaat aantrekken, bij voorbeeld een blauw pak met verticale strepen, dus je lijkt langer en autoritair, in ieder geval, met die pak straal je zelfvertrouwen uit, vertel de verkoper welk doel jij beoogt;

Het beïnvloeden van je perceptie

Manchetknopen

Een competente verkoper zal je op vriendelijk en toeschietelijk wijze benaderen, die zal lachen om jouw grapjes (stel dat je ze maakt). Je zult die persoon wellicht mogen. Wanneer een potentiële koper een verkoper mag, dan wordt er een weg gebaand voor wederkerigheid. Kortom, de verkoper is aardig tegen me, dus ik zal als koper-zijnde zal je belonen d.m.v. de aankoop op deze pak en talloze dure acessoires. Mensen zijn eerder geneigd om zich te associeren met mensen die zij uit een positieve ervaring kennen, waardoor een koper eerder geneigd om dezelfde aankoop op dezelfde plaats te doen.

Let op de invloed van perceptuele contrast, d.w.z., wanneer je veel dure dingen om je heen hebt, dan zullen andere dure kleine artikelen (relatief t.o.v. andere winkels) alleen normaal geprijsd lijken, bijv. je wordt omringd door pakken van duizenden euro's, (die je uiteraard, niet allemaal moet gaan kopen) dan lijkt een paar sokken van 25 euro normaal. Terwijl je misschien bij de hema voor dat bedrag 50 van dezelfde paar sokken kan kopen, als het ware. Kortom, besteed geen geld aan de acessoires die relatief normaal afgeprijsd lijken. Koop die horloge, overhemd, dassen, schoenen, sokken, manchetknopen elders als je geld wilt besparen. In dit geval is het contrast normaliserend.

Perceptuele contrast geldt ook voor hebbedingetjes die in de aanbieding zijn, let op de prijzen van alle artikelen in de winkel. Is alles boven 500 euro en hoger?  Is die paar sokken afgeprijsd van 75 naar 25 euro? Voor paar sokken... wauw, dat lijkt goedkoop. Hier is het contrast versterkend.

Het is verstandig om te bepalen welke winkels goedkoop zijn en welke duur zijn. De aanname, dat goedkoop is duurkoop of dat duur impliceert goede kwaliteit, gaat niet atijd op, wees daar bewust van. Het gaat om de perceptie van de mensen die de pak waarnemen. Als je zegt dat iets duur is, (zonder dat dit werkelijk duizenden euro's gekost heeft), als men die perceptie al hebben, dan is dat ook zo in hun verbeelding. De reden waarom grote bedrijven die pakken verkopen, mega-winst kunnen maken is omdat zij het hele productieproces uitbesteden aan lage-lonen landen. De werkelijke waarde is een optelsom van de materiaalkosten plus productieloonkosten plus vervoerkosten plus verkoperskosten, denk aan provisie voor elk pak voor de verkoper en zijn uurloon. Ik heb geen concrete cijfers gezien, maar ik heb zo'n flauw vermoeden.

Stel dat de pak die je wilt nog enigzins bijgesneden moet worden of op één of een andere manier bewerkt moet worden en je hebt nog niet definitief besloten dat je de pak wil aanschaffen. Een effectieve verkoper zal jou proberen te beïnvloeden, d.m.v. de schaarsheid principe.  Bijvoorbeeld, stel dat je heel snel binnenkort een pak nodig hebt, dit heb je per ongeluk verteld aan de verkoper. De verkoper zal je aan herinneren dat jouw pak (ongeacht welke pak dat moge zijn) wel 4 à 6 weken zou gaan duren om te maken, om je ook indirect te herinneren aan gemiste mogelijke kansen in het verschiet. Het effect van deze herinnering is dat je een enorme aandrang zult voelen om die betreffende pak meteen aan te schaffen. Mensen raken in het algemeen sneller in paniek wanneer zij geloven dat zij dingen zullen mislopen in de toekomst, waardoor zij de helderheid verliezen om verstandige besluiten te nemen.
Afgezien van de schaarste in de tijd, kunnen verkopers producten ook schaars laten lijken. Denk aan zeldzaam geziene pakken die gemaakt zijn door een of een andere belangrijke ontwerper die niet zoveel pakken maakt.

Een wat oudere (uitziende) verkoper, of een verkoper die meent dat hij al langer in het vak zit, zal mogelijk erop aandringen dat je zijn advies moet nemen en vele autoriteitsargumenten geven, zoals: "Uit ervaring heb ik geleerd dat bladibladibla.". Je brein vertelt je dat deze persoon is een autoriteit, je brein zegt: "...dus ik moet wel zijn advies volgen, anders is dat ten nadele van mij, die niet zoveel weet hierover...".

Een goede verkoper zal veel voorstellen doen wat betreft acessoires, opties, combinaties etc. De verkoper zal je veel dingen laten aanraken.  Zij weten echter dat wanneer verkoopartikelen eenmaal aangeraakt worden door een potentiële koper, dan zal dat artikel met een grotere kans gekocht worden door de (hand)tastelijke koper. Ik vermoed dat dit te maken heeft met de commitment en consistentie principes. Wanneer de commitment voor het kopen van een artikel reeds vaag aanwezig is, zal de commitment hieraan versterkt worden door een positieve tastzintuigelijke waarneming, waardoor het consistentie gedeelte van de beïnvloeding zal zegevieren over de rede, (heb ik deze prullaria wel nodig?), kortom, je brein zegt: "oe leuk, koop die hap!".


Contrast

Vermijd cognitieve dissonanties, geen vloekende kleuren (bijv. laat iemand die niet kleurenblind is je ensemble'tje nakijken). Sluit belachelijke contrasten uit, bijv. hele donkere sokken bij een witte broek, behalve als je moet moonwalken als Michael Jackson. De waarnemer van je pak, bij degene waar de coginitieve dissonantie ontstaat, zal bewust of onbewust een oordeel vormen over jouw hele persoon op basis van deze schijnlijke bedrieglijke onbelangrijke "foutje". Wellicht is de perceptie van de waarnemer van jou in één woord samen te vatten: slordig.

Wat ik nog niet genoemd heb is social proof en conformity, vrij vertaald, sociale bewijsvoering en conformiteit, dit is wanneer een groep verkopers menen dat een bepaald voorwerp je mooi staat, bijv. een duur acessoire bij je dure pak, terwijl jij dat ding zelf bij voorbaat afschuwelijk vindt. Er ontstaat een grote aandrang om te conformeren aan de verwachting van die groep, ik kan me voorstellen dat de volgende gedachtes door je hoofd gaat: "Er zijn zoveel mensen die hetzelfde denken, ik zal wel gek zijn. Ik zal het maar moeten kopen."

Deze psychologische beïnvloedingstactieken zoals liking (iemand mogen, vriendelijk zijn), autoriteit, perceptuele contrast,  schaarsheid, wederkerigheid, commitment en consistentie, sociale bewijsvoering en conformiteit zijn ook van toepassing op andere branches. Denk aan een mooie auto die bij je pak past in de autoverkoopbranche. Dat artikel schrijf ik later wel als ik zelf hoger ben opgeklommen en een auto nodig heb.

Onderhandeling

Een beetje ervaren verkoper zal je gelijk een derde oor proberen aan te naaien, maar trap er niet in! Ga in discussie, neem vooraf onderzoek naar prijzen van pakken en gebruik deze feiten die je oppikt tijdens de onderhandeling over de prijs. Maak aantekeningen als je geen goed geheugen hebt. Neem desnoods foto's, maar vraag eerst om de toestemming van de zaak.

Val de verkoper aan op prijs, gebruik wat je weet over de objectieve gemiddelde prijs, maar wees geduldig en beleefd voor de andere onderhandelende partij.

Bijvoorbeeld, zeg dat je twee dassen wil met die pak, all-in. Want dat scheelt jullie twee de moeite. Het scheelt de verkoper de moeite om een nieuwe sukkel te vinden. Voor jouw scheelt het de moeite om die dassen elders vinden. Probeer anders af te dingen, je zegt bijv. "okee, één das dan...". Probeer wederzijdse concessies te maken.

Stel dat je onderhandeling mislukt, je zult die das niet zomaar krijgen met die pak, all-in, want blijkbaar is de prijs echte heel scherp (o, echt waar?), we hebben nog altijd het internet. Bovendien, er is slechts één maat voor alle dassen. Tijdens de onderhandeling, probeer te benadrukken dat jij onderhandelt vanuit het algemene belang, dat je uitgaat van een win-win situatie. Zeg dat je binnenkort (half jaar?) ook voor andere festiviteiten een pak nodig hebt, wat verdomd veel lijkt op een lange-termijn relatie lijkt tussen jou en verkoper.

Conclusie

Een pak kopen vereist voorbereiding. Om te voorkomen dat je te veel betaalt voor iets moet je bewust omgaan met bepaalde beïnvloedingen om je heen.

Links & Bronnen

What I learned from a headhunter for a managerial job opening

First of all, I am an unofficial student of dr. Robert Cialdini's work, with regard to his published work on influence. Since I read and reread his seminal work, the world started to make sense, I started to understand on a deeper level, why people did certain things in a certain way. Especially people who wanted you to be compliant to their way of thinking or commit to their cause and be consistent about it too.

When dealing with these types of people, for instance this headhunter I met, he started with authority, i.e. "We are among the top three headhunters in the world, we have 600-700 shops around the world". This might not be true, but it's damn impressive nonetheless.

Then he proceeded with scarcity, i.e. "We only handle cases starting from 45k euro per annum otherwise it isn't interesting for us; when we request an interview with someone we mean business, we know who are the best", in my opinion the last is also meant to flatter the interviewee, which is liking and still a hint of authority, e.g. "We only interview the best" or some variation of that.

I think this is standard stuff for headhunters.

Suppose that you botch up the interview, by not being completely awake, because you couldn't sleep, with all the excitement bubbling inside of you because you have an interview the following day for a very special position (read: ka-ching). Anyway, you realize half-way you're not getting the job (right now), because you're not doing too well selling yourself, with buzz-words and short soundbites to describe your professional skills and experience, so you start to actually relax and do quite well, because you start to act as yourself and let your guard down.

In this case, apply active listening (from dr. Thomas Gordon), so the interviewer will tell you more than (s)he is actually allowed to tell you. Like the name of the company, so you can skip over the headhunter and go for the company yourself and negotiate.
I don't recommend screwing over headhunters, because they are also human and fallible, but if they start getting sloppy, they should learn like all human beings, e.g. by error, pain etc.

Get a suit for the interview


Okay, the things I learned actually can be summarized in bullets:
  • Watch your body language, give an impression with your body that you're comfortable with talking about yourself and you're enthusiastic about the job prospect;
  • Familiarize yourself with real names of positions you would like to attain further on the (cannibalistic corporate canine-trophic) ladder;
  • If you come from academia or your family and friends tend to talk on an abstract level, please leave the academic at home, be concrete, do not hide behind abstractions as a protection mechanism (i.e. a way to stay safe from a barrage of prying questions);
  • Wear a suit and proportional management shoes for management positions, comb your hair;
  • Wrap around stories based on certain keywords that are related to this position, e.g. I worked day and night on blabla, above and beyond..., (so you're dedicated); I negotiated hard on blabla, (so you're assertive...);
  • Tell something about yourself with interview power words and emotive stories;
  • Apply influence, make the stories believable and compelling, so they end up with the same conclusion or lead them to the same conclusion. Also use the interviewer's name when giving examples;
  • Do not be self-deprecative, do not slag off your previous employer(s), just say you reached a certain level of growth in your profession, you want to excel beyond this level (mind you, excel is a power word);
  • Oh, don't make shit up, they will find out. (I didn't, because I assume this is common sense and eventually you'll probably end up getting black-mailed.);
  • Make sure your references back you up, because don't underestimate the power of social proof.
  • Practice your pitch with an unbiased observer!
Good luck with the job hunt.

Tuesday, October 12, 2010

mutt smtp server support (no local mta)

Suppose you want to use a text-based mail client without having to install and setup a full-fledged mail-transfer agent (mta). Mutt from version 1.5 on, has support for your smtp
server (internet mta) on the internet.

All you need to do is recompile with the extra option --enable-smtp in ./configure.

See the last line of the following command:

# ...
./configure \
  --prefix=/usr \
  --mandir=/usr/man \
  --sysconfdir=/etc/mutt \
  --with-mailpath=/var/spool/mail \
  --enable-pop \
  --enable-imap \
  --with-ssl \
  --enable-locales-fix \
  --without-wc-funcs \
  --build=$ARCH-slackware-linux \
  --enable-smtp # <-- add this extra argument!
# ...
To configure the server you need to create a file called .muttrc in your $HOME and enter the following settings:
#openssl aes-256-cbc -in test.txt -out test.txt.enc # encrypt
#openssl enc -d -aes-256-cbc -in test.txt.enc       # decrypt
#echo "body" | mutt -s "subject" -F muttrc addressee@somewhere.com -a attachment
set smtp_url="smtp://smtp.server.tld"
set realname="Mutt Forevah"
set from="Who-ay-Em Ayemhoo"

Adapt to your smtp server settings accordingly.

Sunday, August 29, 2010

wget: downloader extraordinaire

I always found browsers too bothersome or intrusive when downloading files. The task of downloading files via browsers always require too much clicking and configuring. I'd rather spend time on the task at hand rather than configuring the crap out of some unintuitive graphical user interface.

I always have a shell handy when I'm browsing, so I always download files using wget.

I use an alias to predefine the arguments in $HOME/.bashrc I usually need to download like this:


alias wget="$(which wget) -m -U 'Mozilla/5.0 (compatible; Konqueror/3.2; Linux)' \
-e robots=off --wait 1 -c -nd -nH"


I put a which in there because I'm too lazy to type /usr/bin/wget; the purpose of putting it there is to escape any other current aliases or scripts which might get called using the same invocation name, namely wget.

The arguments there are crafted to make wget ignore robots.txt, to make wget pretend to be a browser. This is generally frowned upon. People call this bad netiquette. As I said, I'd rather get the job done...

If you decide to download recursively, you'll need the following commands:

  • Download recursively with -r --level n, where n is the maximum recursion depth;
  • To reject all files except, you'll need -R<.ext>, where .ext is some type of file extension, for instance .mp3
  • To allow only a certain file extension, you'll need -A<.ext>.

Have fun!

Tuesday, August 24, 2010

Slackware 13.1 woes

Slackware, why have thou forsaken me?

At home:

The upgrade from 13.0 to 13.1 killed my /dev/sdX where / is mounted, apparently 13.1 has no love for reiserfs anymore. (On the other hand, my hard drive might be showing symptoms due to its age)

I hope it's nothing serious. Gotta fix it fast though.

At work:

Xorg on 13.1 does not like me changing keyboard settings from hal, after I done the following:

cp /usr/share/hal/fdi/policy/10osvendor/10-keymap.fdi /etc/hal/fdi/policy/
nano -w /etc/hal/fdi/policy/

Before:
<?xml version="1.0" encoding="ISO-8859-1"?>
<deviceinfo version="0.2">
  <device>
    <match key="info.capabilities" contains="input.keymap">
      <append key="info.callouts.add" type="strlist">hal-setup-keymap</append>
    </match>
    <match key="info.capabilities" contains="input.keys">
      <!-- Restore Ctrl-Alt-Bksp Xserver Zapping -->
      <!--<merge key="input.xkb.options" type="string">terminate:ctrl_alt_bksp</merge>-->

      <!-- Edit (as needed) these four lines in the copied fdi file -->
      <merge key="input.xkb.rules" type="string">base</merge>
      <merge key="input.xkb.model" type="string">evdev</merge>
      <merge key="input.xkb.layout" type="string">us</merge>
      <merge key="input.xkb.variant" type="string"/>
    </match>
  </device>
</deviceinfo>


After:
<?xml version="1.0" encoding="ISO-8859-1"?>
<deviceinfo version="0.2">
  <device>
    <match key="info.capabilities" contains="input.keymap">
      <append key="info.callouts.add" type="strlist">hal-setup-keymap</append>
    </match>
    <match key="info.capabilities" contains="input.keys">
      <!-- Restore Ctrl-Alt-Bksp Xserver Zapping -->
      <!--<merge key="input.xkb.options" type="string">terminate:ctrl_alt_bksp</merge>-->

      <!-- Edit (as needed) these four lines in the copied fdi file -->
      <merge key="input.xkb.rules" type="string">base</merge>
      <merge key="input.xkb.model" type="string">evdev</merge>
      <merge key="input.xkb.layout" type="string">us</merge>
      <merge key="input.xkb.variant" type="string">dvorak</merge>
    </match>
  </device>
</deviceinfo>


Although I like the zero-configuration philosophy behind it; that's why I refuse to set it through the traditional /etc/x11/xorg.conf.

Hald should be deprecated for udev really. The XML overhead is stupid if you can adapt things to your needs through command-line like this:

In $HOME/.bashrc add:

alias aoeu="setxkbmap -model evdev -layout dvorak -variant intl -option grp:win_switch"

Or if you have an EZ-Reach (tm) Typematrix 2030 keyboard like me, you can add this to your /etc/udev/rules.d/99-custom.rules:

SYSFS{idVendor}=="1234", SYSFS{idProduct}=="5678", MODE="660", SYMLINK+="input/typematrix%k", GROUP="plugdev", RUN+="setxkbmap -model evdev -layout dvorak -variant intl -option grp:win_switch ; loadkeys dvorak"

Run the lsusb command to get the idVendor and idProduct values, my lsusb output goes something like this:

...
Bus 004 Device 002: ID 047d:1020 Kensington Expert Mouse Trackball
...

If you own a  Kensington expert trackball (mouse) like me, then you'll probably want to borrow my fdi definition, in the following block of xml code:

<?qxml version='1.0' encoding='UTF-8'?>
<deviceinfo version='0.2'>
  <device>
    <match key='info.capabilities' contains='input.mouse'>
      <merge key='input.x11_driver' type='string'>mouse</merge>
      <merge key="input.x11_options.Device" type="string">/dev/ttyS0</merge>
      <merge key="input.x11_options.Protocol" type="string">ThinkingMouse</merge>
      <merge key="input.x11_options.Emulate3Buttons" type="string">false</merge>
      <merge key="input.x11_options.CorePointer" type="string">On</merge>
      <merge key="input.x11_options.Buttons" type="string">4</merge>
      <merge key="input.x11_options.ButtonMapping" type="string">1 2 3 4</merge>
    </match>
  </device>
</deviceinfo>

End rant.

Once you go slack, you never go back.

Friday, August 20, 2010

Simple script to alert you when websites are down

Suppose your clients and boss would like to know when a site you maintain should go down.

You can write your own script to achieve this goal, where you implement a light-weight http client in python-twisted... but why bother?

There is a thing called reinventing the wheel.


So what do we need?
  1. curl;
  2. logrotate;
  3. mutt;
  4. A list of urls;
  5. A log file to record visits by curl;
  6. A script called site-checker, which one puts in the /etc/cron.hourly/ and chmod 770 /etc/cron.hourly/site-checker:


    #!/bin/bash
    
    #set -e -x
    
    FILELOC=/home/user/bin/websites.list
    LOGFILE=/home/user/bin/log/site-checker.log
    WEBSITES=$(cat $FILELOC | xargs)
    UDATE=$(date -u)
    EMAILADDR=user@nowhere.com
    for w in $WEBSITES
    do
      echo "==Fetched $w on $UDATE==" >> $LOGFILE
      curl -sf $w || echo "Warning: $w is down on $UDATE" | mutt -s "[website down] $UDATE $w" $EMAILADDR $
    done
    

websites.list would contain something like this:

www.microsoft.com
www.google.com
www.sap.com
www.oracle.com

For mutt to send e-mails you either need to setup a MTA (mail-transfer agent) or add a .muttrc file to your $HOME, containing something like this:

set smtp_url="smtp://user@smtp.woohoo.com"
set smtp_pass="tuff2guess"

You also have to prevent your site-checker.log from growing too big with logrotate.

Type this in your bash shell:

echo -e "/home/user/bin/log/*.log {\n\
  daily\n\
  missingok\n\
  size=100k\n\
  rotate 5\n\
}" > /etc/logrotate.d/user-custom\


As with every single script on this blog, you must adapt the script to your own needs.

Happy hacking!


Update 21-dec-2010:
#!/bin/bash
FILELOC=/home/user/bin/websites.list
PEOPLELOC=/home/user/bin/people.list
LOGFILE=/home/user/bin/site-checker.log
WEBSITES=$(cat $FILELOC | xargs)
UDATE=$(date -u)

mail () {
  read message
  domainname=$1
  for addr in $(cat $PEOPLELOC | xargs)
  do
    echo $message | mutt $addr -s "[website down] $UDATE $domainname" 2> /dev/null
  done
}

for w in $WEBSITES
do
  touch $LOGFILE
  echo "==Fetched $w on $UDATE==" >> $LOGFILE
  curl -sf $w || echo "Warning: $w is down on $UDATE" | mail $w
done

Linux: conflict resolution, usb sound card vs. webcam

Here's what you do to resolve a conflict in alsa, caused by some device hijacking your default alsa device to play sound. This could happen to you right after you upgrade your kernel.

  1. Hit cat /proc/asound/cards to locate the offending device;
  2. You prevent the offending modules that belong to the offending device from loading, by blacklisting the in /etc/modprobe.d/blacklist;
  3. Check the actual device you actually want, by hitting cat /proc/asound/cards
    You'll find something like this:


    0 [XYZ123         ]: HDA-boo - HDA ...
                          HDA ... at 0xdbef8000 irq 22
    
  4. Attack /etc/asound.conf by adding these lines:

    pcm.!default {
            type hw
            card XYZ123
    }
    
    ctl.!default {
            type hw
            card XYZ123
    }
    

The last step sets your default audio alsa device.

Hack on!

Friday, August 13, 2010

E-mail relay: emailrelay

Imagine you work in a company with dozens of employees, e-mailing each other, e-mailing clients etc.
  1. You'd like to sample correspondences from time to time, you achieve this by storing the messages and attachments in a database (nosql or sql-based);
  2. ...  but you don't want the hassle of setting up a foll mta (mail transfer agent), with foll-blown policies, like postfix and the like.
  3. .... you also want out-going e-mails scanned for spam and viruses, to prevent from getting blacklisted by other ISPs (in case there's an malware infection somewhere in the local network);
  4. ... you'd like to get jiggy with new open protoclis like xmpp and transfer internal e-mails over xmpp as well, with attachments too;
  5. ... recognize and reroute internal e-mails by bypassing the ISP's smtp server and sending it to the Exchange server or what have you;
  6. ... you want to mount the database which stores the e-mails and apply some clustering algorithm on it to gather data while e-mails are in transit... in other words online learning;
  7. ... you also have VOIP over SIP, which you would like to be stored with the e-mails.

Relaying e-mail


How would you do all that?

Personally, as a programmer, my initial reaction was: I'll roll it myself (using erlang)! I don't have the time to actually do it of course. The most I can do with the given time is to write some bash files to glue existing projects together.

I googled a bit and found a nice project called emailrelay. This wee project does what it advertises. Even better, it's licensed under gpl 3.0.

Why reinvent the wheel? When all the components are already here? Spamassassin, clamav, cron, ejabberd, etc.

So what does it do exactly? It can relay e-mails directly, or do it in two steps: collect and relay. Why is this relevant? It's relevant because mailrelay allows you to use third-party software to manipulate the e-mails in-transit.

For instance, during the collect phase we defer the decision to relay them or not to spamassassin and clamav, and during the relay phase compare the recipient address to the local addresses, and thus bypass the ISP MTA by sending it directly to the recipient (by some protocol i.e. smtp, xmpp, what have you).

Another example, imagine you want to send yourself a TODO item via emailrelay and you want this item synchronized with your existing TODO lists. All you need to do is grab some api/library for your favourite scripting language and voila, you're almost done. You still need to write the glue code though.

You can pick your favourite programming language (e.g. bash, java, python, ruby...) and let mailrelay pass a parameter to your application or script (is there a difference?) and compose whatever behaviour or policy you want during the collect or relay phase.


Some benefits of this approach in contrast to a traditional full-blown MTA:
  • You can test each piece of software at any level;
  • You can adapt to any network topology depending on where the bottle-necks are and how your software pieces should operate optimally (i.e. serial or parallel on one or multiple computers, multiple processes, single processes, vmguest or real machine...);
  • You can mix and match pieces of technology to get what you want. See this example of spamassassin.

The drawbacks of this approach:
  • You have to write your own (standard) policies and (standard) behaviour;
  • No monitoring of crashing components (supervisor a la erlang, anyone?) between the cooperating processes;
Non-monolithic composable software. Ain't it grand?

We use couchdb to store the e-mails and attachments etc. and replicate it to other nodes etc.

Here's some code to get you started.

Daemon initialization code:

#!/bin/bash
set -e -x

ROOT=/home/administrator/email
/usr/local/etc/init.d/couchdb restart
if [ -e $ROOT/scripts/pid-emailrelay-server.pid ]; then
  kill `cat $ROOT/scripts/pid-emailrelay-server.pid`
fi
if [ -e $ROOT/scripts/pid-emailrelay-client.pid ]; then                  
  kill `cat $ROOT/scripts/pid-emailrelay-client.pid`
fi
#server
$(which emailrelay) --as-server --port 2022 --spool-dir $ROOT/spool \
  --filter $ROOT/scripts/emailrelay-server-filter.sh --filter-timeout 60 \
  --pid-file $ROOT/scripts/pid-emailrelay-server.pid --verbose
#client
$(which emailrelay) --as-server --no-smtp --forward --forward-to smtp.somewhere.org:smtp \
  --spool-dir $ROOT/spool --poll 60 \
  --pid-file $ROOT/scripts/pid-emailrelay-client.pid --verbose

## -d = --log --close-stderr
##    = -l    -e
## -q = --log --no-syslog --no-daemon --dont-serve --forward --forward-to <host:port>
##    = -l    -n          -t          -x           -f        -o           <host:port>

Filter script:

#!/bin/sh

set -e -x

ROOT=/home/administrator/email
$(which python) $ROOT/scripts/couch.py email $1 # couchdb python client script, first parameter is the couchdb db
clamdscan --no-summary --move=$ROOT/infected - < $1
$(which spamassassin) "$1" > "$1.tmp"
mv "$1.tmp" "$1"

exit 0

File destroyer / obliterator

I stole the idea here and I wrote a a practical script to embellish on the theme.

#!/bin/sh

#set -e -x
set -e

if [ -f $1 ]; then
  echo "Are you sure? Hit ctrl+c to abort."
  for i in `seq 5`; do
    echo $i && sleep 1;
  done
  # mac osx / bsd: BYTES=$(stat -f%z $1)
  BYTES=$(/usr/bin/du -b $1 | sed 's/\([0-9]\+\).*/\1/')
  /bin/dd if=/dev/urandom of=$1 bs=$BYTES count=1 conv=notrunc
  echo "$1 has been written over with random bits."
  /bin/rm -f $1
  echo "$1 has been removed from the filesystem."
fi

Warning! Achtung baby! Do not use this on sensitive files you may need in the future.

This got me thinking. Imagine you work for a secretive branch of some government and you have sensitive info on your drive you wouldn't want to be caught with dead, like passwords etc. You could build a kill switch for your portable computer(s) with the sensitive files. This kill switch is attached to an artificial pacemaker which senses if you're still alive or not, by checking your heart-beat. This portable computer gets signals from the pacemaker. When it ceases to receive signals, the kill switch is engaged, after 5 minutes with no keep-alive signals from the pacemaker, the portable computer is erased completely.

The trick is to not let the signal get intercepted. Hmmmm... I should write spy-novels with a technical flavour.

Thursday, July 29, 2010

Create your own dvd

Cooking a dvd


I'm just rehashing this and augmenting it.

A dvd looks like something like this, pretty, shiny, cheap


Imagine you created your own private show and you'd like to burn it on dvd, you stored the files as avi files. Let's suppose you already use a bsd or linux os.

Install dvdauthor, growisofs, ffmpeg. Using apt-get or emerge or yum whatever.

After you have ascertained that you have enough disk space (with this command: df -ah). Convert the avi files to mpeg format, with this command:

find path-avis -name "*avi" -exec ffmpeg -i {} -aspect 4:3 -target pal-dvd path-mpgs/{}.mpg \;


Make sure you don't fill up your hard drive because that will lead to file corruption, you won't like that. If possible make sure that path-mpgs is on a non-critical partition, so it won't obliterate or damage any important files.

If you have a wide-screen show, use 16:9 instead of 4:3.

A normal dvd has about 4.4 gigabytes, ascertain first how many converted files, i.e. from avi to mpg, you can fit on it (with this command: du -ah files).

Now use dvdauthor to create the files you need to burn on the dvd later:

dvdauthor -o private-show-disc1/ -t private-show-season1-0{1,2,3}.mpg


Notice the {1,2,3} bit, that stuffs three episodes on the path private-show-disc1/, it shoves private-show-season1-01.mpg and private-show-season1-02.mpg and private-show-season1-03.mpg in the path private-show-disc1. If you can fit another one there, go crazy. Type du -ah private-show-disc1 to check if the size is below the maximum capacity of the dvd; if you go over it, then it's likely the dvd won't burn properly.

Finalize with dvdauthor -o private-show-disc1 -T to create an index, so your dvd player can jump from scene to scene or episode to episode.

Little trick

If you decide to write your own bash script to automate the whole process, you probably need to do some kind of loop to apply a set of operations on a certain pattern.

Suppose you have 13 episodes of private-show. You'd like to loop through it with:

for nr in `seq 1 13`; do something_with_file private-show-s1$nr...; done

This will probably get you into trouble, since there are 3 pairs of episodes that have filenames which start with this pattern. Namely, private-show-s1.title.mpg, private-show-s11.title.mpg, ..s1. and s12. and ..s1. and s13. This will screw up your automated system, due to nameclashes.


Update 15-08-2010: You can prefix some padding of zeroes using this:

for i in $(seq -w 1 13); do  some_command_here; done

Say you want to check the size of the files, just do it like this:

for i in $(seq -w 1 13); do du -ah *1x$i*.mpeg; done

I used to do it like this until I found out about the (-w) switch. The following is a tad bit overkill.

You can fix this problem with this command:

for i in $(seq 1 13); do echo 0$i; done | sed 's/0\(..\)/\1/g' | xargs

and use it like this:

for nr in `for i in $(seq 1 13); do echo 0$i; done | sed 's/0\(..\)/\1/g' | xargs`
do
  something_with private-show-s1$nr.*.mpg
done

For instance:

for  nr in  `for i in $(seq 1 13); do echo 0$i; done | sed 's/0\(..\)/\1/g'  | xargs`
do
  spumux -P MyShow.1x$nr*xml < MyShow.1x$nr*.mpg \
    > MyShow.s1.with-subs/MyShow.s1.1x$nr.with.subs.mpg
done

Subtitles


Suppose you have the some files which contain some reference to srt file type or something else, because people in your private show tend to mumble or speak in some jargon that is hard to follow without prior knowledge of the scene's context and themes. This is where dvdauthor - spumux comes in.

You need an template xml file for each mpeg file and it's corresponding srt file.


<subpictures>
  <stream>
    <textsub filename="subtitles.srt"
       font="LiberationSerif-Regular.ttf"
       characterset="UTF-8"
       horizontal-alignment="left"
       vertical-alignment="bottom"
       left-margin="60"
       bottom-margin="80"
       fontsize="18.0"
    />
  </stream>
</subpictures>

You'll probably need to change two things:
  1. Change subtitles.srt to the srt filename you actually have;
  2. Change the font to something you like and have, have a look at what you got in the path /usr/share/fonts/truetype; and copy that to ~/.spumux/.
  3. Save this template above as spumux.xml.
To generate the xml files you can do something like this:

for i in `ls -1 *srt`
do
  cp spumux.xml ${i%.srt}.xml
  sed -i "s/subtitles.srt/$i/" ${i%.srt}.xml
done

The suffix renaming bash trick I use here is ${i%.srt}, that removes the filename suffix (srt), which is then replaced by an xml suffix, to indicate the file format (xml).

The 2nd sed command replaces the dummy subtitles.srt to something sensible, like the actual srt file, e.g. private-show-s101.srt.

Now run this:

spumux -P private-show-s101.xml < private-show-s101.mpg > private-show-s101.with-sub-en.mpg


To automate this whole process use a bash for loop:

for i in `ls -1 *xml`
do
  spumux -P $i  < ${i%.xml}.mpg > ${i%.xml}-with-sub-en.mpg
done

This can be collapsed to a one-liner:

for i in `ls -1 *xml`; do spumux -P $i  < ${i%.xml}.mpg > ${i%.xml}-with-sub-en.mpg; done

All of these commands depend on the veracity of the assumption that the filenames are very similar. Please modify where this is not the case.

Burning to dvd


Finally, burn the prepared path (private-show-disc1) to a dvd with growisofs -Z /dev/you_know -dvd-video private-show-disc1 and watch it from you dvd player. Enjoy!

Fully-automated script


To create a fully automated script one must homogenize all the funky filenames, or not. If you choose the latter strategy make sure those mpeg files are isolated and grouped together.

The algorithm should be like this more or less:

  1. Convert files to mpeg (if necessary);
  2. Convert to mpeg with subtitles (if srts are provided).
  3. Seperate files into burnable (4.4 to 4.7gb?) units, in strict order, first episode first, last episode last, depending on the size, dvdauthor -o unit$i -t ...files...
  4. For every burnable unit, finalize, dvdauthor -o unit$i -T
  5. Burn every unit to dvd, i.e. growisofs -Z /some/device -dvd /dev/you_know -dvd-video unit$i;
  6. Send e-mail or text-message, whatever to signal change of dvd in case either failure or succes; Build sufficient timeout for changing the dvds.

Sunday, July 25, 2010

Number chunking

I learned some excellent mental math techniques from a book I bought recently by Arthur Benjamin on mental math called Secrets of Mental Math: The Mathemagician's Guide to Lightning Calculation and Amazing Math Tricks. First I was put off by the... hm, how shall I say this, over-enthusiastic way of writing. I found that this informal way of writing did not help my motivation at all when attacking hard (relative to my skill level) problems. Later, I felt more comfortable with this style of writing, when I could hack the problems much quicker, because my ego wasn't longer in the way.

I read it before going to bed, if I can't fall asleep, because solving problems help me relax or when I'm on the throne in the morning (I'm fairly regular, hooray for me).
Do not be mislead by this, I think the book is great

Fairly early on while reading the book, I realized that I have a terrible memory for numbers. I'm quite strong on algebra but I have trouble remembering and recalling numbers. Instead of struggling with something that is hardwired into this gray gooey mass, one might as well resort to tricks of the mind. The book I mentioned is great, because it circumvents the whole nature nurture debate thing with good mental algorithms virtually anyone can learn.

So my next project is to create an python app that will help me do number chunking and unchunking. Chunking is general name for a technique to translate a seemingly unrelated group of information, into something that is more compressed, so that the chunked data is easier to remember and recall. The technique Benjamin discusses in his book is called phonetic chunking, no I call it that, actually he calls it phonetic code.

The phonetic code consists of mapping decimal numbers to certain consonant sounds:

#soundsounds likephonetic alphabet
0z,szero, soundSorry, I'm too lazy to fill this in...
1t,dtear, fat,thud,door?
2nnostril?
3mmonth?
4rrow,war?
5llove,bell?
6j,ch,shjob,cheer?
7k,glock,grow?
8f,vfun,vulva?
9p,bplastic,burp,button,pub?

Other consonants and vowels may be used freely to match words to numbers, thereby chunking them together.

I realize that there is a strong anglo-saxon English linguistic bias here, but I believe this technique is applicable to any language; even clicking ones.

The author suggests a mnemonic to remember the phonetic code by chunking those together as Tony Marloshkovips (s for final 0).

I'd like to master number chunking and unchunking by using an app at regular intervals that I write in python. Just like my mental arithmetic python app.

My first design of the algorithm goes more or less like this:
  • Generate a sample set of long integers;
  • Translate each number into the correct phonetic code;
  • Write an algorithm to map phonetic codes to words from a dictionary, optimizing on certain properties that aid the human memory, (i.e. sex, death, love, any theme that evokes strong emotion);
  • Let the user put in her own chunky words, distil the relevant phonemes from the words and translate them to numbers and compare with the numbers to be learned;
  • Randomly show word sequences for user to translate to numbers;
  • Tag frequently used words for future reference;
  • Record the time to chunk and remember/recall and unchunk long number sequences;
  • Constantly measure the semantic distance of the word used by the human user.
    Some other random ideas:The CMU pronouncing dictionary
    OCZ's neural impulse actuator
    • use dictionary.reference.com as a resource.
    • (dis)prove hypothesis that short semantic distance between words in the chunked phrase lead to improved number chunkability and recall ability;
    • (dis)prove hypothesis that words (by themselves or together) in the chunked phrase that give strongest emotional response lead to improved recall ability or chunkability (use ocz nia for this experiment);
    • Omfg, I just realized I can also used nia for my mental arithmetics training tool.

    Apparently, algorithms exist to index words phonetically, the soundex algorithm by Odel and Russel (1918) and also works on the premise that words are converted to numbers. I just realized that English may not be the best language to do this, since it has 24 letters to indicate 44 distinct phonemes, whereas for instance Russian has a better mapping of phonemes to letters. This is my gut feeling about Russian, I can read cyrillic but I don't speak the language fluently enough to give more examples. I guess someone in academia has already generated the samples of different languages to compare the phoneme to letter correspondence, in n-grams or whatnot.

    I think google has some form of prescience or something. While I was researching and reading up on soundex etc. I suddenly came accross the CMU pronouncing dictionary. This will probably prove to be handy for this application I'm thinking about.

    Thursday, July 22, 2010

    Syntaxhighlighter on blogger blog

    I spent the best part of yesterday trying to get syntaxhighlighter by Alex Gorbatchev to work on this blog via external resources, courtesy of github.com and bitbucket.org.

    I've succeeded as you see (is it broken?):

    def pyfn(a):
      print "Hello Turd!", a
    

    Anyhoo, I tested it and it works (for now).
    So this is how I did it:
    Bitbucket.org commit hash code
    1. I located Alex's code repository, later I'll link to the code from there;
    2. I realised that every commit on the repository has a different hash which is used in all the urls I'll be linking to, e.g. http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64... if you want to use the latest committed code you have to find the latest hash for yourself and replace the hash with the latest in the linking code; 
    3. You'll also need the actual linking code; read the official installation tutorial to understand what you're actually linking to:
    <link href="http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/styles/shCore.css" rel="stylesheet" type="text/css"></link>
    
    <link href="http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/styles/shCoreDefault.css" rel="stylesheet" type="text/css"></link>
    
    <link href="http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/styles/shThemeDefault.css" rel="stylesheet" type="text/css"></link>
    
    <script src="http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/XRegExp.js" type="text/javascript" />
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shCore.js' type='text/javascript'/>
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shBrushBash.js' type='text/javascript'/>
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shBrushPython.js' type='text/javascript'/>
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shBrushErlang.js' type='text/javascript'/>
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shBrushJScript.js' type='text/javascript'/>
    
    <!-- added bit for jQuery -->
    <script src='http://github.com/jquery/jquery/raw/master/speed/jquery-basis.js' type='text/javascript' />
    <script type='text/javascript'>
    $(document).ready(function () {SyntaxHighlighter.all();});
    </script>
    

    Update 23-07-2010: For some reason the latest blogger template editor does not want you to combine a <script src=...> with your own code nested in it. So don't feel tempted to nest your own code with a <script> pull.

    Blogger link to manual template edit
    The final step is to paste this in your <head>-tag. You have to modify your blogger template to do this.
    Edit template

    I used jQuery to hook "SyntaxHighlighter.all()" on to the <body>-tag's onload event. This might be a tad overkill for one function invocation, but I know for certain that I'll use jQuery again on this blog.

    Wednesday, July 21, 2010

    Demonstration python decorator

    I feel that python decorators allow programmers in python to code in a very natural way. Since a ton of blogs out there already have documented what python decorators precisely are or aren't, so I won't go into too much detail here. I probably won't do a good job explaining what they are.

    Ah what the heck, here goes my explanation:

    Babushka (grandma) matryoshka dolls
    Imagine that a decorated function is like a matryoshka doll, a matryoshka doll can be enclosed around with another doll. Python experts might not agree with this analogy, because when a function is decorated it can be accessed by the function outside it, so it does stand dormant in a dark place like a matryoshka doll.

    See the following function declarations:

    def wrap(f)
      # Add before behaviour
      print "start wrap"
      # Do something with f
      # Add after behaviour
      print "end wrap"
      return f # Pass f to the next label or babushka/function
    
    @wrap
    def fun():
      pass # do something interesting...
    

    The previous rows are equivalent to
    fun = wrap(fun)

    The wrapping function (wrap) must return a final function to relabel the newly decorated function with the original unwrapped label, which is fun in our example.

    You could also decorate more than once:
    @huge
    @medium
    @small
    def core()
      pass
    

    Which is equivalent to the following:
    core = huge(medium(small(core)))
    In this example, one must also keep passing the decorated function to the next decorator with a return statement.

    This year I decided to improve my mental arithmetic skills, so I wrote a python application to train on. I was also quite curious as to what the whole python decorator thing was about. I combined my curiosity for python decorators and the desire for a mental arithmetic trainer. The following script is the (partial) end result.

    def timeit(fun):
      from time import time
      def timed(*args):
        ts = time()
        mistakes, op_symbol = fun(*args)
        te = time()
        time_elapsed = te - ts
        print "Time elapsed:", time_elapsed
        return time_elapsed, mistakes, op_symbol
      return timed
    
    class Query(object):
      def __init__(self, op_symbol):
        self.op_symbol = op_symbol
    
      def __call__(self, f):
        # NOTE: you can count the function arguments later to try on a n-ary functions
        def wrapped_binary_f(*args):
          a,b = tuple(args)
          #print "%s %s %s = ?" % (a, self.op_symbol, b) # infix notation
          print a, self.op_symbol, b, " = ?"
          correct_answer = f(a,b)
          user_answer = None
          mistakes = 0
          while not user_answer or user_answer != correct_answer:
            user_answer = None
            try:
              user_answer = input("answer: ")
              if user_answer != correct_answer:
                mistakes+=1
                print "Please try again... %s" % str(mistakes)
              else:
                print "Correct."
                break
            except SyntaxError:
              mistakes+=1
              print "Please try again... %s" % str(mistakes)
            except NameError:
              mistakes+=1
              print "Please try again... %s" % str(mistakes)
          return mistakes, self.op_symbol
        return wrapped_binary_f
        
    @timeit
    @Query('*')
    def multiply(a, b):
      return a * b
      
    @timeit
    @Query('-')
    def subtract(a, b):
      return a - b
      
    @timeit
    @Query('+')
    def add(a, b):
      return a + b

    This example demonstrates three techniques, namely:
    1. Passing arguments to a decorator function (Query class's __init__ function);
    2. Using multiple decorator functions on multiple functions (timeit, Query);
    3. Manipulating arguments passed to the decorator function and the called functions (multiply, subtract, add) and mixing them up
    Notice the nested functions (wrapped_binary_f, timed), this is the main technique that is used to manipulate the arguments of a passed function. The outer function (__call__, timeit) are used to feed the to-be-wrapped function (multiply etc.) and the nested function serves to get the arguments passed to the to-be-wrapped function.

    One can also add arguments to the decorator function by way of a class declaration. First the __init__ function is called to create the object of the class arguments passed, the arguments are stored in the object, then the object is called via the __call__  function. The __call__ function can access the arguments from the members of the object.

    I have included the complete application here, try it out, improve it, enjoy!

    #!/usr/bin/env python
    # -*- coding: iso-8859-1 -*-
    '''
    Command-line interactive script to train mental arithmetic,
    also a demonstration of python decorators.
    Created on May 22, 2010 and refactored several times after...
    '''
    
    def timeit(fun):
      from time import time
      def timed(*args):
        ts = time()
        mistakes, op_symbol = fun(*args)
        te = time()
        time_elapsed = te - ts
        print "Time elapsed:", time_elapsed
        return time_elapsed, mistakes, op_symbol
      return timed
    
    class Query(object):
      def __init__(self, op_symbol):
        self.op_symbol = op_symbol
    
      def __call__(self, f):
        # NOTE: you can count the function arguments later to try on a n-ary function
        def wrapped_binary_f(*args):
          a,b = tuple(args)
          #print "%s %s %s = ?" % (a, self.op_symbol, b) # infix notation
          print a, self.op_symbol, b, " = ?"
          correct_answer = f(a,b)
          user_answer = None
          mistakes = 0
          while not user_answer or user_answer != correct_answer:
            user_answer = None
            try:
              user_answer = input("answer: ")
              if user_answer != correct_answer:
                mistakes+=1
                print "Please try again... %s" % str(mistakes)
              else:
                print "Correct."
                break
            except SyntaxError:
              mistakes+=1
              print "Please try again... %s" % str(mistakes)
            except NameError:
              mistakes+=1
              print "Please try again... %s" % str(mistakes)
          return mistakes, self.op_symbol
        return wrapped_binary_f
        
    @timeit
    @Query('*')
    def multiply(a, b):
      return a * b
      
    @timeit
    @Query('-')
    def subtract(a, b):
      return a - b
      
    @timeit
    @Query('+')
    def add(a, b):
      return a + b
    
    @timeit
    @Query('/')
    def divide(a, b):
      return float(a) / float(b)
    #  from decimal import Decimal, getcontext
    #  from fractions import Fraction # check this shit out, useful if user enters fraction and computer calculates float or Decimal... then check if the same
    #  getcontext().prec=2
    #  return Decimal(a) / Decimal(b)
    
    def generate_samples_naive(maxdecimals, nsamples):
      from random import choice
      return zip([ choice(xrange(10**maxdecimals)) for _i in xrange(nsamples) ],
                 [ choice(xrange(10**maxdecimals)) for _j in xrange(nsamples) ])
    
    
    def generate_samples_positive(maxdecimals, nsamples):
      """
      The lambda here swaps the values in case subtraction results are negative
      """
      from random import choice
      return map (lambda (x,y): (x,y) if x>=y else (y,x), \
                  [ (choice(xrange(10**maxdecimals)), choice(xrange(10**maxdecimals))) \
                   for _i in xrange(nsamples) ])
    
    def get_stats(li):
      """
      Tuple of total time spent solving the problems
      and total number of mistakes
      """
      return sum([ t for t,_,_ in li ]), \
        sum([m for _,m,_ in li])
    
    def get_detailed_stats(li):
      """
      Total number of mistakes per operator type,
      total number of time spent per operator type
      """
      d = {}
      for op_symbol in ['*','-','+','/']:
        d[op_symbol] = (sum([ t for t,_,o in li if o == op_symbol ]),
          sum([ m for _,m,o in li if o == op_symbol ]))
      return d
    
    def show_stats(results):
      (total_time, total_mistakes) = get_stats(results)
      print "Total solved:", len(results)
      print "Total time elapsed:", total_time
      print "Total mistakes made:", total_mistakes
    
      print "Detailed results per operator (time, mistakes):" #, get_detailed_stats(results)
      d = get_detailed_stats(results)
      for k in d.keys():
        (time, mistakes) = d[k]
        print k, "%.2f" % (float(time)/total_time), "%.2f" % (float(mistakes)/total_mistakes)
      print "Done!"
    
    def main(av):
      from optparse import OptionParser
      usage = "usage: %prog [options] arg"
      parser = OptionParser(usage)
    
      # parser.add_option...
    #  parser.add_option("-m", "--max-tries", dest='maxtries', type='int', default=3,
    #                    help='Number of tries before going to the next example')
      parser.add_option("-n", "--nr-samples", dest="nsamples", type="int", default=10,
        help="Per arithmetic operation train on this number of SAMPLES", metavar="SAMPLES")
    #  parser.add_option("--mind", "--min-decimals", dest="maxdecimals", type="int", default=0,
    #    help="Define the lowest number to train expressed as the number of DECIMALS", metavar="DECIMALS")
      parser.add_option("--maxd", "--max-decimals", dest="maxdecimals", type="int", default=2,
        help="Define the largest number to train expressed as the number of DECIMALS", metavar="DECIMALS")
    #  parser.add_option("-s", "--no-stats", dest="sendstats", action="store_false", default=True,
    #    help="Don't send any statistical data to servers, over the internet.")
    #  parser.add_option("-a", "--anonymous", dest="anonymous", action="store_true", default=False,
    #    help="Don't send credentials or any data revealing user's identity over the internet")
      parser.add_option("--nd", "--no-division", dest="train_division", action="store_false", default=False,
        help="Don't train user on division")
      parser.add_option("--na", "--no-addition", dest="train_addition", action="store_false", default=True,
        help="Don't train user on division")
      parser.add_option("--ns", "--no-subtraction", dest="train_subtraction", action="store_false", default=True,
        help="Don't train user on subtraction")
      parser.add_option("--nm", "--no-multiplication", dest="train_multiplication", action="store_false", default=True,
        help="Don't train user on multiplication")
      parser.add_option("--nh", "--no-history", dest="store_history", action="store_false", default=True,
        help="Don't store user history, mind you, the same training samples might be repeated very often")
      parser.add_option("-o", "--ordered-training", dest="ordered", action="store_false", default=True,
        help="Train in this order: addition, subtraction, multiplication, division")
      parser.add_option("-e", "--e-mail-addr", dest="email_addr", type="string", default="anon@anon.org",
        help="This script will modify itself to reflect the e-mail ADDRESS of the user in question", metavar="ADDRESS")
      parser.add_option("-p","--positive-only", dest="positive_only", action="store_true", default=False,
                        help="The result from the operation on a pair of samples will only be POSITIVE.", metavar="POSITIVE")
    
    
      (options, _args) = parser.parse_args(av)
    
      if options.positive_only:
        samples = generate_samples_positive(options.maxdecimals, options.nsamples)
      else:
        samples = generate_samples_naive(options.maxdecimals, options.nsamples)
    
      results = []
      if options.ordered:
        if options.train_subtraction:
          for (a,b) in samples:
            results.append(subtract(a,b))
    
        if options.train_addition:
          for (a,b) in samples:
            results.append(add(a,b))
    
        if options.train_multiplication:
          for (a,b) in samples:
            results.append(multiply(a,b))
    
        if options.train_division:
          for (a,b) in samples:
            results.append(divide(a,b))
    
      else:
        op_options = []
        if options.train_addition:
          op_options.append(add)
        if options.train_multiplication:
          op_options.append(multiply)
        if options.train_subtraction:
          op_options.append(subtract)
        if options.train_divison:
          op_options.append(divide)
        fns = op_options*options.nsamples
        import random
        random.shuffle(fns)
        for (nr,(a,b)) in enumerate(3*samples): # TODO - come division, change to four
          results.append(fns[nr](a,b))
    
      show_stats(results)
    
    
    if __name__ == '__main__':
      from sys import argv as av
      main(av)
    

    Update 30-07-2010: I just realized that you could grab this code and put a gui on top quite easily, by replacing the Query class with something with a gui. Done!

    Tuesday, July 20, 2010

    Pstools part 2: preparing a client

    After discovering pstools, I wanted to write a batch file to make clients more compliant to pstools. I got the grand idea of putting the batch file in the netlogon script of all the users; so all the clients would magically get compliant once a user logs on that specific machine. But alas, this plan fails, because you need system rights or admin rights to run the commands e.g. reg.exe and net.exe, to bore a hole in the local firewall (also, the group policies must be setup accordingly so that admin users have these rights).

    To make a computer compliant you have to either image/ghost over the to-be compliant machine with an already compliant image or run this batch file and pray that it works. Or login with a superuser account on the target client machine and run this from netlogon (netlogon is to expedite things, but this is not compulsory), as I described earlier.

    This command also works on most windows (professional edition) clients:
    runas /user:administrator script.bat
    

    or

    runas /user:administrator@your-ad-network.tld script.bat
    


    @echo off
    REM Make pstools and rdp work on all client machines and add local superuser
    echo "-- Make client compliant for pstools --"
    REM pstools - make compliant from registry
    reg add "hklm\system\currentcontrolset\control" /f /v SCMApiConnectionParam /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\loggedon" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psexec" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psfile" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psgetsid" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psinfo" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\pskill" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\pslist" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psloglist" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\pspasswd" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psservice" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psshutdown" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\pssuspend" /f /v EulaAccepted /t REG_DWORD /d 1
    netsh firewall set portopening TCP 445 ENABLE
    REM md %windir% - give -c a location to push
    REM net share admin=%windir%
    EXIT
    

    I also wrote the following script to create a default rdp (remote desktop) local admin user account to see from rdp what pstools is doing to the (remote) local user; Caveat! Don't let the password get intercepted:

    @echo off
    echo "-- Make client compliant for rdp (remote desktop) --"
    REM "remoteadmin" user (XP machines, untested on vista or 7)
    reg add "hklm\system\currentcontrolset\control\terminal server" /f /v fDenyTSConnections /t REG_DWORD /d 0
    netsh firewall set service remoteadmin enable
    netsh firewall set service remotedesktop enable
    net user remoteadmin Supersecretpassword123 /Add
    net localgroup administrators remoteadmin /Add
    reg add "hklm\software\microsoft\windows nt\currentversion\winlogon\specialaccounts\userlist" /f /v remoteadmin /t REG_DWORD /d 0
    EXIT
    

    Pstools part 3: pushing silent updates

    Suppose that you use clamwin and spybot s&d because your boss is cheap and you sure as hell ain't gonna pay for it yourself, but you have to provide some type of baseline security for your AD (active directory) networked windows clients; and you're too lazy to manually install over 50 machines, while people are using the aforementioned machines.

    Pstools and silent install to the rescue!

    Type the following from a superuser account in the AD network in the command line interface (dos):

    psexec @hosts.txt -s -n 3 -c -v "clamwin-x.y-setup.exe" /sp- /silent /norestart
    psexec @hosts.txt -s -n 3 -c -v "spybotsdxyz.exe" /sp- /silent /suppressmsgboxes /norestart


    Legend:

    parameterexplanation
    @hosts.txta file containing ip-addresses or domain client computer names seperated by endlines
    -srun with system rights
    -n 3try maximum of three times, before concluding that the client is not reachable
    -c -vPush the following file (e.g. clamwin-x.y-setup.exe) to client; -v implies push only if it's newer (spare bandwidth, non-clobber).

    Pstools part 1

    I wrote a wee nice batch file to make a windows-based client computer in an AD (active directory) environment in case pstools, namely psexec, is not cooperative.

    For people in the know, pstools is a lot like ssh for unix (linux, bsd). I understand that there's a mingw?/cygwin ssh daemon out there, but I haven't used it yet.

    Imagine that you're the system administrator and your colleague, let's call her Anna, (who is not in the know) is complaining that she does not have e.g. internet explorer on her computer.

    Normally a hard-working competent administrator would have to walk all over there and take over the account find the icon for them or directly find the binary for the supposedly missing application and double-click it there, physically, manually.

    Say you would like to check the validity of her statement from your current computer without walking all the way over there.

    This is where pstools comes in. Imagine that you can find anyone's ip-address in the lan (local area network), because you assigned names to computers used by your colleagues and put that in the dns, mind you in AD this is registered by the dhcp server.

    You fire up psexec and you type:


    psexec \\roomAcompA -i -U anna -P S3cr3t "C:\Program Files\Internet Explorer\iexplore.exe"


    \\roomAcompA stands for the computer A in room A, suppose that's the computer where Anna works, the parameter -i stands for interactive, which means the gui (if available) will be visible to Anna.

    You call her up, and ask her if anything weird has happened. She'll tell you that internet explorer has magically popped out of nowhere.

    You can also take over her machine by typing:


    psexec \\roomAcompA -s -U administrator -P S00p@S3kr1t "cmd"


    The parameter -s stands for "system", which stands for system rights granted to the current user. This is comparable to local root permissions for unix systems.

    If you're currently running pstools as a superuser on the AD network, the -U and -P parameters could be omitted.

    You could also pass local commands with system rights like this:


    psexec \\roomAcompA -s "cmd" /C "taskkill /pid 0"


    This would unabashedly kill a certain process with pid (process id) 0 with system rights. This is just an example what you could do with psexec.

    To kill processes, pstools has the pslist and pskill.

    Tuesday, July 13, 2010

    Silly sikuli suggestions

    If you haven't heard of sikuli yet then I suggest you pop over to their website or their blog. Let me quickly summarize what it does: a programming language based on jython on the jvm to control gui elements from a desktop environment.

    What a great project! It's one of those things you probably thought of before, then dismissed quickly.

    This is a nice demonstration of what you could do with sikuli; if you follow the link you'd find bejeweled being played by a programmed script.

    I tried coding in sikuli in win7 (windows 7) to run maintenance programs on it, like ccleaner, spybot s&d and clamwin. These things, for instance, clearing out useless files, immunizing the windows registry, scanning for virii, should be routinely done, but nobody really bothers to do them. Sikuli can relieve the burden of clicking from window to window.

    The following list are some ideas I got right after programming a few scripts.:
    • For some automating system administration tasks, you'll probably need password inputs, but protected, so the user can't read the passwords. Let these get scheduled by cron or windows schedule service.
    • Metauserinterface to query user for automated tasks for instance for creating local intranet accounts or accounts for twitter, gmail, etc. One could also provide a list to create accounts in bulk, just do this in sikuli >> file_list = open("names.txt"); # then loop through the names. Edit 4-07-2010: check out the functions popup and input.
    • Cut-and-paste images from image and text editors by way of the clipboard, none of that import stuff, this is a basic feature, although this feature requires some native OS hacking.
    • Port sikuli to pypy for faster image matching calculations; or just improve the algorithms; although I can imagine that this detracts from the goal of the project, since important parts of sikuli must be recoded from the ground-up.
    • Infinite undoability. Sikuli currently has no undo functionality.
    • Plugins, but without ending up like another eclipse. The whole sikuli editor should actually just get plugged into eclipse, but still have a interpreter counterpart so scripts could be run independently of eclipse, scripts that are ready for production could get run with some type of interpreter from the command line. There will be a seperation of concerns, editor, compiler and interpreter.
    • OCR (optical character recognition) plugin, to manipulate gui character output and feed it back to the system.
    Fun stuff to demonstrate with sikuli:
    • Show true introspection in sikuli/jython/jvm by letting sikuli self modify itself to do some abstract task.
    • Show how security flaws can be exploited with sikuli, also doing nasty stuff by using obfuscation.

    I'll try to find the time to implement some of my ideas. For instance creating a plugin for eclipse would be a good idea, since eclipse already has brilliant plugins like pydev which fully supports jython. Although the big-ass images might phrack up the layout of eclipse.

    Sunday, July 11, 2010

    Bridged openvpn on ubuntu

    At my work we use ubuntu "lucid lynx" as the operating system for openvpn. Our "lynx" is  a vmguest running on linx/qemu/kvm. We have linux and windows clients. Actually, all possible clients which support openvpn client mode.

    First I setup our internet-facing shorewall firewall to allow 1194 and port-forward it to our internal openvpn (vm)hosts, using shorewall-rules. We use shorewall version 4.4.6.

    #ACTION SOURCE DEST PROTO DEST_PORT ...
    DNAT wan lan:192.168.11.3:1194 udp 1194 #network alpha
    DNAT wan lan:192.168.22.3:1194 udp 1294 #network beta

    We have a multi-isp setup, that is denoted by the wan (wide area network) under SOURCE. We also have two lans (local area networks) which use this shorewall machine as the gateway to the internet, using two different subnets to communicate within each lan. So I wrote two rules to port-forward from the firewall to the correct openvpn server.

    This is called dNATing (destination network address translation + ing); in practical terms, an openvpn client knocks on a specific port on the firewall, giving the illusion that this whole business is between the firewall and the client, but in fact, the firewall is only passing messages to and fro between the openvpn server inside a lan and the remote client somewhere on the internet, regardless of whether the client itself is behind a firewall or directly on the internet. It just works.

    As I have mentioned earlier we use openvpn, the specific version is 2.1.0.
    I looked far and wide on the internet for openvpn bridging and only found this tutorial to be the easiest to setup. The weird thing is, that the tutorial supposed to be used with a N900 mobile phone from nokia.

    1) Install the software. See the tutorial first.
    2) I also setup the bridge first. As recommended by the tutorial.


    auto lo br0
    iface lo inet loopback

    iface br0 inet static
    address 192.168.11.3
    network 192.168.11.0
    broadcast 192.168.11.255
    netmask 255.255.255.0
    gateway 192.168.11.254
    bridge_ports eth0
    bridge_fd 9
    bridge_hello 2
    bridge_maxage 12
    bridge_stp off

    iface eth0 inet manual
    up ifconfig $IFACE 0.0.0.0 up
    up ip link set $IFACE promisc on
    down ip link set $IFACE promisc off
    down ifconfig $IFACE down
    # do this in /etc/network/interfaces


    3) Next you need to create the keys and certificates. This is the least difficult part.


    sudo cp -a /etc/openvpn/easy-rsa ~/

    chown -R user ~/easy-rsa
    cd ~/easy-rsa
    sudo chmod g+w . ; source ./vars ## execute your new vars file
    ./clean-all ## setup the easy-rsa directory (deletes all keys)
    ./build-dh ## takes a while consider backgrounding
    ./pkitool –initca ## creates ca cert and key
    ./pkitool –server server ## creates a server cert and key
    cd keys
    openvpn –genkey –secret ta.key ## Build a TLS key
    sudo cp {server,ca}.crt {server,ta}.key dh1024.pem /etc/openvpn


    4) Enable ip forwarding in /etc/sysctl.conf

    5) Create the up.sh and down.sh shell scripts to create and kill the tap devices when a openvpn client logs on or off. Mind you that more parameters are passed to the scripts by the openvpn server than what the tutorial lets on. I found this out by turning on the debugging facilities of the openvpn script in /etc/init.d/openvpn, like this:


    #!/bin/sh
    set -e -x
    ..



    Fix the permissions of the up.sh and down.sh files with this:

    user@ovpnubuntu~/ sudo chmod go+rwx /etc/openvpn/{up,down}.sh


    6) Modify the server.conf to reflect your lan's settings; for instance change the ip address of the "local" parameter. In our case it's 192.168.11.3. Port 1194.

    local 192.168.11.3 1194


    Openvpn has a built-in dhcp server, that distributes local ip addresses to openvpn clients. Make sure this is provisioned in your main dhcp server, so there is no chance of ip address collisions.

    # openvpn server configuration file - server.conf
    mode server
    tls-server

    local 192.168.11.3 1194
    proto udp

    dev tap0
    up "/etc/openvpn/up.sh br0"
    down "/etc/openvpn/down.sh br0"

    persist-key
    persist-tun

    #certs
    ca ca.crt
    cert server.crt
    key server.key
    dh dh1024.pem
    tls-auth ta.key 0

    #cipher and compression
    cipher AES-256-CBC
    comp-lzo

    #DHCP
    ifconfig-pool-persist ipp.txt
    ## settings for openvpn dhcp server
    server-bridge 192.168.11.254 255.255.255.0 192.168.11.216 192.168.11.224
    push "dhcp-option DNS 192.168.11.5" # DNS
    push "dhcp-option WINS 192.168.11.5" # windows-flavor DNS
    push "dhcp-option DOMAIN ad.local" # for windows active directory objects or samba4!
    max-clients 8

    #log security
    user nobody
    group nogroup
    keepalive 10 60
    status openvpn-status.log
    log-append server.log
    verb 5


    7) Configure /etc/defaults/openvpn; if you need to change the parameters passed to the openvpn server, modify OPTARGS:


    OPTARGS="--fast-io --verb 5" # fast io for fast connections; verb for verbose debug output


    After that you'll be able to restart the openvpn server automagically when the server goes down and has to reboot for some reason.

    8) Create some keys for openvpn clients; as described in the tutorial:cd ~/easy-rsa ; source ./vars ; ./pkitool {user1,user2,user3,user4,joe,satch,bob,marilyn}

    8a) If for some reason you want to prevent people from getting to your keys and/or certificates in cleartext; you can use openssl to encrypt your certificates and keys, while sending them across some unencrypted medium.

    In my case, I wanted to send myself the keys etc. via e-mail, but I'm ultra-paranoid about big-brother sticking its nose in my beeswax. So I tarred and compressed my files first.

    tar cf ~/openvpn-keys-certs.tar ~/easy-rsa ; bzip -z9 ~/openvpn-keys-certs.tar


    Then I proceeded with encrypting the bzip2 file:

    openssl aes-256-cbc -in openvpn-keys-certs.tar.bz2 -out openvpn-keys-certs.tar.bz2.enc


    This is a form of symmetric encryption, so that means if someone finds out your passphrase you recently wrote down on a piece of paper, anyone who finds that paper can decode your message. Use hard passwords with a handful of these "!@#$%^&*();<>' in different cases with numbers in them.
    Theoretically everything is brute-forceable, if one ignores the cost to crack the code. Unless there is some invariant loop-hole in the encryption algorithm to crack the dictionary of the compression algorithm.

    Decrypt with this command:

    openssl enc -d -aes-256-cbc -in openvpn-keys-certs.tar.bz2.enc -out openvpn-keys-certs.tar.bz2 ; tar xjf openvpn-keys-certs.tar.bz2


    8b) Use asymmetric encryption like gpg. Send the encrypted keys and certificates to the user using her public key. Let her worry about the private key and passphrase. There is gpg4win for windows users.

    9) Write a template client configuration file.

    # template client configuration file - client.ovpn.txt
    # Specify that this is a client
    client

    # Bridge device setting
    dev tap

    # Host name and port for the server (default port is 1194)
    # note: replace with the correct values your server set up
    remote firewall_ip_addr 1194

    # Client does not need to bind to a specific local port
    nobind

    # Keep trying to resolve the host name of OpenVPN server.
    ## The windows GUI seems to dislike the following rule.
    ##You may need to comment it out.
    resolv-retry infinite

    # Preserve state across restarts
    persist-key
    persist-tun

    # SSL/TLS parameters files created previously
    ca ca.crt
    cert replace.crt
    key replace.key

    # Since we specified the tls-auth for server, we need it for the client
    # note: 0 = server, 1 = client
    tls-auth ta.key 1

    # Specify same cipher as server
    cipher AES-256-CBC

    # Use compression
    comp-lzo

    # Log verbosity (to help if there are problems)
    verb 3


    Each client requires three 4 files in total in our setup to connect; a) The common shared (ca.crt) certificate with the server; b) The client certificate; c) The client key; d) A configuration file.

    You need to activate the template by replacing some parameters in the template with some real values. For instance there is no "replace.crt" nor "replace.key". This must be changed to some file name intended to be used by some user. For instance "bob".

    So, given that you already created the key and certificate for "bob", you can create a configuration file using the template like this:

    sed 's/replace/bob/g' client.ovpn.txt > bob-client.ovpn ; sed -i 's/firewall_ip_addr/real_firewall_ip_addr/' bob-client.ovpn

    Working at home with ssh port-forwarding, qemu/kvm and vnc

    Sometimes I work at home because I don't want to get constantly bothered by people barging into my room with trivial requests etc. Every time someone barges into my room, I find that regaining my focus takes too much effort.

    To actually get any work done, I made it possible for me to log into the system by punching a hole in the firewall for ssh traffic. I increased the security by disallowing password authentication and forcing asymmetric encryption authentication, i.e. 2048-bit rsa public key authentication, where I am the sole owner of the private key. Above all, I also limited the number of (internet) locations where this private key can be used, by putting this in the public key (authorized_keys):

    from="*.this.is.my.isp.tld" ssh-rsa MEH...

    At my work we work a lot with ssh and virtual machines. We use linux/qemu/kvm. Kvm can be initialized with the "-vnc :N" parameter, where N is a positive integer. This creates a vnc session on the vmhost. For instance, if one were to use krdc and N is 1, the invocation command would look like something like this:

    user@vmhost~$ krdc vnc://vmhost:5901

    The rule is to add N to 5900. That's how you find the port. You can also run a vnc client on the local network to view the vnc stream. This is very convenient because you're not stuck with one port for multiple vmguests.

    user@somewhere~$ krdc vnc://vmhost:5901

    To make things easier for yourself when working at home, you could port-forward this very remote vnc port to your own localhost with this invocation:

    user@athome~$ ssh -f -N -L 6001:vmhost_internal_ip_addr:5901 firewall_ip_addr
    user@athome~$ krdc vnc://localhost:6001

    The parameters "-f -N" forces this ssh client session in the background. The relevant part is the "-L" parameter. The L probably stands for local, there is also an R counterpart that port-forwards a local port to a remote port.

    This is a form of tunneling where a remote port in some remote (local) area network,  could be plugged in directly into your localhost as if this service is running on your local machine.

    This trick also applies to other tcp-based protocols, for instance the standard rdp (remote desktop protocol) port. For instance you have some windows machines you would like to log on to, as if they are running on you local machine, you do the following:

    user@athome~$ ssh -f -N -L 4489:windows_machine_ip_addr:3389 firewall_ip_addr
    user@athome~$ krdc rdp://localhost:4489


    Voila! Instant connection to a worksite windows machine. This trick also applies to windows machines using putty.