Pages

Thursday, July 29, 2010

Create your own dvd

Cooking a dvd


I'm just rehashing this and augmenting it.

A dvd looks like something like this, pretty, shiny, cheap


Imagine you created your own private show and you'd like to burn it on dvd, you stored the files as avi files. Let's suppose you already use a bsd or linux os.

Install dvdauthor, growisofs, ffmpeg. Using apt-get or emerge or yum whatever.

After you have ascertained that you have enough disk space (with this command: df -ah). Convert the avi files to mpeg format, with this command:

find path-avis -name "*avi" -exec ffmpeg -i {} -aspect 4:3 -target pal-dvd path-mpgs/{}.mpg \;


Make sure you don't fill up your hard drive because that will lead to file corruption, you won't like that. If possible make sure that path-mpgs is on a non-critical partition, so it won't obliterate or damage any important files.

If you have a wide-screen show, use 16:9 instead of 4:3.

A normal dvd has about 4.4 gigabytes, ascertain first how many converted files, i.e. from avi to mpg, you can fit on it (with this command: du -ah files).

Now use dvdauthor to create the files you need to burn on the dvd later:

dvdauthor -o private-show-disc1/ -t private-show-season1-0{1,2,3}.mpg


Notice the {1,2,3} bit, that stuffs three episodes on the path private-show-disc1/, it shoves private-show-season1-01.mpg and private-show-season1-02.mpg and private-show-season1-03.mpg in the path private-show-disc1. If you can fit another one there, go crazy. Type du -ah private-show-disc1 to check if the size is below the maximum capacity of the dvd; if you go over it, then it's likely the dvd won't burn properly.

Finalize with dvdauthor -o private-show-disc1 -T to create an index, so your dvd player can jump from scene to scene or episode to episode.

Little trick

If you decide to write your own bash script to automate the whole process, you probably need to do some kind of loop to apply a set of operations on a certain pattern.

Suppose you have 13 episodes of private-show. You'd like to loop through it with:

for nr in `seq 1 13`; do something_with_file private-show-s1$nr...; done

This will probably get you into trouble, since there are 3 pairs of episodes that have filenames which start with this pattern. Namely, private-show-s1.title.mpg, private-show-s11.title.mpg, ..s1. and s12. and ..s1. and s13. This will screw up your automated system, due to nameclashes.


Update 15-08-2010: You can prefix some padding of zeroes using this:

for i in $(seq -w 1 13); do  some_command_here; done

Say you want to check the size of the files, just do it like this:

for i in $(seq -w 1 13); do du -ah *1x$i*.mpeg; done

I used to do it like this until I found out about the (-w) switch. The following is a tad bit overkill.

You can fix this problem with this command:

for i in $(seq 1 13); do echo 0$i; done | sed 's/0\(..\)/\1/g' | xargs

and use it like this:

for nr in `for i in $(seq 1 13); do echo 0$i; done | sed 's/0\(..\)/\1/g' | xargs`
do
  something_with private-show-s1$nr.*.mpg
done

For instance:

for  nr in  `for i in $(seq 1 13); do echo 0$i; done | sed 's/0\(..\)/\1/g'  | xargs`
do
  spumux -P MyShow.1x$nr*xml < MyShow.1x$nr*.mpg \
    > MyShow.s1.with-subs/MyShow.s1.1x$nr.with.subs.mpg
done

Subtitles


Suppose you have the some files which contain some reference to srt file type or something else, because people in your private show tend to mumble or speak in some jargon that is hard to follow without prior knowledge of the scene's context and themes. This is where dvdauthor - spumux comes in.

You need an template xml file for each mpeg file and it's corresponding srt file.


<subpictures>
  <stream>
    <textsub filename="subtitles.srt"
       font="LiberationSerif-Regular.ttf"
       characterset="UTF-8"
       horizontal-alignment="left"
       vertical-alignment="bottom"
       left-margin="60"
       bottom-margin="80"
       fontsize="18.0"
    />
  </stream>
</subpictures>

You'll probably need to change two things:
  1. Change subtitles.srt to the srt filename you actually have;
  2. Change the font to something you like and have, have a look at what you got in the path /usr/share/fonts/truetype; and copy that to ~/.spumux/.
  3. Save this template above as spumux.xml.
To generate the xml files you can do something like this:

for i in `ls -1 *srt`
do
  cp spumux.xml ${i%.srt}.xml
  sed -i "s/subtitles.srt/$i/" ${i%.srt}.xml
done

The suffix renaming bash trick I use here is ${i%.srt}, that removes the filename suffix (srt), which is then replaced by an xml suffix, to indicate the file format (xml).

The 2nd sed command replaces the dummy subtitles.srt to something sensible, like the actual srt file, e.g. private-show-s101.srt.

Now run this:

spumux -P private-show-s101.xml < private-show-s101.mpg > private-show-s101.with-sub-en.mpg


To automate this whole process use a bash for loop:

for i in `ls -1 *xml`
do
  spumux -P $i  < ${i%.xml}.mpg > ${i%.xml}-with-sub-en.mpg
done

This can be collapsed to a one-liner:

for i in `ls -1 *xml`; do spumux -P $i  < ${i%.xml}.mpg > ${i%.xml}-with-sub-en.mpg; done

All of these commands depend on the veracity of the assumption that the filenames are very similar. Please modify where this is not the case.

Burning to dvd


Finally, burn the prepared path (private-show-disc1) to a dvd with growisofs -Z /dev/you_know -dvd-video private-show-disc1 and watch it from you dvd player. Enjoy!

Fully-automated script


To create a fully automated script one must homogenize all the funky filenames, or not. If you choose the latter strategy make sure those mpeg files are isolated and grouped together.

The algorithm should be like this more or less:

  1. Convert files to mpeg (if necessary);
  2. Convert to mpeg with subtitles (if srts are provided).
  3. Seperate files into burnable (4.4 to 4.7gb?) units, in strict order, first episode first, last episode last, depending on the size, dvdauthor -o unit$i -t ...files...
  4. For every burnable unit, finalize, dvdauthor -o unit$i -T
  5. Burn every unit to dvd, i.e. growisofs -Z /some/device -dvd /dev/you_know -dvd-video unit$i;
  6. Send e-mail or text-message, whatever to signal change of dvd in case either failure or succes; Build sufficient timeout for changing the dvds.

Sunday, July 25, 2010

Number chunking

I learned some excellent mental math techniques from a book I bought recently by Arthur Benjamin on mental math called Secrets of Mental Math: The Mathemagician's Guide to Lightning Calculation and Amazing Math Tricks. First I was put off by the... hm, how shall I say this, over-enthusiastic way of writing. I found that this informal way of writing did not help my motivation at all when attacking hard (relative to my skill level) problems. Later, I felt more comfortable with this style of writing, when I could hack the problems much quicker, because my ego wasn't longer in the way.

I read it before going to bed, if I can't fall asleep, because solving problems help me relax or when I'm on the throne in the morning (I'm fairly regular, hooray for me).
Do not be mislead by this, I think the book is great

Fairly early on while reading the book, I realized that I have a terrible memory for numbers. I'm quite strong on algebra but I have trouble remembering and recalling numbers. Instead of struggling with something that is hardwired into this gray gooey mass, one might as well resort to tricks of the mind. The book I mentioned is great, because it circumvents the whole nature nurture debate thing with good mental algorithms virtually anyone can learn.

So my next project is to create an python app that will help me do number chunking and unchunking. Chunking is general name for a technique to translate a seemingly unrelated group of information, into something that is more compressed, so that the chunked data is easier to remember and recall. The technique Benjamin discusses in his book is called phonetic chunking, no I call it that, actually he calls it phonetic code.

The phonetic code consists of mapping decimal numbers to certain consonant sounds:

#soundsounds likephonetic alphabet
0z,szero, soundSorry, I'm too lazy to fill this in...
1t,dtear, fat,thud,door?
2nnostril?
3mmonth?
4rrow,war?
5llove,bell?
6j,ch,shjob,cheer?
7k,glock,grow?
8f,vfun,vulva?
9p,bplastic,burp,button,pub?

Other consonants and vowels may be used freely to match words to numbers, thereby chunking them together.

I realize that there is a strong anglo-saxon English linguistic bias here, but I believe this technique is applicable to any language; even clicking ones.

The author suggests a mnemonic to remember the phonetic code by chunking those together as Tony Marloshkovips (s for final 0).

I'd like to master number chunking and unchunking by using an app at regular intervals that I write in python. Just like my mental arithmetic python app.

My first design of the algorithm goes more or less like this:
  • Generate a sample set of long integers;
  • Translate each number into the correct phonetic code;
  • Write an algorithm to map phonetic codes to words from a dictionary, optimizing on certain properties that aid the human memory, (i.e. sex, death, love, any theme that evokes strong emotion);
  • Let the user put in her own chunky words, distil the relevant phonemes from the words and translate them to numbers and compare with the numbers to be learned;
  • Randomly show word sequences for user to translate to numbers;
  • Tag frequently used words for future reference;
  • Record the time to chunk and remember/recall and unchunk long number sequences;
  • Constantly measure the semantic distance of the word used by the human user.
    Some other random ideas:The CMU pronouncing dictionary
    OCZ's neural impulse actuator
    • use dictionary.reference.com as a resource.
    • (dis)prove hypothesis that short semantic distance between words in the chunked phrase lead to improved number chunkability and recall ability;
    • (dis)prove hypothesis that words (by themselves or together) in the chunked phrase that give strongest emotional response lead to improved recall ability or chunkability (use ocz nia for this experiment);
    • Omfg, I just realized I can also used nia for my mental arithmetics training tool.

    Apparently, algorithms exist to index words phonetically, the soundex algorithm by Odel and Russel (1918) and also works on the premise that words are converted to numbers. I just realized that English may not be the best language to do this, since it has 24 letters to indicate 44 distinct phonemes, whereas for instance Russian has a better mapping of phonemes to letters. This is my gut feeling about Russian, I can read cyrillic but I don't speak the language fluently enough to give more examples. I guess someone in academia has already generated the samples of different languages to compare the phoneme to letter correspondence, in n-grams or whatnot.

    I think google has some form of prescience or something. While I was researching and reading up on soundex etc. I suddenly came accross the CMU pronouncing dictionary. This will probably prove to be handy for this application I'm thinking about.

    Thursday, July 22, 2010

    Syntaxhighlighter on blogger blog

    I spent the best part of yesterday trying to get syntaxhighlighter by Alex Gorbatchev to work on this blog via external resources, courtesy of github.com and bitbucket.org.

    I've succeeded as you see (is it broken?):

    def pyfn(a):
      print "Hello Turd!", a
    

    Anyhoo, I tested it and it works (for now).
    So this is how I did it:
    Bitbucket.org commit hash code
    1. I located Alex's code repository, later I'll link to the code from there;
    2. I realised that every commit on the repository has a different hash which is used in all the urls I'll be linking to, e.g. http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64... if you want to use the latest committed code you have to find the latest hash for yourself and replace the hash with the latest in the linking code; 
    3. You'll also need the actual linking code; read the official installation tutorial to understand what you're actually linking to:
    <link href="http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/styles/shCore.css" rel="stylesheet" type="text/css"></link>
    
    <link href="http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/styles/shCoreDefault.css" rel="stylesheet" type="text/css"></link>
    
    <link href="http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/styles/shThemeDefault.css" rel="stylesheet" type="text/css"></link>
    
    <script src="http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/XRegExp.js" type="text/javascript" />
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shCore.js' type='text/javascript'/>
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shBrushBash.js' type='text/javascript'/>
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shBrushPython.js' type='text/javascript'/>
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shBrushErlang.js' type='text/javascript'/>
    <script src='http://bitbucket.org/alexg/syntaxhighlighter/raw/9d94fe294c64/scripts/shBrushJScript.js' type='text/javascript'/>
    
    <!-- added bit for jQuery -->
    <script src='http://github.com/jquery/jquery/raw/master/speed/jquery-basis.js' type='text/javascript' />
    <script type='text/javascript'>
    $(document).ready(function () {SyntaxHighlighter.all();});
    </script>
    

    Update 23-07-2010: For some reason the latest blogger template editor does not want you to combine a <script src=...> with your own code nested in it. So don't feel tempted to nest your own code with a <script> pull.

    Blogger link to manual template edit
    The final step is to paste this in your <head>-tag. You have to modify your blogger template to do this.
    Edit template

    I used jQuery to hook "SyntaxHighlighter.all()" on to the <body>-tag's onload event. This might be a tad overkill for one function invocation, but I know for certain that I'll use jQuery again on this blog.

    Wednesday, July 21, 2010

    Demonstration python decorator

    I feel that python decorators allow programmers in python to code in a very natural way. Since a ton of blogs out there already have documented what python decorators precisely are or aren't, so I won't go into too much detail here. I probably won't do a good job explaining what they are.

    Ah what the heck, here goes my explanation:

    Babushka (grandma) matryoshka dolls
    Imagine that a decorated function is like a matryoshka doll, a matryoshka doll can be enclosed around with another doll. Python experts might not agree with this analogy, because when a function is decorated it can be accessed by the function outside it, so it does stand dormant in a dark place like a matryoshka doll.

    See the following function declarations:

    def wrap(f)
      # Add before behaviour
      print "start wrap"
      # Do something with f
      # Add after behaviour
      print "end wrap"
      return f # Pass f to the next label or babushka/function
    
    @wrap
    def fun():
      pass # do something interesting...
    

    The previous rows are equivalent to
    fun = wrap(fun)

    The wrapping function (wrap) must return a final function to relabel the newly decorated function with the original unwrapped label, which is fun in our example.

    You could also decorate more than once:
    @huge
    @medium
    @small
    def core()
      pass
    

    Which is equivalent to the following:
    core = huge(medium(small(core)))
    In this example, one must also keep passing the decorated function to the next decorator with a return statement.

    This year I decided to improve my mental arithmetic skills, so I wrote a python application to train on. I was also quite curious as to what the whole python decorator thing was about. I combined my curiosity for python decorators and the desire for a mental arithmetic trainer. The following script is the (partial) end result.

    def timeit(fun):
      from time import time
      def timed(*args):
        ts = time()
        mistakes, op_symbol = fun(*args)
        te = time()
        time_elapsed = te - ts
        print "Time elapsed:", time_elapsed
        return time_elapsed, mistakes, op_symbol
      return timed
    
    class Query(object):
      def __init__(self, op_symbol):
        self.op_symbol = op_symbol
    
      def __call__(self, f):
        # NOTE: you can count the function arguments later to try on a n-ary functions
        def wrapped_binary_f(*args):
          a,b = tuple(args)
          #print "%s %s %s = ?" % (a, self.op_symbol, b) # infix notation
          print a, self.op_symbol, b, " = ?"
          correct_answer = f(a,b)
          user_answer = None
          mistakes = 0
          while not user_answer or user_answer != correct_answer:
            user_answer = None
            try:
              user_answer = input("answer: ")
              if user_answer != correct_answer:
                mistakes+=1
                print "Please try again... %s" % str(mistakes)
              else:
                print "Correct."
                break
            except SyntaxError:
              mistakes+=1
              print "Please try again... %s" % str(mistakes)
            except NameError:
              mistakes+=1
              print "Please try again... %s" % str(mistakes)
          return mistakes, self.op_symbol
        return wrapped_binary_f
        
    @timeit
    @Query('*')
    def multiply(a, b):
      return a * b
      
    @timeit
    @Query('-')
    def subtract(a, b):
      return a - b
      
    @timeit
    @Query('+')
    def add(a, b):
      return a + b

    This example demonstrates three techniques, namely:
    1. Passing arguments to a decorator function (Query class's __init__ function);
    2. Using multiple decorator functions on multiple functions (timeit, Query);
    3. Manipulating arguments passed to the decorator function and the called functions (multiply, subtract, add) and mixing them up
    Notice the nested functions (wrapped_binary_f, timed), this is the main technique that is used to manipulate the arguments of a passed function. The outer function (__call__, timeit) are used to feed the to-be-wrapped function (multiply etc.) and the nested function serves to get the arguments passed to the to-be-wrapped function.

    One can also add arguments to the decorator function by way of a class declaration. First the __init__ function is called to create the object of the class arguments passed, the arguments are stored in the object, then the object is called via the __call__  function. The __call__ function can access the arguments from the members of the object.

    I have included the complete application here, try it out, improve it, enjoy!

    #!/usr/bin/env python
    # -*- coding: iso-8859-1 -*-
    '''
    Command-line interactive script to train mental arithmetic,
    also a demonstration of python decorators.
    Created on May 22, 2010 and refactored several times after...
    '''
    
    def timeit(fun):
      from time import time
      def timed(*args):
        ts = time()
        mistakes, op_symbol = fun(*args)
        te = time()
        time_elapsed = te - ts
        print "Time elapsed:", time_elapsed
        return time_elapsed, mistakes, op_symbol
      return timed
    
    class Query(object):
      def __init__(self, op_symbol):
        self.op_symbol = op_symbol
    
      def __call__(self, f):
        # NOTE: you can count the function arguments later to try on a n-ary function
        def wrapped_binary_f(*args):
          a,b = tuple(args)
          #print "%s %s %s = ?" % (a, self.op_symbol, b) # infix notation
          print a, self.op_symbol, b, " = ?"
          correct_answer = f(a,b)
          user_answer = None
          mistakes = 0
          while not user_answer or user_answer != correct_answer:
            user_answer = None
            try:
              user_answer = input("answer: ")
              if user_answer != correct_answer:
                mistakes+=1
                print "Please try again... %s" % str(mistakes)
              else:
                print "Correct."
                break
            except SyntaxError:
              mistakes+=1
              print "Please try again... %s" % str(mistakes)
            except NameError:
              mistakes+=1
              print "Please try again... %s" % str(mistakes)
          return mistakes, self.op_symbol
        return wrapped_binary_f
        
    @timeit
    @Query('*')
    def multiply(a, b):
      return a * b
      
    @timeit
    @Query('-')
    def subtract(a, b):
      return a - b
      
    @timeit
    @Query('+')
    def add(a, b):
      return a + b
    
    @timeit
    @Query('/')
    def divide(a, b):
      return float(a) / float(b)
    #  from decimal import Decimal, getcontext
    #  from fractions import Fraction # check this shit out, useful if user enters fraction and computer calculates float or Decimal... then check if the same
    #  getcontext().prec=2
    #  return Decimal(a) / Decimal(b)
    
    def generate_samples_naive(maxdecimals, nsamples):
      from random import choice
      return zip([ choice(xrange(10**maxdecimals)) for _i in xrange(nsamples) ],
                 [ choice(xrange(10**maxdecimals)) for _j in xrange(nsamples) ])
    
    
    def generate_samples_positive(maxdecimals, nsamples):
      """
      The lambda here swaps the values in case subtraction results are negative
      """
      from random import choice
      return map (lambda (x,y): (x,y) if x>=y else (y,x), \
                  [ (choice(xrange(10**maxdecimals)), choice(xrange(10**maxdecimals))) \
                   for _i in xrange(nsamples) ])
    
    def get_stats(li):
      """
      Tuple of total time spent solving the problems
      and total number of mistakes
      """
      return sum([ t for t,_,_ in li ]), \
        sum([m for _,m,_ in li])
    
    def get_detailed_stats(li):
      """
      Total number of mistakes per operator type,
      total number of time spent per operator type
      """
      d = {}
      for op_symbol in ['*','-','+','/']:
        d[op_symbol] = (sum([ t for t,_,o in li if o == op_symbol ]),
          sum([ m for _,m,o in li if o == op_symbol ]))
      return d
    
    def show_stats(results):
      (total_time, total_mistakes) = get_stats(results)
      print "Total solved:", len(results)
      print "Total time elapsed:", total_time
      print "Total mistakes made:", total_mistakes
    
      print "Detailed results per operator (time, mistakes):" #, get_detailed_stats(results)
      d = get_detailed_stats(results)
      for k in d.keys():
        (time, mistakes) = d[k]
        print k, "%.2f" % (float(time)/total_time), "%.2f" % (float(mistakes)/total_mistakes)
      print "Done!"
    
    def main(av):
      from optparse import OptionParser
      usage = "usage: %prog [options] arg"
      parser = OptionParser(usage)
    
      # parser.add_option...
    #  parser.add_option("-m", "--max-tries", dest='maxtries', type='int', default=3,
    #                    help='Number of tries before going to the next example')
      parser.add_option("-n", "--nr-samples", dest="nsamples", type="int", default=10,
        help="Per arithmetic operation train on this number of SAMPLES", metavar="SAMPLES")
    #  parser.add_option("--mind", "--min-decimals", dest="maxdecimals", type="int", default=0,
    #    help="Define the lowest number to train expressed as the number of DECIMALS", metavar="DECIMALS")
      parser.add_option("--maxd", "--max-decimals", dest="maxdecimals", type="int", default=2,
        help="Define the largest number to train expressed as the number of DECIMALS", metavar="DECIMALS")
    #  parser.add_option("-s", "--no-stats", dest="sendstats", action="store_false", default=True,
    #    help="Don't send any statistical data to servers, over the internet.")
    #  parser.add_option("-a", "--anonymous", dest="anonymous", action="store_true", default=False,
    #    help="Don't send credentials or any data revealing user's identity over the internet")
      parser.add_option("--nd", "--no-division", dest="train_division", action="store_false", default=False,
        help="Don't train user on division")
      parser.add_option("--na", "--no-addition", dest="train_addition", action="store_false", default=True,
        help="Don't train user on division")
      parser.add_option("--ns", "--no-subtraction", dest="train_subtraction", action="store_false", default=True,
        help="Don't train user on subtraction")
      parser.add_option("--nm", "--no-multiplication", dest="train_multiplication", action="store_false", default=True,
        help="Don't train user on multiplication")
      parser.add_option("--nh", "--no-history", dest="store_history", action="store_false", default=True,
        help="Don't store user history, mind you, the same training samples might be repeated very often")
      parser.add_option("-o", "--ordered-training", dest="ordered", action="store_false", default=True,
        help="Train in this order: addition, subtraction, multiplication, division")
      parser.add_option("-e", "--e-mail-addr", dest="email_addr", type="string", default="anon@anon.org",
        help="This script will modify itself to reflect the e-mail ADDRESS of the user in question", metavar="ADDRESS")
      parser.add_option("-p","--positive-only", dest="positive_only", action="store_true", default=False,
                        help="The result from the operation on a pair of samples will only be POSITIVE.", metavar="POSITIVE")
    
    
      (options, _args) = parser.parse_args(av)
    
      if options.positive_only:
        samples = generate_samples_positive(options.maxdecimals, options.nsamples)
      else:
        samples = generate_samples_naive(options.maxdecimals, options.nsamples)
    
      results = []
      if options.ordered:
        if options.train_subtraction:
          for (a,b) in samples:
            results.append(subtract(a,b))
    
        if options.train_addition:
          for (a,b) in samples:
            results.append(add(a,b))
    
        if options.train_multiplication:
          for (a,b) in samples:
            results.append(multiply(a,b))
    
        if options.train_division:
          for (a,b) in samples:
            results.append(divide(a,b))
    
      else:
        op_options = []
        if options.train_addition:
          op_options.append(add)
        if options.train_multiplication:
          op_options.append(multiply)
        if options.train_subtraction:
          op_options.append(subtract)
        if options.train_divison:
          op_options.append(divide)
        fns = op_options*options.nsamples
        import random
        random.shuffle(fns)
        for (nr,(a,b)) in enumerate(3*samples): # TODO - come division, change to four
          results.append(fns[nr](a,b))
    
      show_stats(results)
    
    
    if __name__ == '__main__':
      from sys import argv as av
      main(av)
    

    Update 30-07-2010: I just realized that you could grab this code and put a gui on top quite easily, by replacing the Query class with something with a gui. Done!

    Tuesday, July 20, 2010

    Pstools part 2: preparing a client

    After discovering pstools, I wanted to write a batch file to make clients more compliant to pstools. I got the grand idea of putting the batch file in the netlogon script of all the users; so all the clients would magically get compliant once a user logs on that specific machine. But alas, this plan fails, because you need system rights or admin rights to run the commands e.g. reg.exe and net.exe, to bore a hole in the local firewall (also, the group policies must be setup accordingly so that admin users have these rights).

    To make a computer compliant you have to either image/ghost over the to-be compliant machine with an already compliant image or run this batch file and pray that it works. Or login with a superuser account on the target client machine and run this from netlogon (netlogon is to expedite things, but this is not compulsory), as I described earlier.

    This command also works on most windows (professional edition) clients:
    runas /user:administrator script.bat
    

    or

    runas /user:administrator@your-ad-network.tld script.bat
    


    @echo off
    REM Make pstools and rdp work on all client machines and add local superuser
    echo "-- Make client compliant for pstools --"
    REM pstools - make compliant from registry
    reg add "hklm\system\currentcontrolset\control" /f /v SCMApiConnectionParam /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\loggedon" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psexec" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psfile" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psgetsid" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psinfo" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\pskill" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\pslist" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psloglist" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\pspasswd" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psservice" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\psshutdown" /f /v EulaAccepted /t REG_DWORD /d 1
    reg add "hkcu\software\sysinternals\pssuspend" /f /v EulaAccepted /t REG_DWORD /d 1
    netsh firewall set portopening TCP 445 ENABLE
    REM md %windir% - give -c a location to push
    REM net share admin=%windir%
    EXIT
    

    I also wrote the following script to create a default rdp (remote desktop) local admin user account to see from rdp what pstools is doing to the (remote) local user; Caveat! Don't let the password get intercepted:

    @echo off
    echo "-- Make client compliant for rdp (remote desktop) --"
    REM "remoteadmin" user (XP machines, untested on vista or 7)
    reg add "hklm\system\currentcontrolset\control\terminal server" /f /v fDenyTSConnections /t REG_DWORD /d 0
    netsh firewall set service remoteadmin enable
    netsh firewall set service remotedesktop enable
    net user remoteadmin Supersecretpassword123 /Add
    net localgroup administrators remoteadmin /Add
    reg add "hklm\software\microsoft\windows nt\currentversion\winlogon\specialaccounts\userlist" /f /v remoteadmin /t REG_DWORD /d 0
    EXIT
    

    Pstools part 3: pushing silent updates

    Suppose that you use clamwin and spybot s&d because your boss is cheap and you sure as hell ain't gonna pay for it yourself, but you have to provide some type of baseline security for your AD (active directory) networked windows clients; and you're too lazy to manually install over 50 machines, while people are using the aforementioned machines.

    Pstools and silent install to the rescue!

    Type the following from a superuser account in the AD network in the command line interface (dos):

    psexec @hosts.txt -s -n 3 -c -v "clamwin-x.y-setup.exe" /sp- /silent /norestart
    psexec @hosts.txt -s -n 3 -c -v "spybotsdxyz.exe" /sp- /silent /suppressmsgboxes /norestart


    Legend:

    parameterexplanation
    @hosts.txta file containing ip-addresses or domain client computer names seperated by endlines
    -srun with system rights
    -n 3try maximum of three times, before concluding that the client is not reachable
    -c -vPush the following file (e.g. clamwin-x.y-setup.exe) to client; -v implies push only if it's newer (spare bandwidth, non-clobber).

    Pstools part 1

    I wrote a wee nice batch file to make a windows-based client computer in an AD (active directory) environment in case pstools, namely psexec, is not cooperative.

    For people in the know, pstools is a lot like ssh for unix (linux, bsd). I understand that there's a mingw?/cygwin ssh daemon out there, but I haven't used it yet.

    Imagine that you're the system administrator and your colleague, let's call her Anna, (who is not in the know) is complaining that she does not have e.g. internet explorer on her computer.

    Normally a hard-working competent administrator would have to walk all over there and take over the account find the icon for them or directly find the binary for the supposedly missing application and double-click it there, physically, manually.

    Say you would like to check the validity of her statement from your current computer without walking all the way over there.

    This is where pstools comes in. Imagine that you can find anyone's ip-address in the lan (local area network), because you assigned names to computers used by your colleagues and put that in the dns, mind you in AD this is registered by the dhcp server.

    You fire up psexec and you type:


    psexec \\roomAcompA -i -U anna -P S3cr3t "C:\Program Files\Internet Explorer\iexplore.exe"


    \\roomAcompA stands for the computer A in room A, suppose that's the computer where Anna works, the parameter -i stands for interactive, which means the gui (if available) will be visible to Anna.

    You call her up, and ask her if anything weird has happened. She'll tell you that internet explorer has magically popped out of nowhere.

    You can also take over her machine by typing:


    psexec \\roomAcompA -s -U administrator -P S00p@S3kr1t "cmd"


    The parameter -s stands for "system", which stands for system rights granted to the current user. This is comparable to local root permissions for unix systems.

    If you're currently running pstools as a superuser on the AD network, the -U and -P parameters could be omitted.

    You could also pass local commands with system rights like this:


    psexec \\roomAcompA -s "cmd" /C "taskkill /pid 0"


    This would unabashedly kill a certain process with pid (process id) 0 with system rights. This is just an example what you could do with psexec.

    To kill processes, pstools has the pslist and pskill.

    Tuesday, July 13, 2010

    Silly sikuli suggestions

    If you haven't heard of sikuli yet then I suggest you pop over to their website or their blog. Let me quickly summarize what it does: a programming language based on jython on the jvm to control gui elements from a desktop environment.

    What a great project! It's one of those things you probably thought of before, then dismissed quickly.

    This is a nice demonstration of what you could do with sikuli; if you follow the link you'd find bejeweled being played by a programmed script.

    I tried coding in sikuli in win7 (windows 7) to run maintenance programs on it, like ccleaner, spybot s&d and clamwin. These things, for instance, clearing out useless files, immunizing the windows registry, scanning for virii, should be routinely done, but nobody really bothers to do them. Sikuli can relieve the burden of clicking from window to window.

    The following list are some ideas I got right after programming a few scripts.:
    • For some automating system administration tasks, you'll probably need password inputs, but protected, so the user can't read the passwords. Let these get scheduled by cron or windows schedule service.
    • Metauserinterface to query user for automated tasks for instance for creating local intranet accounts or accounts for twitter, gmail, etc. One could also provide a list to create accounts in bulk, just do this in sikuli >> file_list = open("names.txt"); # then loop through the names. Edit 4-07-2010: check out the functions popup and input.
    • Cut-and-paste images from image and text editors by way of the clipboard, none of that import stuff, this is a basic feature, although this feature requires some native OS hacking.
    • Port sikuli to pypy for faster image matching calculations; or just improve the algorithms; although I can imagine that this detracts from the goal of the project, since important parts of sikuli must be recoded from the ground-up.
    • Infinite undoability. Sikuli currently has no undo functionality.
    • Plugins, but without ending up like another eclipse. The whole sikuli editor should actually just get plugged into eclipse, but still have a interpreter counterpart so scripts could be run independently of eclipse, scripts that are ready for production could get run with some type of interpreter from the command line. There will be a seperation of concerns, editor, compiler and interpreter.
    • OCR (optical character recognition) plugin, to manipulate gui character output and feed it back to the system.
    Fun stuff to demonstrate with sikuli:
    • Show true introspection in sikuli/jython/jvm by letting sikuli self modify itself to do some abstract task.
    • Show how security flaws can be exploited with sikuli, also doing nasty stuff by using obfuscation.

    I'll try to find the time to implement some of my ideas. For instance creating a plugin for eclipse would be a good idea, since eclipse already has brilliant plugins like pydev which fully supports jython. Although the big-ass images might phrack up the layout of eclipse.

    Sunday, July 11, 2010

    Bridged openvpn on ubuntu

    At my work we use ubuntu "lucid lynx" as the operating system for openvpn. Our "lynx" is  a vmguest running on linx/qemu/kvm. We have linux and windows clients. Actually, all possible clients which support openvpn client mode.

    First I setup our internet-facing shorewall firewall to allow 1194 and port-forward it to our internal openvpn (vm)hosts, using shorewall-rules. We use shorewall version 4.4.6.

    #ACTION SOURCE DEST PROTO DEST_PORT ...
    DNAT wan lan:192.168.11.3:1194 udp 1194 #network alpha
    DNAT wan lan:192.168.22.3:1194 udp 1294 #network beta

    We have a multi-isp setup, that is denoted by the wan (wide area network) under SOURCE. We also have two lans (local area networks) which use this shorewall machine as the gateway to the internet, using two different subnets to communicate within each lan. So I wrote two rules to port-forward from the firewall to the correct openvpn server.

    This is called dNATing (destination network address translation + ing); in practical terms, an openvpn client knocks on a specific port on the firewall, giving the illusion that this whole business is between the firewall and the client, but in fact, the firewall is only passing messages to and fro between the openvpn server inside a lan and the remote client somewhere on the internet, regardless of whether the client itself is behind a firewall or directly on the internet. It just works.

    As I have mentioned earlier we use openvpn, the specific version is 2.1.0.
    I looked far and wide on the internet for openvpn bridging and only found this tutorial to be the easiest to setup. The weird thing is, that the tutorial supposed to be used with a N900 mobile phone from nokia.

    1) Install the software. See the tutorial first.
    2) I also setup the bridge first. As recommended by the tutorial.


    auto lo br0
    iface lo inet loopback

    iface br0 inet static
    address 192.168.11.3
    network 192.168.11.0
    broadcast 192.168.11.255
    netmask 255.255.255.0
    gateway 192.168.11.254
    bridge_ports eth0
    bridge_fd 9
    bridge_hello 2
    bridge_maxage 12
    bridge_stp off

    iface eth0 inet manual
    up ifconfig $IFACE 0.0.0.0 up
    up ip link set $IFACE promisc on
    down ip link set $IFACE promisc off
    down ifconfig $IFACE down
    # do this in /etc/network/interfaces


    3) Next you need to create the keys and certificates. This is the least difficult part.


    sudo cp -a /etc/openvpn/easy-rsa ~/

    chown -R user ~/easy-rsa
    cd ~/easy-rsa
    sudo chmod g+w . ; source ./vars ## execute your new vars file
    ./clean-all ## setup the easy-rsa directory (deletes all keys)
    ./build-dh ## takes a while consider backgrounding
    ./pkitool –initca ## creates ca cert and key
    ./pkitool –server server ## creates a server cert and key
    cd keys
    openvpn –genkey –secret ta.key ## Build a TLS key
    sudo cp {server,ca}.crt {server,ta}.key dh1024.pem /etc/openvpn


    4) Enable ip forwarding in /etc/sysctl.conf

    5) Create the up.sh and down.sh shell scripts to create and kill the tap devices when a openvpn client logs on or off. Mind you that more parameters are passed to the scripts by the openvpn server than what the tutorial lets on. I found this out by turning on the debugging facilities of the openvpn script in /etc/init.d/openvpn, like this:


    #!/bin/sh
    set -e -x
    ..



    Fix the permissions of the up.sh and down.sh files with this:

    user@ovpnubuntu~/ sudo chmod go+rwx /etc/openvpn/{up,down}.sh


    6) Modify the server.conf to reflect your lan's settings; for instance change the ip address of the "local" parameter. In our case it's 192.168.11.3. Port 1194.

    local 192.168.11.3 1194


    Openvpn has a built-in dhcp server, that distributes local ip addresses to openvpn clients. Make sure this is provisioned in your main dhcp server, so there is no chance of ip address collisions.

    # openvpn server configuration file - server.conf
    mode server
    tls-server

    local 192.168.11.3 1194
    proto udp

    dev tap0
    up "/etc/openvpn/up.sh br0"
    down "/etc/openvpn/down.sh br0"

    persist-key
    persist-tun

    #certs
    ca ca.crt
    cert server.crt
    key server.key
    dh dh1024.pem
    tls-auth ta.key 0

    #cipher and compression
    cipher AES-256-CBC
    comp-lzo

    #DHCP
    ifconfig-pool-persist ipp.txt
    ## settings for openvpn dhcp server
    server-bridge 192.168.11.254 255.255.255.0 192.168.11.216 192.168.11.224
    push "dhcp-option DNS 192.168.11.5" # DNS
    push "dhcp-option WINS 192.168.11.5" # windows-flavor DNS
    push "dhcp-option DOMAIN ad.local" # for windows active directory objects or samba4!
    max-clients 8

    #log security
    user nobody
    group nogroup
    keepalive 10 60
    status openvpn-status.log
    log-append server.log
    verb 5


    7) Configure /etc/defaults/openvpn; if you need to change the parameters passed to the openvpn server, modify OPTARGS:


    OPTARGS="--fast-io --verb 5" # fast io for fast connections; verb for verbose debug output


    After that you'll be able to restart the openvpn server automagically when the server goes down and has to reboot for some reason.

    8) Create some keys for openvpn clients; as described in the tutorial:cd ~/easy-rsa ; source ./vars ; ./pkitool {user1,user2,user3,user4,joe,satch,bob,marilyn}

    8a) If for some reason you want to prevent people from getting to your keys and/or certificates in cleartext; you can use openssl to encrypt your certificates and keys, while sending them across some unencrypted medium.

    In my case, I wanted to send myself the keys etc. via e-mail, but I'm ultra-paranoid about big-brother sticking its nose in my beeswax. So I tarred and compressed my files first.

    tar cf ~/openvpn-keys-certs.tar ~/easy-rsa ; bzip -z9 ~/openvpn-keys-certs.tar


    Then I proceeded with encrypting the bzip2 file:

    openssl aes-256-cbc -in openvpn-keys-certs.tar.bz2 -out openvpn-keys-certs.tar.bz2.enc


    This is a form of symmetric encryption, so that means if someone finds out your passphrase you recently wrote down on a piece of paper, anyone who finds that paper can decode your message. Use hard passwords with a handful of these "!@#$%^&*();<>' in different cases with numbers in them.
    Theoretically everything is brute-forceable, if one ignores the cost to crack the code. Unless there is some invariant loop-hole in the encryption algorithm to crack the dictionary of the compression algorithm.

    Decrypt with this command:

    openssl enc -d -aes-256-cbc -in openvpn-keys-certs.tar.bz2.enc -out openvpn-keys-certs.tar.bz2 ; tar xjf openvpn-keys-certs.tar.bz2


    8b) Use asymmetric encryption like gpg. Send the encrypted keys and certificates to the user using her public key. Let her worry about the private key and passphrase. There is gpg4win for windows users.

    9) Write a template client configuration file.

    # template client configuration file - client.ovpn.txt
    # Specify that this is a client
    client

    # Bridge device setting
    dev tap

    # Host name and port for the server (default port is 1194)
    # note: replace with the correct values your server set up
    remote firewall_ip_addr 1194

    # Client does not need to bind to a specific local port
    nobind

    # Keep trying to resolve the host name of OpenVPN server.
    ## The windows GUI seems to dislike the following rule.
    ##You may need to comment it out.
    resolv-retry infinite

    # Preserve state across restarts
    persist-key
    persist-tun

    # SSL/TLS parameters files created previously
    ca ca.crt
    cert replace.crt
    key replace.key

    # Since we specified the tls-auth for server, we need it for the client
    # note: 0 = server, 1 = client
    tls-auth ta.key 1

    # Specify same cipher as server
    cipher AES-256-CBC

    # Use compression
    comp-lzo

    # Log verbosity (to help if there are problems)
    verb 3


    Each client requires three 4 files in total in our setup to connect; a) The common shared (ca.crt) certificate with the server; b) The client certificate; c) The client key; d) A configuration file.

    You need to activate the template by replacing some parameters in the template with some real values. For instance there is no "replace.crt" nor "replace.key". This must be changed to some file name intended to be used by some user. For instance "bob".

    So, given that you already created the key and certificate for "bob", you can create a configuration file using the template like this:

    sed 's/replace/bob/g' client.ovpn.txt > bob-client.ovpn ; sed -i 's/firewall_ip_addr/real_firewall_ip_addr/' bob-client.ovpn

    Working at home with ssh port-forwarding, qemu/kvm and vnc

    Sometimes I work at home because I don't want to get constantly bothered by people barging into my room with trivial requests etc. Every time someone barges into my room, I find that regaining my focus takes too much effort.

    To actually get any work done, I made it possible for me to log into the system by punching a hole in the firewall for ssh traffic. I increased the security by disallowing password authentication and forcing asymmetric encryption authentication, i.e. 2048-bit rsa public key authentication, where I am the sole owner of the private key. Above all, I also limited the number of (internet) locations where this private key can be used, by putting this in the public key (authorized_keys):

    from="*.this.is.my.isp.tld" ssh-rsa MEH...

    At my work we work a lot with ssh and virtual machines. We use linux/qemu/kvm. Kvm can be initialized with the "-vnc :N" parameter, where N is a positive integer. This creates a vnc session on the vmhost. For instance, if one were to use krdc and N is 1, the invocation command would look like something like this:

    user@vmhost~$ krdc vnc://vmhost:5901

    The rule is to add N to 5900. That's how you find the port. You can also run a vnc client on the local network to view the vnc stream. This is very convenient because you're not stuck with one port for multiple vmguests.

    user@somewhere~$ krdc vnc://vmhost:5901

    To make things easier for yourself when working at home, you could port-forward this very remote vnc port to your own localhost with this invocation:

    user@athome~$ ssh -f -N -L 6001:vmhost_internal_ip_addr:5901 firewall_ip_addr
    user@athome~$ krdc vnc://localhost:6001

    The parameters "-f -N" forces this ssh client session in the background. The relevant part is the "-L" parameter. The L probably stands for local, there is also an R counterpart that port-forwards a local port to a remote port.

    This is a form of tunneling where a remote port in some remote (local) area network,  could be plugged in directly into your localhost as if this service is running on your local machine.

    This trick also applies to other tcp-based protocols, for instance the standard rdp (remote desktop protocol) port. For instance you have some windows machines you would like to log on to, as if they are running on you local machine, you do the following:

    user@athome~$ ssh -f -N -L 4489:windows_machine_ip_addr:3389 firewall_ip_addr
    user@athome~$ krdc rdp://localhost:4489


    Voila! Instant connection to a worksite windows machine. This trick also applies to windows machines using putty.