Silly sikuli suggestions

If you haven't heard of sikuli yet then I suggest you pop over to their website or their blog. Let me quickly summarize what it does: a programming language based on jython on the jvm to control gui elements from a desktop environment.

What a great project! It's one of those things you probably thought of before, then dismissed quickly.

This is a nice demonstration of what you could do with sikuli; if you follow the link you'd find bejeweled being played by a programmed script.

I tried coding in sikuli in win7 (windows 7) to run maintenance programs on it, like ccleaner, spybot s&d and clamwin. These things, for instance, clearing out useless files, immunizing the windows registry, scanning for virii, should be routinely done, but nobody really bothers to do them. Sikuli can relieve the burden of clicking from window to window.

The following list are some ideas I got right after programming a few scripts.:

• For some automating system administration tasks, you'll probably need password inputs, but protected, so the user can't read the passwords. Let these get scheduled by cron or windows schedule service.
• Metauserinterface to query user for automated tasks for instance for creating local intranet accounts or accounts for twitter, gmail, etc. One could also provide a list to create accounts in bulk, just do this in sikuli >> file_list = open("names.txt"); # then loop through the names. Edit 4-07-2010: check out the functions popup and input.
• Cut-and-paste images from image and text editors by way of the clipboard, none of that import stuff, this is a basic feature, although this feature requires some native OS hacking.
• Port sikuli to pypy for faster image matching calculations; or just improve the algorithms; although I can imagine that this detracts from the goal of the project, since important parts of sikuli must be recoded from the ground-up.
• Infinite undoability. Sikuli currently has no undo functionality.
• Plugins, but without ending up like another eclipse. The whole sikuli editor should actually just get plugged into eclipse, but still have a interpreter counterpart so scripts could be run independently of eclipse, scripts that are ready for production could get run with some type of interpreter from the command line. There will be a seperation of concerns, editor, compiler and interpreter.
• OCR (optical character recognition) plugin, to manipulate gui character output and feed it back to the system.

Fun stuff to demonstrate with sikuli:

• Show true introspection in sikuli/jython/jvm by letting sikuli self modify itself to do some abstract task.
• Show how security flaws can be exploited with sikuli, also doing nasty stuff by using obfuscation.

I'll try to find the time to implement some of my ideas. For instance creating a plugin for eclipse would be a good idea, since eclipse already has brilliant plugins like pydev which fully supports jython. Although the big-ass images might phrack up the layout of eclipse.