CrossOver

The past saturday I remembered a sentence I've heard from Grampa Simpon in the early seasons. I didn't remember the complete sentence, just something like

"if you can take all your winnings and risk it all 
on one roulette number and lose, and start again 
and never breath a word about your lost. The world 
is yours and everything is on it."

Just another excelent quote that come out from the Simpsons.

Well after a little search on the Internet, I found, it was part of a very well known poem from Rudyard Kipling, who was a nobel prize winner and also author of The Book of the Jungle. I found the complete poem, so I decided to posted here because it is very nice and pleasant. Also because I wanted a fast place to find it again.

If you can keep your head when all about you Are losing theirs and blaming it on you; If you can trust yourself when all men doubt you, But make allowance for their doubting too; If you can wait and not be tired by waiting, Or being lied about, don't deal in lies, Or being hated, don't give way to hating, And yet don't look too good, nor talk too wise;

If you can dream - and not make dreams your master; If you can think - and not make thoughts your aim, If you can meet with Triumph and Disaster And treat those two impostors just the same; If you can bear to hear the truth you've spoken Twisted by knaves to make a trap for fools, Or watch the things you gave your life to, broken, And stoop and build 'em up with worn-out tools;

If you can make one heap of all your winnings And risk it on one turn of pitch-and-toss, And lose, and start again at your beginnings And never breathe a word about your loss; If you can force your heart and nerve and sinew To serve your turn long after they are gone, And so hold on when there is nothing in you Except the Will which says to them: 'Hold on!'

If you can talk with crowds and keep your virtue, Or walk with Kings - nor lose the common touch, If neither foes nor loving friends can hurt you, If all men count with you, but none too much; If you can fill the unforgiving minute With sixty seconds' worth of distance run, Yours is the Earth and everything that's in it, And - which is more - you'll be a Man, my son!

Who can say now TV isn't culture.

As suggested by a couple of friends I decided to give awesome 3.1 a try. Unfortunately it wasn't available at Debian Sid's repository. So I had to download it from awesome home page and installing it from scratch. Usually this kind of things are very easy, but in this case awesome had a dependency library which was not available at Debian. So it wasn't THAT easy.

Because I didn't want my Debian installation get dirty due instalation of third-party libs on my root filesystem, I decided to install awesome and its dependency libraries in a sort of sandbox. Here it is a very short recipe:

First you have to download xcb-utils-0.3.x, which is the only dependecy not packaged in Debian. Then you must run:

$ ./configure --prefix=/sandboxdir/xcb-util
$ make
$ make install

where /sanboxdir is the sandbox directory where your third-party stuff will be located.

Then you need to download awesome-3.1.tar.gz and untar it inside your sandbox directory. The awesome windows manager uses cmake for the building process and you will need to tell cmake where to find the xcb-util lib recently installed. That's when PKG_CONFIG_PATH enviroment variable came out to solve the issue. This enviroment variable tells pkg-config where to find more metainformation about installed libraries. You have to point PKG_CONFIG_PATH where xcb-utils *.pc files are located.

export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/sandboxdir/xcb-util/lib/pkconfig/

Then you must edit awesomeConfig.cmake and add sandboxdir/xcb-util/lib to the link_directories configuration option.

link_directories(/usr/local/lib sanboxdir/xcb-util/lib)

If you have already installed all the awesome's dependecy libraries, that is all you have to do. Now , you only need to execute make for starting awesome's building proccess. If everything goes well, you will find the awesome binary inside the build directory.

Finally you migth add sandboxdir/xcb-util to your LD_LIBRARY_PATH. You probably want to add the line below into your .bashrc file, in case you are using bash as your shell.

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/sandboxdir/xcb-util/

Well that's it. And you must remeber that all of this will be obsolete when xcb-util is updated on Debian.

Yeaah!, that's great, even is a modest research work I think this means the idea is good and I can continue with this research line for at least finish my master thesis .

Here is the title and the abstract (traslated by me, original is in spanish)

A proposal for network traffic pattern recognition based on genetic algorithms.

Network traffic pattern recognition is one of the main components in today's network instrusion systems In this work, a genetic algorithm for learning a set of rules is presented. The learned rules are going to be used to normal network traffic pattern recognition. This approach is diferent from previous works in which genetic algorithms were used to learn anomalie network traffic instead. In this work ajustements to a canonic genetic algorithm are discussed, mainly covering fitness function related issues and niching techniques to get multiple solutions

You don't know what to do with yours old cpu cabinets? Well, here is an idea: You will need only two CPU cabinet and a Blindex(R) glass to build your own personal geek table for the living room.

If you are lucky and have some basic artistic skills you can also paint them for a more professional look . That's not the case for me, but my girlfriend Lili is a visual artist, so she painted the cabinets with some fancy geek motives, including Ubuntu and Firefox logos among others.

Below, you can see some pictures of the finished table in my appartment living room

A weeks ago a was doing some performance test to [pyrex]. I made an application using genetic algorithms with a very time consuming fitness function in python and I wanted to improve the overall performance of the application. That's why I tried pyrex, a python-like programming language that lets you mix python and C data types and the compiles it into a C extension for python. The pyrex homepage says nothing about perfomance improvements, because the main purpose of pyrex is developing python module wrapping C libraries in a easy way. It's more o less the same thing that SWIG does. But besides this, I wanted to try it.

My Fitness function

The implemetation in pure python of my fitness function can be viewed below. The function is simple, it receives two lists and try to count the times a given position of the list indiv has exactly the same value in the traffic list. If this is true, a weight value is added to count. The weight allow me to favour some position than others.

def compara(indiv,traffic):
    wfeat=[0.1,0.1,0.1,0.1,0.5,0.1,0.4,0.4,0.2,0.3,0.7,0.7,0.7,0.7,0.1]
    count=0

    for instance in traffic:
        for feature in xrange(0,len(indiv)):
            if indiv[feature] >= 0:
                if indiv[feature] == instance[feature]:
                    count+=(1 * wfeat[feature])
                else:
                    count+=(0.25 * wfeat[feature])
            else:
                    count+=(1 * wfeat[feature])
    return count

This simple function is very time consuming because it's invoqued more than 30000 times. So my idea was to implement this function y pyrex and see what happens.

The implementation in pyrex was very easy, is basically python with types. I have to put some array limits like in the wfeat, and the function also has hardcode the bondary check to 14. Note the non-pythonic for implementation, this is the way pyrex recomends to use for. Here is my compara.pyx

def compara (indiv,instance):
    cdef int feature
    cdef float wfeat[14]
    cdef float count
    wfeat[0]=0.1
    wfeat[1]=0.1
    wfeat[2]=0.1
    wfeat[3]=0.1
    wfeat[4]=0.5
    wfeat[5]=0.1
    wfeat[6]=0.4
    wfeat[7]=0.4
    wfeat[8]=0.2
    wfeat[9]=0.3
    wfeat[10]=0.7
    wfeat[11]=0.7
    wfeat[12]=0.7
    wfeat[13]=0.7

    count=0.0
    for feature from 0 <= feature < 14:
        if indiv[feature] >= 0:
            if indiv[feature] == instance[feature]:
                count=count+(1 * wfeat[feature])
            else:
                count=count+(0.25 * wfeat[feature])
        else:
            count=count+(1 * wfeat[feature])
    return count

The next thing I do is to use the pyrex tool to build a C file ready to be built as a python module. I simply do:

pyrex compara.pyx
gcc -c -fPIC -I/usr/include/python2.4/ compara.c

After that I got a compara.so dynamic python module ready to be imported into my application. Yupppie! After that I tested the new pyrex module and for my surprise the result was not the one I expected, the overall time of the application had barelly changed :(. I was very dissapointed. Googling a bit looking for an answer for this bad result I found this [IBM article] that explain my problem. Basically pyrex speedups python programs with a significant amount of numeric calculation. But if a program spend more of its time doing libraries call is just not going to be benefited by pyrex. In order to speed up the code you need to avoid the use of the most native python types as you can. In my case indiv and instance are still python natives types and changing that is not a trivial task.

I have been using the [psyco] JIT compiler with excelente results and with a minimun source code modification needed, but psyco faces the problem of being only supported for IA32 systems and I have a few AMD64 machines that I would like to use... I was hoping pyrex saved me of this problem but this was not the case.

For the last three years I was using wmi as my window manager. wmi is a simple and very basic window manager. But with a few cool features. Strong key bindings (everything could be done with the keyboard). A window tiled mode, which is very usefull to me, because i use to lost my xterm behind all my others windows. A simple status bar, with a very cool feature. You can put everything you want in the status bar via the wmiremote command. So with a simple script like the one below you can have a clock, a battery monitor , information about the song you are playing a so on.

while true
do
    unset infostr
    hora="[`date +%d/%m" "%H:%M`hs]"
    power="[`apm|awk '{print $5,$1}'|tr -d ,`]"
    bat="`cat /proc/apm|awk '{print $7}'`"
    signal="[`cat /proc/net/wireless |grep wlan0|awk '{print $3}'`Db]"
    playing="`mpc|head -n 1|grep -v "vol"` "
    playing_percent=(`mpc|egrep "\[playing.*%"`)
    test -n "$playing"      && infostr+=$playing
    test -n "$playing_percent"  && infostr+=${playing_percent[2]}\
    ${playing_percent[3]}
    test -n "$temp"         && infostr+=$temp
    test -n "$signal"       && infostr+=$signal
    infostr+=$hora
    infostr+=$power
    test -n "$centericq"    && infostr+=$centericq
    wmiremote -t "$infostr"
    sleep 30 
done

But now this is over for me, why? because a found a new window manager, made by the same people who made wmi, Anselm Garbe and others. dwm is the name of this new window manager. The main idea of dwm is to write a window manager in less of 3000 lines of code :).But despite this, dwm has also very cool features. The first one is the the tiled windows autoarragment. That is, you have a window layout predefined. A master windows using most of your screen and if you have more than one window open in your screen these goes to a second smaller area on the left of your screen called stacked area. If you have to open a new aplication, this one goes to the master area and the previous application moves to the stacked area. This is called Zoom.

Its very small size and the fact that is written in C, give me the oportunity to learn how to implement a window manager. Beside this, dwm has all the nice features wmi also have, but because is so small, even me can add a feature or two without a problem. And I actually did it

But this goes in another post!

In my previous post I wrote about cxFreeze, a really cool tool for distributing python applications. cxfreeze puts in a directory all the necessaries modules for running my python application. The next thing to do is to make an archive of the directory and that's it.

makeself.sh is another cool tool to make self-extractable binaries. It is bash script very simple to use. It was used by nvidia, IDsoftware and google besides others.

HOW does it work? Well, as it is indicated in the makeself.sh web site, the makeself.sh script itself is used only to create the archives from a directory of files. The resultant archive is actually a compressed TAR archive, with a small shell script stub at the beginning. This small stub performs all the steps of extracting the files, running the embedded setup script, and removing the temporary files when it's all over

Here is an example of how to use it:

harpo@ws10 $ /home/harpo/tools/makeself/makeself.sh ./dist gaids.run GaIDS ./run

The first parameter tells where to find the application directory, in this case I put my freezed application in the ./dist directory. The next parameter is the name of the self-extracting script, gaids.run. The next one is the name of the application. This name will be displayed when installing/self-extracting the script. Finally the name of the embedded setup script is needed. The setup script will be automatically run when the extraction of the application have finished.

Normaly the setup script would be used to install the application, it is not the case here. I'll use it for running the freezed python application and once it has finished I save the results in a directory with a timestamp as part of the name.

Here is my ./run script:

#!/bin/sh
# Silly script for Gaids autoexecution

data_dir="gaids-"`date +%s`
./gaids
mkdir ../$data_dir
cp *.{dat,log} ../$data_dir/
cp rules-* ../$data_dir/

As I wrote in my previous post the main idea of using cx_Freeze and makeself.sh was to find a simple way to tune a genetic algorithm program. With this script I can simulate the behavior of a tools like Condor. I can write a simple launcher script that send my python application to a list of linux machines and when each of them has finished, they send back the resulting data to central repository, maybe a FTP server o something like that. Note this behavior have not been implemented in the above ./run script, but could be easyly added.

Of couse because we are using Condor at LAPIC, this behavior is not necesary. But could be a good choice in some cases.

The last days I was working with a Genetic Algorithm Program made in Python. For tunning the algorithm I need to run the program for 30 times at least. Then I collect all the data and do some statistics test. Because each run takes more than 20 minutes, I have to use some basic home-made distributed solution.

The basic idea was to use a shell script to run the algorithm with the same parameters in each one of the cluster nodes we have here at LAPIC. Nothing fancy, as every node share the master's filesystem via NFS, a simple ssh to each one of them did the trick.

Now I want more! The cluster has only 12 nodes, but here at LAPIC we have almost 40 computers. Why don't use them? Well, when you want something like this, is when things like Condor come to mind. Condor is a HTC system for do things like this.

Beacuse my GA program was made in python, I need a way to distribute the program to machines which could not have python installed. For that, the best tool I found was cx_Freeze. It is very easy to use, Just move to the cx_Freeze source directory and write:

python MakeFrozenBases.py
python FreezePython.py --no-copy-deps FreezePython.py

These python scripts collect information about your python instalation, and build the binary. So the next thing to do is use the ./FreezePython/ binary to build your program's binary distribution. In my case I did:

./FreezePython -c -O --target-dir=gaids_dist/ \ 
--target-name=gaids gaids.py

With the line above I'm building a binary named gaids based on gaids.py, and FreezePython will put all the necesary dependecies for running the program in the gaids_dist. The distribution directory will look somethin like this.

harpo@ws010:gaids_dist$ ls -1
array.so
binascii.so
collections.so
cPickle.so
cStringIO.so
datetime.so
fcntl.so
gaids
itertools.so
math.so
md5.so
psyco._psyco.so
_random.so
select.so
_socket.so
_ssl.so
strop.so
struct.so
termios.so
time.so
zlib.so

And that's it. All I have to do now is to made a tar archive file with all the dependencies.

Tip: FreezePython will use the python dynamic library if can find it. That's not good for me. Instead I need to link against the static library libpython.a. Fortunally FreezePython do that by default, unless it can find a dynamic library first. FreezePython look at the /usr/lib/python/config for the python library. You should be sure the dynamic version isn't there!

Now it would be nice to have a selfinstaller script... but that is another thing. post.

WOW, more than 2 months from my last post!!, that's a lot time. What was I doing for the last 2 months? Lots of thinks. And because the main idea of this blog is practice my English, In the following posts I will try to remember the most interesting ones.

In March I went to Buenos Aires for 4 days, We met the people of the LSC again. This time we had the oportunity to work with Ben Clifford from de Computer Institute at Chicago University. Ben works with Ian Foster grid research group.

The main idea of this trip was learn about the state of the art of grid research. And of course told Ben about ours current researchs. It was a great experience for me, In a very informal presentation I had to told in english about my master thesis work.

Ben is a great guy, hopefully we will meet again.

After that me and the others guys from LAPIC went to Santa Fe to the PAV school. PAV school, is a school for grad students funded by goverment grants. We stay there for 2 weeks. The first week we took a course about grid technologies with our already known friend, ben Clifford. It was a practical course mostly based on tools like on globus toolkit and condor. The course was great for me, because I did'nt have so much experience with this tools.

The second week we took two courses, one in the morning and the other in the afternoon. The first one was about a UML based model for developing web applications. This course was interesting because gave me the oportunity to review my knowledge about UML. Which is in fact very poor The other one was about Fundational Ontologies, this course was very hard, because we have to work with a high level of abstraction.

After these two weeks, we finally come back to Mendoza, and start working with the homework for approving the courses we took. At this moment we only have sent the work for the Web Applications course. We are waiting for news about Ontologies' teacher to start working on that one.

A few weeks ago I was in Bs. As. for work.I was visiting the FCEN of the University of Buenos Aires (Mostly for work). We have a research project in common with the LSC, the main idea was to exchange ideas about the posibility of publish a few works together.

It was an exausting trip, I did'nt have time to visit any of my friend there, hope next time I could do it. The good or bad news is I have to come back in a few days!

The interesting thing is that Damian, a fellow from the LSC, told me about a new hobbie. Kiteflying!! Yeah, it is interesenting (and wierd) to see the geeks also have the time to enjoy a few hours outdoor.

In Bs. As. there is a big community of kiteflyers, they have meeting for share they experiences, fly their kites together and have a good time enjoying the fresh air. Mostly like a Linux User Group, without the fresh air

Too funny! It's a pitty here in Mendoza City we don't have enough wind to do the same, in town at least.