CrossOver

In my previous post I wrote about cxFreeze, a really cool tool for distributing python applications. cxfreeze puts in a directory all the necessaries modules for running my python application. The next thing to do is to make an archive of the directory and that's it.

makeself.sh is another cool tool to make self-extractable binaries. It is bash script very simple to use. It was used by nvidia, IDsoftware and google besides others.

HOW does it work? Well, as it is indicated in the makeself.sh web site, the makeself.sh script itself is used only to create the archives from a directory of files. The resultant archive is actually a compressed TAR archive, with a small shell script stub at the beginning. This small stub performs all the steps of extracting the files, running the embedded setup script, and removing the temporary files when it's all over

Here is an example of how to use it:

harpo@ws10 $ /home/harpo/tools/makeself/makeself.sh ./dist gaids.run GaIDS ./run

The first parameter tells where to find the application directory, in this case I put my freezed application in the ./dist directory. The next parameter is the name of the self-extracting script, gaids.run. The next one is the name of the application. This name will be displayed when installing/self-extracting the script. Finally the name of the embedded setup script is needed. The setup script will be automatically run when the extraction of the application have finished.

Normaly the setup script would be used to install the application, it is not the case here. I'll use it for running the freezed python application and once it has finished I save the results in a directory with a timestamp as part of the name.

Here is my ./run script:

#!/bin/sh
# Silly script for Gaids autoexecution

data_dir="gaids-"`date +%s`
./gaids
mkdir ../$data_dir
cp *.{dat,log} ../$data_dir/
cp rules-* ../$data_dir/

As I wrote in my previous post the main idea of using cx_Freeze and makeself.sh was to find a simple way to tune a genetic algorithm program. With this script I can simulate the behavior of a tools like Condor. I can write a simple launcher script that send my python application to a list of linux machines and when each of them has finished, they send back the resulting data to central repository, maybe a FTP server o something like that. Note this behavior have not been implemented in the above ./run script, but could be easyly added.

Of couse because we are using Condor at LAPIC, this behavior is not necesary. But could be a good choice in some cases.

The last days I was working with a Genetic Algorithm Program made in Python. For tunning the algorithm I need to run the program for 30 times at least. Then I collect all the data and do some statistics test. Because each run takes more than 20 minutes, I have to use some basic home-made distributed solution.

The basic idea was to use a shell script to run the algorithm with the same parameters in each one of the cluster nodes we have here at LAPIC. Nothing fancy, as every node share the master's filesystem via NFS, a simple ssh to each one of them did the trick.

Now I want more! The cluster has only 12 nodes, but here at LAPIC we have almost 40 computers. Why don't use them? Well, when you want something like this, is when things like Condor come to mind. Condor is a HTC system for do things like this.

Beacuse my GA program was made in python, I need a way to distribute the program to machines which could not have python installed. For that, the best tool I found was cx_Freeze. It is very easy to use, Just move to the cx_Freeze source directory and write:

python MakeFrozenBases.py
python FreezePython.py --no-copy-deps FreezePython.py

These python scripts collect information about your python instalation, and build the binary. So the next thing to do is use the ./FreezePython/ binary to build your program's binary distribution. In my case I did:

./FreezePython -c -O --target-dir=gaids_dist/ \ 
--target-name=gaids gaids.py

With the line above I'm building a binary named gaids based on gaids.py, and FreezePython will put all the necesary dependecies for running the program in the gaids_dist. The distribution directory will look somethin like this.

harpo@ws010:gaids_dist$ ls -1
array.so
binascii.so
collections.so
cPickle.so
cStringIO.so
datetime.so
fcntl.so
gaids
itertools.so
math.so
md5.so
psyco._psyco.so
_random.so
select.so
_socket.so
_ssl.so
strop.so
struct.so
termios.so
time.so
zlib.so

And that's it. All I have to do now is to made a tar archive file with all the dependencies.

Tip: FreezePython will use the python dynamic library if can find it. That's not good for me. Instead I need to link against the static library libpython.a. Fortunally FreezePython do that by default, unless it can find a dynamic library first. FreezePython look at the /usr/lib/python/config for the python library. You should be sure the dynamic version isn't there!

Now it would be nice to have a selfinstaller script... but that is another thing. post.

WOW, more than 2 months from my last post!!, that's a lot time. What was I doing for the last 2 months? Lots of thinks. And because the main idea of this blog is practice my English, In the following posts I will try to remember the most interesting ones.

In March I went to Buenos Aires for 4 days, We met the people of the LSC again. This time we had the oportunity to work with Ben Clifford from de Computer Institute at Chicago University. Ben works with Ian Foster grid research group.

The main idea of this trip was learn about the state of the art of grid research. And of course told Ben about ours current researchs. It was a great experience for me, In a very informal presentation I had to told in english about my master thesis work.

Ben is a great guy, hopefully we will meet again.

After that me and the others guys from LAPIC went to Santa Fe to the PAV school. PAV school, is a school for grad students funded by goverment grants. We stay there for 2 weeks. The first week we took a course about grid technologies with our already known friend, ben Clifford. It was a practical course mostly based on tools like on globus toolkit and condor. The course was great for me, because I did'nt have so much experience with this tools.

The second week we took two courses, one in the morning and the other in the afternoon. The first one was about a UML based model for developing web applications. This course was interesting because gave me the oportunity to review my knowledge about UML. Which is in fact very poor The other one was about Fundational Ontologies, this course was very hard, because we have to work with a high level of abstraction.

After these two weeks, we finally come back to Mendoza, and start working with the homework for approving the courses we took. At this moment we only have sent the work for the Web Applications course. We are waiting for news about Ontologies' teacher to start working on that one.