How to install packages with easy_install or pip through proxy

There are two easy way to install packages for python (regardless of the platform you're using), namely easy_install and pip. The later is actually an overlay on the former.

They work very nicely, except with pesky proxies. Both of them have the ability to download through proxy, but they use different syntax.

For pip you will simply add the option '--proxy' to indicate the proxy address like so:

Whereas easy_install will rely on environment variables, http_proxy, https_proxy , ftp_proxy

If you want those settings to persist, don't forget to add them to your bashrc, or through your windows settings

Enabling WebSSO / Kerberos Auth in Firefox

I've been using Chrome for a while, but i recently switched back to Firefox, at least for work (mostly for memory usage reasons).

 XMarks makes the back and forth transition a piece of cake. The only issue that i had was with Kerberos / WebSSO which was enabled by default on Chrome but not Firefox.

Its a simple configuration change though. all you need to do is access the settings by typing "about:config" in the address bar and search for the following items:

• network.negotiate-auth.trusted-uris: value= mySite1.com,mySite2.com
• network.negotiate-auth.delegation-uris : value= mySite1.com,mySite2.com

• network.automatic-ntlm-auth.trusted-uris: value= mySite1.com,mySite2.com
• network.automatic-ntlm-auth.allow-proxies: value=true

•  network.negotiate-auth.allow-proxies: value=true

And you're good to go.

Control fan speed under windows 7 installed with bootcamp on mac

This week end temperature suddenly came back to 30s (Celcius) which mean winter seems to be over here (HK). Although this is fantastic new as the weather was gorgeous, my trusty mac wasn't so happy. I put him under very heavy load, and cooling didn't seem to work so great.

So i went trying to manually crank up the fan speed. I tried SpeedFan which is a very good tool, but unfortunately doesn't seem to be able to control fan speed on my Mac running windows 7 (from what I've read in various places  this is "normal").

Still I've found a work around, simple increase the default running speed of the fans. It doesn't make my box more noisy, but that's fine for me. Do do that you need to change the default running speed of the fan under MacOs and it'll carry over to Windows7.

I did that using a tool named "Fan Control". Install it, raise the speed to max and all the thresholds to the minimum.

Ant - setting Environment variables in a Hudson/Jenkins build

Today, while trying to setup Jenkins for a project, one of the job includes a ant build for a java project. This build has been working pretty well for some time, but i wanted to automate in inside Jenkins. I encountered a problem when i tried to create the new jenkins job, because the ant build references some environment variables.

How to display the top x biggest folder under unix linux

As the titles mentions here's a little one liner when you want to find out who's are the top disk space users under a certain directory:

du -sk * | sort -nr | cut -f2 | head -5 | xargs du -sh

The command above show the top 5 offenders in a nice readable way (i.e not in kb which i find hard to read):

$ du -sk * | sort -nr | cut -f2 | head -5 | xargs du -sh
 3.2G   Offender1
 1.8G   Offender2
 1.3G   Offender3
 1.1G   Offender4
 603M   Offender5

How to grep for NULL char

Was trying to extract some FIX messages from log files, and more precisely filter out a number of messages like heartbeat. My initial thought was to do something like that:

cat fixlog.log | grep '\[8=FIX.' | tr '\1' '|' | grep -v '\|35=0'

That works fine, except it's a little bit wasteful. What this command above does is, replace all the null character (that are used as delimiter in FIX messages) by "|". So that got me thinking how I could filter out those without having to replace anything. My first idea was:

cat C20120203.log | grep '\[8=FIX.' | grep -v '\00135=D'

That didn't work because grep was interpreting \001 as a back reference. Several variants later (among which the use of sed) and some constructive goggling made me understand that the the problem wasn't coming from grep but from bash and here's the proof.

If you type the following (under linux):

$ echo '\001' | hexdump -C
00000000  5c 30 30 31 0a                                    |\001.|
00000005

The trick is to tell bash do some quoting (escaping), and you do that by prefixing the string with $:

$ echo $'\001' | hexdump -C
00000000  01 0a                                             |..|
00000002

So putting it all together the original command above becomes:

cat fixlog.log | grep '\[8=FIX.' | grep -v $'\00135=0'

The mechanism allow you to grep for any arbitrary ASCII character.

How to use easy_install through a proxy

You first need to make sure that the shell variable http_proxy is defined:

$ env | grep http_proxy
http_proxy=http://your-proxy.com:yourport

On this done you can run simply run easy_install as follow:

sudo -E easy_install your_package

The "-E" parameter for sudo preserves your environment (namely variables) which is not the default behavior.

How to fix windows 7 install stuck at step "Finalizing your settings"

While trying to install windows 7 on my shiny Macbook, after the whole ordeal i had went through (See here and here for more details), I finally had the install going and was at the last step.

By last step I mean my computer had already rebooted several times, went through "Setting up your computer for first time use", and was at the step "Finalizing your settings".

Unfortunatly this last step took forever. By Forever i mean, after leaving my computer at it the whole night, was still stuck there in the morning.

Install Windows 7 with boot camp on mac (Part 2)

Small resume from Part 1:

• I am the (un)happy owner of a late 2008 Macbook Pro
• My super Drive is DEAD
• I have Mac OS X Lion installed on my mac
• For whatever (foolish) reason I want to install Windows 7 on a Bootcamp partition
• I used Bootcamp 4.0.1

In the guide below i will explain how i managed to (finally) install Windows 7 on my bootcamp partition. But before we go further, some important warning. This whole procedure is at best dangerous, and is not for the faint hearted. It can possibly screw up your disk, follow it at your own risk. I strongly advise that you read this guide completely before starting this whole procedure by yourself. Now that you've been warned, let's get started.

Windows 7 with boot camp on macbook late 2008 (Part 1)

In this serie of articles, i will describe the steps I followed while trying to install a Windows 7 in Boot Camp. I will try to make this guide as detailed as possible in hope it'll be helpful to other people. But before i dive in a bit of context, that has its importance.

My computer is a MacBook Pro of late 2008, whose Superdrive is unfortunately dead. Which meant i was not able to simple put in the DVD install of Windows 7 into my Superdrive into my MacBook, which left me with trying to install it from a USB boot disk. Unfortunately, my Mac being from late 2008, it means that usb boot disk was out of the question, which i only discovered after trying to the steps below.

How to install cx_Oracle and oracle client on debian

In this post i'm going to explain how to install and use the python lib named cx_Oracle on a debian box. To make things a little more spicy, i will start from the assumption that oracle is not installed on the box, and that i don't want to install a full fledged oracle server on my box.

So after this procedure i will have a minimal install of oracle libs, that means the bare minimum to use cx_Oracle. More precisely, that means no oracle client, no sqlplus will be on that box.

The first step is to download the oracle instant client and the cx_Oracle library that matches. Out of laziness, i decided to use the pre compiled version of cx_Oracle.

Do download oracle instant client, go on oracle website here. Then choose the platform that / library that matches your oracle server. For me it was "instantclient-basiclite-linux32-11.2.0.2.0.zip"

Once you have downloaded that file unzip it and install it. I personnaly chose to install it in (i.e move the unzipped folder) into /usr/local/lib.

Once this done, you need to define two variables:

export ORACLE_HOME=/usr/local/lib/instantclient_11_1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/instantclient_11_1

Once this is done, we can move on to install cx_Oracle. I downloaded the RPM version that matches my python and oracle version, in my case cx_Oracle-5.1.1-11g-py26-1.i386.rpm.

Being under debian, RPM is not the favored method to install packages. Fortunately there's an utility that allows to convert RPMs to DEB files called alien. Here's the steps to install alien and cx_Oracle:

# install alien
sudo apt-get install alien

# convert cx_Oracle-5.1.1-11g-py26-1.i386.rpm to cx_Oracle-5.1.1-11g-py26-1.i386.deb and install it
alien -i cx_Oracle-5.1.1-11g-py26-1.i386.rpm

Once that done, there's one last step. This installed cx_Oracle library was installed under /usr/lib/python2.6/site-packages. Unfortunatly by default, site-packages wasn't part of the default python search path. There's several ways to go around that which i won't describe at length (more details here under "Modifying Python’s Search Path"). I chose to modify site.py under /usr/lib/python2.6/ (mostly because it's the first but probably not the last package that will be installed in site-packages).

In order to do that I simply edited site.py and searched for addsitepackages and edited it as follow:

def addsitepackages(known_paths):
    """Add site-packages (and possibly site-python) to sys.path"""
    sitedirs = []
    seen = []

    for prefix in PREFIXES:
        if not prefix or prefix in seen:
            continue
        seen.append(prefix)

        if sys.platform in ('os2emx', 'riscos'):
            sitedirs.append(os.path.join(prefix, "Lib", "site-packages"))
        elif os.sep == '/':
            sitedirs.append(os.path.join(prefix, "local/lib",
                                        "python" + sys.version[:3],
                                        "dist-packages"))
            sitedirs.append(os.path.join(prefix, "lib",
                                        "python" + sys.version[:3],
                                        "dist-packages"))
            sitedirs.append(os.path.join(prefix, "lib", "dist-python"))

            ##################
        # added following line to make site-packages part of default search path
            #################
            sitedirs.append(os.path.join(prefix, "lib",
                                        "python" + sys.version[:3],
                                        "site-packages"))

        else:
            sitedirs.append(prefix)
            sitedirs.append(os.path.join(prefix, "lib", "site-packages"))

        if sys.platform == "darwin":
            # for framework builds *only* we add the standard Apple
            # locations. Currently only per-user, but /Library and
            # /Network/Library could be added too
            if 'Python.framework' in prefix:
                sitedirs.append(
                    os.path.expanduser(
                        os.path.join("~", "Library", "Python",
                                     sys.version[:3], "site-packages")))

    for sitedir in sitedirs:
        if os.path.isdir(sitedir):
            addsitedir(sitedir, known_paths)

    return known_paths

After all that, you should be able to type "import cx_Oracle".

Eclipse syntax highlighting: how to configure same occurrence color properly

Eclipse has some nice syntax highlighting feature, especially with the popular add-on "Eclipse Color Theme" from the Market place. The result is pretty nice, at least for my taste:

(here the obsidian theme)

On very nice feature is the ability to highlight automatically same occurrences of same. Unfortunately as you see here the "follow" keyword is totally bloated and unreadable. 

This can be configured, but it's that there's a legion of option for syntax highlighting. This one is actually controlled by "annotations", more specifically "C++ occurences":

And its suddenly much more readable.

Enabling proxy under Debian

In order to enabled a proxy, there's two main places (at least for my use this did the trick, there's probably are others). The first one is simple defining the following variables:

export http_proxy="http://your.proxy.com:your_port"

export ftp_proxy="http://your.proxy.com:your_port"

Secondly in order to have apt-get work through proxy, you need to update /etc/apt/apt.conf.d/proxy (or create it):

Acquire::http::Proxy "http:///your.proxy.com:your_port";

VirtualBox: Creating new VM by duplicating existing one

The other day i tried to create a new VM by duplicating an existing one. Everything worked perfectly ... except all network interface were disable. And it was impossible to mount up those network interfaces.

If you duplicated a new VM from an existing one, you have to do a couple of things. First MAC addresses should be changed in new VM.

Then once the VM is created, disabled all network interface and boot it up. Once the machine is started, there's a couple of files to update for the hostname. I'm using a Debian squeeze, so path may change in other distributions:

/etc/hosts 

/etc/hostname

Then you need to need to edit  /etc/udev/rules.d/.  What happens is it maps the "new" mac address to a new interface. You just need to remove the old one and rename the new one (thanks to the solution from this forum: https://forums.virtualbox.org/viewtopic.php?t=7749)

Using Eclipse as your R IDE

There's a bunch of possible IDE which are R specific, amongst which to name a few:

• RKWard
• R Studio (my personal favorite)
• Rattle
• ...

There's a very long list each with their own pros and cons, which I  don't intend to discuss in this post.

What i do want to discuss is the usage of Eclipse as a R IDE. Why use it compared to other. Well being a developer during the day, I rather like the idea of using the same tool at night so to speak. Another added bonus on top of syntax completion, object inspection, is the integration with an which ever SCM you like to use. For me that's the icing on the cake. Time being of the essence I don't want to look anything and relying on a SCM is a perfect way to do that.

Now enough talk. In order to have that integration working you need a couple of things:

• StatET: that's the eclipse plugin (see here)
• rj : that's the "other side of StatET (see here)
• rJava: that's a java binder with R (see here)

Then follow the excellent step by step guide here on how to setup Eclipse itself in order to work with R.

Enjoy

Mercurial integration with SVN.

I've recently been evaluating integration of SVN with some other distributed source control systems, mainly Git and Mercurial. The idea is to have a distributed source control system that would integrate seamlessly with a central SVN repository (very slow and painfull access over internet). That allows a certain number of things, amongst which there is fast access to commit history, cheap branches to experiment...

At some point I had both Mercurial and Git installed, and decided it was time to do a bit of cleanup. I have a bias towards Mercurial because it has a better integration in Windows explorer through TortoiseHG (you can call that lazyness).

After uninstalling everything I started clean, installed the latest TortoiseHG version (which btw has a very nice GUI), as well as hgsubversion as explained here: Getting started with HGSubversion

Mercurial was working fine but I was running into the following error each time I tried to clone our repo:

hg clone svn+ssh://myuser@myrepo.com/someotherpath
destination directory: someotherpath
abort: Can't create tunnel: The system cannot find the file specified.


That error message has been driving me nuts for a while, and after trying a lot of things (uninstalling everything, installing Git again, checking library versions ...), I stumbled upon the solution :

set SVN_SSH="C:\\Program Files\\PuTTY\\plink.exe"


As simple as that... Just define that variable SVN_SSH and it worked flowlessly. After that I justed defined it in my user environment to make it persist.

Hopefully this will help someone else.

K-means clustering using R

I'm trying to learn R, and I'm a firm believer that there's not better way to learn than by getting your hands dirty. After reading an excellent post on Intelligent Trading blog, it got me thinking how you would do a clustering analysis with R, using K-means.

In the rest of this post I will try to detail the different steps that I followed, in hope that it can be useful to others. In this post I will be using a couple of R packages, namely quantmod, fpc and a few others. The most crucial is quantmod. You can install it with:

They are used for:

• quantmod: this is your bread and butter to ease time series analysis
• graphics, scatterplot3d, gplots RColorBrewer are used for plotting
• fpc: this packages is package dedicated to clustering 

First this article supposes that you already have your data handy in a xts object used by quantmot. If it's not the case have a look at this article.

In the details below, "x" is the name of the object that contains my timeserie. Now let dig into it.

The first thing you need to do is create matrix with the different criteria you want to use for the clustering. In my case I'm going to use three normalized ratios, Close / Open, High / Open,  Low / Open , and then stuff them into a matrix. Then you want to process the actual cluster and display it:

The result looks like:

This works pretty well, except you have to specify a number of cluster you're looking for. Another may to do this is through the pamk function from the fpc package, for for which you don't specify the actual number of cluster, it will be calculated (you provide a range of value though):

Note the difference with kmeans method, the cluster information is packed into a pam object. That why you access the cluster details through zpamk$pamobject.

Enjoy.

Thumbnails for Videos

I recently installed a web gallery to host photos and videos named Zenphoto. It works perfectly to my taste except for a small detail. Video galleries load a flash player named flowplayer, which does the job nicely, but until you click on the actual video to make the flash player load, you can't see a thumbnail of the video.

 The solution to have thumbnails for videos in Zenphotos, the same way you have for photos, is to place a jpg with the same name as the video. For instance if you have a video named a.mp4, create a jpeg named a.jpg in the same folder as the video. This photo will be used as thumbnail for you video.

It's fine to do a few manually ... but call me lazy I don't fancy the extra manual step each time I need to add a new video. So a create a little script to generate the thumbnails. 

Before digging in the details, it assumes a few things. First this only works under linux, so you must not be afraid of command line. Secondly it relies on ffmpeg to extract a frame automatically.

If you don't have ffmpeg and you're using an Ubuntu / Debian type:

Then you into the folder under which the videos are and type:

And paste the following content into the file:

Run the script and you're done.

How to load csv data with quantmod in R

In this article I'm going to describe how to load data in an xts object, which is the preferred object to handle time series quantmod from a CSV file. This article is really for beginner (like myself), hopefully those can be helpful to others.

Let's imagine you have some csv data that look like:

First thing you need to make sure is that you column headers have names that contain  Open, High, Low and Close. This will make you life much easier. Secondly, your csv file should only contain dates, and numerical data, otherwise it's kind of a hassle and you have to go and manually convert data afterwards. So  what your data should really look like is:

Personnally I "massaged" the data in R, but you could very well do that in Excel, Perl, Python, or whatever tool you fancy (or simply correct the data at the source ;) ) but for the sake of example this is how I did it:

Now that you have the correct data, it's pretty simple to load :



You're now the proud owner of an xts object that you can start abusing in all the ways you'd like ;)



Hope this helps

How to extract a DEB file

DEB files are ar archives, which contain three files:

• debian-binary
• control.tar.gz
• data.tar.gz

The actual data is contained into data.tar.gz. You can extract it with the following command:

ar p yourfile.deb data.tar.gz | tar xz