Friday, November 25, 2011

How to use easy_install through a proxy

You first need to make sure that the shell variable http_proxy is defined:
$ env | grep http_proxy
http_proxy=http://your-proxy.com:yourport


On this done you can run simply run easy_install as follow:
sudo -E easy_install your_package
The "-E" parameter for sudo preserves your environment (namely variables) which is not the default behavior.

Tuesday, November 22, 2011

How to fix windows 7 install stuck at step "Finalizing your settings"

While trying to install windows 7 on my shiny Macbook, after the whole ordeal i had went through (See here and here for more details), I finally had the install going and was at the last step.
By last step I mean my computer had already rebooted several times, went through "Setting up your computer for first time use", and was at the step "Finalizing your settings".
Unfortunatly this last step took forever. By Forever i mean, after leaving my computer at it the whole night, was still stuck there in the morning.

Monday, November 21, 2011

Install Windows 7 with boot camp on mac (Part 2)

Small resume from Part 1:
  • I am the (un)happy owner of a late 2008 Macbook Pro
  • My super Drive is DEAD
  • I have Mac OS X Lion installed on my mac
  • For whatever (foolish) reason I want to install Windows 7 on a Bootcamp partition
  • I used Bootcamp 4.0.1

In the guide below i will explain how i managed to (finally) install Windows 7 on my bootcamp partition. But before we go further, some important warning. This whole procedure is at best dangerous, and is not for the faint hearted. It can possibly screw up your disk, follow it at your own risk. I strongly advise that you read this guide completely before starting this whole procedure by yourself. Now that you've been warned, let's get started.

Windows 7 with boot camp on macbook late 2008 (Part 1)


In this serie of articles, i will describe the steps I followed while trying to install a Windows 7 in Boot Camp. I will try to make this guide as detailed as possible in hope it'll be helpful to other people. But before i dive in a bit of context, that has its importance.

My computer is a MacBook Pro of late 2008, whose Superdrive is unfortunately dead. Which meant i was not able to simple put in the DVD install of Windows 7 into my Superdrive into my MacBook, which left me with trying to install it from a USB boot disk. Unfortunately, my Mac being from late 2008, it means that usb boot disk was out of the question, which i only discovered after trying to the steps below.

Thursday, November 17, 2011

How to install cx_Oracle and oracle client on debian

In this post i'm going to explain how to install and use the python lib named cx_Oracle on a debian box. To make things a little more spicy, i will start from the assumption that oracle is not installed on the box, and that i don't want to install a full fledged oracle server on my box.

So after this procedure i will have a minimal install of oracle libs, that means the bare minimum to use cx_Oracle. More precisely, that means no oracle client, no sqlplus will be on that box.

The first step is to download the oracle instant client and the cx_Oracle library that matches. Out of laziness, i decided to use the pre compiled version of cx_Oracle.

Do download oracle instant client, go on oracle website here. Then choose the platform that / library that matches your oracle server. For me it was "instantclient-basiclite-linux32-11.2.0.2.0.zip"

Once you have downloaded that file unzip it and install it. I personnaly chose to install it in (i.e move the unzipped folder) into /usr/local/lib.

Once this done, you need to define two variables:
export ORACLE_HOME=/usr/local/lib/instantclient_11_1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/instantclient_11_1

Once this is done, we can move on to install cx_Oracle. I downloaded the RPM version that matches my python and oracle version, in my case cx_Oracle-5.1.1-11g-py26-1.i386.rpm.

Being under debian, RPM is not the favored method to install packages. Fortunately there's an utility that allows to convert RPMs to DEB files called alien. Here's the steps to install alien and cx_Oracle:
# install alien
sudo apt-get install alien

# convert cx_Oracle-5.1.1-11g-py26-1.i386.rpm to cx_Oracle-5.1.1-11g-py26-1.i386.deb and install it
alien -i cx_Oracle-5.1.1-11g-py26-1.i386.rpm



Once that done, there's one last step. This installed cx_Oracle library was installed under /usr/lib/python2.6/site-packages. Unfortunatly by default, site-packages wasn't part of the default python search path. There's several ways to go around that which i won't describe at length (more details here under "Modifying Python’s Search Path"). I chose to modify site.py under /usr/lib/python2.6/ (mostly because it's the first but probably not the last package that will be installed in site-packages).

In order to do that I simply edited site.py and searched for addsitepackages and edited it as follow:
def addsitepackages(known_paths):
    """Add site-packages (and possibly site-python) to sys.path"""
    sitedirs = []
    seen = []

    for prefix in PREFIXES:
        if not prefix or prefix in seen:
            continue
        seen.append(prefix)

        if sys.platform in ('os2emx', 'riscos'):
            sitedirs.append(os.path.join(prefix, "Lib", "site-packages"))
        elif os.sep == '/':
            sitedirs.append(os.path.join(prefix, "local/lib",
                                        "python" + sys.version[:3],
                                        "dist-packages"))
            sitedirs.append(os.path.join(prefix, "lib",
                                        "python" + sys.version[:3],
                                        "dist-packages"))
            sitedirs.append(os.path.join(prefix, "lib", "dist-python"))

            ##################
        # added following line to make site-packages part of default search path
            #################
            sitedirs.append(os.path.join(prefix, "lib",
                                        "python" + sys.version[:3],
                                        "site-packages"))

        else:
            sitedirs.append(prefix)
            sitedirs.append(os.path.join(prefix, "lib", "site-packages"))

        if sys.platform == "darwin":
            # for framework builds *only* we add the standard Apple
            # locations. Currently only per-user, but /Library and
            # /Network/Library could be added too
            if 'Python.framework' in prefix:
                sitedirs.append(
                    os.path.expanduser(
                        os.path.join("~", "Library", "Python",
                                     sys.version[:3], "site-packages")))

    for sitedir in sitedirs:
        if os.path.isdir(sitedir):
            addsitedir(sitedir, known_paths)

    return known_paths

After all that, you should be able to type "import cx_Oracle".

Wednesday, October 26, 2011

Eclipse syntax highlighting: how to configure same occurrence color properly

Eclipse has some nice syntax highlighting feature, especially with the popular add-on "Eclipse Color Theme" from the Market place. The result is pretty nice, at least for my taste:


(here the obsidian theme)
On very nice feature is the ability to highlight automatically same occurrences of same. Unfortunately as you see here the "follow" keyword is totally bloated and unreadable. 

This can be configured, but it's that there's a legion of option for syntax highlighting. This one is actually controlled by "annotations", more specifically "C++ occurences":


And its suddenly much more readable.


Friday, October 14, 2011

Enabling proxy under Debian



In order to enabled a proxy, there's two main places (at least for my use this did the trick, there's probably are others). The first one is simple defining the following variables:
export http_proxy="http://your.proxy.com:your_port"
export ftp_proxy="http://your.proxy.com:your_port"

Secondly in order to have apt-get work through proxy, you need to update /etc/apt/apt.conf.d/proxy (or create it):
Acquire::http::Proxy "http:///your.proxy.com:your_port";



VirtualBox: Creating new VM by duplicating existing one

The other day i tried to create a new VM by duplicating an existing one. Everything worked perfectly ... except all network interface were disable. And it was impossible to mount up those network interfaces.

If you duplicated a new VM from an existing one, you have to do a couple of things. First MAC addresses should be changed in new VM.

Then once the VM is created, disabled all network interface and boot it up. Once the machine is started, there's a couple of files to update for the hostname. I'm using a Debian squeeze, so path may change in other distributions:

/etc/hosts 
/etc/hostname

Then you need to need to edit  /etc/udev/rules.d/.  What happens is it maps the "new" mac address to a new interface. You just need to remove the old one and rename the new one (thanks to the solution from this forum: https://forums.virtualbox.org/viewtopic.php?t=7749)







Thursday, June 9, 2011

Using Eclipse as your R IDE


There's a bunch of possible IDE which are R specific, amongst which to name a few:
  • RKWard
  • R Studio (my personal favorite)
  • Rattle
  • ...
There's a very long list each with their own pros and cons, which I  don't intend to discuss in this post.

What i do want to discuss is the usage of Eclipse as a R IDE. Why use it compared to other. Well being a developer during the day, I rather like the idea of using the same tool at night so to speak. Another added bonus on top of syntax completion, object inspection, is the integration with an which ever SCM you like to use. For me that's the icing on the cake. Time being of the essence I don't want to look anything and relying on a SCM is a perfect way to do that.

Now enough talk. In order to have that integration working you need a couple of things:
  • StatET: that's the eclipse plugin (see here)
  • rj : that's the "other side of StatET (see here)
  • rJava: that's a java binder with R (see here)

Then follow the excellent step by step guide here on how to setup Eclipse itself in order to work with R.

Enjoy

Monday, May 23, 2011

Mercurial integration with SVN.

I've recently been evaluating integration of SVN with some other distributed source control systems, mainly Git and Mercurial. The idea is to have a distributed source control system that would integrate seamlessly with a central SVN repository (very slow and painfull access over internet). That allows a certain number of things, amongst which there is fast access to commit history, cheap branches to experiment...

At some point I had both Mercurial and Git installed, and decided it was time to do a bit of cleanup. I have a bias towards Mercurial because it has a better integration in Windows explorer through TortoiseHG (you can call that lazyness).

After uninstalling everything I started clean, installed the latest TortoiseHG version (which btw has a very nice GUI), as well as hgsubversion as explained here: Getting started with HGSubversion


Mercurial was working fine but I was running into the following error each time I tried to clone our repo:

hg clone svn+ssh://myuser@myrepo.com/someotherpath
destination directory: someotherpath
abort: Can't create tunnel: The system cannot find the file specified.

That error message has been driving me nuts for a while, and after trying a lot of things (uninstalling everything, installing Git again, checking library versions ...), I stumbled upon the solution :

set SVN_SSH="C:\\Program Files\\PuTTY\\plink.exe"


As simple as that... Just define that variable SVN_SSH and it worked flowlessly. After that I justed defined it in my user environment to make it persist.

Hopefully this will help someone else.

Saturday, May 7, 2011

K-means clustering using R

I'm trying to learn R, and I'm a firm believer that there's not better way to learn than by getting your hands dirty. After reading an excellent post on Intelligent Trading blog, it got me thinking how you would do a clustering analysis with R, using K-means.

In the rest of this post I will try to detail the different steps that I followed, in hope that it can be useful to others. In this post I will be using a couple of R packages, namely quantmod, fpc and a few others. The most crucial is quantmod. You can install it with:



They are used for:

  • quantmod: this is your bread and butter to ease time series analysis
  • graphics, scatterplot3d, gplots RColorBrewer are used for plotting
  • fpc: this packages is package dedicated to clustering 


First this article supposes that you already have your data handy in a xts object used by quantmot. If it's not the case have a look at this article.

In the details below, "x" is the name of the object that contains my timeserie. Now let dig into it.

The first thing you need to do is create matrix with the different criteria you want to use for the clustering. In my case I'm going to use three normalized ratios, Close / Open, High / Open,  Low / Open , and then stuff them into a matrix. Then you want to process the actual cluster and display it:



The result looks like:
This works pretty well, except you have to specify a number of cluster you're looking for. Another may to do this is through the pamk function from the fpc package, for for which you don't specify the actual number of cluster, it will be calculated (you provide a range of value though):

Note the difference with kmeans method, the cluster information is packed into a pam object. That why you access the cluster details through zpamk$pamobject.

Enjoy.

Friday, May 6, 2011

Thumbnails for Videos

I recently installed a web gallery to host photos and videos named Zenphoto. It works perfectly to my taste except for a small detail. Video galleries load a flash player named flowplayer, which does the job nicely, but until you click on the actual video to make the flash player load, you can't see a thumbnail of the video.

 The solution to have thumbnails for videos in Zenphotos, the same way you have for photos, is to place a jpg with the same name as the video. For instance if you have a video named a.mp4, create a jpeg named a.jpg in the same folder as the video. This photo will be used as thumbnail for you video.

It's fine to do a few manually ... but call me lazy I don't fancy the extra manual step each time I need to add a new video. So a create a little script to generate the thumbnails. 


Before digging in the details, it assumes a few things. First this only works under linux, so you must not be afraid of command line. Secondly it relies on ffmpeg to extract a frame automatically.

If you don't have ffmpeg and you're using an Ubuntu / Debian type:


Then you into the folder under which the videos are and type:

And paste the following content into the file:

Run the script and you're done.

Thursday, April 21, 2011

How to load csv data with quantmod in R

In this article I'm going to describe how to load data in an xts object, which is the preferred object to handle time series quantmod from a CSV file. This article is really for beginner (like myself), hopefully those can be helpful to others.

Let's imagine you have some csv data that look like:

First thing you need to make sure is that you column headers have names that contain  Open, High, Low and Close. This will make you life much easier. Secondly, your csv file should only contain dates, and numerical data, otherwise it's kind of a hassle and you have to go and manually convert data afterwards. So  what your data should really look like is:


Personnally I "massaged" the data in R, but you could very well do that in Excel, Perl, Python, or whatever tool you fancy (or simply correct the data at the source ;) ) but for the sake of example this is how I did it:


Now that you have the correct data, it's pretty simple to load :



You're now the proud owner of an xts object that you can start abusing in all the ways you'd like ;)



Hope this helps

Monday, April 18, 2011

How to extract a DEB file

DEB files are ar archives, which contain three files:
  • debian-binary
  • control.tar.gz
  • data.tar.gz
The actual data is contained into data.tar.gz. You can extract it with the following command:
ar p yourfile.deb data.tar.gz | tar xz

Sunday, April 17, 2011

Compiling a linux kernel on a Debian "like" distribution

I recently had to re-compile a kernel, and there's a few preliminaries. You need to install a bunch of packages :

sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge
sudo apt-get install kernel-package

After that you can follow the excellent guide here:

One point which I had missed though is where are the packages once you've finished compiling (although it's written in the guide). So if like they recommend in the you've extracted the kernel source under ~/src/linux-source-2.6.35 for instance, then the deb packages will be in ~/src.


Enjoy.

Sunday, April 10, 2011

Edit Formula: Alternative to F2 for Excel Mac


One of the most used keyboard shortcut is F2, which allows to edit the formula in a cell without using the mouse. Unfortunately that shortcut is different between Office for Mac and Office Windows.

This shortcut was F2 in a PC, but in Excel for Mac it turns into Ctrl + U.  Unfortunately not the most effective shortcut, as it's rather difficult to press Ctrl + U with one hand on most keyboards.


Windows Shortcut: F2

Mac Shortcut: Ctrl + U

Excel Mac: Cell Reference Lock - F4 Alternative


One of the first major problem that I faced when I first began using Excel on a Mac, was the appropriate shortcut replacement for F4, which adds $ signs to cell references to lock cell/range references from moving if dragged to another cell. After a great deal of frustration, and after trawling a number of excel forums I eventually found the solution which is shown below.

Windows ShortcutF4

Mac Shortcut : Cmd + T

Install from Perl modules from CPAN without confirmation

If need to install modules from CPAN, especially if you want to install a whole bunch in one go without having to confirm every ten seconds try running your install as follow:


PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install My::Module'

Additionally if you want to make sure that the install script will pull all the required dependencies (should be by default, but heh ...):



cpan
cpan> o conf prerequisites_policy follow
cpan> o conf commit


That's it.

Tuesday, April 5, 2011

Find closest and fastest repository for your Debian

I recently installed a new distrib with linux mint based on Debian. I've done that scores of times, but one thing i never remember is how to find the best repo (and the fastest of course) based on your location.

There's a very neet package that will do the work for your:
sudo apt-get install netselect-apt
sudo netselect-apt

And Voila, you're done.


Monday, April 4, 2011

Update Debian Squeeze to Mint ~ a.k.a Mintify your Debian

I've been willing to install a new linux for a while, more precisely a Linux Mint distrib for various reasons. Call me lazy i didn't wanted to download and burn a full DVD. So the came the idea to use a Debian install as a base (since there is a version called LMDE base on Debian instead of Ubuntu.

At the time of this post, Debian has recently updated Squeeze to stable, so that the freshly baked Linux Mint isn't too far away (LMDE is supposed to be based on the current testing debian).

If you want to follow the same steps as me, this post assumes a couple of things:

  1. You can and will use command line without fear
  2. You know what debian is and can read log messages or use test installer
  3. You computer is already connected to internet


Some thanks to those two posts which i used as a base and mix together with my own twist:
HOWTO: Create a custom, minimal LMDE install
Mintifying Debian

Enough talk, let's get down to it. First you'll need to download a ISO of Debian, i used the businesscard version. You can find other version fitting to your need / architecture here:
http://www.debian.org/releases/squeeze/debian-installer/

1) Download it, burn it or put it on a USB key, which ever you prefer and install debian.

Once the install is completed, and you've added yourself as a sudoer ,you can proceed.



2) Then you may want to add the contrib and non-free Debian repositories by editing the file /etc/apt/sources.list and adding “contrib non-free” without quotes right at the end of each line ending with “main”.

4) Next you add the Debian Multimedia and the Mint repositories. You can for instance create a file within /etc/apt/sources.list.d called “lmde.list” and add these two lines into it:

deb http://packages.linuxmint.com/ debian main upstream import
deb http://www.debian-multimedia.org squeeze main non-free



5) Updates :

sudo apt-get update
sudo apt-get install linuxmint-keyring
wget -c http://www.debian-multimedia.org/pool/main/d/debian-multimedia-keyring/debian-multimedia-keyring_2010.12.26_all.deb
sudo dpkg -i debian-multimedia-keyring_2010.12.26_all.deb
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade



Now the trick start here. Apparently when Squeeze went from testing to stable a particular package called "gstreamer0.10-pitfdll" has been removed from Debian repo cause the all meta package mint-meta-debian to fall to pieces. With some error message that look like:



The following packages have unmet dependencies:
mint-meta-debian : Depends: mint-meta-codecs but it is not going to be installed



The following packages have unmet dependencies:
mint-meta-codecs : Depends: gstreamer0.10-pitfdll but it is not installable
E: Broken packages





But fear not, here's the solution:
wget -c http://debian.securedservers.com/debian/pool/contrib/g/gstreamer0.10-pitfdll/gstreamer0.10-pitfdll_0.9.1.1+cvs20080215-1_i386.deb
dpkg -i gstreamer0.10-pitfdll_0.9.1.1+cvs20080215-1_i386.deb


Now you can safely proceeed to the Mintifying of your debian:
apt-get install mint-meta-debian

Enjoy.