Copy and paste

is essential for writing code ;)

Nice alias:

alias cb='xclip -selection clipboard'

And also

alias cpath='echo -n "$(pwd)" | cb'

So you can quickly take the current directory and open it in another terminal emulator, or in say firefox.

But the best part is, you can use these from vim (this is unnecessary if you have vim compiled with +clipboard; use vim –version to check):

! cb

to cut

r ! cb -o

to paste.

I had trouble getting it to work because I had the alias in ~/.zshrc, which is not read when zsh is run non-interactive. Changing to ~/.zshenv fixes.

Advertisements
Posted in zsh | Leave a comment

Preparse and load a script into Sage

There’s probably already a way to do this, but writing this function is easier than reading the doc.

def sload(name):
        os.system('sage -preparse "{0}".sage'.format(name))
        _ip.magic('run -i {0}'.format(name))
Posted in python | Leave a comment

The Awesomest Conditional Expression

I was struggling for a whole day to figure out why my tiny KDE Plasma applet wouldn’t display right.

...
   22 void Foo::init()
   23 {
   24     const char *iconName = "folder-yellow";
   25 
   26     m_icon = new Plasma::IconWidget(KIcon(iconName, NULL), QString());
   27     setPopupIcon(m_icon->icon());
   28 
   29     Plasma::ToolTipManager::self()->registerWidget(this);
   30     update();
   31 
   32     registerAsDragHandle(m_icon);
   33     setAspectRatioMode(Plasma::ConstrainedSquare);
   34 }
   35 
   36 QWidget* Foo::widget()
   37 {
   38     if (m_main == NULL)
   39     {
   40         m_main = new QLabel("Hi!");
   41     }
   42 
   43     return m_main;
   44 }
   45 

I wanted it to display as an icon in the panel, but instead the “Hi” label was showing.

And then I found where KDE makes the decision:

  282         //Applet on desktop
  283         if ((!parentApplet || parentApplet->isContainment() ) && icon 
    && (!icon->svg().isEmpty() || !icon->icon().isNull())
    && ((f != Plasma::Vertical && f != Plasma::Horizontal) ||
  284             ((f == Plasma::Vertical && parentSize.width() >= minimum.width()) ||
  285              (f == Plasma::Horizontal && parentSize.height() >= minimum.height())))) {

(popupapplet.cpp).

Ah now it makes sense…

Well, the problem seems to be that if the applet fits in its panel spot, it isn’t replaced by an icon. This feature is not unreasonable but it would be nice if PopupApplet Class Reference made some mention of it.

Posted in KDE | Leave a comment

Complex-valued linear models

Someone has probably already written code to do this. But I couldn’t find it in CRAN, so here goes.

Oh no lm() won’t take complex numbers! (or rather, it’ll take them, but it’ll discard the imaginary part.)

Easy enough fix. Split into real.

y = a*x
y1 + y2*i = (a1 + a2*i) * (x1 + x2*i)
y1 = a1*x1 - a2*x2
y2 = a1*x2 + a2*x1

So, if we’re looking to discover the coefficients a1 and a2, we can split the vectors y and x like

Re(y1)  Re(x1)  -Im(x1)
Im(y1)  Im(x1)   Re(x1)
...     ...      ...
break1 = function(X) {
        do.call(c, lapply(X, function(x) { c(Re(x), Im(x)) }))
}

break2 = function(X) {
        do.call(c, lapply(X, function(x) { c(-Im(x), Re(x)) }))
}

So if we have a function to do a complex fit, the first thing is to make new variables based on the inputs.

fit.complex = function(Y, X.List) {

        # Split into real variables
        YF = break1(Y)
        XF.List = do.call(c, lapply(X.List,
                function(x) { list(break1(x), break2(x)) } ))

        # ...
}

Then put those into a data.frame and make an appropriate formula.

        # Make the data.fram
        Data = data.frame(Y = YF)
        X.Names = paste('X', 1:length(XF.List), sep='')

        for (N in seq_along(XF.List)) {
                Data[[ X.Names[[N]] ]] = XF.List[[N]]
        }

        Formula = paste("Y ~ ", paste(X.Names, collapse='+'), "-1")

It’s important to put the “-1” in the formula so that lm() doesn’t include a constant term. (We might want a constant term, but it would have to look more like c(1, 0, 1, 0, …) because it is complex).

Then do the fit and extract the coefficients.

        Model = lm(as.formula(Formula), data=Data)

        # Make them complex again
        Coeffs = sapply(seq_along(X.List),
                function(N) {
                        ( Model$coefficients[[ X.Names[[2*N-1]] ]]
                        + Model$coefficients[[ X.Names[[2*N]] ]]*1i )
                })
        names(Coeffs) = names(X.List)

        Model$coefficients.complex = Coeffs

The whole function looks like

fit.complex = function(Y, X.List) {

        # Split into real variables
        YF = break1(Y)
        XF.List = do.call(c, lapply(X.List,
                function(x) { list(break1(x), break2(x)) } ))

        # Make the data.fram
        Data = data.frame(Y = YF)
        X.Names = paste('X', 1:length(XF.List), sep='')

        for (N in seq_along(XF.List)) {
                Data[[ X.Names[[N]] ]] = XF.List[[N]]
        }

        # Formula + Model
        Formula = paste("Y ~ ", paste(X.Names, collapse='+'), "-1")
        Model = lm(as.formula(Formula), data=Data)

        # Make them complex again
        Coeffs = sapply(seq_along(X.List),
                function(N) {
                        ( Model$coefficients[[ X.Names[[2*N-1]] ]]
                        + Model$coefficients[[ X.Names[[2*N]] ]]*1i )
                })
        names(Coeffs) = names(X.List)

        Model$coefficients.complex = Coeffs

        Model
}

Now test it

Beta0 = 1 + 3i
Beta1 = 3 - 2i

X = runif(15, 0, 10)
Y = (Beta0 + Beta1*X +
        rnorm(length(X), 0, 0.7) * exp(1i*runif(length(X), 0, 2*pi))
)

Model = fit.complex(Y, list(
         const = 0*X+1,
        linear = X
))

Beta0.Est = Model$coefficients.complex[[1]]
Beta1.Est = Model$coefficients.complex[[2]]
> Beta0.Est
[1] 1.090385+3.017922i
> Beta1.Est
[1] 2.912617-2.030427i

Excellent.

Posted in R | 2 Comments

Starting Apache on first request

I haven’t figured this one out yet. I’m going to describe the things I’ve tried so far.

So first, the story. A long time ago you could run apache through inetd. This is kind of a silly thing to do because apache takes a few seconds to start up, so they dropped the feature.

I don’t want to start apache through inetd, because that would be too slow. But I’d be OK with waiting a few seconds on the first request, if subsequent requests went straight to apache. So I’d like a program that would:

  1. Listen on port 80.
  2. When it gets a request, start apache.
  3. Forward the traffic to apache.
  4. Exit.

Now you might ask, why not just start apache at boot? After all, it’s not using hardly any CPU time when no requests are being serviced. The answer is that it’s not just apache. I have a plethora of fairly bloated services that I like to have available for when I need them, and together they drag out the boot process considerably. RAM isn’t an issue at this point, but they do collectively consume a decent amount of RAM.

So here’s the attempt I made, in (gasp) Perl. I don’t have much experience in socket programming, and it’s just a test so errors are mostly unchecked.

#!/usr/bin/perl

use IO::Socket;
use Getopt::Long;

my $port     = $ARGV[1];
my $service  = $ARGV[0];

print "port    =  $port\n";
print "service =  $service\n";

my $socket = new IO::Socket::INET(
        LocalPort => $port,
        Proto     => 'tcp',
        Listen    => 1,
        ReuseAddr => 1,
        ResuePort => 1
);

$socket or die("cannot create listener socket: $!");

my $connect = $socket->accept();
$socket->shutdown(2);
close($socket);

print STDERR "starting service...\n";
`$service`;

# Should automatically call connect()
my $redirect = new IO::Socket::INET(
        PeerHost => 'localhost',
        PeerPort => $port,
        Proto     => 'tcp'
);

$redirect or die("Failed to establish client connection: $!");

my $pipe_child = fork();

if (! $pipe_child) {
        copy($redirect, $connect, 'server -> client');

        exit(0);
}
else {
        print STDERR "forked $pipe_child to do server -> client\n";

        copy($connect, $redirect, 'server <- client');

        # Reap all children
        waitpid(-1, WNOHANG);
}

sub copy {
        my $a = shift;
        my $b = shift;
        my $name = shift;

        while (<$a>) {
                print $b $_;
        }

        print "done getting lines ($name)\n";

        $b->shutdown(2) or die($!);
        close($b) or die($!);
}

sub owner {
        my $foo = `fuser -n tcp 8000 2>&1`;
        return $foo;
}

Then to use it, run

sudo ./listener '/etc/init.d/apache2 start' 80

Open a web-browser and go to http://localhost/. It works!

But there’s a serious problem. Apache will think the request is coming from localhost. It needs to be tricked somehow, but I do not know how to trick it.

Any ideas?

Posted in networking | Leave a comment

R SQL-ish aggregation

I came to R from SQL. I’ve heard that packages such as reshape and plyr are quite beautiful, but to me they are somewhat non-intuitive. I’m always looking for

SELECT mean(sodium) as mean, variance(sodium) as var 
    FROM texas GROUP BY weight;

I struggled with this a bit. First, it’s tempting to use tapply(), except that’s more of a one-column-in, one-column-out sort of thing. There’s by(), which is nice, and with a short wrapper can be OK. Well here’s what I did.

First, the pretty stuff. Because R, while it is generally considered a practical language, is, to me, more of a pretty language. A great way to make things pretty in R is to abuse dynamic binding.

Say I have a table like this:

> Table
  Alice Bob       Cathy Doug
1     1   1  0.08309254    a
2     2   2 -0.76015321    a
3     3   3  0.30887905    a
4     4   4  0.74900962    a
5     1   5  0.69447607    a
6     2   6  0.32485197    a

I’d like to call select() like this:

select(Table, Alice,
        a = mean(Cathy * Doug),
        b = sum(Cathy - Doug) / sum(Doug)
)

Where ‘Alice’ gives the column to group by, and a=, b= give the columns to create.

So this could look like

select = function(Table, Key, Add.Key=T, ...) {
        Columns = as.list(substitute(list(...)))[-1L]

        S.Key    = substitute(Key)
        Key.Str  = deparse(S.Key)
        Group.By = eval(S.Key, envir=Table)

        ...

}

First, the as.list(substitute(list(…)))[-1L] trick to get the unevaluated expressions for the columns. Then substitute(Key) gets the unevaluated key. deparse(S.Key) gives it a name (better would let the caller specify the name, but I haven’t wanted to do that yet so I didn’t implement it).

Group.By = eval(S.Key, envir=Table)

evaluates the grouping column, so now we have it as a vector, with (hopefully) the same number of rows as the table.

Now the next step turned out to suffer some unusual efficiency problems. The obvious approach is to use by() to split Table into many subtables, and evaluate the output columns on all of those. This works, but when the number of output rows is very large, this is extraordinarily slow.

An implementation using by() would look like this

select1  = function(Table, Key, Add.Key=T, ...) {
        Columns = as.list(substitute(list(...)))[-1L]

        S.Key = substitute(Key)
        Key.Str = deparse(S.Key)
        Group.By = eval(S.Key, envir=Table)
        Table[[Key.Str]] = Group.By

        Group.Gen = function(Group) {
                List.Out = sapply(Columns, function(Col) {
                        eval(Col, envir=Group)
                })

                if (Add.Key) {
                        List.Key = list()
                        List.Key[[Key.Str]] = unique(Group[[Key.Str]])

                        List.Out = c(List.Key, List.Out)
                }

                as.data.frame(List.Out)
        }

        Group.List = by(Table, Group.By, Group.Gen, simplify=F)
        names(Group.List) = c()

        Result  = do.call(rbind, Group.List)

        Result
}
  1. Split it into tables with by().
  2. Evaluate the columns in each subtable.
  3. rbind() the outputs together.

The following implementation, which, instead of using by(), deals with vectors directly (not making a data.frame until the end), is about 10 times faster.

select2  = function(Table, Key, Add.Key=T, ...) {
        Columns = as.list(substitute(list(...)))[-1L]

        S.Key    = substitute(Key)
        Key.Str  = deparse(S.Key)
        Group.By = eval(S.Key, envir=Table)

        Indices  = tapply(1:nrow(Table), Group.By, c)

        Col.List = as.list(Table)

        Col.Out.List = lapply(Columns, function(Column) {
                sapply(Indices, function(Group) {
                        Subset = lapply(Col.List, function(Col) Col[Group])
                        eval(Column, envir=Subset)
                })
        })
        if (Add.Key) {
                Key.List = list();
                Key.List[[Key.Str]] = sapply(Indices, function(Group) {
                        unique(Group.By[Group])
                })

                Col.Out.List = c(Key.List, Col.Out.List)
        }

        Table.Out = do.call(data.frame, Col.Out.List)

        Table.Out
}

With a large table

Table = data.frame(
        Alice = rep(c(1, 2, 3, 4), 1000),
        Bob   = rep(1:2000, 2),
        Cathy = rnorm(4000),
        Doug  = rep('a', 4000)
)

The difference becomes apparent

> system.time(select1(Table, Bob, m=mean(Cathy)))
   user  system elapsed 
  3.000   0.010   3.026 
> system.time(select2(Table, Bob, m=mean(Cathy)))
   user  system elapsed 
  0.290   0.000   0.286 

Unfortunately, when the number of output rows is very, very large, even select2() becomes too slow (and much slower than tapply()) (plyr is even slower). The best solutions I have found are

  • If there’s only one input column and one output column, use tapply().
  • Use sqldf to do the query it SQLite.
Posted in R | 5 Comments

Python method gotcha

Little quirk of Python that drove me crazy.

I love Python decorators. I wanted to make a decorator for lazy evaluation.

class Lazy:

        def __init__(self, f):
                self.f   = f
                self.val = None

        def __call__(self, *a, **b):
                if self.val == None:
                        self.val = self.f(*a, **b)

                return self.val

class A:

        def __init__(self):
                self.a = 5

        @Lazy
        def foo(self):
                print("A.foo")
                return self.a

        def bar(self):
                print("A.bar")
                return self.a

a = A()

a.foo()
a.foo()

Doesn’t work!

Traceback (most recent call last):
  File "demo1.py", line 31, in 
    a.foo()
  File "demo1.py", line 11, in __call__
    self.val = self.f(*a, **b)
TypeError: foo() takes exactly 1 argument (0 given)

What’s going on? It looks like A.foo is not receiving ‘self’. That means it’s not a method — it’s a… callable object.

>>> print(a.foo)
<demo3.Lazy instance at 0x7fc8aed5ddd0>
>>> print(a.bar)
<bound method A.bar of <demo3.A instance at 0x7fc8aed5de18>>

This begs the question — how does Python decide whether something should be

  • A method bound to an object.
  • Just a function defined within a clas.

Well, it turns out, Python looks at the type of objects defined within a class, and if they happen to be a function (not just a callable object), it will make it into a method. A little wrapper fixes the problem.

# -*- coding: utf-8 -*-
def wrap(method):
        def foo(*a, **b):
                return method(*a, **b)

        return foo

def lazy(f):
        return wrap(Lazy(f))
A.foo
A.bar
A.bar

Excellent. Now it’s lazy.

Posted in python | Leave a comment