Not Quite Functional

Bitten by not-quite-a-functional-language again!

I really should have seen it this time, after I’ve already posted on how confusing this can be. But do you remember how certain constructs that look “functional” don’t create local environments?

Sorry about the bit that follows — it’s "research code".

So the offending lines were

    1 
    2 funcs = [
    3     lambda fk: P((fk - A) * B)
    4     for P in polys
    5 ]
    6 
    7 for j in range(p):
    8     P = scipy.special.chebyt(j)
    9     for i in range(N):
   10         fk = f[i]
   11         X[i,j] = funcs[j](fk) * np.exp(-1j*2*np.pi*fk * (dist/v*2))
   12 
   13 

Now you may be wondering, "What’s that at line 8? Why’d he make that variable called P and then not use it?" Well the answer is I used to use it, and then I didn’t, and I forgot to take it away. And that line on its own wouldn’t have been a problem, if it weren’t for what came before it…

    1 [
    2     lambda __: __
    3     for __ in __
    4 ]

By this point I should have known not to write list comprehensions with straight-up lambdas in them. The variable P is not remembered for each cycle of the list comprehension, rather it is iterated through like a regular for-loop.

Of course, I would have noticed if all my fitting coefficients had come out the same/massively ill-conditioned. Think about it: that’s what normally would happen when you make this mistake — all Ps would take on the same value, and the independent variables in the fit would have been identical. And there’s where line 8 comes into play…

Because, not only does P not get held in a closure, it is a local variable of the function, even though it’s never declared in a place where you’d see that. As someone who regularly abuses R‘s carefree scoping rules I shouldn’t complain about Python dumping stuff into the local namespace, but there are downsides to letting local variables be created inadvertently (actually this is a somewhat different problem than the one described there, since the variables are really leaking in from the local environment, and that may be an inevitable downside to dynamic name resolution).

And there’s the mystery: the coefficients came out perfectly ok, because P was being reset in the loop to what it should have been. Only when later I tried to rebuild the fitted model using those same functions in funcs (isn’t that a lovely name?) did things turn out weird. And I was totally mystified.

Anyway, the fix is the same as in R:

    1 funcs = [
    2     (lambda P: lambda fk: P((fk - A) * B))(P)
    3     for P in polys
    4 ]