Haycorn

How Knuth Brushes His Teeth

2016-02-24T00:00:00.000Z

A premature optimization?

Q: Do you have any other insights to offer into how you use maths in your work?

A: For instance, when I brush my teeth I’ve got eight areas to cover, namely Left and right, upper and lower, inside and outside. It’s most efficient to follow a “Hamiltonian path” or “Gray code”:

left upper outside

right upper outside

right upper inside

left upper inside

left lower inside

right lower inside

right lower outside

left lower outside

(source)

AppleScript Tips for Shell Scripting

2015-10-03T00:00:00.000Z

OS X is surprisingly scriptable, including OS X applications. (Especially Apple’s applications). This is the good news. The bad news is that said scripts are difficult to write and debug, and they don’t play very well with Unix-style shell scripts. (Manipulating Unix-style filenames is especially painful.)

Here’s an example of what can be done: select works like open except that it opens a Finder window with the arguments selected:

$ select *.gif

The AppleScript code is complicated and involves far more workarounds than it really should but, hey, it does work.

Tips

Some tips and tricks in no particular order.

Basics

Use & for string concatenation: return "Hello, " & name
Use as [type] to coerce to a string (you’ll need this for sane debugging): return foo as string would be written in C-like languages as return (string) foo
Built-in properties can contain spaces: track number of current track is equivalent to something like currentTrack['trackNumber'] in JavaScript
AppleScript tries to be English-like (never a good idea): assignment is via set foo to bar (not set foo = bar) and you see things like count of every media item of container src

Debugging

log to dump output for debugging, though note that some types are coerced into strings for output–compare the output of log obj and log class of obj if not sure.

Script Editor

osascript’s error messages are pretty awful; for a slightly (slightly!) better development experience, use the Script Editor application. This gets you syntax highlighting, and slightly better log messages.

For best results click on the “page” icon in the bottom of the window to reveal the “messages” tab:

(Unfortunately Script Editor can’t open files that start with a shebang (see below)–you’ll need to copy and paste AppleScript to and from a text editor into Script Editor.)

Shell scripting

There’s a few different ways you can run AppleScript from the command line. First of all, you can use the standard #! shebang with osascript as the interpreter:

$ cat hello.sh
#!/usr/bin/osascript

return "Hello, World"
$ chmod +x hello.sh
$ ./hello.sh
Hello, World

To read arguments, use the run method:

$ cat hello.sh
#!/usr/bin/osascript

on run argv
  return "Hello, " & item 1 of argv
end run
$ ./hello.sh Clem
Hello, Clem

You might often find that you often want to do some pre-processing via bash, especially for anything that involves filenames. This can be achieved (as in the select example) via a bash “heredoc”:

#!/bin/bash

# bash code
if [ $# == 0 ]; then
    echo "usage: $(basename $0) args"
    exit
fi

# AppleScript code
/usr/bin/osascript - "$@" << END

on run argv
  # ...
end run

END

Applications API

Use Script Editor’s “File | Open Dictionary…” to figure out what different applications let you do via AppleScript. Apple applications often expose a reasonably large API; for others you might get something minimal, or even nothing at all.

Spotify, for example, only lets you play/pause/next/prev. You can also get some information about the currently-playing track. One the other hand Apple Photos has a fairly rich API. (Example script.)

Working with filenames

Need the current working directory?

set pwd to do shell script "pwd"

Create file object from Unix-style path (with “/”):

(POSIX file unixpath)

Get Unix-style path from a file object:

(POSIX path file)

For more information on handling filenames, see:

Aliases and Files (from Apple’s documentation)
AppleScript and POSIX paths

Tabs and Makefile

2015-04-20T00:00:00.000Z

The computing world is littered with unfortunate mistakes that managed to make the leap from small-scale inconvenience to a source of grief to millions or billions–at which point so many people and systems have been become accustomed to the error that fixing it is impossible.

One notorious example is the spelling of HTTP header referer, which escaped the Netscape Corporation embedded in the bowels of their web browser without anyone noticing the error. From there it spread to every other web browser (they needed to be compatible), and finally RFC1945, the Hypertext Transfer Protocol standard itself. (At least, I’m pretty sure how things happened, but Wikipedia is inconclusive. Does anyone know?)

Another is the fateful decision of the author of make to have the space character and the (visually identical) tab character do very different things. Very early on, the story goes, the author realized that this was a mistake, but by that point dozens of people were relying on it—backward-compatibility needed to be preserved—and so the tabs had to stay.

Unfortunately I couldn’t find a good source for this story online, so I emailed the author, Stuart Feldman, and asked him if it was true.

From: Michael Stillwell <...@google.com>
Date: Mon, 20 Apr 2015 at 15:33
Subject: make versus tabs
To: Stuart Feldman <...@google.com>

Once upon a time I heard a story that soon after make was
released, one of its first users complained that the requirement
that each command line begin with a tab was suboptimal, and that
some alternative approach should be used. As the story goes, the
author was receptive to this criticism, but explained that sadly
it was now too late to change the way things worked, because too
many people were used to current behavior. The punchline, of
course, was that at this point make had perhaps dozens of users.

I was Googling a bit to try and figure out whether my memory was
correct (it's surprisingly resistant to search), when I
discovered that make's author works at Google, and so it might be
possible to get a definitive answer by emailing you directly.

So, to satisfy my curiosity, perhaps you could tell me: is the
story true? Are there any other details you can recall and share?
And how do you feel about tabs versus spaces versus whitespace
now?

From: Stuart Feldman <...@google.com>
Date: Mon, 20 Apr 2015 at 15:51
Subject: Re: make versus tabs
To: Michael Stillwell <...@google.com>

Story is only partly true.

I used tabs because I was trying to use Lex (still in first
version) and had trouble with some other patterns.

(Make was written over a weekend, rewritten the next weekend ...)

So I gave up on being smart and just used a fixed pattern (^\t)
to indicate rules.

Within a few weeks of writing Make, I already had a dozen friends
who were using it.

So even though I knew that "tab in column 1" was a bad idea, I
didn't want to disrupt my user base.

So instead I wrought havoc on tens of millions.

I have used that example in software engineering lectures.

Side note: I was awarded the ACM Software Systems Award for Make
a decade ago.  In my one minute talk on stage, I began "I would
like to apologize". The audience then split in two - half started
laughing, the other half looked at the laughers.

A perfect bipartite graph of programmers and non-programmers.

Old Code & Large Numbers

2015-04-06T00:00:00.000Z

Found an old passive aggressive (or maybe just aggressive) letter of mine that was published in an Australian science magazine for teens called Double Helix.

Horrible photo. It includes code, in BASIC. Numbered lines. There is a “clear screen” command though I seem to have avoided GOTO.

Not really related, but while looking through Wikipedia for more information on this problem (it’s an an example of a Tag System), I came across some fascinating examples of conjectures that have extremely large (numeric) counter-examples. (i.e. conjectures resistant to computer-derived counter-examples, and common sense.)

They are:

Pólya conjecture - initial counterexample was 1.845 × 10³⁶¹, though now reduced to 906,150,257.
Mertens conjecture - upper bound is e^{1.59×10⁴⁰}.
Skewes’ number - somewhere around e^727.95133.

Scott Aaronson has an intriguing blog post on the biggest number you can write down in 15 seconds using standard math notation.

Promise Patterns

2015-03-29T00:00:00.000Z

Promises are still pretty new, and whilst their place in the Javascript world is by this point pretty secure (most of the newer JS APIs are Promise based), the ways in which they’re used are still evolving: the clunky syntax and nested .then() structures often recall async “callback hell”, there’s no consistency to APIs that return promises, and there’s no design patterns to introduce some order. (If working with push notifications, for example, you’re likely to end up with at least three Promise-returning constructions: a static property (.ready), a getter (.getSubscription()) and a verb (.subscribe()).)

One approach to improving this situation is to lean on ES6’s yield or ES7’s await and use Babel or Traceur to transpile future Javascript into the Javascript of today, but there’s a few things you can do without leaving browser-compatible Javascript.

Use the same name for the promise and the name of the value the promise resolves to

Since you never really need the promise and the value the promise resolves to at the same time, give them the same name–they represent the same thing anyway, and giving them completely separate names makes no sense. (Annotating the types Hungarian-notation style might help, but it turns out that prepending or appending p or promise to variable names looks silly.)

That is, if your promise is foo, then the name of the argument should also be foo:

foo.then(function (foo) {
  // ...
});

Similarly, a function like fetch() does not return a fetchPromise; it returns a response:

var response = fetch(...);
response.then(function (response) {
  // ...
});

Eliminate Promise chaining with functions, objects–any which way you can

Promise hell usually isn’t quite as bad as callback hell, but it can certainly be hell-ish. For example, some Promise-based APIs you may be able to encapsulate common operations on a

Separate Promise code from non-Promise code

A lot of code mixes together logic and the boilerplate that goes with the Promise syntax. Try to avoid mixing the two by moving all the boilerplate somewhere else. (Encapsulating it inside an object might a viable choice.)

It’s possible that it will help to convert simple values into Promises:

Try to write the code you’d write if Promises were built in to the language

When working with Promises, try to imagine what the code would look like if Promises were built in to the language as first class values. (That is, if you could use Promises wherever you would otherwise use values.) Is it possible to actually write code like this, by moving move the Promise-manipulation code somewhere else? If this is possible, is it otherwise a good idea?

printf-style debugging of Promises

Here’s a simple function for console.log-style debugging of Promises:

function DEBUG(s, p) {
  p.then(
    console.log.bind(console, s, "RESOLVED"),
    console.warn.bind(console, s, "REJECTED"),
  );
  return p;
}

Use it like this:

var user = foo.getUser(); // returns a Promise
DEBUG("user = ", user);
user.then(...)

Remember that .then() lambdas can return promises

Resolving a list of promises to arguments

Remember that you can attache multiple “listeners” to a Promise

BBC News Goes Responsive

2015-03-23T00:00:00.000Z

Today, BBC News stopped redirecting mobiles users from www.bbc.co.uk/news to m.bbc.co.uk/news. Now everyone gets the same thing: we are all responsive now, and no shitty redirects. Congratulations to @jcleveley, @sthulb, @sfrench001 and loads of other good and talented folk!

I worked on the BBC News codebase a bit as part of a BBC World Service posting, and it was one of the best projects I’ve ever worked on. I learnt a lot about developing software, but also got to appreciate a bit more strongly the needs of the more marginalised and vulnerable (in terms of bandwidth, hardware, language, freedom of expression) internet users. Not to get all serious or anything but some people out there have a pretty shitty experience and from what I saw the BBC World Service does a pretty good job of trying to make things a little bit better and a little bit clearer.

The language aspect was pretty educational, though. The World Service is available in 27 languages, and for the most part the choice comes down to a combination of usage and need: in many places people would be stuck with only one source of “news” if it weren’t for the World Service. So there’s no French or German news (no French news for people living in France, that is–there’s most definitely definitely news for French speakers living in Africa), but there’s news in Ukrainian, Persian and Kyrgyz. Did you know that Uzbek has three written scripts? (The page is mostly Cyrillic on this page, but there’s also a Latin section, as well as an Arabic tab.) Or that Hausa (34m native speakers!) has apparently not settled on the spelling of the month “April”? (If you know why, please answer this question on Quora!) Or that Burmese is written in such a pretty script?

Anyway, the point is: thank-you and congratulations!

Securing a Personal Website

2015-03-14T00:00:00.000Z

So a while ago I decided that SSL/TLS (why it’s really TLS, not SSL) was the right thing to do, but CloudFlare’s blog, the seemingly endless run of TLS/SSL vulnerabilities, and the various blog posts explaining just how complicated it is to set up TLS (let alone get an A+ rating on SSL Labs’ SSL Test [1][2][3]), got me all depressed.

For a little while I considered holding out for the EFF-supported Let’s Encrypt project, which is supposed to make securing a website as easy as running two commands but: (a) it’s not available yet (it’s arriving “Mid-2015”); and (b) I’m skeptical that they can actually make it as easy as all that unless you’re running a very popular Linux distribution in a very default way, and I kinda wanted to get out of the business of running my own HTTP server anyway.

Sadly it seems that these days, outsourcing TLS is the only simple, secure and performant way to do it, as it is with DNS and SMTP. (This also provides a few other bells and whistles that would otherwise be days or hours of work to configure and maintain at the same time too, like abuse protection, a reverse proxy and maybe even a CDN.)

I’ve investigated two approaches in a little bit of detail—Gandi’s Simple Hosting and CloudFlare’s Free Plan in Flexible SSL mode—and eventually went with Gandi. CloudFlare has more features, is likely to be faster for anyone not in Paris (especially for the first page load), and is incredibly easy to set up and configure (you don’t even need to buy a certificate), but in the end it’s another piece of infrastructure to configure, if not exactly “manage”, and I already had a Gandi account, and a happy relationship with them. A web site also seems like a fairly important bit of infrastructure to be handled by a free service, especially since you also need to use their DNS servers.

Anyway, the upshot of all this is that this site now runs on Gandi’s PaaS infrastructure, and I have an A+ rating on SSL Labs’s SSL Test. (Buying a certificate through Gandi got me to an A; the A+ comes from enabling HTTP Strict Transport Security, which is a little bit scary since I’m now committed to running https://beebo.org for at least a year.)

Mountain Lion's Complicated Document Model

2012-08-07T00:00:00.000Z

A few thousand words on the Mountain Lion document model, the auto save model, how it differs from previous versions of OS X, and so on. Does this really need to be so complicated? I very much prefer the way Sublime Text handles this problem: a document is continuously auto-saved to some secret location, with the changes only written to disk when you explicitly hit save. As soon as you make some changes to a document, the “unsaved” indicator in the tab changed to “unsaved”, but if you quit the editor at this point: (a) there’s no prompt to get you to save “unsaved” changes; and (b) if you restart the editor, you get everything back the way it was before you quit, including the “unsaved” indicator. (It doesn’t save the undo stack, I just noticed, so it’s not perfect, but it’s so much better than the confusing muddle of Mountain Lion.)

Sugru Tips

2012-06-09T00:00:00.000Z

Some tips from a booklet that came with my packet of Sugru. I was hoping this would be online, so I could just throw the booklet out (the suggestion that you clean you hands by wiping them with tissue paper is excellent—water and soap is surprisingly and frustratingly ineffective), but it’s not, and so I’m typing it out here…

Master Your Craft

mix sugru to create great new colours
or just marble two colours for a funky effect
if you don’t want sugru to stick, use soapy water (it’s a release agent)
to remove sugru, carve most of it off with a sharp knife, then remove the rest with your finger nail and a tissue
carve sugru for a super sharp finish (use a scalpel when it is cured / semi-cured)
add texture, it feels lovely and looks awesome
if you want a super smooth surface—use soapy water and rub gently
build up your hack—sugru sticks to sugru
blend—add a bit of something special—wool, glitter, sand…

In case you didn’t know…

the best way to clean your hands during / after using sugru is to wipe them with tissue paper
sugru cures faster at room temp (21°C)

What SQL Query "Optimisation" Has Done For Me Lately

2012-03-19T00:00:00.000Z

So I assumed databases would have a pretty good SQL optimisation story by now, but no… Two recent disappointments:

MySQL failed to notice that SELECT ... FROM ... WHERE ? IN (foo, bar) was equivalent to ... WHERE foo = ? OR bar = 2 and/or that the latter would be much faster.
Copying a database from one server to another led to a query that took 20 seconds on the source timing out (after 10 minutes) on the copy. Even more bizarrely, the fix was to tell the copy to ignore an index. (The master and copy weren’t exactly the same version of MySQL, but they were both varieties of 5.1.x.)

The first time it sees a query, MongoDB runs multiple query plans in parallel, remembers which one worked best, and uses that the next time. Is there any relational database that does this? Seems like a good idea, given recent experiences… Are there any web pages showing what clever things query optimisers get up to, especially optimisations that aren’t immediately obvious to a human being? (For a gcc example, see Will it Optimize?)

Mobile specific URLs? They're breaking the web

2012-03-13T00:00:00.000Z

Those mobile specific URLs—m.*, mobile.* and the like—they’ve got to go. There’s just too many mobile-specific URLs popping up on Twitter and similar, and since for some reason the browser detection only only pushes people from desktop to mobile, if you’re actually on a desktop browser, you end up stuck there. A poor user experience, and it doesn’t sell many ads, either. People are just going to have to do that shit right—the same URL (i.e. resource) for everyone, and content negotiation (or “responsive design”) thereafter. (Set a cookie summarising what’s in the User-Agent header, if necessary.) It’s not that hard!

State-of-the-art JS compilers: no bytecode, no intermediate representation other than source

2011-07-19T00:00:00.000Z

A brief note; something I discovered recently found surprising: JavaScript compilers, which these days end up essentially compiling the same source code into machine code multiple times (the same function might sometimes take ints and sometimes take floats, for example), do it by going back to the source code and re-parsing the source. In particular, they don’t start from an intermediate representation like an AST or bytecode. This applies both the v8 and Firefox. (Actually it’s not clear that Firefox actually operates this way, but Eich says that it could.) I wonder if they have essentially two versions of the parser, one for unverified source code (better debugging information, but slower) and another for source code known to be syntactically correct? And what does this mean for Parrot?

Modifying protected properties in PHP

2011-07-06T00:00:00.000Z

In a blog post, Matthew Weier O’Phinney pointed out that if class Derived extends class Base, Derived can access protected properties and methods of Base even in the case where an instance of Base is merely the argument to some method. (I expected Derived to be able to access $this->foo where $foo is protected, but I didn’t expect that Derived would be able to access $obj->foo where $obj is an argument.)

This works even if the functions are static, leading to a generic way to modify protected properties:

class Foo {

    protected $message = "jjj";

}

class Foo_Wrapper extends Foo {

    // Allows you to set both public and protected (!) properties

    static function set($obj, $property, $value) {
        $obj->{$property} = $value;
    }

    // Allows you to get both public and protected (!) properties

    static function get($obj, $property) {
        return $obj->{$property};
    }

}

$obj = new Foo();

Foo_Wrapper::set($obj, "message", "kkk");

echo Foo_Wrapper::get($obj, "message"), "\n";

The difference between GPL and Apache-licensed code for web developers: not so much, actually

2011-05-20T00:00:00.000Z

For most web developers, there is no practical difference between using GPL’d code (version 2) and code with a more permissive license such as Apache or BSD. For some reason, a lot of people don’t seem to get this. (One situation in which there is difference: if you’re distributing the actual code itself—if people download your code and run it on their own servers, for example—in this case you need to make the source code available. Another is where your code is “distributed” by being embedded on a hardware device, such as a phone.)

If the code runs on your servers only, you do not need to publicly release your changes. If you’ve written the code for a customer, you need to give them the source code, but there’s no legal obligation for the customer to make the code publicly available to anyone else. (As an example, Google has written a lot of patches to MySQL and Linux, which have been deployed onto their machines. They haven’t had to make any of these patches publicly available—although at least in the case of MySQL, they’ve subsequently made some of these changes public.)

This is more or less a loophole in the GPL—it was written at a time when interactions with software were mostly conducted with software running on your own machine, and the loophole is the reason the AGPL exists. The AGPL has the properties many people seem to think the GPL has: in the case of AGPL-licensed code, if you make modifications and run the modified code on your server, you do need to make the modifications public. (MongoDB is perhaps the most prominent example of software licensed in this way; they have a helpful licensing page describing the circumstances under which you do and don’t have to release any changes you might make.)

A few useful entries from the FSF’s GPL 2 FAQ:

A company is running a modified version of a GPL’ed program on a web site. Does the GPL say they must release their modified sources? – No
Does the GPL allow me to develop a modified version under a nondisclosure agreement? – Yes
What license should I use for CMS templates? – “Templates are minor enough that it is not worth using copyleft to protect them.” Also, calling GPL’d JavaScript code from HTML does not “infect” the HTML with the GPL, but you might want to add a comment making it clear that it does not.

(In reading the FAQ entries, note that “distribution” is not what you’re doing if you deploy software to web servers, or your customers’ web servers. Distribution is where customers get a copy of your software to run on their servers.)

Why Aspect-Oriented Programming? What's wrong with pub/sub (or hooks)?

2011-05-08T00:00:00.000Z

Is there any substantive difference between Aspect-Oriented Programming and the publish/subscribe messaging pattern, where you explicitly publish a message at join points? (I consider hooks and the observer pattern to be pretty equivalent to pub/sub, too.)

To ask the question another way, is there really any point pushing for existing languages to be enhanced to support AOP when you could just use pub/sub and get nearly all the benefits? With pub/sub you avoid all the syntax weirdness, the source code transformation weirdness (if you’re doing it that way), the cultural resistance from introducing AOP to language that doesn’t already have it, the need to learn a new syntax/DSL to express join points…

The main disadvantage of hooks seems to be that you need to be able to modify existing code to use them. They’re also ugly if you want hooks absolutely everywhere, but against this they’re explicit, they can be triggered at absolutely any point, and they don’t need language support.

(Some examples: WordPress, Rails.)

Recommendation: "Writing Testable Code"

2011-04-17T00:00:00.000Z

A few folks at Google have put together a very readable PDF book called Writing Testable Code. It covers four flaws, and explains, in admirable detail and great clarity, why they are a bad thing, and how to fix them. I don’t usually like articles like this—I find them too vague and non-specific—but this is a brilliant exception. The four flaws are:

Constructor does real work
Digging into collaborators
Brittle global state and singletons
Class does too much

(It’s not just about testable code; it’s about good code in general. Testable code is much easier to reason about than untestable code.)

spl_object_hash: what is it good for??

2011-04-16T00:00:00.000Z

Is it possible to do anything useful with PHP’s spl_object_hash() function? It returns an identifier when passed an object, but:

If the object is destroyed, the identifier can be re-used.
If the object is changed, the identifier stays the same.

(In having these two properties, it seems as though the identifier is essentially equivalent to the address of the object in memory.) What on earth is this for?

JavaScript CDNs: for effective caching be specific about the version you want

2011-04-16T00:00:00.000Z

If you use Google’s CDN, if at all possible, you really want to be specific about the version of the library you want. Otherwise, if you say you want 1.7.x, the library will need to be sent to the browser with instructions to check back every so often to see if a newer release in the 1.7 series is available, which in Google’s case is just 1 hour. On the other hand, if you are able to specify that you want Prototype 1.7.0.0—no more and no less—then the browser is able to cache the library for 1 year.

boom:~$ wget --quiet --save-headers -O - \
http://ajax.googleapis.com/ajax/libs/prototype/1.7/prototype.js | grep ^Cache\-Control
Cache-Control: public, must-revalidate, proxy-revalidate, max-age=3600
boom:~$ wget --quiet --save-headers -O - \
http://ajax.googleapis.com/ajax/libs/prototype/1.7.0.0/prototype.js | grep ^Cache\-Control
Cache-Control: public, max-age=31536000

ZendMail and Amazon's Kindle Converter

2011-02-05T00:00:00.000Z

If you’re trying to use Zend_Mail to send email to Amazon’s free PDF-to-Kindle converter, you might like to know that for some reason two headers that Zend_Mail adds to the message—Content-Transfer-Encoding and Content-Disposition—trick Amazon’s converter into thinking that there are no attachments. (These emails are interpreted correctly by pretty much everything else as far as I can tell so I’m not sure if this is Zend_Mail’s fault, or Amazon’s, or both.)

There’s probably several ways to fix this, but I fixed it by creating a new version of Zend_Mail_Transport_Smtp:

class Zend_Mail_Transport_Smtp_Kindle extends Zend_Mail_Transport_Smtp
{
    protected function _prepareHeaders($headers)
    {
        // Remove some headers from the mail enclosure--these somehow confuse
        // Amazon's Kindle converter, leading it to conclude that there are
        // no attachments.  (Even though every other mail client seems to be
        // able to cope.)

        if (array_key_exists("Content-Transfer-Encoding", $headers)) {
            unset($headers["Content-Transfer-Encoding"]);
        }
        if (array_key_exists("Content-Disposition", $headers)) {
            unset($headers["Content-Disposition"]);
        }

        return parent::_prepareHeaders($headers);
    }
}

Anyway, with this “bug” worked around, I can now do this:

My ~/Library/PDF Services/Send to Kindle reads:

#!/bin/bash

TITLE=$(perl -MURI::Escape -e 'print uri_escape(join(" ", @ARGV))' $1)

cat "$3" | curl --data-binary @- "http://beebo.org/api/kindle/?username=ithinkihaveacat&title=$TITLE"

(See “Providing PDF Workflow Options in the Print Dialog” for more information.)

Understanding the Y Combinator by writing about it

2011-01-01T00:00:00.000Z

Motivation

This post is written for those who, like me, have tried and failed to understand the Y Combinator despite the help of Wikipedia, blog posts constructing the Y Combinator in multiple languages, stories on Hacker News, comments on Hacker News… I think I mostly get it now (this comes courtesy of a night in Helsinki where I was up at a strange time due to the fallout from a Helsinki-Stockholm-Helsinki booze cruise)—and so perhaps these notes will help someone else. If nothing else, writing them up helped me! (Maybe the only way to understand it is to write a blog post about it.)

I think a large part of the problem understanding the Y Combinator is that the various bits of information available online are written from several different perspectives, which means that they emphasise different aspects of the Y Combinator story, to wit:

The properties of the Y Combinator function itself. Technically, the Y Combinator is one example of a fixed point combinator, meaning that it’s a higher order function with a particular property (to be described later). Sadly it seems this property is very hard to grok unless you’ve recently spent some time with lambda calculus (or at least it was for me), so discussion is usually motivated by one particular application of the Y Combinator, which is that…
The Y Combinator allows you to write “recursive” functions without explicit recursion. (“Without explicit recursion” means “recursion” without explicitly calling a function named foo in the body of a function that itself is called foo.) Thus, discussion of the Y Combinator is tied up with anonymous (i.e. unnamed) functions. (Note that the Y Combinator-powered version of a function will almost certainly not be any faster than the recursive version, and it’s going to need just as much stack space.) Finally, Y Combinator posts and articles often have an extended section on…
The implementation of the Y Combinator in some language. The Y Combinator is an important part of (theoretical) computer science, and its implementation typically stretches a language’s support for first class functions—it’s often instructive to compare how elegantly (or not) it can be implemented in various languages. (A few examples: JavaScript, Perl, PHP.)

Deriving the Y Combinator

Step 1

As I mentioned above, discussion of the Y Combinator is typically motivated by a particular problem: how to write “recursive” functions without explicit recursion. I don’t know any better way of going about it, so let’s first consider that standard exemplar of recursion, the factorial function, and transform it into a version that uses the Y Combinator. The standard implementation:

var fact = function (n) {
  if (n < 2) return 1;
  return n * fact(n - 1);
};

(I’m using JavaScript in these examples because its widely known and has good support for first class functions.)

Step 2

How might you go about removing the self-referential call to fact() within the function body? (The exact reasons for wanting to do this are not important, but for example suppose that you want to inline the entire call.) One obvious way is to use a keyword provided by the language that refers to the function itself such as JavaScript’s arguments.callee, but, well, this is cheating and we can do better!

Step 3

Our first attempt is to transform the fact() function slightly, so that fact becomes an argument to a wrapper function, instead of a free variable referring to the factorial function. We can then use this argument in the “recursive” call:

var fact_wrapper1 = function (fact) {
  return function (n) {
    if (n < 2) return 1;
    return n * fact(n - 1);
  };
};

At this point, the function contains no references to variables declared outside the scope of the function itself—though overall we haven’t much progress because we also don’t have a working fact() function… If only there were some way to call fact_wrapper1() passing it its own return value as an argument! If we could get this to work, the return value of fact_wrapper1() would be the factorial function we want.

Step 4

Okay, so that attempt didn’t quite work out; let’s try a slightly different approach. Perhaps it’s possible to modify fact_wrapper1() so that it can be passed as an argument to itself. (This currently isn’t possible because fact_wrapper1()’s argument (fact) is a simple function that takes an integer argument and returns an integer; by contrast fact_wrapper1() itself is a higher-order function that receives a function as an argument, returning a simple function that takes an integer argument and returning an integer.) If we can pass the function to itself, we eliminate the need to figure out how to extract just its return value. fact_wrapper1() can be transformed into such a function as follows:

var fact_wrapper2 = function (f) {
  return function (n) {
    if (n < 2) return 1;
    return n * f(f)(n - 1);
  };
};

Note that the only real change is that f(f)(n - 1) replaces fact(n - 1). This is because f(f) returns a function, which we then apply to n - 1.

Step 5

Let’s just check to see if fact_wrapper2 can really be used as an argument to itself:

> var fact = fact_wrapper2(fact_wrapper2);
> fact(5);
120

Okay, that works: so far, so good! (You can copy-and-paste all these examples into a NodeJS session.)

We can also make fact_wrapper2() look a bit more like fact_wrapper1() by moving the weird f(f) call into a helper function:

var fact_wrapper2 = function (f) {
  var g = function (n) {
    return f(f)(n);
  };
  return function (n) {
    if (n < 2) return 1;
    return n * g(n - 1);
  };
};

Step 6

By adding another helper function, we can avoid the need for fact_wrapper2 to appear twice, and if we inline its definition, we can avoid the need to give the wrapper function a name at all:

var recur = function (f) {
  return f(f);
};

var fact = recur(function (f) {
  var g = function (n) {
    return f(f)(n);
  };
  return function (n) {
    if (n < 2) return 1;
    return n * g(n - 1);
  };
});

We still get the same result:

> fact(5);
120

Step 7

Next, let’s transform Step 6 by eliminating the local variable g, instead passing it in as an argument of a new inlined function:

var recur = function (f) {
  return f(f);
};

var fact = recur(function (f) {
  return (function (g) {
    return function (n) {
      // unchanged from above
      if (n < 2) return 1;
      return n * g(n - 1);
    };
  })(function (n) {
    return f(f)(n);
  });
});

Step 8

This lets us move the factorial function itself out of the helper code:

var fact_core = function (g) {
  return function (n) {
    if (n < 2) return 1;
    return n * g(n - 1);
  };
};

var recur = function (f) {
  return f(f);
};

var fact = recur(function (f) {
  return fact_core(function (n) {
    return f(f)(n);
  });
});

Step 9

Almost there! At this point we can also inline recur() itself:

var fact_core = function (g) {
  // same as above
  return function (n) {
    if (n < 2) return 1;
    return n * g(n - 1);
  };
};

var fact = (function (g) {
  // the inlined recur() function
  return g(g);
})(function (f) {
  // same as above; now an argument to the inlined recur()
  return fact_core(function (n) {
    return f(f)(n);
  });
});

Step 10

Finally, we can turn the helper code into a standalone function (i.e. eliminating the reference to fact_core in the body of Step 9’s fact by converting it into an argument), which we’ll call Y():

var Y = function (h) {
  return (function (f) {
    return f(f);
  })(function (f) {
    return h(function (n) {
      return f(f)(n);
    });
  });
};

var fact_core = function (g) {
  // same as above
  return function (n) {
    if (n < 2) return 1;
    return n * g(n - 1);
  };
};

var fact = Y(fact_core);

Hurrah! We have finally achieved Y Combinator:

> fact(5)
120

Notes and Observations

Neither the function Y() nor fact_core() are self-referential—these names do not appear anywhere in the code. A lot of other functions are involved, but since they’re all anonymous, they can’t refer to themselves either. Hence, the Y Combinator can be used to implement self-referential (i.e. recursive) programs in languages that don’t otherwise support it.

The function fact() is technically the “fixed point” of the function fact_core() and the Y Combinator is a function for finding such fixed points. The fixed point p of a function f is a function that satisfies the condition f(p) = p. What this actually means is somewhat hard to get one’s head around (see the Wikipedia page for more), but it’s easy to check that fact_core() and fact() do have these properties:

> fact_core(fact)(5);
120
> fact(5)
120

This means that the following expressions are all (mathematically) equivalent to the function fact:

Y(fact_core)
fact_core(Y(fact_core))
fact_core(fact)
fact_core(fact_core(fact_core(Y(fact_core))))

References

Wikipedia’s page on the fixed point combinator
Deriving the Y Combinator in 7 Easy Steps – JavaScript
Deriving the Y Combinator – JavaScript
The Why of Y (PDF) – Richard P. Gabriel; Scheme
Fixed-point combinators in JavaScript: Memoizing recursive functions – JavaScript
Y-Combinator in PHP – PHP
The PHP 5.3 Y-Combinator – PHP