'); for (var i = 0; i < 5; i++) { var sentence = generate('S'); document.writeln('
• ' + sentence + '
'); } document.writeln('

I liked the pictures, but that heads-up they gave before the race was slimy. I wondered, "how can I make a better, cheaper service for finding photos of myself (and put these guys out of business :P)?" The result is bibfind.com: a crowd-sourced photo tagging system that uses the bib numbers worn by runners to identify them in photos.

Steve McConnell recommends you "use more parentheses than you think you need" (pg738 in Code Complete, 2nd edition). That solution is very practical, but it amounts to giving in to the language. I'm an idealistic young kid, and I think we should make the languages suit us instead of changing ourselves to suit the languages. I've also taken enough philosophy courses that just thinking bores me, so here's some action:

The code that accomplishes this feat is simple; view source to see exactly how it's done. The heavy lifting is accomplished in under 10 lines of code, and those aren't very dense lines either.

Now the real question is, will I find some open source language and implement this thoughtful grouping? We'll see...

Thinking about this data, it occurred to me that a simple bar chart or histogram would be inappropriate, because I'd have to choose some arbitrary point at which to start drawing the bars, say midnight, with the result that 11pm would end up on the other side of the chart. That would be wrong, because in reality one comes right after the other. To solve that problem, I decided to plot my calls in a circle. The result is that the angular distance between two data measures on the chart corresponds directly with their temporal distance (fancy term isn't it?).

Similarly, the "small multiple" (Tufte's term) plots for each weekday are arranged in a flattened clockwise loop to keep temporally adjacent days near to each other.

Initially, I envisioned this plot as a circular histogram, and that's how I coded it. Then I had the idea of making those small multiple plots for the individual days, and I realized that I'd need to use a consistent scale for comparisons between those plots to be meaningful. So it turned into a circular bar chart.

I liked the way that looked, but one of Tufte's points kept nagging me: he says that the amount of visual real-estate you give to the representations of your data points should be directly proportional to the magnitude of those data points (I'm paraphrasing, accurately I hope). Looking at my circular bar chart, I realized that I was violating that principle, and that he was right about it. I varied the radius of each circular "bar" in proportion to the amount of calls during the corresponding hour, but the area of a pie slice (segment of a circle) is proportional to the square of the radius of the pie. That means that the area of the circular bar for an hour with 5 calls would be less than half the area for the circular bar representing 10 calls. So I scrapped the bars in favor of lines; they're one-dimensional, and so are the data they represent.

The lines for daytime hours are colored in shades of yellow, and nighttime ones in shades of blue. Tufte says that most people can distinguish blue from other colors, even if they're colorblind, and as a bonus, there's some semantic relationship between those colors and the hours of the day they correspond with (skycolor). Unfortunately the yellow is hard to see against a white background. Expletive deleted.

the making of

To get at my phone's call data, I mangled mrflip's perl code for extracting the SQLite database from the iphone's backup files (see this thread). Then I put together some python scripts to digest the call data into a form that's conducive to charting and send it down the pipeline to a buggy little piece of Processing code that draws the actual graphics. The bugs have to do with the calculation of the low, median, and high values that are labeled on the charts, but I touched up the graphic in Photoshop to mask the issues.

That cover-up was a profound philosophical moment for me. I had the option of tightening my code to handle all cases, pass all unit tests (hah! as if I wrote those), and be prepared for use by total strangers. But as the gears in my mind spun up to design this bulletproof algorithm, the monkeywrench of cool rationality fell in. Taking that code from functional with bugs to bugless would have taken a disproportionate amount of time. I'm ok with mostly bugless, because this code is throwaway; it was just a tool I made as one step in the process of generating the chart.

This is what I like most about being able to code: if I don't have the tools I need to make something, I can make the tools.

Let me start by saying that I think it's great that neat opportunities for web developers are posted on this list. Keep 'em coming! But as a software engineer I tend to have reservations about "I have an idea; now I need an 
engineer" type requests, so I thought I'd lay out my thought process here for the benefit of everyone. Before I start, a caveat: I may not be representative of all web developers; I believe there are websites where you can go to post a job description and have someone remotely code your site for you, but I'm not the sort of engineer that looks for jobs there, so please take what I have to say with this in mind.

First question I usually have is "what's the idea?" I have a mountain of ideas to implement whenever I find the time and inclination, so an overly vague description is unlikely to get me interested. Of course I understand the desire to protect your idea, but for me at least, the description has to be more of a sales pitch than a "need handyman to build 4'x6' doghouse"-style ad in the paper.

The secrecy also raises a big red flag for me. I think: he's protecting the idea, so he's in love with it, and without any detail on his background, I have to assume that the idea person ("Idea" from now on) is non-technical. 
From that combination I conclude that I'll have to do more work than Idea realizes and thus suggests, Idea will be frustrated by the slow rate of progess relative to his expectations, AND Idea will believe that the idea is worth at least as much as my implementation (which is taking so damn long!), so he'll be unwilling to give up an appropriate amount of equity/cash. All of these conclusions might be wrong. Idea might be a PhD in CS who got an MBA, now works at a nice consulting firm, and just wants to do this idea on the side but doesn't have time to code it himself. But I need to hear this up front, otherwise I assume the worst (described above) and nothing is going to get me to play a role in that sad story.

After that initial stuff is out of the way, I'm curious what I can expect from this project. Is it going to be fun from a technical standpoint? Web development involves a lot of mindless drudgery, particularly when it comes to tweaking the visual aspects of a website. So either the idea should be so exciting that I won't mind that stuff, there should be a graphic designer taking care of front-end details, or the financial prospects have to be good enough that I can slog through it (but this option is probably not the most preferable). Which brings me to...

What's it going to pay? I'm not necessarily interested in exact dollars, but more, what sort of cash-equity combination is it. If it's all cash, this might be the sort of job for one of those rent-a-coder websites; if it's all equity, then how will my equity compare to Idea's? If it's not all one or the other, what sort of mix is it (and I'm still interested in how my proportion compares to Idea's)? If the equity is significant (and it probably should be to get our interests aligned) then sell me on the business merits. I've already got a fancy piece of paper entitling me to thousands of shares in a tech-startup, so if you want to sell me another, it had better be even fancier (I'm joking :P) or I also need to be convinced that yours will be worth the ink it's printed with.

--------------------------

Hopefully that gave you a peek into the software engineer's perspective on ideas needing an engineer, because propositions that take this perspective into account are way more likely to get my attention (and other engineers too I'd guess). It probably sounded one-sided, because I'm sure I'm biased here. I understand that we engineer's need business people too; there are plenty of business administration tasks that I'm sure I'd find even less interesting than making the corners on some box render correctly across all the browsers out there. So good luck with whatever idea you're pursuing!

This technique visualizes the meaning of a word in addition to calling it by name.

Here's how it works: in the text field at the bottom you enter a number. All entries in Pascal's Triangle divisible by the number you entered will be shown, provided they are in the part of Pascal's Triangle that's shown in the applet. The powers of 2 are the most interesting; I like 64. Have fun!

These things start searching before I'm done typing, which is distracting and inefficient. They're distracting, because the first part of my search may not yield results that are relevant once the whole thing is in. For example, see the screenshots below of me searching for "free running" with Google Suggest. The results after typing "free" don't overlap at all with the results after the full query "free running" is entered.

These live search boxes are also inefficient, because if they run a search while I'm typing, another search will be necessary when I'm done; but as long as I'm typing I can't click on the intermediate results, so they're wasted. In the iTunes screenshot below I'm searching for "bon jovi" but it does a premature search for "bon". That does bring up the right results, but it brings up some wrong results along with them, and if it had been patient enough to wait the milliseconds for me to finish typing " jovi" I would have gotten the exact results I wanted.

I admit that when my computer is running smoothly these intermediate results just flash by until I've finished typing and my final results are shown. But when my computer isn't at full processing capacity, the intermediate searches can be slow enough that they cause iTunes to freeze temporarily, and interfere with me typing out my full query. That's what really gets me about live search boxes with this behavior: they can make it harder to do the search I want to do.

Maybe an analogy is useful. In the gym at my school the men's locker room has toilets with those infrared sensors that lets them detect when they're occupied, and flush when appropriate. When functioning correctly, this system works exactly how I want the live search to work: the device accepts input from the user, and when it has detected that the user is finished inputting, it processes the input appropriately. Unfortunately, the sensor systems don't work properly, and they frequently process input while it's still coming in. As with the search boxes above, this operation is inefficient (here in terms of water usage) and believe me it's distracting. But the correct functioning of these toilet systems gives the perfect analog of my solution to the live search box problem.

sidenote: I feel justified using toilet humor to describe this problem because Jef Raskin did it too; "A search is either incremental or excremental." (footnote on p.126 of Jef Raskin, The Humane Interface, 2000).

my solution

The problem was that these search boxes start searching before I'm done typing, so the solution should be apparent. It's to make the damn boxes wait for me (and you and everyone else) to finish typing! A simple way to accomplish this would be to wait half a second or so after the user types each character, and if another character hasn't been typed in that half-second, assume the user is done typing and run the query. The problem with this approach is that users can have different typing speeds, and a one-size-fits-all delay of 0.5 seconds (or 0.3 or 1.2 or whatever) before searching may not be appropriate to all of them. If 0.5 seconds is too long, there will be an annoying lag before the search is run, and if it's too short, then the search box will be at least as bad as the ones I just finished griping about.

Because individual variation is the issue, we need a solution that varies with the user. This can be accomplished with some really basic artificial intelligence. At a high level, my method is to watch how fast the user types by counting the milliseconds between keystrokes, fit a simple probabilistic model (specifically, a Gaussian) to that data to describe the user's typing speed, and then use that model to determine the amount of time beyond which it's very unlikely for the user to type another character. Once that (probabilistic) upper limit has passed without any typing activity, I run the search.

Now for the details. That talk about probability probably made this sound more complicated than it is. Basically, I keep a running average of the amount of time between keystrokes, and that gives the best estimate of how much time there will be between this keystroke and the next. But this is an average, and it may be that some keystrokes are a bit slower and some are a bit faster than the average; to account for that I also track the running standard deviation of keystroke times, which is a measure of how much they vary. By the empirical rule (and the assumption that the Gaussian is an appropriate model for these keystroke times), we know that just about all keystrokes are going to be shorter than the average plus 3 standard deviations (sigmas). So my probabilistic upper limit is set at the average plus 3 sigma, plus a little fudge factor that (hopefully) doesn't add an uncomfortable delay, but prevents short hesitations from triggering a search.

the result