Interviews in practice, part 2

There's this old Tweet by Zach Holman that I think about a lot that reads "Programming interviews would be wildly different if people realized that most tech jobs are pretty boring and easy".

I agree with the Tweet and was reminded about this recently when I found out an infra team that's a cousin to my team has been trying to hire a "performance engineer" for a maybe half a year or a year now. My manager was on their last hiring panel and told me that they rejected the candidate and said something like "we really want someone more like Dan Luu". While that's flattering, I think it's a bit ironic since I think they would've rejected me if I interviewed!

With no other information, this is a fairly safe bet, since I basically have the highest rate of interview failure of anyone I know (probably true if you adjust for confidence).

But in particular, they rejected the candidate for being too specialized. If anyone's too specialized, it's me! Almost all of my background is in chip design and verification (before my current job, I'd spent one year in a software role and the rest of my time in hardware roles). For someone who makes chips, I have an incredibly broad background in various areas of chip design (logic, microcode, verification, etc.). But for as software position, you'd probably consider someone with over a decade of experience writing assembly to pretty specialized and hardware is even more specialized than that.

I think the hiring panel said that should hire someone like me is that the team doesn't need enough performance engineers to hire a specialist and, since taking this job, I've worked on a pretty wide variety of problems, from narrow code optimization, to JVM tuning, to applied stats, to debugging miscellaneous problems for various teams, etc. Last Friday, I did some basic performance tuning of a kind I'd never looked into before and found something which appears to be worth $3m/yr to $5m/yr, which is pretty normal for the kind of things I've been turning up. The thing is, I hadn't done any of these things before this job (except for code optimization, but I've been optimizing JVM code here, which is quite different from anything I'd ever touched before).

Despite (in retrospect) pretty obviously being able to do my job, I was actually really surprised when I didn't fail my team's interview. I got asked a question about how to debug a certain kind of performance problem, I said I'd look into X, Y, and Z. My interviewers (this was a panel interview asked what tools I'd use and I told them that I'd google it, but there must be some kind of tool for each one, and if they didn't mind me using my phone, I could probably give them the correct incantation within a couple minutes.

I've seen enough of these interviews to know that this kind of response will almost always result in a rejection, often with some kind of commentary about my deficiencies attached. For example, something vaguely similar happened when I interviewed at Jane Street, and someone later told me that they were glad they hired person X instead of me to build Y since I clearly didn't have a lot of programming experience and they built very high performance systems where you had to have a lot of experience (they cited millions of QPS through a system; of course how hard this is depends on the exact requirements, but I've worked on systems with many orders of magnitude higher performance as measured by QPS, including a system that ran on a million machines (at the time, probably much more now) with greater QPS on each machine).

Unlike most teams, my team was ok with an interviewee saying "hmm, I'd Google it". I've heard people say that they're ok with this, but when I've actually said this it's pretty clear that most people are not ok with this. Of course it depends on what you say this about, but there are things that people consider foundational things that everyone should know, and if you say this about one of those things, you're likely to get a rejection.

I think the big conceit here is that what we're doing is somehow incredibly hard and you need to be some kind of wizard to do it and if you don't already have the arcane knowledge, you're not a wizard. But... it's just not that hard. On JVM tuning, I knew that I didn't know about the JVM when I took the job, so I asked Nitsak Wikart (a well-known JVM expert) for suggestions on reading material, read the material, and then immediately found a bunch of tuning changes in our systems, each worth hundreds of thousands to millions of dollars a year. Now, if that was it, you might be able to argue that I'm just a wizard, but the next thing I did was write a tutorial (internal to the company, since it has specific information on how to get metrics from our dashboards, etc.) explaining how anyone can do this themselves, and I've since heard from people saying that they did the exact same thing. It took maybe a day to write the tutorial, which indicates that this stuff just can't be that hard -- if someone can read a tutorial written in a day and then go find a million dollars of performance waste, it just can't be that hard. I haven't gotten around to writing the tutorial for every kind of performance work I've done yet, but I've written three tutorials on three different topics and the same thing has happened for each kind of tutorial -- people with little to no experience doing performance work have been able to pick up the tutorials and do valuable work.

Afterward reading these tutorials and doing some of this kind of work, people have told me that it was much easier than they expected. The perception that this kind of thing is hard and unapproachable is one of the hardest things about it, maybe the hardest.

We, and I think most other companies, reject a lot of people and put a lot of load on our hiring pipelines by asking people to have all sorts esoteric knowledge to solve problems when people could just Google that stuff on the job, or failing that, email someone asking for a links to things to read.

Appendix!

This isn't to say that all performance work is easy. For another company, I consulted on a system that targeted 40ns per transaction, which takes a bit of care because this means you can't afford, on average, one cache miss per transaction (you're looking at roughly 50ns when going to RAM, plus the overhead of the failed lookups in the cache hierarchy, on the order of 40 cycles for a Skylake), but, in a sense, even this kind of hardness is just a matter of knowing what gotchas to be careful of and making sure you don't do anything on the list if things you should avoid.