You'd have to be very smart to come up with that

When I look at system designs that have produced failed or extremely problematic systems, a lot of them fall into the category that I think of as "you'd have to be very smart to build something so problematic". A "normal person" would've produced a straightforward system that basically works, but someone who's very smart managed to build something with a staggering level of complexity that either doesn't work or has extremely problematic failure modes that a simple system couldn't. This isn't really an original observation, see (just for example) this comment from a former Google engineer who was very senior at Google: https://twitter.com/apenwarr/status/1264032934132158464 

I don't want to cite any specific examples of systems that I know of, so instead I'm going to cite a comment that I think is in the same class, something I think is wrong that you'd have to be unusually smart to come up with. And BTW, I don't mean to pick on this comment in particular, it just happens to be one that was made in public recently.

I did an analysis of car safety and then made an off-handed comment that one of the silliest things I've ever done was to buy a used car around the turn of the century and then drive it forever.

Fatalities per mile driven are down ~35% since ~2000 and this underestimates the impact because fleet age is up maybe 10% to 40% (different data sets give you different results here and none are really weighted by miles driven, so getting the right number here is non-trivial). Additionally, I didn't own a particularly safe car for its age, but I could own a particularly safe car for its age, which would make the differential between my old subcompact car and current-generation safe car looks like it's maybe 4x per mile. If you think this is too extreme, you can de-rate this to 2x and that won't change the result, but I think 4x is in the plausible range if we look at what changes have occurred since 2000. Just for example, depending on what estimate you look at, electronic stability control alone reduces road deaths by 30% to 45%. The upper end estimate is 1.8x just from one technology. If you add in everything else, 4x isn't an unreasonable mid-range estimate, a top-end estimate would be higher.

Someone, who I think is obviously very smart, dropped in to say that it probably wasn't worth my time to even shop for a newer car since, based on the NHSTA's value of a life at $10M and, "presumably" that safety improvements added each year are near the cost-benefit threshold, you only buy "a few hundred dollars" of safety, making the value in safety less than the time cost of buying a new car as well as less than the maintenance cost of having an old car.

I think this line of reasoning wouldn't have occurred to any normal person. If you can take your greatest risk of dying (which is driving for people across a wide age range) and reduce it by 4x by spending well under 0.5% of your income (amortized, and I think this is the right number for anyone with my title at a big company, and I don't have a particularly fancy title), that seems like an easy choice.

But then if you want to do the computation, I don't think any of this person's starting points make sense, either. If we're doing the personal computation for whether or not we should buy a car, the NHSTA's value of a life is meaningless to us. We have some income and there's a decision to trade off some fraction our income for some stochastic increase in lifespan. If, tomorrow, the NHSTA decides to value a life at $100M instead of $10M, that doesn't have any impact on what tradeoff we should make. The other part of the argument relies on the assumption that car manufacturers are doing some kind of computation that must involve the depreciation of vehicles so that the marginal safety impact of a newer vehicle is traded off against the cost of an older vehicle. I don't think this makes sense, but even if the comment made sense, we could just do the more direct computation of expected life expectancy change vs. cost, there's no reason to bring multiple levels of extra indirection into it that can only possibly make the computation less accurate.

I happen to have already done a rough computation for the cost of driving a motorcycle vs. commuting time saved, and the computation here is a similar kind of computation, so I know the relevant baseline numbers. This computation varies a lot by person and I think it's pretty misleading to use averages and it's difficult to get the appropriate non-average number in many cases, but here's a rough sketch of the computation.

First, why it's problematic to use averages: if you look at up middle-aged person's risk of death, things that will be pretty high up on the list include: car accident, heart attack, drug overdose, suicide, firearms, and stroke will also be somewhere up there if they're at the upper end of the middle-aged range, and you'll also see other accident risk if you aggregate it together.

As we've noted, car accident fatality risk can easily be changed by buying a different car (and surely also varies across drivers, but it's hard to say how much you should discount or increase risk based on your driving style). Heart attack risk is something people have a lot of control over (stroke, much less so) and I'm in a very low risk bucket for that. Drug overdose risk even more so, the risk for me seems close to zero. I would guess that's also true of suicide for me, although maybe I'll be surprised and likewise for a firearm death. But that means that, for me, most of the major causes of death outside of driving a car are relatively low, which makes my relative risk of dying in a car accident higher than we'd expect from averages.

Anyway, if I make a rough guess at what this means and we ignore covid, if I were to die in the next year, it doesn't seem unreasonable to guess that I'd have about a 20% chance of dying in a car accident (after removing motorcycle, pedestrian, and other non-occupant deaths). That's assuming I'm driving an "averagely safe" car in an average fashion. Let's say, hypothetically, I would move from a car that's half as safe as the average car to a car that's twice as safe, so assuming I'm an average driver, my relative odds of dying in the old subcompact are  (0.8 +  (0.2 * 2)) / (0.8 +  (0.2 * 0.5)), or about 1.33x my odds of dying when driving a relatively new, safe, car. If you ask a developer at a big company, would you want to earn an extra 0.5% income by taking on a 33% increased risk of death, I think most people would say no.

You can also carry out this computation from a life-expectancy standpoint and you'll get what I think is a similarly obvious answer there. I'm not going to write this down here because the computation is longer (you really want to do a summation over some gross data that can't be summed symbolically), but feel free to work through it yourself for parameters that match your life, if you're interested. And BTW, I think that life expectancy computations understate the obviousness of the choice, since, in a naive life expectancy calculation, each year has the same value, and having a significant probability of having your life cut severely short seems worse than losing the equivalent number of expected years at the end of your life.

But my point isn't that you can see that this is wrong if you carry out some kind complex calculation to see that this smart comment is wrong, it's that smart people are capable of coming up with convoluted but incorrect explanations that wouldn't even occur to a normal person. Some people seem to do this kind of reflexively?

This is one reason I'm not super excited about interviews designed to pick out "smart people". There's a kind of verbal quick-wittedness and concomitant ability to come up with rationalizations for anything that codes as smart that's easy to spot in interviews. The kinds of smarts that are actually useful on the job are much harder to spot in interviews, so interviews at places that are really obsessed with hiring the smartest people often turn up people who are smart in a way that doesn't really lead them to the right answer.