Finding the Balance Between Code Optimization and Maintainability

Where is the line between optimized code and unmaintainable code?

It's not really a line. It's barely even in opposition.

There exist complex algorithms that would be hard for the average developer to read or maintain. It's rare that you'll need such an algorithm, but if you really need it, then you know you need it.

And once that algorithm is well implemented, ideally you don't need to maintain it much, if at all. (Except where the algorithm is the business logic.)

A clean design will isolate complexity from code that might eventually need to be maintained (e.g., business logic). Build it as if it's going to be a generic library (in fact, look around to see if it already exists as a generic library!).

If you do find problems in your implementation of the complex algorithm, then you assign your best algorithm expert to fixing it, or your swap out that algorithm for another—because if it's sufficiently isolated, the rest of the code shouldn't depend on implementation details.

For garden variety code that doesn't need to be optimized by an obscure algorithm to extract the very maximum performance, leaning on standard tools and patterns can produce code that's reasonably optimized and readable and maintainable.

Using set/map/dict structures when you need to look things up rather than performing a linear find. Memoizing functions. Using algebra instead of iteration. Performing the entire query you need instead of performing a dozen queries of the same data (related: using proper joins). Creating a design that doesn't require unnecessary overhead to accomplish something that should be trivial.

All of the above and more can create code that's sufficiently fast that you'd need to perform precision measurements to detect the speed difference of any less maintainable optimizations, whereas ignoring the above and similar techniques can mean a web page that takes 90 seconds to update instead of 100ms (true story).

So the default code you write should be readable and first-pass optimized and maintainable. No “line” required, most of the time.

And the only time you should invest in more optimization is if you have identified a performance problem that needs to be addressed. Only in this case should you even consider adding complexity that impacts maintenance—and most of the time that complexity should be isolated to minimize its damage.

There's your threshold. “When the additional optimization is demonstrated to be needed.” But remember that doesn't mean you shouldn't write optimized code all the time, especially when it doesn't harm readability. Just not heavily optimized code that adds unnecessary complexity for no real benefit.

It shouldn't be a game of “how can I apply this obscure algorithm.” Instead you should be using standard tools that have reasonable performance by default, and only look for obscure algorithms when the normal approach won't do the job. (Or where there is no normal approach.)