When should you take performance into consideration?

Before publishing my previous post on rewriting an if, I knew some people would hate it, because the refactored construct is less performant. Although I think performance is important, relevant performance improvements are, apart from in tight loops, hardly ever to find in language constructs. To put it more bluntly, they are a waste of time.