Karl Seguin also discusses the use of Object Relational Mappers, and Data Mappers and how he has a growing dislike of them.
Now, I don't think you can lump *all* ORM's into that camp, some are very intelligent about when they retrieve data and how much they retrieve, but some aren't. I don't think you can say whether using an ORM is good or bad, it's all dependent on the situation.
I am a proponent of SQL, I think it's the best tool for the job, but sometimes, for some projects, it's not feasible to write all the SQL by hand, which makes ORMs handy. I think that working with individual objects (with simple enough data models), is negligibly slower than using SQL in things like desktop applications. However, when you get into applications under heavy load such as high-traffic web applications, the convenience of not writing some SQL is quickly outweighed by the performance effects. Though some ORMs implement pretty advanced caching strategies, so that's something else you'd have to roll on your own to help with performance.
The problem comes down to the features of ORMs being useful, and at some point even if you dont' use an ORM, you're probably going to try to add some of the same features to your codebase. Pretty soon your data access layer is just as cumbersome and just as "slow". The trick is finding the right balance of features to raw performance, and tailoring your code to the project at hand.