Staged build: Keeping build time reasonably low

Keeping the build fast is a necessity for CI to provide rapid feedback. Developers need to know quickly that the build has been broken in order to fix it and allow other team mates to work with a functional version. But what to do when building your project takes a long time? Say two hours? Or if you have extensive testing that involves use of another servers and it takes about a day to complete? Should you reject the continuous integration idea at all or is there a solution for you?