Build fail only on the build server with delay-signed assemblies

I got that error from our build machine. It crashed wonderfully and would tell me that an assembly could not be loaded. After 1 hour of search I finally found the problem.