Getting around Silverlight 2 beta 1 error 4002 changing visibility

While working on Bumble Beegger we came across a daunting problem that only seemed to happen under certain conditions. Silverlight error message ErrorCode: 4002 ErrorType: ManagedRuntimeError Message: System.Exception: Error HRESULT E_FAIL has been returned from a call to a COM component.