I have been working on a lot of projects where I utilized open source third party libraries such as NHibernate and Castle Framework. When the libraries do not depend on each other there is no problem.
The problems starts when you add a third library such as the MvcContrib WindsorControllerFactory, since they might be compiled towards a older version than the latest version of its dependencies. In that case, you need to start compiling stuff and get the sources for everything. And then you need to manage these dependencies in an organized manner.
When your project depend on a handful libraries the task is manageable, but when the number of dependencies grows greater the task becomes very time consuming and sometimes very frustrating.
I did evaluate Hornget a while back, which is a package manager initiative developed by the Scottish .NET user group. However, I believe that they didn't really get all the way in their efforts as it didn't really work in a satisfying manner.
Now, Phil Haack has a post about a coming package manager that is currently in alpha. Check out his post and also watch the short intro video on the NuPack Codeplex website. It has a PowerShell console as well as a Visual Studio 2010 add on. It certainly looks extremely promising!
I am looking forward to the RTM of NuPack!