You have described pretty much exactly what I have been doing. I load the kernel from tftp, manually set bootargs (I have been using a mircosd card for my rootfs), and then use bootm to boot the kernel. I use the same procedure to boot the stock kernel (replace tftp with reading from nand) and it works fine.
I am having this same problem. The only differences I see are that I am using version 2010q1-188 for my cross compiler, and I am loading the uImage off tftp initially. The only thing that I have gotten to work for me is the image already on the internal flash.
It looks like it may not be fully uncompressing the kernel? As appears to jump to booting the kernel much sooner.