I am running Freeswitch on a Guruplug using the latest debian.
As you say Ubuntu no longer support that version of the ARM processor but Debian still does
As for compiling I did it on the plug directly (I am booting from an 8Gig USB so space is not an issue)
It does take some time but since I am not smart enough to work out all these cross compiling thingy it has to do for me
For compiling I simply followed the Freeswithc wiki to install it for a Debian.
Freeswitch runs really well on my guruplug and I am sure it will run as well on your sheevaplug.