You should always dedicate fixed amount of RAM for Xen dom0.
This can be done by specifying "dom0_mem=512M" option for Xen hypervisor (usually xen.gz) in grub.conf/menu.lst. This makes sure the initial size of memory allocated for dom0 is 512 MB (replace with the amount of memory you want), and the rest of the RAM is available for other guests in Xen hypervisor. See this grub.conf example for GRUB1:
title Xen 4.1.0 / pv_ops dom0 kernel 2.6.32.36
root (hd0,0)
kernel /xen-4.0.gz dom0_mem=512M loglvl=all guest_loglvl=all
module /vmlinuz-2.6.32.36 ro root=/dev/sda2 console=hvc0 earlyprintk=xen nomodeset
module /initrd-2.6.32.36.img
The next step is to configure xend to make sure dom0 memory is never ballooned down while starting new guests.
This can be done by editing /etc/xen/xend-config.sxp and modifying the "dom0-min-mem" option to (dom0-min-mem 512) and also change the "enable-dom0-ballooning" option to (enable-dom0-ballooning no). These options will make sure xend never takes any memory away from dom0.
After making these changes to grub.conf and to xend-config.sxp, reboot the system. After reboot you will notice dom0 has only 512 MB of memory, and the rest of the RAM is available in Xen hypervisor as a free memory. You can run "xm list" to verify the amount of memory dom0 has, and "xm info" to verify the amount of free memory in Xen hypervisor.