To get started, open VMware, click on the New Virtual machine option, select Custom (Advanced) and click Next.
In the next step select an installation device (e.g. DVD ROM) or add a disk image.
When selecting a guest operating system choose Windows Server 2008 x64. Or select Windows Server 2008 in case you wish to install a 32 bit version.
In the easy installation section, add a product key, name and password. Note: you can leave the product key blank and proceed in case you wish to evaluate server 2008 or activate it later.
Specify a location to save your virtual machine. Make sure that the partition you use should have at least 20GB of free hard disk space.
Next Specify the number of cores to allot to the virtual machine, make sure that you do not allocate all the cores as that can led to system instability.
Select the amount of memory of virtual machine. I would recommend allotting not more than half of the system memory to avoid instability and crashes.
When configuring network settings, choose NAT or bridged networking to make sure that the virtual machine is able to connect to the internet. A bridged network is also more useful for emulating a domain environment to connect the server with virtual client operating systems. In the next step, create a new virtual hard disk, choose a hard disk type and allocate space. At least 20 GB should be allocated to avoid running out of disk space. Although the initial size requirement is about 5 GB, however, installation of updates, software and server components is likely to consume a lot more space.
After that, verify that all parameters are properly configured and click Finish. The operating system will normally boot automatically, however, if it does not, click any key to continue.
The rest of the installation will take place according to normal procedure within VMware.
You will be prompted to change password at first login after which you can start configuring and using Windows Server 2008 R2 x64. The same procedures more or less apply for installing a 32-bit version. However, unlike a 32-bit OS a 64-bit machine will not install without configuring the bios settings (as mentioned before).