At Loss for Words

I, for an example, often work with and maintain few Windows 2003 32-bit servers with 8 GB of RAM and even one with 16 GB of RAM. And even some of 32-bit server processes running there will happily use more than 4 gigabytes when necessary. How is it possible? Or rather ask: Why shouldn't it be possible?

The numbers may seem relevant because 2³² bytes is ~4 GB, but in fact this is just a coincidence. How could even it be related? Number of bytes of memory available to the computer is related to address bus width ...not to a native register/operand size.

Do you remember how much memory was 16-bit DOS able to address? It was more than 2¹⁶ bytes, wasn't it?

Here is a short checklist, follow me:

• http://technet.microsoft.com/en-us/library/cc758523(WS.10).aspx
  Comparison of Server 2003 editions. It says that 32-bit Datacenter Edition can use 64 GB RAM ...hmm, what, really?
• http://en.wikipedia.org/wiki/Pentium_Pro
  The first 32-bit Intel processor with 36-bit address bus. AHAAAA
• http://en.wikipedia.org/wiki/Physical_Address_Extension
  This one describes how it is typically used, you can skip this one.
• http://en.wikipedia.org/wiki/Virtual_memory
  If you know little about programming and what a pointer size is, you may find this article very interesting.

So, when the address bus on your old Pentium Pro is 36 bits wide, this means 2³⁶ = 64 GB of addressable memory. 32-bit Windows NT5 kernel certainly can make use of that. Then why does your Windows XP shows 3.5 GB of RAM instead of 4 GB, when you bought those two fancy cheap 2 GB sticks? Or even more? I'll tell you. It is because of crappy drivers and poor driver programmers!

It was like 2003 or somewhere around when few enthusiasts started building their Windows XP machines with 4 GB of RAM. And their machines started to crash. And they all said "shitty Windows XP". But the professionals already had the experience. And the solutions was easy: do not let poor drivers in. But the users didn't listen. And crashing grown. To solve this problem, Microsoft released SP1 (or was it SP2, not sure) in which the virtual address space was hardwired and limited to 4 GB. Problem solved. Now users do not complain about crashing, but about being limited to 3.5 GB of memory or so.

Nice story, isn't it? But I didn't actually answered the question...

Why only 3.5 GB?

Actually the answer can be deduced from reading checklist above, but I'll tell you. Since this 4 GB limit is imposed because of device drivers, it is imposed on a level where device drivers map memory of devices managed by them into the virtual address space. The buffers on your soundcard needs to be assigned an addresses so the driver can pass them the audio data. The large memory on your video accelerator needs to be assigned an addresses so the driver can write there your textures, geometry and shaders. Remaining addresses, up to 4 GB, are mapped to plain RAM. As some of this virtual address space is already taken by drivers, you won't get all the 4 GB of RAM, there is nowhere to map it.

And one bad news at the end, the situation hasn't changed with Windows Vista.