Comparison and Difference between VMFS 3 and VMFS 5

New Unified 1MB File Block Size

Earlier versions of VMFS used 1, 2, 4 or 8MB file blocks. These larger blocks were needed to create large files (>256GB). These different file blocks sizes are no longer needed to create large files on VMFS-5. Very large files can now be created on VMFS-5 using the new unified 1MB file blocks. Earlier versions of VMFS will still have to use larger file blocks to create large files.
Large Single Extent Volumes
In earlier versions of VMFS, the largest single extent was 2TB - 512 bytes. An extent is a partition on which one can place a VMFS. To create a 64TB VMFS-5, one needed to create 32 x 2TB extents/partitions and join them together. With VMFS-5, this limit for a single extent/partition has been increased to 64TB.

Smaller Sub-Blocks
VMFS-5 introduces smaller sub-blocks. Sub-blocks are now 8KB rather than 64KB as used in the earlier versions. With VMFS-5, small files (< 8KB, but > 1KB) in size will consume only 8KB rather than 64KB. This will reduce the amount of disk space stranded by small files. Also, there are many more sub-blocks in VMFS-5 than there were in VMFS-3 (32,000 on VMFS-5 compared to approximately 4,000 on VMFS-3).
Small File Support
VMFS-5 introduces support for very small files. For files less than or equal to 1KB, VMFS-5 uses the file descriptor location in the metadata for storage rather than file blocks. When these files grow beyond 1KB, they will then start to use the new 8KB sub-blocks.
Increased File Count
VMFS-5 introduces support for greater than 120,000 files, a four-fold increase when compared to the number of files supported on VMFS-3, which was approximately 30,000.

GPT

VMFS-5 now uses GPT partition table rather that MBR table as used by earlier version of VMFS extending the maximum partition size to 64TB which was limited to 2TB in earlier verions of VMFS.

Limitation of upgrading filesystem from VMFS-3 to VMFS-5

While a VMFS-3 which is upgraded to VMFS-5 provides you with most of the capabilities as a newly created VMFS-5, there are some differences.

No Uniform Block Size

VMFS-5 upgraded from VMFS-3 continues to use the previous file-block size, which may be larger than the unified 1MB file-block size.

No New Sub-Block Size

VMFS-5 upgraded from VMFS-3 continues to use 64KB sub-blocks and not the new 8KB sub-blocks. This can also lead to stranded/unused disk space. The upgraded VMFS-5 also continues to use the original number of sub-blocks from the VMFS-3.

No Increase to the Maximum Number of Files per Datastore

VMFS-5 upgraded from VMFS-3 continues to have a file limit of 30,720 rather than new file limit of > 100,000 for newly created VMFS-5.

Uses MBR

VMFS-5 upgraded from VMFS-3 continues to use MBR (Master Boot Record) partition type; when the VMFS-5 volume has grown beyond 2TB, it automatically and seamlessly switches from MBR to GPT (GUID Partition Table) with no impact on the running virtual machines.

Starts on Sector 128

VMFS-5 upgraded from VMFS-3 continues to have its partition starting on sector 128. Newly created VMFS-5 partitions will have their partition starting at sector 2048.

References:
VMFS 5 Upgrade from VMFS 3

Related Articles:
Configuration maximum comparison for VMware vSphere 5.0, 5.1 and 5.5
Minimum H/W requirements for VMware vCenter Server 4.x and 5.x
Comparison and Difference between vSphere 5.1 and 5.5