Stacks up to
Blue Gems can be used to turn Chester into Snow Chester by placing at least one in each of his inventory slots on a full moon. Snow Chester's special ability is preserving food longer like an Ice Box.
Reign of Giants
In the Shipwrecked DLC, Fishing with Trawl Net in Deep Ocean has a small chance to catch a Blue Gem. The chance is higher during Hurricane Season and Dry Season. Magma Piles found in Magma Fields also have a chance to drop Blue Gems when dug.