The single-family homes are a rarity in America's wealthiest ZIP Code.