Why is it that ABO3 perovskites generally do not exhibit negative thermal expansion (NTE) over a wide temperature range, whereas layered perovskites of the same chemical family often do? It is generally accepted that there are two key ingredients that determine the extent of NTE: the presence of soft phonon modes that drive contraction (have negative Grüneisen parameters); and anisotropic elastic compliance that predisposes the material to the deformations required for NTE along a specific axis. This difference in thermal expansion properties is surprising since both ABO3 and layered perovskites often possess these ingredients in equal measure in their high-symmetry phases. Using first principles calculations and symmetry analysis, we show that in layered perovskites there is a significant enhancement of elastic anisotropy due to symmetry breaking that results from the combined effect of layering and condensed rotations of oxygen octahedra. This feature, unique to layered perovskites of certain symmetry, is what allows uniaxial NTE to persist over a large temperature range. This fundamental insight means that symmetry and the elastic tensor can be used as descriptors in high-throughput screening and to direct materials design.