In this talk, we explain different roles that symmetries and approximate symmetries can play in machine learning models. We define approximately equivariant graph neural networks and we show a bias-variance tradeoff when selecting the symmetries to enforce. We explain how to see equivariant functions as gradients of invariant functions, and we show how to use these ideas in self-supervised learning.