You can enforce exact symmetry in neural networks with the right sort of model structure. For permutation invariance in particular, take a look at Deep Sets: https://arxiv.org/abs/1703.06114