This is due to the recursivity of it. I think you’ll might want to go over the mathematical derivation of this, as it might give you the proof you’re looking for (which wasn’t shown here). I’d start with the Wikipedia page: https://en.wikipedia.org/wiki/Bellman_equation