Yep, I don't think you are pushing him off this hand. Overpair is a good hand. On the flop, no way is he folding. Maybe a push on the turn wins the pot, but on a board like that, I'm gonna be slightly concerned about a set, so I'm not usually pushing.
You were either gonna lose it all or win a small pot. If you pushed too hard early you might have pushed him off. He must have read you for 2 big cards. he wasn't smart enough to know he was beat than got super lucky. You had to call getting 5-1.