I have only had a slight problem with one ladder in particular. I'll not say which one.Most ladders you can go over the top and down the other side with ease. The one I mentioned, you have get on it and then climb around it horizontally to the opposite side. That is always a good trick if going over the top causes you to fall.
Your results may depend on a setting in the Options / Game Setting. I have mine set to Attach - Jump. So I just go up to a ladder, jump, and I am on it immediately. Then, in most cases, I can go up and over the top, even if it is a short ways up.
Another thing I do is to always go down a ladder facing out, away from the wall. The view is more impressive that way.And I can see what I am getting into before I arrive.