From the pic, it appears that the ladder bracket is not snug on the left side (as you stated), and may be:
1. The bracket is not bent to the contour of the tailgate (it kind of looks like this might be a contributor).. If you look at the bracket, Right by the "L" in Land Rover, it is centered...