You shouldn't move things by setting position/rotation explicitly. What happens is that you move the object, but you don't update it's rotational velocity. When things collide with it, they won't know how fast it's moving. They will only know that they somehow started overlapping with it and will get pushed apart by a corrective force that is much smaller than the one it needs for a collision with a moving object.
Not quite sure what you are trying to do with the pin joint, but I think maybe you need some other kind of joint. A pin joint attaches the two anchor points on each body by a pin or rod so that the anchor points stay the same distance apart as when the joint was created.