Tina, my initial thought is that this may not be a gender issue, but rather a weight issue. I am curious to know if there is a difference in size and weight between the female and the male riders. If there is, I suggest that you have your vet give your horse a through exam, and maybe even have a qualified equine chiropractor examine him.
If the weight is not significantly different between the male and the female riders, I would suggest doing some lunging sessions with you handling the lunge line and a male rider on your horse’s back. Keep a close eye on his level of relaxation: are his ears soft and turning back and forth? Is he soft and long in the neck? Is his tail relaxed? Reward signs of relaxation and calmly work through signs of stress. Have your male riders give crest scratches, and, before mounting and just before dismounting, have them give him a treat. Going forward, all “feel good” experiences (food, going out to pasture, grooming, etc.) should be conducted by the male rider whenever possible. Good luck!