Popeye's gym sign indicates it is for women. However, the locker room contains a shaving soap mug and a small shaving mirror loosely hanging from a peg in front of a locker. (The mirror would not be suitable for a woman's needs.)
When Bluto blocks Olive from entering Popeye's gym, his body blocks the entire doorway. When he later goes through the narrower side entrance (access to the locker room), he walks right through the door without stooping or turning sideways.