While men might not think twice before taking an afternoon off here and there to hit the links, women appear to be much more self-conscious about up and leaving work.

That's a finding from a survey conducted by Karen Firestone, the CEO of financial services firm Aureus Asset Management. Firestone summarized her findings on the HBR Blog Network.

The Guilt Test

The survey split executives with at least 15 years of experience into all-men and all-women groups. She asked whether and how often they leave work early on a weekday afternoon to play golf or participate in other leisure activities, either to serve business ends or just for fun.

Firestone only received 40 responses on her survey, which obviously isn't a massive sample set. But the findings do pass the sniff test, sounding like what you might expect.

The men who responded to the survey, at a clip of 90 percent, take the occasional weekday afternoon off for leisure time. That, compared to 40 percent for their women counterparts. Both genders would used this time away from the office for business and personal time at about an equal rate. And those who didn't leave early occasionally--again, overwhelmingly women--said work was the reason.

This suggests women leaders "feel less comfortable being away from work than their male peers, who have generations of experience with such pastimes," Firestone writes. In other words, even as women see progress in assuming leadership positions, they still have to battle perceptions once they get there.

Feelings About Work and Labels

This isn't a new idea, but serves as further support for it. The Pantene advertisement below, which went viral in November and December, touched on similar themes.

That's not to say men don't face any similar issues. One of the revelations about Twitter co-founder Jack Dorsey in Nick Bilton's book about the company's rise: Dorsey apparently ruffled some feathers in the company's early days by leaving the office for a 6 p.m. yoga session daily.

The book revealed plenty of Dorsey's other faults, but if startup culture scoffs at leaving work at 6 for yoga appointments, that's a real problem regardless of gender. But at least Dorsey felt he had the right to leave and did so, something woman leaders appear to struggle with.