After the November election, Casey Patten was so intent on staying above the partisan fray he commissioned a billboard for his Washington, D.C.-based sandwich chain that exhorted: "Less Politics, More Hoagies." Still, this week Patten's business, Taylor Gourmet, suddenly became the subject of boycotts and threats after Patten attended a small-business roundtable, called the Small Business Leaders Listening Session, with President Donald Trump. That blowback is as troubling as it is unsurprising.

Interactions between CEOs and politicians naturally attract suspicion and criticism. (Yesterday, the CEO of Uber announced he was leaving Trump's economic advisory panel for those reasons, after a massive #deleteuber campaign went viral.) But unlike CEOs of big companies, small-business owners don't approach meetings with the President as opportunities to curry favor for their specific companies or industries. Patten wasn't maneuvering to beat out Potbelly for a White House catering deal. And he wasn't lauding Trump's agenda or character. He was given the unique chance to tell the guy in charge what was troubling him and other small business owners, and he decided to seize it.

Small company owners need every powerful ear they can get. If they withhold counsel from a government official because he or she might be unpopular or controversial, the small business community suffers.

As a Washington, D.C.-based business owner, this wasn't the first time Taylor Gourmet has been invited to these sorts of meetings. Patten had previously participated in similar events with President Obama, with no fallout. He took the meetings with both Presidents very seriously. "I was never interested in a photo-op," says Patten, who spent most of last weekend preparing his talking points, after receiving the invitation Friday evening.

On Monday, Patten got the chance to go to the White House and present his concerns to the new President: He wanted to see a tax credit for companies that hire and train 16-to-24-year-olds. He thought every entrepreneur working with the Small Business Administration should be assigned a counselor to help navigate the red tape. Mostly he wanted to address the immigration issues that were stressing out his employees. "To understand my workforce is to understand a lot about diversity in the restaurant community," says Patten.

Then, unexpectedly, he found himself part of a group observing as Trump signed an executive order about regulations. When the photo of him shaking hands with Trump appeared on a Washington neighborhood blog, social media erupted in fury, including threats--"Watch your back." "I hope you die."--against the entrepreneur, his family, and his business. Patten spent the next day visiting his 11 sandwich shops to explain to his 350 employees--many of whom were confused and angry--why he met with Trump and what was discussed.

Deborah Carey, co-founder of New Glarus Brewing in New Glarus, Wisconsin, experienced something similar when she visited the Obama White House to relay her opinions--on one occasion to Obama himself--on issues like skilled labor and tax credits for major equipment. "There is a right wing radio station in Milwaukee that tried hard to vilify me and my business," says Carey. "I received very nasty letters. There were repeated attempts to boycott my brewery."

Like Patten, Carey describes herself as middle-of-the-road politically. (Although in 2014 there was talk of her running for governor of Wisconsin on the Democratic ticket.) She says if Donald Trump wanted to pick her brain she wouldn't think twice. "He can read my Facebook and see what I say about him," says Carey. "If he really sincerely wants to know what I think, more power to him.

"When the leader of the free world asks you for your advice, you have an obligation to go make your case for your fellow small businesses or the people in your community," says Carey. "It's your public duty."

And that's not true only for the leader of the free world. Amy Handlin, a long-time New Jersey assemblywoman and author of the book Be Your Own Lobbyist, says small businesses owners must try to get in front of any politician who will listen to them. "Whether you agree or disagree with your representative from town hall to Capitol Hill to the White House, those representatives have tremendous influence over the environment of business," says Handlin. "Withholding information that you know as a business owner is completely counterproductive because it will only encourage bad decisions."

Of course most of Patten's troubles arise from that photo with Trump at the executive order signing, an event that caught him and the other visiting business owners by surprise. Such optics implicitly convey support for the President and his agenda. Ideally those who were not wholly supportive wouldn't have participated. But to expect a small business owner, standing in the Oval Office surrounded by officials and his peers, to decline to have his photo snapped with the President is probably unrealistic.

Joy Weatherup Anthis, another small business owner present at the Trump meeting, says the order signing occurred only after a substantive 90-minute discussion of business subjects. Anthis, founder of JWA Construction Management, in Syracuse New York, is a Republican and voted for Trump. But she agrees with Carey that when it comes to representing the small-business community, she is bipartisan. "I would have said the exact same things I said to [Trump] to Obama or Bush or Clinton," says Anthis. "I would go in there and meet with any of them."

Anthis sympathizes with Patten, who she points out is vulnerable because he operates a consumer-based business in the public eye "Somebody made a comment in our local paper that I am going to boycott JWA," she says. "But there's not a lot of individuals that need a construction management company that does municipal work."

Fortunately for Patten, he says the calls for a Taylor Gourmet boycott haven't actually made a dent in his business. Along with a chorus of local customers, it turns out another small business owner has come to his defense--D.C.-based celebrity chef Jose Andre, who tweeted that showing respect for the President is quite different than advocating on his behalf.