In a 2004 Berkshire Hathaway annual meeting, Warren Buffett was fielding questions from people in the audience.

A 14-year-old from California stood in front of the greatest investor of our generation and asked the billionaire Buffett what advice he would give a young person on becoming successful. 

Buffett's response, eligible for any age group as it relates to business and career development, is still a classic. He said:

It's better to hang out with people better than you. Pick out associates whose behavior is better than yours and you'll drift in that direction.

As the famous saying goes, we are the average of the five people we spend the most time with. And depending on who you associate with at work, that can potentially disrupt your thinking, self-esteem, and how you make decisions.

Plain and simple, if you spend time with positive and optimistic people, you're likely to become more positive and optimistic. Do the opposite and hang around negative complainers who view life as a half-empty glass, well, you know where you're probably headed.

To cut your losses and keep your sanity, consider rooting out associates that no longer serve you as you move toward your True North. That's not to say ditch your closest peers and coworkers since you depend on them for work. What I am saying is to reevaluate your networks and filter out the bad in order to filter in the good. 

Who should you ditch?

In growing your influence as a leader, business owner, or professional so others will gravitate to your inner circle, there are four types of people that you want to avoid.

1. Ditch people who lack integrity.

Integrity is so crucial for success, Buffett once said that you should never hire someone without it, no matter how smart they are. People operating with integrity can be trusted; you never have to worry about their actions, or whether they're hiding anything from anyone.

While we may associate those who lack integrity as public figures who embezzle money or sexually harass women, it is much more subtle than that in your own workplaces. Watch for people who take shortcuts and consistently behave lazy, careless, or greedy.

Small concessions may fly below the radar especially for influential colleagues pushing for closing that big sale or maneuvering to get that promotion.

In a study published in Harvard Business Review, researchers found that if they could get people to cheat a little the first time, they'd be willing to cheat a bit more the second time, and finally cheat big the third time.

2. Ditch negative people

Negativity is damaging to the workplace. It may be found in gossip, lying, slander, deceit, selfishness, and pessimism. And the consequences can lead to distrust within a team, a decrease in employee engagement, or even liability issues if harassment takes place.

Negativity sucks energy from people and hampers productivity and performance. When persistent, a negative attitude in the workplace stifles creativity. Because of this, leaders at every level must be proactive in maintaining a culture of respect and positivity.

3. Ditch people who show no love of learning

Buffett credits his success to the fact that he is a learning machine. He estimates that he spends 80 percent of his working day reading and thinking. Buffett has what Stanford professor Carol Dweck calls a "growth mindset." In her research, Dweck found that people with a growth mindset believe that their most basic abilities can be developed through dedication and hard work -- brains and talent are just the starting point. She says, "This view creates a love of learning and a resilience that is essential for great accomplishment." 

On the flip side, avoid people with what Dweck calls a "fixed mindset." They believe their qualities are fixed traits and therefore cannot change, which has some risk factors. It can prevent important skill development and growth, which could sabotage your health and happiness.

Buffett was once asked how a person can get smarter. The Oracle of Omaha held up stacks of paper and said he "read 500 pages like this every day. That's how knowledge builds up, like compound interest." 

4. Ditch self-centered people who only look after themselves

As depicted in Buffett's biography, "The Snowball: Warren Buffett and the Business of Life," Buffett once was asked by Georgia Tech students about his greatest success and greatest failure, to which he responded: "When you get to my age, you'll really measure your success in life by how many of the people you want to have love you actually do love you."

He adds, "I know people who have a lot of money ... but the truth is that nobody in the world loves them....that's the ultimate test of how you have lived your life."

Buffett nails it with one final statement on the secret to being loved: "The trouble with love is that you can't buy it ... The only way to get love is to be lovable ... The more you give love away, the more you get."