The lore of innovation is filled with stories about remarkable persistence.  The origin story of Post-It notes involves Spencer Silver spending years telling colleagues about a weak adhesive he created while trying to develop a strong one.  Eventually, his adhesive came to the attention of fellow 3M employee Art Fry who first used it to create bookmarks for his hymnal. 

Similarly, after James Dyson had the insight to use the design of an industrial cyclone to eliminate the bag in a vacuum cleaner, he spent five years developing prototypes before he was ready to bring the product to market.

So, perseverance is an important part of innovation and success in general.  Indeed, much has been written about the concept of grit that Angela Duckworth has written about extensively.  Persistence and focus on an important goal are key aspects of grit. 

But, successful people don't just stick blindly to an idea if they are not making progress.  An alternate perspective comes from research on sunk costs.  A sunk cost is any time, money, effort, or other resource that you have already put into a project.  Economic theory suggests that sunk costs should not affect your decision about whether to continue to pursue a project, because those resources are already gone.  Nonetheless, many people will continue to work on a project far longer than they should, simply because of the amount they have invested in the project already.

Research by Richard Larrick, Richard Nisbett, and James Morgan surveyed researchers working for a university and found that those who were willing to walk away from projects that were not succeeding even though they had already spent a lot of time or effort on them were more successful than those who persisted on projects just because they had already been working on them for a while. 

This conflict suggests that the most successful people are the ones who are best able to persist on promising projects while being able to walk away from projects that are unlikely to bear fruit.

So, how do you know that it is time to walk away from a project?

You have a better alternative

The fundamental problem with persistence is the opportunity cost with continuing on a project.  Whenever you devote a limited resource to a project, you have necessarily taken that resource away from other projects you might pursue.  That means that you have to evaluate continued effort on a project by taking into account what you might accomplish if you devoted your effort elsewhere.

As exciting as it can be to swing or the fences on a project, you may find that there are other projects that have a high chance of success, a decent reward for completing them, and a shorter time horizon for completing them.  When you have a better alternative, you should strongly consider putting a project on hold to focus on that alternative.

Notice, though, that the idea is that you are putting the long-term project on hold.  Just because you walk away from a project for a while does not mean that you are giving up on it entirely.  You are just re-prioritizing in the moment.

You are making progress on something

The idea of prioritizing relates to a second aspect of persistence.  Just because you continue working on a long-term project does not mean that should be the only thing on your plate.  Successful people do persist on project, but they also make sure that the collection of things they are working on includes both projects that can be completed in a timely fashion as well as innovative projects with a big potential reward that will take longer to complete.

If you find that you are not bringing any projects to completion, then it is time to rethink the amount of effort you are spending on a long-term project.  You might choose to walk away from that project entirely, but at a minimum you need to take on more projects that can be completed quickly.  It is best to think about the work you do in your career as a portfolio of resource investments rather than an all-or-none bet.

The upside of success is small

As rewarding as it can be to succeed at solving a problem, it is important to ensure that the problem you are working on remains an important one to solve.  Research on negotiation suggests that people are often motivated to reach an agreement, even if they would be better off economically by walking away from the negotiating table.  Likewise, you may be motivated to complete a project simply to ensure that you finish what you started.

However, as psychologically satisfying as it may be in the moment to reach a satisfactory end to a project, it is also important to keep an eye on the value of that success.  Sometimes the world changes while you are engaged in a project.  Other technologies or companies may come along that dampen the value of what you are working on.  It is useful to keep an eye on the likely value of the success of your project.  At some point, it may be worth walking away from a project just because the ultimate success of that project no longer has the significant upside you believed it would have when you started the project.