Disobedient robots may sound scary, but even scarier is a robot that follows every human command without fail.

Researchers at Tufts University's Human-Robot Interaction Lab have taught robots how to question commands that would have negative consequences--like walking off the edge of a table--an important step in the development of artificial intelligence. Robots can even change their minds after initially refusing a direct order, provided a human can offer a logical reason as to why the robot should override its first response.

To see one of Tufts's researchers convince a robot to follow his command after the robot's initial refusal, check out the video below.

\r\n

While there are obvious benefits to developing increasingly sophisticated A.I. technology, there are also some not-so-obvious risks. Of particular concern for Tufts professor Matthias Scheutz is what happens when humans begin developing relationships with so-called \"social robots\" used in elder care and other situations where companionship is one of the primary functions.

\r\n

\"These are contexts where you use robots with vulnerable populations, such as people who might have dementia,\" Scheutz says. \"These people are even more likely to invest emotions and invest in the relationship, even though the robots cannot reciprocate.\"

\r\n

French robotics company Aldebaran bills its Pepper robot as \"the first humanoid robot designed to live with humans.\" Originally created for Japanese mobile phone company SoftBank Mobile as a device to greet and welcome shoppers in stores, Pepper can recognize certain human facial expressions and react accordingly. For example, if the robot senses someone is sad, it can play their favorite song to try to cheer them up, or play interactive games through a display tablet mounted on its chest. Japanese industrial automation company Aist makes an interactive robot that looks like a seal called Paro. Used in elder care settings, it has been shown to improve people's moods.

\r\n

As robots like Pepper and Paro become increasingly sophisticated and interactive, however, there is a significant risk that they can cause more psychological harm than good, according to Scheutz. \"People might interpret the robot's gestures or eye gaze in a particular way and form expectations based on that, and the robot will inevitably disappoint those expectations,\" he says. \"People will be tricked into believing that the robot has capabilities and something inside it that it doesn't.\"

\r\n

Even for people who don't suffer from any mental condition, there are risks involved in using robots for companionship, Scheutz says. \"If you take people who are just lonely, and they project something into this robot that's not there, it's going to be very frustrating.\" 

\r\n

Though social robots are not available yet for consumer purchase in the U.S., robotic machines like the Roomba vacuum have been around for years, so the technology is already making its way into U.S. households. Scheutz believes the earliest applications for other robots in homes are likely to be basic services like cleaning and delivery, before \"companion\" robots become a consumer item. \"Eventually, we will notice that we have a whole bunch of robots in our homes that do different things, and maybe they'll talk to each other,\" he says. 

\r\n

What Scheutz doesn't want to see is robotics companies throwing around the term \"emotional robot\" when referring to any product. It could create confusion he thinks might eventually lead to humans experiencing psychological harm.

\r\n

\"It's absurdly bogus,\" he says. \"There's nothing in a robot that has anything to do with emotions.\" 

","inc_code_only_text":null,"inc_pubdate":"2016-02-04 16:40:00","inc_promo_date":"2016-02-04 16:40:00","inc_custom_pubdate":null,"inc_feature_image_override":"","inc_feature_image_background_color_override":null,"inc_show_feature_imageflag":true,"inc_feature_image_style":"pano","inc_image_caption_override":null,"inc_autid":0,"inc_typid":1,"inc_staid":7,"inc_serid":0,"inc_prtid":0,"inc_activeflag":true,"inc_copyeditedflag":false,"inc_flag_for_reviewflag":false,"inc_lock_articleflag":false,"inc_react_displayflag":true,"inc_filelocation":"graham-winfrey/robots-are-saying-no-to-humans-but-thats-not-the-problem.html","inc_override_url":null,"inc_hide_article_sidebarflag":false,"inc_custom_sidebar":null,"inc_show_read_moreflag":true,"inc_display_video_at_bottomflag":false,"inc_autoplay_videoflag":true,"inc_full_width_read_moreflag":false,"inc_custom_footer":null,"inc_custom_teaser":null,"inc_hide_video_prerollflag":false,"inc_custom_css":null,"inc_custom_javascript":null,"inc_canonical_url":null,"inc_meta_keywords":null,"inc_column_name_override":null,"inc_newsworthyflag":false,"inc_notepad":"","inc_track_changesflag":false,"time_updated":"2018-05-17 11:50:03","channels":[{"id":7,"cnl_name":"Innovate","cnl_filelocation":"innovate","cnl_featuretype":"None","cnl_custom_color":"9DC786","cnl_calculated_color":null,"cnl_contributor_accessflag":true,"cnl_custom_article_footer":null,"cnl_global_nav_background_color":null,"cnl_global_nav_background_gradient_start":null,"cnl_global_nav_background_gradient_end":null,"cnl_iflid":0,"sortorder":null}],"categories":[],"primarychannelarray":null,"authors":[{"id":3828,"aut_name":"Graham Winfrey","aut_usrid":631397,"aut_base_filelocation":"graham-winfrey","aut_imgid":359288,"aut_twitter_id":"GrahamWinfrey","aut_title":"Senior editor, Inc.","aut_blurb":"Graham Winfrey is a senior editor at Inc.com. He was previously a reporter at IndieWire, a staff writer at Inc., and an associate editor at Private Equity International magazine. He has written for Business Insider, CNBC, Slate, The Week, and Worth, among others.","aut_footer_blurb":"Graham Winfrey is a senior editor at Inc.com. He was previously a reporter at IndieWire, a staff writer at Inc., and an associate editor at Private Equity International magazine. He has written for Business Insider, CNBC, Slate, The Week, and Worth, among others.","aut_column_name":null,"aut_atyid":1,"aut_newsletter_location":null,"authorimage":"https://www.incimages.com/uploaded_files/image/100x100/WinfreyHeadshot_359288.jpg","sortorder":null}],"images":[{"id":80192,"sortorder":null}],"inlineimages":[],"photoEssaySlides":null,"readMoreArticles":null,"slideshows":[],"videos":[],"bzwidgets":null,"relatedarticles":null,"comparisongrids":[],"products":[],"keys":["Innovate","Graham Winfrey","Inc.com staff writer"],"meta_description":"One of the real dangers for human-robotic interaction is making robots seem more intelligent than they are.","brandview":null,"internationalversion":[],"imagemodels":[{"id":80192,"img_foreignkey":"200365378-010","img_gettyflag":true,"img_reusableflag":false,"img_rightsflag":false,"img_usrid":0,"img_pan_crop":"{\"x\":0,\"y\":35,\"h\":928,\"w\":2000,\"img_h\":1509,\"img_w\":2000}","img_tags":null,"img_reference_name":"getty_200365378-010_200015092000928035.jpg","img_caption":null,"img_custom_credit":null,"img_bucketref":null,"img_panoramicref":"getty_200365378-010_200015092000928035.jpg","img_super_panoramicref":null,"img_tile_override_imageref":null,"img_skyscraperref":null,"img_gallery_imageref":null,"credit":"Getty Images","sizes":{"panoramic":{"original":"uploaded_files/image/getty_200365378-010_200015092000928035.jpg","1230x1672":"uploaded_files/image/1230x1672/getty_200365378-010_200015092000928035_80192.jpg","1940x900":"uploaded_files/image/1940x900/getty_200365378-010_200015092000928035_80192.jpg","1270x734":"uploaded_files/image/1270x734/getty_200365378-010_200015092000928035_80192.jpg","0x734":"uploaded_files/image/0x734/getty_200365378-010_200015092000928035_80192.jpg","1150x540":"uploaded_files/image/1150x540/getty_200365378-010_200015092000928035_80192.jpg","970x450":"uploaded_files/image/970x450/getty_200365378-010_200015092000928035_80192.jpg","640x290":"uploaded_files/image/640x290/getty_200365378-010_200015092000928035_80192.jpg","635x367":"uploaded_files/image/635x367/getty_200365378-010_200015092000928035_80192.jpg","0x367":"uploaded_files/image/0x367/getty_200365378-010_200015092000928035_80192.jpg","575x270":"uploaded_files/image/575x270/getty_200365378-010_200015092000928035_80192.jpg","385x240":"uploaded_files/image/385x240/getty_200365378-010_200015092000928035_80192.jpg","336x336":"uploaded_files/image/336x336/getty_200365378-010_200015092000928035_80192.jpg","300x520":"uploaded_files/image/300x520/getty_200365378-010_200015092000928035_80192.jpg","300x200":"uploaded_files/image/300x200/getty_200365378-010_200015092000928035_80192.jpg","284x160":"uploaded_files/image/284x160/getty_200365378-010_200015092000928035_80192.jpg","155x90":"uploaded_files/image/155x90/getty_200365378-010_200015092000928035_80192.jpg","100x100":"uploaded_files/image/100x100/getty_200365378-010_200015092000928035_80192.jpg","50x50":"uploaded_files/image/50x50/getty_200365378-010_200015092000928035_80192.jpg"}}}],"formatted_text":"<p>Disobedient <a href="https://www.inc.com/graham-winfrey/introducing-pepper-the-friendly-humanoid-robot.html">robots</a> may&nbsp;sound scary, but even scarier is a robot that follows every human command without fail.</p>\r\n<p>Researchers at <a target="_blank" rel="nofollow" href="http://hrilab.tufts.edu/">Tufts University's Human-Robot Interaction Lab</a> have taught robots how to question commands that would have negative consequences--like walking&nbsp;off the edge of a table--an important step in the development of <a href="https://www.inc.com/graham-winfrey/how-artificial-intelligence-will-transform-your-business.html">artificial intelligence</a>. Robots can even change&nbsp;their minds after initially&nbsp;refusing a direct&nbsp;order, provided a human can offer a logical reason as to why the robot should override its first response.</p>\r\n<p>To see one of Tufts's&nbsp;researchers convince&nbsp;a robot to follow his&nbsp;command after the robot's initial refusal, check out the video below.</p>\r\n<p><iframe frameborder="0" height="315" src="https://www.youtube.com/embed/0tu4H1g3CtE" width="420"></iframe></p>\r\n<p>While there are obvious benefits to developing increasingly sophisticated A.I. technology, there are also some not-so-obvious risks. Of particular concern for Tufts professor Matthias Scheutz is what happens when humans begin developing relationships with so-called "social robots" used in&nbsp;elder care and other situations where companionship is&nbsp;one of the primary functions.</p>\r\n<p>"These are contexts where you use robots with vulnerable populations, such as people who might have dementia," Scheutz says. "These people are even more likely to invest emotions and invest in the relationship, even though the robots cannot reciprocate."</p>\r\n<p>French robotics company Aldebaran bills its&nbsp;<a target="_blank" rel="nofollow" href="https://www.aldebaran.com/en/cool-robots/pepper">Pepper</a> robot as "the first humanoid robot designed to live with humans." Originally created for Japanese mobile phone company SoftBank Mobile as a device to greet and welcome shoppers in stores, Pepper can recognize certain human facial expressions and react accordingly. For example, if the robot senses someone is sad, it&nbsp;can play their&nbsp;favorite song to try to cheer them up, or play interactive games through a display tablet mounted on its chest.&nbsp;Japanese industrial automation company Aist&nbsp;makes an&nbsp;interactive robot that looks like a seal called Paro. Used in elder care settings, it&nbsp;has been shown to improve people's moods.</p>\r\n<p>As robots like Pepper and Paro become increasingly sophisticated and&nbsp;interactive, however,&nbsp;there is a significant risk that they can cause more psychological harm than good, according to Scheutz.&nbsp;"People might interpret the robot's gestures or eye gaze in a particular way and form expectations based on that, and the robot will inevitably disappoint those expectations," he says. "People will be tricked into believing that the robot has capabilities and something inside it that it doesn't."</p>\r\n<p>Even for people who don't suffer from any mental condition, there are risks involved in using robots for companionship, Scheutz&nbsp;says.&nbsp;"If you take people who are just lonely, and they project something into this robot that's not there, it's going to be very frustrating."&nbsp;</p>\r\n<p>Though social robots are not&nbsp;available yet for consumer purchase&nbsp;in the U.S., robotic machines like the Roomba vacuum have been around for years, so the technology is already making its way into U.S. households. Scheutz believes&nbsp;the earliest applications for other robots in homes are likely to be basic services like&nbsp;cleaning and&nbsp;delivery, before "companion" robots become a consumer item.&nbsp;"Eventually, we will notice that we have a whole bunch of robots in our homes that do different things, and maybe they'll talk to each other," he says.&nbsp;</p>\r\n<p>What&nbsp;Scheutz doesn't want to see is&nbsp;robotics companies throwing around the&nbsp;term "emotional robot" when referring to any product. It could create confusion he thinks might eventually lead to humans experiencing psychological harm.</p>\r\n<p>"It's absurdly bogus," he says. "There's nothing in a robot that has anything to do with emotions."&nbsp;</p>","adinfo":{"c_type":"article","showlogo":true,"cms":"inc87732","video":"no","aut":["graham-winfrey"],"channelArray":{"topid":"7","topfilelocation":"innovate","primary":["innovate"],"primaryFilelocation":["innovate"],"primaryname":["Innovate"]},"adzone":"/4160/mv.inc/innovate/innovate/innovate"},"seriesname":null,"editorname":null,"commentcount":null,"inc5000companies":[],"inc5000list":{"id":null,"ifl_list":null,"ifl_year":null,"ifl_custom_data_description":null,"ifl_filelocation":null,"ifl_sharetext":null,"ifl_data_endpoint":null,"ifl_columns":null,"ifl_rows_per_page":null,"ifl_filter_columns":null,"ifl_permanently_hidden_columns":null,"ifl_extra_large_hidden_columns":null,"ifl_large_hidden_columns":null,"ifl_medium_hidden_columns":null,"ifl_small_hidden_columns":null,"ifl_extra_small_hidden_columns":null,"ifl_currency":null,"ifl_enable_accent_rule_topflag":false,"ifl_enable_accent_rule_bottomflag":false,"ifl_table_accent_rule_color":null,"ifl_table_header_background_color":null,"ifl_table_header_text_color":null,"ifl_table_row_stripe_color":null,"ifl_enable_filterflag":false,"ifl_filter_background_color":null,"ifl_filter_dropdown_border_color":null,"ifl_filter_dropdown_text_color":null,"ifl_enable_pagination_topflag":false,"ifl_enable_pagination_bottomflag":false,"ifl_pagination_bar_color":null,"ifl_filter_reset_button_color":null,"ifl_methodology":null,"ifl_pubdate":null,"ifl_default_sort":null,"companylist":null},"companies":[],"buyerzonewidgets":[],"photoEssaySlideModels":null,"custom_article_footer":null,"ser_footer_blurb":null,"dayssincepubdate":1355,"trackingpixel":"","promotions":null,"inline_script_tags":[],"loadedFully":true,"promoimage":"https://www.incimages.com/uploaded_files/image/970x450/getty_200365378-010_200015092000928035_80192.jpg","largepromoimage":"https://www.incimages.com/uploaded_files/image/1940x900/getty_200365378-010_200015092000928035_80192.jpg","pubdate":"2016-02-04T16:40:00.000Z","author":"Graham Winfrey","authortwitter":"@GrahamWinfrey","articlesection":"Innovate"},"server811378":{"id":237357,"inc_homepage_headline":"The Myth of the First Mover Advantag","inc_homepage_headline_ab_test":"First Mover Disadvantage?","inc_headline":"The Myth of the First Mover Advantage","inc_filelocation":"https://www.inc.com/howard-tulllman/the-myth-of-the-first-mover-advantage.html?icid=readmoretext_ab_pretest","tilefeatureimage":"https://www.incimages.com/uploaded_files/image/300x200/getty_477554544_2000184820009280919_404959.jpg","tilefeatureimageX2":"https://www.incimages.com/uploaded_files/image/600x400/getty_477554544_2000184820009280919_404959.jpg","brandview":null,"loadedFully":false},"server811379":{"id":237143,"inc_homepage_headline":"Simon Sinek: Adam Grant Made Me Sick With Envy Until I Learned the Value of a Great Rival","inc_homepage_headline_ab_test":"Simon Sinek: Here's Why Everyone Should Have a Work Rival","inc_headline":"Simon Sinek: Adam Grant Made Me Sick With Envy Until I Learned the Value of a Great Rival","inc_filelocation":"https://www.inc.com/jessica-stillman/simon-sinek-heres-why-everyone-should-have-a-work-rival .html?icid=readmoretext_ab_pretest","tilefeatureimage":"https://www.incimages.com/uploaded_files/image/300x200/getty_655011988_2000127020009280126_404971.jpg","tilefeatureimageX2":"https://www.incimages.com/uploaded_files/image/600x400/getty_655011988_2000127020009280126_404971.jpg","brandview":null,"loadedFully":false},"server811380":{"id":237313,"inc_homepage_headline":"Handle Negative Feedback Like a Pro: 6 Expert Tips","inc_homepage_headline_ab_test":"6 Ways to Deal With Negative Feedback Constructively","inc_headline":"Handle Negative Feedback Like a Pro: 6 Expert Tips","inc_filelocation":"https://www.inc.com/young-entrepreneur-council/handle-negative-feedback-like-a-pro-6-expert-tips.html?icid=readmoretext_ab","tilefeatureimage":"https://www.incimages.com/uploaded_files/image/300x200/getty_916520080_404877.jpg","tilefeatureimageX2":"https://www.incimages.com/uploaded_files/image/600x400/getty_916520080_404877.jpg","brandview":null,"loadedFully":false},"server811381":{"id":237325,"inc_homepage_headline":"Netflix Co-Founder Says Entrepreneurs Should Constantly Remind Themselves That 'Nobody Knows Anything'","inc_homepage_headline_ab_test":"Netflix Co-Founder Has 3 Words of Advice for Anyone Who Wants to Turn Their Idea Into Something Real","inc_headline":"Netflix Co-Founder Says Entrepreneurs Should Constantly Remind Themselves That 'Nobody Knows Anything'","inc_filelocation":"https://www.inc.com/carmine-gallo/netflix-co-founder-says-entrepreneurs-should-constantly-remind-themselves-that-nobody-knows-anything.html?icid=readmoretext_ab","tilefeatureimage":"https://www.incimages.com/uploaded_files/image/300x200/getty_968049386_404901.jpg","tilefeatureimageX2":"https://www.incimages.com/uploaded_files/image/600x400/getty_968049386_404901.jpg","brandview":null,"loadedFully":false}},"videoHash":{"server811382":{"id":237173,"inc_homepage_headline":null,"inc_homepage_headline_ab_test":null,"inc_headline":"These Female Founders Shared Their Best Advice for Overcoming Failure and Self-Doubt","inc_deck":"Holly Whitaker, founder of online recovery school Tempest, and Kate Ryder, founder of healthcare company Maven, shared the advice that helped them succeed through failure, self-doubt and countless business challenges. ","inc_filelocation":"https://www.inc.com/video/these-female-founders-shared-their-best-advice-for-overcoming-failure-and-self-doubt.html","video":true,"vidid":"15799","tilefeatureimage":"https://www.incimages.com/uploaded_files/image/300x200/Ep2_Kate_Holly_POSTER_TILE_404660.png","tilefeatureimageX2":"https://www.incimages.com/uploaded_files/image/600x400/Ep2_Kate_Holly_POSTER_TILE_404660.png"}},"gridHash":{},"productHash":{},"user":{"loggedIn":false},"mustReadArticles":{"articles":[],"videos":[],"desktoplogo":null,"mobilelogo":null,"logolink":null,"containedInList":null},"articleBundleHash":{"server811386":{"articleId":"server811377","videos":[],"grids":[],"slideshows":[],"slides":[],"readmorearticles":[237357,237143,237313,237325],"images":["server811383"],"channels":["server811384"],"authors":["server811385"]}},"articlePage":{"infiniteArticleCollection":[{"articleBundleId":"server811386"}]},"imageHash":{"server811383":{"id":80192,"img_foreignkey":"200365378-010","img_gettyflag":true,"img_reusableflag":false,"img_rightsflag":false,"img_usrid":0,"img_pan_crop":"{\"x\":0,\"y\":35,\"h\":928,\"w\":2000,\"img_h\":1509,\"img_w\":2000}","img_tags":null,"img_reference_name":"getty_200365378-010_200015092000928035.jpg","img_caption":null,"img_custom_credit":null,"img_bucketref":null,"img_panoramicref":"getty_200365378-010_200015092000928035.jpg","img_super_panoramicref":null,"img_tile_override_imageref":null,"img_skyscraperref":null,"img_gallery_imageref":null,"credit":"Getty Images","sizes":{"panoramic":{"original":"uploaded_files/image/getty_200365378-010_200015092000928035.jpg","1230x1672":"uploaded_files/image/1230x1672/getty_200365378-010_200015092000928035_80192.jpg","1940x900":"uploaded_files/image/1940x900/getty_200365378-010_200015092000928035_80192.jpg","1270x734":"uploaded_files/image/1270x734/getty_200365378-010_200015092000928035_80192.jpg","0x734":"uploaded_files/image/0x734/getty_200365378-010_200015092000928035_80192.jpg","1150x540":"uploaded_files/image/1150x540/getty_200365378-010_200015092000928035_80192.jpg","970x450":"uploaded_files/image/970x450/getty_200365378-010_200015092000928035_80192.jpg","640x290":"uploaded_files/image/640x290/getty_200365378-010_200015092000928035_80192.jpg","635x367":"uploaded_files/image/635x367/getty_200365378-010_200015092000928035_80192.jpg","0x367":"uploaded_files/image/0x367/getty_200365378-010_200015092000928035_80192.jpg","575x270":"uploaded_files/image/575x270/getty_200365378-010_200015092000928035_80192.jpg","385x240":"uploaded_files/image/385x240/getty_200365378-010_200015092000928035_80192.jpg","336x336":"uploaded_files/image/336x336/getty_200365378-010_200015092000928035_80192.jpg","300x520":"uploaded_files/image/300x520/getty_200365378-010_200015092000928035_80192.jpg","300x200":"uploaded_files/image/300x200/getty_200365378-010_200015092000928035_80192.jpg","284x160":"uploaded_files/image/284x160/getty_200365378-010_200015092000928035_80192.jpg","155x90":"uploaded_files/image/155x90/getty_200365378-010_200015092000928035_80192.jpg","100x100":"uploaded_files/image/100x100/getty_200365378-010_200015092000928035_80192.jpg","50x50":"uploaded_files/image/50x50/getty_200365378-010_200015092000928035_80192.jpg"}}}},"channelHash":{"server811384":{"id":7,"cnl_name":"Innovate","cnl_filelocation":"innovate","cnl_featuretype":"None","cnl_custom_color":"9DC786","cnl_calculated_color":null,"cnl_contributor_accessflag":true,"cnl_custom_article_footer":null,"cnl_global_nav_background_color":null,"cnl_global_nav_background_gradient_start":null,"cnl_global_nav_background_gradient_end":null,"cnl_iflid":0,"sortorder":null}},"authorHash":{"server811385":{"id":3828,"aut_name":"Graham Winfrey","aut_usrid":631397,"aut_base_filelocation":"graham-winfrey","aut_imgid":359288,"aut_twitter_id":"GrahamWinfrey","aut_title":"Senior editor, Inc.","aut_blurb":"Graham Winfrey is a senior editor at Inc.com. He was previously a reporter at IndieWire, a staff writer at Inc., and an associate editor at Private Equity International magazine. He has written for Business Insider, CNBC, Slate, The Week, and Worth, among others.","aut_footer_blurb":"Graham Winfrey is a senior editor at Inc.com. He was previously a reporter at IndieWire, a staff writer at Inc., and an associate editor at Private Equity International magazine. He has written for Business Insider, CNBC, Slate, The Week, and Worth, among others.","aut_column_name":null,"aut_atyid":1,"aut_newsletter_location":null,"authorimage":"https://www.incimages.com/uploaded_files/image/100x100/WinfreyHeadshot_359288.jpg","sortorder":null}},"author":{"authorData":{},"isFetching":false,"isFetched":false,"error":null},"companyProfile":[],"channel":{"channel":[],"isFetching":false,"isFetched":false,"error":null},"guide":{"guide":{},"isFetching":false,"isFetched":false,"error":null},"sitemap":{"sitemap":[],"isFetching":false,"isFetched":false,"error":null},"hostname":"node.inc.com","homePage":{"topArticles":[],"packages":[],"isFetching":false,"isFetched":false,"error":null},"ifl":{"ifl":[],"isFetching":false,"isFetched":false,"error":null}};