User-agent: * Disallow: /apps Disallow: /discussions Disallow: /cgi-bin Disallow: /sendit Disallow: /soundoff Disallow: /resources/inc500.5172005 Disallow: /resources/inc500.20050819 Disallow: /resources/inc500.postDART Disallow: /resources/inc500.preDART Disallow: /resources/inc500.tmp Disallow: /inctv/video/featured-video Disallow: /inctv/video/featured-audio Disallow: /inc5000dev Disallow: /adops Disallow: /adsales Disallow: /atlas Disallow: /brandnet Disallow: /cache Disallow: /clients Disallow: /conf Disallow: /doubleclick Disallow: /lib Disallow: /models Disallow: /modules Disallow: /users Disallow: /lib/twitter/start.php #block rest api from indexing. a.b Disallow: /rest #Disallow robots #added these from 404 list A.B Disallow: /home/ Disallow: /&* Disallow: /<* Disallow: /&rdquo* Disallow: /index.php Disallow: /guides/ Disallow: /ss/ Disallow: /draft/ #Disallow robots #User-Agent: Googlebot #Allow: / User-agent: Twitterbot Disallow: /&* Disallow: /<* Disallow: /”* User-Agent: Googlebot-Mobile Allow: / Disallow: /&* Disallow: /<* Disallow: /”* User-Agent: msnbot Allow: / Disallow: /&* Disallow: /<* Disallow: /”* User-Agent: bingbot Allow: / Disallow: /&* Disallow: /<* Disallow: /”* # Adsense User-Agent: Mediapartners-Google Allow: / # Blekko User-Agent: ScoutJet Allow: / User-Agent: Yandex Allow: / User-agent: baiduspider Allow: / User-agent: DuckDuckBot Allow: / User-agent: BomboraBot Allow: / User-agent: grapeshot Allow: / # CommonCrawl User-agent: ccbot Allow: / Sitemap: https://www.inc.com/sitemap/sitemap_index.xml Sitemap: https://www.inc.com/sitemap/sitemap_news.xml Sitemap: https://www.inc.com/sitemap/sitemap_videos.xml Sitemap: https://www.inc.com/sitemap/sitemap_brandview_542.xml Sitemap: https://www.inc.com/sitemap/sitemap_grow_6.xml Sitemap: https://www.inc.com/sitemap/sitemap_innovate_7.xml Sitemap: https://www.inc.com/sitemap/sitemap_lead_4.xml Sitemap: https://www.inc.com/sitemap/sitemap_magazines.xml Sitemap: https://www.inc.com/sitemap/sitemap_money_2.xml Sitemap: https://www.inc.com/sitemap/sitemap_politics_419.xml Sitemap: https://www.inc.com/sitemap/sitemap_technology_5.xml Sitemap: https://www.inc.com/sitemap/sitemap_wire_374.xml Sitemap: https://www.inc.com/sitemap/sitemap_profiles.xml