Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Looks like they are implicitly permitting google via robots.txt:

http://www.victoriassecret.com/robots.txt

    User-agent: ia_archiver    
    Disallow: /    

    User-agent: *    
    Disallow: /commerce/application?namespace=shoppingBag&origin=myMain.jsp&event=link.viewBag    
    Disallow: /commerce/logoff.vs    
    Disallow: /commerce/signin.vs    
    Disallow: /commerce/registration.vs    
    Disallow: /commerce/logon.vs    
    Disallow: /commerce/addressbook/index.jsp    
    Disallow: /commerce/changePasswordJsf.vs    

    Sitemap: http://www.victoriassecret.com/sitemap.xml


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: