User-agent: *
Disallow:/dwr
Disallow:/dashboard
Disallow:/message/
Disallow:/postmanage/
Disallow:/notice
Disallow:/followed
Disallow:/follow
Disallow:/theme/
Disallow:/newblog
Disallow:/setting
Disallow:/inviteelist
Disallow:/submit
Disallow:/like
Disallow:/logout
Disallow:/edit/
Disallow:/reblog/
Disallow:/control
Disallow:/comment.do
Disallow:/mydomainr.do
Disallow:/invite
Disallow:/art/checkorder
Disallow:/art/orderpayresult
Disallow:/art/ordersnapshot
Disallow:/benefit/
Sitemap: http://www.lofter.com/sitemap.xml
User-agent: *代表爬的所有数据
Disallow代表不可爬的内容 比如个人信息这些隐私内容
网友评论