太原SEO
专注关键词排名

如何编写robots协议优化你的网站 太原SEO-robots协议

搜索引擎是不会直接与网站对话的,做优化的目的就是让搜索引擎蜘蛛更容易更便捷的抓取网站,对于robots文件搜索引擎是可以直接和他对话的,所以制作robots协议至关重要的,robots协议也称为爬虫协议、机器人协议 等,robots协议全称是“网络爬虫排除标准”(Robots Exclusion Protocol),网站通过robots协议告诉搜索引擎哪些页面可以抓取,哪些页面不能抓取,下面李毅就从robots协议的写法、作用来简单介绍下robots协议怎么制作。

robots协议

robots协议

一、robots的写法

1、User-agent: * 这里的*代表的所有的搜索引擎种类,*是一个通配符

2、Disallow: /admin/ 这里定义是禁止爬寻admin目录下面的目录

3、Disallow: /require/ 这里定义是禁止爬寻require目录下面的目录

4、Disallow: /ABC/ 这里定义是禁止爬寻ABC目录下面的目录

5、Disallow: /cgi-bin/*.htm 禁止访问/cgi-bin/目录下的所有以”.htm”为后缀的URL(包含子目录)

6、Disallow: /*?* 禁止访问网站中所有包含问号 (?) 的网址

7、Disallow: /.jpg$ 禁止抓取网页所有的.jpg格式的图片

8、Disallow:/ab/adc.html 禁止爬取ab文件夹下面的adc.html文件

9、Allow: /cgi-bin/ 这里定义是允许爬寻cgi-bin目录下面的目录

10、Allow: /tmp 这里定义是允许爬寻tmp的整个目录

11、Allow: .htm$ 仅允许访问以”.htm”为后缀的URL

12、Allow: .gif$ 允许抓取网页和gif格式图片

13、Sitemap: 网站地图 告诉爬虫这个页面是网站地图

二、robots协议的作用

1、引导搜索引擎蜘蛛抓取指定栏目或内容;

2、网站改版或者URL重写优化时候屏蔽对搜索引擎不友好的链接;

3、屏蔽死链接、404错误页面;

4、屏蔽无内容、无价值页面;

5、屏蔽重复页面,如评论页、搜索结果页;

6、屏蔽任何不想被收录的页面;

7、引导蜘蛛抓取网站地图;

三、考虑新手自己写不了robots协议,下面就是本站自己的robots文件

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /trackback/
Disallow: /comments/
Disallow: /attachment/
Disallow: /comments/feed
Disallow: /feed
Disallow: /*/feed
Disallow: /*/comment-page-*
Disallow: /*?replytocom=*
Disallow: /*/trackback
Disallow: /?s=*
Disallow: /*/?s=*\
Disallow: /wp-*.php

Sitemap: http://www.liyonyi.com/sitemap.html

制作好的robots文件就直接上传到网站的根目录下就好,新手站长就拿上李毅编写的robots协议把xml地图替换掉就好,编写robots协议、让网站优化更高效。

赞(1) 打赏
【本站文章禁止任何媒体转载,侵权必究!】太原SEO - 专注关键词排名「免费指导」 » 如何编写robots协议优化你的网站
分享到: 更多 (0)

评论 27

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
  1. #1

    It’s the best time to make a few plans for the long
    run and it’s time to be happy. I’ve learn this put up and if I may just I wish
    to suggest you some interesting issues or advice.
    Maybe you could write subsequent articles referring to this article.

    I desire to learn even more things approximately it!
    Hey would you mind letting me know which hosting company you’re working with?

    I’ve loaded your blog in 3 completely different browsers and I must say this blog loads a lot quicker then most.

    Can you suggest a good internet hosting provider at a reasonable price?
    Thanks, I appreciate it! Woah! I’m really enjoying the template/theme of this blog.
    It’s simple, yet effective. A lot of times it’s tough to get
    that “perfect balance” between user friendliness and appearance.
    I must say you’ve done a fantastic job with this.
    Additionally, the blog loads super fast for me on Safari.

    Excellent Blog! http://Www.cspan.net/

    Kristen3周前 (10-29)回复
    • Thank you for visiting my website. The web server hosts you. You can choose resellerclub. I hope it can help you.

      李 毅3周前 (10-30)回复
  2. #2

    If you have children, why don’t you obtain the best online piano course for the kids as well.
    Now then, with Professional Skill is your goal, for starters, have the PROFESSIONAL ATTITUDE.
    I began hearing ABBA when I involved 15 roughly and leaving
    school because obviously I had heard a few of their songs about the radio and TV but somehow I came across an ABBA
    cassette and played it then played it over and over.

    먹튀다잡아2周前 (11-07)回复
  3. #3

    What’s uр, thiѕ weekend is ⲣleasant fօr me, as tthis time i am reading this enormous
    educational article here at myʏ home.

    TheLotter2周前 (11-08)回复
    • Thank you very much for visiting my website.

      李 毅2周前 (11-08)回复
  4. #4

    I’d like to find out more? I’d want to find out some additional information.

    camera wifi1周前 (11-08)回复
    • Please continue to pay attention to my new article.

      李 毅1周前 (11-09)回复
  5. #5

    It’s remarkable for me to have a website, which is good designed for my know-how.
    thanks admin

    camera wifi ngoài trời không dây1周前 (11-09)回复
    • Thank you very much for your visit. I will continue to work hard.

      李 毅1周前 (11-09)回复
  6. #6

    Hello to all, the contents existing at this web site are in fact remarkable for people knowledge, well, keep up
    the good work fellows.

  7. #7

    Veryy goօd informɑtion. Lucky me I гan across yoᥙr site by accident (stumbleupon).

    І’νe book-marked іt for later!

  8. #8

    I have been surfing online more than 4 hours today, yet I never found any interesting article like yours.
    It’s pretty worth enough for me. Personally, if
    all webmasters and bloggers made good content as you did, the net will be a lot more useful
    than ever before.|
    I couldn’t refrain from commenting. Perfectly written!|
    I’ll immediately snatch your rss feed as I can’t in finding your email subscription link
    or newsletter service. Do you have any? Kindly allow me recognise in order that I could subscribe.
    Thanks.|
    It is perfect time to make some plans for the future and it’s time
    to be happy. I have read this post and if I could I desire
    to suggest you some interesting things or suggestions.
    Maybe you could write next articles referring
    to this article. I desire to read more things about it!|
    It is the best time to make some plans for the future and it is time to be happy.
    I have read this post and if I could I wish to recommend you few fascinating things or suggestions.
    Perhaps you can write subsequent articles relating
    to this article. I desire to learn even more things approximately
    it!|
    I have been browsing online greater than 3 hours as of late, but I by no
    means discovered any interesting article like yours.
    It’s lovely value enough for me. In my view, if all web
    owners and bloggers made excellent content material as you did, the
    internet shall be a lot more helpful than ever before.|
    Ahaa, its good dialogue regarding this paragraph here at this webpage,
    I have read all that, so at this time me also commenting here.|
    I am sure this article has touched all the internet visitors, its really really nice
    article on building up new weblog.|
    Wow, this article is pleasant, my sister is analyzing these things, thus I am going
    to convey her.|
    bookmarked!!, I like your web site!|
    Way cool! Some very valid points! I appreciate you penning this
    write-up and also the rest of the site is extremely good.|
    Hi, I do think this is an excellent website. I stumbledupon it 😉 I am going to come back once again since i have book-marked it.
    Money and freedom is the greatest way to change, may you be rich and
    continue to help others.|
    Woah! I’m really loving the template/theme of this site.
    It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appearance.
    I must say that you’ve done a great job with this.
    Additionally, the blog loads very quick for me on Firefox.
    Superb Blog!|
    These are genuinely impressive ideas in on the topic
    of blogging. You have touched some fastidious things here.
    Any way keep up wrinting.|
    Everyone loves what you guys are usually up too.
    This sort of clever work and exposure! Keep up the superb works guys I’ve added you
    guys to my own blogroll.|
    Hi! Someone in my Myspace group shared this site with
    us so I came to take a look. I’m definitely loving the
    information. I’m book-marking and will be tweeting this to my
    followers! Terrific blog and superb style and
    design.|
    I really like what you guys are usually up too.
    This kind of clever work and reporting! Keep up the superb works guys I’ve incorporated you guys to my personal blogroll.|
    Hey there would you mind sharing which blog platform you’re
    working with? I’m planning to start my own blog in the near future but I’m having
    a difficult time choosing between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design and style seems different then most blogs and I’m looking for
    something completely unique. P.S Sorry for getting off-topic but I had to ask!|
    Howdy would you mind letting me know which web host you’re utilizing?

    I’ve loaded your blog in 3 completely different
    web browsers and I must say this blog loads a lot
    quicker then most. Can you suggest a good web hosting provider at
    a reasonable price? Kudos, I appreciate it!|
    I like it when people get together and share ideas.
    Great site, keep it up!|
    Thank you for the good writeup. It in fact was a amusement account it.
    Look advanced to more added agreeable from you!

    By the way, how can we communicate?|
    Hey just wanted to give you a quick heads up. The words in your article seem to be running off the screen in Ie.

    I’m not sure if this is a formatting issue or something to do with browser compatibility but I thought I’d post to let you know.
    The style and design look great though! Hope you get the
    issue resolved soon. Thanks|
    This is a topic that is close to my heart… Many thanks!
    Exactly where are your contact details though?|
    It’s very simple to find out any topic on web as compared to books, as
    I found this article at this web site.|
    Does your website have a contact page? I’m having trouble locating it but, I’d like
    to send you an e-mail. I’ve got some ideas for your blog you might
    be interested in hearing. Either way, great blog and I look forward to seeing
    it develop over time.|
    Hello! I’ve been reading your web site for a
    long time now and finally got the courage to go ahead and give you a shout out from
    Huffman Tx! Just wanted to say keep up the good work!|
    Greetings from California! I’m bored to tears at work so I decided to browse your blog on my iphone during lunch break.
    I enjoy the knowledge you provide here and can’t wait to take a look when I get home.

    I’m amazed at how fast your blog loaded on my cell phone ..
    I’m not even using WIFI, just 3G .. Anyhow, good site!|
    Its like you read my thoughts! You appear to understand a lot approximately this, such as
    you wrote the ebook in it or something. I
    believe that you simply can do with a few percent to power the message home a bit, however other than that, this is wonderful blog.

    An excellent read. I’ll certainly be back.|
    I visited multiple blogs however the audio quality for
    audio songs present at this site is genuinely excellent.|
    Howdy, i read your blog occasionally and i own a similar one and i was just wondering if you get a lot of spam
    comments? If so how do you protect against it, any plugin or anything you can recommend?

    I get so much lately it’s driving me insane so any assistance is very much appreciated.|
    Greetings! Very helpful advice within this article!
    It’s the little changes that produce the most important changes.
    Thanks a lot for sharing!|
    I truly love your site.. Very nice colors & theme.
    Did you make this amazing site yourself? Please reply back as I’m planning to create my own personal blog and would
    like to find out where you got this from or what the theme
    is named. Kudos!|
    Hello there! This blog post could not be written much better!
    Looking through this post reminds me of my previous roommate!
    He continually kept preaching about this. I most certainly
    will send this article to him. Pretty sure he will have a great read.
    Thanks for sharing!|
    Whoa! This blog looks exactly like my old one! It’s on a completely different
    topic but it has pretty much the same page layout and design. Great choice of colors!|
    There is definately a great deal to know about this topic.

    I love all of the points you’ve made.|
    You made some really good points there. I looked on the internet to find out more about the issue and found most people will go along with
    your views on this website.|
    Hi there, I check your blog on a regular basis.
    Your writing style is witty, keep up the good work!|
    I simply couldn’t go away your site before suggesting that I actually enjoyed the usual
    information an individual supply to your guests? Is going to be
    again incessantly in order to investigate cross-check
    new posts|
    I want to to thank you for this good read!! I definitely loved every bit of it.
    I have you bookmarked to check out new things you post…|
    Hi there, just wanted to mention, I loved this post.
    It was helpful. Keep on posting!|
    Hi there, I enjoy reading all of your post. I like to write a
    little comment to support you.|
    I always spent my half an hour to read this website’s
    content everyday along with a cup of coffee.|
    I every time emailed this blog post page to all my friends, because if like to read it then my
    contacts will too.|
    My developer is trying to convince me to move to .net from PHP.
    I have always disliked the idea because of the costs.
    But he’s tryiong none the less. I’ve been using
    WordPress on a number of websites for about a year and
    am anxious about switching to another platform. I have heard excellent things about blogengine.net.

    Is there a way I can transfer all my wordpress content
    into it? Any kind of help would be greatly appreciated!|
    Howdy! I could have sworn I’ve been to your blog before but after going through a few of the articles I realized it’s new to me.
    Anyhow, I’m definitely pleased I found it and I’ll be bookmarking it and checking back frequently!|
    Great work! This is the kind of info that are meant to be shared around the
    web. Shame on the seek engines for no longer positioning
    this post upper! Come on over and talk over with my web site
    . Thank you =)|
    Heya i’m for the first time here. I came across this board and I find It really useful & it helped me
    out much. I hope to give something back and aid others like you aided me.|
    Howdy, I do believe your web site might be having web browser compatibility problems.
    When I look at your web site in Safari, it looks fine but when opening in I.E.,
    it’s got some overlapping issues. I just wanted to provide you
    with a quick heads up! Apart from that, fantastic website!|
    Somebody essentially lend a hand to make critically articles I might state.
    That is the very first time I frequented your web page and up
    to now? I surprised with the research you made
    to make this particular put up extraordinary.

    Great activity!|
    Heya i am for the primary time here. I came across this board and I in finding It really useful & it
    helped me out a lot. I’m hoping to offer one thing back
    and help others like you aided me.|
    Hello there! I simply want to give you a big thumbs up for your great info you have got here on this post.
    I am returning to your site for more soon.|
    I all the time used to read piece of writing in news papers but now as
    I am a user of web thus from now I am using net for content, thanks to web.|
    Your method of describing everything in this post is in fact
    good, every one be able to without difficulty be aware of it, Thanks a lot.|
    Hello there, I discovered your blog by means of Google while searching for a related topic, your website came up,
    it appears to be like great. I’ve bookmarked it in my google bookmarks.

    Hello there, simply changed into aware of your blog through Google,
    and found that it is truly informative. I’m gonna watch
    out for brussels. I will be grateful if you proceed
    this in future. Numerous people will likely be benefited from your writing.
    Cheers!|
    I am curious to find out what blog platform you’re working with?
    I’m having some minor security issues with my latest blog and I
    would like to find something more secure. Do you have any suggestions?|
    I am extremely impressed with your writing skills as well as with the layout on your weblog.
    Is this a paid theme or did you modify it yourself? Either way keep up
    the excellent quality writing, it’s rare to see a great blog
    like this one nowadays.|
    I am extremely impressed together with your writing skills as smartly as with the layout to your weblog.
    Is this a paid theme or did you modify it your self? Either way stay up the excellent
    high quality writing, it’s uncommon to peer a
    nice blog like this one today..|
    Hello, Neat post. There’s a problem with your site in internet explorer,
    could check this? IE still is the market leader and a big component to people
    will omit your great writing because of this problem.|
    I’m not sure where you are getting your information, but great topic.
    I needs to spend some time learning much more or understanding more.
    Thanks for excellent information I was looking for this info for
    my mission.|
    Hi, i think that i saw you visited my blog so i came to “return the favor”.I’m attempting to find things to improve my web site!I suppose
    its ok to use a few of \

    • Thank you very much for your support. I recommend ResellerClub as your website hosting provider. If you have other questions, you can add my Facebook jasonliyi and email to my Google mailbox jasonqazwsx.521@gmail.com.

  9. #9

    Yօu made sⲟme g᧐od points tһere. I looкeɗ on tһe
    internet fߋr the topic ɑnd found mߋst persons wіll approve
    with your site.

    monaruggieri4天前回复
  10. #10

    I always used to study post in news papers Ƅut now as I am a user of internet thus from now I am using net fоr articles or reviews, thankѕ to web.

    sophie_griswold4天前回复
  11. #11

    Jᥙѕt want to say your article is as astounding.
    The clarity in yߋur post іs simply spectacular and і
    can assume yoս’гe an expert оn this subject. Ϝine with your
    permission let mе to grab yoᥙr feed to keеp uр
    to ɗate ᴡith forthcoming post. Τhanks a million and please carry
    ߋn the gratifying ᴡork.

    rhodacolbert4天前回复
  12. #12

    Ꭲhanks for sharing yօur thoughts οn robots协议.
    Ꭱegards

    luissizemore4天前回复
  13. #13

    Do you mind if I quote а couple of ʏour articles as ⅼong as Ι provide
    credit ɑnd sources back t᧐ yοur site? Μy website iѕ in tһe very ѕame niche
    as yourѕ and my visitors wօuld dеfinitely benefit fгom
    ɑ lot of the information you provide here. Please
    let me know if this ok with you. Aⲣpreciate it!

    molly_pelsaert4天前回复
  14. #14

    Ꮃe are a bunch of volunteers ɑnd opening a new scheme in our community.
    Үour web site offered ᥙs ԝith helpful infоrmation tⲟ work օn. Yoս have performed а formidable job and our entire community wilⅼ be grateful t᧐ ʏou.

  15. #15

    Hi, I do think this is an excellent site. I stumbledupon it 😉 I am going to return yet again since I book-marked it.
    Money and freedom is the best way to change,
    may you be rich and continue to guide other people.

  16. Thank you for visiting my website.

    李 毅1周前 (11-10)回复

太原SEO - 专注关键词排名「免费指导」

SEO新手入门太原SEO服务项目

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏