Quality Rater Guidelines 2023: Whats new? (2024)

Google released an new version of the Quality Rater Guidelines on 16.November 2023. In this article you can find what’new in the QRG.

What are the Google Quality Rater Guidelines?

The Google Quality Rater Guidelines are a set of criteria that Google uses to guide human evaluators in assessing the quality of search results. These guidelines help ensure that the search results are of high quality, relevant, and useful to users. Here are the key aspects of these guidelines:

1. Purpose of the Page: Evaluators are instructed to determine the main purpose of a page. This can include providing information, selling products, entertaining users, or any other specific purpose.
2. Experience, Expertise, Authoritativeness, and Trustworthiness (E-E-A-T): This is a crucial factor in assessing the quality of a page, especially for YMYL (Your Money or Your Life) content, which includes topics like health, finance, legal issues, etc. High-quality pages must demonstrate expertise, authoritativeness, and trustworthiness.
3. Content Quality and Amount: The guidelines emphasize the importance of high-quality content. This includes the accuracy, originality, thoroughness, and comprehensiveness of the content presented.
4. Website Information and Reputation: The reputation of the website and the information about the creators of the content are also important. This involves checking reviews, expert opinions, news articles, and other credible sources of information about the site.
5. User Experience: The overall user experience provided by the website, including its design, usability, and the safety of the user, is a significant factor.
6. Needs Met Rating: This criterion assesses how well a page meets the needs of users based on their search queries. This includes considering the intent of the search and how effectively the content addresses that intent.

These guidelines are used by Google’s human raters to evaluate search results, but they do not directly impact search engine rankings. However, they provide insights into what Google considers high-quality content, which can indirectly influence how content is optimized for search engines.

Whats new in the Quality Rater Guidelines 2023?

The Google Quality Rater Guidelines from 2023 incorporate several updates and changes, aimed at maintaining relevance and effectiveness in assessing the quality of web pages and search results. The changes are as follow with some examples:

• Simplified Needs Met Scale Definitions: Added specific guidance for different types of Needs Met ratings.

Part 3: Need Met Rating Guideline

Rating Description A special rating category, which only applies to certain queries and results. All or almost all Fully Meets (FullyM) users would be immediately and fully satisfied by the result and would not need to view other results to satisfy their need. Highly Meets (HM) Very helpful for many or most users. Some users may wish to see additional results. Moderately Meets (MM) Helpful for many users OR very helpful for some users. Some or many users may wish to see additional results. Slightly Meets (SM) Helpful for fewer users. There is a connection between the query and the result, but not a strong or satisfying connection. Many or most users would wish to see additional results. Fails to Meet (FailsM) Completely fails to meet the needs of the users. All or almost all users would wish to see additional results. Please note that you may assign in-between ratings.

13.1 Rating Result Blocks: Block Content and Landing Pages

Which part of the result do you rate? It depends on both the query and the result block:

13.1.1

Query and User Intent Result Rating Fully Meets Explanation specific, clear, and unambiguous. ● All or almost all users would be completely satisfied by the result4users issuing that query would not need additional results to fully satisfy the user intent. In other words, the Fully Meets rating should be reserved for results that are the <complete and perfect response or answer= so that no other results are necessary for all or almost all users to be fully satisfied.

13.2.

specific, clear, and unambiguous. ● All or almost all users would be completely satisfied by the result4users issuing that query would not need additional results to fully satisfy the user intent. In other words, the Fully Meets rating should be reserved for results that are the <complete and perfect response or answer= so that no other results are necessary for all or almost all users to be fully satisfied.

See Also

Google's Search Quality Evaluators Guidelines Cliff Notes + Audiobook

● The user is looking for a very specific fact or piece of information and the result block provides the information thoroughly, accurately, and clearly, and is from a high quality source. No other results would be needed. Before using the Fully Meets rating for queries seeking a very specific fact or piece of information, you must check for accuracy and confirm that the information is supported by expert consensus where such consensus exists. Fully Meets may apply in other situations as well. Ask yourself whether the result block alone is the perfect and complete result that would fully satisfy all or almost all users. Be conservative when using the Fully Meets rating. When in doubt, consider a lower rating. Note: If a result block is very close to being fully satisfying, but the block alone may not be sufficient in order to fully satisfy all or almost all users, a rating of Highly Meets+ may be appropriate. Sometimes you have to think about how specific the user need is and will need to make a judgment call on whether it is specific enough to possibly have a Fully Meets rating.

Query: [titanic imdb] The query has clear intent to go User Intent: Go to the IMDb to the IMDb page for Titanic. page for Titanic. Query: [www.yahoo.c0m] Even though this is an imperfect URL query, it’s clear the user User Intent: Go to the Yahoo wants to go to the Yahoo website. website.

Query: [how to find Mac OS complete and correct answer version] prominently in the result block, in a way that is easy for users to User Intent: Find out how to read, and comes from a high locate the OS version of a quality, authoritative source for Mac. Apple product information. The result block Fully Meets the user need. The result block displays the Query: [new york city complete and correct answer population 2012] prominently in the result block, in a way that is easy for users to User Intent: Find the read4it Fully Meets the user population of New York City need. back in 2012.

13.3.

Highly Meets medical and scientific information pages must represent well-established scientific/medical consensus unless the user is clearly seeking an alternative viewpoint. For all information pages, you must check for accuracy and confirm that the information is supported by expert consensus where such consensus exists. A query can have many Highly Meets results. Have high standards for the Highly Meets rating. This is especially important for queries with many on-topic results. interpretations, think about how likely the query interpretation is and how helpful the result is. ● A very helpful result for a dominant interpretation should be rated Highly Meets, because it is very helpful for many or most users. Some queries with a dominant interpretation have a FullyM result. ● A very helpful result for a common interpretation may be Highly Meets or Moderately Meets , depending on how likely the interpretation is. ● A very helpful result for a very minor interpretation may be Slightly Meets or lower because few users may be interested in that interpretation. ● There are some interpretations that are so unlikely that results should be rated FailsM . We call these <no chance= interpretations.

This is the official website for Query: [trader joes] Trader Joe’s, which has highly authoritative information on the User Location: Charlotte, businesses’ stores, promotions, North Carolina recipes, news, etc. User Intent: There are Note: This result block is not two possible user intents: Fully Meets because users who most users probably want want information to visit the store to visit a nearby location or have to do a bit of work to find go to the website. that information4it is not

User Intent: There are Note: This result block is not two possible user intents: Fully Meets because users who most users probably want want information to visit the to visit the car dealership dealership have to do a bit of or go to the website. work to find that information4it is not immediately available.

User Intent: Find Note: This result block is not information about this Fully Meets4 even though there library or go to the is a website link, it is not very website. satisfying for users who want to go to the website because it is not as informative as a web search result block with a title, snippet, and URL. There is only one library in the user location (Belmont, California) . The result block Query: [belmont library] shows information about the library, and is especially helpful User Location: Belmont, for users who want to visit the California location. User Intent: Find Note: This result block is not information about this Fully Meets4 even though there library or go to the is a website link, it is not very website. satisfying for users who want to go to the website because it is not as informative as a web search result block with a title, snippet, and URL. The user intent is too broad to have a Fully Meets result. However, this is The Query: [decemberists] Decemberists’ official website and has a lot of content including User Intent: Learn more tour dates, music videos, the about the band, The latest album, etc. The result is Decemberists. uniquely authoritative and most users would be very satisfied by this website. The user intent is too broad to have a Fully Meets result. Query: [celine dion] However, this is Celine Dion’s official website and has a lot of User Intent: Learn more content including recent news, about the singer Celine Dion. videos, etc. The result is uniquely authoritative and most users would be very satisfied by this website.

• Modernized Examples: Focused on popular platforms and video formats, removing outdated examples.
• Expanded Rating Guidance: Specifically for forum pages.
• Consistent Language Updates: Across various sections.
• Refinement of Page Quality Rating: Focused on Main Content Quality, Website and Content Creator Reputation, and E-E-A-T.

14.0

Here is some guidance about assigning Needs Met and Page Quality ratings: ● Useless results should always be rated FailsM , even if the landing page has a high Page Quality rating. Useless is useless. ● On-topic, helpful, but low Page Quality results should get lower Needs Met ratings than on-topic, helpful, and high Page Quality results. The Needs Met scale encompasses all aspects of <helpfulness,= and many users find low Page Quality results less helpful than high Page Quality results. Your ratings should reflect this. ● The HM rating should be given to helpful, high Page Quality pages that are a good fit for the query. The HM rating may also be used for results that are very helpful, medium quality, and have other very desirable characteristics, such as very recent information. ● The HM rating may not be appropriate if a page has low Page Quality or has any other undesirable characteristics, such as outdated or inaccurate information, or if it is a poor fit for the query. We have very high standards for the HM rating. ● SM is often an appropriate rating for low quality but on-topic pages. However, a page can have such low Page Quality that it is useless for nearly all queries. Gibberish pages are a good example of pages with low Page Quality that should be rated FailsM . An exception to this is queries with clear website intent, where the target website should be rated FullyM even if the page has low Page Quality. ● Remember that if a page lacks a beneficial purpose, it should always be rated Lowest Page Quality – regardless of the page’s Needs Met rating or how well-designed the page may be. Please review Section 4.1 for a summary of other types of Lowest Page Quality pages. Query and User Intent Result Block and LP, Needs Met Rating, Page Quality Rating Explanation Needs Met : This page would be helpful for few users. Page Quality : Read this content out loud. The article on the LP is shallow and unintentionally humorous. The MC has four paragraphs of text, but it basically says that electric cars use Query: [how do electric electricity and have to be cars work] charged. The page also lacks E-E-A-T. Low is an appropriate User Intent: Find rating. information about how electric cars work. Needs Met : The query asks how electric cars work, but the article on the LP is about mobility scooters. Page Quality : There is a lot of helpful MC on the landing page on Wikipedia.org. The page has high E-E-A-T. Medium+ to High is an appropriate rating.

Query and User Intent Result Block and LP, Needs Met Rating, Page Quality Rating Explanation

15.2.3.

15.2.3 Needs Met Rating for Clear p*rn Intent Queries For very clear p*rn intent queries, assign a rating to the p*rn landing page based on how helpful it is for the user. Even though there is p*rn intent, the page should still be assigned a p*rn flag. Do not simply rate all p*rn pages for p*rn queries as MM or HM . Even though the query is p*rn and the result is p*rn, the page must fit the query and be helpful to get a high Needs Met rating. Pages that provide a poor user experience, such as pages that try to download malicious software, should also receive low ratings, even if they have some images appropriate for the query. Query User Intent Landing Page Rating p*rn Flag? [freeones], English (US) Navigate to the Freeones Example:

15.4.2

Videos are often an example where foreign language pages are helpful and desired. Think about user intent and what pages are good for users. If the query <asks= for a foreign language song, band, film, sporting event, etc., then a video of the song, band, film, sporting event, etc. is helpful since it can probably be understood or enjoyed even though it is in a foreign language. For these types of queries, foreign language results are often expected. If the video is someone talking about the song, band, film, or event, the result probably cannot be understood or enjoyed and should be assigned a FailsM rating and the Foreign Language flag. Note: If you are unable to evaluate the Page Quality rating of a Foreign Language result, you do not need to assign a Page Quality rating and can leave the slider at N/A.

16.0

16.0 Rating Queries with Multiple Interpretations and Intents Some queries really only have one meaning. Consider the query [iphone], English (US). There may be different user intents for this query (research iPhones, buy an iPhone, go to the iPhone page on Apple’s website), but all users are basically referring to the same thing: the phone made by Apple, Inc. Some queries truly have different possible meanings. Consider the query [apple], English (US). Some users may want to find more information on the computer brand or the fruit. We refer to these different meanings as query interpretations. When giving Needs Met ratings for results involving different query

17.0

because Discover is a popular credit card company. Query: [credit cards] Page Quality : Discover is a company with User Intent: Users are high E-E-A-T that offers services for credit probably looking to sign cards, banking, etc. with a good reputation. up for a credit card High+ to Highest is an appropriate rating. online, or want to This page offers a list of top credit cards in research credit cards a variety of categories. before signing up.

• Refreshed YMYL (Your Money Your Life) Focus: Concentrated on high-accuracy topics to prevent significant harm.

2.3

2.3 Your Money or Your Life (YMYL) Topics Pages on the World Wide Web are about a vast variety of topics. Some topics have a high risk of harm because content about these topics could significantly impact the health, financial stability, or safety of people, or the welfare or well-being of society. We call these topics <Your Money or Your Life= or YMYL.

13.3

13.3 Highly Meets (HM) A rating of Highly Meets is assigned to results that meet the needs of many or most users. Highly Meets results are highly satisfying and a good <fit= for the query. In addition, they often have some or all of the following characteristics: high quality, authoritative, entertaining, and/or recent (e.g., breaking news on a topic). In order to receive a Highly Meets rating, information pages such as encyclopedia articles and news articles must be accurate and highly credible. Highly Meets medical and scientific information pages must represent well-established scientific/medical consensus unless the user is clearly seeking an alternative viewpoint. For all information pages, you must check for accuracy and confirm that the information is supported by expert consensus where such consensus exists. A query can have many Highly Meets results. Have high standards for the Highly Meets rating. This is especially important for queries with many on-topic results. interpretations, think about how likely the query interpretation is and how helpful the result is. ● A very helpful result for a dominant interpretation should be rated Highly Meets, because it is very helpful for many or most users. Some queries with a dominant interpretation have a FullyM result. ● A very helpful result for a common interpretation may be Highly Meets or Moderately Meets , depending on how likely the interpretation is. ● A very helpful result for a very minor interpretation may be Slightly Meets or lower because few users may be interested in that interpretation. ● There are some interpretations that are so unlikely that results should be rated FailsM . We call these <no chance= interpretations.

• Device Agnostic Language: To be applicable across all device types.
• General Minor Updates: Including language, examples, and fixing of typos.

These changes reflect Google’s ongoing efforts to improve the relevance and quality of search results, emphasizing the importance of content quality, authoritativeness, and user experience. The 2023 guidelines show a greater emphasis on modernization, with specific attention to popular platforms and video formats, and a simplified approach to Needs Met ratings. The 2022 guidelines focused more on broadening the applicability of the guidelines to various types of websites and refining the core pillars of Page Quality Rating​.

Summary of the Quality Rater Guidelines

The Google Quality Rater Guidelines provide comprehensive instructions for evaluating the quality of web pages and search results. Here’s a detailed summary of the key sections:

1. Introduction: The guidelines begin with an overview of the purpose of the document, emphasizing its role in improving search results by providing human feedback to Google’s algorithms.
2. Understanding Webpages and Websites: This section instructs raters on how to evaluate a website’s purpose, its content, and the context of the information provided.
3. Page Quality Rating: It delves into the criteria for assessing the quality of a page, focusing on factors like experience, expertise, authoritativeness, trustworthiness (E-E-A-T), and the quality of content.
4. Needs Met Rating: This part focuses on how well a page meets a user’s search intent. It includes various scenarios and examples to illustrate different levels of user satisfaction.
5. Understanding User Intent: The guidelines emphasize the importance of understanding what users are looking for when they perform a search and how this affects the evaluation of search results.
6. Mobile User Needs: Given the increasing use of mobile devices for internet access, this section addresses the unique needs and challenges of mobile users.
7. Additional Rating Tasks and Instructions: This segment covers various other tasks that raters may encounter, such as rating dictionary results, understanding the user location, and handling misspelled queries.
8. Appendix and Additional Resources: The document concludes with an appendix that provides additional resources and explanations to assist raters in their tasks.

Throughout the document, the emphasis is on providing accurate, unbiased ratings based on the guidelines, with numerous examples and scenarios to guide raters in various situations. The guidelines also highlight the evolving nature of web content and the importance of staying current with new types of content and changing user behaviors.

Different Versions of the Quality Rater Guidelines

Here are different Versions of the QRG to download:

1. Quality Rater Guidelines 2014
2. Quality Rater Guidelines 2015
3. Quality Rater Guidelines May 2017
4. Quality Rater Guidelines July 2017
5. Quality Rater Guidelines 2022