Google Play Store Listing Experiments: How to A/B Test Your App Icon and Screenshots

Last reviewed against the current Google Play Console documentation. Interface labels and available options can change; where this article and your Play Console disagree, your Console is authoritative.

Most listing changes are guesses. A designer prefers the bolder icon, the founder prefers the screenshot with the cleanest background, and the change ships because someone had conviction. Whether it helped is usually unknown, because nothing measured it.

Google Play Store listing experiments exist to replace that guess with a comparison: real store traffic, split between your current listing and a variant, scored on a metric you choose. This guide covers what the feature does, how to set one up, how to read the result, and where the honest limits are.

Quick answer: what a Store Listing Experiment is

The feature is documented in Run A/B tests on your store listing. Global experiments were renamed to default graphics experiments; the functionality is otherwise unchanged.

What a Store Listing Experiment actually does

For a published app, you can test a variant of a supported asset against your current listing. Visitors are split between the current listing and the experimental variant, and Google reports which one performed better on the metric you selected.

Three consequences follow from that definition, and each one prevents a common misreading:

• It is a comparison, not an audit. The experiment tells you which of two versions got more of the chosen action. It does not tell you whether the listing is good, whether the app is good, or why people behave the way they do.
• It only measures what the metric measures. Install clicks and open clicks are acquisition signals. They are not retention, revenue, or product quality.
• It applies to a listing, not to a ranking. Nothing in the documentation says a winning experiment improves keyword rankings. Better listing performance and search ranking are separate systems.

Google also states clearly that the outcome is a recommendation, not an automatic change. You decide whether to apply a variant.

Which store listing elements can you test?

Google documents the supported assets by experiment type. The list is narrower than the listing itself, and it is worth internalising because it shapes what questions you can actually ask.

• App icon — available in both experiment types.
• Feature graphic — available in both experiment types.
• Screenshots — available in both experiment types.
• Short and full descriptions — documented for localized experiments, where the experiment covers text as well as graphics.

What is not on the list matters too. Your app name, the category, the developer name, the video, and pricing are not documented as experiment assets. If you want to change one of those, you change it directly and accept that you will not have a controlled comparison — or you find a related variable you can test, such as the screenshot that carries the same message as a name change.

If you need the exact technical requirements for producing these assets — icon dimensions, screenshot formats, alt text, device-specific rules — those are covered in the Google Play store listing guide rather than repeated here. An experiment does not relax those requirements: variants still have to pass the same asset rules and the same policy review.

Default graphics vs localized experiments

Google documents two experiment types, and the difference is which audience sees your variant. Choosing the wrong one is the single easiest way to end up with a result you cannot act on.

The exclusion rule that surprises people

If you have added localized graphic assets in any language, users viewing your listing in that language are excluded from your default graphics experiment — even when you are testing the icon. Google’s example is explicit: with English as the default language and a French feature graphic, users viewing in French are excluded from a default graphics experiment, icon included.

Practically, that means a default graphics experiment measures the audience that sees your default graphics, not everyone who visits your listing. If your most important market has localized assets, a global result may not represent it.

How many can run at once

Google’s documentation states that, for each app, you can run one default graphics experiment or up to five localized experiments at the same time. That sentence describes one or the other. Treat “one of each simultaneously” as something to confirm in your own Console rather than assuming, especially if both tests would touch the same audience.

Before you start: prerequisites to check

Experiments are for published apps, and they are not offered on every listing at every moment.

• The app must be published. Google describes experiments as testing variants against your current version for published apps.
• The listing must be fully rolled out. If Set up experiment is not available, follow the prompt at the top of the listing’s Details page to roll out the listing and get it ready, then return.
• The listing must be eligible for the asset you want to test. A different language needs a listing in that language before a localized experiment can measure it.
• Variants must pass review. Assets you provide are submitted through the normal publishing flow, so plan for review time before you expect the test to start.
• Traffic must be sufficient. The setup page estimates the time and volume needed; a listing without enough visitors may never produce a decisive result.

One more practical prerequisite: decide in advance what you will do with each possible outcome. If “apply the variant only if it wins on install clicks, otherwise keep the current listing” is not agreed before the test starts, the result becomes a debate instead of a decision.

How to set up a store listing experiment

The steps below follow the current Play Console documentation. Label wording can change, so match the meaning rather than memorising the exact words.

1. Open the Store listings page. In Play Console, go to Grow users > Store presence > Store listings.
2. Start the experiment. In the Experiment column for the listing you want, choose Set up. Alternatively, open the listing’s Details page with the row arrow and choose Set up experiment if it is available.
3. Name it. Experiment names are for your team only and are limited to 50 characters. Use a name that still makes sense months later, such as icon-search-vs-onboarding.
4. Choose the type. Select Graphics experiment on default listing or Experiment on a localized listing. For a localized experiment, choose the languages to test.
5. Choose the target metric. Pick Unique user install clicks, Unique user open clicks, or Unique user pre-registration clicks. This is the metric that decides the result.
6. Read the estimate. The creation page estimates the time and the number of acquisitions, opens or pre-registrations needed for a statistically significant result. If the estimate is implausible for your timeline, change the design rather than starting anyway.
7. Review advanced settings if needed. The default settings are described as giving results in the shortest time with reasonable accuracy. Adjusting them can extend the run: number of variants (up to two), experiment audience percentage, minimum detectable effect, and confidence level.
8. Pick the asset. Choose the asset type you want to test. Google recommends testing changes to one asset at a time so you can be more certain about what caused the difference.
9. Add the variants. Provide the alternative asset or assets. Make the change big enough to be a real difference — a two-pixel adjustment is not a hypothesis.
10. Review and save. Check the variants, then click Save. Return to the Details page or go to the publishing overview and submit the changes for review.

What the advanced settings mean

• Number of variants: how many experimental versions compete with the current listing. Google documents up to two.
• Experiment audience percentage: the share of visitors who see a variant instead of the current listing. That share is split equally across variants.
• Minimum detectable effect: the smallest difference that will be treated as a real win. If the observed difference is smaller, the experiment is considered a draw.
• Confidence level: how often the confidence interval contains the true performance. Raising it reduces the chance of a false positive, and usually means a longer run.

A smaller minimum detectable effect or a higher confidence level means you are asking the test to detect a subtler difference, which needs more data. That is why the estimate moves when you change these settings.

How to choose a useful hypothesis and variants

A weak hypothesis produces a result you cannot use, even when the experiment “wins”. The structure below keeps the test interpretable.

1. Start with a claim about behaviour

A useful hypothesis names the change, the reason it should matter, and the audience. “A screenshot showing the search feature as the first image will produce more install clicks from people browsing for a note-taking app, because it answers ‘can it find my notes?’ before they scroll.” That is testable. “Make the listing better” is not.

2. Change one thing

Google explicitly recommends testing one asset at a time. If you replace the icon and the first three screenshots at once, a win tells you the combination worked, not which part did the work — and you have no way to keep the good half.

3. Make the variants meaningfully different

An A/B test compares two answers to a question. If the variants differ only in a shade of blue, the honest expected outcome is a draw, which is information but usually not the information you needed. Test a different concept, not a refinement of the same one.

4. Match the test to the traffic you have

Traffic is the constraint that decides how much you can learn. A listing with large daily volume can resolve small differences; a new app with a few dozen visitors a day may only be able to detect a large one. Localized experiments concentrate traffic in a language, which can help — unless that language is a small share of your audience, in which case the test may never finish.

How to choose a target metric

Google documents three target metrics, and the definitions are narrower than the names suggest:

• Unique user install clicks — users who installed your app and did not have it installed on any other devices at the time.
• Unique user open clicks — installed users who opened your app.
• Unique user pre-registration clicks — users who pre-registered for your app.

Choosing between them is a question about the decision you are trying to make:

• Installs answer “does this asset get more people to install?” That is the usual question for icon and screenshot tests.
• Opens answer a different question: among people who installed, does the asset attract users who actually launch the app? It can be a useful sanity check when a screenshot promises something the app does not deliver quickly.
• Pre-registrations only make sense for an app using pre-registration, where the goal is interest before launch.

What these metrics do not measure

Two further limits are worth stating plainly. Results are not projectable to a different audience: a win among users viewing the default language says nothing certain about a localized listing you did not test. And a win does not transfer to a different seasonal or campaign context, because the traffic mix changes.

How to read experiment results

Results live on the Experiments tab, in the Experiment column on the Listings tab, and on a listing’s Details page. The Results section summarises performance on your target metric, explains the outcome using the confidence interval and minimum detectable effect, and shows a recommended action where one applies.

When more than one variant performs better than the current listing, or the outcome is a draw, the decision is yours. Review the numbers and pick the variant you want to apply — or keep what you have. That is a judgment call, and it is worth writing down the reason so the next test can build on it.

How long an experiment should run

There is no universal duration. Google’s setup page estimates the time needed based on your traffic and settings, and the estimate is the closest thing to an official answer. Two documented facts help you plan:

• Experiments stop automatically after six months. After that, no new data is collected, no variants are applied, and traffic reverts to the current listing. The completed experiment remains available to review.
• Google tells you when a result is close. If you have less than a week before a result is expected, the interface advises you.

Stopping early to “take the win” cuts against the whole point: the confidence interval and minimum detectable effect were chosen for the full sample. If the result still says “More data needed”, waiting is the correct action.

Turning on notifications

To be notified by email when experiments complete, set your preferences on the Notifications page in Play Console. Without them, a finished experiment can sit unread for weeks.

Troubleshooting: why an experiment looks stuck

Common A/B testing mistakes

1. Testing a bundle of changes. If three assets change at once, the result cannot be attributed. Test one asset.
2. Treating a draw as failure. A draw means the alternatives were equivalent on your metric. That is a usable answer, and it protects you from shipping a change that only looked better.
3. Stopping the moment a variant pulls ahead. Early differences are the least reliable. The interface tells you when the data is sufficient; wait for it.
4. Running a test you cannot power. If the estimate says months and your launch is next week, the experiment cannot answer your question. Change the question.
5. Testing a language with almost no traffic. A localized experiment is only useful where the audience exists.
6. Optimising the icon while the screenshots carry the decision. Test the asset most likely to change behaviour, not the easiest one to redesign.
7. Ignoring the exclusion rule. A default graphics experiment excludes users who have localized graphics in their language, so a global average may not represent your key market.
8. Assuming installs equal quality. An asset that oversells can raise installs and hurt retention. Validate a winning creative against what the app actually does.
9. Never re-testing. Audiences and markets shift. An asset that won a year ago is a hypothesis again.
10. Forgetting policy. A variant that misrepresents the app is rejected at review regardless of how well it might have performed.

Experiments vs custom store listings

These two features are often confused because both change what users see. They answer different questions.

• An experiment compares alternatives to learn which performs better. It is temporary, split-tested, and ends with a decision.
• A custom store listing targets a segment with a tailored listing. It runs continuously for that audience and is not a controlled test.

Google documents custom listings for audiences such as users in different countries, pre-registered users, users who search for specific keywords, and inactive users, as well as for visitors arriving from a unique custom listing URL or a Google Ads campaign. A detail worth knowing before you plan around it: custom store listings do not include automated translations, so an audience with a different language needs an added translation if you want them to read the listing in that language.

The practical relationship is sequential. Use a custom listing to serve a segment better. Use an experiment when you want evidence about which version of an asset works — including inside a custom listing, since Google documents experiments for both default and custom listings.

A practical experiment checklist

Frequently asked questions

What are Google Play Store Listing Experiments?

A Play Console feature that compares variants of supported listing assets against your current listing using real store traffic and a chosen metric. See Google’s experiments documentation.

Can I A/B test my Google Play app icon?

Yes. The icon is documented as a testable asset in both default graphics and localized experiments. The variant still has to meet the normal icon requirements and pass review.

Can I test screenshots in Google Play Console?

Yes. Screenshots are testable in both experiment types, and Google recommends changing one asset at a time so the cause of any difference is clear.

What is the difference between default graphics and localized experiments?

A default graphics experiment tests icon, feature graphic, or screenshots in your default listing language and can reach all users of that listing. A localized experiment tests graphics and/or descriptions in up to five chosen languages and is shown only to users viewing those languages.

How long should a Store Listing Experiment run?

Until the Console reports a result. Google estimates the required time and traffic on the setup page, and experiments stop automatically after six months with traffic reverting to the current listing. There is no universal fixed duration.

What does “More data needed” mean?

Not enough data has been collected to return a result. Leave the experiment running; Google advises you when a result is expected within a week.

What should I do if the result is a draw?

Review the results and decide which variant, if any, to apply. A draw usually means the variants were too similar or the difference was below your minimum detectable effect; the next test should use a more distinct concept.

Can an experiment guarantee more installs?

No. An experiment identifies which of two versions performed better on one metric during one period. It cannot guarantee installs, retention, revenue, or rankings.

What is the difference between an experiment and a custom store listing?

An experiment is a temporary, split test that ends with a decision about which version is better. A custom store listing is a permanent variant shown to a specific audience, such as users in a country, pre-registrants, users searching particular keywords, or traffic from a custom URL or ads campaign.

Can I test multiple changes at the same time?

You can run several experiments within the documented limits — one default graphics experiment or up to five localized experiments per app — but Google recommends testing one asset at a time so you can attribute the result.

Does a winning experiment improve my keyword rankings?

That is not documented. A better-performing listing and search ranking are different systems; treat ranking claims as unproven and avoid relying on listing experiments to change them.

Do experiments work for custom store listings?

Google’s documentation states that you can run experiments for default and custom store listings. Confirm the option in the listing you intend to test, since availability depends on the listing’s state.

Conclusion: test the decision, not the decoration

Store listing experiments are most useful when they answer a question you were going to answer by opinion. That means a hypothesis worth stating, one asset changed at a time, a metric that matches the decision, and enough patience to let the data arrive.

The limits are part of the method, not a flaw in it. A result speaks only to the metric, the audience, and the period it covered. It cannot promise installs, retention, revenue, or rankings — and treating a draw or “More data needed” honestly is what keeps the next test worth running.

Start with the asset most likely to change behaviour, give the test the traffic it needs, apply only what you are willing to keep, and write down what you learned. That is a slower process than guessing, and it is the only one that compounds.

Official sources

• Google Play Console Help — Run A/B tests on your store listing
• Google Play Console Help — Custom store listings
• Google Play Console Help — Pre-registration
• Google Play Console — Store listing experiments overview