App store A/B testing

What each store lets you test

The asymmetry matters for planning. On Android you can test copy, so description and short-description variants are cheap experiments. On iOS the only text lever is the metadata itself, which is not testable — you change it, watch impressions, and treat it as a before-and-after rather than a controlled test.

Designing a test that produces a usable answer

1. 1

   Pick one element and one hypothesis

   Write it as a sentence: 'Leading the first screenshot with the outcome rather than the interface will raise conversion.' If you cannot write it, the test is not ready.

2. 2

   Estimate the sample you need

   Small lifts need large samples. As a working rule, aim for several thousand product page views per variant; a low-traffic app should test bigger, bolder changes rather than subtle ones.

3. 3

   Check the calendar

   Do not start during a metadata change, a store feature, a paid campaign burst, or a seasonal peak in your category.

4. 4

   Run to the plan, not to the first good day

   Early leads reverse constantly. Decide the duration up front and read the result at the end of it.

5. 5

   Ship and re-baseline

   Roll out the winner, then record new baseline metrics before designing the next test.

What to test first

1. Icon — the largest single conversion lever, and the only element visible in every search result.
2. First screenshot — most viewers never swipe, so this is effectively your whole listing.
3. First screenshot caption — outcome-led wording against feature-led wording.
4. Preview video presence — a video helps in some categories and hurts in others; test rather than assume.
5. Screenshot order — reordering an existing set is the cheapest test available.

Mistakes that void the result

• Changing metadata mid-test, which changes the mix of traffic arriving at the page.
• Testing two elements at once and attributing the win to the one you prefer.
• Calling the test early because a variant is ahead on day three.
• Testing during a seasonal spike, when intent is unusually high and results do not generalise.
• Reading installs rather than conversion rate, which conflates reach with persuasion.

Worked example: a test that reversed after week one

A photo app tested two icons. After four days the new variant led by a comfortable margin and the team was ready to call it. They let it run the full three weeks instead, and the result reversed: the incumbent finished ahead, with a wider margin than the early lead the challenger had held.

The reversal is not unusual and the cause is mundane. The first days of a test are weighted by whatever traffic mix happened to arrive — a weekend, a paid burst, a feature placement — and small samples produce large swings. Store traffic is weekday-patterned, so any test shorter than two full weeks is partly measuring the calendar.

The useful outcome was not the icon. It was that the team adopted a fixed rule afterwards: run length declared before launch, one variable, no mid-test app releases, and no calling a result early regardless of how convincing it looks. The rule costs a fortnight per test and removes the class of decision that gets quietly reversed six months later when the numbers stop matching the story.

Rules that keep results usable

Test discipline is mostly a short list of pre-commitments:

• Declare the run length before the test starts, and hold to it.
• One variable per cell — the whole value of a test is attribution.
• No app releases, price changes or campaign bursts mid-test.
• Read conversion rate rather than install count, so traffic volume does not confound the result.
• Record the losing variant and why, so the same idea is not retested in a year.
• Apply the winner everywhere it is relevant, including other listings and paid creative, rather than only where it was tested.