A/B Testing

The MAX A/B Testing feature allows you to use real-time data to test, and to measure the impact of, any change that you make to your waterfall configuration.

You can experiment with different ad monetization strategies and identify opportunities that will maximize LTV. Such strategies might include adding or removing a network to see incrementality, adding a new price point, or experimenting with regional waterfall optimizations.

See also

• “Test for Success: How to Accelerate Growth Through A/B Testing” from AppLovin’s Blog.
• “3 Solid A/B Tests to Boost Your Revenue” from AppLovin’s Blog.
• Lucas Yalap, CTO of YSO Corp, discusses how they A/B test their monetization setup
• “What is LTV?” from AppLovin’s Blog.

Creating an A/B Test

Take the following steps to create an A/B test:

1. 在 MAX 控制面板中选择 MAX > Mediation > Manage > Ad Units。

2. Click the ad unit for which you would like to establish an A/B test. The Edit Ad Unit window appears.

3. 从⋯菜单中的 Default Waterfall 选项卡，选择 Create A/B Test。

4. 然后，您会看到一个描述该过程的小弹出对话框。勾选 “Copy existing ad unit configuration…” 复选框，然后点击 Create A/B Test 以将现有的瀑布流复制到新的测试中。

   注意

   MAX copies your existing waterfall structure into the test structure you create. You only need to make those changes that you want to test in your new Test Ad Unit. You do not have to build a complete new waterfall from scratch.

5. 当您创建新的 A/B 测试时，默认情况下 MAX 会将您的 Test 配置应用于 50% 的用户。您可以修改此值，例如，如果您希望在较低流量下运行更长时间的测试。为此，请在创建 A/B 测试时更改 Test Group Allocation 的值。

6. 您会在 Default Waterfall 选项卡中看到一个名为 Test 的新切换开关。在此切换设置中，您会看到您的测试配置 (最初与您复制的瀑布流配置相同)。为您的测试输入一个新的 A/B Test Name，然后开始根据您要测试的内容更改瀑布流。

   AppLovin recommends that you name your A/B tests with a taxonomy that reflects the start time for the A/B test (for example, “0812_01UTC_FBBiddingAdded”). This makes it easier for you to keep track of changes and to monitor the results.

   

7. 完成区分测试瀑布流的更改后，点击 Save。MAX 广告单元页面更新以显示您的新 A/B 测试。

8. 当您返回广告单元页面时，默认视图仍将是您原始的控制瀑布流。点击 Control 切换按钮，以切换到 Test，并比较您的两个瀑布流。

   您可以通过在 Default Waterfall 选项卡的 ⋯ 菜单中选择 Preview Waterfall 来可视化每个瀑布流设置。然后，在 Waterfall Preview 对话框中，选择您想要可视化瀑布流的国家。然后，您会看到该国家瀑布流中非竞价方广告平台的可视化展示 (以及补充瀑布流的任何竞价广告平台的列表)。

   

Monitoring Results

Monitor the impact of your test on requests, impressions, and revenue, for 24–48 hours after the test begins. Do this in MAX > Mediation > Analyze > Advanced Reporting or MAX > Mediation > Analyze > A/B Tests. You then can either promote your test waterfall to all your users, or you can deprecate it and revert to the original control configuration. Your A/B test should appear in production and be present in your reports no more than 20 minutes after you save it.

Any change that you notice reflected in requests, impressions, and revenue indicates a change in the IMP/DAU, REQ/DAU, and ARPDAU. This is probably a result of the changes you made in your test waterfall.

The example screenshot above shows a 28% drop in IMP/DAU, a 16% decrease in REQ/DAU, and a 27% drop in ARPDAU. If these results continue after a few hours (to generate a larger sample), this test would be a good one to deprecate, due to the potential revenue loss.

提示

AppLovin recommends that you promote changes that have been stable for 24 hours, above 10,000 impressions per day, and above 2% incrementality, for sustained improvement in your ARPDAU. Deprecate any test that does not result in a lift.

To deprecate or promote a test:

1. 前往广告单元页面的 Default Waterfall 选项卡。
2. 将 Control/Test 切换至 Test。
3. 从 ⋯ 菜单中选择 End A/B Test。
4. 在出现的 End A/B Test 对话框中，点击 Deprecate Test 或 Promote Test。
5. 然后点击 Save 保存。您的更改将在几分钟内在生产环境流量中生效。

Understanding Results

MAX 为测试组中产生超过 10,000 次展示的测试提供 A/B 测试结果页面。本页显示的是测试生成这些展示的次日结果。 A/B Tests 页面 (MAX > Mediation > Analyze > A/B Tests) 列出并链接到这些页面。

You can read this A/B test result page to understand the Network Revenue share-of-voice changes, the ARPDAU lift anticipated with the test, and many other data points that will help you make the right decision to promote or deprecate the test.