Voice   >   Insights

How to improve optimization efficiency through A/B Test?

Some tips to Improve Your Waterfall Efficiency

04 March 2020 at 01:00 PM

AdTiming Staff AdTiming
During the optimization of waterfall, developers often need to optimize the tiered floor price, test the tiering effect of a new platform, add or remove tiering. These operations will inevitably cause the overall waterfall data, which in turn affects the monetization benefits. Based on this situation, it is extremely significant to find an alternative with low cost, low risk, and high efficiency. The A / B Test can distribute the traffic into proportions. The divided traffic can be set with different monetization schemes, which can minimize the risks caused by changes in waterfall. Let's take a closer look at the method and principle of A/B Test.

What is A / B Test?

The premise of A / B Test is to first create a hypothesis about a factor, and then divide the traffic users into two groups, A and B, and set different mediation settings for each group. By analyzing the data indicators, we can find out which strategy is more conducive to achieve the expectation, and eventually switch all traffic to the solution that meets the goal.

Generally speaking, A / B Test is often divided into Group A (Control Group) and Group B (Test Group). By comparing the results of the monetization data of the two groups, it helps developers to identify which monetization strategy is better.

What are the preparations before the A / B Test starts?

1. Understand the test objectives of A / B Test

First, analyze the effectiveness of the data of the existing monetization strategy, find the variables that are expected to achieve better results, and form a test group. It is important to choose a reasonable control group and test group variables for testing. The unique variable method is the most intuitive and easiest way to analyze the factors that affect the results. Once the test goals are determined, the creation of the A / B Test can begin.

Secondly, the contrast dimension of the A / B scheme needs to be clarified before the test. So which data dimensions can be improved? Using AdTiming's foreground, you can find that there are the following dimensions to test:

● Ad Network dimension
You can compare the monetization efficiency of the two sets of waterfalls by adding layers or deleting layers.

● Segment traffic dimension
You can compare different groups of monetization for different advertising frequencies, user access times, device models, and versions. It is also possible to estimate eCPM based on the user's multiple attributes for audience granularity.

● Dimensions of sales methods
It includes Header Bidding, automatic waterfall, and manual waterfall. Header Bidding requests multiple Programmatic Buyers in parallel to perform real-time bidding for each Ad Request, without wasting every opportunity for high Ad Request prices.

2. Test samples and test cycle planning

Traffic division refers to randomly dividing the current population into a proportion, one as the control group and the other as the test group. Of course, the specific ratio can be customized during the test. For example, due to the test group's uncertainty in the data, for the sake of insurance, the traffic division ratio between the control group and the test group can be 70% and 30%, 80% and 20%, etc. Thus, even the data fluctuates greatly, and it will not have a great impact on the overall revenue.

The test period refers to the entire time period after the time the test group is created from the time when the test group is turned on to the time the test group is closed. The length of the test cycle depends on the tested sample size and often reflects the stability of the data volume.

After the optimal test plan is obtained, the strategy of gradually releasing the traffic can be adopted to maximize the monetization profit.

3. Planning and preparation of Group B Mediation

After the test objectives of Group B are determined, relevant preparations need to be made in advance for the variables to be tested. Since Group A and Group B cannot use the same instance, first we need to create the placements of the relevant platform of Group B and complete the setting of the floor price, then create a new instance and fill in the corresponding platform placement ID, and finally A / B Test Type B to complete.

What are the usual considerations for creating group B?

1. Factors that determine the rationality of waterfall-level data

A. Prioritization of eCPM and waterfall levels should match. The level of eCPM plays a crucial role in revenue, and it is also an important indicator for developers. In monetization optimization, different Ad Networks have different eCPM floor price algorithms, which means that even if the floor price is set from high to low according to the priority of the waterfall level, the actual eCPM may not be the same. So we need to adjust the order of the waterfall level according to the actual eCPM level when the amount of data reaches a certain accumulation.

B. The fill rate of different waterfall levels should be reasonable. There are typical data problems in monetized data:

Typical problem No.1
The fill rate of the first layer is too high, even reaching more than 80%, which means that the request level can basically be displayed by the first layer, that is, the bottom price of the first layer is set too low and the revenue cannot be maximized.

Solution: We use Group B test, add one or more layers as the first level on the basis of the current level, and increase the reserve price to ensure the maximum profit.

After the group A and group B add layers and adjust the prices of other levels, the waterfall level is as follows:

Typical problem No.2:
The fill rate of the first layer is too low, and the floor price is set unreasonably, which means that most of the display data is displayed at the bottom level, which has a huge impact on revenue.

Solution: We use Group B tests to lower all the floor price settings for all levels except the bottom level to ensure a certain fill rate and impression volume to maximize revenue.

After the group A and group B add layers and adjust the prices of other levels, the waterfall level is as follows: C. The waterfall level is too finely divided. If the magnitude of the entire placements is not very large, extremely fine division will cause the impression volume of each layer to be very low, and the algorithm of each Ad Network has a certain relationship with the impression, and it will also affect the eCPM as a whole, thereby affecting profits in the end.

D. The eCPM gap between adjacent levels should be reasonable. This is also an aspect that is relatively easy to be ignored in the monetization. In addition to observing the fill rate and eCPM, you also need to pay attention to the eCPM value before each level. If the spacing is too large, you can increase the middle level to increase revenue.

2. The actual effect of the new ADN platform placement

At present, AdTiming supports 12 Ad Networks. In the early stage, the number of platforms is often limited due to some objective factors. For newly added platforms, the performance can be preset based on historical data collected in advance. Different platforms have different performances in different categories, countries, and advertising situations. Preset test groups based on historical experience data can make better use of them while reduce unnecessary time wastage. The only variable in group B compared to group A is to increase the level of the new Ad Networks, and test by traffic division to get the optimal return.

3. Test variables must be unique

The uniqueness of variable selection is very important when performing A / B Test. When analyzing the final data, you can clearly understand the influencing factors of data changes. Therefore, when we carry out plan B, we need to choose the only variable factor with plan A, and at the same time we test waterfall B plan, then plan A should not be changed during period B, otherwise it will affect the judgment of the effect of variable A and B.

4. Test confidence time and data magnitude

How to more accurately determine which of the A and B solutions is optimal? This requires sufficient time and data accumulation. In time, it usually takes more than 3 days to complete the test in order to rule out accidental effects due to seasonality, the platform itself, and other reasons; in data, it usually needs to reach 5k or more for reference. The above data is for reference only. The specific confidence time and data may vary depending on the application.

How is AB Test implemented in the system?

On the AdTiming platform, you can implement AB Test in a more concise way. The specific steps are as follows:

Step 1:Navigate to Mediation to AB test and click "Create A / B Test" Step 2: Define your AB Test Step 3: Configuration needs to be completed before starting A / B Test First, navigate directly to the Ad Network Setup-B TEST Group to set it up, and configure the relevant parameters of the integrated Ad Network in Operation. Second, after completing the Ad Network, start the related configuration of Mediation Setting. Third, start formal testing. Step 4: Analyze the report and terminate the test


At this point, you have a certain understanding of the relevant content of A / B Test. It is a low-risk test function in improving the realization of profit, which is very useful to developers.