Section2:Plate Uniformity and Signal Variability Assessment

From Assay Guidance Wiki

Overview

All assays should have a plate uniformity assessment. For new assays the plate uniformity study should be run over 3 days to assess uniformity and separation of signals, using DMSO at the concentration to be used in screening. For assay transfers (See Section A for the definition of an assay transfer) the plate uniformity study need be only 2 days.

The actual variability tests are conducted on three types of signals.

• "Max" signal: This measures the maximum signal. For agonist assays this would be maximal response of an agonist; for potentiator assays this would be an EC10 concentration of a standard agonist (the actual percentage is as per protocol and may not be 10% in some cases) plus maximal concentration of a standard potentiator. For inhibition type assays this would be a reaction with an EC80 concentration of a standard agonist (again the actual percentage is as per protocol, and may not be 80%). For inverse agonist assays this would be the untreated constitutively active condition in the presence of DMSO alone.
• "Min" signal: This measures the background signal. For agonist assays this is the basal signal. For potentiator assays this is an EC10 concentration of agonist. For inhibitor assays, including receptor-binding assays, this is an EC80 concentration of the standard agonist plus a maximally inhibiting concentration of a standard antagonist (preferred) or unstimulated reaction.
• "Mid" signal: This estimates the signal variability at some point between the maximum and minimum signals. Typically, for agonist assays the mid-point is reached by adding an EC50 concentration of a full agonist/activator compound; for potentiator assays it is an EC10 concentration of agonist plus EC50 concentration of a potentiator; and for inhibitor assays it is an EC80 concentration of an agonist plus an IC50 concentration of a standard inhibitor to each well.

N.B. If calibration of the signals is required then the concentration levels and all analyses are to be conducted on the calibrated responses and not the raw plate reader counts. It is a requirement that the raw signals lie within the range of the calibration curve, ie at most 1-2% of the wells lie outside the calibration range (i.e. above the fitted top or below the fitted bottom of the calibration curve).

Two different plate formats exist for the plate uniformity studies: an Interleaved-Signal format where all signals are on all plates, but varied systematically so that over all plates on a given day each signal is observed in each well, and a Uniform-Signal plate format where each signal is run uniformly across entire plates. There are no universal advantages to either format. The Interleaved-Signal format can be used in all instances and requires fewer plates. The Uniform-Signal format is easier to run, and more useful for detecting non-uniform signals, but takes more plates in total. It also should not be used if signals vary across plates on a given day. See Section C.3.d. for examples of when they should not be used.

Interleaved-Signal Format

Procedure

You should use the following plate layouts, for which Excel analysis templates have been developed. These layouts have a combination of wells producing max, min, and mid signals on a plate with proper statistical design. Use the same plate formats on all days of the test. Do not change the concentration producing the mid point signal over the course of the test. See Section C.2.d. for a further discussion about midpoint accuracy. The trials should use independently prepared reagents and preferably be run on separate days.

Summary Signal Calculations and Plate Acceptance Criteria

The points below describe these calculations and acceptance criteria. The overall requirement for the signals is that the raw signals are sufficiently tight and that there is sufficient separation between the max and min signals to conduct screening. Calculations and acceptance criteria are summarized as follows.

1. Outliers should be flagged with an asterisk in the plate input section. The outliers should be “obvious”, and the rate of outliers should be less than 2 percent (i.e. on average less than 2 on a 96 well plate, 8 on a 384 well plate).
2. Compute the mean (AVG), SD, and CV (of the mean) for each signal (max, mid, min) on each plate. Note that the CV should be calculated taking into account the number of wells per test compound per concentration that will be used in the production assay. For example if in the production assay duplicate wells will be run for each concentration of each test substance then

   More generally, if there will be n wells per test compound per concentration then

   The acceptance criterion are that the CV’s of each signal be less than or equal to 20%. Note that the min signal often fails to meet this criterion, especially for those assays whose min signal mean is very low. An alternate acceptance criterion for the min signal is SDmin ≤ both SDmid and SDmax. All plates should pass all signal criteria (ie all Max and Mid signals should have CV’s less than 20% and all Min signals should either pass the CV criteria or all Min signals should pass the SD criteria).
3. For each of the mid-signal wells, compute a percent activity for agonist or stimulation assay relative to the means of the max and min signals on that plate, For inhibition assays compute percent inhibition for each mid-signal well, where %Inhibition = 100 - %Activity.
4. Compute the mean and SD for the mid-signal percent activity values on each plate. The acceptance criterion is SDmid ≤ 20 on all plates.
5. Compute a Signal Window (SW) or Z’ factor (Z’) for each plate, as described below. The acceptance criterion SW ≥ 2 or Z’ ≥ 0.4 on all plates (either all SW’s ≥ 2 or all Z’ ≥ 0.4).

The formula for the signal window is:

where n is the number of replicates of the test substance that will be used in the production assay. Instead of the SW the Z’ factor can be used to evaluate the signal separation, where the only difference is the denominator (AVGmax – AVGmin) is used instead of SDmax. The complete formula is:

If one assumes that the SD of the max signal is at least as large as the SD of the min signal, then the Z’ factor will be within a specific range for a given signal window, as illustrated in the following graph. Note that Z’ values greater than 1 are possible only if AVGmax < AVGmin, and so the templates also check that all Z’ values are less than 1.

The recommended acceptance criterion is Z’ factor ≥ 0.4, which is comparable to a SW ≥ 2. Either measure could be used.

Spatial Uniformity Assessment

A scatter plot (see examples below) can reveal patterns of drift, edge effects and other systematic sources of variability. The response is plotted against well number, where the wells are ordered either by row first, then by column, or by column first, then by row. The overall requirement is that plates do not exhibit material edge or drift effects. In general drift or edge effects < 20% are considered non-material, and effects seen only on a single or few plates, and not the predominant pattern are also considered non-material. Some guidelines to detecting and dealing with these problems follow.

No drift or edge effects

The following two plots (of the same data) show an example where there are no edge effects or drift.

Drift

Use the max and mid signals to look for drift. Consider drift associated with the min only if the mean signal is greater than 10% of the maximum signal. Look for significant trends in the signal from left-to-right and top-to-bottom. If you observe drift that exceeds 20% then you have material drift effects. In the example below, the mean of column 1 is 10.6, while the mean of column 10 is 13.8, and the overall mean is 12.2. The drift is 26% [(13.8-10.6)/12.2], and therefore should be investigated.

Edge Effects

Edge effects can contribute to variability, and spotting them can be a helpful troubleshooting technique. Edge effects are sometimes due to evaporation from wells that are incubated for long periods of time. Edge effects can also be caused either by a short incubation time or by plate stacking – these conditions allow the edge wells to reach the desired incubation temperature faster than the inner wells. Edge effects may show up in the data as represented in the following example.

Note: Because of the vertical axis scale, problems in the min and even mid signals may not be visible. Adjusting the scale to highlight the min and mid scales may be necessary to properly examine these signals.

Inter-Plate and Inter-Day Tests

The normalized mid signal should not show any significant shift across plates or days. “Significant” depends to a certain extent on the typical slopes encountered in dose response curves. Thus plate-to-plate or day-to-day variation in the mid point percent activity needs to be assessed in light of the steepness of the dose-response curves of the assay. For receptor binding assays, and other assays with a slope parameter of 1, a 15% difference can correspond to a two-fold change in potency. The template will translate the mean normalized mid-signal to potency shifts across plates and days. There should not be a potency shift ≥2 between any two plates within a day, or ≥2 between any two average day mid point %activities. For functional assays whose slopes may not equal 1 you can enter a “typical” slope into the template. This should be based on the slope of a dose-response curve for the substance used to generate the mid point signal.

For these calculations to have utility the mid point %Inhibition/Activity should be “near” the midpoint. Values within the range of 30-70% are ideal. Studies with mean values outside this range should be discussed with a statistician, especially before any studies are repeated solely for this reason. Also note that the conditions used to obtain the midpoint should not be changed over the course of the plate uniformity study.

Summary of Acceptance Criteria

1. Intra-plate Tests: Each plate should have a
   CVmax and CVmid ≤ 20%,
   CVmin ≤ 20% or SDmin ≤ min(SDmid, SDmax),
   Normalized SDmid ≤ 20,
   SW ≥ 2 or Z’ ≥ 0.4.
2. No material edge, drift or other spatial effects. Note that the templates do not check this criterion
3. Inter-plate and Inter-Day Tests: The normalized average mid-signal should not translate into a fold shift
   > 2 within days,
   > 2 across any two days.

384-well Plate Uniformity Studies

384-well plates contain 16 rows by 24 columns, and one 384-well plate contains the equivalent of four 96-well plates. Two different formats of interleaved plate uniformity templates have been developed. The first layout expands the 96-well plate format into 4 squares. The plate layouts are as follows:

The second is useful for assays using certain automation equipment such as Tecan and Beckman. In that case column 1 of the 96-well plate corresponds to columns 1 and 2 of the 384-well plate, and is laid out in 8 pairs of columns. The plate layouts for it are as follows:

The analysis and acceptance criterion are exactly the same as for 96-well format Plate Uniformity Studies. See Section 2.C.2e for a summary of the acceptance criterion.

Uniform-Signal Plate Layouts

Uniform-Signal plate layouts are an alternative format to conduct the plate uniformity studies. Their main advantage is easier execution since all wells on each plate are exactly the same, and together with heat maps provide for a straightforward assessment of spatial properties. The disadvantages are that this format requires twice as many plates as the Interleaved-Signal format, and that the normalizing calculations are quite artificial in that max and min signals are not on-plate signals and therefore may produce misleading results. See Section C.3.d for further elaboration of this point.

Procedure

Max, Mid and Min signals are prepared as defined in Section C.1. Two plates are run for each signal, making six plates per day. On each plate all wells are the same, i.e. either all Max, all Mid, or all Min. The number of days required is the same as for the Interleaved-Signal layout: three days for new assays, two days for transfers of previously validated assays.

Summary Calculations and Plate Acceptance Criterion

The actual calculations will be performed by the template. Details of the calculations are as follows:

1. Compute the mean (AVG), standard deviation (SD) and Coefficient of Variation (CV) for each plate (as per the Interleaved-Signal format the CV’s should reflect the number of wells per test-condition envisioned in the production assay). Requirements are the same as for Interleaved-Signal format: The CV of each plate should be less than 20%. For the Min plates having SD ≤ SDmid and SDmax, where is the combined standard deviation from the two Mid plate SD’s, and similarly for the Min and Max signals.
2. For each of the Mid signal plates, compute the percent activity for agonist or stimulation assays, and percent inhibition for antagonist or inhibition assays (including binding assays). In this format the calculation is where AVGmin is the average taken over the two Min plate averages, and AVGmax is the average taken over the two Max plate averages. Percent Inhibition = 100 - %Activity.
3. Compute the SD of the normalized signals on each Mid plate. The acceptance criterion is SD%mid ≤ 20.
4. Compute the Z’ factor and/or the SW for each day. The formulas are the same as in Section C.2.b, except that AVGmax and AVGmin are defined as in point 2 above, and SDmax and SDmin are defined as in point 1 above. The acceptance criterion is either all Z’ ≥ 0.4 or all SW ≥ 2.

Spatial Uniformity Assessment

The Excel template provides scatterplots of the plate signals combined across plates and days and is interpreted in a similar manner as the Interleaved-Signal format. The criterion for acceptance is the same as for the interleaved format: No drift or edge effects that exceed 20% of the mean. Also as in the Interleaved-Signal format the presence of these effects should be apparent as the predominant effect, and not seen just in single isolated plates for the assay to be failed by this criterion.

The following example illustrates a spatially uniform result, an edge effect, and a drift effect. Day 1 shows an acceptably uniform result. Day 2 shows an assay with a significant edge effect (25% from the mean edge value to the mean of the interior), and Day 3 shows an assay with significant drift (25% change in mean value from left to right as compared to the average in the middle). If patterns are similar or worse than those depicted in Day 2 or Day 3 then the assay does not pass the spatially uniform requirement.

Inter-Plate and Inter-Day Tests

The Inter-plate and inter-day tests are exactly the same as in Section C.2.d, except the definitions of %Activity and %Inhibition defined above (Section C.3.a) are used in the tests.

Impact of Plate Variation on Validation Results

The Uniform-Signal format does make the assumption that plate variation within each run day is negligible. If this assumption is not correct then many of the diagnostic tests described here will be misleading, and the Interleaved-Signal format should be used instead. In particular, Z’ factors and/or Signal Windows may be incorrect in either direction, and the Inter-plate and Inter-Day tests could possibly fail acceptable assays.

The following example illustrates the problem. The raw signals of one day of an Interleaved-Signal format Plate Uniformity Study are shown on the left in Panel A. The Max and Mid raw signals vary across the 3 plates (Panel A, Plates 1-3), but note that the %Activity is very stable across the 3 plates (Panel B, Plates 1-3). The maximum fold shift across plates is 1.2. The Midpoint Percent Activity plot (Panel B) shows what can happen if you don’t have on-plate Max and Min controls. The three left-hand panels show the plates normalized to their own controls while, to mimic the Uniform-Signal protocol with its off-plate controls, the right hand columns of Panel B show each plate’s mid signal normalized to the plate 3 controls, i.e. “Plate 1” shows the actual plate 1 mid signal normalized to the plate 3 Max and Min signals, “Plate 2” shows the actual plate 2 mid signals normalized to the plate 3 Max and Min signals and “Plate 3” is the plate 3 mid signals normalized to their own controls. In the presence of plate variation the off-plate controls do not effectively normalize the assay. As Panel B shows, plate-to-plate variation in the raw signals can induce the appearance of significant mid-point variation when in fact there is little variation in signals properly normalized to on-plate controls. In this example using off-plate controls Plates 1-3 have a max fold shift of 2.0 which does not pass the inter-plate acceptance criterion.