Issues with alpha values in texture-mapped billboards

Let's say you have a texture map with alpha values, for a plant.  These generally consist of the plant on a background of a solid color.  The background may be black, white or some other color, which in theory doesn't matter because the alpha values will mask it away.  As an example, consider the top of this palm tree:

color (RGB)

alpha (A)

However, on the soft edges of the plant, there are boundary texels which contain some combination of the plant's color and the background.  The contribution from the background often produces a "fuzzy halo" effect when the texture map is actually used:

Against the sky, the palm doesn't look so bad.  This is because that's the color the palm was originally photographed against, so traces of that color are blended into its halo.  However, against dark areas of the terrain, it looks quite wrong.

The color we actually want, on the soft edge texels, is the original plant color, without the contribution of the background.  How can we achieve this?  By subtracting the background and normalizing the remaining color to preserve intensity.  The algorithm is to guess the unblended color by looking at the current color, consider the difference between it and the background, and move away from the background along this color vector by the weighted amount:

result = background + (color - background) / alpha

When we run this on the palm tree, we get the following, compared with the original:

original

processed to guess true edge colors

This produces a large improvement, but we can do even better.  When mip-mapping is used, it's possible for the background to still bleed through, due to the sub-sampling of the smaller mip-levels.  The solution is to fill in the background areas of the texture with a blend of the closest non-background texels.  This produces images which look like the following, with different conditions for blending outwards from neighboring texels:

1 neighbor

2 neighbors

3 neighbors

It looks funny, but consider that the alpha values still define the palm crisply from within this odd image.  Here is the result in the 3D scene, halos are gone:

This texture processing (treating edges and spreading texels outward) is now an option in VTBuilder.

The impressive commercial package SpeedTree does something very similar:

Original RGB

Original Alpha

RGB processed