Television broadcasts of events can be enhanced by providing graphics that are blended with other images or video to convey information to viewers. Such information can be related to the event, people involved in the event, or any other subject that the broadcaster desires. For example, during the broadcast of a football game a graphic can be employed to indicate the field location to which an offensive team must advance in order to obtain a first down. Logos and other graphics can also be blended with a video of an event. In some instances, such as adding a first down line or an advertisement to an image of a playing field, it is desired to blend the graphic with the video such that the graphic appears to be part of the live event, and not superimposed on the video. For example, it may be desired to add a line or logo to a video presentation of a sporting event such that the line or logo appears to be painted on the field.
When blending graphics with a video, broadcasters may want to vary the degree to which the graphic is blended. For example, when blending graphicsto appear to be part of the event, it may be desired to vary the amount of blending to account for occlusions. That is, if a person or object is positioned in front of the location where the graphic is to be added, the graphic would look more realistic if the graphic is not drawn over the person or object. A broadcaster may also want to add a graphic to a video representation of a scene such that the graphic shows some of the texture of the underlying scene. A broadcaster may also wish to vary the blending of an image to show hard edges or soft edges. In order to accomplish these results, a blendingcoefficient can be generated for each pixel in the graphic to indicate the degree of blending between the graphic and the program video. One example of such a blending coefficient is an alpha signal that is used with a keyer for blending a graphic.
Most attempts to generate blending coefficients have utilized chroma-key techniques. Chroma-keying usually entails choosing a color in a video signal to be replaced with a graphic. In most instances, use of the color in the video is controlled and the blending is limited to either replacing or not replacing, without degrees of blending. On the other hand, a broadcast of a live event such as a sporting event may include many different colors and lighting conditions, different hue and brightness conditions, and different color balances that are not controlled by the broadcaster. During the broadcast of a football game, the broadcaster has no control over the color and lighting of either the field or players' uniforms. In addition, changing weather patterns can also affect the appearance of certain colors. For example, a broadcaster may wish to blend a graphic to a video of a sporting event such that thegraphic appears to be painted on the field. In such a case, a system should be able to make a distinction between a "green" of a grass field, whereblending is to occur, and a "green" of a player's uniform, where no blending is to occur. The standard chroma-key techniques cannot perform all of the above functionality in real time, and cannot make all the necessary fine distinctions between the colors appearing at a live event.
Accordingly, a system is needed for generating a blending coefficient that provides a broadcaster with flexibility in blending a graphic to the video presentation of a live event. A graphic and video are blended by controlling the relative transparency of corresponding pixels in the graphic and the video through the use of blending coefficients. One example of a blendingcoefficient is an alpha signal used in conjunction with a keyer. The value of a blending coefficient for a pixel in the graphic is based on the luminance and chrominance characteristics of a neighborhood of pixels in the video. Inclusions and exclusions are set up which define how the neighborhood of pixels is used to create or change a particular blending characteristic.
The graphic can be blended with a video representation of a live event such that the graphic appears to be part of the scene captured in the video, rather than superimposed on top of the video. For example, if the graphic is added to a broadcast of a sporting event, the graphic will appear to be part of the field.
The present invention makes use of inclusions and exclusions. An inclusion is a set of characteristics that describe a class of pixels that can be modified in the video. An exclusion is a set of characteristics that describe a set of pixels that are not to be modified in the video. When blending thegraphic to a video, the graphic may be blended on a pixel by pixel basis taking into account whether a given pixel in the video is an inclusion or an exclusion. In an alternate embodiment, the video and graphic are blended at a polygon level.
In one embodiment, the creation of an inclusion or an exclusion comprises an identification of luminance and/or chrominance characteristics for pixels. The process of creating inclusions and exclusions includes the steps of choosing an area of a video frame which identifies a set of pixels. An indication is made as to whether the set of pixels will be used to create an inclusion or an exclusion. If the pixels are to be used to generate an inclusion, then a display filter is created which describes the characteristics of the pixels in the set. If the pixels are to be used to create an exclusion, then the system generates a protection filter which describes the characteristics of the pixels in the set.
In one embodiment, a graphic is divided into a number of polygons. One method for dividing the graphic into a number of polygons includes creating border lines and a number of edge lines. In one alternative, all the border lines and edge lines are parallel to each other. Each polygon has its vertices on edge lines and/or border lines. Each of the vertices on the border lines are given a nominal blending coefficient. The vertices on the border lines can also receive nominal blending coefficients or a constant blending coefficient. By adding additional edge lines and changing blending coefficients for the edge lines, the amount of blending of a graphic can be varied throughout the graphic.
In one embodiment, each of the nominal blending coefficients for the vertices on the edge lines are multiplied by edge fractions. An edge fraction for a particular pixel of a graphic is created by a method which includes selecting a set of pixels in the video. The set of pixels includes a first pixel in the video having a position in the video frame corresponding to the position of the pixel in the graphic that is under consideration. The other members of the set may include those neighboring pixels surrounding the first pixel. The set of pixels are examined one pixel at a time to determine what fraction of the pixels are an inclusion and not an exclusion. Other embodiments may use other methods for creating an edge fraction.
Once the blending coefficients are set for each of the vertices of the polygons, blending coefficients can be determined for each pixel in each polygon. Using the blending coefficients, the graphic can be blended with the video.