Cnet.com: How We Test HDTV's

 

Today’s TV choices cover a broad spectrum of screen sizes, aspect ratios, features, technologies, and prices. But what ltimately distinguishes one TV from another is how good the image looks.

Test and reference equipment

The most important piece of test equipment is a trained, expert eye. Test patterns and the latest gear are no substitute for a knowledgeable, keen-eyed evaluator with a background in reviewing similar types of TVs. All of our TV testers, namely David Katzmaier, Matthew Moskovciak, and Kevin Miller, have extensive experience reviewing and calibrating displays.

Our primary mechanical test device is a Konica Minolta CS-2000 spectroradiometer, which replaced an older CS-200 in June 2008. The CS-2000 improves upon the CS-200 in its capability to measure very low-luminance sources. The spectraradiometer accurately reads chromaticity and luminance from any type of display, including CRT, plasma, LCD, and DLP, in direct-view as well as front- and rearprojection configurations. Chromaticity, also known as colour temperature, measures the hue of a display at given levels of brightness and is expressed in degrees Kelvin or as X and Y coordinates on the CIE chromaticity diagram. Luminance is a measure of brightness.

 

The reference and test gear in CNET’s TV lab also includes:

Current reference display: As of September 2008, CNET uses the Pioneer Elite Kuro PRO-111FD, on long-term loan from the manufacturer, as the primary reference display for colour and black level performance.

  • Sencore VP-403C: A signal generator that outputs a variety of test patterns at various resolutions and formats, including all HDTV resolutions and 1080p, and can output to composite, S-Video, component video, DVI/HDMI, RF ATSC, and RGB. Our secondary generator, rarely used, is an Accupel HDG-3000.
  • Gefen 2x8 HDMI Distribution Amplifier: An eight-output HDMI distribution amplifier that can send either of two HDMI sources to as many as eight displays simultaneously without any signal degradation. We use this for side-byside comparison testing.
  • Accell UltraAV HDMI 4-8 Switch: An eight-output HDMI distribution amplifier that can send any of four HDMI sources to as many as eight displays simultaneously without any signal degradation. We use this for side-by-side comparison testing as well.
  • Extron DA6 YUV A: A six- output component-video/RGBHV distribution amplifier that can send those sources to as many as six different displays simultaneously without any signal degradation. We also use this for side-by-side comparison testing.
  • Sony PlayStation 3 Blu-ray player (reference)
  • Oppo DV-980H DVD player
  • DirecTV HR20 high-def DVR
  • Velocity Micro CineMagix Grand Theatre Home-theatre PC
  • Monoprice cables

While we use a variety of DVD and Blu-ray sources, including reference-quality theatrical releases, our primary test material includes:

* HD Basics (Blu-ray)

* HQV Benchmark (DVD)

* HQV Benchmark (Blu-ray)

* FPD Benchmark Software for Professional (Blu-ray)

* DisplayMate Multimedia Edition (PC)

 

TV review sample information

Unless noted otherwise, CNET HDTV reviews are based on one reviewer’s hands-on experience with a single particular sample of one model. While our experiences are usually representative of other samples with the same name by the same manufacturer, we can’t always be sure of that since performance can vary somewhat from sample to sample--particularly if newer samples receive updated firmware, or if manufacturers make changes without updating the model name. We typically review models as quickly as possible, so we often receive early versions of firmware that are sometimes corrected later. However, we never review preproduction samples. All of the samples used in CNET HDTV reviews represent, as far as we can tell, shipping models. Sometimes a firmware version will have a direct effect on the performance of a television, and thus on its final review score. When this is the case and we’re made aware of it--usually after a CNET reviewer or a reader finds a performance-related problem--we’ll note the firmware version in the review and post related follow-up information in a note referenced in the review body.

It’s worth noting that CNET obtains nearly all its review samples directly from manufacturers, typically by an editor asking a public relations representative for the desired model. This, unfortunately, can lead to manufacturers sending non-representative samples, or even tampering with the units before they are sent, to help ensure more-positive reviews. If we spot a blatant case of tampering, we’ll note it in the review, but we can’t always prove it (and in case you’re wondering, no, we’ve never spotted a case of tampering that we could prove enough to mention in a review). If a manufacturer cannot ship us a sample or doesn’t want us to review a particular set, we sometimes buy the model in question ourselves. We’d like to move to a buy-only methodology for procuring our review samples, but unfortunately that’s not possible at this time.

 

Test procedure

Below we outline the procedures used in CNET TV tests. We strive to consistently test all TVs we review using the procedure below, but in some cases we do not. In cases where not all of the test procedures are followed, we’ll note the missing items in the review.

Unless otherwise noted, all CNET HDTV reviews take place in a completely darkened environment. While we realize that many people don’t always watch TV in the dark, we use a dark environment ourselves for a number of reasons. Most important, darkness eliminates the variable of light striking the TV’s screen, so the display has to stand on its own in the most-demanding situation. It makes differences in image quality easier to spot, especially perceived black-level performance, which is severely affected by ambient light. Darkness also allows viewers at home to more easily match the experiences written about by the CNET reviewer. Finally, darkness is the environment we find most satisfying for watching high-quality material that really tests a display.

 

Calibration

Before we perform formal evaluations of HDTVs, we first calibrate their picture settings to achieve peak performance in our dark room. While it may seem more realistic to test TVs in the default picture settings, those settings are nearly always designed for maximum brightness, saturation, and impact on the showroom floor. That might sound desirable, but we believe a more natural, realistic picture looks better--in other words, one that most accurately reproduces the incoming signal. Unfortunately, most of the picture presets on HDTVs are not designed for that kind of accurate reproduction, which is one reason why we perform calibrations. Another is to provide a level playing field for comparisons (see below).

Note that, unlike many of the third-party TV calibrations offered today, such as those available from contractors certified by groups like the ISF, the calibrations performed for CNET TV reviews no longer utilize settings in the hidden “service menus” of televisions.

Nearly all HDTVs have these menus, and previously we would access them to better calibrate our review samples. In the last couple of years, however, we have begun posting our ideal dark-room picture settings as part of our reviews, and since users cannot typically assess those service menus (at least, not without voiding the warranty), we decided to no longer use them in our calibrations. We recommend that TV viewers avoid accessing the service menus themselves, because without proper training they can do more harm than good. Happily, many new HDTVs offer ample controls to achieve optimum picture quality without having to resort to service menus.

 

The calibrations we perform follow a few steps:

Adjust maximum light output to 40ftl (footlambert), as measured on the CS-2000 from a Window 100% w/New Pluge pattern (Chapter 24) from the 1080p section of the Advanced Video Test Patterns menu of Digital Video Essentials: HD Basics (Blu-ray). Achieved using a combination of contrast, brightness and, where appropriate, backlight and IRIS controls. This light level was chosen for high contrast in a dark room without being bright enough to cause eyestrain over long viewing periods from appropriate seating distances.

  • Adjust black level to maximum darkness while still showing full shadow detail, as observed by the reviewer via the PLUGE w/Gray Scale (CH1) and Reverse Gray Ramps & Steps (CH3, 4) patterns from DVE: HD Basics. Achieved using a combination of contrast, brightness, and, where appropriate, backlight and IRIS controls.
  • Adjust colour temperature to the setting that comes closest to X=0.3127 Y=0.329 (which corresponds somewhat with 6500K) on the CIE chromaticity diagram, as measured on the CS-2000 from window patterns in 5% increments from 100% to 15% on DVE: HD Basics (CH24-41). If the TV has fine colour temperature controls that allow adjustment beyond presets such as Warm, Cool, and so forth, those are adjusted as well.
  • Adjust colour controls to maximum saturation and accurate hue, without introducing colour imbalance, as observed by the reviewer with the help of colour filters and the SMPTE 75% Colour Bars w/Gray Scale and 75% Bars w/ Gray Reference from DVE: HD Basics. Supplemental colour references include the current colour reference display compared to the TV in question watching demonstration material from DVE.
  • Adjust gamma, if available, to come closest to the reference of 2.2, as measured on the CS-200 from window patterns in 5% increments from 100% to 15% on DVE: HD Basics (CH24-41).
  • Adjust colour management and/or colour space and gamut controls, if available, to bring primary and secondary colours to the closest approximation of the SMPTE HD Spec for DTV Colours (see the Geek Box below). Colour source is a full-raster 100 IRE pattern from the Sencore VP403 signal generator.
  • Adjust other controls where appropriate for source and material, including energy saver controls (which often impact light output), video processing, noise reduction, dynamic contrast, and so on. In general we observe all permutations of these controls and adjust to the settings that look best to our eyes, noting the difference where appropriate. As a rule, we avoid turning on controls that adjust the picture on the fly according to program content, as well as those that automatically adjust to room lighting.

All of our picture settings are published in the Tips section of the review, so owners of that model TV can try CNET’s settings and see if they want to use them. We don’t typically calibrate the TV for more than one source or lighting condition.

 

Side-by-side comparison

Nearly every HDTV CNET reviews is compared to others in the room during the evaluation. This is a direct, side-by-side comparison; the TVs are literally lined up next to one another and compared in real-time, with the reviewer recording observations on a laptop computer. We use numerous sources fed through a switch and a distribution amplifier-- a device that can feed multiple TVs the exact same signal with no degradation. TVs being compared often share similar price points, screen sizes, and other characteristics, but can just as often be more or less expensive or have different characteristics to better illustrate major differences (such as between LCD and plasma, or an extremely expensive set versus a less-expensive model). These comparisons allow CNET’s to make definitive, in-context statements about virtually every area of a TV’s performance, and their accuracy depends on each of the TVs sharing a level playing field. For that reason, we compare only calibrated televisions. We know of no other professional publication that regularly performs side-by-side comparisons as a part of nearly every review.

 

Image-quality test

We perform a broad range of tests on all televisions we review--too numerous to list them all here--but following is a brief description of some of those tests. We use DVD, HDTV, and Blu-ray sources, as applicable, from the test-pattern generators and test discs referenced above. Most comments in a TV review’s performance section are based on observations of a high-quality Blu-ray movie. We choose Blu-ray movies because they represent the highest-quality sources typically available to HDTV viewers today. We use a variety of films, ideally a different one in every review, to better illustrate that performance characteristics are universal and apply regardless of which movie’s being watched (they also make the reviews more fun to read and write). An argument can be made for using the same movie every time, and we do have a few scenes in certain films that we return to over and over, but in general we prefer the variety of different films. CNET does not usually evaluate HDTV audio quality. We believe that anyone who cares would be better served investing in a separate audio system; the least expensive models will nearly always outperform a TV’s built-in audio.

 

Black-level performance:

We comment on the depth of black a TV is capable of producing, a characteristic that leads to more realistic pictures, higher contrast, and more “pop” and colour saturation. This is currently a subjective test, measured by eye against a reference display, although we are taking steps to measure contrast ratio and black level quantitatively sometime in 2008. We also talk about shadow detail, typically reflected as gamma, in this section, also on a subjective basis.

 

Colour accuracy:

We evaluate the combination of colour temperature, colour decoding, and primary and secondary colour accuracy. With the exception of colour decoding, which is difficult to quantify, we include objective results for this area in the Geek box (see below for details). In the review body, we also address the effects of colour accuracy, particularly with skin tones and other delicate areas.

 

Video processing:

This broad range of tests includes objective measurements (1080i and 1080p resolution capabilities, 1080i de-interlacing), the latter of which is a part of the Geek box, and subjective tests with both patterns and real-world material. As of September 2008, we also began testing for motion resolution, which has both subjective and objective elements and so is usually reported as a range, e.g “between 300 and 400 lines.” If a TV has motion processing, such as 120Hz with dejudder, we also address its real-world effects in this section. We’ll also talk about excessive video noise here, if we can trace its fault to the TV, as well as other miscellaneous issues and artefacts not dealt with elsewhere.

 

Uniformity:

With LCDs and rear-projection sets, we use this section to address backlight uniformity across the screen, making subjective observations with full-raster test patterns from the Sencore VP403 and flat-colour scenes, such as shots of skies, from program material.

Plasma screens usually have perfect uniformity, so we don’t typically address it in those reviews. We also talk about off-angle viewing in this section, using similar material and subjective comparisons, and again, plasma is usually exempt. Finally, if we notice any false contouring, we’ll note it here.

 

Bright lighting performance:

Televisions with unusual screen coatings, such as antiglare or antireflective, receive an evaluation in this section. If a TV lacks a specialized screen, we usually don’t address this issue; see our Plasma vs. LCD comparison under “reflectivity of screen” for general guidelines; rear-projection TVs generally reflect less light than either flat-panel technology.

 

Standard-definition tests:

All of the above tests are typically made using high-definition sources, but their results usually apply (with the exception of video processing) to standard-def sources as well. We typically use the HQV Benchmark DVD to address standard-definition performance, hooking up a DVD player via component video at 480i resolution and choosing the Movie (or equivalent) picture preset is available--although if another picture setting markedly improves standard-def performance, we’ll note that in the review. We check the colour bar/resolution patterns, the video-based processing patterns, the detail pattern, the noise reduction patterns, and the 2:3 pull-down pattern. Most of these tests are relatively subjective and use real-world program material, so we discuss their results individually in this section, often in the context of other displays.

Note: Many people have their displays hooked up to HD cable, satellite, or other external sources that perform SD-to-HD processing internally. In these cases, the standarddefinition processing of the display is irrelevant. Since the box is taking the standard-def channel (480i or 480p) and converting it to high-def (1080i or 720p), the TV’s processing makes no difference. Many boxes offer settings that control what kind of processing is done; typically when you select “native” mode or otherwise elect to skip internal processing, the TV’s standard-def processing again comes into play.

 

PC tests:

For HDTVs with analogue VGA inputs, we test their capabilities as large computer monitors. We’ll generally set our reference PC to the native resolution of the display, if possible, and run through select tests in the “Sharpness and Resolution” and “Screen Pixel Resolution” sections of DisplayMate MultiMedia Edition, including “Scaled Fonts,” “Horizontal Line Resolution,” and “Vertical Line Resolution.” We’ll report the results, which are typically pass/fail, as well as similar results using the TV’s HDMI input connected to a DVI output on the PC.

 

About the Geek box

The Geek box is where we put test objective results. The numbers therein adhere to the following guidelines to arrive at Good, Average, or Poor scores.

 

Before colour temp

Example result: 6,057/6,784K

This is the out-of-the-box colour temperature in Kelvin, measured using the best-available user-menu presets, usually the Warm colour temperature and the Movie picture preset. We measure both the high and low light levels from Digital Video Essentials: HD Basics Blu-ray disc (Advanced Video Test Patterns, Chapters 24-41). The exact levels we use depend on the type of TV being reviewed, but they’re generally 20% and 80%. The rating is based on how close the measurements come to 6500K.

  • Good: +/- less than 300K
  • Average: +/- 300K to 600K
  • Poor: +/- more than 600K

 

After colour temp

Example result: 6,511/6,634K

This is the colour temperature after calibration using the user-menu controls beyond the presets. If such controls, which typically affect red, green and blue white balance, are not present, we list the result as N/A. The procedure is the same as the before colourtemp measurement, but our standards are stricter here since a calibrated TV should come much closer to 6500K.

  • Good: +/- less than 100K
  • Average: +/- 100K to 300K
  • Poor: +/- more than 300K

 

Before greyscale variation

Example result: +/- 132K

This is shorthand for greyscale tracking, which measures the consistency of the TV’s greyscale, or colour of grey at different light levels, before calibration. Using DVE: HD Basics (CH24-41), we measure each window between levels 15% and 100% in 5% increments, record the variation from 6500K, and average the results.

  • Good: +/- less than 300K
  • Average: +/- 300K to 600K
  • Poor: +/- more than 600K

 

After greyscale variation

Example result: +/- 592K

Here, we use the same procedure as above, but we take the measurement after usermenu calibration. Once again, our standards are stricter for postcalibration results.

  • Good: +/- less than 100K
  • Average: +/- 100K to 300K
  • Poor: +/- more than 300K

 

Colour of red, green, and blue

Example result: X=0.18, Y=0.068

For this test, we send a full-brightness, full-raster pattern of the appropriate colour to the TV’s highest-quality input from the Sencore VP403 signal generator, then measure the X and Y coordinates with the Konica Minolta CS-200.

We compare the results to the SMPTE HD Spec for DTV Colours to determine how close the actual colours of red, green, and blue come to the spec. The score is based on an average deviation from the spec, which is a result of adding the difference between X and the spec to the difference between Y and the spec.

 

Overscan

Example result: 4.6 percent

Using the Overscan pattern from DVE: HD Basics (CH7), we measure the percentage of picture that’s cropped before calibration on every side. Zero picture loss is ideal.

  • Good: less than 3 percent
  • Average:3 percent to 6 percent
  • Poor: more than 6 percent

 

Defeatable edge enhancement

Example result: Yes

With the Sencore VP403 use the Sharpness pattern to determine whether all traces of edge enhancement can be eliminated.

  • Good: No edge enhancement present after adjustment of user menu controls.
  • Poor: Visible edge enhancement present regardless of adjustment.
  • No average score possible

 

480i 2:3 pull-down detection

Example result: Pass

We use the HQV Benchmark’s DVD-based Film Detail Test to determine whether the TV has the basic ability to recognize 24fps, film-based images and properly convert them to the video frame rate of 30fps. Note that this test covers 480i material only, not 1080i.

  • Good: No visible moire in the grandstands (denoted by “Pass”)
  • Poor: Moire appears in the grandstands (denoted by “Fail”)
  • No average score possible

 

1080i video resolution

Example result: Pass

We use the HQV Benchmark on Blu-ray’s Video Resolution Loss Test to determine whether the display can preserve all of the incoming resolution of a 1080i video (30 fps) signal and convert it to the display’s native resolution.

  • Good: Fine horizontal lines visible in corner boxes (denoted by “Pass”)
  • Poor: Black and white strobing visible in boxes (denoted by “Fail”)
  • No average score possible

 

1080i film resolution

Example result: Fail

We use the HQV Benchmark on Blu-ray’s Film Resolution Loss Test to determine whether the display can properly recognize film-based content recorded at 24fps and properly convert it to the display’s native resolution without losing detail.

  • Good: Fine horizontal lines visible in corner boxes (denoted by “Pass”)
  • Poor: Boxes exhibit strobing and/or vertical bands (denoted by “Fail”)
  • No average score possible

For a complete overview of our TV testing methodology, scroll to the top of this page.

 

The Juice box: TV power consumption tests

In October 2008, to comply with the new Energy Star 3.0 specification, CNET completely revamped its test procedure for the Juice Box, which records power consumption performance for every TV we review. Check out the About the Juice Box section of the Quick Guide to TV power consumption for more information.

Application Note

Download

Cnet.com: How We Test HDTV's

READ MORE