The Deceiving Metrics at DxOMark – And What to Look for in (Camera) Reviews

Photography is both an expensive hobby as well as an expensive profession, so it’s only understandable that we do not want to buy under-performing gear for hundreds, if not thousands, of dollars.

You don’t want neither body nor lens to be a considerable “bottleneck” that limits your image quality, so you read reviews before your purchase and compare the pros and cons — before spending thousands of dollars on body and lens.

But what makes a “good” body, what makes a “good” lens? What “metrics” can you put to the test? How do you measure “sharpness” and what even is sharp? Where do you even get your reviews from? What to look out for?

What lens should I buy? What body should I buy? Which review to trust? Who to distrust? Questions upon questions! Here, I’m using the metrics at DxOMark as an example on how a technically correct assessment may not be as effective in helping to make a purchasing decision. In my opinion, there’s a better way.

Buyer’s Reviews are not enough, sadly!

Buyer’s reviews on sites like Amazon are easy to understand; lenses and bodies have stars and the better the stars and the more reviews they have, the better the purchase would be — allegedly. However, Amazon’s reviews are all over the place. People rate something with 3 stars and call it “everything’s fine” or others are simply frustrated about a delayed delivery and rate it one star instead, not even reviewing or remotely discussing the product they have originally bought.

Decorative image, Metrics at DxOMark and a person holding a camera.
Your view on your gear will always be a subjective one. Source: Sergio Rodriguez/Unsplash

Furthermore, you’ll never know how competent or demanding the reviewing person is. Some value weight over image quality, others are regarding ergonomics highly over everything else. All of them might be right but not for you (or me, in that regard) personally. Can you be sure someone mentioning a lack of image stabilization really did activate the lens stabilization? Or did they just assume it was active and didn’t check it further? It wouldn’t come to a surprise to me that this happened more than once, simply because people also worry about their broken computer whilst all they had to do was to connect it to a power source.

All reviews above 4 stars are basically saying that the “package was delivered fine and the product was as I hoped it to be”. This does not mean that you or me would agree with it and rate it the same way. Not to mention that reviews can also be bought and faked quite easily, so we need something else to make up our mind before trying the products ourselves!

– Advertisement –

Standardized Tests — But how?

To rightly assess what gear to buy, standardized tests with charts, backed up with reproducible data, seem to be the right choice to go. Rightly so, one might argue, when a test chart gets pictured in higher apparent resolution with sharper lines, the lens itself is sharper and thus better. But what even is sharp, and is it the only reason to buy or not buy a lens?

Does the standardized test factor in other criteria, like whether an included image stabilization in the lens is existent or even active on a steady tripod? Does the “vibration reduction” introduce vibration even? What about the grip and feel of the lens (“the ergonomics”) — are they accounted for? And if so, can they be standardized?

Additionally, what most people consider as “objective reviewing” is giving a run-down of the technical specifications of a body and lens. Albeit with varying accuracy.

Symbolic image about standardized testing. Source: @nguyendhn/Unsplash

Some say the Sony A7RIII (ILCE-7RM3) offers 14-bit RAW pictures. Sony even states it on their own website that they do. What most “reviewers” out there miss, however, is that the 14-bit RAW files are uncompressed and only a lossy compression is available at 14-bit (rendering it practically useless compared to 12-bit lossless compression, in my opinion). These are, apparently minor, problems that make or break a camera body for me, because uncompressed 14-bit RAW means individual files with over 100 megabyte each.

Testing a product for its quality and usability does not mean solely running down the numbers specified in a datasheet. Instead, it needs clear criteria on how to handle and evaluate certain benefits and flaws of a product so that the viewer of the review can decide on their own what and how they have to value each criteria.

A review is no more than a valuated opinion that aids the reader of that review into making their own decisions.

The Flaws in DxOMark’s Metrics

Metrics at DxOMark for cameras and lenses look professional, clean and reproduceable. Almost with reliable precision, they seem to tell you which lens is trash and which is a must-have. Their lens score goes from 51 (very good) to 6 (very bad), their camera sensor score from 102 (very good) to 25 (very bad). In fact, their score is compounded out of various metrics they gather at “DxO Image Labs”. The metrics at DxOMark are further discussed down below.

Camera Sensors Database at DxOMark

Camera sensor metrics at DxOMark are divided into three categories. That is: color depth, dynamic range and low-light ISO performance they refer to as “Portrait”, “Landscape” and “Sports” respectively. In short, “Portrait” describes the vividness of colors, “Landscape” the ability of the sensor to capture various ranges of light from bright to dark and “Sports” as low-noise images up to a certain ISO value.

These DxO Metrics together make up the total score averaging out the total score of a camera sensor. The way how they assess each camera sensor is further detailed here. Admittedly, these measures do seem solid and sound. However, you’ll notice a certain trend in regards to sensors, that being that a medium-format sensor will innately perform better than a full-frame sensor and the full-frame sensor innately better than the APS-C (crop) sensor will. Does this mean, it’s the “best” choice to go for a full-frame or even “medium-format” sensor simply because the data says so? Well, no.

Image of an RX1 camera sensor built into the camera.
A typical 35mm camera sensor. Brunox983 at Pixabay

Choosing the right type of sensor mostly correlates with what you want to achieve in your photography and with your budget. APS-C sensor cameras will be, inevitably, cheaper than full-frame models.

Yes, a larger sensor will lead to a better noise performance. A smaller sensor will increase your perceived focal length by the given crop factor instead. But, I argue once again that a crop-sensor camera (albeit with more noise) can still make images of the same quality when handled properly. It all depends on the image you want to achieve.

Therefore, sensor comparisons should only be made between camera sensors of the same size. If so, the values indeed could be taken into consideration for choosing a certain camera when the use case for that one is set out clearly. Then again, assuming vast differences between modern camera sensors usually means to make mountains out of molehills. Modern cameras are often advanced enough that the differences between those should not weigh in greatly.

More so, for already established (hobby) photographers with a lens line-up, it’d be best to compare (if a new body is needed) the lenses by sensor size and mount instead of all other cameras to it. Buying a vastly different camera mount and thus all new lenses is simply not worthwhile nowadays because all brands do offer some adequate quality.

Reducing a camera to its sensor is not the only concern!

Just the camera sensor doesn’t make a whole camera body. The body itself is much more than just the sensor; in fact, each brand of camera manufacturer follow their own design philosophies in regards to ergonomics, menu design, color rendering, preference of certain body-specific aspects (faster shooting, better image compression, longer battery life, better viewfinder, …), which has to play an additional role in selecting your future camera that simply is not taken into account when looking at the DxOMark metrics. Otherwise you wouldn’t be happy with the camera if only the sensor were top-of-the-line and nothing else.

– Advertisement –

Lens Database at DxOMark

Whereas the “Camera” category solely deals with the sensor and is prone to underrating APS-C and other cropped sensors, it becomes even more crucial in combination with a crop sensor camera and the tested lenses.

Metrics at DxOMark for lenses have 5 categories that make up the “DXOMARK Score”. These being: “Sharpness”, “Distortion”, “Vignetting”, “Transmission” and “Chromatic Aberration”.

Lens Sharpness Metrics at DxOMark

The lens sharpness metrics at DxOMark for the individual lenses may seem like they accurately represent the acuity of a certain lens and assess whether the produced image will be soft or sharp. However, what may not be obvious at first glance, is that this metric is — in fact — to be seen in relation to the (pre-)selected Camera Megapixel count. If the chosen camera only has 24 megapixel, the sharpness cannot be higher than 24 megapixel in that regard.

Think about it this way. A Carl Zeiss Milvus f/1.4 85mm lens, arguably one of the more qualitatively high lenses in terms of sharpness, scores only a “21” on sharpness when combined with a Nikon D5600 (24,2 MP), whereas it reaches a score of a whopping “45” when combined with a Nikon D850 (45,7 MP).

Seeing these sharpness measurements as a relative instead of an absolute value may be something that’s worth to keep in mind. Arguably, their measuring is acute and precise but their representation and context isn’t so clear as a user of their site might think it is. If you are, for whatever reason, are looking for the sharpest lens available, check it against the camera’s megapixel count first.

Distortion, Vignetting, Transmission and Chromatic Aberration

Arguably, the other metrics are fine as is and definitely a valid concern when evaluating different lenses. All of these are, by no means, “breaking” issues. If your lens is distorted greatly or has a ludicrous amount of vignetting, it’s easy to fix this in post-processing or call it “artistic” and leave the flaws of the lens as is and work with it instead.

Transmission, the ability to get most out of the currently available light, is a different story. To get sufficiently “lit”/”bright” images, you need a longer exposure for this to work. If you’re shooting fast paced action that requires a noise-free, low-ISO value, shot, then it’d be appropriate to search for the highest transmission possible in your desired focal range.

Same goes for Chromatic Aberration. If you’re often working with thin objects, things on the border of your frame, bright-dark contrasts and all the like, you’d also would be looking out for that value.

What’s missing …

When looking for the lens to buy, the one-size-fits-all type of lens, there’s way more to it than their sharpness and chromatic aberration. Review sites like DxOMark or any other individual reviewer often do not focus on the things I, personally, am interested in. They’re often overtly technical and argue “lens is sharp, therefore it’s good” but fail to look at different – and more – aspects of it. Some are even so technical to go through the individual glasses in the lens and their different coatings. Heck, how should I know what this means. Be honest, neither do you.

Instead! Talk about the built-in image stabilization! Talk about the ergonomics of the lens! How does it feel in your hand; does it weigh too much? Is the autofocus of the lens too slow, or even inaccurate? Should you rely on manual focus instead? If the lens has zoom, does it work fast and precise? Is it easy to grip? What are the alternatives in the same price range? What about different brands? What material are they made out of? Are they weather-sealed? More often than not, these concerns are only rarely addressed by standardized metrics.

Personal Anecdote about Useless Reviews

If you are looking for a super-telezoom lens to capture some birds up close or try to get your foot into general wildlife photography, you’ll inevitably stumble over the 150-600mm models that Tamron and Sigma have to offer. Tamron has two, the older G1 and the newer G2. Sigma also has two. The Sigma Contemporary and the Sigma Sports.

The thing — the problem — is that most reviews compare the Tamron G2 with the Sigma Sports and not the equivalent Sigma Contemporary. Those two may look alike but are vastly different in their specifications apart from being equally reasonably sharp. The Contemporary model is made out of plastic and not metal, is therefore lighter but also not as weather-sealed as the Sports model. Sports, on the other hand, is sturdier and made out of metal. But also twice as expensive as the Contemporary model. The Tamron G2 is equal to the Contemporary in most of its specifications and surrounding properties. So, why not compare the G2 to the Contemporary? I am not any smarter after watching that review.

My Recommendation — Check for Complaints!

Instead of relying on 5-star-reviews and standardized metrics, see if and how many people you can find that complain about things instead of praising others. Everybody always has something to complain about — so do I, in fact, right now — and this is important. (But it’s also important to filter complaints. As always, it depends.)

The general idea is that every person complaining about something had different expectations and priorities when going for something new to buy. Generally speaking, most people share the same concerns and want to get the “optimum”, the best value for the lowest price. That value being a mixture of ergonomics, image quality, size, cost, durability and many different things more solely depending on the individual person. So, if something is out of the norm, people will remark that something is too large, too clunky, too small, too loud or too complicated to use.

Now it’s the fun part — filtering out what is really important to you and oneself. If you’re photographing sports, you most likely won’t care about a loud and noisy camera but would definitely care about the noise it produces when shooting weddings or wild and shy animals to not distract or even scare bride and bird away.

A traveler’s priority for a new camera and lens would probably be the weight and size they have to carry. They would rate portability over image quality, maybe — whereas a studio photographer likely doesn’t mind carrying an extra pound if the quality of the final image is better.

It doesn’t seem too complicated but it gets worse. Some people, like me, are very picky about a lot of things, so they’ll complain a lot and even about things other people don’t even notice much. So, apart from knowing exactly what to prioritize and see if it’s an issue for others, you’ll also have to know who to listen to. Not only you’ll have to regard their own profession, goals and priorities, but also their level of “acceptable error” and general tolerance to “imperfection” in anything.

There’s nothing wrong with looking at standardized metrics at DxOMark or any other place but it’s also equally important to know how these metrics come together and what bias the reviewer might have. If this is put into perspective with other people’s complaints or concerns about lens/body performance and adjusted to ones own priorities, one can make a purchase that’s desirable and of value for many years to come.

– Advertisement –