Face-off: Apple vs. Adobe face recognition

So here is a question: what’s the best way to catalogue and tag your pictures? Is it Lightroom Classic? Lightroom Cloud? Is it Apple Photos? Is it something else? Maybe it’s a lot of things. If you are a high-volume imaging-type person, you’ve probably wondered how to deal with things like tagging people. The most macabre application, of course, is the funeral collage. But say you have tens of thousands of pictures of family members and want to print a chronological photo album. Then what? Face recognition features in software may be your best bet. From a time standpoint, they may be your only choice. The problem is that different software has different competencies.

Apple Photos

Something like Photos is designed to group pictures, more or less automatically, around people, events, dates, or geography. Think of it as your iPhone application on steroids. Photos is not big on user control. It is not even engineered to do anything with folders except display them if that’s how photos were imported.

Face recognition in Photos is incremental and behind the scenes: it only finds faces when you are not actively using the program, and over time, it batches up groups of pictures which you confirm or deny as a named person in your Faces collection. To establish your Faces collection, you have to put names on faces in a frame where faces have been detected. This tends to mean that face recognition proceeds by which faces the user thinks are most important. As it should be.

Unlike Lightroom, Photos does not presume that detected faces are unique. It applies a threshold such that if it detects Faces A, B, C, and D, and they are close enough, they are treated as the same (unnamed) person. As such, naming one person can have the unintended effect of tagging a bunch of false matches. Either way, you can error correct by right-clicking the ones you see that are wrong.

My assessment of Photos is that it is not suitable as a face-recognition tool if you have hundreds of thousands of images, for several reasons:

  • Its catalogs are gigantic, even if you use “referenced” images. Photos loves it some big previews, no matter what you do. For scale, my referenced Photos library is 250gb where my entire Lightroom Classic library folder is 40gb (both excluding original image files – so Photos sucks up 6x the space).
  • The face recognition process appears to be mostly (if not completely local), it runs in spare processor cycles, and in my experience, can cause kernel panic. Hand-in-hand with this is the fact that you can never actually turn Photos off. It’s part of MacOS.
  • There does not appear to be any indication that Photos actually writes metadata to files. So when you move to a new application, you’re starting from zero.
  • You can’t really use it in conjunction with a grown-up asset management system like Lightroom.

Photos is, however, good for generating hilariously off-base collections of photos (memories) with weird auto-generated titles (“Celebrate good times” with a crying baby as the cover photo). Or collections based on the date a bunch of pictures taken over decades were scanned (such as my 42,600 pictures apparently taken on December 12, 2008). I actually have no idea where these are generated. But they are funny.

I’m sure Photos is really good for those funeral collages, though.

Lightroom Classic (LrC)

Something like Lightroom Classic (LrC) is designed around manipulating, filtering, and outputting large numbers of pictures at once. This is, indeed, the killer app for handling large volumes of photos, and becomes a single interface for everything. It’s OK, but not great, for face recognition.

To put it mildly, LrC’s face-recognition is processor- and disk-intensive. The best way to use it is to use it on a few hundred photos at a time so that your identifications don’t swamp everything in your collection in a recalculation. LrC is good at showing you different faces all at once, as single images, so you can get cracking on identifying as many new “people” as you have patience for in one sitting.

The top level of the Faces module shows you (i) “Named People” and (ii) “Unnamed People.” You need to name at least one “Unnamed” person to start. After a while, the system will try to start putting names on “Unnamed” people. If you have a Named person named “John Doe” and are presented with an image that is “John Doe?” you can click the check box to confirm it and the X box to remove the suggestion (clicking again removes the detected face zone, such as if the system mistook a 1970s stereo for someone’s face).

Once you have done that, you can drill down on a “Named” person to see what pictures are “Confirmed” and what pictures are “Similar.” Again, to move from Similar to Confirmed requires an affirmative call. Here, you only get a check box. There is no “Not John Doe” option, which means that every possible match is shown, ranked in what LrC thinks is similarity. This is actually problematic because as you confirm more pictures, the number of “Similar” pictures rises exponentially. This puts a huge computational drag on things.

Wherever it happens, confirmation of a face’s identity is an affirmative process that is repeated for each picture (you can select several). This prevents false IDs based on grouping disparate real people into one “face,” but it also makes tagging excruciatingly repetitive. And slow. Highlighting faces to group-confirm or identify can have the “highlight” lagging far after your click. And God help you if you click six pictures and then try to type a name into one to rename all six. It works about half the time. The other half, it auto-completes with a totally unintended name. If you accidentally confirm the wrong face for a given name, you can highlight the errant thumbnail and hit Delete (this is not well documented).

Critically, the top level of the Faces module (where you see all named people as thumbnails) is the only place where the system puts a “most likely name” on unnamed people. Otherwise, looking at any particular “Named Person,” the same person – Bob – might show up as a similar for John Doe. And when you switch to Richard Roe, Bob will show up as a “similar” for him as well. This is part of the reason why people for whom you have 10 actual pictures always show up with 20,000 “similars.”

A big advantage of LrC over other solutions is that you can see and tag faces within specific folders, collections, or filmstrips. This lets you make context-sensitive decisions about who is who. For example, I am pretty sure that my kids did not exist in the 1970s. Or I might know that only 6 people are represented on a single roll of film that constitutes a folder in my library.

When a name is confirmed on a picture, that name is written as a keyword to the metadata in the library. It appears that XMP files (if you chose that option for RAW files) are written with the actual coordinates of faces in the picture, which allows some recovery if you have to rebuild a library from scratch. The important thing is that a picture is keyworded with the right names. Face zones are nice but not quite as critical in the long run because in reality, you only really care whether a picture contains John Doe or Richard Roe, not which one is which in a picture of both.

Always save your metadata to files if working with TIFFs/JPEGs/scans (Command+S) or “always write XMP” with RAW camera files. This helps keep your options open if you want to get divorced from Adobe. Or if your Lightroom library goes wheels-up and you have to rebuild from zero. There is no explanation for why this program just doesn’t write an XMP for every file. It would make things easier.

Lightroom [CC or “cloud”]

What a hot mess. The only thing that really works about Lr CC is face recognition. The rest of it is a flashy, underpowered toy that despite being “cloud” based can still consume massive amounts of hard drive space and processing power. If your photos are in the Adobe cloud, or synched from LrC, the program works with smart previews.

Adobe’s Sensei technology is a frighteningly good face-recognition system. In the People view (mutually exclusive with the Folders view), it takes all of your photos and groups them according to what it thinks is the same face (like Apple Photos). Put a name on that face, and it might ask you if this other stack over here is the same face. It is extremely fast (because it runs in the cloud). Sensei can also identify objects, and to some degree, places in photos. Naturally, the most important people in your life have the highest counts, and you can sort unnamed faces by count and work your way down. Things break down when 400 people have 15 pictures apiece, though…

The system, though, has some amazing limitations that are pretty clearly engineered in by a company that is trying to move everyone to its walled garden. Two of these four bear directly on the issue of why a hard drive – and keeping your own metadata local – is your ladder out of that walled garden.

First, metadata transfers to Lr are one-way. The program can absorb keywords applied in LrC, but not recognized faces/zones, and nothing you input in Lr can ever rain down on LrC. There is no programming-related reason that prevents metadata from flowing the other way, aside from intentionally engineering this out of being possible β€” so that you are eventually forced to store all your stuff in Adobe’s per-month-subscription storage space. Because paying a monthly to use programs that aren’t really being updated – like LrC – was not bad enough.

Second, you cannot force face recognition on arbitrary subsets of your library, at least very efficiently or intuitively. If you came at this program assuming that it would be like LrC, you would conclude that there is no way to do this. Instead, you have to select a group of pictures and hit Command/Control-K (for “keyword” – how intuitive…) to see the faces present in the picture or group. Lr then shows you the single picture with the face boxes – and the collection of faces in the picture on the right panel. This is great – but why is it so hard to find? You also get the impression that when you do this, the face boxes are generated on the fly. But the critical defect here is that the “named faces” that are shown as thumbnails are even smaller than the other face thumbnails in Lr.

Third, when asked to “consolidate” two faces, there is no way to flip between the two collections. This is an oversight – you are not asked to name a person based on one photo, but for some reason you are asked to make a consolidation decision that could have catastrophic consequences β€” based on a single fuzzy thumbnail. If in doubt, sit it out.

Finally, you can’t push face recognition data back down to LrC. So if you use LrC, you basically end up with completely separate face-recognition data sets based on the same photos. This is a big-time fail.

Upshot

Well, in terms of applications you can access for a Mac right now, the options are ok – but not great. Stay tuned for Part 2, in which we look at a way to leverage LrC and LR CC against each other to speed things up.

Tags: , , ,

One response to “Face-off: Apple vs. Adobe face recognition”

  1. Jack says :

    This was very helpful, thanks. And please hurry with part II as I’m smack dab in the middle of trying to figure all this out! πŸ™‚

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: