Adobe face recognition: beat the system?

The Kobayashi Maru test is not a test of character unless you see the world in terms of “go down in dignity with the [star]ship” or “be a coward.” Or whatever Nick Meyer thought the outcomes would be. Captain Kirk won the test by not accepting a binary decision tree. This is exactly how you should approach any problem that looks like it is unwinnable. Rewrite the simulation. Use a screwdriver as a chisel.

One of the ways you can do this is to ignore the process as presented completely, decide your goal state, and then selectively use whatever is available to get there. Face recognition is exactly such an exercise. Adobe would have you select one of two suboptimal tools (Lightroom Classic or Lightroom) and have you build out the recognition process and leave it in the platform where it started.

Not believing in the no-win scen-ah-ri-o (sorry, Shatner), I started with the first principle:

What is the purpose of face recognition in photos?

This is actually a really good question. The way the process on Lightroom proceeds (either version), you think the purpose is to name every person in every photo and know what precise face goes with every name. This view assumes that you are a photojournalist who needs to capture stuff. You will go bat crazy trying to achieve this goal if your back catalog is hundreds of thousands of pictures and you use Lightroom Classic (“Classic”) as your primary tool.

Let’s face it – you are (at least this year) a work-at-home salary man, not Gene Capa. The real utility of face recognition is to pull up all pictures of someone you actually care about. You need it for a funeral. For a birthday party. For blackmail.

That does not actually require you to identify precise faces, just to know that one face in the picture is the one you want. You already know this person’s name and how the person looks. And even if you didn’t remember, a collection of pictures of that person – no matter who else was in or out of the shot – would have one subject in common. You would know within a few pictures who John Smith was.

Taking this view, a face identification is just another keyword.

It’s not even 100% clear that you would ever need it done in advance, on spec, or before you had a real need to use it.

What do we know about face recognition in LrC vs LR?

Our statement of problem: 250,000 images of various people, some memorable and some not. I want to get to being able to pull up all pictures of John, Joe, Jane, or Bill. And I want this capability to last longer than my patience with Lightroom cloud. I want to be able to ditch Lightroom, even Classic, one day and change platforms without losing my work.

When you are figuring out a work flow, or trying to, it’s helpful to consider what your tools can and cannot do; hence, with Classic and Cloud, start breaking down the capabilities.

• Both recognize faces with rudimentary training.
• Cloud is much faster than Classic and tends to have fewer false hits (due to Sensei)
• Both can do face recognition within a subset of photos.
• Classic can an apply keywords to images that Cloud can see/
• Cloud cannot create keywords that Classic can see.
• LrC has better keyword capabilities, period.
• You can make an album in Cloud and have it (and its contents) show up as a collection in Classic.
• You can put things in one of these items in either program and have it show up in the other.

Do these suggest anything? No? Let’s step through.

Step-by-Step

Let’s talk about some preliminaries that no one ever seems to address.

Order of operations. If you are starting from zero, you should identify faces in the import every time you import something. Not only are names of near-strangers fresher in your mind, it also prevents the kind of effort we are about to explore.

What’s my name? You must have a naming convention and a normalized list of names. It doesn’t matter whether you pick someone’s nickname, real name, married name, whatever. Whatever you decide for a person must be treated consistently. Is my name Machine Planet? Planet Machine? PlanetMachine? This has implications for Classic, where you can’t simply type a two-word name (Bill Jones) into the text search box without getting everyone named Bill and everyone named Jones. For that you might want to concatenate both names together (unless you want to use keywords in the hierarchical filters). In Cloud, the program can sort by first and last name, so there is value in leaving these separate.

Stay in the moment. Although you might be tempted to run learning against every single picture you have at once, this leads to a congested Faces view (or People view), slow recalculation on Classic and a lot of frustration. Do a day or a week at a time. Or an event. This will give you far fewer faces from which to choose, and fewer faces to identify. Likewise, if there is a large group picture in the set, focus your effort on tagging everyone in it. This will set up any additional Identified People in Classic and will kickstart Cloud.

Who’s your friend? You next need to decide who is worth doing a lot of work to ID. You are not going to do iterative identification (especially on Classic) with people you don’t care about. Leave their faces unidentified. Or better yet, delete the face zones. This is a very small amount of effort in a 200-shot session or a 36-shot roll of scanned pictures.

Start in Cloud. This part is not intuitive at all. Go ahead and sync (do not migrate!) all your pictures to Lightroom mobile. This consumes no storage space on the Adobe plan. If there are a lot that have no humans, use a program like Excire Search to detect pictures with at least one face pointed at the camera. This is a reasonable cut, since there are few pictures you would bother tagging that have one face, solely in profile.

The synch process will take forever. I don’t think there is a lot of point in preserving the Classic folder structure when you do this; I would just make a collection like “Color 2000-2010” in the Classic synched collections and dump your targets into that (n.b. a collection in Classic is just an alias to your pictures; making a collection does not change the folder arrangement on your computer). We are only using Cloud for face recognition; its foldering is too rudimentary and inflexible to be useful – although right-clicking in Classic to make folders (or groups of folders) into synched folders will let you adopt the Classic organization in Cloud, albeit flattened, without re-synching. Again, not very useful. Also, for reasons described further on, you want to have a relatively clean folder panel in Cloud because you will be making some albums, and you don’t need extra clutter.

Ok. Let the synch run its course, or start your identification work on Cloud as it goes. Cloud will start aggregating what it thinks is the same face into face groups, which you then must name. Start naming these according to the convention you chose. I would put the People view to sort by “count,” which naturally puts the most important people at the top (you have the most pictures of them). Let’s say you name one face group “John Smith.”

Crossing over

The process so far is pretty generic. To start crossing things over to Classic, you need to make folders (“albums”) in Cloud. Start with one per important person (“___ John Smith”). Search for that person. Dump the search results into the album. You can always add more later.

Now flip back to Classic. You will see collections under “From Lightroom.” Voilà! One of them is “John Smith.”

Now you can do one of two things.

You can simply make a quick check to make sure there are no pictures included that obviously are not John Smith. But after you do that, or not, you can mass-keyword everything in that collection “John Smith.” If you named John Smith consistently with any pre-existing Classic face identification of John Smith (i.e., not two different variations of the name), your searches will now have the benefit of both tools. Save those keywords down to the JPG/TIFF files (Control-S/Command-S) or XML files (same), and you will forever have them, regardless of whether you leave the Adobe infrastructure. In fact, many computer-level file indexes can find JPGs and TIFFs by embedded keywords (which the index sees as text).

Congratulations. Now you’ve highjacked Sensei into doing the dirty work on Classic.

With a small but not overwhelming amount of creativity, you could use a technique like this to cross-check your past Classic calls.

STOP HERE AND GO TO “CALIBRATING YOUR EFFORTS” UNLESS YOU ARE A MASOCHIST

Second, if you’ve missed your OCD meds, you can also use the results of this to inform your Classic face-recognition process.

a. Select this “From Lightroom–>John Smith” collection and flip to Faces view in Classic.

You are now seeing all “Named People” and all “Unnamed People.” Unnamed people are shown by who Photoshop thinks they are most likely to be. You can sort Unnamed people in various ways, but however you do it, you want to get John Smith?s in a contiguous section where you can then confirm or X out. By going into this in the From Lightroom–>John Smith collection, you are not waiting for recalculations against every photo you have – just the ones that Sensei thought should have John Smith.

So the cool trick is this: if you see 106 pictures in From Lightroom–>John Smith, then you know you are probably going to be done when you have 106 confirmed pictures of him. Or done enough. John can only appear in a picture once. There will be a margin of error due to how closely Classic can approximate Sensei, but you can get to about 90% of the Sensei results without a lot of trouble. This is a bit better than Classic on its own, where more pictures of John Smith at an earlier age might be really buried down in the near matches. Further, Classic is something of a black hole for similar pictures because unlike Cloud with Sensei, there is no minimum required similarity score to be a suspected match.

b. You can, of course, drill down on John Smith as a Named Person. You don’t have much control over how “Similar” pictures are ordered (I believe it is degree of match for the face), but here, you can confirm a much more concentrated set (after you decide how to deal with the “fliers” who are not John Smith).

One other technique I have developed while in the “confirming” stage is that it may be easier to confirm en masse (even if some are wrong) down to the point where the “not John Smiths” are about a third of the results in a row of Similar faces. A small number of “fliers” can be removed by going up to the Confirmed pictures, selecting them, and hitting delete. Trying to select huge swaths of unconfirmed faces in Similar and then unselecting scattered fliers tends to really slow things down. As in Classic really slows down as it tries to read metadata from everything you selected.

Incidentally including and then manually removing a few fliers from Confirmed does not seem to affect accuracy (because every recomputation of similarity is on the then-current set of Confirmed faces – changing that set changes the computation). If you have 99 pictures that are right and one that is wrong, it won’t even change the accuracy appreciably. If in Confirmed, you have 995 pictures that are John Smith and 5 that are not, again, the bigger set of correct ones will predominate future calculations.

Next, at some point, especially with siblings, Classic is going to reach a point where Jane Smith (John’s Sister) is going to show up as a lot of the “Similars” with John Smith. When this happens, go back to Faces (top level, always within From Lightroom–>John Smith), click on her, and confirm a bunch of her pictures. When you go back to Named Person John Smith, a lot of the noise will be gone, and hopefully more John Smiths will be visible in a concentrated set you can bulk-confirm.

Crossing back (optional)

I did write “iterate,” right? You might want to keep your Cloud face IDs as complete as possible, since there is not 100% correspondence between results from the methods used by the two platforms. This is relevant if you have already trained Classic on John Smith.

1. In Classic, note the count in your From Lightroom–>John Smith collection. Say it’s 106 pictures.
2. Do a search from your Classic Library for all pictures of John Smith. If you used a space in the name, add Keywords to the field chooser menus (via preferences) and select that line.
3. Drag all of those results to From Lightroom–>John Smith.
4. Flip to Cloud. They are now in that “John Smith” album. Or they will be when it synchs.
5. Select all the pictures in the “John Smith” album.
6. Hit Control-K (or Command-K) to bring up keywords and detected/recognized faces in the “John Smith” album.
7. Now name any faces that are blanks – but should be John Smith.
8. Now from the All Pictures view, search for John Smith and drag all his pictures to the John Smith album.
9. In Classic, check your count. If it’s say 128 pictures, now, that means that Cloud took your examples and found more John Smiths. And now they are ID’ed in Classic as well.
10. Switch to Faces and confirm the 22 additional faces as John Smith. Now both systems have identical results.

Calibrating your efforts

For searches for random people, Cloud is still the best because it requires very little training. That said, for randos, you are using a tool that does not give you any permanent results. That’s probably ok for people who you don’t really care about. Or if you plan to be on Cloud forever.

For close friends and family, you may just run the “Crossing Over” exercise. I would do it in groups: do a bunch of albums on Cloud (say seven people), then do a bunch of naming on Classic (their collections), etc.

If you are really a neat-freak or compulsive, you could use the “Crossing back” step. But Sensei is reasonably good at what it does, so the marginal effect of adding Classic results to Sensei may not be much. If you have Excire, you might use it to find pictures that look like a picture of John Smith, which will give you a third means of concurrence.

The thing to remember about face recognition is that it is miraculous but also imperfect. It has to detect a face and then it has to identify a face. It doesn’t see how you see. Efficiency works at cross-purposes to accuracy.

But it is still vastly better than trying all of this on your own.