Automated Photo Tagging System

Business Objective

The objective is to maximize meaningful tagging that drives engagement while respecting privacy and maintaining user trust through accurate, non-creepy suggestions. Photo tagging serves multiple business purposes: increases engagement (tagged users get notified, view photos, interact), builds social graph (reveals relationships and social contexts), drives content distribution (tagged photos reach broader audiences), and creates training data for face recognition models.

The privacy dimension is critical and differentiates this from pure face recognition. We're not just identifying faces; we're suggesting tags to the uploader based on their social context. A person might appear in a photo but shouldn't be suggested if they're not friends with the uploader or if they've opted out of tagging suggestions. The system must balance utility with privacy expectations and regulatory requirements like GDPR.

There's a significant cost to false positives. Suggesting the wrong person is embarrassing and damages trust in the feature. Users might disable it entirely after a few bad suggestions. We need very high precision, arguably more important than recall. It's okay to miss some taggable people; it's not okay to suggest wrong people.

The "creepy factor" is real. If we suggest tagging someone the uploader barely knows or hasn't interacted with recently, it feels like surveillance even if technically accurate. We need to incorporate social context and recency, not just face matching.

ML Objective

From an ML perspective, this is a ranking problem conditioned on face recognition. Given a photo with detected faces and the uploader's social context, we need to rank potential people to suggest for each face. The pipeline is: detect faces in image, generate face embeddings for each detected face, retrieve candidate people (from uploader's friends/connections), rank candidates by probability they're the person in this face, and present top-K suggestions per face with confidence scores.

The core challenge is combining multiple signals: visual similarity (does the face match this person?), social context (are they friends? close friends? recently interacted?), photo context (location, event, other tagged people), historical patterns (who does uploader typically tag?), and privacy preferences (has person opted out? restricted tagging?).

We're not doing open-world face recognition (identifying anyone in the world). We're doing closed-set ranking within the uploader's social network, which is more tractable but still challenging with thousands of potential candidates per user.