ARKit face tracking without reading a single pixel

I built an iPhone app that reminds you to smile a few times a day. One optional screen shows, in real time, whether you are actually smiling. It uses the TrueDepth camera and never reads a camera frame.

That sounds like a contradiction, so this post is about why it isn't, and what the decision cost me.

The problem with pointing a camera at someone's face

The feature is simple: you press start, the app tells you whether your mouth corners are actually going up, and after five seconds it shows a small summary. Useful, because most people are surprisingly bad at knowing what their own face is doing.

But the moment an app asks for the front camera to look at your face, three separate problems arrive at once.

Users don't believe you. "It's processed on-device" is a sentence anyone can type. There is no way for a user to verify it from the outside, and there have been enough incidents that the default assumption is suspicion.

App review looks harder. Data from the TrueDepth API carries extra restrictions, and an app in a health-adjacent category that also touches faces is going to get read carefully.

And you inherit a liability. Every frame you hold is something that can leak, be subpoenaed, or be quietly repurposed by a future version of yourself who needs a metric.

The interesting part is that none of the three is required by the feature. They come from the implementation, not the requirement.

What ARKit actually hands you

When you run an ARFaceTrackingConfiguration, the data arrives in roughly three tiers, and they are wildly different in sensitivity.

The third tier is the one everyone skips past, and it is the only one my feature needs.

"Are you smiling right now?" is answered by two numbers: mouthSmileLeft and mouthSmileRight. That's it. Those two floats cannot be turned back into a face. They cannot identify you. They cannot tell you what room you are in or who else is in it.

What the app reads, and what it never touches

The whole surface is three things.

• Two coefficients for how far each mouth corner is raised
• The angle between the face and the camera — only so the UI can say "turn towards the camera"
• An ambient light estimate — only so the UI can say "it's too dark in here"

And the list of what it deliberately does not do is longer, which is the point:

• It does not read ARFrame.capturedImage. Not once, not for a preview, not for debugging
• It does not create photos, video, depth maps, or a face mesh
• It does not do face recognition or identity verification, and builds no biometric template — the app cannot tell who you are
• It does not attempt emotion analysis, appearance scoring, or any health inference
• It does not persist the values anywhere: not to files, the photo library, a database, user defaults, backups, or logs
• It sends nothing to a server, because there is no server

When the screen closes, the values are gone from memory. Reopening it starts from nothing — there is no previous session to compare against, by construction.

The camera preview is off by default. If you switch it on, it is drawn for that session and never captured. Most people leave it off, which surprised me; a bar that moves is apparently more legible than your own face.

The shape of it

Illustrative rather than literal, but the structure is this — you take the anchor, read two keys, and never reach for the frame:

There is no session(_:didUpdate frame:) in the app. That is not an oversight — it is the whole design. If the frame never enters your code, you cannot accidentally log it, cache it, or hand it to a crash reporter.

What this buys you

A privacy policy made of checkable sentences. This is the part I underestimated. Most privacy policies are written in a register of reassurance: "we take your privacy seriously", "data is processed securely". None of that is falsifiable, so none of it earns trust.

When your implementation is this narrow, you can write sentences that are just facts. "The app reads two mouth-corner coefficients, a face-to-camera angle, and an ambient light estimate." "It does not read the pixels of the camera frame." Someone can disagree with those, which is exactly what makes them worth reading.

A smaller review surface. The features you didn't build are features nobody has to evaluate.

Permission that is genuinely optional. Because the live check is one card on the home screen rather than the core loop, camera access is requested only when you open it. Decline and every other part of the app behaves identically. An app that degrades gracefully when you say no is a different product from one that nags.

What it costs

I'd rather not pretend this is free.

I cannot debug from user data. When someone writes to say the detection felt wrong, I have nothing. No recording, no coefficient trace, not even a count of how often the feature is opened. I ask questions and guess.

Nothing improves on its own. There is no corpus accumulating, no threshold being tuned against real usage. Whatever heuristic I ship is the heuristic, until I sit down and change it deliberately.

The feature is device-limited. Face tracking needs the TrueDepth camera, so on unsupported hardware this one screen is unavailable. That was tolerable only because the feature is optional; if it had been the core loop, this would have been a much worse trade.

And I gave up the metric that would tell me if any of this mattered. I don't know how many people use the live check. I have opinions, and no data.

Those costs are real, and for a team with investors and a roadmap they might be disqualifying. For one person shipping a small app, they were cheaper than the alternative.

The general version

The framing that actually did the work here wasn't privacy-first or local-first. It was narrower:

What is the smallest piece of this signal that answers my question?

A camera frame answers "what does this person look like right now". I never needed that. I needed "are the corners of the mouth up", which is two floats. Everything else was surplus I would have had to protect, justify, and eventually explain.

The cheapest data to secure is data you never took.

---

The app is SmileDay, on the App Store. Fair warning: the interface is Korean-only right now, so it is more useful as a reference than as something to install. The privacy policy is Korean too, but the section on face data is the part described above, written out in full.