Build a Body-Tracking App for iPhone and Android From One Sentence

Short answer: describe the app in a sentence, such as "a push-up counter that draws my skeleton and tells me when I'm not going low enough", and choose iPhone + Android when you start the project. Orchard builds a real React Native app with live body tracking from MediaPipe Pose, and you open it on your own phone in seconds by scanning a QR code in the free Expo Go app.

Body tracking used to be one of the harder things to put in an app. You needed a pose model, a camera pipeline, the maths to turn points into joint angles, and a way to count reps without double counting every wobble. Then you needed a phone to test it on, because a laptop browser cannot show you whether the counter keeps up with a real push-up. This guide covers what body tracking can do in an app you build from a description, how it runs on your phone, and where its limits are.

What the app actually sees

Body tracking in the iPhone + Android option uses MediaPipe Pose, which finds 33 points on the body in the live camera feed: the nose, eyes and ears, shoulders, elbows, wrists, hands, hips, knees, ankles, heels and toes. It reports them about 20 times a second, from either the front or the back camera.

Each point comes in two forms. One is its position on the screen, which is what the app uses to draw a skeleton over you as you move. The other is its position in 3D space, measured in metres around the hips, which is what the app uses to work out true joint angles: how bent your elbow is, how deep your squat goes, whether your hips are level. Every point also carries a visibility score, so the app knows when your wrist has left the frame instead of guessing where it went.

What you can build with it

Because the app knows where your joints are and how they bend, a lot of fitness and movement ideas become one sentence away:

• Rep counters: push-ups, squats, curls, lunges. A rep counts when a joint angle goes below one threshold and then back above another, so a shaky hold at the bottom never counts twice.
• Form coaching: the app tracks the deepest point of each rep, so it can tell you to go lower, keep your back straight, or slow down, and show it on screen as you move.
• Yoga and posture: compare your joint angles with a target pose and tell you which limb to adjust, or nudge you when your shoulders creep forward.
• Dance and movement games: score how closely someone matches a move, or turn a jump or a wave into a game control.
• Physio and rehab exercises: count prescribed movements and record range of motion over time.

A good example is RepCount, a push-up counter with live body tracking and form coaching. It draws your skeleton over the camera, counts each push-up as you do it, and coaches your form. It was built from a description, not a template.

The agent designs it, not a template

Orchard's agent writes the whole app from what you describe and designs it from scratch. There are no template screens or preset themes, so a yoga app and a boxing app do not end up looking like the same app with different colours. The tracking itself is a ready part the agent can use; what to measure, how to count, and how the screen looks are decided for your app.

You can be specific. "Count squats, only when my thighs go past parallel, and flash the screen green on a good rep with a short vibration" is a perfectly good prompt. Haptics, animation and sound all work on the phone too.

How it runs on your phone

This is where the iPhone + Android option is different. Instead of watching your app in a browser and imagining the camera, you test it on your actual phone:

1. Install Expo Go, a free app on the App Store and Google Play.
2. Scan the QR code on the Phone app tab in Orchard with your phone's camera, or tap "Open in Expo Go" if you are already on your phone.
3. The app opens in seconds with the real camera, so the tracking runs on you, live.
4. Ask for changes in plain English. The QR code stays the same; reload the app in Expo Go (shake the phone or reopen it) to get the latest version.

The tracking runs on the phone itself. Orchard also shows the app inside a phone frame in your browser, but camera features like body tracking show a short "open it on your phone" note there, because a browser preview cannot film you doing a squat.

The honest limits

Body tracking is impressive, but it is a camera looking at a person, and that comes with conditions:

• The body needs to be in view. For a squat counter that means your whole body, or at least the part that matters. Prop the phone up and step back. If a joint leaves the frame, the app knows it has lost you rather than counting nonsense.
• Light matters. Decent lighting gives steadier points. A dark room makes tracking jumpy.
• It needs an internet connection. The tracking model loads over the internet when the camera starts, and the preview itself needs a connection.
• It is body pose, not face or depth. AR, LiDAR and face tracking are not available in the iPhone + Android option. If your idea depends on those, start a project with the iPhone app option instead, which builds native Swift and supports them.

Shipping it

On a paid plan, the Deploy tab sends your app to TestFlight and then to the App Store, using your own Apple developer account. For Android, it builds an APK you can install on any Android phone for testing, plus the file Google Play needs, and it can upload builds to Google Play's internal testing track once you connect your Play developer account. Apple and Google decide their own review times, so nobody can promise you a date.

FAQ

Can I build a body-tracking app without coding? Yes. Describe what the app should watch and what it should count or check, choose iPhone + Android, and Orchard writes the app, including the tracking, the rep logic and the screens.

Does it work on both iPhone and Android? Yes. The iPhone + Android option builds one React Native app that runs on both, and you can preview it on either kind of phone through Expo Go.

How accurate is the body tracking? It uses MediaPipe Pose with 33 body points about 20 times a second. It works best with the relevant part of your body fully in view and decent light.

Can I test it without a phone? You can see the app's screens in the browser preview, but the tracking itself needs a real camera, so it runs on your phone. Scanning the QR code takes seconds.

Can it do AR or face tracking too? Not in the iPhone + Android option. Those need the native iPhone app option, which builds Swift and supports AR, LiDAR and face tracking.

Got a rep counter, a form coach or a yoga trainer in mind? Describe it at Orchard, choose iPhone + Android, and open it on your own phone.