Naruto Hand Sign Battle

Naruto Hand Sign Battle is a two-player browser game where players speak a jutsu, perform webcam-detected hand signs, and resolve attacks in real time. The project combines LLM-driven jutsu generation, computer vision sign validation, live game state sync, and WebRTC video into a playable anime-style battle loop.

The core idea is simple:

1. Two players join the same match in the browser.
2. A player speaks a jutsu name or asks for chakra recovery.
3. Gemini turns that voice input into a structured action.
4. If it is a jutsu, the game generates a hand-sign sequence and spell animation.
5. The player performs the signs on camera.
6. Roboflow validates the signs.
7. The backend resolves damage, chakra cost, and the next state of the match.

This gives the game three layers at once:

• Fantasy layer: Naruto-inspired jutsu, chakra, animations, announcer voice, themed UI.
• Interaction layer: webcam, microphone, live voice, hand-sign performance.
• Systems layer: turn logic, validation, animation timing, multiplayer sync.

• Voice-first gameplay: players do not just click abilities, they say them out loud.
• Physical input: attacks depend on actually performing hand signs in front of the camera.
• AI as game master: Gemini acts like a dramatic announcer and generates structured game actions.
• Live PvP loop: state updates, video, and casting all happen during the same browser session.

• React 18
• TypeScript
• Vite
• Zustand for client state
• Framer Motion for UI and battle motion
• Sonner for toasts
• @google/genai for Gemini Live in the browser
• @fishjam-dev/ts-client for WebRTC media signaling
• @mediapipe/tasks-vision for headband / face overlay features

• FastAPI
• Uvicorn
• Pydantic
• httpx
• python-dotenv
• google-genai
• inference-sdk for Roboflow
• fishjam-server-sdk

• Docker Compose for local multi-service orchestration
• Fishjam for WebRTC media
• Caddy for production reverse proxy / TLS

flowchart LR
  A[React Frontend] -->|REST| B[FastAPI Backend]
  A -->|WebSocket game state| B
  A -->|WebRTC signaling| C[Fishjam]
  B -->|Management API| C
  B -->|LLM requests| D[Gemini]
  B -->|Vision inference| E[Roboflow]

• The backend creates a game ID and stores match state in memory.
• Players join from separate browser tabs or devices.
• Game state is pushed through WebSocket and polling-assisted presence logic.
• The match automatically transitions from waiting to active once both players are present.

• The frontend opens a Gemini Live session using an ephemeral token from the backend.
• Gemini receives microphone audio and returns tool calls.
• It can call:
  • declare_jutsu
  • chakra_recovery
• For jutsu casting, Gemini returns:
  • jutsu name
  • hand-sign sequence
  • creativity score
  • description
  • spell animation type

• During casting, the frontend captures frames from the local camera at intervals.
• The backend sends frames to Roboflow.
• Roboflow predicts the sign class.
• The backend compares the predicted sign to the expected next sign.
• Once all signs are validated, the turn can resolve successfully.

• Successful casts deal damage based on:
  • sign count
  • creativity score
  • completion speed
• Failed casts still cost chakra.
• Chakra recovery restores +20 chakra.
• The backend broadcasts the updated state to both players.

• Fishjam handles player video/audio transport through WebRTC.
• The game UI renders local and opponent feeds side by side.
• Additional overlays such as headbands and spell animations are rendered on top of the gameplay scene.

The repo is designed to run as:

• Frontend: Vite dev server
• Backend: FastAPI / Uvicorn
• Media server: Fishjam

Recommended local setup:

docker compose up --build

This starts:

• backend on http://localhost:8000
• fishjam on http://localhost:5002

Then run the frontend separately:

cd frontend
npm install
npm run dev

Open:

• http://localhost:5173 for frontend dev
• http://localhost:8000 for backend

Vite proxies /api and /ws to the backend in development.

The production stack in this repo is:

• Caddy in front
• FastAPI backend
• Fishjam media server

docker-compose.prod.yml runs:

• caddy
• backend
• fishjam

Caddy proxies:

• /socket/* to Fishjam
• /ws/* to the backend
• everything else to the backend

This keeps the public surface relatively simple while still supporting:

• browser app/API traffic
• websocket state updates
• WebRTC signaling/media bootstrap

backend/                 FastAPI app, game logic, Gemini, Roboflow integration
frontend/                React app, UI, hooks, overlays, casting flow
frontend/public/         Static themed assets
docker-compose.yml       Local backend + Fishjam
docker-compose.prod.yml  Production stack with Caddy
Caddyfile                Reverse proxy configuration
requirements.txt         Python dependencies

Create a root .env file from the example:

cp .env.example .env

Important variables:

For WebRTC in Docker, you will usually also need:

export FJ_WEBRTC_TURN_IP=<your-lan-or-public-ip>

Then start compose:

docker compose up --build

For production builds with a split frontend/API host, copy:

cp frontend/env.production.example frontend/.env.production

Then set:

• VITE_API_BASE_URL
• VITE_WS_BASE_URL
• VITE_FISHJAM_HOST
• VITE_FISHJAM_PORT if needed

Do not commit frontend/.env.production.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn backend.main:app --reload --host 127.0.0.1 --port 8000

cd frontend
npm install
npm run dev

• A game starts in waiting state.
• Once both players are connected, it becomes active.
• Each player has:
  • HP
  • chakra
  • optional pending jutsu state
• Completed jutsu trigger a spell animation before turn resolution.
• The backend persists runtime game state in games_state.json.

This project explores what happens when you combine:

• anime-inspired game design
• LLM tool calling
• real-time browser voice
• computer vision
• multiplayer state sync
• WebRTC media

It is part game prototype, part AI interaction experiment, and part real-time systems playground.

• games_state.json is local runtime state and should not be committed.
• Gemini is used through server-issued ephemeral tokens, so the frontend does not need a permanent Gemini API key in normal operation.
• Fishjam WebRTC setup depends on a reachable IP or hostname; loopback often does not work for real device-to-device media.
• Chrome is the safest default browser for microphone, camera, and WebRTC behavior during development.