Use an adapter, not provider-specific UI code
The safest architecture is to keep the browser focused on capture and interaction while a server-side gateway owns authentication, provider selection, request normalization, and logging. The browser should never receive a long-lived provider key.
Define a small internal contract for starting a session and requesting a response. Normalize transcript, image frame, locale, session ID, timing metadata, and safety flags. A provider adapter can then translate that contract to API Mart today or an approved SeedRealtime endpoint later.
- Browser: permissions, media preview, speech controls, cancellation
- Server route: validation, rate limiting, secrets, provider adapter
- Provider adapter: model name, payload mapping, error normalization
- Observability: latency, failures, usage, user-visible outcome
Design the session lifecycle before streaming everything
A useful first version does not need to stream every raw frame. Start a session, capture speech, sample a relevant frame, and send a bounded request. This provides a predictable privacy and cost envelope while the product team validates the core use case.
When true streaming access is available, extend the same contract with session tokens, incremental media transport, server events, interruption signals, and explicit shutdown. Make start, reconnect, timeout, and end states visible to the user.
- Idle → permission request → preview → connected → ended
- Abort stale requests when a newer user turn supersedes them
- Limit image size, frame frequency, transcript length, and duration
- Return clear retryable and non-retryable errors
Production checklist for latency, privacy, and migration
Measure capture time, upload time, provider latency, first output, and total turn completion separately. A fast model cannot compensate for oversized frames, distant regions, blocked microphones, or a user interface that hides connection state.
Plan migration as a configuration and adapter change. Keep model-specific fields out of shared UI state, test both providers against the same scenarios, and use a controlled rollout. Confirm official terms, data retention, and regional routing before enabling real customer media.
- Store secrets only in encrypted server environment variables
- Apply origin checks, rate limits, payload limits, and abuse controls
- Document consent, retention, deletion, and human review policies
- Run a side-by-side evaluation before changing the default adapter
FAQ
Frequently asked questions
Can I put an API Mart key in browser JavaScript?
No. Keep provider credentials on the server and expose only a narrowly scoped application route with validation, rate limits, and appropriate authentication.
Do I need WebRTC for the first prototype?
Not necessarily. A bounded transcript-plus-frame request can validate the experience. Use a streaming transport when the use case and approved model endpoint genuinely require continuous media.
How do I switch to an official endpoint later?
Keep a provider-neutral request contract and implement a new server adapter. Compare outputs and latency, then change the configured adapter through a controlled rollout.