#26 Design an AI Chatbot System (ChatGPT-like) — Solution

✦ AI-Generated Solution · System Design (full-stack, frontend-emphasis) · Comprehensive This question is scored mostly on frontend architecture + streaming, with a deliberately stateless, no-database backend.

1. Requirements

Functional

• Authenticated access before chatting.
• Send a message, receive an AI response.
• Streaming responses token-by-token (not all at once).
• Ephemeral sessions — refresh starts a brand-new conversation; nothing persisted server-side.

Constraints (given)

• No backend database for messages/history.
• All conversation data lives in the browser.
• Emphasis on full-stack reasoning, especially the client.

2. Architecture

The only backend component is a thin BFF / proxy that (a) holds the model API key so it never reaches the browser, (b) enforces auth and rate limits, and (c) relays the streamed tokens. There is no message store.

3. Streaming: SSE vs WebSocket (the core trade-off)

Recommendation: SSE. A chat completion is a one-way token stream; SSE is simpler, proxy-friendly, and has built-in reconnection. Reach for WebSockets only if you need bidirectional features (live collaboration, interrupt/typing signals with server push). The modern variant is fetch() with a ReadableStream reading the text/event-stream body, which also lets you POST the prompt (raw EventSource is GET-only).

// Client: stream tokens from the BFF
async function streamChat(messages, onToken, signal) {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${token}` },
    body: JSON.stringify({ messages, stream: true }),
    signal,                                   // AbortController -> "stop generating"
  });
  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buf = "";
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buf += decoder.decode(value, { stream: true });
    for (const line of buf.split("\n\n")) {            // SSE frames split on blank line
      if (line.startsWith("data: ")) {
        const data = line.slice(6);
        if (data === "[DONE]") return;
        onToken(JSON.parse(data).delta);               // append token to UI
      }
    }
    buf = buf.slice(buf.lastIndexOf("\n\n") + 2);
  }
}

4. Frontend Architecture

• Component tree: App → AuthGate → ChatView → (MessageList(virtualized), Composer). Token deltas append to the last assistant message.
• State management: a single store (Redux/Zustand/Context+reducer) holding messages[], streamingStatus (idle | streaming | error), and the in-flight AbortController. Keep streaming text in a dedicated field and commit to the message on [DONE] to avoid re-rendering the whole list per token.
• Efficient streaming render: append to a single text node / use requestAnimationFrame batching; virtualize the message list so long chats stay smooth; auto-scroll only when the user is at the bottom.
• Browser storage: because sessions are ephemeral, keep the live conversation in in-memory state; optionally mirror to sessionStorage (cleared on tab close) — explicitly not localStorage, to honor "refresh starts fresh." Mention storage-quota handling if persistence were ever wanted.

5. Auth & Credential Management

• User authenticates via OAuth/OIDC → short-lived access token (in memory) + refresh token in an HttpOnly, Secure, SameSite cookie (not readable by JS → mitigates XSS token theft).
• The model API key never touches the browser — the BFF injects it server-side. The browser only ever holds the user's token.
• BFF enforces per-user rate limits and usage quotas before proxying.

6. Error Handling & Resilience

• Network/stream interruption: detect a dropped reader; show a "retry" affordance. Because state is client-side, you can re-send the conversation so far to continue.
• API timeout / 429: exponential backoff with jitter; surface a friendly message; preserve the user's typed input.
• Partial responses: keep already-streamed tokens; mark the message as incomplete so the user can regenerate.
• Stop generation: AbortController.abort() cancels the fetch and tells the BFF to close the upstream stream.

7. Summary