流式输出

安装依赖

调用流式输出

测试

之前文章《用 LangChain 驱动本地 Ollama 模型》讲叙了使用 LangChain 进行大模型对话。
大模型的响应时间一般都会比较长，那么如何考虑给用户更好的体验呢？

图片#B #S #R #60% #auto

流式输出

类似打字机一样的效果，按token输出。

安装依赖

pip install -U uvicorn "fastapi[standard]" "langchain[openai]"

调用流式输出

核心方法：stream/astream

import json

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import StreamingResponse
from langchain_openai import ChatOpenAI

app = FastAPI()
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)


@app.post("/api/bot/chat")
async def bot_chat(request: dict):
    query = request.get("query", "你好")

    llm = ChatOpenAI(
        model="qwen3.5:35b",
        base_url="http://192.168.31.24:4000",
        api_key="your api key",
        temperature=0.7,
        streaming=True,
    )

    system_prompt = (
        "你是一个会展示思考过程的AI。"
    )

    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": query},
    ]

    async def generate():
        # 直接用 LLM astream，逐 token 流出
        async for chunk in llm.astream(messages):
            content = chunk.content or ""
            if not content:
                continue
            yield json.dumps({
                "type": "chunk",
                "content": content,
            }, ensure_ascii=False) + "\n"

        yield json.dumps({"type": "done"}) + "\n"

    return StreamingResponse(
        generate(),
        media_type="application/x-ndjson",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",
        },
    )


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)

测试

curl --location --request POST 'http://127.0.0.1:8000/api/bot/chat' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "请一步步思考：2+3等于多少？"
}'

图片#B #S #R #60% #auto

思考过程

提示词

在大模型的响应参数里，只有部分模型是带有reasoning，如果要兼容大部分模型，就要换种方式，输出时带标签标识。

#... 其它不变

system_prompt = (
    "你是一个会展示思考过程的AI。\n"
    "请先输出你的思考过程（用<THINK>标签包裹），"
    "然后再输出最终答案（用<FINAL>标签包裹）。\n\n"
    "示例：\n"
    "<THINK>这里是推理过程</THINK>\n"
    "<FINAL>这里是最终答案</FINAL>"
)

#... 其它不变

页面实现

通过fetch实现简单示例效果

<!DOCTYPE html>
<html lang="zh">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>MiMo Chat</title>
  <style>
        * {
          margin: 0;
          padding: 0;
          box-sizing: border-box;
        }

        :root {
          --bg: #0a0a0f;
          --surface: #12121a;
          --border: #2a2a3a;
          --think-bg: #13130a;
          --think-border: #3a3a1a;
          --think-text: #d4a843;
          --think-label: #b8941f;
          --answer-text: #e0e0e0;
          --accent: #6366f1;
          --accent-glow: rgba(99, 102, 241, 0.15);
          --muted: #6b6b80;
        }

        body {
          font-family: 'Inter', -apple-system, sans-serif;
          background: var(--bg);
          color: var(--answer-text);
          min-height: 100vh;
          display: flex;
          flex-direction: column;
          align-items: center;
        }

        .chat-container {
          width: 100%;
          max-width: 760px;
          padding: 24px 16px;
          display: flex;
          flex-direction: column;
          gap: 20px;
        }

        .input-area {
          position: sticky;
          top: 0;
          z-index: 10;
          background: var(--bg);
          padding: 16px 0;
          display: flex;
          gap: 10px;
        }

        .input-area textarea {
          flex: 1;
          background: var(--surface);
          border: 1px solid var(--border);
          border-radius: 12px;
          padding: 12px 16px;
          color: var(--answer-text);
          font-family: inherit;
          font-size: 14px;
          resize: none;
          outline: none;
          min-height: 44px;
          max-height: 120px;
          transition: border-color 0.2s;
        }

        .input-area textarea:focus {
          border-color: var(--accent);
          box-shadow: 0 0 0 3px var(--accent-glow);
        }

        .send-btn {
          background: var(--accent);
          border: none;
          border-radius: 12px;
          padding: 0 20px;
          color: #fff;
          font-weight: 600;
          font-size: 14px;
          cursor: pointer;
          transition: all 0.2s;
          white-space: nowrap;
        }

        .send-btn:hover {
          opacity: 0.85;
        }

        .send-btn:disabled {
          opacity: 0.4;
          cursor: not-allowed;
        }

        .message {
          background: var(--surface);
          border: 1px solid var(--border);
          border-radius: 16px;
          overflow: hidden;
          animation: slideUp 0.3s ease;
        }

        @keyframes slideUp {
          from {
            opacity: 0;
            transform: translateY(12px);
          }
          to {
            opacity: 1;
            transform: translateY(0);
          }
        }

        .message-header {
          padding: 12px 16px;
          border-bottom: 1px solid var(--border);
          display: flex;
          justify-content: space-between;
          font-size: 12px;
          color: var(--muted);
        }

        .message-body {
          padding: 16px;
        }

        .think-section {
          background: var(--think-bg);
          border: 1px solid var(--think-border);
          border-radius: 10px;
          margin-bottom: 12px;
          overflow: hidden;
          display: none;
        }

        .think-section.visible {
          display: block;
        }

        .think-header {
          padding: 10px 14px;
          display: flex;
          align-items: center;
          gap: 8px;
          cursor: pointer;
          user-select: none;
          font-size: 13px;
          color: var(--think-label);
          font-weight: 500;
        }

        .think-arrow {
          display: inline-block;
          transition: transform 0.25s;
          font-size: 10px;
        }

        .think-arrow.collapsed {
          transform: rotate(-90deg);
        }

        .think-content {
          padding: 0 14px 12px;
          font-family: 'JetBrains Mono', monospace;
          font-size: 13px;
          line-height: 1.75;
          color: var(--think-text);
          white-space: pre-wrap;
          word-break: break-word;
          max-height: 800px;
          opacity: 1;
          overflow: hidden;
          transition: max-height 0.3s, padding 0.3s, opacity 0.3s;
        }

        .think-content.collapsed {
          max-height: 0;
          padding: 0 14px;
          opacity: 0;
        }

        .answer-content {
          font-size: 15px;
          line-height: 1.8;
          white-space: pre-wrap;
          word-break: break-word;
        }

        .cursor {
          display: inline-block;
          width: 2px;
          height: 1.1em;
          background: var(--accent);
          margin-left: 1px;
          animation: blink 0.7s step-end infinite;
          vertical-align: text-bottom;
        }

        @keyframes blink {
          50% {
            opacity: 0;
          }
        }

        .loading-dots {
          display: inline-flex;
          gap: 3px;
          margin-left: auto;
        }

        .loading-dots span {
          width: 5px;
          height: 5px;
          background: var(--think-label);
          border-radius: 50%;
          animation: bounce 1.2s infinite;
        }

        .loading-dots span:nth-child(2) {
          animation-delay: 0.15s;
        }

        .loading-dots span:nth-child(3) {
          animation-delay: 0.3s;
        }

        @keyframes bounce {
          0%, 80%, 100% {
            transform: scale(0.5);
            opacity: 0.3;
          }
          40% {
            transform: scale(1);
            opacity: 1;
          }
        }
    </style>
</head>
<body>
<div class="chat-container">
  <div class="input-area">
        <textarea id="queryInput" placeholder="输入问题..." rows="1" onkeydown="if(event.key==='Enter'&&!event.shiftKey){event.preventDefault();sendMessage();}">
            请一步步思考：2+3等于多少？
        </textarea>
    <button class="send-btn" id="sendBtn" onclick="sendMessage()">发送</button>
  </div>
  <div id="messages"></div>
</div>

<script>
    let isSending = false;

    async function sendMessage() {
      if (isSending) return;
      const query = document.getElementById('queryInput').value.trim();
      if (!query) return;

      isSending = true;
      document.getElementById('sendBtn').disabled = true;

      const msgId = crypto.randomUUID().replace(/-/g, '');
      const conversationId = localStorage.getItem('convId') || crypto.randomUUID().replace(/-/g, '');
      localStorage.setItem('convId', conversationId);

      const msgEl = createMessage(msgId);
      document.getElementById('messages').prepend(msgEl);

      const state = {
        raw: '',
        lastThinkLen: 0,
        lastAnswerLen: 0,
        thinkCollapsed: false,
      };

      try {
        const resp = await fetch('http://127.0.0.1:8000/api/bot/chat', {
          method: 'POST',
          headers: {'Content-Type': 'application/json'},
          body: JSON.stringify({
            msgId, conversationId, query,
            isEditedQuery: false,
            modelConfig: {enableThinking: true, webSearchStatus: "disabled", model: ""},
            multiMedias: [],
          }),
        });

        const reader = resp.body.getReader();
        const decoder = new TextDecoder();
        let buffer = '';

        while (true) {
          const {done, value} = await reader.read();
          if (done) break;

          buffer += decoder.decode(value, {stream: true});
          const lines = buffer.split('\n');
          buffer = lines.pop();

          for (const line of lines) {
            if (!line.trim()) continue;
            let data;
            try {
              data = JSON.parse(line);
            } catch (e) {
              continue;
            }

            if (data.type === 'chunk') {
              state.raw += data.content;
              render(msgEl, state);
            } else if (data.type === 'done') {
              render(msgEl, state);
              removeCursor(msgEl);
            }
          }
        }

        if (buffer.trim()) {
          try {
            const d = JSON.parse(buffer);
            if (d.type === 'chunk') state.raw += d.content;
          } catch (e) {
          }
        }
        render(msgEl, state);
        removeCursor(msgEl);

      } catch (err) {
        msgEl.querySelector('.answer-content').textContent = '错误: ' + err.message;
      } finally {
        isSending = false;
        document.getElementById('sendBtn').disabled = false;
      }
    }

    /**
     * 当标签被 chunk 切断时（如 </THI → NK>），正则的 |$ 兜底
     * 会把部分标签（</THI）当成内容显示。
     *
     * 示例：
     *   raw = "<THINK>2+3等于5</THI"
     *   thinkMatch[1] = "2+3等于5</THI"      ← 正则兜底，残留标签混入
     *   raw.includes('</THINK>') = false    ← 完整标签还没到
     *   stripPartial(thinkText, '</THI')    ← 从末尾剥掉残留
     *   → "2+3等于5"                         ← 干净
     *
     *   raw = "<THINK>2+3等于5</THINK>"
     *   thinkMatch[1] = "2+3等于5"           ← 正则精确匹配，无残留
     *   raw.includes('</THINK>') = true
     *   → 不需要剥
     */
    function render(msgEl, state) {
      const raw = state.raw;

      const thinkMatch = raw.match(/<THINK>([\s\S]*?)(?:<\/THINK>|$)/);
      const answerMatch = raw.match(/<FINAL>([\s\S]*?)(?:<\/FINAL>|$)/);

      let thinkText = thinkMatch ? thinkMatch[1] : '';
      let answerText = answerMatch ? answerMatch[1] : '';

      // 标签未完整时，剥掉末尾的残留片段
      if (!raw.includes('</THINK>')) {
        thinkText = stripPartial(thinkText, '</THINK>');
      }
      if (!raw.includes('</FINAL>')) {
        answerText = stripPartial(answerText, '</FINAL>');
      }

      // Think 区域
      if (thinkText.length > 0) {
        const section = msgEl.querySelector('.think-section');
        const content = msgEl.querySelector('.think-content');

        section.classList.add('visible');

        if (thinkText.length > state.lastThinkLen) {
          const delta = thinkText.substring(state.lastThinkLen);
          content.appendChild(document.createTextNode(delta));
          content.scrollTop = content.scrollHeight;
          state.lastThinkLen = thinkText.length;
        }

        if (raw.includes('</THINK>') && !state.thinkCollapsed) {
          state.thinkCollapsed = true;
          const dots = section.querySelector('.loading-dots');
          if (dots) dots.style.display = 'none';
          section.querySelector('.think-arrow').classList.add('collapsed');
          content.classList.add('collapsed');
        }
      }

      // Answer 区域
      if (answerText.length > 0) {
        const content = msgEl.querySelector('.answer-content');

        if (answerText.length > state.lastAnswerLen) {
          const delta = answerText.substring(state.lastAnswerLen);

          const oldCursor = content.querySelector('.cursor');
          if (oldCursor) oldCursor.remove();

          content.appendChild(document.createTextNode(delta));
          const cursor = document.createElement('span');
          cursor.className = 'cursor';
          content.appendChild(cursor);

          state.lastAnswerLen = answerText.length;
        }
      }
    }

    /**
     * 从文本末尾剥掉部分标签
     * 比如 stripPartial("2+3等于5</THI", "</THINK") → "2+3等于5"
     * 逐个尝试 "</", "</T", "</TH", ... 直到完整标签
     */
    function stripPartial(text, fullTag) {
      for (let i = 2; i <= fullTag.length; i++) {
        const suffix = fullTag.substring(0, i);
        if (text.endsWith(suffix)) {
          return text.slice(0, -suffix.length);
        }
      }
      return text;
    }

    function createMessage(msgId) {
      const el = document.createElement('div');
      el.className = 'message';
      el.id = `msg-${msgId}`;
      el.innerHTML = `
            <div class="message-header">
                <span>AI 助手</span>
                <span class="timestamp">${new Date().toLocaleTimeString()}</span>
            </div>
            <div class="message-body">
                <div class="think-section">
                    <div class="think-header" onclick="toggleThink(this)">
                        <span class="think-arrow">▼</span>
                        <span>思考过程</span>
                        <div class="loading-dots"><span></span><span></span><span></span></div>
                    </div>
                    <div class="think-content"></div>
                </div>
                <div class="answer-content"></div>
            </div>
        `;
      return el;
    }

    function removeCursor(msgEl) {
      msgEl.querySelectorAll('.cursor').forEach(c => c.remove());
    }

    function toggleThink(header) {
      header.querySelector('.think-arrow').classList.toggle('collapsed');
      header.parentElement.querySelector('.think-content').classList.toggle('collapsed');
    }
</script>
</body>
</html>

图片#B #S #R #60% #auto

思考模型

如果业务需求需要大模型输出JSON格式，可以考虑使用仅思考模型，在回复前总会思考，后续有时间说这个方案

BugShare

手撸AI对话助手带上思考过程

流式输出

安装依赖

调用流式输出

测试

思考过程

提示词

页面实现

思考模型

More Products