From 0d4d7b208be8c2144e2003cc34db1af84ad0d946 Mon Sep 17 00:00:00 2001 From: Yige Date: Mon, 17 Aug 2026 01:25:46 -0400 Subject: [PATCH] =?UTF-8?q?feat(web-access):=20v3.0=20=E5=86=85=E7=BD=AE?= =?UTF-8?q?=E6=B5=8F=E8=A7=88=E5=99=A8=E8=83=BD=E5=8A=9B=E9=9D=A2=E5=90=8C?= =?UTF-8?q?=E6=AD=A5=E2=80=94=E2=80=94=E5=88=A0=E9=99=A4=20Python=20Playwr?= =?UTF-8?q?ight=20=E5=9B=9E=E8=90=BD=EF=BC=88S31=EF=BC=89=20(#85)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## 变更内容 / What 浏览器升级计划 S31 最终验收信号：web-access 技能与内置受管浏览器新能力面对齐，**删除「用户手工启动调试 Chrome + Python Playwright」回落路径**。 - **删除回落路径**：Prerequisites: Chrome CDP Setup、Layer 3 CDP Browser（Python Playwright 模板）、Installation Note（pip install playwright）整段删除；references/cdp-browser.md 文件删除；jina-reader.md 的 CDP 引用改为 page.extract-text - **订正陈旧断言**： - 「没有批量取文通道」→ BrowserSnapshot mode:text / page.extract-text（maxBytes/cursor 分页，超出截断给 nextCursor） - 「page.evaluate 基本不可用」→ 返回真实值（expression/awaitPromise，超预算截断标 truncated）；仍走人工闸门 - 「截图前必须 tab.activate / 串行截图 / BROWSER_TAB_HOST_NOT_FOUND」→ S36 订正：Agent 单标签会话免 activate；多标签后台 tab 秒级报 BROWSER_VIEWPORT_UNAVAILABLE；命令超时只 stop 不 close，标签页可重试 - 「accessibility 超限即失败」→ 尊重 depth + maxBytes 截断翻页（S8） - 「只有整页截图」→ clip{x,y,width,height,scale≤4} + captureBeyondViewport - **provides.tools 加 BrowserScript**（code-mode；信任级别等同 Bash） - **新增选用规则（D4 唯一约束机制）**：反检测站点一律优先 input.*（#1808 输入拟真 + 身份一致性）；page.element 写类仅用于表单批量填充等站点不检测场景；JS 直调 el.click() 为禁止回退 - **新增 fetch.browser 配方**：page.evaluate 页面上下文跑 fetch（带 origin Cookie、同 origin、受 Grant origins 约束）——登录态取站内接口的正解 - **能力速查**：page.element 九 op / page.wait 九 until / inline wait 块 / loc= 方言 / BrowserScript / 跨源 iframe 快照（S35） - **版本** 2.2.1 → 3.0.0（删除回落层为 breaking）；source_hash 重算；required_client_version 维持 10.0.98（新能力在正文标注 10.0.112+） ## Why v2.x 时代回落路径存在的每一条理由（无批量取文、evaluate 不可用、截图必须串行 activate）均已被 S2–S14/S35/S36 覆盖；文档继续引导用户手工起调试 Chrome 会误导新 Agent 走已废弃路径。 双语同步修改（SKILL.md / SKILL.zh-CN.md heading 数一致，i18n-validate 通过）。 - [x] CLA --- skills/web-access/SKILL.md | 366 +++++++----------- skills/web-access/SKILL.zh-CN.md | 338 ++++++---------- skills/web-access/references/browser-tools.md | 301 ++++++++++---- skills/web-access/references/cdp-browser.md | 330 ---------------- skills/web-access/references/jina-reader.md | 15 +- 5 files changed, 482 insertions(+), 868 deletions(-) delete mode 100644 skills/web-access/references/cdp-browser.md diff --git a/skills/web-access/SKILL.md b/skills/web-access/SKILL.md index f8270ea..e2b8153 100644 --- a/skills/web-access/SKILL.md +++ b/skills/web-access/SKILL.md @@ -5,18 +5,17 @@ description: >- — searching for current information, fetching public web pages, browsing login-gated sites (微博/小红书/B站/飞书/Twitter), comparing products, researching topics, gathering documentation, or summarizing news. - This skill orchestrates four complementary layers: (1) WebSearch + WebFetch + This skill orchestrates three complementary layers: (1) WebSearch + WebFetch for public pages, (2) Jina Reader as the default token-optimization layer for - heavy/JS-rendered pages, (3) the governed built-in browser (isolated - BrowserSpace + Cookie import) to reach and interact with login-gated sites, - and (4) Chrome DevTools Protocol via Python Playwright as the fallback for - bulk text extraction and complex automation. Always cite source URLs. + heavy/JS-rendered pages, and (3) the governed built-in browser (isolated + BrowserSpace + Cookie import + bulk text extraction + waits + code mode) to + reach, interact with, and read login-gated sites. Always cite source URLs. Use when 用户提到 联网搜索、上网查、 查资料、抓取网页、研究、调研、最新资讯、文档查询、对比、竞品、技术文档、 新闻、网址、URL、找一下、搜一下、查一下、小红书、B站、微博、飞书、Twitter、 推特、X、知乎、公众号、已登录、登录状态。 license: Complete terms in LICENSE.txt -version: 2.2.1 +version: 3.0.0 type: procedural risk_level: low status: enabled @@ -27,13 +26,14 @@ tags: - fetch - research - browsing + - browser - cdp - - playwright provides: tools: - BrowserManage - BrowserSnapshot - BrowserAct + - BrowserScript - BrowserImport - BrowserShare - SitePatternRead @@ -41,7 +41,7 @@ provides: - LocalBookmarks metadata: author: desirecore - updated_at: '2026-08-06' + updated_at: '2026-08-17' i18n: default_locale: en-US source_locale: zh-CN @@ -50,17 +50,17 @@ metadata: - en-US zh-CN: name: 联网访问 - short_desc: 联网搜索、网页抓取、内置受管浏览器登录态访问、研究调研工作流 - description: 四层联网访问工具包——搜索公开页面、Jina 优化抓取、内置受管浏览器登录态访问、Python Playwright CDP 兜底。 + short_desc: 联网搜索、网页抓取、内置受管浏览器登录态访问与取文、研究调研工作流 + description: 三层联网访问工具包——搜索公开页面、Jina 优化抓取、内置受管浏览器完成登录态访问、交互与取文。 body: ./SKILL.zh-CN.md - source_hash: sha256:ce46e1982de8cd65 + source_hash: sha256:20c98f047378220a translated_by: human en-US: name: Web Access short_desc: Web search, page fetching, logged-in access via the governed built-in browser, research workflows - description: A four-layer web-access toolkit — search public pages, fetch heavy pages via Jina Reader, reach logged-in sites through the governed built-in browser, and fall back to Chrome CDP. + description: A three-layer web-access toolkit — search public pages, fetch heavy pages via Jina Reader, and reach, interact with, and read logged-in sites through the governed built-in browser. body: ./SKILL.md - source_hash: sha256:ce46e1982de8cd65 + source_hash: sha256:20c98f047378220a translated_by: human market: icon: >- @@ -73,9 +73,8 @@ market: stroke="url(#wa-a)" stroke-width="1" stroke-opacity="0.35"/> category: research @@ -90,28 +89,30 @@ market: ## L0: One-line Summary -A four-layer web-access toolkit — search public pages, optimize fetches via Jina Reader, reach login-gated sites through the governed built-in browser, and fall back to Chrome CDP. +A three-layer web-access toolkit — search public pages, optimize fetches via Jina Reader, and reach, interact with, and read login-gated sites through the governed built-in browser (v3.0 ships bulk text extraction, waits, and code mode in-browser; the Python Playwright fallback is gone). ## L1: Overview & Use Cases ### Capability -web-access is a **procedural skill** that provides four complementary layers of web access: +web-access is a **procedural skill** that provides three complementary layers of web access: - **L1** (WebSearch + WebFetch): public, static pages - **L2** (Jina Reader): JS-rendered heavy pages, saving tokens by default -- **L3-fast** (governed built-in browser — **rewritten in v2.1**): reach and *interact with* logged-in / interactive sites — isolated BrowserSpace per task, zero Python dependency, every action carries a signed receipt. **Reading long article text is still L2/L3-fallback's job** — see the extraction note below -- **L3-fallback** (Chrome CDP + Python Playwright): backup for complex automation (long waits, race conditions, custom in-browser scripts) +- **L3** (governed built-in browser, capability surface completed in v3.0): reach, *interact with*, and **read** logged-in / interactive sites — isolated BrowserSpace per task, zero Python dependency, every action carries a signed receipt. Bulk text extraction (`page.extract-text`), discriminated waits (`page.wait`), and code mode (`BrowserScript`) all close the loop inside this layer -### v2.1 — governed built-in browser (default-hidden, exposed only after Skill activation) +The v2.x fourth layer — "user manually launches a debug Chrome + Python Playwright CDP" — was removed in v3.0: every reason it existed for (no bulk text channel, evaluate unusable, screenshots must activate-serialize) is now covered by the built-in browser, see the cheatsheet below. -When you call `Skill('web-access')`, the following 8 tools are injected into the current session so the LLM can drive the built-in browser directly: +### v3.0: governed built-in browser (default-hidden, exposed only after Skill activation) + +When you call `Skill('web-access')`, the following 9 tools are injected into the current session so the LLM can drive the built-in browser directly: | Tool | Purpose | |------|---------| | BrowserManage | Create/destroy isolated BrowserSpace, start sessions, manage tabs | -| BrowserSnapshot | `semantic` / `accessibility` / `visual` page snapshots — the primary way to read a page | -| BrowserAct | One governed action per call: navigate, click, type, scroll, screenshot, … | +| BrowserSnapshot | `semantic` / `text` / `accessibility` / `visual` snapshots — the primary way to read a page | +| BrowserAct | One governed action per call: navigate, input, extract text, wait, element ops, screenshot, … | +| BrowserScript | **Code mode**: one async JS script issues browser commands back-to-back, eliminating per-action round trips (trust level equals Bash) | | BrowserImport | Import Cookies from the user's Chrome/Edge/Firefox/Safari profile (human-approved; **needs `browser.import.*` granted by the Host — not available to a plain `create_space` session**) | | BrowserShare | Delegate a Space/Session to another Agent (isolated / snapshot / copy-on-write / live) | | SitePatternRead / SitePatternWrite | Per-domain "site experience" (AgentFS three-layer) | @@ -119,20 +120,22 @@ When you call `Skill('web-access')`, the following 8 tools are injected into the > **Important**: before `Skill('web-access')` is called, none of these tools appear in the LLM tools list — default conversations don't pay their token cost. See [references/browser-tools.md](references/browser-tools.md). > -> **Removed in v2.1**: `BrowserListTabs` / `BrowserNavigate` / `BrowserEval` / `BrowserClick` / `BrowserScreenshot` / `BrowserScroll` / `BrowserSetFiles` / `BrowserCloseTab` and the cdp-proxy behind them are retired. Calling them now returns "该旧 BrowserXxx/cdp-proxy 入口已停用". Requires client v10.0.98+. +> **Removed in v2.1**: `BrowserListTabs` / `BrowserNavigate` / `BrowserEval` / `BrowserClick` / `BrowserScreenshot` / `BrowserScroll` / `BrowserSetFiles` / `BrowserCloseTab` and the cdp-proxy behind them are retired. Calling them now returns "该旧 BrowserXxx/cdp-proxy 入口已停用". Requires client v10.0.98+; `page.extract-text` / `page.element` / `page.wait` / inline wait blocks / cross-origin iframe snapshots need v10.0.112+, and `BrowserScript` needs a build containing S17/S18. ### Use Cases - The user needs to search for current information or research a specific topic - The user needs to fetch public web content or technical documentation -- The user needs to access logged-in sites (Xiaohongshu, Bilibili, Weibo, Feishu, Twitter, etc.) +- The user needs to access logged-in sites (Xiaohongshu, Bilibili, Weibo, Feishu, Twitter, etc.) and **read the body text** +- The user needs to pull data from a site's own API in a logged-in context (lists, comments, orders, …) - The user needs to compare products, aggregate news, or investigate API/library versions ### Core Value -- **Four-layer progression**: from lightweight search to heavy JS rendering to logged-in access — pick on demand -- **Token optimization**: Jina Reader cuts token usage by 50–80% by default -- **Logged-in session reuse**: attach to the user's Chrome via CDP in the fallback layer, or — where the Host has granted `browser.import.*` — import their Cookies into an isolated Space via BrowserImport; either way, no re-login required +- **Three-layer progression**: from lightweight search to heavy JS rendering to logged-in access — pick on demand +- **Token optimization**: Jina Reader cuts token usage by 50–80% by default; `page.extract-text`'s maxBytes/cursor paging keeps even long logged-in articles under control +- **Logged-in session reuse**: where the Host has granted `browser.import.*`, BrowserImport brings the user's Cookies into an isolated Space — no re-login required +- **Zero external dependencies**: no Python/Playwright install, no manually launched debug Chrome ## L2: Detailed Specification @@ -144,53 +147,6 @@ When you complete a research task, you **MUST** cite all source URLs in your res If any fetch fails, explicitly tell the user which URL failed and which fallback you used. ---- - -## Prerequisites: Chrome CDP Setup (for login-gated sites) - -**Required for the L3-fallback layer** (Python Playwright) — and, in practice, still the reliable way to reuse a login. The L3-fast built-in browser can skip it only when the Host has granted `browser.import.*` so that `BrowserImport` actually works; otherwise set this up. - -### One-time setup - -Launch a dedicated Chrome instance with remote debugging enabled: - -**macOS**: -```bash -/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \ - --remote-debugging-port=9222 \ - --user-data-dir="${DESIRECORE_ROOT}/chrome-profile" -``` - -**Linux**: -```bash -google-chrome \ - --remote-debugging-port=9222 \ - --user-data-dir="${DESIRECORE_ROOT}/chrome-profile" -``` - -**Windows (PowerShell)**: -```powershell -& "C:\Program Files\Google\Chrome\Application\chrome.exe" ` - --remote-debugging-port=9222 ` - --user-data-dir="$env:USERPROFILE\.desirecore\chrome-profile" -``` - -After launch: -1. Manually log in to the sites you need (Xiaohongshu, Bilibili, Weibo, Feishu, …) -2. Leave this Chrome window open in the background -3. Verify the debug endpoint: `curl -s http://localhost:9222/json/version` should return JSON - -### Verify CDP is ready - -Before any CDP operation, always run: -```bash -curl -s http://localhost:9222/json/version | python3 -c "import sys,json; d=json.load(sys.stdin); print('CDP ready:', d.get('Browser'))" -``` - -If the command fails, tell the user: "Please launch Chrome with the remote debugging port enabled (see the Prerequisites section of the web-access skill)." - ---- - ## Tool Selection Decision Tree ``` @@ -207,12 +163,12 @@ User intent │ (Jina Reader = default for JS-rendered content, saves tokens) │ ├─ "Read this login-gated page" (Xiaohongshu/Bilibili/Weibo/Feishu/Twitter/Zhihu/WeChat) - │ ├─→ Reach it: BrowserManage(create_space/start_session) → BrowserAct(tab.navigate) - │ │ → tab.activate + page.screenshot to confirm you landed on the content - │ │ (semantic snapshot has no body text). Logged-in? BrowserImport only if - │ │ browser.import.* was granted — otherwise reuse the login via CDP below. - │ └─→ Extract the text: verify CDP ready, then python3 playwright.connect_over_cdp() - │ → page.content() → Jina Reader / BeautifulSoup + │ └─→ BrowserManage(create_space/start_session) → BrowserAct(tab.navigate) + │ → BrowserAct(page.extract-text) ← body text read out directly, maxBytes/cursor paging + │ Logged in? BrowserImport only if browser.import.* was granted + │ + ├─ "Pull data from the site's API in a logged-in context" + │ └─→ fetch.browser recipe: run fetch inside BrowserAct(page.evaluate) with that origin's cookies │ ├─ "API documentation / GitHub / npm package info" │ └─→ Prefer official API endpoints over scraping HTML: @@ -221,24 +177,19 @@ User intent │ - PyPI: curl https://pypi.org/pypi//json │ └─ "Real-time interactive task" (click, fill form, scroll, screenshot) - ├─→ **Default: built-in browser** (BrowserManage → BrowserAct → BrowserSnapshot — - │ see references/browser-tools.md, no Python needed) - └─→ Fallback: CDP + Python Playwright (references/cdp-browser.md) when the built-in browser is insufficient - (e.g., complex race conditions, multi-event waits, long-running in-browser scripts) + └─→ built-in browser (BrowserManage → BrowserAct → BrowserSnapshot — + see references/browser-tools.md, no Python needed) ``` -### Four-layer strategy summary +### Three-layer strategy summary | Layer | Use case | Primary tool | Token cost | |-------|----------|--------------|------------| | L1 | Public, static | `WebFetch` | Low | | L2 | JS-heavy, long articles, token savings | `Bash curl r.jina.ai` | **Lowest** (Markdown pre-cleaned) | -| **L3-fast** | **Login-gated navigation & interaction (PRIMARY)** | **built-in browser (BrowserManage / BrowserAct / BrowserSnapshot)** | Medium | -| L3-fallback | Complex automation (race / long-wait / custom scripts) | `Bash + Python Playwright CDP` | Medium | +| **L3** | **Login-gated navigation, interaction & extraction (PRIMARY)** | **built-in browser (BrowserManage / BrowserAct / BrowserSnapshot / BrowserScript)** | Medium | -**Default priority**: L1 for simple public pages → L2 for heavy → **L3-fast for login-gated** → L3-fallback only when the built-in browser is insufficient. - ---- +**Default priority**: L1 for simple public pages → L2 for heavy → **L3 for login-gated (body text and in-site API data included)**. ## Supported Sites Matrix @@ -248,16 +199,14 @@ User intent | GitHub README, issues, PRs | `gh api` (best) → L1 WebFetch | Prefer API | | Hacker News, Reddit | L1 WebFetch | Public content | | Medium, Dev.to | L2 Jina Reader | JS-rendered, member gates | -| Twitter/X | L3 CDP (or L2 Jina with `x.com`) | Login required for full thread | -| Xiaohongshu (xiaohongshu.com) | L3 CDP | Login required | -| Bilibili (bilibili.com) | L3 CDP | Login needed for video desc/comments | -| Weibo (weibo.com) | L3 CDP | Long posts require login | -| Zhihu (zhihu.com) | L3 CDP | Long articles + comments require login | -| Feishu Docs (feishu.cn) | L3 CDP | Login required | +| Twitter/X | L3 (or L2 Jina with `x.com`) | Login required for full thread | +| Xiaohongshu (xiaohongshu.com) | L3 built-in browser + BrowserImport | Login required; body text via page.extract-text | +| Bilibili (bilibili.com) | L3 built-in browser + BrowserImport | Login needed for video desc/comments | +| Weibo (weibo.com) | L3 built-in browser + BrowserImport | Long posts require login | +| Zhihu (zhihu.com) | L3 built-in browser + BrowserImport | Long articles + comments require login | +| Feishu Docs (feishu.cn) | L3 built-in browser + BrowserImport | Login required | | WeChat Official Accounts (mp.weixin.qq.com) | L2 Jina Reader | Usually public, Jina cleans better | -| LinkedIn | L3 CDP | Login wall | - ---- +| LinkedIn | L3 built-in browser + BrowserImport | Login wall | ## Tool Reference @@ -280,7 +229,7 @@ WebFetch(url="https://example.com/article") Tips: - Results cached for 15 min - Returns cleaned Markdown with title + URL + body -- If body < 200 chars or looks garbled → escalate to Layer 2 (Jina) or Layer 3 (CDP) +- If body < 200 chars or looks garbled → escalate to Layer 2 (Jina) or Layer 3 (built-in browser) ### Layer 2: Jina Reader (default for heavy pages) @@ -298,111 +247,94 @@ Why Jina is the default token-saver: See [references/jina-reader.md](references/jina-reader.md) for advanced endpoints and rate limits. -### Layer 3: CDP Browser (login-gated access) +### Layer 3: built-in browser (login-gated access) -Use Python Playwright's `connect_over_cdp()` to attach to the user's running Chrome (which already has login cookies). **No re-login needed.** +The full command surface, capability tiers, and boundaries are in [references/browser-tools.md](references/browser-tools.md). The loop: -**Minimal template**: -```bash -python3 << 'PY' -from playwright.sync_api import sync_playwright +1. `BrowserManage(create_space)` → `BrowserManage(start_session)` for a sessionId + first tab +2. Need login state → `BrowserImport` (only if the Host granted `browser.import.*`; without that grant the user's Cookies cannot be reused — tell the user and continue without login or abort) +3. `BrowserAct(tab.navigate)` to reach the page +4. `BrowserSnapshot(semantic)` for interactive-element `ref` handles (cross-origin iframe elements are in the same tree with globally sequential refs) +5. Interact via `input.*` (humanized trajectories) or `page.element` (bulk form writes); wait for results via `page.wait` or inline wait blocks +6. Read body text via `BrowserSnapshot(text)` or `BrowserAct(page.extract-text)`; pull API data via the fetch.browser recipe +7. `BrowserManage(close_session)` when done -TARGET_URL = "https://www.xiaohongshu.com/explore/..." +Multi-step sequences (navigate→snapshot→click→wait→extract) can be done in one `BrowserScript` run, skipping the per-action IPC round trips. -with sync_playwright() as p: - browser = p.chromium.connect_over_cdp("http://localhost:9222") - context = browser.contexts[0] # reuse user's default context (has cookies) - page = context.new_page() - page.goto(TARGET_URL, wait_until="domcontentloaded") - page.wait_for_timeout(2000) # let lazy content load - html = page.content() - page.close() +## L3 Cheatsheet (v3.0) -# Print first 500 chars to verify -print(html[:500]) -PY -``` - -**Extract text via BeautifulSoup** (no Jina round-trip): -```bash -python3 << 'PY' -from playwright.sync_api import sync_playwright -from bs4 import BeautifulSoup - -with sync_playwright() as p: - browser = p.chromium.connect_over_cdp("http://localhost:9222") - page = browser.contexts[0].new_page() - page.goto("https://www.bilibili.com/video/BV...", wait_until="networkidle") - html = page.content() - page.close() - -soup = BeautifulSoup(html, "html.parser") -title = soup.select_one("h1.video-title") -desc = soup.select_one(".video-desc") -print("Title:", title.get_text(strip=True) if title else "N/A") -print("Desc:", desc.get_text(strip=True) if desc else "N/A") -PY -``` - -See [references/cdp-browser.md](references/cdp-browser.md) for: -- Per-site selectors (Xiaohongshu / Bilibili / Weibo / Zhihu / Feishu) -- Scrolling & lazy-load patterns -- Screenshot & form-fill recipes -- Troubleshooting connection issues - ---- - -## L3-fast: Built-in Browser Cheatsheet (v2.1) - -**Only after you call `Skill('web-access')` will the following tools appear in `tools[]`.** - -| Tool | One-line example | -|------|-----------------| -| `BrowserManage({ action: 'create_space', name, persistence: 'ephemeral' })` | One isolated Space per task | -| `BrowserManage({ action: 'start_session', spaceId, capabilities })` | Start a session, returns sessionId + first tab | -| `BrowserManage({ action: 'list_tabs' \| 'create_tab' \| 'close_session' })` | Tab / lifecycle management | -| `BrowserAct({ action: 'tab.navigate', params: { url } })` | Navigate the current tab | -| `BrowserSnapshot({ mode: 'semantic' })` | Read the page: interactive elements + `ref` handles | -| `BrowserAct({ action: 'input.click', params: { ref } })` | Click by snapshot `ref`, never by raw x/y | -| `BrowserAct({ action: 'input.text', params: { text } })` | Type into the focused element | -| `BrowserAct({ action: 'input.wheel', params: { x: 640, y: 400, deltaX: 0, deltaY: 720 } })` | Scroll to trigger lazy loading — `x`/`y` (or `ref`) is **required**, and the session needs `browser.input.pointer.wheel` | -| `BrowserAct({ action: 'tab.activate', params: { bounds } })` → `page.screenshot` | **activate first, then screenshot** | -| `BrowserImport({ action: 'discover' \| 'create_plan' \| 'dry_run' \| 'apply' \| 'rollback' \| 'list_plans' })` | Reuse the user's login cookies — those 6 are the **complete** action set; happy path is `discover → create_plan → dry_run → apply`. **Needs `browser.import.*`, which `create_space` does not grant** (see below) | - -Full API and edge cases: see [references/browser-tools.md](references/browser-tools.md). - -**How to read a page** — pick by what you need: +### Reading a page: pick the channel by need | You need | Use | Note | |----------|-----|------| -| Interactive elements + `ref` handles | `BrowserSnapshot({ mode: 'semantic' })` | Buttons / inputs / links only — **no article text** | -| Article text on a small/simple page | `BrowserSnapshot({ mode: 'accessibility' })` | Returns StaticText nodes; fails with `BROWSER_RESULT_TOO_LARGE` on real content pages (2 MB result cap, and the `depth` argument is currently ignored by the host) | -| Article text on a real page | L2 Jina Reader (public) or L3-fallback Playwright (login-gated) | The built-in browser has no working bulk text-extraction channel yet | -| What the page looks like | `tab.activate` → `BrowserAct({ action: 'page.screenshot' })` | Full-page only; read it visually | +| Interactive elements + `ref` / `loc=` handles | `BrowserSnapshot({ mode: 'semantic' })` | Buttons/inputs/links + per-line `[ref=eN]` and (when producible) `[loc=...]` stable selectors; cross-origin iframes in the same tree | +| Article body text | `BrowserSnapshot({ mode: 'text' })` or `BrowserAct({ action: 'page.extract-text' })` | markdown/text formats; beyond maxBytes it truncates and hands back a nextCursor for paging — no error | +| Accessibility tree | `BrowserSnapshot({ mode: 'accessibility' })` | Respects `depth` (default 50, max 100) and the maxBytes budget — truncates + pages instead of failing wholesale | +| What the page looks like | `BrowserSnapshot({ mode: 'visual' })` or `BrowserAct({ action: 'page.screenshot' })` | Pixels land directly in the result (vision models read them in place); `clip={x,y,width,height,scale}` for element-level crops (scale up to 4) and `captureBeyondViewport` for full-page capture | -`page.evaluate` is **not** an extraction channel: it needs human approval per call and its string/object return values come back as `[REDACTED:browser-runtime-value]`. +### Command surface at a glance + +`BrowserAct` actions grouped by purpose (full enum in the tool schema): + +- **tab.***: `navigate` / `back` / `forward` / `reload` / `activate` / `close` +- **input.***: `move` / `click` / `double-click` / `drag` / `wheel` / `touch` / `pinch` / `key` / `text` — humanized input (#1808: consistent UA/UA-CH identity + real trajectories); the preferred interaction channel on anti-bot sites +- **page.element** (discriminated op × selector, nine ops): write ops `fill` / `select-option` / `check` / `uncheck` / `scroll-into-view`; read ops `get-attribute` / `bounding-box` / `count` / `all-inner-texts`. Selectors speak the `loc=` dialect or a snapshot `ref` (with snapshotId). `fill` refuses `input[type=password]` +- **page.wait** (discriminated until, nine values): poll-type `load` / `domcontentloaded` / `networkidle` / `selector` / `url` / `timeout` return `waited:false` on timeout; event-type `request` / `response` / `download` throw on timeout. Default 10s, max 60s +- **Inline wait blocks**: `params.wait` (isomorphic to page.wait params) on `tab.navigate` / `input.click` / `input.key` / `page.element{op:"fill"}` — one receipt completes "act→wait for result", the waiter registers before the action, no cross-IPC race +- **page.evaluate**: `{ expression, awaitPromise }`, return values cross as-is (over-budget results truncate with a `truncated` flag, never throw); capability `browser.page.evaluate` sits behind the human gate (allow-all mode skips the card) +- **page.extract-text / page.screenshot / page.wait**: see the table above and the fetch.browser recipe + +`loc=` selector dialect (S7/S9): `e` (must carry the snapshotId of the snapshot that issued the ref), `loc=css:` / `loc=role:` / `loc=text:` / `loc=testid:`, bare CSS, composable with `internal:nth/last/scope/filter`. Unknown prefixes fail explicitly — never silently degrade to CSS. + +### Choosing the interaction channel: input.* vs page.element + +- **On anti-bot sites (Xiaohongshu/Weibo/Bilibili etc.) always prefer `input.*`**: it rides the #1808 humanized input pipeline (coordinate dispatch, humanized trajectories, auditable visualization) plus the identity layer that keeps UA/UA-CH free of Electron/Headless tells +- **`page.element` write ops fit bulk form filling on sites that don't detect automation**: one call fills/selects/checks, far faster than per-element input.click + input.text +- **Red line**: `page.element` deliberately has no click — pointer actions must go through `input.*`; invoking `el.click()` via JS bypasses the entire humanization investment and is an explicitly forbidden fallback + +### The fetch.browser recipe: pull API data with login state + +The right way to pull a site's own API (lists, comments, orders, any JSON) in a logged-in context: run `fetch` **in the page context** — it carries that origin's cookies automatically, stays same-origin, is bounded by Grant origins, and rides the existing `page.evaluate` gate. It is a wrapper usage of `BrowserAct({ action: 'page.evaluate' })`: + +```yaml +BrowserAct: + action: page.evaluate + params: + expression: | + fetch('/api/v1/comments?page=1&size=20', { + headers: { accept: 'application/json' } + }).then(r => r.text()) + awaitPromise: true # default true; waits for the returned Promise to settle +``` + +Notes: +- `tab.navigate` to any page on the site first (establishes the origin and cookies), then fire the fetch; use a relative path so it is same-origin by construction +- Return values cross as-is; for large JSON take `.text()` and slice it yourself, or page through multiple calls +- Only the current tab's origin is reachable (Grant origins constraint); for another site's API, navigate there first +- `page.evaluate` is a human-gated capability: outside allow-all mode an approval card appears — explain the purpose to the user ### Recommended flow (Xiaohongshu example) ``` 1. BrowserManage({ action: 'create_space', name: 'xhs-note', persistence: 'ephemeral' }) 2. BrowserManage({ action: 'start_session', spaceId, capabilities: [...] }) - ← an explicit list *narrows* the lease; include browser.input.pointer.wheel if you will scroll -3. Reuse the login: L3-fallback Playwright against the user's own Chrome is the default path. - BrowserImport({ action: 'discover' → 'create_plan' → 'dry_run' → 'apply' }) only works if the - Host granted browser.import.* separately — create_space alone does not. If it is denied, fall back. + ← an explicit list *narrows* the lease; include browser.input.pointer.wheel to scroll + and browser.observe.snapshot to extract text +3. Reuse the login: only when the Host granted browser.import.*, + BrowserImport({ action: 'discover' → 'create_plan' → 'dry_run' → 'apply' }). + Without that grant the user's cookies cannot be reused — say so, then continue + as logged-out or abort. 4. BrowserAct({ action: 'tab.navigate', params: { url: 'https://www.xiaohongshu.com/explore/abc123' } }) -5. BrowserSnapshot({ mode: 'semantic' }) ← element refs for interaction (no body text) - tab.activate + page.screenshot ← confirm the note actually rendered - → extract the note text via L3-fallback Playwright -6. SitePatternRead({ domain: 'xiaohongshu.com' }) ← read accumulated experience -7. At task end → BrowserManage({ action: 'close_session', sessionId }) -8. If you find a new pitfall → SitePatternWrite({ domain, scope: 'agent', mode: 'merge', content }) +5. BrowserSnapshot({ mode: 'semantic' }) ← interaction refs (cross-origin iframes in the same tree) +6. BrowserAct({ action: 'page.extract-text', params: { format: 'markdown' } }) + ← body text read out directly; if too long, pass back the returned nextCursor to continue +7. Need to confirm rendering → BrowserSnapshot({ mode: 'visual' }) (pixels readable in place) +8. SitePatternRead({ domain: 'xiaohongshu.com' }) ← read accumulated experience +9. At task end → BrowserManage({ action: 'close_session', sessionId }) +10. If you find a new pitfall → SitePatternWrite({ domain, scope: 'agent', mode: 'merge', content }) ``` ---- - -## Site Experience Accumulation (v2.0) +## Site Experience Accumulation When the task ends and you've discovered new anti-bot pitfalls, effective selectors, or platform quirks, call: @@ -411,7 +343,7 @@ SitePatternWrite({ domain: "xiaohongshu.com", scope: "agent", // agent=shared (Git-tracked, can be published); user=private mode: "merge", // merge appends; replace overwrites - content: "## Known pitfalls\n- 2026-05: ...", + content: "## Known pitfalls\n- 2026-08: ...", confidence: "medium" }) ``` @@ -427,8 +359,6 @@ SitePatternRead({ domain: "xiaohongshu.com" }) Content containing cookies / tokens / phone numbers / emails will **automatically downgrade scope='user'** and notify you. ---- - ## Common Workflows Read [references/workflows.md](references/workflows.md) for detailed templates: @@ -437,11 +367,9 @@ Read [references/workflows.md](references/workflows.md) for detailed templates: - News aggregation & timelines - API/library version investigation -Read [references/cdp-browser.md](references/cdp-browser.md) for login-gated site recipes (Xiaohongshu / Bilibili / Weibo / Zhihu / Feishu). - Read [references/jina-reader.md](references/jina-reader.md) for Jina Reader positioning, rate limits, and advanced endpoints. ---- +Read [references/browser-tools.md](references/browser-tools.md) for the full built-in browser command surface, capability tiers, and known boundaries. ## Quick Workflow: Multi-Source Research @@ -455,27 +383,22 @@ Read [references/jina-reader.md](references/jina-reader.md) for Jina Reader posi 7. Report with inline [source](url) citations + a Sources list at the end ``` ---- - ## Anti-Patterns (Avoid) - ❌ **Using WebFetch on obviously heavy sites** — Medium, Twitter, Xiaohongshu will waste tokens or fail. Jump straight to L2/L3. -- ❌ **Launching headless Chrome instead of CDP attach** — loses user's login state, triggers anti-bot, slow cold start. Always use `connect_over_cdp()` to attach to the user's existing session. - ❌ **Fetching one URL at a time when you need 5** — batch in a single message. - ❌ **Trusting a single source** — cross-check ≥ 2 sources for non-trivial claims. - ❌ **Fetching the search result page itself** — WebSearch already returns snippets; fetch the actual articles. - ❌ **Ignoring the cache** — WebFetch caches 15 min, reuse freely. -- ❌ **Scraping when an API exists** — GitHub, npm, PyPI, Wikipedia all have JSON APIs. +- ❌ **Scraping when an API exists** — GitHub, npm, PyPI, Wikipedia all have JSON APIs; a logged-in site's own API goes through the fetch.browser recipe. - ❌ **Forgetting the year in time-sensitive queries** — "best AI models" returns 2023 results; "best AI models 2026" returns current. -- ❌ **Hardcoding login credentials in scripts** — always rely on the user's pre-logged CDP session. +- ❌ **Hardcoding login credentials in scripts** — login state can only come from Cookies imported via BrowserImport. - ❌ **Citing only after the fact** — collect URLs as you fetch, not from memory afterwards. -- ❌ **(v2.1) Writing Python heredoc when the built-in browser would do** — slow, requires Python+Playwright install, and bloats context. Prefer L3-fast; fall back to Python only when the built-in browser is insufficient (race / long-wait / custom scripts). -- ❌ **(v2.1) Calling `page.screenshot` before `tab.activate`** — tabs are parked off-screen at `(-10000,-10000,1x1)` and have no compositing surface, so the capture stalls until the 30s deadline **and the tab host is destroyed**; every later call then fails with `BROWSER_TAB_HOST_NOT_FOUND`. Always `tab.activate` with real bounds first. -- ❌ **(v2.1) Reaching for `page.evaluate` to read a page** — it needs human approval on every call, its string/object return values come back as `[REDACTED:browser-runtime-value]` (only number/boolean/null survive), and anti-debug sites stall it for tens of seconds. Read pages with `BrowserSnapshot` instead. -- ❌ **(v2.1) Discovering new pitfalls and not writing a site-pattern** — next time the same Agent runs the task, it'll repeat the same mistakes. Anything that took 2+ steps to figure out is worth `SitePatternWrite(scope='agent', mode='merge')`. -- ❌ **(v2.1) Writing cookies / phone numbers to scope='agent'** — that layer is Git-tracked and may be published to the marketplace. SitePatternWrite auto-downgrades, but don't deliberately write secrets to the agent layer. - ---- +- ❌ **(v3.0) Using page.element for bulk interaction on anti-bot sites** — it is a direct JS call that bypasses humanized trajectories; on anti-bot sites always use `input.*`; keep `page.element` for bulk form filling where the site doesn't detect automation. +- ❌ **(v3.0) Reading body text from screenshots** — `page.extract-text` / `BrowserSnapshot(text)` hand you markdown/text with paged budgets; save screenshots for layout confirmation and CAPTCHAs where you truly must look. +- ❌ **(v3.0) Tolerating per-action round trips when BrowserScript would do** — navigate→snapshot→click→wait→extract runs as one script; remember BrowserScript's trust level equals Bash and the script source passes one human approval. +- ❌ **(v3.0) Discovering new pitfalls and not writing a site-pattern** — next time the same Agent runs the task, it'll repeat the same mistakes. Anything that took 2+ steps to figure out is worth `SitePatternWrite(scope='agent', mode='merge')`. +- ❌ **(v3.0) Writing cookies / phone numbers to scope='agent'** — that layer is Git-tracked and may be published to the marketplace. SitePatternWrite auto-downgrades, but don't deliberately write secrets to the agent layer. ## Example Interaction @@ -484,26 +407,15 @@ Read [references/jina-reader.md](references/jina-reader.md) for Jina Reader posi **Agent workflow**: ``` 1. Recognize → Xiaohongshu is an L3 logged-in site -2. Check CDP: curl -s http://localhost:9222/json/version - ├─ Failure → prompt the user to launch Chrome in debug mode, abort - └─ Success → continue -3. Bash: python3 connect_over_cdp script → page.goto(url) → page.content() -4. BeautifulSoup extract h1 title, .note-content, .comments +2. BrowserManage(create_space + start_session) + Need login state → if browser.import.* was granted, run the BrowserImport four steps; + otherwise tell the user the login cannot be reused and continue with the public part +3. BrowserAct(tab.navigate → the note URL) +4. BrowserAct(page.extract-text, format: markdown) + ← body text read out directly; page with nextCursor if over budget 5. When returning to the user: - Cite the original URL - - If content is long, run it through Jina to save tokens -6. Tell the user: "Fetched via your logged-in session, original link: [xhs](url)" + - Quote facts from the extracted text with source links +6. Tell the user: "Fetched via the built-in browser, original link: [xhs](url)" +7. BrowserManage(close_session) ``` - ---- - -## Installation Note - -CDP features require Python + Playwright installed: - -```bash -pip3 install playwright beautifulsoup4 -python3 -m playwright install chromium # only needed if user hasn't installed Chrome -``` - -If `playwright` is not installed when the user requests a login-gated site, run the install commands in Bash and explain you're setting up the browser automation dependency. diff --git a/skills/web-access/SKILL.zh-CN.md b/skills/web-access/SKILL.zh-CN.md index 51b7d2a..74a9588 100644 --- a/skills/web-access/SKILL.zh-CN.md +++ b/skills/web-access/SKILL.zh-CN.md @@ -4,28 +4,30 @@ ## L0：一句话摘要 -四层联网访问工具包——搜索公开页面、Jina 优化抓取、内置受管浏览器登录态访问（v2.1）、Python Playwright CDP 兜底。 +三层联网访问工具包——搜索公开页面、Jina 优化抓取、内置受管浏览器完成登录态访问与交互（v3.0 起取文/等待/脚本全部内置，不再需要 Python Playwright 兜底）。 ## L1：概述与使用场景 ### 能力描述 -web-access 是一个**流程型技能（Procedural Skill）**，提供四层互补的联网访问能力： +web-access 是一个**流程型技能（Procedural Skill）**，提供三层互补的联网访问能力： - **L1**（WebSearch + WebFetch）：公开页面，轻量 - **L2**（Jina Reader）：JS 渲染的重页面，默认节省 Token -- **L3-fast**（内置受管浏览器，**v2.1 重写**）：到达并*操作*登录态/交互站点——每个任务独立 BrowserSpace 隔离、零 Python 依赖、每次动作都有可审计回执。**抽取长正文仍归 L2 / L3-fallback**，见下方取文说明 -- **L3-fallback**（Chrome CDP + Python Playwright）：复杂自动化场景兜底（长等待、特殊 race condition 等） +- **L3**（内置受管浏览器，v3.0 能力面补全）：到达、操作并**读取**登录态/交互站点——每个任务独立 BrowserSpace 隔离、零 Python 依赖、每次动作都有可审计回执。批量取文（`page.extract-text`）、判别式等待（`page.wait`）、代码模式（`BrowserScript`）都在本层内闭环 -### v2.1 重写：内置受管浏览器（默认隐藏，激活后才暴露） +v2.x 时代的第四层「用户手工启动调试 Chrome + Python Playwright CDP」已在 v3.0 移除：它依赖的每一条理由（无批量取文通道、evaluate 不可用、截图必须串行 activate）都被内置浏览器的新能力覆盖，见下方速查。 -调用 `Skill('web-access')` 加载本技能时，以下 8 个工具被注入到当前会话，让 LLM 直接驱动内置浏览器： +### v3.0：内置受管浏览器（默认隐藏，激活后才暴露） + +调用 `Skill('web-access')` 加载本技能时，以下 9 个工具被注入到当前会话，让 LLM 直接驱动内置浏览器： | 工具 | 用途 | |------|------| | BrowserManage | 建/销隔离 BrowserSpace、启动会话、管理标签页 | -| BrowserSnapshot | `semantic` / `accessibility` / `visual` 快照——读页面的主通道 | -| BrowserAct | 一次调用一个受管动作：导航、点击、输入、滚动、截图…… | +| BrowserSnapshot | `semantic` / `text` / `accessibility` / `visual` 四种快照——读页面的主通道 | +| BrowserAct | 一次调用一个受管动作：导航、输入、取文、等待、元素操作、截图…… | +| BrowserScript | **代码模式**：一段异步 JS 连续下发浏览器命令，消除逐动作往返（信任级别等同 Bash） | | BrowserImport | 从用户 Chrome/Edge/Firefox/Safari 配置导入 Cookie（需人工审批；**还需 Host 授予 `browser.import.*`，普通 `create_space` 会话拿不到**） | | BrowserShare | 把 Space/Session 委派给其他 Agent（隔离 / 快照 / 写时复制 / 实时共享） | | SitePatternRead / SitePatternWrite | 按域名累积"站点经验"（AgentFS 三层） | @@ -33,20 +35,22 @@ web-access 是一个**流程型技能（Procedural Skill）**，提供四层互 > **重要**：未调用 Skill('web-access') 之前，这些工具**不会**出现在 LLM 的 tools 列表里——默认对话不消耗其 token。详见 [references/browser-tools.md](references/browser-tools.md)。 > -> **v2.1 已移除**：`BrowserListTabs` / `BrowserNavigate` / `BrowserEval` / `BrowserClick` / `BrowserScreenshot` / `BrowserScroll` / `BrowserSetFiles` / `BrowserCloseTab` 及其背后的 cdp-proxy 已停用，调用会返回「该旧 BrowserXxx/cdp-proxy 入口已停用」。本版要求客户端 v10.0.98+。 +> **v2.1 已移除**：`BrowserListTabs` / `BrowserNavigate` / `BrowserEval` / `BrowserClick` / `BrowserScreenshot` / `BrowserScroll` / `BrowserSetFiles` / `BrowserCloseTab` 及其背后的 cdp-proxy 已停用，调用会返回「该旧 BrowserXxx/cdp-proxy 入口已停用」。本版要求客户端 v10.0.98+；`page.extract-text` / `page.element` / `page.wait` / 内联 wait 块 / 跨源 iframe 快照需 v10.0.112+，`BrowserScript` 需包含 S17/S18 的更新版本。 ### 使用场景 - 用户需要搜索当前信息或研究特定主题 - 用户需要抓取公开网页内容或技术文档 -- 用户需要访问登录态站点（小红书、B站、微博、飞书、Twitter 等） +- 用户需要访问登录态站点（小红书、B站、微博、飞书、Twitter 等）并**读出正文** +- 用户需要在登录态下调站内接口取数（列表、评论、订单等） - 用户需要对比产品、聚合新闻或调查 API/库版本 ### 核心价值 -- **四层递进**：从轻量搜索到重度 JS 渲染到登录态访问，按需选择 -- **Token 优化**：Jina Reader 默认减少 50-80% Token 消耗 -- **登录态复用**：兜底层 CDP 连用户 Chrome，或在 Host 已授予 `browser.import.*` 时用 BrowserImport 把 Cookie 导入隔离 Space；两条路都不必重新登录 +- **三层递进**：从轻量搜索到重度 JS 渲染到登录态访问，按需选择 +- **Token 优化**：Jina Reader 默认减少 50-80% Token 消耗；`page.extract-text` 的 maxBytes/cursor 分页让登录态长文也可控 +- **登录态复用**：Host 授予 `browser.import.*` 时用 BrowserImport 把 Cookie 导入隔离 Space，不必重新登录 +- **零外部依赖**：不再要求 Python/Playwright 安装，也不再要求用户手工启动调试 Chrome ## L2：详细规范 @@ -58,53 +62,6 @@ When you complete a research task, you **MUST** cite all source URLs in your res If any fetch fails, explicitly tell the user which URL failed and which fallback you used. ---- - -## Prerequisites: Chrome CDP Setup (for login-gated sites) - -**L3-fallback 层（Python Playwright）必需**，实际上也仍是复用登录态最稳的一条路。L3-fast 内置浏览器只有在 Host 已授予 `browser.import.*`、`BrowserImport` 真能用的前提下才可以跳过这一步；否则照样要配。 - -### One-time setup - -Launch a dedicated Chrome instance with remote debugging enabled: - -**macOS**: -```bash -/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \ - --remote-debugging-port=9222 \ - --user-data-dir="${DESIRECORE_ROOT}/chrome-profile" -``` - -**Linux**: -```bash -google-chrome \ - --remote-debugging-port=9222 \ - --user-data-dir="${DESIRECORE_ROOT}/chrome-profile" -``` - -**Windows (PowerShell)**: -```powershell -& "C:\Program Files\Google\Chrome\Application\chrome.exe" ` - --remote-debugging-port=9222 ` - --user-data-dir="$env:USERPROFILE\.desirecore\chrome-profile" -``` - -After launch: -1. Manually log in to the sites you need (小红书、B站、微博、飞书 …) -2. Leave this Chrome window open in the background -3. Verify the debug endpoint: `curl -s http://localhost:9222/json/version` should return JSON - -### Verify CDP is ready - -Before any CDP operation, always run: -```bash -curl -s http://localhost:9222/json/version | python3 -c "import sys,json; d=json.load(sys.stdin); print('CDP ready:', d.get('Browser'))" -``` - -If the command fails, tell the user: "请先启动 Chrome 并开启远程调试端口（见 web-access 技能的 Prerequisites 部分）。" - ---- - ## Tool Selection Decision Tree ``` @@ -121,12 +78,12 @@ User intent │ (Jina Reader = default for JS-rendered content, saves tokens) │ ├─ "Read this login-gated page" (小红书/B站/微博/飞书/Twitter/知乎/公众号) - │ ├─→ 到达：BrowserManage(create_space/start_session) → BrowserAct(tab.navigate) - │ │ → tab.activate + page.screenshot 确认落到正文页 - │ │ （semantic 快照不含正文，不能用来读内容） - │ │ 要登录态：仅在已授予 browser.import.* 时用 BrowserImport，否则走下面的 CDP - │ └─→ 取正文：确认 CDP 就绪后 python3 playwright.connect_over_cdp() - │ → page.content() → Jina Reader / BeautifulSoup + │ └─→ BrowserManage(create_space/start_session) → BrowserAct(tab.navigate) + │ → BrowserAct(page.extract-text) ← 正文直接读出，支持 maxBytes/cursor 分页 + │ 要登录态：仅在已授予 browser.import.* 时用 BrowserImport + │ + ├─ "Pull data from the site's API in a logged-in context" + │ └─→ fetch.browser 配方：BrowserAct(page.evaluate) 里跑 fetch（带该 origin 的 Cookie） │ ├─ "API documentation / GitHub / npm package info" │ └─→ Prefer official API endpoints over scraping HTML: @@ -135,24 +92,19 @@ User intent │ - PyPI: curl https://pypi.org/pypi//json │ └─ "Real-time interactive task" (click, fill form, scroll, screenshot) - ├─→ **Default: 内置受管浏览器** (BrowserManage → BrowserAct → BrowserSnapshot — - │ see references/browser-tools.md, no Python needed) - └─→ Fallback: CDP + Python Playwright (references/cdp-browser.md) when 内置浏览器 is insufficient - (e.g., complex race conditions, multi-event waits, long-running in-browser scripts) + └─→ 内置受管浏览器 (BrowserManage → BrowserAct → BrowserSnapshot — + see references/browser-tools.md, no Python needed) ``` -### 四层策略总结 +### 三层策略总结 | Layer | Use case | Primary tool | Token cost | |-------|----------|--------------|------------| | L1 | Public, static | `WebFetch` | Low | | L2 | JS-heavy, long articles, token savings | `Bash curl r.jina.ai` | **Lowest** (Markdown pre-cleaned) | -| **L3-fast** | **登录态导航与交互 (PRIMARY)** | **内置受管浏览器（BrowserManage / BrowserAct / BrowserSnapshot）** | Medium | -| L3-fallback | 复杂自动化（race / long-wait / 自定义脚本） | `Bash + Python Playwright CDP` | Medium | +| **L3** | **登录态导航、交互与取文 (PRIMARY)** | **内置受管浏览器（BrowserManage / BrowserAct / BrowserSnapshot / BrowserScript）** | Medium | -**Default priority**: L1 for simple public pages → L2 for heavy → **L3-fast for login-gated** → L3-fallback only when 内置浏览器不够用。 - ---- +**Default priority**: L1 for simple public pages → L2 for heavy → **L3 for login-gated（含正文与站内接口取数）**。 ## Supported Sites Matrix @@ -162,16 +114,14 @@ User intent | GitHub README, issues, PRs | `gh api` (best) → L1 WebFetch | Prefer API | | Hacker News, Reddit | L1 WebFetch | Public content | | Medium, Dev.to | L2 Jina Reader | JS-rendered, member gates | -| Twitter/X | L3 CDP (or L2 Jina with `x.com`) | Login required for full thread | -| 小红书 (xiaohongshu.com) | L3 CDP | 强制登录 | -| B站 (bilibili.com) | L3 CDP | 视频描述/评论需登录 | -| 微博 (weibo.com) | L3 CDP | 长微博需登录 | -| 知乎 (zhihu.com) | L3 CDP | 长文+评论需登录 | -| 飞书文档 (feishu.cn) | L3 CDP | 必须登录 | +| Twitter/X | L3（或 L2 Jina with `x.com`） | Login required for full thread | +| 小红书 (xiaohongshu.com) | L3 内置浏览器 + BrowserImport | 强制登录；正文走 page.extract-text | +| B站 (bilibili.com) | L3 内置浏览器 + BrowserImport | 视频描述/评论需登录 | +| 微博 (weibo.com) | L3 内置浏览器 + BrowserImport | 长微博需登录 | +| 知乎 (zhihu.com) | L3 内置浏览器 + BrowserImport | 长文+评论需登录 | +| 飞书文档 (feishu.cn) | L3 内置浏览器 + BrowserImport | 必须登录 | | 公众号 (mp.weixin.qq.com) | L2 Jina Reader | 通常公开，Jina 处理更干净 | -| LinkedIn | L3 CDP | 登录墙 | - ---- +| LinkedIn | L3 内置浏览器 + BrowserImport | 登录墙 | ## Tool Reference @@ -194,11 +144,11 @@ WebFetch(url="https://example.com/article") Tips: - Results cached for 15 min - Returns cleaned Markdown with title + URL + body -- If body < 200 chars or looks garbled → escalate to Layer 2 (Jina) or Layer 3 (CDP) +- If body < 200 chars or looks garbled → escalate to Layer 2 (Jina) or Layer 3 (built-in browser) -### Layer 2: Jina Reader (default for heavy pages) +### Layer 2: Jina Reader（重页默认） -Jina Reader (`r.jina.ai`) is a free public proxy that renders pages server-side and returns clean Markdown. Use it as the **default** for any page where WebFetch produces garbled or truncated output, and as the **preferred** extractor for JS-heavy SPAs. +Jina Reader (`r.jina.ai`) 免费公共代理，服务端渲染页面并返回干净 Markdown。WebFetch 输出乱码/截断时的默认升级路径，JS 重页面的首选抓取器。 ```bash curl -sL "https://r.jina.ai/https://example.com/article" @@ -212,111 +162,93 @@ Why Jina is the default token-saver: See [references/jina-reader.md](references/jina-reader.md) for advanced endpoints and rate limits. -### Layer 3: CDP Browser (login-gated access) +### Layer 3: 内置受管浏览器（登录态与交互） -Use Python Playwright's `connect_over_cdp()` to attach to the user's running Chrome (which already has login cookies). **No re-login needed.** +完整命令面、能力档位与边界条件见 [references/browser-tools.md](references/browser-tools.md)。这里的循环是： -**Minimal template**: -```bash -python3 << 'PY' -from playwright.sync_api import sync_playwright +1. `BrowserManage(create_space)` → `BrowserManage(start_session)` 拿 sessionId + 首个 tab +2. 需要登录态 → `BrowserImport`（仅当 Host 已授予 `browser.import.*`；没有这层授权就无法复用用户 Cookie，告诉用户并按无登录态继续或放弃） +3. `BrowserAct(tab.navigate)` 到达页面 +4. `BrowserSnapshot(semantic)` 拿可交互元素 `ref`（跨源 iframe 的元素也在同一棵树里，ref 全局连续编号） +5. 交互用 `input.*`（拟真轨迹）或 `page.element`（表单批量写）；等结果用 `page.wait` 或内联 wait 块 +6. 取正文用 `BrowserSnapshot(text)` 或 `BrowserAct(page.extract-text)`；取接口数据用 fetch.browser 配方 +7. 任务收尾 `BrowserManage(close_session)` -TARGET_URL = "https://www.xiaohongshu.com/explore/..." +多动作连续编排（导航→快照→点击→等待→取文）可用 `BrowserScript` 一段脚本完成，省去逐动作 IPC 往返。 -with sync_playwright() as p: - browser = p.chromium.connect_over_cdp("http://localhost:9222") - context = browser.contexts[0] # reuse user's default context (has cookies) - page = context.new_page() - page.goto(TARGET_URL, wait_until="domcontentloaded") - page.wait_for_timeout(2000) # let lazy content load - html = page.content() - page.close() +## L3 速查（v3.0） -# Print first 500 chars to verify -print(html[:500]) -PY -``` - -**Extract text via BeautifulSoup** (no Jina round-trip): -```bash -python3 << 'PY' -from playwright.sync_api import sync_playwright -from bs4 import BeautifulSoup - -with sync_playwright() as p: - browser = p.chromium.connect_over_cdp("http://localhost:9222") - page = browser.contexts[0].new_page() - page.goto("https://www.bilibili.com/video/BV...", wait_until="networkidle") - html = page.content() - page.close() - -soup = BeautifulSoup(html, "html.parser") -title = soup.select_one("h1.video-title") -desc = soup.select_one(".video-desc") -print("Title:", title.get_text(strip=True) if title else "N/A") -print("Desc:", desc.get_text(strip=True) if desc else "N/A") -PY -``` - -See [references/cdp-browser.md](references/cdp-browser.md) for: -- Per-site selectors (小红书/B站/微博/知乎/飞书) -- Scrolling & lazy-load patterns -- Screenshot & form-fill recipes -- Troubleshooting connection issues - ---- - -## L3-fast: 内置受管浏览器速查（v2.1） - -**只在你调用 `Skill('web-access')` 加载本技能后，下面这组工具才会出现在 tools[] 里。** - -| 工具 | 一行示例 | -|------|---------| -| `BrowserManage({ action: 'create_space', name, persistence: 'ephemeral' })` | 每个任务一个隔离 Space | -| `BrowserManage({ action: 'start_session', spaceId, capabilities })` | 启动会话，返回 sessionId 与首个标签页 | -| `BrowserManage({ action: 'list_tabs' \| 'create_tab' \| 'close_session' })` | 标签页与生命周期管理 | -| `BrowserAct({ action: 'tab.navigate', params: { url } })` | 当前标签页导航 | -| `BrowserSnapshot({ mode: 'semantic' })` | 读页面：可交互元素 + `ref` 句柄 | -| `BrowserAct({ action: 'input.click', params: { ref } })` | 按快照 `ref` 点击，不要用裸 x/y | -| `BrowserAct({ action: 'input.text', params: { text } })` | 向聚焦元素输入文本 | -| `BrowserAct({ action: 'input.wheel', params: { x: 640, y: 400, deltaX: 0, deltaY: 720 } })` | 滚动触发懒加载 —— `x`/`y`（或 `ref`）**必填**，且会话要带 `browser.input.pointer.wheel` | -| `BrowserAct({ action: 'tab.activate', params: { bounds } })` → `page.screenshot` | **先 activate 再截图** | -| `BrowserImport({ action: 'discover' \| 'create_plan' \| 'dry_run' \| 'apply' \| 'rollback' \| 'list_plans' })` | 复用用户登录态 Cookie —— 这 6 个就是**全部**动作，主流程走 `discover → create_plan → dry_run → apply`。**需要 `browser.import.*`，而 `create_space` 不会授予**（见下） | - -完整 API 与边界条件见 [references/browser-tools.md](references/browser-tools.md)。 - -**怎么读页面** —— 按需求选通道： +### 读页面：按需求选通道 | 你要什么 | 用什么 | 说明 | |----------|--------|------| -| 可交互元素 + `ref` 句柄 | `BrowserSnapshot({ mode: 'semantic' })` | 只有按钮/输入框/链接，**不含正文文本** | -| 小页面的正文 | `BrowserSnapshot({ mode: 'accessibility' })` | 返回 StaticText 节点；真实内容页会报 `BROWSER_RESULT_TOO_LARGE`（结果上限 2 MB，且 `depth` 参数当前被宿主忽略） | -| 真实页面的正文 | L2 Jina Reader（公开页）或 L3-fallback Playwright（登录态） | 内置浏览器目前没有可用的批量取文通道 | -| 页面长什么样 | `tab.activate` → `BrowserAct({ action: 'page.screenshot' })` | 只有整页截图，靠看图读 | +| 可交互元素 + `ref` / `loc=` 句柄 | `BrowserSnapshot({ mode: 'semantic' })` | 按钮/输入框/链接 + 每行 `[ref=eN]` 与（能产出时）`[loc=...]` 稳定选择器；跨源 iframe 元素同树 | +| 页面正文 | `BrowserSnapshot({ mode: 'text' })` 或 `BrowserAct({ action: 'page.extract-text' })` | markdown/text 两种格式；超 maxBytes 截断并给 nextCursor 续读（cursor 分页），不报错 | +| 辅助功能树 | `BrowserSnapshot({ mode: 'accessibility' })` | 尊重 `depth`（默认 50 上限 100）与 maxBytes 预算，超出截断+翻页，不再整体报错 | +| 页面长什么样 | `BrowserSnapshot({ mode: 'visual' })` 或 `BrowserAct({ action: 'page.screenshot' })` | 像素直接进结果（视觉模型当场看）；支持 `clip={x,y,width,height,scale}` 元素级裁剪放大（scale 最大 4）与 `captureBeyondViewport` 整页截图 | -`page.evaluate` **不是**取文通道：每次调用需人工审批，且字符串/对象返回值会被替换成 `[REDACTED:browser-runtime-value]`。 +### 命令面速览 + +`BrowserAct` 的 `action` 按用途分组（完整枚举见工具 schema）： + +- **tab.***：`navigate` / `back` / `forward` / `reload` / `activate` / `close` +- **input.***：`move` / `click` / `double-click` / `drag` / `wheel` / `touch` / `pinch` / `key` / `text` —— 走输入拟真（#1808：UA/UA-CH 身份一致 + 真实轨迹），反检测站点的交互首选 +- **page.element**（判别式 op × selector，九 op）：写类 `fill` / `select-option` / `check` / `uncheck` / `scroll-into-view`；读类 `get-attribute` / `bounding-box` / `count` / `all-inner-texts`。selector 说 `loc=` 方言或快照 `ref`（配 snapshotId）。`fill` 拒绝 `input[type=password]` +- **page.wait**（判别式 until，九种）：轮询型 `load` / `domcontentloaded` / `networkidle` / `selector` / `url` / `timeout` 超时返回 `waited:false`；事件型 `request` / `response` / `download` 超时抛错。默认 10s、上限 60s +- **内联 wait 块**：`tab.navigate` / `input.click` / `input.key` / `page.element{op:"fill"}` 的 `params.wait`（形态与 page.wait 同构）——一条回执完成「动作→等结果」，等待器先于动作注册，无跨 IPC 竞态 +- **page.evaluate**：`{ expression, awaitPromise }`，返回值原样过界（超预算截断并标 `truncated`，不抛错）；能力档 `browser.page.evaluate` 走人工闸门（allow-all 模式免卡片） +- **page.extract-text / page.screenshot / page.wait**：见上表与 fetch.browser 配方 + +`loc=` 选择器方言（S7/S9）：`e<序号>`（须配签发该 ref 的快照 snapshotId）、`loc=css:` / `loc=role:` / `loc=text:` / `loc=testid:`、裸 CSS，可叠 `internal:nth/last/scope/filter`。未知前缀显式报错，绝不静默降级。 + +### 交互通道选用：input.* vs page.element + +- **反检测站点（小红书/微博/B站等）一律优先 `input.*`**：走 #1808 的输入行为拟真（坐标派发、拟真轨迹、可视化可审计），配套身份一致性层让 UA/UA-CH 不带 Electron/Headless 痕迹 +- **`page.element` 写类适用于表单批量填充等站点不检测的场景**：一次调用完成 fill/select-option/check，比逐元素 input.click+input.text 快得多 +- **红线**：`page.element` 刻意不含 click——指针动作必须走 `input.*`，用 JS 直调 `el.click()` 会绕开全部拟真投入，属于明确禁止的回退 + +### fetch.browser 配方：带登录态取接口数据 + +登录态下取站内接口（列表、评论、订单等 JSON）的正解：在**页面上下文**里跑 `fetch`——自动带该 origin 的 Cookie，同 origin、受 Grant origins 约束，走 `page.evaluate` 已有闸门。这是 `BrowserAct({ action: 'page.evaluate' })` 的封装用法： + +```yaml +BrowserAct: + action: page.evaluate + params: + expression: | + fetch('/api/v1/comments?page=1&size=20', { + headers: { accept: 'application/json' } + }).then(r => r.text()) + awaitPromise: true # 默认 true；表达式返回 Promise 时等它 settle +``` + +要点： +- 先 `tab.navigate` 到该站任意页面（建立 origin 与 Cookie），再发 fetch；路径写相对路径，天然同 origin +- 返回值原样过界；大 JSON 用 `.text()` 拿原文自己截取，或分页多次取 +- 只能访问当前 tab origin（Grant origins 约束）；跨站接口请先导航过去 +- `page.evaluate` 属人工闸门能力：非 allow-all 模式会弹审批卡，向用户说明用途即可 ### 推荐流程（小红书示例） ``` 1. BrowserManage({ action: 'create_space', name: 'xhs-note', persistence: 'ephemeral' }) 2. BrowserManage({ action: 'start_session', spaceId, capabilities: [...] }) - ← 显式列表是做减法；要滚动就必须把 browser.input.pointer.wheel 列进去 -3. 复用登录态：默认走 L3-fallback Playwright 连用户自己的 Chrome。 - BrowserImport({ action: 'discover' → 'create_plan' → 'dry_run' → 'apply' }) 只有在 Host 另行 - 授予 browser.import.* 时才走得通 —— 光靠 create_space 拿不到；被拒就直接回落。 + ← 显式列表是做减法；要滚动就把 browser.input.pointer.wheel 列进去， + 要取文就带 browser.observe.snapshot +3. 复用登录态：仅在 Host 已授予 browser.import.* 时走 + BrowserImport({ action: 'discover' → 'create_plan' → 'dry_run' → 'apply' })。 + 没有这层授权就无法复用用户 Cookie——如实告诉用户，按无登录态继续或放弃。 4. BrowserAct({ action: 'tab.navigate', params: { url: 'https://www.xiaohongshu.com/explore/abc123' } }) -5. BrowserSnapshot({ mode: 'semantic' }) ← 只拿交互用的元素 ref（不含正文） - tab.activate + page.screenshot ← 确认确实渲染出了笔记 - → 正文抽取走 L3-fallback Playwright -6. SitePatternRead({ domain: 'xiaohongshu.com' }) ← 读累积经验 -7. 任务结束 → BrowserManage({ action: 'close_session', sessionId }) -8. 如发现新陷阱 → SitePatternWrite({ domain, scope: 'agent', mode: 'merge', content }) +5. BrowserSnapshot({ mode: 'semantic' }) ← 拿交互元素 ref（跨源 iframe 同树） +6. BrowserAct({ action: 'page.extract-text', params: { format: 'markdown' } }) + ← 正文直接读出；太长就传上一页返回的 nextCursor 续读 +7. 需要确认渲染效果 → BrowserSnapshot({ mode: 'visual' })（像素直接可看） +8. SitePatternRead({ domain: 'xiaohongshu.com' }) ← 读累积经验 +9. 任务结束 → BrowserManage({ action: 'close_session', sessionId }) +10. 如发现新陷阱 → SitePatternWrite({ domain, scope: 'agent', mode: 'merge', content }) ``` ---- - -## 站点经验积累（v2.0 新增） +## 站点经验积累 任务结束如果发现新的反爬陷阱、有效选择器、平台特征，调用： @@ -325,7 +257,7 @@ SitePatternWrite({ domain: "xiaohongshu.com", scope: "agent", // agent=共享（受 Git 管理，发布给其他用户）；user=私有 mode: "merge", // merge 追加，replace 覆盖 - content: "## 已知陷阱\n- 2026-05: ...", + content: "## 已知陷阱\n- 2026-08: ...", confidence: "medium" }) ``` @@ -341,8 +273,6 @@ SitePatternRead({ domain: "xiaohongshu.com" }) 含 cookie / token / 手机号 / 邮箱时 SitePatternWrite **自动降级 scope='user'** 并提示。 ---- - ## Common Workflows Read [references/workflows.md](references/workflows.md) for detailed templates: @@ -351,11 +281,9 @@ Read [references/workflows.md](references/workflows.md) for detailed templates: - 新闻聚合与时间线 (News aggregation) - API/库版本调查 (Library version investigation) -Read [references/cdp-browser.md](references/cdp-browser.md) for login-gated site recipes (小红书/B站/微博/知乎/飞书). - Read [references/jina-reader.md](references/jina-reader.md) for Jina Reader positioning, rate limits, and advanced endpoints. ---- +Read [references/browser-tools.md](references/browser-tools.md) for the full built-in browser command surface, capability tiers, and known boundaries. ## Quick Workflow: Multi-Source Research @@ -369,27 +297,22 @@ Read [references/jina-reader.md](references/jina-reader.md) for Jina Reader posi 7. Report with inline [source](url) citations + a Sources list at the end ``` ---- - ## Anti-Patterns (Avoid) - ❌ **Using WebFetch on obviously heavy sites** — Medium, Twitter, 小红书 will waste tokens or fail. Jump straight to L2/L3. -- ❌ **Launching headless Chrome instead of CDP attach** — loses user's login state, triggers anti-bot, slow cold start. Always use `connect_over_cdp()` to attach to the user's existing session. - ❌ **Fetching one URL at a time when you need 5** — batch in a single message. - ❌ **Trusting a single source** — cross-check ≥ 2 sources for non-trivial claims. - ❌ **Fetching the search result page itself** — WebSearch already returns snippets; fetch the actual articles. - ❌ **Ignoring the cache** — WebFetch caches 15 min, reuse freely. -- ❌ **Scraping when an API exists** — GitHub, npm, PyPI, Wikipedia all have JSON APIs. +- ❌ **Scraping when an API exists** — GitHub, npm, PyPI, Wikipedia all have JSON APIs; 登录态站内接口走 fetch.browser 配方. - ❌ **Forgetting the year in time-sensitive queries** — "best AI models" returns 2023 results; "best AI models 2026" returns current. -- ❌ **Hardcoding login credentials in scripts** — always rely on the user's pre-logged CDP session. +- ❌ **Hardcoding login credentials in scripts** — 登录态只能来自 BrowserImport 导入的 Cookie. - ❌ **Citing only after the fact** — collect URLs as you fetch, not from memory afterwards. -- ❌ **(v2.1) 在能用内置浏览器时仍写 Python heredoc** — 慢、依赖 Python+Playwright 安装、上下文体积大。优先 L3-fast；只在内置浏览器不够（race / 长等待 / 自定义脚本）时才回退。 -- ❌ **(v2.1) 没 `tab.activate` 就直接 `page.screenshot`** — 标签页默认停在窗口外 `(-10000,-10000,1x1)`，没有合成表面，截图会卡满 30 秒 deadline **并把标签页宿主打掉**，之后所有调用都报 `BROWSER_TAB_HOST_NOT_FOUND`。务必先用真实 bounds `tab.activate`。 -- ❌ **(v2.1) 用 `page.evaluate` 读页面** — 每次调用都需人工审批，返回的字符串/对象会被治理策略替换为 `[REDACTED:browser-runtime-value]`（只有 number/boolean/null 能穿透），且反调试站点会把它挂起几十秒。读页面请用 `BrowserSnapshot`。 -- ❌ **(v2.1) 任务结束发现新陷阱却不写 site-pattern** — 下次同 Agent 再做相同任务会重复踩坑。任何"花了 2+ 步才搞清楚的细节"都值得 `SitePatternWrite(scope='agent', mode='merge')`。 -- ❌ **(v2.1) 把含 cookie / 手机号的内容写到 scope='agent'** — 这层会被 Git 提交、可能发布到市场。SitePatternWrite 会自动降级，但你不该故意往 agent 层写敏感信息。 - ---- +- ❌ **(v3.0) 在反检测站点用 page.element 批量交互** — 它是 JS 直调，绕开拟真轨迹；反检测站点一律 `input.*`，`page.element` 只用于不检测场景的表单批量填充。 +- ❌ **(v3.0) 用截图靠"看图读字"取正文** — `page.extract-text` / `BrowserSnapshot(text)` 直接给 markdown/text，带分页预算；截图留给版面确认与验证码这类必须看图的场合。 +- ❌ **(v3.0) 逐动作往返还能忍时不换 BrowserScript** — 导航→快照→点击→等待→取文五连用一段脚本完成；但记住 BrowserScript 信任级别等同 Bash，脚本源码要过一次人工审批。 +- ❌ **(v3.0) 任务结束发现新陷阱却不写 site-pattern** — 下次同 Agent 再做相同任务会重复踩坑。任何"花了 2+ 步才搞清楚的细节"都值得 `SitePatternWrite(scope='agent', mode='merge')`。 +- ❌ **(v3.0) 把含 cookie / 手机号的内容写到 scope='agent'** — 这层会被 Git 提交、可能发布到市场。SitePatternWrite 会自动降级，但你不该故意往 agent 层写敏感信息。 ## Example Interaction @@ -398,26 +321,15 @@ Read [references/jina-reader.md](references/jina-reader.md) for Jina Reader posi **Agent workflow**: ``` 1. 识别 → 小红书是 L3 登录态站点 -2. 检查 CDP：curl -s http://localhost:9222/json/version - ├─ 失败 → 提示用户启动 Chrome 调试模式，终止 - └─ 成功 → 继续 -3. Bash: python3 connect_over_cdp 脚本 → page.goto(url) → page.content() -4. BeautifulSoup 提取 h1 title、.note-content、.comments +2. BrowserManage(create_space + start_session) + 需要登录态 → 已授予 browser.import.* 就走 BrowserImport 四步； + 没授予 → 告诉用户无法复用登录态，按公开可见部分继续 +3. BrowserAct(tab.navigate → 笔记 URL) +4. BrowserAct(page.extract-text, format: markdown) + ← 正文直接读出；超预算就按 nextCursor 续读 5. 返回给用户时： - 引用原 URL - - 若内容很长，用 Jina 清洗一遍节省 token -6. 告知用户：「已通过你的登录态抓取，原链接：[xhs](url)」 + - 引用正文事实，标注来源链接 +6. 告知用户：「已通过内置浏览器抓取，原链接：[xhs](url)」 +7. BrowserManage(close_session) ``` - ---- - -## Installation Note - -CDP features require Python + Playwright installed: - -```bash -pip3 install playwright beautifulsoup4 -python3 -m playwright install chromium # only needed if user hasn't installed Chrome -``` - -If `playwright` is not installed when the user requests a login-gated site, run the install commands in Bash and explain you're setting up the browser automation dependency. diff --git a/skills/web-access/references/browser-tools.md b/skills/web-access/references/browser-tools.md index c764744..9148e0b 100644 --- a/skills/web-access/references/browser-tools.md +++ b/skills/web-access/references/browser-tools.md @@ -1,26 +1,15 @@ -# 内置受管浏览器工具速查（L3-fast） +# 内置受管浏览器工具速查（L3） -> v2.1 起，本层从「cdp-proxy 驱动用户自己的 Chrome」改为「DesireCore 内置受管浏览器」。 +> v2.1 起本层从「cdp-proxy 驱动用户自己的 Chrome」改为「DesireCore 内置受管浏览器」；v3.0 补全能力面（批量取文 / 元素操作 / 等待 / 代码模式）并移除 Python Playwright 回落。 > 每个任务跑在独立 BrowserSpace 里（Cookie / Storage / 缓存互不串扰），每个动作都经过 > Capability → Grant → Lease → Origin → Host fencing 校验并留下可审计回执。 > -> 要求客户端 **v10.0.98+**。旧的 `BrowserListTabs` / `BrowserNavigate` / `BrowserEval` / -> `BrowserClick` / `BrowserScreenshot` / `BrowserScroll` / `BrowserSetFiles` / -> `BrowserCloseTab` 与其背后的 cdp-proxy 已停用，调用会直接返回 +> 要求客户端 **v10.0.98+**。`page.extract-text` / `page.element` / `page.wait` / 内联 wait 块 / +> 跨源 iframe 快照需 **v10.0.112+**；`BrowserScript` 需包含 S17/S18 的版本。旧的 +> `BrowserListTabs` / `BrowserNavigate` / `BrowserEval` / `BrowserClick` / `BrowserScreenshot` / +> `BrowserScroll` / `BrowserSetFiles` / `BrowserCloseTab` 与其背后的 cdp-proxy 已停用，调用会直接返回 > 「该旧 BrowserXxx/cdp-proxy 入口已停用；请改用 BrowserManage、BrowserSnapshot 或 BrowserAct」。 -## 何时用内置浏览器 vs Python Playwright - -| 场景 | 推荐 | -|------|------| -| 到达并操作登录态站点（小红书 / B站 / 微博 / 飞书 / 知乎） | **内置浏览器**（登录态优先走 L3-fallback CDP；仅在已授予 `browser.import.*` 时才用 BrowserImport 导入 Cookie） | -| 抽取登录态站点的长正文 | Python Playwright（内置浏览器没有批量取文通道，见下） | -| 简单点击 / 滚动 / 截图 | **内置浏览器** | -| 多个任务要互不串扰地并行 | **内置浏览器**（一任务一 Space） | -| 需要复杂等待逻辑（wait_for_selector + race condition） | Python Playwright（cdp-browser.md） | -| 需要在浏览器内运行长时间脚本（>30 s） | Python Playwright（内置浏览器单条命令 30 s deadline） | -| 需要对元素做放大裁剪截图 | Python Playwright（内置浏览器只有整页截图） | - ## 前置条件 1. 客户端 v10.0.98+，`~/.desirecore/config/browser.json` 里 `electron-embedded` 或 @@ -68,27 +57,52 @@ BrowserManage: ### BrowserSnapshot -读页面的**主通道**。返回可交互元素及其 `ref` 句柄、`rect`、`disabled` 状态。 +读页面的**主通道**。四种 mode： ```yaml BrowserSnapshot: - mode: semantic # semantic（默认）| accessibility | visual + mode: semantic # semantic（默认）| text | accessibility | visual sessionId: bss_xxx # 当前 Agent 有多个会话时用于消歧 - tabId: btab_xxx + tabId: btab_xxx # 非活动标签页时用于消歧 + options: {...} # 各 mode 专属参数，见下 ``` -提示： -- `semantic` 只列 button / input / a 等可交互元素，**不含图片和正文文本节点** -- 元素 `name` 往往是 placeholder（如「请输入」），无标签时只能靠 `rect.y` 排序定位 -- 页面重排后旧 `ref` 会失效——**每次交互前重新取快照** -- `accessibility` 能拿到 StaticText 正文，但**只在很简单的页面上可用**：实测 - example.com 正常返回，维基百科条目一律 `BROWSER_RESULT_TOO_LARGE`（回执上限 2 MB）， - 且 schema 里的 `depth` 参数目前被宿主忽略（硬编码 depth=50），调小也没用 -- 因此**取真实页面正文仍要靠 L2 Jina Reader（公开页）或 L3-fallback Playwright（登录态）** +**semantic** —— 可交互元素清单（button / input / a 等）+ `ref` 句柄。每行形如 +`[ref=e12] [loc=...] 按钮 提交`： +- `loc=` 是该元素的稳定选择器（S7）：仅当能产出时才给出，五级优先级 data-testid → + 唯一 CSS 安全 id → name 属性 → 链接 href（剥 query 前缀匹配）→ role+name +- 跨源 iframe（OOPIF）的元素并入同一棵树，ref 全局连续编号（S35） +- 页面重排后旧 `ref` 会失效——**每次交互前重新取快照**；带 ref 的命令要同时给签发该 + ref 的快照 `snapshotId` +- `options`：`scope`（viewport=视口内元素优先截断）、`maxElements`（默认 400 上限 2000）、 + `maxBytes`（默认 256KB 上限 1MB）、`cursor` 翻页、`limit` + +**text** —— 正文批量取文通道之一（S3/S4）： +```yaml +BrowserSnapshot: + mode: text + options: + format: markdown # markdown（默认，保留标题/列表/表格）| text（纯文本） + maxBytes: 65536 # 默认 65536，最大 524288；超出不报错，截断并返回 nextCursor + cursor: t1:...:... # 上一页返回的 nextCursor，同参数续读 + scope: full_page # full_page（默认）| viewport | ref（配 ref + snapshotId 取子树） + includeLinks: true # 链接目标写进正文（只写 http/https） + includeTables: true # 表格保留逐行结构 +``` + +**accessibility** —— AX 树（S8 起**尊重 `depth`、预算截断不炸**）： +`options.depth`（默认 50 上限 100）、`maxElements`、`maxBytes`、`cursor`。真实内容页不再 +`BROWSER_RESULT_TOO_LARGE` 整体失败——超预算截断给 nextCursor 续读。 + +**visual** —— 截图（像素直接进结果）： +`options.format`（png/jpeg）、`quality`、`captureBeyondViewport`（整页）、 +`clip={x,y,width,height,scale}`（元素级裁剪放大，`scale` 最大 4）。 ### BrowserAct -一次调用一个受管动作。完整 `action` 见工具 schema，常用的： +一次调用一个受管动作。`action` 分组： + +**tab.*** —— `navigate` / `back` / `forward` / `reload` / `activate` / `close`： ```yaml BrowserAct: @@ -96,10 +110,15 @@ BrowserAct: params: { url: https://www.xiaohongshu.com/explore/... } ``` +**input.*** —— `move` / `click` / `double-click` / `drag` / `wheel` / `touch` / `pinch` / +`key` / `text`。走 #1808 输入拟真（坐标派发、拟真轨迹、身份一致性），**反检测站点的交互首选**： + ```yaml BrowserAct: action: input.click - params: { ref: bref_xxx } # 用快照 ref；坐标会因页面重排失效 + params: + ref: e12 # 语义快照签发的短号 ref（不要裸 x/y） + snapshotId: snap-xxx # 签发该 ref 的快照回执里的 snapshotId，必带 ``` ```yaml @@ -114,37 +133,120 @@ BrowserAct: params: { deltaX: 0, deltaY: 720, x: 640, y: 400 } ``` -```yaml -BrowserAct: - action: tab.activate # 截图前必须先做这一步 - params: { bounds: { x: 0, y: 0, width: 1280, height: 900 } } -``` +带元素 ref 的 `input.*` 必须在 params 顶层同时带同一条快照回执的 `snapshotId`——ref 序号跨 +快照会重复，缺 snapshotId 一律拒绝执行。 + +**page.element** —— 判别式元素命令（op × selector，九 op，S10）。**表单批量填充等站点 +不检测场景用**；反检测站点交互走 input.*： ```yaml BrowserAct: - action: page.screenshot - params: { format: png } # 结果落 artifact store，回执给 artifact.id / sha256 / bytes + action: page.element + params: + op: fill # 写类：fill | select-option | check | uncheck | scroll-into-view + # 读类：get-attribute | bounding-box | count | all-inner-texts + selector: loc=css:input#email # loc= 方言或 e12 / ref=e12（配 snapshotId） + value: user@example.com # fill / select-option 用 + snapshotId: bsnp_xxx # selector 用 ref 时必带 ``` +- 写类走 `browser.input.keyboard` 能力档（与 input.text 同档）；读类走 `browser.observe.snapshot` 只读档 +- `fill` 对 `input[type=password]` **一律拒绝** +- `selector` 方言：`e<序号>` / `ref=e<序号>`、`loc=css:` / `loc=role:` / `loc=text:` / + `loc=testid:`、裸 CSS，可叠 `internal:nth/last/scope/filter`。未知前缀（如 `loc=xpath:`）显式报错，绝不静默降级 + +**page.wait** —— 判别式等待（until，九种，S11）。等待不改页面状态，走只读档： + +```yaml +BrowserAct: + action: page.wait + params: + until: networkidle # load | domcontentloaded | networkidle | selector | url | timeout + # | request | response | download + timeoutMs: 10000 # 默认 10000，上限 60000 + idleMs: 500 # networkidle 静默窗，默认 500 +``` + +- 轮询型（load/domcontentloaded/networkidle/selector/url/timeout）超时返回 `waited: false`，不抛错 +- 事件型（request/response/download）超时**抛错** +- `waitForFunction` 刻意不在枚举里——任意 JS 走 page.evaluate 的能力档与审批 + +**内联 wait 块**（S14）—— `tab.navigate` / `input.click` / `input.key` / `page.element{op:"fill"}` +支持 `params.wait`（形态与 page.wait 参数同构），一条回执完成「动作→等结果」，等待器先于动作注册， +不跨两次 IPC 出竞态： + +```yaml +BrowserAct: + action: tab.navigate + params: + url: https://example.com/login + wait: { until: networkidle, timeoutMs: 15000 } +``` + +**page.evaluate** —— 页面上下文求值（S2 起返回真实值）： + +```yaml +BrowserAct: + action: page.evaluate + params: + expression: document.title + awaitPromise: true # 默认 true +``` + +- 返回值**原样过界**（字符串/对象/数组都真实返回）；超预算截断并标 `truncated`，不抛错 +- 能力档 `browser.page.evaluate` 属人工闸门：非 allow-all 模式每次调用弹审批卡 +- 登录态取站内接口走 [fetch.browser 配方]（见下节） + +**page.extract-text** —— 与 `BrowserSnapshot mode:text` 同款取文通道，作为动作下发： +scope（viewport/full_page/ref）+ ref + snapshotId、format（markdown/text）、maxBytes（默认 +65536 最大 524288）、cursor（`^t1:` 续读）、includeLinks、includeTables。超出预算截断 + nextCursor。 + +**page.screenshot** —— 与 `BrowserSnapshot mode:visual` 同款：`format`、`quality`、 +`captureBeyondViewport`、`clip={x,y,width,height,scale}`。结果落 artifact store，回执给 +artifact.id / sha256 / bytes；截图像素直接进结果（见「截图」节）。 + +### BrowserScript（代码模式，S17+S18） + +一段异步 JS 在 Worker 里跑，通过注入的 `page` / `tab` / `input` / `snapshot` / `console` / +`performance` 门面连续下发命令——**导航→快照→点击→等待→取文一气呵成**，消除逐动作往返： + +```yaml +BrowserScript: + code: | + const snap = await snapshot.semantic(); + receipt.log(snap.content); + await input.click({ ref: 'e12', snapshotId: snap.snapshotId }); + await page.wait({ until: 'networkidle' }); + receipt.log(await page['extract-text']({ format: 'markdown' })); + sessionId: bss_xxx # 可选，多会话消歧 + tabId: btab_xxx # 可选，缺省用会话活动 tab + totalBudgetMs: 180000 # 默认 180000，上限 600000；审批等待不计入 + maxCommands: 500 # 每次运行命令条数上限 +``` + +- ❗**信任级别等同 Bash**：Worker 不是沙箱，脚本可访问 Node fs/net/child_process。两层审批： + 脚本源码先过一次与 Bash 同档的人类审批（allow-all / ask-external 豁免；脚本源码不可记忆）， + 其内每个受管浏览器能力再按「能力 × 本次运行」各问一次 +- helper 白名单 24 个：`snapshot.*` / `page.extract-text` / `tab.*` / `input.*` / + `page.evaluate` / `page.wait` / `page.element` / `console.read` / `performance.metrics`； + `console.log(page)` 可输出各 helper 的 signature/params/example 文档 +- 会话被用户接管、Lease 轮换 ⇒ **硬停**：后续命令一律不再下发 +- 需要 `browser.*` 相应能力；`receipt.log(...)` 或 `console.log` 输出文本 + ### BrowserImport（**需人工审批 + 额外授权**） -把用户浏览器里的登录态 Cookie 导入当前 Space。 +把用户浏览器里的登录态 Cookie 导入当前 Space——**唯一的登录态复用通道**。 > **前置条件（先看这里，别直接试）**：`browser.import.discover` / > `browser.import.cookies.inspect` / `browser.import.cookies` 三个能力**不在 `agentDefault` 里**， > 而 `BrowserManage(create_space)` 建的 Agent grant 就是按 `agentDefault` 签的。也就是说 > **仅靠 create_space / start_session 走不通 BrowserImport**，必须由 Host/用户侧另行授予 import -> 能力（`agentElevated` 或 Workbench 路径）。没有这层授权就别在这条路上耗——**直接回落 -> L3-fallback（Python Playwright 连用户已登录的 Chrome）**，那是当前更稳的登录态复用方式。 +> 能力（`agentElevated` 或 Workbench 路径）。没有这层授权就别在这条路上耗——如实告诉用户 +> 「当前无法复用你的登录态」，按无登录态继续或放弃。 动作枚举只有这 6 个：`discover` | `create_plan` | `dry_run` | `apply` | `rollback` | `list_plans` （**没有 `plan`**）。完整流程是 `discover → create_plan → dry_run → apply`： -```yaml -BrowserImport: - action: discover # 列出可导入的来源，返回不透明 sourceProfileId -``` - ```yaml BrowserImport: action: create_plan # 域名授权、来源、冲突策略都在这一步定死 @@ -155,57 +257,85 @@ BrowserImport: conflictStrategy: newer-wins # keep-target | replace-target | newer-wins | fail-on-conflict ``` -```yaml -BrowserImport: - action: dry_run # 先看命中多少条，再决定要不要真的写入 - planId: bimp_xxx # create_plan 返回的 ID，前缀是 bimp_ -``` - -```yaml -BrowserImport: - action: apply # 只认 planId；域名/策略在 create_plan 时已固化 - planId: bimp_xxx -``` - -写坏了用 `action: rollback` + 同一个 `planId` 回退；`list_plans` 查当前 Space 的历史 plan。 - -解密与过滤全在 Host 侧完成，**Cookie 值不会进入 Agent 上下文或审计日志**。 +`dry_run` 先看命中多少条再 `apply`（只认 `create_plan` 返回的 `planId`，前缀 `bimp_`）；写坏了 +用 `rollback` + 同一个 planId 回退；`list_plans` 查历史。解密与过滤全在 Host 侧完成， +**Cookie 值不会进入 Agent 上下文或审计日志**。 ### BrowserShare 把 Space / Session 委派给另一个 Agent，`shareMode` 可选 `snapshot`（只读副本）、 `copy-on-write`（写时复制）、`live-shared`（实时共享）、`handoff`（移交控制权）。 +## fetch.browser 配方：带登录态取接口数据 + +站内接口（列表、评论、订单等 JSON）在登录态下的正解——**页面上下文跑 `fetch`**，自动带该 +origin 的 Cookie，同 origin、受 Grant origins 约束，走 `page.evaluate` 已有闸门： + +```yaml +BrowserAct: + action: page.evaluate + params: + expression: | + fetch('/api/v1/comments?page=1&size=20', { + headers: { accept: 'application/json' } + }).then(r => r.text()) + awaitPromise: true +``` + +- 先 `tab.navigate` 到该站任意页面建立 origin 与 Cookie，再发 fetch；路径写**相对路径** +- 只能访问当前 tab origin；跨站接口先导航过去 +- 大 JSON 用 `.text()` 拿原文自己截取，或分页多次取 +- 非 allow-all 模式会弹审批卡（page.evaluate 人工闸门），向用户说明用途即可 + +## 交互通道选用：input.* vs page.element（反检测决策） + +| 场景 | 用什么 | 原因 | +|------|--------|------| +| 反检测站点（小红书/微博/B站等）的一切点击/输入 | **`input.*`** | 坐标派发 + #1808 拟真轨迹 + 身份一致性（UA/UA-CH 无 Electron/Headless 痕迹） | +| 表单批量填充等**站点不检测**的场景 | **`page.element` 写类** | 一条命令完成 fill/select-option/check，比逐元素 input.* 快得多 | +| 需要元素属性/坐标/计数/批量文本 | `page.element` 读类 | 只读档（browser.observe.snapshot），无写审批 | +| 任何「用 JS 直调 el.click()」的想法 | ❌ 禁止 | 绕开全部拟真投入（S10 红线）；指针动作只走 input.* | + ## 截图：像素直接给你，通常不需要再 Read -> **需要客户端 10.0.100+。** 更老的版本上截图只回元数据； -> 若需要画面信息请改用 `semantic` / `accessibility` 快照，或回落 Jina / Playwright。 +`BrowserSnapshot mode:visual` 与 `BrowserAct page.screenshot` 会把截图像素作为 image 块 +**直接放进工具结果**——视觉模型当场就能看，不必再调 `Read`。 -`BrowserSnapshot` `mode: visual` 与 `BrowserAct` `page.screenshot` 会把截图像素作为 -image 块**直接放进工具结果**——视觉模型当场就能看，不必再调 `Read`。 - -只有这几种情况才需要走 `result.artifact.absolutePath` 再 `Read` 一次： - -- 结果里明确写了「截图已保存，但…未附带像素」（超出单图预算、或体积过大） -- 你需要的是**原始分辨率**，而附带的像素被压缩过 - -对同一张图既看 image 块又 `Read` 一遍，只会让它在上下文里占两份。 +- **元素级裁剪放大**：`options.clip={x,y,width,height,scale}`（`scale` 最大 4）。先用 + `semantic` 快照拿到元素坐标，再截那一块并放大——验证码、小按钮在整页截图里只有几十像素， + 看不清时用它。**不必走 `cdp.raw`** +- **整页截图**：`captureBeyondViewport: true`（超出视口部分也截） +- 只有结果里明确写了「截图已保存，但…未附带像素」（超预算/体积过大）或需要原始分辨率时， + 才走 `result.artifact.absolutePath` 再 `Read` 一次 当前模型不支持视觉输入时，结果会明说「你看不到它的内容」——**此时不要凭空描述画面**， -改用 `semantic` / `accessibility` 快照拿页面信息。 +改用 `semantic` / `text` 快照拿页面信息。 + +## 视口与 tab.activate（S36 起） + +Agent 会话（actor ≠ user）的标签页**常驻离屏原位，保住合成表面**，不再停放成 1×1： + +| 场景 | 行为 | +|------|------| +| embedded，单标签会话 | 无需 `tab.activate` 即可截图/取视口（presentation 即呈现） | +| embedded，多标签会话的后台 tab | 仍需先 `tab.activate`；未激活的视口快判**秒级**报 `BROWSER_VIEWPORT_UNAVAILABLE`，不再挂满 30s，也不会毁掉标签页 | +| standalone | 后台 tab 可直接截图 | + +多 Space 依然可以并发导航/快照/输入，互不串扰。 ## 已知边界（照做，别试探） | 边界 | 说明 | |------|------| -| **截图前必须 `tab.activate`** | 标签页默认停在 `(-10000,-10000,1x1)`，没有合成表面。直接截图会卡满 30 s deadline，**并把标签页宿主打掉**，之后全部报 `BROWSER_TAB_HOST_NOT_FOUND` | -| **同时只有一个可见标签页** | `tab.activate` 绑定主窗口、全局互斥。多 Space 可以并发导航/快照/输入，但截图必须逐个 activate 串行 | -| **`page.evaluate` 基本不可用** | 每次调用需人工审批；返回的字符串/对象被替换为 `[REDACTED:browser-runtime-value]`（只有 number/boolean/null 穿透）；反调试站点会把它挂起几十秒。读页面用 `BrowserSnapshot` | -| **`cdp.raw` 需人工审批** | `browser.raw_cdp.*` 属于永远人工闸门的能力，无人值守流程用不了。**元素级裁剪不必走它**（需 10.0.100+）——`BrowserSnapshot` 的 `options.clip={x,y,width,height,scale}` 直接支持（`scale` 最大 4）：先用 `semantic` 快照拿到元素坐标，再截那一块并放大。验证码、小按钮在整页截图里只有几十像素，看不清时用它 | -| **没有批量取文通道** | `semantic` 不含正文，`accessibility` 在真实页面上超限，`page.evaluate` 被审批+脱敏。要正文请回落 Jina / Playwright | -| **单条命令 30 s deadline** | 超时即判 `browser.host.gone`，会话作废 | -| **用户真实操作会抢控制权** | 在可见标签页上**点击、按键、滚轮、触摸**会触发 `trusted-user-input` 并递增 control epoch，打断 Agent 的租约。**鼠标只是划过不会**（默认策略 `intentional-input`，需 10.0.100+；更老版本上划过也会打断），所以用户看着页面移动光标是安全的 | -| **artifact 不要走 `/save`** | 该接口会弹系统「另存为」对话框等人点。**用回执里的 `result.artifact.absolutePath`**（需 10.0.100+；更老版本没有这个字段，只能放弃直接读，改用页面内提取）（工具会把它登记进本次会话的可读白名单），不要自己拼路径——`${DESIRECORE_ROOT}` 这类变量在路径展开里不认（只认 `~` / `$HOME` / `$USERPROFILE`），拼出来是相对路径、`Read` 会报「文件不存在」。默认保留 24 小时 | +| **`page.evaluate` 走人工闸门** | 能力 `browser.page.evaluate` 属 always-human-gate：非 allow-all 模式每次弹审批卡。返回值已原样过界（超预算截断标 `truncated`），登录态取接口走 fetch.browser 配方 | +| **`cdp.raw` 需人工审批** | `browser.raw_cdp.*` 属于永远人工闸门的能力，无人值守流程用不了。元素级裁剪 / 整页截图用 `clip` / `captureBeyondViewport`，别走它 | +| **`page.element` 没有 click** | 指针动作必须走 `input.*`（拟真轨迹、可审计）；JS 直调 `el.click()` 是明确禁止的回退 | +| **`fill` 拒绝密码框** | `input[type=password]` 一律拒绝——密码输入只走 `input.text`（拟真键入） | +| **单条命令 30 s deadline** | 超时只掐掉**这一条命令**（stop 加载），标签页仍可用，重试即可；不再打掉宿主。等待类命令超时连 stop 都跳过 | +| **多标签后台 tab 先 activate** | embedded 多标签会话里后台 tab 截图前先 `tab.activate`，否则秒级 `BROWSER_VIEWPORT_UNAVAILABLE` | +| **用户真实操作会抢控制权** | 在可见标签页上**点击、按键、滚轮、触摸**会触发 `trusted-user-input` 并递增 control epoch，打断 Agent 的租约。**鼠标只是划过不会**（默认策略 `intentional-input`），所以用户看着页面移动光标是安全的 | +| **BrowserScript 信任级别等同 Bash** | 脚本可访问 Node fs/net/child_process；源码过一次与 Bash 同档审批 + 每能力一问；接管/Lease 轮换即硬停 | +| **artifact 不要走 `/save`** | 该接口会弹系统「另存为」对话框等人点。**用回执里的 `result.artifact.absolutePath`**（工具会把它登记进本次会话的可读白名单），不要自己拼路径——`${DESIRECORE_ROOT}` 这类变量在路径展开里不认（只认 `~` / `$HOME` / `$USERPROFILE`），拼出来是相对路径、`Read` 会报「文件不存在」。默认保留 24 小时 | ## SitePatternRead / SitePatternWrite @@ -228,17 +358,18 @@ SitePatternWrite: | 错误 | 原因 | 解决 | |------|------|------| | `该旧 BrowserXxx/cdp-proxy 入口已停用` | 还在调 v2.0 的旧工具 | 改用 BrowserManage / BrowserSnapshot / BrowserAct | -| `BROWSER_TAB_HOST_NOT_FOUND` | 标签页宿主已销毁（多因上一条命令超时） | 重建 Session；检查是否漏了 `tab.activate` | -| `BROWSER_COMMAND_DEADLINE_EXCEEDED` | 单条命令超 30 s | 截图先 activate；避免 `page.evaluate` | +| `BROWSER_TAB_HOST_NOT_FOUND` | 标签页宿主真的没了（渲染进程崩溃/销毁） | 重建 Session。命令超时**不再**引发此错误——超时只 stop 这一条命令，标签页保留 | +| `BROWSER_COMMAND_DEADLINE_EXCEEDED` | 单条命令超 30 s | 标签页仍停在上一页/空白页，直接重试；慢加载常态不算故障 | +| `BROWSER_VIEWPORT_UNAVAILABLE` | embedded 多标签会话的后台 tab 没有可用视口 | 先 `tab.activate` 再截图（秒级快判，不是挂死） | | `BROWSER_TOOL_SESSION_FORBIDDEN` | 会话已关闭/崩溃，或不属于当前 Agent | 重新 `list_sessions`，必要时重建 | | `BROWSER_TOOL_SESSION_AMBIGUOUS` | 当前 Agent 有多个会话且未传 sessionId | 显式传 `sessionId` | -| `BROWSER_HUMAN_APPROVAL_REQUIRED` | 触到人工闸门能力（evaluate / raw_cdp / 上传下载 / Cookie 导入等） | 向用户说明用途并等待审批，或换用无需审批的路径 | -| `BROWSER_RESULT_TOO_LARGE` | 回执超过 2 MB（`accessibility` 快照最常见） | 改用 `semantic` 快照 + 截图；取正文回落 Jina / Playwright | +| `BROWSER_HUMAN_APPROVAL_REQUIRED` | 触到人工闸门能力（evaluate / raw_cdp / 上传下载 / Cookie 导入等） | 向用户说明用途并等待审批，或换用无需审批的路径（取文用 extract-text，裁剪用 clip） | +| `BROWSER_RESULT_TOO_LARGE` | 回执超过结果上限 | text/semantic/accessibility 都有 maxBytes + cursor 分页，调小 maxBytes 续读即可，不再整体失败 | ## 调用链路 ``` -Agent → BrowserManage/Snapshot/Act → browser-use service（Capability/Grant/Lease/Policy 校验） +Agent → BrowserManage/Snapshot/Act/Script → browser-use service（Capability/Grant/Lease/Policy 校验） → BrowserHost（electron-embedded 或 standalone-managed）→ Chromium ↑ 每步产出带 digest 的回执，写入审计事件流 ``` diff --git a/skills/web-access/references/cdp-browser.md b/skills/web-access/references/cdp-browser.md deleted file mode 100644 index 40cdd14..0000000 --- a/skills/web-access/references/cdp-browser.md +++ /dev/null @@ -1,330 +0,0 @@ -# CDP Browser Access — Login-Gated Sites Manual - -Detailed recipes for accessing sites that require the user's login session, via Chrome DevTools Protocol (CDP) + Python Playwright. - -**Precondition**: Chrome is already running with `--remote-debugging-port=9222` and the user has manually logged in to the target sites. See the main SKILL.md `Prerequisites` section for the launch command. - ---- - -## Why CDP attach, not headless - -| Approach | Login state | Anti-bot | Speed | Cost | -|----------|-------------|----------|-------|------| -| Headless Playwright (new context) | ❌ Empty cookies | ❌ Flagged as bot | Slow cold start | Re-login pain | -| `playwright.chromium.launch(headless=False)` | ❌ Fresh profile | ⚠ Sometimes flagged | Slow | Same | -| **CDP attach (`connect_over_cdp`)** | ✅ User's real cookies | ✅ Looks human | Instant | Zero friction | - -**Rule**: For any login-gated site, always attach to the user's running Chrome. - ---- - -## Core Template - -Every CDP script follows this shape: - -```python -from playwright.sync_api import sync_playwright - -def fetch_with_cdp(url: str, wait_selector: str | None = None) -> str: - """Attach to user's Chrome via CDP, fetch URL, return HTML.""" - with sync_playwright() as p: - browser = p.chromium.connect_over_cdp("http://localhost:9222") - # browser.contexts[0] is the user's default context (with cookies) - context = browser.contexts[0] - page = context.new_page() - try: - page.goto(url, wait_until="domcontentloaded", timeout=30000) - if wait_selector: - page.wait_for_selector(wait_selector, timeout=10000) - else: - page.wait_for_timeout(2000) # generic settle - return page.content() - finally: - page.close() - # DO NOT call browser.close() — that would close the user's Chrome! - -if __name__ == "__main__": - html = fetch_with_cdp("https://example.com") - print(html[:1000]) -``` - -**Critical**: Never call `browser.close()` when using CDP attach — you'd kill the user's Chrome. Only close the page you opened. - ---- - -## Site Recipes - -### 小红书 (xiaohongshu.com) - -```python -from playwright.sync_api import sync_playwright -from bs4 import BeautifulSoup - -NOTE_URL = "https://www.xiaohongshu.com/explore/XXXXXXXX" - -with sync_playwright() as p: - browser = p.chromium.connect_over_cdp("http://localhost:9222") - page = browser.contexts[0].new_page() - page.goto(NOTE_URL, wait_until="domcontentloaded") - page.wait_for_selector("#detail-title", timeout=10000) - page.wait_for_timeout(1500) # let images/comments load - html = page.content() - page.close() - -soup = BeautifulSoup(html, "html.parser") -title = (soup.select_one("#detail-title") or {}).get_text(strip=True) if soup.select_one("#detail-title") else None -desc = (soup.select_one("#detail-desc") or {}).get_text(" ", strip=True) if soup.select_one("#detail-desc") else None -author = soup.select_one(".author-wrapper .username") -print("Title:", title) -print("Author:", author.get_text(strip=True) if author else None) -print("Desc:", desc) -``` - -**Selectors** (may drift over time — update if they fail): -- Title: `#detail-title` -- Description: `#detail-desc` -- Author: `.author-wrapper .username` -- Images: `.swiper-slide img` -- Comments: `.parent-comment .content` - -### B站 (bilibili.com) - -```python -from playwright.sync_api import sync_playwright -from bs4 import BeautifulSoup - -VIDEO_URL = "https://www.bilibili.com/video/BVxxxxxxxxx" - -with sync_playwright() as p: - browser = p.chromium.connect_over_cdp("http://localhost:9222") - page = browser.contexts[0].new_page() - page.goto(VIDEO_URL, wait_until="networkidle") - page.wait_for_timeout(2000) - html = page.content() - page.close() - -soup = BeautifulSoup(html, "html.parser") -print("Title:", soup.select_one("h1.video-title").get_text(strip=True) if soup.select_one("h1.video-title") else None) -print("UP:", soup.select_one(".up-name").get_text(strip=True) if soup.select_one(".up-name") else None) -print("Desc:", soup.select_one(".desc-info-text").get_text(" ", strip=True) if soup.select_one(".desc-info-text") else None) -``` - -**Tip**: For B站 evaluations, the [公开 API](https://api.bilibili.com/x/web-interface/view?bvid=XXXX) often returns JSON without needing CDP. Try it first: - -```bash -curl -s "https://api.bilibili.com/x/web-interface/view?bvid=BVxxxxxxxxx" | python3 -m json.tool -``` - -### 微博 (weibo.com) - -```python -WEIBO_URL = "https://weibo.com/u/1234567890" # or /detail/xxx - -# Same CDP template -# Selectors: -# .Feed_body_3R0rO .detail_wbtext_4CRf9 — post text -# .ALink_default_2ibt1 — user link -# article[aria-label="微博"] — each feed item -``` - -**Note**: Weibo uses React + heavy obfuscation. Selectors change frequently. If selectors fail, pipe the HTML through Jina for clean Markdown: - -```python -html = fetch_with_cdp(WEIBO_URL) -# Save to temp file, then: -import subprocess -result = subprocess.run( - ["curl", "-sL", f"https://r.jina.ai/{WEIBO_URL}"], - capture_output=True, text=True -) -print(result.stdout) -``` - -### 知乎 (zhihu.com) - -```python -ANSWER_URL = "https://www.zhihu.com/question/123/answer/456" - -# Selectors: -# h1.QuestionHeader-title — question title -# .RichContent-inner — answer body -# .AuthorInfo-name — author -``` - -Zhihu works with CDP but often also renders enough metadata server-side for Jina to work: - -```bash -curl -sL "https://r.jina.ai/https://www.zhihu.com/question/123/answer/456" -``` - -Try Jina first, fall back to CDP if content is truncated. - -### 飞书文档 (feishu.cn / larksuite.com) - -```python -DOC_URL = "https://xxx.feishu.cn/docs/xxx" - -# Feishu uses heavy virtualization — must scroll to load all content. -# Recipe: - -from playwright.sync_api import sync_playwright - -with sync_playwright() as p: - browser = p.chromium.connect_over_cdp("http://localhost:9222") - page = browser.contexts[0].new_page() - page.goto(DOC_URL, wait_until="domcontentloaded") - page.wait_for_selector(".docs-render-unit", timeout=15000) - - # Scroll to bottom repeatedly to load lazy content - last_height = 0 - for _ in range(20): - page.evaluate("window.scrollTo(0, document.body.scrollHeight)") - page.wait_for_timeout(800) - h = page.evaluate("document.body.scrollHeight") - if h == last_height: - break - last_height = h - - # Extract text - text = page.evaluate("() => document.body.innerText") - page.close() - -print(text) -``` - -### Twitter / X - -```python -TWEET_URL = "https://x.com/username/status/1234567890" - -# Selectors: -# article[data-testid="tweet"] — tweet container -# div[data-testid="tweetText"] — tweet text -# div[data-testid="User-Name"] — author -# a[href$="/analytics"] — view count anchor (next sibling has stats) -``` - -Twitter is aggressive with anti-bot. CDP attach usually works, but set a generous wait: - -```python -page.goto(url, wait_until="networkidle", timeout=45000) -page.wait_for_selector('article[data-testid="tweet"]', timeout=15000) -``` - ---- - -## Common Patterns - -### Pattern 1: Scroll to load lazy content - -```python -def scroll_to_bottom(page, max_steps=30, pause_ms=800): - last = 0 - for _ in range(max_steps): - page.evaluate("window.scrollTo(0, document.body.scrollHeight)") - page.wait_for_timeout(pause_ms) - h = page.evaluate("document.body.scrollHeight") - if h == last: - return - last = h -``` - -### Pattern 2: Screenshot a specific element - -```python -element = page.locator("article").first -element.screenshot(path="/tmp/article.png") -``` - -### Pattern 3: Extract structured data via JavaScript - -```python -data = page.evaluate("""() => { - const items = document.querySelectorAll('.list-item'); - return Array.from(items).map(el => ({ - title: el.querySelector('.title')?.innerText, - url: el.querySelector('a')?.href, - })); -}""") -print(data) -``` - -### Pattern 4: Fill a form and click - -```python -page.fill("input[name=q]", "search query") -page.click("button[type=submit]") -page.wait_for_load_state("networkidle") -``` - -### Pattern 5: Clean HTML via Jina after extraction - -When selectors are unreliable, dump the full page HTML and let Jina do the cleaning: - -```python -html = page.content() -# Save to file, serve via local HTTP, or just pipe the original URL: -import subprocess -clean_md = subprocess.run( - ["curl", "-sL", f"https://r.jina.ai/{url}"], - capture_output=True, text=True -).stdout -print(clean_md) -``` - ---- - -## Troubleshooting - -### `connect_over_cdp` fails with `ECONNREFUSED` - -Chrome is not running with remote debugging. Tell the user: -> "请先用下面的命令启动 Chrome： -> `/Applications/Google\\ Chrome.app/Contents/MacOS/Google\\ Chrome --remote-debugging-port=9222 --user-data-dir=\"${DESIRECORE_ROOT}/chrome-profile\"` -> 然后手动登录需要抓取的网站，再让我继续。" - -### `browser.contexts[0]` is empty - -Chrome was launched but no windows are open. Ask the user to open at least one tab and navigate anywhere. - -### Playwright not installed - -```bash -pip3 install playwright beautifulsoup4 -# No need for `playwright install` — we're attaching to existing Chrome, not downloading a new browser -``` - -### Site detects automation - -Despite CDP attach, some sites (Cloudflare-protected, Instagram) may still detect automation. Options: -1. Use Jina Reader instead (`curl -sL https://r.jina.ai/`) — often succeeds where Playwright fails -2. Ask the user to manually copy the visible content -3. Use the site's public API if available - -### Content is truncated - -The page uses virtualization or lazy loading. Apply Pattern 1 (scroll to bottom) before calling `page.content()`. - -### `page.wait_for_selector` times out - -The selector is stale — the site updated its DOM. Dump `page.content()[:5000]` and inspect manually, or fall back to Jina Reader. - ---- - -## Security Notes - -- **Never log or print cookies** from `context.cookies()` even during debugging -- **Never extract and store** the user's session tokens to files -- **Never use the CDP session** to perform writes (post, comment, like) unless the user explicitly requested it -- The `${DESIRECORE_ROOT}/chrome-profile` directory contains the user's credentials — treat it as sensitive -- If the user asks to "log in automatically", refuse and explain they must log in manually in the Chrome window; the skill only reads already-authenticated sessions - ---- - -## When NOT to use CDP - -- **Public static sites** → use L1 `WebFetch`, it's faster -- **Heavy SPAs without login walls** → use L2 Jina Reader, it's cheaper on tokens -- **You need thousands of pages** → CDP is not built for scale; look into proper scrapers - -CDP is specifically the "right tool" for: **small number of pages + login required + human-like behavior needed**. diff --git a/skills/web-access/references/jina-reader.md b/skills/web-access/references/jina-reader.md index 19db9cc..e614807 100644 --- a/skills/web-access/references/jina-reader.md +++ b/skills/web-access/references/jina-reader.md @@ -95,19 +95,8 @@ curl -sL "https://r.jina.ai/$URL" > /tmp/jina-cache.md ### Handle very long articles Jina returns the full article in one response. For articles > 50K chars, pipe through `head` or extract specific sections with Python/awk before feeding back to the model context. -### Combine with CDP -When you use L3 CDP to fetch a login-gated page, you can pipe the resulting HTML through Jina for clean Markdown instead of parsing with BeautifulSoup: - -```python -html = fetch_with_cdp(url) # from references/cdp-browser.md -# Now convert via Jina (note: Jina fetches the URL itself, not your HTML) -# So this only works if the content is already visible without login: -import subprocess -md = subprocess.run(["curl", "-sL", f"https://r.jina.ai/{url}"], - capture_output=True, text=True).stdout -``` - -For truly login-gated content, you must parse the HTML directly (BeautifulSoup) since Jina can't log in on your behalf. +### Combine with the built-in browser +For login-gated pages, the built-in browser already has its own extraction channel with paged budgets — `BrowserAct({ action: 'page.extract-text', params: { format: 'markdown', maxBytes: 65536 } })` returns clean Markdown directly (beyond maxBytes it truncates and hands back a nextCursor). Jina can't log in on your behalf, so use Jina for public pages and `page.extract-text` for logged-in ones; there is no need to round-trip HTML through Jina anymore. ---