diff --git a/skills/web-access/SKILL.md b/skills/web-access/SKILL.md index d235e56..934bc0f 100644 --- a/skills/web-access/SKILL.md +++ b/skills/web-access/SKILL.md @@ -5,17 +5,17 @@ description: >- — searching for current information, fetching public web pages, browsing login-gated sites (微博/小红书/B站/飞书/Twitter), comparing products, researching topics, gathering documentation, or summarizing news. - This skill orchestrates three complementary layers: (1) WebSearch + WebFetch + This skill orchestrates four complementary layers: (1) WebSearch + WebFetch for public pages, (2) Jina Reader as the default token-optimization layer for heavy/JS-rendered pages, and (3) the governed built-in browser (isolated BrowserSpace + Cookie import + bulk text extraction + waits + code mode) to - reach, interact with, and read login-gated sites. Always cite source URLs. + reach, interact with, and read login-gated sites, and (4) the user's own Chrome over CDP when they ask for it by name. Always cite source URLs. Use when 用户提到 联网搜索、上网查、 查资料、抓取网页、研究、调研、最新资讯、文档查询、对比、竞品、技术文档、 新闻、网址、URL、找一下、搜一下、查一下、小红书、B站、微博、飞书、Twitter、 推特、X、知乎、公众号、已登录、登录状态。 license: Complete terms in LICENSE.txt -version: 3.0.1 +version: 3.2.0 type: procedural risk_level: low status: enabled @@ -41,7 +41,7 @@ provides: - LocalBookmarks metadata: author: desirecore - updated_at: '2026-08-18' + updated_at: '2026-08-21' i18n: default_locale: en-US source_locale: zh-CN @@ -51,16 +51,16 @@ metadata: zh-CN: name: 联网访问 short_desc: 联网搜索、网页抓取、内置受管浏览器登录态访问与取文、研究调研工作流 - description: 三层联网访问工具包——搜索公开页面、Jina 优化抓取、内置受管浏览器完成登录态访问、交互与取文。 + description: 联网访问工具包——搜索公开页面、Jina 优化抓取、内置受管浏览器完成登录态访问与取文,以及用户点名时接管他自己的 Chrome。 body: ./SKILL.zh-CN.md - source_hash: sha256:20c98f047378220a + source_hash: sha256:4d3bc4221b2d6b09 translated_by: human en-US: name: Web Access short_desc: Web search, page fetching, logged-in access via the governed built-in browser, research workflows - description: A three-layer web-access toolkit — search public pages, fetch heavy pages via Jina Reader, and reach, interact with, and read logged-in sites through the governed built-in browser. + description: A web-access toolkit — search public pages, fetch heavy pages via Jina Reader, reach and read logged-in sites through the governed built-in browser, and drive the user's own Chrome over CDP on request. body: ./SKILL.md - source_hash: sha256:20c98f047378220a + source_hash: sha256:4d3bc4221b2d6b09 translated_by: human market: icon: >- @@ -89,19 +89,21 @@ market: ## L0: One-line Summary -A three-layer web-access toolkit — search public pages, optimize fetches via Jina Reader, and reach, interact with, and read login-gated sites through the governed built-in browser (v3.0 ships bulk text extraction, waits, and code mode in-browser; the Python Playwright fallback is gone). +A web-access toolkit — search public pages, optimize fetches via Jina Reader, reach/interact with/read login-gated sites through the governed built-in browser, and drive the user's own Chrome over CDP when they ask for it by name. ## L1: Overview & Use Cases ### Capability -web-access is a **procedural skill** that provides three complementary layers of web access: +web-access is a **procedural skill** that provides four complementary layers of web access: - **L1** (WebSearch + WebFetch): public, static pages - **L2** (Jina Reader): JS-rendered heavy pages, saving tokens by default - **L3** (governed built-in browser, capability surface completed in v3.0): reach, *interact with*, and **read** logged-in / interactive sites — isolated BrowserSpace per task, zero Python dependency, every action carries a signed receipt. Bulk text extraction (`page.extract-text`), discriminated waits (`page.wait`), and code mode (`BrowserScript`) all close the loop inside this layer -The v2.x fourth layer — "user manually launches a debug Chrome + Python Playwright CDP" — was removed in v3.0: every reason it existed for (no bulk text channel, evaluate unusable, screenshots must activate-serialize) is now covered by the built-in browser, see the cheatsheet below. +- **L3-external** (the user's own Chrome, attached via CDP + Python Playwright): **take this route when the user names their own browser** — their login state, their window, visible to them the whole time and theirs to take over at any moment + +A note of history on L3-external: v3.0 deleted it outright, on the grounds that "every technical reason it existed for (no bulk text channel, evaluate unusable, screenshots must activate-serialize) is now covered by the built-in browser". That technical judgement was correct — **as a fallback for when the built-in browser isn't enough, it genuinely isn't needed any more**. But the deletion took with it a completely different use case: the user wanting *their own* browser. That has nothing to do with capability, and the built-in browser cannot stand in for it, so v3.2 restores it as a peer option **triggered by user intent**. Note it is no longer a fallback; see "Two browsers — pick by user intent" below. ### v3.0: governed built-in browser (default-hidden, exposed only after Skill activation) @@ -132,10 +134,10 @@ When you call `Skill('web-access')`, the following 9 tools are injected into the ### Core Value -- **Three-layer progression**: from lightweight search to heavy JS rendering to logged-in access — pick on demand +- **Layered progression**: from lightweight search to heavy JS rendering to logged-in access — pick on demand; plus the user's own browser whenever they name it - **Token optimization**: Jina Reader cuts token usage by 50–80% by default; `page.extract-text`'s maxBytes/cursor paging keeps even long logged-in articles under control - **Logged-in session reuse**: where the Host has granted `browser.import.*`, BrowserImport brings the user's Cookies into an isolated Space — no re-login required -- **Zero external dependencies**: no Python/Playwright install, no manually launched debug Chrome +- **Zero external dependencies by default**: the built-in browser needs no Python/Playwright install and no manually launched debug Chrome (L3-external does, and only when the user asks for it) ## L2: Detailed Specification @@ -147,6 +149,55 @@ When you complete a research task, you **MUST** cite all source URLs in your res If any fetch fails, explicitly tell the user which URL failed and which fallback you used. +## Prerequisites: Chrome CDP Setup (L3-external only) + +**Only needed when taking the L3-external route** (the user named their own browser). The built-in +browser has no prerequisites. + +### One-time setup + +Have the user launch Chrome with remote debugging enabled: + +**macOS**: +```bash +/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \ + --remote-debugging-port=9222 \ + --user-data-dir="${DESIRECORE_ROOT}/chrome-profile" +``` + +**Linux**: +```bash +google-chrome \ + --remote-debugging-port=9222 \ + --user-data-dir="${DESIRECORE_ROOT}/chrome-profile" +``` + +**Windows (PowerShell)**: +```powershell +& "C:\Program Files\Google\Chrome\Application\chrome.exe" ` + --remote-debugging-port=9222 ` + --user-data-dir="$env:USERPROFILE\.desirecore\chrome-profile" +``` + +After launch: +1. The user logs in manually to the sites they need +2. That Chrome window stays open +3. Verify the debug endpoint: `curl -s http://localhost:9222/json/version` should return JSON + +### Verify readiness before every operation + +```bash +curl -s http://localhost:9222/json/version | python3 -c "import sys,json; d=json.load(sys.stdin); print('CDP ready:', d.get('Browser'))" +``` + +If it fails, tell the user: "请先启动 Chrome 并开启远程调试端口(见 web-access 技能的 Prerequisites 部分)" +— **then wait for them.** Don't switch to the built-in browser just because it could also do the job. + +⚠️ When attached over CDP, **never call `browser.close()`** — that would close the user's own Chrome. +Only close the page you opened. Full recipes in [references/cdp-browser.md](references/cdp-browser.md). + +--- + ## Tool Selection Decision Tree ``` @@ -177,20 +228,66 @@ User intent │ - PyPI: curl https://pypi.org/pypi//json │ └─ "Real-time interactive task" (click, fill form, scroll, screenshot) - └─→ built-in browser (BrowserManage → BrowserAct → BrowserSnapshot — + ├─→ **User named "my own / my machine's / the external browser"** → L3-external: + │ verify CDP is ready (see Prerequisites), then python3 playwright.connect_over_cdp() + │ If it isn't ready, give them the launch command and wait — don't quietly switch to the built-in one + └─→ **Otherwise (default)**: built-in browser (BrowserManage → BrowserAct → BrowserSnapshot — see references/browser-tools.md, no Python needed) ``` -### Three-layer strategy summary +### Two browsers — pick by user intent, not by difficulty + +DesireCore can drive **two** browsers. They are peer options: + +| | L3 built-in governed browser | L3-external — the user's own browser | +|---|---|---| +| What it is | A browser instance inside the app (the `Browser*` tools) | The Chrome installed on the user's machine, attached via CDP + Python Playwright | +| Login state | Isolated; needs `browser.import.*` granted by the Host before `BrowserImport` can pull cookies | **Literally the user's own session** — nothing to import | +| Can the user see it | Agent tabs are offscreen by default; must be presented to the workbench | **It's their own window** — visible throughout, theirs to take over | +| Prerequisite | None | User must launch Chrome with `--remote-debugging-port=9222` (see Prerequisites) | +| Default | ✅ yes | When the user names it | + +> The login-state row is easy to misread as "the built-in browser can't reuse the user's login +> state" — that isn't what it says. Precisely: it **can't reuse it directly**. `Browser*` cannot see +> the windows or tabs of the user's external browser and cannot read its live session; but once the +> Host grants `browser.import.*`, `BrowserImport` can carry that login state into an isolated Space +> **by importing cookies**. The distinction is "take over that live session" (not possible) versus +> "import a copy of the cookies" (possible, once authorized). Only without that grant is the login +> state genuinely unreusable — and then you say so plainly. + +**The layer is chosen by user intent, not by technical difficulty.** v3.0 deleted this layer as +"a fallback for when the built-in browser isn't enough" — and as a fallback, it really isn't needed +any more. But that deletion also removed a **different** use case: the user wanting *their own* +browser. That has nothing to do with capability — their login state lives in their Chrome, and they +want to watch it happen and take over when they choose. The built-in browser cannot stand in for that. + +**If the user named one, use the one they named:** + +- "my own / my machine's / the external browser / my Chrome" → **L3-external**. Verify CDP readiness + per Prerequisites first; if it isn't ready, give them the launch command and wait. **Do not switch + to the built-in browser just because it could also do the job** +- "the built-in browser", or nothing named → **L3 built-in** (default, no prerequisites) +- Genuinely unclear which they mean → ask, don't guess + +⚠️ Either way, the user must be able to tell which one you actually used. Never use wording that fits +both — "the local browser", "the managed browser on your machine", "your local browser is now open". +When what they asked for and what you're giving differ, the wording has to make that visible. + +### Layer strategy summary | Layer | Use case | Primary tool | Token cost | |-------|----------|--------------|------------| | L1 | Public, static | `WebFetch` | Low | | L2 | JS-heavy, long articles, token savings | `Bash curl r.jina.ai` | **Lowest** (Markdown pre-cleaned) | | **L3** | **Login-gated navigation, interaction & extraction (PRIMARY)** | **built-in browser (BrowserManage / BrowserAct / BrowserSnapshot / BrowserScript)** | Medium | +| L3-external | **User named their own browser**; or their personal login state is needed and `BrowserImport` is unavailable | `Bash + Python Playwright connect_over_cdp` (see references/cdp-browser.md) | Medium | **Default priority**: L1 for simple public pages → L2 for heavy → **L3 for login-gated (body text and in-site API data included)**. + +> L3-external is deliberately absent from this ordering: it isn't chosen by "is the layer capable +> enough" but by **the user naming it**. When the user wants their own browser, go there directly — +> even if the built-in browser could do the job. See "Two browsers — pick by user intent" above. ## Supported Sites Matrix | Site | Recommended Layer | Notes | diff --git a/skills/web-access/SKILL.zh-CN.md b/skills/web-access/SKILL.zh-CN.md index 74a9588..7ada3f4 100644 --- a/skills/web-access/SKILL.zh-CN.md +++ b/skills/web-access/SKILL.zh-CN.md @@ -4,19 +4,21 @@ ## L0:一句话摘要 -三层联网访问工具包——搜索公开页面、Jina 优化抓取、内置受管浏览器完成登录态访问与交互(v3.0 起取文/等待/脚本全部内置,不再需要 Python Playwright 兜底)。 +联网访问工具包——搜索公开页面、Jina 优化抓取、内置受管浏览器完成登录态访问与交互,以及用户点名时接管他自己的 Chrome。 ## L1:概述与使用场景 ### 能力描述 -web-access 是一个**流程型技能(Procedural Skill)**,提供三层互补的联网访问能力: +web-access 是一个**流程型技能(Procedural Skill)**,提供四层互补的联网访问能力: - **L1**(WebSearch + WebFetch):公开页面,轻量 - **L2**(Jina Reader):JS 渲染的重页面,默认节省 Token - **L3**(内置受管浏览器,v3.0 能力面补全):到达、操作并**读取**登录态/交互站点——每个任务独立 BrowserSpace 隔离、零 Python 依赖、每次动作都有可审计回执。批量取文(`page.extract-text`)、判别式等待(`page.wait`)、代码模式(`BrowserScript`)都在本层内闭环 -v2.x 时代的第四层「用户手工启动调试 Chrome + Python Playwright CDP」已在 v3.0 移除:它依赖的每一条理由(无批量取文通道、evaluate 不可用、截图必须串行 activate)都被内置浏览器的新能力覆盖,见下方速查。 +- **L3-external**(用户自己的 Chrome,经 CDP + Python Playwright 接管):**用户点名要用他自己那个浏览器时走这条**——他的登录态、他的窗口、他能全程看着并随时接管 + +关于 L3-external 的一段历史:v3.0 曾把它整个删掉,理由是「它存在的每一条技术理由(无批量取文通道、evaluate 不可用、截图必须串行 activate)都已被内置浏览器覆盖」。那个技术判断没错,**作为「内置浏览器不够用时的兜底」它确实不再需要**。但删除时顺带丢掉了一个完全不同的用例:用户想用**他自己那个**浏览器。这跟能力够不够无关,内置浏览器替代不了,所以 v3.2 把它作为一条**由用户意图触发**的平级选择恢复回来——注意它不再是 fallback,判据见下方「两个浏览器,按用户意图选」。 ### v3.0:内置受管浏览器(默认隐藏,激活后才暴露) @@ -47,10 +49,10 @@ v2.x 时代的第四层「用户手工启动调试 Chrome + Python Playwright CD ### 核心价值 -- **三层递进**:从轻量搜索到重度 JS 渲染到登录态访问,按需选择 +- **分层递进**:从轻量搜索到重度 JS 渲染到登录态访问,按需选择;用户点名时还可直接用他自己的浏览器 - **Token 优化**:Jina Reader 默认减少 50-80% Token 消耗;`page.extract-text` 的 maxBytes/cursor 分页让登录态长文也可控 - **登录态复用**:Host 授予 `browser.import.*` 时用 BrowserImport 把 Cookie 导入隔离 Space,不必重新登录 -- **零外部依赖**:不再要求 Python/Playwright 安装,也不再要求用户手工启动调试 Chrome +- **默认零外部依赖**:内置浏览器不要求 Python/Playwright,也不要求用户手工启动调试 Chrome(L3-external 需要,且仅在用户点名时才用) ## L2:详细规范 @@ -62,6 +64,54 @@ When you complete a research task, you **MUST** cite all source URLs in your res If any fetch fails, explicitly tell the user which URL failed and which fallback you used. +## Prerequisites: Chrome CDP Setup(仅 L3-external 需要) + +**只有走 L3-external(用户点名要用他自己的浏览器)时才需要这一步。** 内置浏览器零前置条件。 + +### One-time setup + +让用户带远程调试端口启动 Chrome: + +**macOS**: +```bash +/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \ + --remote-debugging-port=9222 \ + --user-data-dir="${DESIRECORE_ROOT}/chrome-profile" +``` + +**Linux**: +```bash +google-chrome \ + --remote-debugging-port=9222 \ + --user-data-dir="${DESIRECORE_ROOT}/chrome-profile" +``` + +**Windows (PowerShell)**: +```powershell +& "C:\Program Files\Google\Chrome\Application\chrome.exe" ` + --remote-debugging-port=9222 ` + --user-data-dir="$env:USERPROFILE\.desirecore\chrome-profile" +``` + +启动后: +1. 用户在这个 Chrome 里手工登录需要的站点 +2. 这个 Chrome 窗口保持开着 +3. 验证调试端点:`curl -s http://localhost:9222/json/version` 应返回 JSON + +### 每次操作前先验就绪 + +```bash +curl -s http://localhost:9222/json/version | python3 -c "import sys,json; d=json.load(sys.stdin); print('CDP ready:', d.get('Browser'))" +``` + +失败就告诉用户:「请先启动 Chrome 并开启远程调试端口(见 web-access 技能的 Prerequisites 部分)」, +**然后等他**——不要因为内置浏览器也能做就擅自改用内置的。 + +⚠️ 用 CDP attach 时**绝不能调 `browser.close()`**,那会关掉用户自己的 Chrome;只关你开的 page。 +完整配方见 [references/cdp-browser.md](references/cdp-browser.md)。 + +--- + ## Tool Selection Decision Tree ``` @@ -92,20 +142,62 @@ User intent │ - PyPI: curl https://pypi.org/pypi//json │ └─ "Real-time interactive task" (click, fill form, scroll, screenshot) - └─→ 内置受管浏览器 (BrowserManage → BrowserAct → BrowserSnapshot — + ├─→ **用户点名「我本机的 / 我自己的 / 外部的浏览器」** → L3-external: + │ 先验 CDP 就绪(见 Prerequisites),再 python3 playwright.connect_over_cdp() + │ 没就绪就给启动命令并等他,不要擅自改用内置浏览器 + └─→ **其余情况(默认)**:内置受管浏览器 (BrowserManage → BrowserAct → BrowserSnapshot — see references/browser-tools.md, no Python needed) ``` -### 三层策略总结 +### 两个浏览器,按用户意图选,不按能力难度选 + +DesireCore 能驱动**两个**浏览器,它们是平级的选项: + +| | L3 内置受管浏览器 | L3-external 用户自己的浏览器 | +|---|---|---| +| 是什么 | 应用内的浏览器实例(`Browser*` 工具族) | 用户机器上装的 Chrome,经 CDP + Python Playwright 接管 | +| 登录态 | 独立隔离;需 Host 授予 `browser.import.*` 才能用 `BrowserImport` 导 Cookie | **就是用户本人的登录态**,无需导入 | +| 用户能看到吗 | Agent 开的标签页默认离屏,需展示到工作台 | **就在用户自己的窗口里**,他能全程看着、随时接管 | +| 前置条件 | 无 | 用户需先带 `--remote-debugging-port=9222` 启动 Chrome(见 Prerequisites) | +| 默认 | ✅ 是 | 用户点名时 | + +> 登录态那一栏容易读成「内置浏览器复用不了用户的登录态」——不是那个意思。准确说法是 +> **不能直接复用**:`Browser*` 看不见用户外部浏览器的窗口与标签页,也读不到它的活动会话; +> 但 Host 授予 `browser.import.*` 后,`BrowserImport` 可以**以导入 Cookie 的方式**把那边的 +> 登录态搬进隔离 Space。区别在于「直接接管那个会话」(做不到)与「导入一份 Cookie」 +> (授权后可以)。没有这层授权时才是真的复用不了——那时如实告诉用户。 + +**选层判据是用户意图,不是技术难度。** v3.0 把这一层当作「内置浏览器不够用时的兜底」删掉过, +那个技术判断本身没错(取文、evaluate、截图这些内置浏览器现在都能做),但它顺带删掉的是一个 +**完全不同的用例**:用户想用**他自己那个**浏览器。那跟能力够不够无关——他的登录态在他自己的 +Chrome 里,他想亲眼看着操作、随时接管。这个需求内置浏览器替代不了。 + +**用户点名了就按点名的来:** + +- 说「我本机的 / 我自己的 / 外部浏览器 / 我的 Chrome」→ 走 **L3-external**。先按 + Prerequisites 验 CDP 就绪;没就绪就告诉他启动命令并等他,**不要因为「内置浏览器也能做」 + 就擅自改用内置的** +- 说「内置浏览器」或没点名 → 走 **L3 内置浏览器**(默认,零前置条件) +- 拿不准他指哪个 → 问一句,别猜 + +⚠️ 无论走哪条,都要让用户能分辨你实际用了哪个。不要用「本地浏览器」「本机的受管浏览器」 +「已启动本地浏览器」这种两边都像的说法——用户要的和你给的不是一回事时,措辞必须让他一眼看出来。 + +### 分层策略总结 | Layer | Use case | Primary tool | Token cost | |-------|----------|--------------|------------| | L1 | Public, static | `WebFetch` | Low | | L2 | JS-heavy, long articles, token savings | `Bash curl r.jina.ai` | **Lowest** (Markdown pre-cleaned) | | **L3** | **登录态导航、交互与取文 (PRIMARY)** | **内置受管浏览器(BrowserManage / BrowserAct / BrowserSnapshot / BrowserScript)** | Medium | +| L3-external | **用户点名要用他自己的浏览器**;或需要他本人的登录态而 `BrowserImport` 不可用 | `Bash + Python Playwright connect_over_cdp`(见 references/cdp-browser.md) | Medium | **Default priority**: L1 for simple public pages → L2 for heavy → **L3 for login-gated(含正文与站内接口取数)**。 +> L3-external 不在这条默认排序里,因为它不由「能力够不够」决定,而由**用户点名**决定: +> 用户要他自己那个浏览器时直接走它,哪怕内置浏览器也做得到。判据见上方 +> 「两个浏览器,按用户意图选」。 + ## Supported Sites Matrix | Site | Recommended Layer | Notes | diff --git a/skills/web-access/references/cdp-browser.md b/skills/web-access/references/cdp-browser.md new file mode 100644 index 0000000..40cdd14 --- /dev/null +++ b/skills/web-access/references/cdp-browser.md @@ -0,0 +1,330 @@ +# CDP Browser Access — Login-Gated Sites Manual + +Detailed recipes for accessing sites that require the user's login session, via Chrome DevTools Protocol (CDP) + Python Playwright. + +**Precondition**: Chrome is already running with `--remote-debugging-port=9222` and the user has manually logged in to the target sites. See the main SKILL.md `Prerequisites` section for the launch command. + +--- + +## Why CDP attach, not headless + +| Approach | Login state | Anti-bot | Speed | Cost | +|----------|-------------|----------|-------|------| +| Headless Playwright (new context) | ❌ Empty cookies | ❌ Flagged as bot | Slow cold start | Re-login pain | +| `playwright.chromium.launch(headless=False)` | ❌ Fresh profile | ⚠ Sometimes flagged | Slow | Same | +| **CDP attach (`connect_over_cdp`)** | ✅ User's real cookies | ✅ Looks human | Instant | Zero friction | + +**Rule**: For any login-gated site, always attach to the user's running Chrome. + +--- + +## Core Template + +Every CDP script follows this shape: + +```python +from playwright.sync_api import sync_playwright + +def fetch_with_cdp(url: str, wait_selector: str | None = None) -> str: + """Attach to user's Chrome via CDP, fetch URL, return HTML.""" + with sync_playwright() as p: + browser = p.chromium.connect_over_cdp("http://localhost:9222") + # browser.contexts[0] is the user's default context (with cookies) + context = browser.contexts[0] + page = context.new_page() + try: + page.goto(url, wait_until="domcontentloaded", timeout=30000) + if wait_selector: + page.wait_for_selector(wait_selector, timeout=10000) + else: + page.wait_for_timeout(2000) # generic settle + return page.content() + finally: + page.close() + # DO NOT call browser.close() — that would close the user's Chrome! + +if __name__ == "__main__": + html = fetch_with_cdp("https://example.com") + print(html[:1000]) +``` + +**Critical**: Never call `browser.close()` when using CDP attach — you'd kill the user's Chrome. Only close the page you opened. + +--- + +## Site Recipes + +### 小红书 (xiaohongshu.com) + +```python +from playwright.sync_api import sync_playwright +from bs4 import BeautifulSoup + +NOTE_URL = "https://www.xiaohongshu.com/explore/XXXXXXXX" + +with sync_playwright() as p: + browser = p.chromium.connect_over_cdp("http://localhost:9222") + page = browser.contexts[0].new_page() + page.goto(NOTE_URL, wait_until="domcontentloaded") + page.wait_for_selector("#detail-title", timeout=10000) + page.wait_for_timeout(1500) # let images/comments load + html = page.content() + page.close() + +soup = BeautifulSoup(html, "html.parser") +title = (soup.select_one("#detail-title") or {}).get_text(strip=True) if soup.select_one("#detail-title") else None +desc = (soup.select_one("#detail-desc") or {}).get_text(" ", strip=True) if soup.select_one("#detail-desc") else None +author = soup.select_one(".author-wrapper .username") +print("Title:", title) +print("Author:", author.get_text(strip=True) if author else None) +print("Desc:", desc) +``` + +**Selectors** (may drift over time — update if they fail): +- Title: `#detail-title` +- Description: `#detail-desc` +- Author: `.author-wrapper .username` +- Images: `.swiper-slide img` +- Comments: `.parent-comment .content` + +### B站 (bilibili.com) + +```python +from playwright.sync_api import sync_playwright +from bs4 import BeautifulSoup + +VIDEO_URL = "https://www.bilibili.com/video/BVxxxxxxxxx" + +with sync_playwright() as p: + browser = p.chromium.connect_over_cdp("http://localhost:9222") + page = browser.contexts[0].new_page() + page.goto(VIDEO_URL, wait_until="networkidle") + page.wait_for_timeout(2000) + html = page.content() + page.close() + +soup = BeautifulSoup(html, "html.parser") +print("Title:", soup.select_one("h1.video-title").get_text(strip=True) if soup.select_one("h1.video-title") else None) +print("UP:", soup.select_one(".up-name").get_text(strip=True) if soup.select_one(".up-name") else None) +print("Desc:", soup.select_one(".desc-info-text").get_text(" ", strip=True) if soup.select_one(".desc-info-text") else None) +``` + +**Tip**: For B站 evaluations, the [公开 API](https://api.bilibili.com/x/web-interface/view?bvid=XXXX) often returns JSON without needing CDP. Try it first: + +```bash +curl -s "https://api.bilibili.com/x/web-interface/view?bvid=BVxxxxxxxxx" | python3 -m json.tool +``` + +### 微博 (weibo.com) + +```python +WEIBO_URL = "https://weibo.com/u/1234567890" # or /detail/xxx + +# Same CDP template +# Selectors: +# .Feed_body_3R0rO .detail_wbtext_4CRf9 — post text +# .ALink_default_2ibt1 — user link +# article[aria-label="微博"] — each feed item +``` + +**Note**: Weibo uses React + heavy obfuscation. Selectors change frequently. If selectors fail, pipe the HTML through Jina for clean Markdown: + +```python +html = fetch_with_cdp(WEIBO_URL) +# Save to temp file, then: +import subprocess +result = subprocess.run( + ["curl", "-sL", f"https://r.jina.ai/{WEIBO_URL}"], + capture_output=True, text=True +) +print(result.stdout) +``` + +### 知乎 (zhihu.com) + +```python +ANSWER_URL = "https://www.zhihu.com/question/123/answer/456" + +# Selectors: +# h1.QuestionHeader-title — question title +# .RichContent-inner — answer body +# .AuthorInfo-name — author +``` + +Zhihu works with CDP but often also renders enough metadata server-side for Jina to work: + +```bash +curl -sL "https://r.jina.ai/https://www.zhihu.com/question/123/answer/456" +``` + +Try Jina first, fall back to CDP if content is truncated. + +### 飞书文档 (feishu.cn / larksuite.com) + +```python +DOC_URL = "https://xxx.feishu.cn/docs/xxx" + +# Feishu uses heavy virtualization — must scroll to load all content. +# Recipe: + +from playwright.sync_api import sync_playwright + +with sync_playwright() as p: + browser = p.chromium.connect_over_cdp("http://localhost:9222") + page = browser.contexts[0].new_page() + page.goto(DOC_URL, wait_until="domcontentloaded") + page.wait_for_selector(".docs-render-unit", timeout=15000) + + # Scroll to bottom repeatedly to load lazy content + last_height = 0 + for _ in range(20): + page.evaluate("window.scrollTo(0, document.body.scrollHeight)") + page.wait_for_timeout(800) + h = page.evaluate("document.body.scrollHeight") + if h == last_height: + break + last_height = h + + # Extract text + text = page.evaluate("() => document.body.innerText") + page.close() + +print(text) +``` + +### Twitter / X + +```python +TWEET_URL = "https://x.com/username/status/1234567890" + +# Selectors: +# article[data-testid="tweet"] — tweet container +# div[data-testid="tweetText"] — tweet text +# div[data-testid="User-Name"] — author +# a[href$="/analytics"] — view count anchor (next sibling has stats) +``` + +Twitter is aggressive with anti-bot. CDP attach usually works, but set a generous wait: + +```python +page.goto(url, wait_until="networkidle", timeout=45000) +page.wait_for_selector('article[data-testid="tweet"]', timeout=15000) +``` + +--- + +## Common Patterns + +### Pattern 1: Scroll to load lazy content + +```python +def scroll_to_bottom(page, max_steps=30, pause_ms=800): + last = 0 + for _ in range(max_steps): + page.evaluate("window.scrollTo(0, document.body.scrollHeight)") + page.wait_for_timeout(pause_ms) + h = page.evaluate("document.body.scrollHeight") + if h == last: + return + last = h +``` + +### Pattern 2: Screenshot a specific element + +```python +element = page.locator("article").first +element.screenshot(path="/tmp/article.png") +``` + +### Pattern 3: Extract structured data via JavaScript + +```python +data = page.evaluate("""() => { + const items = document.querySelectorAll('.list-item'); + return Array.from(items).map(el => ({ + title: el.querySelector('.title')?.innerText, + url: el.querySelector('a')?.href, + })); +}""") +print(data) +``` + +### Pattern 4: Fill a form and click + +```python +page.fill("input[name=q]", "search query") +page.click("button[type=submit]") +page.wait_for_load_state("networkidle") +``` + +### Pattern 5: Clean HTML via Jina after extraction + +When selectors are unreliable, dump the full page HTML and let Jina do the cleaning: + +```python +html = page.content() +# Save to file, serve via local HTTP, or just pipe the original URL: +import subprocess +clean_md = subprocess.run( + ["curl", "-sL", f"https://r.jina.ai/{url}"], + capture_output=True, text=True +).stdout +print(clean_md) +``` + +--- + +## Troubleshooting + +### `connect_over_cdp` fails with `ECONNREFUSED` + +Chrome is not running with remote debugging. Tell the user: +> "请先用下面的命令启动 Chrome: +> `/Applications/Google\\ Chrome.app/Contents/MacOS/Google\\ Chrome --remote-debugging-port=9222 --user-data-dir=\"${DESIRECORE_ROOT}/chrome-profile\"` +> 然后手动登录需要抓取的网站,再让我继续。" + +### `browser.contexts[0]` is empty + +Chrome was launched but no windows are open. Ask the user to open at least one tab and navigate anywhere. + +### Playwright not installed + +```bash +pip3 install playwright beautifulsoup4 +# No need for `playwright install` — we're attaching to existing Chrome, not downloading a new browser +``` + +### Site detects automation + +Despite CDP attach, some sites (Cloudflare-protected, Instagram) may still detect automation. Options: +1. Use Jina Reader instead (`curl -sL https://r.jina.ai/`) — often succeeds where Playwright fails +2. Ask the user to manually copy the visible content +3. Use the site's public API if available + +### Content is truncated + +The page uses virtualization or lazy loading. Apply Pattern 1 (scroll to bottom) before calling `page.content()`. + +### `page.wait_for_selector` times out + +The selector is stale — the site updated its DOM. Dump `page.content()[:5000]` and inspect manually, or fall back to Jina Reader. + +--- + +## Security Notes + +- **Never log or print cookies** from `context.cookies()` even during debugging +- **Never extract and store** the user's session tokens to files +- **Never use the CDP session** to perform writes (post, comment, like) unless the user explicitly requested it +- The `${DESIRECORE_ROOT}/chrome-profile` directory contains the user's credentials — treat it as sensitive +- If the user asks to "log in automatically", refuse and explain they must log in manually in the Chrome window; the skill only reads already-authenticated sessions + +--- + +## When NOT to use CDP + +- **Public static sites** → use L1 `WebFetch`, it's faster +- **Heavy SPAs without login walls** → use L2 Jina Reader, it's cheaper on tokens +- **You need thousands of pages** → CDP is not built for scale; look into proper scrapers + +CDP is specifically the "right tool" for: **small number of pages + login required + human-like behavior needed**.