desirecore/market
2
0
Fork 0
mirror of https://git.openapi.site/https://github.com/desirecore/market.git synced 2026-09-06 17:23:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(web-access): v3.0 内置浏览器能力面同步——删除 Python Playwright 回落（S31） (#85)

Browse Source 
## 变更内容 / What

浏览器升级计划 S31 最终验收信号：web-access 技能与内置受管浏览器新能力面对齐，**删除「用户手工启动调试 Chrome +
Python Playwright」回落路径**。

- **删除回落路径**：Prerequisites: Chrome CDP Setup、Layer 3 CDP Browser（Python
Playwright 模板）、Installation Note（pip install
playwright）整段删除；references/cdp-browser.md 文件删除；jina-reader.md 的 CDP 引用改为
page.extract-text
- **订正陈旧断言**：
- 「没有批量取文通道」→ BrowserSnapshot mode:text /
page.extract-text（maxBytes/cursor 分页，超出截断给 nextCursor）
- 「page.evaluate 基本不可用」→ 返回真实值（expression/awaitPromise，超预算截断标
truncated）；仍走人工闸门
- 「截图前必须 tab.activate / 串行截图 / BROWSER_TAB_HOST_NOT_FOUND」→ S36 订正：Agent
单标签会话免 activate；多标签后台 tab 秒级报 BROWSER_VIEWPORT_UNAVAILABLE；命令超时只 stop 不
close，标签页可重试
  - 「accessibility 超限即失败」→ 尊重 depth + maxBytes 截断翻页（S8）
  - 「只有整页截图」→ clip{x,y,width,height,scale≤4} + captureBeyondViewport
- **provides.tools 加 BrowserScript**（code-mode；信任级别等同 Bash）
- **新增选用规则（D4 唯一约束机制）**：反检测站点一律优先 input.*（#1808 输入拟真 +
身份一致性）；page.element 写类仅用于表单批量填充等站点不检测场景；JS 直调 el.click() 为禁止回退
- **新增 fetch.browser 配方**：page.evaluate 页面上下文跑 fetch（带 origin Cookie、同
origin、受 Grant origins 约束）——登录态取站内接口的正解
- **能力速查**：page.element 九 op / page.wait 九 until / inline wait 块 / loc=
方言 / BrowserScript / 跨源 iframe 快照（S35）
- **版本** 2.2.1 → 3.0.0（删除回落层为 breaking）；source_hash
重算；required_client_version 维持 10.0.98（新能力在正文标注 10.0.112+）

## Why

v2.x 时代回落路径存在的每一条理由（无批量取文、evaluate 不可用、截图必须串行 activate）均已被
S2–S14/S35/S36 覆盖；文档继续引导用户手工起调试 Chrome 会误导新 Agent 走已废弃路径。

双语同步修改（SKILL.md / SKILL.zh-CN.md heading 数一致，i18n-validate 通过）。

- [x] CLA
This commit is contained in:
yi-ge
2026-08-17 01:25:46 -04:00
committed by GitHub
parent 43239669e5
commit 0d4d7b208b
5 changed files with 482 additions and 868 deletions
Download Patch File Download Diff File Expand all files Collapse all files

330
skills/web-access/references/cdp-browser.md
View File

@@ -1,330 +0,0 @@
# CDP Browser Access — Login-Gated Sites Manual
Detailed recipes for accessing sites that require the user's login session, via Chrome DevTools Protocol (CDP) + Python Playwright.
**Precondition**: Chrome is already running with `--remote-debugging-port=9222` and the user has manually logged in to the target sites. See the main SKILL.md `Prerequisites` section for the launch command.
---
## Why CDP attach, not headless
| Approach | Login state | Anti-bot | Speed | Cost |
|----------|-------------|----------|-------|------|
| Headless Playwright (new context) | ❌ Empty cookies | ❌ Flagged as bot | Slow cold start | Re-login pain |
| `playwright.chromium.launch(headless=False)` | ❌ Fresh profile | ⚠ Sometimes flagged | Slow | Same |
| **CDP attach (`connect_over_cdp`)** | ✅ User's real cookies | ✅ Looks human | Instant | Zero friction |
**Rule**: For any login-gated site, always attach to the user's running Chrome.
---
## Core Template
Every CDP script follows this shape:
```python
from playwright.sync_api import sync_playwright
def fetch_with_cdp(url: str, wait_selector: str | None = None) -> str:
"""Attach to user's Chrome via CDP, fetch URL, return HTML."""
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp("http://localhost:9222")
# browser.contexts[0] is the user's default context (with cookies)
context = browser.contexts[0]
page = context.new_page()
try:
page.goto(url, wait_until="domcontentloaded", timeout=30000)
if wait_selector:
page.wait_for_selector(wait_selector, timeout=10000)
else:
page.wait_for_timeout(2000) # generic settle
return page.content()
finally:
page.close()
# DO NOT call browser.close() — that would close the user's Chrome!
if __name__ == "__main__":
html = fetch_with_cdp("https://example.com")
print(html[:1000])
```
**Critical**: Never call `browser.close()` when using CDP attach — you'd kill the user's Chrome. Only close the page you opened.
---
## Site Recipes
### 小红书 (xiaohongshu.com)
```python
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
NOTE_URL = "https://www.xiaohongshu.com/explore/XXXXXXXX"
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp("http://localhost:9222")
page = browser.contexts[0].new_page()
page.goto(NOTE_URL, wait_until="domcontentloaded")
page.wait_for_selector("#detail-title", timeout=10000)
page.wait_for_timeout(1500) # let images/comments load
html = page.content()
page.close()
soup = BeautifulSoup(html, "html.parser")
title = (soup.select_one("#detail-title") or {}).get_text(strip=True) if soup.select_one("#detail-title") else None
desc = (soup.select_one("#detail-desc") or {}).get_text(" ", strip=True) if soup.select_one("#detail-desc") else None
author = soup.select_one(".author-wrapper .username")
print("Title:", title)
print("Author:", author.get_text(strip=True) if author else None)
print("Desc:", desc)
```
**Selectors** (may drift over time — update if they fail):
- Title: `#detail-title`
- Description: `#detail-desc`
- Author: `.author-wrapper .username`
- Images: `.swiper-slide img`
- Comments: `.parent-comment .content`
### B站 (bilibili.com)
```python
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
VIDEO_URL = "https://www.bilibili.com/video/BVxxxxxxxxx"
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp("http://localhost:9222")
page = browser.contexts[0].new_page()
page.goto(VIDEO_URL, wait_until="networkidle")
page.wait_for_timeout(2000)
html = page.content()
page.close()
soup = BeautifulSoup(html, "html.parser")
print("Title:", soup.select_one("h1.video-title").get_text(strip=True) if soup.select_one("h1.video-title") else None)
print("UP:", soup.select_one(".up-name").get_text(strip=True) if soup.select_one(".up-name") else None)
print("Desc:", soup.select_one(".desc-info-text").get_text(" ", strip=True) if soup.select_one(".desc-info-text") else None)
```
**Tip**: For B站 evaluations, the [公开 API](https://api.bilibili.com/x/web-interface/view?bvid=XXXX) often returns JSON without needing CDP. Try it first:
```bash
curl -s "https://api.bilibili.com/x/web-interface/view?bvid=BVxxxxxxxxx" | python3 -m json.tool
```
### 微博 (weibo.com)
```python
WEIBO_URL = "https://weibo.com/u/1234567890" # or /detail/xxx
# Same CDP template
# Selectors:
# .Feed_body_3R0rO .detail_wbtext_4CRf9 — post text
# .ALink_default_2ibt1 — user link
# article[aria-label="微博"] — each feed item
```
**Note**: Weibo uses React + heavy obfuscation. Selectors change frequently. If selectors fail, pipe the HTML through Jina for clean Markdown:
```python
html = fetch_with_cdp(WEIBO_URL)
# Save to temp file, then:
import subprocess
result = subprocess.run(
["curl", "-sL", f"https://r.jina.ai/{WEIBO_URL}"],
capture_output=True, text=True
)
print(result.stdout)
```
### 知乎 (zhihu.com)
```python
ANSWER_URL = "https://www.zhihu.com/question/123/answer/456"
# Selectors:
# h1.QuestionHeader-title — question title
# .RichContent-inner — answer body
# .AuthorInfo-name — author
```
Zhihu works with CDP but often also renders enough metadata server-side for Jina to work:
```bash
curl -sL "https://r.jina.ai/https://www.zhihu.com/question/123/answer/456"
```
Try Jina first, fall back to CDP if content is truncated.
### 飞书文档 (feishu.cn / larksuite.com)
```python
DOC_URL = "https://xxx.feishu.cn/docs/xxx"
# Feishu uses heavy virtualization — must scroll to load all content.
# Recipe:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp("http://localhost:9222")
page = browser.contexts[0].new_page()
page.goto(DOC_URL, wait_until="domcontentloaded")
page.wait_for_selector(".docs-render-unit", timeout=15000)
# Scroll to bottom repeatedly to load lazy content
last_height = 0
for _ in range(20):
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
page.wait_for_timeout(800)
h = page.evaluate("document.body.scrollHeight")
if h == last_height:
break
last_height = h
# Extract text
text = page.evaluate("() => document.body.innerText")
page.close()
print(text)
```
### Twitter / X
```python
TWEET_URL = "https://x.com/username/status/1234567890"
# Selectors:
# article[data-testid="tweet"] — tweet container
# div[data-testid="tweetText"] — tweet text
# div[data-testid="User-Name"] — author
# a[href$="/analytics"] — view count anchor (next sibling has stats)
```
Twitter is aggressive with anti-bot. CDP attach usually works, but set a generous wait:
```python
page.goto(url, wait_until="networkidle", timeout=45000)
page.wait_for_selector('article[data-testid="tweet"]', timeout=15000)
```
---
## Common Patterns
### Pattern 1: Scroll to load lazy content
```python
def scroll_to_bottom(page, max_steps=30, pause_ms=800):
last = 0
for _ in range(max_steps):
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
page.wait_for_timeout(pause_ms)
h = page.evaluate("document.body.scrollHeight")
if h == last:
return
last = h
```
### Pattern 2: Screenshot a specific element
```python
element = page.locator("article").first
element.screenshot(path="/tmp/article.png")
```
### Pattern 3: Extract structured data via JavaScript
```python
data = page.evaluate("""() => {
const items = document.querySelectorAll('.list-item');
return Array.from(items).map(el => ({
title: el.querySelector('.title')?.innerText,
url: el.querySelector('a')?.href,
}));
}""")
print(data)
```
### Pattern 4: Fill a form and click
```python
page.fill("input[name=q]", "search query")
page.click("button[type=submit]")
page.wait_for_load_state("networkidle")
```
### Pattern 5: Clean HTML via Jina after extraction
When selectors are unreliable, dump the full page HTML and let Jina do the cleaning:
```python
html = page.content()
# Save to file, serve via local HTTP, or just pipe the original URL:
import subprocess
clean_md = subprocess.run(
["curl", "-sL", f"https://r.jina.ai/{url}"],
capture_output=True, text=True
).stdout
print(clean_md)
```
---
## Troubleshooting
### `connect_over_cdp` fails with `ECONNREFUSED`
Chrome is not running with remote debugging. Tell the user:
> "请先用下面的命令启动 Chrome：
> `/Applications/Google\\ Chrome.app/Contents/MacOS/Google\\ Chrome --remote-debugging-port=9222 --user-data-dir=\"${DESIRECORE_ROOT}/chrome-profile\"`
> 然后手动登录需要抓取的网站，再让我继续。"
### `browser.contexts[0]` is empty
Chrome was launched but no windows are open. Ask the user to open at least one tab and navigate anywhere.
### Playwright not installed
```bash
pip3 install playwright beautifulsoup4
# No need for `playwright install` — we're attaching to existing Chrome, not downloading a new browser
```
### Site detects automation
Despite CDP attach, some sites (Cloudflare-protected, Instagram) may still detect automation. Options:
1. Use Jina Reader instead (`curl -sL https://r.jina.ai/<url>`) — often succeeds where Playwright fails
2. Ask the user to manually copy the visible content
3. Use the site's public API if available
### Content is truncated
The page uses virtualization or lazy loading. Apply Pattern 1 (scroll to bottom) before calling `page.content()`.
### `page.wait_for_selector` times out
The selector is stale — the site updated its DOM. Dump `page.content()[:5000]` and inspect manually, or fall back to Jina Reader.
---
## Security Notes
- **Never log or print cookies** from `context.cookies()` even during debugging
- **Never extract and store** the user's session tokens to files
- **Never use the CDP session** to perform writes (post, comment, like) unless the user explicitly requested it
- The `${DESIRECORE_ROOT}/chrome-profile` directory contains the user's credentials — treat it as sensitive
- If the user asks to "log in automatically", refuse and explain they must log in manually in the Chrome window; the skill only reads already-authenticated sessions
---
## When NOT to use CDP
- **Public static sites** → use L1 `WebFetch`, it's faster
- **Heavy SPAs without login walls** → use L2 Jina Reader, it's cheaper on tokens
- **You need thousands of pages** → CDP is not built for scale; look into proper scrapers
CDP is specifically the "right tool" for: **small number of pages + login required + human-like behavior needed**.