mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-07-12 12:37:32 +00:00
7522b0203409334511d8e0cd20ccb20344ff9ad7
17 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Katsoragi
|
7522b02034 |
fix(parser): parse Gemma 3/4 custom tool calling tokens (#5033)
* fix: parse Gemma 3/4 custom tool calling tokens in parser * test: cover Gemma tool call parsing --------- Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com> |
||
|
botinate
|
69b9bb0869 |
fix(agent): execute fenced tool calls with inline args and route bare email tool names (#3681)
* fix(agent): execute fenced tool calls with inline args and bare email tool names
Two bugs made local (Ollama) models unable to use email tools, leaving
raw fences like ```list_email_accounts {}``` in the chat:
1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a
tool call with args on the same line ("```list_email_accounts {}")
never matched and was never executed. The fence now matches with
optional spaces/newline after the tag.
2. Even when parsed, bare email tool names had no dispatch branch in
tool_execution.py and fell through to "Unknown tool type". They now
route to the email MCP server as mcp__email__<name>, matching how
function_call_to_tool_block already maps them for native callers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(security): block all bare email tool names for non-admins; harden fence-tag regex
Review follow-up on #3681 (thanks @vgalin):
1. Routing bare email names made 10 of the 14 email tools executable by
non-admin owners — is_public_blocked_tool() runs on the bare name
before dispatch, and NON_ADMIN_BLOCKED_TOOLS only listed 4. Define the
full email tool set once (BUILTIN_EMAIL_TOOLS in tool_security.py) and
derive the blocklist, the fence tags (TOOL_TAGS), the bare-name
dispatch, and the native-call mapping from it so they can't drift.
This also fixes 4 tools (search_emails, draft_email, draft_email_reply,
ai_draft_email_reply) that were missing from the old tool_schemas copy
and therefore unreachable even for native function-calling models.
2. The relaxed fence regex from the previous commit could prefix-match
longer fence tags: ```python3 parsed as tool "python" with content
"3\nprint(...)" and executed as code. Add a (?![\w-]) boundary after
the tag.
Tests: test_public_agent_policy_blocks_sensitive_tools now covers all 14
bare email names + the mcp__email__ form; new tests/test_fenced_inline_args.py
pins inline-args parsing, the python3/hyphenated-tag non-matches, and
strip/parse display mirroring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(security): gate bare and mcp-qualified email names together; stop executing Markdown info strings
Review follow-up on #3681 (thanks @RaresKeY):
1. P1: execute_tool_block() checked disabled_tools / the turn ToolPolicy
only against the incoming block name, then the bare-email branch
qualified it to mcp__email__<name> and called the MCP manager. Plan
mode and the MCP settings toggle write the QUALIFIED name into the
denylist, so a bare fence like ```list_emails``` sailed past a
mcp__email__list_emails entry. Both gates now match on both
spellings (bare <-> mcp__email__-qualified), in either direction.
2. P2: the relaxed fence regex accepted arbitrary same-line text after
a recognized tag, which made ordinary Markdown info strings
executable: ```python title="example.py" ran as a python tool call.
Same-line content now only counts as tool input when it starts with
{ or [ (JSON args); anything else leaves the fence as display text,
and strip_tool_blocks mirrors that (the fence stays visible).
Tests: disabled-tools alias regression (qualified entry blocks bare
name and vice versa, never reaching the MCP manager), ToolPolicy alias
regression, python/bash title="..." non-execution + display retention,
and inline JSON-array args still parsing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(security): reject brace-style fence metadata; cover the full email set in the friendly toggle
Review follow-up round 3 on #3681 (thanks @RaresKeY):
1. Brace-style fence metadata no longer executes. The previous narrowing
still treated any same-line {/[ after a recognized tag as tool input,
so ```bash {title="setup"} ran as a bash call. The fence header is now
captured separately and judged by one predicate shared between
parse_tool_blocks and strip_tool_blocks (_fenced_tool_call), so the
execute and display decisions can't disagree: same-line content only
counts as inline args when the tag is NOT a code tag (bash/python
never take same-line args — that text is Markdown fence attributes)
AND the inline text (plus any continuation lines) parses as standalone
JSON. ```bash {title="setup"}, ```python {"title":"example.py"} and
```list_emails {title="x"} all stay visible and inert.
2. The friendly `disable_tool email` toggle covered 3 of the 14 email
tools (mcp__email__{list_emails,read_email,send_email}); the other
bare aliases this PR routes stayed executable after an operator
disabled email. The alias now derives from BUILTIN_EMAIL_TOOLS in
BOTH spellings — bare (function-schema hiding, bare-fence dispatch)
and mcp__email__* (MCP schema hiding, qualified runtime blocks) —
so the toggle and the runtime gate can't drift apart.
Tests: brace/bracket metadata regressions for parse and strip symmetry
(code tags, invalid-JSON inline on a JSON tool, multi-line inline JSON
still parsing), and disable_tool/enable_tool email covering all 14 names
in both spellings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(email): close remaining email-tool registry drift; classify every email tool for plan mode
Deep self-review follow-up on #3681. Three review rounds each found another
hand-maintained copy of the email tool list that had drifted; this commit
hunts down ALL remaining copies and pins them to BUILTIN_EMAIL_TOOLS.
The same 5 tools (search_emails, draft_email, draft_email_reply,
ai_draft_email_reply, download_attachment) were missing from every
advertising surface, so they were dispatchable but never offered:
- FUNCTION_TOOL_SCHEMAS: native function-calling models never saw them
(the round-1 fix covered dispatch only); schemas added, mirroring the
email server's inputSchema definitions.
- TOOL_SECTIONS: fenced-block models were never told about them; prompt
sections added.
- tool_index: absent from the RAG embedding registry (never retrievable),
the email keyword hints, and the scheduled assistant's always-available
set — the latter two now derive from BUILTIN_EMAIL_TOOLS.
- agent_loop._DOMAIN_TOOL_MAP["email"], tool_policy._COMMON_TOOL_NAMES,
the assistant tool-selector UI groups (assistant.js), and the default
Assistant crew seed (task_scheduler) now derive from / cover the set.
Plan mode now classifies every email tool explicitly:
- list_email_accounts and search_emails join PLAN_MODE_READONLY_TOOLS.
Without this, list_email_accounts sat in the plan-mode bare denylist
(schema-derived) while its qualified form passed the MCP read-only
filter — and the round-2 bare/qualified alias gate would have blocked
the qualified call too, regressing read-only email discovery in plan
mode.
- draft_email, draft_email_reply, ai_draft_email_reply, and
download_attachment join the fail-closed mutator backstop (drafts
create documents; download_attachment writes to disk).
Tests: tests/test_email_registry_sync.py pins every registry (including
the email server source and assistant.js) to BUILTIN_EMAIL_TOOLS and
asserts the plan-mode partition, so the next email tool can't drift; a
parse/strip mirror grid covers 192 fence shapes (tag x header x body)
asserting executed <=> stripped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: move the email alias rule into tool_security; extract the assistant seed constant
Code-quality pass over the PR's own changes:
- The bare<->qualified email aliasing rule lived inline in the generic
dispatcher (_execute_tool_block_impl). It is policy knowledge, so it
moves next to BUILTIN_EMAIL_TOOLS as email_tool_policy_names(); the
dispatcher just consumes it, and the rule gets its own unit test
(including the mcp__email__<not-a-tool> and mcp__other__ non-alias
cases).
- The default Assistant's enabled_tools list was an inline literal
inside the CrewMember seed, and its registry-sync test asserted a
source-code substring. Extracted to DEFAULT_ASSISTANT_ENABLED_TOOLS
so the test imports and checks the actual value.
- _fenced_tool_call return type tightened to Optional[Tuple[str, str]].
No behavior change; suite green (3295 passed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* revert: move the email registry consolidation to a follow-up PR
Per review feedback on scope, this PR stays narrow: fenced inline-args
parsing, bare email tool routing, and the directly required safety
gates. This commit reverts the registry/advertising consolidation from
|
||
|
nopoz
|
fbe3a0d73b |
fix(security): prevent ReDoS in XML and args tool-call parsers (#4941)
* fix(security): prevent ReDoS in XML and args tool-call parsers
Four py/polynomial-redos sinks in tool_parsing.py ran lazy/greedy regexes over
untrusted model output (tool-call markup is attacker-influenced via prompt
injection). When the closing delimiter was absent, each rescanned to
end-of-string from every opener -> O(n^2):
- args => { ... } in _parse_tool_call_block: greedy \{([\s\S]*)\} restarted
from every `args:{` opener. Now finds the opener once and takes through the
last `}` (rfind) — equivalent capture, O(n).
- _XML_INVOKE_RE: lazy <invoke ...>([\s\S]*?)</invoke>. Now _iter_xml_invoke
pairs each opener with the first reachable </invoke> and stops when none is.
- _XML_DIRECT_TOOL_RE and the <tag>([\s\S]*?)</\1> param scan in
_parse_tool_code_block: lazy backreference patterns. Now _iter_backref_blocks
pairs each opener with the nearest matching closer and memoizes tag names
with no remaining closer, so an opener flood stays O(n).
All four are output-equivalent to the originals on well-formed tool-call markup;
the lazy patterns remain defined (still re-exported via agent_tools) but no
longer drive a finditer over untrusted text. Adds tests/test_redos_xml_tool_parsers.py
pinning correctness and bounding the opener-flood inputs (old paths took 4-15s).
* fix(security): harden invoke-parameter and distinct-name tag scans
Forward-only the two residual ReDoS paths in the XML/tool parsers that the
outer-delimiter fix left quadratic:
- _parse_xml_invoke parsed <parameter> with _XML_PARAM_RE.finditer, so a
closed <invoke> body full of unclosed <parameter> openers rescanned the
body from every opener (O(n^2), ~11s at 8k openers). Now scans forward-only
via _iter_named_blocks, factored out of _iter_xml_invoke.
- _iter_backref_blocks only memoized repeated missing tag names; a flood of
distinct unclosed names searched the suffix once per name (O(n^2)). It now
indexes every closer by name in one linear pass and binary-searches per
opener (O(n log n)). Covers the direct and tool_code backref scans.
Output-equivalent to the prior scanners (200k randomized trials match the
memoized version for both the direct ci=True and tool_code ci=False configs).
Adds regressions for the closed-invoke parameter flood and the distinct-name
floods (45k openers now run in ~0.05s, were 5-6s).
|
||
|
nopoz
|
c098355778 |
fix(security): prevent ReDoS in LLM-output tool/think parsers (#4704)
* fix(security): prevent ReDoS in LLM-output tool/think parsers The regexes that parse untrusted model output in text_helpers.py and tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole response, they degrade to O(n^2) when the closing delimiter is absent: the engine rescans to end-of-string from every opener. Model output is untrusted, so a prompt-injected or malicious model can stall the agent loop with many unclosed openers (measured ~25s on a 60KB <thought flood). - text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the Gemma <|channel>...<channel|> subs when no <channel|> closer is present. - tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check for their closing delimiter. With no closer the regex cannot match, so skipping is equivalent; only the wasted O(n^2) rescan is removed. Resolves CodeQL py/polynomial-redos #230, #231, #232, #233, #235, #236, #524. The _XML_OPEN_TOOL_CALL_RE alerts (#234, #477) are false positives (its greedy [\s\S]*\Z is linear) and left untouched. * fix(security): close ReDoS gaps in tool/think parsers from review Addresses two review findings on the closer-guard approach: - Whole-string "closer exists?" checks were bypassable: a stale closer before an opener flood, or a closer with no reachable inner `}`, kept the guard true while every opener still rescanned to end-of-string (O(n^2)). Replace the substring guards with `_iter_delimited`, a forward-only scan that pairs each opener with a *later* closer and stops once none is reachable (O(n)). `parse_tool_blocks` and `strip_tool_blocks` (via `_strip_delimited`) both use it for the [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats. Verified equivalent to the original regexes on well-formed inputs. - `<thought([^>]*)>` dropped the tag-name boundary and corrupted unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`: the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*` overlap) while restoring the boundary; capture is byte-for-byte identical for real `<thought ...>` openers. Adds regressions for stale-closer-before-opener, closer-present-without- inner-brace, and the <thoughtful>/<thoughts> passthrough. * fix(security): close Gemma channel ReDoS guard flagged in review vdmkenny noted the same bypassable whole-string guard remained in text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma thought/response channel subs. A stale `<channel|>` before a `<|channel>thought` opener flood keeps the guard true while every opener still rescans to end-of-string (measured ~7.3s at 4k openers). Replace it with `_sub_delimited`, the same forward-only scan used for the tool-call parsers: pair each opener with a later closer, stop when none is reachable (O(n)). Verified output-equivalent to the original capture regexes on well-formed multi-channel inputs; the stale-closer case now runs in <2ms. Adds a regression for stale-closer-before-opener on the Gemma path. * fix(security): harden strip_think() think-tag ReDoS flagged in review The earlier fixes hardened normalize_thinking_markup and the delimiter scanners, but the production entrypoint strip_think() still ran _THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag _THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to end-of-string from every opener on a "many openers, no closer" flood. On the prior head, malformed `<think` / `<thinking` / `<thought` floods took 6-14s through strip_think(). The shipped `<thought>` normalization had the same residual: the single-opener case was linear but an opener flood was still O(n^2) (~4.4s). - Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing forward-only _sub_delimited scan (pair each opener with the first reachable closer, stop when none is reachable). One pass collapses sequential and nested blocks as before. - Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`) so a no-`>` opener flood can't drive a single match attempt to end-of-string. Identical capture for well-formed think/thought tags. - email_helpers._strip_think: compute had_think from the single linear _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which had the same O(n^2) on the email reply/summary/extraction paths. All flood variants now finish in <10ms (were 6-14s). Output verified byte-for-byte identical to the prior implementation over a 34-case corpus (nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds strip_think() timing regressions for malformed openers, opener floods (all three tag names), the closed-opener flood, and the malformed-closer flood. * docs: trim verbose comments in think-tag ReDoS fix |
||
|
Dividesbyzer0
|
2e16394b41 | fix(agent): parse misfenced read_file calls (#4799) | ||
|
pewdiepie-archdaemon
|
993d504de3 | Clear remaining CodeQL path and parser alerts | ||
|
pewdiepie-archdaemon
|
fbdec22dcb | CodeQL hardening for cookbook sync | ||
|
pewdiepie-archdaemon
|
92daf4e560 | Cookbook launch and gallery upload fixes | ||
|
Kenny Van de Maele
|
cdae9879f2 |
feat(agent): add manage_bg_jobs tool to inspect and kill background bash jobs (#4577)
Detached bash jobs (#!bg) could be launched and auto-reported on completion, but the agent had no way to act on a running one: no on-demand output read and no kill (it blocked until the 1h max-runtime). bg_jobs had the pieces (_read_output, list_for_session, internal _kill) but none was exposed. Adds: - bg_jobs.kill(job_id): tears down the process tree, marks the job killed, and sets followed_up so the monitor does not also auto-continue a deliberate kill. - manage_bg_jobs registry tool with actions list / output / kill, scoped to the chat that launched the job (cross-session access reads as not found). - Wiring: TOOL_HANDLERS/TAGS, function schema, RAG index + keyword hints, parser name map, dispatch (threads session_id via _direct_fallback). Gated like bash (NON_ADMIN_BLOCKED_TOOLS; plan-mode mutator). - agent_loop: background-job intent regex maps to the files domain (and the tool joins _DOMAIN_TOOL_MAP[files]) so short commands like 'kill that job' are not dropped by the low-signal gate that skips tool retrieval. - bg launch message tells the model to call manage_bg_jobs itself for check/stop rather than printing raw tool syntax to the user. Tests: tests/test_bg_job_tools.py (kill semantics, per-chat scoping, actions, and the intent classifier). |
||
|
Dividesbyzer0
|
33c26bab88 | fix(agent): parse raw json web search calls (#4088) | ||
|
Lucas Daniel
|
0a324f20d2 |
fix(agent): stop treating illustrative Markdown fences as tool calls for native function-calling models (#3356)
* fix(agent): stop executing illustrative Markdown fences as tool calls for native function-calling models _resolve_tool_blocks fell back to the textual parse_tool_blocks() fenced-block parser whenever a model produced no native tool_calls, regardless of whether that model has a reliable native function-calling channel. Native models (GPT/Claude/Grok/Qwen3/DeepSeek-V, etc. - _is_api_model true) commonly write illustrative ```bash/```python/```json examples in guide-only prose; the fallback parser matched these and executed them as real commands, sometimes looping for several rounds as the model tried to clarify with more examples (#3222). Restrict the textual fenced-block fallback to non-native models, which rely on it as their only tool-invocation channel. Native models are trusted to use their structured tool_calls channel for real invocations; when they don't emit one, a bare fence in their response is prose, not an action. The native tool_calls path itself is untouched. This sits one layer below #3088's guide-only policy enforcement: that PR blocks tool exposure/execution on explicit no-tools requests, while this fixes the parser so ordinary illustrative fences are never misread as calls in the first place, on any turn. * fix(agent): gate only the fenced-example pattern for native models, preserve DSML/invoke recovery and persistence _resolve_tool_blocks previously short-circuited the entire textual parser (tool_blocks = [] if is_api_model else parse_tool_blocks(...)) for native function-calling models with no native tool_calls. That also dropped Patterns 2-5 (explicit [TOOL_CALL]/<invoke>/<tool_code>/DSML markup leaked into content as text), which are real calls a model couldn't emit on its structured channel (e.g. DeepSeek-V falling back to DSML), not illustrative examples. parse_tool_blocks/strip_tool_blocks now take a skip_fenced flag that gates ONLY Pattern 1 (the fenced ```bash/```python/```json block matcher). _resolve_tool_blocks passes skip_fenced=is_api_model so fenced examples stop being executed for native models while [TOOL_CALL]/<invoke>/<tool_code>/DSML stay fully active and recoverable. cleaned_round mirrors the same gate when persisting round text, so an illustrative fence that wasn't executed isn't stripped from saved/reloaded history either (it was streaming once and then disappearing on reload). |
||
|
Nicholai
|
33edc40eae |
fix: route misfenced web lookups to web tools
Fixes #3067 |
||
|
nubs
|
08e543d1ff | fix(tool-parsing): don't ship unconvertible <invoke> fence content to the code executor (#2926) | ||
|
Afonso Coutinho
|
3175d7ca21 | fix: tool-block parsing crashes on a non-string input (#1628) | ||
|
Tatlatat
|
acfdcf346c |
fix(agent): map native google_search and surface empty rounds
Models (notably Gemini) emit a native 'google_search' function call, but the agent loop had no mapping for it, so the call failed to convert, the round produced 0 chars and 0 tool blocks, and generation died silently — the web client hung on 'waiting for first token' with no error (also #443). - Map google_search / google_search_retrieval / google_search_grounding to the web_search tool, and read Gemini's 'queries' array (falling back to 'query'). - In stream_agent_loop, when a round yields no response text and no tool events, emit a visible fallback message instead of leaving the user hanging. - Give the unknown-tool execution branch an explicit exit_code=1 so the failure is logged as an error rather than 'n/a'. Unknown/unconvertible tool names still return None (unchanged) so they are dropped safely rather than executed. Added tests covering the google_search mapping, the queries array, and unknown/invalid-JSON returning None. |
||
|
Rifqi Akram
|
5b1e56407b |
Add SSRF-guarded web fetch agent tool
* feat(web-fetch): add web_fetch tool to read a specific URL's content * test(web-fetch): add SSRF coverage and fail closed on empty DNS resolution Add explicit SSRF regression tests for the web_fetch path covering loopback, private LAN ranges, link-local/metadata, IPv6 private/local, redirect-into-private, and unsupported schemes. Harden _public_http_url to fail closed when a hostname resolves to no addresses. |
||
|
pewdiepie-archdaemon
|
e5c99a5eee | Odysseus v1.0 |