The grep tool bypassed the sensitive-file deny-list that read_file,
write_file, and edit_file all respect. Two code paths fixed:
1. ripgrep path: adds --glob exclusion patterns for each entry in
_SENSITIVE_FILE_PATTERNS (id_rsa, known_hosts, authorized_keys, etc.)
2. Pure-Python os.walk fallback: checks _is_sensitive_path() before
opening each file, skipping files that match the deny-list
Fixes#5011
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Slice 2c of the route-domain reorganization (#4082/#4071, per
specs/architecture-runtime-inventory.md §6.3). Moves memory_routes.py into
routes/memory/, leaving a backward-compat sys.modules shim at the old path.
Pure file reorganization, no behavior change.
The shim uses sys.modules replacement (same pattern as the merged gallery
#4903 and research #4975 slices) so that `import routes.memory_routes`,
`from routes.memory_routes import X`, `importlib.import_module(...)`, and
the `import ... as mr` + `monkeypatch.setattr(mr, ...)` pattern used by
test_memory_routes_session_owner.py / test_memory_owner_isolation.py all
operate on the same module object the application uses.
The canonical module does NOT depend on the shim — routes/memory/
memory_routes.py imports only from services/, core/, src/, and stdlib (zero
internal routes/ coupling).
Four source-introspection test sites repointed to the new canonical path:
- test_direct_upload_limits.py
- test_upload_limits_centralized.py (two dict keys)
- test_vision_owner_scope.py
Adds tests/test_memory_routes_shim.py to pin the sys.modules shim contract
(legacy and canonical paths resolve to the same module object; monkeypatch
via legacy alias reaches the canonical module).
Verified: compileall clean; full suite 4219 passed, 3 skipped.
* fix(agent): execute fenced tool calls with inline args and bare email tool names
Two bugs made local (Ollama) models unable to use email tools, leaving
raw fences like ```list_email_accounts {}``` in the chat:
1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a
tool call with args on the same line ("```list_email_accounts {}")
never matched and was never executed. The fence now matches with
optional spaces/newline after the tag.
2. Even when parsed, bare email tool names had no dispatch branch in
tool_execution.py and fell through to "Unknown tool type". They now
route to the email MCP server as mcp__email__<name>, matching how
function_call_to_tool_block already maps them for native callers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(security): block all bare email tool names for non-admins; harden fence-tag regex
Review follow-up on #3681 (thanks @vgalin):
1. Routing bare email names made 10 of the 14 email tools executable by
non-admin owners — is_public_blocked_tool() runs on the bare name
before dispatch, and NON_ADMIN_BLOCKED_TOOLS only listed 4. Define the
full email tool set once (BUILTIN_EMAIL_TOOLS in tool_security.py) and
derive the blocklist, the fence tags (TOOL_TAGS), the bare-name
dispatch, and the native-call mapping from it so they can't drift.
This also fixes 4 tools (search_emails, draft_email, draft_email_reply,
ai_draft_email_reply) that were missing from the old tool_schemas copy
and therefore unreachable even for native function-calling models.
2. The relaxed fence regex from the previous commit could prefix-match
longer fence tags: ```python3 parsed as tool "python" with content
"3\nprint(...)" and executed as code. Add a (?![\w-]) boundary after
the tag.
Tests: test_public_agent_policy_blocks_sensitive_tools now covers all 14
bare email names + the mcp__email__ form; new tests/test_fenced_inline_args.py
pins inline-args parsing, the python3/hyphenated-tag non-matches, and
strip/parse display mirroring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(security): gate bare and mcp-qualified email names together; stop executing Markdown info strings
Review follow-up on #3681 (thanks @RaresKeY):
1. P1: execute_tool_block() checked disabled_tools / the turn ToolPolicy
only against the incoming block name, then the bare-email branch
qualified it to mcp__email__<name> and called the MCP manager. Plan
mode and the MCP settings toggle write the QUALIFIED name into the
denylist, so a bare fence like ```list_emails``` sailed past a
mcp__email__list_emails entry. Both gates now match on both
spellings (bare <-> mcp__email__-qualified), in either direction.
2. P2: the relaxed fence regex accepted arbitrary same-line text after
a recognized tag, which made ordinary Markdown info strings
executable: ```python title="example.py" ran as a python tool call.
Same-line content now only counts as tool input when it starts with
{ or [ (JSON args); anything else leaves the fence as display text,
and strip_tool_blocks mirrors that (the fence stays visible).
Tests: disabled-tools alias regression (qualified entry blocks bare
name and vice versa, never reaching the MCP manager), ToolPolicy alias
regression, python/bash title="..." non-execution + display retention,
and inline JSON-array args still parsing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(security): reject brace-style fence metadata; cover the full email set in the friendly toggle
Review follow-up round 3 on #3681 (thanks @RaresKeY):
1. Brace-style fence metadata no longer executes. The previous narrowing
still treated any same-line {/[ after a recognized tag as tool input,
so ```bash {title="setup"} ran as a bash call. The fence header is now
captured separately and judged by one predicate shared between
parse_tool_blocks and strip_tool_blocks (_fenced_tool_call), so the
execute and display decisions can't disagree: same-line content only
counts as inline args when the tag is NOT a code tag (bash/python
never take same-line args — that text is Markdown fence attributes)
AND the inline text (plus any continuation lines) parses as standalone
JSON. ```bash {title="setup"}, ```python {"title":"example.py"} and
```list_emails {title="x"} all stay visible and inert.
2. The friendly `disable_tool email` toggle covered 3 of the 14 email
tools (mcp__email__{list_emails,read_email,send_email}); the other
bare aliases this PR routes stayed executable after an operator
disabled email. The alias now derives from BUILTIN_EMAIL_TOOLS in
BOTH spellings — bare (function-schema hiding, bare-fence dispatch)
and mcp__email__* (MCP schema hiding, qualified runtime blocks) —
so the toggle and the runtime gate can't drift apart.
Tests: brace/bracket metadata regressions for parse and strip symmetry
(code tags, invalid-JSON inline on a JSON tool, multi-line inline JSON
still parsing), and disable_tool/enable_tool email covering all 14 names
in both spellings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(email): close remaining email-tool registry drift; classify every email tool for plan mode
Deep self-review follow-up on #3681. Three review rounds each found another
hand-maintained copy of the email tool list that had drifted; this commit
hunts down ALL remaining copies and pins them to BUILTIN_EMAIL_TOOLS.
The same 5 tools (search_emails, draft_email, draft_email_reply,
ai_draft_email_reply, download_attachment) were missing from every
advertising surface, so they were dispatchable but never offered:
- FUNCTION_TOOL_SCHEMAS: native function-calling models never saw them
(the round-1 fix covered dispatch only); schemas added, mirroring the
email server's inputSchema definitions.
- TOOL_SECTIONS: fenced-block models were never told about them; prompt
sections added.
- tool_index: absent from the RAG embedding registry (never retrievable),
the email keyword hints, and the scheduled assistant's always-available
set — the latter two now derive from BUILTIN_EMAIL_TOOLS.
- agent_loop._DOMAIN_TOOL_MAP["email"], tool_policy._COMMON_TOOL_NAMES,
the assistant tool-selector UI groups (assistant.js), and the default
Assistant crew seed (task_scheduler) now derive from / cover the set.
Plan mode now classifies every email tool explicitly:
- list_email_accounts and search_emails join PLAN_MODE_READONLY_TOOLS.
Without this, list_email_accounts sat in the plan-mode bare denylist
(schema-derived) while its qualified form passed the MCP read-only
filter — and the round-2 bare/qualified alias gate would have blocked
the qualified call too, regressing read-only email discovery in plan
mode.
- draft_email, draft_email_reply, ai_draft_email_reply, and
download_attachment join the fail-closed mutator backstop (drafts
create documents; download_attachment writes to disk).
Tests: tests/test_email_registry_sync.py pins every registry (including
the email server source and assistant.js) to BUILTIN_EMAIL_TOOLS and
asserts the plan-mode partition, so the next email tool can't drift; a
parse/strip mirror grid covers 192 fence shapes (tag x header x body)
asserting executed <=> stripped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: move the email alias rule into tool_security; extract the assistant seed constant
Code-quality pass over the PR's own changes:
- The bare<->qualified email aliasing rule lived inline in the generic
dispatcher (_execute_tool_block_impl). It is policy knowledge, so it
moves next to BUILTIN_EMAIL_TOOLS as email_tool_policy_names(); the
dispatcher just consumes it, and the rule gets its own unit test
(including the mcp__email__<not-a-tool> and mcp__other__ non-alias
cases).
- The default Assistant's enabled_tools list was an inline literal
inside the CrewMember seed, and its registry-sync test asserted a
source-code substring. Extracted to DEFAULT_ASSISTANT_ENABLED_TOOLS
so the test imports and checks the actual value.
- _fenced_tool_call return type tightened to Optional[Tuple[str, str]].
No behavior change; suite green (3295 passed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* revert: move the email registry consolidation to a follow-up PR
Per review feedback on scope, this PR stays narrow: fenced inline-args
parsing, bare email tool routing, and the directly required safety
gates. This commit reverts the registry/advertising consolidation from
db29046 and 016ce47 (native schemas, prompt sections, RAG description
index + keyword hints, assistant always-available set, guide-only
known-names union, frontend tool-selector groups, default assistant
seed, and their sync tests) — all of that moves to a dedicated
follow-up PR together with the _EMAIL_TOOL_HINTS finding.
Kept here because the narrow scope needs them:
- email_tool_policy_names() in tool_security + its use in the
execute_tool_block gates and its unit test (refactor of this PR's own
round-2 alias fix),
- list_email_accounts in PLAN_MODE_READONLY_TOOLS (the alias gate works
both ways, and the schema-derived plan-mode bare denylist would
otherwise block the qualified read-only call too),
- the parse/strip mirror grid test (parser scope),
- the narrow registry sync tests (email server <-> BUILTIN_EMAIL_TOOLS
match, fence-tag coverage, non-admin blocklist coverage).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(email): execute empty email fences with empty args; reject non-object JSON args
Two gaps found by replaying captured local-model traffic against the
narrowed branch:
1. ```list_email_accounts``` with NO body — a shape gemma really emits
for no-arg tools — was silently dropped (parse skips empty content),
so the model concluded email was broken: the original #337 symptom
through a different door. Empty fences whose tag is a built-in email
tool now dispatch with {} args and the tool's own validation answers
(e.g. an empty send_email returns "to is required" instead of
silence). Empty bash/python/other fences keep skipping, and strip
stays mirrored (the fence was executed, so it is removed).
2. The fence parser accepts JSON arrays as inline args, but the email
dispatch parsed only objects — an array silently became {} args.
Non-object JSON now returns a correctable "arguments must be a JSON
object" error before reaching the MCP server (same class as #3966).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(security): classify all email tools for plan mode statically; reject invalid email JSON bodies
Review follow-up round 5 on #3681 (thanks @RaresKeY):
1. This PR makes every BUILTIN_EMAIL_TOOLS name fence-taggable, so each
one must be explicitly classified for plan mode — the draft tools and
download_attachment were in neither the read-only allowlist nor the
static denylist, leaving their bare-alias plan-mode safety dependent
on the MCP read-only inventory being present and current.
search_emails joins PLAN_MODE_READONLY_TOOLS (explicit, not
allowed-by-omission); draft_email, draft_email_reply,
ai_draft_email_reply, and download_attachment join the fail-closed
_PLAN_MODE_KNOWN_MUTATORS backstop. (Moved back from the #4053 split:
the partition is directly required for this PR to merge
independently.)
2. The classic tag/body fence form reaches execution unvalidated (only
INLINE args are JSON-checked by the parser), so a body like
{account: "work"} silently became {} args and read the DEFAULT
mailbox instead of the intended one. JSON-looking bodies that fail to
parse now return a correctable "not valid JSON" error before reaching
the MCP server.
Tests: a partition invariant (every email tool is explicitly read-only
or plan-mode-denied), a mutating-alias probe that uses only the static
denylist with a fake MCP manager (no inventory layer), and the
body-form invalid-JSON regression.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(tool-dispatch): decode inline JSON args for legacy MCP tools; reject all non-object email bodies
Review follow-up round 6 on #3681 (thanks @RaresKeY) — both pre-existing
on this branch, surfaced by the relaxed inline-args parser:
1. The relaxed parser accepts inline JSON for every non-code tag, but
the legacy line-based arg builders (web_search/web_fetch/read_file/
write_file/generate_image/manage_memory) wrapped the whole JSON
string as the query/url/path/prompt — so `web_search {"query": "x"}`
executed as a search for the literal string `{"query": "x"}`.
_build_mcp_args now uses a fenced JSON object directly when it carries
the tool's primary arg key (query/url/path/prompt/action). Keyed off
membership so it can't drift; an object without the primary key (e.g.
a freeform JSON query, or bare object content for write_file) falls
through to the line parser unchanged. Also fixes the same corruption
for the classic newline-JSON form.
2. The bare-email dispatch only rejected bodies starting with { or [, so
a non-empty non-JSON body like `account: work` still fell through to
{} args and silently read the DEFAULT mailbox. Now ANY non-empty body
must decode to a JSON object or it returns a correctable error; only a
truly empty body keeps the no-arg path (```list_email_accounts```).
Tests: inline-JSON arg decoding for the five legacy tools plus the
freeform and missing-primary-key fallbacks; the email body rejection
extended to cover the brace-looking and bare `key: value` shapes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(tool-dispatch): drop dead manage_memory JSON-decode entry; pin the live-path invariant
Self-audit catch on the round-6 fix. manage_memory was added to
_MCP_JSON_PRIMARY_KEYS, but _build_mcp_args is only reached via
_call_mcp_tool, which only runs for _MCP_TOOL_MAP tools — and
manage_memory isn't one (its tag routes through dispatch_ai_tool ->
do_manage_memory, which line-parses). So the round-6 decode for
manage_memory was dead code: the unit test exercising _build_mcp_args
passed while a real `manage_memory {"action": ...}` fence still parsed
the whole JSON blob as the action.
Remove the dead entry and add test_mcp_json_primary_keys_are_all_live,
which asserts every JSON-primary tool is in _MCP_TOOL_MAP so a dead
decode can't be added again. The same inline-JSON corruption for
manage_memory and the other tools that route through positional
dispatchers (create_session, ui_control, send_to_session, search_chats,
the document tools, etc.) is pre-existing (dev corrupts their newline
JSON form too) and tracked separately; the proper fix there is to route
fenced JSON through function_call_to_tool_block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(tool-dispatch): decode inline JSON in WriteFileTool (its live path); round-6 fix was on the dead MCP path
Self-audit: round 6 claimed to fix inline JSON args for write_file via
_build_mcp_args, but there is no filesystem MCP server, so write_file
always runs through _direct_fallback -> WriteFileTool, never through
_build_mcp_args. WriteFileTool — unlike its siblings ReadFileTool /
WebSearchTool / WebFetchTool, which all decode JSON — took lines[0] as
the path, so `write_file {"path": "/tmp/x", "content": "y"}` wrote to a
file literally named with the JSON blob. The round-6 _build_mcp_args
entry decoded correctly but on a path that never executes (same class
as the manage_memory dead entry), and the round-6 unit test passed on
that dead path.
WriteFileTool now decodes a JSON object carrying "path" (matching
ReadFileTool directly above it), and the comment on _MCP_JSON_PRIMARY_KEYS
records that only generate_image has a live MCP server today — the other
entries are defense-in-depth for the MCP path; the live fix for each
server-less tool is in its handler.
Test: test_write_file_inline_json_args drives the LIVE path
(execute_tool_block with no MCP) and asserts the intended path is used —
verified to fail without the handler fix. web_search/web_fetch/read_file
were already correct (their handlers decode); write_file was the gap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(strip-fence): derive the live-strip TOOL_TAGS from the real set
Semantic conflict from the dev merge that textual auto-merge didn't flag:
dev added test_live_strip_email_tool_fences.py whose _tool_tags() helper
source-scrapes only the TOOL_TAGS literal `{...}`, which worked on dev
because the email tool names were listed inline there. This branch makes
TOOL_TAGS the single source — `{...} | BUILTIN_EMAIL_TOOLS` — so the email
names are no longer in the literal and the scraper missed them, leaving the
email-fence strip assertions failing even though TOOL_TAGS does contain them
at runtime.
Import the real TOOL_TAGS instead of scraping source, so the test mirrors
exactly what GET /api/tools serves (sorted(TOOL_TAGS)) and the live
EXEC_FENCE_RE derives from — robust to however the set is composed. The
source-level frontend/route guards in the same file are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: botinate <285686135+botinate@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Michael