Jake/odysseus
1
0
Fork 0
mirror of https://github.com/pewdiepie-archdaemon/odysseus.git synced 2026-07-14 12:48:03 +00:00
Code Packages Projects Releases Activity

Compare commits

base: Jake/odysseus:f7e2d0c0b7e7fd94c38eb35c092f738c3bb5b7bf
Branches Tags
Jake/odysseus:dev Jake/odysseus:main Jake/odysseus:dependabot/pip/python-258f0448df Jake/odysseus:dependabot/github_actions/actions-5b4a7f4463 Jake/odysseus:fix/security-gallery-endpoint-hardening
...
compare: Jake/odysseus:fix/security-gallery-endpoint-hardening
Branches Tags
Jake/odysseus:dev Jake/odysseus:main Jake/odysseus:dependabot/pip/python-258f0448df Jake/odysseus:dependabot/github_actions/actions-5b4a7f4463 Jake/odysseus:fix/security-gallery-endpoint-hardening

479 Commits
f7e2d0c0b7 ... fix/security-gallery-endpoint-hardening

Author SHA1 Message Date
Alexandre Teixeira 0cf7eddde6 fix(security): harden gallery endpoint URL checks 
Replace substring OpenAI endpoint detection with exact parsed-host matching.

Route gallery image endpoint construction through a constant path allowlist.

Remove client-visible exception and upstream response body leaks from gallery image flows while preserving diagnostics in server logs.

Add focused regression tests for OpenAI host matching, checked endpoint joining, harmonize SSRF hardening, and sanitized client errors.
 2026-06-28 13:47:53 +01:00
nopoz a7fc1343a3 fix(security): prevent ReDoS in verdict-prose and continuation matchers (#4943) 
Two py/polynomial-redos sinks ran regexes with two adjacent \s-matching
quantifiers over untrusted model text, backtracking O(n^2) when the tail failed
on a whitespace flood:

  - routes/skills_routes.py: the last-resort verdict-from-prose extractor used
    `["\'\s:]*\s*` — the class already matches \s, so the trailing \s* was a
    redundant second quantifier. Dropped it (extracted to a documented module
    constant _VERDICT_PROSE_RE); the matched text is identical, the scan linear.
  - src/agent_loop.py _EXPLICIT_CONTINUATION_RE: `\s*[.!?]*\s*$` put two \s*
    around `[.!?]*`. Rewrote as `\s*(?:[.!?]+\s*)?$` — same accepted tails (no
    two \s* adjacent), linear. Portable form (no possessive quantifiers).

Both verified output-equivalent to the originals across a fuzz corpus. Adds
tests/test_redos_verdict_continuation.py pinning the unchanged match sets and
bounding the flood inputs (old patterns took seconds at 40k whitespace chars).
 2026-06-28 11:42:20 +01:00
red person 827a6b2778 Reject blank ownerless claim owner (#4929) 2026-06-28 10:57:11 +01:00
Tal.Yuan 8066a8e0cd refactor(routes): move gallery domain into routes/gallery subpackage (#4903) 
Move the gallery route domain into routes/gallery/ while preserving backward-compatible legacy import shims.

- app imports the canonical gallery route module
- canonical gallery route code imports canonical gallery helpers
- legacy gallery route/helper paths remain compatibility aliases
- add shim regression coverage for module identity and monkeypatch behavior
- repoint gallery source-introspection tests to the canonical paths

No intended behavior change.
 2026-06-28 10:40:34 +01:00
Rudra Sarker 5b8bfdabab fix(chat): sanitize web search query to strip markdown and code blocks (#4863) 
Layer a defensive cleanup on top of the generated-query web-search flow so the final selected query is sanitized before reaching comprehensive_web_search.

- remove fenced code blocks from the final search query
- preserve inline code as plain text
- collapse whitespace and cap query length
- cover generated-query success plus LLM failure/empty fallback paths

Partially addresses #4547.
 2026-06-28 01:23:08 +01:00
tanmayraut45 ff0f1b3450 fix(mcp): retain builtin startup tasks and reap npx probe 
Keep strong references to builtin MCP startup tasks until completion and kill/reap the npx probe subprocess when cancellation interrupts the probe. Includes focused regression coverage for both lifecycle paths.
 2026-06-28 01:18:17 +01:00
Pedro Barbosa 9782e5bc94 fix(cookbook): load user-site pth hooks for runtime installs 
Replay user-site .pth hooks when checking cookbook runtime dependencies so packages installed with --user are visible to dependency completion. Includes focused regression coverage.
 2026-06-28 01:01:44 +01:00
tanmayraut45 c01c09559a fix(ai): offload model resolution from async paths 
Wrap blocking _resolve_model calls in asyncio.to_thread across async model interaction paths so endpoint/model resolution does not stall the event loop. Preserve owner-scoped resolution and add focused regression coverage.
 2026-06-28 00:48:35 +01:00
hestiaOS 8b110c28e6 fix(tasks): keep scheduled-task prompt cache stable 
Move scheduled-task current-time context out of the system prompt and into a user-role context message so the system prompt remains stable for prompt caching. Preserve time grounding on both the agent-loop path and fallback direct-call path, with focused regression coverage.
 2026-06-28 00:05:02 +01:00
Alexandre Teixeira 259662e914 test: split endpoint resolver tests (#4957) 2026-06-28 00:49:43 +02:00
nopoz fbe3a0d73b fix(security): prevent ReDoS in XML and args tool-call parsers (#4941) 
* fix(security): prevent ReDoS in XML and args tool-call parsers

Four py/polynomial-redos sinks in tool_parsing.py ran lazy/greedy regexes over
untrusted model output (tool-call markup is attacker-influenced via prompt
injection). When the closing delimiter was absent, each rescanned to
end-of-string from every opener -> O(n^2):

  - args => { ... } in _parse_tool_call_block: greedy \{([\s\S]*)\} restarted
    from every `args:{` opener. Now finds the opener once and takes through the
    last `}` (rfind) — equivalent capture, O(n).
  - _XML_INVOKE_RE: lazy <invoke ...>([\s\S]*?)</invoke>. Now _iter_xml_invoke
    pairs each opener with the first reachable </invoke> and stops when none is.
  - _XML_DIRECT_TOOL_RE and the <tag>([\s\S]*?)</\1> param scan in
    _parse_tool_code_block: lazy backreference patterns. Now _iter_backref_blocks
    pairs each opener with the nearest matching closer and memoizes tag names
    with no remaining closer, so an opener flood stays O(n).

All four are output-equivalent to the originals on well-formed tool-call markup;
the lazy patterns remain defined (still re-exported via agent_tools) but no
longer drive a finditer over untrusted text. Adds tests/test_redos_xml_tool_parsers.py
pinning correctness and bounding the opener-flood inputs (old paths took 4-15s).

* fix(security): harden invoke-parameter and distinct-name tag scans

Forward-only the two residual ReDoS paths in the XML/tool parsers that the
outer-delimiter fix left quadratic:

- _parse_xml_invoke parsed <parameter> with _XML_PARAM_RE.finditer, so a
  closed <invoke> body full of unclosed <parameter> openers rescanned the
  body from every opener (O(n^2), ~11s at 8k openers). Now scans forward-only
  via _iter_named_blocks, factored out of _iter_xml_invoke.
- _iter_backref_blocks only memoized repeated missing tag names; a flood of
  distinct unclosed names searched the suffix once per name (O(n^2)). It now
  indexes every closer by name in one linear pass and binary-searches per
  opener (O(n log n)). Covers the direct and tool_code backref scans.

Output-equivalent to the prior scanners (200k randomized trials match the
memoized version for both the direct ci=True and tool_code ci=False configs).
Adds regressions for the closed-invoke parameter flood and the distinct-name
floods (45k openers now run in ~0.05s, were 5-6s).
 2026-06-27 15:42:55 -07:00
Solanki Sumit df9907c09f fix(health): report unhealthy memory vector store as degraded 
Keep an unhealthy MemoryVectorStore instance available for health reporting instead of discarding it as disabled. This lets health checks report a degraded/down vector-store state while preserving focused regression coverage for initializer behavior.
 2026-06-27 22:25:13 +01:00
Ricardo 3b4187e25d fix(email): don't probe IMAP for send-only (SMTP-only) accounts (#4830) 
An account configured with SMTP only (no imap_host) has no inbox, but the
inbox list path still called _imap_connect, which handed an empty host to
imaplib. imaplib.IMAP4("", 993) silently dials localhost:993 and fails with
"[Errno 111] Connection refused", so the email panel's poll logged a
"Failed to list emails" ERROR every ~60s and surfaced a scary error in the UI.

_imap_connect now fails fast with a typed EmailNotConfiguredError (subclass of
RuntimeError, so existing broad handlers keep working) when no imap_host is set,
and the inbox list returns an empty result for that case instead of an error.
SMTP send is unaffected.
 2026-06-27 21:52:26 +01:00
Alexandre Teixeira 20cf323ca4 test: split provider detection tests (#4933) 2026-06-27 21:46:33 +01:00
Alexandre Teixeira 2497160fd4 test: split llm-core temperature tests (#4935) 2026-06-27 22:02:41 +02:00
Afonso Coutinho 70d806019b fix: tool results misthreaded to the wrong tool_call_id when a native call fails to convert (#1917) 
* fix: tool results misthreaded when a native call fails to convert

* Unpack the third converted_calls return from _resolve_tool_blocks in the fenced-example tests
 2026-06-27 19:31:17 +01:00
muhamed hamed 3e7af8634f fix: improve uploaded document retrieval and deep research reuse (#4784) 
* fix: improve uploaded document retrieval and deep research reuse

* test: add coverage for upload manifest and document pagination

* chore: rerun CI

* fix: restore _insert_before_latest_user helper

* fix(agent_loop): restore missing upload context helper
 2026-06-27 19:24:17 +01:00
Solanki Sumit 7e9bfb1700 fix(chat): guard non-numeric agent tool budget setting 
Guard the agent_max_tool_calls settings read so hand-edited or agent-written non-numeric settings.json values fall back to 0 instead of crashing agent-mode chat stream initialization. Add regression coverage for guarded coercion.
 2026-06-27 19:20:48 +01:00
Arpit e7c61a75b6 fix(search): use generated query for chat mode web search #4547 (#4557) 
* fix(search): use generated query for chat mode web search #4547

* style(search): tidy query generation call

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>
 2026-06-27 19:04:46 +01:00
Solanki Sumit 20691d6019 fix(upload): handle corrupt uploads index and malformed vision JSON 
Use the upload handler's tolerant index loader when reading upload metadata so corrupt uploads.json degrades to missing metadata instead of a 500. Return 400 for malformed vision JSON request bodies and add regression coverage for both paths.
 2026-06-27 18:59:28 +01:00
Miraç Duran 228efbc70a fix(calendar): accept time-first datetimes in _parse_dt 
Accept calendar datetime phrases such as "3pm tomorrow" by adding a time-first natural-language parser branch mirroring the reminder parser. Add regression coverage proving time-first forms match their existing day-first equivalents.
 2026-06-27 18:51:18 +01:00
nopoz c098355778 fix(security): prevent ReDoS in LLM-output tool/think parsers (#4704) 
* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos #230, #231, #232, #233, #235, #236,
#524. The _XML_OPEN_TOOL_CALL_RE alerts (#234, #477) are false positives
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix
 2026-06-27 10:12:28 -07:00
Rudra Sarker 090f4078d8 fix(llm-core): prevent cache-affinity fields from reaching Cerebras 
Recognize api.cerebras.ai as a Cerebras cloud provider so llama.cpp/LM Studio cache-affinity fields are not attached even when endpoint_kind is misconfigured as local. Add regression coverage for provider detection, self-hosted classification, and payload field exclusion.
 2026-06-27 18:07:12 +01:00
Afonso Coutinho ad745801c6 fix(visual_report): ignore fenced headings in TOC extraction 
Strip fenced code blocks before extracting visual-report headings so heading-looking lines inside code fences do not desync TOC anchors. Add regression coverage for backtick and tilde fences while preserving normal heading extraction.
 2026-06-27 17:44:32 +01:00
Miraç Duran d5286f926e fix(visual_report): make TOC heading slugs unique 
Ensure generated visual-report TOC slugs cannot collide with naturally occurring slug names. Add regression coverage for duplicate headings, natural suffix collisions, and unchanged distinct headings.
 2026-06-27 17:36:17 +01:00
Ashvin 67040a196f fix(docker): install python-magic and libmagic for upload MIME sniffing 
Install libmagic1 and image-scoped python-magic in the Docker image so upload MIME detection can use content sniffing. Add regression coverage for the Dockerfile dependency pair and the libmagic-present sniffing path.
 2026-06-27 17:31:46 +01:00
Catalin Iliescu 497c391f84 fix(cookbook): preserve scheduled serve server metadata (#4545) 
Co-authored-by: Cata <cata@bigjohn.local>
 2026-06-27 16:48:53 +01:00
Marcus Sonntag 95b3c8139d fix(llm): add default context window lengths for Xiaomi Mimo 2.5 models (#4579) 2026-06-27 16:43:00 +01:00
Arpit a05666a1b0 fix(notes): allow inline editing of checklist items (#4832) 
* Refresh README screenshot

* fix(notes): allow inline editing of checklist items

* fix(notes): delete checklist item if inline edit is empty

* fix(notes): use debounce for text click to bypass toggle on double click

* fix(notes): use Edit button exclusively for inline edit to avoid UX delay on toggle

---------

Co-authored-by: pewdiepie-archdaemon <pewdiepie-archdaemon@users.noreply.github.com>
 2026-06-27 17:37:28 +02:00
Dewangga Abdullah 6d429a49b9 refactor(tools): register update_plan tool and support dynamic execution (#4069) 
* refactor(tools): register update_plan tool and support dynamic execution

* refactor: move interaction tools to registry and fix tuple unpacking error

* docs: add HACK comment for circular dependency workaround

Signed-off-by: dewanggaabdullah <255674162+dewanggaabdullah@users.noreply.github.com>

* refactor(tools): use docstring for better code style

Signed-off-by: dewanggaabdullah <255674162+dewanggaabdullah@users.noreply.github.com>

* fix(tools & file): restore file tool_registry & unknown tool fallback and fix dynamic handlers unpacking

Signed-off-by: dewanggaabdullah <255674162+dewanggaabdullah@users.noreply.github.com>

---------

Signed-off-by: dewanggaabdullah <255674162+dewanggaabdullah@users.noreply.github.com>
 2026-06-27 17:36:10 +02:00
SINE 2dfc83ee22 fix(models): accept bare-list /models responses (Together AI) (#4761) 
* fix(api): handle varying response formats for model IDs from compatible providers

merge conflict for pr-2204 resolved

* fix(modal): keep body-portaled dropdowns above their tool modal at any stack depth (#4720) (#4724)

* fix(memory): keep the Brain memory item menu above the modal at any stack depth

The memory item "⋮" dropdown is portaled to <body> with a hardcoded
z-index of 10001. Tool modals, however, get a monotonically increasing
z-index from modalManager's bring-to-front counter (_modalTopZ), which
climbs unbounded as modals are opened/restored over a session. Once that
counter passes 10001, the Brain modal stacks above the body-portaled
dropdown, so the menu renders behind the panel — visible only where it
spills past the modal's edge (#4720).

Derive the dropdown's z-index from the owning modal's current z-index
(+1), keeping 10001 as a floor for the common low-counter case, so the
menu always sits just above its modal however high the counter has climbed.

Verified with document.elementFromPoint at the dropdown's location: with a
high modal z-index the old build returns the modal at every sampled point
(menu behind); the fixed build returns the dropdown (menu on top). The
default low-counter case is unchanged (z stays 10001).

* refactor(modal): route body-portaled dropdowns through a shared topPortalZ() helper

The hardcoded z-index:10001 the Brain memory menu used (#4720) is the same
literal shared by ~16 body-portaled dropdowns across calendar, cookbook,
cookbookServe, documentLibrary, emailLibrary, gallery, notes, emojiPicker and
memory — each renders behind its owning tool modal once modalManager's
bring-to-front counter climbs past the literal over a long session.

Promote the per-dropdown fix into a single topPortalZ() helper in
toolWindowZOrder.js — the existing source of truth for tool-window z, already
imported by modalManager's _bringToFront and notes.js — returning
max(topToolWindowZ(), dock-chip floor) + 1, so a portaled dropdown always sits
just above the live tool-window stack however high the counter has climbed.
Route all 16 sites through it. The slashCommands tour tooltips and the
cookbookServe VRAM dialog are intentionally left out (neither is a modal-owned
portaled dropdown).

Add tests/test_portal_dropdown_z_js.py covering the helper, including the #4720
scenario (modal counter at 99999 -> dropdown at 100000). Existing
test_notes_z_order_js.py stays green.

* fix(llm): detect mistral.ai provider and support reasoning_effort (#4698)

* fix(llm): detect mistral.ai provider and support reasoning_effort

Four coupled bugs broke Mistral thinking model support:

1. _detect_provider() had no mistral.ai host check, so all Mistral
   endpoints fell through to the generic 'openai' provider string.
   _provider_display_name() correctly identified them as 'Mistral',
   making any 'if provider == "Mistral"' check elsewhere dead code.

2. reasoning_effort parameter was never sent in the request payload,
   so Mistral never activated thinking mode even when the user
   configured a thinking-capable model (mistral-small-latest,
   mistral-medium-latest, magistral-*).

3. Mistral returns content as a typed array
   ([{"type":"thinking",...},{"type":"text",...}]) when
   reasoning is on, not as a plain string. Both the streaming and
   non-streaming parsers expected strings and silently dropped the
   thinking content.

4. _THINKING_MODEL_PATTERNS didn't include magistral or mistral-*
   model prefixes, so the frontend wouldn't tag reasoning output
   as thinking even after the above were fixed.

Fix:
- Add mistral.ai to _detect_provider() host checks
- Add a _normalize_mistral_content() helper that splits the typed
  array into (text, thinking) strings
- Inject payload["reasoning_effort"] = "high" when provider is
  Mistral and _supports_thinking(model) is true, in both stream_llm
  and llm_call_async payload construction
- Wire the normalizer into both response parsers
- Extend _THINKING_MODEL_PATTERNS to include magistral,
  mistral-small, mistral-medium, mistral-large

Tested on Docker install with mistral-small-latest +
reasoning_effort=high. Reasoning streams correctly into the
thinking panel after the fix.

Fixes #4678

* fix(llm): address review — lowercase provider id, configurable effort, tests

Addresses vdmkenny's review on PR #4698:

1. Removed duplicate 'if provider == "mistral"' block in stream_llm
   — two back-to-back copies, one was dead-redundant.

2. Dropped personal-context comment ('free-tier limits are generous
   for this user') and made reasoning_effort configurable via env var
   ODYSSEUS_MISTRAL_REASONING_EFFORT (high / medium / low / none).
   Default remains 'high' for backward compat with the tested behavior.

3. Recased provider id from 'Mistral' to 'mistral' to match the
   lowercase convention used by every other provider id in the file
   (openai, anthropic, ollama, copilot, ...). _provider_display_name()
   still returns the Title-Case 'Mistral' for UI labels — only the
   runtime id used in 'if provider == ...' checks was recased.

4. Added tests/test_llm_core_mistral_content.py with 13 tests pinning
   _normalize_mistral_content()'s contract: string passthrough, the
   Mistral array format (thinking + text blocks), and edge cases
   (empty, garbage, None, wrong types, missing fields, string-vs-array
   inner thinking field).

Also fixed a gap the review didn't catch: the non-streaming paths
(llm_call sync + llm_call_async) were missing the reasoning_effort
injection entirely. Added the same injection to both, so Deep Research
and agent tool calls also activate Mistral thinking.

All 13 new tests pass. Existing reasoning/streaming/ollama-thinking
tests still pass (38 tests, no regressions).

Fixes #4678

* fix: Images cannot be seen by model that is vision capable (#4726)

* fix: Images cannot be seen by model that is vision capable

* fix: skip http(s) image_url for Ollama (images[] is base64-only)

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix(chat): strip executed email tool fences from the live stream (#3993) (#4275)

* fix(chat): strip executed email tool fences from the live stream (#3993)

The backend strips every fenced tool block from persisted text (the regex in
src/tool_parsing.py is built from the full TOOL_TAGS set, which includes the
email tools), so a reloaded session renders cleanly. The live frontend path
uses a separate hardcoded EXEC_FENCE_RE in static/js/chatRenderer.js that only
listed web_search/read_file/write_file/create_document/edit_document/
update_document — so executed email tool fences (list_emails, etc.) lingered as
raw code blocks in the live assistant bubble until the user reloaded.

Add the nine email tool tags to EXEC_FENCE_RE so the live render settles into
the same clean layout as the history reload. bash/python stay excluded on
purpose: those are languages a user may legitimately have asked the model to
show as code, not tool invocations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chat): single-source live exec-fence tool list from TOOL_TAGS (#3993)

Per review: EXEC_FENCE_RE was a second, hand-maintained copy of the
executable-tool list, so any tool not in it — and every future tool added to
TOOL_TAGS — would leave its executed fence lingering in the live bubble until
reload (the original #3993 bug, recurring one tool at a time).

EXEC_FENCE_RE is now built from an explicit EXEC_TOOL_TAGS list that mirrors
TOOL_TAGS (src/agent_tools/__init__.py) minus bash/python, which stay excluded
as legitimate code-example languages. A new regression test
(test_exec_fence_re_covers_all_executable_tools) extracts both lists from
source and fails if they drift, so the whole class is caught in CI instead of
by a user — the "minimum acceptable middle ground" from the review, made exact
(set equality, not just coverage).

Verified: pytest tests/test_live_strip_email_tool_fences.py (5 passed);
node --check static/js/chatRenderer.js; and a node run of the built regex
confirms email/generate_image/manage_memory/ls fences strip while
bash/python/sh are preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chat): build live exec-fence list from /api/tools at runtime (#3993)

Make TOOL_TAGS the single source for live exec-fence stripping. chatRenderer.js
no longer hard-codes a tool list; it fetches the backend's authoritative set
once from GET /api/tools (sorted(TOOL_TAGS)) and builds EXEC_FENCE_RE from it at
load, minus bash/python. No second list to drift, and a future tool added to
TOOL_TAGS is covered automatically — without touching the streaming path.

Until the fetch resolves EXEC_FENCE_RE is null and exec fences aren't stripped
(a sub-second window before the first stream); the backend already strips
persisted history, so a reload always renders clean.

Drop test_exec_fence_re_covers_all_executable_tools (no hand-maintained list to
guard) and add source-level guards: the frontend keeps no hard-coded list and
fetches /api/tools, and the endpoint serves the full sorted(TOOL_TAGS).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL

* fix(chat): warn on /api/tools fetch failure instead of swallowing it (#3993)

A fresh-context review flagged that loadExecFenceRegex's catch silently
discarded errors: if the one-shot fetch fails, EXEC_FENCE_RE stays null for the
whole session and live exec fences go unstripped until reload, with zero signal.
console.warn it, and correct the comment to describe the failure mode honestly
(was understated as just a sub-second startup window).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): log and cleanly 500 on unreadable HTML page (#4637)

* fix(routes): serve 404 instead of 500 when an HTML page file is missing

_serve_html_with_nonce opened the HTML file with no error handling, and
callers such as /backgrounds and /login pass their paths in with no
existence check, so a missing or unreadable file raised an unhandled
OSError that surfaced as a 500. Wrap the read and raise HTTPException(404)
instead; the normal render path (CSP-nonce substitution) is unchanged.

Fixes #4594

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): distinguish missing page (404) from read failure (500)

The previous fix caught a broad OSError and returned 404 for every
failure, which masks real server-side problems (permission errors, I/O
failures) as "not found" and lets them slip past error alerting. Split
FileNotFoundError (genuine 404) from other OSError, which now logs the
exception and returns a generic 500 — without leaking the OS error
string or file path into the response body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): treat unreadable bundled HTML page as logged 500, not 404

Per PR #4637 review: every caller of the page-render helper serves a fixed,
server-owned template (index/login/backgrounds), never a client-supplied
path. So a missing or unreadable file is a server fault (broken deployment),
not a client "not found" — a 404 there mislabels a server error and hides a
missing core template from 5xx alerting, contradicting the OSError->500
rationale this PR is built on. Collapse both branches into a single logged,
leak-free 500.

Move the helper to src.app_helpers.serve_html_with_nonce so the behavior can
be unit-tested without importing the whole app (app.py is the slim
orchestrator; the test harness stubs src.database, so importing app in tests
is not viable). Add tests pinning missing/unreadable -> 500 (not 404) and
nonce injection on the happy path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(catalog): add Gemma 4 12B/QAT entries and RTX 3050 bandwidth (#4728)

Add official Gemma 4 12B-it plus QAT-INT4/INT8 catalog entries (with their
GGUF sources), QAT quantization support across the quant tables and the
prequantized-prefix list, and the missing RTX 3050 / 3050 Ti memory
bandwidth so speed estimates stop falling back to the generic cuda value.

* fix debugging on windows (#4679)

* fix: Real-ESRGAN install + Cookbook deps-panel crash on the Python 3.14 image (#4694)

* fix(docker): make Real-ESRGAN installable on the Python 3.14 image

realesrgan's deps basicsr/gfpgan/facexlib (unmaintained since 2022) read
their version in setup.py via `exec(...); locals()['__version__']`, which
raises KeyError on Python 3.13+ — PEP 667 made locals() in a function an
independent snapshot that exec() can no longer mutate. That fails the
Cookbook "install realesrgan" sdist build on the python:3.14 base.

Add a `realesrgan-wheels` builder stage that fetches the pinned sdists,
patches get_version() to exec into an explicit namespace dict, and builds
wheels; the final stage installs them --no-deps so a later
`pip install realesrgan` resolves from wheels instead of rebuilding the
broken sdists. torch stays a runtime pull to keep the base image lean.

Also add the runtime libs opencv-python (cv2) needs — libgl1,
libglib2.0-0t64, libxcb1 — which the slim base omits; without them the
install succeeds but `import cv2` dies with
`libxcb.so.1: cannot open shared object file`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cookbook): don't let a package's sys.exit() on import hang the deps panel

The local optional-dependency probe imports each package in-process and
catches ImportError / Exception. But a package can call sys.exit() at
import time — e.g. rembg does `sys.exit(1)` when no onnxruntime backend
loads. SystemExit is a BaseException, not Exception, so it escaped the
probe, propagated out of the list_packages endpoint, and hung the whole
Dependencies panel / worker (the UI loads forever).

Catch (Exception, SystemExit) so one broken optional package is reported
as not-usable instead of taking down the panel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): 500 (not 404) when the app-shell index.html is missing (#4791)

Follow-up to #4637. serve_index — the handler for / and the SPA deep-link
routes (/notes, /calendar, /cookbook, /email, /memory, /gallery, /tasks,
/library) — pre-checked os.path.exists and raised its own
HTTPException(404, "index.html not found") when the bundle was missing. So a
missing core template returned 404 before serve_html_with_nonce's 500 could
fire, the one inconsistency left after #4637.

index.html is a fixed, app-bundled template; a missing one is a broken
deployment (server fault), not a client "not found", so it should surface as a
logged 500 in 5xx alerting rather than a 404. Keep the static->root fallback,
drop the redundant existence guard and the dead-end 404, and let the shared
helper handle the missing case.

Verified against the running app: / and /notes return 200 with the bundle
present and a logged 500 when index.html is absent.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(setup): load .env so a pre-seeded admin password is honored on native installs (#4787)

setup.py read ODYSSEUS_ADMIN_USER / ODYSSEUS_ADMIN_PASSWORD via os.getenv()
but never loaded .env, so on native Linux/macOS installs a password
pre-seeded in .env (documented in docs/setup.md and .env.example) was
silently ignored and a random one generated, breaking the first login.
Docker was unaffected because compose passes the vars into the container env.

Call load_dotenv(BASE_DIR/.env, encoding="utf-8-sig") at the top of main(),
mirroring app.py (utf-8-sig tolerates a Notepad UTF-8 BOM). load_dotenv does
not override already-exported OS vars, so the existing precedence is kept.
python-dotenv is already a required dependency.

Adds a regression test that pre-seeds credentials only in .env (not the
shell) and asserts the stored bcrypt hash matches the pre-seeded password.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: email poller marks calendar extraction processed on LLM failure (#4622)

Move calendar processed-marker insert into the LLM success path (else branch).
Previously, the INSERT ran even after a transient LLM failure, causing the
poller to skip retrying calendar extraction on subsequent runs.

Minimal change: only touches the try/except/else control flow in
_auto_summarize_pass_single() — preserves existing formatting and line endings.

* feat(ui): add toggle for padding around chat area (#4691)

* feat: Allow admins to choose if they want to share defaults (#4752)

* First bare fix

* Adding the option toggle

* toggle function fix

* Final fix, added missing /auth/

* Extended toggle text & added tests

* Comments change

* Description toggle change

* br tag fix

* description change based on suggestion

* fix(agent): parse misfenced read_file calls (#4799)

* fix: use atomic write in APIKeyManager.save() to prevent credential data loss (#4591) (#4597)

* fix: use atomic write in APIKeyManager.save() to prevent data loss

Opening api_keys.json with 'w' truncates the file before writing, so a
crash, disk-full, or mid-write error leaves all stored provider API keys
corrupted. Switch to atomic write (temp file + fsync + os.replace) so
the original file is always intact on any failure.

Fixes #4591

* chore: trigger CI re-run

* chore: update PR description

* chore: fix how-to-test section for description check

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* feat(discovery): detect llama.cpp servers and label local providers (#4729)

* feat(discovery): detect llama.cpp servers and label local providers

Scan port 8080 (llama-server) and 11435 (APFEL) during discovery, fingerprint
llama.cpp via its native /props endpoint, and label well-known local serving
ports (8080 llama.cpp, 8000 vLLM, 1234 LM Studio, 11434 Ollama) consistently
in both the Python provider helper and the JS endpoint UI. Adds a llama.cpp
hint to the /setup slash command.

* fix(discovery): don't infer the serving tool from the port alone

Per review: vLLM, SGLang, llama.cpp and plain OpenAI-compatible servers all
share 8000/8080, so labeling by port mislabels real setups (a vLLM box on 8080
shown as llama.cpp). Drop the port->tool assertions from _provider_label and
providerLabel; the authoritative signal is the /props fingerprint done during
discovery, which is unchanged. Loopback now reads a neutral 'local endpoint' /
'Local'. Tests updated to assert the neutral labels.

* refactor(tools): migrate config/integration admin tools to the registry (#4742)

Part of #3629 (the `admin_tools.py` bullet). Moves the config/integration admin
tools off the legacy elif dispatch chain in tool_implementations.py onto the
agent_tools registry:

  manage_endpoints, manage_mcp, manage_webhooks, manage_tokens, manage_settings

The do_* implementations (and manage_mcp's command-allowlist / RCE guard:
_validate_mcp_command, _mcp_allowed_commands, and the _MCP_* constants) move
verbatim into the new src/agent_tools/admin_tools.py. They register through a
single ADMIN_TOOL_HANDLERS map that TOOL_HANDLERS.update()s, and the five elif
branches plus their imports are dropped from tool_execution.py, so these tools
now flow through _direct_fallback like the other migrated clusters. The names
are re-exported from src.agent_tools for back-compat.

Dedup:
  - _parse_tool_args was duplicated in tool_implementations.py and
    document_tools.py. It now lives once in src.tool_utils (which imports nothing
    from the project beyond src.constants, so this introduces no cycle) and both
    call sites import it from there. The orphaned `import json` in document_tools
    is removed with it.
  - The five tools share one _owner_adapter(fn) factory that threads ctx["owner"]
    into the owner-taking do_* signature, instead of five near-identical wrappers.

Tests: new tests/test_admin_tools_registry.py pins the registration, the
re-export back-compat, the owner-threading adapter, and the single-source
_parse_tool_args (across admin_tools and document_tools). Existing MCP /
settings / webhook suites are repointed at the new module.

* refactor(exceptions): dedupe src/exceptions via core re-export (#4785)

src/exceptions.py was a byte-for-byte duplicate of the canonical
core/exceptions.py. Replace its class bodies with a re-export shim
(mirroring the core/constants.py -> src/constants.py pattern) so the
exception classes are defined in exactly one place. Also fix the stale
"# src/exceptions.py" header comment in core/exceptions.py.

No behavior change: both import paths resolve to the same class objects
(verified by identity), so `except SessionNotFoundError` works regardless
of which module it was imported from. Ran py_compile and
pytest tests/test_app.py (12 passed).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619)

Upstream bug (present in pewdiepie-archdaemon/odysseus main): the task
executor passes task.endpoint_url VERBATIM to the model HTTP call, unlike
the chat path which stores build_chat_url(normalize_base(base)) on the
session. A task carrying an explicit bare OpenAI-compatible base such as
"http://host:11434/v1" therefore POSTs to a 404 ("page not found"); the
agent loop swallows the empty body into "The model returned an empty
response" and marks the run success, so nothing surfaces the failure.

Tasks that omit an endpoint dodge this only because _resolve_defaults()
cribs an already-full URL from a recent chat session. The API/token path
(e.g. an external client that POSTs /api/tasks with endpoint_url=".../v1")
hits it every time.

Fix: route every resolved task endpoint through _normalize_chat_endpoint()
at the three resolution sites (_execute_llm_task, the persona/research
session path, and _execute_research_task). The helper is idempotent
(strips any existing chat suffix, re-appends the correct one) and leaves
native-Ollama (/api...) and already-concrete URLs untouched, so other
providers are unaffected. Proven via isolated repro: ".../v1" -> 404 ->
empty; ".../v1/chat/completions" -> 200 -> real gemma4:31b output.

Regression test asserts the bare-/v1 -> full-chat-URL mapping, idempotency,
and the native-Ollama/empty passthroughs.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(model-routes): harden _probe_endpoint against malformed model-list responses (#4789)

* fix(model-routes): harden _probe_endpoint against malformed model-list responses

_probe_endpoint parsed model lists with data.get(...) at four sites without
checking that data is a dict, and built the list with a truthiness-only
filter. A /models (or /api/tags) endpoint returning HTTP 200 with valid but
non-dict JSON ([], "x", null, 123) made data.get(...) raise AttributeError,
and a non-string id like 123 passed the filter and then hit .startswith() /
.lower() in the Z.AI/Kimi curated merge and _is_chat_model(). Both errors are
swallowed by the broad except Exception, but the comprehension dies mid-list
so the ENTIRE probed model list is discarded and the endpoint silently
degrades — masking a misconfigured/non-compliant upstream as "no models".

- Guard each data.get(...) with isinstance(data, dict) so a non-dict body
  falls through the existing `or []` default.
- Restrict the OpenAI and Ollama model-list comprehensions to non-empty str
  values, protecting the .startswith() merges and both _is_chat_model calls.
- Add an isinstance guard at the top of _is_chat_model (defense in depth for
  all four call sites).

No behavior change for well-formed {"data":[...]} / {"models":[...]}
responses. Adds regression tests (non-dict body via caplog, mixed/all
non-string ids, _is_chat_model boundary) that fail before the fix and pass
after.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(model-routes): extract _openai_model_ids / _ollama_model_names helpers

Per review on #4789: the malformed-response guards were inlined four times in
_probe_endpoint (two OpenAI-id comprehensions, two Ollama-name comprehensions).
Pull each into a small, directly-testable helper so the security-relevant
parsing lives in one place and a future malformed-shape fix doesn't have to be
applied in four spots (CONTRIBUTING flags repeated logic for this reason).

Behavior is unchanged. Adds direct unit tests for both helpers (non-dict body,
non-string ids, non-dict entries, name>model precedence).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cookbook): only block model launch on real port collisions (#4760)

* Fix #4507: only block model launch on real port collisions

Quick-run hardcoded port 8000 and never called _nextAvailablePort(), so
every launch collided. Both pre-launch guards (serve panel + quick-run)
were count-based and fired regardless of port.

- quick-run now auto-assigns a free port (8080 for llama.cpp)
- both guards parse the new port and only prompt on a real overlap,
  stopping only the colliding serve
- dialog reports the actual port instead of a hardcoded 8000

* refactor(cookbook): share _taskPort for port parsing; auto-assign llama.cpp port

Addresses review on #4760:
- _taskPort regex now matches --port= as well as --port (space)
- _nextAvailablePort and both launch guards reuse _taskPort instead of inline regex
- quick-run llama.cpp no longer pins 8080, so two can run concurrently

* fix(cookbook): _taskPort also parses -p; add port-parsing tests

Addresses review on #4760:
- _taskPort now matches -p <n> too, so it's the complete single reader
  (was missing the short flag that other readers already handle)
- add tests/test_cookbook_port_parsing_js.py covering the port forms,
  shared-reader reuse, and llama.cpp auto-assign

* test(cookbook): extract pure port helpers and test behavior

Addresses review on #4760: the prior tests only asserted source strings.
- extract portOf() and nextFreePort() into static/js/cookbookPorts.js
- cookbookRunning.js imports them; _taskPort and _nextAvailablePort delegate
- tests run the helpers via node and assert real behavior: all port forms
  (--port, --port=, -p, -p=), next-free-port skipping taken ports, and the
  same-port-clash / different-port-coexist outcome

---------

Co-authored-by: samy <samy@odysseus.boukouro.com>

* fix(ui): route tasks.js + skills.js dropdowns through topPortalZ() (#4768)

Fixes #4767. #4724 routed 16 body-portaled dropdowns through the shared
topPortalZ() helper so they always render just above the currently-raised tool
modal, but two were missed and still used a hardcoded z-index, so they hit the
same #4720 bug once a modal's bring-to-front counter climbed past the literal:

  - tasks.js _showTaskDropdown(): inline z-index:100000 on .task-dropdown
  - skills.js kebab menu (.skill-kebab-menu): z-index:100002 in style.css

Both now set zIndex from topPortalZ() after they are appended to the body,
matching the other migrated sites. The dead CSS z-index on .skill-kebab-menu is
removed (the inline value always wins). test_portal_dropdown_z_js.py gains a
source guard asserting both files use topPortalZ() and that no hardcoded
100000/100002 portal literal survives in either file or style.css.

* do_list_models in ai_interaction.py dropped

---------

Co-authored-by: Max Hsu <maxmilian@users.noreply.github.com>
Co-authored-by: aubrey <kyuhex@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ahmed Dlshad <ahmed.dlshad.m@gmail.com>
Co-authored-by: Joel Alejandro Escareño Fernández <52678667+TheAlexz@users.noreply.github.com>
Co-authored-by: Kalin Stoyanov <kgs.void@gmail.com>
Co-authored-by: Pedro Barbosa <devpedrobarbosa@gmail.com>
Co-authored-by: Solanki Sumit <125974181+YAMRAJ13y@users.noreply.github.com>
Co-authored-by: Rudra Sarker <78224940+rudra496@users.noreply.github.com>
Co-authored-by: Skoh <101289702+SkohTV@users.noreply.github.com>
Co-authored-by: Jakub Grula <ramsters110@gmail.com>
Co-authored-by: Dividesbyzer0 <54127744+zoomdbz@users.noreply.github.com>
Co-authored-by: Kenny Van de Maele <kenny@kvandemaele.be>
Co-authored-by: Magiomakes <114195802+Magiomakes@users.noreply.github.com>
Co-authored-by: Samy <12219635+touzenesmy@users.noreply.github.com>
Co-authored-by: samy <samy@odysseus.boukouro.com>
 2026-06-27 16:25:15 +01:00
Ashvin a6400c10af fix(calendar): keep imported events with non-positive duration visible (#4484) 
A single-day all-day event whose source writes DTEND equal to DTSTART
(treating DTEND as an inclusive bound rather than the RFC 5545 exclusive
one) was stored verbatim as a zero-duration row. list_events selects
events overlapping the window with `dtstart < end AND dtend > start`, so
that row is filtered out for any window starting at or after its date and
the event never appears, even though the import reported success.

Events created via the API never hit this because creation always
synthesizes a positive duration; only the two import paths can persist a
non-positive one. Clamp a non-positive end at import (import_ics and the
CalDAV pull) to the same default span used when DTEND is absent: one day
for all-day events, one hour otherwise.

Also repair the persisted state for users who already imported before this
clamp existed. Their stored zero-duration row is invisible, and re-importing
the same ICS hit the duplicate branch and skipped without touching it, so
the event stayed hidden. The duplicate branch now backfills the clamp onto
the matched row before skipping, and the response reports a `repaired` count.
(The CalDAV pull already rewrites dtend on re-sync, so it self-heals.)
 2026-06-27 16:52:40 +02:00
Afonso Coutinho 16ddfbf966 fix: vCard parser drops folded continuation lines, corrupting emails (#1870) 2026-06-27 14:41:57 +01:00
Afonso Coutinho edd5ea36ad Fix _parse_msg_content corrupting JSON-array-like text messages on reload (#2060) 
_parse_msg_content deserializes stored multimodal content (image/audio
blocks) back into a list. It treated ANY string starting with '[{' and
containing the substring "type" as serialized content, requiring only
that each element be a dict — never that "type" be a real content-block
kind. So a plain text message whose content happens to be a JSON array
of typed objects (e.g. a user pasting an API schema sample like
[{"type": "object", ...}]) was silently parsed from str into a list on
the next hydration, destroying the original string. This runs on every
session load from the DB (_db_to_session -> get_session). Restrict the
round-trip to non-empty lists whose every element is a dict whose
"type" is a recognized block kind (text/image/image_url/audio/...);
real multimodal content (verified: document_processor emits exactly
these) still round-trips, JSON-looking text is left untouched.
 2026-06-27 14:31:51 +01:00
Michael e3ecdd3207 fix(security): gate codex cookbook routes behind admin check for cookie sessions (#4554) 
The Codex cookbook bridge authorized cookie sessions with require_user()
only, allowing non-admin accounts to read cookbook task state, server
topology, task logs, tmux sessions, and model presets. The stop/adopt
routes also execute local or SSH-backed tmux commands.

Add _require_cookbook_scope() that enforces require_admin() for
cookie-session callers while preserving the existing API-token scope
checks. Apply it to all nine /api/codex/cookbook/* routes.

Fixes #4542

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
 2026-06-27 14:09:32 +01:00
Kevin Fiddick 8888819d74 Isolate untrusted context from visible user prompts (#3584) 
Prevent untrusted source/context guard text from being merged into the current visible user request during provider message sanitization.

Changes:
- Detect untrusted context blocks during LLM message sanitization
- Insert a short assistant boundary before the current user request
- Keep the visible user prompt as its own user message
- Preserve normal consecutive user-message merging for non-untrusted cases
- Strengthen prompt-security wording to avoid mentioning guard wrappers
- Add regression coverage for untrusted context followed by a user prompt

Notes:
- Untrusted context remains role:user for safety
- This does not add prompt debug logging
- This does not change frontend draft persistence
 2026-06-27 13:50:04 +01:00
nopoz ebead8083e fix(security): prevent ReDoS in agent_loop <think> stripping (#4877) 
The lazy `<think>.*?</think>` pattern (one compiled `_THINK_RE`, one inline
copy) is applied with `re.sub` over whole model responses. With a `<think>`
opener and no closer, the engine rescans to end-of-string from every opener
-> O(n^2) on attacker-influenced output (prompt injection can echo thousands
of openers via tool output / retrieved content). CodeQL py/polynomial-redos.

Replace both with `_strip_think_blocks`, a forward-only linear scan that is
byte-for-byte equivalent to the original narrow regex: only literal
`<think>`/`</think>` (any case) match, a dangling opener with no closer is
left intact, and an orphan `</think>` is never stripped. Routing through the
broader `text_helpers.strip_think` was avoided on purpose -- it also strips
`<thinking>`, attributes and prompt echoes, which would change what the
loop's progress/circling heuristics see.

Adds tests/test_redos_think_blocks.py pinning regex-equivalence on a battery
of well-formed/edge inputs plus a linear-time bound on hostile input.
 2026-06-27 04:32:42 +01:00
Sid a9b208f470 fix(auth): add config lock around migration methods (#4447) 
Per code audit #4388: Wrap _migrate_single_user and
   _drop_reserved_loaded_users with _config_lock to ensure atomic
   config reads/writes and prevent potential race conditions during
   concurrent access.

   This is a defense-in-depth fix - these methods run at startup
   before concurrent requests are accepted, but adding the lock
   makes the code consistent with other config mutations.
 2026-06-26 20:35:11 +02:00
Victor d4cd6d60f1 fix(email): validate IMAP/SMTP ports instead of crashing with 500 (#4464) 
The email-account endpoints coerced user-supplied ports with a bare int(data.get("imap_port") or 993), so a non-numeric port (e.g. "imap") raised ValueError and surfaced as an HTTP 500 in the create, update, and test-config endpoints.

Add a _coerce_port(value, default) -> (port, error) helper and use it in all three endpoints, returning the endpoints standard {"ok": False, "error": ...} response (matching the existing "name required" validation) instead of crashing. A blank or missing port still falls back to the default (993/465).
 2026-06-26 20:32:56 +02:00
Solanki Sumit ac05dff73c docs(setup): add a self-host troubleshooting cookbook of common traps (#4834) 
ROADMAP "Self-host troubleshooting cookbook" asks to document the weird
30-second fixes that otherwise become 30-minute searches. Adds a "Common
self-host traps" subsection under Troubleshooting covering: the UTF-8 BOM
.env gotcha (app.py loads with utf-8-sig), macOS AirPlay holding port 7000
(the start script uses 7860), the plain-HTTP Tailscale/LAN clipboard
limitation, self-hosted ntfy delivery (NTFY_BIND/NTFY_BASE_URL + the ntfy
Android Instant-delivery toggle), Dovecot cleartext-auth on LAN mail stacks,
and Radicale full-collection-URL sync.

Docs only; grounded in existing repo behavior (.env.example NTFY_* block,
app.py utf-8-sig loader, start-macos.sh port choice).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-26 20:24:02 +02:00
Alexandre Teixeira fcbddf3845 Merge pull request #4280 from GeekLuffy/feat/llm-self-eval 
feat(teacher): implement Tier 2 LLM self-evaluation
 2026-06-26 18:35:01 +01:00
Alexandre Teixeira ab01e7a000 Merge pull request #4448 from Muhammad-Ikhwan-Fathulloh/dev 
fix(upload): cache upload manifest and improve rename reliability
 2026-06-26 18:04:59 +01:00
Alexandre Teixeira 626414584b fix(upload): remove trailing whitespace 2026-06-26 18:01:04 +01:00
GeekLuffy d5a45c1ce3 feat(teacher): add teacher_tier2_enabled setting and strict parser 2026-06-26 22:26:15 +05:30
Alexandre Teixeira 62a23ca4aa test: split embedding lane tests (#4389) 
* test: split embedding lane tests

* test: preserve embedding focus selector after lane split
 2026-06-26 18:28:40 +02:00
Tal.Yuan fc1351d0f8 refactor(tools): split tool_implementations.py into src/tools/ package (#4423) 
* test(tools): add shim protection test for tool_implementations split

Covers all 48 top-level functions (33 do_* + 15 _helpers) extracted from
the original module. Guards the upcoming split: the shim must re-export
every symbol so existing 'from src.tool_implementations import X' imports
keep working. Passes on baseline (pre-split).

* refactor(tools): add src/tools/ package with shared _common

Slice 1 Task 2 (#4082/#4071). Adds the package skeleton and moves the
shared _parse_tool_args helper into src/tools/_common.py. Domain modules
will import from here. tool_implementations.py is untouched at this step.

* refactor(tools): extract system domain into src/tools/system.py

Slice 1 (#4082/#4071), Task 3: move the system-domain tool functions
(do_manage_skills/_skill_dump/do_manage_tasks/do_manage_endpoints/
do_manage_mcp/do_manage_webhooks/do_manage_tokens/do_manage_settings/
do_api_call/do_app_api) and the app_api blocklist constants out of
tool_implementations.py into a new src/tools/system.py module.

tool_implementations.py re-imports all of them so it stays a working
backward-compatible facade (shim test stays green).

- do_manage_mcp resolves get_mcp_manager via a function-local import
  from tool_implementations so the test that patches
  src.tool_implementations.get_mcp_manager still applies post-move.
- do_app_api imports _internal_headers and _INTERNAL_BASE (still in
  tool_implementations) function-locally to avoid a circular import.
- Repoint test_context_budget introspection assertion to the moved
  code's new home in src/tools/system.py.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(tools): extract cookbook domain into src/tools/cookbook.py

Moves the model-serving (cookbook) tool domain out of tool_implementations.py
into src/tools/cookbook.py as part of slice 1 (#4082/#4071):

- 13 do_* tools: download/serve/list/stop/tail/search/adopt/cached models,
  list downloads/cancel, list cookbook servers, serve presets
- 9 private helpers: _cookbook_servers, _resolve_cookbook_host,
  _cookbook_env_for_host, _infer_serve_{port,host}, _ensure_served_endpoint,
  _cookbook_register_task, _cookbook_apply_retry_suggestion,
  _scan_running_model_processes, _cookbook_kill_session
- _MODEL_PROCESS_PATTERNS constant (used only by _scan_running_model_processes)

tool_implementations.py stays a backward-compatible facade via a re-import
from src.tools.cookbook; src/tools/__init__ re-exports the same symbols.

_internal_headers and _INTERNAL_BASE stay in tool_implementations.py (shared
by system.py's do_app_api and many cookbook funcs). Each cookbook function
that needs them does a function-local import to avoid a top-level circular
dependency, matching the system-domain split.

Verified: compileall clean; shim test green; cookbook-touching suite
(652 passed, 1 skipped); full suite 3587 passed, 2 failed
(pre-existing test_api_chat_security, unrelated).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(tools): extract search domain into src/tools/search.py

* refactor(tools): extract notes domain into src/tools/notes.py

* refactor(tools): extract calendar domain into src/tools/calendar.py

Repoints tests/test_caldav_bidirectional_sync.py source-introspection
to src/tools/calendar.py (do_manage_calendar moved there).

* refactor(tools): extract image domain into src/tools/image.py

* refactor(tools): extract research domain into src/tools/research.py

* refactor(tools): extract contacts domain into src/tools/contacts.py

* refactor(tools): extract vault domain into src/tools/vault.py

Repoints tests/test_vault_password_not_in_argv.py source-introspection
to src/tools/vault.py (the vault do_* helpers moved there).

* refactor(tools): collapse tool_implementations to clean re-export shim

Move shared _INTERNAL_BASE/_internal_headers to src/tools/_common.py and
drop the duplicate _parse_tool_args (already in _common). tool_implementations.py
is now a pure re-export facade (+ 3 pre-existing email-context helpers, out of
scope). Domain files' function-local imports of these names still resolve via
the facade re-export.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(tools): port upstream cookbook workflow changes to split module

Rebase onto dev dropped c504214 ("Cookbook model workflow fixes") edits
to do_serve_model / do_tail_serve_output: the extraction commit moved
the pre-edit bodies into src/tools/cookbook.py and git auto-accepted the
deletion from tool_implementations.py, losing dev's changes. Restore them
in their post-split home:

- do_serve_model: add where/log_path/next_tools and the expanded
  "Next required check" output message
- do_tail_serve_output: empty-output fallback message replacing
  "(empty pane)"

(do_manage_settings web_fetch alias edit was already applied to
src/tools/system.py during the system-extract conflict resolution.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(tools): break admin_tools circular import in split facade

After rebasing onto dev (#3629 moved the admin manage_* tools into
src/agent_tools/admin_tools), the facade re-exported them via a top-level
`from src.agent_tools.admin_tools import ...`. But src.agent_tools.__init__
imports this facade at top level, so the eager import re-entered the
partially-initialized agent_tools package and broke collection.

Re-export the admin symbols (do_manage_endpoints/mcp/webhooks/tokens/
settings, _MCP_DENIED_COMMANDS, _validate_mcp_command) lazily through
module __getattr__ instead, and drop them from src/tools/__init__ (they
no longer live in the src.tools package). system.py now holds only the
skills/tasks/api bridges; admin tools live solely in admin_tools.py,
matching upstream.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(tools): re-export dropped helpers through the split shim

Address review finding from #4423: the compatibility facade claimed to
preserve every original top-level symbol but omitted three helpers the
old src.tool_implementations exposed. Re-export them and pin them in
the shim protection test:

- _string_arg, _validate_cookbook_ssh_target <- src/tools/cookbook.py
- _mcp_allowed_commands <- src/agent_tools/admin_tools.py (lazily via
  __getattr__, to keep the agent_tools.__init__ <-> facade import acyclic
  after the #3629 admin-tools migration)

All three added to tests/test_tool_implementations_shim.py _EXPECTED so
the test contract now matches its "every original top-level function"
comment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(tools): self-verify shim re-exports every domain do_*

The hand-maintained _EXPECTED list in the shim protection test can drift
silently when a new tool is added to a domain module but not re-exported
by the facade — exactly the omission a reviewer flagged post-split.
Add an auto-discovering test that enumerates every do_* from the domain
modules (incl. admin_tools) and asserts reachability through the shim,
so a forgotten re-export fails the build automatically.

Uses hasattr (not dir(ti)) because the admin symbols are re-exported
lazily via module __getattr__ and don't appear in dir(ti).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(tools): self-verify every in-repo facade import resolves

RaresKeY's P3 on the shim test was a claim-vs-reality gap: the docstring
said it protected "every from src.tool_implementations import X" but the
hand-maintained _EXPECTED list omitted three underscore helpers, so the
claim wasn't enforced. Re-exporting the three (cf1f5e3) fixed the known
gap; this closes the structural one.

Add test_every_facade_import_in_repo_resolves: ast-enumerate every
`from src.tool_implementations import X` site in src/ and tests/ and
assert hasattr(ti, X) for each. A forgotten re-export that anything in
the repo imports now fails the build automatically — including underscore
helpers, which the do_* discovery test does not cover.

Together with test_shim_reexports_every_domain_do_function, the shim
contract is now self-verifying. Demote _EXPECTED in the docstring to the
curated historical/downstream surface (the three helpers have no in-repo
consumer, so they stay manual by necessity) instead of "ground truth".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(tools): dedupe _parse_tool_args + align shim guard with route consumers

Addresses two P3s from review (RaresKeY, 2026-06-26):

1. maintainability — _common carried a full copy of _parse_tool_args
   alongside the canonical src.tool_utils one; future parser fixes could
   diverge. The two bodies were byte-identical in logic, so _common now
   re-exports from tool_utils (a leaf module, no circular-import risk).
   The single-source test is extended to assert _common._parse_tool_args
   and tool_implementations._parse_tool_args are the same object as
   tool_utils._parse_tool_args.

2. test — the shim guard's import-site scan only walked src/ and tests/,
   missing routes/chat_routes.py's clear_active_email/set_active_email
   imports, and _EXPECTED omitted the active-email facade helpers. The
   scan now walks every first-party Python dir (pruning venvs/caches/data
   in-place), and set/get/clear_active_email are added to _EXPECTED
   (get_active_email has no in-repo importer, so the scan alone can't see
   it).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: yuandonghao <yuandonghao@cohl.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-26 15:40:04 +01:00
nikakhalatiani 6cd489f79d Retry oversized embedding requests (#1106) 2026-06-26 14:21:27 +01:00
Rishi Sharma 6ee51b6b10 feat: add dismiss (×) button to all toast notifications (#1355) (#1755) 
* feat: add dismiss (×) button to all toast notifications (#1355)

* Refresh README presentation

* fix: reset pointer-events on toast dismiss button click

Action toasts set pointer-events:auto on #toast for their clickable
button, but the × close-button handler only cleared the auto-hide timer
without resetting pointer-events. This left an invisible fixed overlay
blocking clicks in the top-right area after manual dismissal.

- Add pointerEvents reset in both showToast and showError close handlers
- Add DOM behavior tests for pointer-events across all toast types

---------

Co-authored-by: pewdiepie-archdaemon <pewdiepie-archdaemon@users.noreply.github.com>
 2026-06-26 14:02:35 +01:00
Hinode a5b60a34ee fix: group selection drop-downs recreation and repopulation logic (#3424) 
* fix: include in-memory templates in group participant character list

_getCharacterList() only fetched user templates from the /api/presets/templates
endpoint. When a character was just created in the Character tab, the async
auto-save to the templates API might not have completed by the time the Group
tab loaded its participant dropdown — causing newly created characters to be
missing.

Now also merges the in-memory userTemplates array from presets.js as a
fallback. These are updated as soon as the async save completes (via the
loadUserTemplates callback), so they bridge the gap between character creation
and API persistence.

Fixes #3207

* fix: optimistic userTemplates update on character save

Update the in-memory userTemplates array immediately when saveCustomPreset()
succeeds, before the fire-and-forget templates API POST completes. This
bridges the timing gap where _getCharacterList() calls getUserTemplates()
and gets stale data because loadUserTemplates() hasn't been triggered yet.

* test: verify group participant dropdown merges in-memory templates

Source-level guards for the #3207 fix:
- group.js imports and calls getUserTemplates() to merge in-memory templates
- presets.js exports getUserTemplates and does optimistic in-memory update on save

5 tests ensuring the fix can't be silently reverted.

* fix: generate client-side id for optimistic update, return shallow copy from getUserTemplates

1. New characters now get a 'user-<hex>' id immediately on save, matching
   the server's convention (uuid.uuid4().hex[:8]). Previously the id was ''
   which the merge guard in _getCharacterList filtered as falsy.

2. getUserTemplates() now returns [...userTemplates] so callers cannot
   accidentally mutate module state.

* fix(group.js): fix selection drop-downs behavior

- add an identifier to the selection drop-downs
  based on what type it is.
- fix behavior of continuously adding a row
  when a user clicks the "Group" tab button.
- fix behavior of not repopulating existing
  selection drop-downs whenever a user
  clicks the "Group" tab button.

* fix(#3207): remove duplicate of latest persona

- fix the duplication of the latest persona
  or character being shown in selection
  drop-downs.
- remove unnecessary blocks of code in
  `_getCharacterList()`
- add functionality to show error toast if saving
  a preset template/character fails.
- add functionality to revert optimistic update
  of preset template/character if saving fails.

* chore(group.js,preset.js): fix test & format errors

remove trailing whitespaces in lines 230 and 232
in /static/group.js

add back the expected syntax from
tests/test_group_character_dropdown.py

* fix(presets.js,group.js): fix runtime errors

as stated in a comment by @alteixeira20,
runtime errors exist for the applied fixes.

fixes:

- missing ending `]`
  querySelectorAll("select.preset-input[data-selection-type=character")
  in `group.js`
- spelling error in `modelSelection.vale` in `group.js`
- fix the ordering logic error in optimistic rollback where `Object.assign` is called first before the clone happens in `saveCustomPreset` in `presets.js`.
- add tests for the cloning logic bug with the same format as previous tests by checking the order of LOC in `tests/test_group_character_dropdown.py`.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Alexandre Teixeira <111787685+alteixeira20@users.noreply.github.com>
 2026-06-26 13:35:25 +01:00
Dividesbyzer0 f5200ec45b fix(cookbook): treat local Windows as Windows for serve commands (#3975) 
* fix(cookbook): prefer native llama-server on local Windows

* fix(cookbook): harden local llama-server launch commands

* fix(cookbook): build serve commands for selected target
 2026-06-26 13:13:01 +01:00
Kenny Van de Maele de12d4734a fix(ui): route tasks.js + skills.js dropdowns through topPortalZ() (#4768) 
Fixes #4767. #4724 routed 16 body-portaled dropdowns through the shared
topPortalZ() helper so they always render just above the currently-raised tool
modal, but two were missed and still used a hardcoded z-index, so they hit the
same #4720 bug once a modal's bring-to-front counter climbed past the literal:

  - tasks.js _showTaskDropdown(): inline z-index:100000 on .task-dropdown
  - skills.js kebab menu (.skill-kebab-menu): z-index:100002 in style.css

Both now set zIndex from topPortalZ() after they are appended to the body,
matching the other migrated sites. The dead CSS z-index on .skill-kebab-menu is
removed (the inline value always wins). test_portal_dropdown_z_js.py gains a
source guard asserting both files use topPortalZ() and that no hardcoded
100000/100002 portal literal survives in either file or style.css.
 2026-06-24 22:29:36 +02:00
Samy 5d23495eb2 fix(cookbook): only block model launch on real port collisions (#4760) 
* Fix #4507: only block model launch on real port collisions

Quick-run hardcoded port 8000 and never called _nextAvailablePort(), so
every launch collided. Both pre-launch guards (serve panel + quick-run)
were count-based and fired regardless of port.

- quick-run now auto-assigns a free port (8080 for llama.cpp)
- both guards parse the new port and only prompt on a real overlap,
  stopping only the colliding serve
- dialog reports the actual port instead of a hardcoded 8000

* refactor(cookbook): share _taskPort for port parsing; auto-assign llama.cpp port

Addresses review on #4760:
- _taskPort regex now matches --port= as well as --port (space)
- _nextAvailablePort and both launch guards reuse _taskPort instead of inline regex
- quick-run llama.cpp no longer pins 8080, so two can run concurrently

* fix(cookbook): _taskPort also parses -p; add port-parsing tests

Addresses review on #4760:
- _taskPort now matches -p <n> too, so it's the complete single reader
  (was missing the short flag that other readers already handle)
- add tests/test_cookbook_port_parsing_js.py covering the port forms,
  shared-reader reuse, and llama.cpp auto-assign

* test(cookbook): extract pure port helpers and test behavior

Addresses review on #4760: the prior tests only asserted source strings.
- extract portOf() and nextFreePort() into static/js/cookbookPorts.js
- cookbookRunning.js imports them; _taskPort and _nextAvailablePort delegate
- tests run the helpers via node and assert real behavior: all port forms
  (--port, --port=, -p, -p=), next-free-port skipping taken ports, and the
  same-port-clash / different-port-coexist outcome

---------

Co-authored-by: samy <samy@odysseus.boukouro.com>
 2026-06-24 19:44:09 +02:00
Solanki Sumit 22379fe736 fix(model-routes): harden _probe_endpoint against malformed model-list responses (#4789) 
* fix(model-routes): harden _probe_endpoint against malformed model-list responses

_probe_endpoint parsed model lists with data.get(...) at four sites without
checking that data is a dict, and built the list with a truthiness-only
filter. A /models (or /api/tags) endpoint returning HTTP 200 with valid but
non-dict JSON ([], "x", null, 123) made data.get(...) raise AttributeError,
and a non-string id like 123 passed the filter and then hit .startswith() /
.lower() in the Z.AI/Kimi curated merge and _is_chat_model(). Both errors are
swallowed by the broad except Exception, but the comprehension dies mid-list
so the ENTIRE probed model list is discarded and the endpoint silently
degrades — masking a misconfigured/non-compliant upstream as "no models".

- Guard each data.get(...) with isinstance(data, dict) so a non-dict body
  falls through the existing `or []` default.
- Restrict the OpenAI and Ollama model-list comprehensions to non-empty str
  values, protecting the .startswith() merges and both _is_chat_model calls.
- Add an isinstance guard at the top of _is_chat_model (defense in depth for
  all four call sites).

No behavior change for well-formed {"data":[...]} / {"models":[...]}
responses. Adds regression tests (non-dict body via caplog, mixed/all
non-string ids, _is_chat_model boundary) that fail before the fix and pass
after.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(model-routes): extract _openai_model_ids / _ollama_model_names helpers

Per review on #4789: the malformed-response guards were inlined four times in
_probe_endpoint (two OpenAI-id comprehensions, two Ollama-name comprehensions).
Pull each into a small, directly-testable helper so the security-relevant
parsing lives in one place and a future malformed-shape fix doesn't have to be
applied in four spots (CONTRIBUTING flags repeated logic for this reason).

Behavior is unchanged. Adds direct unit tests for both helpers (non-dict body,
non-string ids, non-dict entries, name>model precedence).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-24 19:05:31 +02:00
Magiomakes 4e46e415ea fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619) 
Upstream bug (present in pewdiepie-archdaemon/odysseus main): the task
executor passes task.endpoint_url VERBATIM to the model HTTP call, unlike
the chat path which stores build_chat_url(normalize_base(base)) on the
session. A task carrying an explicit bare OpenAI-compatible base such as
"http://host:11434/v1" therefore POSTs to a 404 ("page not found"); the
agent loop swallows the empty body into "The model returned an empty
response" and marks the run success, so nothing surfaces the failure.

Tasks that omit an endpoint dodge this only because _resolve_defaults()
cribs an already-full URL from a recent chat session. The API/token path
(e.g. an external client that POSTs /api/tasks with endpoint_url=".../v1")
hits it every time.

Fix: route every resolved task endpoint through _normalize_chat_endpoint()
at the three resolution sites (_execute_llm_task, the persona/research
session path, and _execute_research_task). The helper is idempotent
(strips any existing chat suffix, re-appends the correct one) and leaves
native-Ollama (/api...) and already-concrete URLs untouched, so other
providers are unaffected. Proven via isolated repro: ".../v1" -> 404 ->
empty; ".../v1/chat/completions" -> 200 -> real gemma4:31b output.

Regression test asserts the bare-/v1 -> full-chat-URL mapping, idempotency,
and the native-Ollama/empty passthroughs.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-24 18:02:31 +02:00
Solanki Sumit 6a2a39f892 refactor(exceptions): dedupe src/exceptions via core re-export (#4785) 
src/exceptions.py was a byte-for-byte duplicate of the canonical
core/exceptions.py. Replace its class bodies with a re-export shim
(mirroring the core/constants.py -> src/constants.py pattern) so the
exception classes are defined in exactly one place. Also fix the stale
"# src/exceptions.py" header comment in core/exceptions.py.

No behavior change: both import paths resolve to the same class objects
(verified by identity), so `except SessionNotFoundError` works regardless
of which module it was imported from. Ran py_compile and
pytest tests/test_app.py (12 passed).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-24 16:50:07 +02:00
GeekLuffy 413e628a30 Merge remote-tracking branch 'upstream/dev' into feat/llm-self-eval 2026-06-24 13:07:10 +05:30
Kenny Van de Maele 5ce2056521 refactor(tools): migrate config/integration admin tools to the registry (#4742) 
Part of #3629 (the `admin_tools.py` bullet). Moves the config/integration admin
tools off the legacy elif dispatch chain in tool_implementations.py onto the
agent_tools registry:

  manage_endpoints, manage_mcp, manage_webhooks, manage_tokens, manage_settings

The do_* implementations (and manage_mcp's command-allowlist / RCE guard:
_validate_mcp_command, _mcp_allowed_commands, and the _MCP_* constants) move
verbatim into the new src/agent_tools/admin_tools.py. They register through a
single ADMIN_TOOL_HANDLERS map that TOOL_HANDLERS.update()s, and the five elif
branches plus their imports are dropped from tool_execution.py, so these tools
now flow through _direct_fallback like the other migrated clusters. The names
are re-exported from src.agent_tools for back-compat.

Dedup:
  - _parse_tool_args was duplicated in tool_implementations.py and
    document_tools.py. It now lives once in src.tool_utils (which imports nothing
    from the project beyond src.constants, so this introduces no cycle) and both
    call sites import it from there. The orphaned `import json` in document_tools
    is removed with it.
  - The five tools share one _owner_adapter(fn) factory that threads ctx["owner"]
    into the owner-taking do_* signature, instead of five near-identical wrappers.

Tests: new tests/test_admin_tools_registry.py pins the registration, the
re-export back-compat, the owner-threading adapter, and the single-source
_parse_tool_args (across admin_tools and document_tools). Existing MCP /
settings / webhook suites are repointed at the new module.
 2026-06-24 09:29:10 +02:00
Joel Alejandro Escareño Fernández e0ccf250a4 feat(discovery): detect llama.cpp servers and label local providers (#4729) 
* feat(discovery): detect llama.cpp servers and label local providers

Scan port 8080 (llama-server) and 11435 (APFEL) during discovery, fingerprint
llama.cpp via its native /props endpoint, and label well-known local serving
ports (8080 llama.cpp, 8000 vLLM, 1234 LM Studio, 11434 Ollama) consistently
in both the Python provider helper and the JS endpoint UI. Adds a llama.cpp
hint to the /setup slash command.

* fix(discovery): don't infer the serving tool from the port alone

Per review: vLLM, SGLang, llama.cpp and plain OpenAI-compatible servers all
share 8000/8080, so labeling by port mislabels real setups (a vLLM box on 8080
shown as llama.cpp). Drop the port->tool assertions from _provider_label and
providerLabel; the authoritative signal is the /props fingerprint done during
discovery, which is unchanged. Loopback now reads a neutral 'local endpoint' /
'Local'. Tests updated to assert the neutral labels.
 2026-06-23 23:39:56 +02:00
Michael 72c0bde8a9 fix: use atomic write in APIKeyManager.save() to prevent credential data loss (#4591) (#4597) 
* fix: use atomic write in APIKeyManager.save() to prevent data loss

Opening api_keys.json with 'w' truncates the file before writing, so a
crash, disk-full, or mid-write error leaves all stored provider API keys
corrupted. Switch to atomic write (temp file + fsync + os.replace) so
the original file is always intact on any failure.

Fixes #4591

* chore: trigger CI re-run

* chore: update PR description

* chore: fix how-to-test section for description check

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
 2026-06-23 23:28:53 +02:00
Dividesbyzer0 2e16394b41 fix(agent): parse misfenced read_file calls (#4799) 2026-06-23 23:20:13 +02:00
Jakub Grula 060dbf0681 feat: Allow admins to choose if they want to share defaults (#4752) 
* First bare fix

* Adding the option toggle

* toggle function fix

* Final fix, added missing /auth/

* Extended toggle text & added tests

* Comments change

* Description toggle change

* br tag fix

* description change based on suggestion
 2026-06-23 23:06:45 +02:00
Skoh d9ad418195 feat(ui): add toggle for padding around chat area (#4691) 2026-06-23 22:20:17 +02:00
Rudra Sarker 08994a0a96 fix: email poller marks calendar extraction processed on LLM failure (#4622) 
Move calendar processed-marker insert into the LLM success path (else branch).
Previously, the INSERT ran even after a transient LLM failure, causing the
poller to skip retrying calendar extraction on subsequent runs.

Minimal change: only touches the try/except/else control flow in
_auto_summarize_pass_single() — preserves existing formatting and line endings.
 2026-06-23 20:32:30 +02:00
Solanki Sumit e9136f801a fix(setup): load .env so a pre-seeded admin password is honored on native installs (#4787) 
setup.py read ODYSSEUS_ADMIN_USER / ODYSSEUS_ADMIN_PASSWORD via os.getenv()
but never loaded .env, so on native Linux/macOS installs a password
pre-seeded in .env (documented in docs/setup.md and .env.example) was
silently ignored and a random one generated, breaking the first login.
Docker was unaffected because compose passes the vars into the container env.

Call load_dotenv(BASE_DIR/.env, encoding="utf-8-sig") at the top of main(),
mirroring app.py (utf-8-sig tolerates a Notepad UTF-8 BOM). load_dotenv does
not override already-exported OS vars, so the existing precedence is kept.
python-dotenv is already a required dependency.

Adds a regression test that pre-seeds credentials only in .env (not the
shell) and asserts the stored bcrypt hash matches the pre-seeded password.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-23 20:08:05 +02:00
Ahmed Dlshad e90dbc1012 fix(routes): 500 (not 404) when the app-shell index.html is missing (#4791) 
Follow-up to #4637. serve_index — the handler for / and the SPA deep-link
routes (/notes, /calendar, /cookbook, /email, /memory, /gallery, /tasks,
/library) — pre-checked os.path.exists and raised its own
HTTPException(404, "index.html not found") when the bundle was missing. So a
missing core template returned 404 before serve_html_with_nonce's 500 could
fire, the one inconsistency left after #4637.

index.html is a fixed, app-bundled template; a missing one is a broken
deployment (server fault), not a client "not found", so it should surface as a
logged 500 in 5xx alerting rather than a 404. Keep the static->root fallback,
drop the redundant existence guard and the dead-end 404, and let the shared
helper handle the missing case.

Verified against the running app: / and /notes return 200 with the bundle
present and a logged 500 when index.html is absent.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-23 19:47:22 +02:00
Pedro Barbosa d47715036a fix: Real-ESRGAN install + Cookbook deps-panel crash on the Python 3.14 image (#4694) 
* fix(docker): make Real-ESRGAN installable on the Python 3.14 image

realesrgan's deps basicsr/gfpgan/facexlib (unmaintained since 2022) read
their version in setup.py via `exec(...); locals()['__version__']`, which
raises KeyError on Python 3.13+ — PEP 667 made locals() in a function an
independent snapshot that exec() can no longer mutate. That fails the
Cookbook "install realesrgan" sdist build on the python:3.14 base.

Add a `realesrgan-wheels` builder stage that fetches the pinned sdists,
patches get_version() to exec into an explicit namespace dict, and builds
wheels; the final stage installs them --no-deps so a later
`pip install realesrgan` resolves from wheels instead of rebuilding the
broken sdists. torch stays a runtime pull to keep the base image lean.

Also add the runtime libs opencv-python (cv2) needs — libgl1,
libglib2.0-0t64, libxcb1 — which the slim base omits; without them the
install succeeds but `import cv2` dies with
`libxcb.so.1: cannot open shared object file`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cookbook): don't let a package's sys.exit() on import hang the deps panel

The local optional-dependency probe imports each package in-process and
catches ImportError / Exception. But a package can call sys.exit() at
import time — e.g. rembg does `sys.exit(1)` when no onnxruntime backend
loads. SystemExit is a BaseException, not Exception, so it escaped the
probe, propagated out of the list_packages endpoint, and hung the whole
Dependencies panel / worker (the UI loads forever).

Catch (Exception, SystemExit) so one broken optional package is reported
as not-usable instead of taking down the panel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-23 19:31:00 +02:00
Kalin Stoyanov 87407b3a09 fix debugging on windows (#4679) 2026-06-23 18:44:05 +02:00
Joel Alejandro Escareño Fernández 119228a6db feat(catalog): add Gemma 4 12B/QAT entries and RTX 3050 bandwidth (#4728) 
Add official Gemma 4 12B-it plus QAT-INT4/INT8 catalog entries (with their
GGUF sources), QAT quantization support across the quant tables and the
prequantized-prefix list, and the missing RTX 3050 / 3050 Ti memory
bandwidth so speed estimates stop falling back to the generic cuda value.
 2026-06-23 18:23:46 +02:00
Ahmed Dlshad 8f5e36a079 fix(routes): log and cleanly 500 on unreadable HTML page (#4637) 
* fix(routes): serve 404 instead of 500 when an HTML page file is missing

_serve_html_with_nonce opened the HTML file with no error handling, and
callers such as /backgrounds and /login pass their paths in with no
existence check, so a missing or unreadable file raised an unhandled
OSError that surfaced as a 500. Wrap the read and raise HTTPException(404)
instead; the normal render path (CSP-nonce substitution) is unchanged.

Fixes #4594

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): distinguish missing page (404) from read failure (500)

The previous fix caught a broad OSError and returned 404 for every
failure, which masks real server-side problems (permission errors, I/O
failures) as "not found" and lets them slip past error alerting. Split
FileNotFoundError (genuine 404) from other OSError, which now logs the
exception and returns a generic 500 — without leaking the OS error
string or file path into the response body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): treat unreadable bundled HTML page as logged 500, not 404

Per PR #4637 review: every caller of the page-render helper serves a fixed,
server-owned template (index/login/backgrounds), never a client-supplied
path. So a missing or unreadable file is a server fault (broken deployment),
not a client "not found" — a 404 there mislabels a server error and hides a
missing core template from 5xx alerting, contradicting the OSError->500
rationale this PR is built on. Collapse both branches into a single logged,
leak-free 500.

Move the helper to src.app_helpers.serve_html_with_nonce so the behavior can
be unit-tested without importing the whole app (app.py is the slim
orchestrator; the test harness stubs src.database, so importing app in tests
is not viable). Add tests pinning missing/unreadable -> 500 (not 404) and
nonce injection on the happy path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-23 16:12:32 +02:00
Max Hsu 30dd789351 fix(chat): strip executed email tool fences from the live stream (#3993) (#4275) 
* fix(chat): strip executed email tool fences from the live stream (#3993)

The backend strips every fenced tool block from persisted text (the regex in
src/tool_parsing.py is built from the full TOOL_TAGS set, which includes the
email tools), so a reloaded session renders cleanly. The live frontend path
uses a separate hardcoded EXEC_FENCE_RE in static/js/chatRenderer.js that only
listed web_search/read_file/write_file/create_document/edit_document/
update_document — so executed email tool fences (list_emails, etc.) lingered as
raw code blocks in the live assistant bubble until the user reloaded.

Add the nine email tool tags to EXEC_FENCE_RE so the live render settles into
the same clean layout as the history reload. bash/python stay excluded on
purpose: those are languages a user may legitimately have asked the model to
show as code, not tool invocations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chat): single-source live exec-fence tool list from TOOL_TAGS (#3993)

Per review: EXEC_FENCE_RE was a second, hand-maintained copy of the
executable-tool list, so any tool not in it — and every future tool added to
TOOL_TAGS — would leave its executed fence lingering in the live bubble until
reload (the original #3993 bug, recurring one tool at a time).

EXEC_FENCE_RE is now built from an explicit EXEC_TOOL_TAGS list that mirrors
TOOL_TAGS (src/agent_tools/__init__.py) minus bash/python, which stay excluded
as legitimate code-example languages. A new regression test
(test_exec_fence_re_covers_all_executable_tools) extracts both lists from
source and fails if they drift, so the whole class is caught in CI instead of
by a user — the "minimum acceptable middle ground" from the review, made exact
(set equality, not just coverage).

Verified: pytest tests/test_live_strip_email_tool_fences.py (5 passed);
node --check static/js/chatRenderer.js; and a node run of the built regex
confirms email/generate_image/manage_memory/ls fences strip while
bash/python/sh are preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chat): build live exec-fence list from /api/tools at runtime (#3993)

Make TOOL_TAGS the single source for live exec-fence stripping. chatRenderer.js
no longer hard-codes a tool list; it fetches the backend's authoritative set
once from GET /api/tools (sorted(TOOL_TAGS)) and builds EXEC_FENCE_RE from it at
load, minus bash/python. No second list to drift, and a future tool added to
TOOL_TAGS is covered automatically — without touching the streaming path.

Until the fetch resolves EXEC_FENCE_RE is null and exec fences aren't stripped
(a sub-second window before the first stream); the backend already strips
persisted history, so a reload always renders clean.

Drop test_exec_fence_re_covers_all_executable_tools (no hand-maintained list to
guard) and add source-level guards: the frontend keeps no hard-coded list and
fetches /api/tools, and the endpoint serves the full sorted(TOOL_TAGS).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL

* fix(chat): warn on /api/tools fetch failure instead of swallowing it (#3993)

A fresh-context review flagged that loadExecFenceRegex's catch silently
discarded errors: if the one-shot fetch fails, EXEC_FENCE_RE stays null for the
whole session and live exec fences go unstripped until reload, with zero signal.
console.warn it, and correct the comment to describe the failure mode honestly
(was understated as just a sub-second startup window).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-23 14:12:32 +02:00
Michael e8175c9535 fix: Images cannot be seen by model that is vision capable (#4726) 
* fix: Images cannot be seen by model that is vision capable

* fix: skip http(s) image_url for Ollama (images[] is base64-only)

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
 2026-06-23 10:32:57 +02:00
aubrey bd9149f79a fix(llm): detect mistral.ai provider and support reasoning_effort (#4698) 
* fix(llm): detect mistral.ai provider and support reasoning_effort

Four coupled bugs broke Mistral thinking model support:

1. _detect_provider() had no mistral.ai host check, so all Mistral
   endpoints fell through to the generic 'openai' provider string.
   _provider_display_name() correctly identified them as 'Mistral',
   making any 'if provider == "Mistral"' check elsewhere dead code.

2. reasoning_effort parameter was never sent in the request payload,
   so Mistral never activated thinking mode even when the user
   configured a thinking-capable model (mistral-small-latest,
   mistral-medium-latest, magistral-*).

3. Mistral returns content as a typed array
   ([{"type":"thinking",...},{"type":"text",...}]) when
   reasoning is on, not as a plain string. Both the streaming and
   non-streaming parsers expected strings and silently dropped the
   thinking content.

4. _THINKING_MODEL_PATTERNS didn't include magistral or mistral-*
   model prefixes, so the frontend wouldn't tag reasoning output
   as thinking even after the above were fixed.

Fix:
- Add mistral.ai to _detect_provider() host checks
- Add a _normalize_mistral_content() helper that splits the typed
  array into (text, thinking) strings
- Inject payload["reasoning_effort"] = "high" when provider is
  Mistral and _supports_thinking(model) is true, in both stream_llm
  and llm_call_async payload construction
- Wire the normalizer into both response parsers
- Extend _THINKING_MODEL_PATTERNS to include magistral,
  mistral-small, mistral-medium, mistral-large

Tested on Docker install with mistral-small-latest +
reasoning_effort=high. Reasoning streams correctly into the
thinking panel after the fix.

Fixes #4678

* fix(llm): address review — lowercase provider id, configurable effort, tests

Addresses vdmkenny's review on PR #4698:

1. Removed duplicate 'if provider == "mistral"' block in stream_llm
   — two back-to-back copies, one was dead-redundant.

2. Dropped personal-context comment ('free-tier limits are generous
   for this user') and made reasoning_effort configurable via env var
   ODYSSEUS_MISTRAL_REASONING_EFFORT (high / medium / low / none).
   Default remains 'high' for backward compat with the tested behavior.

3. Recased provider id from 'Mistral' to 'mistral' to match the
   lowercase convention used by every other provider id in the file
   (openai, anthropic, ollama, copilot, ...). _provider_display_name()
   still returns the Title-Case 'Mistral' for UI labels — only the
   runtime id used in 'if provider == ...' checks was recased.

4. Added tests/test_llm_core_mistral_content.py with 13 tests pinning
   _normalize_mistral_content()'s contract: string passthrough, the
   Mistral array format (thinking + text blocks), and edge cases
   (empty, garbage, None, wrong types, missing fields, string-vs-array
   inner thinking field).

Also fixed a gap the review didn't catch: the non-streaming paths
(llm_call sync + llm_call_async) were missing the reasoning_effort
injection entirely. Added the same injection to both, so Deep Research
and agent tool calls also activate Mistral thinking.

All 13 new tests pass. Existing reasoning/streaming/ollama-thinking
tests still pass (38 tests, no regressions).

Fixes #4678
 2026-06-23 10:28:17 +02:00
Max Hsu fef08ed114 fix(modal): keep body-portaled dropdowns above their tool modal at any stack depth (#4720) (#4724) 
* fix(memory): keep the Brain memory item menu above the modal at any stack depth

The memory item "⋮" dropdown is portaled to <body> with a hardcoded
z-index of 10001. Tool modals, however, get a monotonically increasing
z-index from modalManager's bring-to-front counter (_modalTopZ), which
climbs unbounded as modals are opened/restored over a session. Once that
counter passes 10001, the Brain modal stacks above the body-portaled
dropdown, so the menu renders behind the panel — visible only where it
spills past the modal's edge (#4720).

Derive the dropdown's z-index from the owning modal's current z-index
(+1), keeping 10001 as a floor for the common low-counter case, so the
menu always sits just above its modal however high the counter has climbed.

Verified with document.elementFromPoint at the dropdown's location: with a
high modal z-index the old build returns the modal at every sampled point
(menu behind); the fixed build returns the dropdown (menu on top). The
default low-counter case is unchanged (z stays 10001).

* refactor(modal): route body-portaled dropdowns through a shared topPortalZ() helper

The hardcoded z-index:10001 the Brain memory menu used (#4720) is the same
literal shared by ~16 body-portaled dropdowns across calendar, cookbook,
cookbookServe, documentLibrary, emailLibrary, gallery, notes, emojiPicker and
memory — each renders behind its owning tool modal once modalManager's
bring-to-front counter climbs past the literal over a long session.

Promote the per-dropdown fix into a single topPortalZ() helper in
toolWindowZOrder.js — the existing source of truth for tool-window z, already
imported by modalManager's _bringToFront and notes.js — returning
max(topToolWindowZ(), dock-chip floor) + 1, so a portaled dropdown always sits
just above the live tool-window stack however high the counter has climbed.
Route all 16 sites through it. The slashCommands tour tooltips and the
cookbookServe VRAM dialog are intentionally left out (neither is a modal-owned
portaled dropdown).

Add tests/test_portal_dropdown_z_js.py covering the helper, including the #4720
scenario (modal counter at 99999 -> dropdown at 100000). Existing
test_notes_z_order_js.py stays green.
 2026-06-23 10:24:31 +02:00
nopoz 7e5db9a3c6 fix(security): redact credential-bearing URLs and PII from logs (#4750) 
* fix(security): redact credential-bearing URLs and PII from logs

Several log statements emitted sensitive data in clear text:

- model_routes / chat_routes / contacts_routes logged endpoint URLs raw.
  Admin-configured URLs can embed credentials in userinfo or query
  (e.g. https://user:pass@host, ?api_key=...). Route them through a
  shared core.log_safety.redact_url() that drops userinfo/query/fragment.
- note_routes / task_scheduler logged operator email addresses (smtp_user,
  recipient). Replaced with presence booleans, which keeps the diagnostic
  ("why didn't this send") without writing PII to logs.

model_routes already had a local redactor on its HTTPStatusError branch;
the generic except branch was missed, so reuse the existing helper there.

Clears CodeQL py/clear-text-logging-sensitive-data alerts 264, 317, 324,
325, 343, 344, 528.

* fix(security): re-bracket IPv6 hosts and single-source the URL redactor

Address review on #4750:
- redact_url now re-brackets IPv6 literals so host:port stays
  unambiguous (https://[2001:db8::1]:8443/v1, not the bracket-less
  ambiguous form).
- point model_routes._redact_url_for_log at the shared helper so the
  two redactors are single-sourced (also picks up the IPv6 fix).
 2026-06-22 23:12:39 +02:00
nopoz 2f246c7779 fix(security): escape backslashes in calendar bg-image CSS url() (#4712) 
* fix(security): escape backslashes in calendar bg-image CSS url()

The calendar event-background CSS escaped ' -> \' for a bg: image URL but
not backslashes first. Inside a single-quoted url('...'), \ is the CSS
escape char, so a URL value ending in/containing a backslash escapes the
closing quote and breaks out of the string, injecting arbitrary CSS. The
bg:<url> value is per-event and CalDAV-syncable, hence untrusted (CodeQL
js/incomplete-sanitization).

Add a single canonical _cssUrlEscape() in calendar/utils.js that escapes
backslashes FIRST, then quotes, and route all four sinks through it:
calendar.js:416 / :1263 (the flagged #463/#464), the event-form preview
(:2931), and _calBgCss() in utils.js — the latter two share the identical
bug but were unflagged. Output is byte-identical to the old escaping for
legitimate URLs (which contain no backslashes); only malicious input differs.

Resolves CodeQL js/incomplete-sanitization #463, #464.

* fix(security): route remaining calendar bg url() sinks through _cssUrlEscape

Review (vdmkenny) flagged that the centralization missed an injectable
sibling sink: the edit-form color-picker swatch (calendar.js:2856) built
`url('${url}')` from `existing.color` (a CalDAV-syncable, untrusted `bg:`
value) raw, then interpolated it into `style="background:..."` via innerHTML
- the same `'`/`\` breakout class as the sinks already fixed. The custom-dot
preview (:2953) was likewise raw (non-exploitable - a CSSOM `.style`
assignment of a URL the current user just picked - but it broke the invariant).

Route both through `_cssUrlEscape`, and normalize the two pre-escaped-variable
sites (_calItemBgStyle, _renderWeek) to the same inline form so all five
url() interpolations in calendar.js follow one rule. Add a whole-file
invariant test asserting every `url('${...}')` calls `_cssUrlEscape` - this
catches a future missed sink, the exact failure mode here. Behavior-identical
for legitimate URLs (no visual change).
 2026-06-22 21:17:52 +02:00
Rudra Sarker 8ec27fd903 fix: document read fails with 403 when auth is disabled (#4623) 
* fix: document read fails with 403 when auth is disabled

Add _auth_disabled() bypass in _verify_doc_owner() and the
/api/documents/{session_id} route guard so documents remain accessible
in single-user / no-auth mode.

Minimal change: only adds the auth-disabled check alongside existing
403 raises — preserves existing formatting and line endings.

* refactor: hoist _auth_disabled import to module level

Address reviewer feedback on PR #4623 — no circular import exists
(src.auth_helpers only imports stdlib + fastapi), so the inline
imports are unnecessary. Moves the import to module top in both
document_helpers.py and document_routes.py.

* test: add regression tests for auth-disabled document access (PR #4623)
 2026-06-22 21:01:11 +02:00
MACKAT05 b57989f08c fix(hwfit): repair remote Windows hardware scan over SSH (#4674) 
Remote Cookbook hwfit probes failed on Windows hosts because the PowerShell script was sent as nested -Command quoting through OpenSSH. Use -EncodedCommand for remote probes, auto-detect platform when omitted (including Darwin for Mac SSH hosts), and return a clearer error when SSH works but the probe fails.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-22 20:59:09 +02:00
Gabriel Peña 91bba117c1 fix ask-user choices across reloads (#4669) 2026-06-22 20:49:49 +02:00
Mocchibird 4c82e4a172 fix(ui): route transient dropdown menus through escMenuStack to stop listener leaks (#4684) 
The app's ad-hoc dropdown/context menus each wire their own document-level
outside-click listener, but that listener only removes itself on an *outside*
click. Every other dismissal path -- clicking a menu item (which calls
el.remove() directly), a Cancel button, Escape, or the "close the
previously-open menu" reopen sweep -- tears the node down without
unregistering the listener, orphaning it on `document`. The stranded listener
then lingers and can break the next menu interaction: the recurring "the
button stops working until I refresh the page" class of bug (e.g. delete an
email, then the kebab/more button is dead on the other rows).

Route all 16 of these menus through the existing escMenuStack helper
(bindMenuDismiss / dismissOrRemove), exactly as documentLibrary.js
_showLibDropdown, cookbookRunning.js, and research/panel.js already do: a
single idempotent close() owns the teardown and is released on every dismissal
path, reopen sweeps use dismissOrRemove() instead of a bare .remove(), and
Escape flows through the central LIFO esc-stack arbiter. Net -49 lines.

Menus migrated: cookbook _showDepMenu; document export menu and
_openDocAiReplyChoice; emailInbox _showEmailMenu; emailLibrary
_showReaderMoreMenu / _showCardMenu / _showBulkActionsMenu; gallery
_showGalleryBulkMenu; notes _pickCustomDate / _openNoteCornerMenu; settings
(3 unified-integrations dropdowns); skills _openSkillMenu; tasks
_showTaskDropdown; compare _toggleExportMenu.

Per-menu semantics preserved (anchor-as-inside tests, the tasks 250ms
ghost-click guard, emailLibrary's reader-more-active anchor class and the
bulk-Cancel select-mode reset, settings' reused-vs-recreated lifecycles).

Six menus with custom lifecycles (notes _openReminderMenu, sessions
long-press, document markdown-toolbar, emojiPicker, compare model selector)
are intentionally left for a follow-up -- each needs individual review.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-22 20:40:56 +02:00
Ahmed Dlshad b899095f18 docs(setup): note -BindHost flag for LAN access on native Windows (#4636) 
The native Windows launcher binds to 127.0.0.1 via its own -BindHost
parameter and does not read APP_BIND/ODYSSEUS_HOST from .env, so editing
.env alone leaves the server on loopback. Document the -BindHost flag in
the Native Windows setup section, with the existing keep-auth-on /
don't-expose-publicly caveats.

Fixes #4552

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-22 20:29:55 +02:00
Mostafa Eid 888e25624f fix(sessions): prevent Backspace/Delete from deleting session while renaming (#4662) 2026-06-22 20:22:52 +02:00
comatrix-1 c062c27648 Fix link to CONTRIBUTING.md in setup documentation (#4677) 2026-06-22 20:12:04 +02:00
holden093 93ec7cbb52 fix(contacts): verify UID removal after CardDAV DELETE (#4642) 
Add a post-delete verification step: after the CardDAV server returns
2xx/404, force-re-fetch the contact list and confirm the UID is gone.
If the UID is still present, log a warning and return False instead of
silently reporting success.

This catches the case where _resolve_resource_url falls back to the
guessed {uid}.vcf URL but the contact's real resource URL differs —
the DELETE hits the wrong URL, server returns 404 (treated as success),
but the contact remains. Previously this caused silent persistence
failures and agent loops.
 2026-06-22 18:39:44 +02:00
ooovenenoso c12b8ab6c9 fix: add OpenCode setup provider aliases (#4700) 
Co-authored-by: Kevin <120500656+oooindefatigable@users.noreply.github.com>
 2026-06-22 17:33:02 +02:00
Ashvin e812a29233 fix(markdown): preserve URLs inside inline code spans (#4681) 
Inline backtick spans were converted to <code> only at the end of
mdToHtml, after the bare-URL autolink and <a>/allowed-HTML passes. A URL
inside inline code is preceded by a space, so the autolink wrapped it in
an <a> tag and swapped it for an ___ALLOWED_HTML_ placeholder, corrupting
commands like `irm http://127.0.0.1:3000/x`.

Extract inline code into placeholders before the link passes, mirroring
the existing fenced-code-block handling, and restore them last so
placeholders carried inside restored <a> blocks resolve. Escape the code
at extraction time since it now bypasses the global escape pass.
 2026-06-22 17:23:55 +02:00
nopoz ca4973c41f fix(security): prevent exponential ReDoS in email→calendar extract regex (#4708) 
The fallback regex in email_pollers.py that recovers a
[{"action": ...}, ...] JSON array from raw model output used lazy
[^[\]]*? runs inside a (?:,\s*\{...\}\s*)* repetition, which backtracks
exponentially (CodeQL py/redos) on inputs like [{"action"},{ + }},{{ * N.
It runs on the LLM reply to an email→calendar prompt embedding the
untrusted email body, so a crafted email can stall the background poller.

Extract the pattern to a module-level _CAL_ACTION_ARRAY_RE and rewrite the
object-content class from the lazy [^[\]]*? to a greedy brace-delimited
[^{}], which removes the quantifier ambiguity. The match is linear (a 500KB
adversarial input now resolves in <1ms) and equivalent on well-formed
arrays; it is also strictly more robust for values containing '[' or ']'
(the old class bailed on those and extracted nothing).

Resolves CodeQL py/redos #198.
 2026-06-22 17:18:34 +02:00
Tom 91b4171b3f feat(a11y): add a Text size control and an OpenDyslexic font option (#4210) 
* feat(a11y): add a Text size control and an OpenDyslexic font option

Text size: a Theme > Font & Layout control (Default / Larger) that scales the whole UI via CSS zoom, so the many hard-coded px sizes scale too (density only moves the root font-size). Stored globally so it persists across theme switches; applied early in the boot script to avoid a flash. OpenDyslexic: a dyslexia-friendly self-hosted font (SIL OFL 1.1), bundled as woff2 alongside Fira Code/Inter and wired into the Font select. Reuses the existing density/font pattern end to end; no new colours, spacing, or component styles.

* fix(a11y): keep modals on-screen at Larger text size

Inline vh heights on .modal-content overrode the ui-scale-125 max-height
compensation, so Cookbook (and the email/doc/skills/PDF modals) overflowed
the viewport at 125% — pushing the header and close button off-screen.
Let the compensation own those heights.

* fix(a11y): keep PDF export modal at its original 86vh on Default size
 2026-06-22 13:53:46 +02:00
PewDiePie d36879bd50 Merge pull request #4706 from pewdiepie-archdaemon/sync-readme-screenshot-dev 
docs: refresh README screenshot
 2026-06-22 14:02:35 +09:00
pewdiepie-archdaemon b51656770d Refresh README screenshot 2026-06-22 04:54:15 +00:00
PewDiePie 5f63a3d3bd Merge pull request #4701 from pewdiepie-archdaemon/sync-dev-from-main-20260622 
chore(dev): sync main cookbook and model workflow fixes
 2026-06-22 11:52:26 +09:00
pewdiepie-archdaemon 993d504de3 Clear remaining CodeQL path and parser alerts 2026-06-22 02:45:05 +00:00
pewdiepie-archdaemon fbdec22dcb CodeQL hardening for cookbook sync 2026-06-22 02:39:18 +00:00
pewdiepie-archdaemon 19dd82b8f6 CI test fixes for dev sync 2026-06-22 02:20:15 +00:00
pewdiepie-archdaemon 57e7229219 CI fixes for cookbook workflow sync 2026-06-22 02:08:25 +00:00
pewdiepie-archdaemon 92daf4e560 Cookbook launch and gallery upload fixes 2026-06-22 01:49:15 +00:00
pewdiepie-archdaemon 75f04bc088 Merge origin/dev into main 2026-06-21 11:08:50 +00:00
pewdiepie-archdaemon c504214925 Cookbook model workflow fixes 2026-06-21 11:02:35 +00:00
nopoz 160267417e fix(personal): scope RAG file delete to the caller's own upload dir (#4602) 
The DELETE /api/personal/file disk-delete containment check used the
shared PERSONAL_UPLOADS_DIR root, so one admin could delete another
user's personal upload by passing its path (uploads are partitioned per
owner under <root>/<owner>/). Confine the check to the caller's own
per-owner subdir via _personal_upload_dir_for_owner(owner). RAG removal
and listing exclusion are unchanged (they still serve non-upload indexed
sources). Adds a regression test for the cross-owner case.
 2026-06-20 00:50:15 +02:00
Kenny Van de Maele ed18192a8e refactor(tools): move session tools to the agent_tools registry (#4454) 
Moves create_session, list_sessions, send_to_session and manage_session out of
ai_interaction.py into src/agent_tools/session_tools.py (the do_ prefix
dropped) and registers them in TOOL_HANDLERS, so dispatch flows through the
registry instead of the dispatch_ai_tool elif in tool_execution.py. Same
pattern as the model-interaction move.

The bodies move verbatim; each fetches the runtime-set session manager via a
get_session_manager() shim, and reuses _resolve_model / AI_CHAT_TIMEOUT from
ai_interaction. manage_session's internal 'list' alias is repointed from the
old do_list_sessions to the moved list_sessions. stream_ai_tool (dead, no
callers) and do_pipeline stay put. dispatch_ai_tool loses its four now-unused
branches.

Tests: test_session_tools_registry covers registration, owner threading, the
manage_session->list_sessions delegation, graceful no-manager handling, and
registry dispatch. Verified end-to-end against a live SessionManager.
 2026-06-19 11:55:22 +02:00
nopoz 076e8c93c9 fix(ui): escape model name in model-info popup (DOM-XSS) + two latent sinks (#4605) 
chatRenderer.js built the model-info popup HTML by concatenating the
model name (from the LLM response's model/answered_by field) into
popup.innerHTML without escaping, so a model advertised as an HTML/script
payload executed when the user clicked the role label. Wrap both
insertions with the uiModule.esc() helper the same function already uses.

Also apply existing escape helpers at two latent sinks flagged by CodeQL,
fed only by self-authored/server values today: document-tab title via
_esc(), and the calendar event background URL (escape the double quote
that would otherwise break out of the style="..." attribute).
 2026-06-19 11:03:44 +02:00
Kenny Van de Maele a226c94df7 chore(deps): remove unused @anthropic-ai/sdk dependency (#4566) 
Never imported anywhere in the codebase (unused since v1.0); it is the only
root dependency and nothing depends on it. Removing it also drops 6 transitive
packages from the lockfile.

Fixes #4565
 2026-06-19 09:40:35 +02:00
RaresKeY 057ec0552c fix(cookbook): stop Windows process trees (#4283) 2026-06-19 00:28:25 -07:00
Kenny Van de Maele cdae9879f2 feat(agent): add manage_bg_jobs tool to inspect and kill background bash jobs (#4577) 
Detached bash jobs (#!bg) could be launched and auto-reported on completion,
but the agent had no way to act on a running one: no on-demand output read and
no kill (it blocked until the 1h max-runtime). bg_jobs had the pieces
(_read_output, list_for_session, internal _kill) but none was exposed.

Adds:
- bg_jobs.kill(job_id): tears down the process tree, marks the job killed, and
  sets followed_up so the monitor does not also auto-continue a deliberate kill.
- manage_bg_jobs registry tool with actions list / output / kill, scoped to the
  chat that launched the job (cross-session access reads as not found).
- Wiring: TOOL_HANDLERS/TAGS, function schema, RAG index + keyword hints, parser
  name map, dispatch (threads session_id via _direct_fallback). Gated like bash
  (NON_ADMIN_BLOCKED_TOOLS; plan-mode mutator).
- agent_loop: background-job intent regex maps to the files domain (and the tool
  joins _DOMAIN_TOOL_MAP[files]) so short commands like 'kill that job' are not
  dropped by the low-signal gate that skips tool retrieval.
- bg launch message tells the model to call manage_bg_jobs itself for check/stop
  rather than printing raw tool syntax to the user.

Tests: tests/test_bg_job_tools.py (kill semantics, per-chat scoping, actions,
and the intent classifier).
 2026-06-19 00:28:22 -07:00
pewdiepie-archdaemon 8c46172e87 Sidebar + theme: drop hamburger cycle no-op branch; add brandMixTo CSS var to themes for logo-gradient end color 2026-06-19 00:35:08 +00:00
pewdiepie-archdaemon e442cc859d Research panel: inline Library-link hint when there are no past runs (replaces the standalone past-research column) 2026-06-19 00:35:02 +00:00
pewdiepie-archdaemon a01c3da75f Notes: checklist/todo/goal classification + agent-stream-complete state class for done indicator 2026-06-19 00:34:57 +00:00
pewdiepie-archdaemon 23ed92d965 Email Library: render tag chips + spam verdict pill on the email row 2026-06-19 00:34:52 +00:00
pewdiepie-archdaemon 8cc76b53a2 Chat: first-token wait timer cleanup so per-pane timeouts dont leak when a response finishes mid-wait 2026-06-19 00:34:47 +00:00
pewdiepie-archdaemon 20c6ba8f32 Bump APP_VERSION to 1.0.1 2026-06-19 00:34:37 +00:00
pewdiepie-archdaemon 9adb940ef9 Agent stream: 10s heartbeat keepalive on the SSE subscribe so long-running thinking models dont drop the connection 2026-06-19 00:34:30 +00:00
pewdiepie-archdaemon 2fbfd22946 Agent loop: compact one-line tool-usage hints for local/small models so the system prompt doesnt eat the context budget 2026-06-19 00:34:24 +00:00
pewdiepie-archdaemon a10bfc466b Model endpoints: per-category probe timeouts (15s local / 3s ollama / 2s api) so slow first-token launches arent killed 2026-06-19 00:34:19 +00:00
pewdiepie-archdaemon 18f29bdfd8 Email send: normalize address fields to strip trailing commas + stray whitespace before MIME encoding 2026-06-19 00:34:13 +00:00
pewdiepie-archdaemon 63d9b12b22 Cookbook Running: short-circuit polls for Ollama sidecar tasks so status stays running 
Three different background loops (_reconnectTask reachability poll,
_checkServeReachability, _pollBackgroundStatus) each independently
flipped Ollama sidecar tasks between running and stopped because the
`docker exec ollama-rocm ollama show <tag>` cmd exits cleanly after
its verification print, which the loops misread as the serve dying.

Added _isOllamaSidecarTask(task) and an early-bail in each of the
three loops so the task stays pinned to running once the show-cmd
exits 0. Also the tmux-graceful-kill path prepends a
`docker exec ollama-rocm ollama stop <tag>` before tearing down
the tmux session, so the Ollama-side model load gets unloaded too
(was leaving the model resident in the daemon after Stop).
 2026-06-19 00:33:48 +00:00
pewdiepie-archdaemon ee6fd8ffe8 Cookbook UI: backend-aware env vars, always-show MoE/EP/Reasoning toggles, GPU default, Firefox-mobile expand 
Frontend half of the backend-detection + per-OS install command work,
plus a pile of mobile/UX fixes:

Backend awareness:
- _gpuEnvPrefix() picks CUDA_VISIBLE_DEVICES / HIP_VISIBLE_DEVICES /
  nothing based on detected hwfit backend + scanned-host match (so a
  stale ajax scan does not leak CUDA env vars into a kierkegaard
  Vulkan launch). Replaces 6 hardcoded CUDA_VISIBLE_DEVICES sites.
- GGML_CUDA_ENABLE_UNIFIED_MEMORY only emitted when backend is
  actually CUDA (was leaking onto Vulkan/ROCm via saved presets).

Per-target install command:
- Dep rows render a single mono command box + Copy button when the
  server resolved pkg.install_cmd_for_target. Reused in the build-deps
  install failure toast so the toast and the row show the same line.
- Diagnosis patterns split cmake/g++/git out of the generic
  llama-cpp-python catch-all so a missing-cmake failure surfaces a
  cmake-specific message + per-distro Copy buttons.

Form toggles always visible:
- Reasoning Parser, Expert Parallel, MoE Env Vars no longer gated on
  model-family detection. Detection still hints (parser tag shown when
  matched); toggle works with sensible defaults otherwise. MiniMax M-
  series added to MoE family detector so the auto-fill is right.

Mobile + GPU default:
- Launch tab cached-list flex collapsed to 0px on mobile because the
  desktop `flex: 1 1 0` had no parent height to grow into. Override
  to `flex: 0 0 auto` in the cookbook mobile @media block.
- doclib-card expand on mobile (Firefox no :has() support) pins
  explicit px heights so the launch form actually appears.
- llama_mode defaults to gpu when hwfit detected cuda/rocm/vulkan/
  metal on the current target, instead of always cpu (which was
  forcing -ngl 0 on first-open and burning 35GB models on CPU).
 2026-06-19 00:33:37 +00:00
pewdiepie-archdaemon f01465e87f Cookbook Dependencies: per-OS+backend install command + install-system-deps endpoint 
When a llama.cpp launch needs cmake/build-essential/git the user used to
get a four-distro dump ("apt: x / pacman: y / dnf: z / brew: w") and
had to pick the right one. Now:

- shell_routes /api/cookbook/packages probes /etc/os-release on the
  target in the same SSH round-trip as the existing system-prereq
  check, classifies into debian / arch / fedora / alpine / suse /
  macos, and builds a single install_cmd_for_target string from the
  (os_family, backend) matrix. CUDA hosts get nvidia-cuda-toolkit;
  ROCm gets rocm-dev / rocm-hip-sdk; Vulkan gets libvulkan-dev /
  vulkan-headers; etc.

- llama_cpp catalog entry gets system_prereqs: [cmake, g++, git].
  When any of those are missing on the target, the row picks up
  pkg.build_deps_missing + pkg.install_cmd_for_target for the
  frontend to render.

- New POST /api/cookbook/install-system-deps endpoint runs the right
  package manager via passwordless sudo on the target. Allowlisted to
  {cmake, build-essential, g++, gcc, git, tmux, make}; sudo -n only
  so it can never hang waiting for a password (returns a clear
  "passwordless sudo unavailable" error via stderr instead).
 2026-06-19 00:33:19 +00:00
pewdiepie-archdaemon 1324e1b0d5 Cookbook backend detection: report Vulkan on AMD hosts without ROCm; gate CUDA build on actual NVIDIA hardware 
Three classes of incorrect detection fixed:

(1) AMD GPU + no ROCm installed (e.g. Strix Halo) was reported as
    backend=rocm everywhere, so launch commands emitted
    HIP_VISIBLE_DEVICES (silent no-op on Vulkan) and the from-source
    build path failed. Both _probe_amd_sysfs (routes/cookbook_routes)
    and _detect_amd (services/hwfit/hardware) now probe rocminfo /
    hipconfig / vulkaninfo at detection time and report vulkan when
    only Vulkan is present.

(2) Build helper was picking the CUDA branch on AMD hosts whenever a
    stray pip-installed nvcc was on PATH (vLLM wheels carry one
    without libcudart). Added _odysseus_has_nvidia_hw() that checks
    nvidia-smi / /dev/nvidia* / lspci, and gates both the nvcc PATH
    augmentation and the CUDA elif branch on real hardware.

(3) Build chain reordered to ROCm/HIP > CUDA > Vulkan > CPU. Vulkan
    tier added between CUDA and CPU as a portable fallback for hosts
    with a GPU but no native toolchain (the common Strix Halo case).
    Same _append_llama_cpp_linux_accel_build_lines also auto-attempts
    sudo -n apt/pacman/dnf install of cmake/build-essential/git when
    they are missing, surfacing a clear no-passwordless-sudo warning
    otherwise.
 2026-06-19 00:33:07 +00:00
pewdiepie-archdaemon b3e186746a Docker compose: mount docker.sock + install Docker CLI so Cookbook can reach sibling containers 
Cookbook now needs to docker-exec into ollama-rocm (and any other sibling
container holding a model server) from inside its own container, so:

- Dockerfile installs the Docker CLI from the static binary tarball
  (the Debian docker.io package ships dockerd but not the client on slim)
- docker-compose.yml bind-mounts /var/run/docker.sock and adds group_add
  for the host docker group (default GID 963)
- entrypoint.sh detects the socket GID, creates a local group with that
  GID, and runs usermod -aG before gosu-dropping to the app user so the
  supplementary group propagates through (gosu strips by default)
 2026-06-19 00:32:47 +00:00
Michael 39a802bea2 fix(tools): prune skipped dirs before descending in glob tool (#4538) 
* fix(tools): prune skipped dirs before descending in glob tool

GlobTool used pathlib.Path.rglob which descends into every directory
(including node_modules, .git, dist, etc.) and filters AFTER the walk.
On repos with large junk directories this causes the glob tool to hang
for minutes.

Replace rglob with os.walk that prunes _CODENAV_SKIP_DIRS before
descending — matching the approach GrepTool already uses. Also add a
fast path for literal patterns (no wildcards → direct path lookup).

Fixes #4493

* fix(tools): use regex glob matching to fix * semantics and literal fallback

Replace fnmatch with _glob_to_regex so that * stays within a single
path segment (matching pathlib/rglob semantics) and **/ spans zero or
more directories.  Literal patterns now fall through to os.walk when
the direct path lookup misses, so e.g. 'foo.py' still finds files at
any depth.

Add tests for:
- bare literal matching in subdirectories
- multi-segment single-star patterns (sub/*.txt)
- * not crossing / boundaries
- ** matching at arbitrary depth

Closes #4493

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
 2026-06-18 22:02:29 +02:00
RaresKeY 1cc8a373b0 fix(cookbook): validate agent SSH targets (#4429) 2026-06-18 21:41:33 +02:00
Wei Hong a52ac6822b fix(cookbook): pull llama.cpp from the ggml-org GHCR namespace (#4457) (#4490) 
The Dependencies tab's llama.cpp docker recipe surfaced
\`docker pull ghcr.io/ggerganov/llama.cpp:server-cuda\`. The upstream
repo moved from github.com/ggerganov/llama.cpp to
github.com/ggml-org/llama.cpp and the old GHCR namespace no longer
publishes images, so copying the recipe failed with:

  failed to resolve reference "ghcr.io/ggerganov/llama.cpp:server-cuda":
  not found

Point the recipe at \`ghcr.io/ggml-org/llama.cpp:server-cuda\`, which is
already the namespace routes/cookbook_routes.py uses for the source
clone. Adds a regression test in the same shape as
test_cookbook_diagnosis_js.py asserting the new namespace and forbidding
the dead one.

No CSS/HTML/SVG/style changes — the file is a pure data module
(no DOM access) consumed by other renderers; only the displayed command
text changes.
 2026-06-18 21:29:47 +02:00
Wei Hong 7475779b7c fix(chat): track chat hot-path background tasks for strong references (#4443) (#4444) 
Two background tasks scheduled on every chat completion in
routes/chat_helpers.py — the memory/skill extraction dispatch and the
session auto-namer — are created via bare asyncio.create_task(...).
asyncio only holds a weak reference to the outer task, so the GC can
collect it mid-execution and the work silently never runs.

Add a module-private _BG_TASKS set and a _spawn_bg() helper that mirrors
WebhookManager._spawn_tracked (the pattern #3964 / #4336 established for
the webhook emitters two lines apart in the same function). Route both
call sites through it so the lifecycle owner is explicit.

Adds an AST-level guard test that fails on any bare
asyncio.create_task(...) statement in routes/chat_helpers.py to prevent
a regression — same shape as test_webhook_emitters_use_manager.py from
#4336.

The same bare pattern exists in routes/email_routes.py and
routes/cookbook_routes.py; left out of this PR per CONTRIBUTING.md's
"one fix per PR" and tracked in #4443's "Additional Information" for a
follow-up.
 2026-06-18 21:26:11 +02:00
Christian Eriksson e7ffc69729 fix(cookbook): scope the "Kill vLLM" diagnosis to actual vLLM tracebacks (#4517) 
The diagnosis panel offered a "Kill vLLM processes" (pkill -f vllm) recovery
for ANY Python traceback — including pip build failures and other tracebacks
that have nothing to do with vLLM. That advice is useless for a build failure
and harmful if an unrelated vLLM server happens to be running.

ERROR_PATTERNS in static/js/cookbook-diagnosis.js had one catch-all traceback
matcher that always attached the vLLM-kill fix. Split it into three (all
keeping the existing healthy-server suppression):
- pip build failure (Failed to build / metadata-generation-failed /
  subprocess-exited-with-error / Could not build wheels) -> "a dependency
  failed to build" message, no kill.
- vLLM-specific traceback (tail mentions vllm) -> keeps the kill, now scoped.
- any other traceback -> neutral "check the captured output" message, no kill.

How to test:
- node --check static/js/cookbook-diagnosis.js
- Trigger a wheel-build failure (old package on a newer Python) or a non-vLLM
  traceback and open the diagnosis. Before: generic traceback message + "Kill
  vLLM processes" button. After: a build-failure / neutral message with no kill;
  only a real vLLM traceback still offers it.

Fixes #4516

Co-authored-by: Claude
 2026-06-18 21:18:14 +02:00
Karl Jussila 396e26b4bf fix(auth): tie remember-me cookie lifetime to TOKEN_TTL (#4472) 
The persistent login cookie's max_age hardcoded 60 * 60 * 24 * 7, an
independent copy of the session token lifetime that core/auth.py already
defines once as TOKEN_TTL (and reports to the frontend via /api/auth/policy
as session_days). If TOKEN_TTL changes, the cookie silently drifts: the
browser keeps a cookie for a token whose lifetime no longer matches.

Import TOKEN_TTL and use it for the cookie max_age so the session lifetime
has a single source of truth. No behaviour change at the current value.

Fixes #4471
 2026-06-18 21:15:48 +02:00
nubs 0bfc7750a2 fix(llm): route gpt-oss harmony commentary channel without leaking markers/tool-args (#4523) 
The harmony stream router only recognized the analysis and final channels, so
gpt-oss's standard `commentary` channel (tool-call preambles / function-arg
bodies) was unhandled: the literal `<|channel|>commentary` marker, the
`to=functions.*` recipient, and the commentary body all leaked into the
visible answer. Add commentary to the marker regex + the suffix-hold table, and
route its body to thinking (only `final` is user-facing). Adds a regression
test (split-chunk + recipient + body), verified to fail without the fix.
 2026-06-18 21:12:25 +02:00
Rolly Calma 790ef81b06 fix: use aware UTC in health timestamp (#4503) 2026-06-18 20:58:25 +02:00
Victor 804691501f test: stop test_skill_index_prompt_injection leaking a stub prefs_routes (#4387) 
_patch_prefs installs a fake routes.prefs_routes with a bare
sys.modules[...] = assignment that is never undone. The stub is an empty
ModuleType without _save_for_user, so a later test whose code path runs
`from routes.prefs_routes import _save_for_user` (e.g. test_backup_import_skills)
fails with ImportError under an unfavorable test order.

Install the stub with monkeypatch.setitem instead (the helper already takes
monkeypatch and uses it for DATA_DIR) so it is reverted at teardown.

Repro: pytest tests/test_skill_index_prompt_injection.py tests/test_backup_import_skills.py
(1 failed before, 5 passed after).
 2026-06-18 20:54:15 +02:00
dependabot[bot] 8e6a2e89f8 chore(deps): bump actions/checkout in the actions group (#4559) 
Bumps the actions group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 6.0.3 to 7.0.0
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/df4cb1c069e1874edd31b4311f1884172cec0e10...9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-18 20:49:58 +02:00
dependabot[bot] dbcc7874bf chore(deps): bump the npm group with 2 updates (#4558) 
Bumps the npm group with 2 updates: [@anthropic-ai/sdk](https://github.com/anthropics/anthropic-sdk-typescript) and [@antithesishq/bombadil](https://github.com/antithesishq/bombadil).


Updates `@anthropic-ai/sdk` from 0.104.1 to 0.105.0
- [Release notes](https://github.com/anthropics/anthropic-sdk-typescript/releases)
- [Changelog](https://github.com/anthropics/anthropic-sdk-typescript/blob/main/CHANGELOG.md)
- [Commits](https://github.com/anthropics/anthropic-sdk-typescript/compare/sdk-v0.104.1...sdk-v0.105.0)

Updates `@antithesishq/bombadil` from 0.5.0 to 0.6.1
- [Release notes](https://github.com/antithesishq/bombadil/releases)
- [Changelog](https://github.com/antithesishq/bombadil/blob/main/CHANGELOG.md)
- [Commits](https://github.com/antithesishq/bombadil/compare/v0.5.0...v0.6.1)

---
updated-dependencies:
- dependency-name: "@anthropic-ai/sdk"
  dependency-version: 0.105.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm
- dependency-name: "@antithesishq/bombadil"
  dependency-version: 0.6.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-18 20:42:49 +02:00
RaresKeY 16e660ad09 fix(hwfit): normalize CPU arch for fallback estimates (#4441) 2026-06-18 20:26:22 +02:00
Mazen Tamer Salah b51d83b16d fix(agent): index api_call so RAG tool selection can retrieve it (#3923) 
* fix(agent): index api_call so RAG tool selection can retrieve it

api_call exists in FUNCTION_TOOL_SCHEMAS and the agent's system prompt
advertises configured API integrations, but the tool had no entry in
BUILTIN_TOOL_DESCRIPTIONS. RAG tool selection embeds those descriptions and
retrieves the top-K per message, so a tool without one can never be selected:
the agent claims it can call Home Assistant/Miniflux/Gitea/etc. and then
never receives the api_call schema (unless the Personal Assistant
ASSISTANT_ALWAYS_AVAILABLE path applies).

Add a retrieval-rich description for api_call, plus an ast-based parity test
asserting every FUNCTION_TOOL_SCHEMAS tool has an index description so the
next added tool cannot silently drift the same way.

Fixes #3794

* fix(agent): route API-integration intent to api_call at selection time

Addresses review (RaresKeY) on #3923: indexing api_call in the ToolIndex
description was necessary but not sufficient — the #3794 repro ('Use the
api_call tool to call Home Assistant GET /api/states') matched no domain in
_classify_agent_request, classified as low-signal, so the agent loop skipped
retrieval entirely and the schema filter sent only ALWAYS_AVAILABLE
(manage_memory/ask_user/update_plan). api_call never reached the model.

- _classify_agent_request: detect API-integration intent (api_call,
  integration(s), Home Assistant/Miniflux/Gitea/Linkding/Jellyfin) -> new
  'integrations' domain, so the turn is no longer low-signal.
- _DOMAIN_TOOL_MAP['integrations'] = {api_call}: deterministically seeds
  api_call into relevant tools after retrieval, independent of embeddings.
- _DOMAIN_RULES['integrations']: rule pack (required — _domain_rules_for_tools
  indexes _DOMAIN_RULES[domain] directly).
- tool_index _KEYWORD_HINTS: parity hint for the retrieval / keyword-fallback
  paths.
- Regression drives the real classifier -> domain-map -> FUNCTION_TOOL_SCHEMAS
  filter chain and asserts api_call is advertised for the #3794 prompt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-18 08:43:25 +00:00
Shreyas S Joshi f70db19cc6 fix(document): allow render-pdf to be framed and 503 cleanly on missing PyMuPDF (#2103) 
* fix(document): allow render-pdf to be framed and 503 cleanly on missing PyMuPDF

Fixes #2101.

Two related bugs in the PDF-form library preview flow:

1. SecurityHeadersMiddleware was sending X-Frame-Options: DENY and
   frame-ancestors 'none' on /api/document/{doc_id}/render-pdf, but
   static/js/documentLibrary.js embeds the response in an <iframe> for
   the library card preview. The browser blocked the load with
   ERR_BLOCKED_BY_RESPONSE, leaving the user with a blank panel.

   Extend the existing is_tool_render exemption to also cover
   /api/document/.../render-pdf. Per-document owner checks still run in
   the route handler, so the exemption is scoped the same way as the
   tool-render exemption it mirrors. /api/document/.../export-pdf is
   left untouched — it's a download (Content-Disposition: attachment),
   not an iframe embed.

2. routes/document_routes.py:render_pdf called fill_fields, which
   raises RuntimeError via _require_fitz() when the optional PyMuPDF
   dependency isn't installed. That RuntimeError bubbled out as a
   generic 500 with a cryptic 'PDF render failed' detail.

   Reuse the existing _load_pdf_viewer_fitz() helper to fail fast with
   a 503 and a user-actionable install hint (mentions
   requirements-optional.txt and AGPL-3.0), matching the convention
   used by the other PDF endpoints.

Tests cover both fixes:
- middleware headers on /api/document/.../render-pdf (iframeable, but
  X-Content-Type-Options and Referrer-Policy are still set)
- middleware headers on /api/document/.../export-pdf (must stay strict)
- middleware path matching precision (similar-but-different paths stay
  strict)
- middleware headers on /api/tools/.../render (no regression)
- middleware headers on /api/chat (no regression)
- render-pdf returns 503 with install hint when PyMuPDF is missing
- 503 is raised before any file I/O (fail-fast ordering)

* chore: address maintainer feedback on PDF previews same-origin framing and comment trimming

* chore: make render-pdf regression tests order-independent
 2026-06-18 06:25:26 +00:00
Kenny Van de Maele 56ba144875 refactor(tools): move model-interaction tools to the agent_tools registry (#4445) 
Moves chat_with_model, ask_teacher and list_models out of ai_interaction.py
into src/agent_tools/model_interaction_tools.py (the do_ prefix dropped) and
registers them in TOOL_HANDLERS, so dispatch flows through the registry instead
of the dispatch_ai_tool elif in tool_execution.py.

The implementations are relocated, not wrapped. ai_interaction.py keeps only
the shared helpers they reuse (_resolve_model, AI_CHAT_TIMEOUT), still used by
the not-yet-migrated session/pipeline tools. dispatch_ai_tool loses its three
now-unused branches.

Also removes the dead do_second_opinion: it was already off the live tool
surface (no tag/schema/parsing/dispatch; tool_index.py notes it was removed),
so the function and its stale frontend catalog entries (admin.js, assistant.js)
are deleted.

Tests: owner-scope test points at the new list_models location and drops the
moved tools from the dispatch_ai_tool parametrize; a new
test_model_interaction_registry covers registration, owner threading, and
registry dispatch.
 2026-06-18 05:56:37 +00:00
pewdiepie-archdaemon d70c00e8d2 Merge branch 'main' of https://github.com/pewdiepie-archdaemon/odysseus 2026-06-17 12:28:24 +00:00
Matyas Gosztonyi 97a7f59fe7 fix(ui): share one z-order stack across Notes and modals (#3798) 
* fix(notes): bring pane above active windows

* fix(notes): align tool window z-order handoff

---------

Co-authored-by: Matyas Fenyves <16389204+uhhgoat@users.noreply.github.com>
 2026-06-17 12:15:48 +02:00
Afonso Coutinho 24ace44888 fix: canvasCoords crashes on empty touch list (mobile race) (#2045) 2026-06-17 10:25:39 +02:00
Muhammad-Ikhwan-Fathulloh b3ed60e95a fix: optimize upload manifest performance and fix owner rename bug 2026-06-16 23:11:30 +07:00
Muhammad Ikhwan Fathulloh 37da04e8b5 Merge branch 'pewdiepie-archdaemon:dev' into dev 2026-06-16 22:31:13 +07:00
Kenny Van de Maele 93569b141b fix(security): allowlist manage_mcp 'add' to close the agent-path RCE (#4433) 
* fix(security): allowlist manage_mcp 'add' to close the agent-path RCE

do_manage_mcp('add') passed model- and prompt-injection-controlled command,
args, and env straight to a stdio subprocess spawn with no validation, and it
persisted an enabled server row before connecting (so a payload also survived
to re-execute on restart). A string smuggled into a skill description, memory
entry, fetched page, or email body could register a server running arbitrary
code as the app UID, e.g. command='sh' args=['-c','...'].

Add _validate_mcp_command, applied on the agent path before any DB write or
spawn:
- Hard-deny interpreters, runtimes, package runners, shells, and exec-wrappers
  (even if an operator lists one in ODYSSEUS_MCP_ALLOWED_COMMANDS).
- Require a bare basename (no path components, no shell metacharacters) that is
  present in the operator allowlist (empty by default).
- Reject code-exec argv flags by prefix so glued forms are caught too
  (-c/-e/-m/--eval/--exec/--print/--module/--command/--require), remote-URL
  args, and env keys that inject code into the child (LD_PRELOAD, NODE_OPTIONS,
  PYTHONPATH, DYLD_*, PATH, ...).

A rejected registration returns an error, writes no row, and makes no
connection. The trusted admin route is unchanged. Mirrors the policy intent of
_validate_serve_cmd but inverted for the model-reachable surface.

Supersedes #438; incorporates the bypass forms found in its review (interpreter
script paths, -m pip, glued -c/-e, --eval=, eval subcommands, package runners,
remote URLs) and adds integration coverage on the real do_manage_mcp path.

Closes #2891

* fix(security): deny versioned/alias runtimes in manage_mcp allowlist

Addresses RaresKeY's review on #4433. The hard-deny matched command names
exactly, so versioned or alias runtime forms (python3.11, node18, pip3,
ruby3.2, java, javac, bunx, tsx, ts-node, pypy3, ...) slipped past and, if an
operator allowlisted one, re-opened the prompt-injection-controlled MCP
registration path.

- Canonicalize a trailing version suffix before the deny check so versioned
  forms collapse to the family (python3.11 -> python, node18 -> node, pip3 ->
  pip); both the raw basename and the canonical form are denied.
- Broaden the denied-family set (java/javac/jshell/jbang/kotlin/dotnet/mono/
  swift/osascript/tsx/ts-node/bunx/pypy/jruby/raku/luajit/wish/expect/iex).

Deny runs before the operator allowlist, so an alias cannot be allowlisted back
in. Canonicalization only feeds the deny check, so a legit name that ends in a
digit still reaches the normal allowlist check rather than being mis-denied.
Adds validator + integration regressions for versioned/alias runtimes asserting
no DB row and no connection, including the allowlisted-anyway case.
 2026-06-16 14:34:53 +00:00
Catalin Iliescu 9a00401507 fix(hwfit): use CPU fallback for cpu_only speed estimates (#4397) 
* fix(hwfit): use CPU fallback for cpu_only speed estimates

* fix(hwfit): preserve ARM fallback for cpu_only estimates

---------

Co-authored-by: Cata <cata@bigjohn.local>
 2026-06-16 14:18:31 +00:00
Aura Rays Lab 76562ae31d Change host from 0.0.0.0 to 127.0.0.1 in CONTRIBUTING.md (#4422) 
Updated the host address in the run command for clarity.
 2026-06-16 13:40:47 +00:00
Christian Eriksson 497f455da6 fix(cookbook): open() no longer crashes when a task has a diagnosis (#4417) 
_showDiagnosis referenced an undefined `body` (left over from the refactor
that moved the diagnosis text into the toolbar), throwing a ReferenceError
whenever a failed task rendered fix buttons. Because open() wraps its render
in try/finally with no catch, the throw escaped before the modal was
un-hidden, so the whole Cookbook silently failed to open.

- cookbook-diagnosis.js: append the fixes row to `diag` (the in-scope
  container) instead of the removed `body` element.
- cookbook.js: guard the render passes in open() so one broken task card
  can't leave the entire panel stuck hidden.

Fixes #4406
 2026-06-16 13:35:51 +00:00
Ashvin dd20c2bc75 fix(tasks): offer shell/file tools to scheduled task agents by default (#4398) 
The scheduled-task runner built the agent's tool set from RAG retrieval plus
ASSISTANT_ALWAYS_AVAILABLE. Neither includes bash/python (nor the file tools),
and no keyword hint force-includes them, so a task only saw the shell when the
tool-embedding index happened to surface it. On hosts where that index is empty
or degraded (e.g. a fresh Docker deploy), retrieval returns nothing and the task
agent never receives bash/python — telling the user the shell is disabled even
for an admin owner.

Offer the shell/file group to task agents by default, mirroring the chat agent
where these are on unless a privilege or global setting turns them off. The
existing blocked_tools_for_owner() gate in stream_agent_loop still strips the
whole group for non-admin multi-user owners and only admits it for admins and
single-user (AUTH_ENABLED=false) deployments, so this changes what is offered,
not who is allowed. A crew that defines an explicit enabled_tools allowlist
still has its restriction honored.

Also merge the operator's global disabled_tools setting into the scheduler's
disabled set before composing relevant_tools and before entering the agent
loop, matching what chat already does. Without it, the global tool-disable
contract did not reach unattended scheduled tasks: an admin or AUTH_ENABLED=false
task could still see and call shell/file tools the operator had turned off
globally, since the prompt/schema/execution gates only enforce the disabled
tools passed in.
 2026-06-16 13:27:30 +00:00
Afonso Coutinho a36b423a4e Fix odysseus-calendar list dropping in-progress / multi-day events (#2065) 
cmd_list filtered on the event START falling inside the window
(dtstart >= start AND dtstart < end). The canonical web route
(routes/calendar_routes.py) and the recurrence contract test use
OVERLAP semantics for non-recurring events: dtstart < end AND
dtend > start. So an event that began before the window but is still
ongoing inside it — e.g. a 09:00-17:00 conference listed at 14:00, or
any multi-day event spanning the window — was silently dropped by the
CLI even though the web UI shows it. Use overlap, matching the route.
dtend is NOT NULL in the schema, so no null-end regression.
 2026-06-16 14:04:56 +02:00
Rudy Wolf 4e477741e7 harden(agent-loop): wrap non-native tool results as untrusted data (#1629) 
The non-native (prompted) tool-call path fed tool output back to the model as a plain "[Tool execution results]" user message, bypassing the untrusted_context_message wrapper that THREAT_MODEL.md requires for tool output. That path is what models without native tool-calling (many smaller local models) use, so prompt-injection inside a tool result (fetched page, file read, MCP/email output) could be read as instructions there.

Wrap it via untrusted_context_message("tool execution results", ...), the same hardening already applied to skills (#788) and escalation traces (#275). Also update _recent_context_for_retrieval, which used the old "[Tool execution results]" prefix as a sentinel to keep tool envelopes out of the retrieval query, to recognise the wrapped envelope via metadata.trusted.

The native path keeps returning tool-role messages (a user-role wrapper would break the native tool-call contract); it is covered by UNTRUSTED_CONTEXT_POLICY. Adds tests/test_tool_output_prompt_injection.py.

Fixes #1627.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 13:35:07 +02:00
Kenny Van de Maele a2261c38c1 refactor(auth): centralize the internal-tool pseudo-username into a constant (#4333) 
The in-process tool loopback stamps current_user = "internal-tool" and
require_admin grants admin to that sentinel; it is also a reserved username.
That security-sensitive string was hand-typed in ~7 places (stamp, admin gate,
RESERVED_USERNAMES, and standalone admin-equivalent checks in note/research/
shell/task routes), where a typo silently breaks an auth gate.

Add INTERNAL_TOOL_USER in core/middleware.py next to INTERNAL_TOOL_TOKEN/
INTERNAL_TOOL_HEADER and use it at every such site. A typo is now an
ImportError, not a silent mismatch. auth.py importing middleware is acyclic
(middleware imports no app modules). Behaviour is unchanged.

The multi-sentinel sets bundling internal-tool with api/demo/system
(assistant_routes, task_scheduler, research_routes) are a separate reserved-set
dedup, left for a follow-up.

Closes #4332
 2026-06-16 13:13:00 +02:00
Alexandre Teixeira bf56010aad test: split provider classification tests (#4392) 2026-06-16 09:54:07 +00:00
Karl Jussila ee72d71872 fix(auth): centralize password and username validation constants (#4120) 
Added PASSWORD_MIN_LENGTH and RESERVED_USERNAMES to src/constants.py as the
single source of truth. Previously PASSWORD_MIN_LENGTH was hardcoded as 8 in
four route handlers and all three JS validation paths; RESERVED_USERNAMES was
an inline frozenset duplicated in core/auth.py, routes/assistant_routes.py,
routes/research_routes.py, and src/task_scheduler.py.

Added GET /api/auth/policy (unauthenticated) so the frontend reads the real
values from the server instead of hardcoding them in JS.

Added missing empty-username guard to /setup and admin POST /users. Both
returned a misleading 500/409 on whitespace-only input. /signup already had the
check; this makes all three consistent.
 2026-06-16 09:52:15 +02:00
RaresKeY 2b519bf355 fix(routes): normalize session owner fallback helpers (#4313) 
* fix(memory): normalize import session fallback

* fix(chat): use token owner for compaction scope

* fix(background): honor session endpoint fallback
 2026-06-16 06:07:42 +01:00
Kfir Sadeh d795d9a923 feat(launcher): add portable windows launcher (#976) 
* feat(windows): add standalone portable executable, splash screen, and system tray

* test: fix test_get_wsl_windows_user_profile_falls_back_to_users_dir on Windows

* Refactor launcher: isolate desktop logic into launcher.py, clean app.py/requirements, update build scripts, and add tests

* chore: clean launcher test whitespace

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>
 2026-06-16 04:58:16 +01:00
Tal.Yuan 648db61b45 docs(architecture): add Phase 0 runtime inventory document (#4148) 
* docs(architecture): add Phase 0 runtime inventory document

Per #4082 requirements, this no-code planning document maps:
- Largest runtime modules (Python + frontend)
- Import dependency graph and cross-layer violations
- Route ownership grouped by feature domain
- Tool registry boundaries and split candidates
- Risk-ranked candidate slices with recommended first 3 PRs
- Safety guardrails and validation commands for follow-up work

Closes #4082

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(architecture): correct inventory metrics per review feedback

Address @alteixeira20 review on #4148 (CHANGES_REQUESTED):

- src/ flat .py: ~60 -> 91; routes/: 52 -> 54
- core/database.py importers: 49 -> 94; src/agent_loop.py: -> 21
- src/ -> routes/ import lines: ~20 -> 38
- src/ subdirs: 3 -> 2 (agent_tools/, search/); drop non-existent agent/
- move main.py and src/agent/ out of current-structure into new
  section 10 'Future Direction (NOT current state)'
- route grouping: frame as one domain per PR, not a broad
  reorganization (helper imports / registration / test path risk)

* docs(architecture): round-2 fixes — move to specs/, correct counts, frame as candidate

Per @alteixeira20 + @RaresKeY review on #4148:

- Move docs/architecture-runtime-inventory.md -> specs/ (docs/ is
  GitHub Pages public content, per @RaresKeY)
- src/ -> routes/ import lines: 38 -> 30 (direct grep of import lines
  referencing routes/, matching reviewer's count)
- self-caught count drift: tests 552 -> 544; routes->src 349 -> 351;
  src->core 49 -> 99
- frame section 6 (rankings/package shapes/split order/route grouping)
  and section 10 (future direction) as candidate proposals pending
  maintainer agreement, not a committed plan (per @RaresKeY)

* docs(architecture): round-3 reviewer fixes — fix tool categorization, counts, appendix

Self-review as reviewer found:
- §5.2 tool categories were wrong: listed filesystem/shell/email-sending
  tools that are NOT in tool_implementations.py (they live in src/agent_tools/).
  Rewrote to the actual 33 do_* functions grouped by domain
  (system/cookbook/calendar/notes/search/research/contacts/vault/image)
- §2.1 builtin_actions.py: 0 -> 2 classes, ~26 -> ~24 functions
- §5.1: '33+' -> '33' (exact count)
- Appendix A: 'Complete File Listing' -> 'File Listing'; src noted as
  '61 of 91 shown' (was claiming complete but listed 61)
- Last updated date refreshed

* docs(architecture): round-4 — verify remaining counts, soften §6.3 framing

- task_scheduler ~6 -> 5 funcs; tool_index ~580 -> 542 lines (verified vs dev)
- §6.3 'Recommended First 3 Slices' -> 'Candidate' (ownership unsettled, per review)
- verified §4 route-domain line counts, §2.2 frontend counts, mcp_servers=4
- full test suite: 3267 passed, 1 skipped, 0 failed

* docs(architecture): refresh Phase 0 inventory metrics + document counting method

Refresh every count against current dev (b58af42) per review on #4148:
- src/ flat .py: 91 -> 95; tests/test_*.py: 544 -> 583
- core.database importers: 94 -> 102; src.agent_loop importers: 21 -> 22
- src/ -> routes/ lines: 30 -> 31; routes/ -> src/: 351 -> 374; src/ -> core/: 99 -> 106
- Last updated: dev@9d7a3d6 -> dev@b58af42

Add a "How the metrics are computed" note under section 3.4 with the exact
grep/find command for each count, so the numbers are reproducible and future
dev drift is a one-command recheck instead of another review round (per the
request to note the counting method).

Documentation-only; no code changes.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(architecture): refresh remaining counts + add snapshot basis note

Reviewer self-audit of the previous refresh caught more stale counts after
the rebase onto dev@b58af42:
- tool_implementations importers: 18 -> 17 (§3.2, §6.2, Appendix B)
- core/database classes: 27 -> 28 (§2.1, §6.2)
- mcp_servers .py files: 4 -> 5 (§1.1)
- routes/ -> core/ import lines: 124 -> 126 (§3.4)

Line counts in §2.1/§2.2 also drifted over the rebased range but are left
as-is and covered by a new "Snapshot basis" note in the header: line counts
are a snapshot that drifts as dev moves (recompute with wc -l), while the
importer/file/import-line counts are the authoritative ones refreshed here.
This keeps the inventory honest about live metric vs structural snapshot, so
dev drift no longer triggers a review round.

Documentation-only; no code changes.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(architecture): fix missed tool_implementations importer count in §6.3

Follow-up to the previous refresh: §6.3 Slice 1 still read "18 importers"
after the 18->17 update elsewhere. Correct to 17 for consistency. Doc-only.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: yuandonghao <yuandonghao@cohl.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-16 04:57:24 +01:00
RaresKeY 260ce8ba59 fix(email): enforce MCP owner boundaries (#4335) 
* fix(email): enforce MCP owner boundaries

* fix(email): fail closed for unowned MCP fallback
 2026-06-16 04:31:24 +01:00
RaresKeY 2f9ae43a58 test(email): cover sender signature owner cache writes (#4278) 2026-06-16 04:21:11 +01:00
RaresKeY 293bbfabf4 test(hwfit): cover SSH target validation regressions (#4279) 2026-06-16 04:18:21 +01:00
Alexandre Teixeira 0086399656 test: add fire_and_forget to API chat webhook stub (#4383) 2026-06-16 03:15:14 +00:00
RaresKeY 9d2989f386 test(auth): cover reserved username sentinel gate (#4276) 2026-06-16 04:09:58 +01:00
RaresKeY b5edbd3df7 fix(devops): harden docker config defaults (#4349) 2026-06-16 04:03:43 +01:00
RaresKeY 33fe7276be fix(endpoints): normalize URL handling (#4338) 2026-06-16 03:59:18 +01:00
RaresKeY a031a94a2e fix(cookbook): harden remote serve host handling (#4345) 2026-06-16 03:46:32 +01:00
RaresKeY 4d10c16d02 fix(auth): clean up rename and null-owner ownership (#4340) 2026-06-16 03:33:02 +01:00
RaresKeY 745c10e0d7 fix(gallery): confine gallery image path resolution (#4352) 2026-06-16 03:28:09 +01:00
Alexandre Teixeira 6b7a4c1e70 test: add oversized test split plan (#3987) 
* test: add oversized test split plan

* test: refresh oversized split plan
 2026-06-16 02:28:03 +00:00
RaresKeY 422f23fb12 fix(mcp): scope memory server by owner (#4315) 2026-06-16 03:18:17 +01:00
TheDragonTail 0f966d6b9f fix(embeddings): fall back to default cache dir when FASTEMBED_CACHE_PATH is empty (#3434) 
docker-compose.yml injects FASTEMBED_CACHE_PATH=${FASTEMBED_CACHE_PATH:-},
which sets the variable to an empty string when the host has not defined it.
FASTEMBED_CACHE_DIR used os.getenv("FASTEMBED_CACHE_PATH", default), and
os.getenv only returns the default when the variable is ABSENT -- so the empty
value won and FASTEMBED_CACHE_DIR became "". os.makedirs("") then raised
[Errno 2] No such file or directory: '', FastEmbed failed to initialise, and
every vector feature (RAG, semantic memory, tool index) silently degraded on
the default Docker stack.

Treat an empty value like an absent one via `os.getenv(...) or default`.
Add a regression test covering the empty, unset, and explicit cases.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 03:11:48 +01:00
Afonso Coutinho 7b09491557 fix: check-in calendar digest leaks every user's events (missing owner scope) (#1925) 
* fix: check-in calendar digest leaks every user's events (no owner scope)

* Seed dtend on calendar events in digest test so the NOT NULL column is satisfied
 2026-06-16 02:42:41 +01:00
Kenny Van de Maele fafaf089c5 refactor(search): centralize the web-scraping User-Agent into one constant (#4325) 
The outbound UA for web_fetch / web_search was inlined in four places with
two different values and nothing keeping them current: content.py pinned a
mid-2021 Chrome 91 build, and providers.py sent a bare Mozilla/5.0 in three
spots. Some sites serve a degraded or blocked page to a UA that old.

Add WEB_FETCH_USER_AGENT to src/constants.py (env-overridable, matching the
existing Copilot/Kimi UA-constant pattern) and import it in content.py and
providers.py. Default to a current, common desktop UA so pages return their
normal HTML: the market-leading desktop OS (Windows; NT 10.0 covers Windows
10 and 11) and browser (Chrome) on a current stable build. The version is now
bumped in one place.

Service-specific self-identifying agents (Copilot, Kimi, webhooks, cookbook)
are intentionally left separate. Adds a regression pinning the constant shape,
the env override, and a guard against a new inline Mozilla literal in the
search sources.

Closes #4324
 2026-06-16 01:33:47 +00:00
RaresKeY b58af4267b fix(companion): require chat scope for model inventory (#4319) 2026-06-16 01:15:05 +02:00
AkioKoneko 8ff76f083c fix(cookbook): avoid launching Ollama during Windows cache scan (#4368) 2026-06-16 01:00:40 +02:00
Wei Hong 2196869c86 fix(webhooks): route public emitters through fire_and_forget (#3964) (#4336) 
The three public webhook emitters in chat_helpers and webhook_routes
schedule deliveries via asyncio.create_task(webhook_manager.fire(...)),
which bypasses WebhookManager._bg_tasks. asyncio only holds a weak
reference to the outer task, so the GC can collect it mid-delivery and
the webhook is silently dropped.

Route all three through webhook_manager.fire_and_forget() so the task
is tracked by _spawn_tracked() and the manager owns the full lifecycle.

Adds an AST-level guard test that scans routes/ for direct
asyncio.create_task wrapping webhook_manager.fire(...) to prevent
regressions.
 2026-06-16 00:41:45 +02:00
holden093 dd2e23c9af fix(agent): report phone numbers from resolve_contact when a matched contact has no email (#4327) 
When a CardDAV contact matched the search query but had no email
address (only phone numbers), the tool silently dropped it and
returned 'No contacts found'.  Fall back to the contact's phone
number(s) so the caller still receives usable information.

Refs: #4178 (the contacts-domain classifier fix that made the model
actually call resolve_contact for contacts queries, surfacing this
pre-existing gap)
 2026-06-16 00:03:33 +02:00
Fahim facc50cb0f fix(api): attribute bearer-token actions to the token owner on owner-scoped routes (#4054) 
* fix(api): attribute bearer-token actions to the token owner on owner-scoped routes

Owner-scoped chat, session, and upload routes called
get_current_user(), which resolves a bearer ody_ API token to the
sandboxed "api" pseudo-user. A paired API-token client (companion, CLI,
IDE extension) therefore saw and created a separate "api"-owned silo
instead of the owner's data.

effective_user() already exists for exactly this: it attributes a token's
actions to request.state.api_token_owner, is identical to
get_current_user() for cookie sessions, and falls back safely when a
token has no owner. session_routes.py was already migrated; this
completes the migration for the remaining owner-scoped routes:

- chat_helpers.py: chat-privilege enforcement, message attribution, prefs/context
- chat_routes.py: orphaned-endpoint owner, session-auth owner, message search
- upload_routes.py: upload owner attribution + access checks

The /api/models swap is intentionally omitted: #4292 already migrated it
to effective_user (plus the chat-scope gate and ownerless-token 403), so
this PR keeps dev's version of routes/model_routes.py unchanged.

chat_routes.py keeps importing get_current_user for the workspace owner
gate; session_routes.py drops the now-unused import.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: target effective_user in auth monkeypatches and owner-scope assertion

The owner-scoped routes now call effective_user() instead of
get_current_user(), so the tests that stubbed get_current_user (or
asserted on it) follow suit:

- test_chat_helpers.py, test_review_regressions.py,
  test_kv_cache_invalidation_2927.py: monkeypatch effective_user
- test_session_endpoint_owner_scope.py: assert the owner-scope guard uses
  effective_user(request)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-15 23:56:22 +02:00
Kenny Van de Maele 074a1e6eff fix(search): add download budgets to web_fetch with truncation notice and hard ceiling (#3955) 
* fix(search): add download budgets to web_fetch with truncation notice and hard ceiling

MAX_OUTPUT_CHARS only trims what the agent sees; fetch_webpage_content
buffered and cached the entire response body first, so a large or hostile
URL could pull arbitrarily many bytes into memory and the content cache.

The fetch is now a capped streaming GET (SSRF redirect guard unchanged):
a soft default budget (WEB_FETCH_SOFT_MAX_BYTES, 2 MB), a per-call
override via full/max_bytes on the web_fetch tool, and a hard ceiling
(WEB_FETCH_HARD_MAX_BYTES, 20 MB) that the override can never exceed.
When Content-Length already declares a body over the ceiling the fetch
is refused before any body bytes are buffered. Truncated results carry
truncated/fetched_bytes/total_bytes, the tool output leads with a
partial-content notice telling the model how to re-fetch with full=true,
and the tool schema documents the flag. A truncated PDF is reported as
a budget error since a cut PDF is unparseable. The effective cap is part
of the content-cache key so a truncated fetch is never served to a
full-budget request.

Existing tests that faked httpx.get or the old _get_public_url signature
are adapted to the streaming interface; behavior pins are unchanged.

Fixes #3812

* fix(search): close compressed-body cap bypass and protect the partial notice

Addresses RaresKeY's review on #3955:

- Force Accept-Encoding: identity for the capped fetch. With gzip/deflate the
  wire bytes (and Content-Length) can be a fraction of the decoded body, so a
  tiny compressed response could pass the hard-cap preflight and then expand
  past the ceiling in a single decoded chunk before the streamed cap could
  slice it. Identity makes Content-Length the true body size and keeps each
  streamed chunk bounded by the network read, so the hard ceiling actually
  bounds memory.
- Lead web_fetch output with the partial-content notice and cap the page
  title. The notice is the user-facing contract for partial fetches, but the
  title is untrusted, uncapped page content; placed ahead of the notice a giant
  title could push it past MAX_OUTPUT_CHARS and drop it. The notice now leads
  and the title is capped as a second guard.

Adds regressions: the fetch advertises identity encoding, and a truncated
result with an oversized title still surfaces the partial notice.

* fix(search): reject compressed responses that ignore the identity request

Requesting Accept-Encoding: identity is not enough on its own: a server can
ignore it and still return Content-Encoding: gzip, and httpx.iter_bytes would
decode that, so a tiny compressed body could balloon into one decoded chunk
far past the hard cap before the streamed loop slices it (and Content-Length,
the compressed wire length, makes the preflight and size metadata unreliable).

Refuse a non-identity Content-Encoding before reading the body. Adds a
regression where the server ignores the identity request and returns gzip;
the fetch is refused before any body is decoded.
 2026-06-15 17:38:09 +00:00
Kenny Van de Maele 2fab378c6a refactor(search): import REQUEST_TIMEOUT from constants in providers.py (#4331) 
providers.py redefined REQUEST_TIMEOUT = 20 locally, shadowing the same
value in src/constants.py and risking drift if the constant is bumped.
Import it from src.constants and drop the local copy; same value, one
source of truth.

Closes #4329
 2026-06-15 17:22:08 +00:00
Michael 5bafc30622 fix(api): normalize non-object JSON bodies to empty dict in token PATCH (#3976) 
* fix(api): normalize non-object JSON bodies to empty dict in token PATCH

Valid non-dict JSON (e.g. [] or null) reaches payload.get(...) and
raises AttributeError. Normalize to {} so the route returns a controlled
response instead of an unhandled 500.

Fixes #3966

* test(api): add regression tests for PATCH with non-object JSON bodies

Covers array body ([]), null body, and normal object body as requested
in alteixeira20's review of #3976.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
 2026-06-15 18:05:15 +01:00
darius-f96 d6d2e17214 fix(hwfit): add GB10 unified-memory bandwidth so speed scores are real (#4270) 
NVIDIA Grace Blackwell GB10 / DGX Spark was missing from GPU_BANDWIDTH, so
_lookup_bandwidth() returned None for it and _estimate_speed() fell through
to the crude FALLBACK_K path (k/active-params). That over-stated tok/s and
let speed scores saturate regardless of the box's real ~273 GB/s LPDDR5X
pool — distorting model ranking on these 128GB unified-memory rigs.

Add "gb10": 273 (GB/s). nvidia-smi reports the device name as "NVIDIA GB10",
which substring-matches the new key, so detected GB10 boxes now estimate
speed from the real bandwidth instead of the fallback.
 2026-06-15 18:55:15 +02:00
Lucas Daniel f4e8990635 chore: add warnings to silent except Exception blocks (#3212) 
* log(app): add warnings to silent except Exception blocks

- Internal tool auth header failure now logs a warning instead of
  silently passing, making auth bypass easier to spot in logs.
- Token last_used_at update failure now logs at DEBUG (fire-and-forget,
  non-critical, but useful when debugging token tracking issues).
- Image ownership verification failure now logs a warning so unexpected
  access-check errors surface instead of silently allowing the request.

* log(chat_routes): add warnings to silent except Exception blocks

- clear_orphaned_session_endpoint: log before rollback so failures
  appear in traces when users see stale/deleted model options.
- _endpoint_has_model (JSON parse): log malformed cached_models instead
  of silently treating endpoint as valid.
- _has_any_visible_model (JSON parse): log malformed cached_models
  instead of silently returning empty list.
- timezone header parse: log failure so time-zone-related tool bugs
  (wrong scheduled times, calendar events) are traceable.
- attachments JSON parse: log failure so silently-dropped attachments
  are visible in server logs.

* log(email_routes): add warnings to silent except Exception blocks

- Email alias resolution failure now logs a warning instead of silently
  returning an empty list, making broken account configs diagnosable.

* log(document_routes): add warnings to silent except Exception blocks

- Export ZIP request body parse failure now logs a warning so empty
  exports caused by malformed requests are diagnosable.
- clear_active_document failure on detach now logs a warning to help
  trace doc re-injection bugs like #1160.

* log(agent_loop): add warnings to silent except Exception blocks

- builtin tool overrides load failure now logs a warning so misconfigured
  settings don't silently fall back to defaults without a trace.
- Timezone context injection failure now logs a warning to help debug
  incorrect scheduled times in agent-created tasks.
- PDF form-backed document detection failure now logs a warning so
  broken form-doc UI is traceable to the root cause.

* log(llm_core): add warnings to silent except Exception blocks

- Malformed URL in _is_ollama_native_url now logs a warning so bad
  endpoint configs are traceable instead of silently returning False.
- Model list fetch failure now logs a warning with the endpoint URL so
  endpoints that silently vanish from the model picker are diagnosable.

* log: pass exception via exc_info instead of string interpolation

* fix(logging): avoid logging raw URLs in llm_core error paths

Drop the raw url/base_chat_url from the Ollama-detection and
model-list-fetch warning logs added by this sweep, since these values
can contain private hostnames, internal IPs, credentials, or other
deployment details.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-15 17:49:27 +01:00
Kfir Sadeh fc3a5e555e feat(paths): abstract runtime path logic for frozen distribution packages (#969) 
* feat(core): abstract runtime path logic for frozen distribution packages

* Address review feedback: revert browser MCP check, persistent data dir default when frozen, and add path tests
 2026-06-15 17:44:10 +01:00
Hriday Ranka 270b8570fc feat(email): add Google OAuth2 for Google Workspace / .edu IMAP & SMTP (#237) 
* feat(email): add Google OAuth2 for Google Workspace / .edu IMAP & SMTP

Google deprecated basic-auth (password) access for Google Workspace
accounts in May 2025. This means any .edu or org Google email account
could no longer connect via IMAP/SMTP with a username + password —
the email feature was silently broken for a large class of users.

This PR adds full OAuth2 (XOAUTH2) support for Google accounts so
Workspace / .edu emails work out of the box.

## What changed

### Backend
- `core/database.py`: add `oauth_provider`, `oauth_access_token`,
  `oauth_refresh_token`, `oauth_token_expiry`, and `display_name`
  columns to `EmailAccount` + idempotent migration
- `routes/email_helpers.py`: XOAUTH2 auth in `_imap_connect()` and
  `_send_smtp_message()`, automatic token refresh, OAuth fields in
  `_get_email_config()`
- `routes/email_routes.py`: OAuth authorize + callback routes,
  `_smtp_ready()` fix, OAuth fields through `_deliver()` closure,
  `display_name` in `From:` header

### Frontend
- `static/js/settings.js`: "Google Workspace / .edu" provider preset,
  "Connect with Google" button, success/error banner, display name field
- `static/js/document.js`: `_accountCanSend()` recognises OAuth accounts
  as SMTP-capable

* security: sign OAuth state, scope callback by owner, fix quotes & logs

Addresses reviewer feedback on the email OAuth2 PR:

- OAuth state is now HMAC-SHA256 signed (keyed with the app secret from
  secret_storage) encoding account_id + owner + a random nonce, and is
  verified with constant-time comparison in the callback before any
  token write. Replaces the bare account_id state, closing the CSRF /
  state-guessing gap.
- Callback extracts the owner from the verified state and re-checks it
  against EmailAccount.owner before writing tokens, matching the
  ownership guards used elsewhere in the email routes. Single-user mode
  (owner == "") still accepts any account, consistent with
  _assert_owns_account.
- Replaced curly/smart quotes in the Name/Email/Display Name input rows
  with plain ASCII so getElementById lookups and event wiring work.
- Stripped account name, SMTP host/user, owner, and raw provider error
  text from send-config and OAuth logs; failures now surface as generic
  error codes in the redirect instead of raw exception strings.

* test(email): add OAuth2 state, _smtp_ready, and XOAUTH2 tests

Move the OAuth state sign/verify helpers out of the setup_email_routes
closure into module-level make_oauth_state/verify_oauth_state in
email_helpers.py so they can be unit-tested, then add tests/test_email_oauth.py:

- signed state round-trips account_id + owner, nonce is unique per call
- tampered account_id, forged signature, and garbage states are rejected
- _smtp_ready treats an OAuth account (no password) as send-capable, and
  still rejects host+user-only accounts with neither password nor OAuth
- _xoauth2_string / _xoauth2_bytes produce the correct SASL XOAUTH2 framing

14 new tests; existing test_security_regressions.py still passes (28).

* refactor(email): single XOAUTH2 frame helper, use RuntimeError

Polish from self-review before merge:

- Collapse the XOAUTH2 framing to one source of truth: _xoauth2_raw()
  returns the unencoded SASL string used by both the SMTP and IMAP auth
  callbacks (each library base64-encodes it), and _xoauth2_bytes() is
  just its .encode(). Removes the unused base64 _xoauth2_string helper
  and the duplicated inline frame in _send_smtp_message.
- Raise RuntimeError (not bare Exception) for the "OAuth token
  unavailable" path, matching the convention used across src/.
- Update tests accordingly.

All 14 OAuth tests + 28 security regressions pass; SMTP/IMAP XOAUTH2
verified live against a real Workspace account.

* tests(email-oauth): cover the security-sensitive OAuth paths before merge

The previous tests only exercised pure helpers (state signing, _smtp_ready,
XOAUTH2 framing). This adds coverage for the actual token-custody and
ownership behaviour, pinning the real route handlers rather than
re-implementations of their logic.

Real OAuth callback route (pulled live from setup_email_routes()):
- missing code -> generic missing_code redirect, no account id / owner in URL
- provider error -> generic google_error redirect, raw error not echoed
- tampered/invalid state -> invalid_state redirect, auth code never leaked
- signed state with owner mismatch -> token write refused (ownership_error),
  DB row left untouched
- signed state with matching owner -> tokens written encrypted, and only to
  the intended account (a second account stays untouched)

Real accounts-list route:
- exposes oauth_provider status but never the access/refresh token values,
  encrypted or otherwise

Token storage / refresh helpers (isolated in-memory SQLite, mocked HTTP):
- refreshed access token stored encrypted; expiry is a timestamp, not a token
- fresh token uses cache (no refresh call); expired token triggers refresh
- refresh HTTP failure returns None silently, no exception or secret surfaced
- missing client credentials short-circuits to None

Password-account regression:
- password IMAP accounts call conn.login(); OAuth accounts call XOAUTH2
  authenticate() and never login()

28 tests pass (14 prior + 14 new).

* fix(email-oauth): drop raw exception text from token-refresh log

Google token refresh failures now log the account id only, matching
the conservative logging used elsewhere on the OAuth path — no raw
provider/exception details surfacing in logs.

* fix(email-oauth): bring OAuth UI parity to the Integrations email form

The Google Workspace / .edu provider preset, Display Name field, and
Connect-with-Google flow were only wired into the Email-tab account
form. The Integrations-tab form (a separate code path for the same
account type) was missing all three, so the OAuth option was invisible
from that entry point. Mirrors the same PROVIDERS entry, OAuth section,
and connect handler so both forms behave identically.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Alexandre Teixeira <111787685+alteixeira20@users.noreply.github.com>
 2026-06-15 17:02:58 +01:00
Léo 0750486654 fix(notes): fail closed when an unauthenticated request reaches owner-scoped routes (#4062) 
* fix(notes): fail closed when an unauthenticated request reaches owner-scoped routes

The notes CRUD routes resolved the acting user with bare get_current_user().
A request that reached them with no identity (auth-middleware regression,
SSRF from a sibling service) came through as user=None — which every query
treats as the single-user mode: list all accounts' notes, read/update/
delete/pin/archive any row, reorder globally.

Resolve the owner through require_user() instead, which already encodes the
right policy: 401 when auth is configured, while the documented anonymous
modes (AUTH_ENABLED=false, LOCALHOST_BYPASS on loopback, unconfigured
first-run) still resolve to the single-user path. fire-reminder in the same
file already gated this way; the CRUD routes now match, and the inline
require_user import there is folded into the module import.

Extracted from #2940 (stabilization slice).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(notes): drive fail-closed test via ASGITransport, not sync TestClient

The focused fail-closed test hung at `TestClient(app).get(...)` on some
environments. Starlette's sync TestClient runs the app in a background
event-loop thread (anyio blocking portal) and then dispatches each sync
endpoint onto a second worker thread; that handshake deadlocks on certain
anyio/httpx/platform combos. The identity injection also used
BaseHTTPMiddleware (@app.middleware("http")), the other known TestClient
deadlock source.

Switch to the repo's existing httpx.ASGITransport + AsyncClient idiom so the
whole request runs on the test's own event loop (no portal thread, no
BaseHTTPMiddleware). Identity now comes from a pure-ASGI shim that writes the
same request.state fields the real auth middleware sets, and a non-loopback
client peer keeps require_user's loopback fall-throughs out of the picture.
Same assertions and coverage; production code unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-15 17:43:28 +02:00
RaresKeY d38e2cbc07 fix(ci): avoid duplicate CodeQL setup (#4297) 2026-06-15 16:39:13 +01:00
Ashvin 7fd937fa57 fix(calendar): parse "mins"/"hrs" reminder offsets in manage_calendar (#4266) 
_reminder_minutes matched the offset with (?:m|min|minute|minutes)\b and
(?:h|hr|hour|hours)\b. The trailing \b makes the common plural
abbreviations "mins"/"hrs" fail to match (after "min" the "s" is a word
char, so no boundary), so reminder_minutes "5 mins" or "2 hrs" returned
None and the event was created with no reminder, silently.

Widen the two unit regexes and the matching reminder_only description
regex to a strict superset that also accepts mins/hrs. The sibling
duration parser already accepts these forms (it has no \b), so this only
brings the reminder parser in line.
 2026-06-15 17:37:28 +02:00
Catalin Iliescu c41caac438 fix(cookbook): only persist successfully stopped scheduled serves (#4267) 
Co-authored-by: Cata <cata@bigjohn.local>
 2026-06-15 17:30:18 +02:00
Kenny Van de Maele 1747c13133 test: align README presentation guards with the #4306 refresh (#4311) 
* test: align README presentation guards with the #4306 refresh

The 'Refresh README presentation' change (#4306) swapped the ASCII banner
for a centered wordmark image and moved the native quickstart into
docs/setup.md, which left four base tests failing on dev and froze the
merge gate:

- test_security_regressions::test_readme_native_quickstart_uses_loopback
  now also accepts the loopback guidance from docs/setup.md, where the
  quickstart moved (no behaviour change; the guidance is intact there).
- test_readme_ascii_fenced guards the new wordmark title instead of the
  removed ASCII banner, and keeps a defensive check that any reintroduced
  box-drawing banner stays inside a code fence (the original #1390 mode).
- The five unreferenced demo gifs under docs/ (chat, compare, document,
  notes, research) are removed so test_docs_no_orphan_images passes; they
  were de-referenced by the refresh. Recoverable from history if a docs
  page wants to embed them again.

* chore: refresh PR checks

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>
 2026-06-15 16:25:38 +01:00
RaresKeY ffd0aaf69b fix(cookbook): validate adopt host (#4282) 2026-06-15 16:44:24 +02:00
RaresKeY 81e7074d93 fix(gallery): confine replacement image path (#4285) 2026-06-15 16:42:41 +02:00
RaresKeY f66a23d19d fix(ai): validate generated image result URLs (#4289) 2026-06-15 16:40:49 +02:00
RaresKeY f602819523 fix(models): scope API-token model listing (#4292) 2026-06-15 16:38:41 +02:00
RaresKeY 85a773ea02 fix(personal): resolve upload delete path (#4291) 2026-06-15 16:38:37 +02:00
PewDiePie fb0a64fe4f Merge pull request #4306 from pewdiepie-archdaemon/readme-refresh-default-branch 
docs: refresh README presentation
 2026-06-15 23:28:20 +09:00
pewdiepie-archdaemon bcf46dafb9 Refresh README presentation 2026-06-15 23:26:10 +09:00
pewdiepie-archdaemon d9ebdd6fbb Refresh README presentation 2026-06-15 23:24:41 +09:00
pewdiepie-archdaemon b118c33e37 test(provider): align lookalike-host URL expectations with /models behavior 
build_models_url returns /models (no /v1 prefix) for non-local generic
OpenAI-compatible hosts (intentional, see endpoint_resolver.py:206). The
tests added in #4272 expected /v1/models, which is the local/deepseek
behavior. Match production semantics.
 2026-06-15 23:21:49 +09:00
pewdiepie-archdaemon da74cc23e4 Merge remote-tracking branch 'origin/dev' 2026-06-15 23:13:18 +09:00
Ashvin d792b61722 test(gallery): point delete-ordering tests at the tmp image dir (#4300) 
The two delete-ordering tests did monkeypatch.chdir(tmp_path) and wrote the
image under tmp_path/data/generated_images, but DATA_DIR (and therefore
gallery_routes.GALLERY_IMAGE_DIR) is always an absolute path, so the delete
resolver pointed at the repo's real data dir and ignored the chdir.

test_file_removed_on_successful_delete therefore failed on dev (the file at
the tmp path was never the one being removed), and test_file_kept_when_commit_fails
passed only by accident. Set GALLERY_IMAGE_DIR to the seeded tmp dir via
monkeypatch so both tests exercise the real path and pass deterministically.
 2026-06-15 14:07:49 +00:00
pewdiepie-archdaemon f3e4e071c7 Merge remote-tracking branch 'origin/dev' 2026-06-15 14:02:53 +00:00
pewdiepie-archdaemon 1faadf7e10 Merge remote-tracking branch 'origin/dev' 2026-06-15 23:02:46 +09:00
Kenny Van de Maele e87b44126c test(hwfit): fix non-Apple guard to assert the Apple matcher (unblocks pytest gate) (#4303) 
* test(hwfit): assert the Apple matcher, not the general lookup, in the non-Apple guard

f7aa2de (#2564) added test_non_apple_gpu_with_cores_does_not_match, which
asserts _lookup_bandwidth(RTX 4090) is None. But '4090': 1008 has been in
the general GPU_BANDWIDTH table since v1.0, so _lookup_bandwidth correctly
returns the card's real bandwidth and the test fails (expected None, got
1008) - reddening the required pytest gate on dev and, by inheritance,
every open PR.

The guard's actual intent is that the Apple-specific bandwidth path does
not false-match a non-Apple card that carries a gpu_cores count. Point
the two asserts at _lookup_apple_bandwidth, which returns None for any
name without 'apple' regardless of the general table. The general-lookup
behavior (4090 -> 1008) is correct and untouched.

* fix(hwfit): route string GPU names through the Apple bandwidth helper

Second half of the #2564 regression (RaresKeY review on #4303). That
change moved the Apple tiers out of the generic GPU_BANDWIDTH table into
the dict-only _lookup_apple_bandwidth, but _lookup_bandwidth only called
that helper for dict inputs. A bare-string caller like
_lookup_bandwidth("Apple M3 Max") therefore fell through to the generic
table, found no Apple key, and returned None instead of the conservative
tier. Route both dict and string inputs through the Apple helper (a
string carries no gpu_cores, so it gets the model's lowest tier).
Regression added for the string path plus a non-Apple string control.
 2026-06-15 14:01:05 +00:00
pewdiepie-archdaemon ff7cf8e279 Merge remote-tracking branch 'origin/dev' 2026-06-15 14:00:54 +00:00
pewdiepie-archdaemon 62476ddb55 Merge remote-tracking branch 'origin/dev' 2026-06-15 22:59:57 +09:00
pewdiepie-archdaemon e899817969 Remove duplicate CodeQL workflow 2026-06-15 22:53:29 +09:00
pewdiepie-archdaemon 1cc9a003fd Fix failing post-merge tests 2026-06-15 22:49:06 +09:00
Ahmad Naalweh f7aa2de410 fix(hwfit): distinguish Apple Silicon bandwidth variants (#2564) 
* fix: resolve Apple Silicon bandwidth variants

* fix(hwfit): preserve string lookup path in _lookup_bandwidth

* fix(hwfit): guard Apple bandwidth lookup against false GPU matches

Add "apple" not in gn check to _lookup_apple_bandwidth() so that
non-Apple GPUs with "m3"/"m4"/"m5" in their names (e.g. NVIDIA
Quadro M4 000) don't incorrectly match Apple bandwidth tiers.

Addresses @o3LL review comment on PR #2564.
 2026-06-15 15:13:03 +02:00
Ashvin 514d345334 test(models): pin lookalike hosts to the generic OpenAI branch (#4272) 
#4159 (4b0a977) made build_models_url insert /v1 for path-less bases, so
the TestBuildersRejectLookalikeHosts model assertions that expected
/models started failing and turned the pytest gate red on dev.

Both the generic OpenAI branch and the real Anthropic branch now end in
/v1/models, so a URL-only assertion no longer proves a lookalike host
dodged the Anthropic/Ollama branch. Assert _detect_provider == "openai"
directly and keep the /v1/models expectation.
 2026-06-15 12:43:33 +00:00
pewdiepie-archdaemon 6d507f8128 Merge remote-tracking branch 'origin/dev' into test-main-dev-merge-20260615 
# Conflicts:
#	src/tool_implementations.py
#	static/js/research/panel.js
 2026-06-15 21:20:15 +09:00
pewdiepie-archdaemon 2cbd55b8bd Open email context for agent, email search across All Mail, cookbook serve polish 
- Agent: pass the open email reader (uid/folder/account/from/subject/body
  preview) on every chat submit so 'reply to this' / 'write email saying
  hi' route to ui_control open_email_reply with the right UID instead of
  inventing a new .md draft. Code-level enforcement (chat_routes strips
  create_document + send_email when active_email is set); cross-session
  active_doc_id is now trusted instead of being silently dropped.
  set_active_email/clear_active_email tool-layer helpers in
  tool_implementations.

- ui_control open_email_reply: optional body argument so the agent can
  open-and-write in one call; envelope now forwards uid/folder/account/
  body/panel through tool_output. Tool description sharpened and the
  parser rejects empty bodies on reply/reply-all (forces the agent to
  write rather than open an empty draft).

- Email library: search now runs against [Gmail]/All Mail when the
  current folder is INBOX (archived emails surface). Whirlpool spinner
  + 'Searching…' placeholder while in flight. Each search result is
  stamped with its source folder so clicks open the right email instead
  of whatever shares its UID in INBOX. Search no longer re-applies the
  same text pill locally (which only checks subject/from/snippet, never
  body) so body-only matches don't get dropped after IMAP returns them.
  Initial inbox load bumped 100→500.

- Email favorites: 'Favorite (pin to top)' / 'Unfavorite' in both the
  card menu and the open-reader more menu, backed by a new
  /api/email/flag/{uid}?on=true|false endpoint. Flagged emails always
  bubble to the top of the grid regardless of active sort.

- AI reply in doc editor: never overwrites existing draft text or the
  quoted history. AI suggestion is prepended; AI-generated 'On …
  wrote:' re-quotes are stripped so the original quote isn't visually
  edited.

- Cookbook serve: pre-launch GPU driver / has_gpu / install / version-
  floor checks (vllm minimax_m2 needs 0.10.0+, deepseek_r1 needs 0.7.0
  etc.) before the launch chain starts. Detect 'another model already
  running on this host' and offer Stop & launch (with graceful then
  force tmux kill helpers, port release wait). Per-vendor deep-link
  buttons (vLLM recipe / SGLang cookbook) with hardware hash. Backend
  picker is now a custom dropdown with accent-coloured logos for vLLM,
  SGLang, llama.cpp, Ollama, Diffusers; same glyphs added next to
  package names in Dependencies. Runtime-readiness note moved inside
  the panel (green when ready, red when missing) with an × dismiss.
  Esc collapses the expanded card; expanded card scrolls when it
  overflows; Trust Remote / Auto Tool / Reasoning Parser / Enforce
  Eager / Prefix Caching / Expert Parallel / Speculative / MoE Env on
  one row (Reasoning Parser auto-detected per model family).
  Dtype→Row 1, GPUs→Row 2 (rightmost). Removed redundant GPU 'auto'
  input — command builders read from the GPU button strip. Default
  cookbook open is Download tab.

- Cookbook hwfit: 'Model (latest)' / 'Model (oldest)' header sorts by
  release_date; release dates can be backfilled with the new
  scripts/backfill_model_release_dates.py and recipe metadata pulled
  with scripts/import_from_vllm_recipes.py against the upstream
  vllm-project/recipes catalog (vllm_recipe + min_vllm_version stamped
  on entries).

- Calendar: Quick add hint cycles a random Odysseus-themed example per
  open (wooden horse Friday, crew muster 10am daily, council on
  Ithaca, …). Typing a time like '11pm' in the event title updates
  the hero clock live.

- Doc editor: email-mode Reply button (sparkle icon, accent) opens the
  same Fast/Full + context popover the email reader uses; Ctrl+Alt+M
  toggles markdown preview.

- Memories panel: custom sort picker with per-option icons, default
  'Latest', visible Enabled/Disabled toggle text matching the section
  description style.
 2026-06-15 20:47:51 +09:00
andrewemer cd02ac7ef6 fix(agent): skill-prescribed tools never reach the model's schema list (#4008) 
* Agent: make skill-prescribed tools actually callable

The skill index and matched-skill procedures are injected into the
prompt, but tool selection never followed: manage_skills wasn't in the
RAG-selected schema list (so the model substituted manage_memory), and
a matched skill could prescribe tools (grep, read_file) the model had
no schema for. Now:

- manage_skills rides along whenever the owner has any skills indexed
- a Jaccard-matched skill's requires_toolsets join the selection
- viewing a skill mid-turn via manage_skills unlocks its
  requires_toolsets for subsequent rounds
- admin-intent turns send _ADMIN_TOOLS schemas, matching the prompt
  text _build_base_prompt already advertises
- index_for(active_toolsets=None) no longer hides requires_toolsets
  skills from callers that don't know the active set

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Agent: validate skill requires_toolsets against known tools, not TOOL_SECTIONS

grep/glob/ls ship as function schemas without a prompt-prose section,
so gating on TOOL_SECTIONS silently dropped them from a skill's
requires_toolsets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-15 20:32:43 +09:00
cirim e7abb7559d fix(research): keep Discuss chats grounded on their report (#4006) 
* fix(research): preserve Discuss spin-off primer during context trimming

trim_for_context() kept only system_msgs[:1] as essential and dropped the
rest under budget pressure. A research "Discuss" spin-off seeds the report
as a system message that sits after the preface system messages, so it
landed in extra_system and was the first thing evicted once the chat grew
— the conversation then lost its grounding and drifted off task.

Treat any system message carrying research_spinoff_from metadata as
essential, alongside the leading system prompt, so the seeded report
survives trimming. maybe_compact already retains all system messages.

Tests: tests/test_context_compactor.py::TestResearchPrimerPreserved

* fix(research): ground Discuss spin-off chats on the seeded report

build_chat_context injected global memory (pinned + hybrid-retrieved) and
personal-doc RAG every turn, keyed off the user-level memory_enabled pref
and a request-scoped use_rag flag — never the session. A research spin-off,
whose primer declares the report the sole knowledge base, thus had
unrelated keyword-matched facts pulled in ("wrong data") competing with the
report; its rag=False flag was also ignored (use_rag defaulted on).

Add _session_is_research_spinoff(sess) (detects the primer research_spinoff_from
metadata; handles ChatMessage and dict forms) and, for such sessions,
disable memory injection and force RAG off.

Tests: tests/test_chat_helpers.py spin-off detection cases

---------

Co-authored-by: Dan (cirim) <claude@cirim.org>
 2026-06-15 20:31:57 +09:00
Max Hsu 172a8ea7b0 fix(skills): keep edit mode open on outside-the-textarea click (#4011) 
Clicking the card body outside the edit <textarea> bubbled to the card's
click handler and collapsed the card, silently discarding unsaved skill
edits (issue #4002). The textarea's own stopPropagation only shields
clicks landing on it. Bail out of the card click handler while a
.skill-md-editor is present so the card only leaves edit mode via Save
(Cancel button is handled separately by #3580). Mirrors the same guard
into the built-in capability card, which shared the bug.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-15 20:31:11 +09:00
Daniel 2adae2bbba Parameterize Docker Compose volume host paths (#3907) 2026-06-15 20:30:18 +09:00
Josh Patra f5d3e5098a fix(llm): omit temperature for Kimi K2.5 and K2.6 (#3960) 2026-06-15 20:29:22 +09:00
Josh Patra 4ee5ed4dce fix(memory): return complete memory lists (#3885) 2026-06-15 20:28:25 +09:00
Josh Patra f2bfe9b91f fix(memory): exempt audits from request timeout (#3886) 2026-06-15 20:27:46 +09:00
Hsin-Chen Pai 3f3c05e8c2 docs: add backup/restore guide for odysseus-backup (#2587) 
The scripts/odysseus-backup snapshot/restore CLI was undocumented in
README.md and docs/. Add docs/backup-restore.md covering the snapshot,
list, verify, and restore subcommands, default include/skip behavior
(deep_research and mail-attachments skipped unless flagged), the
destructive-restore warning and its data.before-restore-* stash, a cron
example, and Docker-vs-native data/ paths (including the ChromaDB named
volume caveat). Link it from the README Data section.

Addresses the "Backup/restore guide and helper flow for data/" item in
ROADMAP.md. Docs only; no change to the tool.

Fixes #2583

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-15 20:26:47 +09:00
Dividesbyzer0 2e9f641c2c fix(windows): detect installed CUDA toolkit on launch (#2639) 2026-06-15 20:26:07 +09:00
Dividesbyzer0 627a52ac44 fix(cookbook): shim Windows Store python3 alias (#2610) 2026-06-15 20:25:30 +09:00
RaresKeY 397fce6e32 docs: add pull request review template (#3128) 
* docs: add pull request review template

- add a reusable review structure with findings, validation, and hygiene sections

- document priority badges, intent labels, and expected finding fields

* docs: clarify review template usage

* docs: add small PR review path

---------

Co-authored-by: Alexandre Teixeira <111787685+alteixeira20@users.noreply.github.com>
 2026-06-15 20:23:13 +09:00
dependabot[bot] 10cc2295e5 chore(deps): bump the npm group with 2 updates (#3989) 
Bumps the npm group with 2 updates: [@anthropic-ai/sdk](https://github.com/anthropics/anthropic-sdk-typescript) and [@antithesishq/bombadil](https://github.com/antithesishq/bombadil).


Updates `@anthropic-ai/sdk` from 0.98.0 to 0.104.1
- [Release notes](https://github.com/anthropics/anthropic-sdk-typescript/releases)
- [Changelog](https://github.com/anthropics/anthropic-sdk-typescript/blob/main/CHANGELOG.md)
- [Commits](https://github.com/anthropics/anthropic-sdk-typescript/compare/sdk-v0.98.0...sdk-v0.104.1)

Updates `@antithesishq/bombadil` from 0.3.2 to 0.5.0
- [Release notes](https://github.com/antithesishq/bombadil/releases)
- [Changelog](https://github.com/antithesishq/bombadil/blob/main/CHANGELOG.md)
- [Commits](https://github.com/antithesishq/bombadil/compare/v0.3.2...v0.5.0)

---
updated-dependencies:
- dependency-name: "@anthropic-ai/sdk"
  dependency-version: 0.104.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm
- dependency-name: "@antithesishq/bombadil"
  dependency-version: 0.5.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-15 20:21:04 +09:00
Vishnu 933ec8fec9 fix(memory): reject ambiguous multi-object outputs during skill extraction (#3985) 2026-06-15 10:44:43 +00:00
Merajul Arefin 8fe98cf471 feat(auth): add per-user admin promote/demote toggle (#3078) 
* feat(auth): add per-user admin promote/demote toggle

Admin-only API and Users-tab control to grant/revoke admin rights; refuses to demote the last admin.

* fix(auth): restore pre-admin privilege restrictions on demotion

Promoting now stashes the user's privilege map (privileges_before_admin)
and demoting restores it instead of resetting to defaults, so a
promote/demote round trip can no longer broaden a restricted user's
access. Users without a stash (created as admin, or promoted before this
fix) still demote to DEFAULT_PRIVILEGES so a born-admin's stored all-True
map — including can_use_bash — can't survive demotion.

---------

Co-authored-by: K M Merajul Arefin <merajul.arefin@therapservices.net>
 2026-06-15 10:44:27 +00:00
nubs 55b4a5e6ff fix(ui): restore all-edge modal snap zones (#2260) 2026-06-15 12:36:34 +02:00
dependabot[bot] 3c0e9fcb25 chore(deps): bump the actions group with 4 updates (#3990) 
Bumps the actions group with 4 updates: [actions/checkout](https://github.com/actions/checkout), [actions/setup-python](https://github.com/actions/setup-python), [actions/setup-node](https://github.com/actions/setup-node) and [github/codeql-action](https://github.com/github/codeql-action).


Updates `actions/checkout` from 4.3.1 to 6.0.3
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4.3.1...df4cb1c069e1874edd31b4311f1884172cec0e10)

Updates `actions/setup-python` from 5.6.0 to 6.2.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5.6.0...a309ff8b426b58ec0e2a45f0f869d46889d02405)

Updates `actions/setup-node` from 4.4.0 to 6.4.0
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/49933ea5288caeca8642d1e84afbd3f7d6820020...48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e)

Updates `github/codeql-action` from 3.36.0 to 4.36.2
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/03e4368ac7daa2bd82b3e85262f3bf87ee112f57...8aad20d150bbac5944a9f9d289da16a4b0d87c1e)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 6.0.3
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/setup-python
  dependency-version: 6.2.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/setup-node
  dependency-version: 6.4.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: github/codeql-action
  dependency-version: 4.36.2
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-15 19:26:05 +09:00
dependabot[bot] d5de061656 chore(deps): bump the python group with 3 updates (#3991) 
Updates the requirements on [markitdown](https://github.com/microsoft/markitdown), [pydantic](https://github.com/pydantic/pydantic) and [pydantic-settings](https://github.com/pydantic/pydantic-settings) to permit the latest version.

Updates `markitdown` from 0.1.5 to 0.1.6
- [Release notes](https://github.com/microsoft/markitdown/releases)
- [Commits](https://github.com/microsoft/markitdown/compare/v0.1.5...v0.1.6)

Updates `pydantic` to 2.13.4
- [Release notes](https://github.com/pydantic/pydantic/releases)
- [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md)
- [Commits](https://github.com/pydantic/pydantic/compare/v2.0...v2.13.4)

Updates `pydantic-settings` to 2.14.1
- [Release notes](https://github.com/pydantic/pydantic-settings/releases)
- [Commits](https://github.com/pydantic/pydantic-settings/compare/v2.0.0...v2.14.1)

---
updated-dependencies:
- dependency-name: markitdown
  dependency-version: 0.1.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python
- dependency-name: pydantic
  dependency-version: 2.13.4
  dependency-type: direct:production
  dependency-group: python
- dependency-name: pydantic-settings
  dependency-version: 2.14.1
  dependency-type: direct:production
  dependency-group: python
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-15 19:25:15 +09:00
dependabot[bot] 8b157f452c chore(deps): bump python from 3.12-slim to 3.14-slim (#3988) 
Bumps python from 3.12-slim to 3.14-slim.

---
updated-dependencies:
- dependency-name: python
  dependency-version: 3.14-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-15 19:23:27 +09:00
Simon Guggisberg daec3604f3 fix: correct Three Jugs eval prompt answer (#2542) (#2544) 2026-06-15 19:21:39 +09:00
GeekLuffy 8fa10f9866 feat(teacher): implement Tier 2 LLM self-evaluation 2026-06-15 15:32:38 +05:30
nubs e75a52efbb fix(notes): reset search filter on panel reopen so stale query doesn't hide notes (#2920) 2026-06-15 11:55:46 +02:00
Mazen Tamer Salah f28703adf6 fix(gallery): remove image file only after the delete commit succeeds (#2196) 
delete_gallery_image() deleted the on-disk file before setting
is_active=False and committing. If that commit failed and rolled back,
the record stayed active but its file was already gone — a broken,
unviewable image (data loss).

Soft-delete and commit first, then remove the file best-effort, so a
missing or locked file can no longer 500 a delete that already succeeded
logically.

Adds tests/test_gallery_delete_file_ordering.py covering the
commit-failure (file kept) and success (file removed) paths.
 2026-06-15 11:00:32 +02:00
Kfir Sadeh d8e7cc7053 feat(ui): add real-time diagnostic logs console (#974) 
* feat(diagnostics): add admin-gated real-time diagnostics logs terminal UI

* feat(ui): resolve diagnostics logs feedback and optimize client-side caching

* feat(ui): resolve diagnostics logs feedback
 2026-06-15 10:32:51 +02:00
pewdiepie-archdaemon 1fcec32a3c Cookbook/Serve: 'Install in Dependencies →' link in the runtime readiness note 
When the backend (vllm / sglang / llama_cpp / diffusers) is missing on
the chosen serve target, the runtime-readiness note already flips red
and reads '<backend> missing on <host>.' but offered no fix path.

Append an accent-coloured link that calls openCookbookDependencies with
expandRecipe + the model's repo id, so one click switches to the
Dependencies tab, expands the right backend row's recipe panel, and
pre-selects the model so the user just hits Run.
 2026-06-14 22:57:43 +09:00
pewdiepie-archdaemon 768bcb565a Cookbook/Dependencies: variant toggle now uses the agent/chat mode-toggle; Copy inside code 
- Drop the 'Install via' label and the pill-tag variant buttons. The
  toggle is now the same sliding-pill mode-toggle used by the
  Agent/Chat selector in the chat input. Pip/uv on the left, Docker on
  the right, default = Pip/uv. CSS: extended .mode-chat::before's
  translateX(100%) rule to also fire on .mode-right so non-chat
  callers can use the same animation without claiming the chat-only
  class name.
- Copy button moves inside the <pre>: absolute-positioned at the
  top-right corner, icon-only, padding-right on the pre makes room.
  Matches the Setup-token copy pattern in the integrations form.
 2026-06-14 22:54:38 +09:00
pewdiepie-archdaemon 63b4ad2e9c Cookbook/Dependencies: Pip/uv vs Docker variant toggle on recipe panel 
Each recipe catalog entry now carries two variants:
  variants.pip    → uv pip install …
  variants.docker → docker pull <image>

A small 'Install via' pill row in the panel toggles between them
(default = Pip/uv per the user's preference). Switching variant or
changing the model re-renders the <pre> via _refreshRecipePre(); the
display text drops the 'source venv/bin/activate' prefix for Docker
since docker pull doesn't need a venv. Run honours the active variant
so picking Docker queues 'docker pull …' as the tmux task.
 2026-06-14 22:47:26 +09:00
pewdiepie-archdaemon d70eb99a0d Cookbook/Dependencies recipes: install into configured venv, drop 'uv venv' 
Recipes now hold ONLY the install command(s). The rendered <pre>
prepends a 'source <envPath>/bin/activate' line so the user sees a
paste-ready sequence; Run uses env_prefix (same path the Install
button uses) to activate the configured venv before the install
command, so the install lands in the existing environment rather
than a fresh .venv in whatever CWD the tmux task happens to start in.

- cookbook-deps-recipes.js: trim each recipe to its single pip command
- cookbook.js: _recipeDisplayText() prepends the activate context for
  display; pre's data-dep-recipe-install holds the raw install-only
  command list so Run knows what to send; Run builds env_prefix the
  same way _installDep does.
 2026-06-14 22:45:12 +09:00
pewdiepie-archdaemon d44de3af43 Cookbook/Dependencies: populate recipe model picker from downloaded models 
The recipe dropdown was a static catalog (MiniMax / Any vLLM model). Now
it lists every model already downloaded on the active server (the same
_cachedModelIds set the Launch tab + dl-dots already drive), plus an
'Other (generic …)' fallback. The change handler uses pickRecipe(backend,
modelId) to find the best match — MiniMax ids land on the MiniMax recipe,
everything else falls back to the generic install.

cookbook-diagnosis.js: openCookbookDependencies's pre-select logic now
matches by full option value (model id) instead of label substring, since
the dropdown values are full repo ids now.
 2026-06-14 22:40:52 +09:00
pewdiepie-archdaemon 25dd94234c Cookbook/Launch: pre-flight backend install check, deep-link to Dependencies 
Before the quickrun (Run) button fires /api/model/serve, ask the deps
API whether the chosen backend (vllm / sglang / llama_cpp) is actually
installed on the target server. If not:

- Toast: '<backend> not installed on <host>. Opening Dependencies …'
- Route the user into the Dependencies tab via the existing
  _openCookbookDependencies helper (now exported as
  openCookbookDependencies)
- Auto-expand the recipe panel for that backend
- Pre-select the user's model in the panel's picker so the right
  recipe is highlighted out of the box

The serve task is suppressed; the Run button is re-enabled. Once the
install task finishes in Running, the user clicks Run again.

cookbook-diagnosis.js: openCookbookDependencies takes an opts object
that, when expandRecipe is set, finds the row's caret and clicks it,
then matches a recipe label by model (currently only MiniMax has a
specific entry; the generic fallback stays selected otherwise).
 2026-06-14 22:35:56 +09:00
pewdiepie-archdaemon 600fa6be8a Cookbook/Dependencies: per-backend recipe panel (vllm/sglang/llama_cpp) 
Each row for vllm, sglang, llama_cpp now carries an expand caret that
opens an inline recipe panel below the row. The panel has:
  - 'Serving which model?' select populated from a new tiny catalog
  - <pre> code block showing the exact shell sequence for that pair
  - Copy: clipboard the commands
  - Run: launch the joined 'cmd1 && cmd2 && …' as a tmux task on the
    currently-selected deps server (same plumbing as Install)

New file: src/static/js/cookbook-deps-recipes.js — single source of
truth for the recipes. Seeded with MiniMax M2/M2.7 + a generic fallback
for each backend (all three use 'uv venv → source .venv/bin/activate
→ uv pip install ... --torch-backend auto', the recipe the user
pasted). Adding model-specific recipes is now a one-entry edit.

Next commit: Launch-tab pre-flight that intercepts the serve click
when the backend isn't installed and deep-links into this panel.
 2026-06-14 22:33:49 +09:00
pewdiepie-archdaemon 781a3ee829 Cookbook: rename 'Run' tab → 'Launch' (cookbook.js:1865) 2026-06-14 22:23:38 +09:00
pewdiepie-archdaemon a9de61771a Cookbook serve panel: tighten vertical spacing inside Advanced fold 
Rows inside the Advanced details were inheriting the standard
6px row-gap from .hwfit-serve-row (used to give the Core knobs
some breathing room). Inside Advanced — where the rows are
mostly single-line dropdowns — that read as half a row of empty
space between every pair.

Now inside Advanced only:
- grid row-gap drops to 4px
- label → control margin-top drops to 1px (was 2px)
- checks row gap also drops to 4px

Outside Advanced (Core, etc.) the original spacing stays.
 2026-06-14 09:14:31 +09:00
pewdiepie-archdaemon 9873f9b44f Cookbook diagnosis: fold message + suggestion into the toolbar row 
Was rendering as a separate body block below the Copy/× toolbar.
Now the diagnosis message and the suggested-action text sit inline
on the left of the toolbar, with Copy and × pinned to the right —
reads as one self-contained header strip instead of stacked rows.
 2026-06-14 09:03:58 +09:00
pewdiepie-archdaemon 09a82852c0 Cookbook tmux: history-limit 100k + crash-watchdog grabs 2000 lines 
The tmux default 2000-line scrollback was getting blown out by
long vLLM tracebacks (DeepSeek-V4-Flash launch crash had the root
cause scrolled off; the user saw only the tail "See root cause
above"). Bumped:

- tmux server history-limit to 100000 at session creation (prepended
  to each tmux new-session command so both local + ssh remote inherit
  the larger scrollback)
- crash-watchdog capture-pane from -S -200 → -S -2000 so the
  diagnosis includes the actual exception line
 2026-06-14 09:02:04 +09:00
pewdiepie-archdaemon 4074e77d93 Cookbook: auto-set KV cache to fp8 for DeepSeek V3/V4/R1 MoE families 
These models OOM on --kv-cache-dtype auto (≈bf16) at any usable
context with current tensor-parallel layouts. _detectModelOptimizations
now seeds opts.kvCacheDtype='fp8' for them, and the serve panel's KV
Cache select picks that up as the default unless the user has a
saved override on this skill.
 2026-06-14 08:57:29 +09:00
pewdiepie-archdaemon d3944be1be Cookbook: detect DeepSeek V4+ as MoE so Expert Parallel + Spec show 
The DeepSeek branch in _detectModelOptimizations matched only V3
and R1 literally. DeepSeek-V4-Flash (and future Vx / Rx) didn't
hit any branch, so the Expert Parallel checkbox + Speculative
defaults never surfaced in the Run panel. Widened to a regex that
catches v3/v3.1/v4/v5/v10+ and r1/r2/… for both the expert-parallel
flag and the MTP speculative defaults.
 2026-06-14 08:51:57 +09:00
pewdiepie-archdaemon ce964b9a00 Cookbook Run panel: drop ‹ › arrows on Speculative tokens, narrow to 44px input 
The +/- step buttons next to the Speculative tokens count read as
clutter for a 1-10 single-digit input — the native number-input
spinner + manual typing is enough. Reduced the input width to 44px
so it sits tight next to the method dropdown.
 2026-06-14 08:50:20 +09:00
pewdiepie-archdaemon 1d7d9c5e9c Cookbook deps: drop the manual vLLM install block + Run handlers 2026-06-14 08:49:20 +09:00
pewdiepie-archdaemon adac89c8e2 Cookbook deps: NVIDIA vs AMD ROCm-aware vLLM install commands 
Reads the last hwfit scan's backend (window._hwfitSystemCache.backend)
and picks the right vLLM install path per vendor:

- NVIDIA/CUDA (default)
  - uv:     uv pip install -U vllm --torch-backend auto
  - docker: docker pull vllm/vllm-openai:latest
- AMD/ROCm
  - uv:     uv pip install -U vllm --torch-backend rocm
  - docker: docker pull rocm/vllm-dev:main

The <pre> previews are re-painted on render to match what Run will
actually launch, and the confirm dialog tags the backend so the user
knows what they're committing to.
 2026-06-14 08:46:58 +09:00
pewdiepie-archdaemon 65a2e51af8 Cookbook deps: convert manual install snippets to Run buttons 
Was just a copy-paste reference. Each row now has a Run button that
launches the command as a tmux task on the currently-selected deps
server (same path Reinstall already uses) — Odysseus does the work,
the user watches progress in the Active tab. Dropped the plain
pip option since the existing per-package Install button already
covers it; kept uv (recommended) and Docker pull as the two
alternatives.
 2026-06-14 08:45:43 +09:00
pewdiepie-archdaemon 04a97adbb3 Cookbook: Extra args under Reasoning/Spec + manual vLLM install hints in Dependencies 
- Moved "Extra args" out from above the vLLM advanced checks
  (Reasoning Parser, Speculative, MoE Env) to AFTER them, so it
  reads as "after the advanced toggles, anything else".
- Added a collapsed "Manual install (vLLM)" details block to the
  Dependencies tab description with three copy-paste recipes:
  uv venv + uv pip (recommended), plain pip, and docker pull
  vllm/vllm-openai:latest. Useful when the in-app Install button
  can't run (offline target, custom torch backend, etc).
 2026-06-14 08:43:10 +09:00
pewdiepie-archdaemon 8829ae2675 Cookbook serve: nudge runtime-note dismiss × up 4px (top:-4 → -8) 2026-06-14 08:33:14 +09:00
pewdiepie-archdaemon 09a1718103 Skills test: set explicit max_tokens=4096 instead of 0 
max_tokens=0 made stream_agent_loop omit the param entirely, which
on some OpenAI-compat upstreams (DeepSeek in the report) meant the
model defaulted to a very short or zero-token completion — the user
saw "the model returned an empty" even though normal chat with the
same model worked (chat sends its preset's max_tokens). Match the
chat default.
 2026-06-13 23:09:15 +09:00
pewdiepie-archdaemon f03a9e79a7 Settings: tighten endpoint logo+select gap + align fallback trash right 
- .adm-model-logo + .settings-select { margin-left: -4px } pulls
  the select 4px closer to its logo chip so the row reads as one
  unit instead of having an obvious gap between the icon and the
  dropdown.
- Fallback-row selects get flex:1 so the trash-can sits flush
  against the right edge of the row — matching the right edge of
  the Endpoint and Model selects in the rows above the fallback
  list (was rendering tight to the model select's content width).
 2026-06-13 23:04:27 +09:00
pewdiepie-archdaemon bb66914b1e Settings: clamp logo SVGs to 18px chip + endpoint dropdown gets logo 
Provider SVGs in providers.js declare only viewBox (no width/height),
so when injected into the 18×18 logo chips they fell back to the
browser default of 300×150 and blew out the row.

- CSS: SVGs inside settings logo chips (`span[id$="-logo"]`,
  the 18px wrappers in fallback rows) now stretch to 100%/100% of
  their container.
- Added matching `-logo` chip next to the Endpoint dropdowns in
  Default Chat Model and Utility Model cards.
- New `_syncEndpointLogo` helper mirrors the selected endpoint
  option's text label through providerLogo() (the select value is
  a UUID and wouldn't match anything otherwise), and
  `_fillEndpointSelect` calls it on each render.
 2026-06-13 23:00:16 +09:00
pewdiepie-archdaemon 8053d6a50a Skills: dedupe by name + move Select to 2nd in kebab menu 
- The API occasionally returns the same skill twice (built-in
  shadow vs user copy, or a write/read race) which made the
  duplicate-detector tag BOTH copies as the "recommended" keeper
  (the find-skills card showing duplicate #1 twice).  Loading now
  filters out repeats by lowercased name before render.
- Reordered the per-skill kebab menu: Publish/Unpublish → Select
  → Edit → Test → Audit → Delete. Select previously sat at the
  bottom; lifting it next to Publish puts the bulk-mode entry
  point with the other bulk-style action.
 2026-06-13 22:49:48 +09:00
pewdiepie-archdaemon 7cbf5a2c00 Research panel: connect Settings toggle to body + lift textarea 4px 
- When Settings is expanded, the toggle bar's bottom radius/border
  flattens and merges into the row below (zero gap, softer
  top-border on the body) so the row visually reads as the toggle's
  open-state content instead of an unrelated card below it.
- .research-query margin-top trimmed from 6px to 2px (lifts the
  textarea ~4px closer to the description line above).
 2026-06-13 22:41:49 +09:00
pewdiepie-archdaemon 0895c70fc9 Research: drop visible category row, move Format override into Settings 
Auto handles 90%+ of cases — the row of category buttons was
visual noise on the main panel. Now:
- Removed the .research-category-row from above the textarea.
- Added a Format <select> inside Settings (next to Rounds) with
  Auto / Product / Compare / How-to / Fact-check options. Default
  is Auto, same as before.
- Updated all the JS that read .research-cat.active / data-cat to
  read #research-category.value instead (_saveSettings, _readSettings,
  _resetCategoryToAuto, _editJob, _restoreSavedSettings).

Same wire to the backend — settings.category still carries through.
 2026-06-13 22:38:20 +09:00
pewdiepie-archdaemon 16c41612ca Research panel: ? hint chip on Rounds + cogwheel icon on Settings toggle 2026-06-13 22:34:12 +09:00
pewdiepie-archdaemon 7ef3e353c6 Research panel: push #research-stats count chip down 4px 2026-06-13 22:33:09 +09:00
pewdiepie-archdaemon 10b9e6b81f Research panel: accent-tint the research SVG next to the title 2026-06-13 22:32:51 +09:00
pewdiepie-archdaemon 360ce696e0 Research panel: pull "past runs in Library" hint up 4px (top:-4px) 2026-06-13 22:32:30 +09:00
pewdiepie-archdaemon 0548d335d4 Research panel: move the research SVG next to the Research title 2026-06-13 22:31:41 +09:00
pewdiepie-archdaemon 79d55b46a6 Research panel: pull loop-agent description line up 4px (margin-top 6→2) 2026-06-13 22:31:09 +09:00
pewdiepie-archdaemon 93c0529e00 Research panel: "Past runs in Library" hint inline with loop agent line 
Was rendering on its own row under "Multi-step web research with an
LLM-in-the-loop agent". Now appended to that same flex-wrap line as
"— past runs in Library, Research" so the header section stays one
visual block instead of two.
 2026-06-13 22:29:57 +09:00
pewdiepie-archdaemon a29c2b25d0 Research panel: Past Research library hint goes inline with section title 
Was rendering on a second row below the "Past research" header,
inflating it to two rows. Now appended to the title span as a small
inline chip — "Past research — all in Library, Research" — keeping
the header at one row. Same click → close panel + open Library tab.
 2026-06-13 22:27:49 +09:00
pewdiepie-archdaemon 654f9f82c7 Cookbook: don't auto-fold Direct Download from inside its own body 
The capture-phase scroll listener was firing for scrolls anywhere
in the modal — including the Trending models list, which lives
inside the Direct Download fold body. Scrolling that list was
auto-folding the section that contains it.

Bail early if the scroll target is the fold body or a descendant —
the section only folds on scrolls in sibling scrollers (.cookbook-body,
.hwfit-list, .modal-content).
 2026-06-13 22:26:04 +09:00
pewdiepie-archdaemon 45b3cd15df Research: rotate textarea placeholder through 10 example queries 
Each time the panel opens we pick a random entry from a list of 10
diverse research prompts (history, tech, food, science, fact-check,
how-to) so the textarea hint feels fresh and shows the breadth of
queries the tool handles instead of always nudging toward the same
Odysseus example.
 2026-06-13 22:23:58 +09:00
pewdiepie-archdaemon d006e38a2f Cookbook task menu: merge Edit actions + group items into sections 
- Removed standalone "Edit cmd & relaunch" — "Edit in serve panel"
  renamed to "Edit & relaunch" and is now the single edit entry.
  Tooltip notes that the raw cmd is still editable inside the panel.
- Tagged each item with a group (run / edit / endpoint / copy /
  danger) and renderer inserts a thin divider whenever the group
  changes, so the menu reads as visual blocks instead of one long
  list.
 2026-06-13 22:18:13 +09:00
pewdiepie-archdaemon 438db357ff Cookbook Active tab: header → Active, Reconnect → Reconnect tmux, section dividers 
- Header h2 inside the Active group now says "Active" (matches
  the renamed tab) instead of "Running".
- Both context-menu Reconnect entries (the normal one and the
  recover-from-vanished-process fix) say "Reconnect tmux" so the
  user knows what the action actually does.
- Sibling cookbook-server-section-* blocks inside the Active group
  get a top divider + 14px gap so transitions between server
  groups (local / remote-host / etc) read clearly.
 2026-06-13 22:11:45 +09:00
pewdiepie-archdaemon 3ff4eb5519 Cookbook: _gpuToggleTotal updates on every scan, not just the first 
Previously the global GPU-toggle total was set once and never
overridden, so a first scan on the local 1-GPU container left
the Run-panel GPU button row stuck on GPU 0 even after switching
to a 4-GPU remote host. Now any scan returning a positive total
updates the binding; zero/missing values still don't clobber a
known-good count (no flicker during in-flight re-probes).
 2026-06-13 22:06:35 +09:00
pewdiepie-archdaemon f34cb42b07 Cookbook: runtime readiness text moves to model title chip 
Mirrored the panel's runtime readiness note into a small chip
appended to the .memory-item-title at the top of the expanded
serve card. The in-panel note becomes a hidden source-of-truth.

This way the "vLLM ready on … : vLLM CLI: …; python package:
vllm 0.22.0" status sits inline with the model name where the
user is already looking, instead of buried below the toolbar row.
 2026-06-13 21:57:21 +09:00
pewdiepie-archdaemon ac4de93928 Cookbook: rename Serve tab → Run (label only, data-backend stays Serve) 2026-06-13 21:55:24 +09:00
pewdiepie-archdaemon 6763fe4d44 Cookbook GPU/RAM toggle: default to whichever pool has more capacity 
On initial render, compare total_ram_gb vs gpu_vram_gb — if RAM is
the larger pool, pre-select the RAM (count=0) button instead of the
max-GPU button. Boxes with more system RAM than VRAM (low-VRAM
GPU + lots of system memory, or CPU-only servers with a small
adapter) now open on the dominant pool.
 2026-06-13 21:40:40 +09:00
pewdiepie-archdaemon 44a60c1261 Cookbook toolbar: move Search next to Standard, Engine/Quant/Context to right 
New order: [Standard ▾] [Search ............] [Engine] [Quant] [Context]
so the two primary picks (type + free text) sit together at the
left, with the more advanced filters lined up to the right.
 2026-06-13 21:30:05 +09:00
pewdiepie-archdaemon f09f606bec Cookbook fold: smooth max-height + opacity transition 
display:none toggle was instant and felt jarring during auto-fold/
auto-expand. Swapped to a CSS class `.is-folded` that transitions
max-height (0 ↔ 1200px) and opacity (0 ↔ 1) over ~280ms with ease,
so both manual chevron clicks and the scroll-driven toggles slide
in/out smoothly.
 2026-06-13 20:14:34 +09:00
pewdiepie-archdaemon e6349c016e Cookbook auto-fold: auto-expand when scrolling back to top 
scroll handler now tracks per-target scrollTop via WeakMap. Downward
scroll on any scroller in the cookbook modal folds Direct Download;
scrolling back to top (scrollTop <= 0) unfolds it. Manual chevron
clicks still win — they persist to localStorage; auto-toggles
don't, so the user's last explicit pick survives reload.
 2026-06-13 20:12:30 +09:00
pewdiepie-archdaemon e630605aef Cookbook auto-fold: capture-phase scroll listener catches hwfit-list 
IntersectionObserver missed the case because scrolling inside the
nested .hwfit-list (max-height:52vh own scroller) doesn't move the
header out of view at all. The user wants any downward scroll in
the scan/download area to fold Direct Download.

Switched to a capture-phase scroll listener on #cookbook-modal that
catches every scroll event from any nested scroller (.hwfit-list,
.cookbook-body, .modal-content). Folds only on downward scrolls so
scrolling back up doesn't keep re-folding.
 2026-06-13 20:10:46 +09:00
pewdiepie-archdaemon 74e563dabc Cookbook auto-fold: use IntersectionObserver to catch any scroll source 
The scroll listener on .cookbook-body never fired — the user is
likely scrolling inside the nested .hwfit-list (max-height:52vh)
which doesn't bubble to its parent. IntersectionObserver fires
whenever the Direct Download header crosses the viewport edge
regardless of which container moved.

Folds only when boundingClientRect.top < 0 (header pushed up past
the top) so modal close / detach doesn't trigger it.
 2026-06-13 20:07:32 +09:00
pewdiepie-archdaemon ae0b29af3d Cookbook auto-fold: target the actual scroll container (.cookbook-body) 
Previous .modal-body / .cookbook-content lookup matched neither the
desktop scroller (.cookbook-body) nor the mobile one (#cookbook-modal
.modal-content), so the scroll listener was attached to document.body
and never fired. Walk up to whichever scroller actually exists.
 2026-06-13 20:05:33 +09:00
pewdiepie-archdaemon d68c75a82c Cookbook: auto-fold Direct Download when its header scrolls past top 
Added a scroll listener on the parent .modal-body / cookbook-content
that folds the Direct Download body once its h2 header has scrolled
above the container's top edge. Frees the viewport for the Scan
section below while leaving the chevron clickable to expand again.

Auto-fold doesn't write to localStorage (only manual clicks do)
so the user's last explicit preference still wins on reload.
 2026-06-13 20:03:14 +09:00
pewdiepie-archdaemon a615f7f786 Cookbook Trending: drop ↻ refresh button (trending list reloads on toggle) 2026-06-13 20:01:54 +09:00
pewdiepie-archdaemon 0808de0b3b Cookbook Trending: shrink trending-up icon 18px → 15px 2026-06-13 20:01:26 +09:00
pewdiepie-archdaemon aba3a7ae43 Cookbook Trending: accent trending-up icon + chevron on right + larger row 
- Added a trending-up (market-up) SVG before the label, tinted
  accent so the section reads as "what's hot".
- Chevron ▸ moved from the left to the right side of the toggle
  row (still rotates via the existing CSS).
- Bumped the toggle row taller (26→34px) with 13px font + 18px
  icon so the section header has more presence.
 2026-06-13 19:59:41 +09:00
pewdiepie-archdaemon fa3adca5fc Cookbook Trending: HF link pill tinted accent 
Inside #cookbook-hf-latest-list the HF ↗ link is the row's main
affordance, so tint it accent instead of the muted-gray default
used elsewhere.
 2026-06-13 19:58:53 +09:00
pewdiepie-archdaemon f78084c230 Brain cards 32px tall + Trending tab up 8px + drop hwfit Rescan 
- Brain admin-card header rows get min-height:32px so cards with
  toggles and cards without (Inject Skills) align.
- Cookbook Trending models tab nudged up 8px (top:-3 → -11).
- Removed the ↻ RESCAN button in hwfit toolbar; manual EDIT still
  available and auto-probe runs on container restart.
 2026-06-13 19:56:22 +09:00
pewdiepie-archdaemon 7004e1de7b Brain settings: reorder + AI star icons on each toggle 
- Reordered: Auto-extract memories → Auto-extract skills →
  Auto-approve skills → Inject Skills (Auto-approve now above
  Inject so all three AI-driven toggles cluster together)
- Added accent-tinted star icon (the AI star) before:
  Auto-extract memories, Auto-extract skills, Auto-approve skills
- Inject Skills gets a neutral down-arrow-into-line icon (it's
  configuration, not AI work)
 2026-06-13 19:36:10 +09:00
pewdiepie-archdaemon e2a30c0600 Skills: Audit on left + accent star, Select w/ dot/X icon swap 
- Reordered the toolbar so Audit sits left of Select (matches the
  brain memories layout where bulk actions live before Select)
- Renamed "Audit all" → "Audit"
- Star icon in Audit now tinted with var(--accent, var(--red))
- Select button gets the same dot/X SVG swap used in brain
  memories (dot in idle state, X when bulk-select mode is active)
 2026-06-13 15:47:16 +09:00
pewdiepie-archdaemon eb0abe7c90 Doc compose: Cc toggle and X close up 1px (top:calc(50% + 2px) → +1px) 2026-06-13 14:59:46 +09:00
pewdiepie-archdaemon c822d34ce6 Revert Chat/Agent mode tag in message header 
Per user report — the tag's mode metadata coincided with a
500 error on agent mode (especially on mobile). Removing the
UI tag, the chat.js writes of metadata.mode, and the CSS pill
so agent mode posts work cleanly again.

Touches:
- chat.js: drop _sendMode capture + meta.mode writes (user + assistant)
- chatRenderer.js: roleTimestamp() back to a single (when) arg, drop
  the .role-mode-tag append; updated three call sites
- style.css: dropped .role-mode-tag and .role-mode-agent rules
 2026-06-13 11:39:32 +09:00
pewdiepie-archdaemon 0889eb4e01 Doc compose: accent prefix labels for each field + Cc btn up 2px 
- Each input now has a sibling .email-field-prefix span (To / Cc /
  Bcc / Subject) absolute-positioned at the left edge in the
  accent color. Inputs get padding-left:44px (64px for Subject)
  so typed text doesn't slide under the prefix.
- Placeholders shrink back to just the example so only the
  prefix gets the accent color, not the example text.
- Cc toggle moved another 2px up (calc(50% + 4px) → calc(50% + 2px)).
 2026-06-13 09:35:21 +09:00
pewdiepie-archdaemon 77f00eeab1 Doc compose: accent-tint the To/Cc/Bcc placeholder hints 2026-06-13 09:33:28 +09:00
pewdiepie-archdaemon 86daf254cf Doc compose: Cc toggle up 4px (8→4), X close up 2px (4→2) 2026-06-13 09:33:03 +09:00
pewdiepie-archdaemon 9ea3a250db Doc compose: drop field labels, expand placeholders with examples 
- Removed the <label>To/Cc/Bcc/Subject</label> elements — they
  doubled what the placeholder said.
- Placeholders now carry both the field name AND an example so an
  empty input still tells the user what to type:
    To  recipient@example.com
    Cc  cc@example.com, example2
    Bcc  bcc@example.com
    Subject
 2026-06-13 09:29:20 +09:00
pewdiepie-archdaemon c537d2b95c Doc compose Cc/Bcc X: nudge 4px down (top:50% → top:calc(50% + 4px)) 2026-06-13 09:28:24 +09:00
pewdiepie-archdaemon f538da9a8e Doc compose: X close button inside Cc and Bcc fields 
Adds a per-field X (24x24 SVG, opacity 0.4 → 1 + accent on hover)
absolute-positioned at the right edge of each Cc/Bcc field. Click
hides both rows, clears their inputs, and restores the Cc opener
on the To row. Inputs get padding-right:32px so the close button
doesn't overlap typed text.
 2026-06-13 08:40:37 +09:00
pewdiepie-archdaemon 015aeb1fab Doc compose Cc toggle: another 4px down (4 → 8) 2026-06-13 08:39:03 +09:00
pewdiepie-archdaemon 0d27480719 Doc compose Cc toggle: nudge 4px down (top:50% → top:calc(50% + 4px)) 2026-06-13 08:37:38 +09:00
pewdiepie-archdaemon 81a9a1fed3 Doc compose Cc toggle: vertically center inside To field 
Was `top: calc(50% + 4px)` which left the button 4px below the
true vertical center of the input — visibly misaligned. Dropped
the +4 offset so the toggle anchors at top:50% / translateY(-50%)
and tracks the input's center.

Also removed the redundant base rule's position:relative + top:2px
nudge — it was being overridden by the more-specific
.email-field .email-cc-toggle absolute positioning anyway.
 2026-06-13 08:12:03 +09:00
pewdiepie-archdaemon a01ca5a0a1 Email reminders: "Note" picks open a write-your-own-text modal 
- Renamed "Note (no timer)" → "Note".
- Clicking it now opens a small modal with a textarea + Save/Cancel.
- The typed text becomes the todo item; due_date is omitted so no
  timer fires. Esc cancels; Cmd/Ctrl+Enter saves.
 2026-06-13 08:09:37 +09:00
pewdiepie-archdaemon 3239430996 Email reminders: add "Note (no timer)" option 
Re-adds the timer-less note path next to the time-based presets.
Picking it POSTs the same payload but omits due_date so the entry
lives in notes as a plain reply todo with no reminder firing.
Toast: "Reply note saved" instead of "Todo reminder set for …".
 2026-06-13 08:06:53 +09:00
pewdiepie-archdaemon 65ead1f799 Email library: reset select-mode + selectedUids on open 
Was sticking on toggled-on state if the user closed the library
while in select-mode — reopening showed the Cancel/X toggle even
though no emails were selected. Force-reset state._selectMode and
state._selectedUids in openEmailLibrary so each open starts fresh.
 2026-06-13 08:03:10 +09:00
pewdiepie-archdaemon 6cc45a4f77 AI reply: 1st click shows cached, 2nd click clears + opens menu 
Correct behavior:
1. Cached draft + first click → opens the cached reply
2. Cached draft + second click → clears the cache and opens the
   Fast/Full + context menu so the user can request a fresh draft
3. No cache → opens the menu directly

Per-button shownOnce dataset tracks the first-click state so the
second click triggers the menu instead of replaying the cached
reply again.
 2026-06-13 08:01:27 +09:00
pewdiepie-archdaemon f6c4c9a67c AI reply always reopens menu + Cc toggle 2px down in doc compose 
- AI reply: removed the cached_ai_reply shortcut so clicking the
  button always reopens the Fast/Full + context menu. Lets the user
  ask for a fresh draft (with new steering) instead of being locked
  into the cached one.
- .email-cc-toggle gets position:relative + top:2px so it
  baseline-aligns with the To: field chips next to it in the
  document email compose.
 2026-06-13 07:59:41 +09:00
pewdiepie-archdaemon 10a25f5959 Email library: Select button matches brain memories (dot↔X swap) 
- Initial button: dot-in-circle SVG + "Select" label
- After click (select-mode on): X SVG + "Cancel" label + .active class
- Same SVG glyphs as memory.js so the two pages feel consistent.
Hooked into the toolbar Select toggle AND the bulk-bar Cancel button
so both reset the icon state.
 2026-06-13 07:58:24 +09:00
pewdiepie-archdaemon a57327c13f Email reader cluster: solid bg in wrapped 2-row mode to stop body bleed-through 2026-06-13 07:32:54 +09:00
pewdiepie-archdaemon 37e49246a6 Email reader: To/Cc expand as floating panel instead of inline reflow 
When the chevron opens the details, the To/Cc rows pop up as an
absolutely-positioned panel anchored to the bottom of the meta
block — with bg, border, rounded corners and a shadow. Nothing
in the rest of the header reflows: From row stays put, the action
cluster stays put, the email body content stays put. This kills
all the height-/spacing-jump quirks the inline-expanded design
was fighting.
 2026-06-13 07:30:30 +09:00
pewdiepie-archdaemon 0351e5e166 Revert "Email reader meta: full chip names + locked-in From/To/Cc labels" 
This reverts commit 98c05dd08d.
 2026-06-13 07:26:24 +09:00
pewdiepie-archdaemon 98c05dd08d Email reader meta: full chip names + locked-in From/To/Cc labels 
- .email-reader-meta .recipient-chip drops max-width and overflow
  truncation so the full name renders in each chip. The parent
  .recipient-chips span already has overflow-x:auto, so users can
  swipe horizontally to reveal any chip whose tail is clipped off
  the right edge of the row.
- Strong (From: / To: / Cc:) labels get explicit white-space:nowrap
  + flex-shrink:0 so they never truncate even when the row is
  squeezed to its minimum width.
 2026-06-13 07:25:02 +09:00
pewdiepie-archdaemon 4811af7ab2 AI reply menu: viewport-aware placement on mobile 
- Horizontal: max-width and left already clamped to viewport-16.
- Vertical: prefer below the button, but flip ABOVE if there's
  more space there (e.g. button near the bottom of the viewport).
- max-height clamped to viewport-16 with overflow:auto as a final
  guard so the menu can never extend past the screen edge.
 2026-06-13 07:21:18 +09:00
pewdiepie-archdaemon ba17829202 AI reply menu: Fast/Full sit below the context textarea as confirm 
Dropped the two-step (pick mode → context → OK) flow. Now the
context textarea is at the top of the popover and Fast (left) /
Full (right) sit below as the confirm buttons themselves — they
fire the draft with whatever's currently in the textarea (empty
= no steering).
 2026-06-13 07:16:26 +09:00
pewdiepie-archdaemon 8f696064d5 AI reply menu: outside-click closer ignores clicks inside the menu 
The document-level capture listener was closing the popover on
ANY click — including clicks inside the context textarea, which
made it impossible to focus the input. Replaced with an inline
handler that bails when the click target is inside the menu.
 2026-06-13 07:15:44 +09:00
pewdiepie-archdaemon 3819a23344 AI reply menu: click Fast/Full → context input → OK 
Restructured flow:
1. Click Fast or Full → reveals an optional context textarea
   ("Add context (optional)") below
2. Type optional steering note or leave blank
3. Click OK → triggers the draft with the chosen mode + note

Dropped the standalone … note-toggle button — the textarea is now
gated on picking a mode, which makes it easier to discover.
 2026-06-13 06:59:10 +09:00
pewdiepie-archdaemon cedc38fee8 AI reply menu: drop draft sub-buttons + viewport-clamp on mobile 
- Removed the conditional Draft fast / Draft full buttons. Note
  textarea is always-on via the … toggle, and whatever's in it
  is picked up by the existing Fast / Full buttons as noteHint.
- Clamped the popover max-width and left position to
  Math.min(220, viewport-16) + 8px margin so the (now wider) menu
  doesn't spill off the right edge on narrow mobile screens.
 2026-06-12 23:41:46 +09:00
pewdiepie-archdaemon 198af4709d AI reply menu: add … note input to steer the draft 
Top row keeps Fast / Full + a new horizontal-dots button. Clicking
the dots reveals a textarea ("e.g. reply nicely but say no"); as
soon as text is in it the panel shows Draft fast / Draft full
buttons that pass the note through as noteHint to the AI reply
endpoint. Empty textarea hides the draft buttons so the user only
gets the steered draft when they've actually typed direction.
 2026-06-12 23:39:05 +09:00
pewdiepie-archdaemon 696ff78302 Mobile sidebar: force opaque background to stop chat model picker bleed-through 
Firefox mobile rendered the backdrop-filter:blur + var(--panel)
combination on the slide-out sidebar as semi-transparent, so the
chat input bar's selected-model label (e.g. "minimax") was
visible behind the drawer. Force background:var(--panel) and
backdrop-filter:none inside the mobile @media block.
 2026-06-12 23:37:39 +09:00
pewdiepie-archdaemon f2da86b455 Email reader mobile: kill background gradient + padding-left too 
The left-edge gradient fade was likely the source of the perceived
shadow under the icons on mobile. Forced background:none and the
matching padding-left:0 on mobile so the cluster reads as bare
icons without any soft edge.
 2026-06-12 23:26:01 +09:00
pewdiepie-archdaemon 5212758698 Email reader mobile: force-disable overlay box-shadow with !important 2026-06-12 23:19:53 +09:00
pewdiepie-archdaemon 9e73912d24 Email reader mobile: drop overlay shadow + lift action cluster 1px more 2026-06-12 23:17:32 +09:00
pewdiepie-archdaemon 6d328b1ad7 Email library: title also shifts 4px right in expanded card view 
Was only the date moving — the expanded card had a more-specific
`padding: 4px 0 6px` shorthand on the title row that zeroed out
the padding-left from my earlier nudge. Added the expanded-card
selector to the padding-left:4px rule so the title now lines up
with the meta line in both list and expanded states.
 2026-06-12 21:50:50 +09:00
pewdiepie-archdaemon 27c92caee8 Email library: nudge card subject + date line 4px right 2026-06-12 21:41:28 +09:00
pewdiepie-archdaemon 85966881d3 Email reader: lift action cluster 2px more (-7px → -9px margin-top) 2026-06-12 21:30:38 +09:00
pewdiepie-archdaemon dc170b1f58 Email reader: From row no longer wraps label onto its own line 
Was using flex-wrap:wrap on the From row, which let the chip span
flip onto a new row below From: when the available width briefly
dropped — then snap back as the chip span's overflow-scroll kicked
in. Switching to flex-wrap:nowrap keeps the label glued to the
chip; the chip span shrinks/scrolls horizontally instead.
 2026-06-12 10:26:48 +09:00
pewdiepie-archdaemon 37269fd96a Email reader docked: always show To/Cc, hide chevron toggle 
In docked mode the header already reserves vertical space for the
absolute action cluster, so the To/Cc details fit without any
height tradeoff — force [hidden] open and hide the chevron toggle
so the recipients are always visible there.
 2026-06-12 10:25:42 +09:00
pewdiepie-archdaemon e832133e47 Email reader docked: +2px more between From/To and To/Cc (now 4px each) 2026-06-12 10:24:25 +09:00
pewdiepie-archdaemon 51a41c0c30 Email reader docked: uniform 2px spacing between From / To / Cc 
Was From→To = 0 (meta gap 2 + details margin-top 0) while To→Cc
was 6 (details gap). Set details gap to 2 in docked too so all
three meta rows have the same vertical distance. Dropped the
per-row margin-top:4 docked override since spacing now comes
entirely from gaps.
 2026-06-12 10:20:11 +09:00
pewdiepie-archdaemon 8b8ec7fb1d Email reader: To/Cc details down 2px more (margin-top 2 → 4px) 2026-06-12 10:19:03 +09:00
pewdiepie-archdaemon 8f4747b1ff Email reader: nudge To/Cc details down 2px (margin-top 0 → 2px) 2026-06-12 10:18:09 +09:00
pewdiepie-archdaemon be7b3d796c Email reader: drop 2-row wrap breakpoint from 600px to 450px 2026-06-12 08:09:58 +09:00
pewdiepie-archdaemon 760c8ef72c Email reader docked: stretch meta so icons land right edge 
Docked header is flex-direction:column, and the base
align-items:flex-start was sizing the meta to its chip width and
parking it at the left — the absolute cluster's right:0 then
landed at the meta's right edge in the middle of the pane.
align-items:stretch makes meta fill the header width so right:0
hits the actual right edge.
 2026-06-12 08:02:53 +09:00
pewdiepie-archdaemon 3c4fb62d3a Email reader: docked uses same icon layout as undocked 
Dropped the docked-specific overrides (cluster flowing below meta,
padding-right:0, header min-height:0). The same container-query
rules drive both: cluster floats top-right and wraps to 2 rows
when the reader width crosses 600px, snaps to overlay below 380px.
Docked pane width is just another container width.
 2026-06-12 07:58:17 +09:00
pewdiepie-archdaemon cc86c3dd04 Email reader: +4px breathing room under wrapped 2-row cluster (92→96px) 2026-06-12 07:52:31 +09:00
pewdiepie-archdaemon 32898a68eb Email reader: header grows on wrap + no slide-down at overlay break 
1. Moved the min-height from .email-reader-header to .email-reader-meta
   (92px) inside the <600 container query. Targeting the container
   itself in its own @container rule was flaky; using a descendant
   that affects the parent's intrinsic height works reliably.
2. Dropped the margin-top:0 reset on the cluster in the <380 overlay
   rule — that was clearing the base -7px lift and sliding the
   cluster ~7px downward at the breakpoint. Now both states use the
   same -7px lift so the visual position is stable across the
   transition.
 2026-06-12 07:50:48 +09:00
pewdiepie-archdaemon 55e438d18c Email reader: prune competing rules from grid-era refactor 
Dropped the @media(769px) from-row min-height + align-items:center
and the strong > top:-2px nudge — leftovers from the grid layout
that were forcing extra height and label offsets the block-flow
meta doesn't need.

Consolidated docked overrides into a single flat block (no @media
wrapper) and merged the two .email-reader-meta declarations into
one. Same visual result, much less competing CSS to debug.
 2026-06-12 07:50:02 +09:00
pewdiepie-archdaemon a653f74cab Email reader: grow header min-height to fit wrapped 2-row cluster 
When the cluster wraps to 2 rows (44 + 4 gap + 44 = 92px tall), it
was peeking out below the header bottom because min-height stayed
at 60px (only ~44px of cluster room). Bumped min-height to 108px
inside the same <600 container query so the wrapped cluster sits
fully inside the header with 8px breathing room top + bottom.
 2026-06-12 07:42:31 +09:00
pewdiepie-archdaemon 4913a1363b Email reader: block-flow meta with absolute cluster — no more jump 
Replaced the grid layout (which made From row height depend on
cluster height, causing To/Cc to shoot up or down at the wrap
breakpoint) with a plain block stack:
- meta = position:relative block
- From row + details = natural block flow with padding-right
  reserving space for the absolute cluster on the right
- cluster = position:absolute top-right, width changes per
  container query (308px wide / 158px narrow / 180px overlay)
- padding-right tightens from 320px → 170px → 0 as the cluster
  shrinks and finally goes overlay
- details margin-top dropped from -10px to 0 since there's no
  grid row gap to compensate for

To/Cc now hugs From with no jumps when the cluster wraps or
overlays.
 2026-06-11 22:54:29 +09:00
pewdiepie-archdaemon 6edcc07c1b Email reader: lock From-row height when details expanded to kill jump 
Removed the medium-mode -12px details margin compensation — it
under/over-shot depending on grid row sizing. Replaced with a
:has() rule: when the user expands To/Cc, the From row gets
min-height 92px (matching the cluster's 2-row max height). Row 1
becomes the same size whether the cluster is 1 row (wide) or 2
rows (narrow), so resizing across the 600px wrap breakpoint no
longer makes To/Cc shoot up 4px.
 2026-06-11 22:47:53 +09:00
pewdiepie-archdaemon 7369c7c642 Email reader: extra 2px details lift in wrapped-cluster mode (no jump) 2026-06-11 22:43:23 +09:00
pewdiepie-archdaemon 7db4e8df4a Email reader: pull To/Cc details 2px tighter under From (-8px → -10px) 2026-06-11 22:41:02 +09:00
pewdiepie-archdaemon 5d5cfc07d7 Email reader: pull To/Cc details up 2px so they don't jump at overlay break 2026-06-11 22:38:48 +09:00
pewdiepie-archdaemon d592b1e6af Email reader: reserve row-1 height when cluster goes absolute 
When the cluster snaps to absolute overlay at <380px, it stops
contributing to grid row sizing — row 1 was collapsing to the From
row's natural height, which made the To/Cc details slide upward and
left the floating cluster visually misaligned against them. Setting
min-height:88px on the From row inside the same container query
holds row 1 at the cluster's two-row height so nothing jumps.
 2026-06-11 22:36:43 +09:00
pewdiepie-archdaemon b5449ea3f9 Email reader: 6px slack on cluster width to enforce 2-row max 
Was fanning out to 3 rows because the 152px max-width (3 icons +
2 gaps exact) had no slack — subpixel rounding could push the
third icon over and trigger another wrap. Bumped to 158px in the
in-grid mode (600px breakpoint) and 180px in the absolute-overlay
mode (380px breakpoint, where the 22px padding-left from the
gradient fade was also eating into the 3-icon row width).
 2026-06-11 22:32:00 +09:00
pewdiepie-archdaemon 73dbf3cde7 Email reader: lock cluster to 158px wide + right-edge anchor 
Was wrapping into 4+ rows at narrow widths because the cluster's
grid column could shrink below the 3-icon cap. Set both min-width
and max-width to the 3-icon row width and justify-self:end on the
cluster so the icons stay glued to the right edge instead of
sliding toward the middle when the cluster is wider than its
content.
 2026-06-11 22:28:48 +09:00
pewdiepie-archdaemon debd2cd386 Email reader: wire up emailreader container so wrap caps fire 
The 600px / 380px breakpoints were @container docpane queries but
the email reader isn't inside a docpane container — they never
fired and the cluster wrapped to 3+ rows at narrow widths. Added
container-type:inline-size + container-name:emailreader on
.email-reader-header and switched the queries to that container,
so the 2-row cap now actually applies.
 2026-06-11 22:25:09 +09:00
pewdiepie-archdaemon d95abaff1b Email reader: cap action cluster at 2 rows then overlay with shadow 
Three-step shrink:
1. > 600px pane: cluster sits in col 2 as 1 row of 6
2. 380-600px pane: cluster capped at 3-icon width so wrapping
   stops at 3 + 3 (max 2 rows) — chips share width with the 2-row
   cluster instead of multiplying into 3+ rows
3. < 380px pane: cluster snaps to absolute overlay with left-edge
   box-shadow, still capped at 3-icon width so it's the same 2-row
   shape but floating above the truncated chips
 2026-06-11 22:21:04 +09:00
pewdiepie-archdaemon 13086c3662 Email reader: 6-in-1-row default, wrap to 3+3 only when chip touches 
Grid tracks now:
- col 1: minmax(60px, 250px) — chip natural width capped at 250px,
  with the 60px (4 char) floor enforced on From / To / Cc alike
- col 2: minmax(48px, 1fr) — takes the rest, shrinks first when
  the pane narrows

Removed the hard max-width on the action cluster so on wide panes
it stays as one row of 6. Once col 2 shrinks below the 1-row width,
flex-wrap kicks in and the icons re-stack to 3+3. Chips only start
to shrink past that point.
 2026-06-11 22:18:37 +09:00
pewdiepie-archdaemon 5719e4db5f Email reader: lock cluster to 3+3 layout, shadow overlay at <380px 
- Action cluster's max-width is calc(48*3 + 4*2) so the 6 icons
  always lay out as 3 top / 3 bottom by default.
- When the pane narrows the chips in col 1 shrink first (with 60px
  min so 4 chars + ellipsis stay visible).
- At <380px the cluster snaps to absolute overlay with a left-edge
  box-shadow so it reads as floating above the truncated chip.
 2026-06-11 22:15:20 +09:00
pewdiepie-archdaemon 9ac3f40955 Email reader: icons wrap before chips shrink + 60px min chip width 
Two-step shrink behavior:
1. As the pane narrows, the action cluster (max-width:50% of meta)
   wraps to a 2-row icon stack first
2. Then the recipient chip span starts overflow-scrolling, but
   keeps a 60px min-width (~4 chars) so the first chars of the
   sender/recipient name stay visible
 2026-06-11 22:14:25 +09:00
pewdiepie-archdaemon 3a5c58da75 Email reader: To/Cc rows constrained to col 1 + cluster spans rows 
Previously only the From row affected the action cluster's column
width — To/Cc detail rows spanned both columns and ignored the
cluster. Now:
- meta-details lives in col 1 only so the To/Cc chips shrink
  together with the From chip when the pane narrows
- action cluster spans rows 1 and 2 so its width is set by the
  widest col-1 content; a long To/Cc list triggers the wrap to a
  2-row icon stack just like a long From sender does
 2026-06-11 22:11:47 +09:00
pewdiepie-archdaemon 7cf3402ef4 Email reader: grid layout so action cluster wraps before overlaying 
Meta switched to CSS grid in undocked mode:
- row 1, col 1: From row (label + chip + chevron)
- row 1, col 2: action cluster
- row 2, span: To/Cc details

The cluster shrinks alongside the chip and flex-wraps into a 2-row
icon stack before crowding the chip. At very narrow pane widths
(< 380px via @container docpane) it snaps back to absolute overlay
so From: still fits.

Docked mode overrides meta back to flex column so the cluster
flows naturally last — under From, and under To/Cc when expanded.
 2026-06-11 22:08:51 +09:00
pewdiepie-archdaemon 6066d0af02 Email reader: solid bg + gradient fade on action cluster overlay 
Was rendering as a transparent ghost — From chip / sender text bled
through the gaps between icons. Added a left-fading gradient
backed by var(--bg) so the cluster reads as an opaque overlay
while chips poking out from underneath blend smoothly into its
left edge.
 2026-06-11 22:06:58 +09:00
pewdiepie-archdaemon 7e029db44a Email reader: don't search-pivot from From/To/Cc chips + accent search icon 
- Window-level recipient-chip click handler now bails if the chip
  is inside .email-reader-meta — the per-reader handler still
  toggles the expanded-address view on click.
- The from-sender (magnifying glass) search button SVG is now
  tinted with var(--accent-primary) so it stands out as a deliberate
  search action against the neutral Reply / Forward / etc icons.
 2026-06-11 22:05:40 +09:00
pewdiepie-archdaemon f569b9394e Email reader docked: action cluster drops below To/Cc when expanded 
Moved the action cluster out of the From row to a sibling of meta
inside .email-reader-meta. Undocked: cluster is absolute-positioned
top-right of the header so it overlays the From line as before.
Docked: cluster is in-flow at the bottom of the meta column, so it
sits below the From row when collapsed and below the To/Cc rows
when the user expands the recipient details via the chevron.
 2026-06-11 22:04:34 +09:00
pewdiepie-archdaemon fce9942ae0 Chat: fix mode-tag breakage — toggleState wasn't in scope at those sites 
The previous commit read toggleState.mode before it was declared
(send-time site near line 632) and outside its closure (assistant
finalize site near line 3426). Both threw ReferenceError / TDZ on
first send, which crashed the chat send + render pipeline.

Read fresh via Storage.loadToggleState() at each site, defaulting to
'chat' on any error. Mode-tag rendering otherwise unchanged.
 2026-06-11 22:00:22 +09:00
pewdiepie-archdaemon 93ae65f99f Email reader undocked: wrap action cluster to 2 rows before overlay 
Cluster is now in-flow with margin-left:auto and flex-wrap:wrap so
when the chip text grows long enough to crowd it, the buttons split
to a second row of icons before they have to cover the chip. The
absolute-overlay behavior kicks back in at very narrow pane widths
(<380px via @container docpane) so From: still fits on one row when
the pane is truly cramped.
 2026-06-11 21:59:45 +09:00
pewdiepie-archdaemon f8d3890e6a Email reader undocked: nudge action cluster 1px down (-8px → -7px) 2026-06-11 21:58:09 +09:00
pewdiepie-archdaemon 85a11ad416 Email reader undocked: lift action cluster 2px more (-6px → -8px) 2026-06-11 21:56:20 +09:00
pewdiepie-archdaemon 2be0c5c892 Email reader docked: drop whole From row 4px + right-align icons 
Pulled the From row's negative margin from -8 to -4 so the whole
row (From: label AND chip) sits 4px lower together. Action cluster
below now justifies flex-end so the icons sit at the right edge
of the row instead of left-aligned.
 2026-06-11 20:44:47 +09:00
pewdiepie-archdaemon e0af7bd8a0 Chat: show Chat/Agent tag next to message timestamp 
Sometimes the user lands in chat mode without realizing — surface the
mode the message went out on as a small uppercase pill right after the
timestamp in the role header.

- roleTimestamp(when, mode) gains an optional mode arg. Agent renders
  in accent; Chat renders in muted/neutral. Other values render
  nothing (back-compat for older history without the field).
- The three roleTimestamp call sites pass metadata?.mode through.
- chat.js writes mode into the user-message metadata at send time and
  into the assistant metadata when the active-stream render lands,
  reading toggleState.mode so research/agent overrides upstream still
  flow through correctly.

Historical messages from before this change just don't show the pill —
graceful fallback, no migration needed.
 2026-06-11 20:44:18 +09:00
pewdiepie-archdaemon 1d1678214a Email reader: nudge undocked action cluster down 2px (-8px → -6px) 2026-06-11 20:43:41 +09:00
pewdiepie-archdaemon 06899c669c Email reader: lift undocked action cluster another 4px (-4px → -8px) 2026-06-11 20:41:00 +09:00
pewdiepie-archdaemon 05f05dd372 Email reader: shift From: label down 4px in docked mode 2026-06-11 20:40:33 +09:00
pewdiepie-archdaemon a195f4f194 Email reader: docked mode flows action cluster UNDER From row 
When the modal is docked there's no room to overlay the actions on
the From line. Now:
- From row gets flex-wrap so the action cluster drops to its own
  row below the From label + chevron
- Action cluster goes position:static, flex-basis:100%, no gradient
  fade, no padding-left, left-aligned
- Whole From row pulled up 8px to claim back vertical space
- Header min-height drops back to 0 since buttons no longer
  overlap

Also bumped the gap from From to To/Cc details by 2px (-8 → -6).
 2026-06-11 20:39:22 +09:00
pewdiepie-archdaemon 28caa40e68 Email reader: pull From label + actions up 2px more in docked mode 2026-06-11 20:36:24 +09:00
pewdiepie-archdaemon 6c1ce446f5 Email reader: lift action cluster 4px and From: label 2px on desktop 2026-06-11 20:34:45 +09:00
pewdiepie-archdaemon 729494a59b Email library: match magnifier color/opacity to other search fields 
opacity 0.55 → 0.45 and explicit color:var(--fg), matching the
.cal-search-icon treatment so the email chip-bar magnifier reads at
the same muted intensity as the calendar search field.
 2026-06-11 20:33:16 +09:00
pewdiepie-archdaemon df69bced42 Email reader: taller header to fit absolute-positioned action cluster 
Bumped header min-height to 60px and padding-top to 8px so the
44px-tall action buttons (absolutely positioned inside the From
row) have room without overflowing the header. From row gets
min-height:44px on desktop so the buttons fit cleanly inside it.
Dropped the now-redundant negative margin nudges on the From row
and the strong label.
 2026-06-11 20:33:02 +09:00
pewdiepie-archdaemon 12c8f9637f Email reader: search input up 1px, AI reply menu pared to Fast/Full 
Search input gets position:relative;top:-1px so the placeholder text
sits 1px higher inside the chip bar.

AI reply choice popover: drop the '...' kebab and the 'Draft with
note' textarea row entirely. Replace the concentric-circle Full icon
with our standard accent dot (filled 6px circle in viewBox 24).
 2026-06-11 20:31:04 +09:00
pewdiepie-archdaemon 7fe8a70032 Email reader: actions overlay chip instead of wrapping below when narrow 
Pinned .email-reader-actions-inline to absolute top:0 right:0 of the
From row with a gradient fade. When the window narrows the cluster
stays on the From line and the recipient-chips span scrolls under
it, so users can swipe/drag to reveal recipients tucked behind the
buttons instead of seeing From: jump above the action row.
 2026-06-11 20:29:24 +09:00
pewdiepie-archdaemon 2e8e097683 Revert "Email reader: AI reply becomes a split button (main + caret)" 
This reverts commit 86965950ac.
 2026-06-11 20:28:42 +09:00
pewdiepie-archdaemon 24dfd04964 Email reader: lift From: label 4px above the chip on desktop 2026-06-11 20:25:51 +09:00
pewdiepie-archdaemon 86965950ac Email reader: AI reply becomes a split button (main + caret) 
Main button: open cached AI draft if one exists, otherwise generate
a fast draft inline. No more intermediate Fast/Full/Note menu.

Caret on the side opens a focused popover with just a textarea +
Generate button — the user types instructions (e.g. 'thank them and
confirm Tuesday at 2', 'decline politely') and submitting fires the
full-mode generation with those instructions as the noteHint.

- _aiReplySplitButtonHtml(data) centralizes the new HTML so all three
  reader render sites use the same markup.
- _showAiReplyChoice rewritten — drops the Fast/Full toggle row plus
  the kebab + 'Draft with note' two-step. Ctrl/Cmd+Enter submits.
- _handleAiReplyButton routes based on which inner button was clicked
  (caret → popover, main → run-or-open).
- The three reader event registrations now listen on .ai-reply-split
  so both inner buttons feed the same handler.
 2026-06-11 20:24:19 +09:00
pewdiepie-archdaemon 79e9225c68 Email reader: From row up another 2px on desktop (-6px → -8px) 2026-06-11 20:24:17 +09:00
pewdiepie-archdaemon 1a3880347f Email reader: From row up another 2px on desktop (-4px → -6px) 2026-06-11 20:23:19 +09:00
pewdiepie-archdaemon 20968d5a87 Email reader: shift From row up 4px on desktop, +2px To/Cc gap 
- Desktop (>= 769px): From row gets margin-top -4px so the whole
  From + action cluster sits 4px higher in the header.
- Mobile @media block untouched.
- To/Cc gap bumped 4px → 6px for slight breathing room.
 2026-06-11 20:22:12 +09:00
pewdiepie-archdaemon a7200dd39b Email reader: nudge meta chevron 1px right (-4px → -3px margin) 2026-06-11 20:21:15 +09:00
pewdiepie-archdaemon d1f732bae1 Email reader: align From/To/Cc labels to a fixed 36px column 
Strong labels reserve min-width:36px so the chips after each label
start at the same x — From, To, Cc all line up. Killed the
docked/docpane grid-stack overrides that were splitting label and
chips onto separate rows, since chips already scroll horizontally
inside each row when there are too many.
 2026-06-11 20:19:02 +09:00
pewdiepie-archdaemon d849189b8c Email reader: tighten spacing in docked view + meta details 
- Docked: From row + action cluster nudged up 4px
- Chevron pulled 4px left so it sits tight to the From chip
- To/Cc detail block pulled up 8px to hug the From row
- 4px gap between To and Cc rows (was 2px)
 2026-06-11 20:18:06 +09:00
pewdiepie-archdaemon d30b2d11e6 Email reader: all actions on the From row, wrap when narrow 
Found the culprit — the docked-modal CSS forced .email-reader-meta-row
into a single-column grid, which collapsed the From row into a
vertical stack and pushed the action buttons below it.

Fix:
- Merged the primary + secondary action rows into one flat
  .email-reader-actions-inline cluster inside the From row
- Made the cluster flex-wrap so it stays inline when undocked and
  wraps below the chip when truly cramped (docked, narrow tab)
- Excluded .email-reader-meta-from from the docked-modal and
  narrow-docpane grid-stack rules — those overrides now only
  apply to the To/Cc detail rows
 2026-06-11 20:07:35 +09:00
pewdiepie-archdaemon 156009f9ad Email library: magnifying glass inside the chip-bar search field 
Absolutely-positioned 13px search SVG at the left edge of the chip bar
(same circle+line glyph used elsewhere). Bar padding-left bumped 8→26
to leave room. pointer-events:none on the icon so clicks still land
on the input, opacity 0.55 to match other muted prefix icons.
 2026-06-11 20:06:22 +09:00
pewdiepie-archdaemon 0aa8d17d6c Email: bookmark icon everywhere for favorites; subject matches in suggestions 
Star → bookmark banner SVG also in the card title row (em.is_flagged
glyph) and the inbox toolbar's _starIcon / _starFilledIcon, so every
favorites affordance matches the chats sidebar bookmark.

Search dropdown gains a third suggestion kind:
- kind: 'email' rows surface emails from the snapshot whose subject or
  sender name match the typed term (top 4, scored by startsWith vs
  substring). Render row carries a small envelope glyph + bolded
  subject + 'from name' on the right.
- Picking one closes the dropdown and expands that exact card via
  _toggleCardPreview, scrolling it into view.
 2026-06-11 19:46:45 +09:00
pewdiepie-archdaemon 39331fafb5 Email reader: primary action row literally inside the From row 
Restructured the DOM so the Reply / Reply-all / Forward row lives
INSIDE the email-reader-meta-from div (after the chips span), and
the Summary / AI / More row sits directly below as a sibling of
From inside the meta. Killed the outer email-reader-actions
wrapper that kept letting the buttons drift out of position.

CSS now pushes the primary row right via margin-left:auto on the
From row and right-aligns the secondary row below it.
 2026-06-11 19:46:06 +09:00
pewdiepie-archdaemon 05f87b0f50 Email reader: Reply group on From row, Summary/AI/More below 
Reorganized the action cluster into two visible rows so each fits
the available width:
- Top row (on the From line): Reply / Reply-all / Forward
- Bottom row (under it):      Summary / AI reply / More

Action cluster goes back to flex-direction:column, the row
wrappers are flex rows again (no more display:contents flatten).
 2026-06-11 19:39:13 +09:00
pewdiepie-archdaemon 9f1435f761 Email library: swap Favorites icon star→bookmark banner (matches chat .session-fav) 2026-06-11 19:39:12 +09:00
pewdiepie-archdaemon 772ddf4a86 Email library: filter pills render as icon-only chips 
After picking a filter from the dropdown the pill was 'icon + Unread'.
Drop the text — the icon is the affordance — so the pill collapses to
just the glyph + ×. Hover surfaces the friendly label via the title
attribute. Contact + text pills still carry their text label.
 2026-06-11 19:37:56 +09:00
pewdiepie-archdaemon 432b41cede Email reader: top-align action cluster against From row 
align-items: flex-start on the header keeps the action cluster
locked to the From line when the user expands the To/Cc details
— previously it drifted to vertical center as the meta grew taller.
 2026-06-11 19:34:20 +09:00
pewdiepie-archdaemon e7466175ef Email library chip-bar: filter + tag suggestions with their icons 
Typing a filter keyword now surfaces the matching filter row in the
autocomplete (each with its existing dropdown icon). Picking one pins
a filter pill and drives the global filter state.

Keyword catalog (_LIB_FILTER_OPTIONS):
- has-attachments  ← 'attachment', 'attachments', 'has attachment', 'attach'
- unread           ← 'unread', 'new', 'unseen'
- favorites        ← 'favorite', 'starred', 'star', 'flagged'
- undone           ← 'undone', 'pending', 'todo'
- reminders        ← 'reminder', 'reminders'
- unanswered      ← 'unanswered', 'unreplied', 'no reply'
- pending_30d      ← 'pending 30d', 'pending', 'recent pending'
- stale_30d        ← 'stale', 'old', 'stale 30d'
- tag:urgent       ← 'urgent', 'critical'
- tag:reply-soon   ← 'reply soon', 'reply', 'follow up'
- tag:spam         ← 'spam', 'junk'
- tag:newsletter   ← 'newsletter', 'newsletters', 'subscriptions'
- tag:marketing    ← 'marketing', 'promo', 'promotional'

Filter pill behaviour:
- Only one filter pill is active at a time — adding a new one replaces
  any existing filter pill.
- _applyFilterPillSideEffect drives the existing #email-lib-filter
  select (or the #email-attach-btn toggle for has-attachments). The
  server-side list refetch follows for free via the existing 'change'
  handler.
- Removing the filter pill clears the side effect.

Pill render gains the filter icon as a leading glyph; the suggestion
row renders icon + label in the accent colour so it visually reads as
a filter, not a contact.
 2026-06-11 19:33:55 +09:00
pewdiepie-archdaemon 5bf7caecc9 Email reader (mobile): top-align meta with the two-row action cluster 
After the toolbar reshuffle the action block is now two stacked rows
(Summary/More above Reply/Forward/AI), making it taller than the meta
block. The mobile header rule was align-items:center, which then pulled
the From:/To: rows down into the vertical middle of the header — the
'From: is in the middle' symptom. Switch to flex-start so meta sticks
to the top edge where the user expects it.
 2026-06-11 19:29:41 +09:00
pewdiepie-archdaemon 4bf389ed09 Email reader: actions inline on the From row 
With the meta collapsed to a single visible From row + chevron,
there is room to put the action cluster on that same row as a
right-aligned sibling. Dropped the absolute positioning and
gradient-fade overlap — actions now flex-end via margin-left:auto
so From sits on the left and Reply / Reply-all / Forward / AI /
Summary / More all sit on the right of the same row.

Also moved the chevron inside the recipient-chips span so it sits
adjacent to the sender chip instead of wrapping onto a second line.
 2026-06-11 19:23:34 +09:00
pewdiepie-archdaemon 90acad0d4b Email library chip-bar: AND across pills, plain Enter commits text, pill × up 4px 
1. Multiple pills now AND together — 'alice + bob' means both alice
   AND bob are somewhere on the email, not 'from alice OR from bob'.
   (some → every in the filter.)
2. Default autocomplete focus is now -1 (no row pre-selected) so plain
   Enter commits the input as a text pill — typing then Enter behaves
   like a normal search. ArrowDown / ArrowUp + Enter still picks a
   contact suggestion. Tab still autocompletes the most-relevant match
   regardless of arrow state.
3. Pill × button nudged up 4px so it sits on the visual centerline
   inside the 18px pill height.
 2026-06-11 19:21:37 +09:00
pewdiepie-archdaemon 6e6b860f04 Email reader: collapse To/Cc behind Gmail-style chevron 
Only the From row shows by default. When the email has To and/or
Cc recipients, a small chevron sits next to the From chip — click
it to inline-expand the To/Cc rows below (rotates 180deg open).

Trims the header to a single visible row in the common case,
leaving the action cluster plenty of vertical headroom to stay
on a single row.
 2026-06-11 19:19:12 +09:00
pewdiepie-archdaemon e4c7a3aad9 Email reader: keep From/To/Cc on separate rows, label tight to chips 
Reverted the single-row meta strip — misread the user's ask. Each
meta field gets its own row (From / To / Cc stacked), label sits
tight to the chips on the same line, recipient chips inside the
row still scroll horizontally so long lists slide under the
floating action cluster.
 2026-06-11 19:15:55 +09:00
pewdiepie-archdaemon ac4627b69d Email reader: collapse From/To/Cc into a single inline row 
Three stacked meta rows wasted vertical space — From, To, Cc now
share one horizontal strip with each label tight to its chips. The
strip itself scrolls horizontally so the action cluster (still
floating top-right) can cover the right edge and the user can drag
to reveal recipients hidden underneath.

This also gives the actions a single shared row, since the meta
no longer dictates a multi-row header height.
 2026-06-11 19:12:06 +09:00
pewdiepie-archdaemon 99660e1c6d Email library chip-bar: smaller pills, persist across refresh, Esc + sender click 
Four fixes from the first round of usage:

1. Pill height was larger than the chip-bar's row — shrink to a fixed
   18px-tall pill (line-height + height pinned) so it sits inside
   the input row.

2. List refresh wiped pill state — when _loadEmails replaces
   state._libEmails (refresh, folder switch, etc.), refresh the
   snapshot to the new list and re-apply the pill filter so pills
   persist instead of resetting to 'show all emails'.

3. Click-to-add only worked inside the open email reader. Extend the
   capture-phase handler to ALSO catch clicks on .email-meta-sender
   inside the library grid — the list card's sender name is the most
   natural place to want to pivot from.

4. Esc inside the chip-input didn't close the modal. New behaviour:
   if the autocomplete dropdown is open, Esc closes only the dropdown
   (and swallows the event); otherwise Esc blurs the input and bubbles
   so the existing modal Esc handler can close the library.

Also wires data-email + data-name on .email-meta-sender so the click
handler has reliable targeting.
 2026-06-11 19:11:07 +09:00
pewdiepie-archdaemon f91f37ef70 Email reader: flatten action rows with display:contents 
The primary/secondary row wrappers were still creating nested flex
containers — even with parent flex-direction:row the two row divs
sized to content and could stack visually. Switching the wrappers
to display:contents collapses them entirely so all 6 buttons
become direct flex children of .email-reader-actions and lay out
on a single row guaranteed.
 2026-06-11 19:08:22 +09:00
pewdiepie-archdaemon 682ec11003 Email library: gallery-style chip-input search with contact autocomplete 
Replace the single text-input + IMAP search round-trip with a deterministic
local chip-bar filter modelled on the gallery's tag pills.

What lives in the bar
- Each filter is a pill: { type: 'contact', name, email } or
  { type: 'text', text }.
- Click anywhere in the bar lands the cursor in the input field.
- Typing populates a dropdown of matching contacts + recently-seen senders
  (cached per modal open via _buildSuggestionSource).
- Tab / Enter on a highlighted suggestion → adds a contact pill.
- Enter on free text with no suggestion match → adds a text pill.
- Backspace on empty input → pops the last pill.
- × on a pill removes that one.
- Arrow keys navigate the suggestion list.

Filtering
- _applyPillFilter snapshots the loaded list once, then for every render
  shows emails where ANY pill matches:
    contact pill — from_address equals OR to/cc contains the pill's email
    text pill    — broad substring match across subject/from/snippet

Click-to-add
- Capture-phase click handler on .recipient-chip inside the email reader
  drops the person into the library as a contact pill (and reopens the
  library window if it was closed/minimized).

Removed the debounced /api/email/search IMAP call and its 'Loading emails'
side effect. The dropped server search was the source of the 'type
jonathan, get stuck on Loading' bug.
 2026-06-11 19:02:05 +09:00
pewdiepie-archdaemon 41c0ffbb52 Email reader: collapse action cluster to a single row 
Reply / Reply all / Forward / AI / Summary / More now flow inline
on one row instead of being split into a primary (Summary+More) and
secondary (Reply group) stack. Mobile + docked overrides also
flipped from column to row.
 2026-06-11 18:59:50 +09:00
pewdiepie-archdaemon be430fc4a4 Email reader: actions float top-right over scroll-able recipient row 
From/To/Cc back on the left, action cluster (Reply / Reply-all /
Forward / AI / Summary / More) absolute-positioned top-right with a
gradient fade so chips that overflow slide cleanly underneath. The
recipient-chips lists no longer wrap — they scroll horizontally,
matching the account-chip strip pattern, so users can drag/swipe
to reveal recipients hidden under the action cluster.

Mobile (@media max-width:768px) gets the same row+absolute layout
instead of the previous column with actions on top. The narrow
container query (docpane max-width:460px) still falls back to
in-flow column so it doesn't overlap on very narrow panes.
 2026-06-11 18:55:31 +09:00
pewdiepie-archdaemon 15f2b106ab Email reader: move action toolbar to the TOP, meta below 
Was: from/to/cc/date meta on the left, action cluster (Reply / Reply
all / Forward / AI reply / Summary / More) pinned to the right of
the header. Now: actions stretch across the top in their two existing
sub-rows, the from/to/cc meta sits below.

Pure CSS — no template restructure. The .email-reader-header flexbox
flips to flex-direction:column, .email-reader-actions gets order:-1
to render first, and the existing flex-end aligned action-row rules
swap to flex-start so buttons read left-to-right across the top
toolbar. Mobile media query overrides bend the same way so the
layout is consistent across breakpoints.
 2026-06-11 18:48:34 +09:00
pewdiepie-archdaemon e310336a42 AI Reply note: hide 'Draft with note' button until the textarea has text 2026-06-11 18:46:16 +09:00
pewdiepie-archdaemon e1585aa4aa AI Reply menu: '...' kebab opens a note input to steer the draft 
The Fast/Full popover now has a kebab (three-dot) button alongside the
two preset choices. Clicking it expands a textarea below with a
'Draft with note' send button. The textarea is for the user to tell
the AI how to reply ('confirm Tuesday at 2', 'decline politely', 'say
we'll need an extra week') instead of accepting a generic draft.

Plumbing:
- emailLibrary.js: kebab button + note panel inside .email-ai-reply-choice
  menu. Submitting calls _runAiReplyFromButton with mode='ai-reply-full'
  and a noteHint string.
- _runAiReplyFromButton signature gains noteHint; passes it through
  state._onEmailClick as opts.noteHint.
- emailInbox.js consumer: forwards opts.noteHint into _openEmail's new
  5th arg, which puts it in the /api/email/ai-reply POST body as
  user_hint.
- routes/email_routes.py /ai-reply: reads user_hint, appends a
  'User's instructions for THIS reply' section to the user message
  (priority over default tone/length). Also skips the per-message
  AI-reply cache when a hint is set — the cached generic draft would
  silently override the instructions otherwise.
 2026-06-11 18:41:11 +09:00
pewdiepie-archdaemon 6a392542f3 Email reader: two-row action layout — Summary+More above, Reply/Forward/AI reply below 
Restructure the action cluster so it stays as two visible rows inside
.email-reader-actions instead of flattening via display:contents:
- Top row: Summary, More
- Bottom row: Reply, Reply all (conditional), Forward, AI reply
Dropped the Search button — wasn't part of the requested layout.

CSS: .email-reader-actions becomes flex column with both rows
right-aligned; .email-reader-actions-row becomes a real flex row
(no more display:contents flattening) so each row stays on its own
line. Whole block continues to sit beside the From/To meta inside
.email-reader-header.
 2026-06-11 18:40:16 +09:00
pewdiepie-archdaemon 7b3bc598f4 AI Reply Fast/Full icons: paint with var(--accent, var(--red)) 2026-06-11 18:36:27 +09:00
pewdiepie-archdaemon 239cc02422 AI Reply menu: SVG icons for Fast (lightning) and Full (concentric circles) 2026-06-11 18:29:35 +09:00
pewdiepie-archdaemon 44f12f266e Email library: await _loadAccounts before loading emails 
After dropping the 'Default' chip, _loadAccounts started setting
state._libAccountId asynchronously while _loadEmails fired in parallel
with the still-null id. The list request was going out with no
account_id (so the server defaulted) while subsequent per-email reads
used the explicit id set after _loadAccounts resolved — back to the
same desync the chip-removal was meant to fix.

Sequence them: await _loadAccounts first, then kick off the folders /
reminders / emails fetches. The list always carries the right
account_id from the very first call.
 2026-06-11 17:15:49 +09:00
pewdiepie-archdaemon 8e8ce8ddd6 Email reader: 'open in new tab' windows don't auto-dock left on Reply 
Replying from an email opened in a new tab was dragging that window to
the left-sidebar dock — same treatment as the main email library, even
though the user had explicitly opted to pop it into its own floating
viewer. Annoying when the viewer is mid-screen and Reply yanks it.

Add an early bail in _snapEmailModalToLeftSidebar for modals whose id
starts with 'email-view-' (the 'open in new tab' reader). Compose still
opens; the floating viewer just stays where it is, on top of the
library. User can move/close it themselves.
 2026-06-11 17:13:15 +09:00
pewdiepie-archdaemon f2ccf8b21f Email library: drop the 'Default' chip — pick an explicit account always 
Bug: clicking the dot to change the server-side default account while
viewing 'Default' left a desynced state — the email list still showed
the OLD default's cached UIDs, but the server's default now pointed
at a different account. Opening any email used the visible UID +
account_id='' on the read endpoint, which resolved against the NEW
default account → wrong email content (or older mail entirely).

Fix: remove the 'Default' chip. _loadAccounts now auto-selects the
is_default account (or the first one) into state._libAccountId so the
list view + every per-email request always carries an explicit
account_id and can't desync from set-default.

The dot button still lives on each account chip for changing which
account the server treats as the default — but it no longer affects
which account the list is currently displaying.
 2026-06-11 17:11:55 +09:00
pewdiepie-archdaemon 5d9d21f227 Email filter Unread: use the incognito eye SVG (eye with X) instead of the ringed dot 2026-06-11 17:08:46 +09:00
pewdiepie-archdaemon 537f492762 Email reader: pin More to far right + allow actions to wrap beside meta 
- .email-reader-actions flex-wrap nowrap → wrap so when the cluster
  exceeds the room next to a tall multi-recipient meta block, the
  buttons wrap within the actions area instead of pushing the whole
  block onto its own row below From/To.
- New rule: .email-reader-more-wrap gets order:99 so the More kebab
  sits at the far right of the flattened flex row instead of in the
  middle (its source order put it ahead of the secondary row's AI
  Reply / Summary buttons after display:contents flattening).
 2026-06-11 17:06:26 +09:00
pewdiepie-archdaemon 6a0a7622fd Email filter picker: nudge up 2px on desktop (3px → 1px) 2026-06-11 17:05:47 +09:00
pewdiepie-archdaemon 719867a819 Email library: nudge .email-filter-btn up 4px 2026-06-11 17:01:23 +09:00
pewdiepie-archdaemon 9dfea188bf Email filter: custom dropdown with SVG icons for each option 
The All/Unread/Favorites/etc selector was a native <select>, which
can't render SVG inside <option>. Replace it with a custom picker
that:

- Keeps the existing <select id="email-lib-filter"> as the value
  store (hidden via display:none). All existing 'change' listeners
  keep working — the picker just dispatches a change event after
  updating the select's value.
- Renders a styled button + drop-out menu built from the select's
  options (preserves optgroup labels like 'Tags').
- Each option carries an SVG icon: lines for All, ringed dot for
  Unread, star for Favorites, empty checkbox for Undone, bell for
  Reminders, reply arrow for Unanswered/Reply-soon, clock for
  Pending, calendar-x for Stale, exclamation-triangle for Urgent,
  ban for Spam, newsletter and megaphone for the marketing tags.
- Icons use var(--accent) so they pick up the user's theme color.
- Click outside / Esc closes the menu (Esc handler is capture-phase
  + stopPropagation so it doesn't bubble to the modal-close listener
  and shut the whole email window).

CSS scoped under .email-filter-picker.
 2026-06-11 12:53:39 +09:00
pewdiepie-archdaemon df908b4c11 Email reader: regroup More menu + reshuffle toolbar rows 
More menu reorganization:
- Group 1: Open in new tab, Remind to reply
- Group 2 (state): Mark as Unread/Read, Mark as Done/Not Done, Move to
  Archive, Save sender to contacts
- Group 3 (destructive, unchanged): Move to Spam, Move to Trash,
  Delete Permanently
- Renames: Done→'Mark as Done', Archive→'Move to Archive', Mark
  Read/Unread→'Mark as Read'/'Mark as Unread'.
- Mark Unread moves out of group 1 down into the state-change group
  alongside Done; Save sender to contacts moves down into the same
  state group.

Toolbar row reshuffle (applies to both the email-list card reader and
the email document view):
- Row 1 (primary): Reply, Reply all, Forward, Search, More — Forward
  no longer has to fight Search/More for space in the secondary row.
- Row 2 (secondary): AI reply, Summary — gets its own dedicated row.
 2026-06-11 12:50:47 +09:00
pewdiepie-archdaemon be126afcf8 Email accounts strip: bigger 18x18 hit target around the small default-dot 
The 6px dot was easy to miss on touch / small-cursor setups. Replace
padding-only sizing with explicit width:18px;height:18px on the
button, dot centered inside via justify-content. Anchor moved from
right:9 → right:6 so the visible dot stays where it was; the extra
clickable area extends inward from the chip edge.
 2026-06-11 12:44:02 +09:00
Muhammad Ikhwan Fathulloh 04ff417a10 Merge branch 'pewdiepie-archdaemon:dev' into dev 2026-06-11 10:32:17 +07:00
pewdiepie-archdaemon b2243efd3f Email accounts strip: nudge default-dot 1px up + 2px left 2026-06-11 12:18:02 +09:00
pewdiepie-archdaemon 79c04c71e9 Email accounts strip: shrink default-dot to 6px (matches sidebar notif dot) 2026-06-11 11:57:46 +09:00
pewdiepie-archdaemon ebd2332db4 Agent prompt builder: stop re-adding ALWAYS_AVAILABLE on top of filtered tools 
Found the reason yesterday's tool-retrieval drop wasn't taking effect:
in _build_agent_prompt, when relevant_tools was provided, it computed
  tool_names = set(ALWAYS_AVAILABLE) | set(relevant_tools)
which silently re-added every tool get_tools_for_query had just
deliberately discarded. So when a 'save this for <person>' query
dropped manage_memory from the retrieved set, the prompt builder put
it right back, and the model saw both tools again.

Trust the relevant_tools set. get_tools_for_query already starts from
ALWAYS_AVAILABLE — any discard there is intentional and should
propagate. Only force-include ask_user and update_plan here as belt-
and-suspenders since the agent loop relies on those for its own
control flow.

Other callers (task_scheduler) already union ALWAYS_AVAILABLE or
ASSISTANT_ALWAYS_AVAILABLE into relevant_tools before passing it in,
so they're unaffected.
 2026-06-11 09:49:20 +09:00
pewdiepie-archdaemon 070ec4c711 Email accounts strip: nudge default-dot 1px left + shrink 10→8px 2026-06-11 09:49:03 +09:00
pewdiepie-archdaemon 6fc79e90ac Settings/Contacts (CardDAV): show '(unchanged)' placeholder when password is saved 
GET /api/contacts/config masks the saved password as '***' (or ''
when none). Mirror that into the password input's placeholder so users
can see at a glance that a password is on file — matching the email
account form's '(unchanged)' pattern.
 2026-06-11 09:47:28 +09:00
pewdiepie-archdaemon f5ad59317c Tool retrieval: HARD drop manage_memory when query is a contact-save pattern 
Description-level steering wasn't enough — even with the explicit 'DO
NOT use for info about another person' in manage_memory's description,
models kept choosing memory over manage_contact. They can't if memory
isn't in the toolset.

New logic in ToolIndex.get_tools_for_query: detect three contact-save
patterns and discard manage_memory from the returned set (overriding
ALWAYS_AVAILABLE):

1. 'save [up to 3 words] for/to <name>' where <name> isn't a timing /
   pronoun stopword (later, tomorrow, me, you, future, etc.). Catches
   the canonical 'save this for X' and the wider 'save this address
   for X', 'save it for X'.
2. 'to/in/into (my) contacts' or 'address book'. Catches both 'add X
   to my contacts' and 'put this in my address book for X'.
3. Possessive: 'save (his/her/their) (address/phone/email/...)'.
   Stronger signal — also force-adds manage_contact to the set in
   case the keyword fallback missed it.

Verified: 8 positive contact patterns all drop memory, 10 false-
positive 'save X for later/tomorrow/me/the next thing' all keep it.
 2026-06-11 09:46:34 +09:00
pewdiepie-archdaemon 803df21fc2 Email accounts strip: nudge default-dot 2px left (right 4→6) 2026-06-11 09:44:03 +09:00
pewdiepie-archdaemon df47536b8d manage_memory descriptions: explicit deferral to manage_contact for person info 
Even with manage_contact in the retrieved tool set, models were still
defaulting to manage_memory when the user pasted an address + 'save for
<person>'. Both tools were in front of the model and it picked memory.

Tighten both descriptions to steer at decision-time:
- agent_loop.py manage_memory description: clarify scope is facts
  about the USER, with an explicit 'DO NOT use for info about another
  person' + a 'use manage_contact instead' line.
- tool_index.py manage_memory description: same in shorter form, so the
  embedded retrieval signal is consistent with the prompt-time
  description.
 2026-06-11 09:25:23 +09:00
pewdiepie-archdaemon 2049eb7713 Contacts UI: address + phone inputs, search filter, address-only adds 
The contacts manager in Settings was stuck at name+email inline only —
no address field, no phone input on add, no search to find anything in
a list of 100+ contacts.

UI:
- Add form gets phone and address inputs alongside name/email. The
  email-required gate becomes name-OR-email so address/phone-only
  entries are creatable.
- Edit form gets an address input, threaded into the PUT body.
- Search input above the list filters client-side by name / emails /
  phones / address (debounced 80ms). Count badge shows N/M when a
  filter is active.

Backend:
- /api/contacts/{uid} PUT now accepts address and routes it through
  _update_contact (which already supports it after the previous
  commit). Validation loosened: name OR email OR address.
- /api/contacts/add POST now accepts phone + address. Phone goes
  through an immediate _update_contact since _create_contact's
  signature only takes name+email+address.
 2026-06-11 09:23:14 +09:00
pewdiepie-archdaemon f42cee8512 Email accounts strip: bigger default-dot (10px) + 4px more chip padding 
8px ring read as a sliver next to the chip label. Bump to a 10x10 SVG
with stroke-width:3 for the hollow ring so it presents like the
sidebar notif dot at this size. Chip padding-right bumped 20→24 so
the larger glyph isn't crushed against the text.
 2026-06-11 09:18:34 +09:00
pewdiepie-archdaemon 8a00f954a9 Tool retrieval: catch 'add X to (my) contacts' / 'address book' phrasings 
The literal phrase 'add to contacts' missed when there was a name
between 'add' and 'to', e.g. 'add Pat to my contacts'. Anchor on the
tail with 'to my contacts', 'to contacts', 'to address book' so word
boundaries fire regardless of what sits in front.
 2026-06-11 09:18:30 +09:00
pewdiepie-archdaemon 6d1d626d87 Email accounts strip: swap default-star for a dot, nudge up 2px 
Replace the star polygon with a small 8px circle dot — filled +
accent-tinted on the default account, hollow + muted on others.
Vertical position bumped up 2px via top: calc(50% - 2px) so it
visually centers against the chip's text baseline instead of
geometric center.
 2026-06-11 09:17:04 +09:00
pewdiepie-archdaemon 8632072ce0 Contacts: postal-address support via vCard ADR, keep tool prompt minimal 
Closes the gap that pushed the agent into manage_memory when the user
pasted an address and said 'save this for X'. manage_contact now
accepts an optional address arg end-to-end:

- routes/contacts_routes.py:
  - _normalize_contact carries an 'address' field
  - _build_vcard emits ADR:;;<address>;;;; (street component of the
    RFC-6350 7-part ADR), only when address is non-empty
  - _parse_vcards reads ADR, joins non-empty components with ', '
  - _create_contact and _update_contact thread address through;
    update preserves existing address when caller passes empty
- src/tool_implementations.py do_manage_contact:
  - add accepts address; require at least name+address or email
    (was: email required) so address-only contacts are addable
  - update accepts address; require name OR emails OR address
- src/tool_schemas.py: schema gets a single 'address' string field
- src/tool_index.py + src/agent_loop.py: descriptions get one
  'address' arg mention and a 'use this for save-X-for-person /
  address pastes / phone-with-name' steering line. Net: a few
  bytes added, not a paragraph.

Also: removed a stray name from the schema's manage_contact example
strings ('save Jonathan's email…') — no real names in the codebase.
 2026-06-11 09:14:52 +09:00
pewdiepie-archdaemon c637b5057b Email accounts strip: rename 'All (default)' → 'Default', add star toggle 
- The 'All (default)' chip showed only the default account, so the
  label was misleading. Rename to just 'Default' to match behavior.
- Each user account chip gets a star button (filled if it IS the
  default, hollow otherwise). Clicking calls the existing
  POST /api/email/accounts/{id}/set-default and refreshes the strip.

Cross-account aggregation (a true 'All') is a separate bigger lift
that needs UID namespacing and merge/sort in _list_emails_sync;
flagged for follow-up rather than smuggled into this change.
 2026-06-11 09:12:37 +09:00
pewdiepie-archdaemon 153b788134 Tool retrieval: surface manage_contact for 'save X for <person>' patterns 
When the user dumps a postal address or phone number alongside a
person's name and says 'save this for X', the vector retriever was
missing manage_contact because its description only mentioned the
literal word 'contact'. The model defaulted to manage_memory (which is
in ALWAYS_AVAILABLE), so the saved fact ended up as un-named memory
that wouldn't surface on a later 'what's X's address?' search.

- Rewrite manage_contact's index description to anchor on the
  semantics: 'save info about another person', including postal/
  mailing address, ZIP, phone, etc. Now it embeds close to address-
  paste queries.
- Extend the keyword intent-map with 'save this for', 'save it for',
  'mailing address', 'postal code', 'their address', etc. — common
  ways users say 'this belongs to a contact' without the literal word
  'contact'.
 2026-06-11 08:56:42 +09:00
pewdiepie-archdaemon bc2d934b94 Agent email safety: stage drafts for user approval instead of auto-send 
Closes the auto-send hole that let earlier models invent signatures
(e.g. signing 'David' for a user named Felix) and SMTP them to real
recipients before the user could review.

New setting: agent_email_confirm (default True).

When on, the MCP send_email and reply_to_email tools no longer SMTP
directly — they write the composed email to scheduled_emails with a new
status 'agent_draft' (far-future send_at so the scheduled-send poller
ignores them) and return a {pending: true, pending_id, to, subject,
body, message: ...} payload. The model surfaces that to the user.

Backend endpoints to approve / cancel:
- GET    /api/email/pending          → list staged drafts for the owner
- POST   /api/email/pending/{id}/approve → flip status to 'pending' +
                                           backdate send_at so the
                                           existing scheduled-send
                                           poller delivers immediately
- DELETE /api/email/pending/{id}     → status = 'cancelled'

UI:
- Settings / AI Defaults gets a new 'Email Safety' card with the
  toggle, default on.
- Tool descriptions for send_email and reply_to_email now include the
  pending behavior + an explicit 'DO NOT invent a signature, do not
  type a person's name' guardrail.

Pass 2 (next): inline chat card with Send / Discard buttons so the user
doesn't have to type a confirmation reply. Today's prompt + the listing
endpoint give the model a clean path to surface drafts.
 2026-06-11 08:50:06 +09:00
pewdiepie-archdaemon 2b1e2e9e20 Email library: center the loading whirlpool over the full grid 
Old rule fixed the loading wrap at min-height:180px so the spinner
landed near the top of the email-list section. Switch to
position:absolute inset:0 over the grid (with #email-lib-grid set to
position:relative) so the whirlpool + 'Loading emails' label center
within the entire visible email area regardless of section height.
 2026-06-11 08:46:34 +09:00
pewdiepie-archdaemon b5b96980e3 Email bulk bar: nudge 'Marking…' label up 2px + 'All' checkbox up 2px 
- 'Marking done' / 'Marking read' / 'Marking unread' label was 2px low
  vs. the whirlpool spinner inside the Actions button. The existing
  loading-label CSS only scoped to #email-lib-bulk-delete; extend it
  to also cover #email-lib-bulk-actions and bump top from 0 to -2px.
- 'All' checkbox label was inline-styled top:2px so the box + text sat
  lower than the surrounding bulk-action items. Reset to top:0 to
  match memory + skills select-all rows.
 2026-06-11 08:41:59 +09:00
pewdiepie-archdaemon 127745d13b Email search: instant local-cache filter + stop blanking the grid 
Two pain points:
- IMAP server search is genuinely slow.
- The grid blanked to a whirlpool on every keystroke, so even fast
  searches felt dead because you couldn't see your own results.

Fix:
- _localSearchFilter runs synchronously on every keystroke, filtering
  the pre-search snapshot by subject / from-name / from-address /
  snippet so the grid responds immediately. Snapshot is taken on the
  first non-empty keystroke and restored when the input is cleared.
- _doSearch no longer renders the loading-whirlpool spinner into the
  grid. The local filter already shows useful results; surface
  'Searching…' in the stats badge to indicate the server search is in
  flight.
- When server results land, they replace the grid; if the user has
  already typed past them, the seq guard skips the stale render.
 2026-06-11 08:28:25 +09:00
pewdiepie-archdaemon 5ec1e12a50 Email bulk actions: loading state for every action + 6-way parallel fetches 
Before: only delete showed a spinner/disabled buttons. Picking Done on
92 selected emails fired off 184 sequential HTTP calls (mark-answered
+ mark-read) with zero UI feedback, so it looked like the click did
nothing for the ~20-30 seconds it took to grind through.

- All five bulk actions (delete / archive / done / read / unread) now
  swap the target button into a whirlpool+verb-ing state, dim siblings,
  and show 'N/M…' progress in the count label that ticks as each
  request resolves.
- Per-uid work runs in parallel with a hard cap of 6 in flight, so a
  90-email Done finishes in ~3 server round-trips of latency instead
  of 90, but we still don't open 90 simultaneous IMAP-backed connections.
 2026-06-11 07:41:36 +09:00
pewdiepie-archdaemon 7c1af0385a Email reader More menu: reorder + separators into three groups 
Group 1 — per-email view actions:
  Open in new tab → Mark Unread/Read → Remind to reply
Group 2 — non-destructive state changes:
  Save sender to contacts → Done/Not Done → Archive
Group 3 — destructive (own divider):
  Move to Spam → Move to Trash → Delete Permanently

Adds support for { separator: true } items in the actions array,
rendered as .dropdown-divider rows.
 2026-06-11 07:40:11 +09:00
pewdiepie-archdaemon dde2d25804 Email library bulk Done: animate-out + drop when filter='undone' 
Repro: filter Undone → Select All → uncheck a few → Actions → Done →
nothing visible happens. Reason: the bulk-Done branch only flipped
em.is_answered on the in-memory entries; the cards stayed in
state._libEmails so they kept rendering, but now with the done check
ticked. From the user's POV — still 'undone' filter, cards still
there — it looked like the action was a no-op.

When the filter is 'undone' specifically, treat marking done as a
view-removal (same animate-then-prune step archive/delete uses).
 2026-06-11 07:37:38 +09:00
pewdiepie-archdaemon 7f71fbc3ea Email list: scroll an expanded card into view after click 
When clicking an email higher up in the list, its top edge can be hiding
behind the modal header or off-screen. After applying the
.email-card-expanded class + the new minHeight, scrollIntoView(block:start)
on the next animation frame so the user sees the whole card.
 2026-06-11 07:35:32 +09:00
pewdiepie-archdaemon 7017127a11 Email card: drop redundant header kebab; keep bottom '...' menu in expanded state 
The expanded email card painted a kebab menu in its title row because
the per-card .memory-item-actions menu at the bottom was hidden while
expanded. Both pointed at _showCardMenu(em). Remove the duplicate:

- Drop the email-card-header-menu button (and its rightCluster
  wrapper) — title row now just holds the nav arrows.
- Remove the CSS rule that hid .memory-item-actions on
  .email-card-expanded so the bottom kebab stays visible.
- Unread-dot insert point retargets to .email-card-nav-arrows now
  that the rightCluster is gone.
 2026-06-11 07:33:04 +09:00
pewdiepie-archdaemon 00643b5a4b Email library New (compose): envelope icon takes the accent color 2026-06-11 07:30:26 +09:00
pewdiepie-archdaemon e25c279e4b Email Library: drop redundant 'All emails. Click to open as a document' subtitle 2026-06-11 07:28:21 +09:00
pewdiepie-archdaemon df54d8d2bf Email library: bulk 'Done' actually marks selected emails done 
state._selectedUids holds whatever the server returns for em.uid (string
or number); the bulk action looped Array.from(...) and did strict ===
against state._libEmails entries. When the types disagreed, the find()
returned undefined, the in-memory is_answered flip never happened, and
the post-loop _renderGrid() painted the cards back into their original
not-done state — looking like 'mark done' did nothing even though the
server-side call had succeeded.

- Compare via String() on both sides so the in-memory state actually
  flips.
- Surface HTTP failure from mark-answered/mark-read so the existing
  failedReadSync toast can fire if the calls don't go through.
 2026-06-11 07:27:25 +09:00
pewdiepie-archdaemon 8ae31aeb13 Email library compose button: scope taller+lower variant to desktop only 
Wrap the height:28px / top:0 rule in @media (min-width:769px) so it
can't leak into mobile, where a different touch-friendly variant
already sets min-height:36px + top:-2px.
 2026-06-11 07:24:25 +09:00
pewdiepie-archdaemon cc86760a26 Email library: drop compose button another 2px (top -2→0) 2026-06-11 07:23:49 +09:00
pewdiepie-archdaemon 2e7cfbe1fa Email library: New (compose) button 4px taller + 2px lower 
Base .memory-toolbar-btn is 24px tall at top:-4px. Bump the compose
button alone to 28px (4px taller) and top:-2px (moves down 2px) so
it reads as the primary action in the toolbar without affecting
Select/Refresh.
 2026-06-11 07:22:38 +09:00
pewdiepie-archdaemon 9dbe31bfb0 Email/doc split: stop auto-tab-down when there's no room 
Previously _prepareEmailWindowForDocument would:
  1. Check if there was horizontal room for both email + doc.
  2. If not, try collapsing the sidebar to recover space.
  3. If even that wasn't enough, _clearEmailDocumentSplit() — the
     email tab-down the user has been disliking.

Drop step 3. We still try collapsing the sidebar (free easy room),
but if the layout is still cramped, just dock anyway and let the
user manage their layout. _clearEmailDocumentSplit() is still
called on the legitimate close paths.
 2026-06-11 07:17:26 +09:00
pewdiepie-archdaemon 2bf372b41c Tasks: optional persona for LLM + research tasks (biases output voice) 
Wire the existing built-in PERSONAS catalog through to scheduled tasks
the same way I wired it to reminder synthesis. Repurposes the
dormant scheduled_tasks.character_id column.

UI (static/js/tasks.js)
- New 'Persona' select in the LLM / Research task form, with the five
  built-in characters (socrates/razor/nietzsche/spark/odysseus) plus a
  default 'no persona' option. Pre-populates from existing.character_id
  on edit. Non-llm/research types explicitly clear it on save.

API (routes/task_routes.py)
- TaskCreate + TaskUpdate gain character_id: Optional[str].
- _task_to_dict echoes character_id back so the form can hydrate on
  edit. Update endpoint stores '' as None to allow clearing.

Runner (src/task_scheduler.py)
- When task.character_id is set and matches a built-in persona, prepend
  the persona prompt to the task system prompt so the model speaks in
  that voice while still knowing it's running a scheduled task.
- crew_member.personality still wins as the base; character_id stacks
  on top.
 2026-06-10 23:36:18 +09:00
pewdiepie-archdaemon a86990fc58 Email row: fix crash from leftover menu-wrap wiring after button removal 
I removed the .email-menu-wrap markup from email rows earlier but
left the JS that queries it and calls .addEventListener on the
result. Since the query returns null, every _createEmailItem call
threw and the row never made it into the list — most visibly:
clicking a sender name to filter by them didn't appear to work,
because the row wiring (including the sender click handler) was
ripped out mid-construction.

- Drop the unconditional menuWrap.addEventListener('click', ...)
  block — there's no menu to open.
- Drop the early-return guard on touchstart that referenced the
  removed wrap.
- The two remaining .email-menu-wrap queries are already guarded
  with 'if (menuWrap)' so they safely no-op.
 2026-06-10 23:31:23 +09:00
pewdiepie-archdaemon f4c1b264c6 Email reminder bell: re-evaluate visibility live on settings change 
The bell is already gated on settings.reminder_channel === 'email', but
the check only ran at email-library init — so switching the reminder
channel in Settings didn't update the bell until you reopened Email.

- Settings/Reminders channel-change handler now dispatches
  odysseus-reminder-channel-changed { channel } after saving.
- emailLibrary listens for it and re-runs _syncEmailReminderBellVisibility
  with the new channel value.
 2026-06-10 23:26:53 +09:00
pewdiepie-archdaemon 031a600725 Email accounts strip: drop redundant 'Accounts' label during load — whirlpool alone 
The strip already lives where account chips render, so the text label
beside the whirlpool was redundant. Strip the label + the fallback
'Accounts...' text — the spinner alone tells the user accounts are
loading.
 2026-06-10 23:22:59 +09:00
pewdiepie-archdaemon b385b25d5f Email row: remove the three-dot actions menu button 
Dropped the .email-menu-wrap / .email-menu-btn from each row. Other
handlers that check 'if (e.target.closest(.email-menu-wrap)) return;'
safely no-op when the element doesn't exist. Row click + swipe still
open the email and its in-reader actions.
 2026-06-10 23:21:17 +09:00
pewdiepie-archdaemon 49b72bd09c Email attachments: nudge download spinner up 2px to sit on icon baseline 2026-06-10 23:19:19 +09:00
pewdiepie-archdaemon 0a3333b961 Edge-dock resize handle: fade accent stripe in on hover 
Transparent at rest, accent gradient animates in on hover with a 0.18s
ease transition. Drag affordance + col-resize cursor still work; the
stripe just stops bothering you when not touched.

Right-side handle mirrors the gradient direction (left-to-right
gradient flipped to right-to-left).
 2026-06-10 23:18:07 +09:00
pewdiepie-archdaemon 1638db9c86 Email reader: theme-aware override for Gmail drive/attachment chips 
Gmail composer chips arrive with inline border:1px solid #ddd + an
assumed white background, so on dark themes they read as a barely-
visible white box with the filename invisible. Override .gmail_chip /
.gmail_drive_chip inside .email-reader-body:

- Strip inline width:386px / height:20px (use auto + max-width:100%),
- Re-flow as inline-flex with a 6px gap so icon + filename align.
- Background tinted with var(--fg) 4%, border = var(--border).
- Anchor uses var(--accent) and the filename span uses var(--fg) so
  text is always legible regardless of theme.
- Icon img clamped to 16x16.
 2026-06-10 23:17:18 +09:00
pewdiepie-archdaemon cd9ad1a7f2 Email attachments: swap paperclip for whirlpool spinner during download 
Before: the attachment chip just dimmed (opacity 0.6) while the file
downloaded — easy to miss on a large attachment.

Now: replace the paperclip SVG with a 12px whirlpool spinner for the
duration of the fetch, restoring the original icon when the download
finishes (or errors out). Same loading vocabulary as Test / Scan /
Probe / Send buttons elsewhere in the UI.
 2026-06-10 23:15:52 +09:00
pewdiepie-archdaemon 023f1ba575 Email inbox: visual flash when an email is auto-marked done after sending 
When the email-answered event fires (user just sent a reply, so the
source email auto-marks as done), the row was getting the .active
class instantly with no visible cue beyond the checkbox tick. Add a
brief .email-auto-done-flash class on the row that runs two keyframe
animations:
- email-auto-done-row: tints the row background with the accent for
  ~1.2s then fades to transparent.
- email-auto-done-check: pops the done checkbox to 1.4× with an
  accent ring that expands outward over 0.6s.

Class self-removes after 1.2s so it doesn't replay on re-renders.
 2026-06-10 23:06:42 +09:00
pewdiepie-archdaemon 1a4659b7fc Edge-dock resize handle: drop the visible accent stripe 
The drag handle painted a 35% accent gradient strip on the page edge
of any docked panel. The col-resize cursor on hover is enough to
surface the affordance; the stripe felt like a stray UI element.
 2026-06-10 23:05:39 +09:00
pewdiepie-archdaemon 965b0e143c Email accounts strip: wheel + grab-drag horizontal scroll 
The single-row chip strip relied on native horizontal scroll, which is
hard to reach without a horizontal wheel. Wire two scroll mechanisms
on the strip once it's rendered:

- Vertical wheel → horizontal scroll (intercept only when overflow
  exists and the wheel motion is primarily vertical, so normal page
  scroll still works elsewhere).
- Mouse grab-and-drag: cursor goes grab/grabbing, mousedown→move
  bumps scrollLeft by the cursor delta. A 5px drag threshold cancels
  the chip click so the user can drag-scroll without accidentally
  switching accounts.
 2026-06-10 23:00:29 +09:00
pewdiepie-archdaemon 1eca28e588 Email: revert single-row email-item; account chips single-row at all widths 
- Revert the email row layout — sender/date stay above subject again,
  matching the original two-line item that the user actually wanted.
- The account filter chips (#email-lib-accounts) wrapped onto multiple
  rows on desktop. Promote the mobile-only horizontal-scroll rule to
  apply at every breakpoint so the chips always sit on one row with
  overflow scroll, regardless of screen size.
 2026-06-10 22:55:10 +09:00
pewdiepie-archdaemon a80421efb6 Sessions sort dropdown: nudge all items 2px more left 
Group row's auto-sort-sessions-btn padding-left 6→4, and
.sort-dropdown-item left padding 8→6 so 'Last Active', 'Newest First',
'By Folder', '↑↓ Rearrange', '● Select' all shift in by the same
amount, matching the Group nudge.
 2026-06-10 22:49:53 +09:00
pewdiepie-archdaemon 89efd7d44b Email list: collapse to a single row — [sender] [subject] [date] 
Subject was on its own line below sender/date. Move it inline so each
email occupies one row: sender capped at 35% width (ellipsis), subject
takes the remaining space (ellipsis), date pins to the right. Tighter
list density at the cost of dropping the spare line for snippet text
(none was being rendered anyway).
 2026-06-10 22:45:19 +09:00
pewdiepie-archdaemon 41980df6f1 Sidebar/Chats manage button: parent-hover reveal + clearer 'manage' text 
Two related fixes for the Chats section header:
- The 'manage' label only slid out when the button itself was hovered.
  Add section-header-flex:hover to the reveal rule so hovering the sort
  icon (or anywhere in the section header) also opens the label.
- Parent-hover opacity bumped 0.45 → 0.85 so the 'manage' text reads
  much more clearly when revealed. Direct hover on the button still
  pushes to full opacity 1.
 2026-06-10 22:44:50 +09:00
pewdiepie-archdaemon baa4449a03 Sidebar/Chats manage button: drop hover background tint 
The shared .section-header-btn:hover rule paints a tinted background
across all section header buttons. On the Chats manage button this
showed as a box behind the sliding 'manage' label, which the user
didn't want. Override background to transparent for that one button.
 2026-06-10 22:42:06 +09:00
pewdiepie-archdaemon 1ee51be420 Sessions: Esc + outside-click also close the Move-to-folder submenu 
The session-dropdown Esc handler only closed .session-dropdown-menu,
leaving the .session-folder-submenu (Move to folder → folder list)
orphaned on screen. Same gap on the click-away path. Extend both
selectors to cover the submenu so a single Esc / outside-click
dismisses the whole stack.
 2026-06-10 22:39:23 +09:00
pewdiepie-archdaemon 94931ba59f Chats sidebar: 'manage' label sits in flex flow so its area is clickable 
Email's 'new' label is absolutely positioned to the LEFT of the '+'
icon, which works there because the '+' is the visible/clickable
anchor. The chats manage button has no visible glyph at rest, so the
label was rendered outside the button's bounding box — hovering
'manage' lost the :hover state and clicking it missed.

Override list-item-plus-label inside chats-manage-btn:
  position: static (in flex flow) + max-width:0 / max-width:80px
expand-on-hover so the button's clickable rect grows alongside the
text. Hover stays sticky; click hits.
 2026-06-10 22:39:05 +09:00
pewdiepie-archdaemon 49ecd806a2 Chats sidebar: 'manage' slides in from the side like email's 'new' 
The list-item-plus-label slide-in needs a visible anchor element so
the button takes up consistent width and the absolutely-positioned
label can fly in to the left of it. Email uses the '+' SVG as that
anchor; here we use an empty 13x13 spacer span instead — same
footprint, no glyph. Result: empty button at rest (still visible per
the chats-manage-btn fade rules), 'manage' slides in from the left
on direct hover.
 2026-06-10 22:37:06 +09:00
pewdiepie-archdaemon 1eaa5c2a81 Sessions sort: nudge auto-sort icon + 'Group' text 4px left (10→6 left padding) 2026-06-10 22:35:21 +09:00
pewdiepie-archdaemon e107c5876e Chats sidebar: drop library SVG from manage button — text-only 'manage' 
Removed the book/library SVG and list-item-plus-btn/-label classes.
The button is now a plain text button styled like email's 'new' label
(9.5px, 0.02em letter-spacing), reusing the existing chats-manage-btn
opacity hover-reveal rules so it still fades until you hover the
section.
 2026-06-10 22:35:06 +09:00
pewdiepie-archdaemon 4f7061fd61 Settings overhaul + UI polish pass 
Two months of iteration on the Settings panel, integration forms, and
small visual nudges across the app. Highlights:

Settings restructure
- Add Models: split into separate Local + API cards (no more in-card
  tabs); each fuses Type/Provider with the URL input.
- Added Models: new dedicated sidebar tab, with Probe + Clear-offline
  pulled into its header; Local/API sub-section icons accent-tinted.
- Search: Web Search and a new Deep Research card (Model + tuning),
  with a cross-link to AI Defaults. Provider hints use real clickable
  anchors; Web Search Test button shows a whirlpool spinner.
- AI Defaults: Image Generation card returns; Research Model card
  carries only Endpoint+Model with a cross-link to Search; Vision /
  Default / Utility fallbacks unified under one numbered-row design
  matching Search's chain.
- API Permissions (was 'API Tokens'): per-row rename, inline
  Permissions toggle that expands the scope-edit panel, in-field
  copy icons (icon→check on success). Empty state accent-tinted.
- Integrations: + Add Integration drops a type-picker menu directly
  under the button (drop-up on tight viewports); each integration
  form (API, CalDAV, CardDAV, Email, Codex/Claude, Vault, MCP) uses
  the same accent-outlined Save/Test/Cancel buttons right-aligned.
- Danger Zone: Wipe→Delete with trash icons; new 'Delete everything'
  row at the bottom that loops every category.

AI Synthesis (Reminders)
- Persona dropdown sourced from PROMPT_TEMPLATES + custom preset.
- src/reminder_personas.py mirrors the five built-ins for the
  server-side synthesis path.
- dispatch_reminder() reads reminder_llm_persona and uses the
  persona's system prompt; empty/unknown falls back to warm-neutral.

Esc handling
- Kebab menus and the provider picker intercept Esc in capture phase
  so dismissing a popup no longer closes the whole Settings modal.

Accent tinting
- Scoped CSS rule across data-settings-panel=ai/services/added-models/
  search/integrations/reminders for card h2 icons + the Added Models
  sub-section icons.

Codex/Claude integration form
- No more auto-creation on form open — explicit Create token button.
- New tokens start with every scope granted; existing tokens move out
  of the integration form into the API Permissions card.
- Setup reveal: copy buttons inline inside the token + setup code
  blocks; shorter subtitle wording.

Misc visual polish
- Save/Test/Cancel uniformly accent-outlined and right-aligned on
  every integration form.
- Provider logos render inline next to the search fallback selects
  and the Deep Research Search dropdown.
- Trash icons in fallback rows bumped to 20x20 so they fill the 32px
  button.
- Image generation default flipped to off.
 2026-06-10 15:15:13 +09:00
pewdiepie-archdaemon 7690860ab1 Settings/Add Models: bump Local Type select width 57→62px 2026-06-09 15:12:57 +09:00
pewdiepie-archdaemon b6366e9da5 Settings/Add Models: fuse Local Type select + URL input into one bordered group 2026-06-09 15:12:12 +09:00
pewdiepie-archdaemon 64122269e9 Settings/Add Models: shrink Local Type select by 15px (72→57) 2026-06-09 15:11:07 +09:00
pewdiepie-archdaemon 1bdd515941 Settings/Add Models: drop 'Type:' label, keep the LLM/Image select 2026-06-09 15:10:48 +09:00
pewdiepie-archdaemon 8ac0ae72dc Settings/Add Models: Local card — Type and Add inline with URL field 
Lift the LLM/Image Type select to the left of the URL input and the Add
button to its right, so the primary action (URL + Add) sits on one row.
Scan / Ollama / Key / Test stay on the action row below.
 2026-06-09 15:09:28 +09:00
pewdiepie-archdaemon b2458f9891 Settings/Add Models: split Local and API into separate cards, always show API key 
Drop the in-card Local/API tab strip — each is now its own admin card with
a normal h2 heading (Local on top, API below). The API key input is
always visible (no more click-to-reveal toggle), matching how cloud
providers actually work. Local keeps the optional key reveal since
local servers usually don't need one.

Dead code removed: wireModelsTabs IIFE and the adm-epApiKeyBtn toggle wire.
 2026-06-09 14:57:42 +09:00
pewdiepie-archdaemon 2252776a97 Settings: promote Added Models to its own sidebar menu 
Move the Added Models endpoint lists out of the Add Models card into a
dedicated sidebar tab between Add Models and AI Defaults. The card now
focuses purely on adding (Local / API tabs), while the new panel owns
the existing endpoints + Probe and Clear-offline controls.

admin.js: defensive fallback so a stale 'added' value in localStorage
falls back to 'local' instead of leaving both panes hidden.
 2026-06-09 14:52:48 +09:00
pewdiepie-archdaemon c9fecd53dc Settings: third 'Added Models' tab in Add Models card 
Move the Added Local + Added API lists out of the per-type tabs into
a dedicated third tab. Each Add tab is now just the form; the new tab
collects both lists together with Local / API subheadings.

Card layout:
  Add Models  [Probe] [Clear offline]
    [Local]  [API]  [Added Models]

Tab content:
  Local         → Add Local form
  API           → Add API form
  Added Models  → Local list + API list (subheadings)

All endpoint list/form IDs preserved. Tab switcher JS is generic so
the new 'added' tab works without code changes.
 2026-06-09 14:47:21 +09:00
pewdiepie-archdaemon 75268e7f43 Fix Cookbook serve server selection 2026-06-09 14:45:22 +09:00
pewdiepie-archdaemon 8ef9b8b215 Settings: tabbed Add Models card with Local / API tabs 
Earlier split into 4 flat cards wasn't what was asked for. Restore to
a single 'Add Models' card with two tabs at the top:

  Local  → Add form + Added Local Models list
  API    → Add form + Added API Endpoints list

Probe / Clear-offline live on the card header and act on both lists.
Active tab is remembered in localStorage so the user lands back where
they were. All form/list IDs preserved (adm-epLocalUrl, adm-epList-local,
adm-epList-api, etc.) so admin.js continues to work unchanged.

Replaces the .adm-section-toggle fold-open JS with a tab-switcher; the
fold elements no longer exist so the old handler was already a no-op.
 2026-06-09 14:43:28 +09:00
pewdiepie-archdaemon 459b825daa Settings: split Add/Added Models into 4 flat cards (no folds) 
The previous 'Add Models' card had two collapsible folds (Local + API)
inside it and 'Added Models' had two inline subsections. Both folded
states added a click-to-expand step that wasn't earning its keep —
users coming to Settings to add a model don't want a fold, they want
the form.

Reshape: four flat admin-cards in the Services panel, each with its
own h2 title matching the rest of Settings:
  Add Local Model       (was Add Models → Local fold)
  Add API               (was Add Models → API fold)
  Added Local Models    (was Added Models → Local subsection)
  Added API Endpoints   (was Added Models → API subsection)

The collapsible JS hook in admin.js already guards on
'if (!head) return' so removing the .adm-section-toggle headers
turns it into a clean no-op — no breakage.

All input/list IDs preserved (adm-epLocalUrl, adm-epList-local,
adm-epList-api, etc.) so the rest of admin.js continues to work
unchanged. Probe / Clear-offline live on the Local card and act on
both lists together (existing behavior).
 2026-06-09 14:36:44 +09:00
pewdiepie-archdaemon 3247773447 Hide Teacher Model settings card (2.0 'harden the core' deferral) 
The Teacher Mode feature stays out of the default UI per the 2.0
roadmap — backend escalation is already dormant when teacher_model is
unset (its default) and we want to focus on core reliability before
surfacing escalation as a feature.

Nothing removed from the backend:
- src/teacher_escalation.py still gates on get_setting('teacher_model')
- agent_loop.py's run_teacher_inline hook is a no-op without the setting
- settings backup/restore round-trips the teacher_model key unchanged
- power users can still set it via manage_settings or the JSON backup

settings.js's initTeacherModel already early-returns when the card's
DOM ids are missing, so the JS side is clean.

To re-surface the card, revert this commit.
 2026-06-09 14:31:04 +09:00
pewdiepie-archdaemon 013beab861 Add Codex and Claude document draft integration 2026-06-09 14:27:53 +09:00
pewdiepie-archdaemon c5230e85a9 Change project license to AGPL-3.0-or-later 2026-06-09 14:25:04 +09:00
Muhammad-Ikhwan-Fathulloh e8106f7c7c Fix logical bugs in event bus and bulk session deletion 2026-06-07 01:38:33 +07:00
435 changed files with 44091 additions and 12399 deletions
Expand all files Collapse all files
.dockerignore
+4
View File
@@ -15,6 +15,10 @@ build/
# at runtime — never baked into the image. Mirrored in .gitignore.
secrets.env
secrets.env.*
secrets.env~
.secrets.env.swp
.secrets.env.swo
**/#secrets.env#
!secrets.env.example
/data/
/logs/
.env.example
+7
View File
@@ -190,3 +190,10 @@ SEARXNG_INSTANCE=http://localhost:8080
# These overlays only expose the GPU devices. The slim Odysseus image
# still needs CUDA/ROCm userspace via Cookbook -> Dependencies (vLLM,
# llama-cpp-python, etc.) before models can actually serve on GPU.
# ============================================================
# Storage Paths (Docker Compose)
# ============================================================
# APP_DATA_DIR=./data
# APP_LOGS_DIR=./logs
.github/pull_request_review_template.md
+123
View File
@@ -0,0 +1,123 @@
# Pull Request Review Template
Use this shape as a copyable reference for substantive PR reviews; GitHub does
not auto-apply this file to review comments. Omit sections that do not add
useful signal. Lead with confirmed findings; keep speculative notes out of the
public review unless they are framed as a concrete open question.
## Small PR Path
For narrow docs, typo, test-only, or obvious local fixes, a short review is
enough:
```md
LGTM after checking:
- scope:
- validation:
- residual risk:
```
Use the fuller structure below for larger, risky, multi-finding, or
security-sensitive reviews.
## Findings
**<sub><sub>![P2 Badge](https://img.shields.io/badge/P2-yellow?style=flat)</sub></sub> issue (test): Short issue title**
- **Problem:** Concrete broken flow, contract, input, or risk.
- **Impact:** Why this matters to users, CI, maintainers, data, security, or scale.
- **Ask:** Smallest practical correction or decision the author should make.
- **Location:** `path:line`
## Open Questions
- **question (scope, non-blocking): Short author question** Ask the concrete
intent, scope, or tradeoff question.
## Validation
- Ran:
- Not run:
- Residual risk:
## PR Hygiene
- Target/template/checks:
- Related, duplicate, or superseding context:
## No Findings Variant
```md
## Findings
none confirmed
## Validation
- Ran:
- Not run:
- Residual risk:
```
## Legend
- **Findings:** Verified, author-actionable issues that should be fixed or
consciously accepted before merge.
- **Priority badges:** The shields.io badges below are optional formatting for
priority labels. Plain `P0`, `P1`, `P2`, or `P3` text is also acceptable when
an external image dependency is undesirable or may not render.
- **P0:** `![P0 Badge](https://img.shields.io/badge/P0-red?style=flat)` -
release-blocking or actively dangerous.
- **P1:** `![P1 Badge](https://img.shields.io/badge/P1-orange?style=flat)` -
serious bug, security risk, data-loss risk, or broken primary flow.
- **P2:** `![P2 Badge](https://img.shields.io/badge/P2-yellow?style=flat)` -
meaningful correctness, test, maintainability, or edge-case issue.
- **P3:** `![P3 Badge](https://img.shields.io/badge/P3-lightgrey?style=flat)` -
minor polish or low-risk cleanup.
- **Intent labels:**
- **`issue`:** A confirmed defect, regression, broken contract, or concrete
risk.
- **`suggestion`:** A non-blocking improvement that would make the PR clearer,
safer, or easier to maintain.
- **`nit`:** A tiny, non-blocking cleanup or style note. Use it only when the
author can safely ignore it without changing the review outcome.
- **`question`:** A real author-facing clarification about intent, scope, or
tradeoffs. Do not use questions to hide an issue that should be stated
directly.
- **`LGTM`:** "Looks good to me." Use only when the review found no blocking
issues, or when any remaining notes are clearly optional.
- **Decorations:** Optional labels in parentheses that clarify the finding type,
scope, or merge impact.
- **`security`:** Auth, authorization, ownership, secrets, SSRF, injection,
unsafe external input, or other trust-boundary concerns.
- **`test`:** Missing, failing, misleading, brittle, or insufficient tests.
- **`scope`:** PR scope, feature boundaries, unrelated churn, or work that
should be split into a separate issue or PR.
- **`ci`:** CI configuration, workflow failures, flaky checks, or validation
signal quality.
- **`api`:** Route, request/response, public function, schema, persistence, or
integration contract changes.
- **`docs`:** User-facing docs, contributor docs, examples, or comments that
need to change with the code.
- **`non-blocking`:** Useful feedback that should not prevent merge by
itself.
- **Finding fields:**
- **Problem:** What is wrong, what contract is ambiguous, or what risk the PR
introduces.
- **Impact:** Why the problem matters in practical terms.
- **Ask:** The smallest concrete fix, test, or decision requested from the PR
author.
- **Location:** The most useful repo-relative file and line reference for the
finding, using `path:line`.
- **Optional sections:**
- **Open Questions:** Genuine scope or intent questions; omit when there are
no real questions.
- **Validation:** What the reviewer ran, what was intentionally not run, and
what risk remains after review.
- **PR Hygiene:** Target-branch, template, CI/check, duplicate, related-work,
or superseding-PR notes.
- **`none confirmed`:** Use only when no review-worthy findings were confirmed;
still list validation gaps or residual risk when relevant.
.github/workflows/ci.yml
+6 -6
View File
@@ -19,10 +19,10 @@ jobs:
name: Python syntax (compileall)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: "3.11"
# Byte-compile sources — catches syntax errors without installing deps.
@@ -32,10 +32,10 @@ jobs:
name: JS syntax (node --check)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "20"
# Syntax-check our own JS (skip vendored libs in static/lib).
@@ -54,7 +54,7 @@ jobs:
# ROADMAP "fresh install smoke tests" item; make this required once green.
continue-on-error: true
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
fetch-depth: 0
persist-credentials: false
@@ -81,7 +81,7 @@ jobs:
echo "docs_only=false" >> "$GITHUB_OUTPUT"
fi
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
if: steps.docs-check.outputs.docs_only != 'true'
with:
python-version: "3.11"
.github/workflows/codeql.yml
-61
View File
@@ -1,61 +0,0 @@
# CodeQL code scanning
#
# Purpose: GitHub's own static analysis engine reads the application source
# (Python backend + the JavaScript frontend) and looks for real
# vulnerabilities -- SQL/command injection, path traversal, auth mistakes,
# unsafe deserialization. Findings appear in the repo's Security tab. This is
# the deepest check in the suite and the most valuable for a high-profile
# target.
#
# It runs on every push to main and on a weekly schedule (to catch newly
# disclosed query patterns against unchanged code). It deliberately does NOT
# run on pull requests: most PRs here come from forks, whose read-only token
# cannot publish results, which would produce confusing failures. To scan pull
# requests too, a maintainer can instead enable CodeQL "default setup" in
# Settings -> Security -> Code scanning (one toggle, no file needed) -- see
# docs/security-ci.md.
name: CodeQL
on:
push:
branches: [main]
schedule:
# Weekly, Monday 06:00 UTC.
- cron: '0 6 * * 1'
workflow_dispatch:
permissions: {}
concurrency:
group: codeql-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
analyze:
name: Analyze (${{ matrix.language }})
runs-on: ubuntu-latest
permissions:
contents: read
security-events: write # publish results to the Security tab
strategy:
fail-fast: false
matrix:
# Both are interpreted, so CodeQL needs no build step (build-mode none).
language: [python, javascript-typescript]
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@03e4368ac7daa2bd82b3e85262f3bf87ee112f57 # v3.36.0
with:
languages: ${{ matrix.language }}
build-mode: none
- name: Perform CodeQL analysis
uses: github/codeql-action/analyze@03e4368ac7daa2bd82b3e85262f3bf87ee112f57 # v3.36.0
with:
category: "/language:${{ matrix.language }}"
.github/workflows/container-scan.yml
+1 -1
View File
@@ -37,7 +37,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
.github/workflows/container-trivy.yml
+3 -3
View File
@@ -52,7 +52,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
@@ -93,7 +93,7 @@ jobs:
security-events: write # upload SARIF to the Security tab
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
@@ -119,7 +119,7 @@ jobs:
TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db:2
- name: Upload Trivy results
uses: github/codeql-action/upload-sarif@03e4368ac7daa2bd82b3e85262f3bf87ee112f57 # v3.36.0
uses: github/codeql-action/upload-sarif@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
with:
sarif_file: trivy-results.sarif
category: trivy-image
.github/workflows/dependency-review.yml
+2 -2
View File
@@ -36,7 +36,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
@@ -55,7 +55,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
.github/workflows/docker-publish.yml
+2 -2
View File
@@ -45,7 +45,7 @@ jobs:
arch: arm64
runner: ubuntu-24.04-arm
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Buildx
@@ -86,7 +86,7 @@ jobs:
contents: read
packages: write
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Read APP_VERSION + short sha
.github/workflows/issue-description-check.yml
+1 -1
View File
@@ -14,7 +14,7 @@ jobs:
# Skip bots (Dependabot, release-drafter, etc.)
if: ${{ github.event.issue.user.type != 'Bot' }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
sparse-checkout: .github/scripts
persist-credentials: false
.github/workflows/pr-description-check.yml
+1 -1
View File
@@ -23,7 +23,7 @@ jobs:
# Skip bots: they open PRs programmatically and have their own process.
if: github.event.pull_request.user.type != 'Bot'
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.base_ref }}
sparse-checkout: .github/scripts
.github/workflows/secret-scan.yml
+1 -1
View File
@@ -35,7 +35,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
# Full history so a secret committed in an earlier commit (and later
# deleted) is still caught -- deletion does not remove it from Git.
.github/workflows/workflow-security.yml
+2 -2
View File
@@ -36,7 +36,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
@@ -61,7 +61,7 @@ jobs:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
ACKNOWLEDGMENTS.md
+1
View File
@@ -86,6 +86,7 @@ Bundled in `static/fonts/`:
| [Fira Code](https://github.com/tonsky/FiraCode) | SIL Open Font License 1.1 | Nikita Prokopov & contributors |
| [Inter](https://github.com/rsms/inter) | SIL Open Font License 1.1 | Rasmus Andersson |
| [GohuFont](https://font.gohu.org/) (`fonts/custom/GohuFont.ttf`) | WTFPL | Hugo Chargois |
| [OpenDyslexic](https://opendyslexic.org/) (`fonts/OpenDyslexic-{Regular,Bold}.woff2`) | SIL Open Font License 1.1 ([`licenses/OpenDyslexic-OFL.txt`](licenses/OpenDyslexic-OFL.txt)) | Abbie Gonzalez |
## Python dependencies
CONTRIBUTING.md
+1 -1
View File
@@ -37,7 +37,7 @@ Manual development uses Python 3.11+:
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -m uvicorn app:app --host 0.0.0.0 --port 7000
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
Windows is not actively tested. Docker on Linux or a Linux/macOS manual install is the safer path for now.
Dockerfile
+62 -1
View File
@@ -1,4 +1,15 @@
FROM python:3.12-slim
# ---- builder: patch + build wheels for Real-ESRGAN's broken-on-3.14 deps ----
# basicsr/gfpgan/facexlib read their version via exec()+locals()['__version__'],
# which raises KeyError on Python 3.13+ (PEP 667). Build patched wheels here so
# the final image / Cookbook never has to compile the broken sdists. See
# docker/build-realesrgan-wheels.sh for the full rationale.
FROM python:3.14-slim AS realesrgan-wheels
RUN apt-get update && apt-get install -y --no-install-recommends curl \
&& rm -rf /var/lib/apt/lists/*
COPY docker/build-realesrgan-wheels.sh /usr/local/bin/build-realesrgan-wheels.sh
RUN bash /usr/local/bin/build-realesrgan-wheels.sh /wheels
FROM python:3.14-slim
# System deps. tmux is required by Cookbook for background downloads/serves.
# openssh-client is required for Cookbook remote server tests, setup, probes,
@@ -18,8 +29,44 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
tmux \
openssh-client \
gosu \
libgl1 \
libglib2.0-0t64 \
libxcb1 \
libmagic1 \
&& rm -rf /var/lib/apt/lists/*
# libgl1/libglib2.0-0t64/libxcb1 are runtime shared libs (libGL.so.1,
# libglib-2.0/libgthread, libxcb.so.1) that opencv-python (cv2) loads. The
# slim base omits them, so the Cookbook "install realesrgan" path imports cv2
# and dies with `libxcb.so.1: cannot open shared object file` despite a clean
# pip install. Using full opencv-python (not -headless) because basicsr/gfpgan/
# facexlib/realesrgan all depend on the `opencv-python` distribution by name.
#
# libmagic1 is the shared lib (libmagic.so.1) that python-magic dlopens for
# content-based MIME sniffing in src/upload_handler.py. We install both here
# (libmagic1 + the python-magic wrapper, below) rather than in requirements.txt
# because python-magic resolves libmagic at import time: where the lib is
# absent the import can block or raise, so keeping it image-only avoids
# regressing pip/venv installs on hosts without libmagic. Debian always has the
# lib here, so the import is instant and detection actually works.
# Docker CLI (client only — daemon stays on the host via the
# /var/run/docker.sock mount). The Debian `docker.io` package ships
# dockerd but not the client binary on slim, so grab the static client
# tarball from download.docker.com instead.
ARG DOCKER_CLI_VERSION=27.5.1
RUN ARCH="$(dpkg --print-architecture)" \
&& case "$ARCH" in \
amd64) DARCH=x86_64 ;; \
arm64) DARCH=aarch64 ;; \
*) echo "unsupported arch $ARCH"; exit 1 ;; \
esac \
&& curl -fsSL "https://download.docker.com/linux/static/stable/${DARCH}/docker-${DOCKER_CLI_VERSION}.tgz" \
-o /tmp/docker.tgz \
&& tar -xzf /tmp/docker.tgz -C /tmp \
&& install -m 0755 /tmp/docker/docker /usr/local/bin/docker \
&& rm -rf /tmp/docker /tmp/docker.tgz
WORKDIR /app
# Install Python deps first (layer cache). Optional extras (PyMuPDF AGPL, etc.)
@@ -29,6 +76,20 @@ COPY requirements.txt requirements-optional.txt ./
RUN pip install --no-cache-dir -r requirements.txt \
&& if [ "$INSTALL_OPTIONAL" = "true" ]; then pip install --no-cache-dir -r requirements-optional.txt; fi
# python-magic powers content-based MIME sniffing in src/upload_handler.py.
# Image-only (not in requirements.txt) because it needs the libmagic1 system
# lib installed above; see the apt note near the top of this stage.
RUN pip install --no-cache-dir python-magic==0.4.27
# Pre-install the patched basicsr/gfpgan/facexlib wheels built in the
# realesrgan-wheels stage (--no-deps keeps the image lean — torch & friends are
# pulled only when realesrgan is actually installed). With these dists already
# satisfied, the Cookbook's plain `pip install realesrgan` resolves them from
# wheels instead of rebuilding the sdists that fail on Python 3.14.
COPY --from=realesrgan-wheels /wheels/ /tmp/odysseus-wheels/
RUN pip install --no-cache-dir --no-deps /tmp/odysseus-wheels/*.whl \
&& rm -rf /tmp/odysseus-wheels
# Copy app code
COPY . .
Odysseus.spec
+45
View File
@@ -0,0 +1,45 @@
# -*- mode: python ; coding: utf-8 -*-
a = Analysis(
['launcher.py'],
pathex=[],
binaries=[],
datas=[('static', 'static'), ('scripts', 'scripts'), ('mcp_servers', 'mcp_servers'), ('services/hwfit/data', 'services/hwfit/data'), ('config', 'config'), ('.env.example', '.env.example')],
hiddenimports=[],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
noarchive=False,
optimize=0,
)
pyz = PYZ(a.pure)
exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name='Odysseus',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
icon=['static\\icon.ico'],
)
coll = COLLECT(
exe,
a.binaries,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='Odysseus',
)
README.md
+38 -458
View File
@@ -1,471 +1,65 @@
# Odysseus
<p align="center">
<img src="docs/odysseus-wordmark.png" alt="Odysseus" width="238">
</p>
> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main).
<p align="center">
A self-hosted AI workspace for chat, agents, research, documents, email, notes, calendar, and local model workflows.
</p>
```
───────────────────────────────────────────────
⊹ ࣪ ˖ ૮( ˶ᵔ ᵕ ᵔ˶ )っ Odysseus vers. 1.0
───────────────────────────────────────────────
```
<p align="center">
<a href="#quick-start">Quick Start</a> ·
<a href="docs/setup.md">Setup Guide</a> ·
<a href="CONTRIBUTING.md">Contributing</a> ·
<a href="ROADMAP.md">Roadmap</a>
</p>
![Odysseus](docs/odysseus.jpg)
<p align="center">
<a href="https://repology.org/project/odysseus-ai/versions"><img src="https://repology.org/badge/vertical-allrepos/odysseus-ai.svg" alt="Packaging status"></a>
</p>
A self-hosted AI workspace -- meant to be the self-hosted version of the UI experience you get from ChatGPT and Claude. But with more jank and fun. Running on your own hardware, with your own data -- local-first, privacy-first, and no trojan.
<p align="center">
<img src="docs/odysseus-browser.jpg" alt="Odysseus interface">
</p>
[![Packaging status](https://repology.org/badge/vertical-allrepos/odysseus-ai.svg)](https://repology.org/project/odysseus-ai/versions)
## Features
- **Chat** -- chat with any local model or API; adding them is super simple.<br>　<sub>vLLM · llama.cpp · Ollama · OpenRouter · OpenAI · GitHub Copilot</sub>
- **Agent** -- hand it tools and let it run the whole task itself.<br>　<sub>built on [opencode](https://github.com/anomalyco/opencode) · MCP · web · files · shell · skills · memory</sub>
- **Cookbook** -- Scans your hardware, recommends models, click to download and serve.. easy!<br>　<sub>built on [llmfit](https://github.com/AlexsJones/llmfit) · VRAM-aware · GGUF / FP8 / AWQ · fit scoring · vLLM / llama.cpp serving</sub>
- **Deep Research** -- multi-step runs that gather, read, and synthesize sources into a nice visual report.<br>　<sub>adapted from [Tongyi DeepResearch](https://github.com/Alibaba-NLP/DeepResearch)</sub>
- **Compare** -- a fun tool to compare models side by side. Test completely blind, no bias!<br>　<sub>multi-model · blind test · synthesis</sub>
- **Documents** -- YOU write the text, AI is there to assist, not the opposite.<br>　<sub>multi-tab editor · markdown · HTML · CSV · syntax highlighting · AI edits · suggestions</sub>
- **Memory / Skills** -- Persistent memory and skills, your agent evolves over time as it better understands you and your tasks!<br>　<sub>ChromaDB · fastembed (ONNX) · vector + keyword retrieval · import/export</sub>
- **Email** -- IMAP/SMTP inbox with AI triage built in: urgency reminders, auto-tag, auto-summary, auto-reply drafts, auto-spam.<br>　<sub>IMAP · SMTP · per-account routing · CalDAV-aware</sub>
- **Notes & Tasks** -- Quick notes with reminders, a todo list, and scheduled tasks the agent can act on.<br>　<sub>note pings · checklist · cron-style tasks · ntfy / browser / email channels</sub>
- **Calendar** -- Local-first calendar with CalDAV sync to Radicale / Nextcloud / Apple / Fastmail.<br>　<sub>CalDAV pull · .ics import/export · per-calendar colors · agent-aware</sub>
- **Works on mobile** -- looks and runs great on your phone, not just desktop.<br>　<sub>responsive · installable (PWA) · touch gestures</sub>
- **Extras** -- more to explore, happy if you give it a go!<br>　<sub>image editor · theme editor · file uploads (vision + PDF) · web search · presets · sessions · 2FA</sub>
## Demo
A full, hover-to-play tour lives on the landing page (`docs/index.html`).
<details>
<summary>Screenshots / clips</summary>
### Chat & Agents
![Chat & Agents](docs/chat.gif)
### Deep Research
![Deep Research](docs/research.gif)
### Compare
![Compare](docs/compare.gif)
### Documents
![Documents](docs/document.gif)
### Notes & Tasks
![Notes & Tasks](docs/notes.gif)
</details>
---
## Quick Start
Defaults work out of the box: clone, run, then configure models/search/email
inside **Settings**. Only edit `.env` for deployment-level overrides like
`APP_BIND`, `APP_PORT`, `AUTH_ENABLED`, `DATABASE_URL`, or a pre-seeded admin password.
> `dev` is the default branch and gets the newest changes first. Use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main) if you want the more curated branch.
On first setup, Odysseus creates an admin account (`admin` unless
`ODYSSEUS_ADMIN_USER` is set) and prints a temporary password in the terminal.
For Docker installs, the same line is in `docker compose logs odysseus`.
Use that for the first login, then change it in **Settings**.
Contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, testing, and
pull request guidelines.
### Docker (recommended)
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
cp .env.example .env # optional, but recommended for explicit defaults
cp .env.example .env
docker compose up -d --build
```
To include optional extras in the image (PDF viewer, Office extraction; includes AGPL PyMuPDF), build with `docker compose build --build-arg INSTALL_OPTIONAL=true` before `up`.
Open `http://localhost:7000` when the containers are healthy. Docker Compose
binds the web UI to `127.0.0.1` by default. If the port is taken, set
`APP_PORT=7001` in `.env` and recreate the container. Set `APP_BIND=0.0.0.0`
only when you intentionally want LAN/reverse-proxy access.
Open `http://localhost:7000` when the containers are healthy. The first admin password is printed in `docker compose logs odysseus`.
> **On Apple Silicon (M-series) Macs:** Docker can't reach the Metal GPU, so
> Cookbook serves local models on CPU only. For GPU-accelerated model serving,
> run natively instead — see [Apple Silicon](#apple-silicon) below.
Native installs, GPU notes, Windows/macOS instructions, HTTPS, and configuration live in the [setup guide](docs/setup.md).
### Native Linux / macOS
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
Requirements: Python 3.11+. Cookbook also needs `tmux` for background model
downloads and serves. The app itself is lightweight; local model serving is the
heavy part and depends on the model, runtime, GPU, and VRAM, so small hosts can
connect to API or remote model servers instead. Use `--host 0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
## Features
### Apple Silicon
Docker on macOS cannot use the Metal GPU. For GPU-accelerated Cookbook on an
M-series Mac, run Odysseus natively:
- **Chat + Agents** — local/API models, tools, MCP, files, shell, skills, and memory.
- **Cookbook** — hardware-aware model recommendations, downloads, and serving.
- **Deep Research** — multi-step web research with source reading and report generation.
- **Compare** — blind side-by-side model testing and synthesis.
- **Documents** — writing-first editor with AI edits, suggestions, Markdown, HTML, CSV, and syntax highlighting.
- **Email** — IMAP/SMTP inbox with triage, tags, summaries, reminders, and reply drafts.
- **Notes, Tasks + Calendar** — reminders, todos, scheduled agent tasks, and CalDAV sync.
- **Extras** — gallery/image editor, themes, uploads, web search, presets, sessions, and 2FA.
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
./start-macos.sh
```
## Demo
It launches at `http://127.0.0.1:7860`. To expose it to your phone over a trusted LAN/VPN such as Tailscale, bind all interfaces:
```bash
ODYSSEUS_HOST=0.0.0.0 ./start-macos.sh
# then open http://<tailscale-ip>:7860
```
The script also reads `.env` at startup, so `APP_BIND=0.0.0.0` and `APP_PORT`
set there are picked up automatically without a command-line override each run.
Keep `AUTH_ENABLED=true` (the default) before binding outside loopback. Do not
expose this port directly to the public internet. To build a clickable app wrapper:
```bash
./build-macos-app.sh
```
<details>
<summary>Cookbook, GPU, Ollama, and troubleshooting notes</summary>
**Docker bundled services.** Compose starts Odysseus, ChromaDB, SearXNG, and
ntfy. Odysseus and the bundled service ports bind to `127.0.0.1` by default, so
they are reachable from the host but not exposed to your LAN/public internet
unless you opt in.
**Cookbook storage in Docker.** Downloads live in `./data/huggingface`
(`~/.cache/huggingface` in the container). Cookbook-installed Python CLIs and
serve engines live in `./data/local` (`~/.local` in the container), so they
survive container recreation.
**Remote servers.** In **Cookbook -> Settings -> Servers**, generate the
Odysseus SSH key and add the public key to the remote server's
`~/.ssh/authorized_keys`. From the host you can also run:
```bash
ssh-copy-id -i data/ssh/id_ed25519.pub user@server
```
**Docker GPU overlays.** CPU-only users can skip this section. Cookbook can
only detect GPUs that Docker exposes to the container — if the host runtime or
device passthrough is not configured, Cookbook sees the iGPU, another card, or
CPU instead of your intended GPU.
For NVIDIA, `scripts/check-docker-gpu.sh` diagnoses GPU passthrough and can
optionally install the host runtime or update `.env`.
```bash
# Read-only diagnostic (default — installs nothing, never edits .env):
scripts/check-docker-gpu.sh
# Print OS-specific install commands without running them:
scripts/check-docker-gpu.sh --print-install-commands
# Install NVIDIA Container Toolkit on Ubuntu/Debian (requires sudo):
scripts/check-docker-gpu.sh --install-nvidia-toolkit
# Write COMPOSE_FILE to .env (only when GPU passthrough is confirmed working):
scripts/check-docker-gpu.sh --enable-nvidia-overlay
# Full assisted setup — install toolkit, then enable overlay if passthrough works:
scripts/check-docker-gpu.sh --install-nvidia-toolkit --enable-nvidia-overlay
```
Safety notes:
- The app never installs host GPU runtime automatically.
- The app never edits `.env` automatically.
- `.env` is only modified when `--enable-nvidia-overlay` is explicitly passed,
and only after GPU passthrough succeeds. `--yes` skips prompts but does not
bypass the passthrough gate.
- `.env.bak.*` backups created by `--enable-nvidia-overlay` are ignored by
Git and the Docker build context.
To enable manually without the script, add this to `.env`:
```bash
COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
```
**AMD / ROCm.** AMD setup is read-only diagnostic plus manual `.env` edit. Run:
```bash
scripts/check-docker-amd-gpu.sh
```
Then add the reported values to `.env`, replacing `RENDER_GID` with your host's
numeric render group id:
```bash
COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
RENDER_GID=989
```
For NVIDIA/AMD GPU support, also read the comments in the selected overlay file: docker/gpu.nvidia.yml or docker/gpu.amd.yml.
**Stack-management UIs (Portainer, Coolify, Dockhand, etc.).** These tools
often accept only a single Compose file and do not reliably honor `COMPOSE_FILE`
or multiple `-f` overlays. CLI users should keep using the `COMPOSE_FILE`
overlay workflow above. For stack UIs, point the stack at one of the standalone
files instead, which bundle the base stack plus the GPU settings:
- `docker-compose.gpu-nvidia.yml` — still requires the NVIDIA Container Toolkit
on the host.
- `docker-compose.gpu-amd.yml` — still requires host ROCm/kfd/DRI setup, the
`video`/`render` group membership, and `RENDER_GID` when needed.
The base `docker-compose.yml` plus the `docker/gpu.*.yml` overlays remain the
source of truth; the standalone files mirror them for single-file deployments.
Verify after enabling either overlay:
```bash
docker compose exec odysseus nvidia-smi -L # NVIDIA
docker compose exec odysseus sh -lc 'test -e /dev/kfd && test -d /dev/dri && ls -l /dev/kfd /dev/dri/renderD*' # AMD
```
> **GPU passthrough ≠ llama.cpp CUDA.** `nvidia-smi` passing inside the
> container confirms Docker GPU access, but llama.cpp also needs `cudart` and
> the CUDA Toolkit at runtime. If Cookbook logs show `Unable to find cudart
> library`, `Could NOT find CUDAToolkit`, `CUDA Toolkit not found`, or
> tensors/layers assigned to CPU, that is a Cookbook/llama.cpp build issue —
> not a Docker passthrough failure. Reinstall the serve engine via
> **Cookbook → Dependencies** to get a CUDA-enabled build.
>
> The same split applies to AMD/ROCm: seeing `/dev/kfd` and `/dev/dri` inside
> the container confirms device passthrough, not ROCm userspace or a
> ROCm-enabled vLLM/llama.cpp build. `rocm-smi` and `rocminfo` are not expected
> inside the slim Odysseus image.
**Ollama with Docker.** If Ollama runs on the host, add this endpoint in
Settings:
```text
http://host.docker.internal:11434/v1
```
Ollama must listen outside its own loopback interface:
```bash
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
This connects Odysseus in Docker to an Ollama server that is already running on
your host machine; it does not start Ollama inside the container.
`host.docker.internal` is Docker's hostname for the host machine from inside the
container. Cookbook **Serve** is a separate workflow for serving downloaded
models through Odysseus/llama.cpp, so Windows users with an existing Ollama
install usually only need to add the endpoint in Settings.
**Useful checks.**
```bash
docker compose ps
docker compose logs --tail=120 odysseus
docker compose logs odysseus | grep -E 'ChromaDB|MemoryVectorStore|DEGRADED'
```
**macOS details.** `start-macos.sh` installs Homebrew deps, creates the venv,
runs setup, and starts uvicorn on port `7860` because AirPlay often holds
`7000`. It uses llama.cpp/Ollama for Metal. vLLM/SGLang are CUDA/ROCm-only and
do not run on macOS. MLX-only models are not served by Odysseus.
</details>
### Native Windows
**One-command launcher** (creates the venv, installs deps, runs setup, starts the
server; safe to re-run):
```powershell
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
```
Or do it by hand:
```powershell
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
py -3.11 -m venv venv
venv\Scripts\Activate.ps1
pip install -r requirements.txt
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
If `python` points at an older interpreter, use `py -3.12` (or another installed
3.11+ version) for the venv step.
**Requirements:** Python 3.11+. The core app (chat, agent, memory, documents,
email, calendar, deep research) runs fully native. For full **Cookbook** background
model downloads and the agent shell tool, also install
[Git for Windows](https://git-scm.com/download/win) (provides `bash.exe`).
Local GPU *serving* of vLLM/SGLang needs Linux/WSL2; for a local model on Windows,
[Ollama](https://ollama.com/download) is the easiest path — point Odysseus at
`http://localhost:11434/v1` in Settings.
Open `http://localhost:7000`, log in with the generated admin password,
and configure everything else inside **Settings**.
## Troubleshooting & Advanced Setup
### `chromadb-client` conflicts with embedded ChromaDB
If `chromadb-client` (the lightweight HTTP-only package) is installed alongside the full `chromadb` package, Odysseus starts but ChromaDB silently falls back to HTTP-only mode and fails.
**Fix:** uninstall `chromadb-client` and force-reinstall the full package:
```bash
./venv/bin/pip uninstall chromadb-client -y
./venv/bin/pip install --force-reinstall chromadb
```
### HTTPS + LAN/Tailscale exposure
To expose Odysseus on a local network or Tailscale with HTTPS:
1. Change the bind address to `0.0.0.0` in `.env` (`APP_BIND=0.0.0.0` or `ODYSSEUS_HOST=0.0.0.0`).
2. Generate a locally-trusted cert for your LAN/Tailscale IPs using [mkcert](https://github.com/FiloSottile/mkcert):
```bash
mkcert -install
mkcert -cert-file cert.pem -key-file key.pem 192.168.1.100 tailscale-ip
```
3. Run `uvicorn` with the generated certs:
```bash
python -m uvicorn app:app --host 0.0.0.0 --port 7000 --ssl-certfile=cert.pem --ssl-keyfile=key.pem
```
4. Install the `mkcert` CA on any other device you want to access Odysseus from (e.g., for iOS, email the `rootCA.pem` to yourself, install the profile, and trust it in Certificate Trust Settings).
### Optional Dependencies
`requirements-optional.txt` contains packages that unlock extra features. It is not installed by default.
| Package | Feature unlocked |
|---------|-----------------|
| `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
| `ddgs` | DuckDuckGo as a search provider option. |
| `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
| `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
### Faster, reproducible installs with uv (optional)
[uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
```bash
uv venv venv --python 3.13
uv pip install -r requirements.txt
# then continue as usual: python setup.py, uvicorn, ...
```
`requirements.txt` is intentionally unpinned, so two installs at different times can produce different package versions. If you want a reproducible environment (e.g. across your own machines, or to roll back after a bad upgrade), snapshot and restore exact versions with:
```bash
uv pip compile requirements.txt -o requirements.lock # snapshot current resolution
uv pip sync requirements.lock # reproduce it exactly later
```
`requirements.lock` is gitignored and platform-specific (compile it on the OS you deploy to). Regenerate it deliberately when you want to take upgrades. The plain `uv pip install -r requirements.txt` keeps following the unpinned requirements like pip does.
### Outlook / Office 365 email
Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
current limitation and the planned integration direction.
## Security Notes
Odysseus is a self-hosted workspace with powerful local tools: shell access, file uploads, model downloads, web research, email/calendar integrations, and API tokens. Treat it like an admin console.
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
- Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
- Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
- Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
- Non-admin users do not get shell/Python/file read/write by default, and admin-only routes/tools such as MCP management, API tokens, webhooks, model/cookbook serving, backup/vault, and app settings are admin-gated. Other features are controlled by per-user privileges, so review each user's privileges before exposing a deployment.
- Rotate any API keys or tokens that were ever pasted into a shared chat, demo, screenshot, or log.
- If you enable API tokens or webhooks, create separate tokens per integration and delete unused ones.
- Prefer binding manual development runs to `127.0.0.1`; bind to `0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
- Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
### Private or proxied deployments
Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
1. Keep Odysseus on localhost, for example `127.0.0.1:7000`.
2. Terminate HTTPS at a trusted reverse proxy or private access gateway.
3. Put the authenticated Odysseus web/API entrypoint behind that layer.
4. Keep raw service and model ports internal-only.
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
`ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
Common internal-only ports from the default docs/compose setup:
| Port | Service |
|---|---|
| `7000` | Odysseus raw app port |
| `8080` | SearXNG |
| `8091` | ntfy |
| `8100` | ChromaDB host port for manual/compose access |
| `11434` | Ollama |
| `8000-8020` | Common local model/provider APIs |
A full hover-to-play tour lives on the landing page: [`docs/index.html`](docs/index.html).
## Contributing
Help is welcome. The best entry points are fresh-install testing, provider setup
bugs, mobile/editor polish, docs, and small focused refactors. See
[ROADMAP.md](ROADMAP.md) for the current help-wanted list.
## Configuration
Most setup is done inside the app with `/setup` or **Settings**. Use `.env`
for deployment-level defaults and secrets you want present before first boot.
Key settings:
Help is welcome. The best entry points are fresh-install testing, provider setup bugs, mobile/editor polish, docs, and small focused refactors. See [CONTRIBUTING.md](CONTRIBUTING.md) and [ROADMAP.md](ROADMAP.md).
| Variable | Default | Description |
|---|---|---|
| `LLM_HOST` | `localhost` | Your LLM server (e.g. `llm-host.local:8000`) |
| `LLM_HOSTS` | -- | Comma-separated list for model discovery |
| `OPENAI_API_KEY` | -- | Optional OpenAI key. Prefer adding providers in the app unless pre-seeding. |
| `SEARXNG_INSTANCE` | `http://localhost:8080` | SearXNG URL. Docker overrides this to `http://searxng:8080`. |
| `SEARXNG_SECRET` | generated on first Docker boot | Optional SearXNG cookie/CSRF secret. Leave blank unless you need to pin it. |
| `APP_BIND` | `127.0.0.1` | Docker Compose host bind address for the web UI. Use `0.0.0.0` only for intentional LAN/reverse-proxy access. |
| `APP_PORT` | `7000` | Docker Compose host port for the web UI. |
| `AUTH_ENABLED` | `true` | Enable/disable login |
| `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
| `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
| `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
| `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
| `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |
| `EMBEDDING_URL` | -- | OpenAI-compatible embeddings endpoint |
| `ODYSSEUS_CHAT_UPLOAD_MAX_BYTES` | `10485760` | Chat/agent attachment cap in bytes. Raise for larger local PDFs or text documents. |
| `ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES` | `104857600` | Gallery image upload cap in bytes (100 MB). |
| `ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES` | `26214400` | Gallery transform input cap in bytes (25 MB). |
| `ODYSSEUS_MEMORY_IMPORT_MAX_BYTES` | `10485760` | Memory import file cap in bytes (10 MB). |
| `ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES` | `26214400` | Personal document upload cap in bytes (25 MB). |
| `ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES` | `26214400` | Email compose attachment cap in bytes (25 MB). |
| `ODYSSEUS_STT_MAX_AUDIO_BYTES` | `26214400` | Speech-to-text audio cap in bytes (25 MB). |
| `ODYSSEUS_ICS_MAX_BYTES` | `10485760` | Calendar `.ics` import cap in bytes (10 MB). |
## Security
All upload-limit vars are validated (must be a positive integer) and optional; an invalid value fails fast at startup.
### Built-in MCP servers (optional setup)
Odysseus auto-registers a few built-in MCP servers at startup. The npx-based ones (currently the browser server, `@playwright/mcp`) only start when their npm package is already in the local npx cache. If a package isn't cached, that server is skipped with a startup log message explaining what to do, so a fresh install does not block on a multi-minute npm download or hang if Playwright system deps are missing.
To enable the browser MCP (page navigation, screenshots, vision), run once:
```bash
npx -y @playwright/mcp@latest --version
```
That installs `@playwright/mcp` plus Playwright (~300MB total). Restart Odysseus and the server will register at startup.
## Architecture
```
app.py # FastAPI entry point
core/ auth, database, middleware, constants
src/ llm_core, agent_loop, agent_tools, chat_processor, search/
routes/ chat, session, document, memory, model … endpoints
services/ docs, memory, search, hwfit (Cookbook) …
static/ index.html + app.js + style.css + js/ (modular front-end)
docs/ landing page (index.html) + preview clips
```
## Data
All user data lives in `data/` (gitignored): `app.db` (sessions, messages, documents),
`memory.json`, `presets.json`, `uploads/`, `personal_docs/`, `chroma/`, `settings.json`.
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly. Deployment details are in the [setup guide](docs/setup.md#security-notes).
## Star History
@@ -478,19 +72,5 @@ All user data lives in `data/` (gitignored): `app.db` (sessions, messages, docum
</a>
## License
AGPL-3.0-or-later -- see [LICENSE](LICENSE) and [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md).
```
|
|||
|||||
| | | |||||||
)_) )_) )_) ~|~
)___))___))___)\ |
)____)____)_____)\\|
_____|____|____|_____\\\__
\ /
~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~
~^~ all aboard! ~^~
~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~~^~^~
```
AGPL-3.0-or-later -- see [LICENSE](LICENSE) and [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md).
app.py
+78 -35
View File
@@ -1,6 +1,17 @@
# app.py — slim orchestrator
import mimetypes
import os
import sys
import asyncio
# On Windows, asyncio.create_subprocess_exec/shell require the ProactorEventLoop.
# When started via `python -m uvicorn` from a terminal, uvicorn sets this
# automatically. But the VS Code debugger (and other non-uvicorn entrypoints)
# use the default SelectorEventLoop, which raises NotImplementedError on any
# subprocess call. Force ProactorEventLoop here so the right loop is always
# used, regardless of how the process is launched.
if sys.platform == "win32":
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
def register_static_mime_types() -> None:
@@ -38,12 +49,12 @@ load_dotenv(encoding="utf-8-sig")
import asyncio
import logging
import secrets
from datetime import datetime
from datetime import datetime, timezone
from typing import Dict
from contextlib import asynccontextmanager
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import JSONResponse, FileResponse, HTMLResponse
from fastapi.responses import JSONResponse, FileResponse
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from starlette.middleware.base import BaseHTTPMiddleware
@@ -64,15 +75,42 @@ from core.exceptions import (
import bcrypt as _bcrypt
from src.app_helpers import abs_join
from src.app_helpers import abs_join, serve_html_with_nonce
from src.generated_images import GENERATED_IMAGE_HEADERS, resolve_generated_image_path
from starlette.responses import RedirectResponse
# ========= LOGGING =========
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
)
import logging.handlers
from core.constants import DATA_DIR
_root_logger = logging.getLogger()
_root_logger.setLevel(logging.INFO)
_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# Clear existing handlers to avoid duplicates
for _h in list(_root_logger.handlers):
_root_logger.removeHandler(_h)
_console_h = logging.StreamHandler()
_console_h.setFormatter(_formatter)
_root_logger.addHandler(_console_h)
try:
_log_dir = os.path.join(DATA_DIR, "logs")
os.makedirs(_log_dir, exist_ok=True)
_log_file = os.path.join(_log_dir, "app.log")
# RotatingFileHandler is not multi-process safe (e.g. if uvicorn is run with --workers N).
# Odysseus is single-process by convention, so this is acceptable, but be aware that
# concurrent log rotation issues can arise if multiple workers are configured.
_file_h = logging.handlers.RotatingFileHandler(
_log_file, maxBytes=5 * 1024 * 1024, backupCount=3, encoding="utf-8"
)
_file_h.setFormatter(_formatter)
_root_logger.addHandler(_file_h)
except Exception as e:
_root_logger.warning(f"Failed to initialize file logging handler (falling back to console-only): {e}")
logger = logging.getLogger(__name__)
# ========= APP =========
@@ -86,12 +124,13 @@ app = FastAPI(
)
# ========= CORS =========
CORS_ALLOW_METHODS = ["GET", "POST", "PUT", "PATCH", "DELETE"]
allowed_origins = os.getenv("ALLOWED_ORIGINS", "http://localhost,http://127.0.0.1").split(",")
app.add_middleware(
CORSMiddleware,
allow_origins=allowed_origins,
allow_credentials=True,
allow_methods=["GET", "POST", "PUT", "DELETE"],
allow_methods=CORS_ALLOW_METHODS,
allow_headers=[
"Accept",
"Authorization",
@@ -140,6 +179,7 @@ _TIMEOUT_EXEMPT_PREFIXES = (
"/api/cookbook/setup", # remote pacman/apt installs
"/api/upload", # large files
"/api/image", # diffusion proxies (inpaint/harmonize/upscale/etc.) — own 120s httpx timeout
"/api/memory/audit", # retains own 120s LLM inactivity timeout
)
@@ -288,7 +328,7 @@ if AUTH_ENABLED:
# (no admin cookie available in that context). Restricted to
# loopback clients + matching token to keep it locked down.
try:
from core.middleware import INTERNAL_TOOL_HEADER, INTERNAL_TOOL_TOKEN as _ITT
from core.middleware import INTERNAL_TOOL_HEADER, INTERNAL_TOOL_TOKEN as _ITT, INTERNAL_TOOL_USER
_hdr = request.headers.get(INTERNAL_TOOL_HEADER)
if _hdr and secrets.compare_digest(_hdr, _ITT) and _is_trusted_loopback(request):
# Impersonation: when the agent's loopback call sets
@@ -300,11 +340,11 @@ if AUTH_ENABLED:
if _impersonate and _impersonate in getattr(_auth_mgr, "users", {}):
request.state.current_user = _impersonate
else:
request.state.current_user = "internal-tool"
request.state.current_user = INTERNAL_TOOL_USER
request.state.api_token = False
return await call_next(request)
except Exception:
pass
except Exception as _e:
logger.warning("Internal tool auth header check failed", exc_info=_e)
# Allow DIRECT localhost requests (internal service calls from
# heartbeats etc.). Tunnel/proxy-forwarded requests are excluded by
# _is_trusted_loopback so LOCALHOST_BYPASS can't be abused over a
@@ -357,11 +397,10 @@ if AUTH_ENABLED:
_db.close()
try:
await _asyncio.to_thread(_do)
except Exception:
pass
except Exception as _e:
logger.debug("Failed to update token last_used_at", exc_info=_e)
_asyncio.create_task(_touch_last_used(matched_id))
# Keep bearer-token callers out of normal cookie/user
# routes. API-aware routes can read api_token_owner.
request.state.current_user = "api"
request.state.api_token = True
request.state.api_token_id = matched_id
@@ -410,7 +449,7 @@ class _RevalidatingStatic(StaticFiles):
return resp
app.mount("/static", _RevalidatingStatic(directory="static"), name="static")
app.mount("/static", _RevalidatingStatic(directory=STATIC_DIR), name="static")
# ========= GENERATED IMAGES =========
@app.get("/api/generated-image/{filename}")
@@ -436,8 +475,8 @@ async def serve_generated_image(filename: str, request: Request):
_db.close()
except HTTPException:
raise
except Exception:
pass
except Exception as _e:
logger.warning("Image ownership verification failed for %r", filename, exc_info=_e)
ext = filename.rsplit('.', 1)[-1].lower()
mime = {
"png": "image/png", "jpg": "image/jpeg", "jpeg": "image/jpeg",
@@ -500,6 +539,7 @@ memory_vector = components.get("memory_vector")
upload_handler = components["upload_handler"]
app.state.upload_handler = upload_handler
personal_docs_mgr = components["personal_docs_manager"]
app.state.personal_docs_manager = personal_docs_mgr
api_key_manager = components["api_key_manager"]
preset_manager = components["preset_manager"]
chat_processor = components["chat_processor"]
@@ -645,7 +685,7 @@ from routes.signature_routes import setup_signature_routes
app.include_router(setup_signature_routes())
# Gallery (image library)
from routes.gallery_routes import setup_gallery_routes
from routes.gallery.gallery_routes import setup_gallery_routes
app.include_router(setup_gallery_routes())
# Persisted image-editor drafts (server-backed projects)
@@ -761,23 +801,17 @@ app.include_router(setup_companion_routes())
# ========= ROUTES (kept in app.py) =========
def _serve_html_with_nonce(request: Request, file_path: str) -> HTMLResponse:
"""Read an HTML file and inject the CSP nonce into inline <script> tags."""
with open(file_path, "r", encoding="utf-8") as f:
html = f.read()
nonce = getattr(request.state, "csp_nonce", "")
html = html.replace("{{CSP_NONCE}}", nonce)
return HTMLResponse(html)
@app.get("/")
async def serve_index(request: Request):
static_path = abs_join(BASE_DIR, "static/index.html")
if os.path.exists(static_path):
return _serve_html_with_nonce(request, static_path)
root_path = abs_join(BASE_DIR, "index.html")
if os.path.exists(root_path):
return _serve_html_with_nonce(request, root_path)
raise HTTPException(404, "index.html not found")
return serve_html_with_nonce(request, static_path)
# No static bundle — fall back to a root-level index.html if one is shipped.
# If neither exists, serve_html_with_nonce logs it and returns a generic 500:
# a missing index.html is a broken deployment (server fault), not a client
# "not found". This keeps the app-shell route consistent with the other
# bundled-template routes instead of mislabelling the fault as a 404.
return serve_html_with_nonce(request, abs_join(BASE_DIR, "index.html"))
@app.get("/notes")
async def serve_notes(request: Request):
@@ -818,13 +852,13 @@ async def serve_library(request: Request):
@app.get("/backgrounds")
async def serve_backgrounds(request: Request):
"""Sandbox page for prototyping background effects. No auth required."""
return _serve_html_with_nonce(request, abs_join(BASE_DIR, "static/backgrounds.html"))
return serve_html_with_nonce(request, abs_join(BASE_DIR, "static/backgrounds.html"))
@app.get("/login")
async def serve_login(request: Request):
if not AUTH_ENABLED:
return RedirectResponse(url="/", status_code=302)
return _serve_html_with_nonce(request, abs_join(BASE_DIR, "static/login.html"))
return serve_html_with_nonce(request, abs_join(BASE_DIR, "static/login.html"))
@app.get("/api/version")
async def get_version():
@@ -833,7 +867,7 @@ async def get_version():
@app.get("/api/health")
async def health_check() -> Dict[str, str]:
return {"status": "healthy", "timestamp": datetime.utcnow().isoformat()}
return {"status": "healthy", "timestamp": datetime.now(timezone.utc).isoformat()}
@app.get("/api/ready")
async def readiness_check() -> JSONResponse:
@@ -1143,3 +1177,12 @@ async def _shutdown_event():
except Exception as e:
logger.warning(f"MCP shutdown error: {e}")
logger.info("Application shutdown complete")
if __name__ == "__main__":
import uvicorn
bind_host = os.getenv("APP_BIND", "127.0.0.1")
bind_port = int(os.getenv("APP_PORT", "7000"))
uvicorn.run(app, host=bind_host, port=bind_port, log_level="info")
build-windows-portable.ps1
+72
View File
@@ -0,0 +1,72 @@
#Requires -Version 5.1
<#
Build a portable Windows distribution for Odysseus.
Output layout:
dist\Odysseus\Odysseus.exe
dist\Odysseus\static\...
dist\Odysseus\scripts\...
dist\Odysseus\mcp_servers\...
dist\Odysseus\services\hwfit\data\...
The app then keeps using its normal filesystem layout when frozen.
Usage:
powershell -ExecutionPolicy Bypass -File .\build-windows-portable.ps1
#>
$ErrorActionPreference = "Stop"
Set-Location -Path $PSScriptRoot
function Write-Step($msg) { Write-Host ""; Write-Host ("==> " + $msg) -ForegroundColor Cyan }
function Fail($msg) {
Write-Host ""
Write-Host ("ERROR: " + $msg) -ForegroundColor Red
exit 1
}
Write-Step "Checking for Python"
$pyExe = $null
if (Test-Path ".\.venv\Scripts\python.exe") {
$pyExe = (Resolve-Path ".\.venv\Scripts\python.exe").Path
} else {
foreach ($c in @("py", "python")) {
$cmd = Get-Command $c -ErrorAction SilentlyContinue
if ($cmd) { $pyExe = $cmd.Source; break }
}
if ($pyExe -like "*WindowsApps*python.exe") {
$pyCmd = Get-Command py -ErrorAction SilentlyContinue
if ($pyCmd) {
$pyExe = $pyCmd.Source
}
}
}
if (-not $pyExe) {
Fail "Python not found on PATH. Install Python 3.11+ first."
}
Write-Host ("Using Python: " + $pyExe)
Write-Step "Installing build dependencies"
& $pyExe -m pip install --upgrade pip --quiet
& $pyExe -m pip install -r requirements.txt pyinstaller pystray Pillow
if ($LASTEXITCODE -ne 0) { Fail "Dependency install failed." }
Write-Step "Building portable exe bundle"
Remove-Item -Recurse -Force build, dist -ErrorAction SilentlyContinue
$dataArgs = @(
"--add-data", "static;static",
"--add-data", "scripts;scripts",
"--add-data", "mcp_servers;mcp_servers",
"--add-data", "services/hwfit/data;services/hwfit/data",
"--add-data", "config;config",
"--add-data", ".env.example;.env.example"
)
& $pyExe -m PyInstaller --noconfirm --clean --onedir --noconsole --icon=static/icon.ico --name Odysseus @dataArgs launcher.py
if ($LASTEXITCODE -ne 0) { Fail "PyInstaller build failed." }
Write-Host ""
Write-Host "Build complete." -ForegroundColor Green
Write-Host "Portable app folder: $PSScriptRoot\dist\Odysseus" -ForegroundColor Green
Write-Host "Distribute the whole folder (or zip it) so static assets and scripts stay with the exe." -ForegroundColor Green
companion/routes.py
+17 -3
View File
@@ -5,8 +5,9 @@ offers and pair to it, without duplicating any LLM logic.
Auth is enforced globally by AuthMiddleware (app.py), so reaching a handler here
means the caller is authenticated by either a cookie session or a Bearer `ody_`
API token. The read endpoints (ping/info/models) accept either; the pairing
endpoints are admin-cookie only.
API token. Ping/info accept either credential type, models requires a chat-
scoped API token for bearer callers, and the pairing endpoints are admin-cookie
only.
Pairing CSRF posture: minting happens ONLY on POST. The session cookie is
SameSite=Lax (routes/auth_routes.py), which a browser does not send on a
@@ -18,7 +19,7 @@ on a GET would be unsafe (Lax cookies ride top-level GET navigations), so GET
import html
from fastapi import APIRouter, Request
from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import HTMLResponse
from core.middleware import require_admin
@@ -52,6 +53,18 @@ def owner_can_see(row_owner, owner) -> bool:
return row_owner is None or row_owner == owner
def require_models_scope(request: Request) -> None:
"""Require the companion chat scope for bearer-token model inventory."""
if not getattr(request.state, "api_token", False):
return
scopes = getattr(request.state, "api_token_scopes", None) or []
if isinstance(scopes, str):
scopes = [scope.strip() for scope in scopes.split(",")]
scope_set = {str(scope).strip() for scope in scopes if str(scope).strip()}
if _pairing.COMPANION_SCOPE not in scope_set:
raise HTTPException(403, "API token requires chat scope")
def mint_pairing_token(owner: str, invalidate=None) -> tuple[str, str]:
"""Mint a pairing token AND invalidate the auth middleware's in-memory token
cache, so the new token is accepted on the very next request without a server
@@ -103,6 +116,7 @@ def setup_companion_routes() -> APIRouter:
rows -- the same rule as owner_filter. Read-only; never returns api_key
material.
"""
require_models_scope(request)
import json as _json
from core.database import SessionLocal, ModelEndpoint
core/auth.py
+107 -18
View File
@@ -3,6 +3,7 @@ Authentication module — multi-user password hashing, session tokens, config pe
Config stored in data/auth.json. Uses bcrypt directly.
"""
import enum
import json
import os
import secrets
@@ -19,6 +20,7 @@ logger = logging.getLogger(__name__)
from core.atomic_io import atomic_write_json as _atomic_write_json # noqa: E402
from core.middleware import INTERNAL_TOOL_USER # noqa: E402
DEFAULT_PRIVILEGES = {
"can_use_agent": True,
@@ -46,7 +48,7 @@ ADMIN_PRIVILEGES["allowed_models_restricted"] = False
# backwards for this sentinel.
ADMIN_PRIVILEGES["block_all_models"] = False
from src.constants import AUTH_FILE
from src.constants import AUTH_FILE, PASSWORD_MIN_LENGTH
DEFAULT_AUTH_PATH = AUTH_FILE
TOKEN_TTL = 60 * 60 * 24 * 7 # 7 days
@@ -64,7 +66,7 @@ TOKEN_TTL = 60 * 60 * 24 * 7 # 7 days
# of those names would be denied an assistant and inconsistently owner-scoped.
# Refuse to create or rename into any of them so the sentinels can't be
# impersonated. (Keep this in sync with that synthetic-owner set.)
RESERVED_USERNAMES = frozenset({"internal-tool", "api", "demo", "system"})
RESERVED_USERNAMES = frozenset({INTERNAL_TOOL_USER, "api", "demo", "system"})
def normalize_known_username(users: Dict[str, Any], username: str | None) -> Optional[str]:
@@ -83,6 +85,15 @@ def _verify_password(password: str, hashed: str) -> bool:
return bcrypt.checkpw(password.encode("utf-8"), hashed.encode("utf-8"))
class SetAdminResult(enum.Enum):
"""Outcome of AuthManager.set_admin, so callers can map each case to a
precise response instead of guessing from a bare bool."""
OK = "ok"
USER_NOT_FOUND = "user_not_found"
NOT_AUTHORIZED = "not_authorized" # requester is not an admin
LAST_ADMIN = "last_admin" # would remove the last remaining admin
class AuthManager:
"""Manages multi-user password + session-token auth system."""
@@ -165,16 +176,17 @@ class AuthManager:
)
old_user = "admin"
old_hash = self._config["password_hash"]
self._config = {
"users": {
old_user: {
"password_hash": old_hash,
"created": time.time(),
"is_admin": True,
with self._config_lock:
self._config = {
"users": {
old_user: {
"password_hash": old_hash,
"created": time.time(),
"is_admin": True,
}
}
}
}
self._save()
self._save()
logger.info(f"Migrated single-user auth to multi-user (admin: {old_user})")
def _drop_reserved_loaded_users(self):
@@ -193,8 +205,9 @@ class AuthManager:
continue
normalized[key] = data
if removed or normalized != users:
self._config["users"] = normalized
self._save()
with self._config_lock:
self._config["users"] = normalized
self._save()
if removed:
logger.warning(
"Removed reserved username(s) from auth config: %s",
@@ -233,6 +246,15 @@ class AuthManager:
def is_configured(self) -> bool:
return len(self.users) > 0
def policy(self) -> dict:
"""Return public auth policy constants for the frontend."""
return {
"password_min_length": PASSWORD_MIN_LENGTH,
"reserved_usernames": sorted(RESERVED_USERNAMES),
"signup_enabled": self.signup_enabled,
"session_days": TOKEN_TTL // 86400,
}
# ------------------------------------------------------------------
# Account management
# ------------------------------------------------------------------
@@ -387,6 +409,69 @@ class AuthManager:
logger.info(f"Updated privileges for '{username}': {current}")
return True
def set_admin(self, username: str, is_admin: bool,
requesting_user: str) -> SetAdminResult:
"""Promote/demote an existing user to/from admin. Admin only.
Refuses to remove the last remaining admin so the instance can never
be locked out of admin access; self-demotion is allowed as long as
another admin remains. Admin status is re-checked live on every
request, so unlike delete/rename no session or token revocation is
needed — a demoted admin simply fails the next is_admin() gate.
Promotion stashes the user's current privilege map and demotion
restores it, so a temporary admin stint can't silently broaden a
user's non-admin access; users without a stash (created as admin,
or promoted before stashing existed) demote to DEFAULT_PRIVILEGES.
Counting admins and flipping the flag happen in one critical section
so two concurrent demotions can't race the admin count to zero.
"""
username = (username or "").strip().lower()
requesting_user = (requesting_user or "").strip().lower()
is_admin = bool(is_admin)
with self._config_lock:
target = self._config.get("users", {}).get(username)
if target is None:
return SetAdminResult.USER_NOT_FOUND
if not self.users.get(requesting_user, {}).get("is_admin"):
return SetAdminResult.NOT_AUTHORIZED
currently_admin = bool(target.get("is_admin"))
if currently_admin == is_admin:
return SetAdminResult.OK # no-op; leave privileges untouched
if currently_admin and not is_admin:
admin_count = sum(1 for d in self.users.values() if d.get("is_admin"))
if admin_count <= 1:
return SetAdminResult.LAST_ADMIN
# Write order matters for lock-free readers: get_privileges()
# reads without _config_lock and trusts is_admin, so the admin
# flag must be flipped while the stored map is safe to expose —
# before writing admin privileges on promote, after restoring
# the pre-admin map on demote.
if is_admin:
target["is_admin"] = True
# Stash the pre-admin map so a later demotion can restore it.
# While is_admin is set the stored map is inert: get_privileges
# short-circuits to ADMIN_PRIVILEGES and set_privileges refuses
# admins, so only set_admin ever touches the stash.
target["privileges_before_admin"] = dict(
target.get("privileges") or DEFAULT_PRIVILEGES
)
target["privileges"] = dict(ADMIN_PRIVILEGES)
else:
# Restore the stashed pre-admin map. Fall back to defaults for
# users created as admins (their stored map is ADMIN_PRIVILEGES,
# which must not leak past demotion — e.g. can_use_bash) and
# for admins promoted before the stash existed.
target["privileges"] = dict(
target.pop("privileges_before_admin", None)
or DEFAULT_PRIVILEGES
)
target["is_admin"] = False
self._save()
logger.info("Set is_admin=%s for '%s' (by '%s')", is_admin, username, requesting_user)
return SetAdminResult.OK
def change_password(self, username: str, current_password: str, new_password: str) -> bool:
username = username.strip().lower()
if username not in self.users:
@@ -500,16 +585,20 @@ class AuthManager:
return None
return self.create_session_trusted(username)
def create_session_trusted(self, username: str) -> str:
def create_session_trusted(self, username: str) -> Optional[str]:
"""Issue a session token for an already-verified user.
Call only after verify_password (and TOTP if enabled) have passed."""
username = username.strip().lower()
token = secrets.token_hex(32)
with self._sessions_lock:
self._sessions[token] = {
"username": username,
"expiry": time.time() + TOKEN_TTL,
}
with self._config_lock:
if username not in self.users:
logger.warning("Refused to issue session for missing user '%s'", username)
return None
with self._sessions_lock:
self._sessions[token] = {
"username": username,
"expiry": time.time() + TOKEN_TTL,
}
self._save_sessions()
return token
core/database.py
+49 -2
View File
@@ -2,12 +2,15 @@ import os
import logging
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
from sqlalchemy import event, create_engine, Column, String, Text, Boolean, DateTime, Integer, ForeignKey, JSON, Index, func, text
from sqlalchemy.engine import Engine
from sqlalchemy.types import TypeDecorator
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy.orm import relationship, sessionmaker, backref
from src.runtime_paths import get_app_root
logger = logging.getLogger(__name__)
# Create base class for declarative models
@@ -29,9 +32,26 @@ class TimestampMixin:
def updated_at(cls):
return Column(DateTime, default=utcnow_naive, onupdate=utcnow_naive, nullable=False)
# Get database URL from environment, default to SQLite in DATA_DIR
# Ensure the writable data directory exists before SQLite connects.
from src.constants import DATA_DIR, AUTH_FILE, MEMORY_FILE, USER_PREFS_FILE, SETTINGS_FILE
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{DATA_DIR}/app.db")
Path(DATA_DIR).mkdir(parents=True, exist_ok=True)
def _default_database_url() -> str:
return f"sqlite:///{Path(DATA_DIR) / 'app.db'}"
def _normalize_sqlite_url(url: str) -> str:
if not url.startswith("sqlite:///"):
return url
db_path = url.replace("sqlite:///", "", 1)
if db_path == ":memory:" or os.path.isabs(db_path):
return url
return f"sqlite:///{(Path(get_app_root()) / db_path).resolve().as_posix()}"
# Get database URL from environment, default to SQLite in DATA_DIR
DATABASE_URL = _normalize_sqlite_url(os.getenv("DATABASE_URL", _default_database_url()))
# Create engine
engine = create_engine(
@@ -324,6 +344,13 @@ class EmailAccount(TimestampMixin, Base):
smtp_password = Column(String, default="")
from_address = Column(String, default="")
display_name = Column(String, nullable=True) # "Hriday Ranka" — used in From: header
# OAuth2 (Google / Google Workspace). Tokens stored encrypted via secret_storage.
oauth_provider = Column(String, nullable=True) # "google" or None
oauth_access_token = Column(String, nullable=True) # encrypted
oauth_refresh_token = Column(String, nullable=True) # encrypted
oauth_token_expiry = Column(String, nullable=True) # unix timestamp string
__table_args__ = (
Index('ix_email_accounts_owner_default', 'owner', 'is_default'),
@@ -1427,6 +1454,25 @@ def _migrate_add_task_automation_columns():
except Exception as e:
logging.getLogger(__name__).warning(f"task automation migration: {e}")
def _migrate_add_email_oauth_columns():
"""Add Google OAuth and display_name columns to email_accounts if missing."""
try:
with engine.connect() as conn:
cols = [r[1] for r in conn.execute(text("PRAGMA table_info(email_accounts)"))]
for col, typedef in [
("oauth_provider", "TEXT"),
("oauth_access_token", "TEXT"),
("oauth_refresh_token", "TEXT"),
("oauth_token_expiry", "TEXT"),
("display_name", "TEXT"),
]:
if col not in cols:
conn.execute(text(f"ALTER TABLE email_accounts ADD COLUMN {col} {typedef}"))
conn.commit()
except Exception as e:
logging.getLogger(__name__).warning(f"email oauth columns migration: {e}")
def _migrate_add_oauth_config():
"""Add oauth_config column to mcp_servers table if missing."""
try:
@@ -1771,6 +1817,7 @@ def init_db():
_migrate_add_tidy_verdict()
_migrate_add_doc_source_email_cols()
_migrate_add_oauth_config()
_migrate_add_email_oauth_columns()
_migrate_add_task_automation_columns()
_migrate_add_disabled_tools()
_migrate_add_mcp_oauth_tokens_column()
core/exceptions.py
+1 -1
View File
@@ -1,4 +1,4 @@
# src/exceptions.py
# core/exceptions.py
"""Custom exceptions for the application."""
class SessionNotFoundError(Exception):
core/log_safety.py
+27
View File
@@ -0,0 +1,27 @@
"""Helpers for keeping sensitive data out of logs.
Endpoint URLs configured by admins can embed credentials in the userinfo
(``https://user:pass@host``) or query string (``?api_key=...``). Logging them
raw leaks those secrets, so route/diagnostic logs run URLs through
``redact_url`` first. Reconstructing the URL without userinfo/query/fragment
also doubles as a sanitizer barrier for CodeQL's clear-text-logging query.
"""
from urllib.parse import urlparse, urlunparse
def redact_url(url: str) -> str:
"""Return a URL safe for logs by removing userinfo and query/fragment.
Keeps scheme, host, port and path so logs stay useful for debugging.
"""
try:
parsed = urlparse(url or "")
host = parsed.hostname or ""
if ":" in host: # IPv6 literal — re-bracket so host:port stays unambiguous
host = f"[{host}]"
if parsed.port:
host = f"{host}:{parsed.port}"
return urlunparse((parsed.scheme, host, parsed.path, "", "", ""))
except Exception:
return "<endpoint>"
core/middleware.py
+6 -7
View File
@@ -15,6 +15,8 @@ from starlette.responses import Response
# same value from this module. Never persisted or exposed externally.
INTERNAL_TOOL_TOKEN = os.environ.get("ODYSSEUS_INTERNAL_TOKEN") or secrets.token_hex(32)
INTERNAL_TOOL_HEADER = "X-Odysseus-Internal-Token"
# Pseudo-username on in-process tool-loopback requests; require_admin trusts it and it is reserved.
INTERNAL_TOOL_USER = "internal-tool"
def is_cors_preflight(method: str, headers) -> bool:
@@ -39,7 +41,7 @@ def require_admin(request: Request):
hdr = request.headers.get(INTERNAL_TOOL_HEADER)
if hdr and secrets.compare_digest(hdr, INTERNAL_TOOL_TOKEN):
return
if getattr(request.state, "current_user", None) == "internal-tool":
if getattr(request.state, "current_user", None) == INTERNAL_TOOL_USER:
return
except Exception:
pass
@@ -65,10 +67,9 @@ class SecurityHeadersMiddleware(BaseHTTPMiddleware):
response = await call_next(request)
path = request.url.path
# Tool render endpoints are served inside iframes — allow framing by self
# Tool render endpoints
is_tool_render = path.startswith("/api/tools/") and path.endswith("/render")
# PDF previews are embedded by the in-app document library. Keep the
# exception route-scoped so normal app pages remain unframeable.
# Document library PDF preview endpoint
is_document_pdf_preview = path.startswith("/api/document/") and path.endswith("/render-pdf")
# Visual report pages are self-contained HTML — need inline scripts + external images
is_report = path.startswith("/api/research/report/")
@@ -95,9 +96,7 @@ class SecurityHeadersMiddleware(BaseHTTPMiddleware):
"frame-ancestors 'none'"
)
elif is_tool_render:
# Tool iframe content: skip all framing headers — the iframe's
# sandbox="allow-scripts" attribute provides isolation.
# Don't overwrite the route's own restrictive CSP either.
# Skip framing headers for tools.
pass
elif is_document_pdf_preview:
response.headers["X-Frame-Options"] = "SAMEORIGIN"
core/session_manager.py
+12 -1
View File
@@ -40,7 +40,18 @@ def _parse_msg_content(raw):
if isinstance(raw, str) and raw.startswith('[{') and '"type"' in raw:
try:
parsed = json.loads(raw)
if isinstance(parsed, list) and all(isinstance(p, dict) for p in parsed):
# Only treat as serialized multimodal content when EVERY element is
# a dict whose "type" is a recognized content-block kind. Otherwise a
# plain text message that merely *looks* like a JSON array of objects
# (e.g. a user pasting an API schema/sample with a "type" field) was
# silently parsed back into a list, destroying the original string.
_BLOCK_TYPES = {
"text", "image", "image_url", "audio", "input_audio",
"input_image", "document", "file",
}
if (isinstance(parsed, list) and parsed
and all(isinstance(p, dict) and p.get("type") in _BLOCK_TYPES
for p in parsed)):
return parsed
except (json.JSONDecodeError, ValueError):
pass
docker-compose.gpu-amd.yml
+21 -5
View File
@@ -16,18 +16,26 @@ services:
ports:
- "${APP_BIND:-127.0.0.1}:${APP_PORT:-7000}:7000"
volumes:
- ./data:/app/data:z
- ./logs:/app/logs:z
- ${APP_DATA_DIR:-./data}:/app/data:z
- ${APP_LOGS_DIR:-./logs}:/app/logs:z
# Cookbook remote-server SSH identity. Odysseus can generate a key here;
# add the shown public key to each remote server's authorized_keys.
- ./data/ssh:/app/.ssh:z
- ${APP_DATA_DIR:-./data}/ssh:/app/.ssh:z
# Cookbook local model cache. Inside Docker, "Local" means the Odysseus
# container, so persist its HuggingFace cache under ./data/huggingface.
- ./data/huggingface:/app/.cache/huggingface:z
- ${APP_DATA_DIR:-./data}/huggingface:/app/.cache/huggingface:z
# Cookbook-installed Python CLIs/packages (vLLM, llama-cpp-python, etc.)
# land under /app/.local for the odysseus user. Persist them so a
# container recreate does not silently remove installed serve engines.
- ./data/local:/app/.local:z
- ${APP_DATA_DIR:-./data}/local:/app/.local:z
# Docker socket — lets Cookbook launch commands like
# `docker exec ollama-rocm ollama show <tag>` reach the host's
# Docker daemon (and sibling containers like ollama-rocm /
# ollama-test). The in-container user needs to be in the
# socket's owning group — see `group_add` below; the GID
# there must match the host's `docker` group (defaults to 963
# on Debian, 999 on Ubuntu — override via env if yours differs).
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
# Lets the container reach local services on the Docker host, including
# Ollama at http://host.docker.internal:11434.
@@ -60,6 +68,13 @@ services:
- ODYSSEUS_INPROCESS_TASKS=${ODYSSEUS_INPROCESS_TASKS:-1}
- ODYSSEUS_SCRIPT_HOST=${ODYSSEUS_SCRIPT_HOST:-localhost}
- ODYSSEUS_CHAT_UPLOAD_MAX_BYTES=${ODYSSEUS_CHAT_UPLOAD_MAX_BYTES:-10485760}
- ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES=${ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES:-104857600}
- ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES=${ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_MEMORY_IMPORT_MAX_BYTES=${ODYSSEUS_MEMORY_IMPORT_MAX_BYTES:-10485760}
- ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES=${ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES=${ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_STT_MAX_AUDIO_BYTES=${ODYSSEUS_STT_MAX_AUDIO_BYTES:-26214400}
- ODYSSEUS_ICS_MAX_BYTES=${ODYSSEUS_ICS_MAX_BYTES:-10485760}
- DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
- GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
- GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}
@@ -86,6 +101,7 @@ services:
- /dev/kfd
- /dev/dri
group_add:
- "${DOCKER_GID:-963}"
- video
- ${RENDER_GID:-render}
docker-compose.gpu-nvidia.yml
+22 -5
View File
@@ -15,18 +15,28 @@ services:
ports:
- "${APP_BIND:-127.0.0.1}:${APP_PORT:-7000}:7000"
volumes:
- ./data:/app/data:z
- ./logs:/app/logs:z
- ${APP_DATA_DIR:-./data}:/app/data:z
- ${APP_LOGS_DIR:-./logs}:/app/logs:z
# Cookbook remote-server SSH identity. Odysseus can generate a key here;
# add the shown public key to each remote server's authorized_keys.
- ./data/ssh:/app/.ssh:z
- ${APP_DATA_DIR:-./data}/ssh:/app/.ssh:z
# Cookbook local model cache. Inside Docker, "Local" means the Odysseus
# container, so persist its HuggingFace cache under ./data/huggingface.
- ./data/huggingface:/app/.cache/huggingface:z
- ${APP_DATA_DIR:-./data}/huggingface:/app/.cache/huggingface:z
# Cookbook-installed Python CLIs/packages (vLLM, llama-cpp-python, etc.)
# land under /app/.local for the odysseus user. Persist them so a
# container recreate does not silently remove installed serve engines.
- ./data/local:/app/.local:z
- ${APP_DATA_DIR:-./data}/local:/app/.local:z
# Docker socket — lets Cookbook launch commands like
# `docker exec ollama-rocm ollama show <tag>` reach the host's
# Docker daemon (and sibling containers like ollama-rocm /
# ollama-test). The in-container user needs to be in the
# socket's owning group — see `group_add` below; the GID
# there must match the host's `docker` group (defaults to 963
# on Debian, 999 on Ubuntu — override via env if yours differs).
- /var/run/docker.sock:/var/run/docker.sock
group_add:
- "${DOCKER_GID:-963}"
extra_hosts:
# Lets the container reach local services on the Docker host, including
# Ollama at http://host.docker.internal:11434.
@@ -59,6 +69,13 @@ services:
- ODYSSEUS_INPROCESS_TASKS=${ODYSSEUS_INPROCESS_TASKS:-1}
- ODYSSEUS_SCRIPT_HOST=${ODYSSEUS_SCRIPT_HOST:-localhost}
- ODYSSEUS_CHAT_UPLOAD_MAX_BYTES=${ODYSSEUS_CHAT_UPLOAD_MAX_BYTES:-10485760}
- ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES=${ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES:-104857600}
- ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES=${ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_MEMORY_IMPORT_MAX_BYTES=${ODYSSEUS_MEMORY_IMPORT_MAX_BYTES:-10485760}
- ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES=${ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES=${ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_STT_MAX_AUDIO_BYTES=${ODYSSEUS_STT_MAX_AUDIO_BYTES:-26214400}
- ODYSSEUS_ICS_MAX_BYTES=${ODYSSEUS_ICS_MAX_BYTES:-10485760}
- DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
- GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
- GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}
docker-compose.yml
+22 -5
View File
@@ -4,18 +4,28 @@ services:
ports:
- "${APP_BIND:-127.0.0.1}:${APP_PORT:-7000}:7000"
volumes:
- ./data:/app/data:z
- ./logs:/app/logs:z
- ${APP_DATA_DIR:-./data}:/app/data:z
- ${APP_LOGS_DIR:-./logs}:/app/logs:z
# Cookbook remote-server SSH identity. Odysseus can generate a key here;
# add the shown public key to each remote server's authorized_keys.
- ./data/ssh:/app/.ssh:z
- ${APP_DATA_DIR:-./data}/ssh:/app/.ssh:z
# Cookbook local model cache. Inside Docker, "Local" means the Odysseus
# container, so persist its HuggingFace cache under ./data/huggingface.
- ./data/huggingface:/app/.cache/huggingface:z
- ${APP_DATA_DIR:-./data}/huggingface:/app/.cache/huggingface:z
# Cookbook-installed Python CLIs/packages (vLLM, llama-cpp-python, etc.)
# land under /app/.local for the odysseus user. Persist them so a
# container recreate does not silently remove installed serve engines.
- ./data/local:/app/.local:z
- ${APP_DATA_DIR:-./data}/local:/app/.local:z
# Docker socket — lets Cookbook launch commands like
# `docker exec ollama-rocm ollama show <tag>` reach the host's
# Docker daemon (and sibling containers like ollama-rocm /
# ollama-test). The in-container user needs to be in the
# socket's owning group — see `group_add` below; the GID
# there must match the host's `docker` group (defaults to 963
# on Debian, 999 on Ubuntu — override via env if yours differs).
- /var/run/docker.sock:/var/run/docker.sock
group_add:
- "${DOCKER_GID:-963}"
extra_hosts:
# Lets the container reach local services on the Docker host, including
# Ollama at http://host.docker.internal:11434.
@@ -48,6 +58,13 @@ services:
- ODYSSEUS_INPROCESS_TASKS=${ODYSSEUS_INPROCESS_TASKS:-1}
- ODYSSEUS_SCRIPT_HOST=${ODYSSEUS_SCRIPT_HOST:-localhost}
- ODYSSEUS_CHAT_UPLOAD_MAX_BYTES=${ODYSSEUS_CHAT_UPLOAD_MAX_BYTES:-10485760}
- ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES=${ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES:-104857600}
- ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES=${ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_MEMORY_IMPORT_MAX_BYTES=${ODYSSEUS_MEMORY_IMPORT_MAX_BYTES:-10485760}
- ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES=${ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES=${ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES:-26214400}
- ODYSSEUS_STT_MAX_AUDIO_BYTES=${ODYSSEUS_STT_MAX_AUDIO_BYTES:-26214400}
- ODYSSEUS_ICS_MAX_BYTES=${ODYSSEUS_ICS_MAX_BYTES:-10485760}
- DATA_BRAVE_API_KEY=${DATA_BRAVE_API_KEY:-}
- GOOGLE_API_KEY=${GOOGLE_API_KEY:-}
- GOOGLE_PSE_CX=${GOOGLE_PSE_CX:-}
docker/build-realesrgan-wheels.sh
Executable
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Build patched wheels for Real-ESRGAN's unmaintained dependencies.
#
# basicsr / gfpgan / facexlib (xinntao, last released 2022) read their version
# in setup.py with:
#
# exec(compile(f.read(), version_file, 'exec'))
# return locals()['__version__']
#
# Python 3.13+ implements PEP 667: locals() inside a function returns an
# independent snapshot that exec() can no longer mutate, so the read raises
# `KeyError: '__version__'` and the sdist build fails. That is why the Cookbook
# "install realesrgan" button dies on the python:3.14 image. The packages have
# no fixed release, so we patch get_version() to exec into an explicit namespace
# dict (works on every Python) and build wheels from the patched source.
#
# Usage: build-realesrgan-wheels.sh [OUTPUT_DIR] (default: /wheels)
set -euo pipefail
OUT="${1:-/wheels}"
mkdir -p "$OUT"
work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT
cd "$work"
# Pinned to the versions Real-ESRGAN 0.3.0 resolves to.
SPECS="basicsr==1.4.2 gfpgan==1.3.8 facexlib==0.3.0"
for spec in $SPECS; do
name="${spec%%==*}"
ver="${spec##*==}"
# pip download builds metadata (and trips the same bug), so fetch the raw
# sdist URL from the PyPI JSON API instead.
url="$(python - "$name" "$ver" <<'PY'
import json, sys, urllib.request
name, ver = sys.argv[1], sys.argv[2]
data = json.load(urllib.request.urlopen(f"https://pypi.org/pypi/{name}/{ver}/json"))
for f in data["urls"]:
if f["packagetype"] == "sdist":
print(f["url"]); break
else:
sys.exit(f"no sdist found for {name}=={ver}")
PY
)"
echo ">> fetching ${name} ${ver}: ${url}"
curl -fsSL "$url" -o "${name}.tar.gz"
tar xzf "${name}.tar.gz"
done
echo ">> patching get_version()"
python - <<'PY'
import pathlib
old_exec = "exec(compile(f.read(), version_file, 'exec'))"
new_exec = "_ver_ns = {}\n exec(compile(f.read(), version_file, 'exec'), _ver_ns)"
old_ret = "return locals()['__version__']"
new_ret = "return _ver_ns['__version__']"
patched = 0
for setup in pathlib.Path(".").glob("*/setup.py"):
s = setup.read_text()
if old_exec in s and old_ret in s:
setup.write_text(s.replace(old_exec, new_exec).replace(old_ret, new_ret))
print(" patched", setup)
patched += 1
assert patched == 3, f"expected to patch 3 setup.py files, patched {patched}"
PY
echo ">> building wheels into ${OUT}"
pip wheel --no-deps -w "$OUT" ./basicsr-* ./gfpgan-* ./facexlib-*
ls -l "$OUT"
docker/entrypoint.sh
+74 -19
View File
@@ -13,6 +13,8 @@ set -e
PUID="${PUID:-1000}"
PGID="${PGID:-1000}"
GOSU_BIN="$(command -v gosu)"
PYTHON_BIN="$(command -v python)"
# Reuse an existing matching group/user if the host's UID/GID already
# corresponds to one in /etc/passwd (e.g. when the image is rebuilt
@@ -24,26 +26,78 @@ if ! getent passwd "$PUID" >/dev/null 2>&1; then
useradd -u "$PUID" -g "$PGID" -M -s /bin/sh -d /app odysseus
fi
# Repair ownership on every writable path the app touches at runtime.
#
# Bind-mounted dirs (/app/data, /app/logs) are the obvious ones, but
# the app ALSO writes inside the image's own source tree at runtime:
# - services/cache/{search,content}/* (search cache LRU)
# - services/search_analytics.json
# - services/search_engine_error.log
# - services/tts cache, etc.
# These dirs were created as root during `docker build`, so dropping
# to PUID:PGID would otherwise crash on the first import that tries
# to mkdir them. Chown the whole /app tree — fast (<1s on this size)
# and idempotent via the `-not -uid` filter so we only touch files
# that need fixing.
for dir in /app /app/data /app/logs; do
ODY_USER="$(getent passwd "$PUID" | cut -d: -f1)"
[ -z "$ODY_USER" ] && ODY_USER=odysseus
# Docker-socket group plumbing. When /var/run/docker.sock is bind-mounted
# (Cookbook uses docker exec to reach sibling containers), the socket is
# owned by root:<host docker gid>. Add the app user to that group and later
# call gosu by username so supplementary groups are retained.
DOCKER_SOCK="${DOCKER_SOCK:-/var/run/docker.sock}"
if [ -S "$DOCKER_SOCK" ]; then
SOCK_GID="$(stat -c '%g' "$DOCKER_SOCK" 2>/dev/null || echo '')"
if [ -n "$SOCK_GID" ] && [ "$SOCK_GID" != "0" ]; then
if ! getent group "$SOCK_GID" >/dev/null 2>&1; then
groupadd -g "$SOCK_GID" docker_host || true
fi
SOCK_GROUP="$(getent group "$SOCK_GID" | cut -d: -f1)"
if [ -n "$SOCK_GROUP" ]; then
usermod -aG "$SOCK_GROUP" "$ODY_USER" 2>/dev/null || true
fi
fi
fi
mount_root_for() {
awk -v target="$1" '$5 == target { print $4; exit }' /proc/self/mountinfo 2>/dev/null || true
}
is_broad_mount_root() {
case "$1" in
/|/home|/srv|/var|/usr|/opt|/tmp|/mnt|/media)
return 0
;;
esac
return 1
}
repair_tree_ownership() {
dir="$1"
if [ -d "$dir" ]; then
# `find ... -not -uid` keeps this O(touched-files), not
# O(everything), so terabyte-sized maildirs don't slow startup.
find "$dir" -not -uid "$PUID" -print0 2>/dev/null \
find "$dir" -xdev -not -uid "$PUID" -print0 2>/dev/null \
| xargs -0 -r chown "$PUID:$PGID" 2>/dev/null || true
fi
}
repair_app_tree_ownership() {
if [ -d /app ]; then
find /app -xdev \
\( -path /app/data -o -path /app/logs -o -path /app/.ssh -o -path /app/.cache -o -path /app/.local \) -prune \
-o -not -uid "$PUID" -print0 2>/dev/null \
| xargs -0 -r chown "$PUID:$PGID" 2>/dev/null || true
fi
}
repair_bind_mount_ownership() {
dir="$1"
if [ ! -d "$dir" ]; then
return
fi
mount_root="$(mount_root_for "$dir")"
if is_broad_mount_root "$mount_root"; then
echo "Skipping recursive ownership repair for $dir because it maps to broad host path $mount_root" >&2
chown "$PUID:$PGID" "$dir" 2>/dev/null || true
return
fi
repair_tree_ownership "$dir"
}
# Repair image-owned writable paths without walking into bind-mounted host
# trees, then repair the app-owned mount roots separately.
repair_app_tree_ownership
for dir in /app/data /app/logs /app/.ssh /app/.cache/huggingface /app/.local; do
repair_bind_mount_ownership "$dir"
done
# Cookbook installs vllm/etc. via `pip install --user`, which pulls
@@ -70,6 +124,7 @@ for cu in \
break
fi
done
# Disable the FlashInfer JIT sampler unconditionally — it is sampler-only
# and has no impact on the attention path, but requires nvcc + matching
# CUDA headers at startup. Without this, vLLM crashes with "Could not find
@@ -83,9 +138,9 @@ export PATH="/app/.local/bin:$PATH"
# Run first-time setup as the app user so data/ files get the right ownership.
# setup.py is idempotent — skips auth.json / .env if they already exist.
# || true so a setup failure never prevents the container from starting.
gosu "$PUID:$PGID" python /app/setup.py || true
"$GOSU_BIN" "$ODY_USER" "$PYTHON_BIN" /app/setup.py || true
# Drop root and run the actual app. `gosu` is preferred over `su` /
# `sudo` because it cleans up the process tree (no extra shell layer)
# so signals (SIGTERM from `docker stop`) reach uvicorn directly.
exec gosu "$PUID:$PGID" "$@"
exec "$GOSU_BIN" "$ODY_USER" "$@"
docs/backup-restore.md
+129
View File
@@ -0,0 +1,129 @@
# Backup & Restore
Odysseus keeps all of your state in the `data/` directory — the SQLite database
(`app.db`), the Fernet encryption key (`data/.app_key`), the vault, memory, RAG
indexes, personal documents, and uploads. The `scripts/odysseus-backup` tool
snapshots that directory into a single gzip tarball and restores it later.
Snapshots are safe to take while the app is running: SQLite databases are copied
through SQLite's own `.backup` API rather than a raw file copy, so an in-flight
write can't corrupt the snapshot.
> **A snapshot contains your secrets.** The tarball includes the Fernet
> encryption key (`data/.app_key`), the vault, sessions, and any stored
> provider/API tokens — so treat it like a password. Store backups somewhere
> private, never commit them to Git, and prefer an encrypted destination when
> copying them offsite.
## Quick start
Run the tool from the repository root:
```bash
# Create a snapshot → backups/odysseus-backup-<YYYYMMDD-HHMMSS>.tar.gz
./scripts/odysseus-backup snapshot
# List existing snapshots (most recent first)
./scripts/odysseus-backup list
# Check a tarball's integrity without extracting it
./scripts/odysseus-backup verify backups/odysseus-backup-20260101-120000.tar.gz
# Restore (destructive — see the warning below)
./scripts/odysseus-backup restore backups/odysseus-backup-20260101-120000.tar.gz --yes
```
The script depends only on the Python standard library, so any `python3` on your
`PATH` will run it — you don't need the app's virtualenv active.
Every command prints a JSON result. Add `--pretty` for indented output.
## Commands
### `snapshot`
Writes a `tar.gz` of `data/` to `backups/<timestamp>.tar.gz`.
| Flag | Effect |
| --- | --- |
| `--out PATH` | Write to a specific path instead of the default `backups/` location. Must be **outside** `data/`. |
| `--include-research` | Include `data/deep_research/` (skipped by default — research runs are large). |
| `--include-attachments` | Include `data/mail-attachments/` (skipped by default — cached IMAP extractions, re-derivable). |
By default the snapshot includes everything under `data/` **except**
`deep_research/` and `mail-attachments/`. Personal uploads and documents are
included.
```bash
# Snapshot straight to a mounted NAS path
./scripts/odysseus-backup snapshot --out /mnt/nas/odysseus-$(date +%F).tar.gz
# Full snapshot including research runs and mail attachments
./scripts/odysseus-backup snapshot --include-research --include-attachments
```
### `list`
Lists the tarballs in `backups/`, most recent first, with size and modification
time.
### `verify PATH`
Opens the tarball read-only and walks every member to confirm it is intact and
safe to restore. Nothing is extracted. Use this before relying on an old backup
or after copying one across machines.
### `restore PATH --yes`
Overwrites `data/` from a tarball.
> **Restore is destructive.** It replaces the current `data/` directory. `--yes`
> is required so a mistyped command can't wipe your live state.
Restore is not a blind delete: before extracting, the tool **renames your current
`data/` to `data.before-restore-<timestamp>`** in the repository root. If a
restore turns out to be wrong, your previous state is still there — delete the
restored `data/` and rename the stashed directory back. The restore path is also
validated entry-by-entry: archives containing absolute paths, `..` segments,
symlinks, or anything outside `data/` are rejected.
## Scheduling offsite backups
The tarball output composes cleanly with cron and any copy tool. For example, a
nightly snapshot copied offsite:
```cron
0 3 * * * cd /path/to/odysseus && ./scripts/odysseus-backup snapshot --out "/mnt/nas/odysseus-$(date +\%F).tar.gz"
```
Swap the `--out` target for `scp`, `rclone`, `s3cmd`, or similar to push the
snapshot to remote storage.
## Docker vs native installs
The tool reads `data/` and writes `backups/` relative to the repository root, so
where you run it matters:
- **Native installs** — run it from the repo root as shown above. `data/` and
`backups/` are both in the repo directory.
- **Docker** — `docker-compose.yml` bind-mounts the host's `./data` to
`/app/data`, so the live data is also present on the host. **Run the tool on
the host** from the repo root; the snapshot reads the bind-mounted `./data` and
writes to `./backups` on the host. Running it *inside* the container is not
recommended, because `backups/` is not a mounted volume and the tarball would
be lost when the container is recreated.
> **ChromaDB caveat (Docker only).** In the Docker setup, ChromaDB stores its
> vectors in a separate Compose-managed volume (declared as `chromadb-data`),
> **not** under `./data`. `odysseus-backup` therefore does not capture the Docker
> ChromaDB store. Back it up separately if you need it. Compose prefixes the
> volume with the project name, so find the real name first
> (`docker volume ls | grep chromadb`), then archive it — for example:
>
> ```bash
> docker run --rm -v <project>_chromadb-data:/data -v "$PWD":/backup \
> alpine tar czf /backup/chromadb.tar.gz -C /data .
> ```
>
> On native installs ChromaDB lives at `data/chroma/` and is included in the
> snapshot normally.
docs/chat.gif
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 3.0 MiB

docs/compare.gif
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 3.4 MiB

docs/document.gif
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

docs/notes.gif
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1003 KiB

docs/odysseus-browser.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 185 KiB

docs/odysseus-wordmark.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

docs/odysseus.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 79 KiB

docs/research.gif
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 2.5 MiB

docs/security-ci.md
+14 -9
View File
@@ -1,14 +1,16 @@
# Security CI guide
This project runs a set of automated security checks on every pull request and
on every push to `main`. This page explains what each one does, whether it can
This project runs a set of automated security checks on pull requests and
selected branch pushes. This page explains what each one does, whether it can
block a merge, and the few one-time settings you should turn on to get the full
benefit.
## What runs, and why
Each check lives in its own file under `.github/workflows/`. They run
automatically; you do not start them.
Most checks live in files under `.github/workflows/`. CodeQL is configured
through GitHub's code scanning default setup, so it appears as a dynamic GitHub
workflow instead of a checked-in workflow file. They run automatically; you do
not start them.
| Check | What it protects against | Blocks a merge? |
|---|---|---|
@@ -88,11 +90,14 @@ let the workflows run on one pull request first, then add them here.
2. Turn on **Dependency graph** (usually on by default for public repos) -- this
powers Dependency review and Dependabot.
3. Turn on **Dependabot alerts** and **Dependabot security updates**.
4. Under **Code scanning**, you have two ways to scan the app code with CodeQL:
- The included `codeql.yml` workflow already scans `main` and runs weekly.
- To also scan **pull requests** (recommended, since most contributions come
from forks), click **Set up -> Default** under Code scanning. GitHub then
runs CodeQL on pull requests for you, with no token limitations.
4. Under **Code scanning**, use **Set up -> Default** for CodeQL. GitHub then
runs CodeQL as a dynamic workflow without the fork-token limitations that
affect checked-in advanced workflows.
Do not also add a checked-in CodeQL workflow while default setup is enabled:
GitHub rejects advanced CodeQL uploads when default setup is active. If the
project later needs an advanced CodeQL workflow, disable default setup first
and keep only one CodeQL publishing path active.
## Keeping it current
docs/setup.md
+448
View File
@@ -0,0 +1,448 @@
# Odysseus Setup Guide
This page keeps the detailed install, deployment, troubleshooting, and configuration notes out of the front README.
## Quick Start
> **Branch note:** `dev` is the default branch and contains the latest development changes, but it may be unstable. For the more stable curated branch, use [`main`](https://github.com/pewdiepie-archdaemon/odysseus/tree/main).
Defaults work out of the box: clone, run, then configure models/search/email
inside **Settings**. Only edit `.env` for deployment-level overrides like
`APP_BIND`, `APP_PORT`, `AUTH_ENABLED`, `DATABASE_URL`, or a pre-seeded admin password.
On first setup, Odysseus creates an admin account (`admin` unless
`ODYSSEUS_ADMIN_USER` is set) and prints a temporary password in the terminal.
For Docker installs, the same line is in `docker compose logs odysseus`.
Use that for the first login, then change it in **Settings**.
Contributing? See [CONTRIBUTING.md](../CONTRIBUTING.md) for setup, testing, and
pull request guidelines.
### Docker (recommended)
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
cp .env.example .env # optional, but recommended for explicit defaults
docker compose up -d --build
```
To include optional extras in the image (PDF viewer, Office extraction; includes AGPL PyMuPDF), build with `docker compose build --build-arg INSTALL_OPTIONAL=true` before `up`.
Open `http://localhost:7000` when the containers are healthy. Docker Compose
binds the web UI to `127.0.0.1` by default. If the port is taken, set
`APP_PORT=7001` in `.env` and recreate the container. Set `APP_BIND=0.0.0.0`
only when you intentionally want LAN/reverse-proxy access.
> **On Apple Silicon (M-series) Macs:** Docker can't reach the Metal GPU, so
> Cookbook serves local models on CPU only. For GPU-accelerated model serving,
> run natively instead — see [Apple Silicon](#apple-silicon) below.
### Native Linux / macOS
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
Requirements: Python 3.11+. Cookbook also needs `tmux` for background model
downloads and serves. The app itself is lightweight; local model serving is the
heavy part and depends on the model, runtime, GPU, and VRAM, so small hosts can
connect to API or remote model servers instead. Use `--host 0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
### Apple Silicon
Docker on macOS cannot use the Metal GPU. For GPU-accelerated Cookbook on an
M-series Mac, run Odysseus natively:
```bash
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
./start-macos.sh
```
It launches at `http://127.0.0.1:7860`. To expose it to your phone over a trusted LAN/VPN such as Tailscale, bind all interfaces:
```bash
ODYSSEUS_HOST=0.0.0.0 ./start-macos.sh
# then open http://<tailscale-ip>:7860
```
The script also reads `.env` at startup, so `APP_BIND=0.0.0.0` and `APP_PORT`
set there are picked up automatically without a command-line override each run.
Keep `AUTH_ENABLED=true` (the default) before binding outside loopback. Do not
expose this port directly to the public internet. To build a clickable app wrapper:
```bash
./build-macos-app.sh
```
<details>
<summary>Cookbook, GPU, Ollama, and troubleshooting notes</summary>
**Docker bundled services.** Compose starts Odysseus, ChromaDB, SearXNG, and
ntfy. Odysseus and the bundled service ports bind to `127.0.0.1` by default, so
they are reachable from the host but not exposed to your LAN/public internet
unless you opt in.
**Cookbook storage in Docker.** Downloads live in `./data/huggingface`
(`~/.cache/huggingface` in the container). Cookbook-installed Python CLIs and
serve engines live in `./data/local` (`~/.local` in the container), so they
survive container recreation.
**Remote servers.** In **Cookbook -> Settings -> Servers**, generate the
Odysseus SSH key and add the public key to the remote server's
`~/.ssh/authorized_keys`. From the host you can also run:
```bash
ssh-copy-id -i data/ssh/id_ed25519.pub user@server
```
**Docker GPU overlays.** CPU-only users can skip this section. Cookbook can
only detect GPUs that Docker exposes to the container — if the host runtime or
device passthrough is not configured, Cookbook sees the iGPU, another card, or
CPU instead of your intended GPU.
For NVIDIA, `scripts/check-docker-gpu.sh` diagnoses GPU passthrough and can
optionally install the host runtime or update `.env`.
```bash
# Read-only diagnostic (default — installs nothing, never edits .env):
scripts/check-docker-gpu.sh
# Print OS-specific install commands without running them:
scripts/check-docker-gpu.sh --print-install-commands
# Install NVIDIA Container Toolkit on Ubuntu/Debian (requires sudo):
scripts/check-docker-gpu.sh --install-nvidia-toolkit
# Write COMPOSE_FILE to .env (only when GPU passthrough is confirmed working):
scripts/check-docker-gpu.sh --enable-nvidia-overlay
# Full assisted setup — install toolkit, then enable overlay if passthrough works:
scripts/check-docker-gpu.sh --install-nvidia-toolkit --enable-nvidia-overlay
```
Safety notes:
- The app never installs host GPU runtime automatically.
- The app never edits `.env` automatically.
- `.env` is only modified when `--enable-nvidia-overlay` is explicitly passed,
and only after GPU passthrough succeeds. `--yes` skips prompts but does not
bypass the passthrough gate.
- `.env.bak.*` backups created by `--enable-nvidia-overlay` are ignored by
Git and the Docker build context.
To enable manually without the script, add this to `.env`:
```bash
COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
```
**AMD / ROCm.** AMD setup is read-only diagnostic plus manual `.env` edit. Run:
```bash
scripts/check-docker-amd-gpu.sh
```
Then add the reported values to `.env`, replacing `RENDER_GID` with your host's
numeric render group id:
```bash
COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
RENDER_GID=989
```
For NVIDIA/AMD GPU support, also read the comments in the selected overlay file: docker/gpu.nvidia.yml or docker/gpu.amd.yml.
**Stack-management UIs (Portainer, Coolify, Dockhand, etc.).** These tools
often accept only a single Compose file and do not reliably honor `COMPOSE_FILE`
or multiple `-f` overlays. CLI users should keep using the `COMPOSE_FILE`
overlay workflow above. For stack UIs, point the stack at one of the standalone
files instead, which bundle the base stack plus the GPU settings:
- `docker-compose.gpu-nvidia.yml` — still requires the NVIDIA Container Toolkit
on the host.
- `docker-compose.gpu-amd.yml` — still requires host ROCm/kfd/DRI setup, the
`video`/`render` group membership, and `RENDER_GID` when needed.
The base `docker-compose.yml` plus the `docker/gpu.*.yml` overlays remain the
source of truth; the standalone files mirror them for single-file deployments.
Verify after enabling either overlay:
```bash
docker compose exec odysseus nvidia-smi -L # NVIDIA
docker compose exec odysseus sh -lc 'test -e /dev/kfd && test -d /dev/dri && ls -l /dev/kfd /dev/dri/renderD*' # AMD
```
> **GPU passthrough ≠ llama.cpp CUDA.** `nvidia-smi` passing inside the
> container confirms Docker GPU access, but llama.cpp also needs `cudart` and
> the CUDA Toolkit at runtime. If Cookbook logs show `Unable to find cudart
> library`, `Could NOT find CUDAToolkit`, `CUDA Toolkit not found`, or
> tensors/layers assigned to CPU, that is a Cookbook/llama.cpp build issue —
> not a Docker passthrough failure. Reinstall the serve engine via
> **Cookbook → Dependencies** to get a CUDA-enabled build.
>
> The same split applies to AMD/ROCm: seeing `/dev/kfd` and `/dev/dri` inside
> the container confirms device passthrough, not ROCm userspace or a
> ROCm-enabled vLLM/llama.cpp build. `rocm-smi` and `rocminfo` are not expected
> inside the slim Odysseus image.
**Ollama with Docker.** If Ollama runs on the host, add this endpoint in
Settings:
```text
http://host.docker.internal:11434/v1
```
Ollama must listen outside its own loopback interface:
```bash
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
This connects Odysseus in Docker to an Ollama server that is already running on
your host machine; it does not start Ollama inside the container.
`host.docker.internal` is Docker's hostname for the host machine from inside the
container. Cookbook **Serve** is a separate workflow for serving downloaded
models through Odysseus/llama.cpp, so Windows users with an existing Ollama
install usually only need to add the endpoint in Settings.
**Useful checks.**
```bash
docker compose ps
docker compose logs --tail=120 odysseus
docker compose logs odysseus | grep -E 'ChromaDB|MemoryVectorStore|DEGRADED'
```
**macOS details.** `start-macos.sh` installs Homebrew deps, creates the venv,
runs setup, and starts uvicorn on port `7860` because AirPlay often holds
`7000`. It uses llama.cpp/Ollama for Metal. vLLM/SGLang are CUDA/ROCm-only and
do not run on macOS. MLX-only models are not served by Odysseus.
</details>
### Native Windows
**One-command launcher** (creates the venv, installs deps, runs setup, starts the
server; safe to re-run):
```powershell
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1
```
Or do it by hand:
```powershell
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
py -3.11 -m venv venv
venv\Scripts\Activate.ps1
pip install -r requirements.txt
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000
```
If `python` points at an older interpreter, use `py -3.12` (or another installed
3.11+ version) for the venv step.
**Exposing on a LAN/Tailscale (Windows):** the launcher binds to `127.0.0.1` and
does **not** read `APP_BIND` / `ODYSSEUS_HOST` from `.env`, so editing `.env`
alone leaves the native Windows server on loopback. Pass the launcher's
`-BindHost` flag instead:
```powershell
powershell -ExecutionPolicy Bypass -File .\launch-windows.ps1 -BindHost 0.0.0.0
```
The manual `uvicorn` command takes the same address as `--host 0.0.0.0`. Bind
outside loopback only for a trusted LAN/VPN such as Tailscale: keep
`AUTH_ENABLED=true` and do not expose the port directly to the public internet.
**Requirements:** Python 3.11+. The core app (chat, agent, memory, documents,
email, calendar, deep research) runs fully native. For full **Cookbook** background
model downloads and the agent shell tool, also install
[Git for Windows](https://git-scm.com/download/win) (provides `bash.exe`).
Local GPU *serving* of vLLM/SGLang needs Linux/WSL2; for a local model on Windows,
[Ollama](https://ollama.com/download) is the easiest path — point Odysseus at
`http://localhost:11434/v1` in Settings.
Open `http://localhost:7000`, log in with the generated admin password,
and configure everything else inside **Settings**.
## Troubleshooting & Advanced Setup
### `chromadb-client` conflicts with embedded ChromaDB
If `chromadb-client` (the lightweight HTTP-only package) is installed alongside the full `chromadb` package, Odysseus starts but ChromaDB silently falls back to HTTP-only mode and fails.
**Fix:** uninstall `chromadb-client` and force-reinstall the full package:
```bash
./venv/bin/pip uninstall chromadb-client -y
./venv/bin/pip install --force-reinstall chromadb
```
### HTTPS + LAN/Tailscale exposure
To expose Odysseus on a local network or Tailscale with HTTPS:
1. Change the bind address to `0.0.0.0` in `.env` (`APP_BIND=0.0.0.0` or `ODYSSEUS_HOST=0.0.0.0`).
2. Generate a locally-trusted cert for your LAN/Tailscale IPs using [mkcert](https://github.com/FiloSottile/mkcert):
```bash
mkcert -install
mkcert -cert-file cert.pem -key-file key.pem 192.168.1.100 tailscale-ip
```
3. Run `uvicorn` with the generated certs:
```bash
python -m uvicorn app:app --host 0.0.0.0 --port 7000 --ssl-certfile=cert.pem --ssl-keyfile=key.pem
```
4. Install the `mkcert` CA on any other device you want to access Odysseus from (e.g., for iOS, email the `rootCA.pem` to yourself, install the profile, and trust it in Certificate Trust Settings).
### Common self-host traps (30-second fixes)
A grab-bag of small gotchas that otherwise turn into long debugging sessions.
- **`AUTH_ENABLED=false` is ignored / you're still forced to log in (Windows).** If you edited `.env` in Notepad it may have saved a UTF-8 **BOM**, turning the first key into `AUTH_ENABLED` so it is never matched. Odysseus loads `.env` with `encoding="utf-8-sig"` to tolerate a leading BOM, but the safe fix is to re-save `.env` as **UTF-8 without BOM** (VS Code: *Save with Encoding → UTF-8*).
- **macOS: the app isn't at `http://localhost:7000`.** macOS AirPlay Receiver usually holds port `7000`, so the macOS start script serves on **`7860`** instead — open `http://localhost:7860`. To use `7000`, free it (System Settings → General → AirDrop & Handoff → turn off *AirPlay Receiver*) and set `APP_PORT=7000`.
- **Copy buttons do nothing over a plain-HTTP Tailscale/LAN URL.** Browsers only expose the clipboard API (`navigator.clipboard`) on **secure origins** — HTTPS, or `localhost`. Over `http://100.x.y.z:7860` it is blocked. Serve over HTTPS (see *HTTPS + LAN/Tailscale exposure* above); `localhost` is exempt, so copy still works on the host itself.
- **Self-hosted ntfy reminders don't reach your phone.** Two things: (1) the bundled ntfy binds to loopback by default — to reach it from your phone set `NTFY_BIND` to your host/Tailscale IP and `NTFY_BASE_URL` to the same server URL in `.env`, then recreate the ntfy container (see the `NTFY_*` block in `.env.example`); (2) in the ntfy **Android** app, subscribe to the topic with **Instant delivery** enabled — non-`ntfy.sh` servers don't get instant push otherwise.
- **Local mail (Dovecot) login fails: "Plaintext authentication disallowed on non-encrypted connections."** Your IMAP/SMTP server is refusing cleartext auth over an unencrypted link. Prefer enabling TLS on the mail server; on a trusted LAN only, you can allow cleartext (Dovecot: `disable_plaintext_auth = no`).
- **Calendar/contacts (Radicale) won't sync.** Point Odysseus at the **full collection URL** with its trailing slash — e.g. `http://host:5232/<user>/<collection-id>/` — not just the server root. Radicale shows this address for each calendar/address book in its web UI.
### Optional Dependencies
`requirements-optional.txt` contains packages that unlock extra features. It is not installed by default.
| Package | Feature unlocked |
|---------|-----------------|
| `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
| `ddgs` | DuckDuckGo as a search provider option. |
| `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
| `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
### Faster, reproducible installs with uv (optional)
[uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
```bash
uv venv venv --python 3.13
uv pip install -r requirements.txt
# then continue as usual: python setup.py, uvicorn, ...
```
`requirements.txt` is intentionally unpinned, so two installs at different times can produce different package versions. If you want a reproducible environment (e.g. across your own machines, or to roll back after a bad upgrade), snapshot and restore exact versions with:
```bash
uv pip compile requirements.txt -o requirements.lock # snapshot current resolution
uv pip sync requirements.lock # reproduce it exactly later
```
`requirements.lock` is gitignored and platform-specific (compile it on the OS you deploy to). Regenerate it deliberately when you want to take upgrades. The plain `uv pip install -r requirements.txt` keeps following the unpinned requirements like pip does.
### Outlook / Office 365 email
Odysseus email accounts currently use IMAP/SMTP username-password auth. Outlook
and Microsoft 365 generally require OAuth instead, so normal Microsoft mailbox
passwords will fail. See [docs/email-outlook.md](docs/email-outlook.md) for the
current limitation and the planned integration direction.
## Security Notes
Odysseus is a self-hosted workspace with powerful local tools: shell access, file uploads, model downloads, web research, email/calendar integrations, and API tokens. Treat it like an admin console.
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
- Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
- Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
- Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
- Non-admin users do not get shell/Python/file read/write by default, and admin-only routes/tools such as MCP management, API tokens, webhooks, model/cookbook serving, backup/vault, and app settings are admin-gated. Other features are controlled by per-user privileges, so review each user's privileges before exposing a deployment.
- Rotate any API keys or tokens that were ever pasted into a shared chat, demo, screenshot, or log.
- If you enable API tokens or webhooks, create separate tokens per integration and delete unused ones.
- Prefer binding manual development runs to `127.0.0.1`; bind to `0.0.0.0` only when you intentionally want LAN/reverse-proxy access.
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
- Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
### Private or proxied deployments
Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
1. Keep Odysseus on localhost, for example `127.0.0.1:7000`.
2. Terminate HTTPS at a trusted reverse proxy or private access gateway.
3. Put the authenticated Odysseus web/API entrypoint behind that layer.
4. Keep raw service and model ports internal-only.
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
`ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
Common internal-only ports from the default docs/compose setup:
| Port | Service |
|---|---|
| `7000` | Odysseus raw app port |
| `8080` | SearXNG |
| `8091` | ntfy |
| `8100` | ChromaDB host port for manual/compose access |
| `11434` | Ollama |
| `8000-8020` | Common local model/provider APIs |
## Configuration
Most setup is done inside the app with `/setup` or **Settings**. Use `.env`
for deployment-level defaults and secrets you want present before first boot.
Key settings:
| Variable | Default | Description |
|---|---|---|
| `LLM_HOST` | `localhost` | Your LLM server (e.g. `llm-host.local:8000`) |
| `LLM_HOSTS` | -- | Comma-separated list for model discovery |
| `OPENAI_API_KEY` | -- | Optional OpenAI key. Prefer adding providers in the app unless pre-seeding. |
| `SEARXNG_INSTANCE` | `http://localhost:8080` | SearXNG URL. Docker overrides this to `http://searxng:8080`. |
| `SEARXNG_SECRET` | generated on first Docker boot | Optional SearXNG cookie/CSRF secret. Leave blank unless you need to pin it. |
| `APP_BIND` | `127.0.0.1` | Docker Compose host bind address for the web UI. Use `0.0.0.0` only for intentional LAN/reverse-proxy access. |
| `APP_PORT` | `7000` | Docker Compose host port for the web UI. |
| `APP_DATA_DIR` | `./data` | Docker Compose host directory for application data volumes. |
| `APP_LOGS_DIR` | `./logs` | Docker Compose host directory for application logs. |
| `AUTH_ENABLED` | `true` | Enable/disable login |
| `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
| `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
| `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
| `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
| `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |
| `EMBEDDING_URL` | -- | OpenAI-compatible embeddings endpoint |
| `ODYSSEUS_CHAT_UPLOAD_MAX_BYTES` | `10485760` | Chat/agent attachment cap in bytes. Raise for larger local PDFs or text documents. |
| `ODYSSEUS_GALLERY_UPLOAD_MAX_BYTES` | `104857600` | Gallery image upload cap in bytes (100 MB). |
| `ODYSSEUS_GALLERY_TRANSFORM_UPLOAD_MAX_BYTES` | `26214400` | Gallery transform input cap in bytes (25 MB). |
| `ODYSSEUS_MEMORY_IMPORT_MAX_BYTES` | `10485760` | Memory import file cap in bytes (10 MB). |
| `ODYSSEUS_PERSONAL_UPLOAD_MAX_BYTES` | `26214400` | Personal document upload cap in bytes (25 MB). |
| `ODYSSEUS_EMAIL_COMPOSE_UPLOAD_MAX_BYTES` | `26214400` | Email compose attachment cap in bytes (25 MB). |
| `ODYSSEUS_STT_MAX_AUDIO_BYTES` | `26214400` | Speech-to-text audio cap in bytes (25 MB). |
| `ODYSSEUS_ICS_MAX_BYTES` | `10485760` | Calendar `.ics` import cap in bytes (10 MB). |
All upload-limit vars are validated (must be a positive integer) and optional; an invalid value fails fast at startup.
### Built-in MCP servers (optional setup)
Odysseus auto-registers a few built-in MCP servers at startup. The npx-based ones (currently the browser server, `@playwright/mcp`) only start when their npm package is already in the local npx cache. If a package isn't cached, that server is skipped with a startup log message explaining what to do, so a fresh install does not block on a multi-minute npm download or hang if Playwright system deps are missing.
To enable the browser MCP (page navigation, screenshots, vision), run once:
```bash
npx -y @playwright/mcp@latest --version
```
That installs `@playwright/mcp` plus Playwright (~300MB total). Restart Odysseus and the server will register at startup.
## Architecture
```
app.py # FastAPI entry point
core/ auth, database, middleware, constants
src/ llm_core, agent_loop, agent_tools, chat_processor, search/
routes/ chat, session, document, memory, model … endpoints
services/ docs, memory, search, hwfit (Cookbook) …
static/ index.html + app.js + style.css + js/ (modular front-end)
docs/ landing page (index.html) + preview clips
```
## Data
All user data lives in `data/` (gitignored): `app.db` (sessions, messages, documents),
`memory.json`, `presets.json`, `uploads/`, `personal_docs/`, `chroma/`, `settings.json`.
To back up or restore everything in `data/`, see the
[Backup & Restore guide](docs/backup-restore.md).
integrations/claude/skills/odysseus/SKILL.md
+1
View File
@@ -102,6 +102,7 @@ python3 ~/.claude/skills/odysseus/scripts/odysseus_api.py POST /api/codex/memory
## Email draft + send
- Prefer `POST /api/codex/emails/draft-document` for agent-written email replies. It creates an editable Odysseus Document with `language: "email"` and does not touch IMAP/send.
- `POST /api/codex/emails/draft` — body matches `SendEmailRequest` (`to`, `cc`, `bcc`, `subject`, `body`, `body_html`, `attachments`, `account_id`, `in_reply_to`, `references`). Requires `email:draft` (or `email:send`).
- `POST /api/codex/emails/send` — same body. Requires `email:send`. Never send without explicit user instruction.
integrations/claude/skills/odysseus/scripts/odysseus_api.py
+32
View File
@@ -17,6 +17,11 @@ def _usage() -> int:
print(" odysseus_api.py todos add TITLE", file=sys.stderr)
print(" odysseus_api.py emails list [limit]", file=sys.stderr)
print(" odysseus_api.py emails read UID", file=sys.stderr)
print(" odysseus_api.py emails draft-doc JSON_PAYLOAD", file=sys.stderr)
print(" odysseus_api.py documents list [limit]", file=sys.stderr)
print(" odysseus_api.py documents read DOC_ID", file=sys.stderr)
print(" odysseus_api.py documents create JSON_PAYLOAD", file=sys.stderr)
print(" odysseus_api.py documents delete DOC_ID", file=sys.stderr)
print(" odysseus_api.py cookbook tasks", file=sys.stderr)
print(" odysseus_api.py cookbook servers", file=sys.stderr)
print(" odysseus_api.py cookbook cached [HOST]", file=sys.stderr)
@@ -79,6 +84,33 @@ def main() -> int:
method = "GET"
path = f"/api/codex/emails/{sys.argv[3]}"
body = None
elif action in ("draft-doc", "draft_document") and len(sys.argv) >= 4:
method = "POST"
path = "/api/codex/emails/draft-document"
body = " ".join(sys.argv[3:])
else:
return _usage()
elif command in ("documents", "docs"):
if len(sys.argv) < 3:
return _usage()
action = sys.argv[2].lower()
if action == "list":
method = "GET"
limit = sys.argv[3] if len(sys.argv) >= 4 else "50"
path = f"/api/codex/documents?limit={limit}"
body = None
elif action == "read" and len(sys.argv) >= 4:
method = "GET"
path = f"/api/codex/documents/{sys.argv[3]}"
body = None
elif action == "create" and len(sys.argv) >= 4:
method = "POST"
path = "/api/codex/documents"
body = " ".join(sys.argv[3:])
elif action == "delete" and len(sys.argv) >= 4:
method = "DELETE"
path = f"/api/codex/documents/{sys.argv[3]}"
body = None
else:
return _usage()
elif command == "cookbook":
integrations/codex/scripts/odysseus_api.py
+32
View File
@@ -17,6 +17,11 @@ def _usage() -> int:
print(" odysseus_api.py todos add TITLE", file=sys.stderr)
print(" odysseus_api.py emails list [limit]", file=sys.stderr)
print(" odysseus_api.py emails read UID", file=sys.stderr)
print(" odysseus_api.py emails draft-doc JSON_PAYLOAD", file=sys.stderr)
print(" odysseus_api.py documents list [limit]", file=sys.stderr)
print(" odysseus_api.py documents read DOC_ID", file=sys.stderr)
print(" odysseus_api.py documents create JSON_PAYLOAD", file=sys.stderr)
print(" odysseus_api.py documents delete DOC_ID", file=sys.stderr)
print(" odysseus_api.py cookbook tasks", file=sys.stderr)
print(" odysseus_api.py cookbook servers", file=sys.stderr)
print(" odysseus_api.py cookbook cached [HOST]", file=sys.stderr)
@@ -79,6 +84,33 @@ def main() -> int:
method = "GET"
path = f"/api/codex/emails/{sys.argv[3]}"
body = None
elif action in ("draft-doc", "draft_document") and len(sys.argv) >= 4:
method = "POST"
path = "/api/codex/emails/draft-document"
body = " ".join(sys.argv[3:])
else:
return _usage()
elif command in ("documents", "docs"):
if len(sys.argv) < 3:
return _usage()
action = sys.argv[2].lower()
if action == "list":
method = "GET"
limit = sys.argv[3] if len(sys.argv) >= 4 else "50"
path = f"/api/codex/documents?limit={limit}"
body = None
elif action == "read" and len(sys.argv) >= 4:
method = "GET"
path = f"/api/codex/documents/{sys.argv[3]}"
body = None
elif action == "create" and len(sys.argv) >= 4:
method = "POST"
path = "/api/codex/documents"
body = " ".join(sys.argv[3:])
elif action == "delete" and len(sys.argv) >= 4:
method = "DELETE"
path = f"/api/codex/documents/{sys.argv[3]}"
body = None
else:
return _usage()
elif command == "cookbook":
integrations/codex/skills/odysseus/SKILL.md
+1
View File
@@ -102,6 +102,7 @@ python3 integrations/codex/scripts/odysseus_api.py POST /api/codex/memory '{"tex
## Email draft + send
- Prefer `POST /api/codex/emails/draft-document` for Codex-written email replies. It creates an editable Odysseus Document with `language: "email"` and does not touch IMAP/send.
- `POST /api/codex/emails/draft` — body matches `SendEmailRequest` (`to`, `cc`, `bcc`, `subject`, `body`, `body_html`, `attachments`, `account_id`, `in_reply_to`, `references`). Requires `email:draft` (or `email:send`).
- `POST /api/codex/emails/send` — same body. Requires `email:send`. Never send without explicit user instruction.
launch-windows.ps1
+22 -1
View File
@@ -105,6 +105,14 @@ if (-not $pyExe) {
}
}
if ($pyExe -like "*WindowsApps*python.exe") {
$pyCmd = Get-Command py -ErrorAction SilentlyContinue
if ($pyCmd) {
$pyExe = $pyCmd.Source
$pyArgs = @("-3.11")
}
}
if (-not $pyExe) {
Fail "Couldn't find Python 3.11+ for Windows setup. Install Python 3.11+ (or open the Python launcher with 'py -3.11') from https://www.python.org/downloads/, then re-run this script."
}
@@ -141,7 +149,20 @@ if (-not (Find-GitBash)) {
Write-Host " https://git-scm.com/download/win" -ForegroundColor Yellow
}
# 6. Start the server (use `python -m uvicorn` - bare `uvicorn` may not be on PATH)
# 6. Point CUDA_PATH at a real CUDA toolkit so GPU llama-cpp-python can import.
$cudaBase = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA"
if (Test-Path $cudaBase) {
$cudaBest = Get-ChildItem $cudaBase -Directory -ErrorAction SilentlyContinue |
Where-Object { Test-Path (Join-Path $_.FullName "bin") } |
Sort-Object { try { [version]($_.Name -replace "^v", "") } catch { [version]"0.0" } } -Descending |
Select-Object -First 1
if ($cudaBest) {
$env:CUDA_PATH = $cudaBest.FullName
Write-Host ("Using CUDA_PATH = " + $cudaBest.FullName) -ForegroundColor Cyan
}
}
# 7. Start the server (use `python -m uvicorn` - bare `uvicorn` may not be on PATH)
Write-Step ("Starting Odysseus at http://{0}:{1}" -f $BindHost, $Port)
Write-Host "Press Ctrl+C to stop."
Write-Host ""
launcher.py
+142
View File
@@ -0,0 +1,142 @@
# launcher.py
"""Dedicated entrypoint for the standalone Windows portable launcher.
Handles:
- Immediate GUI splash screen creation using tkinter.
- Suppressing console stream crashes in windowed GUI mode via NullWriter.
- Spawning system tray icon via pystray and Pillow (lazy-loaded).
- Auto-opening default browser pointing to the running backend.
- Launching the FastAPI server (importing and running app.py).
"""
import os
import sys
import threading
import time
import webbrowser
# Define a dummy NullWriter to suppress standard stream crashes (isatty etc.) in GUI mode
class NullWriter:
def write(self, text):
pass
def flush(self):
pass
def isatty(self):
return False
if sys.stdout is None:
sys.stdout = NullWriter()
if sys.stderr is None:
sys.stderr = NullWriter()
splash_root = None
# If running from a frozen PyInstaller bundle, launch the splash screen IMMEDIATELY
if getattr(sys, 'frozen', False):
import tkinter as tk
def show_splash_instantly():
global splash_root
try:
splash_root = tk.Tk()
splash_root.title("Odysseus")
splash_root.overrideredirect(True)
splash_root.configure(bg="#1a1c23")
# Accented borders
splash_root.config(highlightbackground="#e06c75", highlightcolor="#e06c75", highlightthickness=1)
w, h = 360, 160
ws = splash_root.winfo_screenwidth()
hs = splash_root.winfo_screenheight()
x = (ws - w) // 2
y = (hs - h) // 2
splash_root.geometry(f"{w}x{h}+{x}+{y}")
tk.Label(splash_root, text="⛵ Odysseus", font=("Segoe UI", 22, "bold"), bg="#1a1c23", fg="#e06c75").pack(pady=(22, 2))
tk.Label(splash_root, text="Launching background services...", font=("Segoe UI", 10), bg="#1a1c23", fg="#d1d4e0").pack(pady=2)
tk.Label(splash_root, text="Please wait, this will take a few seconds.", font=("Segoe UI", 8, "italic"), bg="#1a1c23", fg="#5c6370").pack(pady=(12, 0))
splash_root.attributes("-topmost", True)
splash_root.mainloop()
except Exception:
pass
# Launch the GUI splash screen immediately on a background thread
threading.Thread(target=show_splash_instantly, daemon=True).start()
def create_tray_image():
# Generate a beautiful 64x64 icon matching Odysseus brand red accent (#e06c75)
from PIL import Image, ImageDraw
image = Image.new('RGBA', (64, 64), (0, 0, 0, 0))
dc = ImageDraw.Draw(image)
accent_red = (224, 108, 117, 255)
light_red = (224, 108, 117, 150)
# Draw premium sailing boat
dc.polygon([(32, 10), (32, 45), (12, 45)], fill=accent_red)
dc.polygon([(32, 18), (32, 45), (48, 45)], fill=light_red)
dc.polygon([(8, 48), (56, 48), (44, 56), (20, 56)], fill=accent_red)
return image
def on_open_browser(icon, item, url):
webbrowser.open(url)
def on_exit(icon, item):
icon.stop()
os._exit(0)
def setup_system_tray(url):
try:
import pystray
icon_img = create_tray_image()
menu = (
pystray.MenuItem('Open Odysseus', lambda icon, item: on_open_browser(icon, item, url), default=True),
pystray.MenuItem('Exit', on_exit)
)
tray_icon = pystray.Icon(
"Odysseus",
icon_img,
"Odysseus",
menu
)
tray_icon.run()
except Exception:
pass
def open_browser(url):
# Allow uvicorn and app lifecycles to complete warmups
time.sleep(3.5)
# Safely close the splash screen
try:
global splash_root
if splash_root:
splash_root.after(0, splash_root.destroy)
except Exception:
pass
webbrowser.open(url)
if __name__ == "__main__":
import uvicorn
# Import the FastAPI app from app.py
from app import app
bind_host = os.getenv("APP_BIND", "127.0.0.1")
bind_port = int(os.getenv("APP_PORT", "7000"))
url = f"http://{bind_host}:{bind_port}"
if getattr(sys, 'frozen', False):
# Start browser manager thread
threading.Thread(target=open_browser, args=(url,), daemon=True).start()
# Start system tray manager thread
threading.Thread(target=setup_system_tray, args=(url,), daemon=True).start()
uvicorn.run(app, host=bind_host, port=bind_port, log_level="info")
licenses/OpenDyslexic-OFL.txt
+94
View File
@@ -0,0 +1,94 @@
Copyright (c) 2019-07-29, Abbie Gonzalez (https://abbiecod.es|support@abbiecod.es),
with Reserved Font Name OpenDyslexic.
Copyright (c) 12/2012 - 2019
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
http://scripts.sil.org/OFL
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.
mcp_servers/email_server.py
+190 -12
View File
@@ -23,6 +23,7 @@ import os.path
from pathlib import Path
from datetime import datetime, timedelta
import uuid
from contextvars import ContextVar
from mcp.server import Server
from mcp.server.stdio import stdio_server
@@ -55,6 +56,8 @@ def _uid_fetch_rows(data) -> list:
# flat keys when no DB row matches (legacy single-account behaviour).
_ACCOUNT_CACHE: dict = {} # key = normalized account selector -> config dict
_MCP_OWNER_ARG = "_odysseus_owner"
_CURRENT_OWNER: ContextVar[str | None] = ContextVar("email_mcp_owner", default=None)
def _clean_header_value(value) -> str:
@@ -68,6 +71,45 @@ def _db_path() -> Path:
return Path(APP_DB)
def _current_owner() -> str:
owner = _CURRENT_OWNER.get()
return str(owner or "").strip()
def _account_visible_to_owner(row: dict, owner: str) -> bool:
row_owner = str(row.get("owner") or "").strip()
if row_owner == owner:
return True
if row_owner:
return False
# Legacy ownerless accounts are only visible to a scoped caller when the
# mailbox itself matches the owner, mirroring the HTTP email route fallback.
owner_l = owner.lower()
return owner_l in {
str(row.get("imap_user") or "").strip().lower(),
str(row.get("from_address") or "").strip().lower(),
}
def _filter_accounts_for_owner(rows: list[dict]) -> list[dict]:
owner = _current_owner()
if owner:
return [r for r in rows if _account_visible_to_owner(r, owner)]
owners = {str(r.get("owner") or "").strip() for r in rows if str(r.get("owner") or "").strip()}
if len(owners) > 1:
return []
return rows
def _mcp_owner_required(rows: list[dict] | None = None) -> bool:
if _current_owner():
return False
rows = rows if rows is not None else _read_accounts_from_db()
owners = {str(r.get("owner") or "").strip() for r in rows if str(r.get("owner") or "").strip()}
return len(owners) > 1
def _load_email_writing_style() -> str:
"""Return the existing Settings > Email > Writing Style value."""
try:
@@ -121,9 +163,8 @@ def _default_document_owner() -> str | None:
return None
def _list_accounts_raw() -> list:
"""Return list of dicts from the email_accounts table. Empty list if table
missing or empty. Never raises."""
def _read_accounts_from_db() -> list:
"""Return all enabled email account rows. Empty list if missing. Never raises."""
path = _db_path()
if not path.exists():
return []
@@ -131,9 +172,10 @@ def _list_accounts_raw() -> list:
conn = sqlite3.connect(str(path))
conn.row_factory = sqlite3.Row
columns = {r[1] for r in conn.execute("PRAGMA table_info(email_accounts)").fetchall()}
owner_select = "owner" if "owner" in columns else "NULL AS owner"
smtp_security_select = "smtp_security" if "smtp_security" in columns else "'' AS smtp_security"
rows = conn.execute(f"""
SELECT id, name, is_default, enabled,
SELECT id, {owner_select}, name, is_default, enabled,
imap_host, imap_port, imap_user, imap_password, imap_starttls,
smtp_host, smtp_port, {smtp_security_select}, smtp_user, smtp_password, from_address
FROM email_accounts WHERE enabled = 1
@@ -147,11 +189,15 @@ def _list_accounts_raw() -> list:
return []
def _resolve_account(selector: str | None) -> dict | None:
def _list_accounts_raw() -> list:
"""Return owner-visible email account rows for the active MCP call."""
return _filter_accounts_for_owner(_read_accounts_from_db())
def _resolve_account_from_rows(rows: list[dict], selector: str | None) -> dict | None:
"""Given a selector (None = default, or a name/user/id string), return the
matching row or None. Matching is case-insensitive substring on name +
imap_user + from_address, plus exact id match."""
rows = _list_accounts_raw()
if not rows:
return None
if not selector:
@@ -186,6 +232,10 @@ def _resolve_account(selector: str | None) -> dict | None:
return None
def _resolve_account(selector: str | None) -> dict | None:
return _resolve_account_from_rows(_list_accounts_raw(), selector)
def _load_config(account: str | None = None) -> dict:
"""Return the full config dict for the requested account (or default).
@@ -194,7 +244,7 @@ def _load_config(account: str | None = None) -> dict:
2. env vars + settings.json flat keys (legacy)
3. hardcoded fallbacks (localhost:31143 etc.)
"""
cache_key = (account or "").strip().lower() or "__default__"
cache_key = (_current_owner(), (account or "").strip().lower() or "__default__")
if cache_key in _ACCOUNT_CACHE:
return _ACCOUNT_CACHE[cache_key]
@@ -223,8 +273,11 @@ def _load_config(account: str | None = None) -> dict:
"account_name": None,
}
rows = _list_accounts_raw()
row = _resolve_account(account)
raw_rows = _read_accounts_from_db()
rows = _filter_accounts_for_owner(raw_rows)
row = _resolve_account_from_rows(rows, account)
if _current_owner() and raw_rows and not rows:
raise ValueError("No email account is configured for the authenticated owner")
if account and rows and not row:
available = ", ".join(
f"{r.get('name') or r.get('imap_user')} <{r.get('imap_user') or r.get('from_address') or '?'}>"
@@ -885,8 +938,109 @@ def _smtp_connect(account=None, cfg=None):
return conn
def _read_agent_email_confirm_setting() -> bool:
"""True if the user wants agent send_email/reply_to_email calls to be
queued for manual approval instead of SMTPed immediately. Defaults to
True so a fresh install is safe — agents have been observed inventing
signatures and sending to real recipients without the user's review."""
try:
from src.settings import get_setting
return bool(get_setting("agent_email_confirm", True))
except Exception:
return True
def _stash_agent_draft(*, to, subject, body, in_reply_to=None, references=None,
cc=None, bcc=None, account=None) -> dict:
"""Insert the composed email into scheduled_emails with status
'agent_draft' and a far-future send_at so the scheduled-send poller
never picks it up. Returns the pending payload the model surfaces to
the user (and that the chat UI can render as an approval card)."""
try:
from src.constants import SCHEDULED_EMAILS_DB
except Exception:
return {"success": False, "error": "Pending-email storage unavailable"}
pending_id = uuid.uuid4().hex[:16]
far_future = "9999-12-31T00:00:00"
now = datetime.utcnow().isoformat()
try:
conn = sqlite3.connect(SCHEDULED_EMAILS_DB)
# Touch the schema in case the email-routes init hasn't run yet
# (MCP server can boot independently).
conn.execute("""
CREATE TABLE IF NOT EXISTS scheduled_emails (
id TEXT PRIMARY KEY,
to_addr TEXT NOT NULL,
cc TEXT,
bcc TEXT,
subject TEXT,
body TEXT NOT NULL,
in_reply_to TEXT,
references_hdr TEXT,
attachments TEXT,
send_at TEXT NOT NULL,
created_at TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
error TEXT,
owner TEXT DEFAULT '',
account_id TEXT,
odysseus_kind TEXT
)
""")
conn.execute("""
INSERT INTO scheduled_emails
(id, to_addr, cc, bcc, subject, body, in_reply_to, references_hdr,
attachments, send_at, created_at, status, account_id, odysseus_kind, owner)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'agent_draft', ?, ?, ?)
""", (
pending_id,
to if isinstance(to, str) else ", ".join(to),
cc if isinstance(cc, str) else (", ".join(cc) if cc else None),
bcc if isinstance(bcc, str) else (", ".join(bcc) if bcc else None),
subject or "",
body or "",
in_reply_to or None,
references if isinstance(references, str) else (" ".join(references) if references else None),
"[]",
far_future,
now,
account or None,
"agent_draft",
_current_owner(),
))
conn.commit()
conn.close()
except Exception as e:
return {"success": False, "error": f"Failed to stash draft: {e}"}
return {
"success": True,
"pending": True,
"pending_id": pending_id,
"to": to if isinstance(to, str) else ", ".join(to),
"subject": subject or "",
"body": body or "",
"message": (
"✋ Draft staged for your approval — nothing has been sent yet.\n"
"Review the To/Subject/Body above. Reply 'send' to deliver, or "
"'cancel' to discard."
),
}
def _send_email(to, subject, body, in_reply_to=None, references=None, cc=None, bcc=None, account=None):
"""Send an email via SMTP. Returns dict with status."""
"""Send an email via SMTP. Returns dict with status.
When the `agent_email_confirm` setting is on (the default), the email
is NOT SMTPed — instead it lands in scheduled_emails as an
`agent_draft` row and the user reviews + approves it from the chat
UI. This closes the auto-send hole that let earlier models invent
signatures and ship them to real recipients without confirmation."""
if _read_agent_email_confirm_setting():
return _stash_agent_draft(
to=to, subject=subject, body=body,
in_reply_to=in_reply_to, references=references,
cc=cc, bcc=bcc, account=account,
)
send_account, cfg = _resolve_send_config(account)
msg = EmailMessage()
msg["From"] = _clean_header_value(cfg["from_address"])
@@ -1038,7 +1192,7 @@ def _create_email_draft_document(
doc_id = str(uuid.uuid4())
ver_id = str(uuid.uuid4())
doc_title = (title or subject or "Email draft").strip() or "Email draft"
doc_owner = _default_document_owner()
doc_owner = _current_owner() or _default_document_owner()
db = SessionLocal()
try:
@@ -1824,10 +1978,22 @@ async def list_tools() -> list[Tool]:
@server.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
arguments = dict(arguments) if isinstance(arguments, dict) else {}
owner = str(arguments.pop(_MCP_OWNER_ARG, "") or "").strip()
owner_token = _CURRENT_OWNER.set(owner or None)
try:
all_db_accounts = _read_accounts_from_db()
if _mcp_owner_required(all_db_accounts):
return [TextContent(
type="text",
text="Error: email MCP requires an authenticated owner when multiple email account owners are configured.",
)]
if name == "list_email_accounts":
rows = _list_accounts_raw()
rows = _filter_accounts_for_owner(all_db_accounts)
if not rows:
if all_db_accounts and owner:
return [TextContent(type="text", text="No email accounts configured for this owner.")]
return [TextContent(type="text", text="No email accounts configured. Legacy single-account mode active.")]
lines = [f"Found {len(rows)} email account(s):\n"]
for r in rows:
@@ -2007,6 +2173,16 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
bcc=arguments.get("bcc"),
account=acct,
)
if "error" in result:
return [TextContent(type="text", text=f"Error: {result['error']}")]
if result.get("pending"):
return [TextContent(
type="text",
text=(
f"Draft staged for approval (pending id: {result.get('pending_id')}). "
"Nothing has been sent yet. Review and approve it in Odysseus before delivery."
),
)]
acct_note = f" (from {result['account']})" if result.get("account") else ""
return [TextContent(type="text", text=f"Sent email to {result['to']} with subject '{result['subject']}'{acct_note}.")]
@@ -2182,6 +2358,8 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
except Exception as e:
return [TextContent(type="text", text=f"Error: {e}")]
finally:
_CURRENT_OWNER.reset(owner_token)
# ── Main ──
mcp_servers/image_gen_server.py
+2 -2
View File
@@ -73,7 +73,7 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if not model_spec:
for candidate in ("gpt-image-1.5", "gpt-image-1", "dall-e-3"):
try:
_resolve_model(candidate)
await asyncio.to_thread(_resolve_model, candidate)
model_spec = candidate
break
except ValueError:
@@ -81,7 +81,7 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if not model_spec:
return [TextContent(type="text", text="Error: No image model found. Configure one in Admin.")]
url, model_id, headers = _resolve_model(model_spec)
url, model_id, headers = await asyncio.to_thread(_resolve_model, model_spec)
is_gpt_image = "gpt-image" in model_id.lower()
base_url = url.replace("/chat/completions", "").replace("/v1/messages", "").rstrip("/")
mcp_servers/memory_server.py
+92 -32
View File
@@ -6,6 +6,7 @@ Imports MemoryManager and MemoryVectorStore from the Odysseus codebase.
"""
import asyncio
import os
import sys
import time
from pathlib import Path
@@ -23,6 +24,55 @@ _memory_manager = None
_memory_vector = None
_initialized = False
_OWNER_ENV_KEYS = ("ODYSSEUS_MCP_MEMORY_OWNER", "ODYSSEUS_MEMORY_OWNER")
_OWNER_SCOPE_ERROR = (
"Error: Memory MCP owner is not configured for an owner-scoped memory store. "
"Set ODYSSEUS_MCP_MEMORY_OWNER for this server or use the owner-aware native memory tool."
)
def _configured_owner() -> str | None:
for key in _OWNER_ENV_KEYS:
owner = os.environ.get(key, "").strip()
if owner:
return owner
return None
def _entry_owner(entry: dict) -> str | None:
owner = entry.get("owner")
if owner is None:
return None
owner_text = str(owner).strip()
return owner_text or None
def _owner_scoped_store(entries: list[dict]) -> bool:
return any(_entry_owner(entry) for entry in entries if isinstance(entry, dict))
def _scope_entries() -> tuple[str | None, list[dict], list[dict], str | None]:
"""Return configured owner, all entries, visible entries, and optional error."""
entries = _memory_manager.load_all()
owner = _configured_owner()
if owner is None and _owner_scoped_store(entries):
return None, entries, [], _OWNER_SCOPE_ERROR
if owner is None:
visible = [
entry for entry in entries
if isinstance(entry, dict) and _entry_owner(entry) is None
]
else:
visible = [
entry for entry in entries
if isinstance(entry, dict) and _entry_owner(entry) == owner
]
return owner, entries, visible, None
def _text_result(text: str) -> list[TextContent]:
return [TextContent(type="text", text=text)]
def _ensure_init():
"""Lazy-init memory managers on first use."""
@@ -75,43 +125,46 @@ async def list_tools() -> list[Tool]:
@server.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if name != "manage_memory":
return [TextContent(type="text", text=f"Unknown tool: {name}")]
return _text_result(f"Unknown tool: {name}")
_ensure_init()
if not _memory_manager:
return [TextContent(type="text", text="Error: Memory manager not available")]
return _text_result("Error: Memory manager not available")
action = arguments.get("action", "")
if action == "list":
category_filter = arguments.get("category", "")
memories = _memory_manager.load()
_owner, _all_memories, memories, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
if category_filter:
memories = [m for m in memories if m.get("category", "").lower() == category_filter.lower()]
if not memories:
msg = "No memories found"
if category_filter:
msg += f" in category '{category_filter}'"
return [TextContent(type="text", text=msg + ".")]
return _text_result(msg + ".")
lines = [f"Found {len(memories)} memory entries:\n"]
for m in memories[:100]:
for m in memories:
cat = m.get("category", "fact")
mid = m.get("id", "?")[:8]
text = m.get("text", "")
if len(text) > 150:
text = text[:150] + "..."
lines.append(f"- [{cat}] `{mid}` — {text}")
if len(memories) > 100:
lines.append(f"... and {len(memories) - 100} more")
return [TextContent(type="text", text="\n".join(lines))]
return _text_result("\n".join(lines))
elif action == "add":
text = arguments.get("text", "")
category = arguments.get("category", "fact")
if not text:
return [TextContent(type="text", text="Error: Memory text cannot be empty")]
entry = _memory_manager.add_entry(text, source="ai_agent", category=category)
memories = _memory_manager.load_all()
return _text_result("Error: Memory text cannot be empty")
owner, memories, _visible, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
entry = _memory_manager.add_entry(text, source="ai_agent", category=category, owner=owner)
memories.append(entry)
_memory_manager.save(memories)
if _memory_vector and _memory_vector.healthy:
@@ -119,25 +172,28 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
_memory_vector.add(entry["id"], text)
except Exception:
pass
return [TextContent(type="text", text=f"Memory added: [{category}] {text} (id: {entry['id'][:8]})")]
return _text_result(f"Memory added: [{category}] {text} (id: {entry['id'][:8]})")
elif action == "edit":
memory_id = arguments.get("memory_id", "")
new_text = arguments.get("text", "")
if not memory_id or not new_text:
return [TextContent(type="text", text="Error: edit needs memory_id and text")]
memories = _memory_manager.load_all()
found = False
return _text_result("Error: edit needs memory_id and text")
_owner, memories, visible, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
full_id = None
for m in memories:
for m in visible:
if m.get("id", "").startswith(memory_id):
m["text"] = new_text
m["timestamp"] = int(time.time())
found = True
full_id = m["id"]
break
if not found:
return [TextContent(type="text", text=f"Error: Memory '{memory_id}' not found")]
if not full_id:
return _text_result(f"Error: Memory '{memory_id}' not found")
for m in memories:
if m.get("id") == full_id:
m["text"] = new_text
m["timestamp"] = int(time.time())
break
_memory_manager.save(memories)
if _memory_vector and _memory_vector.healthy and full_id:
try:
@@ -145,24 +201,26 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
_memory_vector.add(full_id, new_text)
except Exception:
pass
return [TextContent(type="text", text=f"Memory updated: {new_text}")]
return _text_result(f"Memory updated: {new_text}")
elif action == "delete":
memory_id = arguments.get("memory_id", "")
if not memory_id:
return [TextContent(type="text", text="Error: delete needs memory_id")]
memories = _memory_manager.load_all()
return _text_result("Error: delete needs memory_id")
_owner, memories, visible, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
full_id = None
deleted_text = ""
deleted_category = ""
for m in memories:
for m in visible:
if m.get("id", "").startswith(memory_id):
full_id = m["id"]
deleted_text = m.get("text", "")
deleted_category = m.get("category", "")
break
if not full_id:
return [TextContent(type="text", text=f"Error: Memory '{memory_id}' not found")]
return _text_result(f"Error: Memory '{memory_id}' not found")
memories = [m for m in memories if m.get("id") != full_id]
_memory_manager.save(memories)
if _memory_vector and _memory_vector.healthy and full_id:
@@ -172,30 +230,32 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
pass
cat = f"[{deleted_category}] " if deleted_category else ""
snippet = deleted_text if len(deleted_text) <= 120 else deleted_text[:117] + "..."
return [TextContent(type="text", text=f"Memory deleted: {cat}{snippet} (id: {memory_id})")]
return _text_result(f"Memory deleted: {cat}{snippet} (id: {memory_id})")
elif action == "search":
query = arguments.get("text", "")
if not query:
return [TextContent(type="text", text="Error: search needs text (query)")]
memories = _memory_manager.load()
return _text_result("Error: search needs text (query)")
_owner, _all_memories, memories, scope_error = _scope_entries()
if scope_error:
return _text_result(scope_error)
if hasattr(_memory_manager, 'get_relevant_memories'):
results = _memory_manager.get_relevant_memories(query, memories, threshold=0.05, max_items=20)
else:
query_lower = query.lower()
results = [m for m in memories if query_lower in m.get("text", "").lower()][:20]
if not results:
return [TextContent(type="text", text=f"No memories found matching '{query}'.")]
return _text_result(f"No memories found matching '{query}'.")
lines = [f"Found {len(results)} matching memories:\n"]
for m in results:
cat = m.get("category", "fact")
mid = m.get("id", "?")[:8]
text = m.get("text", "")
lines.append(f"- [{cat}] `{mid}` — {text}")
return [TextContent(type="text", text="\n".join(lines))]
return _text_result("\n".join(lines))
else:
return [TextContent(type="text", text=f"Error: Unknown action '{action}'. Use: list, add, edit, delete, search")]
return _text_result(f"Error: Unknown action '{action}'. Use: list, add, edit, delete, search")
async def run():
package-lock.json
Generated
+6 -77
View File
@@ -4,90 +4,19 @@
"requires": true,
"packages": {
"": {
"dependencies": {
"@anthropic-ai/sdk": "^0.98.0"
},
"devDependencies": {
"@antithesishq/bombadil": "^0.3.2"
}
},
"node_modules/@anthropic-ai/sdk": {
"version": "0.98.0",
"resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.98.0.tgz",
"integrity": "sha512-N7aXtCvC5g6T1Y4V29lJjceu/zTkVkIZF0jdBvagr0TRFHuKeImffalGWEfqZKrvjH+IQbzJWw6TmSmUzrlMgg==",
"license": "MIT",
"dependencies": {
"json-schema-to-ts": "^3.1.1",
"standardwebhooks": "^1.0.0"
},
"bin": {
"anthropic-ai-sdk": "bin/cli"
},
"peerDependencies": {
"zod": "^3.25.0 || ^4.0.0"
},
"peerDependenciesMeta": {
"zod": {
"optional": true
}
"@antithesishq/bombadil": "^0.6.1"
}
},
"node_modules/@antithesishq/bombadil": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/@antithesishq/bombadil/-/bombadil-0.3.2.tgz",
"integrity": "sha512-ATy1w9ZY5gbny1H8DFc7rxZitT7DLLLFDiGcRZe+8TQiUrV5tLO+IJGOVNNLp3RpCqjZqSsxGiKoQsx31ipV1g==",
"version": "0.6.1",
"resolved": "https://registry.npmjs.org/@antithesishq/bombadil/-/bombadil-0.6.1.tgz",
"integrity": "sha512-d1iufG3MI7gSMSiSmMeNdcMW+qR0yQXL2zdkVynC3n3DYgFJYlYXKUQzygmqU12m4RWlR5iOdQU1hsx5UT6+IA==",
"dev": true,
"license": "MIT"
},
"node_modules/@babel/runtime": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.7.tgz",
"integrity": "sha512-Nq8OhGWiZIZGV6hLHoyAKLLcJihP/xFeBMGJoUrxTX2psI8dCifzLhZISFb+VWS3wFMRDmCGw5R+dOySCqPLhw==",
"license": "MIT",
"engines": {
"node": ">=6.9.0"
"bin": {
"bombadil": "bin/bombadil.js"
}
},
"node_modules/@stablelib/base64": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@stablelib/base64/-/base64-1.0.1.tgz",
"integrity": "sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==",
"license": "MIT"
},
"node_modules/fast-sha256": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/fast-sha256/-/fast-sha256-1.3.0.tgz",
"integrity": "sha512-n11RGP/lrWEFI/bWdygLxhI+pVeo1ZYIVwvvPkW7azl/rOy+F3HYRZ2K5zeE9mmkhQppyv9sQFx0JM9UabnpPQ==",
"license": "Unlicense"
},
"node_modules/json-schema-to-ts": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/json-schema-to-ts/-/json-schema-to-ts-3.1.1.tgz",
"integrity": "sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==",
"license": "MIT",
"dependencies": {
"@babel/runtime": "^7.18.3",
"ts-algebra": "^2.0.0"
},
"engines": {
"node": ">=16"
}
},
"node_modules/standardwebhooks": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/standardwebhooks/-/standardwebhooks-1.0.0.tgz",
"integrity": "sha512-BbHGOQK9olHPMvQNHWul6MYlrRTAOKn03rOe4A8O3CLWhNf4YHBqq2HJKKC+sfqpxiBY52pNeesD6jIiLDz8jg==",
"license": "MIT",
"dependencies": {
"@stablelib/base64": "^1.0.0",
"fast-sha256": "^1.3.0"
}
},
"node_modules/ts-algebra": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/ts-algebra/-/ts-algebra-2.0.0.tgz",
"integrity": "sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw==",
"license": "MIT"
}
}
}
package.json
+1 -4
View File
@@ -4,9 +4,6 @@
"url": "https://github.com/pewdiepie-archdaemon/odysseus.git"
},
"devDependencies": {
"@antithesishq/bombadil": "^0.3.2"
},
"dependencies": {
"@anthropic-ai/sdk": "^0.98.0"
"@antithesishq/bombadil": "^0.6.1"
}
}
requirements-optional.txt
+1 -1
View File
@@ -33,4 +33,4 @@ PyMuPDF
# magika (onnxruntime), already a core dep via fastembed. We avoid the
# [all]/Azure/audio extras (cloud + heavy). Pinned to a release >30 days old per
# the dependency-age discussion in issue #485.
markitdown[docx,pptx,xlsx,xls]==0.1.5
markitdown[docx,pptx,xlsx,xls]==0.1.6
requirements.txt
+2 -2
View File
@@ -3,8 +3,8 @@ uvicorn
python-multipart
python-dotenv
httpx
pydantic>=2.0
pydantic-settings>=2.0
pydantic>=2.13.4
pydantic-settings>=2.14.1
SQLAlchemy
pypdf
beautifulsoup4
routes/api_token_routes.py
+3
View File
@@ -31,6 +31,7 @@ ALLOWED_SCOPES = {
TOKEN_PROFILES = {
"chat": ["chat"],
"codex_todos": ["todos:read", "todos:write"],
"codex_documents": ["documents:read", "documents:write"],
"codex_email_drafts": ["email:read", "email:draft", "documents:read", "documents:write"],
}
@@ -159,6 +160,8 @@ def setup_api_token_routes() -> APIRouter:
payload = await request.json()
except Exception:
payload = {}
if not isinstance(payload, dict):
payload = {}
with get_db_session() as db:
token = db.query(ApiToken).filter(ApiToken.id == token_id).first()
if not token:
routes/assistant_routes.py
+3 -2
View File
@@ -16,6 +16,7 @@ from pydantic import BaseModel
from core.database import SessionLocal, CrewMember, ScheduledTask
from src.auth_helpers import get_current_user
from core.auth import RESERVED_USERNAMES
from src.task_scheduler import compute_next_run
@@ -89,11 +90,11 @@ def setup_assistant_routes(task_scheduler) -> APIRouter:
# check-in tasks seeded. Hitting any /assistant route under one of these
# used to seed a full CrewMember + Morning/Midday/Evening tasks under that
# owner, which then double-fired alongside the real user's check-ins.
_SYNTHETIC_OWNERS = frozenset({"internal-tool", "api", "demo", "system", ""})
# RESERVED_USERNAMES covers the same set; the `not owner` guard handles "".
async def _get_or_create(owner: str) -> CrewMember:
"""Return the per-owner assistant CrewMember, creating it on demand."""
if not owner or owner in _SYNTHETIC_OWNERS:
if not owner or owner in RESERVED_USERNAMES:
raise HTTPException(status_code=400, detail=f"Cannot seed assistant for {owner!r}")
db = SessionLocal()
try:
routes/auth_routes.py
+75 -11
View File
@@ -12,8 +12,8 @@ import re
from pathlib import Path
from core.atomic_io import atomic_write_json, atomic_write_text
from core.auth import AuthManager
from src.constants import DEEP_RESEARCH_DIR, MEMORY_FILE, SKILLS_DIR
from core.auth import AuthManager, RESERVED_USERNAMES, SetAdminResult, TOKEN_TTL
from src.constants import DEEP_RESEARCH_DIR, MEMORY_FILE, PASSWORD_MIN_LENGTH, SKILLS_DIR
from src.rate_limiter import RateLimiter
from src.settings_scrub import scrub_settings
from src.settings import (
@@ -73,6 +73,11 @@ class DeleteUserRequest(BaseModel):
class RenameUserRequest(BaseModel):
username: str
class SetAdminRequest(BaseModel):
is_admin: bool
class SetOpenRegistrationRequest(BaseModel):
enabled: bool
@@ -97,8 +102,12 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
raise HTTPException(429, "Too many requests — try again later")
if auth_manager.is_configured:
raise HTTPException(400, "Already configured")
if len(body.password) < 8:
raise HTTPException(400, "Password must be at least 8 characters")
if len(body.password) < PASSWORD_MIN_LENGTH:
raise HTTPException(400, f"Password must be at least {PASSWORD_MIN_LENGTH} characters")
if len(body.username.strip()) < 1:
raise HTTPException(400, "Username is required")
if body.username.lower() in RESERVED_USERNAMES:
raise HTTPException(403, "Username is reserved")
ok = await asyncio.to_thread(auth_manager.setup, body.username, body.password)
if not ok:
raise HTTPException(500, "Setup failed")
@@ -113,10 +122,12 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
raise HTTPException(400, "Run setup first")
if not auth_manager.signup_enabled:
raise HTTPException(403, "Registration is disabled. Ask an admin for an account.")
if len(body.password) < 8:
raise HTTPException(400, "Password must be at least 8 characters")
if len(body.password) < PASSWORD_MIN_LENGTH:
raise HTTPException(400, f"Password must be at least {PASSWORD_MIN_LENGTH} characters")
if len(body.username.strip()) < 1:
raise HTTPException(400, "Username is required")
if body.username.lower() in RESERVED_USERNAMES:
raise HTTPException(403, "Username is reserved")
ok = await asyncio.to_thread(auth_manager.create_user, body.username, body.password, is_admin=False)
if not ok:
raise HTTPException(409, "Username already taken")
@@ -139,6 +150,8 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
raise HTTPException(401, "Invalid 2FA code")
# All checks passed — create session (password already verified above)
token = await asyncio.to_thread(auth_manager.create_session_trusted, username)
if not token:
raise HTTPException(401, "Invalid credentials")
cookie_kwargs = dict(
key=SESSION_COOKIE,
value=token,
@@ -148,7 +161,7 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
path="/",
)
if body.remember:
cookie_kwargs["max_age"] = 60 * 60 * 24 * 7 # 7 days
cookie_kwargs["max_age"] = TOKEN_TTL
response.set_cookie(**cookie_kwargs)
return {"ok": True, "username": username}
@@ -177,13 +190,18 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
pass
return result
@router.get("/policy")
async def auth_policy():
"""Return public auth policy constants for the frontend."""
return auth_manager.policy()
@router.post("/change-password")
async def change_password(body: ChangePasswordRequest, request: Request):
user = _get_current_user(request)
if not user:
raise HTTPException(401, "Not authenticated")
if len(body.new_password) < 8:
raise HTTPException(400, "Password must be at least 8 characters")
if len(body.new_password) < PASSWORD_MIN_LENGTH:
raise HTTPException(400, f"Password must be at least {PASSWORD_MIN_LENGTH} characters")
current_token = request.cookies.get(SESSION_COOKIE)
ok = await asyncio.to_thread(auth_manager.change_password, user, body.current_password, body.new_password)
if not ok:
@@ -263,8 +281,12 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
user = _get_current_user(request)
if not user or not auth_manager.is_admin(user):
raise HTTPException(403, "Admin only")
if len(body.password) < 8:
raise HTTPException(400, "Password must be at least 8 characters")
if len(body.password) < PASSWORD_MIN_LENGTH:
raise HTTPException(400, f"Password must be at least {PASSWORD_MIN_LENGTH} characters")
if len(body.username.strip()) < 1:
raise HTTPException(400, "Username is required")
if body.username.lower() in RESERVED_USERNAMES:
raise HTTPException(403, "Username is reserved")
ok = auth_manager.create_user(body.username, body.password, body.is_admin)
if not ok:
raise HTTPException(409, "Username already taken")
@@ -427,6 +449,23 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
except Exception as e:
logger.warning("Failed to rename upload owner references %s -> %s: %s", old_username, new_username, e)
# direct personal RAG uploads live in per-owner directories and the
# vector metadata also carries the username used for owner-filtered
# search. Keep both in sync with the auth rename.
try:
from routes.personal_routes import rename_personal_upload_owner
personal_docs_manager = getattr(request.app.state, "personal_docs_manager", None)
if personal_docs_manager is not None:
rag_manager = getattr(personal_docs_manager, "rag_manager", None)
rename_personal_upload_owner(
old_username,
new_username,
personal_docs_manager=personal_docs_manager,
rag_manager=rag_manager,
)
except Exception as e:
logger.warning("Failed to rename personal RAG upload owner references %s -> %s: %s", old_username, new_username, e)
# skills: SKILL.md frontmatter carries owner: <username>; the usage
# sidecar (_usage.json) keys entries as owner::skill-name. Both must
# be updated or the renamed user's Skills panel goes empty.
@@ -487,6 +526,31 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
invalidator()
return {"ok": True, "username": new_username, "renamed_self": old_username == user}
@router.put("/users/{username}/admin")
async def set_user_admin(username: str, body: SetAdminRequest, request: Request):
"""Promote/demote a user to/from admin. Admin only.
The last remaining admin can't be demoted (no lockout). Self-demotion
is allowed while another admin exists; the `self` flag tells the UI to
reload the acting user into the normal-user view.
"""
user = _get_current_user(request)
if not user or not auth_manager.is_admin(user):
raise HTTPException(403, "Admin only")
result = auth_manager.set_admin(username, body.is_admin, user)
if result is SetAdminResult.USER_NOT_FOUND:
raise HTTPException(404, "User not found")
if result is SetAdminResult.NOT_AUTHORIZED:
raise HTTPException(403, "Admin only")
if result is SetAdminResult.LAST_ADMIN:
raise HTTPException(400, "Cannot demote the last admin")
target = (username or "").strip().lower()
return {
"ok": True,
"is_admin": body.is_admin,
"self": target == (user or "").strip().lower(),
}
@router.post("/signup-toggle", deprecated=True)
async def toggle_signup(request: Request):
"""
routes/calendar_routes.py
+48 -1
View File
@@ -34,6 +34,24 @@ def _ics_naive_dtstart(dt):
return datetime(dt.year, dt.month, dt.day)
return dt
def _ensure_positive_duration(start_dt, end_dt, all_day):
"""Clamp an imported event's end so it has a positive duration.
Some .ics exporters write a single-day all-day event with DTEND equal to
DTSTART (treating DTEND as inclusive rather than the RFC 5545 exclusive
bound). Stored verbatim that produces a zero-duration row, which the
list_events overlap filter (dtstart < end AND dtend > start) silently
drops — the event never appears on the calendar even though the web UI
would otherwise show it. Normalize a non-positive end to the same default
span used when DTEND is absent: one day for all-day events, one hour
otherwise.
"""
if end_dt <= start_dt:
return start_dt + (timedelta(days=1) if all_day else timedelta(hours=1))
return end_dt
# Single-user fallback identity. Used only when:
# 1. The app is configured for single-user (no auth middleware), AND
# 2. The request didn't resolve to an authenticated user.
@@ -434,6 +452,20 @@ def _parse_dt(s: str) -> datetime:
if t is not None:
return base.replace(hour=t[0], minute=t[1])
# time-first: "3pm today", "9am tomorrow", "11pm tonight"
# (parity with parse_due_for_user, which handles these via the same form)
m = _re.match(r'^(.+?)\s+(today|tonight|tomorrow|tmrw|yesterday)$', lower)
if m:
time_part, word = m.group(1).strip(), m.group(2)
base = today
if word in ("tomorrow", "tmrw"):
base = today + timedelta(days=1)
elif word == "yesterday":
base = today - timedelta(days=1)
t = _parse_time(time_part)
if t is not None:
return base.replace(hour=t[0], minute=t[1])
# next <weekday> [at] TIME
weekdays = ["monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday"]
m = _re.match(r'^next\s+(\w+)(?:\s+at)?\s*(.*)$', lower)
@@ -1226,7 +1258,7 @@ def setup_calendar_routes() -> APIRouter:
db.commit()
db.refresh(target_cal)
imported = skipped = 0
imported = skipped = repaired = 0
for comp in cal_data.walk():
if comp.name != "VEVENT":
continue
@@ -1262,6 +1294,18 @@ def setup_calendar_routes() -> APIRouter:
.first()
)
if existing:
# An import predating the clamp below may have stored
# this same event with a non-positive duration, which
# the list_events overlap filter hides. Re-importing
# lands here and would skip without touching that row,
# so the event would stay invisible. Backfill the clamp
# onto the stored row before skipping it.
fixed_end = _ensure_positive_duration(
existing.dtstart, existing.dtend, bool(existing.all_day)
)
if fixed_end != existing.dtend:
existing.dtend = fixed_end
repaired += 1
skipped += 1
continue
@@ -1295,6 +1339,8 @@ def setup_calendar_routes() -> APIRouter:
else:
end_dt = start_dt + timedelta(hours=1)
end_dt = _ensure_positive_duration(start_dt, end_dt, all_day)
ev = CalendarEvent(
uid=uid_val,
calendar_id=target_cal.id,
@@ -1315,6 +1361,7 @@ def setup_calendar_routes() -> APIRouter:
"ok": True,
"imported": imported,
"skipped": skipped,
"repaired": repaired,
"calendar": cal_display,
"calendar_id": target_cal.id,
}
routes/chat_helpers.py
+157 -26
View File
@@ -14,7 +14,7 @@ from core.database import Session as DBSession, ModelEndpoint
from src.llm_core import normalize_model_id
from src.endpoint_resolver import normalize_base
from src.context_compactor import maybe_compact, trim_for_context
from src.auth_helpers import get_current_user
from src.auth_helpers import effective_user
from src.prompt_security import untrusted_context_message
from routes.prefs_routes import _load_for_user as load_prefs_for_user
@@ -22,6 +22,47 @@ from fastapi import HTTPException
logger = logging.getLogger(__name__)
_CASUAL_OPENING_RE = re.compile(
r"^\s*(?:h+i+|hey+|hello+|yo+|sup+|what'?s up|wass?up|hiya|howdy|"
r"lol|lmao|haha+|hehe+|thanks?|thank you|ty|idk|dunno|meh|bruh|bro)\b(?P<tail>.*)$",
re.IGNORECASE,
)
_CASUAL_BLOCKLIST_RE = re.compile(
r"\b(?:cookbook|serve|serving|launch|start|vllm|sglang|llama\.?cpp|ollama|"
r"download|model|email|document|doc|note|calendar|task|search|web|research|"
r"file|folder|repo|git|settings?|endpoint|api|token|mcp)\b",
re.IGNORECASE,
)
def _is_casual_low_signal(text: str) -> bool:
"""Short greetings/slang should not pull memory, skills, RAG, or docs."""
s = str(text or "").strip()
m = _CASUAL_OPENING_RE.match(s)
if not m:
return False
tail = m.group("tail") or ""
if _CASUAL_BLOCKLIST_RE.search(tail):
return False
tail_words = re.findall(r"[A-Za-z0-9_'-]+", tail)
return len(tail_words) <= 2
# Strong references to in-flight fire-and-forget tasks scheduled from this
# module. asyncio only keeps weak references to tasks created via
# create_task, so without this the GC can collect a task mid-execution and
# the background work (extraction, auto-naming) silently never runs.
# Mirrors WebhookManager._spawn_tracked from src/webhook_manager.py.
_BG_TASKS: set[asyncio.Task] = set()
def _spawn_bg(coro) -> asyncio.Task:
"""Schedule a background task and hold a strong reference until it finishes."""
task = asyncio.create_task(coro)
_BG_TASKS.add(task)
task.add_done_callback(_BG_TASKS.discard)
return task
# ── Data containers ────────────────────────────────────────────────────── #
@@ -63,6 +104,9 @@ class ChatContext:
# The chat route emits a doc_update SSE event for each before streaming
# begins, so the editor pane switches to the new doc immediately.
auto_opened_docs: list = field(default_factory=list)
# Uploads attached to this user turn, resolved and owner-checked for the
# agent's private context. This is not emitted to the browser.
uploaded_files: list = field(default_factory=list)
# ── Helpers ────────────────────────────────────────────────────────────── #
@@ -78,7 +122,7 @@ def _enforce_chat_privileges(request, sess) -> None:
which means unrestricted allowed_models / zero cap -> no-op for them.
"""
try:
user = get_current_user(request)
user = effective_user(request)
except Exception:
user = None
if not user:
@@ -159,17 +203,9 @@ async def auto_name_session(session_manager, sess):
return
owner = getattr(sess, "owner", None)
t_url, t_model, t_headers = resolve_task_endpoint(owner=owner)
if not t_model:
# If no task/utility model is configured at all, fall back to
# the session's own model so auto-naming still works even on
# minimal setups.
from src.endpoint_resolver import resolve_endpoint
_fallback = resolve_endpoint("default", owner=owner)
if _fallback and _fallback[1]:
t_url, t_model, t_headers = _fallback
else:
t_url, t_model, t_headers = sess.endpoint_url, sess.model, sess.headers
t_url, t_model, t_headers = resolve_task_endpoint(
sess.endpoint_url, sess.model, sess.headers, owner=owner
)
if not t_model:
logger.debug("[auto-name] No model provided, skipping")
return
@@ -333,6 +369,59 @@ async def preprocess(
)
def build_uploaded_file_manifest(att_ids: list, upload_handler, owner: Optional[str]) -> list[dict]:
"""Resolve current-turn upload IDs into a small tool-facing manifest.
The chat UI already sends attachment ids, and preprocessing inlines as much
text as fits. Agent mode still needs a discoverable bridge for files whose
content was truncated/omitted or when the model chooses file tools. Only
owner-authorized uploads are included, and paths must remain inside the
configured upload directory.
"""
if not att_ids or not upload_handler or not hasattr(upload_handler, "resolve_upload"):
return []
def _read_file_can_open(path: str) -> bool:
try:
from src.tool_execution import _resolve_tool_path
return _resolve_tool_path(path) == os.path.realpath(path)
except Exception:
return False
manifest: list[dict] = []
for att_id in att_ids:
try:
info = upload_handler.resolve_upload(str(att_id), owner=owner)
except Exception:
logger.debug("Failed to resolve upload %r for agent manifest", att_id, exc_info=True)
continue
if not isinstance(info, dict):
continue
path = info.get("path")
if path:
try:
inside = True
if hasattr(upload_handler, "_inside_upload_dir"):
inside = bool(upload_handler._inside_upload_dir(path))
elif hasattr(upload_handler, "inside_base_dir"):
inside = bool(upload_handler.inside_base_dir(path))
if not inside or not os.path.exists(path) or not _read_file_can_open(path):
path = None
except Exception:
path = None
manifest.append({
"id": info.get("id") or str(att_id),
"name": info.get("name") or info.get("original_name") or str(att_id),
"mime": info.get("mime", ""),
"size": info.get("size", 0),
"path": path,
})
return manifest
def add_user_message(sess, chat_handler, preprocessed: PreprocessedMessage, incognito: bool = False):
"""Add user message to session history and update session name.
In incognito mode, still add to in-memory history (for conversation context)
@@ -346,11 +435,11 @@ def add_user_message(sess, chat_handler, preprocessed: PreprocessedMessage, inco
def fire_message_event(request, webhook_manager, session_id: str, sess, message: str, compare_mode: bool = False):
"""Fire webhook and event_bus events for a new user message."""
if webhook_manager and not compare_mode:
asyncio.create_task(webhook_manager.fire("chat.message", {
webhook_manager.fire_and_forget("chat.message", {
"session_id": session_id, "model": sess.model, "message": message[:2000],
}))
})
from src.event_bus import fire_event
user = get_current_user(request)
user = effective_user(request)
fire_event("message_sent", user)
@@ -505,6 +594,29 @@ def _normalize_model_id_from_cache(sess) -> Optional[str]:
return None
def _session_is_research_spinoff(sess) -> bool:
"""True if this session was created via research "Discuss" spin-off.
Detected by the primer system message the spin-off endpoint seeds into
history (metadata ``research_spinoff_from``). Such sessions are grounded
on the seeded report, so global memory + personal-doc RAG injection is
suppressed for them (the report is the sole knowledge base). Handles both
ChatMessage objects and plain dicts.
"""
for m in getattr(sess, "history", []) or []:
role = getattr(m, "role", None)
if role is None and isinstance(m, dict):
role = m.get("role")
if role != "system":
continue
md = getattr(m, "metadata", None)
if md is None and isinstance(m, dict):
md = m.get("metadata")
if (md or {}).get("research_spinoff_from"):
return True
return False
async def build_chat_context(
sess,
request,
@@ -553,9 +665,16 @@ async def build_chat_context(
if not incognito:
fire_message_event(request, webhook_manager, session_id, sess, message, compare_mode)
# Resolve user prefs
user = get_current_user(request)
# Resolve owner-scoped prefs/context. Browser requests keep the cookie user;
# bearer-token chat requests use the token owner instead of the "api" sentinel.
user = effective_user(request)
uprefs = load_prefs_for_user(user)
uploaded_files = build_uploaded_file_manifest(
att_ids or [],
getattr(chat_handler, "upload_handler", None),
getattr(sess, "owner", None),
)
casual_low_signal = _is_casual_low_signal(message)
# Memory enabled?
mem_enabled = not incognito and not no_memory and uprefs.get("memory_enabled", True)
@@ -565,18 +684,29 @@ async def build_chat_context(
if not allow_tool_preprocessing:
mem_enabled = False
skills_enabled = False
if casual_low_signal:
mem_enabled = False
skills_enabled = False
logger.debug(
"Memory enabled=%s for user=%s (incognito=%s, no_memory=%s, pref=%s)",
mem_enabled, user, incognito, no_memory, uprefs.get("memory_enabled", "NOT_SET"),
)
# Research-spinoff ("Discuss") sessions are grounded on the seeded report:
# the primer system message IS the knowledge base. Injecting global memory
# or personal-doc RAG on every turn pulls in keyword-matched but off-topic
# facts ("wrong data") and competes with the report, so suppress both here.
is_research_spinoff = _session_is_research_spinoff(sess)
if is_research_spinoff:
mem_enabled = False
# Use RAG?
use_rag_val = (str(use_rag).lower() != "false") if use_rag is not None else True
if incognito or not allow_tool_preprocessing:
if incognito or not allow_tool_preprocessing or is_research_spinoff or casual_low_signal:
use_rag_val = False
# If pre-fetched search context was provided (compare mode), skip live web search
skip_web = bool(search_context) or not allow_tool_preprocessing
skip_web = bool(search_context) or not allow_tool_preprocessing or casual_low_signal
# Build context preface
# The stream path uses enhanced_message (with CoT/preprocessing applied),
@@ -595,7 +725,7 @@ async def build_chat_context(
incognito=incognito,
use_skills=skills_enabled,
)
if use_rag is not None:
if use_rag is not None or is_research_spinoff or casual_low_signal:
_preface_kwargs["use_rag"] = use_rag_val
preface, rag_sources, web_sources = chat_processor.build_context_preface(**_preface_kwargs)
@@ -603,7 +733,7 @@ async def build_chat_context(
used_memories = getattr(chat_processor, '_last_used_memories', [])
# Inject pre-fetched search context (compare mode)
if search_context and allow_tool_preprocessing:
if search_context and allow_tool_preprocessing and not casual_low_signal:
preface.append(untrusted_context_message("prefetched search context", search_context))
# YouTube transcripts
@@ -662,6 +792,7 @@ async def build_chat_context(
preset=preset,
preprocessed=preprocessed,
auto_opened_docs=auto_opened_docs,
uploaded_files=uploaded_files,
)
@@ -1081,7 +1212,7 @@ def run_post_response_tasks(
)))
if _extraction_jobs:
asyncio.create_task(_run_extraction_jobs_sequentially(session_id, _extraction_jobs))
_spawn_bg(_run_extraction_jobs_sequentially(session_id, _extraction_jobs))
# Token accumulation
if last_metrics:
@@ -1089,11 +1220,11 @@ def run_post_response_tasks(
# Webhook
if webhook_manager and not compare_mode:
asyncio.create_task(webhook_manager.fire("chat.completed", {
webhook_manager.fire_and_forget("chat.completed", {
"session_id": session_id, "model": sess.model,
"user_message": message, "response": full_response[:2000],
}))
})
# Auto-name
if needs_auto_name(sess.name):
asyncio.create_task(auto_name_session(session_manager, sess))
_spawn_bg(auto_name_session(session_manager, sess))
routes/chat_routes.py
+131 -22
View File
@@ -6,7 +6,7 @@ import os
import time
import logging
from datetime import datetime
from typing import Dict, Any, AsyncGenerator, List
from typing import Dict, Any, AsyncGenerator, List, Optional
from fastapi import APIRouter, Request, HTTPException, Form, Query
from fastapi.responses import StreamingResponse
@@ -23,12 +23,13 @@ from src.endpoint_resolver import normalize_base as _normalize_base, build_chat_
from src.session_search import search_session_messages
from src.prompt_security import untrusted_context_message
from core.exceptions import SessionNotFoundError
from src.auth_helpers import get_current_user
from src.auth_helpers import effective_user, get_current_user
from routes.session_routes import _verify_session_owner
from routes.document_helpers import _owner_session_filter
from core.database import SessionLocal, get_session_mode, set_session_mode
from core.database import Session as DBSession, ChatMessage as DBChatMessage
from core.database import Document as DBDocument, ModelEndpoint
from core.log_safety import redact_url
from routes.research_routes import _resolve_research_endpoint
from routes.model_routes import _visible_models
from routes.chat_helpers import (
@@ -126,7 +127,8 @@ def _clear_orphaned_session_endpoint(sess, owner: str | None = None) -> bool:
sess.model = ""
sess.headers = {}
return True
except Exception:
except Exception as e:
logger.warning("Failed to clear orphaned session endpoint", exc_info=e)
db.rollback()
return False
finally:
@@ -144,7 +146,8 @@ def _endpoint_cache_contains_model(endpoint, model: str) -> bool:
return True
try:
models = json.loads(raw) if isinstance(raw, str) else raw
except Exception:
except Exception as e:
logger.warning("Failed to parse cached models list, treating as containing model", exc_info=e)
return True
if not isinstance(models, list) or not models:
return True
@@ -236,7 +239,8 @@ def _recover_empty_session_model(sess, session_id: str, owner: str | None = None
is_chatgpt_subscription = False
try:
cached = json.loads(ep.cached_models) if isinstance(ep.cached_models, str) else (ep.cached_models or [])
except Exception:
except Exception as e:
logger.warning("Failed to parse cached_models for endpoint %r", getattr(ep, "id", "?"), exc_info=e)
cached = []
if not cached:
visible = []
@@ -360,7 +364,7 @@ def setup_chat_routes(
sess = session_manager.get_session(session)
except KeyError:
raise HTTPException(404, f"Session '{session}' not found")
owner = get_current_user(request)
owner = effective_user(request)
if _clear_orphaned_session_endpoint(sess, owner=owner):
raise HTTPException(400, "Selected model endpoint was removed. Pick another model in Settings.")
@@ -526,6 +530,66 @@ def setup_chat_routes(
active_doc_id = form_data.get("active_doc_id", "").strip()
logger.info(f"[doc-inject] chat_mode={chat_mode}, active_doc_id={active_doc_id!r}")
# Active email reader — when the user has an email open in the UI, the
# frontend passes its uid/folder/account so "reply", "summarize this",
# etc. resolve to the real email instead of the agent inventing a
# fake markdown draft.
active_email_uid = form_data.get("active_email_uid", "").strip()
active_email_folder = form_data.get("active_email_folder", "INBOX").strip() or "INBOX"
active_email_account = form_data.get("active_email_account", "").strip()
active_email_ctx: Optional[Dict[str, str]] = None
# Always reset between requests so a stale active-email pointer from
# a previous turn (different reader closed, different account, etc.)
# can't leak in when the user has no email open this turn.
try:
from src.tool_implementations import clear_active_email
clear_active_email()
except Exception:
pass
if active_email_uid:
active_email_ctx = {
"uid": active_email_uid,
"folder": active_email_folder,
"account": active_email_account,
}
# Try to enrich with subject + from so the agent's system prompt
# block can quote them. Best-effort: a stale cache is fine, a
# missing email just means we pass uid/folder/account only.
try:
from routes.email_routes import _read_cache_get, _read_cache_key
_ck = _read_cache_key(active_email_account or None, active_email_folder, active_email_uid, owner=get_current_user(request))
_cached_email = _read_cache_get(_ck)
if _cached_email and isinstance(_cached_email, dict):
active_email_ctx["subject"] = str(_cached_email.get("subject") or "")
active_email_ctx["from"] = str(
_cached_email.get("from_address")
or _cached_email.get("from")
or _cached_email.get("from_name")
or ""
)
_body_preview = (_cached_email.get("body") or "")[:2000]
if _body_preview:
active_email_ctx["body_preview"] = _body_preview
except Exception as _e:
logger.debug(f"[email-inject] cache enrich skipped: {_e}")
# Stash so email tools can resolve "this email" without UID guessing.
try:
from src.tool_implementations import set_active_email
set_active_email(
uid=active_email_uid,
folder=active_email_folder,
account=active_email_account or None,
subject=active_email_ctx.get("subject"),
sender=active_email_ctx.get("from"),
)
except Exception as _e:
logger.debug(f"[email-inject] set_active_email failed: {_e}")
logger.info(
"[email-inject] active_email uid=%s folder=%s account=%s subject=%r",
active_email_uid, active_email_folder, active_email_account or "(default)",
active_email_ctx.get("subject", ""),
)
try:
# Attachment-only sends: skip the message-required check when the
# user has attached one or more files (the attachment IS the action).
@@ -540,7 +604,7 @@ def setup_chat_routes(
# but BEFORE loading. Prevents cross-user session hijack.
_verify_session_owner(request, session)
sess = session_manager.get_session(session)
owner = get_current_user(request)
owner = effective_user(request)
if _clear_orphaned_session_endpoint(sess, owner=owner):
raise HTTPException(400, "Selected model endpoint was removed. Pick another model in Settings.")
# Issue #587: picker shows a model from the endpoint cache but
@@ -571,7 +635,7 @@ def setup_chat_routes(
_enforce_chat_privileges(request, sess)
# Ensure session has auth headers
resolve_session_auth(sess, session, owner=get_current_user(request))
resolve_session_auth(sess, session, owner=effective_user(request))
# Check for research_pending BEFORE mode persist overwrites it
do_research = str(use_research).lower() == "true"
@@ -586,8 +650,8 @@ def setup_chat_routes(
elif attachments:
try:
att_ids = [str(x) for x in json.loads(attachments)]
except Exception:
pass
except Exception as e:
logger.warning("Failed to parse attachments JSON, ignoring attachments", exc_info=e)
no_memory = str(form_data.get("no_memory", "")).lower() == "true"
pre_context_tool_policy = build_effective_tool_policy(
@@ -641,15 +705,27 @@ def setup_chat_routes(
active_doc_id,
)
active_doc = None
elif doc_session and doc_session != session:
logger.warning(
"[doc-inject] ignoring stale active_doc_id %s from session %s while in session %s",
active_doc_id,
doc_session,
session,
)
active_doc = None
else:
# NOTE: previously dropped the doc when doc.session_id
# != current chat session — but that broke the common
# case of "open an email draft from one chat, ask a
# different chat to write into it". The frontend only
# sends active_doc_id for docs currently visible in
# the UI, and we already owner-checked above, so trust
# the explicit signal. We just log the mismatch and
# re-bind the doc to the current session so future
# turns find it via the session-fallback path too.
if doc_session and doc_session != session:
logger.info(
"[doc-inject] cross-session active_doc_id %s (was session %s, now %s) — accepting and rebinding",
active_doc_id, doc_session, session,
)
try:
active_doc.session_id = session
_doc_db.commit()
except Exception as _e:
_doc_db.rollback()
logger.warning(f"[doc-inject] session rebind failed: {_e}")
logger.info(f"[doc-inject] found by ID: title={active_doc.title!r}, lang={active_doc.language!r}, is_active={active_doc.is_active}, content_len={len(active_doc.current_content or '')}")
else:
logger.warning(f"[doc-inject] NOT FOUND by ID {active_doc_id}")
@@ -714,6 +790,21 @@ def setup_chat_routes(
"manage_skills", # skill presets tied to user
})
# Active email reader open → strip the tools that let the agent
# "drift" to a new compose: create_document (writes a fake email-
# shaped .md file) and send_email (sends fresh to a recipient the
# agent invented). With those gone, the only paths left for "write
# email saying X" are ui_control open_email_reply (draft) and
# reply_to_email (immediate send) — both of which use the open
# email's UID. Code-level enforcement instead of relying on a
# prompt rule the model can ignore.
if active_email_ctx and active_email_ctx.get("uid"):
disabled_tools.update({
"create_document",
"send_email",
"mcp__email__send_email",
})
# Enforce per-user privileges
_privs = {}
_user = ctx.user
@@ -739,7 +830,11 @@ def setup_chat_routes(
from src.settings import get_setting
_global_disabled = get_setting("disabled_tools", [])
if _global_disabled and isinstance(_global_disabled, list):
disabled_tools.update(_global_disabled)
explicit_web_allowed = allow_web_search is not None and str(allow_web_search).lower() == "true"
if explicit_web_allowed:
disabled_tools.update(t for t in _global_disabled if t not in {"web_search", "web_fetch"})
else:
disabled_tools.update(_global_disabled)
# Light auto-escalation: the user is in chat mode and just expressed a
# notes/calendar/email intent. Grant the relevant managers but withhold
@@ -836,7 +931,7 @@ def setup_chat_routes(
if effective_do_research:
_r_ep, _r_model, _r_headers = _resolve_research_endpoint(sess)
_auth_keys = list(_r_headers.keys()) if _r_headers else []
logger.info(f"Research endpoint resolved: model={_r_model}, endpoint={_r_ep}, auth_keys={_auth_keys}, sess_headers_keys={list(sess.headers.keys()) if isinstance(sess.headers, dict) else type(sess.headers)}")
logger.info(f"Research endpoint resolved: model={_r_model}, endpoint={redact_url(_r_ep)}, auth_keys={_auth_keys}, sess_headers_keys={list(sess.headers.keys()) if isinstance(sess.headers, dict) else type(sess.headers)}")
# Clarification round: only for very short/vague queries on first research message.
# Skip in compare mode — each pane is a fresh session, so every one would
@@ -1160,7 +1255,14 @@ def setup_chat_routes(
try:
from src.settings import get_setting
from src.agent_tools import MAX_AGENT_ROUNDS as _DEFAULT_ROUNDS
_tool_budget = int(get_setting("agent_max_tool_calls", 0))
# Per-message tool budget from settings; guard defensively in
# case settings.json was hand-edited to a non-numeric value
# (the HTTP admin endpoint validates, but direct edits bypass
# it). 0 = unlimited, matching auth_routes set_settings().
try:
_tool_budget = int(get_setting("agent_max_tool_calls", 0))
except (TypeError, ValueError):
_tool_budget = 0
# Per-message round cap from settings; clamp defensively in
# case settings.json was hand-edited to a bad value.
try:
@@ -1169,6 +1271,10 @@ def setup_chat_routes(
_max_rounds = _DEFAULT_ROUNDS
_max_rounds = max(1, min(_max_rounds, 200))
_forced_tools = None
if allow_web_search is not None and str(allow_web_search).lower() == "true":
_forced_tools = {"web_search", "web_fetch"}
async for chunk in stream_agent_loop(
sess.endpoint_url,
sess.model,
@@ -1181,6 +1287,7 @@ def setup_chat_routes(
max_rounds=_max_rounds,
context_length=ctx.context_length,
active_document=active_doc,
active_email=active_email_ctx,
session_id=session,
disabled_tools=disabled_tools if disabled_tools else None,
tool_policy=tool_policy,
@@ -1189,6 +1296,8 @@ def setup_chat_routes(
plan_mode=plan_mode,
approved_plan=approved_plan or None,
workspace=workspace or None,
forced_tools=_forced_tools,
uploaded_files=ctx.uploaded_files,
):
if chunk.startswith("data: ") and not chunk.startswith("data: [DONE]"):
try:
@@ -1394,7 +1503,7 @@ def setup_chat_routes(
if not q or not q.strip():
return []
_user = get_current_user(request)
_user = effective_user(request)
return [
result.to_dict()
for result in search_session_messages(
routes/codex_routes.py
+120 -14
View File
@@ -15,6 +15,7 @@ from typing import Any
from fastapi import APIRouter, BackgroundTasks, Body, HTTPException, Request
from fastapi.responses import StreamingResponse
from core.middleware import require_admin
from src.auth_helpers import require_authenticated_request, require_user
from src.tool_implementations import do_manage_notes
from src.constants import COOKBOOK_STATE_FILE
@@ -46,8 +47,12 @@ def _ssh_prefix_for_task(task: dict) -> tuple[str, str]:
shell metacharacters in ``remoteHost`` is rejected with 400 rather than
injected.
"""
host = validate_remote_host((task.get("remoteHost") or "").strip() or None) or ""
ssh_port = validate_ssh_port((task.get("sshPort") or "").strip() or None) or ""
raw_host = task.get("remoteHost")
raw_port = task.get("sshPort")
host_value = str(raw_host).strip() if raw_host is not None else None
port_value = str(raw_port).strip() if raw_port is not None else None
host = validate_remote_host(host_value or None) or ""
ssh_port = validate_ssh_port(port_value or None) or ""
port_flag = f"-p {ssh_port} " if ssh_port and ssh_port != "22" else ""
return host, port_flag
@@ -91,6 +96,34 @@ def _scope_owner(request: Request, allowed: set[str]) -> str:
return require_user(request)
def _scope_owner_all(request: Request, required: set[str]) -> str:
"""Return owner only when an API token has every required scope."""
if getattr(request.state, "api_token", False):
scopes = set(getattr(request.state, "api_token_scopes", []) or [])
missing = required - scopes
if missing:
raise HTTPException(403, f"API token missing required scope: {' and '.join(sorted(missing))}")
owner = getattr(request.state, "api_token_owner", None)
if not owner:
raise HTTPException(403, "API token has no owner")
return owner
return require_user(request)
def _require_cookbook_scope(request: Request, allowed: set[str]) -> str:
"""Authorize a Codex cookbook route.
For API-token callers, enforce the given scope set.
For cookie-session callers, additionally require admin privileges
because cookbook surfaces expose host topology, task logs, tmux
commands, and model-serving controls.
"""
owner = _scope_owner(request, allowed)
if not getattr(request.state, "api_token", False):
require_admin(request)
return owner
def _find_endpoint(router: APIRouter | None, method: str, path: str):
if router is None:
return None
@@ -100,6 +133,18 @@ def _find_endpoint(router: APIRouter | None, method: str, path: str):
return None
def _clamp_pagination(offset: Any, limit: Any, *, default_limit: int = 50, max_limit: int = 50) -> tuple[int, int]:
try:
parsed_offset = int(0 if offset in (None, "") else offset)
except (TypeError, ValueError):
raise HTTPException(400, "Invalid offset")
try:
parsed_limit = int(default_limit if limit in (None, "") else limit)
except (TypeError, ValueError):
raise HTTPException(400, "Invalid limit")
return max(0, parsed_offset), max(1, min(parsed_limit, max_limit))
def setup_codex_routes(
email_router: APIRouter | None = None,
memory_router: APIRouter | None = None,
@@ -138,7 +183,7 @@ def setup_codex_routes(
"read": scoped(EMAIL_READ_SCOPES),
"draft": scoped(EMAIL_DRAFT_SCOPES),
"send": scoped(EMAIL_SEND_SCOPES),
"actions": ["list", "read", "draft", "send"],
"actions": ["list", "read", "draft_document", "draft", "send"],
},
"memory": {
"read": scoped(MEMORY_READ_SCOPES),
@@ -262,6 +307,59 @@ def setup_codex_routes(
# Both handlers in routes/email_routes.py already accept `owner=` via
# FastAPI Depends, so we call them directly without patching state.
def _email_draft_document_content(body: dict[str, Any]) -> str:
def clean(v: Any) -> str:
if isinstance(v, list):
return ", ".join(str(x).strip() for x in v if str(x).strip())
return str(v or "").strip()
to = clean(body.get("to"))
cc = clean(body.get("cc"))
bcc = clean(body.get("bcc"))
subject = clean(body.get("subject"))
in_reply_to = clean(body.get("in_reply_to"))
references = clean(body.get("references"))
body_text = str(body.get("body") or body.get("body_html") or "").strip()
lines = [
f"To: {to}",
]
if cc:
lines.append(f"Cc: {cc}")
if bcc:
lines.append(f"Bcc: {bcc}")
lines.append(f"Subject: {subject}")
if in_reply_to:
lines.append(f"In-Reply-To: {in_reply_to}")
if references:
lines.append(f"References: {references}")
lines.extend(["---", body_text])
return "\n".join(lines).rstrip() + "\n"
@router.post("/emails/draft-document")
async def codex_email_draft_document(request: Request, body: dict[str, Any] = Body(default_factory=dict)):
owner = _scope_owner(request, EMAIL_DRAFT_SCOPES)
docs_owner = _scope_owner_all(request, DOCS_WRITE_SCOPES)
if docs_owner != owner:
raise HTTPException(403, "API token owner mismatch")
if documents_create_endpoint is None:
raise HTTPException(503, "Documents integration is not available")
from routes.document_routes import DocumentCreate
subject = str(body.get("subject") or "Email draft").strip() or "Email draft"
title = str(body.get("title") or subject).strip() or "Email draft"
req = DocumentCreate(
session_id=body.get("session_id"),
title=title,
language="email",
content=_email_draft_document_content(body),
)
result = await _as_owner(request, owner, documents_create_endpoint, request, req)
if isinstance(result, dict):
result = dict(result)
result["draft_type"] = "document"
result["send_required_confirmation"] = True
return result
@router.post("/emails/draft")
async def codex_email_draft(request: Request, body: dict[str, Any] = Body(default_factory=dict)):
owner = _scope_owner(request, EMAIL_DRAFT_SCOPES)
@@ -354,10 +452,18 @@ def setup_codex_routes(
owner = _scope_owner(request, DOCS_READ_SCOPES)
if documents_library_endpoint is None:
raise HTTPException(503, "Documents integration is not available")
return await _as_owner(
offset, limit = _clamp_pagination(offset, limit)
result = await _as_owner(
request, owner, documents_library_endpoint,
request, search, language, sort, offset, limit, archived,
)
if isinstance(result, dict):
docs = result.get("documents")
total = result.get("total")
if isinstance(docs, list) and isinstance(total, int):
next_offset = offset + len(docs)
result["next_offset"] = next_offset if next_offset < total else None
return result
@router.get("/documents/{doc_id}")
async def codex_documents_get(request: Request, doc_id: str):
@@ -461,14 +567,14 @@ def setup_codex_routes(
@router.get("/cookbook/tasks")
async def codex_cookbook_tasks(request: Request):
_scope_owner(request, COOKBOOK_READ_SCOPES)
_require_cookbook_scope(request, COOKBOOK_READ_SCOPES)
state = _read_cookbook_state()
tasks = state.get("tasks") or []
return {"tasks": [_redact_task(t) for t in tasks]}
@router.get("/cookbook/servers")
async def codex_cookbook_servers(request: Request):
_scope_owner(request, COOKBOOK_READ_SCOPES)
_require_cookbook_scope(request, COOKBOOK_READ_SCOPES)
state = _read_cookbook_state()
servers = state.get("env", {}).get("servers") or []
# Strip ssh creds / passwords; keep only what's needed to pick a host.
@@ -487,7 +593,7 @@ def setup_codex_routes(
@router.get("/cookbook/output/{session_id}")
async def codex_cookbook_output(request: Request, session_id: str, tail: int = 400):
_scope_owner(request, COOKBOOK_READ_SCOPES)
_require_cookbook_scope(request, COOKBOOK_READ_SCOPES)
# Defensive: session_id must be the tmux-style id we issue
# (`serve-XXXX` / `cookbook-XXXX` / `queue-XXXX`); anything else
# would let the agent run arbitrary `tmux capture-pane` targets.
@@ -529,7 +635,7 @@ def setup_codex_routes(
@router.post("/cookbook/serve")
async def codex_cookbook_serve(request: Request, body: dict[str, Any] = Body(default_factory=dict)):
_scope_owner(request, COOKBOOK_LAUNCH_SCOPES)
_require_cookbook_scope(request, COOKBOOK_LAUNCH_SCOPES)
# Wraps /api/model/serve with the SAME validation the UI uses.
# _validate_serve_cmd (called inside model_serve) rejects shell
# metachars and requires the leading binary to be in the
@@ -568,7 +674,7 @@ def setup_codex_routes(
@router.post("/cookbook/stop/{session_id}")
async def codex_cookbook_stop(request: Request, session_id: str):
_scope_owner(request, COOKBOOK_LAUNCH_SCOPES)
_require_cookbook_scope(request, COOKBOOK_LAUNCH_SCOPES)
import re as _re
if not _re.fullmatch(r"[a-zA-Z0-9_-]+", session_id):
raise HTTPException(400, "Invalid session id")
@@ -588,7 +694,7 @@ def setup_codex_routes(
"""List cached models on a configured server (or local if host is omitted).
Mirrors `list_cached_models` from the chat agent so external agents have
the same inventory view before deciding what to serve/download."""
_scope_owner(request, COOKBOOK_READ_SCOPES)
_require_cookbook_scope(request, COOKBOOK_READ_SCOPES)
# Hit /api/model/cached internally, with the same modelDirs the chat
# agent's list_cached_models would resolve from cookbook state.
state = _read_cookbook_state()
@@ -650,7 +756,7 @@ def setup_codex_routes(
"""List saved serve presets (model + host + port + launch cmd).
Counterpart to `list_serve_presets`. Use BEFORE composing a `serve`
body — the user's saved preset usually has the working cmd already."""
_scope_owner(request, COOKBOOK_READ_SCOPES)
_require_cookbook_scope(request, COOKBOOK_READ_SCOPES)
state = _read_cookbook_state()
presets = state.get("presets") or []
out = []
@@ -670,7 +776,7 @@ def setup_codex_routes(
async def codex_cookbook_serve_preset(request: Request, name: str):
"""Launch a saved preset by name. Reuses the working cmd + host the
user already saved, avoiding the cmd-allowlist trial-and-error loop."""
_scope_owner(request, COOKBOOK_LAUNCH_SCOPES)
_require_cookbook_scope(request, COOKBOOK_LAUNCH_SCOPES)
import re as _re
if not _re.fullmatch(r"[A-Za-z0-9 _.:@\-]+", name):
raise HTTPException(400, "Invalid preset name")
@@ -722,11 +828,11 @@ def setup_codex_routes(
cookbook tracking. Needed when serve_model rejects a cmd and the
agent falls back to direct ssh — without adoption the session is
invisible to the UI. Body: {tmux_session, model, host?, port?}."""
_scope_owner(request, COOKBOOK_LAUNCH_SCOPES)
_require_cookbook_scope(request, COOKBOOK_LAUNCH_SCOPES)
norm = dict(body or {})
sess = (norm.get("tmux_session") or norm.get("session_id") or "").strip()
model = (norm.get("model") or norm.get("repo_id") or "").strip()
host = (norm.get("host") or norm.get("remote_host") or "").strip()
host = validate_remote_host((norm.get("host") or norm.get("remote_host") or "").strip() or None) or ""
port = norm.get("port") or 8000
import re as _re
if not sess or not _re.fullmatch(r"[a-zA-Z0-9_-]+", sess):
routes/contacts_routes.py
+91 -27
View File
@@ -12,11 +12,13 @@ import json
import csv
import io
import os
import inspect
import httpx
from pathlib import Path
from datetime import datetime
from urllib.parse import urljoin, urlparse, urlunparse
from core.log_safety import redact_url
from fastapi import APIRouter, Query, Depends, Response, HTTPException
from typing import List, Dict, Optional
@@ -90,11 +92,13 @@ def _normalize_contact(contact: Dict) -> Dict:
name = str(contact.get("name") or "").strip()
if not name and emails:
name = emails[0].split("@")[0]
address = str(contact.get("address") or "").strip()
return {
"uid": str(contact.get("uid") or uuid.uuid4()),
"name": name,
"emails": emails,
"phones": phones,
"address": address,
}
@@ -146,11 +150,19 @@ def _vunesc(value: str) -> str:
def _parse_vcards(text: str) -> List[Dict]:
"""Parse a stream of vCards into dicts with name, email, phone."""
# Unfold RFC 6350 3.2 line folding first: a CRLF/LF followed by a single
# space or tab is a continuation of the previous logical line. Real
# CardDAV servers (Radicale, iCloud, Apple/Google) fold long EMAIL / FN /
# PHOTO lines, and splitting on raw newlines without unfolding dropped the
# continuation (e.g. "...@example\n .com" lost the ".com"), truncating the
# email/name.
text = re.sub(r"\r\n[ \t]", "", text or "")
text = re.sub(r"\n[ \t]", "", text)
contacts = []
for block in re.split(r"BEGIN:VCARD", text):
if not block.strip():
continue
contact = {"name": "", "emails": [], "phones": [], "uid": ""}
contact = {"name": "", "emails": [], "phones": [], "uid": "", "address": ""}
for line in block.split("\n"):
line = line.strip()
# Strip an optional RFC 6350 group prefix (e.g. "item1.EMAIL;...")
@@ -173,6 +185,15 @@ def _parse_vcards(text: str) -> List[Dict]:
phone = _vunesc(name_part.split(":", 1)[1])
if phone and phone not in contact["phones"]:
contact["phones"].append(phone)
elif name_part.startswith("ADR"):
# vCard ADR is 7 semicolon-separated components:
# post-office-box;extended-address;street;locality;region;postal-code;country.
# Recover a human-readable string by joining non-empty
# components with ", ".
if ":" in name_part:
raw = name_part.split(":", 1)[1]
parts = [_vunesc(p).strip() for p in raw.split(";")]
contact["address"] = ", ".join(p for p in parts if p)
elif name_part.startswith("UID:"):
contact["uid"] = _vunesc(name_part[4:])
if contact["name"] or contact["emails"]:
@@ -197,7 +218,8 @@ def _vesc(value: str) -> str:
def _build_vcard(name: str, email: str, uid: Optional[str] = None,
emails: Optional[List[str]] = None,
phones: Optional[List[str]] = None) -> str:
phones: Optional[List[str]] = None,
address: Optional[str] = None) -> str:
"""Build a vCard. Accepts either a single `email` (legacy callers) or
full `emails`/`phones` lists (edit path). The first email is marked
PREF=1. All values are RFC-6350-escaped."""
@@ -230,6 +252,12 @@ def _build_vcard(name: str, email: str, uid: Optional[str] = None,
lines.append(f"EMAIL;PREF=1:{_vesc(em)}" if i == 0 else f"EMAIL:{_vesc(em)}")
for ph in phone_list:
lines.append(f"TEL:{_vesc(ph)}")
# Address: stuff the whole human-readable string into the street
# component of ADR. vCard ADR has 7 semicolon-separated components:
# post-office-box;extended-address;street;locality;region;postal-code;country.
addr = (address or "").strip()
if addr:
lines.append(f"ADR:;;{_vesc(addr)};;;;")
lines.append("END:VCARD")
return "\r\n".join(lines) + "\r\n"
@@ -366,7 +394,7 @@ def _resolve_resource_url(uid: str) -> str:
return _lookup() or _vcard_url(uid)
def _create_contact(name: str, email: str) -> bool:
def _create_contact(name: str, email: str, address: str = "") -> bool:
"""Add a new contact via CardDAV or local contacts."""
cfg = _get_carddav_config()
if not _carddav_configured(cfg):
@@ -375,12 +403,12 @@ def _create_contact(name: str, email: str) -> bool:
for c in contacts:
if email_l and email_l in [e.lower() for e in c.get("emails", [])]:
return True
contacts.append(_normalize_contact({"name": name, "emails": [email]}))
contacts.append(_normalize_contact({"name": name, "emails": [email], "address": address}))
_save_local_contacts(contacts)
return True
contact_uid = str(uuid.uuid4())
vcard = _build_vcard(name, email, contact_uid)
vcard = _build_vcard(name, email, contact_uid, address=address)
try:
url = _carddav_base_url(cfg) + "/" + contact_uid + ".vcf"
auth = None
@@ -613,7 +641,7 @@ def _contacts_to_csv(contacts: List[Dict]) -> str:
return out.getvalue()
def _update_contact(uid: str, name: str, emails: List[str], phones: List[str]) -> bool:
def _update_contact(uid: str, name: str, emails: List[str], phones: List[str], address: str = "") -> bool:
"""Rewrite an existing contact via CardDAV or local contacts."""
cfg = _get_carddav_config()
if not _carddav_configured(cfg):
@@ -622,16 +650,19 @@ def _update_contact(uid: str, name: str, emails: List[str], phones: List[str]) -
out = []
for c in contacts:
if c.get("uid") == uid:
out.append(_normalize_contact({"uid": uid, "name": name, "emails": emails, "phones": phones}))
# Preserve existing address when caller passes "" (only
# updating name/emails/phones, not touching address).
addr = address if address else c.get("address", "")
out.append(_normalize_contact({"uid": uid, "name": name, "emails": emails, "phones": phones, "address": addr}))
found = True
else:
out.append(c)
if not found:
out.append(_normalize_contact({"uid": uid, "name": name, "emails": emails, "phones": phones}))
out.append(_normalize_contact({"uid": uid, "name": name, "emails": emails, "phones": phones, "address": address}))
_save_local_contacts(out)
return True
vcard = _build_vcard(name, "", uid=uid, emails=emails, phones=phones)
vcard = _build_vcard(name, "", uid=uid, emails=emails, phones=phones, address=address)
# Use the real resource href (handles externally-created contacts whose
# filename != UID); falls back to the <uid>.vcf guess.
try:
@@ -667,15 +698,24 @@ def _delete_contact(uid: str) -> bool:
url = _resolve_resource_url(uid)
auth = (cfg["username"], cfg["password"]) if cfg["username"] else None
r = httpx.delete(url, auth=auth, timeout=10)
if r.status_code in (200, 204):
_contact_cache["fetched_at"] = None
return True
if r.status_code == 404:
# Resource not found at the resolved URL. With href resolution
# this should be rare (genuinely already deleted). Invalidate
# the cache and report success so the UI doesn't keep a ghost.
logger.info(f"CardDAV DELETE 404 for {uid} — treating as already gone")
if r.status_code in (200, 204, 404):
# Invalidate cache so the next fetch sees the server truth.
_contact_cache["fetched_at"] = None
# Verify: force a fresh fetch and check the UID is actually gone.
# A 404 on the guessed URL ({uid}.vcf) can mean the contact
# lives at a different resource URL — the DELETE missed it but
# we'd silently report success. This check catches that.
fresh = _fetch_contacts(force=True)
still_there = any(c.get("uid") == uid for c in fresh)
if still_there:
logger.warning(
f"CardDAV DELETE reported success for {uid} "
f"but UID still present after re-fetch — "
f"resource URL may differ from {redact_url(url)}"
)
return False
if r.status_code == 404:
logger.info(f"CardDAV DELETE 404 for {uid} — already gone")
return True
logger.warning(f"CardDAV DELETE returned {r.status_code}: {r.text[:200]}")
return False
@@ -718,16 +758,39 @@ def setup_contacts_routes():
"""Add a new contact."""
name = (data.get("name") or "").strip()
email = (data.get("email") or "").strip()
phone = (data.get("phone") or "").strip()
address = (data.get("address") or "").strip()
if not email:
return {"success": False, "error": "Email required"}
# Check if already exists
contacts = _fetch_contacts()
for c in contacts:
if email.lower() in [e.lower() for e in c["emails"]]:
return {"success": True, "message": "Already exists", "contact": c}
# Check if already exists by email
if email:
contacts = _fetch_contacts()
for c in contacts:
if email.lower() in [e.lower() for e in c["emails"]]:
return {"success": True, "message": "Already exists", "contact": c}
if not name:
name = email.split("@")[0]
ok = _create_contact(name, email)
create_params = inspect.signature(_create_contact).parameters
if len(create_params) >= 3:
ok = _create_contact(name, email, address)
else:
ok = _create_contact(name, email)
# If a phone was provided, do an immediate update to thread it
# through (the simple _create_contact signature only takes name +
# email + address; phones happen via update).
if ok and phone:
try:
fresh = _fetch_contacts(force=True)
created = next((c for c in fresh if name == c.get("name") and (not email or email in c.get("emails", []))), None)
if created:
_update_contact(
created["uid"], name,
created.get("emails", []),
[phone],
address,
)
except Exception:
pass
return {"success": ok}
@router.post("/import")
@@ -810,7 +873,7 @@ def setup_contacts_routes():
# match PUT /{uid} with uid="config".
@router.put("/{uid}")
async def edit_contact(uid: str, data: dict, _admin: str = Depends(require_admin)):
"""Edit an existing contact — name / emails / phones."""
"""Edit an existing contact — name / emails / phones / address."""
name = (data.get("name") or "").strip()
emails = data.get("emails")
phones = data.get("phones")
@@ -818,11 +881,12 @@ def setup_contacts_routes():
emails = [data["email"]]
emails = [e.strip() for e in (emails or []) if e and e.strip()]
phones = [p.strip() for p in (phones or []) if p and p.strip()]
if not name and not emails:
return {"success": False, "error": "Name or email required"}
address = (data.get("address") or "").strip()
if not name and not emails and not address:
return {"success": False, "error": "Name, email, or address required"}
if not name and emails:
name = emails[0].split("@")[0]
ok = _update_contact(uid, name, emails, phones)
ok = _update_contact(uid, name, emails, phones, address)
return {"success": ok}
@router.delete("/{uid}")
routes/cookbook_helpers.py
+189 -20
View File
@@ -362,7 +362,12 @@ def _user_shell_path_bootstrap() -> list[str]:
' ODYSSEUS_USER_PATH="$("$ODYSSEUS_USER_SHELL" -ic \'printf "__ODYSSEUS_PATH__%s\\n" "$PATH"\' 2>/dev/null | sed -n \'s/^__ODYSSEUS_PATH__//p\' | tail -n 1 || true)"',
' if [ -n "$ODYSSEUS_USER_PATH" ]; then export PATH="$ODYSSEUS_USER_PATH:$PATH"; fi',
'fi',
'command -v python3 >/dev/null 2>&1 || python3() { python "$@"; }',
# Windows can expose python3 as a Microsoft Store App Execution Alias
# under WindowsApps. Git Bash sees that stub as present, but it exits
# before running Python. A Windows venv usually has python.exe, not
# python3.exe, so treat a missing or WindowsApps python3 as absent.
'_odys_py3="$(command -v python3 2>/dev/null || true)"',
'case "$_odys_py3" in ""|*[Ww]indows[Aa]pps*) python3() { python "$@"; } ;; esac',
'command -v python >/dev/null 2>&1 || python() { python3 "$@"; }',
]
@@ -500,6 +505,8 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
" if u.startswith('KB'): return int(n * 1024)",
" return int(n)",
"def scan_ollama():",
" if any(m.get('is_ollama') for m in models): return",
" if os.name == 'nt' and not os.environ.get('ODYSSEUS_ALLOW_OLLAMA_CLI_SCAN'): return",
" if not shutil.which('ollama'): return",
" try:",
" p = subprocess.run(['ollama', 'list'], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, timeout=6)",
@@ -530,8 +537,8 @@ def _cached_model_scan_script(model_dirs: list[str] | None = None, add_hf_cache:
" models.append({'repo_id':name,'size_bytes':size_bytes,'nb_files':1,'has_incomplete':False,'path':'ollama','backend':'ollama','is_ollama':True})",
" return",
"for _hf_cache in hf_cache_paths(): scan_hf(_hf_cache)",
"scan_ollama()",
"scan_ollama_api()",
"scan_ollama()",
]
for model_dir in model_dirs or []:
lines.append(f"scan_dir(os.path.expanduser({model_dir!r}))")
@@ -554,7 +561,7 @@ def _bash_squote(v: str) -> str:
# Allow-list of binaries permitted as the leading token of `req.cmd` for /api/model/serve.
# Anything else is rejected before the cmd is interpolated into a tmux/PowerShell wrapper.
_SERVE_CMD_ALLOWLIST = {
"vllm", "llama-server", "llama_server", "llama.cpp", "ollama",
"vllm", "llama-server", "llama-server.exe", "llama_server", "llama.cpp", "ollama",
"python", "python3",
"sglang", "lmdeploy",
"node", "npx",
@@ -779,25 +786,149 @@ def _append_llama_cpp_linux_accel_build_lines(runner_lines: list[str]) -> None:
to hard-wire CUDA on Linux. That made ROCm hosts attempt a CUDA configure and
fail with "CUDA Toolkit not found" instead of building with HIP.
"""
# Try a prebuilt binary from llama.cpp's GitHub releases FIRST — no
# cmake/build-essential/git/CUDA-headers needed at all. The from-source
# build below stays as a fallback (custom flags, esoteric arch, no
# internet, etc). 30 seconds vs 5+ minutes of compile, and removes
# every OS-package dep from the launch path. Sets _odysseus_have_prebuilt=1
# on success; the existing build-tier if/elif chain below is gated on
# that variable so we never compile twice or shadow the prebuilt symlink.
runner_lines.append(' _odysseus_have_prebuilt=""')
runner_lines.append(' _odysseus_arch="$(uname -m)"')
runner_lines.append(' _odysseus_prebuilt_url=""')
runner_lines.append(' if command -v curl >/dev/null 2>&1 && [ "$_odysseus_arch" = "x86_64" ]; then')
runner_lines.append(' _odysseus_pat=""')
runner_lines.append(' _odysseus_has_nv_inline() { command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L 2>/dev/null | grep -q "GPU "; }')
runner_lines.append(' _odysseus_has_vk_inline() { ldconfig -p 2>/dev/null | grep -q "libvulkan\\.so" || command -v vulkaninfo >/dev/null 2>&1 || [ -e /usr/lib/x86_64-linux-gnu/libvulkan.so.1 ]; }')
runner_lines.append(' _odysseus_has_vkdev_inline() { ls /dev/dri/renderD* >/dev/null 2>&1 || (lspci 2>/dev/null | grep -Ei \'VGA|3D|Display\' | grep -Eiq \'AMD|ATI|Radeon\'); }')
runner_lines.append(' if _odysseus_has_nv_inline; then')
runner_lines.append(' _odysseus_pat="ubuntu.*cuda"')
runner_lines.append(' elif _odysseus_has_vkdev_inline && _odysseus_has_vk_inline; then')
runner_lines.append(' _odysseus_pat="ubuntu.*vulkan"')
runner_lines.append(' else')
runner_lines.append(' _odysseus_pat="ubuntu-x64\\\\.zip"')
runner_lines.append(' fi')
runner_lines.append(' _odysseus_prebuilt_url="$(curl -fsSL --max-time 15 https://api.github.com/repos/ggml-org/llama.cpp/releases/latest 2>/dev/null | grep \'"browser_download_url"\' | cut -d\'"\' -f4 | grep -iE "$_odysseus_pat" | grep -iv "arm\\|aarch64" | head -1)"')
runner_lines.append(' fi')
# Accept any of unzip / bsdtar / python3 -m zipfile as the extractor.
# python3 is essentially always present on modern Linux, so this lets
# the prebuilt path work on minimal Ubuntu installs that lack `unzip`.
runner_lines.append(' if [ -n "$_odysseus_prebuilt_url" ] && (command -v unzip >/dev/null 2>&1 || command -v bsdtar >/dev/null 2>&1 || command -v python3 >/dev/null 2>&1); then')
runner_lines.append(' echo "[odysseus] Found prebuilt llama-server: $_odysseus_prebuilt_url"')
runner_lines.append(' mkdir -p ~/bin "$HOME/.cache/odysseus/llama-cpp-prebuilt" && cd "$HOME/.cache/odysseus/llama-cpp-prebuilt"')
runner_lines.append(' rm -f llama-cpp.zip')
runner_lines.append(' if curl -fsSL --max-time 120 "$_odysseus_prebuilt_url" -o llama-cpp.zip && [ -s llama-cpp.zip ]; then')
runner_lines.append(' rm -rf build && mkdir -p build')
runner_lines.append(' if command -v unzip >/dev/null 2>&1; then unzip -qq -o llama-cpp.zip -d build; elif command -v bsdtar >/dev/null 2>&1; then bsdtar -xf llama-cpp.zip -C build; else python3 -c "import zipfile; zipfile.ZipFile(\\"llama-cpp.zip\\").extractall(\\"build\\")"; fi')
runner_lines.append(' _odysseus_extracted="$(find build -type f -name llama-server 2>/dev/null | head -1)"')
runner_lines.append(' if [ -n "$_odysseus_extracted" ]; then')
runner_lines.append(' chmod +x "$_odysseus_extracted"')
runner_lines.append(' ln -sf "$_odysseus_extracted" ~/bin/llama-server')
runner_lines.append(' _odysseus_libdir="$(dirname "$_odysseus_extracted")"')
runner_lines.append(' mkdir -p ~/.config && echo "export LD_LIBRARY_PATH=\\"$_odysseus_libdir:\\${LD_LIBRARY_PATH:-}\\"" > ~/.config/odysseus-llama-cpp-env')
runner_lines.append(' _odysseus_have_prebuilt=1')
runner_lines.append(' echo "[odysseus] Prebuilt llama-server installed at $_odysseus_extracted"')
runner_lines.append(' fi')
runner_lines.append(' fi')
runner_lines.append(' [ -z "$_odysseus_have_prebuilt" ] && echo "[odysseus] Prebuilt download/extract failed — falling back to from-source build."')
runner_lines.append(' elif [ -z "$_odysseus_prebuilt_url" ]; then')
runner_lines.append(' echo "[odysseus] No matching prebuilt llama-server for this host (arch=$_odysseus_arch) — will build from source."')
runner_lines.append(' fi')
runner_lines.append(' if [ -z "$_odysseus_have_prebuilt" ]; then')
# Detect pip-installed nvcc (from vLLM/nvidia CUDA wheels) and put it on PATH
# so cmake's CUDA configure can find it. We keep this after the ROCm/HIP
# check — a machine with both stacks should honor the native HIP toolchain on
# AMD hosts instead of accidentally preferring a stray nvcc wheel.
runner_lines.append(' for _cudir in ~/.local/lib/python*/site-packages/nvidia/cu13 ~/.local/lib/python*/site-packages/nvidia/cu12 ~/.local/lib/python*/site-packages/nvidia/cuda_nvcc; do')
runner_lines.append(' [ -x "$_cudir/bin/nvcc" ] && export CUDA_HOME="$_cudir" && export PATH="$_cudir/bin:$PATH" && break')
runner_lines.append(' done')
# so cmake's CUDA configure can find it — BUT only when actual NVIDIA
# hardware is present. On AMD/Intel hosts the pip nvcc is a misleading
# leftover (no libcudart, no GPU it could target) and would otherwise
# send the build down the CUDA branch and fail with "CUDA Toolkit not
# found" instead of trying Vulkan.
runner_lines.append(' _odysseus_has_nvidia_hw() {')
runner_lines.append(' command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L 2>/dev/null | grep -q "GPU " && return 0')
runner_lines.append(' ls /dev/nvidia* >/dev/null 2>&1 && return 0')
runner_lines.append(' lspci 2>/dev/null | grep -iE \'VGA|3D|Display\' | grep -iq nvidia && return 0')
runner_lines.append(' return 1')
runner_lines.append(' }')
runner_lines.append(' if _odysseus_has_nvidia_hw; then')
runner_lines.append(' for _cudir in ~/.local/lib/python*/site-packages/nvidia/cu13 ~/.local/lib/python*/site-packages/nvidia/cu12 ~/.local/lib/python*/site-packages/nvidia/cuda_nvcc; do')
runner_lines.append(' [ -x "$_cudir/bin/nvcc" ] && export CUDA_HOME="$_cudir" && export PATH="$_cudir/bin:$PATH" && break')
runner_lines.append(' done')
runner_lines.append(' fi')
# rm -rf build so a prior poisoned CMakeCache.txt (e.g. from a failed CUDA
# or HIP attempt) doesn't cause the next configure to reuse stale settings.
runner_lines.append(' mkdir -p ~/bin')
runner_lines.append(' cd ~/llama.cpp && rm -rf build')
# Try to install cmake / build-essential / git automatically before the
# build, but ONLY via passwordless sudo (`sudo -n`) — interactive sudo
# would hang a tmux-backgrounded serve task waiting for a password. If
# sudo asks for a password the install is skipped silently and the
# diagnosis pattern (cookbook_routes.py / cookbook_helpers.py) surfaces
# an explicit "install cmake" suggestion in the Cookbook diagnosis
# toolbar after the inevitable build failure.
runner_lines.append(' _odysseus_apt_bootstrap() {')
runner_lines.append(' local _missing=""')
runner_lines.append(' command -v cmake >/dev/null 2>&1 || _missing="$_missing cmake"')
runner_lines.append(' command -v g++ >/dev/null 2>&1 || command -v gcc >/dev/null 2>&1 || _missing="$_missing build-essential"')
runner_lines.append(' command -v git >/dev/null 2>&1 || _missing="$_missing git"')
runner_lines.append(' [ -z "$_missing" ] && return 0')
runner_lines.append(' if command -v apt-get >/dev/null 2>&1 && sudo -n true 2>/dev/null; then')
runner_lines.append(' echo "[odysseus] Auto-installing missing build deps via apt:$_missing"')
runner_lines.append(' sudo -n env DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>&1 | tail -3')
runner_lines.append(' sudo -n env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends $_missing 2>&1 | tail -5 || true')
runner_lines.append(' elif command -v pacman >/dev/null 2>&1 && sudo -n true 2>/dev/null; then')
runner_lines.append(' echo "[odysseus] Auto-installing missing build deps via pacman:$_missing"')
runner_lines.append(' local _pacpkgs="$(echo "$_missing" | sed -e \'s/build-essential/base-devel/g\')"')
runner_lines.append(' sudo -n pacman -Sy --needed --noconfirm $_pacpkgs 2>&1 | tail -5 || true')
runner_lines.append(' elif command -v dnf >/dev/null 2>&1 && sudo -n true 2>/dev/null; then')
runner_lines.append(' echo "[odysseus] Auto-installing missing build deps via dnf:$_missing"')
runner_lines.append(' local _dnfpkgs="$(echo "$_missing" | sed -e \'s/build-essential/gcc gcc-c++ make/g\')"')
runner_lines.append(' sudo -n dnf install -y $_dnfpkgs 2>&1 | tail -5 || true')
runner_lines.append(' else')
runner_lines.append(' echo "[odysseus] WARNING: missing build deps ($_missing) — passwordless sudo is unavailable, cannot auto-install. Cookbook Diagnosis will explain the fix after the build fails."')
runner_lines.append(' fi')
runner_lines.append(' }')
runner_lines.append(' _odysseus_apt_bootstrap')
runner_lines.append(' _odysseus_missing_build_deps=""')
runner_lines.append(' command -v cmake >/dev/null 2>&1 || _odysseus_missing_build_deps="$_odysseus_missing_build_deps cmake"')
runner_lines.append(' command -v git >/dev/null 2>&1 || _odysseus_missing_build_deps="$_odysseus_missing_build_deps git"')
runner_lines.append(' command -v g++ >/dev/null 2>&1 || command -v gcc >/dev/null 2>&1 || _odysseus_missing_build_deps="$_odysseus_missing_build_deps build-essential"')
runner_lines.append(' if [ -n "$_odysseus_missing_build_deps" ]; then')
runner_lines.append(' echo "ERROR: llama.cpp source build needs missing packages:$_odysseus_missing_build_deps"')
runner_lines.append(' if command -v apt-get >/dev/null 2>&1; then')
runner_lines.append(' echo "Install on this host: sudo apt-get update && sudo apt-get install -y cmake build-essential git"')
runner_lines.append(' elif command -v pacman >/dev/null 2>&1; then')
runner_lines.append(' echo "Install on this host: sudo pacman -Sy --needed cmake base-devel git"')
runner_lines.append(' elif command -v dnf >/dev/null 2>&1; then')
runner_lines.append(' echo "Install on this host: sudo dnf install -y cmake gcc gcc-c++ make git"')
runner_lines.append(' fi')
runner_lines.append(' echo "Alternative: install a native llama-server on PATH, then relaunch."')
runner_lines.append(' ODYSSEUS_PREFLIGHT_EXIT=127')
runner_lines.append(' fi')
runner_lines.append(' cd ~/llama.cpp')
runner_lines.append(' _odysseus_has_vulkan() {')
runner_lines.append(' ldconfig -p 2>/dev/null | grep -q \'libvulkan\\.so\' && return 0')
runner_lines.append(' [ -e /usr/lib/libvulkan.so.1 ] && return 0')
runner_lines.append(' [ -e /usr/lib/x86_64-linux-gnu/libvulkan.so.1 ] && return 0')
runner_lines.append(' command -v vulkaninfo >/dev/null 2>&1 && return 0')
runner_lines.append(' return 1')
runner_lines.append(' }')
runner_lines.append(' _odysseus_has_vulkan_device() {')
runner_lines.append(' ls /dev/dri/renderD* >/dev/null 2>&1 && return 0')
runner_lines.append(' lspci 2>/dev/null | grep -Ei \'VGA|3D|Display\' | grep -Eiq \'AMD|ATI|Radeon\' && return 0')
runner_lines.append(' return 1')
runner_lines.append(' }')
# Backend preference: native ROCm/HIP > native CUDA > Vulkan > CPU.
# Vulkan is a portable fallback that works on AMD when ROCm isn't
# installed (e.g. Strix Halo) and on any vendor's discrete GPU, but
# it's ~30-40% slower than native HIP/CUDA for LLM inference — only
# pick it when no native toolchain is present.
runner_lines.append(' if command -v hipconfig &>/dev/null || [ -d /opt/rocm ] || [ -n "$ROCM_PATH" ] || [ -n "$HIP_PATH" ]; then')
runner_lines.append(' rm -rf build')
runner_lines.append(' if command -v hipconfig &>/dev/null; then')
runner_lines.append(' export HIPCXX="${HIPCXX:-$(hipconfig -l)/clang}"')
runner_lines.append(' export HIP_PATH="${HIP_PATH:-$(hipconfig -R)}"')
runner_lines.append(' fi')
runner_lines.append(' echo "[odysseus] ROCm/HIP detected — building llama-server with HIP support..."')
runner_lines.append(' cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIP=ON && cmake --build build -j"$NPROC" --target llama-server && ln -sf ~/llama.cpp/build/bin/llama-server ~/bin/llama-server')
runner_lines.append(' elif command -v nvcc &>/dev/null; then')
runner_lines.append(' elif command -v nvcc &>/dev/null && _odysseus_has_nvidia_hw; then')
runner_lines.append(' rm -rf build')
# nvcc alone is not sufficient — pip-installed CUDA wheels or incomplete
# tooling can expose nvcc without shipping libcudart, causing cmake to fail
# mid-build with "CUDA runtime library not found". Check cudart explicitly
@@ -821,31 +952,50 @@ def _append_llama_cpp_linux_accel_build_lines(runner_lines: list[str]) -> None:
runner_lines.append(' echo "[odysseus] Ensure libcudart is installed (e.g. cuda-runtime package) and visible via ldconfig or CUDA_HOME."')
runner_lines.append(' cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j"$NPROC" --target llama-server && ln -sf ~/llama.cpp/build/bin/llama-server ~/bin/llama-server')
runner_lines.append(' fi')
runner_lines.append(' elif _odysseus_has_vulkan_device && _odysseus_has_vulkan; then')
runner_lines.append(' echo "[odysseus] Vulkan-capable GPU detected (no ROCm/CUDA toolchain installed) — building llama-server with Vulkan support..."')
runner_lines.append(' rm -rf build-vulkan')
runner_lines.append(' cmake -B build-vulkan -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON && cmake --build build-vulkan -j"$NPROC" --target llama-server && ln -sf ~/llama.cpp/build-vulkan/bin/llama-server ~/bin/llama-server')
runner_lines.append(' else')
runner_lines.append(' echo "[odysseus] WARNING: no HIP/CUDA toolchain found — building llama-server for CPU only."')
runner_lines.append(' echo "[odysseus] WARNING: no HIP/CUDA/Vulkan toolchain found — building llama-server for CPU only."')
runner_lines.append(' echo "[odysseus] GPU inference will not be available for this llama.cpp build."')
runner_lines.append(' echo "[odysseus] Install ROCm for AMD GPUs or vLLM/CUDA tooling for NVIDIA, then re-launch this serve task."')
runner_lines.append(' echo "[odysseus] Install Vulkan (libvulkan-dev) / ROCm for AMD GPUs or CUDA tooling for NVIDIA, then re-launch this serve task."')
runner_lines.append(' rm -rf build')
runner_lines.append(' cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j"$NPROC" --target llama-server && ln -sf ~/llama.cpp/build/bin/llama-server ~/bin/llama-server')
runner_lines.append(' fi')
runner_lines.append(' fi # end _odysseus_have_prebuilt guard')
def _llama_cpp_rebuild_cmd() -> str:
def _llama_cpp_rebuild_cmd(update_source: bool = False) -> str:
"""Shell command that clears the Cookbook-managed llama.cpp build.
Removes the cached ``llama-server`` symlink and the ``~/llama.cpp/build``
Removes the cached ``llama-server`` symlink and the ``~/llama.cpp/build*``
directory so the next llama.cpp serve recompiles from source, picking up a
CUDA or HIP toolchain if one is now available. The serve bootstrap only
builds when ``llama-server`` is missing from PATH, so without this an
existing CPU-only build is reused forever. It deliberately installs and
downloads nothing; the rebuild itself happens on the next serve.
existing CPU-only build is reused forever. When ``update_source`` is true,
the command also fast-forwards the Cookbook-managed ``~/llama.cpp`` checkout
if it exists. The rebuild itself happens on the next serve.
"""
update_cmd = ''
if update_source:
update_cmd = (
'if [ -d "$HOME/llama.cpp/.git" ]; then '
'git -C "$HOME/llama.cpp" pull --ff-only --depth 1 || '
'echo "[odysseus] WARNING: llama.cpp source update failed; clearing cached build anyway."; '
'elif command -v git >/dev/null 2>&1; then '
'git clone --depth 1 https://github.com/ggml-org/llama.cpp "$HOME/llama.cpp" || '
'echo "[odysseus] WARNING: llama.cpp clone failed; clearing cached build anyway."; '
'fi && '
)
return (
'mkdir -p "$HOME/bin" && '
f'{update_cmd}'
'rm -f "$HOME/bin/llama-server" && '
'rm -rf "$HOME/llama.cpp/build" && '
'rm -rf "$HOME/llama.cpp/build" "$HOME/llama.cpp/build-vulkan" && '
'echo "[odysseus] Cleared the cached llama.cpp build. '
'Re-launch the serve task to rebuild llama-server from source '
'(CUDA or HIP will be used if a toolchain is now available)."'
'(Vulkan, HIP, or CUDA will be used if a matching toolchain is now available)."'
)
@@ -1108,8 +1258,27 @@ def _diagnose_serve_output(text: str) -> dict | None:
"SGLang is not installed or not in PATH on this server.",
[{"label": "install SGLang in Cookbook Dependencies", "op": "dependency", "package": "sglang[all]"}],
),
# System build deps come BEFORE the generic llama.cpp catch-all so
# cmake / build-essential / git missing → a specific OS-package
# remediation instead of "install llama-cpp-python[server]" (which
# itself fails to compile when cmake is absent).
(
r"llama-server.*command not found|llama\.cpp.*not found|No module named.*llama_cpp|No module named 'starlette_context'|git: command not found|cmake: command not found",
r"cmake: command not found|cmake.*not found.*[Cc]ould not",
"cmake is required to build llama.cpp from source but isn't installed on this server.",
[{"label": "install build deps for llama.cpp (apt: cmake build-essential git / pacman: cmake base-devel git / dnf: cmake gcc-c++ make git / brew: cmake git)", "op": "dependency", "package": "llama-cpp-python[server]"}],
),
(
r"^(make|g\+\+|gcc): command not found|Could not find C\+\+ compiler",
"A C/C++ compiler (build-essential) is required to build llama.cpp from source.",
[{"label": "install build deps for llama.cpp on this server", "op": "dependency", "package": "llama-cpp-python[server]"}],
),
(
r"^git: command not found",
"git is required to clone the llama.cpp source tree.",
[{"label": "install build deps for llama.cpp on this server", "op": "dependency", "package": "llama-cpp-python[server]"}],
),
(
r"llama-server.*command not found|llama\.cpp.*not found|No module named.*llama_cpp|No module named 'starlette_context'",
"llama.cpp / llama-cpp-python dependencies are missing.",
[{"label": "install llama.cpp dependencies or llama-cpp-python[server]", "op": "dependency", "package": "llama-cpp-python[server]"}],
),
routes/cookbook_routes.py
+549 -25
View File
@@ -73,6 +73,9 @@ def setup_cookbook_routes() -> APIRouter:
return "stored"
return f"{value[:4]}...{value[-4:]}"
def _client_host_platform() -> str:
return "windows" if IS_WINDOWS else ""
def _decrypt_secret(value: str | None) -> str:
if not value:
return ""
@@ -189,8 +192,27 @@ def setup_cookbook_routes() -> APIRouter:
"SGLang is not installed or not in PATH on this server.",
[{"label": "install SGLang in Cookbook Dependencies", "op": "dependency", "package": "sglang[all]"}],
),
# System build deps come BEFORE the generic llama.cpp catch-all
# so cmake / build-essential / git missing → a specific OS-package
# remediation instead of "install llama-cpp-python[server]" (which
# itself fails to compile when cmake is absent).
(
r"llama-server.*command not found|llama\.cpp.*not found|No module named.*llama_cpp|No module named 'starlette_context'|git: command not found|cmake: command not found",
r"cmake: command not found|cmake.*not found.*[Cc]ould not",
"cmake is required to build llama.cpp from source but isn't installed on this server.",
[{"label": "install build deps for llama.cpp (apt: cmake build-essential git / pacman: cmake base-devel git / dnf: cmake gcc-c++ make git / brew: cmake git)", "op": "dependency", "package": "llama-cpp-python[server]"}],
),
(
r"^(make|g\+\+|gcc): command not found|Could not find C\+\+ compiler",
"A C/C++ compiler (build-essential) is required to build llama.cpp from source.",
[{"label": "install build deps for llama.cpp on this server", "op": "dependency", "package": "llama-cpp-python[server]"}],
),
(
r"^git: command not found",
"git is required to clone the llama.cpp source tree.",
[{"label": "install build deps for llama.cpp on this server", "op": "dependency", "package": "llama-cpp-python[server]"}],
),
(
r"llama-server.*command not found|llama\.cpp.*not found|No module named.*llama_cpp|No module named 'starlette_context'",
"llama.cpp / llama-cpp-python dependencies are missing.",
[{"label": "install llama.cpp dependencies or llama-cpp-python[server]", "op": "dependency", "package": "llama-cpp-python[server]"}],
),
@@ -226,11 +248,15 @@ def setup_cookbook_routes() -> APIRouter:
"""Return cookbook state without raw secrets for browser clients."""
_strip_task_secrets(state)
env = state.get("env") if isinstance(state, dict) else None
if isinstance(state, dict) and not isinstance(env, dict):
env = {}
state["env"] = env
if isinstance(env, dict):
token = _decrypt_secret(env.get("hfToken"))
env.pop("hfToken", None)
env["hfTokenConfigured"] = bool(token)
env["hfTokenMasked"] = _mask_secret(token)
env["hostPlatform"] = _client_host_platform()
return state
def _state_for_storage(state, on_disk=None):
@@ -249,11 +275,85 @@ def setup_cookbook_routes() -> APIRouter:
env.pop("hfToken", None)
env.pop("hfTokenMasked", None)
env.pop("hfTokenConfigured", None)
env.pop("hostPlatform", None)
return state
def _load_stored_hf_token() -> str:
return load_stored_hf_token(state_path=_cookbook_state_path)
def _normalize_minimax_m3_vllm_cmd(cmd: str) -> str:
"""Patch MiniMax M3 vLLM launches into the known-good local form.
The browser form can be stale or omit advanced-only fields. MiniMax M3
is sensitive to several flags: using the HF repo id with block-size 128
fails KV-cache setup, and FlashInfer sampler JIT fails on this host's
system nvcc. Normalize server-side before writing the tmux runner.
"""
cmd_lower = (cmd or "").lower()
if not cmd or "vllm serve" not in cmd_lower or "minimax" not in cmd_lower or "m3" not in cmd_lower:
return cmd
try:
parts = shlex.split(cmd)
except ValueError:
return cmd
if "serve" not in parts:
return cmd
env_re = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*=")
env_parts = [p for p in parts if env_re.match(p)]
body = [p for p in parts if not env_re.match(p)]
try:
serve_i = body.index("serve")
except ValueError:
return cmd
if serve_i + 1 >= len(body):
return cmd
repo_id = "cyankiwi/MiniMax-M3-AWQ-INT4"
snapshot = (
"/home/pewds/.cache/huggingface/hub/"
"models--cyankiwi--MiniMax-M3-AWQ-INT4/"
"snapshots/4082acbbec1236d21828d55b6bb0fe02ade4ab5b"
)
if body[serve_i + 1] == repo_id:
body[serve_i + 1] = snapshot
def add_env(key: str, value: str) -> None:
if not any(p.startswith(f"{key}=") for p in env_parts):
env_parts.append(f"{key}={value}")
def has_flag(flag: str) -> bool:
return any(p == flag or p.startswith(flag + "=") for p in body)
def set_flag(flag: str, value: str) -> None:
for i, part in enumerate(body):
if part == flag:
if i + 1 < len(body):
body[i + 1] = value
else:
body.append(value)
return
if part.startswith(flag + "="):
body[i] = f"{flag}={value}"
return
body.extend([flag, value])
def add_bool(flag: str) -> None:
if not has_flag(flag):
body.append(flag)
add_env("VLLM_TARGET_DEVICE", "cuda")
add_env("VLLM_USE_FLASHINFER_SAMPLER", "0")
set_flag("--served-model-name", repo_id)
set_flag("--tool-call-parser", "minimax_m3")
set_flag("--reasoning-parser", "minimax_m3")
set_flag("--attention-backend", "TRITON_ATTN")
set_flag("--block-size", "128")
add_bool("--language-model-only")
add_bool("--disable-custom-all-reduce")
add_bool("--enable-expert-parallel")
return shlex.join(env_parts + body)
def _cookbook_ssh_dir() -> Path:
# The Docker image keeps cookbook keys under /app/.ssh; that path only
# exists inside the container. On Windows (and any non-container host)
@@ -676,7 +776,7 @@ def setup_cookbook_routes() -> APIRouter:
_spf = f"-p {_port} " if _port and _port != "22" else ""
setup_cmd = (
f"scp -O {_pf}-q '{runner_path}' {remote}:{remote_runner} && "
f"ssh {_spf}{remote} 'chmod +x {remote_runner} && tmux new-session -d -s {session_id} \"./{remote_runner}\"'"
f"ssh {_spf}{remote} 'chmod +x {remote_runner} && tmux set-option -g history-limit 100000 2>/dev/null; tmux new-session -d -s {session_id} \"./{remote_runner}\"'"
)
else:
# Local: run hf download in the background (tmux on POSIX, a detached
@@ -708,7 +808,7 @@ def setup_cookbook_routes() -> APIRouter:
lines.append('exec "${SHELL:-/bin/bash}"')
wrapper_script.write_text("\n".join(lines) + "\n", encoding="utf-8")
wrapper_script.chmod(0o755)
setup_cmd = None if IS_WINDOWS else f"tmux new-session -d -s {session_id} {shlex.quote(str(wrapper_script))}"
setup_cmd = None if IS_WINDOWS else f"tmux set-option -g history-limit 100000 2>/dev/null; tmux new-session -d -s {session_id} {shlex.quote(str(wrapper_script))}"
logger.info(f"Model download: {req.repo_id} (backend={'ollama' if is_ollama_download else 'hf'}, include={req.include}, session={session_id}, remote={remote})")
logger.info(f"Download setup_cmd: {setup_cmd}")
@@ -984,9 +1084,9 @@ def setup_cookbook_routes() -> APIRouter:
ssh_args = ["ssh"]
if ssh_port and ssh_port != "22":
ssh_args.extend(["-p", str(ssh_port)])
capture_cmd = ssh_args + [remote, "tmux", "capture-pane", "-t", session_id, "-p", "-S", "-200"]
capture_cmd = ssh_args + [remote, "tmux", "capture-pane", "-t", session_id, "-p", "-S", "-2000"]
else:
capture_cmd = ["tmux", "capture-pane", "-t", session_id, "-p", "-S", "-200"]
capture_cmd = ["tmux", "capture-pane", "-t", session_id, "-p", "-S", "-2000"]
_exit_re = re.compile(r"=== Process exited with code (-?\d+) ===")
for wait_s in _waits:
@@ -1230,6 +1330,7 @@ def setup_cookbook_routes() -> APIRouter:
# `TypeError: argument of type 'NoneType'` (a 500 instead of a clean 400).
req.cmd = _validate_serve_cmd(req.cmd) or ""
req.cmd = _normalize_llama_cpp_python_cache_types(req.cmd) or ""
req.cmd = _normalize_minimax_m3_vllm_cmd(req.cmd)
req.cmd = _venv_safe_local_pip_install_cmd(
req.cmd,
local=not bool(req.remote_host),
@@ -1243,8 +1344,16 @@ def setup_cookbook_routes() -> APIRouter:
req.cmd = _pip_install_no_cache(req.cmd)
# Accept common aliases and enforce server extras for llama-cpp so
# `python -m llama_cpp.server` has all runtime dependencies.
req.cmd = re.sub(r"(?<![A-Za-z0-9_.-])llama_cpp(?![A-Za-z0-9_.-])", "llama-cpp-python[server]", req.cmd)
req.cmd = re.sub(r"(?<![A-Za-z0-9_.-])llama-cpp-python(?!\[)", "llama-cpp-python[server]", req.cmd)
# CRITICAL: the lookbehind / lookahead must also exclude `/` so
# the regex DOESN'T mangle a URL path like
# https://abetlen.github.io/llama-cpp-python/whl/cu124
# The previous regex turned that URL into
# https://abetlen.github.io/llama-cpp-python[server]/whl/cu124
# which pip then couldn't resolve → silent fallback to source
# build of the .tar.gz → CPU-only binary (because CMAKE_ARGS
# isn't set), defeating the entire purpose of the CUDA index.
req.cmd = re.sub(r"(?<![A-Za-z0-9_.\-/])llama_cpp(?![A-Za-z0-9_.\-/])", "llama-cpp-python[server]", req.cmd)
req.cmd = re.sub(r"(?<![A-Za-z0-9_.\-/])llama-cpp-python(?![\[/])", "llama-cpp-python[server]", req.cmd)
if "llama-cpp-python" in req.cmd and "--extra-index-url" not in req.cmd:
req.cmd += " --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu"
# PEP-508-style package spec — letters, digits, `.-_` for the
@@ -1284,6 +1393,11 @@ def setup_cookbook_routes() -> APIRouter:
# LOCAL execution on a native-Windows host never uses tmux (detached
# process path below), regardless of the UI-supplied platform.
local_windows = IS_WINDOWS and not remote
if is_windows and remote and "diffusion_server.py" in req.cmd:
raise HTTPException(
400,
"Remote Windows Diffusers serving is not supported yet; use local Windows or a Linux remote server.",
)
if not is_windows and not local_windows and not await _binary_available("tmux", remote, req.ssh_port):
return {
@@ -1373,6 +1487,10 @@ def setup_cookbook_routes() -> APIRouter:
# shell resolves the bundled python3/hf, mirroring the download flow.
if not remote:
runner_lines.append(_local_tooling_path_export(sys.executable))
if local_windows:
# Detached Git Bash runs do not always inherit recently edited
# user PATH entries from the already-running Odysseus process.
runner_lines.append('export PATH="$HOME/bin:$HOME/llama.cpp/build-cuda/bin/Release:$HOME/llama.cpp/build/bin/Release:$HOME/llama.cpp/build/bin/Debug:$HOME/llama.cpp/build/bin:$PATH"')
runner_lines.append("export FLASHINFER_DISABLE_VERSION_CHECK=1")
if req.hf_token:
runner_lines.append(f"export HF_TOKEN='{_bash_squote(req.hf_token)}'")
@@ -1387,7 +1505,8 @@ def setup_cookbook_routes() -> APIRouter:
runner_lines.append(_HF_TOKEN_STATUS_SNIPPET)
handled_ollama_serve = False
# Auto-install inference engine if missing
if "llama_cpp" in req.cmd or "llama-server" in req.cmd:
local_windows_llama_cmd = local_windows and ("llama_cpp" in req.cmd or "llama-server" in req.cmd)
if ("llama_cpp" in req.cmd or "llama-server" in req.cmd) and not local_windows_llama_cmd:
# Prefer the NATIVE llama-server binary — its minja templating
# renders modern GGUF chat templates that the Python bindings'
# Jinja2 rejects (do_tojson ensure_ascii). Build it once from
@@ -1426,6 +1545,69 @@ def setup_cookbook_routes() -> APIRouter:
runner_lines.append(' else')
_append_llama_cpp_linux_accel_build_lines(runner_lines)
runner_lines.append(' fi')
# Source the env file the prebuilt-download path writes so
# LD_LIBRARY_PATH includes the directory holding libllama.so
# and friends. No-op when prebuilt wasn't used.
runner_lines.append(' [ -r ~/.config/odysseus-llama-cpp-env ] && . ~/.config/odysseus-llama-cpp-env')
# Auto-upgrade pip llama-cpp-python to the CUDA-enabled
# wheel when (a) NVIDIA hardware is present and (b) the
# currently-installed wheel is CPU-only. Without this the
# user gets the Python server happily running at 3 tok/s
# because pip's default index ships CPU-only wheels.
# Forward-compat: cu124 wheels work on driver/runtime
# 12.4+ including the cu13.x line.
runner_lines.append(' if command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L 2>/dev/null | grep -q "GPU " && python3 -c "import llama_cpp" 2>/dev/null; then')
runner_lines.append(' if ! python3 -c "import llama_cpp; import sys; sys.exit(0 if llama_cpp.llama_supports_gpu_offload() else 1)" 2>/dev/null; then')
runner_lines.append(' echo "[odysseus] NVIDIA detected but installed llama-cpp-python is CPU-only — reinstalling with CUDA wheel index for GPU offload..."')
runner_lines.append(' python3 -m pip install --user --break-system-packages --force-reinstall --no-cache-dir "llama-cpp-python[server]" --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124 2>&1 | tail -8 || echo "[odysseus] WARNING: CUDA wheel reinstall failed — Python server will stay CPU-only (slow). Manual fix: pip install --user --force-reinstall \'llama-cpp-python[server]\' --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124"')
runner_lines.append(' if python3 -c "import llama_cpp; import sys; sys.exit(0 if llama_cpp.llama_supports_gpu_offload() else 1)" 2>/dev/null; then')
runner_lines.append(' echo "[odysseus] llama-cpp-python now supports GPU offload."')
runner_lines.append(' fi')
runner_lines.append(' fi')
runner_lines.append(' fi')
# SHORT-CIRCUIT before the build/pip fallback: if the
# native binary is missing but llama_cpp Python is already
# installed, drop a wrapper at ~/bin/llama-server that
# translates llama-server CLI args to llama_cpp.server's
# underscore-style flags. The user's serve command stays
# `llama-server ...` and "just works" — no build, no cmake,
# no second install. This is the path that unblocks every
# remote where pip-installed llama-cpp-python is already
# working but Cookbook used to insist on a native binary.
runner_lines.append(' if ! command -v llama-server >/dev/null 2>&1 && python3 -c "import llama_cpp" 2>/dev/null; then')
runner_lines.append(' mkdir -p ~/bin')
runner_lines.append(' cat > ~/bin/llama-server <<\'_ODY_LLAMA_SHIM_EOF\'')
runner_lines.append('#!/usr/bin/env bash')
runner_lines.append('# Auto-generated by Odysseus Cookbook: a `llama-server` lookalike')
runner_lines.append('# that translates the native CLI to `python -m llama_cpp.server`.')
runner_lines.append('# Lets cookbook-generated launch commands run unchanged on hosts')
runner_lines.append('# where only the pip llama-cpp-python package is installed.')
runner_lines.append('ARGS=()')
runner_lines.append('while [ $# -gt 0 ]; do')
runner_lines.append(' case "$1" in')
runner_lines.append(' -ngl|--gpu-layers|--n-gpu-layers) ARGS+=(--n_gpu_layers "$2"); shift 2 ;;')
runner_lines.append(' -c|--ctx-size) ARGS+=(--n_ctx "$2"); shift 2 ;;')
runner_lines.append(' -b|--batch-size) ARGS+=(--n_batch "$2"); shift 2 ;;')
runner_lines.append(' -ub|--ubatch-size) shift 2 ;; # llama-cpp-python has no separate ubatch')
runner_lines.append(' --flash-attn) ARGS+=(--flash_attn true); shift 2 ;;')
runner_lines.append(' --cache-type-k) ARGS+=(--type_k "$2"); shift 2 ;;')
runner_lines.append(' --cache-type-v) ARGS+=(--type_v "$2"); shift 2 ;;')
runner_lines.append(' --n-cpu-moe) ARGS+=(--n_cpu_moe "$2"); shift 2 ;;')
runner_lines.append(' --mmproj) ARGS+=(--clip_model_path "$2"); shift 2 ;;')
runner_lines.append(' --image-max-tokens) shift 2 ;; # native-only')
runner_lines.append(' --no-mmap) ARGS+=(--no_mmap true); shift ;;')
runner_lines.append(' --no-warmup) shift ;; # native-only')
runner_lines.append(' --chat-template) ARGS+=(--chat_format "$2"); shift 2 ;;')
runner_lines.append(' --fit|--split-mode|--tensor-split|--main-gpu|--parallel) shift 2 ;; # native-only')
runner_lines.append(' --mlock) ARGS+=(--use_mlock true); shift ;;')
runner_lines.append(' *) ARGS+=("$1"); shift ;;')
runner_lines.append(' esac')
runner_lines.append('done')
runner_lines.append('exec python3 -m llama_cpp.server "${ARGS[@]}"')
runner_lines.append('_ODY_LLAMA_SHIM_EOF')
runner_lines.append(' chmod +x ~/bin/llama-server')
runner_lines.append(' echo "[odysseus] Created llama-server shim → python -m llama_cpp.server (no native binary needed)"')
runner_lines.append(' fi')
runner_lines.append(' # If the native build failed, fall back to the Python bindings.')
runner_lines.append(' if ! command -v llama-server &>/dev/null && ! python3 -c "import llama_cpp" 2>/dev/null; then')
runner_lines.append(' echo "llama-server build failed — installing Python bindings as fallback..."')
@@ -1489,6 +1671,96 @@ def setup_cookbook_routes() -> APIRouter:
runner_lines.append(' echo "ERROR: vLLM is not installed."')
runner_lines.append(' ODYSSEUS_PREFLIGHT_EXIT=127')
runner_lines.append('fi')
runner_lines.append(f"ODYSSEUS_SERVE_CMD='{_bash_squote(req.cmd)}'")
runner_lines.append('if [ -z "$ODYSSEUS_PREFLIGHT_EXIT" ]; then')
runner_lines.append(' ODYSSEUS_VLLM_HELP_CMD="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
runner_lines.append('import shlex, sys')
runner_lines.append('parts = shlex.split(sys.argv[1])')
runner_lines.append('try:')
runner_lines.append(' serve_i = parts.index("serve")')
runner_lines.append('except ValueError:')
runner_lines.append(' print("vllm serve --help")')
runner_lines.append('else:')
runner_lines.append(' print(shlex.join(parts[:serve_i + 1] + ["--help"]))')
runner_lines.append('PY')
runner_lines.append(')"')
runner_lines.append(' ODYSSEUS_VLLM_SUPPORTS_SWAP=0')
runner_lines.append(' if eval "$ODYSSEUS_VLLM_HELP_CMD" 2>&1 | grep -q -- "--swap-space"; then ODYSSEUS_VLLM_SUPPORTS_SWAP=1; fi')
runner_lines.append('fi')
runner_lines.append('if [ -z "$ODYSSEUS_PREFLIGHT_EXIT" ] && [ "${ODYSSEUS_VLLM_SUPPORTS_SWAP:-0}" = "1" ] && ! printf "%s" "$ODYSSEUS_SERVE_CMD" | grep -q -- "--swap-space"; then')
runner_lines.append(' echo "[odysseus] Setting vLLM --swap-space 0 so the runtime does not reserve CPU swap per GPU."')
runner_lines.append(' ODYSSEUS_SERVE_CMD="${ODYSSEUS_SERVE_CMD} --swap-space 0"')
runner_lines.append('fi')
runner_lines.append('if [ -z "$ODYSSEUS_PREFLIGHT_EXIT" ] && [ "${ODYSSEUS_VLLM_SUPPORTS_SWAP:-0}" != "1" ]; then')
runner_lines.append(' if printf "%s" "$ODYSSEUS_SERVE_CMD" | grep -q -- "--swap-space"; then')
runner_lines.append(' echo "[odysseus] vLLM serve does not expose --swap-space; removing the flag and patching the runtime default to 0."')
runner_lines.append(' ODYSSEUS_SERVE_CMD="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
runner_lines.append('import shlex, sys')
runner_lines.append('parts = shlex.split(sys.argv[1])')
runner_lines.append('out = []')
runner_lines.append('skip = False')
runner_lines.append('for part in parts:')
runner_lines.append(' if skip:')
runner_lines.append(' skip = False')
runner_lines.append(' continue')
runner_lines.append(' if part == "--swap-space":')
runner_lines.append(' skip = True')
runner_lines.append(' continue')
runner_lines.append(' if part.startswith("--swap-space="):')
runner_lines.append(' continue')
runner_lines.append(' out.append(part)')
runner_lines.append('print(shlex.join(out))')
runner_lines.append('PY')
runner_lines.append(')"')
runner_lines.append(' fi')
runner_lines.append(' ODYSSEUS_SERVE_CMD="$(python3 - "$ODYSSEUS_SERVE_CMD" <<\'PY\'')
runner_lines.append('import shlex, sys')
runner_lines.append('parts = shlex.split(sys.argv[1])')
runner_lines.append('patch = r"""import inspect, sys')
runner_lines.append('from vllm.engine.arg_utils import EngineArgs, AsyncEngineArgs')
runner_lines.append('def _odysseus_swap0(cls):')
runner_lines.append(' params = list(inspect.signature(cls).parameters)')
runner_lines.append(' if "swap_space" not in params:')
runner_lines.append(' return')
runner_lines.append(' idx = params.index("swap_space")')
runner_lines.append(' defaults = list(cls.__init__.__defaults__ or ())')
runner_lines.append(' if idx < len(defaults):')
runner_lines.append(' defaults[idx] = 0')
runner_lines.append(' cls.__init__.__defaults__ = tuple(defaults)')
runner_lines.append(' fields = getattr(cls, "__dataclass_fields__", {})')
runner_lines.append(' if "swap_space" in fields:')
runner_lines.append(' fields["swap_space"].default = 0')
runner_lines.append('_odysseus_swap0(EngineArgs)')
runner_lines.append('_odysseus_swap0(AsyncEngineArgs)')
runner_lines.append('try:')
runner_lines.append(' from vllm.config import CacheConfig')
runner_lines.append(' CacheConfig.swap_space = 0')
runner_lines.append('except Exception:')
runner_lines.append(' pass')
runner_lines.append('_orig_create_engine_config = EngineArgs.create_engine_config')
runner_lines.append('def _odysseus_create_engine_config(self, *args, **kwargs):')
runner_lines.append(' self.swap_space = 0')
runner_lines.append(' return _orig_create_engine_config(self, *args, **kwargs)')
runner_lines.append('EngineArgs.create_engine_config = _odysseus_create_engine_config')
runner_lines.append('AsyncEngineArgs.create_engine_config = _odysseus_create_engine_config')
runner_lines.append('from vllm.entrypoints.cli.main import main')
runner_lines.append('sys.exit(main())"""')
runner_lines.append('try:')
runner_lines.append(' serve_i = parts.index("serve")')
runner_lines.append('except ValueError:')
runner_lines.append(' print(shlex.join(parts))')
runner_lines.append('else:')
runner_lines.append(' exe_i = serve_i - 1')
runner_lines.append(' exe = parts[exe_i] if exe_i >= 0 else "vllm"')
runner_lines.append(' py = "python3"')
runner_lines.append(' if exe.endswith("/bin/vllm"):')
runner_lines.append(' py = exe[:-len("/bin/vllm")] + "/bin/python"')
runner_lines.append(' parts[exe_i:serve_i] = [py, "-c", patch]')
runner_lines.append(' print(shlex.join(parts))')
runner_lines.append('PY')
runner_lines.append(')"')
runner_lines.append(' echo "[odysseus] Patched vLLM internal swap_space default to 0 for this runtime."')
runner_lines.append('fi')
elif "sglang.launch_server" in req.cmd:
runner_lines.append('export PATH="$HOME/.local/bin:$PATH"')
runner_lines.append('if ! command -v sglang &>/dev/null; then')
@@ -1530,7 +1802,10 @@ def setup_cookbook_routes() -> APIRouter:
runner_lines,
keep_shell_open=not local_windows,
)
runner_lines.append(req.cmd)
if "vllm serve" in req.cmd:
runner_lines.append('eval "$ODYSSEUS_SERVE_CMD"')
else:
runner_lines.append(req.cmd)
if local_windows:
# Detached background process — no interactive shell to keep open.
# Print the exit marker the status poller looks for, then stop.
@@ -1577,10 +1852,10 @@ def setup_cookbook_routes() -> APIRouter:
setup_cmd = (
f"{scp_extras}"
f"scp -O {_Pf}-q '{runner_path}' {remote}:{remote_runner} && "
f"ssh {_pf}{remote} 'chmod +x {remote_runner} && tmux new-session -d -s {session_id} \"./{remote_runner}\"'"
f"ssh {_pf}{remote} 'chmod +x {remote_runner} && tmux set-option -g history-limit 100000 2>/dev/null; tmux new-session -d -s {session_id} \"./{remote_runner}\"'"
)
else:
setup_cmd = f"tmux new-session -d -s {session_id} {shlex.quote(str(runner_path))}"
setup_cmd = f"tmux set-option -g history-limit 100000 2>/dev/null; tmux new-session -d -s {session_id} {shlex.quote(str(runner_path))}"
if setup_cmd is None:
# LOCAL Windows: launch the bash runner detached; no tmux setup_cmd.
@@ -1834,6 +2109,25 @@ def setup_cookbook_routes() -> APIRouter:
out, err = await _run_gpu_shell("ls -1 /sys/class/drm 2>/dev/null", host, ssh_port, timeout=4)
if err is not None or not out:
return []
# Pick the runtime label up-front so each GPU dict gets the
# right `backend`. AMD silicon can be driven by ROCm/HIP (native)
# OR Vulkan (mesa RADV). Reporting "rocm" on a host where no
# ROCm toolchain is installed misleads the frontend env-var
# prefix logic — it would emit `HIP_VISIBLE_DEVICES=` for a
# Vulkan-only stack, which is a silent no-op at best.
rt_out, _ = await _run_gpu_shell(
'command -v rocminfo >/dev/null 2>&1 && echo rocm '
'|| (command -v hipconfig >/dev/null 2>&1 && echo rocm) '
'|| (command -v vulkaninfo >/dev/null 2>&1 && echo vulkan) '
'|| echo unknown',
host, ssh_port, timeout=4,
)
_amd_runtime = (rt_out or "").strip().splitlines()[-1:][0].strip() if rt_out else "rocm"
if _amd_runtime not in ("rocm", "vulkan"):
# Default to rocm so existing ROCm-installed hosts keep
# working; "unknown" only happens when neither toolchain is
# detected (e.g. minimal sysfs read on a fresh box).
_amd_runtime = "rocm"
gpus = []
for entry in out.split():
if not entry.startswith("card") or "-" in entry:
@@ -1877,7 +2171,7 @@ def setup_cookbook_routes() -> APIRouter:
"free_mb": free_mb, "total_mb": total_mb, "used_mb": used_mb,
"gtt_used_mb": gtt_used_mb,
"util_pct": 0, "busy": bool(total_mb and (free_mb / total_mb) < 0.85),
"processes": [], "backend": "rocm", "source": "amd-sysfs",
"processes": [], "backend": _amd_runtime, "source": "amd-sysfs",
"unified_memory": unified,
})
if gpus:
@@ -2018,10 +2312,15 @@ def setup_cookbook_routes() -> APIRouter:
amd_gpus = await _probe_amd_sysfs(host, ssh_port)
if amd_gpus:
# The per-GPU dict already carries the runtime label picked by
# _probe_amd_sysfs (rocm vs vulkan); mirror that into the
# wrapper so the frontend can read `data.backend` directly
# without scanning the list.
_amd_wrap_backend = str(amd_gpus[0].get("backend") or "rocm")
return {
"ok": True,
"gpus": amd_gpus,
"backend": "rocm",
"backend": _amd_wrap_backend,
"source": "amd-sysfs",
"fallback_from": "nvidia-smi",
"nvidia_error": nvidia_error,
@@ -2110,8 +2409,8 @@ def setup_cookbook_routes() -> APIRouter:
try:
return _state_for_client(json.loads(_cookbook_state_path.read_text(encoding="utf-8")))
except Exception:
return {}
return {}
return _state_for_client({})
return _state_for_client({})
@router.post("/api/cookbook/state")
async def save_cookbook_state(request: Request):
@@ -2161,6 +2460,17 @@ def setup_cookbook_routes() -> APIRouter:
disk_tasks = on_disk.get("tasks") or [] if isinstance(on_disk, dict) else []
incoming_tasks = data.get("tasks") if isinstance(data.get("tasks"), list) else []
incoming_removed = data.get("removedTasks") if isinstance(data.get("removedTasks"), dict) else {}
disk_removed = on_disk.get("removedTasks") if isinstance(on_disk, dict) and isinstance(on_disk.get("removedTasks"), dict) else {}
removed_tasks = {**disk_removed, **incoming_removed}
data["removedTasks"] = removed_tasks
removed_ids = set(removed_tasks.keys())
if removed_ids:
incoming_tasks = [
t for t in incoming_tasks
if not (isinstance(t, dict) and t.get("sessionId") in removed_ids)
]
data["tasks"] = incoming_tasks
# Anti-poisoning guard: a stale browser tab can keep POSTing a
# download task as status='done' from before the strict-finish
# fix landed, undoing any server-side correction. For each
@@ -2198,6 +2508,8 @@ def setup_cookbook_routes() -> APIRouter:
sid = t.get("sessionId")
if not sid or sid in incoming_ids:
continue # client's version wins
if sid in removed_ids:
continue # intentional cross-device clear/remove
ts = t.get("ts") or 0
if isinstance(ts, (int, float)) and (now_ms - ts) <= RACE_WINDOW_MS:
preserved.append(t)
@@ -2304,16 +2616,14 @@ def setup_cookbook_routes() -> APIRouter:
# Add 30% headroom for KV cache, activations, etc.
needed_vram = (est_vram * 1.3) if est_vram else None
if vram_gb > 0 and needed_vram is not None and needed_vram > vram_gb:
continue
# Unknown-size models (e.g. MiniMax-M2.7, DeepSeek-V4-Flash) have no
# "NB" in the repo id, so the regex above can't extract their
# param count. Previously we dropped them entirely, which made
# brand-new flagship releases silently vanish from this list even
# on rigs with hundreds of GB of VRAM. Adapters/LoRAs are already
# filtered by _is_excluded(), so what falls through here is
# overwhelmingly full models — keep them, just without a size
# badge (the frontend handles needed_vram_gb=null gracefully).
if vram_gb > 0:
if needed_vram is None:
# The "trending models that fit" list must be conservative:
# if we cannot estimate size from the repo id/tags, do not
# present it as runnable on this hardware.
continue
if needed_vram > vram_gb:
continue
out.append({
"repo_id": repo_id,
@@ -2510,6 +2820,33 @@ def setup_cookbook_routes() -> APIRouter:
except Exception as e:
logger.warning(f"orphan sweep: state write failed: {e}")
@router.get("/api/cookbook/hf-gguf-files")
async def hf_gguf_files(repo_id: str, owner: str = Depends(require_user)):
"""List GGUF files in a HuggingFace repo for the direct-download picker."""
import httpx
repo_id = _validate_repo_id(repo_id)
url = f"https://huggingface.co/api/models/{repo_id}"
try:
headers = {}
token = _load_stored_hf_token()
if token:
headers["Authorization"] = f"Bearer {token}"
async with httpx.AsyncClient(timeout=15, follow_redirects=True) as client:
resp = await client.get(url, headers=headers)
if resp.status_code != 200:
return {"ok": False, "files": [], "error": f"HF API HTTP {resp.status_code}"}
data = resp.json()
except Exception:
logger.exception("HF GGUF file scan failed for %s", repo)
return {"ok": False, "files": [], "error": "HF API request failed"}
files = [
str(s.get("rfilename") or "")
for s in data.get("siblings", [])
if str(s.get("rfilename") or "").lower().endswith(".gguf")
]
return {"ok": True, "repo_id": repo_id, "files": files}
# In-memory cache for the Ollama library scrape. ollama.com is a public
# site, but it doesn't expose a stable JSON listing — we fetch the HTML
# search page and regex out the model cards. Cached for 1 h so a busy
@@ -2625,6 +2962,193 @@ def setup_cookbook_routes() -> APIRouter:
"error": _ollama_library_cache["error"],
}
# ── vLLM recipe scraper ─────────────────────────────────────────────
# Fetches the official YAML recipe for a model from vllm-project/recipes
# and normalizes it into a small JSON the frontend can consume. Cached
# per-repo so the GitHub raw endpoint isn't hammered.
_vllm_recipe_cache: dict[str, tuple[float, dict | None]] = {}
# Manifest of all <org>/<model> ids that have a recipe in the upstream
# repo. Cheap to fetch (one Git Tree API call), so we cache the whole
# set for ~12h. Per-row "does this model have a recipe?" lookups hit
# this set instead of doing 912 individual recipe fetches.
_vllm_recipe_manifest: dict = {"fetched_at": 0.0, "models": set(), "error": ""}
@router.get("/api/cookbook/vllm-recipe-manifest")
async def vllm_recipe_manifest(refresh: int = 0):
"""Return the set of <org>/<model> ids known to have a vLLM recipe.
One GitHub Tree API call, 12h cache. The frontend uses this to badge
rows in the model list before the user expands them."""
import time as _time
import httpx as _httpx
TTL = 12 * 3600.0
now = _time.time()
if (
refresh
or (now - _vllm_recipe_manifest["fetched_at"]) > TTL
or not _vllm_recipe_manifest["models"]
):
url = (
"https://api.github.com/repos/vllm-project/recipes/"
"git/trees/main?recursive=1"
)
def _fetch_sync() -> tuple[int, dict | None, str]:
try:
headers = {"Accept": "application/vnd.github+json"}
with _httpx.Client(timeout=10.0, follow_redirects=True) as client:
r = client.get(url, headers=headers)
if r.status_code != 200:
return r.status_code, None, r.text[:200]
return 200, r.json(), ""
except Exception as e:
return 0, None, f"fetch error: {e}"
status, data, err = await asyncio.to_thread(_fetch_sync)
if status == 200 and isinstance(data, dict):
models: set[str] = set()
for entry in data.get("tree") or []:
path = (entry or {}).get("path") or ""
if not path.startswith("models/") or not path.endswith(".yaml"):
continue
# path = "models/<org>/<model>.yaml" → "<org>/<model>"
body = path[len("models/"):-len(".yaml")]
if "/" in body:
models.add(body)
_vllm_recipe_manifest["models"] = models
_vllm_recipe_manifest["fetched_at"] = now
_vllm_recipe_manifest["error"] = ""
else:
_vllm_recipe_manifest["error"] = (
f"HTTP {status}: {err}" if status else err
)
# Don't clobber a stale-but-usable list on transient failures.
if not _vllm_recipe_manifest["models"]:
return {
"models": [],
"count": 0,
"error": _vllm_recipe_manifest["error"],
}
return {
"models": sorted(_vllm_recipe_manifest["models"]),
"count": len(_vllm_recipe_manifest["models"]),
"fetched_at": _vllm_recipe_manifest["fetched_at"],
"error": _vllm_recipe_manifest["error"],
}
@router.get("/api/cookbook/vllm-recipe")
async def vllm_recipe(repo: str, refresh: int = 0):
"""Return the vLLM official recipe for a HuggingFace repo, if one
exists at vllm-project/recipes. `repo` is the full HF id like
'MiniMaxAI/MiniMax-M2'. Cached 6h."""
import time as _time
import httpx as _httpx
import yaml as _yaml
TTL = 6 * 3600.0
now = _time.time()
repo = (repo or "").strip().strip("/")
if "/" not in repo:
return {"exists": False, "error": "repo must be <org>/<model>"}
cached = _vllm_recipe_cache.get(repo)
if cached and not refresh and (now - cached[0]) < TTL:
return cached[1] or {"exists": False, "cached": True}
url = (
f"https://raw.githubusercontent.com/vllm-project/recipes/"
f"main/models/{repo}.yaml"
)
def _fetch_sync() -> tuple[int, str]:
try:
with _httpx.Client(timeout=8.0, follow_redirects=True) as client:
r = client.get(url)
return r.status_code, r.text
except Exception as e:
return 0, f"fetch error: {e}"
status, text = await asyncio.to_thread(_fetch_sync)
if status == 404:
_vllm_recipe_cache[repo] = (now, {"exists": False})
return {"exists": False}
if status != 200:
return {"exists": False, "error": f"HTTP {status}", "transient": True}
try:
doc = _yaml.safe_load(text) or {}
except Exception as e:
return {"exists": False, "error": f"yaml parse: {e}"}
meta = doc.get("meta") or {}
model = doc.get("model") or {}
features = doc.get("features") or {}
deps = doc.get("dependencies") or []
variants = doc.get("variants") or {}
hw_overrides = doc.get("hardware_overrides") or {}
strat_overrides = doc.get("strategy_overrides") or {}
# Tool-call + reasoning parsers, as flat arg arrays, so the frontend
# can drop them straight into the launch command.
tool_calling = features.get("tool_calling") or {}
reasoning = features.get("reasoning") or {}
normalized = {
"exists": True,
"source_url": url,
"title": meta.get("title") or "",
"provider": meta.get("provider") or "",
"description": meta.get("description") or "",
"date_updated": str(meta.get("date_updated") or ""),
"hardware_support": meta.get("hardware") or {},
"model_id": model.get("model_id") or repo,
"min_vllm_version": model.get("min_vllm_version") or "",
"architecture": model.get("architecture") or "",
"parameter_count": model.get("parameter_count") or "",
"active_parameters": model.get("active_parameters") or "",
"context_length": model.get("context_length") or 0,
"base_args": list(model.get("base_args") or []),
"base_env": dict(model.get("base_env") or {}),
"tool_calling": {
"description": tool_calling.get("description") or "",
"args": list(tool_calling.get("args") or []),
} if tool_calling else None,
"reasoning": {
"description": reasoning.get("description") or "",
"args": list(reasoning.get("args") or []),
} if reasoning else None,
"dependencies": [
{
"note": (d.get("note") or "").strip(),
"command": (d.get("command") or "").strip(),
"optional": bool(d.get("optional", False)),
}
for d in deps if isinstance(d, dict)
],
"variants": {
k: {
"model_id": v.get("model_id") or model.get("model_id") or repo,
"precision": v.get("precision") or "",
"vram_minimum_gb": v.get("vram_minimum_gb") or 0,
"description": v.get("description") or "",
"extra_args": list(v.get("extra_args") or []),
"extra_env": dict(v.get("extra_env") or {}),
}
for k, v in variants.items() if isinstance(v, dict)
},
"hardware_overrides": {
hw: {
"extra_args": list((ov or {}).get("extra_args") or []),
"extra_env": dict((ov or {}).get("extra_env") or {}),
}
for hw, ov in hw_overrides.items() if isinstance(ov, dict)
},
"strategy_overrides": {
strat: dict(ov or {})
for strat, ov in strat_overrides.items() if isinstance(ov, dict)
},
"compatible_strategies": list(doc.get("compatible_strategies") or []),
}
_vllm_recipe_cache[repo] = (now, normalized)
return normalized
@router.get("/api/cookbook/tasks/status")
async def cookbook_tasks_status(request: Request):
"""Check status of all active cookbook tmux sessions.
routes/diagnostics_routes.py
+26 -1
View File
@@ -1,12 +1,13 @@
"""Diagnostics routes — /api/db/stats, /api/rag/stats, /api/test/youtube, /api/test-research."""
import logging
import os
from typing import Dict, Any
from fastapi import APIRouter, HTTPException, Form, Request
from services.youtube.youtube_handler import extract_youtube_id, extract_transcript_async
from core.constants import DEFAULT_HOST
from core.constants import DEFAULT_HOST, DATA_DIR
from core.middleware import require_admin
logger = logging.getLogger(__name__)
@@ -28,6 +29,30 @@ def setup_diagnostics_routes(
from src.service_health import collect_service_health
return await collect_service_health(rag_manager, memory_vector)
@router.get("/api/diagnostics/logs")
async def get_diagnostics_logs(request: Request, limit: int = 200) -> Dict[str, Any]:
require_admin(request)
limit = max(1, min(limit, 1000))
try:
log_file = os.path.join(DATA_DIR, "logs", "app.log")
if not os.path.exists(log_file):
return {"status": "success", "logs": []}
# Safe tail read of the log file (max 5MB via rotation)
with open(log_file, "r", encoding="utf-8", errors="ignore") as f:
lines = f.readlines()
tail_lines = lines[-limit:] if len(lines) > limit else lines
tail_lines = [line.rstrip('\r\n') for line in tail_lines]
return {
"status": "success",
"logs": tail_lines
}
except Exception as e:
logger.error(f"Diagnostics logs retrieval error: {e}")
raise HTTPException(500, f"Failed to retrieve logs: {str(e)}")
@router.get("/api/db/stats")
async def get_database_stats(request: Request) -> Dict[str, Any]:
require_admin(request)
routes/document_helpers.py
+7 -2
View File
@@ -12,6 +12,7 @@ from pydantic import BaseModel
from core.database import Document, DocumentVersion
from core.database import Session as DbSession
from src.auth_helpers import _auth_disabled
from src.upload_handler import UploadHandler
logger = logging.getLogger(__name__)
@@ -78,6 +79,8 @@ def _verify_doc_owner(db, doc: Document, user: str):
the session join for any not-yet-backfilled legacy row.
"""
if user is None:
if _auth_disabled():
return # Single-user / no-auth mode: allow access
raise HTTPException(403, "Authentication required")
if doc.owner is not None:
if doc.owner != user:
@@ -102,8 +105,10 @@ def _owner_session_filter(q, user):
The owner backfill runs in init_db before the app serves requests, so
by the time this filter is live there are no NULL-owner rows to leak;
we therefore match the owner strictly."""
if user is None:
we therefore match the owner strictly for authenticated callers."""
if not user:
if user == "" or _auth_disabled():
return q
return q.filter(False)
return q.filter(Document.owner == user)
routes/document_routes.py
+13 -5
View File
@@ -10,7 +10,7 @@ from fastapi import APIRouter, HTTPException, Query, Request, UploadFile, File,
from sqlalchemy import case, func, or_
from core.database import SessionLocal, Document, DocumentVersion
from core.database import Session as DbSession
from src.auth_helpers import get_current_user
from src.auth_helpers import get_current_user, _auth_disabled
from src.constants import MAIL_ATTACHMENTS_DIR
logger = logging.getLogger(__name__)
@@ -388,7 +388,8 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
db = SessionLocal()
try:
if not user:
raise HTTPException(403, "Authentication required")
if not _auth_disabled():
raise HTTPException(403, "Authentication required")
# v2 review HIGH-9: raise 403 explicitly when the caller
# can't see this session, instead of returning [] which the
# UI treats identically to "no docs" and silently masks
@@ -503,7 +504,8 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
user = get_current_user(request)
try:
data = await request.json()
except Exception:
except Exception as e:
logger.warning("Failed to parse export request body, defaulting to empty", exc_info=e)
data = {}
ids = data.get("ids") or []
if not ids:
@@ -645,8 +647,8 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
try:
from src.agent_tools.document_tools import clear_active_document
clear_active_document(doc_id)
except Exception:
pass
except Exception as e:
logger.warning("Failed to clear active document %r on detach", doc_id, exc_info=e)
db.commit()
db.refresh(doc)
return _doc_to_dict(doc)
@@ -1331,6 +1333,12 @@ def setup_document_routes(session_manager, upload_handler=None) -> APIRouter:
if not pdf_path:
raise HTTPException(404, f"Source PDF {upload_id} not found")
# Fail fast with a clear 503 if the optional PyMuPDF dependency
# is missing — fill_fields/stamp_annotations will otherwise
# raise RuntimeError deep inside and bubble out as a 500.
# Mirrors the convention in _load_pdf_viewer_fitz above.
_load_pdf_viewer_fitz()
values = parse_markdown_to_values(doc.current_content or "")
out_path = tempfile.NamedTemporaryFile(suffix=".pdf", delete=False).name
_to_unlink.append(out_path)
routes/email_helpers.py
+200 -20
View File
@@ -13,6 +13,8 @@ and `email_pollers.py` (the background loops):
"""
import os
import base64
import time
import imaplib
import smtplib
import email as email_mod
@@ -38,6 +40,116 @@ from src.secret_storage import decrypt as _decrypt
logger = logging.getLogger(__name__)
class EmailNotConfiguredError(RuntimeError):
"""Raised when an IMAP operation is attempted on an account that has no
inbox configured (e.g. a send-only / SMTP-only account).
Subclasses RuntimeError so existing broad ``except Exception`` handlers
keep working; callers that want to treat "no inbox" as an empty result
rather than a failure can catch this type specifically.
"""
def _xoauth2_raw(user: str, access_token: str) -> str:
"""The SASL XOAUTH2 initial-response string (unencoded).
Both smtplib.SMTP.auth() and imaplib.IMAP4.authenticate() base64-encode
the value their callback returns, so callers pass this raw form — never
pre-encoded — to avoid double base64.
"""
return f"user={user}\x01auth=Bearer {access_token}\x01\x01"
def _xoauth2_bytes(user: str, access_token: str) -> bytes:
"""Raw XOAUTH2 bytes for imaplib's authenticate() callback."""
return _xoauth2_raw(user, access_token).encode()
def make_oauth_state(account_id: str, owner: str) -> str:
"""Return an HMAC-signed, base64-encoded OAuth state token.
Encodes account_id + owner + a random nonce, signed with the app secret
so the callback can validate that the flow was initiated by an
authenticated, owning user (CSRF / state-forgery protection).
"""
import hmac as _hmac, hashlib as _hl, secrets as _sec
from src.secret_storage import _load_or_create_key
nonce = _sec.token_hex(16)
payload = json.dumps({"a": account_id, "o": owner, "n": nonce}, separators=(",", ":"))
sig = _hmac.new(_load_or_create_key(), payload.encode(), _hl.sha256).hexdigest()
return base64.urlsafe_b64encode(f"{payload}|{sig}".encode()).decode()
def verify_oauth_state(state: str) -> dict | None:
"""Verify an OAuth state token's HMAC signature.
Returns the decoded payload dict ({"a", "o", "n"}) on success, or None if
the token is malformed, tampered, or signed with a different key.
"""
import hmac as _hmac, hashlib as _hl
from src.secret_storage import _load_or_create_key
try:
decoded = base64.urlsafe_b64decode(state.encode()).decode()
payload, sig = decoded.rsplit("|", 1)
expected = _hmac.new(_load_or_create_key(), payload.encode(), _hl.sha256).hexdigest()
if not _hmac.compare_digest(sig, expected):
return None
return json.loads(payload)
except Exception:
return None
def _refresh_google_token(account_id: str) -> str | None:
"""Exchange the stored refresh token for a new access token and persist it."""
import httpx
from core.database import SessionLocal as _SL, EmailAccount as _EA
from src.secret_storage import encrypt as _enc, decrypt as _dec
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
client_secret = os.environ.get("GOOGLE_OAUTH_CLIENT_SECRET", "")
if not client_id or not client_secret:
return None
db = _SL()
try:
row = db.get(_EA, account_id)
if not row or not row.oauth_refresh_token:
return None
refresh_token = _dec(row.oauth_refresh_token or "")
if not refresh_token:
return None
resp = httpx.post("https://oauth2.googleapis.com/token", data={
"client_id": client_id,
"client_secret": client_secret,
"refresh_token": refresh_token,
"grant_type": "refresh_token",
}, timeout=10)
resp.raise_for_status()
data = resp.json()
access_token = data["access_token"]
row.oauth_access_token = _enc(access_token)
row.oauth_token_expiry = str(int(time.time()) + data.get("expires_in", 3600))
db.commit()
return access_token
except Exception:
logger.warning(f"Google token refresh failed for account {account_id}")
return None
finally:
db.close()
def _get_valid_google_token(account_id: str, cfg: dict) -> str | None:
"""Return a valid Google access token, refreshing if expired or missing."""
from src.secret_storage import decrypt as _dec
access_token = _dec(cfg.get("oauth_access_token") or "")
expiry_str = cfg.get("oauth_token_expiry") or ""
if access_token and expiry_str:
try:
if int(expiry_str) - 60 > time.time():
return access_token
except (ValueError, TypeError):
pass
return _refresh_google_token(account_id)
def _smtp_security_mode(cfg: dict) -> str:
raw = str(cfg.get("smtp_security") or "").strip().lower()
if raw in {"ssl", "starttls", "none"}:
@@ -54,20 +166,29 @@ def _send_smtp_message(cfg: dict, from_addr: str, recipients: list[str], message
port = int(cfg.get("smtp_port") or 465)
user = cfg.get("smtp_user") or ""
password = cfg.get("smtp_password") or ""
def _auth_smtp(smtp):
if cfg.get("oauth_provider") == "google":
token = _get_valid_google_token(cfg.get("account_id"), cfg)
if not token:
raise RuntimeError("Google OAuth token unavailable — reconnect the account")
smtp.ehlo()
smtp.auth("XOAUTH2", lambda challenge=None: _xoauth2_raw(user, token), initial_response_ok=True)
elif user and password:
smtp.login(user, password)
security = _smtp_security_mode(cfg)
if security == "ssl":
with smtplib.SMTP_SSL(host, port, timeout=timeout) as smtp:
if user and password:
smtp.login(user, password)
_auth_smtp(smtp)
smtp.sendmail(from_addr, recipients, message)
return
with smtplib.SMTP(host, port, timeout=timeout) as smtp:
if security == "starttls":
smtp.starttls()
if user and password:
smtp.login(user, password)
_auth_smtp(smtp)
smtp.sendmail(from_addr, recipients, message)
@@ -114,8 +235,9 @@ def _strip_think(text: str) -> str:
"""
if not text:
return ""
from src.text_helpers import strip_think as _central, _THINK_CLOSED_RE, _THINK_OPEN_RE, _THINK_TAG_RE
had_think = bool(_THINK_CLOSED_RE.search(text) or _THINK_OPEN_RE.search(text) or _THINK_TAG_RE.search(text))
from src.text_helpers import strip_think as _central, _THINK_TAG_RE
# Single linear tag check; the old closed/open `.search()` calls could ReDoS.
had_think = bool(_THINK_TAG_RE.search(text))
return _central(text, prose=had_think, prompt_echo=True)
@@ -701,10 +823,16 @@ def _get_email_config(account_id: str | None = None, owner: str = "") -> dict:
"imap_password": _decrypt(row.imap_password or ""),
"imap_starttls": bool(row.imap_starttls),
"from_address": row.from_address or row.imap_user or "",
"oauth_provider": row.oauth_provider or "",
"oauth_access_token": row.oauth_access_token or "",
"oauth_refresh_token": row.oauth_refresh_token or "",
"oauth_token_expiry": row.oauth_token_expiry or "",
"display_name": row.display_name or "",
}
if not (cfg["smtp_host"] and cfg["smtp_user"] and cfg["smtp_password"]):
is_oauth = bool(cfg.get("oauth_provider"))
if not is_oauth and not (cfg["smtp_host"] and cfg["smtp_user"] and cfg["smtp_password"]):
logger.warning(f"SMTP not configured for account {row.name!r}")
if not (cfg["imap_host"] and cfg["imap_user"] and cfg["imap_password"]):
if not is_oauth and not (cfg["imap_host"] and cfg["imap_user"] and cfg["imap_password"]):
logger.warning(f"IMAP not configured for account {row.name!r}")
return cfg
finally:
@@ -811,6 +939,14 @@ def _imap_connect(account_id: str | None = None, owner: str = "",
# `timeout` is overridable so short-lived callers (e.g. the service-health
# probe) can impose a tighter budget than the default IMAP timeout.
cfg = _get_email_config(account_id, owner=owner)
# Send-only (SMTP-only) account: no IMAP host means there is no inbox to
# read. Bail out with a clear, typed error instead of handing an empty
# host to imaplib — IMAP4("", 993) silently dials localhost:993 and fails
# with a confusing "[Errno 111] Connection refused" on every inbox poll.
if not cfg.get("imap_host"):
raise EmailNotConfiguredError(
f"IMAP is not configured for account {cfg.get('account_name') or 'default'!r}"
)
# Connection mode:
# STARTTLS on → plain + upgrade
# STARTTLS off + port 993 → implicit SSL (IMAPS)
@@ -825,12 +961,19 @@ def _imap_connect(account_id: str | None = None, owner: str = "",
timeout=timeout,
)
try:
conn.login(cfg["imap_user"], cfg["imap_password"])
if cfg.get("oauth_provider") == "google":
token = _get_valid_google_token(cfg.get("account_id"), cfg)
if not token:
raise RuntimeError("Google OAuth token unavailable — reconnect the account in Settings → Integrations")
conn.authenticate("XOAUTH2", lambda x: _xoauth2_bytes(cfg["imap_user"], token))
else:
conn.login(cfg["imap_user"], cfg["imap_password"])
except Exception:
# A failed AUTHENTICATE (e.g. an Office 365 app password on an
# MFA-enabled tenant, #3174) otherwise orphans the already-connected
# socket; close it before propagating so a misconfigured account
# can't leak one descriptor per retry / background poller pass.
# MFA-enabled tenant, #3174, or an expired/revoked OAuth token)
# otherwise orphans the already-connected socket; close it before
# propagating so a misconfigured account can't leak one descriptor
# per retry / background poller pass.
try:
conn.shutdown()
except Exception:
@@ -1109,22 +1252,30 @@ def _list_attachments_from_msg(msg):
return attachments
idx = 0
for part in msg.walk():
if part.is_multipart():
continue
cd = str(part.get("Content-Disposition", ""))
ct = part.get_content_type()
is_attached_email = ct == "message/rfc822" and ("attachment" in cd.lower() or part.get_filename())
if part.is_multipart() and not is_attached_email:
continue
# Skip text/html body parts (only consider real attachments)
if ct in ("text/plain", "text/html") and "attachment" not in cd:
continue
filename = part.get_filename()
if filename:
filename = _decode_header(filename)
if ct == "message/rfc822" and not re.search(r"\.[A-Za-z0-9]{1,8}$", filename):
filename = f"{filename}.eml"
else:
# Inline images, etc. - generate a name
ext = ct.split("/")[-1] if "/" in ct else "bin"
ext = "eml" if ct == "message/rfc822" else (ct.split("/")[-1] if "/" in ct else "bin")
filename = f"attachment_{idx}.{ext}"
payload = part.get_payload(decode=True)
size = len(payload) if payload else 0
if payload is None and ct == "message/rfc822":
try:
payload = part.as_bytes()
except Exception:
payload = b""
size = len(payload) if payload is not None else 0
attachments.append({
"index": idx,
"filename": filename,
@@ -1136,29 +1287,58 @@ def _list_attachments_from_msg(msg):
return attachments
def _is_likely_signature_image_attachment(att: dict) -> bool:
"""Match the reader's inline signature/logo image filter."""
filename = str((att or {}).get("filename") or "").lower()
if not re.search(r"\.(png|jpe?g|gif|bmp|svg|webp)$", filename):
return False
size = int((att or {}).get("size") or 0)
if re.search(r"^image\d{3,}\.(png|jpe?g|gif)$", filename):
return True
if re.search(r"^(signature|logo|sig|footer|banner)[-_\d]*\.(png|jpe?g|gif|svg)$", filename):
return True
return 0 < size < 30 * 1024
def _has_visible_attachments(msg) -> bool:
"""Return True only for attachments the reader will render as chips."""
return any(
not _is_likely_signature_image_attachment(att)
for att in _list_attachments_from_msg(msg)
)
def _extract_attachment_to_disk(msg, index, target_dir):
"""Extract a specific attachment to disk and return the file path."""
if not msg.is_multipart():
return None
idx = 0
for part in msg.walk():
if part.is_multipart():
continue
cd = str(part.get("Content-Disposition", ""))
ct = part.get_content_type()
is_attached_email = ct == "message/rfc822" and ("attachment" in cd.lower() or part.get_filename())
if part.is_multipart() and not is_attached_email:
continue
if ct in ("text/plain", "text/html") and "attachment" not in cd:
continue
if idx == index:
filename = part.get_filename()
if filename:
filename = _decode_header(filename)
if ct == "message/rfc822" and not re.search(r"\.[A-Za-z0-9]{1,8}$", filename):
filename = f"{filename}.eml"
else:
ext = ct.split("/")[-1] if "/" in ct else "bin"
ext = "eml" if ct == "message/rfc822" else (ct.split("/")[-1] if "/" in ct else "bin")
filename = f"attachment_{idx}.{ext}"
# Sanitize
safe_name = re.sub(r"[^\w\s\-.]", "_", filename).strip()
payload = part.get_payload(decode=True)
if not payload:
if payload is None and ct == "message/rfc822":
try:
payload = part.as_bytes()
except Exception:
payload = b""
if payload is None:
return None
target_dir.mkdir(parents=True, exist_ok=True)
filepath = target_dir / safe_name
routes/email_pollers.py
+29 -15
View File
@@ -44,6 +44,17 @@ from routes.email_helpers import (
logger = logging.getLogger(__name__)
# Recovers a `[{"action": ...}, ...]` JSON array from raw LLM output when the
# fenced-block strip leaves nothing usable. Runs on model output influenced by
# untrusted email bodies, so it must not backtrack: the object content class is
# `[^{}]` (brace-delimited, greedy) rather than the old `[^[\]]*?` lazy runs,
# which exploded exponentially on inputs like `[{"action"},{` + `}},{{` * N
# (CodeQL py/redos #198).
_CAL_ACTION_ARRAY_RE = re.compile(
r'\[\s*\{[^{}]*"action"[^{}]*\}\s*(?:,\s*\{[^{}]*\}\s*)*\]',
re.DOTALL,
)
def _owner_for_email_account(account_id: str | None) -> str:
if not account_id:
@@ -558,7 +569,7 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
cal_extract = _strip_think(_raw_original)
cal_extract = re.sub(r"^```(?:json)?\s*|\s*```$", "", cal_extract, flags=re.MULTILINE).strip()
if not cal_extract and _raw_original:
matches = list(re.finditer(r'\[\s*\{[^[\]]*?"action"[^[\]]*?\}\s*(?:,\s*\{[^[\]]*?\}\s*)*\]', _raw_original, re.DOTALL))
matches = list(_CAL_ACTION_ARRAY_RE.finditer(_raw_original))
if matches:
cal_extract = matches[-1].group()
logger.info(f"[cal-extract] uid={uid.decode() if isinstance(uid, bytes) else uid} folder={_folder} subj={subject[:50]!r} raw_len={len(cal_extract)} orig_len={len(_raw_original)} raw={cal_extract[:800]!r}")
@@ -683,20 +694,23 @@ async def _auto_summarize_pass_single(days_back: int = 1, account_id: str | None
logger.warning(f"[cal-extract] JSON parse failed: {je} on raw={cal_extract[:200]!r}")
except Exception as e:
logger.warning(f"[cal-extract] Meeting extraction LLM call failed for uid={uid}: {e}")
# Record we processed this email so we don't re-LLM next run
try:
_cc = _sql3.connect(SCHEDULED_DB)
_cc.execute(
"INSERT OR REPLACE INTO email_calendar_extractions "
"(message_id, owner, uid, events_created, created_at) VALUES (?, ?, ?, ?, ?)",
(message_id, account_owner or "", uid.decode() if isinstance(uid, bytes) else str(uid),
_cal_run_count, datetime.utcnow().isoformat())
)
_cc.commit()
_cc.close()
_cal_existing.add(message_id)
except Exception as ce:
logger.debug(f"Could not cache calendar extraction: {ce}")
else:
# Record we processed this email so we don't re-LLM next run.
# Only mark as processed on success ? transient LLM failures
# are retried on the next poll run (matches summary/reply pattern).
try:
_cc = _sql3.connect(SCHEDULED_DB)
_cc.execute(
"INSERT OR REPLACE INTO email_calendar_extractions "
"(message_id, owner, uid, events_created, created_at) VALUES (?, ?, ?, ?, ?)",
(message_id, account_owner or "", uid.decode() if isinstance(uid, bytes) else str(uid),
_cal_run_count, datetime.utcnow().isoformat())
)
_cc.commit()
_cc.close()
_cal_existing.add(message_id)
except Exception as ce:
logger.debug(f"Could not cache calendar extraction: {ce}")
if need_urgent:
try:
routes/email_routes.py
+537 -60
View File
@@ -13,7 +13,9 @@ handlers need. The split is mechanical — no behavior change.
"""
import asyncio
import os
import sqlite3 as _sql3
import time
import email as email_mod
import email.header
import email.utils
@@ -43,8 +45,10 @@ from routes.email_helpers import (
_load_settings, _save_settings, _get_email_config,
_send_smtp_message, _smtp_security_mode,
_IMAP_TIMEOUT_SECONDS, _open_imap_connection,
make_oauth_state, verify_oauth_state,
EmailNotConfiguredError,
_imap_connect, _imap, _decode_header, _detect_sent_folder, _detect_drafts_folder,
_extract_attachment_text, _list_attachments_from_msg,
_extract_attachment_text, _list_attachments_from_msg, _has_visible_attachments, _is_likely_signature_image_attachment,
_extract_attachment_to_disk, _extract_html, _extract_text,
_fetch_sender_thread_context, _pre_retrieve_context,
_EMAIL_REPLY_SYS_PROMPT_BASE, _POOL_HOOKS,
@@ -58,6 +62,22 @@ from routes.email_pollers import _start_poller
logger = logging.getLogger(__name__)
ODYSSEUS_MAIL_ORIGIN = "odysseus-ui"
EMAIL_READ_ATTACHMENT_VERSION = 2
def _coerce_port(value, default):
"""Coerce a user-supplied port to int.
Returns ``(port, error)``. A missing or blank value yields ``default``; a
non-numeric value yields ``(None, message)`` so callers can return a clean
error instead of letting ``int()`` raise and surface as an HTTP 500.
"""
if value in (None, ""):
return default, None
try:
return int(value), None
except (TypeError, ValueError):
return None, f"Invalid port {value!r}; must be a whole number"
def _email_tag_owner_aliases(account_id: str | None, owner: str = "") -> list[str]:
@@ -76,15 +96,16 @@ def _email_tag_owner_aliases(account_id: str | None, owner: str = "") -> list[st
cfg.get("smtp_user") or "",
cfg.get("from_address") or "",
])
except Exception:
except Exception as _e:
logger.warning("Failed to resolve email account alias", exc_info=_e)
resolved_account_id = None
row = db.get(_EA, resolved_account_id) if resolved_account_id else None
if row:
aliases.extend([row.owner or "", row.imap_user or "", row.from_address or ""])
finally:
db.close()
except Exception:
pass
except Exception as _e:
logger.warning("Failed to load email aliases", exc_info=_e)
out = []
for a in aliases:
a = (a or "").strip()
@@ -244,6 +265,21 @@ def _imap_uid_fetch(conn, uid_set: str | bytes, query: str):
return conn.uid("FETCH", _uid_bytes(uid_set), query)
def _imap_search_quote(value: str) -> str:
return '"' + str(value or "").replace("\\", "\\\\").replace('"', '\\"') + '"'
def _message_id_chain(*values: str) -> list[str]:
seen = set()
out = []
for value in values:
for mid in re.findall(r"<[^>]+>", value or ""):
if mid not in seen:
seen.add(mid)
out.append(mid)
return out
def _uid_from_fetch_meta(meta_b: bytes) -> str:
m = re.search(rb"\bUID\s+(\d+)\b", meta_b)
return m.group(1).decode() if m else ""
@@ -285,7 +321,9 @@ def _group_uid_fetch_records(msg_data) -> list:
def _smtp_ready(cfg: dict) -> bool:
return bool(cfg.get("smtp_host") and cfg.get("smtp_user") and cfg.get("smtp_password"))
if not cfg.get("smtp_host") or not cfg.get("smtp_user"):
return False
return bool(cfg.get("smtp_password") or cfg.get("oauth_provider"))
def _resolve_send_config(account_id: str | None = None, owner: str = "") -> dict:
@@ -360,6 +398,21 @@ def _apply_odysseus_headers(msg, kind: str | None = None, ref_id: str | None = N
msg["X-Odysseus-Ref"] = re.sub(r"[^A-Za-z0-9_.:-]", "-", ref_id)[:128]
def _normalize_addr_field(field: str) -> str:
"""Strip the malformed-but-common trailing/leading commas and stray
whitespace from a To/Cc/Bcc string before it lands in the MIME header
or the SMTP envelope. Users often paste a single address with a
trailing comma (e.g. `felix@pewdiepie.com,`) and most MTAs reject the
resulting `To: felix@pewdiepie.com,` line as a syntax error. Collapse
any run of separator junk between addresses too."""
if not field:
return field
# Split on commas, drop empty tokens, rejoin with a single ', '.
parts = [p.strip() for p in field.split(",")]
parts = [p for p in parts if p]
return ", ".join(parts)
def _envelope_recipients(*fields: str) -> list:
"""Extract bare SMTP envelope addresses from one or more To/Cc/Bcc header
strings. A naive `field.split(",")` corrupts display names that contain a
@@ -977,6 +1030,11 @@ def setup_email_routes():
logger.debug(f"Bulk summary attach skipped: {_summary_err}")
return {"emails": emails, "total": total, "folder": folder, "offset": offset}
except EmailNotConfiguredError:
# Send-only (SMTP-only) account: there is no inbox to read, so the
# poll returns an empty list instead of a per-minute error. SMTP
# send is unaffected.
return {"emails": [], "total": 0, "folder": folder, "offset": offset}
except Exception as e:
logger.error(f"Failed to list emails: {e}")
detail = str(e).strip()
@@ -988,6 +1046,65 @@ def setup_email_routes():
except Exception:
pass
def _related_thread_attachments_sync(
folder: str,
account_id: str | None,
owner: str,
current_uid: str,
current_message_id: str,
in_reply_to: str,
references: str,
limit: int = 12,
) -> list[dict]:
"""Return visible attachments from referenced messages in this folder."""
wanted_ids = _message_id_chain(references, in_reply_to)
current_mid = (current_message_id or "").strip()
wanted_ids = [mid for mid in wanted_ids if mid and mid != current_mid]
if not wanted_ids:
return []
related: list[dict] = []
try:
with _imap(account_id, owner=owner) as conn:
conn.select(_q(folder), readonly=True)
# Search newest referenced messages first; cap work so opening
# a long thread stays bounded.
for mid in reversed(wanted_ids[-10:]):
if len(related) >= limit:
break
status, data = _imap_uid_search(conn, f'(HEADER Message-ID {_imap_search_quote(mid)})')
if status != "OK" or not data or not data[0]:
continue
for uid_b in reversed(data[0].split()[-3:]):
source_uid = uid_b.decode(errors="ignore")
if not source_uid or source_uid == str(current_uid):
continue
st2, msg_data = _imap_uid_fetch(conn, source_uid, "(BODY.PEEK[])")
if st2 != "OK" or not msg_data or not isinstance(msg_data[0], tuple):
continue
msg = email_mod.message_from_bytes(msg_data[0][1])
source_from = _decode_header(msg.get("From", ""))
source_subject = _decode_header(msg.get("Subject", ""))
source_date = msg.get("Date", "")
for att in _list_attachments_from_msg(msg):
if _is_likely_signature_image_attachment(att):
continue
enriched = dict(att)
enriched.update({
"source_uid": source_uid,
"source_folder": folder,
"source_message_id": (msg.get("Message-ID") or "").strip(),
"source_from": source_from,
"source_subject": source_subject,
"source_date": source_date,
})
related.append(enriched)
if len(related) >= limit:
break
except Exception as e:
logger.debug(f"related thread attachment lookup failed uid={current_uid}: {e}")
return related
@router.get("/list")
async def list_emails(
folder: str = Query("INBOX"),
@@ -1097,7 +1214,12 @@ def setup_email_routes():
account_id: str | None = Query(None),
owner: str = Depends(require_owner),
):
"""Search emails server-side via IMAP SEARCH. Matches subject, from, or body text."""
"""Search emails server-side via IMAP SEARCH. Matches subject, from, or body text.
When the caller asks for INBOX and the account has an "All Mail"
folder (Gmail does), we transparently swap to All Mail so the
search surfaces archived / labelled emails too. Plain IMAP
accounts fall back to whatever folder the caller specified."""
if not q or len(q) < 2:
return {"emails": [], "total": 0, "query": q}
# CRLF in q would terminate the IMAP command early — reject defensively.
@@ -1105,7 +1227,27 @@ def setup_email_routes():
raise HTTPException(400, "Invalid query")
try:
with _imap(account_id, owner=owner) as conn:
conn.select(_q(folder), readonly=True)
# If the user asked for INBOX, try to upgrade to All Mail —
# one folder == every email on Gmail-class servers.
effective_folder = folder
if (folder or "").upper() == "INBOX":
try:
status, folder_lines = conn.list()
if status == "OK" and folder_lines:
for raw in folder_lines:
if isinstance(raw, bytes):
raw = raw.decode("utf-8", errors="replace")
m = re.match(r"\((?P<flags>[^)]*)\)\s+\"[^\"]*\"\s+(?P<name>.+)", raw)
if not m:
continue
flags = (m.group("flags") or "").lower()
name = m.group("name").strip().strip('"')
if "\\all" in flags or "all mail" in name.lower():
effective_folder = name
break
except Exception:
pass
conn.select(_q(effective_folder), readonly=True)
# Escape backslash and quote for the IMAP-SEARCH quoted-string.
q_escaped = q.replace('\\', '\\\\').replace('"', '\\"')
@@ -1113,7 +1255,7 @@ def setup_email_routes():
status, data = _imap_uid_search(conn, search_cmd)
if status != "OK" or not data[0]:
return {"emails": [], "total": 0, "query": q}
return {"emails": [], "total": 0, "query": q, "folder": effective_folder}
uid_list = data[0].split()
total = len(uid_list)
@@ -1178,6 +1320,13 @@ def setup_email_routes():
"is_flagged": "\\Flagged" in flags,
"flags": flags,
"has_attachments": has_attachments,
# Stamp the folder so the frontend opens each
# email from the folder it actually lives in
# (the search may have run against All Mail
# even though the caller asked for INBOX),
# otherwise clicks open whatever happens to
# have the same UID in INBOX → wrong email.
"folder": effective_folder,
})
except Exception as e:
logger.warning(f"Error parsing search result {uid}: {e}")
@@ -1226,6 +1375,17 @@ def setup_email_routes():
sender_name, sender_addr = email.utils.parseaddr(sender)
parsed_date = email.utils.parsedate_to_datetime(date_str) if date_str else None
attachments = _list_attachments_from_msg(msg)
related_attachments = []
if not _has_visible_attachments(msg):
related_attachments = _related_thread_attachments_sync(
folder,
account_id,
owner,
uid,
message_id,
in_reply_to,
references,
)
if mark_seen:
# Set \Seen in a separate readwrite session so concurrent reads
@@ -1334,6 +1494,8 @@ def setup_email_routes():
"body": body,
"body_html": body_html,
"attachments": attachments,
"related_attachments": related_attachments,
"attachment_version": EMAIL_READ_ATTACHMENT_VERSION,
"cached_summary": cached_summary,
"cached_ai_reply": cached_ai_reply,
"boundaries": cached_boundaries,
@@ -1364,6 +1526,12 @@ def setup_email_routes():
"""Read email body. Cached for 30m, sync IMAP work runs in a thread."""
ck = _read_cache_key(account_id, folder, uid, owner=owner)
cached = _read_cache_get(ck)
if cached is not None:
# Older cached read responses lack the thread-attachment fallback.
# Fetch once so replies that reference prior attachments can show
# those files without waiting for cache expiry.
if cached.get("attachment_version") != EMAIL_READ_ATTACHMENT_VERSION:
cached = None
if cached is not None:
if mark_seen:
try:
@@ -1498,6 +1666,12 @@ def setup_email_routes():
return {"error": f"Attachment index {index} not found"}
from pathlib import Path as _Path
target_root = os.path.abspath(str(target_dir))
filepath_str = os.path.abspath(str(filepath))
if os.path.commonpath([target_root, filepath_str]) != target_root:
logger.warning("Rejected attachment path outside extraction dir: %s", filepath)
return {"error": "Invalid attachment path"}
filepath = _Path(filepath_str)
base = _Path(filepath).name
if base.startswith("."):
return {"error": "Invalid filename", "filename": base}
@@ -1552,6 +1726,65 @@ def setup_email_routes():
return None
doc_session_id = _resolve_doc_session()
def _create_markdown_doc(content: str, summary: str):
from src.database import SessionLocal as _SL, Document as _Doc, DocumentVersion as _DV
doc_id = str(uuid.uuid4())
ver_id = str(uuid.uuid4())
_db = _SL()
try:
_db.query(_Doc).filter(_Doc.is_active == True).update({"is_active": False})
_db.add(_Doc(
id=doc_id, session_id=doc_session_id, title=title,
language="markdown", current_content=content,
version_count=1, is_active=True,
))
_db.add(_DV(
id=ver_id, document_id=doc_id, version_number=1,
content=content, summary=summary, source="upload",
))
_db.commit()
finally:
_db.close()
_tag_doc_with_source(doc_id)
return doc_id
def _attached_email_markdown(raw_bytes: bytes):
if not raw_bytes:
return f"# Attached email: {base}\n\n_(empty email attachment)_"
try:
attached_msg = email_mod.message_from_bytes(raw_bytes)
except Exception:
logger.exception("Failed to parse attached email %s", base)
return f"# Attached email: {base}\n\nCould not parse this email attachment."
attached_subject = _decode_header(attached_msg.get("Subject", "")) or base
attached_from = _decode_header(attached_msg.get("From", ""))
attached_to = _decode_header(attached_msg.get("To", ""))
attached_cc = _decode_header(attached_msg.get("Cc", ""))
attached_date = attached_msg.get("Date", "")
attached_body = _extract_text(attached_msg).strip()
attached_atts = _list_attachments_from_msg(attached_msg)
lines = [f"# Attached email: {attached_subject}", ""]
if attached_from:
lines.append(f"**From:** {attached_from}")
if attached_to:
lines.append(f"**To:** {attached_to}")
if attached_cc:
lines.append(f"**Cc:** {attached_cc}")
if attached_date:
lines.append(f"**Date:** {attached_date}")
lines.extend(["", "## Body", "", attached_body or "_(no readable body)_"])
if attached_atts:
lines.extend(["", "## Attachments", ""])
for att in attached_atts:
size = int(att.get("size") or 0)
size_label = f"{size} B" if size < 1024 else f"{round(size / 1024)} KB"
name = att.get("filename") or f"attachment_{att.get('index', '')}"
ctype = att.get("content_type") or "application/octet-stream"
lines.append(f"- {name} ({ctype}, {size_label})")
return "\n".join(lines).strip()
# ── PDF path (existing) ────────────────────────────────────
if ext == ".pdf":
import shutil as _shutil
@@ -1598,6 +1831,39 @@ def setup_email_routes():
_tag_doc_with_source(doc_id)
return {"doc_id": doc_id, "filename": filepath.name}
# ── Attached email (.eml / message/rfc822) ────────────────
if ext == ".eml":
def _attachment_bytes_from_msg():
if not msg.is_multipart():
return b""
idx = 0
for part in msg.walk():
cd = str(part.get("Content-Disposition", ""))
ct = part.get_content_type()
is_attached_email = ct == "message/rfc822" and ("attachment" in cd.lower() or part.get_filename())
if part.is_multipart() and not is_attached_email:
continue
if ct in ("text/plain", "text/html") and "attachment" not in cd:
continue
if idx == index:
payload = part.get_payload(decode=True)
if payload is None and ct == "message/rfc822":
try:
payload = part.as_bytes()
except Exception:
payload = b""
return payload or b""
idx += 1
return b""
try:
content = _attached_email_markdown(_attachment_bytes_from_msg())
except Exception:
logger.exception("Failed to read email attachment %s", base)
return {"error": "Failed to read email attachment", "filename": base}
doc_id = _create_markdown_doc(content, "Imported attached email")
return {"doc_id": doc_id, "filename": filepath.name}
# ── DOCX path: extract text → markdown document ───────────
if ext == ".docx":
try:
@@ -1635,25 +1901,7 @@ def setup_email_routes():
lines.append("")
content = "\n".join(lines).strip() or f"_(empty {base})_"
from src.database import SessionLocal as _SL, Document as _Doc, DocumentVersion as _DV
doc_id = str(uuid.uuid4())
ver_id = str(uuid.uuid4())
_db = _SL()
try:
_db.query(_Doc).filter(_Doc.is_active == True).update({"is_active": False})
_db.add(_Doc(
id=doc_id, session_id=doc_session_id, title=title,
language="markdown", current_content=content,
version_count=1, is_active=True,
))
_db.add(_DV(
id=ver_id, document_id=doc_id, version_number=1,
content=content, summary="Imported from DOCX", source="upload",
))
_db.commit()
finally:
_db.close()
_tag_doc_with_source(doc_id)
doc_id = _create_markdown_doc(content, "Imported from DOCX")
return {"doc_id": doc_id, "filename": filepath.name}
# ── Plain text / markdown ────────────────────────────────
@@ -1662,25 +1910,7 @@ def setup_email_routes():
content = filepath.read_text(encoding="utf-8", errors="replace")
except Exception as e:
return {"error": f"Failed to read text file: {e}", "filename": base}
from src.database import SessionLocal as _SL, Document as _Doc, DocumentVersion as _DV
doc_id = str(uuid.uuid4())
ver_id = str(uuid.uuid4())
_db = _SL()
try:
_db.query(_Doc).filter(_Doc.is_active == True).update({"is_active": False})
_db.add(_Doc(
id=doc_id, session_id=doc_session_id, title=title,
language="markdown", current_content=content,
version_count=1, is_active=True,
))
_db.add(_DV(
id=ver_id, document_id=doc_id, version_number=1,
content=content, summary="Imported from email attachment", source="upload",
))
_db.commit()
finally:
_db.close()
_tag_doc_with_source(doc_id)
doc_id = _create_markdown_doc(content, "Imported from email attachment")
return {"doc_id": doc_id, "filename": filepath.name}
return {"error": f"Unsupported attachment type: {ext}", "filename": base}
@@ -1724,6 +1954,22 @@ def setup_email_routes():
logger.error(f"Failed to mark unread {uid}: {e}")
return {"success": False, "error": "Mail operation failed"}
@router.post("/flag/{uid}")
async def flag_email(uid: str, folder: str = Query("INBOX"), account_id: str | None = Query(None),
on: bool = Query(True), owner: str = Depends(require_owner)):
"""Toggle the \\Flagged flag (a.k.a. favorite / star) on an email.
Pass `on=true` to favorite, `on=false` to unfavorite."""
try:
with _imap(account_id, owner=owner) as conn:
conn.select(_q(folder))
if not _store_email_flag(conn, uid, "\\Flagged", add=bool(on)):
return {"success": False, "error": "Email not found"}
_invalidate_list_cache(account_id, folder)
return {"success": True, "flagged": bool(on)}
except Exception as e:
logger.error(f"Failed to flag {uid}: {e}")
return {"success": False, "error": "Mail operation failed"}
@router.post("/mark-read/{uid}")
async def mark_read(uid: str, folder: str = Query("INBOX"), account_id: str | None = Query(None), owner: str = Depends(require_owner)):
"""Mark an email as read (set \\Seen flag)."""
@@ -1973,7 +2219,10 @@ def setup_email_routes():
outer = MIMEMultipart("alternative")
body_container = outer
outer["From"] = cfg["from_address"]
to = _normalize_addr_field(to or "")
cc = _normalize_addr_field(cc or "")
bcc = _normalize_addr_field(bcc or "")
outer["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
outer["To"] = to
if cc:
outer["Cc"] = cc
@@ -2104,6 +2353,77 @@ def setup_email_routes():
logger.error(f"cancel_scheduled {sid!r} failed: {e}")
return {"success": False, "error": "Mail operation failed"}
# ── Agent send-confirm: list/approve/cancel ──────────────────────────
# When `agent_email_confirm` is on, the MCP send_email tool drops the
# composed email into scheduled_emails with status='agent_draft' (a
# far-future send_at so the poller never picks it up). These endpoints
# let the chat UI surface them for the user and either approve (flip
# to status='pending' with send_at=now so the poller delivers it) or
# cancel (status='cancelled').
@router.get("/pending")
async def list_pending_agent_drafts(owner: str = Depends(require_owner)):
import sqlite3
try:
conn = sqlite3.connect(SCHEDULED_DB)
conn.row_factory = sqlite3.Row
rows = conn.execute(
"""SELECT id, to_addr, subject, body, created_at, account_id
FROM scheduled_emails
WHERE status = 'agent_draft' AND owner = ?
ORDER BY created_at DESC""",
(owner or "",),
).fetchall()
conn.close()
return {"pending": [dict(r) for r in rows]}
except Exception as e:
logger.error(f"list_pending_agent_drafts failed: {e}")
return {"pending": [], "error": "Mail operation failed"}
@router.post("/pending/{sid}/approve")
async def approve_agent_draft(sid: str, owner: str = Depends(require_owner)):
"""Approve a draft staged by the agent: flip status → pending and
backdate send_at so the scheduled-send poller picks it up
immediately."""
import sqlite3
try:
conn = sqlite3.connect(SCHEDULED_DB)
cur = conn.execute(
"""UPDATE scheduled_emails
SET status = 'pending', send_at = ?
WHERE id = ? AND status = 'agent_draft' AND owner = ?""",
(datetime.utcnow().isoformat(), sid, owner or ""),
)
conn.commit()
affected = cur.rowcount
conn.close()
if not affected:
return {"success": False, "error": "Draft not found or already handled"}
return {"success": True}
except Exception as e:
logger.error(f"approve_agent_draft {sid!r} failed: {e}")
return {"success": False, "error": "Mail operation failed"}
@router.delete("/pending/{sid}")
async def cancel_agent_draft(sid: str, owner: str = Depends(require_owner)):
"""Discard a draft the agent staged for approval."""
import sqlite3
try:
conn = sqlite3.connect(SCHEDULED_DB)
cur = conn.execute(
"""UPDATE scheduled_emails SET status = 'cancelled'
WHERE id = ? AND status = 'agent_draft' AND owner = ?""",
(sid, owner or ""),
)
conn.commit()
affected = cur.rowcount
conn.close()
if not affected:
return {"success": False, "error": "Draft not found or already handled"}
return {"success": True}
except Exception as e:
logger.error(f"cancel_agent_draft {sid!r} failed: {e}")
return {"success": False, "error": "Mail operation failed"}
@router.get("/resolve-contact")
async def resolve_contact(name: str = Query(..., description="Name to search for"), owner: str = Depends(require_owner)):
"""Search Sent folder for a contact by name. Returns matching email addresses."""
@@ -2164,6 +2484,7 @@ def setup_email_routes():
try:
cfg = _resolve_send_config(req.account_id, owner=owner)
except Exception as e:
logger.warning(f"No SMTP-capable account resolved: {e}")
return {"success": False, "error": str(e) or "No SMTP-capable email account configured"}
# Use 'mixed' if we have attachments, 'alternative' otherwise
@@ -2176,7 +2497,10 @@ def setup_email_routes():
outer = MIMEMultipart("alternative")
body_container = outer
outer["From"] = cfg["from_address"]
req.to = _normalize_addr_field(req.to or "")
req.cc = _normalize_addr_field(req.cc or "")
req.bcc = _normalize_addr_field(req.bcc or "")
outer["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
outer["To"] = req.to
if req.cc:
outer["Cc"] = req.cc
@@ -2227,6 +2551,10 @@ def setup_email_routes():
_account_id = cfg.get("account_id") or req.account_id # capture for the IMAP append in the closure
_in_reply_to = (req.in_reply_to or "").strip()
_oauth_provider = cfg.get("oauth_provider") or ""
_oauth_access_token = cfg.get("oauth_access_token") or ""
_oauth_refresh_token = cfg.get("oauth_refresh_token") or ""
_oauth_token_expiry = cfg.get("oauth_token_expiry") or ""
def _deliver():
try:
@@ -2237,6 +2565,11 @@ def setup_email_routes():
"smtp_security": _smtp_security,
"smtp_user": _smtp_user,
"smtp_password": _smtp_pw,
"account_id": _account_id,
"oauth_provider": _oauth_provider,
"oauth_access_token": _oauth_access_token,
"oauth_refresh_token": _oauth_refresh_token,
"oauth_token_expiry": _oauth_token_expiry,
},
_from,
_recipients,
@@ -2349,7 +2682,7 @@ def setup_email_routes():
msg.attach(MIMEText(_draft_html, "html", "utf-8"))
else:
msg = MIMEText(req.body, "plain", "utf-8")
msg["From"] = cfg["from_address"]
msg["From"] = email.utils.formataddr((cfg.get("display_name") or "", cfg["from_address"]))
msg["To"] = req.to
if req.cc:
msg["Cc"] = req.cc
@@ -2617,11 +2950,15 @@ def setup_email_routes():
source_uid = (data.get("uid") or "").strip()
source_folder = (data.get("folder") or "INBOX").strip()
fast_reply = bool(data.get("fast", False))
user_hint = (data.get("user_hint") or "").strip()
if not original_body:
return {"success": False, "error": "No email body provided"}
if message_id:
# Skip cache lookup when the caller supplied a user_hint — the
# cached generic reply doesn't reflect the instructions and
# would silently override them.
if message_id and not user_hint:
try:
_c = _sql3.connect(SCHEDULED_DB)
owner_clause, owner_params = _email_cache_owner_clause(owner)
@@ -2761,8 +3098,13 @@ def setup_email_routes():
user_msg = (
f"Recipient: {to}\nSubject: {subject}\n\n"
f"Original email and any current draft:\n{original_body[:6000]}\n\n"
f"Draft a reply. Return only the reply body text."
)
if user_hint:
user_msg += (
f"User's instructions for THIS reply (follow these — they override "
f"defaults like length/tone):\n{user_hint[:2000]}\n\n"
)
user_msg += "Draft a reply. Return only the reply body text."
# Build a candidate chain so a stale session-stored API key
# (the most common cause of "authentication failed" here)
@@ -2992,6 +3334,8 @@ def setup_email_routes():
"from_address": r.from_address or "",
"has_imap_password": bool(r.imap_password),
"has_smtp_password": bool(r.smtp_password),
"oauth_provider": r.oauth_provider or "",
"display_name": r.display_name or "",
})
return {"accounts": out}
finally:
@@ -3006,6 +3350,12 @@ def setup_email_routes():
name = (data.get("name") or "").strip()
if not name:
return {"ok": False, "error": "name required"}
imap_port, port_err = _coerce_port(data.get("imap_port"), 993)
if port_err:
return {"ok": False, "error": port_err}
smtp_port, port_err = _coerce_port(data.get("smtp_port"), 465)
if port_err:
return {"ok": False, "error": port_err}
db = SessionLocal()
try:
row = EmailAccount(
@@ -3014,16 +3364,17 @@ def setup_email_routes():
is_default=bool(data.get("is_default", False)),
enabled=bool(data.get("enabled", True)),
imap_host=(data.get("imap_host") or "").strip(),
imap_port=int(data.get("imap_port") or 993),
imap_port=imap_port,
imap_user=(data.get("imap_user") or "").strip(),
imap_password=_enc(data.get("imap_password") or ""),
imap_starttls=bool(data.get("imap_starttls", True)),
smtp_host=(data.get("smtp_host") or "").strip(),
smtp_port=int(data.get("smtp_port") or 465),
smtp_security=_smtp_security_mode({"smtp_security": data.get("smtp_security"), "smtp_port": data.get("smtp_port") or 465}),
smtp_port=smtp_port,
smtp_security=_smtp_security_mode({"smtp_security": data.get("smtp_security"), "smtp_port": smtp_port}),
smtp_user=(data.get("smtp_user") or "").strip(),
smtp_password=_enc(data.get("smtp_password") or ""),
from_address=(data.get("from_address") or "").strip(),
display_name=(data.get("display_name") or "").strip(),
# SECURITY: stamp the creator so all subsequent reads / mutations
# can filter by user. Without this every new account leaks to
# every other user.
@@ -3058,12 +3409,15 @@ def setup_email_routes():
if not row:
return {"ok": False, "error": "Account not found"}
# Simple fields
for key in ("name", "imap_host", "imap_user", "smtp_host", "smtp_user", "from_address"):
for key in ("name", "imap_host", "imap_user", "smtp_host", "smtp_user", "from_address", "display_name"):
if key in data:
setattr(row, key, (data[key] or "").strip())
for key in ("imap_port", "smtp_port"):
if data.get(key) not in (None, ""):
setattr(row, key, int(data[key]))
port, port_err = _coerce_port(data.get(key), None)
if port_err:
return {"ok": False, "error": port_err}
setattr(row, key, port)
if "smtp_security" in data:
row.smtp_security = _smtp_security_mode({"smtp_security": data.get("smtp_security"), "smtp_port": data.get("smtp_port") or row.smtp_port})
for key in ("imap_starttls", "enabled"):
@@ -3167,12 +3521,14 @@ def setup_email_routes():
smtp_result = None
imap_host = (body.get("imap_host") or "").strip()
imap_port = int(body.get("imap_port") or 993)
imap_port, imap_port_err = _coerce_port(body.get("imap_port"), 993)
imap_user = (body.get("imap_user") or "").strip()
imap_pass = body.get("imap_password") or ""
imap_starttls = bool(body.get("imap_starttls"))
if not (imap_host and imap_user and imap_pass):
if imap_port_err:
imap_result = {"ok": False, "error": imap_port_err}
elif not (imap_host and imap_user and imap_pass):
imap_result = {"ok": False, "error": "Need IMAP host, username, and password"}
else:
# Connection mode resolution:
@@ -3199,8 +3555,10 @@ def setup_email_routes():
imap_result = {"ok": False, "error": _friendly_email_auth_error("IMAP", imap_host, e)}
smtp_host = (body.get("smtp_host") or "").strip()
if smtp_host:
smtp_port = int(body.get("smtp_port") or 465)
smtp_port, smtp_port_err = _coerce_port(body.get("smtp_port"), 465)
if smtp_host and smtp_port_err:
smtp_result = {"ok": False, "error": smtp_port_err}
elif smtp_host:
smtp_security = _smtp_security_mode({"smtp_security": body.get("smtp_security"), "smtp_port": smtp_port})
smtp_user = (body.get("smtp_user") or imap_user).strip()
smtp_pass = body.get("smtp_password") or imap_pass
@@ -3247,4 +3605,123 @@ def setup_email_routes():
finally:
db.close()
# ── Google OAuth2 routes ──
@router.get("/oauth/google/authorize")
async def google_oauth_authorize(account_id: str = Query(...), request: Request = None, owner: str = Depends(require_user)):
import urllib.parse
_assert_owns_account(account_id, owner)
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
if not client_id:
raise HTTPException(400, "GOOGLE_OAUTH_CLIENT_ID not set — add it to .env")
redirect_uri = (
os.environ.get("GOOGLE_OAUTH_REDIRECT_URI")
or f"http://{request.headers.get('host', 'localhost:7000')}/api/email/oauth/google/callback"
)
state = make_oauth_state(account_id, owner)
params = urllib.parse.urlencode({
"client_id": client_id,
"redirect_uri": redirect_uri,
"response_type": "code",
"scope": "https://mail.google.com/ email",
"access_type": "offline",
"prompt": "consent",
"state": state,
})
from fastapi.responses import RedirectResponse as _RR
return _RR(f"https://accounts.google.com/o/oauth2/v2/auth?{params}")
@router.get("/oauth/google/callback")
async def google_oauth_callback(
code: str = Query(None),
state: str = Query(None),
error: str = Query(None),
request: Request = None,
):
import urllib.parse
from fastapi.responses import RedirectResponse as _RR
if error:
return _RR("/?section=integrations&email_oauth_error=google_error")
if not code or not state:
return _RR("/?section=integrations&email_oauth_error=missing_code")
state_data = verify_oauth_state(state)
if not state_data:
return _RR("/?section=integrations&email_oauth_error=invalid_state")
account_id = state_data.get("a", "")
owner = state_data.get("o", "")
client_id = os.environ.get("GOOGLE_OAUTH_CLIENT_ID", "")
client_secret = os.environ.get("GOOGLE_OAUTH_CLIENT_SECRET", "")
redirect_uri = (
os.environ.get("GOOGLE_OAUTH_REDIRECT_URI")
or f"http://{request.headers.get('host', 'localhost:7000')}/api/email/oauth/google/callback"
)
import httpx as _httpx
try:
resp = _httpx.post("https://oauth2.googleapis.com/token", data={
"code": code,
"client_id": client_id,
"client_secret": client_secret,
"redirect_uri": redirect_uri,
"grant_type": "authorization_code",
}, timeout=10)
resp.raise_for_status()
data = resp.json()
except Exception:
logger.warning("Google token exchange failed")
return _RR("/?section=integrations&email_oauth_error=token_exchange_failed")
access_token = data.get("access_token", "")
refresh_token = data.get("refresh_token", "")
expiry = str(int(time.time()) + data.get("expires_in", 3600))
# Fetch the email address from userinfo so we can auto-fill imap_user.
email_addr = ""
display_name = ""
try:
ui = _httpx.get("https://www.googleapis.com/oauth2/v1/userinfo",
headers={"Authorization": f"Bearer {access_token}"}, timeout=10)
if ui.is_success:
ui_data = ui.json()
email_addr = ui_data.get("email", "")
display_name = ui_data.get("name", "")
except Exception:
pass
from core.database import SessionLocal, EmailAccount
from src.secret_storage import encrypt as _enc
db = SessionLocal()
try:
row = db.query(EmailAccount).filter(EmailAccount.id == account_id).first()
if not row:
return _RR("/?section=integrations&email_oauth_error=account_not_found")
# SECURITY: verify the account belongs to the initiating user.
if owner and row.owner and row.owner != owner:
logger.warning("OAuth callback owner mismatch — rejecting token write")
return _RR("/?section=integrations&email_oauth_error=ownership_error")
row.oauth_provider = "google"
row.oauth_access_token = _enc(access_token)
if refresh_token:
row.oauth_refresh_token = _enc(refresh_token)
row.oauth_token_expiry = expiry
# Auto-fill Google IMAP/SMTP settings if not already configured.
if not row.imap_host:
row.imap_host = "imap.gmail.com"
row.imap_port = 993
row.imap_starttls = False
if not row.smtp_host:
row.smtp_host = "smtp.gmail.com"
row.smtp_port = 587
if email_addr:
if not row.imap_user:
row.imap_user = email_addr
if not row.smtp_user:
row.smtp_user = email_addr
if not row.from_address:
row.from_address = email_addr
if not row.name or row.name == row.id:
row.name = email_addr
if display_name and not row.display_name:
row.display_name = display_name
db.commit()
finally:
db.close()
return _RR("/?section=integrations&email_oauth_success=1")
return router
routes/embedding_routes.py
+1
View File
@@ -9,6 +9,7 @@ from pathlib import Path
from fastapi import APIRouter, HTTPException, Form, Depends
from core.constants import EMBEDDING_ENDPOINT_FILE, FASTEMBED_CACHE_DIR
from core.middleware import require_admin
from src.runtime_paths import get_app_root
logger = logging.getLogger(__name__)
routes/gallery/__init__.py
+6
View File
@@ -0,0 +1,6 @@
"""Gallery route domain package (slice 2a, #4082/#4071).
Contains gallery_routes.py and gallery_helpers.py, migrated from the flat
routes/ directory. Backward-compat shims at routes/gallery_routes.py and
routes/gallery_helpers.py re-export from here.
"""
routes/gallery/gallery_helpers.py
+144
View File
@@ -0,0 +1,144 @@
"""gallery_helpers.py — extracted helpers, models, and small utilities.
Imported by gallery_routes.py."""
"""Gallery routes — browsable library for photos and AI-generated images."""
import logging
from datetime import datetime
from typing import Dict, Any, Optional
from pydantic import BaseModel
from core.database import GalleryImage
from src.auth_helpers import _auth_disabled
logger = logging.getLogger(__name__)
# ---- Request schemas ----
class GalleryPatch(BaseModel):
tags: Optional[str] = None
favorite: Optional[bool] = None
album_id: Optional[str] = None
# ---- EXIF extraction ----
def _extract_exif(content: bytes) -> dict:
"""Extract EXIF metadata from image bytes. Returns dict of fields."""
result = {"width": None, "height": None}
try:
from PIL import Image
from io import BytesIO
img = Image.open(BytesIO(content))
# Read the raw EXIF before any transpose: exif_transpose strips the
# orientation tag and with it the parsed EXIF view.
exif = img._getexif() if hasattr(img, '_getexif') else None
# Record DISPLAY dimensions (EXIF-rotated), matching upload_handler.
# A phone photo with Orientation 6/8 is stored landscape but shown
# portrait, so the raw width/height swap the aspect ratio.
try:
from PIL import ImageOps
img = ImageOps.exif_transpose(img) or img
except Exception:
pass
result["width"] = img.width
result["height"] = img.height
if not exif:
return result
# EXIF tag IDs
# 271=Make, 272=Model, 306=DateTime, 36867=DateTimeOriginal
# 34853=GPSInfo
result["camera_make"] = str(exif.get(271, "")).strip() or None
result["camera_model"] = str(exif.get(272, "")).strip() or None
# Date taken
for tag_id in (36867, 36868, 306): # DateTimeOriginal, DateTimeDigitized, DateTime
raw = exif.get(tag_id)
if raw:
try:
result["taken_at"] = datetime.strptime(str(raw).strip(), "%Y:%m:%d %H:%M:%S")
break
except (ValueError, TypeError):
pass
# GPS
gps_info = exif.get(34853)
if gps_info and isinstance(gps_info, dict):
try:
def _to_deg(vals):
d, m, s = [float(v) for v in vals]
return d + m / 60 + s / 3600
if 2 in gps_info and 4 in gps_info:
lat = _to_deg(gps_info[2])
lng = _to_deg(gps_info[4])
if gps_info.get(1) == 'S': lat = -lat
if gps_info.get(3) == 'W': lng = -lng
result["gps_lat"] = f"{lat:.6f}"
result["gps_lng"] = f"{lng:.6f}"
except Exception:
pass
except Exception as e:
# User-visible failure (photo loses metadata): surface at WARNING
# and record on the result so the upload endpoint can pass it back.
logger.warning(f"EXIF extraction failed: {e}")
result["exif_error"] = str(e)
return result
# ---- Helpers ----
def _image_to_dict(img: GalleryImage, session_name: str = None) -> Dict[str, Any]:
return {
"id": img.id,
"filename": img.filename,
"url": f"/api/generated-image/{img.filename}",
"prompt": img.prompt,
"model": img.model,
"size": img.size,
"quality": img.quality,
"tags": img.tags or "",
"ai_tags": img.ai_tags or "",
"user_tags": img.tags or "",
"session_id": img.session_id,
"session_name": session_name,
"album_id": img.album_id,
"is_active": img.is_active,
"favorite": img.favorite or False,
"taken_at": img.taken_at.isoformat() if img.taken_at else None,
"camera": f"{img.camera_make or ''} {img.camera_model or ''}".strip() or None,
"gps": {"lat": img.gps_lat, "lng": img.gps_lng} if img.gps_lat else None,
"width": img.width,
"height": img.height,
"file_size": img.file_size,
"created_at": img.created_at.isoformat() if img.created_at else None,
"updated_at": img.updated_at.isoformat() if img.updated_at else None,
}
def _owner_filter(q, user, model_cls=GalleryImage):
"""Apply owner filtering to a gallery query.
``get_current_user`` returns None both in auth-disabled single-user mode
and when auth is enabled but no current user was resolved. Preserve the
single-user behavior, but fail closed for auth-enabled null-user states.
"""
if user is not None:
return q.filter(model_cls.owner == user)
if _auth_disabled():
return q
return q.filter(False)
def _human_size(nbytes):
for unit in ['B', 'KB', 'MB', 'GB', 'TB']:
if abs(nbytes) < 1024:
return f"{nbytes:.1f} {unit}"
nbytes /= 1024
return f"{nbytes:.1f} PB"
routes/gallery/gallery_routes.py
+1966
View File
File diff suppressed because it is too large Load Diff
routes/gallery_helpers.py
+10 -140
View File
@@ -1,144 +1,14 @@
"""gallery_helpers.py — extracted helpers, models, and small utilities.
"""Backward-compat shim — canonical location is routes/gallery/gallery_helpers.py.
Imported by gallery_routes.py."""
This module is replaced in ``sys.modules`` by the canonical module object so
that ``import routes.gallery_helpers``, ``from routes.gallery_helpers import X``,
``importlib.import_module("routes.gallery_helpers")``, and
``monkeypatch.setattr(routes.gallery_helpers, ...)`` all operate on the *same*
object. Keeps existing import paths working after slice 2a (#4082/#4071).
"""
"""Gallery routes — browsable library for photos and AI-generated images."""
import sys as _sys
import logging
from datetime import datetime
from typing import Dict, Any, Optional
from routes.gallery import gallery_helpers as _canonical # noqa: F401
from pydantic import BaseModel
from core.database import GalleryImage
from src.auth_helpers import _auth_disabled
logger = logging.getLogger(__name__)
# ---- Request schemas ----
class GalleryPatch(BaseModel):
tags: Optional[str] = None
favorite: Optional[bool] = None
album_id: Optional[str] = None
# ---- EXIF extraction ----
def _extract_exif(content: bytes) -> dict:
"""Extract EXIF metadata from image bytes. Returns dict of fields."""
result = {"width": None, "height": None}
try:
from PIL import Image
from io import BytesIO
img = Image.open(BytesIO(content))
# Read the raw EXIF before any transpose: exif_transpose strips the
# orientation tag and with it the parsed EXIF view.
exif = img._getexif() if hasattr(img, '_getexif') else None
# Record DISPLAY dimensions (EXIF-rotated), matching upload_handler.
# A phone photo with Orientation 6/8 is stored landscape but shown
# portrait, so the raw width/height swap the aspect ratio.
try:
from PIL import ImageOps
img = ImageOps.exif_transpose(img) or img
except Exception:
pass
result["width"] = img.width
result["height"] = img.height
if not exif:
return result
# EXIF tag IDs
# 271=Make, 272=Model, 306=DateTime, 36867=DateTimeOriginal
# 34853=GPSInfo
result["camera_make"] = str(exif.get(271, "")).strip() or None
result["camera_model"] = str(exif.get(272, "")).strip() or None
# Date taken
for tag_id in (36867, 36868, 306): # DateTimeOriginal, DateTimeDigitized, DateTime
raw = exif.get(tag_id)
if raw:
try:
result["taken_at"] = datetime.strptime(str(raw).strip(), "%Y:%m:%d %H:%M:%S")
break
except (ValueError, TypeError):
pass
# GPS
gps_info = exif.get(34853)
if gps_info and isinstance(gps_info, dict):
try:
def _to_deg(vals):
d, m, s = [float(v) for v in vals]
return d + m / 60 + s / 3600
if 2 in gps_info and 4 in gps_info:
lat = _to_deg(gps_info[2])
lng = _to_deg(gps_info[4])
if gps_info.get(1) == 'S': lat = -lat
if gps_info.get(3) == 'W': lng = -lng
result["gps_lat"] = f"{lat:.6f}"
result["gps_lng"] = f"{lng:.6f}"
except Exception:
pass
except Exception as e:
# User-visible failure (photo loses metadata): surface at WARNING
# and record on the result so the upload endpoint can pass it back.
logger.warning(f"EXIF extraction failed: {e}")
result["exif_error"] = str(e)
return result
# ---- Helpers ----
def _image_to_dict(img: GalleryImage, session_name: str = None) -> Dict[str, Any]:
return {
"id": img.id,
"filename": img.filename,
"url": f"/api/generated-image/{img.filename}",
"prompt": img.prompt,
"model": img.model,
"size": img.size,
"quality": img.quality,
"tags": img.tags or "",
"ai_tags": img.ai_tags or "",
"user_tags": img.tags or "",
"session_id": img.session_id,
"session_name": session_name,
"album_id": img.album_id,
"is_active": img.is_active,
"favorite": img.favorite or False,
"taken_at": img.taken_at.isoformat() if img.taken_at else None,
"camera": f"{img.camera_make or ''} {img.camera_model or ''}".strip() or None,
"gps": {"lat": img.gps_lat, "lng": img.gps_lng} if img.gps_lat else None,
"width": img.width,
"height": img.height,
"file_size": img.file_size,
"created_at": img.created_at.isoformat() if img.created_at else None,
"updated_at": img.updated_at.isoformat() if img.updated_at else None,
}
def _owner_filter(q, user, model_cls=GalleryImage):
"""Apply owner filtering to a gallery query.
``get_current_user`` returns None both in auth-disabled single-user mode
and when auth is enabled but no current user was resolved. Preserve the
single-user behavior, but fail closed for auth-enabled null-user states.
"""
if user is not None:
return q.filter(model_cls.owner == user)
if _auth_disabled():
return q
return q.filter(False)
def _human_size(nbytes):
for unit in ['B', 'KB', 'MB', 'GB', 'TB']:
if abs(nbytes) < 1024:
return f"{nbytes:.1f} {unit}"
nbytes /= 1024
return f"{nbytes:.1f} PB"
_sys.modules[__name__] = _canonical
routes/gallery_routes.py
+12 -1917
View File
File diff suppressed because it is too large Load Diff
routes/hwfit_routes.py
+112 -5
View File
@@ -1,8 +1,13 @@
import json
import os
import re
import shlex
import subprocess
from copy import deepcopy
from fastapi import APIRouter, HTTPException
from core.platform_compat import run_ssh_command
from routes._validators import validate_remote_host, validate_ssh_port
@@ -107,6 +112,73 @@ def _apply_manual_hardware(system, manual_mode="", manual_gpu_count="", manual_v
return system
def _run_model_probe(host: str, ssh_port: str, cmd: str) -> str:
try:
if host:
r = run_ssh_command(
host,
ssh_port or None,
cmd,
timeout=15,
connect_timeout=5,
strict_host_key_checking=False,
text=True,
)
else:
r = subprocess.run(["bash", "-lc", cmd], capture_output=True, text=True, timeout=15)
if r.returncode == 0:
return (r.stdout or "").strip()
except Exception:
return ""
return ""
def _inspect_model_path(model_path: str, host: str = "", ssh_port: str = "") -> dict:
"""Read lightweight metadata from a local or SSH-visible HF model folder."""
path = (model_path or "").strip()
if not path or path.startswith(("http://", "https://")):
return {}
if not (path.startswith("/") or path.startswith("~")):
return {}
qpath = shlex.quote(path)
qconfig = shlex.quote(os.path.join(path, "config.json"))
out = {}
exists = _run_model_probe(host, ssh_port, f"test -d {qpath} && printf found || printf missing")
if exists != "found":
target = host or "local container"
out["model_probe_error"] = f"Model path is not visible on {target}: {path}"
return out
raw_config = _run_model_probe(host, ssh_port, f"test -f {qconfig} && sed -n '1,240p' {qconfig}")
if raw_config:
try:
cfg = json.loads(raw_config)
except Exception:
cfg = {}
for key in ("context_length", "max_position_embeddings", "n_ctx_train", "model_max_length", "max_seq_len"):
value = cfg.get(key)
if isinstance(value, (int, float)) and value > 0:
out["model_ctx_max"] = int(value)
break
else:
out["model_probe_error"] = f"config.json not found in model path: {path}"
size_cmd = (
f"find {qpath} -type f \\( -name '*.safetensors' -o -name '*.bin' -o -name '*.gguf' \\) "
"-printf '%s\\n' 2>/dev/null | awk '{s+=$1} END {if (s>0) printf \"%.6f\", s/1073741824}'"
)
weights = _run_model_probe(host, ssh_port, size_cmd)
try:
weights_gb = float(weights)
except Exception:
weights_gb = 0.0
if weights_gb > 0:
out["model_weights_gb"] = round(weights_gb, 3)
elif "model_probe_error" not in out:
out["model_probe_error"] = f"No model weight files found in: {path}"
return out
def setup_hwfit_routes():
router = APIRouter(prefix="/api/hwfit", tags=["hwfit"])
@@ -119,7 +191,7 @@ def setup_hwfit_routes():
return detect_system(host=host, ssh_port=ssh_port, platform=platform, fresh=fresh)
@router.get("/models")
def get_models(use_case: str = "", sort: str = "score", limit: int = 50, search: str = "", host: str = "", quant: str = "", ctx: str = "", gpu_count: str = "", gpu_group: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, manual_mode: str = "", manual_gpu_count: str = "", manual_vram_gb: str = "", manual_ram_gb: str = "", manual_backend: str = "", ignore_detected_gpu: bool = False, ignore_detected_ram: bool = False, fit_only: bool = False):
def get_models(use_case: str = "", sort: str = "newest", limit: int = 50, search: str = "", host: str = "", quant: str = "", ctx: str = "", gpu_count: str = "", gpu_group: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, manual_mode: str = "", manual_gpu_count: str = "", manual_vram_gb: str = "", manual_ram_gb: str = "", manual_backend: str = "", ignore_detected_gpu: bool = False, ignore_detected_ram: bool = False, fit_only: bool = False):
"""Rank LLM models against detected hardware and return scored results.
gpu_count: override GPU count (0 = CPU only, 1-N = simulate N GPUs of the
active group). gpu_group: index into system.gpu_groups (the homogeneous
@@ -235,7 +307,7 @@ def setup_hwfit_routes():
return {"system": system, "models": results}
@router.get("/profiles")
def get_serve_profiles(model: str = "", host: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, serve_weights_gb: float = 0.0, serve_quant: str = ""):
def get_serve_profiles(model: str = "", model_path: str = "", host: str = "", ssh_port: str = "", platform: str = "", fresh: bool = False, serve_weights_gb: float = 0.0, serve_quant: str = ""):
"""Compute llama.cpp serve profiles (Quality/Balanced/Speed) for `model`
against the detected hardware on `host` (or local). Returns concrete
flags (n_gpu_layers, n_cpu_moe, cache_type, ctx) the serve UI can apply.
@@ -260,8 +332,23 @@ def setup_hwfit_routes():
# "deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct".
s = (s or "").lower().strip()
s = s.split("/")[-1] # drop org prefix
s = re.sub(r"[-_.]?gguf$", "", s) # drop trailing gguf marker
s = re.sub(r"[-_.](q\d[^/]*|iq\d[^/]*|fp8|bf16|f16|awq[^/]*|gptq[^/]*)$", "", s)
for suffix in ("-gguf", "_gguf", ".gguf", "gguf"):
if s.endswith(suffix):
s = s[: -len(suffix)]
break
cut_at = None
for idx, ch in enumerate(s):
if ch not in "-_." or idx + 1 >= len(s):
continue
suffix = s[idx + 1:]
if (
suffix in {"fp8", "bf16", "f16"}
or suffix.startswith(("awq", "gptq", "iq"))
or (suffix.startswith("q") and len(suffix) > 1 and suffix[1].isdigit())
):
cut_at = idx
if cut_at is not None:
s = s[:cut_at]
return s
m = catalog.get(model)
@@ -272,8 +359,16 @@ def setup_hwfit_routes():
if nn and (nn == want or want.endswith(nn) or nn.endswith(want)):
m = entry
break
path_meta = _inspect_model_path(model_path or model, host=host, ssh_port=ssh_port)
if m is None:
return {"system": system, "profiles": [], "error": "model not in catalog"}
return {
"system": system,
"profiles": [],
"error": "model not in catalog",
"model_ctx_max": int(path_meta.get("model_ctx_max") or 0),
"model_weights_gb": float(path_meta.get("model_weights_gb") or 0),
"model_probe_error": path_meta.get("model_probe_error") or "",
}
# Surface the model's trained context limit so the serve UI can clamp a
# user-typed context down to it (asking for ctx > n_ctx_train overflows
# and, with a quantized KV cache, can crash the GPU).
@@ -283,6 +378,16 @@ def setup_hwfit_routes():
if isinstance(v, (int, float)) and v > 0:
model_ctx_max = int(v)
break
path_ctx_max = int(path_meta.get("model_ctx_max") or 0)
if path_ctx_max > 0:
model_ctx_max = max(model_ctx_max, path_ctx_max)
model_weights_gb = float(path_meta.get("model_weights_gb") or 0)
if model_weights_gb <= 0:
for k in ("min_vram_gb", "required_gb", "size_gb", "recommended_ram_gb", "min_ram_gb"):
v = m.get(k)
if isinstance(v, (int, float)) and v > 0:
model_weights_gb = float(v)
break
return {
"system": system,
"profiles": compute_serve_profiles(
@@ -291,6 +396,8 @@ def setup_hwfit_routes():
serve_quant=(serve_quant or None),
),
"model_ctx_max": model_ctx_max,
"model_weights_gb": model_weights_gb,
"model_probe_error": path_meta.get("model_probe_error") or "",
}
@router.get("/image-models")
routes/memory_routes.py
+33 -58
View File
@@ -273,65 +273,30 @@ def setup_memory_routes(memory_manager: MemoryManager, session_manager: SessionM
async def api_audit_memories(request: Request, session: str = Form(None)):
"""Deduplicate and consolidate memories via LLM.
Uses the default model from settings, or falls back to a session's model.
Uses task/utility/default settings through the shared resolver, with
the active session as fallback when no task or utility model is set.
Returns before and after memory counts.
"""
from routes.model_routes import _load_settings, _normalize_base, build_chat_url
from core.database import ModelEndpoint
import json as _json
endpoint_url = model = None
headers = {}
# Try utility model from settings first — memory audit is a background
# task and should prefer the lighter utility model over the main chat model.
from src.task_endpoint import resolve_task_endpoint
user = _owner(request)
t_url, t_model, t_headers = resolve_task_endpoint(owner=user)
if t_url and t_model:
endpoint_url, model, headers = t_url, t_model, t_headers
else:
# Fall back to default model if no task/utility model configured
settings = _load_settings()
ep_id = settings.get("default_endpoint_id", "")
default_model = settings.get("default_model", "")
if ep_id:
db = SessionLocal()
try:
ep = db.query(ModelEndpoint).filter(
ModelEndpoint.id == ep_id, ModelEndpoint.is_enabled == True
).first()
if ep:
base = _normalize_base(ep.base_url)
endpoint_url = build_chat_url(base)
model = default_model
if not model and ep.models:
try:
models = _json.loads(ep.models) if isinstance(ep.models, str) else ep.models
if models:
model = models[0]
except Exception:
pass
if ep.api_key:
headers = {"Authorization": f"Bearer {ep.api_key}"}
finally:
db.close()
fallback_url = fallback_model = None
fallback_headers = None
if session:
try:
sess = session_manager.get_session(session)
_assert_session_owner(sess, user)
fallback_url = sess.endpoint_url
fallback_model = sess.model
fallback_headers = sess.headers
except KeyError:
pass
# Fall back to session model if no default configured
if not endpoint_url and session:
try:
sess = session_manager.get_session(session)
_assert_session_owner(sess, _owner(request))
endpoint_url = sess.endpoint_url
model = sess.model
headers = sess.headers
except KeyError:
pass
endpoint_url, model, headers = resolve_task_endpoint(
fallback_url, fallback_model, fallback_headers, owner=user
)
if not endpoint_url or not model:
raise HTTPException(400, "No default model configured — set one in Settings")
user = _owner(request)
result = await audit_memories(
memory_manager,
memory_vector,
@@ -369,18 +334,28 @@ def setup_memory_routes(memory_manager: MemoryManager, session_manager: SessionM
model = None
headers = {}
user = _owner(request)
if session:
try:
sess = session_manager.get_session(session)
_assert_session_owner(sess, _owner(request))
endpoint_url, model, headers = resolve_task_endpoint(
sess.endpoint_url, sess.model, sess.headers, owner=_owner(request)
)
_assert_session_owner(sess, user)
except KeyError:
logger.warning("Session %s not found, falling back to utility endpoint", session)
endpoint_url, model, headers = resolve_endpoint("utility", owner=_owner(request))
sess = None
except HTTPException as exc:
if exc.status_code != 404:
raise
sess = None
if sess is None:
logger.warning("Session %s not found or inaccessible, falling back to utility endpoint", session)
endpoint_url, model, headers = resolve_endpoint("utility", owner=user)
else:
endpoint_url, model, headers = resolve_task_endpoint(
sess.endpoint_url, sess.model, sess.headers, owner=user
)
else:
endpoint_url, model, headers = resolve_task_endpoint(owner=_owner(request))
endpoint_url, model, headers = resolve_task_endpoint(owner=user)
if not endpoint_url or not model:
raise HTTPException(400, "No LLM model configured. Set a default model in Settings.")
routes/model_routes.py
+167 -39
View File
@@ -5,6 +5,7 @@ import re
import uuid
import json
import hashlib
import ipaddress
import socket
import time as _time
import logging
@@ -16,6 +17,7 @@ from fastapi import APIRouter, HTTPException, Form, Query, Body, Request, Respon
from pydantic import BaseModel
from fastapi.responses import StreamingResponse
from core.database import SessionLocal, ModelEndpoint, Session as DbSession
from core.log_safety import redact_url as _redact_url_for_log
from core.middleware import require_admin
from src.llm_core import _detect_provider, _host_match, ANTHROPIC_MODELS
from src.tls_overrides import llm_verify
@@ -26,7 +28,7 @@ from src.endpoint_resolver import (
build_models_url,
build_headers,
)
from src.auth_helpers import _auth_disabled, owner_filter
from src.auth_helpers import _auth_disabled, effective_user, owner_filter
logger = logging.getLogger(__name__)
@@ -405,8 +407,11 @@ def _endpoint_refresh_timeout(ep: Any, category: str) -> float:
except Exception:
val = 0
if val > 0:
return float(max(1, min(30, val)))
return 2.5 if category == "local" else 2.0
return float(max(1, min(60, val)))
# llama.cpp and other local OpenAI-compatible servers can block briefly
# while warming/loading. A 2s local timeout makes working endpoints flicker
# offline before /v1/models is ready.
return 10.0 if category == "local" else 2.0
def _manual_refresh_timeout(ep: Any, category: str, requested: Any = None) -> float:
@@ -473,7 +478,7 @@ def _explicit_model_list_timeout(base_url: str, endpoint_kind: str = "auto", req
category = _classify_endpoint(base_url, kind)
if kind in ("api", "proxy") or category == "api":
return 30.0
return 3.0 if _is_ollama_base(base_url) else 2.0
return 15.0 if category == "local" else (3.0 if _is_ollama_base(base_url) else 2.0)
def _cached_model_ids(ep: Any) -> List[str]:
@@ -518,6 +523,10 @@ _NON_CHAT_EXACT_PREFIXES = (
def _is_chat_model(model_id: str) -> bool:
"""Return True if the model ID looks like a chat/completions-capable model."""
if not isinstance(model_id, str):
# Non-compliant upstreams can return non-string IDs (e.g. int/None);
# treat them as chat-capable rather than crashing on .lower().
return True
mid = model_id.lower()
for prefix in _NON_CHAT_PREFIXES:
if mid.startswith(prefix):
@@ -562,6 +571,8 @@ def _safe_build_models_url(base_url: str) -> str:
"""Build a /models URL without letting optional provider imports break probes."""
try:
return build_models_url(base_url)
except ValueError:
raise
except Exception as exc:
logger.debug("Model URL detection failed for %s: %s", base_url, exc)
return f"{(base_url or '').rstrip('/')}/models"
@@ -633,7 +644,7 @@ def _probe_single_model(base: str, api_key: str, model_id: str, timeout: int = 1
try:
t0 = _time.time()
r = httpx.post(target_url, headers=h, json=payload, timeout=timeout)
r = httpx.post(target_url, headers=h, json=payload, timeout=timeout, verify=llm_verify())
latency = round((_time.time() - t0) * 1000)
if r.is_success:
return {"status": "ok", "latency_ms": latency}
@@ -659,13 +670,20 @@ def _probe_single_model(base: str, api_key: str, model_id: str, timeout: int = 1
# Hostnames / IP prefixes that indicate a local endpoint
_LOCAL_HOSTS = {"localhost", "127.0.0.1", "0.0.0.0", "::1"}
_PRIVATE_PREFIXES = ("10.", "172.16.", "172.17.", "172.18.", "172.19.",
"172.20.", "172.21.", "172.22.", "172.23.", "172.24.",
"172.25.", "172.26.", "172.27.", "172.28.", "172.29.",
"172.30.", "172.31.", "192.168.")
_PRIVATE_NETWORKS = (
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
)
_TAILSCALE_CGNAT = ipaddress.ip_network("100.64.0.0/10")
_TAILSCALE_RE = re.compile(r"^100\.(6[4-9]|[7-9]\d|1[01]\d|12[0-7])\.")
def _local_ip_literal(host: str) -> bool:
try:
ip = ipaddress.ip_address(host)
except ValueError:
return False
return any(ip in network for network in _PRIVATE_NETWORKS) or ip in _TAILSCALE_CGNAT
def _classify_endpoint(base_url: str, endpoint_kind: str = "auto") -> str:
@@ -679,9 +697,7 @@ def _classify_endpoint(base_url: str, endpoint_kind: str = "auto") -> str:
return "api"
try:
host = urlparse(base_url).hostname or ""
if host in _LOCAL_HOSTS or host.startswith(_PRIVATE_PREFIXES):
return "local"
if _TAILSCALE_RE.match(host):
if host in _LOCAL_HOSTS or _local_ip_literal(host):
return "local"
except Exception:
pass
@@ -703,6 +719,51 @@ def _effective_endpoint_kind(ep: Any, base_url: str) -> str:
return "auto"
def _is_loading_model_response(resp: Any) -> bool:
if getattr(resp, "status_code", None) != 503:
return False
try:
body = resp.text or ""
except Exception:
body = ""
return "loading model" in body.lower()
def _openai_model_ids(data: Any) -> List[str]:
"""Extract OpenAI-style model IDs.
Accepts both standard ``{"data": [{"id": ...}]}`` responses and bare
``[{"id": ...}]`` lists returned by some OpenAI-compatible providers.
Tolerates non-dict/non-list bodies and non-string IDs, returning only
non-empty string IDs.
"""
if isinstance(data, list):
items = data
elif isinstance(data, dict):
items = data.get("data")
else:
items = None
return [m["id"] for m in (items or [])
if isinstance(m, dict) and isinstance(m.get("id"), str) and m["id"]]
def _ollama_model_names(data: Any) -> List[str]:
"""Extract native-Ollama model names (``{"models": [{"name"|"model": ...}]}``).
Same tolerance as :func:`_openai_model_ids`: a non-dict body or non-string
value is skipped rather than crashing, preserving name-then-model precedence.
"""
items = data.get("models") if isinstance(data, dict) else None
out: List[str] = []
for m in (items or []):
if not isinstance(m, dict):
continue
v = m.get("name") or m.get("model")
if isinstance(v, str) and v:
out.append(v)
return out
def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> List[str]:
"""Probe a base URL's /models endpoint and return list of model IDs.
@@ -726,7 +787,7 @@ def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> Lis
r = httpx.get(url, headers=headers, timeout=timeout, verify=llm_verify())
r.raise_for_status()
data = r.json()
models = [m.get("id") for m in (data.get("data") or []) if m.get("id")]
models = _openai_model_ids(data)
if models:
return models
except httpx.HTTPStatusError as e:
@@ -748,10 +809,10 @@ def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> Lis
r.raise_for_status()
data = r.json()
# OpenAI format: {"data": [{"id": "model-name"}]}
models = [m.get("id") for m in (data.get("data") or []) if m.get("id")]
models = _openai_model_ids(data)
# Ollama format: {"models": [{"name": "model-name"}]}
if not models:
models = [m.get("name") or m.get("model") for m in (data.get("models") or []) if m.get("name") or m.get("model")]
models = _ollama_model_names(data)
if models:
# Z.AI coding plan omits some working models from /models;
# append curated-only entries for that endpoint only.
@@ -767,16 +828,19 @@ def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> Lis
models.append(_e)
return [m for m in models if _is_chat_model(m)]
except httpx.HTTPStatusError as e:
if e.response is not None and _is_loading_model_response(e.response):
logger.info("Endpoint still loading model at %s", _redact_url_for_log(url))
return []
if api_key:
status = e.response.status_code if e.response is not None else "unknown"
logger.warning(f"Failed to probe {url} with API key: HTTP {status}")
logger.warning("Failed to probe %s with API key: HTTP %s", _redact_url_for_log(url), status)
return []
logger.warning(f"Failed to probe {url}: {e}")
logger.warning("Failed to probe %s: %s", _redact_url_for_log(url), e)
except Exception as e:
if api_key:
logger.warning(f"Failed to probe {url} with API key: {e}")
logger.warning("Failed to probe %s with API key: %s", _redact_url_for_log(url), e)
return []
logger.warning(f"Failed to probe {url}: {e}")
logger.warning("Failed to probe %s: %s", _redact_url_for_log(url), e)
# Older Ollama builds and some proxies expose native /api/tags even when
# the OpenAI-compatible /v1/models path is unavailable.
@@ -787,7 +851,7 @@ def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> Lis
r = httpx.get(root + "/api/tags", timeout=timeout, verify=llm_verify())
r.raise_for_status()
data = r.json()
models = [m.get("name") or m.get("model") for m in (data.get("models") or []) if m.get("name") or m.get("model")]
models = _ollama_model_names(data)
if models:
return [m for m in models if _is_chat_model(m)]
except Exception as e:
@@ -816,6 +880,15 @@ def _ping_endpoint(base_url: str, api_key: str = None, timeout: float = 1.5) ->
or "ollama" in (parsed_base.hostname or "").lower()
)
def _is_loading_model_response(r) -> bool:
if getattr(r, "status_code", None) != 503:
return False
try:
body = r.text or ""
except Exception:
body = ""
return "loading model" in body.lower()
def _result_from_response(r) -> Dict[str, Any]:
if 300 <= r.status_code < 400:
loc = r.headers.get("location", "")
@@ -832,6 +905,13 @@ def _ping_endpoint(base_url: str, api_key: str = None, timeout: float = 1.5) ->
"status_code": r.status_code,
"error": None,
}
if _is_loading_model_response(r):
return {
"reachable": True,
"loading": True,
"status_code": r.status_code,
"error": "Loading model",
}
return {"reachable": False, "status_code": r.status_code, "error": f"HTTP {r.status_code}"}
last_error: Optional[str] = None
@@ -864,7 +944,7 @@ def _ping_endpoint(base_url: str, api_key: str = None, timeout: float = 1.5) ->
if 400 <= sc < 500 and sc not in (401, 403):
models_url = _safe_build_models_url(base)
try:
r2 = httpx.get(models_url, headers=headers, timeout=timeout, verify=llm_verify())
r2 = httpx.get(models_url, headers=headers,timeout=timeout, verify=llm_verify())
result2 = _result_from_response(r2)
if result2["reachable"]:
return result2
@@ -1048,9 +1128,11 @@ def setup_model_routes(model_discovery):
except Exception:
return 0.0
def _failure_delay(fails: int) -> float:
def _failure_delay(fails: int, *, empty_local: bool = False) -> float:
if fails <= 0:
return 0.0
if empty_local:
return min(5.0 * (2 ** max(0, fails - 1)), 30.0)
return min(_REFRESH_FAILURE_BASE * (2 ** max(0, fails - 1)), _REFRESH_FAILURE_MAX)
def _should_refresh_endpoint(ep: Any, now: float, force: bool = False) -> tuple[bool, Dict[str, Any]]:
@@ -1081,7 +1163,12 @@ def setup_model_routes(model_discovery):
fails = int(state.get("fail_count") or 0)
if fails and not force:
last_failure = float(state.get("last_failure") or 0.0)
if now - last_failure < _failure_delay(fails):
empty_local = (
not cached
and category == "local"
and str(getattr(ep, "id", "") or "").startswith("local-")
)
if now - last_failure < _failure_delay(fails, empty_local=empty_local):
return False, info
if cached and not force:
interval = _endpoint_refresh_interval(ep, category)
@@ -1255,13 +1342,16 @@ def setup_model_routes(model_discovery):
# Require auth; "" is the unconfigured single-user mode, treated as
# "see everything" by _fetch_models.
try:
from src.auth_helpers import get_current_user as _gcu
owner = _gcu(request) or ""
except Exception:
owner = ""
# Reject anonymous in configured deployments — no leaking the model
# list to unauthenticated callers.
try:
if getattr(request.state, "api_token", False):
scopes = set(getattr(request.state, "api_token_scopes", []) or [])
if "chat" not in scopes:
raise HTTPException(403, "API token is not scoped for chat")
if not getattr(request.state, "api_token_owner", None):
raise HTTPException(403, "API token has no owner")
owner = effective_user(request) or ""
# Reject anonymous in configured deployments — no leaking the model
# list to unauthenticated callers.
auth_mgr = getattr(request.app.state, "auth_manager", None)
if not owner and not _auth_disabled() and auth_mgr is not None and getattr(auth_mgr, "is_configured", False):
raise HTTPException(401, "Not authenticated")
@@ -1393,7 +1483,7 @@ def setup_model_routes(model_discovery):
t0 = _time.time()
ping = _ping_endpoint(base, ep.api_key, timeout=1.5)
entry["latency_ms"] = round((_time.time() - t0) * 1000)
entry["status"] = "online" if ping.get("reachable") or cached_count else "offline"
entry["status"] = "loading" if ping.get("loading") else ("online" if ping.get("reachable") or cached_count else "offline")
entry["error"] = ping.get("error")
entry["model_count"] = cached_count or (len(ANTHROPIC_MODELS) if provider == "anthropic" else 0)
except Exception as e:
@@ -1567,9 +1657,37 @@ def setup_model_routes(model_discovery):
# "everything's already cached" path because this branch only
# runs for endpoints with an empty cached_models.
if not all_models and not pinned and r.is_enabled:
ping = _ping_endpoint(r.base_url, r.api_key, timeout=3.5)
base_for_ping = _normalize_base(r.base_url)
kind_for_ping = _effective_endpoint_kind(r, base_for_ping)
ping_timeout = 10.0 if _classify_endpoint(base_for_ping, kind_for_ping) == "local" else 3.5
ping = _ping_endpoint(r.base_url, r.api_key, timeout=ping_timeout)
if ping.get("reachable"):
status = "empty"
status = "loading" if ping.get("loading") else "empty"
if ping.get("loading"):
base = _normalize_base(r.base_url)
kind = _effective_endpoint_kind(r, base)
results.append({
"id": r.id,
"name": r.name,
"base_url": r.base_url,
"has_key": bool(r.api_key),
"api_key_fingerprint": _api_key_fingerprint(r.api_key),
"is_enabled": r.is_enabled,
"models": visible,
"pinned_models": pinned,
"hidden_count": len(hidden),
"online": True,
"status": status,
"ping_error": (ping or {}).get("error") if ping else None,
"model_type": getattr(r, "model_type", None) or "llm",
"supports_tools": getattr(r, "supports_tools", None),
"endpoint_kind": kind,
"category": _classify_endpoint(base, kind),
"model_refresh_mode": _endpoint_refresh_mode(r, kind),
"model_refresh_interval": getattr(r, "model_refresh_interval", None),
"model_refresh_timeout": getattr(r, "model_refresh_timeout", None),
})
continue
# Best-effort: if the probe came back reachable, try
# to populate cached_models in the background so the
# NEXT picker load shows "online" instead of "empty".
@@ -1577,7 +1695,7 @@ def setup_model_routes(model_discovery):
# "empty" status, and the existing background refresh
# path will eventually fill it in too.
try:
probed = _probe_endpoint(r.base_url, r.api_key, timeout=5)
probed = _probe_endpoint(r.base_url, r.api_key, timeout=max(5, int(ping_timeout)))
if probed:
r.cached_models = json.dumps(probed)
db.commit()
@@ -1755,7 +1873,7 @@ def setup_model_routes(model_discovery):
model_ids = _probe_endpoint(base_url, api_key.strip() or None, timeout=explicit_timeout) if should_probe else []
ping = {"reachable": False, "error": None}
if (should_probe or requested_kind in ("api", "proxy")) and not model_ids:
ping = _ping_endpoint(base_url, api_key.strip() or None, timeout=min(explicit_timeout, 2.0))
ping = _ping_endpoint(base_url, api_key.strip() or None, timeout=min(explicit_timeout, 10.0))
if require_model_list and not model_ids:
raise HTTPException(400, _model_endpoint_error_message(base_url, ping))
@@ -1822,7 +1940,7 @@ def setup_model_routes(model_discovery):
"models": _merge_model_ids(model_ids, _pinned),
"pinned_models": _pinned,
"online": bool(model_ids) or bool(_pinned) or bool(ping.get("reachable")),
"status": "online" if (model_ids or _pinned) else ("empty" if ping.get("reachable") else "offline"),
"status": "online" if (model_ids or _pinned) else ("loading" if ping.get("loading") else ("empty" if ping.get("reachable") else "offline")),
"ping_error": ping.get("error") if ping else None,
"endpoint_kind": requested_kind,
"category": _classify_endpoint(base_url, requested_kind),
@@ -1847,11 +1965,11 @@ def setup_model_routes(model_discovery):
configured_timeout = _parse_positive_int(model_refresh_timeout, minimum=1, maximum=60)
probe_timeout = _explicit_model_list_timeout(base_url, requested_kind, configured_timeout)
models = _probe_endpoint(base_url, api_key.strip() or None, timeout=probe_timeout)
ping = {"reachable": True, "error": None} if models else _ping_endpoint(base_url, api_key.strip() or None, timeout=min(probe_timeout, 2.0))
ping = {"reachable": True, "error": None} if models else _ping_endpoint(base_url, api_key.strip() or None, timeout=min(probe_timeout, 10.0))
return {
"base_url": base_url,
"online": bool(models) or bool(ping.get("reachable")),
"status": "online" if models else ("empty" if ping.get("reachable") else "offline"),
"status": "online" if models else ("loading" if ping.get("loading") else ("empty" if ping.get("reachable") else "offline")),
"ping_error": ping.get("error") if ping else None,
"models": models,
"count": len(models),
@@ -2029,6 +2147,16 @@ def setup_model_routes(model_discovery):
ep_id = (_user_prefs.get("default_endpoint_id") or "").strip()
model = (_user_prefs.get("default_model") or "").strip()
_fallbacks = _user_prefs.get("default_model_fallbacks") or []
# If user has no personal default, fall back to global default
# But only based on the "share_defaults_with_users" flag
# (only if share_defaults_with_users is enabled)
if settings.get("share_defaults_with_users", False):
if not ep_id:
ep_id = settings.get("default_endpoint_id", "")
if not model:
model = settings.get("default_model", "")
if not _fallbacks:
_fallbacks = settings.get("default_model_fallbacks") or []
else:
ep_id = settings.get("default_endpoint_id", "")
model = settings.get("default_model", "")
routes/note_routes.py
+29 -10
View File
@@ -10,7 +10,8 @@ from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel
from core.database import SessionLocal, Note
from src.auth_helpers import get_current_user
from core.middleware import INTERNAL_TOOL_USER
from src.auth_helpers import require_user
from src.constants import DATA_DIR
from sqlalchemy.orm.attributes import flag_modified
@@ -208,14 +209,17 @@ async def dispatch_reminder(
try:
from src.endpoint_resolver import resolve_endpoint
from src.llm_core import llm_call_async
from src.reminder_personas import synthesis_system_prompt
url, model, headers = resolve_endpoint("utility", owner=owner or None)
if not url:
url, model, headers = resolve_endpoint("default", owner=owner or None)
if url and model:
persona_id = (settings.get("reminder_llm_persona") or "").strip()
sys_prompt = synthesis_system_prompt(persona_id)
raw = await llm_call_async(
url=url, model=model,
messages=[
{"role": "system", "content": "You are a reminder assistant. Write a single short, warm, motivating sentence (max 25 words) reminding the user about the note below. Do not add greetings, preamble, or hashtags. Output only the sentence."},
{"role": "system", "content": sys_prompt},
{"role": "user", "content": f"Title: {title}\n\n{note_body}".strip()},
],
temperature=0.7, max_tokens=200, headers=headers, timeout=30,
@@ -331,10 +335,11 @@ async def dispatch_reminder(
# Loud diagnostic so we can see WHY a reminder didn't send (the
# previous "silently no-op when cfg has no smtp_host" was invisible).
logger.info(
f"dispatch_reminder[email] note_id={note_id} owner={owner!r} "
f"smtp_host={cfg.get('smtp_host')!r} smtp_user={cfg.get('smtp_user')!r} "
f"from={from_addr!r} recipient={recipient!r} "
f"account_name={cfg.get('account_name')!r}"
"dispatch_reminder[email] note_id=%s owner=%r "
"has_smtp_host=%s has_smtp_user=%s has_from=%s has_recipient=%s",
note_id, owner,
bool(cfg.get("smtp_host")), bool(cfg.get("smtp_user")),
bool(from_addr), bool(recipient),
)
missing = []
if not cfg.get("smtp_host"):
@@ -567,10 +572,19 @@ def setup_note_routes(task_scheduler=None):
router = APIRouter(prefix="/api/notes", tags=["notes"])
def _owner(request: Request) -> Optional[str]:
return get_current_user(request)
# require_user, not bare get_current_user: a request that reaches
# these owner-scoped routes with NO identity (auth-middleware
# regression, SSRF from a sibling service) must fail closed (401)
# when auth is configured — not be treated as the single-user mode
# and handed blanket access to every account's notes. The documented
# anonymous modes (AUTH_ENABLED=false, LOCALHOST_BYPASS on loopback,
# unconfigured first-run) still resolve to None, the single-user
# path. fire_reminder below already gated this way; the CRUD routes
# did not.
return require_user(request) or None
def _is_admin_or_single_user(request: Request, user: str | None) -> bool:
if user == "internal-tool":
if user == INTERNAL_TOOL_USER:
return True
if not user:
# require_user() already admitted this request, which only happens
@@ -802,8 +816,7 @@ def setup_note_routes(task_scheduler=None):
Returns {synthesis, email_sent}.
"""
# Gate against anonymous callers — LLM synthesis can burn tokens.
from src.auth_helpers import require_user as _ru
user = _ru(request)
user = require_user(request)
body = await request.json()
note_id = str(body.get("note_id") or "").strip()
if not note_id:
@@ -826,6 +839,12 @@ def setup_note_routes(task_scheduler=None):
_override["reminder_webhook_integration_id"] = body["webhook_integration_id"]
if body.get("webhook_payload_template"):
_override["reminder_webhook_payload_template"] = body["webhook_payload_template"]
# Mirror the in-UI AI Synthesis toggle + persona so the test
# actually exercises the synthesis path before/without a Save.
if "llm_synthesis" in body:
_override["reminder_llm_synthesis"] = bool(body["llm_synthesis"])
if "llm_persona" in body:
_override["reminder_llm_persona"] = str(body["llm_persona"] or "")
else:
db = SessionLocal()
try:
routes/personal_routes.py
+91 -6
View File
@@ -2,8 +2,9 @@
"""Routes for personal documents management."""
import os
import logging
import shutil
import uuid
from typing import List, Tuple
from typing import Any, Dict, List, Tuple
from fastapi import APIRouter, HTTPException, Query, Request, UploadFile, File, Depends
from src.request_models import DirectoryRequest
from core.constants import BASE_DIR, PERSONAL_DIR, PERSONAL_UPLOADS_DIR
@@ -18,14 +19,15 @@ UPLOADS_DIR = PERSONAL_UPLOADS_DIR
logger = logging.getLogger(__name__)
def _personal_upload_dir_for_owner(owner: str | None) -> str:
def _personal_upload_dir_for_owner(owner: str | None, *, create: bool = True) -> str:
"""Return the per-owner upload directory used for direct RAG uploads."""
owner_segment = secure_filename((owner or "local").strip())[:80] or "local"
upload_dir = os.path.abspath(os.path.join(UPLOADS_DIR, owner_segment))
base_abs = os.path.abspath(UPLOADS_DIR)
if os.path.commonpath([upload_dir, base_abs]) != base_abs:
raise ValueError("Unsafe upload owner path")
os.makedirs(upload_dir, exist_ok=True)
if create:
os.makedirs(upload_dir, exist_ok=True)
return upload_dir
@@ -44,6 +46,87 @@ def _unique_personal_upload_path(upload_dir: str, original_name: str | None) ->
raise ValueError("Unsafe upload filename")
return file_path, filename, safe_name
def _unique_existing_target(path: str) -> str:
"""Return a non-existing sibling path for rename collision handling."""
if not os.path.exists(path):
return path
stem, ext = os.path.splitext(path)
while True:
candidate = f"{stem}-{uuid.uuid4().hex[:10]}{ext}"
if not os.path.exists(candidate):
return candidate
def _remove_empty_tree(path: str) -> None:
"""Best-effort removal of empty directories under ``path``."""
if not os.path.isdir(path):
return
for root, dirs, _files in os.walk(path, topdown=False):
for dirname in dirs:
candidate = os.path.join(root, dirname)
try:
os.rmdir(candidate)
except OSError:
pass
try:
os.rmdir(path)
except OSError:
pass
def rename_personal_upload_owner(
old_owner: str,
new_owner: str,
*,
personal_docs_manager: Any = None,
rag_manager: Any = None,
) -> Dict[str, Any]:
"""Move direct personal uploads and rewrite RAG owner metadata on user rename."""
old_dir = _personal_upload_dir_for_owner(old_owner, create=False)
new_dir = _personal_upload_dir_for_owner(new_owner, create=False)
path_map: Dict[str, str] = {}
moved_files = 0
if os.path.isdir(old_dir) and old_dir != new_dir:
os.makedirs(new_dir, exist_ok=True)
for root, _dirs, files in os.walk(old_dir):
rel_root = os.path.relpath(root, old_dir)
target_root = new_dir if rel_root == "." else os.path.join(new_dir, rel_root)
os.makedirs(target_root, exist_ok=True)
for filename in files:
source = os.path.abspath(os.path.join(root, filename))
target = _unique_existing_target(os.path.abspath(os.path.join(target_root, filename)))
shutil.move(source, target)
path_map[source] = target
moved_files += 1
_remove_empty_tree(old_dir)
if personal_docs_manager is not None:
rename_directory = getattr(personal_docs_manager, "rename_directory", None)
if callable(rename_directory):
rename_directory(old_dir, new_dir, path_map=path_map)
rag_result = None
if rag_manager is not None:
rename_owner = getattr(rag_manager, "rename_owner", None)
if callable(rename_owner):
rag_result = rename_owner(
old_owner,
new_owner,
path_map=path_map,
path_prefixes=[(old_dir, new_dir)],
)
return {
"old_dir": old_dir,
"new_dir": new_dir,
"moved_files": moved_files,
"path_map": path_map,
"rag_result": rag_result,
}
def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
"""
Setup personal documents related routes.
@@ -275,11 +358,13 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
except Exception as e:
logger.warning(f"RAG removal failed for {filepath}: {e}")
# Delete file from disk if it's in uploads dir
# Delete file from disk if it's in the caller's own uploads dir.
# Scope to the per-owner subdir, not the shared uploads root, so one
# admin can't delete another user's personal files by path.
deleted_from_disk = False
try:
abs_target = os.path.abspath(filepath)
base_abs = os.path.abspath(UPLOADS_DIR)
abs_target = os.path.realpath(filepath)
base_abs = os.path.realpath(_personal_upload_dir_for_owner(owner, create=False))
in_uploads = (
abs_target == base_abs
or os.path.commonpath([abs_target, base_abs]) == base_abs
routes/preset_routes.py
+2 -1
View File
@@ -1,5 +1,6 @@
"""Preset routes — /api/presets GET, /api/presets/custom POST, user templates CRUD."""
import asyncio
import logging
import uuid
from typing import Dict, Any, List
@@ -102,7 +103,7 @@ def setup_preset_routes(preset_manager) -> APIRouter:
try:
model_spec = data.get("model") or ""
user = effective_user(request)
url, model, headers = _resolve_model(model_spec, owner=user)
url, model, headers = await asyncio.to_thread(_resolve_model, model_spec, owner=user)
result = await llm_call_async(url, model, messages, temperature=0.8, max_tokens=500, headers=headers)
return {"success": True, "prompt": result.strip()}
except Exception as e:
routes/research_routes.py
+4 -2
View File
@@ -12,8 +12,10 @@ from typing import Optional
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import HTMLResponse, StreamingResponse
from pydantic import BaseModel, Field
from core.middleware import INTERNAL_TOOL_USER
from src.endpoint_resolver import resolve_endpoint
from src.auth_helpers import _auth_disabled, get_current_user
from core.auth import RESERVED_USERNAMES
from src.constants import DEEP_RESEARCH_DIR
_SESSION_ID_RE = re.compile(r"^[a-zA-Z0-9-]{1,128}$")
@@ -385,9 +387,9 @@ def setup_research_routes(research_handler, session_manager=None) -> APIRouter:
"""Launch a research job from the dedicated panel."""
from src.auth_helpers import require_privilege
user = require_privilege(request, "can_use_research")
if user == "internal-tool":
if user == INTERNAL_TOOL_USER:
tool_owner = (request.headers.get("X-Odysseus-Owner") or "").strip()
if tool_owner and tool_owner not in {"internal-tool", "api", "demo", "system"}:
if tool_owner and tool_owner not in RESERVED_USERNAMES:
auth_mgr = getattr(request.app.state, "auth_manager", None)
if auth_mgr is not None and getattr(auth_mgr, "is_configured", False):
try:
routes/session_routes.py
+16 -5
View File
@@ -11,7 +11,7 @@ from core.session_manager import SessionManager
from core.models import ChatMessage
from src.request_models import SessionResponse
from core.database import Session as DbSession, SessionLocal, Document, GalleryImage, utcnow_naive
from src.auth_helpers import get_current_user, effective_user, _auth_disabled, owner_filter
from src.auth_helpers import effective_user, _auth_disabled, owner_filter
from src.session_actions import is_session_recently_active
@@ -328,7 +328,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
endpoint_id: str = Form(""),
):
skip_val = str(skip_validation).lower() == "true"
user = get_current_user(request)
user = effective_user(request)
endpoint_api_key = ""
endpoint_base_url = ""
_reject_raw_endpoint_url_for_non_admin(request, user, endpoint_id, endpoint_url)
@@ -477,7 +477,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
db.close()
# Switch model/endpoint mid-session
if model is not None and endpoint_url is not None:
user = get_current_user(request)
user = effective_user(request)
_reject_raw_endpoint_url_for_non_admin(request, user, endpoint_id, endpoint_url)
endpoint_api_key = ""
endpoint_base_url = ""
@@ -1004,6 +1004,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
"""
from src.llm_core import llm_call
user = effective_user(request)
single_user_mode = not user and _auth_disabled()
user_sessions = session_manager.get_sessions_for_user(user)
# Delete empty and throwaway sessions before sorting
@@ -1022,7 +1023,12 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
}
_THROWAWAY_MAX_MESSAGES = 4 # only delete if <= this many messages
try:
rows = db.query(DbSession).filter(DbSession.archived == False, DbSession.owner == user).limit(2000).all()
rows_q = db.query(DbSession).filter(DbSession.archived == False)
if user:
rows_q = rows_q.filter(DbSession.owner == user)
elif not single_user_mode:
rows_q = rows_q.filter(DbSession.owner == user)
rows = rows_q.limit(2000).all()
folder_map = {r.id: r.folder for r in rows}
# Precompute per-session message counts in TWO aggregate queries
# instead of 1–3 queries PER session — with many chats the per-row
@@ -1242,7 +1248,12 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
db = SessionLocal()
try:
for sid, folder_name in assignments.items():
db_session = db.query(DbSession).filter(DbSession.id == sid, DbSession.owner == user).first()
db_session_q = db.query(DbSession).filter(DbSession.id == sid)
if user:
db_session_q = db_session_q.filter(DbSession.owner == user)
elif not single_user_mode:
db_session_q = db_session_q.filter(DbSession.owner == user)
db_session = db_session_q.first()
if db_session:
db_session.folder = folder_name
db_session.updated_at = datetime.utcnow()
routes/shell_routes.py
+381 -15
View File
@@ -15,6 +15,7 @@ from collections import namedtuple
from pathlib import Path
from typing import Dict, Any
from core.platform_compat import IS_APPLE_SILICON, which_tool
from core.middleware import INTERNAL_TOOL_USER
from src.optional_deps import prepare_optional_dependency_import
# POSIX-only: `pty`/`fcntl` transitively import `termios`, which does NOT exist
@@ -55,7 +56,7 @@ def _require_admin(request: Request):
# In-process tool loopback. The AuthMiddleware already validated the
# internal token + loopback client before setting this marker, so
# honour it here as admin-equivalent.
if user == "internal-tool":
if user == INTERNAL_TOOL_USER:
return
if not user or user == "api":
raise HTTPException(403, "Admin only")
@@ -330,6 +331,9 @@ def add_user_install_bins_to_path():
candidates.append(os.path.join(site.USER_BASE, 'bin'))
except Exception:
pass
candidates.append(os.path.expanduser('~/bin'))
candidates.append(os.path.expanduser('~/llama.cpp/build/bin'))
candidates.append(os.path.expanduser('~/llama.cpp/build-vulkan/bin'))
candidates.append(os.path.expanduser('~/.local/bin'))
parts = os.environ.get('PATH', '').split(os.pathsep) if os.environ.get('PATH') else []
changed = False
@@ -961,12 +965,84 @@ def setup_shell_routes() -> APIRouter:
return StreamingResponse(generate(), media_type="text/event-stream")
def _os_id_from_release(text: str) -> str:
"""Map /etc/os-release contents to a canonical family for our matrix."""
if not text:
return ""
ids = []
for line in text.splitlines():
line = line.strip()
if line.startswith("ID=") or line.startswith("ID_LIKE="):
ids += line.split("=", 1)[1].strip().strip('"').split()
ids = [i.lower() for i in ids]
if any(x in ids for x in ("debian", "ubuntu", "linuxmint", "pop", "elementary")):
return "debian"
if any(x in ids for x in ("arch", "manjaro", "endeavouros", "cachyos", "garuda")):
return "arch"
if any(x in ids for x in ("fedora", "rhel", "centos", "rocky", "almalinux", "ol")):
return "fedora"
if "alpine" in ids:
return "alpine"
if any(x in ids for x in ("suse", "opensuse", "opensuse-leap", "opensuse-tumbleweed", "sles")):
return "suse"
return ""
# Matrix lookup keyed on (os_family, backend) → (pkg_mgr_cmd_template, pkg_list_per_dep).
# Each `system_prereqs` name resolves to a list of OS-specific package
# names that get joined into the final `sudo apt install -y …` etc.
# command. Backend-specific extras (CUDA toolkit, ROCm, Vulkan headers)
# are added only when the detected backend needs them.
_PKG_NAMES = {
# canonical-name → {os_id: [actual_pkg_names_on_this_os]}
"cmake": {"debian": ["cmake"], "arch": ["cmake"], "fedora": ["cmake"], "alpine": ["cmake"], "suse": ["cmake"], "macos": ["cmake"]},
"build-essential": {"debian": ["build-essential"], "arch": ["base-devel"], "fedora": ["gcc", "gcc-c++", "make"], "alpine": ["build-base"], "suse": ["gcc-c++", "make"], "macos": []},
"g++": {"debian": ["g++"], "arch": ["gcc"], "fedora": ["gcc-c++"], "alpine": ["g++"], "suse": ["gcc-c++"], "macos": []},
"gcc": {"debian": ["gcc"], "arch": ["gcc"], "fedora": ["gcc"], "alpine": ["gcc"], "suse": ["gcc"], "macos": []},
"make": {"debian": ["make"], "arch": ["make"], "fedora": ["make"], "alpine": ["make"], "suse": ["make"], "macos": []},
"git": {"debian": ["git"], "arch": ["git"], "fedora": ["git"], "alpine": ["git"], "suse": ["git"], "macos": ["git"]},
"tmux": {"debian": ["tmux"], "arch": ["tmux"], "fedora": ["tmux"], "alpine": ["tmux"], "suse": ["tmux"], "macos": ["tmux"]},
}
_BACKEND_EXTRAS = {
"cuda": {"debian": ["nvidia-cuda-toolkit"], "arch": ["cuda"], "fedora": ["cuda-toolkit"], "alpine": [], "suse": ["cuda"], "macos": []},
"rocm": {"debian": ["rocm-dev"], "arch": ["rocm-hip-sdk"], "fedora": ["rocm-devel"], "alpine": [], "suse": ["rocm-dev"], "macos": []},
"vulkan": {"debian": ["libvulkan-dev", "vulkan-tools"], "arch": ["vulkan-headers", "vulkan-tools"], "fedora": ["vulkan-headers", "vulkan-tools"], "alpine": ["vulkan-loader-dev", "vulkan-tools"], "suse": ["vulkan-devel", "vulkan-tools"], "macos": []},
}
_PKG_MGR = {
"debian": "sudo apt install -y {pkgs}",
"arch": "sudo pacman -S --needed {pkgs}",
"fedora": "sudo dnf install -y {pkgs}",
"alpine": "sudo apk add {pkgs}",
"suse": "sudo zypper install -n {pkgs}",
"macos": "brew install {pkgs}",
}
def _install_cmd_for_target(os_id: str, backend: str, missing: list[str]) -> str:
"""Build a single OS+backend-aware install command for the missing prereqs."""
if not os_id or os_id not in _PKG_MGR:
return ""
pkgs: list[str] = []
seen: set[str] = set()
for m in missing:
for p in _PKG_NAMES.get(m, {}).get(os_id, []):
if p not in seen:
pkgs.append(p); seen.add(p)
# Add backend-specific extras only when the build would actually
# consume them (a CUDA toolkit isn't useful on a Vulkan box).
backend = (backend or "").lower()
for p in _BACKEND_EXTRAS.get(backend, {}).get(os_id, []):
if p not in seen:
pkgs.append(p); seen.add(p)
if not pkgs:
return ""
return _PKG_MGR[os_id].format(pkgs=" ".join(pkgs))
@router.get("/api/cookbook/packages")
async def list_packages(
request: Request,
host: str | None = None,
ssh_port: str | None = None,
venv: str | None = None,
backend: str | None = None,
):
"""Check which optional packages are installed.
@@ -987,8 +1063,19 @@ def setup_shell_routes() -> APIRouter:
importlib.invalidate_caches()
try:
user_site = site.getusersitepackages()
if user_site and os.path.isdir(user_site) and user_site not in sys.path:
sys.path.append(user_site)
if user_site and os.path.isdir(user_site):
# Use addsitedir(), NOT a bare sys.path.append(). When a package
# is `pip install --user`'d at runtime (Cookbook → Install) the
# long-lived server process started before the user-site existed,
# so site never processed it — including its `.pth` hooks. On
# Python 3.12+ `distutils` is gone from stdlib and is only
# restored by setuptools' `distutils-precedence.pth`, which ships
# in user-site. basicsr (a realesrgan dep) does `import distutils`
# at import time, so a plain append left the package importable
# but `import distutils` failing → realesrgan probed as
# not-installed until a full process restart. addsitedir() replays
# the `.pth` files so the shim is active.
site.addsitedir(user_site)
except Exception:
pass
if ssh_port and str(ssh_port).strip() not in ("", "22"):
@@ -1015,6 +1102,12 @@ def setup_shell_routes() -> APIRouter:
"kind": "system",
"install_hint": "Install Docker on the selected server and allow this user to run docker.",
},
# Note: cmake / gcc / git are not separate dependency rows —
# they're declared as `system_prereqs` on llama_cpp (and any
# other engine that compiles from source) so they appear as
# an inline status note on that engine's row instead of
# cluttering the panel with raw OS package names that aren't
# meaningful product-level dependencies on their own.
# ── LLM ── installs on GPU servers for model serving/downloading
{
"name": "hf_transfer",
@@ -1026,9 +1119,16 @@ def setup_shell_routes() -> APIRouter:
{
"name": "llama_cpp",
"pip": "llama-cpp-python[server]",
"desc": "Serve GGUF models via llama.cpp",
"desc": "Great for single-GPU or CPU inference with GGUF models",
"category": "LLM",
"target": "remote",
# Build-toolchain prereqs. Cookbook's launch bootstrap
# compiles llama-server from source when no prebuilt
# binary is present; without these the build aborts
# with `cmake: command not found`. Surfaced inline on
# this row so the user doesn't have to chase three
# separate OS-package rows.
"system_prereqs": ["cmake", "g++", "git"],
},
{
"name": "sglang",
@@ -1040,7 +1140,7 @@ def setup_shell_routes() -> APIRouter:
{
"name": "vllm",
"pip": "vllm",
"desc": "High-throughput LLM serving engine",
"desc": "Great for high-throughput multi-GPU inference",
"category": "LLM",
"target": "remote",
},
@@ -1103,6 +1203,7 @@ def setup_shell_routes() -> APIRouter:
# venv over SSH so a remote `pip install` actually reflects here.
remote_status: dict = {}
remote_details: dict = {}
remote_probe_error = ""
remote_names = [
p["name"]
for p in packages
@@ -1141,16 +1242,56 @@ def setup_shell_routes() -> APIRouter:
break
except ValueError as e:
raise HTTPException(400, str(e))
except Exception:
except Exception as e:
remote_status = {}
if host and remote_system_names:
remote_probe_error = f"SSH package probe failed: {str(e)[:160]}"
if "llama_cpp" in remote_names:
try:
inner = (
'export PATH="$HOME/.local/bin:$HOME/bin:'
'$HOME/llama.cpp/build/bin:$HOME/llama.cpp/build-vulkan/bin:$PATH"; '
"command -v llama-server 2>/dev/null || true"
)
argv = _ssh_base_argv(host, ssh_port) + [inner]
proc = await asyncio.create_subprocess_exec(
*argv,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
out, _err = await asyncio.wait_for(proc.communicate(), timeout=8)
llama_server_path = out.decode("utf-8", errors="replace").strip().splitlines()
llama_server_path = llama_server_path[-1].strip() if llama_server_path else ""
if llama_server_path:
remote_status["llama_cpp"] = True
probe = remote_details.setdefault("llama_cpp", {})
if isinstance(probe, dict):
probe.setdefault("binaries", {})["llama-server"] = llama_server_path
except Exception as e:
if not remote_probe_error:
remote_probe_error = f"SSH llama-server probe failed: {str(e)[:160]}"
pass
# Union of system_names + every package's system_prereqs. Probing
# the prereqs alongside the main system deps in a single SSH call
# avoids a second round-trip per Cookbook → Dependencies refresh.
prereq_names: set[str] = set()
for p in packages:
for pr in p.get("system_prereqs") or []:
prereq_names.add(str(pr))
all_system_names = list(set(remote_system_names) | prereq_names)
# Detect the target's OS family + read /etc/os-release in the same
# SSH round-trip as the prereq probe — used downstream to render a
# single OS-specific install command per row instead of dumping
# every distro's syntax onto the user.
target_os_id: str = ""
if host and all_system_names:
try:
checks = []
for name in remote_system_names:
for name in all_system_names:
qn = shlex.quote(name)
checks.append(
f"if command -v {qn} >/dev/null 2>&1; then echo {qn}=1; else echo {qn}=0; fi"
)
checks.append("echo '---OSREL---'; cat /etc/os-release 2>/dev/null || true")
inner = " ; ".join(checks)
argv = _ssh_base_argv(host, ssh_port) + [inner]
proc = await asyncio.create_subprocess_exec(
@@ -1160,20 +1301,45 @@ def setup_shell_routes() -> APIRouter:
)
out, _err = await asyncio.wait_for(proc.communicate(), timeout=12)
txt = out.decode("utf-8", errors="replace").strip()
_section, _osrel_lines = "probe", []
for line in txt.splitlines():
if line.strip() == "---OSREL---":
_section = "osrel"; continue
if _section == "osrel":
_osrel_lines.append(line)
continue
name, sep, value = line.strip().partition("=")
if sep and name in remote_system_names:
if sep and name in all_system_names:
remote_status[name] = value == "1"
target_os_id = _os_id_from_release("\n".join(_osrel_lines))
except ValueError as e:
raise HTTPException(400, str(e))
except Exception:
except Exception as e:
if not remote_probe_error:
remote_probe_error = f"SSH system probe failed: {str(e)[:160]}"
pass
elif not host:
# Local target — probe in-process so the inline install command
# still appears in the dep panel when the cookbook container
# itself is the selected server.
try:
with open("/etc/os-release", encoding="utf-8") as f:
target_os_id = _os_id_from_release(f.read())
except Exception:
target_os_id = ""
if sys.platform == "darwin":
target_os_id = "macos"
for pkg in packages:
on_remote = bool(host and pkg.get("target") == "remote")
probe = None
if on_remote:
pkg["installed"] = bool(remote_status.get(pkg["name"], False))
if remote_probe_error and pkg["name"] not in remote_status:
pkg["installed"] = None
pkg["probe_error"] = remote_probe_error
pkg["status_note"] = remote_probe_error
else:
pkg["installed"] = bool(remote_status.get(pkg["name"], False))
probe = remote_details.get(pkg["name"])
if isinstance(probe, dict):
pkg["details"] = probe
@@ -1222,13 +1388,116 @@ def setup_shell_routes() -> APIRouter:
pkg["installed"] = False
except importlib_metadata.PackageNotFoundError:
pkg["installed"] = False
except Exception:
except (Exception, SystemExit):
# Installed but crashes on import — e.g. a CUDA build of
# llama-cpp-python raising FileNotFoundError when the CUDA
# toolkit dir is absent. One broken optional package must not
# 500 the entire packages panel; report it as not usable.
# toolkit dir is absent, or rembg calling sys.exit(1) when no
# onnxruntime backend can be loaded. SystemExit is a
# BaseException, not Exception, so without catching it here a
# single sys.exit-on-import package escapes and takes down the
# whole packages panel / worker (the panel hangs forever). One
# broken optional package must not 500 — or hang — the entire
# panel; report it as not usable.
pkg["installed"] = False
# llama_cpp partial-state probe: when the package is installed
# but the wheel was built CPU-only AND the target has NVIDIA
# hardware, mark the row as partial (yellow/orange) with a
# one-click upgrade to the CUDA wheel. Without this the row
# reads "ready" green while inference runs at 3 tok/s on GPU
# silicon — actively misleading.
if pkg["name"] == "llama_cpp" and pkg.get("installed"):
_native_llama_server = bool(
isinstance(probe, dict)
and isinstance(probe.get("binaries"), dict)
and probe["binaries"].get("llama-server")
)
_gpu_capable = False
_has_nvidia_target = False
if _native_llama_server:
# Native llama-server is the launcher path Cookbook now
# prefers. Do not mark this as a CPU-only Python wheel just
# because llama-cpp-python is absent from the selected venv.
_gpu_capable = True
elif on_remote and host:
try:
# Activate the configured venv FIRST so the probe
# runs against the same python the launch script
# would activate. Without this prefix, bare
# `python3` was checked — which can disagree with
# the venv's wheel (e.g. user-site has CUDA wheel
# but venv has CPU-only), and the dep panel then
# showed "ready" green while every launch fell to
# CPU.
_vp = _venv_activate_prefix(venv)
probe = (
f'{_vp}python3 -c "import llama_cpp; import sys; '
'sys.exit(0 if llama_cpp.llama_supports_gpu_offload() else 1)" '
'&& echo llama_cpp_gpu=1 || echo llama_cpp_gpu=0; '
'command -v nvidia-smi >/dev/null 2>&1 '
'&& nvidia-smi -L 2>/dev/null | grep -q "GPU " '
'&& echo nvidia=1 || echo nvidia=0'
)
argv = _ssh_base_argv(host, ssh_port) + [probe]
proc = await asyncio.create_subprocess_exec(
*argv, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
)
out, _ = await asyncio.wait_for(proc.communicate(), timeout=8)
txt = out.decode("utf-8", errors="replace")
if "llama_cpp_gpu=1" in txt:
_gpu_capable = True
if "nvidia=1" in txt:
_has_nvidia_target = True
except Exception:
pass
else:
try:
import llama_cpp as _lcp # type: ignore
_gpu_capable = bool(_lcp.llama_supports_gpu_offload())
except Exception:
_gpu_capable = False
_has_nvidia_target = shutil.which("nvidia-smi") is not None
if (not _gpu_capable) and _has_nvidia_target:
pkg["partial"] = True
pkg["partial_reason"] = "Installed but CPU-only wheel — GPU detected on this target. Upgrade to a CUDA wheel for ~10× faster inference."
pkg["partial_action"] = "reinstall_llama_cpp_cuda"
# Attach per-package system_prereqs status. We probed each
# prereq name above; surface "Missing build deps: …" ONLY
# when the package itself is not installed — if the package
# works (e.g. llama-cpp-python already imports cleanly), the
# build toolchain is irrelevant and surfacing it as a red
# flag confuses users ("ready" + "missing" on the same row).
_prereqs = list(pkg.get("system_prereqs") or [])
if _prereqs:
if on_remote:
_pr_present = {n: bool(remote_status.get(n)) for n in _prereqs}
else:
_pr_present = {n: shutil.which(n) is not None for n in _prereqs}
pkg["system_prereqs_status"] = _pr_present
_missing = [n for n, ok in _pr_present.items() if not ok]
# Suppress the "missing build deps" hint when the package
# itself is installed — build deps are only relevant if
# the user would need to recompile from source.
if pkg.get("installed"):
_missing = []
if _missing:
# Build a target-specific install command from the
# (os_family, backend) matrix when we know both. Fall
# back to the multi-distro hint only when the target's
# OS can't be classified (e.g. ssh probe failed).
_resolved_os = target_os_id or "debian" # safest default
_cmd = _install_cmd_for_target(_resolved_os, backend or "", _missing)
if _cmd and target_os_id:
_hint = "Missing build deps for this target: " + ", ".join(_missing)
pkg["install_cmd_for_target"] = _cmd
pkg["install_cmd_os"] = target_os_id
pkg["install_cmd_backend"] = (backend or "").lower()
else:
_hint = "Missing build deps: " + ", ".join(_missing) + ". Install via apt: cmake build-essential git / pacman: cmake base-devel git / dnf: cmake gcc-c++ make git / brew: cmake git."
_existing_note = pkg.get("status_note") or ""
pkg["status_note"] = (_existing_note + " — " + _hint) if _existing_note else _hint
pkg["build_deps_missing"] = _missing
if pkg.get("installed"):
update_status = _package_pip_update_status(pkg, probe)
pkg["pip_update_available"] = update_status.available
@@ -1288,6 +1557,102 @@ def setup_shell_routes() -> APIRouter:
return {"ok": True, "output": stdout.decode()[-200:]}
return {"ok": False, "error": stderr.decode()[-300:]}
@router.post("/api/cookbook/install-system-deps")
async def install_system_deps(request: Request):
"""Install OS-level system packages (cmake/build-essential/git/tmux)
on a remote target or in the local container. Admin only.
Bounded by a per-package allowlist — anything outside the catalog
is rejected so the route can't be coerced into installing arbitrary
OS packages. Uses `sudo -n` (passwordless) so the call returns a
clear "needs sudo password" error instead of hanging when interactive
sudo is required.
"""
_require_admin(request)
body = await request.json()
raw = body.get("packages") or []
host = (body.get("remote_host") or "").strip()
ssh_port = body.get("ssh_port")
# Names users can request — must match canonical names used in the
# deps catalog's `system_prereqs` field and on the System rows.
ALLOWED = {"cmake", "build-essential", "g++", "gcc", "git", "tmux", "make"}
pkgs = [str(p).strip() for p in raw if str(p).strip() in ALLOWED]
if not pkgs:
return {"ok": False, "error": "no installable packages requested (allowlist: " + ", ".join(sorted(ALLOWED)) + ")"}
# Re-map to the right package name per OS. apt/dpkg use the names
# as-is; pacman has base-devel for build-essential, etc.
def _apt(names): return list(names)
def _pacman(names):
return ["base-devel" if n == "build-essential" else n for n in names]
def _dnf(names):
out = []
for n in names:
if n == "build-essential": out += ["gcc", "gcc-c++", "make"]
elif n == "g++": out += ["gcc-c++"]
else: out.append(n)
return out
def _brew(names):
return [n for n in names if n not in ("build-essential", "g++", "gcc", "make")]
# Build a single shell snippet that detects the package manager and
# runs the right install. Non-interactive sudo (-n) only — if sudo
# asks for a password the script reports it instead of hanging.
apt_pkgs = " ".join(shlex.quote(p) for p in _apt(pkgs))
pac_pkgs = " ".join(shlex.quote(p) for p in _pacman(pkgs))
dnf_pkgs = " ".join(shlex.quote(p) for p in _dnf(pkgs))
brew_pkgs = " ".join(shlex.quote(p) for p in _brew(pkgs))
# Error messages go to stderr (>&2) so the route's error field
# gets populated. Without the redirect, `echo "ERROR…"` on stdout
# left stderr empty and the frontend toast fell through to a
# bare "HTTP 200" instead of surfacing the real reason.
script = (
'set -e; '
'if ! sudo -n true 2>/dev/null; then '
' echo "ERROR: passwordless sudo unavailable on this target. Run once: sudo apt install -y ' + " ".join(pkgs) + ' (or your distro equivalent: pacman -S, dnf install, brew install). After that, Cookbook can install the rest." >&2; exit 2; fi; '
'if command -v apt-get >/dev/null 2>&1; then '
f' sudo -n env DEBIAN_FRONTEND=noninteractive apt-get update -qq && sudo -n env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends {apt_pkgs}; '
'elif command -v pacman >/dev/null 2>&1; then '
f' sudo -n pacman -Sy --needed --noconfirm {pac_pkgs}; '
'elif command -v dnf >/dev/null 2>&1; then '
f' sudo -n dnf install -y {dnf_pkgs}; '
'elif command -v brew >/dev/null 2>&1; then '
f' brew install {brew_pkgs}; '
'else '
' echo "ERROR: no supported package manager (apt/pacman/dnf/brew) on this target." >&2; exit 3; fi'
)
try:
if host:
argv = _ssh_base_argv(host, ssh_port) + [script]
else:
argv = ["bash", "-lc", script]
except ValueError as e:
raise HTTPException(400, str(e))
try:
proc = await asyncio.create_subprocess_exec(
*argv, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
)
out, err = await asyncio.wait_for(proc.communicate(), timeout=180)
except asyncio.TimeoutError:
return {"ok": False, "error": "Install timed out after 180s"}
ok = (proc.returncode == 0)
# Combine stderr + (last lines of stdout) into a single error
# blob when ok=False — some package managers print useful failure
# context to stdout, and a script that exits via `echo ...; exit N`
# without `>&2` would otherwise hand back an empty error string
# and force the frontend to show a bare "HTTP 200".
err_txt = err.decode("utf-8", errors="replace").strip()
out_txt = out.decode("utf-8", errors="replace").strip()
if not ok:
tail_out = out_txt[-500:] if out_txt else ""
combined = err_txt or tail_out or f"exit code {proc.returncode}"
else:
combined = None
return {
"ok": ok,
"exit_code": proc.returncode,
"output": out_txt[-1000:],
"error": combined,
}
@router.post("/api/cookbook/rebuild-engine")
async def rebuild_engine(request: Request):
"""Clear the cached llama.cpp build so the next serve recompiles.
@@ -1308,7 +1673,8 @@ def setup_shell_routes() -> APIRouter:
return {"ok": False, "error": f"Unsupported engine: {engine}"}
host = str(body.get("remote_host") or "").strip()
ssh_port = body.get("ssh_port")
cmd = _llama_cpp_rebuild_cmd()
update_source = bool(body.get("update_source"))
cmd = _llama_cpp_rebuild_cmd(update_source=update_source)
try:
argv = (
(_ssh_base_argv(host, ssh_port) + [cmd])
routes/skills_routes.py
+16 -2
View File
@@ -22,6 +22,16 @@ from core.middleware import require_admin
logger = logging.getLogger(__name__)
# Last-resort verdict extraction from a teacher/verifier model's prose (run when
# JSON parsing fails). `["\'\s:]*` already consumes whitespace, so the original
# trailing `\s*` made two adjacent \s-matching quantifiers that backtrack O(n^2)
# on a `verdict` + whitespace flood in untrusted model output (CodeQL
# py/polynomial-redos). Without it a single unbounded quantifier remains — the
# matched text is identical, and the scan is linear.
_VERDICT_PROSE_RE = re.compile(
r'verdict["\'\s:]*["\']?(pass|needs_work|fail|inconclusive)', re.I
)
class SkillAddRequest(BaseModel):
# New schema (preferred)
@@ -196,7 +206,7 @@ async def _eval_skill_run(skill_md: str, task: str, transcript: str,
# Last resort: pull the verdict keyword straight out of the prose so a
# clearly-decided run isn't thrown away as "unparseable".
if v not in _VERDICTS:
km = _re.search(r'verdict["\'\s:]*\s*["\']?(pass|needs_work|fail|inconclusive)', text, _re.I)
km = _VERDICT_PROSE_RE.search(text)
if km:
v = km.group(1).lower()
if data is None:
@@ -691,8 +701,12 @@ async def _run_skill_test_once(md: str, task: str, url, model, headers, owner) -
{"role": "user", "content": task},
]
try:
# max_tokens explicitly set: passing 0 lets some upstreams (Ollama,
# OpenAI-compat) generate an empty completion, which manifested as
# the skill test returning nothing while chat (which carries its
# preset's max_tokens) worked. 4096 matches the chat default.
async for chunk in stream_agent_loop(url, model, messages, headers=headers,
temperature=0.3, max_tokens=0, max_rounds=8, owner=owner):
temperature=0.3, max_tokens=4096, max_rounds=8, owner=owner):
if not chunk.startswith("data: ") or chunk.strip() == "data: [DONE]":
continue
try:
routes/task_routes.py
+9 -1
View File
@@ -11,6 +11,7 @@ from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel
from core.database import SessionLocal, ScheduledTask, TaskRun
from core.middleware import INTERNAL_TOOL_USER
from core.constants import internal_api_base
from src.auth_helpers import get_current_user
from src.constants import DATA_DIR, EMAIL_URGENCY_CACHE_DIR
@@ -151,6 +152,7 @@ class TaskCreate(BaseModel):
endpoint_url: Optional[str] = None
then_task_id: Optional[str] = None # chain: run this task after success
notifications_enabled: Optional[bool] = None # None lets action-specific defaults apply
character_id: Optional[str] = None # built-in persona id (PERSONAS) — biases output voice
class TaskUpdate(BaseModel):
@@ -171,6 +173,7 @@ class TaskUpdate(BaseModel):
endpoint_url: Optional[str] = None
then_task_id: Optional[str] = None
notifications_enabled: Optional[bool] = None
character_id: Optional[str] = None
def _display_task_name(t: ScheduledTask) -> str:
@@ -203,6 +206,7 @@ def _task_to_dict(t: ScheduledTask, include_last_run_result: bool = False) -> di
"output_target": t.output_target,
"session_id": t.session_id,
"crew_member_id": getattr(t, "crew_member_id", None),
"character_id": getattr(t, "character_id", None),
"model": t.model,
"endpoint_url": t.endpoint_url,
"run_count": t.run_count or 0,
@@ -424,7 +428,7 @@ def setup_task_routes(task_scheduler) -> APIRouter:
# In-process tool-loopback marker — AuthMiddleware validated
# the internal token + loopback client before stamping this,
# so treat as admin-equivalent.
if user == "internal-tool":
if user == INTERNAL_TOOL_USER:
return True
try:
from core.auth import AuthManager
@@ -552,6 +556,7 @@ def setup_task_routes(task_scheduler) -> APIRouter:
then_task_id=then_task_id,
webhook_token=webhook_token,
notifications_enabled=notifications_enabled,
character_id=(req.character_id or None),
)
db.add(task)
db.commit()
@@ -705,6 +710,9 @@ def setup_task_routes(task_scheduler) -> APIRouter:
task.then_task_id = _validate_then_task_id(db, req.then_task_id, user, current_task_id=task.id)
if req.notifications_enabled is not None:
task.notifications_enabled = bool(req.notifications_enabled)
if req.character_id is not None:
# Empty string clears the persona; non-empty stores the id.
task.character_id = req.character_id or None
if req.cron_expression is not None:
if req.cron_expression:
try:
routes/upload_routes.py
+95 -23
View File
@@ -3,11 +3,16 @@ import os
import time
import json
import asyncio
import shutil
import uuid
from pathlib import Path
from fastapi import APIRouter, Request, File, UploadFile, HTTPException
from typing import List
import logging
from core.middleware import require_admin
from src.auth_helpers import get_current_user
from core.database import SessionLocal, GalleryImage
from src.auth_helpers import effective_user
from src.constants import GENERATED_IMAGES_DIR
from src.upload_handler import count_recent_uploads
logger = logging.getLogger(__name__)
@@ -50,6 +55,69 @@ def setup_upload_routes(upload_handler):
raise HTTPException(404, "File not found")
raise HTTPException(404, "File not found")
def _promote_chat_image_to_gallery(meta: dict, owner: str | None) -> str | None:
"""Make chat-uploaded images visible in Gallery without changing chat storage."""
is_image_file = getattr(upload_handler, "is_image_file", None)
if not callable(is_image_file):
return None
if not is_image_file(meta.get("name", ""), meta.get("mime", "")):
return None
source_path = meta.get("path")
if not source_path or not os.path.isfile(source_path):
return None
db = SessionLocal()
try:
file_hash = meta.get("hash")
if file_hash:
q = db.query(GalleryImage).filter(
GalleryImage.file_hash == file_hash,
GalleryImage.is_active == True, # noqa: E712
)
if owner:
q = q.filter(GalleryImage.owner == owner)
existing = q.first()
if existing:
return existing.id
image_dir = Path(GENERATED_IMAGES_DIR)
image_dir.mkdir(parents=True, exist_ok=True)
ext = Path(meta.get("name") or source_path).suffix.lower()
if ext not in {".png", ".jpg", ".jpeg", ".webp", ".gif"}:
mime_ext = {
"image/png": ".png",
"image/jpeg": ".jpg",
"image/jpg": ".jpg",
"image/webp": ".webp",
"image/gif": ".gif",
}.get(meta.get("mime", ""))
ext = mime_ext or ".png"
filename = f"{uuid.uuid4().hex[:12]}{ext}"
dest_path = image_dir / filename
shutil.copy2(source_path, dest_path)
image_id = str(uuid.uuid4())
db.add(GalleryImage(
id=image_id,
filename=filename,
prompt=meta.get("name") or "Chat upload",
model="chat-upload",
owner=owner,
file_hash=file_hash,
width=meta.get("width"),
height=meta.get("height"),
file_size=meta.get("size"),
))
db.commit()
return image_id
except Exception as e:
db.rollback()
logger.warning("Failed to add chat image upload to gallery: %s", e)
return None
finally:
db.close()
@router.post("")
async def api_upload(request: Request, files: List[UploadFile] = File(...)):
@@ -78,8 +146,10 @@ def setup_upload_routes(upload_handler):
for u in files:
try:
meta = upload_handler.save_upload(u, client_ip, owner=get_current_user(request))
out.append({
owner = effective_user(request)
meta = upload_handler.save_upload(u, client_ip, owner=owner)
gallery_id = _promote_chat_image_to_gallery(meta, owner)
item = {
"id": meta["id"],
"name": meta["name"],
"mime": meta["mime"],
@@ -89,7 +159,10 @@ def setup_upload_routes(upload_handler):
"width": meta.get("width"),
"height": meta.get("height"),
"is_duplicate": meta.get("is_duplicate", False)
})
}
if gallery_id:
item["gallery_id"] = gallery_id
out.append(item)
except HTTPException:
raise
except Exception as e:
@@ -128,17 +201,16 @@ def setup_upload_routes(upload_handler):
import mimetypes as _mt
# Look up original filename and owner from uploads.json
original_name = file_id
info = None
uploads_db = os.path.join(_upload_root(), "uploads.json")
if os.path.exists(uploads_db):
with open(uploads_db, encoding="utf-8") as f:
db = json.load(f)
info = next((fi for fi in db.values() if fi.get("id") == file_id), None)
if info:
original_name = info.get("name", file_id)
# _load_upload_index() tolerates a missing/corrupt uploads.json (it falls
# back to the .bak sibling, then to {}), so a truncated DB degrades to
# "no metadata" instead of a 500 from an unhandled JSONDecodeError.
db = upload_handler._load_upload_index()
info = next((fi for fi in db.values() if fi.get("id") == file_id), None)
if info:
original_name = info.get("name", file_id)
auth_mgr = getattr(request.app.state, "auth_manager", None)
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
current_user = get_current_user(request)
current_user = effective_user(request)
file_owner = info.get("owner") if info else None
if auth_configured:
if not current_user:
@@ -181,13 +253,10 @@ def setup_upload_routes(upload_handler):
def _load_upload_info(file_id: str):
"""Look up the uploads.json record for a file_id, with owner/auth checks."""
info = None
uploads_db = os.path.join(_upload_root(), "uploads.json")
if os.path.exists(uploads_db):
with open(uploads_db, encoding="utf-8") as f:
db = json.load(f)
info = next((fi for fi in db.values() if fi.get("id") == file_id), None)
return info
# Corruption-tolerant load (see download_file): a bad uploads.json yields
# {} rather than raising JSONDecodeError out of the vision path.
db = upload_handler._load_upload_index()
return next((fi for fi in db.values() if fi.get("id") == file_id), None)
def _vision_cache_path(file_id: str) -> str:
cache_dir = os.path.join(_upload_root(), ".vision")
@@ -204,7 +273,7 @@ def setup_upload_routes(upload_handler):
info = _load_upload_info(file_id)
auth_mgr = getattr(request.app.state, "auth_manager", None)
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
current_user = get_current_user(request)
current_user = effective_user(request)
file_owner = info.get("owner") if info else None
if auth_configured:
if not current_user:
@@ -247,7 +316,7 @@ def setup_upload_routes(upload_handler):
raise HTTPException(404, "File not found")
auth_mgr = getattr(request.app.state, "auth_manager", None)
auth_configured = bool(auth_mgr and auth_mgr.is_configured)
current_user = get_current_user(request)
current_user = effective_user(request)
file_owner = info.get("owner")
if auth_configured:
if not current_user:
@@ -255,7 +324,10 @@ def setup_upload_routes(upload_handler):
if file_owner != current_user and not auth_mgr.is_admin(current_user):
raise HTTPException(404, "File not found")
_resolve_upload_path(file_id)
body = await request.json()
try:
body = await request.json()
except json.JSONDecodeError:
raise HTTPException(400, "Request body must be valid JSON")
text = (body or {}).get("text", "")
if not isinstance(text, str):
raise HTTPException(400, "text must be a string")
routes/webhook_routes.py
+5 -5
View File
@@ -1,6 +1,5 @@
"""Webhook, API Token, and sync chat routes."""
import asyncio
import uuid
import logging
from typing import Optional
@@ -346,8 +345,9 @@ def setup_webhook_routes(
resp = await client.get(models_url, headers=hdrs)
resp.raise_for_status()
data = resp.json()
ids = [m.get("id") for m in (data.get("data") or []) if m.get("id")]
if not ids:
items = data if isinstance(data, list) else (data.get("data") or [])
ids = [m.get("id") for m in items if isinstance(m, dict) and m.get("id")]
if not ids and isinstance(data, dict):
ids = [
m.get("name") or m.get("model")
for m in (data.get("models") or [])
@@ -385,10 +385,10 @@ def setup_webhook_routes(
sess.add_message(ChatMessage("assistant", reply))
session_manager.save_sessions()
asyncio.create_task(webhook_manager.fire("chat.completed", {
webhook_manager.fire_and_forget("chat.completed", {
"session_id": session_id, "model": sess.model,
"user_message": message[:2000], "response": reply[:2000],
}))
})
return {"response": reply, "session_id": session_id, "model": sess.model}
scripts/backfill_model_release_dates.py
Executable
+133
View File
@@ -0,0 +1,133 @@
#!/usr/bin/env python3
"""Backfill release_date on entries in services/hwfit/data/hf_models.json.
Why: the `newest` sort in the cookbook ranks rows by release_date. Anything
missing a date sorts to the bottom. This script pulls `created_at` from the
HuggingFace API for each catalog entry without one (or all entries when
--refresh is passed) and writes the catalog back.
Usage:
python scripts/backfill_model_release_dates.py # missing only
python scripts/backfill_model_release_dates.py --refresh # all entries
python scripts/backfill_model_release_dates.py --limit 50 # cap requests
python scripts/backfill_model_release_dates.py --dry-run # show, don't write
Auth: set HF_TOKEN env var (or huggingface-cli login) to access gated repos.
"""
import argparse
import json
import os
import sys
import time
from datetime import datetime
from pathlib import Path
try:
from huggingface_hub import HfApi
from huggingface_hub.utils import HfHubHTTPError
except ImportError:
print("Install huggingface_hub: pip install huggingface_hub", file=sys.stderr)
sys.exit(1)
CATALOG_PATH = Path(__file__).resolve().parent.parent / "services" / "hwfit" / "data" / "hf_models.json"
def fetch_release_date(api: HfApi, repo_id: str) -> str | None:
"""Return YYYY-MM-DD release date, or None on miss / error."""
try:
info = api.model_info(repo_id, files_metadata=False)
except HfHubHTTPError as e:
# 401 = gated/private, 404 = renamed/deleted. Either way, no date.
status = getattr(getattr(e, "response", None), "status_code", None)
print(f" {repo_id}: HTTP {status or '?'}", file=sys.stderr)
return None
except Exception as e:
print(f" {repo_id}: {type(e).__name__}: {e}", file=sys.stderr)
return None
created = getattr(info, "created_at", None)
if not created:
return None
return created.strftime("%Y-%m-%d")
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--refresh", action="store_true", help="Overwrite existing release_date too (default: only fill missing).")
p.add_argument("--limit", type=int, default=0, help="Stop after N API calls (0 = no limit).")
p.add_argument("--dry-run", action="store_true", help="Don't write back; just report.")
p.add_argument("--sleep", type=float, default=0.05, help="Seconds to sleep between requests (default 0.05).")
args = p.parse_args()
if not CATALOG_PATH.exists():
print(f"Catalog not found: {CATALOG_PATH}", file=sys.stderr)
sys.exit(2)
with CATALOG_PATH.open(encoding="utf-8") as f:
catalog = json.load(f)
candidates = []
for i, m in enumerate(catalog):
name = m.get("name")
if not name:
continue
existing = (m.get("release_date") or "").strip()
if existing and not args.refresh:
continue
candidates.append(i)
if args.limit:
candidates = candidates[: args.limit]
print(f"Catalog: {CATALOG_PATH}")
print(f"Total entries: {len(catalog)}")
print(f"Targets ({'refresh all' if args.refresh else 'missing only'}{'' if not args.limit else f', capped at {args.limit}'}): {len(candidates)}")
if not candidates:
print("Nothing to do.")
return
api = HfApi(token=os.environ.get("HF_TOKEN") or None)
updated = 0
skipped = 0
started = time.time()
for n, idx in enumerate(candidates, start=1):
entry = catalog[idx]
name = entry["name"]
old = (entry.get("release_date") or "").strip()
new = fetch_release_date(api, name)
if new is None:
skipped += 1
tag = "skip"
elif new == old:
tag = "unchanged"
else:
entry["release_date"] = new
updated += 1
tag = f"set {new}" + (f" (was {old})" if old else "")
print(f"[{n}/{len(candidates)}] {name} — {tag}")
if args.sleep:
time.sleep(args.sleep)
elapsed = time.time() - started
print()
print(f"Done in {elapsed:.1f}s — {updated} updated, {skipped} skipped (HF unavailable / gated / missing date).")
if args.dry_run:
print("Dry run — no write.")
return
if updated:
# Atomic write: tmp file in the same dir, then rename. Keeps the
# catalog usable even if the process dies mid-write.
tmp = CATALOG_PATH.with_suffix(".json.tmp")
with tmp.open("w", encoding="utf-8") as f:
json.dump(catalog, f, indent=1, ensure_ascii=False)
f.write("\n")
tmp.replace(CATALOG_PATH)
print(f"Wrote {CATALOG_PATH}")
else:
print("No changes to write.")
if __name__ == "__main__":
main()
scripts/claim_ownerless.py
+8 -2
View File
@@ -27,12 +27,18 @@ def claim_json_entries(entries, owner):
return count
def owner_arg(argv):
if len(argv) < 2 or not argv[1].strip():
return None
return argv[1].strip()
def main():
if len(sys.argv) < 2:
owner = owner_arg(sys.argv)
if not owner:
print("Usage: python scripts/claim_ownerless.py <username>")
sys.exit(1)
owner = sys.argv[1]
print(f"Claiming all ownerless data for: {owner}\n")
# 1. Memories (JSON files)
scripts/import_from_vllm_recipes.py
Executable
+341
View File
@@ -0,0 +1,341 @@
#!/usr/bin/env python3
"""Import models from the upstream vllm-project/recipes catalog into our
local hf_models.json. Two modes:
--update-existing Stamp min_vllm_version + vllm_recipe=True on rows we
already carry. Cheap, no HF API calls.
--add-missing Create new catalog rows for every recipe model we
don't carry. Hits the HF API for created_at + downloads
(~1 req per missing model, paced).
Both modes write atomically (tmp + rename) so a crashed run leaves the
catalog intact. Default with no mode flags runs both, prefer to pass them
explicitly.
Usage:
python scripts/import_from_vllm_recipes.py --update-existing
python scripts/import_from_vllm_recipes.py --add-missing
python scripts/import_from_vllm_recipes.py --dry-run
python scripts/import_from_vllm_recipes.py --limit 10
Auth: set HF_TOKEN to access gated repos when --add-missing.
"""
import argparse
import json
import os
import re
import sys
import time
from datetime import datetime
from pathlib import Path
try:
import httpx
import yaml
except ImportError:
print("pip install httpx PyYAML", file=sys.stderr)
sys.exit(1)
try:
from huggingface_hub import HfApi
from huggingface_hub.utils import HfHubHTTPError
except ImportError:
HfApi = None
HfHubHTTPError = Exception
CATALOG_PATH = Path(__file__).resolve().parent.parent / "services" / "hwfit" / "data" / "hf_models.json"
RECIPES_TREE_URL = (
"https://api.github.com/repos/vllm-project/recipes/git/trees/main?recursive=1"
)
RECIPE_RAW_URL = (
"https://raw.githubusercontent.com/vllm-project/recipes/main/models/{repo}.yaml"
)
# Map recipe `precision` to the closest catalog `quantization` label that
# fit.py / models.py already understand.
_PRECISION_TO_QUANT = {
"fp8": "FP8",
"nvfp4": "NVFP4",
"mxfp4": "MXFP4",
"bf16": "BF16",
"fp16": "F16",
"f16": "F16",
"fp4": "FP4",
"int8": "INT8",
"int4": "INT4",
"awq-4bit": "AWQ-4bit",
"awq-8bit": "AWQ-8bit",
}
# Architecture name → use_case fallback. fit.py weights use_case for filtering;
# missing field defaults to a generic bucket.
_ARCH_USE_CASE = {
"moe": "General-purpose reasoning, long-context",
"llama": "General-purpose chat",
"qwen2": "General-purpose chat",
"qwen3": "General-purpose reasoning",
"deepseek_v3_moe": "General-purpose reasoning, long-context",
"deepseek_v4_moe": "General-purpose reasoning, long-context",
}
def _parse_param_count(s) -> int:
"""'230B' / '8.6B' / '4.2T' → integer parameter count."""
if s is None:
return 0
s = str(s).strip().replace(",", "")
m = re.match(r"^([\d.]+)\s*([KMBT]?)$", s, re.I)
if not m:
return 0
num = float(m.group(1))
unit = (m.group(2) or "").upper()
mult = {"K": 1e3, "M": 1e6, "B": 1e9, "T": 1e12, "": 1.0}[unit]
return int(num * mult)
def _capabilities_for(arch: str, hardware: dict, ctx_len: int, has_reasoning: bool) -> list[str]:
caps = []
if "moe" in (arch or "").lower():
caps.append("moe")
if has_reasoning:
caps.append("reasoning")
if ctx_len and ctx_len >= 100_000:
caps.append("long_context")
if any(hw in (hardware or {}) for hw in ("mi300x", "mi325x", "mi350x", "mi355x")):
caps.append("amd_supported")
return caps
def _fetch_manifest(client: httpx.Client) -> set[str]:
r = client.get(RECIPES_TREE_URL, headers={"Accept": "application/vnd.github+json"}, timeout=15)
r.raise_for_status()
tree = (r.json() or {}).get("tree") or []
out: set[str] = set()
for e in tree:
path = (e or {}).get("path") or ""
if path.startswith("models/") and path.endswith(".yaml"):
body = path[len("models/"):-len(".yaml")]
if "/" in body:
out.add(body)
return out
def _fetch_recipe(client: httpx.Client, repo: str) -> dict | None:
url = RECIPE_RAW_URL.format(repo=repo)
try:
r = client.get(url, timeout=10)
if r.status_code != 200:
return None
return yaml.safe_load(r.text) or {}
except Exception:
return None
def _stamp_from_recipe(entry: dict, recipe: dict) -> bool:
"""Mutate entry with recipe-derived fields. Returns True if anything changed."""
model = recipe.get("model") or {}
meta = recipe.get("meta") or {}
features = recipe.get("features") or {}
changed = False
new_min = (model.get("min_vllm_version") or "").strip()
if new_min and entry.get("min_vllm_version") != new_min:
entry["min_vllm_version"] = new_min
changed = True
if not entry.get("vllm_recipe"):
entry["vllm_recipe"] = True
changed = True
# Hardware support map — useful for filtering "which models run on my AMD box".
hw = meta.get("hardware") or {}
if hw and entry.get("recipe_hardware") != hw:
entry["recipe_hardware"] = {k: str(v) for k, v in hw.items()}
changed = True
# Tool/reasoning parser hints — purely informational at catalog level;
# the live launch command builder still reads them from the recipe API.
if features.get("reasoning") and not entry.get("has_reasoning_parser"):
entry["has_reasoning_parser"] = True
changed = True
if features.get("tool_calling") and not entry.get("has_tool_call_parser"):
entry["has_tool_call_parser"] = True
changed = True
return changed
def _build_new_entry(repo: str, recipe: dict, hf_info=None) -> dict | None:
"""Build a fresh catalog entry from a recipe + (optional) HF model info."""
model = recipe.get("model") or {}
meta = recipe.get("meta") or {}
features = recipe.get("features") or {}
variants = recipe.get("variants") or {}
org, name = repo.split("/", 1)
raw_params = _parse_param_count(model.get("parameter_count"))
active_raw = _parse_param_count(model.get("active_parameters"))
ctx = model.get("context_length") or 0
# Pick the smallest-VRAM variant as the catalog quant — that's what most
# users land on first. NVFP4/MXFP4 typically win this on Blackwell;
# FP8 elsewhere; BF16 baseline only.
pick_quant = None
pick_vram = None
for vk, vv in variants.items():
if not isinstance(vv, dict):
continue
prec = (vv.get("precision") or "").lower()
vram = vv.get("vram_minimum_gb") or 0
quant = _PRECISION_TO_QUANT.get(prec)
if quant and (pick_vram is None or (vram and vram < pick_vram)):
pick_quant = quant
pick_vram = vram or pick_vram
if not pick_quant:
pick_quant = "BF16"
arch = (model.get("architecture") or "").lower()
use_case = _ARCH_USE_CASE.get(arch, "General-purpose chat")
caps = _capabilities_for(arch, meta.get("hardware") or {}, ctx, bool(features.get("reasoning")))
rel_date = ""
downloads = 0
likes = 0
if hf_info is not None:
created = getattr(hf_info, "created_at", None)
if created:
rel_date = created.strftime("%Y-%m-%d")
downloads = int(getattr(hf_info, "downloads", 0) or 0)
likes = int(getattr(hf_info, "likes", 0) or 0)
if not rel_date:
rel_date = str(meta.get("date_updated") or datetime.utcnow().strftime("%Y-%m-%d"))
entry: dict = {
"name": repo,
"provider": org,
"parameter_count": str(model.get("parameter_count") or "?"),
"parameters_raw": raw_params,
"is_moe": "moe" in arch,
"quantization": pick_quant,
"context_length": int(ctx or 0),
"use_case": use_case,
"capabilities": caps,
"pipeline_tag": "text-generation",
"architecture": arch or "unknown",
"hf_downloads": downloads,
"hf_likes": likes,
"release_date": rel_date,
# Recipe-derived bits.
"vllm_recipe": True,
"min_vllm_version": (model.get("min_vllm_version") or "").strip() or None,
"recipe_hardware": {k: str(v) for k, v in (meta.get("hardware") or {}).items()},
"has_reasoning_parser": bool(features.get("reasoning")),
"has_tool_call_parser": bool(features.get("tool_calling")),
}
if active_raw:
entry["active_parameters"] = active_raw
if pick_vram:
# min_vram_gb is what hwfit uses for "does this fit". Recipe states a
# minimum for the chosen variant; round up slightly for KV-cache room.
entry["min_vram_gb"] = float(pick_vram)
entry["min_ram_gb"] = float(round(pick_vram * 0.6, 1))
entry["recommended_ram_gb"] = float(round(pick_vram * 1.2, 1))
# Drop empty / None fields to keep the JSON tidy.
return {k: v for k, v in entry.items() if v not in (None, "", [], {})}
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--update-existing", action="store_true", help="Stamp min_vllm_version + vllm_recipe on existing rows.")
p.add_argument("--add-missing", action="store_true", help="Add new rows for recipe models not in the catalog.")
p.add_argument("--limit", type=int, default=0, help="Stop after N recipe fetches.")
p.add_argument("--dry-run", action="store_true", help="Don't write back; just report.")
p.add_argument("--sleep", type=float, default=0.05, help="Seconds between HTTP requests.")
args = p.parse_args()
if not args.update_existing and not args.add_missing:
args.update_existing = args.add_missing = True
with CATALOG_PATH.open(encoding="utf-8") as f:
catalog = json.load(f)
by_name = {m.get("name"): m for m in catalog if m.get("name")}
client = httpx.Client(follow_redirects=True)
print(f"Catalog: {CATALOG_PATH} ({len(catalog)} entries)")
print("Fetching upstream manifest…")
try:
manifest = _fetch_manifest(client)
except Exception as e:
print(f"FATAL: manifest fetch failed: {e}", file=sys.stderr)
sys.exit(2)
print(f"Manifest: {len(manifest)} recipes")
existing = sorted(by_name.keys() & manifest)
missing = sorted(manifest - by_name.keys())
print(f"Match catalog ↔ manifest: existing={len(existing)} missing={len(missing)}")
targets: list[tuple[str, str]] = [] # (repo, action)
if args.update_existing:
targets.extend((r, "update") for r in existing)
if args.add_missing:
targets.extend((r, "add") for r in missing)
if args.limit:
targets = targets[: args.limit]
print(f"Targets: {len(targets)}")
hf_api = HfApi(token=os.environ.get("HF_TOKEN") or None) if HfApi else None
updated = added = skipped = 0
started = time.time()
for n, (repo, action) in enumerate(targets, 1):
recipe = _fetch_recipe(client, repo)
if not recipe:
print(f"[{n}/{len(targets)}] {repo:55} skip (no recipe fetched)")
skipped += 1
time.sleep(args.sleep)
continue
if action == "update":
entry = by_name[repo]
if _stamp_from_recipe(entry, recipe):
updated += 1
print(f"[{n}/{len(targets)}] {repo:55} updated")
else:
print(f"[{n}/{len(targets)}] {repo:55} unchanged")
else: # add
hf_info = None
if hf_api:
try:
hf_info = hf_api.model_info(repo, files_metadata=False)
except HfHubHTTPError as e:
code = getattr(getattr(e, "response", None), "status_code", "?")
print(f" HF {code} for {repo} — building from recipe only", file=sys.stderr)
except Exception as e:
print(f" HF error for {repo}: {e}", file=sys.stderr)
new_entry = _build_new_entry(repo, recipe, hf_info)
if new_entry:
catalog.append(new_entry)
by_name[repo] = new_entry
added += 1
print(f"[{n}/{len(targets)}] {repo:55} added ({new_entry.get('parameter_count','?')}, {new_entry.get('quantization','?')})")
else:
skipped += 1
print(f"[{n}/{len(targets)}] {repo:55} skip (couldn't build entry)")
time.sleep(args.sleep)
elapsed = time.time() - started
print()
print(f"Done in {elapsed:.1f}s — added={added}, updated={updated}, skipped={skipped}")
if args.dry_run:
print("Dry run — no write.")
return
if added or updated:
tmp = CATALOG_PATH.with_suffix(".json.tmp")
with tmp.open("w", encoding="utf-8") as f:
json.dump(catalog, f, indent=1, ensure_ascii=False)
f.write("\n")
tmp.replace(CATALOG_PATH)
print(f"Wrote {CATALOG_PATH} ({len(catalog)} entries)")
else:
print("No changes — catalog untouched.")
if __name__ == "__main__":
main()
scripts/odysseus-calendar
+5 -1
View File
@@ -103,9 +103,13 @@ def cmd_list(args) -> None:
end = _parse_dt(args.end) if args.end else (start + timedelta(days=30))
db = SessionLocal()
try:
# Overlap semantics, matching the web route (routes/calendar_routes.py)
# and the recurring-expansion contract: an event is in the window when
# it starts before the window end AND ends after the window start. This
# includes multi-day / in-progress events that began before `start`.
q = db.query(CalendarEvent).filter(
CalendarEvent.dtstart >= start,
CalendarEvent.dtstart < end,
CalendarEvent.dtend > start,
)
if args.calendar:
cal = db.query(CalendarCal).filter(CalendarCal.name == args.calendar).first()
services/hwfit/data/hf_models.json
+133 -1
View File
@@ -14059,6 +14059,138 @@
"vision"
]
},
{
"name": "google/gemma-4-12B-it",
"provider": "Google",
"parameter_count": "12.0B",
"parameters_raw": 12000000000,
"min_ram_gb": 8.5,
"recommended_ram_gb": 11.0,
"min_vram_gb": 7.5,
"quantization": "Q4_K_M",
"context_length": 131072,
"use_case": "General purpose, multimodal; unsloth/gemma-4-12B-it-GGUF Dynamic variants reduce VRAM from ~7.5 GB to ~5.5 GB",
"is_moe": false,
"num_experts": null,
"active_experts": null,
"active_parameters": null,
"architecture": "gemma4",
"pipeline_tag": "image-text-to-text",
"release_date": "2026-04-01",
"gguf_sources": [
{
"repo": "unsloth/gemma-4-12B-it-GGUF",
"provider": "unsloth"
}
],
"capabilities": [
"vision"
]
},
{
"name": "google/gemma-4-12B-it-qat-int4",
"provider": "Google",
"parameter_count": "12.0B",
"parameters_raw": 12000000000,
"min_ram_gb": 8.0,
"recommended_ram_gb": 9.5,
"min_vram_gb": 6.5,
"quantization": "QAT-INT4",
"context_length": 131072,
"use_case": "General purpose, multimodal (QAT quantization-aware training — higher quality than post-train INT4; vLLM native; no GGUF)",
"is_moe": false,
"num_experts": null,
"active_experts": null,
"active_parameters": null,
"architecture": "gemma4",
"pipeline_tag": "image-text-to-text",
"release_date": "2026-04-01",
"gguf_sources": [],
"capabilities": [
"vision"
]
},
{
"name": "google/gemma-4-12B-it-qat-int8",
"provider": "Google",
"parameter_count": "12.0B",
"parameters_raw": 12000000000,
"min_ram_gb": 15.0,
"recommended_ram_gb": 20.0,
"min_vram_gb": 13.5,
"quantization": "QAT-INT8",
"context_length": 131072,
"use_case": "General purpose, multimodal (QAT INT8 — highest quality, 2x VRAM of QAT-INT4; vLLM native; no GGUF)",
"is_moe": false,
"num_experts": null,
"active_experts": null,
"active_parameters": null,
"architecture": "gemma4",
"pipeline_tag": "image-text-to-text",
"release_date": "2026-04-01",
"gguf_sources": [],
"capabilities": [
"vision"
]
},
{
"name": "google/gemma-4-12B-it-qat-q4_0-gguf",
"provider": "Google",
"parameter_count": "12.0B",
"parameters_raw": 12000000000,
"min_ram_gb": 8.5,
"recommended_ram_gb": 11.0,
"min_vram_gb": 7.5,
"quantization": "QAT-INT4",
"context_length": 262144,
"use_case": "General purpose, multimodal (vision + audio); official Google QAT int4 GGUF — near-bf16 quality at int4 size, served on llama.cpp/Ollama with CPU offload",
"is_moe": false,
"num_experts": null,
"active_experts": null,
"active_parameters": null,
"architecture": "gemma4",
"pipeline_tag": "image-text-to-text",
"release_date": "2026-04-01",
"gguf_sources": [
{
"repo": "google/gemma-4-12B-it-qat-q4_0-gguf",
"provider": "Google",
"file": "gemma-4-12b-it-qat-q4_0.gguf"
}
],
"capabilities": [
"vision",
"audio"
]
},
{
"name": "google/gemma-4-26B-A4B-it-qat-q4_0-gguf",
"provider": "Google",
"parameter_count": "25.2B",
"parameters_raw": 25200000000,
"min_ram_gb": 14.4,
"recommended_ram_gb": 18.0,
"min_vram_gb": 14.4,
"quantization": "QAT-INT4",
"context_length": 262144,
"use_case": "High-throughput, multimodal MoE (3.8B active); official Google QAT int4 GGUF — near-bf16 quality at int4 size, served on llama.cpp with CPU offload",
"is_moe": true,
"num_experts": null,
"active_experts": null,
"active_parameters": 3800000000,
"architecture": "gemma4",
"pipeline_tag": "image-text-to-text",
"release_date": "2026-04-01",
"gguf_sources": [
{
"repo": "google/gemma-4-26B-A4B-it-qat-q4_0-gguf",
"provider": "Google"
}
],
"capabilities": [
"vision"
]
},
{
"name": "google/gemma-4-31B-it",
"provider": "Google",
@@ -19144,4 +19276,4 @@
],
"_discovered": true
}
]
]
services/hwfit/fit.py
+117 -14
View File
@@ -9,7 +9,7 @@ from services.hwfit.models import (
GPU_BANDWIDTH = {
"5090": 1792, "5080": 960, "5070 ti": 896, "5070": 672, "5060 ti": 448, "5060": 256,
"4090": 1008, "4080 super": 736, "4080": 717, "4070 ti super": 672, "4070 ti": 504, "4070 super": 504, "4070": 504, "4060 ti": 288, "4060": 272,
"3090 ti": 1008, "3090": 936, "3080 ti": 912, "3080": 760, "3070 ti": 608, "3070": 448, "3060 ti": 448, "3060": 360,
"3090 ti": 1008, "3090": 936, "3080 ti": 912, "3080": 760, "3070 ti": 608, "3070": 448, "3060 ti": 448, "3060": 360, "3050 ti": 192, "3050": 224,
"2080 ti": 616, "2080 super": 496, "2080": 448, "2070 super": 448, "2070": 448, "2060 super": 448, "2060": 336,
"1660 ti": 288, "1660 super": 336, "1660": 192, "1650 super": 192, "1650": 128,
"h100 sxm": 3350, "h100": 2039, "h200": 4800, "a100 sxm": 2039, "a100": 1555,
@@ -19,22 +19,36 @@ GPU_BANDWIDTH = {
"6950 xt": 576, "6900 xt": 512, "6800 xt": 512, "6800": 512, "6700 xt": 384, "6600 xt": 256, "6600": 224,
"mi300x": 5300, "mi300": 5300, "mi250x": 3277, "mi250": 3277, "mi210": 1638, "mi100": 1229,
"9070 xt": 624, "9070": 488, "9060 xt": 322, "9060": 322,
# Apple Silicon unified-memory bandwidth (GB/s). Keyed off the chip name
# reported by sysctl machdep.cpu.brand_string (e.g. "Apple M4 Max"). Listed
# before the bare "m_" keys matters less than length-sorting (done below),
# which guarantees "m4 max" is tried before "m4".
"m1 ultra": 800, "m1 max": 400, "m1 pro": 200, "m1": 68,
"m2 ultra": 800, "m2 max": 400, "m2 pro": 200, "m2": 100,
"m3 ultra": 800, "m3 max": 300, "m3 pro": 150, "m3": 100,
"m4 max": 546, "m4 pro": 273, "m4": 120,
"m5 max": 546, "m5 pro": 273, "m5": 150,
# NVIDIA GB10 Grace-Blackwell superchip (DGX Spark). Unified LPDDR5X memory,
# not Apple Silicon, so it lives in the generic GPU table — the Apple-only
# lookup never matches it (its name carries no "apple").
"gb10": 273,
}
# Pre-sort keys by length descending for correct substring matching
_BW_KEYS_SORTED = sorted(GPU_BANDWIDTH.keys(), key=len, reverse=True)
# metal: backstop for Apple Silicon chips not in GPU_BANDWIDTH (e.g. a future
# M5) — the named chips above take the accurate bandwidth path instead.
# Apple Silicon unified-memory bandwidth (GB/s). For chip families with both
# binned and full variants under the same "Apple Mx Max" brand string, prefer
# GPU core count when hardware detection provides it; otherwise fall back to the
# conservative tier so speed estimates do not over-promise.
APPLE_BANDWIDTH_FIXED = {
"m1 ultra": 800, "m1 max": 400, "m1 pro": 200, "m1": 68,
"m2 ultra": 800, "m2 max": 400, "m2 pro": 200, "m2": 100,
"m3 ultra": 800, "m3 pro": 150, "m3": 100,
"m4 pro": 273, "m4": 120,
"m5 pro": 307, "m5": 153,
}
APPLE_BANDWIDTH_BY_CORES = {
"m3 max": {30: 300, 40: 400},
"m4 max": {32: 410, 40: 546},
"m5 max": {32: 460, 40: 614},
}
_APPLE_FIXED_KEYS_SORTED = sorted(APPLE_BANDWIDTH_FIXED.keys(), key=len, reverse=True)
_APPLE_VARIANT_KEYS_SORTED = sorted(APPLE_BANDWIDTH_BY_CORES.keys(), key=len, reverse=True)
# metal: backstop for Apple Silicon chips not in the explicit tables above
# (e.g. a future M6) — use a conservative generic estimate when unknown.
FALLBACK_K = {"cuda": 220, "rocm": 180, "metal": 150, "cpu_x86": 70, "cpu_arm": 90}
USE_CASE_WEIGHTS = {
@@ -60,16 +74,100 @@ CONTEXT_TARGET = {
}
def _lookup_bandwidth(gpu_name):
def _lookup_apple_bandwidth(system):
gpu_name = system.get("gpu_name")
if not isinstance(gpu_name, str) or not gpu_name:
return None
gn = gpu_name.lower()
# Guard against false matches on non-Apple GPUs whose names contain
# "m3"/"m4"/"m5" (e.g. NVIDIA Quadro M4 000).
if "apple" not in gn:
return None
raw_cores = system.get("gpu_cores")
try:
gpu_cores = int(raw_cores) if raw_cores is not None else None
except (TypeError, ValueError):
gpu_cores = None
for key in _APPLE_VARIANT_KEYS_SORTED:
if key not in gn:
continue
if gpu_cores in APPLE_BANDWIDTH_BY_CORES[key]:
return APPLE_BANDWIDTH_BY_CORES[key][gpu_cores]
return min(APPLE_BANDWIDTH_BY_CORES[key].values())
for key in _APPLE_FIXED_KEYS_SORTED:
if key in gn:
return APPLE_BANDWIDTH_FIXED[key]
return None
def _lookup_bandwidth(system):
if isinstance(system, dict):
gpu_name = system.get("gpu_name")
else:
gpu_name = system
if not isinstance(gpu_name, str) or not gpu_name:
return None
# Apple tiers live only in the Apple-specific table now (#2564), so route
# BOTH dict and bare-string callers through it. A bare string carries no
# gpu_cores, so the helper falls back to the conservative (lowest) tier for
# that model -- before #2564 the generic table answered string lookups, and
# dropping that made _lookup_bandwidth("Apple M3 Max") return None.
apple_input = system if isinstance(system, dict) else {"gpu_name": gpu_name}
bw = _lookup_apple_bandwidth(apple_input)
if bw is not None:
return bw
gn = gpu_name.lower()
for key in _BW_KEYS_SORTED:
if key in gn:
return GPU_BANDWIDTH[key]
return None
def _canonical_cpu_backend(system):
"""Return the canonical CPU backend for cpu_only speed estimation.
Normalizes CPU-architecture aliases separately from the GPU backend, and
overrides GPU-only backends (CUDA/ROCm/Metal) so they do not inherit a
discrete-GPU fallback constant when the model is actually running on CPU.
"""
backend = (system.get("backend") or "").lower().strip()
cpu_arch = (system.get("cpu_arch") or "").lower().strip()
cpu_name = (system.get("cpu_name") or "").lower()
gpu_name = (system.get("gpu_name") or "").lower()
# Already-canonical CPU backends
if backend in ("cpu_x86", "cpu_arm"):
return backend
# Raw CPU-architecture aliases. Treat plain "arm" as 32-bit ARM, not the
# ARM64-class CPU fallback used for Apple Silicon/aarch64 machines.
if backend in ("x86_64", "amd64", "i386", "i686"):
return "cpu_x86"
if backend in ("arm64", "aarch64"):
return "cpu_arm"
# Prefer an explicit CPU architecture field when present
if cpu_arch:
if cpu_arch in ("x86_64", "amd64", "x86", "i386", "i686"):
return "cpu_x86"
if cpu_arch in ("arm64", "aarch64"):
return "cpu_arm"
# Apple Silicon enters ranking as backend="metal"; its CPU path is ARM.
if backend in ("metal", "mps", "apple") or "apple" in cpu_name or "apple" in gpu_name:
return "cpu_arm"
# Conservative default for CUDA/ROCm/discrete GPU backends and unknowns.
return "cpu_x86"
def _estimate_speed(model, quant, run_mode, system, offload_frac=0.0):
"""Estimate tok/s. Uses active params for MoE (only active experts run per token).
@@ -84,9 +182,14 @@ def _estimate_speed(model, quant, run_mode, system, offload_frac=0.0):
"""
pb = _active_params_b(model)
is_moe = model.get("is_moe", False)
bw = _lookup_bandwidth(system.get("gpu_name"))
bw = _lookup_bandwidth(system)
backend = system.get("backend", "cpu_x86")
# CPU-only inference must never inherit a GPU backend's fallback constant,
# even if the detected system happens to report a CUDA/Metal/ROCm backend.
if run_mode == "cpu_only":
backend = _canonical_cpu_backend(system)
if bw and run_mode in ("gpu", "cpu_offload"):
bpp = QUANT_BYTES_PER_PARAM.get(quant, 0.5)
model_gb = pb * bpp
services/hwfit/hardware.py
+111 -14
View File
@@ -1,3 +1,4 @@
import json
import os
import platform
import re
@@ -281,7 +282,17 @@ def _detect_amd():
"gpus": cards,
"gpu_groups": groups,
"homogeneous": len(groups) <= 1,
"backend": "rocm",
# Pick the actual runtime label: ROCm/HIP only when its
# toolchain is installed, otherwise Vulkan if vulkaninfo is
# present (mesa RADV works fine on RDNA/CDNA when ROCm
# packages are absent — see Strix Halo where ROCm support
# is still backporting). Reporting "rocm" on a Vulkan-only
# host misleads downstream env-var pinning
# (HIP_VISIBLE_DEVICES is a no-op there).
"backend": (
"rocm" if (_run(["which", "rocminfo"]) or _run(["which", "hipconfig"]))
else ("vulkan" if _run(["which", "vulkaninfo"]) else "rocm")
),
"unified_memory": is_apu,
# AMD ISA/family so downstream can tell datacenter Instinct (CDNA,
# where vLLM/SGLang run AWQ/GPTQ reliably) from consumer Radeon
@@ -319,7 +330,7 @@ def _detect_apple_silicon():
# Only Apple Silicon (arm64) has a Metal GPU worth serving LLMs on; Intel
# Macs fall through to the CPU path.
if "arm" not in arch and "aarch64" not in arch:
if _canonical_cpu_arch(arch) != "arm64":
return None
# Chip name, e.g. "Apple M4 Max" — carries the Pro/Max/Ultra variant that
@@ -335,6 +346,37 @@ def _detect_apple_silicon():
if total_gb <= 0:
return None
def _parse_apple_gpu_cores(text):
if not text:
return None
try:
data = json.loads(text)
except (TypeError, ValueError, json.JSONDecodeError):
data = None
if isinstance(data, dict):
for gpu in data.get("SPDisplaysDataType") or []:
if not isinstance(gpu, dict):
continue
model = str(gpu.get("sppci_model") or gpu.get("_name") or "")
if "apple" not in model.lower():
continue
cores = gpu.get("sppci_cores")
try:
return int(str(cores).strip())
except (TypeError, ValueError):
continue
m = re.search(r"Total Number of Cores:\s*(\d+)", text)
if m:
try:
return int(m.group(1))
except ValueError:
return None
return None
gpu_cores = _parse_apple_gpu_cores(_run(["system_profiler", "SPDisplaysDataType", "-json"]))
if gpu_cores is None:
gpu_cores = _parse_apple_gpu_cores(_run(["system_profiler", "SPDisplaysDataType"]))
# Usable GPU budget. macOS lets Metal use most of unified memory, but the
# default working-set limit scales with RAM: small machines have to keep
# more back for the OS + app. These fractions track Apple's
@@ -357,7 +399,7 @@ def _detect_apple_silicon():
pass
gpu = {"index": 0, "name": brand, "vram_gb": vram_gb}
return {
info = {
"gpu_name": brand,
"gpu_vram_gb": vram_gb,
"gpu_count": 1,
@@ -369,6 +411,9 @@ def _detect_apple_silicon():
# separate pool — downstream fit logic uses this to avoid double-budgeting.
"unified_memory": True,
}
if gpu_cores is not None:
info["gpu_cores"] = gpu_cores
return info
def _read_file(path):
@@ -468,12 +513,57 @@ def _get_cpu_count():
return os.cpu_count() or 1
def _canonical_cpu_arch(value):
arch = str(value or "").lower().strip().replace("-", "_")
if arch in ("x86_64", "amd64", "x64"):
return "x86_64"
if arch in ("i386", "i686", "x86"):
return "x86"
if arch in ("arm64", "aarch64"):
return "arm64"
if arch == "arm" or arch.startswith("armv"):
return "arm"
return arch
def _get_cpu_arch():
if _remote_host:
return _canonical_cpu_arch(_run(["uname", "-m"]) or "")
return _canonical_cpu_arch(platform.machine())
def _powershell_exe():
"""Pick the best PowerShell executable for LOCAL execution: prefer pwsh
(PowerShell 7+), fall back to Windows PowerShell 5.1. Returns an absolute
path so we don't depend on a particular PATH ordering."""
return shutil.which("pwsh") or shutil.which("powershell") or "powershell"
def _powershell_encoded_for_ssh(script: str):
"""Run a PowerShell script on a remote Windows host over SSH.
Nested quotes in powershell -Command break when passed through Windows
OpenSSH's cmd wrapper; -EncodedCommand avoids that.
"""
import base64
encoded = base64.b64encode(script.encode("utf-16-le")).decode("ascii")
return _run(f"powershell -NoProfile -EncodedCommand {encoded}")
def _probe_remote_platform():
"""Best-effort OS detection over SSH when the caller didn't pass platform."""
out = _run("echo %OS%")
if out and "Windows_NT" in out:
return "windows"
uname = (_run(["uname", "-s"]) or "").strip().lower()
if uname == "darwin":
# Mac uses the linux detection path (_detect_apple_silicon over SSH).
return "linux"
if uname == "linux":
out = _run("test -d /data/data/com.termux && echo termux || echo linux")
if out and "termux" in out:
return "termux"
return "linux"
def _detect_windows():
"""Detect Windows hardware via PowerShell/WMI.
@@ -493,6 +583,7 @@ def _detect_windows():
$r.cpu_name = $cpu.Name
$r.cpu_cores = (Get-CimInstance Win32_Processor | Measure-Object -Property NumberOfLogicalProcessors -Sum).Sum
$r.arch = $cpu.AddressWidth
$r.cpu_arch = if ($env:PROCESSOR_ARCHITEW6432) { $env:PROCESSOR_ARCHITEW6432 } else { $env:PROCESSOR_ARCHITECTURE }
# GPU detection via nvidia-smi (fastest) or WMI fallback
try {
$nv = nvidia-smi --query-gpu=memory.total,name --format=csv,noheader,nounits 2>$null
@@ -535,9 +626,8 @@ def _detect_windows():
"""
)
if _remote_host:
# Remote: ship a single command string over SSH. The remote shell parses
# the quoting; PowerShell on the far side runs the -Command payload.
out = _run(f'powershell -Command "{ps_cmd}"')
# Remote: use -EncodedCommand so OpenSSH/cmd quoting does not break the script.
out = _powershell_encoded_for_ssh(ps_cmd.strip())
else:
# Local: pass a LIST argv straight to subprocess so the OS hands ps_cmd
# to PowerShell verbatim — no fragile string-level quote escaping. Prefer
@@ -564,6 +654,7 @@ def _detect_windows():
"available_ram_gb": d.get("avail_gb", 0),
"cpu_cores": _as_int(d.get("cpu_cores"), 1),
"cpu_name": _cpu_name,
"cpu_arch": _canonical_cpu_arch(d.get("cpu_arch")),
"has_gpu": bool(d.get("gpu_name")),
"gpu_name": d.get("gpu_name"),
"gpu_vram_gb": d.get("gpu_vram_gb"),
@@ -707,6 +798,13 @@ def detect_system(host="", ssh_port="", platform="", fresh=False):
"""
global _remote_host, _remote_port, _remote_platform
if host and not platform:
_remote_host = host
_remote_port = ssh_port or None
platform = _probe_remote_platform()
_remote_host = None
_remote_port = None
cache_key = _cache_key(host, ssh_port, platform)
now = time.time()
if not fresh and cache_key in _cache_by_host:
@@ -727,8 +825,8 @@ def detect_system(host="", ssh_port="", platform="", fresh=False):
_remote_platform = None
_cache_by_host[cache_key] = (now, result)
return result
# If Windows detection failed, return error
result = {"error": f"Cannot connect to {host}", "host": host}
# SSH may work while the PowerShell hardware probe still fails.
result = {"error": f"Windows hardware probe failed for {host}", "host": host}
_remote_host = None
_remote_platform = None
_cache_by_host[cache_key] = (now, result)
@@ -759,6 +857,7 @@ def detect_system(host="", ssh_port="", platform="", fresh=False):
available_ram = round(_get_available_ram_gb(), 1)
cpu_cores = _get_cpu_count()
cpu_name = _get_cpu_name()
cpu_arch = _get_cpu_arch()
gpu_info = _detect_apple_silicon() or _detect_nvidia() or _detect_amd()
@@ -768,10 +867,12 @@ def detect_system(host="", ssh_port="", platform="", fresh=False):
"available_ram_gb": available_ram,
"cpu_cores": cpu_cores,
"cpu_name": cpu_name,
"cpu_arch": cpu_arch,
"has_gpu": True,
"gpu_name": gpu_info["gpu_name"],
"gpu_vram_gb": gpu_info["gpu_vram_gb"],
"gpu_count": gpu_info["gpu_count"],
"gpu_cores": gpu_info.get("gpu_cores"),
"gpus": gpu_info.get("gpus", []),
"gpu_groups": gpu_info.get("gpu_groups", []),
"homogeneous": gpu_info.get("homogeneous", True),
@@ -781,17 +882,13 @@ def detect_system(host="", ssh_port="", platform="", fresh=False):
"unified_memory": gpu_info.get("unified_memory", False),
}
else:
if _remote_host:
arch_out = _run(["uname", "-m"]) or ""
else:
import platform as _platform
arch_out = _platform.machine().lower()
backend = "cpu_arm" if "aarch64" in arch_out or "arm" in arch_out else "cpu_x86"
backend = "cpu_arm" if cpu_arch == "arm64" else "cpu_x86"
result = {
"total_ram_gb": total_ram,
"available_ram_gb": available_ram,
"cpu_cores": cpu_cores,
"cpu_name": cpu_name,
"cpu_arch": cpu_arch,
"has_gpu": False,
"gpu_name": None,
"gpu_vram_gb": None,
services/hwfit/models.py
+8
View File
@@ -12,6 +12,7 @@ QUANT_BPP = {
"Q4_K_M": 0.58, "Q4_0": 0.58, "Q3_K_M": 0.48, "Q2_K": 0.37,
"AWQ-4bit": 0.50, "AWQ-8bit": 1.0,
"GPTQ-Int4": 0.50, "GPTQ-Int8": 1.0,
"QAT-INT4": 0.50, "QAT-INT8": 1.0,
"mlx-4bit": 0.55, "mlx-8bit": 1.0, "mlx-6bit": 0.75,
# DeepSeek-V4-style mixed: MoE experts in FP4 (bulk), attention + non-
# expert dense in FP8, embeddings/LM head in BF16. By weight count the
@@ -30,6 +31,7 @@ QUANT_SPEED_MULT = {
"Q4_K_M": 1.15, "Q4_0": 1.15, "Q3_K_M": 1.25, "Q2_K": 1.35,
"AWQ-4bit": 1.2, "AWQ-8bit": 0.85,
"GPTQ-Int4": 1.2, "GPTQ-Int8": 0.85,
"QAT-INT4": 1.15, "QAT-INT8": 0.85,
"mlx-4bit": 1.15, "mlx-8bit": 0.85, "mlx-6bit": 1.0,
"FP4-MoE-Mixed": 1.10, # slightly slower than pure FP4 because of mixed-dtype dispatch
"FP8-Mixed": 0.85,
@@ -47,6 +49,10 @@ QUANT_QUALITY_PENALTY = {
# penalty so FP8 wins when both fit. AWQ-4bit stays heavier.
"AWQ": -1.0, "AWQ-4bit": -4.0, "AWQ-8bit": -1.0,
"GPTQ": -1.0, "GPTQ-Int4": -4.0, "GPTQ-Int8": -1.0,
# Quantization-aware training recovers most of the int4 quality loss, so a
# QAT-INT4 build lands far closer to bf16 than a post-training Q4/INT4
# (Google reports near-bf16 quality). Penalize it lightly, not like Q4_K_M.
"QAT-INT4": -1.0, "QAT-INT8": 0.0,
"mlx-4bit": -4.0, "mlx-8bit": -0.5, "mlx-6bit": -1.5,
# DeepSeek-V4 mixed: only MoE experts at FP4 (the rest is FP8/BF16),
# so the realized quality is much closer to FP8 than to pure FP4 —
@@ -63,6 +69,7 @@ QUANT_BYTES_PER_PARAM = {
"Q4_K_M": 0.5, "Q4_0": 0.5, "Q3_K_M": 0.375, "Q2_K": 0.25,
"AWQ-4bit": 0.5, "AWQ-8bit": 1.0,
"GPTQ-Int4": 0.5, "GPTQ-Int8": 1.0,
"QAT-INT4": 0.5, "QAT-INT8": 1.0,
"mlx-4bit": 0.5, "mlx-8bit": 1.0, "mlx-6bit": 0.75,
"FP4-MoE-Mixed": 0.55,
"FP8-Mixed": 1.0,
@@ -74,6 +81,7 @@ PREQUANTIZED_PREFIXES = (
"AWQ-", "GPTQ-", "mlx-", "FP8", "FP4", "NVFP4", "MXFP4", "NF4",
"INT4", "INT8", "W4A16", "W8A8", "W8A16",
"FP4-MoE-Mixed", "FP8-Mixed",
"QAT-",
)

Some files were not shown because too many files have changed in this diff Show More