IDE Gap Audit¶

Initial audit: 2026-09-13. Progress update: 2026-09-14.

Current status¶

The detailed implementation record, language support boundaries, release gates and acceptance criteria are maintained in Production readiness. The baseline findings below describe the original audit; they must not be treated as the current feature matrix.

Since the initial audit, the following gaps have been addressed:

IR save/load now preserves native bodies, component template maps, typed configuration sections and modern runtime settings.
Python MCP invocation uses real SDK-managed client sessions, with transport/error/timeout tests. The former live-MCP placeholder finding is resolved for Python.
Java functions/tools require explicit native code; HTTP registry tools and template transformers execute real behavior. Unsupported MCP integrations fail export instead of emitting a stub.
Go is a native executable target in the IR and IDE, with its own graph runtime, generated tests and toolchain build checks.
All targets support the canonical /invoke state/result envelope, with legacy request compatibility in Java/Go.
Capability checks reject unsupported graphs and missing target implementations. The shared pack intersection no longer claims Java streaming/run-event support.
IDE target previews and native editors are wired through export; host builds/tests report actual toolchain failures and native debugging limitations.
SDK version 0.3.2 has package/install/runtime verification. New conformance tests compile and execute generated applications instead of only checking source strings.

RAG, advanced agents, native MCP, durable cross-language checkpoints, streaming/events, full settings/state parity, Go pack/reverse-sync/debug support, and hosted backend/deployment acceptance remain open. Their exact boundaries and completion criteria are listed in the readiness document. Existing langstitch.rag and generated registry helper modules still require implementation and business-behavior tests; the initial suggestion that a small runtime surface alone made RAG ready to enable was too optimistic.

Run ./langstitch-docs/scripts/verify-production.ps1 from the sibling-repository workspace to repeat the build, package and runtime gates. See the reproduction instructions. Do not infer production readiness from the historical test counts below.

Original audit findings¶

This audit tracks design and feature gaps across the three core product layers:

SDK/runtime: langstitch-sdk
Representational layer: langstitch-spec
IDE/canvas: langtailor/_canvas

The product architecture is sound: the IDE edits a graph, the spec defines the durable IR v2 contract, and compilers turn that IR into runnable Python or Spring AI projects. The main risk is drift. Some concepts now exist in one layer before the other layers can faithfully compile, run, validate, or explain them.

Verified Baseline¶

langstitch-spec: npm test and npm run build passed before this audit.
langtailor/_canvas: npm test now passes with 98 tests.
langtailor/_canvas: npm run build was failing from type drift; now passes.
langstitch-sdk: local python -m pytest -q now passes with 154 passed, 3 skipped.

Fixes Completed In This Pass¶

IDE build contract drift is fixed.

The IDE canvas build failed because the app had moved faster than its shared TypeScript contracts. The fixes aligned MCP server definitions, RAG retrieval enums, Spring test fixtures, plugin creator host API usage, dead persistence code, and IR bridge casts.

Files changed:

langtailor/_canvas/src/components/ide/PluginCreatorPanel.tsx
langtailor/_canvas/src/lib/codegen/__tests__/springAiProjectGenerator.test.ts
langtailor/_canvas/src/lib/documentIr.ts
langtailor/_canvas/src/lib/mcp/registry.ts
langtailor/_canvas/src/lib/plugins/registry.ts
langtailor/_canvas/src/lib/templates/buildTemplateDocument.ts
langtailor/_canvas/src/lib/codegen/pythonGenerator.ts
IDE export preflight now has target compatibility analysis.

A new resolver blocks exports before codegen when the selected target cannot support the graph. It catches unsupported node kinds, missing custom component templates, and unsupported checkpointer settings, while warning on current MCP runtime placeholders.

Files changed:

langtailor/_canvas/src/lib/targetCapabilities.ts
langtailor/_canvas/src/lib/__tests__/targetCapabilities.test.ts
langtailor/_canvas/src/lib/codegen/bundleGenerator.ts
SDK local test signal is fixed.

tests/test_server_security.py requires FastAPI and LangGraph because the tests compile and invoke a graph. It only gated on FastAPI, so local environments with server deps but without graph deps failed for the wrong reason. The test now also skips when langgraph is absent.

File changed:

langstitch-sdk/tests/test_server_security.py

Gap 1: IDE Features Outrun Compiler Support¶

The IDE exposes advanced nodes and registries that are only partially supported by the compilers.

Representational layer supports these node kinds in IR v2:

start
end
llm
tool
router
function
subgraph
agent
rag
intent_classifier
hitl
response_transformer
custom

Python IR compiler support is stronger than Spring, but still incomplete:

Supported or partially supported: llm, tool, router, function, subgraph local only, intent_classifier, hitl, response_transformer, custom
Explicitly unsupported: agent, rag
Runtime gap: MCP tool code is emitted, but app/mcp_client.py still raises NotImplementedError for live MCP calls.

Spring AI canvas fallback support is narrower:

Supported or partially supported: llm, function, router, tool, response_transformer, custom
Explicitly unsupported: agent, rag, subgraph, intent_classifier, hitl
Runtime gap: MCP tool client returns a stub payload instead of invoking MCP.

Impact:

Users can design graphs in the IDE that cannot compile for selected targets.
Multi-platform packs can claim python-langstitch and spring-ai while relying on features that are not actually portable.
Marketplace templates and connector packs can look installable but produce weak or stubbed runtime behavior.

Recommended fix:

Extend the new capability resolver into the visible Platform export panel so users see target warnings before clicking export.
Make pack validation fail when supportedPlatforms includes a platform that cannot support the graph's used node kinds, component templates, registry features, or runtime settings.

Gap 2: Multi-Platform Intersection Is Declared But Not Enforced¶

langstitch-spec/src/pack.ts declares the multi-platform intersection:

Node kinds: start, end, llm, tool, router, function, response_transformer, custom
Features: streaming, runEvents

The node-kind intersection mostly matches current compiler reality, but enforcement is missing at the IDE and pack validation layer. The feature intersection is weaker: streaming and runEvents are declared as portable features, but they are not yet implemented consistently across emitted Python and Spring projects.

Impact:

The product has a correct architectural contract, but users do not get early guidance when they leave the portable subset.
A pack can include non-portable runtime expectations without a clear validation error.

Recommended fix:

Extend pack validation to compute used features from IR.
Add a target compatibility panel in the IDE that explains why a graph is or is not portable.
Treat streaming and runEvents as aspirational until both compilers emit working runtime events.

Gap 3: Graph Settings Shape Drift¶

The spec's GraphSettings now includes runtime concerns such as:

server
model
security
logging
deployment
observability
checkpointer

The IDE's graph settings still include older canvas/UI-era fields such as:

checkpoint
snapToGrid
showMinimap
eval
a2a

Some of these are still valid product concepts, but they are mixed across runtime settings and IDE presentation settings. The current bridge casts migrated settings into the IDE type after spec migration. That is acceptable as a boundary fix, but it is not a long-term design.

Impact:

Type drift will return whenever spec settings evolve.
Runtime settings and editor preferences can leak into the wrong layer.
Build-time validation cannot easily explain which setting belongs to which target.

Recommended fix:

Split IDE-only preferences from runtime graph settings.
Add explicit mapping functions between spec settings and IDE settings.
Add tests around v2ToFlat and exportIrDocument for security, logging, deployment, server, and IDE-only layout settings.

Gap 4: SDK Runtime Has Capabilities The IR Compiler Does Not Use¶

The SDK includes runtime modules for agents, A2A, MCP registration, RAG, HITL, tracing, and orchestration. The Python IR compiler does not yet map all IR concepts onto those runtime APIs.

Clear examples:

agent nodes are rejected even though SDK agent registration and A2A client/server support exists.
rag nodes are rejected even though langstitch.rag exists.
MCP tools compile into an emitted client, but live MCP invocation is still a placeholder.

Impact:

SDK looks richer than the visual-to-code path.
Users who start in code can access more features than users who start in the IDE.
The IDE's promise of visual authoring is weakened for advanced agent workflows.

Recommended fix:

Implement Python compiler support for rag first because the runtime already has a small RagPipeline surface and the IDE has RAG pipeline configuration.
Implement agent nodes next by mapping registry, remote, and A2A agents to SDK orchestration helpers.
Replace generated MCP NotImplementedError with a minimal working adapter for stdio first, then SSE/streamable HTTP.

Gap 5: Spring AI Target Is Still A Demo-Grade Target For Advanced Graphs¶

The Spring fallback generator produces runnable Spring Boot shape for basic graphs, but it does not yet map several Spring-native concepts:

No HITL equivalent.
No subgraph composition.
No intent classifier support.
RAG is rejected.
Agent and A2A nodes are rejected.
MCP is a stub.
Some connector templates return placeholder state.asMap() or TODO bodies.

Impact:

Spring AI is useful for demos and simple exports, but users may assume parity with Python.
Multi-platform packs can underdeliver when a component has a spring-ai template that is only a stub.

Recommended fix:

Make Spring AI capability badges explicit in the IDE.
Mark stub connector templates as preview or incomplete in their manifests.
Add Spring compiler tests for every supported intersection node kind.
Prioritize real MCP and response transformer behavior before adding unsupported advanced nodes.

Gap 6: Documentation And Deployment Story Drift¶

The backend/service documentation presents two backend stories:

PHP marketplace/auth/release backend for hosted production.
FastAPI backend for local desktop API and some legacy marketplace/auth paths.

Both exist in code, but the docs mix current and older deployment paths.

Impact:

Developers cannot tell which backend is authoritative for marketplace features.
IDE API fallback behavior is harder to reason about because hosted and local paths use different implementations.

Recommended fix:

Add one top-level backend architecture note in langstitch-backend-service.
Label PHP as hosted marketplace backend if that remains true.
Label FastAPI as local desktop/dev platform API unless the production plan changes.
Remove or quarantine stale deployment instructions.

Suggested Fix Order¶

Keep the build green across langstitch-spec, langtailor/_canvas, and langstitch-sdk.
Wire target capability warnings into the Platform export panel and pack validation.
Implement Python compiler support for RAG nodes.
Replace Python emitted MCP placeholder with a working stdio MCP adapter.
Add Spring AI capability badges and fail-fast pack validation for unsupported Spring features.
Split IDE preferences from runtime graph settings.
Clean backend documentation so hosted marketplace and local desktop backend responsibilities are explicit.

Current Validation Commands¶

From this workspace:

cd langtailor\_canvas
npm test
npm run build

cd ..\..\langstitch-sdk
python -m pytest -q