Case study
Inyeon (인연)
An agentic Git CLI that splits a working tree into commits, writes the messages, reviews the change and drafts the PR, and its first evaluation, which found four bugs that 245 passing tests had missed. All four are fixed.
Results
- valid Conventional Commits
- 93.3%28 of 30 real commits, gpt-4.1-mini. My own messages on the same commits: 66.7%.
- commit type matches mine
- 65%13 of 20 commits that had a conventional message. Always guessing feat scores 45%.
- LLM calls per commit message
- 23,552 input and 306 output tokens on average; 3.1 s p50, 5.2 s p95.
- split agreement (ARI)
- 0.525The default strategy, on 12 held-out cases from repos the threshold was not tuned on. One group per file scores 0.352; the gap's 95% CI, +0.02 to +0.35, only just clears zero.
Problem
Every change ends in the same chores: cutting a messy working tree into commits that each do one thing, writing a message for each, reviewing the diff, and describing the pull request. None of it is hard, all of it is repetitive, and it is exactly the part that gets rushed.
Inyeon runs that whole loop from the terminal in one command, with the model of your choice: a hosted one, or a local one for code that should not leave the machine.
What I built
- 01CLITyper + Rich, diff cut at file boundaries
- 02Enginehosted FastAPI backend, or in-process
- 03Pipelinesplit → commit → review → PR
- 04AgentsLangGraph state graphs with error nodes
- 05ContextChromaDB retrieval over the repo
- 06ModelsOpenAI, Gemini or Ollama, streamed
Seven agents, each a LangGraph state graph with conditional edges and its own error node: split, commit, review, PR, conflict resolution, changelog, and an orchestrator that routes a free-form request to the right one. The commit agent can ask to read files for context; those reads are sandboxed to the repository.
Splitting has four strategies. Directory groups hunks by folder. Semantic embeds each hunk and clusters them (agglomerative, cosine distance, threshold 0.5). Conventional asks the model to classify each file by commit type. Hybrid, the default, groups by directory first and sub-clusters any group larger than three hunks.
The same pipeline runs against a hosted backend or entirely in-process, behind one interface. It skips steps that cannot pay for themselves (no split for a single file, no review for a diff under 500 characters), and a failed review or PR step is skipped rather than failing the run.
Considered and set aside
- Remote-only execution—one pipeline interface now runs against the hosted backend or in-process
- Retrying on error-message text—replaced with retries keyed on status codes
- Failing the run when review or PR fails—those steps degrade to skipped, so the commit still lands
- Eager imports in the local engine—made lazy, so the CLI starts without loading every model client
Reading the numbers
The eval found four bugs. The one-command pipeline read the key message while the commit agent returns commit_message, so the generated message came back empty in 10 of 10 runs. Each step after the first lost its first model call to a connection error, because every step ran in a fresh event loop while reusing one cached client. Semantic splitting never ran at all: the retriever was built with an argument its constructor does not take, and the TypeError was caught and turned into None. And the pinned embedding model, text-embedding-004, had been retired.
All four are fixed. On the same runs afterwards, the commit message came back 10 times out of 10, none of 34 model calls failed, and PR descriptions came back 5 times out of 5 (2 before). Seven regression tests now pin the fixes, and every one fails on the old code. The original tests had passed because their mocks returned a response shape no backend actually produces.
Fixing semantic splitting exposed a fifth problem: on the replacement embedding model, the old 0.5 similarity threshold put every hunk in one group. I recalibrated it to 0.75 on three repos and tested on the other two. Semantic alone scores 0.513, against 0.352 for one group per file, but its interval includes zero. The default strategy is only just better than the baselines, and semantic on its own is not yet shown to be.
Scope: one run per configuration, 30 split and 30 commit cases (12 held out for the threshold), gpt-4.1-mini only, the in-process engine only, and only one split case interleaves edits within a file. The suite now stands at 251 passed, 2 skipped.
Lessons
- Passing tests are not a working product. Each agent had tests, but their mocks agreed with the bug instead of with the backend, and nothing tested the hand-off between agents, which is where the output was lost.
- A swallowed exception can switch a headline feature off without a trace. One except-and-return-None made semantic splitting a silent no-op.
- Start from the dumbest baseline. One commit per file landed inside the confidence interval of every strategy.
- Hosted models expire. A pinned embedding model and a threshold tuned to it both quietly went stale.
NextTune the default strategy's threshold on its own rather than inheriting semantic's, grow the held-out set past 12 cases, test edits interleaved within a file, and add an eval for review quality.