An 11% benchmark lift on Solidity code generation from ontology work, not a bigger model

dApp development has a harsher correctness bar than most code assistance. Solidity that looks right and does not compile is worthless. Solidity that compiles and fails its tests is worse than worthless, because someone has to find out why.

The copilot is measured on SolEval, a benchmark that actually runs the generated tests. There is no partial credit for plausible-looking output.

The naive path to a better copilot is a better model. That path was already taken — the baseline was a frontier model.

So the remaining gains had to come from three places that are not the model: retrieval efficiency over a large, dense domain graph fast enough for interactive use; ontology precision, because retrieval quality is bounded by how cleanly the graph can be queried; and output conformance, turning raw generation into something a test harness will accept.

• CostPaid graph compute, storage, I/O and data-transfer fees, plus separate paid visualisation tooling, made iteration expensive on the incumbent stack — and iteration was the whole method.
• ConcurrencyInteractive copilot use means concurrent queries, which is precisely where JVM-based graph databases degrade.
• ConsistencyReal-time accuracy demands strong data consistency without hand-rolled locking protocols.
• OntologyThe Solidity graph had accumulated redundant, overlapping and generically-typed edges. A graph you cannot query cleanly is a graph you cannot retrieve from cleanly.

Three changes, none of them a model swap. Each is individually unremarkable; the result comes from the fact that they compound.

Retrieval became hybrid: Memgraph’s vector index drives semantic search, then Cypher expands to subjects and objects, attaches chunk texts, and collects relationship types. Semantic search finds the neighbourhood; graph expansion supplies the context. Neither half works alone.

Snapshot isolation came out of the box, so strong consistency needed no manual locking protocol — and deployment stayed flexible across cloud, on-premise and hybrid.

The ontology work is the part that generalises. Solidity KG v0.3 was refined by hand with human-in-the-loop feedback, and every change made the graph smaller.

Edge consolidation. Import-related edges merged into one standardised IMPORTS relation, taking coverage from 2,043 to 7,399 instances. Property cleanup. Redundant edges — HAS OPTION, NAME, HAS ATTRIBUTE — removed entirely. Polarisation. USED-family edges standardised to USES.

And the largest single gain: CONTAIN specialisation. 11,571 generic CONTAIN edges split into LANGUAGE_CONTAINS_FEATURE, FILE_CONTAINS_SYMBOL, CONTRACT_CONTAINS_STORAGE and similar. One edge type that meant “is inside” became several that mean something specific enough to query.

The last stage is a Claude 3.5 Sonnet refactoring agent turning raw API output into production-ready, SolEval-compliant test contracts:

1. 01Standardise — SPDX headers, pragma statements and imports brought to one form.
2. 02Purify — Converted to pure Solidity, stripping the non-Solidity artifacts a model tends to emit around the code.
3. 03Complete — Comprehensive test scenarios generated where the draft was thin.
4. 04Compile — Syntax corrections and type consistency fixes — the pass that moves Compile@1 most directly.
5. 05Conform — Output shaped to the Forge test structure and assertion style the benchmark actually runs.

Two of the three levers involve no model inference at all. The ontology is a versioned artifact a human reviewed edge type by edge type. The retrieval expansion is Cypher. Both are inspectable, diffable and reproducible in a way a prompt is not.

That matters for a benchmark result specifically. When a system beats a baseline, the useful question is which part of the system did it — and a gain that lives in deterministic code is a gain you can still explain a year later, after the model underneath has been replaced twice.

• Pass@10 improved 11.3% over the Claude 4-Sonnet baseline, with Pass@1 up 11.2% — absolute improvements on the SolEval benchmark, which executes the generated tests rather than scoring them for plausibility.
• Compile@1 improved 10.4%. This is the metric the deterministic post-processing pass targets, and the clearest evidence that the gain is not coming from the model.
• Gains held across every function complexity bracket, including functions with five or more parameters — so the system handles complex dependency relationships, not just simple ones.
• Gas optimisation showed the largest advantage of any domain, which is what curated, domain-specific knowledge buys you over general capability.

Hybrid retrieval and ontology refinement compound. Memgraph supplies the efficient traversal engine; the refined ontology supplies data worth traversing. Neither alone produces the result.

Post-processing is not a detail. A deterministic refinement layer between raw generation and delivered artifact carries a large share of both Compile@1 and Pass@k. It is the cheapest place in the whole pipeline to buy accuracy.

Human review earned the ontology gains. Not automation — a person deciding, edge type by edge type, what a relation actually means.