An engineer we spoke to had spent a morning pulling apart three commercial code-check demos before her fire-strategy sign-off meeting. All three had labeled the same corridor "compliant" against the same UK BS 9999 clause. All three were wrong in different ways.
The first had read the door schedule and returned the nominal leaf width, 800 mm, as the clear width. Nominal is the manufacturer's stated leaf size. Clear width is what you actually measure between the door faces at the point of tightest projection. On a door with a 100 mm frame projection and a check strip, clear width can be 65 mm less than nominal. The demo did not know the difference. It called the corridor compliant against a limit the corridor did not meet.
The second had extracted the wrong sentence from the code document. BS 9999 §16.5.1 requires that two independent escape routes be provided from every level. The demo cited the requirement and moved on. The building actually had one route on the top level. The tool never noticed. It quoted the code and marked the clause satisfied because the code sentence was present in the corpus, not because the design met it.
The third one was worse. It had generated a citation to a clause number that did not exist. §16.5.7. The demo confidently reported that §16.5.7 required two escape routes and the building satisfied it. There is no §16.5.7. There is §16.5, subclauses §16.5.1 through §16.5.5, and that is where BS 9999 stops. The demo had hallucinated a clause number and hallucinated a pass against it.
Three failure modes. Three different reasons a submission would clear the tool and get bounced back by the reviewer. All three under a single label: false pass.
What a false pass actually costs
A false pass is not a demo quality problem. It is a professional-indemnity problem. If your compliance tool says a corridor meets code and your senior engineer stamps the drawing on that basis, and Building Control comes back with the corridor red-lined, the tool is not the party in the room. The stamped engineer is.
The design has to be revised, the drawings re-issued, and in a live project every one of those actions has a cost measured in weeks of schedule and hundreds of hours of coordination. On a building where the review is regulator-mandated, the sequence is worse. The false pass gets caught by an outsider, not by the design team. The design team's reputation gets caught with it.
That is the reason "get closer to right" is the wrong frame. The tool cannot be closer. The tool has to abstain when it cannot cite a source, and it has to abstain loud enough that the human upstream of the stamp knows to look.
The three failure modes have one common shape
Look at what the three demos had in common.
Each returned a judgment without a cited source.
- Demo 1 returned "800 mm satisfies the minimum" without pointing at the exact clause and the exact drawing dimension. The drawing dimension was there. It was labeled "leaf width". A human would have noticed the label. The tool did not.
- Demo 2 returned "two escape routes required, provided" without checking the actual route topology. The route-topology data was in the model. The tool cited the requirement and skipped the check.
- Demo 3 returned "clause §16.5.7 requires and is satisfied" citing a clause that did not exist. It generated the citation from the shape of the surrounding clauses, not from the code document.
The common shape: generate a verdict, then generate a plausible citation next to it. The order is wrong. In compliance, the citation is the input. If there is no verbatim, source-matched quote behind a value, there is no value.
This is what we call the cite-or-abstain rule, and it is the first thing Signet is built around.
What cite-or-abstain looks like in practice
When Signet reads a BIM model and a code document, it does two operations that many current tools do as one.
Perception. For every value the tool needs, Signet looks in the source (the model, the drawing schedule, the fire strategy document) and finds the verbatim quote it can cite. If a source-matched quote exists, the value comes forward with the quote attached. If not, the value is dropped. Not returned as low-confidence. Not returned with a warning. Dropped.
Judgment. Separately, a deterministic rules engine takes the cited value and scores it against the code threshold. The engine is code, not a language model. Same input, same output, byte for byte. The verdict does not depend on the prompt or the temperature.
The separation matters. If the language model produces the verdict, then the language model gets to hallucinate the pass. If the language model produces only the cited value and a deterministic engine produces the verdict, then the language model cannot manufacture a false positive. It can only fail to deliver a value, in which case the tool abstains.
The engineer's earlier example, the 800 mm door. Under cite-or-abstain, the flow is:
- Signet looks for the clear width of the escape door. The door schedule lists a nominal leaf width, not a clear width. Signet does not find a verbatim clear-width value.
- Signet abstains and flags the door with a note: "clear width not stated in source; leaf width 800 mm found in schedule but is nominal, not clear width. Upload a door schedule that states measured clear width."
- The rules engine cannot judge the clause without a cited clear width. The clause moves to review, not to pass.
The reviewer sees "abstain" and knows to look. The stamp does not happen on incomplete evidence.
Achieved is not the same as required
There is a second failure mode that shows up in more subtle demos. It is the one where the tool cites the correct clause, but confuses the code limit with the design value.
Every fire strategy document restates the code limits it is designed against. A well-written strategy will say "BS 9999 §5.1.4.2 requires clear widths of not less than 850 mm; this design provides 850 mm." A less well-written strategy will just say "clear widths shall not be less than 850 mm" and leave the design value to the drawings.
If a language model extracts the sentence "clear widths shall not be less than 850 mm" and uses it as the design value, the tool will confidently return "compliant" for a corridor whose actual clear width is 800 mm. The tool has extracted the requirement and treated it as the achievement.
Signet tags every extracted value at extraction time with one of three flags: achieved, required, or unknown. Only achieved values feed the deterministic engine. A required value is a code restatement, and gets ignored for the pass/hold decision. An unknown value abstains.
This is the second built-in refusal. A code limit restated in prose is never mistaken for what the design delivers.
The critic that can only downgrade
A third safeguard sits after the deterministic engine. A second AI, prompted differently and grounded in the same evidence trail, reviews every clause the engine passed and asks: is the cited evidence actually what the engine thinks it is?
The critic has one asymmetry. It can only downgrade a pass to a hold. It cannot promote a hold to a pass.
The reason is direct. A pass that the critic downgrades to a hold surfaces to a human reviewer, who can look at the evidence and decide. A hold that the critic promoted to a pass would slip past the human. False positives leak out of the pipeline unless the pipeline has no path from hold to pass without a human.
Every stage of Signet is designed around that asymmetry. Perception can abstain. Judgment can hold. Verification can downgrade. Only a licensed professional can convert a hold into a pass, and only by explicitly accepting the departure on the record.
What this looks like on a real submission
An engineer runs a submission. The BIM model is a mixed-use tower. The jurisdiction is Dubai. The code pack is the UAE Fire and Life Safety Code plus the referenced NFPA sections.
Signet ingests the model and the fire strategy PDF. Perception runs. Every value the tool needs, it finds a verbatim quote for. Where the fire strategy does not state the value, the model is checked. Where the model does not state the value, the tool abstains and flags the gap.
The rules engine scores each cited value against the corresponding UAE FLSC clause. The critic reviews every pass and downgrades one it does not trust. The engineer reviews the departures and the abstentions, provides the missing data where she can, and stamps the file.
The final matrix has three columns of interest. Pass, with a verbatim citation and a green mark. Hold, with the reason the tool could not decide and the specific evidence that would resolve it. Signed, with the professional's stamp and the date.
That matrix is what Building Control accepts. The engineer's PI insurer, in the event of a claim, has the same three columns to look at. The record does not depend on the tool being trustworthy in general. It depends on the tool being honest about what it did and did not decide.
Where this leaves the vendor conversation
If you are evaluating an AI code-compliance product, the useful question is not "how accurate is it". Accuracy is a summary statistic across a corpus you do not have and cannot audit.
The useful question is: what does the tool do when it cannot cite a source?
If the answer is "it makes its best guess and reports low confidence", the tool will generate false passes and the false passes will make it into your submissions. If the answer is "it drops the value and flags the gap for review", the tool will slow you down on the days the source data is incomplete, which is honest, and it will not put your stamp behind a claim it cannot defend.
Signet is built on the second answer. Every value it returns carries a verbatim citation. Every value it cannot cite is dropped in front of a human. The engineer at the top of this post is the one who ran the demo. Say the word and we will run yours.