Model development and research program

The research layer we work in sits between the foundation model and the application: how models are adapted, routed, grounded, given memory, asked to clarify, and taught to recover during real customer-service conversations. This is where most production failures occur and where relatively modest compute can still produce original, reproducible results.

The program exists to answer one question:

How can an LLM-powered service agent become more reliable, efficient and recoverable over an entire customer journey, rather than merely producing a good answer to an isolated prompt?

Our working thesis is that customer-service intelligence is not identical to general-purpose benchmark intelligence. A capable service agent must know what it knows, recognise missing information, use institutional evidence, maintain user state, recover from errors, invoke tools correctly, minimise customer effort and escalate when appropriate. Each of those is a measurable behaviour, and each can be improved independently of raw model scale.

The program was established in September 2026. Its primary objective is to produce rigorous, reproducible research that improves Airgentic’s technology while contributing useful evidence to the wider LLM and customer-service research community.

• Research first, marketing second — Each study begins with a falsifiable hypothesis, not a desired product claim. Negative and null results are recorded and reported.
• Reproducibility — Where licensing and confidentiality permit, we keep benchmark definitions, evaluation scripts, prompts, model versions, hyperparameters and statistical methods, and version every dataset and model artefact with a manifest (configuration, seed, git commit, checksums, metrics).
• Strong baselines — Every intervention is compared with simple alternatives: better prompting, retrieval-only improvements, a second-model verifier, larger-model escalation, or conventional rules. An architectural idea is only valuable if it beats these.
• Whole-conversation evaluation — Single-turn correctness is insufficient. Primary measures increasingly focus on end-to-end task success, unnecessary turns, recovery, abstention, grounding and customer effort.
• Multiple model families — Key findings are repeated across several open-weight model families and sizes where practical, so conclusions are not an artefact of one model.
• Separation from customer data — Benchmarks and training data are built from public institutional information and synthetic or authored scenarios. Customer conversations are used only where consent, privacy, contract and de-identification requirements are satisfied.
• Pre-registered primary measures — Before a final benchmark run we freeze the hypothesis, primary metrics, test split, intervention variants, stopping criteria and statistical comparison method.

aero-signals-1 is a small encoder classifier for individual visitor turns in website chat conversations. It is the first model Airgentic has trained rather than adopted, and the first project delivered under the Small Models, Specialist Behaviour stream. The current checkpoint is v006, trained on double-judge consensus labels after a second pass that tightened urgent and looking_to_transact.

Research question. How accurately can a very small pretrained encoder classify sentiment, conversational signals and user intent in realistic website chatbot conversations, given only the conversational text?

Why an encoder. The task needs a label, not generated text, and production inference must be cheap enough to run on every turn of every conversation. A fine-tuned encoder produces all labels in one forward pass on commodity CPU hardware. Generative models were considered and rejected for this role on cost, latency and controllability.

Backbone. DeBERTa-v3-xsmall (12 layers, hidden size 384; roughly 22M backbone parameters plus a 128k-token embedding matrix, about 71M in total). The encoder is fully fine-tuned; no adapters.

Heads and loss. Three linear heads over the pooled representation: 3 sentiment logits (cross-entropy), 3 signal logits and 4 intent logits (binary cross-entropy with logits). Total loss is an equally weighted sum.

Input. Conversational text only: the preceding turns (up to four) and the current visitor message, truncated to 256 tokens. Industry, persona, journey, requested labels and other generation metadata are never shown to the model.

Optimisation. AdamW, learning rate 2e-5, weight decay 0.01, 6% linear warm-up, effective batch size 16, gradient clipping at 1.0. Training runs on Apple Silicon (PyTorch MPS) with fp16 autocast; bf16 produced NaNs on this stack and is disabled. Epochs are a ceiling of 20 with early stopping when validation loss fails to improve for three consecutive epochs. The retained checkpoint is the epoch with the best combined F1 across heads.

Thresholds. Each signal and intent label has an independent decision threshold selected on the validation set to maximise F1 (grid 0.05–0.95). Thresholds are stored with the model version. 0.5 is not assumed.

Splits. 80 / 10 / 10 train, validation and test, deterministic given a seed, with near-duplicate messages kept in the same split. For v005 and v006, 500 consensus examples were frozen out of those splits as a future gold set.

All training data is synthetic. Two small pilot datasets (v001, v002) were generated with local open-weight models (Qwen 2.5 7B, Gemma 4) to prove the pipeline and are not used for the current model, since they carry an earlier label ontology.

v003 generation. 15,000 conversations generated with Claude Haiku 4.5 on Amazon Bedrock (Sydney region) via a schema-enforced tool call, validated with Pydantic, with failures logged rather than kept. Diversity is controlled by sampling: 48 industries, 30 audience types, 37 visitor journeys and five English locales (AU, NZ, UK, US, CA), with message-length and writing-style distributions covering typos, one-word replies, sarcasm and non-native phrasing. Half the examples are naturalistic (labels drawn from realistic prevalence); half are targeted, of which 70% are deliberately hard cases such as a calm complaint, a polite stuck user, or urgency without frustration. Generation prompts suppress cue words so the model cannot learn shortcut vocabulary.

Generation method (v003). Labels first: the pipeline sampled a target label set and scenario, then asked the generator to write a conversation exhibiting them. This is the conventional approach and, as described below, it is where the project’s main quality problem originated. The conversations themselves were kept; later versions only change the labels.

Baseline (model v004, data v003). On a held-out test split of 1,519 conversations the first full-scale model reached sentiment accuracy 0.775 and macro-F1 0.739; signals macro-F1 0.795 (frustrated 0.61, urgent 0.88, grateful 0.89); intent macro-F1 0.798 (looking_to_transact 0.75, stuck 0.80, wants_human 0.96, complaint 0.68). Training stopped at epoch 8 on validation loss. These numbers are below what the task should support for a model of this size.

Failure analysis. Reviewing misclassified examples showed that many “errors” were disagreements with implausible ground truth: conversations labelled frustrated with no frustration in the text, purchase intent assigned to availability questions, and complaint labels on neutral how-to questions. Because v003 sampled labels independently of the scenario, the generator was often asked to exhibit combinations that do not occur naturally, and it complied loosely.

Audit. We drew 500 random v003 examples and had them labelled twice by an independent judge (Claude Haiku 4.5, temperature 0) that saw only the conversational text, never the requested labels, industry, persona or journey. Inter-judge agreement was high: Cohen’s κ 0.91–0.98 per label, with both passes agreeing on every label for 83.6% of examples. Agreement between the original generation labels and the judges’ consensus was poor for exactly the labels the model struggled on: looking_to_transact κ 0.23, frustrated 0.28, complaint 0.34, stuck 0.42, sentiment 0.49. Only wants_human (κ 0.94) was reliable. The training labels, not the model, were the bottleneck.

Decisions. (1) Relabel all 15,000 conversations text-first with the same blind double-judge protocol and keep only examples where both passes agree; this becomes data v004. (2) Drop angry (folded into frustrated) and confused, which could not be labelled consistently. (3) Retrain as model v005 with an otherwise unchanged recipe, so the effect of label quality can be isolated.

Relabel (data v004). Finished. 14,999 of 15,000 conversations received two independent judgements (one example was dropped because the judge returned an invalid sentiment value). Both judges agreed on every label for 12,375 conversations (salvage 82.5%). Inter-judge Cohen’s κ remained at least 0.90 on every label. Original generation labels still disagreed with that consensus on the hard classes (looking_to_transact κ 0.22, frustrated 0.24, complaint 0.36). 500 consensus examples were frozen out of training as a future gold set.

v005. Trained on the remaining 11,875 consensus examples. On a held-out test split of 1,220 conversations, sentiment macro-F1 rose from 0.739 to 0.922, signals from 0.795 to 0.884, and intent from 0.798 to 0.896. The jump is the effect of replacing noisy labels-first targets with text-first consensus, not a change of architecture.

Boundary tighten (data v005, model v006). Error review of v005 showed that remaining misses on urgent and looking_to_transact were mostly definition boundaries. We did not regenerate text. A single additional judge pass over the 5,878 consensus examples already marked urgent or looking_to_transact applied stricter rules: urgent requires a stated deadline or action needed now or today; looking_to_transact requires this turn to complete an exchange (short confirmations such as “process it” still count). 776 urgent and 3,337 looking_to_transact positives flipped to false. Model v006 was then trained on that tightened set.

• Synthetic distribution — Training and test data come from the same generator family. Test scores overstate real-world performance to an unknown degree until a human-reviewed set exists.
• Correlated judges — Both relabelling passes, and the later boundary pass, use the same model at temperature 0. High inter-pass agreement demonstrates consistency, not correctness. A systematic bias shared by those passes would be invisible to this protocol.
• Ontology stability — Two labels were removed between v003 and v004. looking_to_transact was labelled as broader purchase intent for v005, then tightened back to “this turn completes the exchange” for v006. Results across versions are not directly comparable on that label.
• Per-turn scope — The model classifies one turn with short context. It does not track state across a conversation or aggregate signals over time; that is a separate problem. Short confirmations such as “process it” still depend on the preceding turns remaining in the 256-token window.
• English only — Five English locales are covered. No claim is made for other languages.
• Class prevalence — After the tighten, looking_to_transact has test prevalence of about 12.5%, so per-label F1 has a wider interval than the sentiment head. Bootstrap confidence intervals have not yet been reported.
• Latency — Mean 44.6 ms per turn on Apple M4 CPU at batch size 1, including tokenisation (p95 47.5 ms; peak RSS about 1.2 GB). This is a development-laptop figure, not the production SKU.