Over the past few weeks, we've been working on something a little different. Instead of using a generative LLM to solve another problem, we asked whether a much smaller specialist model could do one specific job extremely well: understand what is happening in a customer conversation.
The result is aero-signals-1, a 71M parameter classifier fine-tuned from Microsoft's DeBERTa-v3-xsmall.
Not a chatbot
It's not a chatbot and it doesn't generate text. It reads the current visitor message, along with up to four preceding turns, and classifies three different dimensions of the conversation in a single pass:
- Sentiment: positive, neutral, or negative
- Signals: frustrated, urgent, or grateful
- Intent: looking to transact, stuck, wants a human, or complaint
Those distinctions matter. Someone can be perfectly polite but completely stuck. A complaint can be calm. Something important isn't necessarily urgent. And someone asking about a product isn't necessarily trying to transact yet.
aero-signals-1 is designed to recognise those differences cheaply enough that we can run it across every turn of every conversation.
Performance
The current model has 71M parameters and runs in an average of 44.6 ms on an Apple M4 CPU, including tokenisation.
On our held-out synthetic test set, the current v006 checkpoint achieved:
- Sentiment: 0.924 macro-F1
- Conversational signals: 0.902 macro-F1
- Intent: 0.851 macro-F1
This was trained on 15,000 realistic website conversations, not customer data, spanning 48 industries, 30 audience types, 37 visitor journeys and five English locales.
The result was a significant improvement in model performance without changing the underlying architecture.
Putting it to work
We'll put aero-signals-1 to work inside Airgentic, helping us surface things like frustration, urgency, customers getting stuck, requests for human help and transactional intent across thousands of conversations.
It's a small model with a very specific job. And it's our first.
More to come. You can also explore more of our research.
