The problem
Telecom fault resolution runs on institutional knowledge that lives in two places: thousands of pages of engineering documentation, and the heads of engineers who have seen the fault before. Neither is queryable at three in the morning by someone who has not seen it before.
The default resolution path is therefore a site visit. A van, an engineer, a day — often to discover something a document already covered.
A general-purpose LLM is a tempting shortcut and a bad answer. It has no knowledge of BT's specific hardware, procedures or terminology, and its failure mode is the worst possible one for this domain: producing a confident, fluent, wrong instruction. In a context where the output dispatches physical resources or touches live infrastructure, a plausible hallucination is more expensive than no answer at all.
Answers had to be grounded in the actual documentation and traceable back to it. An engineer must be able to check the model's work in one click — otherwise the system just moves the trust problem rather than solving it.
Approach
The system is retrieval-augmented generation, built so that retrieval carries the knowledge and the LLM only carries the language.
1. Document ingestion and chunking
Engineering documentation is parsed and split into chunks sized to preserve procedural coherence. This is the least glamorous part of the build and the one that most determines quality — a chunk boundary through the middle of a numbered procedure produces retrievals that are technically relevant and practically useless.
2. Vector index
FAISS over embedded chunks, tuned for recall over precision at the retrieval stage. It is cheaper to hand the model one irrelevant chunk than to omit the relevant one; the generation step can ignore noise, but it cannot recover information that was never retrieved.
3. Grounded generation
Retrieved context is composed into a prompt that constrains the model to answer from the supplied context only, and to say so explicitly when the context is insufficient. "I don't have documentation covering this" is a correct and valuable output. It routes the case to a human instead of to a wrong answer.
4. Citation
Every answer returns the source chunks it drew on. This is what makes the output auditable, and it is what made engineers willing to use it.
Results
The system reached 87.3% resolution accuracy on evaluated fault cases — issues resolved correctly from documentation without escalation to a site visit. Every answer shipped with its sources attached.
A companion GenAI pipeline project formalised the evaluation side of this work: hybrid TF-IDF retrieval, chain-of-thought prompting, faithfulness scoring and BLEU/ROUGE benchmarking, reducing hallucination rate by 78% against an ungrounded baseline. That figure belongs to the companion project, not to this system.
| Component | Choice |
|---|---|
| Orchestration | LangChain |
| Vector store | FAISS |
| Generation | OpenAI models |
| Serving | FastAPI |
| Resolution accuracy | 87.3% |
| Grounding policy | Context-only, explicit abstention |
What the project taught me
- Retrieval quality dominates model quality. Time spent on chunking and index tuning returned more than time spent on prompt engineering or model selection.
- Abstention is a feature. A system that reliably says "not covered" is more useful than one with a higher raw answer rate and an unknown error rate.
- Citations are the adoption mechanism. Engineers did not need the model to be right every time. They needed to be able to check it in seconds.
- Evaluate faithfulness, not fluency. BLEU and ROUGE describe surface similarity. Faithfulness scoring against retrieved context is what actually tracks whether the answer is grounded.