COMMERCIAL IN CONFIDENCE: please seek permission before sharing
Options for MyService Condition Classifier Change
Meeting: Nick Miller, Gwen Collins, Andrew Miller, Celia Perry, Les Jordan, 3 June 2025
Author: Nick Miller, Gov Law Tech
Talking Points:
- Current status: existing Microsoft LUIS service to be retired in October
- Options
- Recommendation
Demo Input
Options
| Option Number | Description | Demo | Results |
|---|---|---|---|
| One | Turn off service. Change request to SvA. Settle a new user experience with SvA. |
NA | Impact depends on what the new user experience would be. Past analysis shows around 40% of conditions get matched to a decision ready condition with the current approach. So if that suddenly stopped, accepted conditions under the decision ready policies would fall by amount from MyService. |
| Two | Migrate to Microsoft's Conversational Language Understanding. Steps: Gov Law Tech migrates existing config; DVA staff get familiar with new interface. Cost: Minor project to upgrade, test, learn; Contract variation for Gov Law Tech - uncertain scope, T&M. Pro: Probably we will be able to get close to the current status quo, so you can say you are not changing anything, or only making a minor change. No change request required with SvA. Con: We're only using a small part of a large and fairly complex new tool - there will likely be minor hitches and unforeseen wrinkles in the transition. |
||
| Three | Semantic Search against SoP Condition Definitions We use a SOTA sentence-embedding service to search the condition definitions directly from the SoPs. Steps: Gov Law Tech can change the service; DVA needs to test and decide a threshold for a match, or decide to have no threshold; no SvA change request required. Pro: Early testing indicates it is superior than current approach; has a principled, explicable foundation; no ongoing maintenance required from DVA, less work than current setup; simple and low risk to implement. Con: Change management required from DVA: the outcomes will change from the current status quo - possibly more claims accepted, possibly less (depends on thresholds), likely better classification. |
||
| Four | Large Language Model Here we send a list of candidate condition definitions and the user's description to an LLM, and request the best match. Steps: It's a moderate complexity project, involving Gov Law Tech, DVA and SvA. Pro: Significantly improved outcomes for veterans and DVA. Public service medals will rain form the sky. Con: Manage expectations around hallucinations; requires new user experience; political issues around AI; needs high level executive champion. |