Which language model the platform uses is the question everyone asks and the one that matters least: it will change twice before your contract is up, and every serious platform will support the new one within weeks. The criteria that survive the model of the month are about your data, your permissions and your ability to measure and to leave. Six of them, each with the question to ask.
No products are named here, ours included. The disclosure is at the end.
1. Where your content goes and where the model runs
Three separate questions hide inside this one. Which providers can the platform route to, and can you restrict routing per project? Is there an option to run a model on your own infrastructure for the cases that need it? And what are the retention and training terms of every provider in the chain, in writing, including the platform vendor itself. The answer people usually give is a reassurance. The answer you need is a contract clause and a retention period in days.
Ask: for each provider in the routing chain, is my content retained, for how long, and is it used for training?
2. Retrieval quality, measured separately
Most disappointing answers in a document assistant are retrieval failures, not generation failures: the right passage was never handed to the model, which then answered from its own parameters and sounded exactly as confident. That makes RAG quality a property of chunking, indexing and filters more than of the model, and it means retrieval has to be measured on its own before anyone judges the prose.
Ask: can I measure retrieval on my own question set, separately from generation, and see which passages were retrieved for each answer?
3. Identity and permissions
If the platform indexes a document repository where different people see different things, the assistant has to respect that at retrieval time, not by filtering the answer afterwards. Check single sign-on against your own identity provider, role-based access, whether permissions can be scoped per project or knowledge base, and what the audit log records. This is the criterion that most often turns a successful pilot into a blocked rollout.
Ask: are document permissions enforced during retrieval, and what does the audit log show for a single question?
4. Evaluation and observability
Without a way to measure quality, every change is a matter of opinion and every regression is discovered by a user. Look for a golden set of questions with known answers, repeatable evaluation runs, and visibility of cost and latency per project and per use case. Ask how the platform behaves when a provider changes a model version underneath you: can you detect the regression, or do you hear about it in a meeting?
Ask: how do I run a repeatable evaluation against a known answer set, and where do I see cost and latency broken down?
5. Whether you can leave
Portability is cheap to arrange at signature and expensive to arrange later. What can you export: documents, the processed index, prompts, evaluation sets, conversation history, in what formats? Are the integrations you build tied to proprietary constructs or to something standard? A platform confident about its value will not be uncomfortable with this question.
Ask: on the day I leave, what exactly do I take with me, in what format, and what stays behind?
6. The cost model you will actually pay
Per seat, per token, per document, per index refresh, or a blend. The one that catches people is reprocessing: if a thousand page manual is revised monthly, who pays to embed it again, and is that line in the quote? Model a full year with your real volumes and your real update frequency, then ask what happens when usage doubles, because a successful deployment is exactly the case where the invoice surprises somebody.
Ask: for my volumes, what is the twelve month cost, and which line grows if adoption doubles?
How to run the comparison
Write twenty real questions from the people who will use the system, with the answers agreed in advance by someone who knows them. Add five questions the system should refuse or escalate, because how a platform handles what it does not know matters more than how it handles what it does. Run the same set on every candidate, with the same documents, and read the retrieved passages rather than only the answers.
Disclosure
We build one of these platforms, AI Portal. The criteria above are the ones we would want a buyer to apply to us, portability and the cost model included.