AI personality quality is often judged through a few attractive demo conversations. That is insufficient for a product that supports text, voice, photos, short videos and real-time digital-human interaction. The same personality can behave coherently in text but contradict itself when spoken, lose context after a video call, or make a confident claim that is not supported by its source material. A cross-modal scorecard gives product teams a repeatable way to test the complete experience before a release reaches users.
1. Define the dimensions before collecting scores
Use a small number of dimensions that teams can explain: factual grounding, identity consistency, conversational control, multimodal continuity, accessibility, privacy controls and user understanding of the AI’s role. Each dimension needs observable criteria. “Feels natural” is too vague by itself; “correctly handles a user interruption, does not repeat the interrupted sentence, and resumes from the last audible point” can be tested.
Keep safety-critical failures separate from average quality scores. A model that is charming but repeatedly invents creator biography should not receive a passing grade because its latency and voice quality are excellent. Define blocking failures in advance, such as falsely claiming a real person personally sent a message, ignoring a request to stop media capture, or restoring a memory after the user deleted it.
2. Build a balanced test set from real user journeys
Include first-session onboarding, a return visit, a factual question, a correction, a shared photo, a short-video discussion, a switch from text to voice, an interruption and a request to inspect memory. Include normal cases and edge cases. Test different languages, network conditions, screen sizes and accessibility settings. Do not rely only on prompts written by the engineering team; product support tickets and opt-in user research can reveal misunderstandings that internal authors do not anticipate.
Use synthetic test identities or properly authorized creator assets. A test suite should not require copying private user conversations into a shared evaluation dataset. Store the minimum evidence needed to reproduce a failure, redact unrelated personal information and define a retention period for test logs.
3. Measure consistency across modalities
For each test case, compare the intended facts and boundaries across text, voice and video. A spoken response may be shorter than a written response, but it should not introduce new unsupported claims. The face or expression should not suggest certainty when the words are explicitly uncertain. If the user changes from video to text, the system should preserve only the relevant context and should not imply that the camera remains active.
Track turn-taking separately from raw latency. A response that begins quickly but speaks over the user is not a good conversational experience. Measure time to first useful response, interruption success, recovery after network loss, transcript alignment and the proportion of sessions that need a restart. Set thresholds from observed user needs and device conditions rather than copying a competitor’s marketing number.
4. Test memory as a lifecycle, not a single answer
Memory tests should include saving, inspecting, correcting, expiring and deleting an item. Start with a known fact, verify that it is recalled when relevant, change the fact and check whether the old version persists. Delete it, then test a new session, a summary path and any downstream personalization. A user-facing deletion confirmation should be backed by a real system state, not just a toast message.
Also test false memories. If a user says “I might visit Osaka,” the system should not later state as fact that the user lives there. Distinguish a preference, an intention, an unverified statement and a confirmed fact. Memory should have confidence and provenance appropriate to the application, even if those details are not exposed as technical jargon to every user.
5. Score accessibility and user control
Test captions, keyboard navigation, screen-reader labels, reduced motion, adjustable playback and text alternatives to voice. A real-time video feature should have obvious camera and microphone states, a reliable end button and a text fallback. Users should be able to disable proactive messages and choose a quiet period. These are not edge-case niceties; they determine whether the same feature works for people with different abilities, schedules and environments.
Ask evaluators whether they understood what was generated, what was stored and how to change it. A control that exists but cannot be found should count as a failure. A deletion route that takes five screens or uses ambiguous wording is technically present but not meaningfully accessible.
6. Turn the scorecard into a release gate
For each release, publish an internal scorecard with test coverage, blocking failures, known limitations, regressions and the owner of each unresolved issue. Keep a stable benchmark set for comparison and a rotating set for new risks. When a failure is fixed, add the exact scenario to the regression suite. Do not let teams improve a headline score by removing difficult test cases.
NIST’s Generative AI Profile provides a lifecycle-oriented risk framework, and Tuikor AI’s public overview describes an experience spanning AI personality, digital twins, multimodal interaction and long-term memory. A useful scorecard connects those broad capabilities to observable behavior. The goal is not to assign a single magical “human-like” number; it is to know which parts of the experience are reliable, for whom, and under what conditions.
Sources
- NIST, Generative AI Profile (published 26 July 2024).
- Tuikor AI official product overview (accessed 28 October 2026).
