What the actual hell? - How did it come up with a conversation with an Agent when the transcription is 100% accurate - there was only one person leaving a simple message, yet the summary looks like it was talking to the lady and confirming the information and request - this is nuts!
How do I turn off the Summary - there doesn’t seem to be a way to do that in the GUI, and the PHP file that is called - /var/lib/asterisk/bin/emailVoicemailTranscript.php is obscured.
This does seem to raise interesting questions about how to handle LLM misbehaviour reports, given that the normal, what I did, expected output, actual output, bug report format, assumes that the system should behave repeatably but AI models are statistical, and don’t do so. The normal debugging process also assumes that there is an obvious cause and therefore obvious fix, but the human developers of AI systems can’t know the precise effects of changes in system prompts and training data, so can’t understand exactly what went wrong and the precise fix.
In this case, my guess is that an attempt to discourage over summarising has created an incentive to fill out the response, and the LLM has fallen back to the basic operation (predicting the most likely next word, one by one) and tried to create the most common pattern in its training data that reaches the length it has been advised to produce.
That is probably an accurate synopsis of what went wrong, but we just need a toggle in the Module to suppress the Summary if it’s going to Hallucinate like this.
The transcription works very well, and the client is very happy with that - it is much faster to read a transcription of a VM than to listen to it - they are dependent on the transcription because they get over 30 VM’s a day that all have to be responded to in a timely manner.
Unfortunately, human nature being what it is, they can’t just ignore the Summary (even though I have begged them to!) and read the transcription - the Summary comes first in the E-Mail, and when it has text like this, they crash and burn before ever reading it.
I used to use Watson to transcribe their Voicemail’s - its accuracy was about 95% of what Scribe achieves, but if I can’t turn off the Summary, I may have to switch back to it and dump Scribe.
I don’t think having a toggle in Scribe to skip the Summary is too much to ask for…
