How we complete truncated language model responses and why stitching turned out harder than it seems

Our model cut off the answer mid-word — without an error or warning. This happened in a service where several language models work in one chat. We had to devise a mechanism for completing truncated fragments. Stitching turned out harder than it seems: you need not just to finish the word, but to preserve the meaning of the entire list of a hundred items. We tried several approaches and settled on the one that