I would recommend getting into contact with support for the qna maker team for cases like this, however if you're willing to commit some time and try to do it yourself (if you have a lot of questions) then there is an extreme option that should work to force refresh your search instance.
The steps that worked for me are:
- Delete your current frozen indexes from the search instance in azure (everything but test).
- Make an edit to every single QnA pair in your kb (even a blank space at the end of the question works, you just need it to be added for retraining)
- Retrain/save your kb. (refresh the url-populated questions as well)
- Publish your knowledge base again.
All the newly edited questions (in this case all of them) should be freshly added to the index. I am sorry that this is a bit of an extreme option though, so I'd still recommend going to support before this generally.