Enhanced low-latency speaker spotting using selective cluster enrichment