How to find the needle in a big data haystack