r/semanticweb • u/mk_neon_builder • 3d ago

Semantic web search using sentence-transformers/all-MiniLM-L6-v2

https://mindzkonnected.com/blogs/returning-only-what-matters-smarter-web-crawling-with-semantic-search/

Using sentence-transformers/all-MiniLM-L6-v2 to Fix Our Web Crawler

Our web crawler was returning entire pages. A question like "what is the return policy" would get the whole product page back. A 5000 word article when we only needed 200 words.

This was wasteful. One page used 50,000 tokens. The model got confused reading all the noise and started making things up.

We used sentence-transformers/all-MiniLM-L6-v2 to filter. It converts text into numbers that capture meaning. Similar meanings land close together. So it finds chunks that match the question's meaning, not just keywords.

Result: 5,000 tokens per page instead of 50,000. 90 percent savings. No more hallucinations.

If you deal with web crawling or large text, this model is worth trying

4 Upvotes

100% Upvoted

1 comment sorted by

1

u/eldrugo85 2d ago

Did the same on a pdf/epub pipeline, hierarchical notes with embeddings and a rerank on top. What bit me was chunk size. MiniLM truncates past 256 tokens, so my longer chunks were half invisible in the index and I didn't notice for a while. Going smaller retrieves cleaner but strips the context around the hit, which is where the rerank earned its keep. Are you cutting on sentences or fixed windows?