In a materials literature search pipeline, after initially gathering candidates broadly with BM25 or embedding search, it would be good to try guiding the reranker so that it includes synthesis route, characterization, composition range, and failure case simultaneously. A simple rubric can be defined first based on existing search logs or researcher preferences.
At a small scale, you can create a checklist for a “good search result set” for one specific materials topic, and compare the current search system with the rubric-reranker. It is useful to examine not only search accuracy, but also literature diversity, reduction of missing information, and downstream reading time.