adding @Miriam and @nettrom_WMF because we've had some prior discussions about evaluating grid results. Previously we've decided not to take position in the grid into account because there's no agreed upon way to do so, but maybe things have changed.

Just on AC number 3 - we (structured data) are currently kinda doing this using https://media-search-signal-test.toolforge.org/

I took a list of the top 1000-odd queries on the commonswiki index (that I had got from superset a few months ago), plus a list of 1000 random queries I got off @TJones , and ran mediasearch for each one with the bost for each field except one set to zero (and the exception had its boost set to 1). Recorded the score and position of each image in the top 100 and saved them, and that's what's being used to supply the images for the tool at the url above

So now we have a list of images, with a search term used to the find them and the elasticsearch score returned for an individual elasticsearch field, with a rating of -1 (bad), 0 (neutral) or 1 (good). 7777 ratings so far, hoping to get to 1000 ratings per search signal (auxiliary_text, caption, category, heading, redirect.title, suggest, text, title, statement_keywords (we tried file_text too but almost all the scores returned were zero))

@dcausse tbh I don't know exactly what relforge does, but as you know we've gathered a set of labeled search results, and written some code to compare resultsets from different search algorithms (see T271801). Will this ticket add anything to that?

@Cparle I think the idea would be to consolidate what you've built with the existing tools the search team uses. Few examples:

• there's a runSearch.php in CirrusSearch whose purpose is close to runSearches.php you wrote
• there are scripts to compute some scores based on a labelled dataset relforge_engine_score/scorers.py but does not implement an F measure you seem to use

repo: https://gerrit.wikimedia.org/g/wikimedia/discovery/relevanceForge

@matthiasmullie @Cparle The search team is trying to understand why SD chose to build their own tools instead of using the existing tools. They want to own one set of tooling only, and so they need to understand what was missing from the previously existing tools, so they know how to improve (or whether to replace) the existing tools (e.g. with Rated Ranking Evaluator). Can you provide some input? Thanks!

We just didn't realise that it was possible to compute precision/recall/etc scores based on a labeled dataset using relforge. The readme mostly covers how to generate diffs of results from applying different config options to sets of queries. I talked about relforge a bit with the search team, but I guess I didn't really understand exactly what I needed a tool to do until I had written a new one, and so failed to ask them the right questions. FWIW we are not attached to our own tool at all and are more than happy to switch over to using relforge instead (once we figure out how)