Delivered-To: tjphuhs@gmail.com Received: by 2002:adf:f3cb:0:b0:441:282b:bd with SMTP id g11csp981086wrp; Thu, 23 Apr 2026 09:19:00 -0700 (PDT) X-Received: by 2002:a05:690c:698a:b0:7a2:1f26:3d5f with SMTP id 00721157ae682-7b9ecf79bb4mr292805927b3.28.1776961140463; Thu, 23 Apr 2026 09:19:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1776961140; cv=none; d=google.com; s=arc-20240605; b=BdQYBAHHC1e7/36x9mas+TnGXxku9Iag8wLoFEasMqKUB/1bbAGX3HduFjD+UYPczg G+VRt2DZ9GNmFKZ1JBiZFmQxJybDMHZPvm5D88ytntg9xHd+OcExVMRO3S73OcRFeUWs wC0To6Xx9MwxcT+moKNljSWKA8b4P7Vrfkq3cBYpTUuM2kzNKg0HWqliy2YRtUI9X+ID fXlAc3JXICUst9qOuoh5N1pgvYHgP+zQdD/8jsWSulivzDZ8mNaisWQrnaGct6BQs5wF CMMId6+tOyuPFb1pYyqeeVJ6pPIPa/FiTTZio/GSYo38bnmQVc3mPfcITsl2o9Ko6ex1 zUOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=subject:mime-version:date:to:from:list-unsubscribe-post:message-id :list-unsubscribe:dkim-signature; bh=lMPZVC6EkYwa/oOHFv2E6FpUbCrKJjOARUYkoze7xZo=; fh=9zP21OW67FFZD2VV42mGudo540Qp4aalMWxJB5kMYug=; b=gAPh9ycKMiyQjz3hvkU40Z4jTolsG+MVrxDSvmBsuP+5ntONLgOtdM0w/Vc6dqOsRE 2jUOC7MKWWEzqiouINqM3O84EAM4282UB64W36rRIlH/2/ayUpt/RxL0bAKZFCDoWaOd +/ZH8/9Dz88/ItvUdZYQExDmVx+kGp14nNe5NmWjBLVGEMe6EzV44vzJfBSjyeNTS4le vGGxMKZ0mmoNbhV4Bn+qzWicRwG8ARvVWeltRe/FVsZTyWViWhUVEQIAYwPya4SnidAn onarAeHfhmYRteQSOZssVySbl+MaQXmwkGdzLtm29q7Pju/lcI56MUg/rKEdsJeIQPlh DU6A==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@mail.you.com header.s=scph1123 header.b=TPBV1MhP; spf=pass (google.com: domain of msprvs1=20573xxozauxo=bounces-266693-1082@mail.you.com designates 137.22.227.75 as permitted sender) smtp.mailfrom="msprvs1=20573XXoZAUXo=bounces-266693-1082@mail.you.com"; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=mail.you.com Return-Path: Received: from ipyoyc3c.mail.you.com (ipyoyc3c.mail.you.com. [137.22.227.75]) by mx.google.com with ESMTPS id 00721157ae682-7baef46a117si133180757b3.99.2026.04.23.09.19.00 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 23 Apr 2026 09:19:00 -0700 (PDT) Received-SPF: pass (google.com: domain of msprvs1=20573xxozauxo=bounces-266693-1082@mail.you.com designates 137.22.227.75 as permitted sender) client-ip=137.22.227.75; Authentication-Results: mx.google.com; dkim=pass header.i=@mail.you.com header.s=scph1123 header.b=TPBV1MhP; spf=pass (google.com: domain of msprvs1=20573xxozauxo=bounces-266693-1082@mail.you.com designates 137.22.227.75 as permitted sender) smtp.mailfrom="msprvs1=20573XXoZAUXo=bounces-266693-1082@mail.you.com"; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=mail.you.com Return-Path: X-MSFBL: KCTd+TZKtE5L40PZNouj3B7uAlCnphSXpxezOS7AAKE=|eyJtZXNzYWdlX2lkIjo iNjllNmJhNDFlYTY5M2NjNjVlNTMiLCJyIjoidGpwaHVoc0BnbWFpbC5jb20iLCJ 0ZW5hbnRfaWQiOiJzcGMiLCJzdWJhY2NvdW50X2lkIjoiMTA4MiIsImN1c3RvbWV yX2lkIjoiMjY2NjkzIn0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mail.you.com; s=scph1123; t=1776959930; i=@mail.you.com; bh=lMPZVC6EkYwa/oOHFv2E6FpUbCrKJjOARUYkoze7xZo=; h=List-Unsubscribe:Message-ID:List-Unsubscribe-Post:From:To:Date: Content-Type:Subject:From:To:Cc:Subject; b=TPBV1MhPqtd5bGCtSOpe55pf6lCNFakuXADWzdK0Zxsl8mcdd2xYpNzQEtSP5Lg8s 9IjtTDc/p2Y217+5XixWgTql7Eol5Z5FXEtsz9W5F3fYZn+zoUdTjdI8QFsw4TFQ/D IEtJm/7E1W6kauygZSRCG43sQdnCeCHccJu6X1yo= Received: from [10.90.12.143] ([10.90.12.143]) by i-0f4601db939949937.mta1vrest.sd.prd.sparkpost (ecelerity 5.2.0.75575 r(msys-ecelerity:tags/5.2.0.17)) with REST id 35/ED-19534-AB14AE96; Thu, 23 Apr 2026 15:58:50 +0000 List-Unsubscribe: , Message-ID: <35.ED.19534.AB14AE96@i-0f4601db939949937.mta1vrest.sd.prd.sparkpost> List-Unsubscribe-Post: List-Unsubscribe=One-Click From: "You.com Product Team" To: tjphuhs@gmail.com Date: Thu, 23 Apr 2026 15:58:50 +0000 Content-Type: multipart/alternative; boundary="_----7YSw6aCJ7vTZ6rPwEmwSkg===_63/ED-19534-AB14AE96" MIME-Version: 1.0 Subject: Open-source web search evals you can run in minutes --_----7YSw6aCJ7vTZ6rPwEmwSkg===_63/ED-19534-AB14AE96 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" Run standardized evals (SimpleQA, FRAMES, BrowseComp,=0ADeepSearchQA) in yo= ur own environment=E2=80=94no custom infra required.=0AHi there,=0A=0AWhen = building AI systems that rely on web search, search quality=0Adetermines mo= del quality.=0A=0AYou shouldn=E2=80=99t have to build eval infrastructure f= rom scratch,=0Arely on results that can=E2=80=99t be reproduced, or spend w= eeks=0Adebugging benchmarks instead of building your product.=0A=0AWe built= something to do that for you.=0A=0A***************************************= **********=0AThe You.com Web Search Eval Harness (open source)=0A=0A*******= ******************************************=0A=0AAn evaluation framework tha= t lets you benchmark any web search=0Aprovider using industry-standard data= sets consistently and in=0Ayour own environment.=0A=0A************=0AWhat y= ou get=0A=0A************=0A=0ARun 4 widely used benchmarks out of the box= =0A=0A* SimpleQA =E2=86=92 factual accuracy on direct questions=0A* FRAMES = =E2=86=92 multi-hop structured reasoning=0A* BrowseComp =E2=86=92 multi-sou= rce navigation and synthesis=0A* DeepSearchQA =E2=86=92 deep research-style= completeness tasks=0ATogether, they cover everything from factual retrieva= l to=0Aproduction-grade research workflows.=0A=0A********************=0AWha= t it does for you=0A********************=0A=0AInstead of building eval infr= astructure from scratch, the=0Aharness handles:=0A=0A* dataset loading=0A* = query execution across providers=0A* answer synthesis=0A* standardized scor= ing (based on original benchmark methods)=0ASame inputs =E2=86=92 consisten= t outputs =E2=86=92 comparable results across=0Aproviders=0A=0A************= =0AHow it works=0A************=0A=0AClone =E2=86=92 add API keys =E2=86=92 = run:=0A=0Apython src/evals/eval_runner.py --samplers=0Ayou_search_with_live= crawl --datasets simpleqa=0A=0AOr run everything by default. Results are sa= ved incrementally so=0Along runs are safe and resumable.=0A=0A*************= ****=0AWhat you get back=0A*****************=0A=0AA single file that includ= es:=0A=0A* accuracy per benchmark=0A* latency per provider=0A* side-by-side= comparisons across search systems=0A=0A************************=0AWhy we o= pen-sourced this=0A************************=0A=0AWe=E2=80=99re trying to ma= ke it easier to:=0A=0A* evaluate search providers yourself=0A* trust your o= wn results (not snapshots or marketing claims)=0A* focus engineering time o= n building real AI workflows, not eval=0Ainfrastructure=0A=0A**************= ******************************=0AGet Started with the Web Search Eval Harne= ss=0A=0A********************************************=0A=0ARepo: https://cli= cks.you.com/f/a/_iffcyi0pftCMdIvio3oOg~~/AAQRxRA~/XJ5TR_jpItAjWmAmb7nPgh4v-= W5tJ_rjDZ-Hd9_X1DL4ZcKkY79ha9dvLQj6oFYhZBKCIo9e1MAwxtcFsxq2yLjiyJr5dmHwg24a= 8QOmp0171alqZrrhK4knbmCJR9wZqoy9wujzQS791LWoEMEZTA~~=0A=0AYou'll need:=0A= =0A* You.com API key =0A( https://clicks.you.com/f/a/jAyoQbXm0oSnaOcWS7s1hQ= ~~/AAQRxRA~/6HSi8uKwRSCtaX5P2Ffdf2HO9N980F-a1H6OND8MUoBINX76-aEkl2eqneCpL6M= lfw8g0nhA-WWs2I0dxsmnDWIyBItgw9aptbYzEhAfd990OvbThRghCEjOoFC21FVE2dgrNJdP2t= Lz3w_0fBg22zI-afxFMCI88wFgS5ieDv1nzYAN6WKTRL77jVa4Lw2qNYRu0WA17o1lEeiXP8RxY= Tg3bSvj4sV0V6dQSASalFN0KGvgQJEOtxRw1J251DB-OJmw9exNbxe0nvIaAXey_qui7XrS64vb= ixHmkUFIVaN8tPXEsySDozvM4PhpLaP1 )=0A=0A* OpenAI API key (for grading and s= ynthesis models)=0ARead the article =0A( https://clicks.you.com/f/a/U50dtO= pq7oEOaC87xK_YzQ~~/AAQRxRA~/ft6Rxy7M8UPYx2KSHMYp4Ph9bFrhxq3Wh_kWVtq75ILrB18= HRI7ZBEOA4OG-9DwN1iz9Ql7_pP_rNaCPNKxlE_Qk3MnLUL8ss_rXhQx4DugI7S00cUhNBdR2Js= dJgpJYle3wpxwu0zQLGcCebnjBDI3AkgkWDCCRrS45UIh0easQylinHRCFCNvjCL7jrtgQnSn7H= oyROO4GUxbRMVL0COi7KxDJEjhSBSiwNoKJTpVBvRhDSK82wSkp0G5K5NiGBGKuOx6MV8swh6Tn= kBu1lwWf2HdL-an7yBh4K3RAgWgBFZcOXOamnqPZ5WFLyIPlboxwm5Qw_6WtGaAUE0CvIRHQwOF= QzOkorKQ4pYr4xAA~ )=0A -The You.com Team=0A=0ATwitter ( https://clicks.yo= u.com/f/a/4CL2lg7bgeTBgXIO1JEGbA~~/AAQRxRA~/nvXoNW36mY-0nwVuiuOMcUTz_b5o_g8= tRzQAigBoedcmxOLzadg-7YgdJ7tYtneJzYB9LSHumzE2PX-ItJpj2FsOI0kNjikVoBpOK1rUaZ= ivA8t3bBaPjKSkMjVGywrW ) LinkedIn =0A( https://clicks.you.com/f/a/W1mu3R2Z= 4KPLOTRqmt2MGA~~/AAQRxRA~/qEVrBWOBkXYZllVEgGDSPx1dy4Os7N6mf9-lkDxMfKOKIXhx0= yqAiBuMiPyIE-19YrGg730GGVtUB9upWGo09i_1RE_CDoYVgwS9vZJkqF2FR2-t8TfJ0N4Cz6Ts= PIHC ) Discord =0A( https://clicks.you.com/f/a/v-mk6amyvdmBIyJj-cJtTA~~/AA= QRxRA~/WfsK8k0nrZNBTXYAHI9nIS-AMye37Nx40n1HohNiSSZg3Vfbwf72xzm2RNxhkG2fV8sj= xHWUsOU2u0q7NQt_QGq4geZFPvnU9vb73-_SftUBI1tLBn14W-Ato1xxk84N )=0AUnsubscrib= e ( https://clicks.you.com/f/a/0pTBPBQ0jpEpwbxyCct80Q~~/AAQRxRA~/AFkvTu9Baw= wD_C4uX8EOayt9kcSgBeZZgOd91s4OcygSt4FFCPVyzmZZpK3jP2u9N5xVCK2vGAxU0hapC3S23= j-d7l_s76EAVCmlLlVrr068SnBfyNDbVFCtA_ZovtdfhhwsyXRt6tQJQ1E1uspuVcUnvIxTS3zF= uOGn0EOFDlMr6_zSEanKZY0OHYr98j8LAs60mMApNS1xUtUN9ZsMDdc2UJ0cLU2hEuU0kIjvg3S= 75fm9AUMVPqi3X90M1CtOfhaSS6uG14mqPzP8h1HnSb8Ks-W-r_vxLNVJCRJ5RinsGvC5L541Ij= nqOlOSU0SaJbkfzv_WsoGLx5xvba9RvHKZ9XgcKUfaO5KcIFT4ry6kjbSx5055ZBaIJChrvUBD1= wmgz-liZkphPWDzuVsotYIOeo5d_A6kfIqmZzxUqchKuGC_IA2aP7LKRs5V87pgEPc6Nxrm2Kzx= O4xhZomeCSvtSyKs__oHZaQyDxjW5PI9XZINLDHrvFzRJw_1V2lJgu9fyfi03wVRfXLG6TBElZ5= fqSooTYOq814kHjhHRzGEDlgY6UJr8zl7DX_5aMGDRuQQpoHALu2gUvGMHMTikPinlKBSYwce3G= 4W-qXFrdR9_imMVowZd3X5JdxpQdU9BrcJCxFU1k6LRbtAmksrkw~~ )=0A=0AYou.com=0A228= Hamilton Ave, Fl 3=0APalo Alto, CA 94301 --_----7YSw6aCJ7vTZ6rPwEmwSkg===_63/ED-19534-AB14AE96 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="UTF-8" =0A=0A=0A=0A=0A=0A=0A=0A=0A=0A
Run standardized evals (Si= mpleQA, FRAMES, BrowseComp, DeepSearchQA) in your own environment=E2=80=94n= o custom infra required.=0A
=0A<= /tr>
=0A
3D""
=
=0A
=0A
=0A

Hi there,

=0A

When building AI systems that rely on = web search, search quality determines model quality.

=0A

=0AYou shouldn=E2=80=99t have to build eval infras= tructure from scratch, rely on results that can=E2=80=99t be reproduced, or= spend weeks debugging benchmarks instead of building your product.

=0A<= p style=3D"margin:0">We built something to do that for you.

=0A
=0A
=E2=80=8A
=0A<= table class=3D"heading_block block-3" width=3D"100%" border=3D"0" cellpaddi= ng=3D"0" cellspacing=3D"0" role=3D"presentation" style=3D"mso-table-lspace:= 0;mso-table-rspace:0">

The You.= com Web Search Eval Harness (open source)

= =0A

=0AAn evaluation framework tha= t lets you benchmark any web search provider using industry-standard datase= ts consistently and in your own environment.

=0A=

What you get

=0A=0A

Run 4 widely = used benchmarks out of the box

=0A
=0A
    =0A
  • SimpleQA =E2=86=92 factual accuracy on direct questi= ons
    • =0A
  • FRAMES =E2=86=92 multi-hop struct= ured reasoning
    • =0A
  • Brows= eComp =E2=86=92 multi-source navigation and synthesis
    • =0A
  • DeepSearchQA =E2=86=92 deep research-styl= e completeness tasks
    • =0A
=0A=0A

Together, they cover everything from factual = retrieval to production-grade research workflows.

=0A

What it does for= you

=0A
=0A

=0AInstead of building eval infrastructure from scratch, = the harness handles:

=0A
=0A
    =0A
  • dataset loading
    • =0A
  • query execution across providers
    • =0A
  • answer synthesis
    • =0A
  • standardized scoring (based on orig= inal benchmark methods)
    • =0A
=0A
=0A=0A

=0ASame inputs =E2=86=92 consistent outp= uts =E2=86=92 comparable results across providers

How it works

=0A

Clone =E2=86=92 add API keys =E2=86=92 run:

= =0A

=0A    &nb= sp;  python src/evals/eval_runner.py --samplers you_search_with_l= ivecrawl --datasets simpleqa

=0A=0A

Or run everything by default. Results are sav= ed incrementally so long runs are safe and resumable.

What you get b= ack
=0A

A single file that includes:

=0A
    =0A
  • a= ccuracy per benchmark
    • =0A
  • side-by-side comparisons across search systems
    • =0A
=0A

Why = we open-sourced this

=0A
=

We=E2=80=99re trying to make it easier to:

<= /td>
=0A
    =0A
  • = =0Aevaluate search providers yourself
    • =0A
  • trust your own results (not snapshots or marketing claims= )
    • =0A
  • focus engineering = time on building real AI workflows, not eval infrastructure
    • =0A
=0A=0A
=E2=80=8A
=0A

G= et Started with the Web Search Eval Harness=0A

=0A=0A
=E2=80=8A
=0A
<= div class=3D"alignment" align=3D"center">= =0ARead the article=0A<= /a>
=0A
=E2=80=8A
=0A

-The You.com Team

= =0A
=E2=80=8A
=0A
=E2=80=8A
=0A
=0A
=0A
=0A=0A=0A=0A
3D"Twitter"=0A3D"Linke== =0A3D"Discord"
=0A
=0A

=0A=0A=09=0A=09=09=0A=09=09=09=0A=09=09=0A=09=0A=0A
=0A=09=09=09

Unsubscribe=0A=09=09=09
=0A=09=09=09You.com
=0A= =09=09=09228 Hamilton Ave, Fl 3
=0A=09=09=09Palo Alto, CA=C2=A094301

=0A=09=09=09
=0A
=0A
=0A=0A=0D=0A3D""=0D=0A=0A --_----7YSw6aCJ7vTZ6rPwEmwSkg===_63/ED-19534-AB14AE96--