The Text Retrieval Conference Call for Papers

CALL FOR PARTICIPATION
TEXT RETRIEVAL CONFERENCE (TREC) 2024
February 2024 - November 2024


Conducted by:
National Institute of Standards and Technology (NIST)

The Text Retrieval Conference (TREC) workshop series encourages
research in information retrieval and related applications by
providing a large test collection, uniform scoring procedures, and a
forum for organizations interested in comparing their results. Details
about TREC can be found at the TREC web site, http://trec.nist.gov.

You are invited to participate in TREC 2024. TREC 2024 will consist of
a set of tasks known as "tracks". Each track focuses on a particular
subproblem or variant of the retrieval task as described
below. Organizations may choose to participate in any or all of the
tracks. Training and test materials are available from NIST for some
tracks; other tracks will provide instructions for dataset download.

Dissemination of TREC work and results other than in the (publicly
available) conference proceedings is welcomed, but the conditions of
participation specifically preclude any advertising claims based on
TREC results. All retrieval results submitted to NIST are published in
the Proceedings and are archived on the TREC web site with the
submitting organization identified.

TREC participants are added to the TREC Slack instance, and the
primary mode of communication in TREC is Slack.  There is a general
mailing list (trecXXXX@nist.gov) but this is used for major
announcements only.  Some tracks have mailing lists which you should
follow if you are interested in those tracks.

Schedule
--------

 - As soon as possible -- submit your application to participate in
 TREC 2024 as described below.

Submitting an application will add you to the active participants'
mailing list. On March 1st, NIST will announce a new password for the
"active participants" portion of the TREC web site. We accept
applications to participate until late May, but applying earlier means
you can be involved in track discussions. Processing applications
requires some manual effort on our end. Once your application is
processed (at most a few business days), the "Welcome to TREC" email
message with details about TREC participation will be sent to the
email address provided in the application.

 - June--August
Results submission deadline for most tracks. Specific deadlines for
each track will be included in the track guidelines, which will be
finalized in the spring.

 - September 30 (estimated)
Relevance judgments and individual evaluation scores due back to participants.

 - Nov 18--22
TREC 2024 in-person conference at NIST in Gaithersburg, MD, USA with a remote attendance option

Task Description
----------------

Below is a brief summary of the tasks. Complete descriptions of tasks
performed in previous years are included in the Overview papers in
each of the TREC proceedings (in the Publications section of the web
site).

The exact definition of the tasks to be performed in each track for
TREC 2024 is still being formulated. Track discussion takes place on
the track mailing list (or other communication medium). To join the
discussion, follow the instructions for the track as detailed below.

TRECVID, TREC's sister evaluation of multimedia understanding, and
TAC, TREC's sister evaluation of NLP, have been folded back into TREC.
TREC 2024 thus has *fourteen* tracks.  Eight of the tracks are
continuing: AToMiC, AVS, iKAT, MedVidQA, NeuCLIR, Product, ToT, and
VTT.  The six new tracks this year are BioGen, CCU, Lateral Reading,
PLABA, RAG, and RUFEERS.

There are two groups of tracks with a somewhat common focus.  AToMiC,
AVS, MedVidQA, Product, and VTT are all multimedia tasks.  IKAT,
NeuCLIR, BioGen, Lateral Reading, PLABA, and RAG all have a generative
aspect to the task.  (The remaining three tracks, ToT, CCU, and
RUFEERS defy categorization.)


Activities in Extended Video (ActEV)
------------------------------------
The Activity in Extended Video (ActEV) challenge main focus is on
human activity detection in multi-camera video streams.

Activity detection has been an active research area in computer vision
in recent years. The ActEV evaluation is being conducted to assess the
robustness of automatic activity detection for a multi-camera
streaming video environment.

The evaluation is based on a portion of the MEVA
(https://mevadata.org/) Known Facility (KF) datasets and run as a
self-reported leaderboard evaluation.

The evaluation is based on 20 activities as described here:
https://actev.nist.gov/SRL#tab_activities . For both the Activity and
Object Detection (AOD, primary task)

and Activity Detection (AD) tasks, the submitted results are measured
by Probability of Missed Detection (Pmiss) at a Rate of Fixed False
Alarm (RateFA) of 0.1 (Pmiss@0.1RFA)

TRECVID ActEV 2024 test dataset release: March 15, 2024
ActEV SRL Challenge Opens: May 10, 2024
Deadline for ActEV SRL Challenge results submission: September 15, 2024: 4:00 PM EST

Track Web Page: https://actev.nist.gov/SRL


Ad-hoc Video Search (AVS)
-------------------------
This track will evaluate video search engines on retrieving relevant
video shots satisfying textual queries combining different facets such
as people, actions, locations and objects.

The testing dataset is the V3C2 (Vimeo Creative Commons) with total
1.3 Million video shot and average duration of about 9 min. The task
will test systems on 20 new queries and 20 progress queries (fixed
from 2022 - 2024) for a total of 40 queries.

Anticipated timeline: submissions due end of July.

Track coordinators:
Georges Quenot, LIG
George Awad, NIST

Track Web Page: https://www-nlpir.nist.gov/projects/tv2024/avs.html


AToMiC Track
------------
The Authoring Tools for Multimedia Content (AToMiC) Track aims to
build reliable benchmarks for multimedia search systems. The focus of
this track is to develop and evaluate IR techniques for text-to-image
and image-to-text search problems.

Anticipated timeline: Document (Image and Texts) collection available
in January, evaluation topics in May/June, final submissions due in
July.

Track coordinators:
  Jheng-Hong (Matt) Yang, University of Waterloo
  Carlos Lassance, Naver Labs Europe
  Mariya Hendriksen, University of Amsterdam
  Thong Nguyen, University of Amsterdam
  Andrew Yates, University of Amsterdam
  Krishna Srinivasan, Google Research
  Miriam Redi, Wikimedia Foundation
  Jimmy Lin, University of Waterloo

Track Web Page: https://trec-atomic.github.io/
Mailing list: Google group, name: atomic-participants


Biomedical Generative Retrieval (BioGen)
----------------------------------------
The track will evaluate technologies in the domain of biomedical
documents retrieval. Specifically those with generative retrieval
capabilities.

Documents including literature abstracts from the U.S. National
Library of Medicine (MEDLINE) with over 30 million abstracts will be
utilized.

Anticipated timeline: TBD

Track coordinators:
Kirk Roberts, UTHealth
Dina Demner-Fushman, NIH
Steven Bedrick, Oregon Health & Science University
Bill Hersh, Oregon Health & Science University

Track Web Page: TBD


Computational Cultural Understanding (CCU)
-------------------------------------------
This new track focuses on the detection of sociocultural norms in
video recordings of naturally occurring interactions between two or
more people conversing in Mandarin Chinese. Successful communication
entails not only knowing the local language but also understanding the
local cultures and customs. Violation of cultural norms may derail a
conversation and lead to disastrous consequences. As such, detecting
social norms and determining if a speaker is adhering to or violating
them are foundational components in dialogue assistance applications
to facilitate successful communication between individuals who do not
speak a common language and are not familiar with each otherís
culture.

The evaluation will employ a set of about 2500 video recordings of
people in some type of conversations with each other in Mandarin
Chinese.

Anticipated timeline: runs due mid-September 2024

Track coordinators:
Audrey Tong, NIST
Jonathan fiscus, NIST
Leora Morgenstern, SRI
Stephanie Strassel, LDC

Track Web Page: https://www.nist.gov/itl/iad/mig/computational-cultural-understanding-open-evaluation-openccu


Interactive Knowledge Assistance Track (iKAT)
---------------------------------------------
iKAT is about conversational information seeking search.  It is the
successor to the Conversational Assistance Track (CAsT). TREC iKAT
evolves CAsT to focus on supporting multi-path, multi-turn,
multi-perspective conversations. That is for a given topic, the
direction and the conversation that evolves depends not only on the
prior responses but also on the user.  Users are modeled with a
knowledge base of prior knowledge, preferences, and constraints.

Anticipated timeline: runs due August 15

Track coordinators:
  Mohammed Aliannejadi, University of Amsterdam
  Zahra Abbasiantaeb, University of Amsterdam
  Shubham Chatterjee, University of Glasgow
  Jeff Dalton, University of Glasgow

Track Web Page: https://trecikat.com
Mailing list: Google group, name: trec_ikat


Lateral Reading
---------------
Detection of misinformation and its evaluation is constrained by the
need to define truth. Lateral reading is a process people can use to
determine the trustworthiness of information through asking questions
about document sources and evidence and seeking answers via search
engines.

This track's goal is to develop technologies to support and encourage
the use of lateral reading. The track will organize 2 subtasks:

1- Question Generation (given 50 news articles from
ClueWeb22B-English, produce 20 questions for each article)

2- Document Retrieval (given ClueWeb22B-English and 50 news articles
with pooled questions from task 1, return top-10 documents/passages as
sources of answers)

Anticipated timeline: runs due end of June/early July.

Track coordinators:
Mark Smucker, University of Waterloo
Charlie Clarke, University of Waterloo
Dake Zhang, University of Waterloo

Track Web Page: https://trec-lateral-reading.github.io/


Medical Video Question Answering (MedVidQA)
-------------------------------------------
This track aims to use medical instructional video corpus to evaluate
systems on retrieving and localizing visual answers given a medical
query and collection of videos.

Another subtask will include generating a step-by-step textual summary
of the visual instructional segment in a given medical video and a
query.

Anticipated timeline: runs due in mid-August. Track coordinators: Deepak Gupta, NIH Dina Demner-Fushman, NIH Track Web Page: https://medvidqa.github.io/ NeuCLIR Track ------------- Cross-language Information Retrieval (CLIR) has been studied at TREC and subsequent evaluation forums for more than twenty years, but recent advances in neural approaches to information retrieval (IR) warrant a new, large-scale effort that will enable exploration of classical and modern IR techniques for this task. The task will support 3 languages (Russian, Chinese, and Farsi) and evaluate 3 subtasks: - CLIR/MLIR Tasks: Given English topic, return ranking of Russian, Chinese, and Farsi news documents (MT to English provided by organizers). - Technical Docs Task: Given English topic, return ranking of Chinese technical abstracts. - Report Generation Pilot (2024): Generate a multi-paragraph report in English based on information supported in the Chinese, Russian, or Persian document collection. Anticipated timeline: Document collection available in January, submissions due in August. Track coordinators: Dawn Lawrie, Johns Hopkins University Sean MacAvaney, University of Glasgow James Mayfield, Johns Hopkins University Paul McNamee, Johns Hopkins University Douglas W. Oard, University of Maryland Luca Soldaini, Allen Institute for AI Eugene Yang, Johns Hopkins University Track Web Page: https://neuclir.github.io/ Mailing list: Google group, name: neuclir-participants PLABA ------- The goal of the PLABA track is to improve health literacy by adapting biomedical abstracts for the general public using plain language. Anticipated timeline: TBD Track coordinators: Brian Ondov, NIH Dina Demner-Fushman, NIH Hoa Dang, NIST Track Web Page: https://bionlp.nlm.nih.gov/plaba2024/ Product Search Track -------------------- The product search track focuses on IR tasks in the world of product search and discovery. This track seeks to understand what methods work best for product search, improve evaluation methodology, and provide a reusable dataset which allows easy benchmarking in a public forum. The task for systems will be: given a search query, find relevant products. Documents include Amazon products corpus (title, description, etc.). The goal this year is to encourage multimodal queries (image and text) Anticipated timeline: Runs due end of June/early July Track coordinators: Daniel Campos, University of Illinois at Urbana-Champaign Corby Rosset, Microsoft Surya Kallumadi, Lowes ChengXiang Zhai, University of Illinois at Urbana-Champaign Sahiti Labhishetty, University of Illinois at Urbana-Champaign Alessandro Magnani, Walmart Track Web Page: https://trec-product-search.github.io/ Retrieval Augmented Generation (RAG) ------------------------------------- The RAG track aims to enhance retrieval and generation effectiveness to focus on varied information needs in an evolving world. Data sources will include a large corpus and topics that capture long-form definitions, list, and ambiguous information needs. The track will involve 2 subtasks: 1- Retrieval Task : Rank passages for a given queries 2- RAG Task : Generate answers with supporting passage attributes The second task takes the primary focus of the track. Anticipated timeline: runs due end of July. Track coordinators: Ronak Pradeep, University of Waterloo Nandan Thakur, University of Waterloo Jimmy Lin, University of Waterloo Nick Craswell, Microsoft Track Web Page: https://trec-rag.github.io/ Recognizing Ultra Fine-grained Entities, Events, and Relations (RUFEERS) ---------------------------------------------------------------------- The RUFEERS track will evaluate IE systems that identify entities, events, and relations of interest, including the roles (if any) that the entities play in the events and relations. The track will evaluate systems on three tasks. Given an ontology and a set of Washington Post articles: - Task 1: Extract one mention of each event, relation, and argument from each document - Task 2: Extract all mentions of events, relations, and their arguments from each document - Task 3: Extract all mentions of each entity from each document Anticipated timeline: deadline will likely be mid-September. Track coordinators: Shudong Huang, NIST Hoa Dang, NIST Track Web Page: https://tac.nist.gov/2024/RUFEERS/ Tip-of-the-Tongue Track ----------------------- The Tip-of-the-Tongue (ToT) Track focuses on the known-item identification task where the searcher has previously experienced or consumed the item (e.g., a movie) but cannot recall a reliable identifier (i.e., "It's on the tip of my tongue..."). Unlike traditional ad-hoc keyword-based search, these information requests tend to be natural-language, verbose, and complex containing a wide variety of search strategies such as multi-hop reasoning, and frequently express uncertainty and suffer from false memories. The track will run 2 subtasks: - Movie identification (230k wikipedia corpus, given a ToT query, rank Wikipedia articles) - ToT known-item search for new domains with ToT query elicitation (domains include landmarks, celebrities, recipes, objects, etc.) Anticipated timeline: Results due August 31 Track coordinators: Jaime Arguello, University of North Carolina Samarth Bhargav, University of Amsterdam Bhaskar Mitra, Microsoft Research Fernando Diaz, Google Evangelos Kanoulas, University of Amsterdam Track Web Page: https://trec-tot.github.io/ Twitter: @TREC_ToT Mastodon: @TREC_ToT@idf.social Video-to-Text (VTT) ------------------- The video-to-Text track aims to evaluate video captioning systems. Given a short video clip of about 10 sec long, systems should generate a description in 1 sentence to include four important facets as applicable: Who: is in the video What: are they doing Where: are they doing it When: are they doing it The testing data will comprise about 2000 clips with a subtask to measure robustness of systems against real-world noise (e.g. camera shaking, etc) Anticipated timeline: runs due end of May/early June Track coordinators: Asad Butt, John Hopkins University George Awad, NIST Yvette Graham, UCD Afzal Godil, NIST Track Web Page: https://www-nlpir.nist.gov/projects/tv2024/vtt.html Conference Format The conference itself will be used as a forum both for presentation of results (including failure analyses and system comparisons), and for more lengthy system presentations describing retrieval techniques used, experiments run using the data, and other issues of interest to researchers in information retrieval. All groups will be invited to present their results in a joint poster session. Some groups may also be selected to present during plenary talk sessions. Application Details Organizations wishing to participate in TREC 2024 should respond to this call for participation by submitting an application. Participants in previous TRECs who wish to participate in TREC 2024 must submit a new application. To apply, use the new Evalbase web app at http://ir.nist.gov/evalbase. First you will need to create an account and profile, then you can register a participating organization from the main Evalbase page. Any questions about conference participation should be sent to the general TREC email address, trec (at) nist.gov. The TREC Conference series is sponsored by NIST's Information Technology Laboratory (ITL) Information Access Division (IAD) Retrieval Group