Exploring LLMs and RAG at the 2025 AI Winter School (Brown University)

At the 2025 AI Winter School, hosted by the Center for the Fundamental Physics of the Universe at Brown University, I led a 2.5-hour hands-on workshop on using large language models with physics-specific source material.

Participants compared direct model answers with answers generated from passages retrieved from LUX papers and Brown theses.

The corpus covered LUX dark matter calibrations and Brown Particle Astrophysics theses. We asked about the mean D-D neutron energy, the electric fields used in LUX yield measurements, and the origin and energy of low-energy 127Xe calibration events. The saved 127Xe example below shows what changed when the relevant thesis was added.

The Scientific Problem

For a numerical answer, I wanted the source location, units, and stated uncertainty. The checks were:

• which document was used
• which page or text region contained the answer
• whether the retrieved passage actually supports the claim
• whether the model preserved units, qualifiers, and uncertainty language
• whether the answer came from the requested source rather than adjacent but incompatible material

Retrieval Model

The RAG system in the notebooks used a standard dense-retrieval pipeline:

1. parse source documents from a Google Drive directory
2. split the extracted text into overlapping chunks
3. embed each chunk into a vector space
4. embed the user question into the same vector space
5. retrieve the top-ranked chunks by vector similarity
6. pass those chunks, plus the question, to the LLM

As a conceptual example, cosine similarity can rank a question against document chunks:

score(q, c_i) = cos(embed(q), embed(c_i))

This equation illustrates dense retrieval; it does not specify the notebook index’s similarity implementation. Retrieval does not guarantee a correct answer: parsing or ranking can omit the right passage, and the generator can misread a passage that was retrieved.

Two Implementations

The workshop used two parallel Colab notebooks so participants could see the same workflow with different model-serving assumptions.

The hosted path required API access. The Llama model ran in a GPU-backed Colab session with the notebook’s dependencies, Hugging Face model access, and enough GPU memory.

Indexing Parameters

Workshop settings, January 2025. Both historical notebooks pinned llama-index==0.12.3 and used:

Settings.chunk_size = 1000
Settings.chunk_overlap = 100

query_engine = index.as_query_engine(similarity_top_k=5)
response = query_engine.query(question)

• Chunk size: larger chunks preserve more local context, but make retrieval less selective and consume more prompt budget.
• Chunk overlap: overlap reduces boundary artifacts, especially when a definition, figure caption, or table explanation straddles a chunk boundary.
• Embedding model: the embedding model defines the retrieval geometry. It determines which passages are “near” the question before the LLM sees anything.
• Top-k retrieval: increasing similarity_top_k improves recall only if the relevant chunks are somewhere near the top of the ranking; it also adds more irrelevant text for the generator to reconcile.

PDF extraction can separate tables, equations, captions, units, and paragraph references. A chunk that is reasonable for prose may be too small for a table and its caption, while a chunk that preserves a table may be too broad for precise nearest-neighbor retrieval.

What We Inspected

Inspect response.metadata and response.source_nodes to see the retrieved evidence:

response.metadata
response.source_nodes

That output lets the user separate three different failure modes:

Checks for other answers include whether “about 2.45 MeV” preserves the source’s measured neutron energy and statistical/systematic uncertainties, whether a recoil endpoint retains keVnr, and whether neutron-source rates and S1/S2 signal sizes retain their measurement conditions. These are checks to apply, not a list of observed model failures.

Compare the answer with its source chunks, including units, assumptions, and uncertainties. If the trace is wrong, revise the corpus, chunking, embedding model, or prompt and query again.

Incremental Indexing

The notebooks also added new documents to an existing index. The first corpus used LUX D-D calibration papers; the second pass inserted Brown Particle Astrophysics theses:

new_documents = SimpleDirectoryReader(new_llama_index_data_path, recursive=True).load_data()
new_nodes = SimpleNodeParser().get_nodes_from_documents(new_documents)
index.insert_nodes(new_nodes)

A saved corpus-coverage example

The hosted-API notebook’s saved outputs record the same question before and after the thesis insertion:

How low in energy was the ER response measured using 127Xe? Where did the 127Xe come from?

Before insertion, the retrieved passages came from the D-D papers. One discussed cosmogenic 131mXe; the generated answer substituted that isotope and did not identify the 127Xe threshold. After insertion, the saved answer reported a lowest energy deposition of 186 eV and attributed the 127Xe to cosmogenic activation while the xenon was above ground.

The new retrieval trace includes the Huang thesis, pages 77–78 in the notebook metadata. The page-78 passage describes the calibration as “reaching all the way down to the observation of 186 eV energy deposition”; the page-77 passage attributes the isotope to cosmogenic activation before the xenon was moved underground. These are passages saved in response.source_nodes, so the answer’s energy, units, and origin can be checked against the retrieved text. Adding the thesis gave the retriever the passage containing the 186 eV result and the isotope’s origin.

Materials

• Workshop Recording
• OpenAI-based LLM Setup
• Llama-based LLM Setup (historical snapshot)
• Slides (PDF)
• 2025 AI Winter School Event Page