Checklist Reviewer Toolkit

Beyond Static Checklists

Hamed Hemati¹, Alicia Janz¹, Stefan Sandfeld¹

¹IAS-9, Forschungszentrum Jülich

Poster at the HMC Conference 2026 GitHub repo

Start a reproducibility review

ml_repro_paper.pdf

Drop PDF or click to upload

Uploading…

Checklist
Review process

Toolkit Purpose

How do we verify research when publications are scaling faster than human reviewers?

Manual

- Expensive

LLM-Based

- Hallucination

Agentic Workflow

- Powerful but difficult to implement

Toolkit Components

A complete, end-to-end pipeline for rigorous metadata assessment.

1

Data Collection

2

Review Process

3

Human Verification

4

Analysis

Dynamic Process Designer

Build complex review workflows visually.

Our toolkit features a node-based review process designer. Instead of hard-coding the logic, researchers can obtain various outputs for the same collection and checklist by composing tools as "agents".

  • Model-driven agentic process
  • Workflow transparency
  • Explainability for all outputs
Input (PDF)
Question Reviewer
Output (Markdown)

Click GitHub, Materials, or the agentic process card to open details.

Input PDF
Question reviewer
Local DB
Output Checklist / MD

Key Capabilities

Freedom of Choice

Choose your backbone model for each component. Supports local execution (Ollama) and remote (Google GenAI, LiteLLM) with customized reasoning levels.

External Tools

Agents can dynamically call external tools for "claim verification", querying databases, fetching code repositories, or running scripts.

Modular Extension

New components can be added without hard rewiring. Embrace non-deterministic, evolving workflows built on a plug-and-play architecture.

Poster at the HMC Conference 2026

Presented at the HMC Conference 2026
Checklist Reviewer Poster
View Full Poster
Download poster (PDF)