Acelab × Structured AI AEC Hackathon

Run your agent

Find the errors in the construction document set. Claim a team, point us at your repo, and we run it in a sandbox and score the output.

Team

Not signed in

Claim a team name

You get a secret team code. It is shown once — save it.

Already have a code

Event info

Links, docs, and the API keys your team uses tonight.

Sign in with your team name and code on the Submit tab to see the event info and your team's API key.

Leaderboard

Ranked by best test-run F1. Ties break by lower cost, then lower time. Final-run scores stay sealed until the end of the event.

Standings

—
#TeamF1PrecisionRecallCostTimeTest runsFinal

Rules

The contract, the environment, and how scoring works.

How it works

  1. Claim a team name. Save the secret code.
  2. Push your agent to a public GitHub repo with run.sh at the root.
  3. Launch a test run. We clone your repo into an isolated sandbox and execute run.sh.
  4. Your code reads the documents from $DATASET_DIR and writes its findings to $OUTPUT_PATH.
  5. We grade the output against the ground-truth manifest and post the score.
  6. When you are confident, launch your one final run against the hidden validation set.

The contract

Input: $DATASET_DIR is a folder of multiple PDF files. Enumerate the directory — do not hardcode file names. The practice set mirrors this format. The file names are the IDs: cite them exactly in your output's document field.

Output: write output.json to $OUTPUT_PATH:

{
  "errors": [
    {
      "document": "plumbing-spec.pdf",
      "category": "cross-document-conflict",
      "location": "page 3, section 2.1",
      "description": "Schedule specifies 2\" pipe; drawing P-101 shows 1.5\""
    }
  ]
}

Categories: cross-document-conflict, code-violation, unit-error, missing-item.

Environment

  • Python 3.10 and Node 22. The openai, httpx, requests, and pypdf Python packages are pre-installed.
  • OPENROUTER_API_KEY is set in the environment. Use it against https://openrouter.ai/api/v1. Any model, your choice.
  • Network is locked down. Allowed hosts: openrouter.ai, pypi.org, files.pythonhosted.org, registry.npmjs.org. Everything else is blocked and logged.

Limits and scoring

  • 3 test runs, 1 final run per team.
  • 10 minute wall-clock timeout per run.
  • Up to 300 LLM calls per run.
  • Score = F1 of matched errors (precision and recall — spamming guesses hurts you). Cost and wall-clock time break ties.
  • LLM cost is measured server-side through OpenRouter. It cannot be self-reported.