hkdse-subject-grading

HKDSE Subject Grading — Unified Skill

Overview

This skill grades HKDSE elective subject exam papers by extracting rubrics from PDFs, calibrating against prior-year reference data with known levels, then grading each student's typed/printed answer PDF on a per-question basis. It produces per-student JSON results and compiles them into final scores with level assignments (1–5).

Subjects supported: ICT, Music, Ethics and Religious Studies, Tourism and Hospitality Studies, Visual Arts, Biology, Economics, BAFS — and any similar subjects with typed written responses.

Key differences from Chinese Writing grading:

• No dedicated Chinese handwriting scoring dimensions (字體標點, 錯別字 are not standalone grading categories — VLM is still permitted for visual inspection where needed; see WARNING.md)
• No Chinese task 1 / task 2 split
• No per-dimension Chinese writing rubric (content, expression, structure, tone)
• Instead: unified per-question mark allocation from official rubrics

CRITICAL RULES

1. Sub-agents are mandatory. Each student MUST be graded by a dedicated sub-agent. No batch grading of multiple students in a single sub-agent call.

2. Model restrictions:

   • Sub-agents: SAME model as the main agent. All sub-agents MUST use the same model that the main (orchestrating) agent is running on. Do NOT pass a model parameter override to sub-agents — let them inherit the main agent's model.
   • VLM tasks: Kimi K2.5 ONLY. When VLM (vision-language model) API calls are needed — e.g., text extraction from scanned PDFs where PyMuPDF fails, or art creation grading requiring visual inspection — use Kimi K2.5 (kimi-k2.5) exclusively. NO OTHER VLM MODELS ARE ALLOWED.

3. Gemini VLM for PDF extraction: Use gemini-3.1-pro via the Google Vertex AI endpoint as the primary VLM for PDF-to-text extraction. Authenticate using the service account JSON (project-f154aafa-a809-44c8-89f-70e8abc5e53a.json) in the workspace root. A ready-to-use client is provided at scripts/vertex_client.py:

   from scripts.vertex_client import get_openai_client
   client = get_openai_client()
   

   The module handles service-account authentication, token refresh, and base-URL construction automatically. Use model name google/gemini-3.1-pro.

   If the Gemini endpoint is unavailable, rate-limited, or returns errors, fall back to Kimi (KIMI_BASE_URL, KIMI_API_KEY, KIMI_MODEL from env.txt).

4. ALIYUN as secondary fallback: If both Gemini and Kimi are unavailable or rate-limited, switch to the ALIYUN endpoint using ALIYUN_BASE_URL, ALIYUN_API_KEY, and ALIYUN_MODEL from env.txt. You may also use ALIYUN in parallel with Gemini/Kimi when running multiple extraction sub-agents simultaneously to avoid rate limits.

5. PDF rendering DPI: When converting PDF pages to images (e.g., for VLM extraction), default to 150 DPI to ensure text is legible. If processing speed is a concern, lowering DPI is acceptable as a trade-off, but do not go below 100 DPI.

6. No LLM/VLM API calls for generating feedback or analysis text. See WARNING.md. VLM is permitted only for tasks requiring genuine visual inspection — text extraction from scanned PDFs where PyMuPDF fails, and grading inherently visual student work (e.g., art creation, handwriting quality, diagrams). VLM must NOT be used for any dimension assessable from extracted text.

7. Read split-part PDFs in numerical order. Files named _part1.pdf, _part2.pdf, …, _partN.pdf MUST be read in sequence. Concatenate the extracted content in order. Ignore _original.pdf files (these are the unsplit source).

8. Reference data informs level assignment. Grade reference students to compute empirical score ranges per level. Agents may use these ranges, along with rubric criteria and official level descriptors, to define level boundaries. There are no hard restrictions on the level division method.

9. Always set the year before running any operation. Either pass --year YEAR to scripts or export GRADING_YEAR=YYYY at the start.

10. All report and commentary text must be written in Traditional Chinese (繁體中文). This applies to all narrative, feedback, analysis, and section content in generated DOCX reports. English is only permitted for: variable names, file paths, technical identifiers, chart axis labels, and JSON field names. Exception: if the subject being assessed is an English-language subject, English is used throughout.

Workspace Layout

{subject}-grading-workspaces/grading_workspace/
├── .github/skills/hkdse-{subject}-grading/SKILL.md
├── START.md
├── WARNING.md
├── env.txt                     # Local credentials (never commit)
├── env.txt.example             # Template
├── pyproject.toml
├── uv.lock
├── start.sh
├── data/                       # Symlink or mount to top-level data/
│   ├── reference/{subject}/{ref_year}/
│   │   ├── student_answers/level{L}_student{M}.pdf
│   │   ├── rubrics/ OR rubric_and_question/
│   │   ├── question/           (if separate)
│   │   └── reference_mapping.json
│   ├── masked_data/{subject}/{grade_year}/
│   │   ├── student_answers/student{N}.pdf
│   │   ├── rubrics/ OR rubric_and_question/
│   │   └── question/           (if separate)
│   └── groundtruth/{subject}/{grade_year}/groundtruth_mapping.json
├── rubric/{grade_year}/        # Generated rubric artifacts
│   ├── grading_guide.md
│   ├── reference_calibration.md
│   ├── reference_scores.json
│   ├── level_division.json
│   └── calibration/            # Intermediate calibration artifacts
│       └── (draft rubrics, reference grading outputs, score calculations)
├── rubric/reference_data_analysis/  # Insights from reference data (Phase 3)
│   ├── per_level_analysis.md        # Observed patterns per level 1–5
│   ├── rubric_gaps.md               # Gaps/ambiguities in official rubric
│   └── rubric_refinements.md        # Supplementary rubric guidance
├── scripts/                    # Python helper scripts
│   ├── generate_class_report.py
│   ├── generate_student_reports.py
│   ├── validate_extraction.py
│   ├── validate_grading_output.py
│   └── validate_reports.py
├── extracted/{grade_year}/     # Extracted student text
│   └── students/student{N}.txt
└── output/{grade_year}/        # Grading output
    ├── student{N}.json
    └── final_scores.json

Phase 1: Environment Setup

Step 1.1: Install Dependencies and Set Year

source env.txt
uv sync
export GRADING_YEAR={grade_year}   # Set from env.txt or manually

Step 1.2: Verify Data Availability

Confirm the following exist:

• Student answer PDFs in data/masked_data/{subject}/{grade_year}/student_answers/
• Rubric PDFs (check both rubrics/ and rubric_and_question/ directories)
• Question paper PDFs (if separate from rubrics)
• Reference data in data/reference/{subject}/{ref_year}/ with reference_mapping.json

Read BATCH_SIZE from env.txt (default: 5) for parallel sub-agent control.

Phase 2: Rubric Extraction

Step 2.1: Extract Rubric Content

Read ALL rubric PDFs from the masked data directory. Handle these patterns:

1. Single rubric file: rubrics/rubrics.pdf — extract directly
2. Split rubric parts: rubrics/rubrics_part1.pdf … rubrics_partN.pdf — read ALL parts in numerical order, concatenate content
3. Combined rubric + question: rubric_and_question/paper_part1.pdf … paper_partN.pdf — read ALL parts in order; both rubric criteria and question text are interleaved in these files
4. Level descriptors: If rubrics/level_descriptors.pdf exists, ALWAYS read it — it contains official level-band boundaries critical for level assignment

Use PyMuPDF (fitz) for text extraction. Fall back to VLM only if text extraction yields empty or garbled content.

Note on rubric extraction granularity: For rubric/criteria PDFs (which are typically typeset/printed), extracting multiple pages at a time is acceptable. However, for extremely long rubric files (e.g., 17+ split parts), consider extracting in manageable batches of a few pages each to avoid truncation or quality degradation.

Step 2.2: Extract Question Paper

If the question paper is separate from rubrics:

• Single file: question/paper.pdf — extract directly
• Split parts: question/paper_part1.pdf … paper_partN.pdf — read in order
• Files with non-standard names (e.g., Chinese filenames) — read all PDFs in question/

Step 2.3: Build Grading Guide

Create rubric/{grade_year}/grading_guide.md with:

• All questions/tasks listed with their question numbers
• Mark allocation per question and sub-question
• Marking criteria: what earns each mark or mark band
• Level-band boundaries (if present in rubric or level descriptors)
• Any special instructions (e.g., alternative acceptable answers)

Phase 3: Reference Calibration

Step 3.1: Read Reference Data

Read data/reference/{subject}/{ref_year}/reference_mapping.json to get the mapping of student IDs to known levels (1–5).

Format:

{
  "mappings": [
    {"student_id": 1, "filename": "level3_student1.pdf", "level": 3, ...},
    ...
  ]
}

Note: Reference files keep their original level-based names (level{L}_student{M}.pdf). The filename field in the mapping matches the actual file on disk. The level is also encoded in the filename, making it easy for agents to identify which level each student belongs to without parsing the JSON.

Step 3.2: Extract Reference Student Answers

Extract text from ALL reference student answer PDFs in data/reference/{subject}/{ref_year}/student_answers/.

Step 3.3: Read Reference Rubrics

Read rubric PDFs from the REFERENCE year directory (structure may differ from the grading year). Apply the same extraction rules as Phase 2.

Step 3.4: Grade Reference Students (Iterative Rubric Refine

---

Content truncated.