catalog-extractor

Technion Catalog JSON Extractor

Extract academic degree catalogs from Technion PDF course catalogs into structured JSON format for the Sogrim degree-completion checker.

Python Scripts

IMPORTANT: This skill uses pre-built Python scripts in packages/catalog-extractor/. Do NOT recreate these scripts from scratch — use and improve them instead.

Setup (one-time)

cd packages/catalog-extractor
pip install -r requirements.txt

Available scripts

Script	Purpose
extract_pdf.py	Downloads PDF (from URL or local path) and extracts text page-by-page
parse_sections.py	Splits extracted text into catalog sections by detecting bank headers
normalize.py	Utility: course ID padding, bank name detection, rule mappings
build_catalog.py	Builds skeleton catalog JSON from parsed sections
validate_catalog.py	Validates final JSON against expected schema
validate_against_pdf.py	Cross-validates a catalog JSON against the source PDF — checks courses, credits, banks, chains, overflows
main.py	All-in-one pipeline combining all steps

Quick pipeline

cd packages/catalog-extractor
python main.py <pdf_url_or_path> --name "Hebrew catalog name" --reference ../docs/ComputerScience3years2024-2025.json --output ../docs/NewCatalog.json --save-text raw_text.txt --save-sections sections.json

Workflow

1. Install dependencies: Run pip install -r packages/catalog-extractor/requirements.txt if not already installed.
2. Extract text: Run python packages/catalog-extractor/extract_pdf.py <pdf_url_or_path> --output raw_text.txt to get the raw PDF text.
3. Review raw text: Read the extracted text to understand the PDF structure and identify which program/track to extract.
4. Ask which program/track: Present the user with the available programs found in the PDF and ask which one to extract.
5. Parse and build skeleton: Run the pipeline or manually use parse_sections.py + build_catalog.py to get a skeleton JSON.
6. Refine the JSON: The skeleton will have TODOs. Using the raw text and your understanding of the catalog, fill in:
   • total_credit — from the PDF's total credit summary
   • description — Hebrew description of the program
   • Bank credit values — from each section's credit requirements
   • credit_overflows — based on the track type (see patterns below)
   • Complex rules: Chains for שרשרת מדעית, SpecializationGroups for קבוצות התמחות
   • catalog_replacements — from replacement notes in the PDF
7. Validate schema: Run python packages/catalog-extractor/validate_catalog.py <output.json>
8. Cross-validate against PDF: Run python packages/catalog-extractor/validate_against_pdf.py <output.json> <pdf_url> --verbose to compare courses, credits, banks, chains, and overflows against the source PDF. Fix any ERRORs before proceeding.
9. Output JSON: Save to packages/docs/{Faculty}{Track}{Year}.json

Improving the scripts

If the PDF format has changed or the scripts miss something, update the scripts in packages/catalog-extractor/ rather than writing one-off code. This keeps the tooling reusable for future catalogs.

JSON Schema

The output JSON must follow this exact structure:

{
  "_id": { "$oid": "" },
  "name": "<Hebrew name, e.g., מדמח ארבע שנתי 2024-2025>",
  "total_credit": <number>,
  "description": "<Hebrew description of the program>",
  "course_banks": [ ... ],
  "credit_overflows": [ ... ],
  "course_to_bank": { ... },
  "catalog_replacements": { ... },
  "common_replacements": { ... },
  "faculty": "<FacultyName>"
}

Field Details

_id

Always {"$oid": ""} for new catalogs (MongoDB will assign on insert).

name

Hebrew name of the catalog. Pattern: "מדמח <track-type> <year-range>".

• 3-year: "מדמח תלת שנתי 2024-2025"
• 4-year: "מדמח ארבע שנתי 2024-2025"

total_credit

Total credit points required for degree completion. Found in the PDF under "סה"כ נ.ז" or similar.

course_banks

Array of course bank objects. Each bank represents a category of courses:

{
  "name": "<Hebrew bank name>",
  "rule": "<rule type>",
  "credit": <number or null>
}

Common banks and their rules (not all faculties use all banks — read the PDF):

Bank Name (Hebrew)	Rule	Notes
חובה	"All"	Mandatory courses - all must be completed
שרשרת מדעית	{"Chains": [[...], ...]}	Scientific chain - pick one chain (CS only, not in EE)
מתמטי נוסף	{"AccumulateCourses": {"$numberLong": "1"}}	Additional math - accumulate N courses (CS only)
רשימה א	"AccumulateCredit"	List A electives (CS)
רשימה ב	"AccumulateCredit"	List B electives (CS)
ליבה	{"AccumulateCourses": {"$numberLong": "N"}}	Core electives - pick N courses from list
קבוצות התמחות	{"SpecializationGroups": {...}}	Specialization groups
בחירה פקולטית	"AccumulateCredit"	Faculty electives (EE, CompEng) — often receives overflow from קבוצות התמחות
פרויקט	{"AccumulateCourses": {"$numberLong": "N"}}	Project — see Projects section for credit rules
סמינר	{"AccumulateCourses": {"$numberLong": "1"}}	Seminar, credit is null
בחירת העשרה	"Malag"	Enrichment electives (CS)
חינוך גופני	"Sport"	Physical education (CS)
בחירה חופשית	"Elective"	Free electives (CS)

IMPORTANT: The three university-wide banks (בחירת העשרה, חינוך גופני, בחירה חופשית) are standard across ALL Technion faculties — always use these 3 separate banks. Some PDFs group them under "בחירה כלל-טכניונית" as a combined label, but they must be modeled as 3 separate banks with the Malag/Sport/Elective rules. The typical split is: 6 Malag + 2 Sport + 4 Free = 12 total.

Chains rule structure (for שרשרת מדעית):

{
  "Chains": [
    ["01140075"],
    ["01140052", "01140054"],
    ["01340058", "01340020"],
    ["01240120", "01250801"],
    ["01240120", "01240510"],
    ["01240120", "01140052"]
  ]
}

Each inner array is one valid chain. The student must complete one full chain.

SpecializationGroups rule structure (for קבוצות התמחות, 4-year only):

{
  "SpecializationGroups": {
    "groups_list": [
      {
        "name": "סיבוכיות של חישובים",
        "courses_sum": {"$numberLong": "3"},
        "course_list": ["02360306", "02360309", "02360313"],
        "mandatory": [["02360313"]]
      }
    ],
    "groups_number": {"$numberLong": "3"},
    "groups_type": "Double"
  }
}

All course IDs in specialization groups must also be 8-digit zero-padded.

groups_type (optional; omit to get the default "Regular" — never write "Regular" explicitly):

• "Regular" (default) — a mandatory course shared between groups satisfies the mandatory requirement of every group it appears in; credit/course-count is still attributed to a single group. Use for most tracks.
• "Double" — Regular semantics, and a group carrying a double spec (below) may count as two groups.
• {"MandatoryNotShared": ["<id>", ...]} — a mandatory course satisfies at most one group, so distinct mandatory courses are required (e.g. biotech מגמות). The array is the pool of all mandatory courses.

double (optional per-group; only honored when groups_type is "Double"): add it when a group can be completed as a "double" (PDF: "קבוצה בודדת/כפולה"). A double needs more courses and a stricter mandatory selection, and counts as two groups. "X and two of {list}" is encoded by repeating the list sublist — each mandatory sublist must be satisfied by a distinct course:

{
  "name": "תקשורת ואינפורמציה",
  "courses_sum": {"$numberLong": "3"},
  "course_list": ["00460206", "00460205", "02360309"],
  "mandatory": [["00460206"], ["00460205", "02360309", "00460204"]],
  "double": {
    "courses_sum": {"$numberLong": "6"},
    "mandatory": [["00460206"], ["00460205", "02360309", "00460204"], ["00460205", "02360309", "00460204"]]
  }
}

PDF cues: "קבוצה בודדת תמנה N מקצועות; קבוצה כפולה תמנה M מקצועות" gives single/double courses_sum (M is not always 2N — e.g. 4→7); "המקצועות המחייבים לקבוצה כפולה" gives the double mandatory. If a track states a double does not satisfy the requirement (e.g. הנדסת חשמל ומתמטיקה), keep groups_type "Regular" and omit double.

credit_overflows

Defines how excess credits overflow between banks:

[
  {"from": "חובה", "to": "רשימה ב"},
  {"from": "רשימה א", "to": "רשימה ב"},
  ...
]

Overflow patterns vary by faculty and track. The general principle: excess credits flow from more-specific banks to less-specific ones, ending at the broadest elective bank.

Example overflow patterns (CS faculty):

• CS 3-year: חובה→רשימה ב, רשימה א→רשימה ב, פרויקט→רשימה א, שרשרת מדעית→רשימה ב, מתמטי נוסף→רשימה ב, רשימה ב→בחירה חופשית, בחירת העשרה→בחירה חופשית, חינוך גופני→בחירה חופשית
• CS 4-year: Same as above plus: פרויקט→סמינר, סמינר→רשימה א, קבוצות התמחות→רשימה א
• Computer Engineering: חובה→בחירה פקולטית, ליבה→בחירה פקולטית, קבוצות התמחות→בחירה פקולטית, פרויקט→בחירה פקולטית, בחירה פקולטית→בחירה חופשית, בחירת העשרה→בחירה חופשית, חינוך גופני→בחירה חופשית

Example overflow patterns (EE faculty):

• EE main: חובה→בחירה פקולטית, קבוצות התמחות→בחירה פקולטית, בחירה פקולטית→בחירה חופשית, בחירת העשרה→בחירה חופשית, חינוך גופני→בחירה חופשית
• EE Computer & Software Eng: חובה→בחירה פקולטית, ליבה→בחירה פקולטית, קבוצות התמחות→בחירה פקולטית, בחירה פקולטית→בחירה חופשית, בחירת העשרה→בחירה חופשית, חינוך גופני→בחירה חופשית

Always derive overflows from the PDF's credit breakdown. Credits flow from specific → general.

course_to_bank

Maps every course ID to its bank name:

{
  "02340114": "חובה",
  "02360306": "רשימה א",
  "01140075": "שרשרת מדעית",
  ...
}

CRITICAL: Course ID format — Use 8-digit zero-padded course IDs (e.g., "02340114" not "234114"). This is th

---

Content truncated.