data-layer

Data Layer

Architecture

src/data-layer/
├── fetchers/         # Fetch functions (one per data source)
│   └── developer-tools/  # Multi-file fetcher (builder resources, GitHub/npm stats, ranking)
├── index.ts          # Public API - typed getter functions
├── tasks.ts          # KEYS constant + Trigger.dev scheduled tasks
├── storage.ts        # get/set abstraction (Netlify Blobs or mock files)
├── s3.ts             # S3 image upload utility for external images
├── docs.md           # Module documentation
├── mocks/            # Mock data files for local development
└── .env.example      # Environment variables for data-layer/Trigger.dev

src/lib/data/
└── index.ts          # Next.js caching adapter (createCachedGetter)

Environment Variables

The data-layer uses a dedicated .env.local at src/data-layer/.env.local, separate from the root .env.local: cp src/data-layer/.env.example src/data-layer/.env.local, fill in the keys (see .env.example for all options), then pnpm trigger:dev runs tasks locally. GITHUB_TOKEN_READ_ONLY and Sentry vars are shared with the main app (configure in both files); everything else (API keys, Netlify Blobs tokens, S3 credentials, Trigger.dev config) is data-layer only. In production, configure vars in the Trigger.dev project dashboard — the app and data-layer run in separate environments.

Key Files

tasks.ts defines the KEYS constant and the WEEKLY/DAILY/HOURLY task tuples; index.ts holds the one-liner getters — snippets for both under "Adding a New Data Source" below.

storage.ts - Storage Abstraction

get<T>(key) / set(key, data) switch between Netlify Blobs (prod) and local mock JSON files (USE_MOCK_DATA=true for local development).

s3.ts - Image Upload Utility

Centralized S3 upload for external images. Fetchers use this to upload external images to a single S3 bucket, reducing Next.js remotePatterns complexity.

// Upload single image
const s3Url = await uploadToS3(sourceUrl, "events/logos")

// Batch upload (parallel)
const s3Urls = await uploadManyToS3(urls, "apps/banners")

Key features:

• SSRF protection - Blocks private/internal network addresses
• Deduplication - SHA256 hash of source URL as key
• Existence check - Skips if already uploaded
• 5MB size limit - Returns null for large images
• Content-Type detection - From header or URL extension fallback

Rules

1. Getters must be pure passthrough

No transformations in index.ts - just get<T>(KEYS.X):

// Correct
export const getEventsData = () => get<EventItem[]>(KEYS.EVENTS)

// Wrong - no transformations in getters
export const getEventsData = () => {
  const data = await get<EventItem[]>(KEYS.EVENTS)
  return data?.map(transform) ?? null
}

All transformations belong in the fetcher (src/data-layer/fetchers/).

2. KEYS is the single source of truth

All task IDs are defined in KEYS in tasks.ts. The getter in index.ts and the task tuple in WEEKLY/DAILY/HOURLY must use the same key.

3. Expose via lib/data for caching

Add cached wrapper in src/lib/data/index.ts:

export const getEventsData = createCachedGetter(
  dataLayer.getEventsData,
  ["events-data"],
  CACHE_REVALIDATE_DAY // or CACHE_REVALIDATE_HOUR
)

The revalidate parameter is number | false. Passing false is a deliberate pattern to keep a route fully static — a finite revalidate opts the page into ISR, which fails on Netlify for pages reading public/content/ files. Example: getStaticAppsData in src/lib/data/index.ts, used by components embedded in MDX pages (data refreshes only on deploy).

4. Use S3 for external images

External images should be uploaded to S3 in the fetcher to centralize image domains:

// In fetcher - correct
import { uploadToS3 } from "../s3"

const logoUrl = await uploadToS3(event.logoImage, "events/logos")
return { ...event, logoImage: logoUrl ?? "" }

Always handle null returns (upload failures) with fallback/empty string.

5. Keep fetchers isolated from the app

Fetchers run on Trigger.dev — a separate runtime, deployment, and bundle from the Next.js app. They cannot assume the app's filesystem, environment, or modules are available.

Any import or runtime dependency reaching outside src/data-layer/ is a warning sign. Allowed: types (@/lib/types, @/lib/interfaces), pure constants (@/lib/constants), and pure utility functions with no app-runtime dependencies. Not allowed: anything that reads process.cwd(), anything from app/ or public/, anything from src/components/, or src/lib/data/ (which wraps the data layer and would create a cycle).

If a fetcher needs data that lives in the app — content files, frontmatter, etc. — fetch it over the network via the GitHub API and treat the repo as an external system. See fetchGitHubContributors.ts for the pattern. Don't work around this with additionalFiles in trigger.config.ts; bundling app files into the data-layer deployment re-creates the coupling.

Adding a New Data Source

1. Create fetcher in src/data-layer/fetchers/fetchNewData.ts:

   export async function fetchNewData(): Promise<YourDataType> {
     // Fetch and transform data here
   }
   

2. Add key to KEYS in src/data-layer/tasks.ts:

   export const KEYS = {
     // ...existing keys
     NEW_DATA: "fetch-new-data",
   } as const
   

3. Add task tuple to WEEKLY, DAILY, or HOURLY in tasks.ts (getter and tuple must use the same KEYS entry — Rule 2):

   const DAILY: TaskDef[] = [
     // ...existing tasks
     [KEYS.NEW_DATA, fetchNewData],
   ]
   

4. Add getter in src/data-layer/index.ts:

   export const getNewData = () => get<YourDataType>(KEYS.NEW_DATA)
   

5. Add mock file at src/data-layer/mocks/fetch-new-data.json (read when USE_MOCK_DATA=true)

6. Add cached wrapper in src/lib/data/index.ts:

   export const getNewData = createCachedGetter(
     dataLayer.getNewData,
     ["new-data"],
     CACHE_REVALIDATE_HOUR
   )