data-layer

Data Layer

Architecture

src/data-layer/
├── fetchers/         # Fetch functions (one per data source)
│   └── developer-tools/  # Multi-file fetcher (builder resources, GitHub/npm stats, ranking)
├── index.ts          # Public API - typed getter functions
├── tasks.ts          # KEYS constant + Trigger.dev scheduled tasks
├── storage.ts        # get/set abstraction (Netlify Blobs or mock files)
├── s3.ts             # S3 image upload utility for external images
├── docs.md           # Module documentation
├── mocks/            # Mock data files for local development
└── .env.example      # Environment variables for data-layer/Trigger.dev

src/lib/data/
└── index.ts          # Next.js caching adapter (createCachedGetter)

Environment Variables

The data-layer uses a dedicated .env.local file at src/data-layer/.env.local, separate from the main app's root .env.local.

Local Development Setup

1. Copy the example file:

   cp src/data-layer/.env.example src/data-layer/.env.local
   

2. Fill in the required API keys (see .env.example for all options)

3. Run Trigger.dev tasks locally:

   pnpm trigger:dev
   

Variable Categories

• Shared with Main App: GITHUB_TOKEN_READ_ONLY, Sentry vars (configure in both files)
• Data Layer Only: API keys (CoinGecko, Beaconcha.in, Dune, Google, etc.), Netlify Blobs tokens, S3 credentials, Trigger.dev config

Production (Trigger.dev Cloud)

Configure environment variables in your Trigger.dev project dashboard. The main app and data-layer run in separate environments.

Key Files

tasks.ts - Single Source of Truth

Defines all task keys and scheduled jobs:

export const KEYS = {
  ETH_PRICE: "fetch-eth-price",
  L2BEAT: "fetch-l2beat",
  // ...
} as const

const WEEKLY: TaskDef[] = [[KEYS.GITHUB_CONTRIBUTORS, fetchGitHubContributors]]

const DAILY: TaskDef[] = [
  [KEYS.APPS, fetchApps],
  [KEYS.EVENTS, fetchEvents],
]

const HOURLY: TaskDef[] = [
  [KEYS.ETH_PRICE, fetchEthPrice],
  [KEYS.TOTAL_ETH_STAKED, fetchTotalEthStaked],
]

index.ts - Simple Getters

One-liner passthrough functions:

export const getEthPrice = () => get<EthPriceData>(KEYS.ETH_PRICE)
export const getL2beatData = () => get<L2beatData>(KEYS.L2BEAT)

storage.ts - Storage Abstraction

Simple get/set that switches between Netlify Blobs (prod) and local JSON files (dev):

export async function get<T>(key: string): Promise<T | null>
export async function set(key: string, data: unknown): Promise<void>

Uses USE_MOCK_DATA=true env var for local development.

s3.ts - Image Upload Utility

Centralized S3 upload for external images. Fetchers use this to upload external images to a single S3 bucket, reducing Next.js remotePatterns complexity.

// Upload single image
const s3Url = await uploadToS3(sourceUrl, "events/logos")

// Batch upload (parallel)
const s3Urls = await uploadManyToS3(urls, "apps/banners")

Key features:

• SSRF protection - Blocks private/internal network addresses
• Deduplication - SHA256 hash of source URL as key
• Existence check - Skips if already uploaded
• 5MB size limit - Returns null for large images
• Content-Type detection - From header or URL extension fallback

Rules

1. Getters must be pure passthrough

No transformations in index.ts - just get<T>(KEYS.X):

// Correct
export const getEventsData = () => get<EventItem[]>(KEYS.EVENTS)

// Wrong - no transformations in getters
export const getEventsData = () => {
  const data = await get<EventItem[]>(KEYS.EVENTS)
  return data?.map(transform) ?? null
}

All transformations belong in the fetcher (src/data-layer/fetchers/).

2. KEYS is the single source of truth

All task IDs are defined in KEYS in tasks.ts. The getter in index.ts and the task tuple in WEEKLY/DAILY/HOURLY must use the same key.

3. Expose via lib/data for caching

Add cached wrapper in src/lib/data/index.ts:

export const getEventsData = createCachedGetter(
  dataLayer.getEventsData,
  ["events-data"],
  CACHE_REVALIDATE_DAY // or CACHE_REVALIDATE_HOUR
)

The revalidate parameter is number | false. Passing false is a deliberate pattern to keep a route fully static — a finite revalidate opts the page into ISR, which fails on Netlify for pages reading public/content/ files. Example: getStaticAppsData in src/lib/data/index.ts, used by components embedded in MDX pages (data refreshes only on deploy).

4. Use S3 for external images

External images should be uploaded to S3 in the fetcher to centralize image domains:

// In fetcher - correct
import { uploadToS3 } from "../s3"

const logoUrl = await uploadToS3(event.logoImage, "events/logos")
return { ...event, logoImage: logoUrl ?? "" }

Always handle null returns (upload failures) with fallback/empty string.

5. Keep fetchers isolated from the app

Fetchers run on Trigger.dev — a separate runtime, deployment, and bundle from the Next.js app. They cannot assume the app's filesystem, environment, or modules are available.

Any import or runtime dependency reaching outside src/data-layer/ is a warning sign. Allowed: types (@/lib/types, @/lib/interfaces), pure constants (@/lib/constants), and pure utility functions with no app-runtime dependencies. Not allowed: anything that reads process.cwd(), anything from app/ or public/, anything from src/components/, or src/lib/data/ (which wraps the data layer and would create a cycle).

If a fetcher needs data that lives in the app — content files, frontmatter, etc. — fetch it over the network via the GitHub API and treat the repo as an external system. See fetchGitHubContributors.ts for the pattern. Don't work around this with additionalFiles in trigger.config.ts; bundling app files into the data-layer deployment re-creates the coupling.

Adding a New Data Source

1. Create fetcher in src/data-layer/fetchers/fetchNewData.ts:

   export async function fetchNewData(): Promise<YourDataType> {
     // Fetch and transform data here
   }
   

2. Add key to KEYS in src/data-layer/tasks.ts:

   export const KEYS = {
     // ...existing keys
     NEW_DATA: "fetch-new-data",
   } as const
   

3. Add task tuple to WEEKLY, DAILY, or HOURLY in tasks.ts:

   const DAILY: TaskDef[] = [
     // ...existing tasks
     [KEYS.NEW_DATA, fetchNewData],
   ]
   

4. Add getter in src/data-layer/index.ts:

   export const getNewData = () => get<YourDataType>(KEYS.NEW_DATA)
   

5. Add mock file at src/data-layer/mocks/fetch-new-data.json for local development

6. Add cached wrapper in src/lib/data/index.ts:

   export const getNewData = createCachedGetter(
     dataLayer.getNewData,
     ["new-data"],
     CACHE_REVALIDATE_HOUR
   )