twitter-reader

Twitter Reader

Fetch Twitter/X post and article content with full media support.

Reading a single post's text: fxtwitter first (2026-08-30)

For plain post text, prefer the fxtwitter mirror API — login-free, key-free, works direct, and returns the full note_tweet body in tweet.text (the full_text key does not exist; a 2,324-char long-form announcement came back complete):

curl -sS --max-time 20 "https://api.fxtwitter.com/<user>/status/<id>" \
  | python3 -c "import json,sys; t=json.load(sys.stdin)['tweet']; print(t['created_at']); print(t['text'])"

replies is a count, not the reply thread. For X Articles with images, use fetch_article.py below — fxtwitter does not carry article bodies.

X Articles with images: fetch_article.py

uv run --with pyyaml python scripts/fetch_article.py <article_url> [output_dir]

Example:

uv run --with pyyaml python scripts/fetch_article.py \
  https://x.com/HiTw93/status/2040047268221608281 \
  ./Clippings

This will:

• Fetch structured data via twitter-cli (likes, retweets, bookmarks)
• Fetch content with images via jina.ai API
• Download all images to attachments/YYYY-MM-DD-AUTHOR-TITLE/
• Generate complete Markdown with embedded image references
• Include YAML frontmatter with metadata

Metadata and plain article text come from twitter-cli; the image URLs ride in the markdown Jina's reader returns. When Jina refuses (see the section below), the script warns on stderr, keeps the twitter-cli text, and reports Images: 0. The Markdown is still correct; it simply has no pictures. Images: 0 on an article that visibly contains images is the signature of a Jina refusal.

Example Output

Fetching: https://x.com/HiTw93/status/2040047268221608281
--------------------------------------------------
Getting metadata...
Title: 你不知道的大模型训练：原理、路径与新实践
Author: Tw93
Likes: 1648

Getting content and images...
Images: 15

Downloading 15 images...
  ✓ 01-image.jpg
  ✓ 02-image.jpg
  ...

✓ Saved: ./Clippings/2026-04-03-文章标题.md
✓ Images: ./Clippings/attachments/2026-04-03-HiTw93-.../ (15 downloaded)

Jina's reader is intermittent, and its refusals look like success

Anonymous r.jina.ai access to x.com gets blocked for hours at a time when third-party callers abuse the domain. The block hits every anonymous caller and then expires on its own. Both states showed up minutes apart on 2026-09-12.

A refusal does not look like a failure. curl exits 0 and the body is a JSON envelope:

{"data":null,"code":403,"name":"AbuseAlleviationError","status":40305,
 "message":"Anonymous access to domain x.com blocked until <date> ..."}

A lapsed or unfunded key returns the same shape with "code":402, "name":"InsufficientBalanceError". curl --fail catches neither: this endpoint answered HTTP 200 with the envelope in the body. The one signal that holds is the Markdown Content: marker that Jina's reader puts in front of the article body: no marker, no article. fetch_article.py tests for that marker before accepting a response, which keeps a refusal envelope out of the generated Markdown.

scripts/fetch_tweets.sh and scripts/fetch_tweet.py both require JINA_API_KEY and have no second source to fall back to, so they stop working whenever that key lapses. Both check the same marker and fail loudly on a refusal — the Python one raises without writing the output file, the shell one reports the URL on stderr and exits non-zero — rather than handing back an envelope dressed up as a post. No key ships with this repository.

For simple text-only fetching:

# Single tweet
curl "https://r.jina.ai/https://x.com/USER/status/TWEET_ID" \
  -H "Authorization: Bearer ${JINA_API_KEY}"

# Batch fetching
scripts/fetch_tweets.sh url1 url2 url3

Features

Full Article Mode (fetch_article.py)

• ✅ Structured metadata (author, date, engagement metrics)
• ✅ Automatic image download (all embedded media)
• ✅ Complete Markdown with local image references
• ✅ YAML frontmatter for PKM systems
• ✅ Handles X Articles (long-form content)

Simple Mode (Jina API)

• Text-only content
• Intermittent availability (see the section above); both scripts require JINA_API_KEY and have no fallback
• Usable for quick text extraction while Jina is answering

Prerequisites

For Full Article Mode

• uv (Python package manager)
• No additional setup (twitter-cli auto-installed)

For Simple Mode (Jina)

export JINA_API_KEY="your_api_key_here"
# Get from https://jina.ai/

Output Structure

output_dir/
├── YYYY-MM-DD-article-title.md       # Main Markdown file
└── attachments/
    └── YYYY-MM-DD-author-title/
        ├── 01-image.jpg
        ├── 02-image.jpg
        └── ...

What Gets Returned

Full Article Mode

• YAML Frontmatter: source, author, date, likes, retweets, bookmarks
• Markdown Content: Full article text with local image references
• Attachments: All downloaded images in dedicated folder

Simple Mode

• Title: Post author and content preview
• URL Source: Original tweet link
• Published Time: GMT timestamp
• Markdown Content: Text with remote media URLs

URL Formats Supported

• https://x.com/USER/status/ID (posts)
• https://x.com/USER/article/ID (long-form articles)
• https://twitter.com/USER/status/ID (legacy)

Scripts

fetch_article.py

Full-featured article fetcher with image download:

uv run --with pyyaml python scripts/fetch_article.py <url> [output_dir]

fetch_tweet.py

Simple text-only fetcher using Jina API:

python scripts/fetch_tweet.py <tweet_url> [output_file]

fetch_tweets.sh

Batch fetch multiple tweets (Jina API):

scripts/fetch_tweets.sh <url1> <url2> ...

twitter-cli's 500-item ceiling belongs to the tool, not to X

This skill reads metadata through twitter-cli, so the tool's own limits apply. twitter bookmarks and the other timeline commands stop at 500 items however the config is written. That ceiling is a module constant in the client: _ABSOLUTE_MAX_COUNT = 500 in twitter_cli/client.py, applied as min(maxCount, 500) when the client is constructed. Both lines read from version 0.8.5. X's own bookmark timeline goes far deeper than that.

Two consequences are easy to get wrong:

• A run that stops at exactly 500 stopped on its own counter and never reached the point of reading the next cursor. "The platform gave no further page" and "my loop finished" are different claims, and only the first one is evidence. A round number is a reason to look closer, not a boundary.
• maxCount has to sit under the rateLimit: section of the config. Put it under fetch: and the client ignores it without any message, falling back to 200.

Going deeper means calling the GraphQL endpoint directly: the Bookmarks operation with the client's own feature flags, parsed by twitter_cli.parser.parse_timeline_response, paging until the platform stops returning a cursor or returns the same cursor twice. Stop on the platform's signal rather than on a count.

Migration from Jina API

Old workflow:

curl "https://r.jina.ai/https://x.com/..."
# Manual image extraction and download

New workflow:

uv run --with pyyaml python scripts/fetch_article.py <url>
# Automatic image download, complete Markdown