By Sorsa Editorial

Published June 13, 2026. Updated July 3, 2026: pricing reworked to per-1,000-tweet rates, the 100 free requests start added, and the X API cost comparison refreshed. Official X API pay-per-use rates verified July 2026.

Key Takeaway: Building a Twitter (X) dataset for machine learning means collecting tweets from an API, cleaning and deduplicating them, labeling for the task, splitting into train, validation, and test sets, and exporting to a format like JSONL. Since the free API and open scrapers stopped working, fresh collection now runs through a paid data API.

That last point is the practical hurdle. With the free API tier gone since 2023 and scrapers like snscrape and twint no longer working, a fresh, on-topic corpus has to come from a data API. Sorsa API, a read-only Twitter/X alternative, returns public tweets and profiles as plain JSON behind a single header, with search, timeline, and bulk endpoints built for collection. Billing is flat: one call counts as one request whatever it returns, the author profile rides along free, and one bulk call rehydrates up to 100 tweet IDs, which is how the ID-only datasets researchers publish get revived. On batch endpoints Sorsa works out to about $0.02 per 1,000 tweets on the Pro plan (from $0.049 on Starter), and 100 free requests, no card and no expiry, cover the full 40-endpoint surface so a pipeline can be tested before any spend.

Contents

How do you build a Twitter dataset for machine learning?

Building a Twitter dataset runs through five stages: collect raw tweets from an API, clean and deduplicate them, label them for the task, split them into train, validation, and test sets, and export to a training format such as JSONL. The same pipeline serves both a classification dataset and an instruction-tuning set for a language model; only the labeling step changes.

Start from the objective, not the data. A sentiment classifier wants short tweets with a clear positive, negative, or neutral label. An instruction-tuned model wants prompt and response pairs. A domain-adaptation run wants a large pool of in-topic text with light labels or none. Knowing the target decides the query, the volume, and the schema before a single tweet is pulled.

The work splits cleanly. Collection and rehydration are an API problem, covered below with code. Labeling, normalization, and splitting are a data problem you handle locally in Python. The rest of this guide takes each in turn, with the 2026 constraints that break most older tutorials called out where they bite.

Should you build your own dataset or use a public one?

Use a public dataset when one already matches your task; build your own when topic, language, time window, or labels are specific to you. Public sets on Hugging Face and Kaggle are free and fast to load, but they are fixed in scope, often years old, frequently English-only, and many ship as tweet IDs that you must rehydrate before the text exists.

The well-known options are worth knowing first. Sentiment140 holds about 1.6 million tweets labeled for sentiment, and the TweetEval benchmark from Cardiff NLP bundles seven tweet classification tasks (irony, hate, offensive, stance, emoji, emotion, sentiment). For a generic English sentiment or emotion model, loading one of these beats collecting anything.

Build your own when the gap is real:

  • The topic is narrow or recent and no public set covers it.
  • You need a language or region the public sets skip.
  • You need a fresh time window, not tweets from 2017.
  • Your label scheme does not match any existing dataset.

For the sentiment case specifically, the guide to classifying tweet sentiment in Python walks the modeling side once the data exists.

How to collect tweets for a dataset in 2026

Collect tweets through three routes: keyword and operator search for topic corpora, user timelines for per-account corpora, and Lists or Communities for curated sources. The free API tier closed in 2023 and scrapers such as snscrape and twint stopped working, so fresh collection at any volume now goes through a data API that returns search and timeline results directly.

A read-only API turns this into a short loop: query an endpoint, read a page of tweets, follow the cursor, write each tweet to a file. The Sorsa search endpoint accepts standard Twitter advanced search operators like lang:, since:, until:, and -filter:retweets, returns about 20 tweets per page with the author profile attached, and paginates with next_cursor. If the syntax is unfamiliar, build the query first in the visual search query builder.

python
import json, time, requests

API_KEY = "YOUR_SORSA_API_KEY"
BASE = "https://api.sorsa.io/v3"
HEADERS = {"ApiKey": API_KEY, "Content-Type": "application/json"}

def collect(query, target=5000):
    rows, cursor = [], None
    while len(rows) < target:
        body = {"query": query, "order": "latest"}
        if cursor:
            body["next_cursor"] = cursor
        r = requests.post(f"{BASE}/search-tweets", headers=HEADERS, json=body, timeout=30)
        if r.status_code == 429:
            time.sleep(1)
            continue
        r.raise_for_status()
        data = r.json()
        for t in data.get("tweets", []):
            rows.append({
                "id": str(t["id"]),
                "text": t.get("full_text", ""),
                "lang": t.get("lang", ""),
                "created_at": t.get("created_at", ""),
                "username": (t.get("user") or {}).get("username", ""),
            })
        cursor = data.get("next_cursor")
        if not cursor:
            break
    return rows

with open("tweets.jsonl", "w", encoding="utf-8") as f:
    for row in collect('"your topic" lang:en -filter:retweets', target=5000):
        f.write(json.dumps(row, ensure_ascii=False) + "\n")

For per-account corpora, swap the search call for the user timeline endpoint with a username; for curated sources, pull from a curated List or Community feed. To sample for balance, the mentions endpoint adds filters such as min_likes, since_date, and until_date, which let you pull, for example, only well-engaged tweets inside a date range.

Two efficiency notes. Because the author profile rides inside every tweet, you get the handle, follower count, and account age in the same response with no extra call. And when the data fits a batch shape, batch endpoints cut the request count: a single call takes up to 100 tweet IDs or 100 usernames, the main lever for holding request volume down on a large pull.

If you would rather avoid an API, the older scraper route still exists, but expect breakage; the breakdown of why open-source scrapers stopped working explains the trade.

How do you rehydrate a tweet-ID-only dataset?

Rehydration means taking a list of tweet IDs and fetching the current content for each one. Public research datasets are distributed as IDs rather than full tweets because the platform's terms restrict redistributing tweet text, so the text only exists after you look the IDs up against an API. Expect to recover a subset: tweets that were deleted, or posted by suspended or now-private accounts, no longer return.

A bulk endpoint makes this cheap. The Sorsa bulk tweet endpoint accepts up to 100 IDs per call and returns the full tweet objects, so a 50,000 ID file is roughly 500 requests rather than 50,000 single lookups.

python
import requests

API_KEY = "YOUR_SORSA_API_KEY"
URL = "https://api.sorsa.io/v3/tweet-info-bulk"
HEADERS = {"ApiKey": API_KEY, "Content-Type": "application/json"}

def chunks(seq, n=100):
    for i in range(0, len(seq), n):
        yield seq[i:i + n]

ids = [line.strip() for line in open("tweet_ids.txt") if line.strip()]
rehydrated = []
for batch in chunks(ids, 100):
    r = requests.post(URL, headers=HEADERS, json={"tweet_links": batch}, timeout=30)
    r.raise_for_status()
    rehydrated.extend(r.json().get("tweets", []))

print(f"Recovered {len(rehydrated)} of {len(ids)} tweets")

The recovery rate drops the older the ID list is, since more of the original tweets disappear every year. For where ID lists come from and how much of an old event dataset survives, see the guide to recovering older tweets from an ID list.

How to structure and label a dataset for training

Give each record a stable schema and export to JSONL, one object per line. A classification set needs the tweet ID, the text, the label, the language, and the timestamp; an instruction-tuning set needs an instruction, an optional input, and the target output. JSONL is the format the Hugging Face datasets library and most fine-tuning pipelines read directly.

A classification record and an instruction record look like this:

json
{"id": "1782368585664626774", "text": "the new build keeps crashing on launch", "label": "negative", "lang": "en", "created_at": "2026-05-01T10:30:00Z"}
json
{"instruction": "Classify the sentiment of this tweet.", "input": "the new build keeps crashing on launch", "output": "negative"}

Clean before you label. Deduplicate on the tweet ID, drop very short or empty tweets, and filter by the lang field so the set stays in your target language. Following the TweetEval convention from Cardiff NLP, replace URLs with a {{URL}} token and non-verified usernames with {{USERNAME}}, so the model learns language patterns rather than specific links or handles.

For labels, three approaches scale. Hand-label a small gold set for evaluation. Pre-label the bulk with a small existing model, then have a human correct it, which is far faster than labeling from scratch. Or, for instruction tuning, write prompt and response pairs around the tweets. LLM fine-tuning guides consistently stress quality over raw volume: a few thousand clean, well-formed pairs usually beat a noisy set ten times the size.

Finally, split before training so evaluation stays honest. A common split is 80 percent train, 10 percent validation, 10 percent test, shuffled with a fixed seed for reproducibility.

python
import json, random

rows = [json.loads(line) for line in open("tweets.jsonl", encoding="utf-8")]
random.seed(42)
random.shuffle(rows)
n = len(rows)
train = rows[: int(0.8 * n)]
val = rows[int(0.8 * n): int(0.9 * n)]
test = rows[int(0.9 * n):]

What about platform terms and ethics?

Collecting public tweets for your own model is one thing; redistributing the dataset is where the rules bite. X's Developer Policy states that if you share content with third parties, including downloadable datasets, you may distribute only post IDs and user IDs, not the full text, which is exactly why public research datasets ship as IDs.

The current terms set concrete limits. Per X's Developer Policy, you may not distribute more than 1,500,000 post IDs to any one entity within a 30 day period without written permission, while individuals acting on behalf of an academic institution for non-commercial research may share an unlimited number of IDs. Researchers using EU Digital Services Act access fall under separate provisions.

The practical pattern that keeps you inside the lines: keep the full text you collect for your own training, and if you publish a dataset, publish the tweet IDs and a rehydration script so others recreate the text themselves. The terms also prohibit tracking or monitoring sensitive groups and events, so steer collection away from those uses. This is general guidance, not legal advice, and the terms change, so check the current X Developer Agreement before you release anything.

How much does collecting a dataset cost?

Cost depends on volume and on whether the source bills per resource or per request. Per-resource pricing charges for every post and every profile a response returns, so a call that pulls a tweet plus its author is billed twice; per-request pricing counts one call as one unit no matter how much it returns. On a read-heavy dataset pull, that single difference decides the bill.

The official X API charges per resource: each post read is $0.005 and each user profile read is $0.010, capped at 2 million post reads per month, rates verified July 2026 in the Twitter API pricing breakdown. Sorsa bills one request per call with the author profile included free, so on batch endpoints a dataset pull runs from about $0.02 per 1,000 tweets on the Pro plan (from $0.049 on Starter, from $0.018 on Enterprise), and from about $0.01 per 1,000 profiles.

TaskOfficial X API (pay-per-use)Sorsa (Pro, batch base)
1,000 tweets, author profile included$15.00 (1,000 post reads + 1,000 user reads)from $0.02
Rehydrate 1,000 tweet IDs$5.00 (1,000 post reads)from $0.02
Collect 100,000 tweets end to endroughly $500 to $1,500within one $199 Pro month

Sorsa figures use the batch base: one request returns up to 100 tweets or 200 profiles. On any read-heavy pull the flat per-request bill lands far below per-resource pricing, and the author profile costs nothing on the Sorsa side.

Which route fits depends on the workflow, not on a single winner:

  • Writing or posting: the official X API, the compliant path for any write action.
  • Very low read volume, under about 10,000 posts a month: either works, and X pay-per-use stays cheap at that size.
  • Building a dataset, which is read-heavy collection plus rehydration: a flat per-request API like Sorsa, up to 50x cheaper on the same tweets with author profiles included free.

Researchers can also check the discounted academic research access, and the same collection code ports to any stack.

A real dataset build

A small AI team building a finance-sentiment model came to this after pricing the official route. They needed roughly 200,000 recent English tweets about a basket of tickers, labeled positive, negative, or neutral. On per-resource billing the post reads alone came to about $1,000, and author profiles would have doubled or tripled that before any refresh, with the 2 million monthly read cap looming once they planned updates. Moving collection to a flat per-request API pulled the same 200,000 tweets inside roughly $20 of a Pro month's quota, up to 50x cheaper on post reads alone (about 98 percent less), and more once the free author profiles are counted, because search returned 20 tweets per request.

The build stayed small. A keyword query per ticker with a language filter fed a JSONL file; a small classifier pre-labeled the rows and an analyst corrected a sample; URLs and handles were normalized to tokens; and the set was split 80, 10, 10 with a fixed seed. When they later wanted to extend an older public event dataset, they rehydrated its ID list in batches of 100 and accepted that a slice of the original tweets had since been deleted. The lesson they kept was the ordering: define the label scheme and the query first, because re-collecting after a schema change is the expensive mistake, not the API bill.

FAQ

Can you still collect tweets for a dataset in 2026?

Yes, but not through the old tools. The free Twitter API tier closed in 2023, and scrapers such as snscrape and twint no longer work against the current platform. Fresh collection at any real volume now runs through a paid data API that returns search and timeline results, which you page through and write to a file. Public datasets remain an option when one already fits the task.

Where can you get a ready-made Twitter dataset?

Hugging Face and Kaggle host the main public tweet datasets. Sentiment140 holds about 1.6 million sentiment-labeled tweets, and the TweetEval benchmark covers seven classification tasks such as irony, hate, and emotion. Many academic datasets ship as tweet IDs rather than full text, so you rehydrate the IDs against an API before the tweet content exists in your copy.

How do you rehydrate tweet IDs?

Rehydration looks up the current content for a list of tweet IDs. A bulk endpoint makes it cheap: a read-only API such as Sorsa accepts up to 100 IDs per call and returns the full tweet objects, so a 50,000 ID file becomes about 500 requests. Expect to recover a subset, because tweets that were deleted or posted by suspended or private accounts no longer return.

How many tweets do you need to fine-tune a model?

It depends on the task, and quality matters more than volume. For a narrow classification or instruction-following task, a few thousand clean, correctly labeled examples often move the model meaningfully, while a noisy set ten times larger can do less. Define the objective, label a small high-quality set first, and grow it only if evaluation shows the model needs more coverage.

Collecting public tweets for your own model is generally permitted, but redistribution is restricted. X's Developer Policy allows sharing only post IDs and user IDs in published datasets, not full tweet text, with a cap of 1,500,000 IDs per entity per 30 days and an exception for academic institutions. Publish IDs and a rehydration script rather than raw text, and check the current terms before release.

Do you need the official X API to build a dataset?

No. The official X API is required only for write actions such as posting. For collecting public tweets and profiles, a read-only alternative such as the Sorsa API uses a single ApiKey header and bills per request, not per tweet, with the author profile included free. It runs from about $0.02 per 1,000 tweets on Pro batch pricing, and 100 free requests, no card, cover a real test run.

Getting started

The quickest way to size a dataset is to run the query before writing a pipeline. Open the browser-based API playground, run a search with your operators, and read the JSON fields you will map into JSONL. When the query looks right, drop the collection script above into your project and point it at a file.

Sorsa starts with 100 free requests: one-time, no card, no expiry, and valid across all 40 endpoints, which is enough to rehydrate up to 10,000 tweets or pull up to 20,000 profiles while you test. When you are ready to scale, the flat per-request pricing plans run from about $0.02 per 1,000 tweets on batch endpoints, with a flat 20 requests per second and no approval queue.

Reviewed by Keksich, founder of Sorsa, marketer and X API researcher.

This guide draws on our team's hands-on work running the live Sorsa API and its documentation, plus public datasets and platform terms checked at the time of writing. The content-redistribution limits were read from X's Developer Policy, the public-dataset and normalization conventions from the TweetEval dataset on Hugging Face, and the official X API per-resource rates were cross-checked against current public breakdowns. More on our team is on the about page. Verified July 2026.