twitter

go-twitter

Twitter/X scraping library for Go — multi-account pool, GraphQL API, anti-ban, CAPTCHA solving, TOTP 2FA, and session persistence.

Built on go-stealth for TLS fingerprinting, proxy rotation, and rate limiting.

Features

• Account Pool — round-robin rotation with per-account health tracking, per-endpoint rate limits and per-account pacing
• GraphQL API — profiles, timelines, followers, search, conversations, lists, communities, bookmarks, home timelines; likes, retweets, tweets, replies; DMs; media upload
• Anti-Ban — TLS fingerprinting, header ordering, client hints, x-client-transaction-id (xtid), x-xp-forwarded-for (xpff)
• Auth — multi-step login flow with password, TOTP 2FA, CAPTCHA (Capsolver)
• Error Recovery — CSRF recovery from the server's ct0, session refresh behind a relogin circuit breaker, ban cooldown, proxy-failure backoff
• Typed Errors — *APIError plus Is* predicates
• Session Persistence — pluggable SessionStore; JSON file cache with TTL by default
• Proxy Support — per-account proxy, exponential backoff on proxy failures

Install

go get github.com/anatolykoptev/go-twitter

Requires Go 1.26.3 or newer (see go.mod).

Quick Start

cfg := twitter.ClientConfig{
    Accounts: []*twitter.Account{
        {Username: "user1", Password: "pass1", TOTPSecret: "BASE32SECRET"},
        {Username: "user2", AuthToken: "existing_token", CT0: "existing_ct0"},
    },
    DefaultProxy:  "http://proxy:8080",
    CaptchaSolver: captcha.NewCapsolver(os.Getenv("CAPSOLVER_KEY")),
}

client, err := twitter.NewClient(cfg)

// Read (the pool picks an account)
user, _ := client.GetUserByScreenName(ctx, "elonmusk")
tweets, _ := client.GetUserTweets(ctx, user.ID, 50)
followers, _ := client.GetFollowers(ctx, user.ID, 100)

// Search
latest, _ := client.SearchTweets(ctx, "$BTC", twitter.SearchLatest, 20)
people, _ := client.SearchUsers(ctx, "go developer", 20)

// Write (account-specific)
tweetID, _ := client.PostWithAccount(ctx, "user1", "Hello Twitter!")
acc := client.AccountByUsername("user1")
_ = client.LikeTweet(ctx, acc, tweetID)

An account needs either AuthToken + CT0, a saved session, or Username + Password (plus TOTPSecret if 2FA is on). ParseAccounts("user:pass[:auth_token[:ct0[:totp_secret]]],...") builds the slice from a comma-separated string.

API

All methods below are on *twitter.Client. Every one of them needs an authenticated account; there is no guest path for any exported method. Auth requirements are defined by requiresAuth in request.go.

• Pool — the pool picks the account, rotates on failure, and waits up to 5 minutes for an account whose endpoint limit allows the call.
• Given — you pass the *Account that performs the call (account-pinned; no rotation).

Read

Search

SearchPeople and SearchMedia exist as SearchProduct values; SearchUsers uses SearchPeople internally and SearchTweets accepts only Top and Latest.

Write

DM

Media

Pool and account helpers

The social package is a thin client for an external account-leasing service. SearchWithSocial, SearchTweetsWithSocial, SearchUsersWithSocial, GetUserWithSocial, GetUserTweetsWithSocial and GetTweetConversationWithSocial in the root package lease an account, run the call, retry on other accounts, and report the outcome. The repeated-header-less-88 escalation does not fire on this path: each attempt builds a fresh Account, so the count never reaches the threshold — a leased account that keeps answering 88 is reported to the credential service as auth_error, not banned.

A caller-side cancellation or deadline (context.Canceled, context.DeadlineExceeded, or the caller ctx ending mid-request) is NOT the leased account's fault: it reports nothing to go-social (the vocabulary has no neutral status) and stops leasing immediately. Reports are sent on a detached, bounded context, so a success verdict still lands when the caller's ctx died before the response arrived.

Error Handling

Every request makes up to 3 attempts, rotating accounts on pool-based calls. Recovery per error class:

Observability note: each header-less 88 is still recorded as rate_limited in request.duration (and in MetricsHook the first two of the run are rate-limited calls) — that is the per-response truth. The escalation itself is signalled by the account banned (repeated header-less code 88) log warning, a non-rate-limited MetricsHook outcome on the third call, and the pool's alerts.item_deactivated alert — not by a banned outcome on the instrument.

Errors returned by the client wrap *twitter.APIError (Status, Codes, Message, Err). Test them with errors.As or these predicates, which unwrap for you:

twitter.ErrNotFound marks deterministic misses (deleted tweet, suspended or unavailable user): the account worked, the resource is not there. Test it with errors.Is.

twitter.ErrBadRequest marks deterministic caller-input rejections: a GraphQL 400 whose error entries carry code 214 (top-level or inside extensions). A bad cursor, handle or variable is the caller's own error — it is never retried, never rotated to another account, never charged to account health, and never counted as an endpoint fault. A GraphQL 400 WITHOUT code 214 keeps the endpoint-fault classification: a stale features set answers kind:"Validation"/name:"BadRequestError" too (code 336), so the kind/name markers alone do not separate caller errors from stale request shapes — only the numeric code does.

twitter.ErrEndpoint (HTTP 404, or GraphQL-shaped 400 without code 214) and twitter.ErrOverloaded (HTTP 5xx, LoadShed) are endpoint-side faults: the committed request shape or X itself is at fault, so the serving account is never charged and the call rotates. Test all sentinels with errors.Is.

Relogin circuit breaker

Relogin clears an account's credentials before logging in again, so a login outage can destroy a still-valid session. To prevent that, NewClient installs a per-account breaker by default: after 3 consecutive failed relogins for an account, further relogins for it are refused for 30 minutes, and the account's credentials are left intact. A successful relogin resets the count. A refused relogin surfaces as a relogin blocked error on that attempt.

To opt out and always relogin, install an always-allow gate after NewClient:

client.SetAutoReloginGate(twitter.AutoReloginGateFunc(
    func(context.Context, string) (bool, string) { return true, "" }))

Any type implementing AutoReloginGate (Allowed(ctx, username) (bool, reason)) can replace the breaker; a nil gate also allows everything. A custom gate that does not implement RecordFailure/RecordSuccess receives no outcome feedback.

Rate Limiting

Limits are per account and per endpoint (operation name such as Followers), in 15-minute windows. Each account's limiter is seeded with caps measured from x.com's x-rate-limit-limit header:

Endpoints without a seed use ClientConfig.RateLimit (default 50 per 15 minutes). On every response from a pool request, an account-pinned POST or a media upload that carries a positive x-rate-limit-limit, the account's cap for that endpoint is updated to it, so the limiter follows what x.com currently grants. The pool skips accounts whose limiter denies the endpoint.

x.com's headers are the authority. Every response (200 and non-200) that carries both x-rate-limit-remaining and x-rate-limit-reset updates that account's window for the endpoint. An older response never raises remaining inside one window (windows are ordered by their reset timestamp; within a window remaining only falls). When remaining is at most 2 (rateLimitHeadroom, spare slots for in-flight calls) and the reset is still ahead, the endpoint is held back on that account until the reset: the current call still succeeds, and the next pick rotates to another account, so a 429 that the headers announced never happens. While an endpoint's window is live (before its reset) the local request counter is ignored; the counter decides before the first header is seen (cold start), when responses carry no headers, and again once a window has expired without a renewing response. Writes and media uploads record the window but do not rotate. Observed values are exported as the twitter.client.ratelimit.* metrics (see docs/metrics.md).

The *WithSocial helpers lease a fresh single-account client per attempt, so that state would die with the lease. They therefore pass the operation to go-social as ?endpoint=<operation> on acquire and send the last observed window back with the usage report as rate_limit (endpoint, limit, remaining, reset); see social.Client.AcquireAccountFor and ReportUsageWithRateLimit. A go-social that predates these fields ignores them.

On top of that, each account is paced individually on pool requests: a random delay of AccountPaceMin to AccountPaceMin + AccountPaceRandom (default 0.8s to 2.0s) between consecutive requests by the same account. Pacing is keyed by account, so a busy caller does not delay callers using other accounts. Account-pinned POSTs and media uploads are not paced. Disable pacing with AccountPaceDisabled.

Independently of pacing, MaxInFlightPerAccount (default 2) bounds how many requests may be in flight on one account at once: the pool skips an account at the cap, and when every eligible account is full the call waits for a slot on the same ctx-bounded machinery as endpoint-limit waits (pool.wait metric); the account-pinned paths wait for their own account's slot. A negative value disables the cap. Live in-flight counts are exported as the twitter.client.pool.inflight gauge (see docs/metrics.md). Caveat for *WithSocial callers: each lease builds a fresh single-account client, so the cap cannot span concurrent leases of the same underlying account — cross-lease concurrency control belongs to go-social.

Anti-Detection

• xtid — x-client-transaction-id generated from x.com page animation keys, refreshed every 30 minutes. Initialised lazily on the first request carrying an account's cookies (logged-out x.com serves a shell without the bundle). If initialization fails, the header is omitted, retries back off from 60 s to 15 min, and one xtid.init_failed alert per failure episode is sent to PoolAlertHook.
• xpff — x-xp-forwarded-for generated from the session's guest_id and user agent.
• CT0 — never rotated on a timer or replaced with a client-generated value on an established session; x.com validates it server-side, so doing so causes CSRF 353 errors. The client adopts the ct0 from each successful response's set-cookie, persists it to the session file, and on a 353 adopts the server-offered ct0 before falling back to relogin. A generated ct0 is used only at login, when x.com sets none.
• TLS — browser-grade JA3 fingerprints via go-stealth, with a per-account browser profile (AssignBrowserProfile).
• Headers — fixed header ordering and client hints derived from the account's user agent.
• Jitter — randomized delay before every request.

Configuration

ClientConfig fields (zero values take the defaults shown):

Session storage

Sessions (auth_token + ct0) are loaded at startup, re-saved whenever x.com rotates ct0, and deleted on relogin. By default they live in <SessionDir>/<username>.json (FileSessionStore, atomic writes, 0600). Set ClientConfig.SessionStore to keep them elsewhere, for example in the database of your credential service:

type dbStore struct{ db *sql.DB }

func (s dbStore) Load(ctx context.Context, user string) (twitter.Session, error) {
    var ss twitter.Session
    err := s.db.QueryRowContext(ctx, `SELECT auth_token, ct0, saved_at FROM tw_sessions WHERE username=$1`, user).
        Scan(&ss.AuthToken, &ss.CT0, &ss.SavedAt)
    if errors.Is(err, sql.ErrNoRows) {
        return twitter.Session{}, twitter.ErrSessionNotFound
    }
    return ss, err
}
func (s dbStore) Save(ctx context.Context, user string, ss twitter.Session) error {
    _, err := s.db.ExecContext(ctx, `INSERT INTO tw_sessions (username, auth_token, ct0, saved_at) VALUES ($1,$2,$3,$4)
        ON CONFLICT (username) DO UPDATE SET auth_token=$2, ct0=$3, saved_at=$4`, user, ss.AuthToken, ss.CT0, ss.SavedAt)
    return err
}
func (s dbStore) Delete(ctx context.Context, user string) error {
    _, err := s.db.ExecContext(ctx, `DELETE FROM tw_sessions WHERE username=$1`, user)
    return err
}

c, err := twitter.NewClient(twitter.ClientConfig{Accounts: accs, SessionStore: dbStore{db}})

Contract: Load returns ErrSessionNotFound on a miss. Expiry is the store's job: the client applies no TTL, and SessionTTL only configures the default file store. Every call gets a context bounded to 5s, and a store error is logged (username only) but never fails an API request. Stores must be safe for concurrent use. Only credentials leased by the *WithSocial helpers bypass the store; static-config accounts still load the stored session first. A Load error other than ErrSessionNotFound is not treated as a miss: provided credentials are used as-is, and an account with none fails instead of logging in with its password.

Types

type TwitterUser struct {
    ID, Handle, DisplayName, Bio string
    Followers, Following, TweetCount, ListedCount int
    CreatedAt                      time.Time
    IsVerified, HasAvatar, HasBio  bool
}

type Tweet struct {
    ID, AuthorID, AuthorHandle, AuthorName, Text string
    CreatedAt                                    time.Time
    Views, Likes, Retweets, Quotes, ReplyCount   int
    TokenMentions                                []string // extracted $TICKER mentions
}

type DMConversation struct {
    ConversationID string
    Participants   []string // user IDs
    Messages       []DMMessage
}

type DMMessage struct {
    ID, ConversationID, SenderID, Text string
    CreatedAt                          time.Time
}

GraphQL ID maintenance (gql-sync)

x.com rotates GraphQL queryIDs and feature-flag defaults inside its JS bundle; a stale ID or flag set returns HTTP 400. cmd/gql-sync extracts the current values from x.com's live bundles and regenerates queryids_gen.go and features_gen.go. At runtime the effective queryID for an operation is resolved as environment override (TWITTER_QID_*) over runtime-refreshed over generated over committed literal.

Runtime refresh. A process that holds an authenticated account also re-reads the IDs from x.com's own bundle (the authenticated page it already fetches for x-client-transaction-id), so an X rotation heals without a release. A refresh runs in the background, on the first authenticated request, after 6 hours, and after 3 endpoint faults (HTTP 404 / GraphQL-shaped 400) on one operation within 2 minutes; refreshes are at least 15 minutes apart, and consecutive failed or rejected refreshes back off exponentially (15 min, 30 min, ... capped at 24 h; a success resets it; the fault trigger obeys it too). A later refresh overlays the earlier one: ids it did not find are carried forward, never reverted. The whole extraction is validated (every ID must match ^[A-Za-z0-9_-]{1,64}$, at least 40% of the known operations must be present, a malformed ID rejects it all) and swapped in atomically; on any failure the last good IDs stay and one queryid.refresh_failed pool alert fires per ok-to-failed transition (the hook is process-wide: the last NewClient that sets PoolAlertHook owns it). The request that hit the fault is not retried — the next call uses the new ID. Feature flags are only added to, never changed or removed. Endpoints[op] keeps the compiled value; the layered value is what requests use (EndpointURL). Set TWITTER_QID_REFRESH=off to disable. Metrics: docs/metrics.md. Operations that are not reachable from the public bundles (the authenticated read cluster and the like/retweet mutations) are maintained by hand, with the env override as the hotfix path.

go run ./cmd/gql-sync -proxy "$PROXY_URL" -fail-on-empty
# logged-out x.com serves a shell without the bundle; pass an account's cookies
# via the environment (never as a flag; sent to x.com and twitter.com hosts only, over https, never logged):
GQL_SYNC_COOKIES='auth_token=...; ct0=...' go run ./cmd/gql-sync -fail-on-empty

Flags: -out (repo root, default .), -proxy, -fixtures (read from a directory instead of the network), -date, -fail-on-empty (exit non-zero instead of leaving the files untouched when nothing is extracted).

CI

gql-sync drift check (.github/workflows/gql-sync-drift.yml) — runs weekly (Mon 06:00 UTC, plus manual workflow_dispatch). It runs gql-sync against x.com's live JS bundle and, if the generated files changed, opens a PR for human review (it never auto-merges generated values — a wrong queryID/flag default is an HTTP 400 on every call).

Requires repo secret TWITTER_PROXY: a proxy URL that can reach x.com from GitHub's runner IPs (bare datacenter IPs are blocked). The job runs with -fail-on-empty, so a missing proxy or a bundle break exits non-zero and fails closed instead of overwriting the committed IDs with nothing.

preflight (.github/workflows/preflight.yml) — make preflight (gofmt, vet, build, test) on every PR and push to main.

License

MIT