Haystack
Use SocialScrape from Haystack pipelines and agents with the socialscrape-haystack package, a transcript fetcher, a fetcher for any endpoint and a toolset of eight agent tools.
socialscrape-haystack is the official Haystack integration. It has two components that return Haystack Documents, one for video transcripts and one for any endpoint, and a toolset that gives a Haystack Agent the jobs agents do most (a profile, its posts, one post, the comments, a video transcript, a search) plus a tool that runs any other endpoint by id. It builds on the Python SDK, so retries, timeouts and long transcription jobs are handled for you.
Install
pip install -U socialscrape-haystack
export SOCIALSCRAPE_API_KEY="YOUR_KEY"
Python 3.11 or newer and Haystack 2.14 or newer. Create a key in the dashboard; a new account starts with 100 free credits. The source is on GitHub and the package is on PyPI.
The key is a Haystack Secret. Left out, every component reads it from SOCIALSCRAPE_API_KEY with Secret.from_env_var, and to_dict and from_dict store the variable name, never the key.
Transcripts into a pipeline
SocialScrapeTranscriptsFetcher takes video URLs and returns one Document per URL, with the transcript as the content.
from haystack import Pipeline from haystack.components.preprocessors import DocumentSplitter from socialscrape_haystack import SocialScrapeTranscriptsFetcher pipe = Pipeline() pipe.add_component("fetcher", SocialScrapeTranscriptsFetcher()) pipe.add_component("splitter", DocumentSplitter(split_by="word", split_length=300, split_overlap=30)) pipe.connect("fetcher.documents", "splitter.documents") result = pipe.run({"fetcher": {"urls": [ "https://www.youtube.com/watch?v=dQw4w9WgXcQ", "https://www.tiktok.com/@nasa/video/7665075736742530317", ]}}) chunks = result["splitter"]["documents"]
From there, add an embedder and a DocumentWriter to index the chunks, or a prompt builder and a chat generator to summarize them. Each Document has meta with url (also as source), platform, video_id, language, duration_seconds and transcript_source (captions, or ai when the audio was transcribed), and the chunks keep it.
One bad link does not fail the batch. A URL that cannot be fetched is left out of documents and listed in the errors output as {"url", "error"}. A failed request is free, as described in errors. Pass raise_on_failure=True to raise on the first failure instead.
A video with no captions is transcribed from its audio up to 2 minutes. Longer audio needs allow_long_transcription=True; see AI transcription for what that costs.
Any endpoint as Documents
SocialScrapeFetcher takes an endpoint id and its parameters, from the constructor or at run time, and returns Documents.
from socialscrape_haystack import SocialScrapeFetcher
fetcher = SocialScrapeFetcher(max_pages=3)
comments = fetcher.run(
endpoint="youtube.video.comments",
params={"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"},
)["documents"]
profile = fetcher.run(endpoint="tiktok.profile", params={"handle": "nasa"})["documents"][0]
- A list endpoint gives one Document per item and is followed across pages up to
max_pages. The default is 1, because each page is one request.max_itemsstops early. - Any other endpoint, such as a profile, one post or a transcript, gives one Document.
- The content is the item's text: a post's title and text, a comment's text, a transcript's text, a profile's name and bio.
- The item's other fields, its counts and its author go to
meta, flat, withendpoint,platformand the item'surl, also assource. - A refused request raises. With
raise_on_failure=Falseit is returned in theerrorsoutput instead.
The endpoint ids and their parameters are in the API reference. Parameters are checked against the endpoint before the request is made, so a misspelt name is answered at once and costs nothing.
Give an agent the tools
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from socialscrape_haystack import SocialScrapeToolset
agent = Agent(
chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
tools=SocialScrapeToolset(), # reads SOCIALSCRAPE_API_KEY
)
result = agent.run(messages=[ChatMessage.from_user(
"What did NASA post on YouTube this week, and what are people saying in the comments?"
)])
print(result["last_message"].text)
Any chat generator that supports tool calling works. The toolset has eight tools that share one client:
| Tool | What it does |
|---|---|
socialscrape_get_profile | A profile, channel or artist: name, bio, links, verification and follower counts. |
socialscrape_list_posts | The latest posts or videos of a profile, channel or page, one page per call. |
socialscrape_get_post | One post, video, tweet, pin or clip from its URL. |
socialscrape_list_comments | The comments on a post or video, one page per call. |
socialscrape_get_transcript | The transcript of a video from its URL. |
socialscrape_search | Keyword search on one platform, one page per call. |
socialscrape_list_endpoints | Every endpoint id with its parameters and price. It makes no request, so it is free. |
socialscrape_call_endpoint | Runs any endpoint in the API reference by its id, with its parameters by name. |
The platform is an argument, for example {"platform": "tiktok", "handle": "nasa"}. For a post, its comments or a transcript the platform is read from the link. Tools that list things return a cursor; pass it back for the next page, as described in pagination.
A tool answers with a JSON string that holds data (the object, or a list page with items, cursor and has_more), credits_charged and credits_remaining. A request the API refuses is free and comes back as the tool's answer, with the error code first (error: not_found: ...), so the agent can read it and try something else. A missing API key raises.
The toolset takes these arguments:
| Argument | What it does |
|---|---|
api_key | A Haystack Secret. Left out, it is read from SOCIALSCRAPE_API_KEY. |
max_age_days | Accept a cached result up to this many days old. A cached result costs 0 credits; see caching. |
trim | On by default: asks for the smaller response, without null fields and the extra object, which saves context. |
client, async_client | A SocialScrape or AsyncSocialScrape client from the SDK that you configured yourself. |
base_url | API base URL override. |
Credits
Each request is charged like any API request: see credits. socialscrape_list_endpoints makes no request and is free.
Other agent frameworks
For LangChain and LangGraph, see the LangChain guide. For Claude, ChatGPT, Cursor and other MCP clients, use the MCP server.