Firecrawl Data Extraction Pipeline

A pipeline for structured data extraction from websites using Firecrawl API

v1.0.0
firecrawl_data_extraction_pipeline.py
Python · 909 lines
183    class Valves(BaseModel):184        FIRECRAWL_API_KEY: str = Field(default="", description="Firecrawl API key")185        DEFAULT_FORMAT: str = Field(default="markdown", description="Default format for content extraction")186        ONLY_MAIN_CONTENT: bool = Field(default=True, description="Extract only main content")187        WAIT_FOR: int = Field(default=5000, description="Wait time in milliseconds")188        ENABLE_WEB_SEARCH: bool = Field(default=False, description="Enable web search for additional context")189        IGNORE_SITEMAP: bool = Field(default=False, description="Ignore sitemap.xml when extracting data")190        INCLUDE_SUBDOMAINS: bool = Field(default=False, description="Include subdomains when extracting data")191        SHOW_SOURCES: bool = Field(default=False, description="Show sources in the extraction results")192        BLOCK_ADS: bool = Field(default=True, description="Block ads during extraction")193        REMOVE_BASE64_IMAGES: bool = Field(default=True, description="Remove base64 encoded images from content")194        MOBILE: bool = Field(default=False, description="Use mobile user agent")

A pipeline for structured data extraction from websites using Firecrawl API

0
0

A pipeline for structured data extraction from websites using Firecrawl API

0
0