Matasoft Web Search Next-generation web search tool for Open WebUI – combines SearXNG results with static/dynamic page reading (BeautifulSoup + Selenium).

🚀 What is this tool? Matasoft Web Search enriches SearXNG-based search with true content extraction. For every search result, it tries:

Requests + BeautifulSoup for fast sites

Selenium + Chrome where JavaScript or SPA content needs rendering

JS need is detected automatically (4-layer heuristics: block patterns, SPA frameworks, text volume, structure). This means you get real text instead of empty shells or JavaScript templates.

🛠️ Features 🟢 Hybrid search: find and extract real content, not just URLs

🟢 Supports static & JS sites (automatically chooses the best fetch method)

🟢 Automatic fallback from requests to Selenium

🟢 Extracts up to 50,000 words per page

🟢 Parallel processing (up to 5 concurrent workers)

🟢 Strategy metadata (know what method was used per result)

🟢 Zero-cookie, Docker-native

🟢 Production-grade error handling

🧩 Requirements (Docker container / host) These system and Python dependencies must be installed (identical to Web Reader):

Requirement Minimum Version Install Command Python 3.10+ (comes with Open WebUI) BeautifulSoup4 4.9+ pip install beautifulsoup4 lxml html5lib Selenium 4.38+ pip install selenium Google Chrome latest stable apt-get install -y google-chrome-stable ChromeDriver (match Chrome) [see notes or use setup script] Requests 2.25+ pip install requests Pydantic 1.x+ pip install pydantic Also requires a running SearXNG instance for search queries (by default on http://host.docker.internal:8080/search).

The fastest install: use included script install_matasoft_web_search.sh to automate everything.

⚙️ Usage Examples Search and extract:

text Search for "python web scraping frameworks" Deep content extraction from result links: (No extra prompt needed – tool auto-fetches best possible content)

Read single URL:

text Read https://dynamic-site.com and summarize Use cases: Tech research, tracking news, fact-checking, or extracting content from JS-heavy pages.

💡 Notes / Troubleshooting ChromeDriver must match Chrome version (see guide in script or in this README).

If you update Open WebUI (e.g. with Watchtower), rerun install to restore dependencies!

SearXNG must be running for search_web to work (see documentation for docker setup).

🏷️ License MIT – Freedom to use and modify.