logo
Sign In

Web Search

tool

Web Search using SearXNG and Scrap first N Pages

constliakos
@constliakos
·
5 months ago
58K
107
137K
Updated 5 months ago
Created Jun 27, 2024
web_search.py
Python · 322 lines
"""
title: Web Search using SearXNG and Scrape first N Pages
author: constLiakos with enhancements by justinh-rahb and ther3zz
funding_url: https://github.com/open-webui
version: 0.1.14
license: MIT
changelog:
- 0.1.14 - Disable automatic citations when using custom citation events
- 0.1.13 - Change Description for better pattern matching by Autotool
- 0.1.12 - Add Event Emitter Option
"""

Web Search using SearXNG and Scrap first N Pages

IMPORTANT

Recommended: Switch to Open WebUI Native Function Calling

Open WebUI (v7.0.0+) now supports Native Function Calling with built-in tools. While this external tool remains functional, switching to the native implementation is highly recommended for better performance and integration.

Best Practice: Enable Native/Agentic Mode

To get the best results, you should enable Native Mode if your model provider supports it. This allows the LLM to handle tool execution more reliably than the default prompt-based engine.

How to enable:

  • Global: Admin Settings > Models > Select Model > Advanced Parameters > Function Calling: Native
  • Per-Chat: Chat Controls > Advanced Params > Function Calling: Native

⚠️ Critical Configuration for Search

Native search tools typically return snippets only (brief summaries). To ensure the LLM actually reads the page content, you must instruct it to perform a two-step process (Search → Fetch).

Required System Prompt:
Add this to your model's System Prompt to prevent hallucination from empty snippets:

text
"When performing a web search, do NOT rely solely on the search result snippets.
You must automatically use the 'fetch_url' tool to scrape and read the full content of the most relevant search results before generating your answer."
11