logo
Sign In

Three Brained Searches LangSearch OR

tool

An advanced search tool that do three searches in parallel and uses 2 LLMs to perform URL selection and content extraction, returns only the most relevant information.

cernunnos
@cernunnos
·
a year ago
15
1
1.3K
Created Dec 6, 2025
three_brained_searches_langsearch_or.py
Python · 1759 lines
"""
title: Three Smart Searches with LangSearch API, Jina scraper and beautifulsoup scraper
author: Lug, scraper part from Web search v0.2.1 by siwadon
author_url: https://brainedai.netlify.app
description: An advanced search tool that do three searches in parallel and uses 2 LLMs to perform URL selection, and content extraction, returns only the most relevant information.
required_open_webui_version: 0.6.0
requirements: aiohttp, asyncio, requests, beautifulsoup4
version: 0.9.3
license: MIT
"""
import json
import time

Version 0.9.3, it can scrape arxiv paper (HTML or PDF) and other improvements.

A tool specially designed for BDS agent, but work well for any agent.

This tool works in four steps:
Step 1: performs three searches on LangSearch in parallel (“query_1” “query_2” and “query_3”).
Step 2: For each query independently, LLM 1 selects a defined number of URLs from those in the searches responses based on “long_query” if it exists, otherwise based on “query_1/2/3”.
Step 3: The selected URLs are scraped by a cloud scraper (Jina) for query 1 and by an internal scraper (beautifulsoup) for queries 2 and 3; if for queries 2 and 3 there are URLs ending in .pdf, they are sent to the Jina scraper (because beautifulsoup cannot scrape PDFs)
Step 4: The pages scraped by Jina from query 1 and those scraped by BeautifulSoup from queries 2 and 3 are sent together to LLM 2, which selects the interesting information based on “long_query” if it exists, otherwise based on “query_1/2/3”.

The default LLM 2 is fast and free, but if you are doing scientific or complicated research, I recommend using an LLM such as Deepseek V3.2 or Kimi K2 Thinking (for deepseek you can activate “Llm 2 Enable Reasoning”).

Jina scraper without Jina API key scrapes 2 URLs at the same time. With Jina API key, it scrapes 5 URLs at the same time.

The main function of this tool is: three_smart_searches (3 query at same time)

The two additional functions:
summarize_url: scrapes and summarizes web pages
scrape_url: scrapes only web pages.

Important:
You need to have LangSearch Api key and a Open Router Api key.
The minimum parameters to enter are the API keys for OpenRouter (for LLM 1 and LLM 2).

A special version for OpenRouter that allows you to configure fallback models and prioritize providers using a list (provider IDs are separated by commas without spaces) or by enabling “Llm 2 Enable Throughput Sort.” If you enable this option, you disable the list of providers to prioritize and enable prioritization based on throughput, and you also enable the ability to disable providers that are too expensive.
https://openrouter.ai/docs/features/provider-routing

0