"""
Adaptive Memory v3.0 - Advanced Memory System for OpenWebUI
Author: AG

Overview

Adaptive Memory is a sophisticated plugin that provides persistent, personalized memory capabilities for Large Language Models (LLMs) within OpenWebUI. It enables LLMs to remember key information about users across separate conversations, creating a more natural and personalized experience.

The system dynamically extracts, filters, stores, and retrieves user-specific information from conversations, then intelligently injects relevant memories into future LLM prompts.

Key Features

1. Intelligent Memory Extraction

   • Automatically identifies facts, preferences, relationships, and goals from user messages
   • Categorizes memories with appropriate tags (identity, preference, behavior, relationship, goal, possession)
   • Focuses on user-specific information while filtering out general knowledge or trivia

2. Multi-layered Filtering Pipeline

   • Robust JSON parsing with fallback mechanisms for reliable memory extraction
   • Preference statement shortcuts for improved handling of common user likes/dislikes
   • Blacklist/whitelist system to control topic filtering
   • Smart deduplication using both semantic (embedding-based) and text-based similarity

3. Optimized Memory Retrieval

   • Vector-based similarity for efficient memory retrieval
   • Optional LLM-based relevance scoring for highest accuracy when needed
   • Performance optimizations to reduce unnecessary LLM calls

4. Adaptive Memory Management

   • Smart clustering and summarization of related older memories to prevent clutter
   • Intelligent pruning strategies when memory limits are reached
   • Configurable background tasks for maintenance operations

5. Memory Injection & Output Filtering

   • Injects contextually relevant memories into LLM prompts
   • Customizable memory display formats (bullet, numbered, paragraph)
   • Filters meta-explanations from LLM responses for cleaner output

6. Broad LLM Support

   • Generalized LLM provider configuration supporting both Ollama and OpenAI-compatible APIs
   • Configurable model selection and endpoint URLs
   • Optimized prompts for reliable JSON response parsing

7. Comprehensive Configuration System

   • Fine-grained control through "valve" settings
   • Input validation to prevent misconfiguration
   • Per-user configuration options

8. Memory Banks – categorize memories into Personal, Work, General (etc.) so retrieval / injection can be focused on a chosen context

Recent Improvements (v3.0)

1. Optimized Relevance Calculation - Reduced latency/cost by adding vector-only option and smart LLM call skipping when high confidence
2. Enhanced Memory Deduplication - Added embedding-based similarity for more accurate semantic duplicate detection
3. Intelligent Memory Pruning - Support for both FIFO and relevance-based pruning strategies when memory limits are reached
4. Cluster-Based Summarization - New system to group and summarize related memories by semantic similarity or shared tags
5. LLM Call Optimization - Reduced LLM usage through high-confidence vector similarity thresholds
6. Resilient JSON Parsing - Strengthened JSON extraction with robust fallbacks and smart parsing
7. Background Task Management - Configurable control over summarization, logging, and date update tasks
8. Enhanced Input Validation - Added comprehensive validation to prevent valve misconfiguration
9. Refined Filtering Logic - Fine-tuned filters and thresholds for better accuracy
10. Generalized LLM Provider Support - Unified configuration for Ollama and OpenAI-compatible APIs
11. Memory Banks - Added "Personal", "Work", and "General" memory banks for better organization
12. Fixed Configuration Persistence - Resolved Issue #19 where user-configured LLM provider settings weren't being applied correctly

Important Valves

Relevance & Similarity Configuration

• use_llm_for_relevance (bool, default: false) - Whether to use LLM for final relevance scoring (more accurate but higher latency/cost)
• llm_skip_relevance_threshold (float, default: 0.93) - If vector similarities exceed this threshold, skip LLM relevance call for efficiency
• vector_similarity_threshold (float, default: 0.7) - Minimum cosine similarity for initial vector-based memory filtering
• relevance_threshold (float, default: 0.7) - Minimum score for memories to be considered relevant for injection
• embedding_similarity_threshold (float, default: 0.97) - Threshold for considering two memories duplicates when using embedding similarity
• use_embeddings_for_deduplication (bool, default: true) - Use embedding-based similarity for more accurate semantic duplicate detection

Memory Management

• max_total_memories (int, default: 200) - Maximum number of memories per user before pruning
• pruning_strategy (str, default: "fifo") - Strategy for pruning: "fifo" (oldest first) or "least_relevant" (lowest relevance first)
• min_memory_length (int, default: 8) - Minimum length to save a memory
• deduplicate_memories (bool, default: true) - Prevent storing duplicate memories
• enable_short_preference_shortcut (bool, default: true) - Use direct memory save for short preference statements

Summarization Controls

• enable_summarization_task (bool, default: true) - Enable/disable background memory summarization
• summarization_interval (int, default: 7200) - Seconds between summarization runs
• summarization_strategy (str, default: "hybrid") - Clustering strategy: "embeddings", "tags", or "hybrid"
• summarization_min_cluster_size (int, default: 3) - Minimum memories in a cluster for summarization
• summarization_min_memory_age_days (int, default: 7) - Minimum age in days for memories to be considered

LLM Provider Configuration

• llm_provider_type (str, default: "ollama") - Type of LLM provider ("ollama" or "openai_compatible")
• llm_model_name (str, default: "llama3:latest") - Name of the model to use
• llm_api_endpoint_url (str, default: "http://host.docker.internal:11434/api/chat") - API endpoint URL
• llm_api_key (str, default: null) - API key (required for "openai_compatible" providers)

Display Settings

• show_status (bool, default: true) - Show memory operations status in chat
• show_memories (bool, default: true) - Show relevant memories in context
• memory_format (str, default: "bullet") - Format for displaying memories: "bullet", "paragraph", or "numbered"

Error Handling & Filtering

• filter_trivia (bool, default: true) - Filter out general knowledge/trivia
• blacklist_topics (str, default: null) - Comma-separated topics to ignore
• whitelist_keywords (str, default: null) - Comma-separated keywords to force-save
• enable_error_counter_guard (bool, default: true) - Temporarily disable features if error rates spike

Memory Categories

• enable_identity_memories (bool, default: true) - Collect identity information (name, age, etc.)
• enable_preference_memories (bool, default: true) - Collect preference information (likes, dislikes)
• enable_goal_memories (bool, default: true) - Collect goal information (aspirations)
• enable_relationship_memories (bool, default: true) - Collect relationship information (family, friends)
• enable_behavior_memories (bool, default: true) - Collect behavior information (habits, interests)
• enable_possession_memories (bool, default: true) - Collect possession information (things owned)

Memory Banks

• allowed_memory_banks: List[str] = Field(default=["General", "Personal", "Work"], description="List of allowed memory bank names for categorization.")
• default_memory_bank: str = Field(default="General", description="Default memory bank assigned when LLM omits or supplies an invalid bank.")

Adaptive Memory enables dynamic, evolving, personalized memory for LLMs in OpenWebUI, making conversations more natural and responsive over time.
"""