> ## Documentation Index
> Fetch the complete documentation index at: https://docs.expectedparrot.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Log Management
> EDSL provides comprehensive log management capabilities through the **LogManager** class. This powerful tool allows you to filter, analyze, and manage EDSL log files with advanced features for debugging, monitoring, and data analysis.
## Quick Start
```python theme={null}
from edsl.logger import LogManager
# Create LogManager instance
log_manager = LogManager()
# View overview with statistics and available commands
log_manager
```
This displays a rich overview of your log file:
```
LogManager(
π Log file: /Users/john/.edsl/logs/edsl.log
π Total entries: 9,420
π
Date range: 2025-08-15 23:40 to 2025-08-16 09:23
π Levels: INFO:5471, ERROR:3949
π Top logger: edsl.edsl.coop.coop
π‘ Common commands:
.get_filtered_entries(level='ERROR', n=50) # Get recent errors
.to_scenario_list(level='ERROR', since_hours=24) # Convert to scenarios
.analyze_patterns(n=100) # Pattern analysis
.archive() # Archive and clear logs
.clear() # Clear all log entries
)
```
## Core Features
### Log Filtering
Filter log entries using multiple criteria:
```python expandable theme={null}
from edsl.logger import LogManager
log_manager = LogManager()
# Filter by log level
errors = log_manager.get_filtered_entries(level='ERROR', n=50)
warnings_and_above = log_manager.get_filtered_entries(min_level='WARNING')
# Time-based filtering
recent = log_manager.get_filtered_entries(since_hours=24)
last_week = log_manager.get_filtered_entries(since_hours=168) # 7 days
date_range = log_manager.get_filtered_entries(
start_date='2024-08-15',
end_date='2024-08-16'
)
# Pattern matching
auth_issues = log_manager.get_filtered_entries(pattern='authentication|login')
failures = log_manager.get_filtered_entries(pattern='fail.*|error.*', n=100)
# Logger-specific filtering
coop_logs = log_manager.get_filtered_entries(logger_pattern='coop')
job_logs = log_manager.get_filtered_entries(logger_pattern='jobs')
# Complex combinations
recent_job_errors = log_manager.get_filtered_entries(
level='ERROR',
since_hours=12,
logger_pattern='jobs',
n=25
)
```
### Scenario Conversion for Analysis
Convert log entries to EDSL scenarios for advanced analysis:
```python theme={null}
# Convert logs to scenarios
error_scenarios = log_manager.to_scenario_list(level='ERROR', n=50)
# Each log entry becomes a scenario with rich metadata
first_scenario = error_scenarios[0]
print(f"Error level: {first_scenario['level']}")
print(f"Timestamp: {first_scenario['timestamp_str']}")
print(f"Logger: {first_scenario['logger_module']}")
print(f"Message: {first_scenario['message']}")
print(f"Is error: {first_scenario['is_error']}")
print(f"Hour of day: {first_scenario['hour']}")
```
**Available scenario fields:** - `timestamp_str`: Formatted timestamp - `logger_name`: Full logger name - `logger_module`: Last part of logger name - `level`: Log level (ERROR, INFO, etc.) - `level_int`: Numeric level for comparison - `message`: Log message content - `raw_line`: Original log line - `hour`, `minute`, `weekday`: Time components - `date_str`: Date in YYYY-MM-DD format - `is_error`: Boolean for ERROR/CRITICAL levels - `is_warning_or_above`: Boolean for WARNING+ levels - `has_exception`: Boolean if message contains βexceptionβ - `has_failed`: Boolean if message contains βfailβ - `message_length`: Character count of message - `words_count`: Word count of message
### Using Log Scenarios with EDSL
Analyze log entries using EDSL questions:
```python expandable theme={null}
from edsl import QuestionFreeText, QuestionMultipleChoice
# Convert error logs to scenarios
error_scenarios = log_manager.to_scenario_list(level='ERROR', n=25)
# Analyze error causes
analysis_question = QuestionFreeText(
question_name="error_analysis",
question_text="What likely caused this error: {{ message }}?"
)
# Categorize errors
category_question = QuestionMultipleChoice(
question_name="error_category",
question_text="Categorize this error: {{ message }}",
question_options=[
"Authentication/Authorization",
"Network/Connection",
"Data/Validation",
"System/Resource",
"Configuration",
"Other"
]
)
# Run analysis
from edsl import Survey
survey = Survey([analysis_question, category_question])
results = survey.by(error_scenarios).run()
# View results
print(results.select("error_analysis", "error_category").print())
```
### Pattern Analysis
Extract insights from log patterns:
```python expandable theme={null}
# Analyze recent log patterns
analysis = log_manager.analyze_patterns(n=1000)
# View time patterns
print(f"Busiest hour: {analysis['time_patterns']['busiest_hour']}")
print(f"Daily distribution: {analysis['time_patterns']['daily_distribution']}")
# View level distribution
print(f"Level counts: {analysis['level_patterns']}")
# View top loggers
print(f"Top loggers: {analysis['logger_patterns']}")
# Error analysis
if 'error_patterns' in analysis:
error_info = analysis['error_patterns']
print(f"Error percentage: {error_info['error_percentage']:.1f}%")
print(f"Exception count: {error_info['exception_count']}")
```
### Statistics and Reporting
Get comprehensive statistics about your logs:
```python theme={null}
# Overall statistics
stats = log_manager.get_stats()
print(f"Total entries: {stats['total']:,}")
print(f"Date range: {stats['date_range']['earliest']} to {stats['date_range']['latest']}")
print(f"Level distribution: {stats['level_counts']}")
print(f"Top 5 loggers: {list(stats['top_loggers'].items())[:5]}")
# Statistics for filtered entries
recent_errors = log_manager.get_filtered_entries(level='ERROR', since_hours=24)
error_stats = log_manager.get_stats(recent_errors)
print(f"Recent errors: {error_stats['total']}")
```
### Export and Sharing
Export filtered logs or share via Expected Parrot:
```python expandable theme={null}
from pathlib import Path
# Export filtered logs to file
export_path = Path('recent_errors.log')
count = log_manager.export_filtered_logs(
export_path,
level='ERROR',
since_hours=24
)
print(f"Exported {count} entries to {export_path}")
# Share logs via Expected Parrot
from edsl.coop import Coop
coop = Coop()
# Upload filtered logs as file
result = coop.send_log(level='ERROR', n=50, alias='recent-errors')
# Upload as scenarios for collaborative analysis
result = coop.send_log(
level='ERROR',
since_hours=24,
as_scenario_list=True,
alias='error-scenarios-for-analysis'
)
```
## Log Maintenance
### Archive Logs
Create timestamped backups of your logs:
```python theme={null}
# Archive with compression and clear original (default)
archive_path = log_manager.archive()
# Archive only, keep original
backup_path = log_manager.archive(clear_after_archive=False)
# Custom archive location without compression
custom_backup = log_manager.archive(
archive_path=Path('~/my_backups'),
compress=False,
clear_after_archive=False
)
```
The archive process shows detailed progress:
```
π¦ About to archive EDSL log file:
π Source: /Users/john/.edsl/logs/edsl.log
π Entries: 9,420 (2025-08-15 to 2025-08-16)
π¦ Archive: /Users/john/.edsl/archives/edsl_log_20250816_092142.log.gz
ποΈ Compress: Yes
π§Ή Clear after: Yes
Continue? (type 'yes' to confirm):
```
### Clear Logs
Remove all log entries to start fresh:
```python theme={null}
# Clear with confirmation prompt (safe)
success = log_manager.clear()
# Clear immediately for scripts (use with caution)
success = log_manager.clear(confirm=True)
```
The clear process includes safety confirmation:
```
β οΈ About to clear EDSL log file:
π File: /Users/john/.edsl/logs/edsl.log
π Entries: 9,420
β οΈ This action cannot be undone!
Continue? (type 'yes' to confirm):
```
## Advanced Usage
### Custom Log Files
Work with non-default log files:
```python theme={null}
from pathlib import Path
# Custom log file location
custom_log_manager = LogManager(Path('/path/to/custom.log'))
# Check if log file exists
if custom_log_manager.log_file_path.exists():
stats = custom_log_manager.get_stats()
print(f"Custom log has {stats['total']} entries")
```
### Filtering Best Practices
Combine filters for precise results:
```python expandable theme={null}
# Find authentication errors in the last day
auth_errors = log_manager.get_filtered_entries(
level='ERROR',
pattern='auth.*|login.*|token.*',
since_hours=24,
case_sensitive=False
)
# Find slow operations (assuming they log with "slow" keyword)
slow_operations = log_manager.get_filtered_entries(
pattern='slow.*|timeout.*|.*ms$',
since_hours=6,
n=50
)
# Debug specific component
coop_debug = log_manager.get_filtered_entries(
logger_pattern='coop.*',
min_level='DEBUG',
since_minutes=30
)
```
### Performance Optimization
LogManager automatically caches statistics for better performance:
```python theme={null}
# First call computes and caches stats
log_manager = LogManager()
print(log_manager) # Caches statistics
# Subsequent calls use cached data
stats = log_manager.get_stats() # Fast - uses cache
# Cache is cleared after maintenance operations
log_manager.clear(confirm=True) # Clears cache automatically
print(log_manager) # Recomputes statistics
```
## Integration Examples
### Debugging Workflows
```python expandable theme={null}
# Daily error review
def daily_error_review():
log_manager = LogManager()
# Get yesterday's errors
errors = log_manager.get_filtered_entries(
level='ERROR',
since_hours=24
)
if errors:
print(f"Found {len(errors)} errors in last 24 hours:")
for error in errors[-5:]: # Show last 5
print(f" {error.timestamp}: {error.message}")
# Convert to scenarios for analysis
error_scenarios = log_manager.to_scenario_list(entries=errors)
# Analyze with EDSL
from edsl import QuestionFreeText
q = QuestionFreeText(
question_name="priority",
question_text="Rate the priority of this error (1-5): {{ message }}"
)
priorities = q.by(error_scenarios[:10]).run() # Analyze top 10
print(priorities.select("priority").print())
```
### Monitoring Automation
```python expandable theme={null}
# Automated log monitoring
def monitor_system_health():
log_manager = LogManager()
# Check for recent critical issues
critical = log_manager.get_filtered_entries(
level='CRITICAL',
since_minutes=60
)
if critical:
# Send alert via Expected Parrot
from edsl.coop import Coop
coop = Coop()
alert_result = coop.send_log(
level='CRITICAL',
since_minutes=60,
as_scenario_list=True,
alias='critical-alerts'
)
print(f"Alert sent: {alert_result['url']}")
# Weekly log maintenance
from datetime import datetime
if datetime.now().weekday() == 0: # Monday
archive_path = log_manager.archive()
print(f"Weekly archive created: {archive_path}")
```
### Research and Analysis
```python expandable theme={null}
# Research log patterns for insights
def analyze_usage_patterns():
log_manager = LogManager()
# Convert logs to scenarios
scenarios = log_manager.to_scenario_list(n=1000)
# Research questions about usage
from edsl import QuestionMultipleChoice, Survey
q1 = QuestionMultipleChoice(
question_name="time_category",
question_text="What time category is {{ hour }}:00?",
question_options=["Night (0-5)", "Morning (6-11)", "Afternoon (12-17)", "Evening (18-23)"]
)
q2 = QuestionMultipleChoice(
question_name="operation_type",
question_text="What type of operation does this suggest: {{ message }}",
question_options=["API Call", "Data Processing", "User Interface", "System Maintenance", "Other"]
)
survey = Survey([q1, q2])
results = survey.by(scenarios).run()
# Analyze patterns
time_patterns = results.select("time_category").to_list()
operation_patterns = results.select("operation_type").to_list()
print("Usage patterns analysis complete!")
return results
```
## Jupyter Notebook Integration
LogManager provides rich HTML displays in Jupyter notebooks:
```python theme={null}
# In Jupyter notebook
from edsl.logger import LogManager
log_manager = LogManager()
log_manager # Shows rich HTML table with colored log levels
```
The HTML display includes: - Overview with file path, entry count, and date range - Color-coded log level distribution (red for errors, blue for info) - Top loggers table with counts - Interactive command examples with syntax highlighting
## API Reference
### LogManager Class
> *class* LogManager(*log\_file\_path=None*)
Main class for EDSL log management.
| | |
| :------------- | :------------------------------------------------------------------------------------------------------------ |
| **Parameters** | **log\_file\_path** (*Path*\*,\* *optional*) β Optional path to log file. Defaults to `~/.edsl/logs/edsl.log` |
> get\_filtered\_entries(\*\**kwargs*)
Filter log entries by various criteria.
| | |
| :-------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Parameters** | β’ **n** (*int*\*,\* *optional*) β Maximum number of entries to return
β’ **level** (*str* *or* *List*\*\[**str**]\*\*,\* *optional*) β Specific level(s) to include (βDEBUGβ, βINFOβ, βWARNINGβ, βERRORβ, βCRITICALβ)
β’ **min\_level** (*str*\*,\* *optional*) β Minimum level (includes this level and above)
β’ **since\_hours** (*float*\*,\* *optional*) β Include entries from last N hours
β’ **since\_minutes** (*float*\*,\* *optional*) β Include entries from last N minutes
β’ **start\_date** (*str* *or* *datetime*\*,\* *optional*) β Start date for filtering (βYYYY-MM-DDβ)
β’ **end\_date** (*str* *or* *datetime*\*,\* *optional*) β End date for filtering (βYYYY-MM-DDβ)
β’ **pattern** (*str*\*,\* *optional*) β Regex pattern for message content
β’ **logger\_pattern** (*str*\*,\* *optional*) β Regex pattern for logger names
β’ **case\_sensitive** (*bool*\*,\* *optional*) β Whether pattern matching is case sensitive
β’ **reverse** (*bool*\*,\* *optional*) β Return in reverse chronological order |
| **Returns** | List of filtered log entries |
| **Return type** | List\[[LogEntry](/en/latest/logging#logentry-class)] |
> to\_scenario\_list(*entries=None*, \*\**filter\_kwargs*)
Convert log entries to EDSL scenarios.
| | |
| :-------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Parameters** | β’ **entries** (*List*\*\[*[*LogEntry*](/en/latest/logging#logentry-class)*]\*\*,\* *optional*) β Pre-filtered entries to convert
β’ **filter\_kwargs** β Arguments for get\_filtered\_entries() |
| **Returns** | ScenarioList with log data |
| **Return type** | [ScenarioList](/en/latest/scenarios#scenariolist-class) |
> analyze\_patterns(\*\**filter\_kwargs*)
Analyze log patterns and extract insights.
| | |
| :-------------- | :------------------------------------------------------------ |
| **Parameters** | β’ **filter\_kwargs** β Arguments for get\_filtered\_entries() |
| **Returns** | Dictionary with pattern analysis |
| **Return type** | Dict\[str, Any] |
> get\_stats(*entries=None*)
Get statistics about log entries.
| | |
| :-------------- | :--------------------------------------------------------------------------------------------------------------------------- |
| **Parameters** | β’ **entries** (*List*\*\[*[*LogEntry*](/en/latest/logging#logentry-class)*]\*\*,\* *optional*) β Specific entries to analyze |
| **Returns** | Statistics dictionary |
| **Return type** | Dict\[str, Any] |
> export\_filtered\_logs(*output\_path*, \*\**filter\_kwargs*)
Export filtered log entries to file.
| | |
| :-------------- | :---------------------------------------------------------------------------------------------------------------------- |
| **Parameters** | β’ **output\_path** (*Path*) β Path for output file
β’ **filter\_kwargs** β Arguments for get\_filtered\_entries() |
| **Returns** | Number of entries exported |
| **Return type** | int |
> archive(*archive\_path=None*, *clear\_after\_archive=True*, *compress=True*, *confirm=False*)
Archive log file with timestamp.
| | |
| :-------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Parameters** | β’ **archive\_path** (*Path*\*,\* *optional*) β Directory for archive (default: \~/.edsl/archives/)
β’ **clear\_after\_archive** (*bool*) β Clear original after archiving
β’ **compress** (*bool*) β Gzip compress the archive
β’ **confirm** (*bool*) β Skip confirmation prompt |
| **Returns** | Path to created archive |
| **Return type** | Path or None |
> clear(*confirm=False*)
Clear all log entries.
| | |
| :-------------- | :------------------------------------------------ |
| **Parameters** | β’ **confirm** (*bool*) β Skip confirmation prompt |
| **Returns** | Success status |
| **Return type** | bool |
### LogEntry Class
> *class* LogEntry
Represents a parsed log entry.
| | |
| :------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Parameters** | β’ **timestamp** (*datetime*) β When the log entry was created
β’ **logger\_name** (*str*) β Name of the logger
β’ **level** (*str*) β Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
β’ **message** (*str*) β Log message content
β’ **raw\_line** (*str*) β Original log line
β’ **level\_int** (*int*) β Numeric level for comparisons |
## Best Practices
### Safety Guidelines
* Always use confirmation prompts for destructive operations (default behavior)
* Archive logs before clearing in production environments
* Use specific filters to avoid processing unnecessary log entries
* Test complex regex patterns with small datasets first
### Performance Tips
* Use `n` parameter to limit results when exploring large logs
* Combine multiple filters in single calls rather than chaining
* Cache LogManager instances when doing multiple operations
* Consider archiving very large log files before analysis
### Common Patterns
**Error Investigation:**
1. Filter recent errors: `get_filtered_entries(level='ERROR', since_hours=24)`
2. Convert to scenarios for analysis: `to_scenario_list()`
3. Use EDSL questions to categorize and analyze
4. Export findings or share via Expected Parrot
**Regular Maintenance:**
1. Weekly archive: `archive()` (includes clearing)
2. Monitor critical issues: `get_filtered_entries(level='CRITICAL')`
3. Pattern analysis: `analyze_patterns()` for trends
**Debugging Workflows:**
1. Filter by component: `logger_pattern='component_name'`
2. Time-bound investigation: `since_hours=X` for specific incidents
3. Pattern matching: `pattern='specific_error_text'`
4. Export for sharing: `export_filtered_logs()`
## Troubleshooting
### Common Issues
**Log file not found:**
Check the log file path. EDSL logs are typically at `~/.edsl/logs/edsl.log`
**No entries returned:**
* Verify log file has content: `log_manager.get_stats()`
* Check filter criteria are not too restrictive
* Ensure date ranges are correct
**Performance issues with large logs:**
* Use `n` parameter to limit results
* Consider archiving old entries
* Filter by time ranges first, then other criteria
**Permission errors:**
* Ensure read access to log file
* For archive/clear operations, ensure write access to log directory
### Getting Help
* View available commands: `print(log_manager)`
* Check method documentation: `help(log_manager.method_name)`
* View log overview: `log_manager.get_stats()`
* Test filters with small limits: `get_filtered_entries(n=10, ...)`
The LogManager provides powerful capabilities for understanding, analyzing, and maintaining your EDSL logs. Whether youβre debugging issues, monitoring system health, or conducting research on usage patterns, LogManager offers the tools you need for effective log management.