Data Retention¶

KrakenHashes includes a comprehensive data retention system that automatically and securely purges old hashlists and associated data based on configurable retention policies.

Overview¶

The retention system ensures compliance with data retention policies by automatically removing expired data from both the database and filesystem. It includes secure deletion mechanisms to prevent data recovery.

Retention Policy Configuration¶

Default Retention Policy¶

• Purpose: Sets the default number of months after which hashlists are automatically deleted
• Scope: Applies to all hashlists unless overridden by client-specific settings
• Special Values:
  • 0 = Keep forever (no automatic deletion)
  • NULL = Use system default
  • Any positive integer = Number of months to retain data

Client-Specific Retention¶

Each client can have their own retention period that overrides the system default: - Set during client creation or via the client management interface - Takes precedence over the default retention setting - Applies to all hashlists associated with that client

Automatic Purge Process¶

Scheduling¶

The retention purge runs automatically: - On Startup: 15 seconds after backend initialization - Daily: At midnight (server time) - Processes hashlists in batches of 1000 for scalability

What Gets Deleted¶

When a hashlist expires based on retention policy:

1. Database Records:
2. Hashlist record from hashlists table
3. All associations in hashlist_hashes table
4. Orphaned hashes (not associated with any other hashlist)

5. Related records via CASCADE deletion:

   • agent_hashlists entries
   • job_executions entries

6. Filesystem:

7. Hashlist file from /var/lib/krakenhashes/hashlists/
8. File is securely overwritten with random data before deletion

9. Prevents recovery via filesystem forensics

10. PostgreSQL Maintenance:

11. VACUUM ANALYZE runs on affected tables
12. Reclaims storage space
13. Removes dead tuples to prevent WAL recovery

What Does NOT Get Deleted¶

Security Features¶

• Secure File Deletion: Files are overwritten with random data before removal
• Transaction Safety: All database operations occur within a single transaction
• Audit Logging: Complete audit trail of all deletion operations
• VACUUM Operations: Prevents data recovery from PostgreSQL internals

Agent-Side Cleanup¶

Agents automatically clean up old files to prevent storage accumulation:

Retention Policy¶

• 3-day retention for temporary files
• Runs every 6 hours automatically
• Initial cleanup 1 minute after agent startup

Files Cleaned¶

• Hashlists: Removed after 3 days of inactivity
• Rule Chunks: Temporary rule segments deleted after 3 days
• Chunk ID Files: Removed when associated chunks are deleted
• Preserved Files: Base rules, wordlists, and binaries are never auto-deleted

Configuration via API¶

View Current Settings¶

GET /api/admin/settings/retention

Response:

{
  "default_retention_months": 36,
  "last_purge_run": "2025-09-17T10:35:00.391Z"
}

Update Retention Settings¶

PUT /api/admin/settings/retention

Request:

{
  "default_retention_months": 24
}

Requires administrator privileges.

Important Considerations¶

Data Preservation¶

• Clients themselves are never deleted by retention policies
• Hashes shared across multiple hashlists are only deleted when orphaned
• User accounts and system settings are unaffected
• Potfile retains ALL plaintext passwords permanently - must be manually managed

Compliance¶

• Retention policies help meet data protection regulations
• Audit logs provide evidence of proper data disposal
• Secure deletion prevents unauthorized data recovery
• NOTE: Potfile retention may conflict with data protection requirements - consider implementing separate potfile cleanup procedures

Performance Impact¶

• VACUUM operations may temporarily impact database performance
• Purge operations are logged but don't block normal operations
• Large deletions are handled in batches to minimize impact

Potfile Management¶

Monitoring¶

Check retention activity in the backend logs:

docker exec krakenhashes-app tail -f /var/log/krakenhashes/backend/backend.log | grep -i purge

View last purge run time:

SELECT value FROM client_settings WHERE key = 'last_purge_run';