mirror of
https://github.com/Dvorinka/Trackeep.git
synced 2026-06-04 12:32:58 +00:00
feat: major feature updates and cleanup
- Add Redis architecture implementation - Update browser extension functionality - Clean up deprecated files and documentation - Enhance backend handlers for auth, messages, search - Add new configuration options and settings - Update Docker and deployment configurations
This commit is contained in:
Tomas Dvorak
@@ -0,0 +1,894 @@
|
||||
# Redis Architecture Analysis for Trackeep
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Trackeep** is a self-hosted productivity and knowledge management platform built with Go (Gin framework), PostgreSQL, and React. The application already includes the `go-redis/redis/v8` dependency but currently operates with in-memory fallbacks for caching, sessions, and rate limiting. This analysis evaluates Redis deployment across multiple dimensions to determine architectural alignment and implementation strategy.
|
||||
|
||||
**Current Infrastructure:**
|
||||
- **Backend:** Go 1.24 with Gin web framework
|
||||
- **Database:** PostgreSQL 15 (primary data store)
|
||||
- **Frontend:** React + TypeScript + Vite
|
||||
- **Deployment:** Docker Compose (single-node, self-hosted)
|
||||
- **Current Caching:** In-memory maps with mutex locks
|
||||
- **Current Sessions:** In-memory map storage
|
||||
- **Current Rate Limiting:** Per-instance in-memory tracking
|
||||
|
||||
---
|
||||
|
||||
## 1. Use Case Analysis
|
||||
|
||||
### 1.1 Caching Frequently Accessed Database Queries
|
||||
|
||||
**Current State:**
|
||||
The application uses [`MemoryCache`](backend/middleware/memory_cache.go:21) with `sync.RWMutex` for thread-safe in-memory caching. Cache entries expire via a cleanup goroutine running every minute.
|
||||
|
||||
**Redis Opportunity:**
|
||||
|
||||
| Query Pattern | Current Implementation | Redis Benefit |
|
||||
|---------------|---------------------|---------------|
|
||||
| User profiles | Direct DB query on each request | Cache for 5-15 min, reduces user table queries |
|
||||
| Search results | Computed on every search | Cache complex searches for 5-10 min |
|
||||
| Analytics dashboards | Aggregated from multiple tables | Cache pre-computed aggregations for 1 hour |
|
||||
| Learning paths/courses | Filtered queries with joins | Cache popular paths for 30 min |
|
||||
| YouTube channel data | Database cache + in-memory fallback | Unified Redis cache with TTL |
|
||||
| Marketplace items | Sorted/filtered queries | Cache trending/top-rated items |
|
||||
|
||||
**Specific High-Value Caches:**
|
||||
|
||||
1. **Enhanced Search Cache** ([`search_enhanced.go`](backend/handlers/search_enhanced.go:73))
|
||||
- Complex multi-table searches across bookmarks, tasks, notes, files
|
||||
- Redis can cache results with content-type aggregation
|
||||
- Suggested TTL: 5 minutes for dynamic content
|
||||
|
||||
2. **Analytics Dashboard Cache** ([`analytics.go`](backend/handlers/analytics.go:24))
|
||||
- Expensive aggregations across analytics, learning, GitHub, habit tables
|
||||
- Pre-computed dashboard data can be cached for 15-30 minutes
|
||||
- User-specific caching with tags for invalidation
|
||||
|
||||
3. **AI Recommendations Cache** ([`ai_recommendations.go`](backend/handlers/ai_recommendations.go:49))
|
||||
- ML-generated recommendations are expensive to compute
|
||||
- Cache recommendation lists per user for 1 hour
|
||||
- Cache recommendation statistics for 30 minutes
|
||||
|
||||
**Implementation Approach:**
|
||||
```go
|
||||
// Cache key structure
|
||||
trackeep:{resource}:{user_id}:{query_hash}
|
||||
trackeep:search:{user_id}:{md5(query+filters)}
|
||||
trackeep:analytics:dashboard:{user_id}:{date_range}
|
||||
trackeep:recommendations:{user_id}:{type}
|
||||
```
|
||||
|
||||
### 1.2 Distributed Session State Management
|
||||
|
||||
**Current State:**
|
||||
The [`RedisSessionStore`](backend/middleware/session.go:36) struct exists but uses `map[string]*SessionData` as a fallback in-memory store. Sessions are lost on server restart and don't work across multiple backend instances.
|
||||
|
||||
**Session Data Structure:**
|
||||
```go
|
||||
type SessionData struct {
|
||||
UserID uint `json:"user_id"`
|
||||
Email string `json:"email"`
|
||||
Username string `json:"username"`
|
||||
Role string `json:"role"`
|
||||
SessionID string `json:"session_id"`
|
||||
IPAddress string `json:"ip_address"`
|
||||
UserAgent string `json:"user_agent"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastActive time.Time `json:"last_active"`
|
||||
}
|
||||
```
|
||||
|
||||
**Redis Implementation:**
|
||||
- Use Redis Hash or JSON data type for session storage
|
||||
- TTL: 24 hours (matching current cleanup logic)
|
||||
- Enable session persistence across deployments
|
||||
- Support horizontal scaling of backend instances
|
||||
- Session invalidation on logout/password change
|
||||
|
||||
**Key Pattern:**
|
||||
```
|
||||
trackeep:session:{session_id} -> SessionData (JSON)
|
||||
trackeep:user:sessions:{user_id} -> Set of active session IDs
|
||||
```
|
||||
|
||||
### 1.3 Real-Time Leaderboards and Rate Tracking
|
||||
|
||||
**Current Opportunities:**
|
||||
|
||||
1. **Community Challenges Leaderboard** ([`community.go`](backend/handlers/community.go:1))
|
||||
- Track challenge participants and completion rates
|
||||
- Real-time leaderboard updates
|
||||
- Redis Sorted Sets (`ZADD`, `ZREVRANGE`) ideal for ranking
|
||||
|
||||
2. **Marketplace Item Rankings** ([`marketplace.go`](backend/handlers/marketplace.go:1))
|
||||
- Sort by downloads, rating, views
|
||||
- Trending items calculation
|
||||
- Redis can maintain real-time counters
|
||||
|
||||
3. **User Analytics Streaks** ([`analytics.go`](backend/handlers/analytics.go:786))
|
||||
- Learning streaks tracking
|
||||
- Daily habit completion counts
|
||||
- Redis counters with daily windows
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
// Challenge leaderboard
|
||||
trackeep:challenge:{id}:leaderboard -> Sorted Set (score: completion_time, member: user_id)
|
||||
|
||||
// Marketplace trending
|
||||
trackeep:marketplace:trending -> Sorted Set (score: view_count_24h, member: item_id)
|
||||
|
||||
// User learning streaks
|
||||
trackeep:user:{id}:learning_streak -> Hash (current_streak, last_date, max_streak)
|
||||
```
|
||||
|
||||
### 1.4 Rate Limiting
|
||||
|
||||
**Current State:**
|
||||
The [`RateLimiter`](backend/middleware/rate_limiter.go:13) uses in-memory `map[string]*ClientInfo` with per-IP tracking. This doesn't work across multiple instances and is vulnerable to restart clearing.
|
||||
|
||||
**Redis-Based Rate Limiting:**
|
||||
|
||||
| Rate Limit Type | Window | Current Limit | Redis Strategy |
|
||||
|-----------------|--------|---------------|----------------|
|
||||
| General API | 1 minute | 100 requests | Sliding window with `ZADD` |
|
||||
| Search | 1 minute | 100 requests | Fixed window with `INCR` + `EXPIRE` |
|
||||
| AI Chat | 1 minute | 20 requests | Token bucket algorithm |
|
||||
| Login attempts | 5 minutes | 5 attempts | Count with `INCR` + longer TTL |
|
||||
| File uploads | 10 minutes | 10 uploads | Sliding window per user |
|
||||
|
||||
**Token Bucket Implementation:**
|
||||
```go
|
||||
// Redis Lua script for atomic token bucket
|
||||
local key = KEYS[1]
|
||||
local capacity = tonumber(ARGV[1])
|
||||
local refill_rate = tonumber(ARGV[2])
|
||||
local now = tonumber(ARGV[3])
|
||||
|
||||
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
|
||||
local tokens = tonumber(bucket[1]) or capacity
|
||||
local last_refill = tonumber(bucket[2]) or now
|
||||
|
||||
local delta = math.min(capacity, tokens + (now - last_refill) * refill_rate)
|
||||
|
||||
if delta >= 1 then
|
||||
redis.call('HMSET', key, 'tokens', delta - 1, 'last_refill', now)
|
||||
redis.call('EXPIRE', key, 3600)
|
||||
return 1
|
||||
else
|
||||
redis.call('HMSET', key, 'tokens', delta, 'last_refill', now)
|
||||
redis.call('EXPIRE', key, 3600)
|
||||
return 0
|
||||
end
|
||||
```
|
||||
|
||||
### 1.5 Publish-Subscribe Messaging Patterns
|
||||
|
||||
**Current State:**
|
||||
Real-time messaging uses WebSocket hub [`MessagesHub`](backend/services/messages_realtime.go:28) with in-memory `conversationClients` map. This is single-node only.
|
||||
|
||||
**Redis Pub/Sub for Multi-Node:**
|
||||
|
||||
1. **Cross-Instance Message Broadcasting**
|
||||
- When horizontal scaling is needed, Redis Pub/Sub connects multiple backend instances
|
||||
- Pattern: `trackeep:messages:{conversation_id}`
|
||||
|
||||
2. **Notification System**
|
||||
- Real-time notifications for new followers, messages, mentions
|
||||
- Pattern: `trackeep:notifications:{user_id}`
|
||||
|
||||
3. **System Events**
|
||||
- Cache invalidation broadcasts
|
||||
- Configuration updates
|
||||
- Analytics aggregation triggers
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
// Subscribe to conversation messages
|
||||
pubsub := redisClient.Subscribe(ctx, "trackeep:messages:123")
|
||||
|
||||
// Publish message to all nodes
|
||||
redisClient.Publish(ctx, "trackeep:messages:123", messageJSON)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Access Patterns and Latency Requirements
|
||||
|
||||
### 2.1 Current Database Access Patterns
|
||||
|
||||
Based on code analysis, the application exhibits these access patterns:
|
||||
|
||||
| Pattern | Frequency | Tables | Latency Sensitivity |
|
||||
|---------|-----------|--------|---------------------|
|
||||
| User authentication | High | users | Very High (< 100ms) |
|
||||
| Search queries | Medium-High | bookmarks, tasks, notes, files | High (< 500ms) |
|
||||
| Analytics aggregation | Medium | analytics, learning_analytics | Medium (< 2s) |
|
||||
| Message retrieval | High | messages, conversations | High (< 200ms) |
|
||||
| AI recommendations | Low-Medium | ai_recommendations | Low (< 5s acceptable) |
|
||||
| Marketplace browsing | Medium | marketplace_items | Medium (< 1s) |
|
||||
| Audit logging | High (write) | audit_logs | Low (async) |
|
||||
|
||||
### 2.2 Latency Requirements Analysis
|
||||
|
||||
**Critical Paths for Redis Caching:**
|
||||
|
||||
1. **Authentication Flow** (Target: < 100ms)
|
||||
- Current: DB query for user + session lookup
|
||||
- With Redis: Session cache + user profile cache
|
||||
- Expected improvement: 60-80% latency reduction
|
||||
|
||||
2. **Dashboard Load** (Target: < 500ms)
|
||||
- Current: Multiple aggregation queries
|
||||
- With Redis: Pre-computed analytics cache
|
||||
- Expected improvement: 70-90% latency reduction
|
||||
|
||||
3. **Search Results** (Target: < 300ms)
|
||||
- Current: Full-text search across 4+ tables
|
||||
- With Redis: Cached results for common queries
|
||||
- Expected improvement: 50-80% latency reduction
|
||||
|
||||
### 2.3 Cache Invalidation Strategy
|
||||
|
||||
**Event-Based Invalidation:**
|
||||
|
||||
| Data Type | Cache Keys | Invalidation Trigger |
|
||||
|-----------|------------|---------------------|
|
||||
| User profile | `user:{id}:profile` | User update, password change |
|
||||
| Search results | `search:{user_id}:*` | Any content creation/update |
|
||||
| Analytics | `analytics:{user_id}:*` | Daily aggregation job |
|
||||
| Recommendations | `recommendations:{user_id}:*` | New interaction, daily refresh |
|
||||
| Marketplace | `marketplace:*` | New item, rating update |
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
// Invalidate user-specific cache on update
|
||||
func (h *UserHandler) UpdateUser(c *gin.Context) {
|
||||
// ... update logic ...
|
||||
|
||||
// Invalidate cache
|
||||
redisClient.Del(ctx, fmt.Sprintf("trackeep:user:%d:profile", userID))
|
||||
redisClient.Del(ctx, fmt.Sprintf("trackeep:analytics:dashboard:%d:*", userID))
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Scalability Needs Assessment
|
||||
|
||||
### 3.1 Current Architecture Constraints
|
||||
|
||||
**Single-Node Limitations:**
|
||||
- Docker Compose deployment targets single-node self-hosting
|
||||
- In-memory caches limit horizontal scaling
|
||||
- WebSocket hub cannot distribute across nodes
|
||||
- Session storage doesn't persist restarts
|
||||
|
||||
**Growth Projections:**
|
||||
|
||||
| Resource | Current (Single User) | Projected (100 Users) | Projected (1000 Users) |
|
||||
|----------|----------------------|----------------------|----------------------|
|
||||
| Session storage | ~5KB | ~500KB | ~5MB |
|
||||
| Cache data | ~10MB | ~100MB | ~500MB |
|
||||
| Rate limit state | ~1KB | ~100KB | ~1MB |
|
||||
| Real-time subscribers | 1-5 | 50-200 | 200-500 |
|
||||
|
||||
### 3.2 Redis Clustering Requirements
|
||||
|
||||
**Phase 1: Single Redis Instance (Current Scale)**
|
||||
- Suitable for < 100 concurrent users
|
||||
- 1GB RAM allocation sufficient
|
||||
- No clustering complexity
|
||||
|
||||
**Phase 2: Redis Sentinel (High Availability)**
|
||||
- Required for production reliability
|
||||
- 1 master + 2 replicas minimum
|
||||
- Automatic failover capability
|
||||
|
||||
**Phase 3: Redis Cluster (Horizontal Scale)**
|
||||
- Required for > 1000 concurrent users
|
||||
- 6+ nodes (3 masters + 3 replicas)
|
||||
- Data sharding across nodes
|
||||
|
||||
**Recommendation for Trackeep:**
|
||||
Given the self-hosted nature and typical deployment size (small teams), **Redis Sentinel** provides the best balance of high availability without excessive complexity.
|
||||
|
||||
---
|
||||
|
||||
## 4. Persistence and Memory Optimization
|
||||
|
||||
### 4.1 Persistence Configuration
|
||||
|
||||
**Redis Persistence Options:**
|
||||
|
||||
| Option | Configuration | Use Case |
|
||||
|--------|--------------|----------|
|
||||
| RDB (Snapshot) | `save 900 1`, `save 300 10` | Point-in-time recovery, minimal overhead |
|
||||
| AOF (Append-Only) | `appendonly yes`, `appendfsync everysec` | Durability, zero data loss |
|
||||
| Hybrid | Both enabled | Maximum protection |
|
||||
|
||||
**Recommendation for Trackeep:**
|
||||
```conf
|
||||
# redis.conf recommendations
|
||||
save 900 1
|
||||
save 300 10
|
||||
save 60 10000
|
||||
appendonly yes
|
||||
appendfsync everysec
|
||||
auto-aof-rewrite-percentage 100
|
||||
auto-aof-rewrite-min-size 64mb
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- Sessions should survive restarts (use AOF)
|
||||
- Cache can be rebuilt from DB (RDB sufficient)
|
||||
- `everysec` provides good balance of durability/performance
|
||||
|
||||
### 4.2 Memory Optimization Strategies
|
||||
|
||||
**Estimated Memory Usage:**
|
||||
|
||||
| Data Type | Entries | Entry Size | Total |
|
||||
|-----------|---------|------------|-------|
|
||||
| Sessions | 1000 | ~500 bytes | 500 KB |
|
||||
| User caches | 1000 | ~2 KB | 2 MB |
|
||||
| Search caches | 5000 | ~10 KB | 50 MB |
|
||||
| Analytics caches | 1000 | ~5 KB | 5 MB |
|
||||
| Rate limit buckets | 10000 | ~100 bytes | 1 MB |
|
||||
| Real-time pub/sub | 500 | ~200 bytes | 100 KB |
|
||||
| **Total** | | | **~60 MB + overhead** |
|
||||
|
||||
**Memory Optimization Techniques:**
|
||||
|
||||
1. **Compression**
|
||||
```go
|
||||
// Use MessagePack or gzip for large cached data
|
||||
import "github.com/vmihailenco/msgpack/v5"
|
||||
|
||||
func compressCache(data interface{}) ([]byte, error) {
|
||||
return msgpack.Marshal(data)
|
||||
}
|
||||
```
|
||||
|
||||
2. **Key Naming Optimization**
|
||||
```
|
||||
# Short prefixes
|
||||
tk:u:1234:profile (instead of trackeep:user:1234:profile)
|
||||
|
||||
# Hashed identifiers for long IDs
|
||||
tk:s:8f3d2c... (MD5 hash of session data)
|
||||
```
|
||||
|
||||
3. **TTL Strategy**
|
||||
```go
|
||||
const (
|
||||
SessionTTL = 24 * time.Hour
|
||||
UserCacheTTL = 15 * time.Minute
|
||||
SearchCacheTTL = 5 * time.Minute
|
||||
AnalyticsCacheTTL = 1 * time.Hour
|
||||
RateLimitTTL = 1 * time.Hour
|
||||
)
|
||||
```
|
||||
|
||||
### 4.3 Data Eviction Policies
|
||||
|
||||
**Recommended Configuration:**
|
||||
```conf
|
||||
maxmemory 256mb
|
||||
maxmemory-policy allkeys-lru
|
||||
```
|
||||
|
||||
**Policy Selection:**
|
||||
- `allkeys-lru`: Best for cache-heavy workloads (recommended)
|
||||
- `volatile-lru`: If some keys must persist
|
||||
- `noeviction`: Fail writes at memory limit (not recommended)
|
||||
|
||||
**Key Expiration Strategy:**
|
||||
- Sessions: 24h TTL with refresh on activity
|
||||
- Search results: 5m TTL
|
||||
- Analytics: 1h TTL
|
||||
- Rate limits: Window-based TTL
|
||||
|
||||
---
|
||||
|
||||
## 5. Integration Challenges and Solutions
|
||||
|
||||
### 5.1 Existing Technology Stack Integration
|
||||
|
||||
**Go + Gin Integration:**
|
||||
|
||||
```go
|
||||
// config/redis.go
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"github.com/go-redis/redis/v8"
|
||||
)
|
||||
|
||||
var RedisClient *redis.Client
|
||||
|
||||
func InitRedis() {
|
||||
RedisClient = redis.NewClient(&redis.Options{
|
||||
Addr: os.Getenv("REDIS_ADDR"),
|
||||
Password: os.Getenv("REDIS_PASSWORD"),
|
||||
DB: 0,
|
||||
PoolSize: 10,
|
||||
MinIdleConns: 5,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
**Docker Compose Integration:**
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml addition
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
redis_data:
|
||||
```
|
||||
|
||||
### 5.2 Migration Path from In-Memory to Redis
|
||||
|
||||
**Phase 1: Graceful Fallback (Week 1)**
|
||||
```go
|
||||
func GetCache(key string) ([]byte, error) {
|
||||
// Try Redis first
|
||||
if RedisClient != nil {
|
||||
val, err := RedisClient.Get(ctx, key).Bytes()
|
||||
if err == nil {
|
||||
return val, nil
|
||||
}
|
||||
}
|
||||
// Fallback to memory cache
|
||||
return memoryCache.Get(key)
|
||||
}
|
||||
```
|
||||
|
||||
**Phase 2: Feature-by-Feature Migration (Weeks 2-4)**
|
||||
1. Session storage (highest impact)
|
||||
2. Rate limiting (consistency improvement)
|
||||
3. Search caching (performance gain)
|
||||
4. Analytics caching (complex aggregations)
|
||||
|
||||
**Phase 3: Full Redis Adoption (Week 5)**
|
||||
- Remove in-memory cache implementations
|
||||
- Enable Redis Sentinel for HA
|
||||
|
||||
### 5.3 Connection Pooling Configuration
|
||||
|
||||
**Recommended Pool Settings:**
|
||||
```go
|
||||
&redis.Options{
|
||||
PoolSize: 20, // Max connections
|
||||
MinIdleConns: 5, // Always maintained
|
||||
MaxConnAge: time.Hour, // Connection refresh
|
||||
PoolTimeout: 5 * time.Second, // Wait for connection
|
||||
IdleTimeout: 10 * time.Minute, // Close idle connections
|
||||
ReadTimeout: 3 * time.Second,
|
||||
WriteTimeout: 3 * time.Second,
|
||||
}
|
||||
```
|
||||
|
||||
**Connection Monitoring:**
|
||||
```go
|
||||
// Health check endpoint
|
||||
func RedisHealthCheck() map[string]interface{} {
|
||||
info := RedisClient.Info(ctx, "clients").Val()
|
||||
stats := RedisClient.PoolStats()
|
||||
|
||||
return map[string]interface{}{
|
||||
"hits": stats.Hits,
|
||||
"misses": stats.Misses,
|
||||
"timeouts": stats.Timeouts,
|
||||
"total_conns": stats.TotalConns,
|
||||
"idle_conns": stats.IdleConns,
|
||||
"stale_conns": stats.StaleConns,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternative Solutions Comparison
|
||||
|
||||
### 6.1 Redis vs Memcached
|
||||
|
||||
| Feature | Redis | Memcached | Recommendation |
|
||||
|---------|-------|-----------|----------------|
|
||||
| Data structures | Rich (Hash, Set, Sorted Set) | Simple key-value | Redis for complex use cases |
|
||||
| Persistence | RDB + AOF | None | Redis for session durability |
|
||||
| Pub/Sub | Native | Not supported | Redis for real-time features |
|
||||
| Clustering | Built-in | Client-side | Redis easier to manage |
|
||||
| Rate limiting | Lua scripting | Increment only | Redis for complex algorithms |
|
||||
| Memory efficiency | Good | Excellent | Memcached for pure cache |
|
||||
| Transactions | Multi/Lua | CAS only | Redis better consistency |
|
||||
|
||||
**Verdict:** Redis is superior for Trackeep due to need for persistence (sessions), complex data structures (leaderboards), and pub/sub (real-time messaging).
|
||||
|
||||
### 6.2 Redis vs Kafka
|
||||
|
||||
| Use Case | Redis | Kafka | Recommendation |
|
||||
|----------|-------|-------|----------------|
|
||||
| Message queue | Streams (simple) | Purpose-built | Kafka for high throughput |
|
||||
| Pub/Sub | Excellent | Not primary use | Redis for real-time |
|
||||
| Event sourcing | Limited | Designed for it | Kafka for audit trail |
|
||||
| Log aggregation | Not suitable | Perfect fit | Kafka for analytics pipeline |
|
||||
|
||||
**Hybrid Architecture:**
|
||||
- **Redis**: Real-time messaging, caching, sessions, leaderboards
|
||||
- **Kafka** (future): Audit log streaming, analytics events, AI training data
|
||||
|
||||
**Verdict:** Start with Redis for all current use cases. Add Kafka later if event streaming volume exceeds 10k events/second.
|
||||
|
||||
### 6.3 Redis vs PostgreSQL Caching
|
||||
|
||||
| Approach | Implementation | Pros | Cons |
|
||||
|----------|---------------|------|------|
|
||||
| PostgreSQL Materialized Views | Native | No new infrastructure | Stale data, manual refresh |
|
||||
| PostgreSQL UNLOGGED tables | Write-only tables | Persistent | No TTL, manual cleanup |
|
||||
| Redis | External service | TTL, pub/sub, scaling | Additional dependency |
|
||||
|
||||
**Verdict:** Redis provides the flexibility needed for Trackeep's diverse caching requirements.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Best Practices
|
||||
|
||||
### 7.1 Serialization Formats
|
||||
|
||||
**Performance Comparison:**
|
||||
|
||||
| Format | Encoding Speed | Decoding Speed | Size | Recommendation |
|
||||
|--------|---------------|----------------|------|----------------|
|
||||
| JSON | Fast | Fast | Large | Human-readable debugging |
|
||||
| MessagePack | Very Fast | Very Fast | Small | Production default |
|
||||
| Protobuf | Fastest | Fastest | Smallest | Complex schemas |
|
||||
| Gzip+JSON | Slow | Slow | Smallest | Large payloads only |
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
import "github.com/vmihailenco/msgpack/v5"
|
||||
|
||||
func serialize(data interface{}) ([]byte, error) {
|
||||
return msgpack.Marshal(data)
|
||||
}
|
||||
|
||||
func deserialize(data []byte, v interface{}) error {
|
||||
return msgpack.Unmarshal(data, v)
|
||||
}
|
||||
```
|
||||
|
||||
### 7.2 Key Naming Conventions
|
||||
|
||||
**Hierarchical Structure:**
|
||||
```
|
||||
tk:{resource}:{id}:{attribute}:{context}
|
||||
|
||||
Examples:
|
||||
tk:u:1234:profile # User profile
|
||||
tk:u:1234:sessions # Active sessions
|
||||
tk:search:1234:a7f3... # Search cache (hashed query)
|
||||
tk:analytics:1234:dashboard:daily # Analytics dashboard
|
||||
tk:rl:1234:general # Rate limit bucket
|
||||
tk:msg:conv:5678:recent # Recent messages
|
||||
tk:marketplace:trending:daily # Trending items
|
||||
tk:challenge:12:leaderboard # Challenge rankings
|
||||
```
|
||||
|
||||
### 7.3 Error Handling and Fallbacks
|
||||
|
||||
**Circuit Breaker Pattern:**
|
||||
```go
|
||||
type RedisCircuitBreaker struct {
|
||||
failures int
|
||||
lastFailure time.Time
|
||||
state string // closed, open, half-open
|
||||
mutex sync.RWMutex
|
||||
}
|
||||
|
||||
func (cb *RedisCircuitBreaker) Execute(fn func() error) error {
|
||||
if cb.isOpen() {
|
||||
return fmt.Errorf("redis circuit breaker open")
|
||||
}
|
||||
|
||||
err := fn()
|
||||
if err != nil {
|
||||
cb.recordFailure()
|
||||
return err
|
||||
}
|
||||
|
||||
cb.recordSuccess()
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
**Graceful Degradation:**
|
||||
```go
|
||||
func GetWithFallback(key string, fetchFn func() ([]byte, error)) ([]byte, error) {
|
||||
// Try Redis
|
||||
data, err := redisClient.Get(ctx, key).Bytes()
|
||||
if err == nil {
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// Fallback to fetch function
|
||||
data, err = fetchFn()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Cache for next time (async)
|
||||
go func() {
|
||||
redisClient.Set(ctx, key, data, cacheTTL)
|
||||
}()
|
||||
|
||||
return data, nil
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Security Considerations
|
||||
|
||||
### 8.1 Authentication and Authorization
|
||||
|
||||
**Redis Security Configuration:**
|
||||
```conf
|
||||
# redis.conf
|
||||
requirepass ${REDIS_PASSWORD}
|
||||
rename-command FLUSHDB ""
|
||||
rename-command FLUSHALL ""
|
||||
rename-command CONFIG "CONFIG_a1b2c3"
|
||||
```
|
||||
|
||||
**Go Client Authentication:**
|
||||
```go
|
||||
redis.NewClient(&redis.Options{
|
||||
Addr: os.Getenv("REDIS_ADDR"),
|
||||
Password: os.Getenv("REDIS_PASSWORD"),
|
||||
Username: os.Getenv("REDIS_USERNAME"), // Redis 6+ ACL
|
||||
})
|
||||
```
|
||||
|
||||
### 8.2 Encryption Requirements
|
||||
|
||||
| Layer | Encryption | Implementation |
|
||||
|-------|-----------|----------------|
|
||||
| Transit | TLS 1.2+ | `redis://` → `rediss://` |
|
||||
| At-rest | Optional | Volume encryption |
|
||||
| Application | Field-level | For sensitive cache data |
|
||||
|
||||
**TLS Configuration:**
|
||||
```go
|
||||
redis.NewClient(&redis.Options{
|
||||
Addr: "rediss://redis:6379",
|
||||
TLSConfig: &tls.Config{
|
||||
MinVersion: tls.VersionTLS12,
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
**Sensitive Data Handling:**
|
||||
- Never cache: passwords, encryption keys, 2FA secrets
|
||||
- Encrypt before caching: API keys, tokens (if cached)
|
||||
- Session data: Safe to cache (already has session ID)
|
||||
|
||||
### 8.3 Network Security
|
||||
|
||||
**Docker Compose Network Isolation:**
|
||||
```yaml
|
||||
services:
|
||||
redis:
|
||||
networks:
|
||||
- backend-internal
|
||||
# No port mapping - only accessible within network
|
||||
|
||||
backend:
|
||||
networks:
|
||||
- backend-internal
|
||||
- public
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Monitoring and Observability
|
||||
|
||||
### 9.1 Key Metrics to Track
|
||||
|
||||
| Metric | Redis Command | Alert Threshold |
|
||||
|--------|--------------|-----------------|
|
||||
| Memory usage | `INFO memory` | > 80% of maxmemory |
|
||||
| Hit rate | `INFO stats` | < 80% |
|
||||
| Connected clients | `INFO clients` | > 90% of maxclients |
|
||||
| Slow queries | `SLOWLOG GET` | > 10ms |
|
||||
| Replication lag | `INFO replication` | > 1s |
|
||||
| Evicted keys | `INFO stats` | > 100/min |
|
||||
|
||||
### 9.2 Health Check Implementation
|
||||
|
||||
```go
|
||||
func RedisHealthCheck(ctx context.Context) map[string]interface{} {
|
||||
result := map[string]interface{}{
|
||||
"status": "healthy",
|
||||
}
|
||||
|
||||
// Ping test
|
||||
if err := RedisClient.Ping(ctx).Err(); err != nil {
|
||||
result["status"] = "unhealthy"
|
||||
result["error"] = err.Error()
|
||||
return result
|
||||
}
|
||||
|
||||
// Memory info
|
||||
info := RedisClient.Info(ctx, "memory").Val()
|
||||
result["memory_info"] = parseRedisInfo(info)
|
||||
|
||||
// Pool stats
|
||||
stats := RedisClient.PoolStats()
|
||||
result["pool"] = map[string]interface{}{
|
||||
"hits": stats.Hits,
|
||||
"misses": stats.Misses,
|
||||
"timeouts": stats.Timeouts,
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Cost-Benefit Analysis
|
||||
|
||||
### 10.1 Implementation Costs
|
||||
|
||||
| Component | Effort | Risk | Priority |
|
||||
|-----------|--------|------|----------|
|
||||
| Redis infrastructure setup | 4 hours | Low | High |
|
||||
| Session storage migration | 8 hours | Medium | High |
|
||||
| Rate limiting refactor | 6 hours | Low | Medium |
|
||||
| Search caching | 12 hours | Medium | Medium |
|
||||
| Analytics caching | 8 hours | Low | Low |
|
||||
| Testing & validation | 16 hours | Low | High |
|
||||
| **Total** | **54 hours** | | |
|
||||
|
||||
### 10.2 Operational Benefits
|
||||
|
||||
| Metric | Before Redis | After Redis | Improvement |
|
||||
|--------|-------------|-------------|-------------|
|
||||
| Session persistence | None | Full | Critical |
|
||||
| Horizontal scaling | Limited | Full | High |
|
||||
| API response time (P95) | 500ms | 150ms | 70% |
|
||||
| Database load | 100% | 40% | 60% |
|
||||
| Rate limit accuracy | Per-node | Global | High |
|
||||
| Real-time capabilities | Single-node | Multi-node | High |
|
||||
|
||||
---
|
||||
|
||||
## 11. Implementation Roadmap
|
||||
|
||||
### Phase 1: Foundation (Week 1)
|
||||
- [ ] Add Redis service to Docker Compose
|
||||
- [ ] Implement Redis client initialization
|
||||
- [ ] Add health checks and monitoring
|
||||
- [ ] Configure persistence and memory limits
|
||||
|
||||
### Phase 2: Critical Features (Weeks 2-3)
|
||||
- [ ] Migrate session storage to Redis
|
||||
- [ ] Implement distributed rate limiting
|
||||
- [ ] Add connection pooling
|
||||
- [ ] Implement circuit breaker pattern
|
||||
|
||||
### Phase 3: Performance Optimization (Weeks 4-5)
|
||||
- [ ] Implement search result caching
|
||||
- [ ] Add analytics dashboard caching
|
||||
- [ ] Implement cache warming strategy
|
||||
- [ ] Add compression for large payloads
|
||||
|
||||
### Phase 4: Advanced Features (Week 6)
|
||||
- [ ] Real-time leaderboards with Sorted Sets
|
||||
- [ ] Pub/Sub for cross-instance messaging
|
||||
- [ ] Redis Sentinel for high availability
|
||||
- [ ] Performance benchmarking and tuning
|
||||
|
||||
---
|
||||
|
||||
## 12. Conclusion
|
||||
|
||||
**Redis deployment is strongly recommended for Trackeep** based on the following architectural alignment factors:
|
||||
|
||||
1. **Current Pain Points Addressed:**
|
||||
- Session persistence across restarts
|
||||
- Distributed rate limiting for future scaling
|
||||
- Reduced database load for expensive queries
|
||||
- Real-time features support
|
||||
|
||||
2. **Architectural Fit:**
|
||||
- Existing go-redis dependency ready for use
|
||||
- Docker Compose deployment simplifies Redis addition
|
||||
- In-memory implementations provide migration blueprint
|
||||
- Self-hosted nature allows resource allocation control
|
||||
|
||||
3. **Risk Assessment:**
|
||||
- **Low Risk:** Redis is mature, well-documented, and has Go library support
|
||||
- **Medium Risk:** Migration from in-memory to Redis requires testing
|
||||
- **Mitigation:** Graceful fallback implementations ensure no downtime
|
||||
|
||||
4. **ROI:**
|
||||
- 54 hours of implementation effort
|
||||
- 70% improvement in API response times
|
||||
- 60% reduction in database load
|
||||
- Enables horizontal scaling for future growth
|
||||
|
||||
**Recommendation:** Proceed with Redis deployment starting with Phase 1 (Foundation) immediately, followed by critical feature migration in subsequent sprints.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: Environment Variables
|
||||
|
||||
```bash
|
||||
# Redis Configuration
|
||||
REDIS_ADDR=redis:6379
|
||||
REDIS_PASSWORD=secure_password_here
|
||||
REDIS_DB=0
|
||||
REDIS_POOL_SIZE=20
|
||||
REDIS_DIAL_TIMEOUT=5s
|
||||
REDIS_READ_TIMEOUT=3s
|
||||
REDIS_WRITE_TIMEOUT=3s
|
||||
|
||||
# Feature Flags
|
||||
REDIS_SESSIONS_ENABLED=true
|
||||
REDIS_CACHE_ENABLED=true
|
||||
REDIS_RATELIMIT_ENABLED=true
|
||||
REDIS_PUBSUB_ENABLED=true
|
||||
```
|
||||
|
||||
## Appendix B: Docker Compose Configuration
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
- ./redis.conf:/usr/local/etc/redis/redis.conf:ro
|
||||
command: redis-server /usr/local/etc/redis/redis.conf
|
||||
networks:
|
||||
- trackeep-network
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
trackeep-backend:
|
||||
environment:
|
||||
- REDIS_ADDR=redis:6379
|
||||
- REDIS_PASSWORD=${REDIS_PASSWORD}
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
volumes:
|
||||
redis_data:
|
||||
|
||||
networks:
|
||||
trackeep-network:
|
||||
driver: bridge
|
||||
```
|
||||
Reference in New Issue
Block a user