mirror of
https://github.com/Dvorinka/Trackeep.git
synced 2026-06-03 20:12:58 +00:00
feat: major feature updates and cleanup
- Add Redis architecture implementation - Update browser extension functionality - Clean up deprecated files and documentation - Enhance backend handlers for auth, messages, search - Add new configuration options and settings - Update Docker and deployment configurations
This commit is contained in:
Tomas Dvorak
+2393
-2
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,894 @@
|
||||
# Redis Architecture Analysis for Trackeep
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Trackeep** is a self-hosted productivity and knowledge management platform built with Go (Gin framework), PostgreSQL, and React. The application already includes the `go-redis/redis/v8` dependency but currently operates with in-memory fallbacks for caching, sessions, and rate limiting. This analysis evaluates Redis deployment across multiple dimensions to determine architectural alignment and implementation strategy.
|
||||
|
||||
**Current Infrastructure:**
|
||||
- **Backend:** Go 1.24 with Gin web framework
|
||||
- **Database:** PostgreSQL 15 (primary data store)
|
||||
- **Frontend:** React + TypeScript + Vite
|
||||
- **Deployment:** Docker Compose (single-node, self-hosted)
|
||||
- **Current Caching:** In-memory maps with mutex locks
|
||||
- **Current Sessions:** In-memory map storage
|
||||
- **Current Rate Limiting:** Per-instance in-memory tracking
|
||||
|
||||
---
|
||||
|
||||
## 1. Use Case Analysis
|
||||
|
||||
### 1.1 Caching Frequently Accessed Database Queries
|
||||
|
||||
**Current State:**
|
||||
The application uses [`MemoryCache`](backend/middleware/memory_cache.go:21) with `sync.RWMutex` for thread-safe in-memory caching. Cache entries expire via a cleanup goroutine running every minute.
|
||||
|
||||
**Redis Opportunity:**
|
||||
|
||||
| Query Pattern | Current Implementation | Redis Benefit |
|
||||
|---------------|---------------------|---------------|
|
||||
| User profiles | Direct DB query on each request | Cache for 5-15 min, reduces user table queries |
|
||||
| Search results | Computed on every search | Cache complex searches for 5-10 min |
|
||||
| Analytics dashboards | Aggregated from multiple tables | Cache pre-computed aggregations for 1 hour |
|
||||
| Learning paths/courses | Filtered queries with joins | Cache popular paths for 30 min |
|
||||
| YouTube channel data | Database cache + in-memory fallback | Unified Redis cache with TTL |
|
||||
| Marketplace items | Sorted/filtered queries | Cache trending/top-rated items |
|
||||
|
||||
**Specific High-Value Caches:**
|
||||
|
||||
1. **Enhanced Search Cache** ([`search_enhanced.go`](backend/handlers/search_enhanced.go:73))
|
||||
- Complex multi-table searches across bookmarks, tasks, notes, files
|
||||
- Redis can cache results with content-type aggregation
|
||||
- Suggested TTL: 5 minutes for dynamic content
|
||||
|
||||
2. **Analytics Dashboard Cache** ([`analytics.go`](backend/handlers/analytics.go:24))
|
||||
- Expensive aggregations across analytics, learning, GitHub, habit tables
|
||||
- Pre-computed dashboard data can be cached for 15-30 minutes
|
||||
- User-specific caching with tags for invalidation
|
||||
|
||||
3. **AI Recommendations Cache** ([`ai_recommendations.go`](backend/handlers/ai_recommendations.go:49))
|
||||
- ML-generated recommendations are expensive to compute
|
||||
- Cache recommendation lists per user for 1 hour
|
||||
- Cache recommendation statistics for 30 minutes
|
||||
|
||||
**Implementation Approach:**
|
||||
```go
|
||||
// Cache key structure
|
||||
trackeep:{resource}:{user_id}:{query_hash}
|
||||
trackeep:search:{user_id}:{md5(query+filters)}
|
||||
trackeep:analytics:dashboard:{user_id}:{date_range}
|
||||
trackeep:recommendations:{user_id}:{type}
|
||||
```
|
||||
|
||||
### 1.2 Distributed Session State Management
|
||||
|
||||
**Current State:**
|
||||
The [`RedisSessionStore`](backend/middleware/session.go:36) struct exists but uses `map[string]*SessionData` as a fallback in-memory store. Sessions are lost on server restart and don't work across multiple backend instances.
|
||||
|
||||
**Session Data Structure:**
|
||||
```go
|
||||
type SessionData struct {
|
||||
UserID uint `json:"user_id"`
|
||||
Email string `json:"email"`
|
||||
Username string `json:"username"`
|
||||
Role string `json:"role"`
|
||||
SessionID string `json:"session_id"`
|
||||
IPAddress string `json:"ip_address"`
|
||||
UserAgent string `json:"user_agent"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastActive time.Time `json:"last_active"`
|
||||
}
|
||||
```
|
||||
|
||||
**Redis Implementation:**
|
||||
- Use Redis Hash or JSON data type for session storage
|
||||
- TTL: 24 hours (matching current cleanup logic)
|
||||
- Enable session persistence across deployments
|
||||
- Support horizontal scaling of backend instances
|
||||
- Session invalidation on logout/password change
|
||||
|
||||
**Key Pattern:**
|
||||
```
|
||||
trackeep:session:{session_id} -> SessionData (JSON)
|
||||
trackeep:user:sessions:{user_id} -> Set of active session IDs
|
||||
```
|
||||
|
||||
### 1.3 Real-Time Leaderboards and Rate Tracking
|
||||
|
||||
**Current Opportunities:**
|
||||
|
||||
1. **Community Challenges Leaderboard** ([`community.go`](backend/handlers/community.go:1))
|
||||
- Track challenge participants and completion rates
|
||||
- Real-time leaderboard updates
|
||||
- Redis Sorted Sets (`ZADD`, `ZREVRANGE`) ideal for ranking
|
||||
|
||||
2. **Marketplace Item Rankings** ([`marketplace.go`](backend/handlers/marketplace.go:1))
|
||||
- Sort by downloads, rating, views
|
||||
- Trending items calculation
|
||||
- Redis can maintain real-time counters
|
||||
|
||||
3. **User Analytics Streaks** ([`analytics.go`](backend/handlers/analytics.go:786))
|
||||
- Learning streaks tracking
|
||||
- Daily habit completion counts
|
||||
- Redis counters with daily windows
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
// Challenge leaderboard
|
||||
trackeep:challenge:{id}:leaderboard -> Sorted Set (score: completion_time, member: user_id)
|
||||
|
||||
// Marketplace trending
|
||||
trackeep:marketplace:trending -> Sorted Set (score: view_count_24h, member: item_id)
|
||||
|
||||
// User learning streaks
|
||||
trackeep:user:{id}:learning_streak -> Hash (current_streak, last_date, max_streak)
|
||||
```
|
||||
|
||||
### 1.4 Rate Limiting
|
||||
|
||||
**Current State:**
|
||||
The [`RateLimiter`](backend/middleware/rate_limiter.go:13) uses in-memory `map[string]*ClientInfo` with per-IP tracking. This doesn't work across multiple instances and is vulnerable to restart clearing.
|
||||
|
||||
**Redis-Based Rate Limiting:**
|
||||
|
||||
| Rate Limit Type | Window | Current Limit | Redis Strategy |
|
||||
|-----------------|--------|---------------|----------------|
|
||||
| General API | 1 minute | 100 requests | Sliding window with `ZADD` |
|
||||
| Search | 1 minute | 100 requests | Fixed window with `INCR` + `EXPIRE` |
|
||||
| AI Chat | 1 minute | 20 requests | Token bucket algorithm |
|
||||
| Login attempts | 5 minutes | 5 attempts | Count with `INCR` + longer TTL |
|
||||
| File uploads | 10 minutes | 10 uploads | Sliding window per user |
|
||||
|
||||
**Token Bucket Implementation:**
|
||||
```go
|
||||
// Redis Lua script for atomic token bucket
|
||||
local key = KEYS[1]
|
||||
local capacity = tonumber(ARGV[1])
|
||||
local refill_rate = tonumber(ARGV[2])
|
||||
local now = tonumber(ARGV[3])
|
||||
|
||||
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
|
||||
local tokens = tonumber(bucket[1]) or capacity
|
||||
local last_refill = tonumber(bucket[2]) or now
|
||||
|
||||
local delta = math.min(capacity, tokens + (now - last_refill) * refill_rate)
|
||||
|
||||
if delta >= 1 then
|
||||
redis.call('HMSET', key, 'tokens', delta - 1, 'last_refill', now)
|
||||
redis.call('EXPIRE', key, 3600)
|
||||
return 1
|
||||
else
|
||||
redis.call('HMSET', key, 'tokens', delta, 'last_refill', now)
|
||||
redis.call('EXPIRE', key, 3600)
|
||||
return 0
|
||||
end
|
||||
```
|
||||
|
||||
### 1.5 Publish-Subscribe Messaging Patterns
|
||||
|
||||
**Current State:**
|
||||
Real-time messaging uses WebSocket hub [`MessagesHub`](backend/services/messages_realtime.go:28) with in-memory `conversationClients` map. This is single-node only.
|
||||
|
||||
**Redis Pub/Sub for Multi-Node:**
|
||||
|
||||
1. **Cross-Instance Message Broadcasting**
|
||||
- When horizontal scaling is needed, Redis Pub/Sub connects multiple backend instances
|
||||
- Pattern: `trackeep:messages:{conversation_id}`
|
||||
|
||||
2. **Notification System**
|
||||
- Real-time notifications for new followers, messages, mentions
|
||||
- Pattern: `trackeep:notifications:{user_id}`
|
||||
|
||||
3. **System Events**
|
||||
- Cache invalidation broadcasts
|
||||
- Configuration updates
|
||||
- Analytics aggregation triggers
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
// Subscribe to conversation messages
|
||||
pubsub := redisClient.Subscribe(ctx, "trackeep:messages:123")
|
||||
|
||||
// Publish message to all nodes
|
||||
redisClient.Publish(ctx, "trackeep:messages:123", messageJSON)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Access Patterns and Latency Requirements
|
||||
|
||||
### 2.1 Current Database Access Patterns
|
||||
|
||||
Based on code analysis, the application exhibits these access patterns:
|
||||
|
||||
| Pattern | Frequency | Tables | Latency Sensitivity |
|
||||
|---------|-----------|--------|---------------------|
|
||||
| User authentication | High | users | Very High (< 100ms) |
|
||||
| Search queries | Medium-High | bookmarks, tasks, notes, files | High (< 500ms) |
|
||||
| Analytics aggregation | Medium | analytics, learning_analytics | Medium (< 2s) |
|
||||
| Message retrieval | High | messages, conversations | High (< 200ms) |
|
||||
| AI recommendations | Low-Medium | ai_recommendations | Low (< 5s acceptable) |
|
||||
| Marketplace browsing | Medium | marketplace_items | Medium (< 1s) |
|
||||
| Audit logging | High (write) | audit_logs | Low (async) |
|
||||
|
||||
### 2.2 Latency Requirements Analysis
|
||||
|
||||
**Critical Paths for Redis Caching:**
|
||||
|
||||
1. **Authentication Flow** (Target: < 100ms)
|
||||
- Current: DB query for user + session lookup
|
||||
- With Redis: Session cache + user profile cache
|
||||
- Expected improvement: 60-80% latency reduction
|
||||
|
||||
2. **Dashboard Load** (Target: < 500ms)
|
||||
- Current: Multiple aggregation queries
|
||||
- With Redis: Pre-computed analytics cache
|
||||
- Expected improvement: 70-90% latency reduction
|
||||
|
||||
3. **Search Results** (Target: < 300ms)
|
||||
- Current: Full-text search across 4+ tables
|
||||
- With Redis: Cached results for common queries
|
||||
- Expected improvement: 50-80% latency reduction
|
||||
|
||||
### 2.3 Cache Invalidation Strategy
|
||||
|
||||
**Event-Based Invalidation:**
|
||||
|
||||
| Data Type | Cache Keys | Invalidation Trigger |
|
||||
|-----------|------------|---------------------|
|
||||
| User profile | `user:{id}:profile` | User update, password change |
|
||||
| Search results | `search:{user_id}:*` | Any content creation/update |
|
||||
| Analytics | `analytics:{user_id}:*` | Daily aggregation job |
|
||||
| Recommendations | `recommendations:{user_id}:*` | New interaction, daily refresh |
|
||||
| Marketplace | `marketplace:*` | New item, rating update |
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
// Invalidate user-specific cache on update
|
||||
func (h *UserHandler) UpdateUser(c *gin.Context) {
|
||||
// ... update logic ...
|
||||
|
||||
// Invalidate cache
|
||||
redisClient.Del(ctx, fmt.Sprintf("trackeep:user:%d:profile", userID))
|
||||
redisClient.Del(ctx, fmt.Sprintf("trackeep:analytics:dashboard:%d:*", userID))
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Scalability Needs Assessment
|
||||
|
||||
### 3.1 Current Architecture Constraints
|
||||
|
||||
**Single-Node Limitations:**
|
||||
- Docker Compose deployment targets single-node self-hosting
|
||||
- In-memory caches limit horizontal scaling
|
||||
- WebSocket hub cannot distribute across nodes
|
||||
- Session storage doesn't persist restarts
|
||||
|
||||
**Growth Projections:**
|
||||
|
||||
| Resource | Current (Single User) | Projected (100 Users) | Projected (1000 Users) |
|
||||
|----------|----------------------|----------------------|----------------------|
|
||||
| Session storage | ~5KB | ~500KB | ~5MB |
|
||||
| Cache data | ~10MB | ~100MB | ~500MB |
|
||||
| Rate limit state | ~1KB | ~100KB | ~1MB |
|
||||
| Real-time subscribers | 1-5 | 50-200 | 200-500 |
|
||||
|
||||
### 3.2 Redis Clustering Requirements
|
||||
|
||||
**Phase 1: Single Redis Instance (Current Scale)**
|
||||
- Suitable for < 100 concurrent users
|
||||
- 1GB RAM allocation sufficient
|
||||
- No clustering complexity
|
||||
|
||||
**Phase 2: Redis Sentinel (High Availability)**
|
||||
- Required for production reliability
|
||||
- 1 master + 2 replicas minimum
|
||||
- Automatic failover capability
|
||||
|
||||
**Phase 3: Redis Cluster (Horizontal Scale)**
|
||||
- Required for > 1000 concurrent users
|
||||
- 6+ nodes (3 masters + 3 replicas)
|
||||
- Data sharding across nodes
|
||||
|
||||
**Recommendation for Trackeep:**
|
||||
Given the self-hosted nature and typical deployment size (small teams), **Redis Sentinel** provides the best balance of high availability without excessive complexity.
|
||||
|
||||
---
|
||||
|
||||
## 4. Persistence and Memory Optimization
|
||||
|
||||
### 4.1 Persistence Configuration
|
||||
|
||||
**Redis Persistence Options:**
|
||||
|
||||
| Option | Configuration | Use Case |
|
||||
|--------|--------------|----------|
|
||||
| RDB (Snapshot) | `save 900 1`, `save 300 10` | Point-in-time recovery, minimal overhead |
|
||||
| AOF (Append-Only) | `appendonly yes`, `appendfsync everysec` | Durability, zero data loss |
|
||||
| Hybrid | Both enabled | Maximum protection |
|
||||
|
||||
**Recommendation for Trackeep:**
|
||||
```conf
|
||||
# redis.conf recommendations
|
||||
save 900 1
|
||||
save 300 10
|
||||
save 60 10000
|
||||
appendonly yes
|
||||
appendfsync everysec
|
||||
auto-aof-rewrite-percentage 100
|
||||
auto-aof-rewrite-min-size 64mb
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- Sessions should survive restarts (use AOF)
|
||||
- Cache can be rebuilt from DB (RDB sufficient)
|
||||
- `everysec` provides good balance of durability/performance
|
||||
|
||||
### 4.2 Memory Optimization Strategies
|
||||
|
||||
**Estimated Memory Usage:**
|
||||
|
||||
| Data Type | Entries | Entry Size | Total |
|
||||
|-----------|---------|------------|-------|
|
||||
| Sessions | 1000 | ~500 bytes | 500 KB |
|
||||
| User caches | 1000 | ~2 KB | 2 MB |
|
||||
| Search caches | 5000 | ~10 KB | 50 MB |
|
||||
| Analytics caches | 1000 | ~5 KB | 5 MB |
|
||||
| Rate limit buckets | 10000 | ~100 bytes | 1 MB |
|
||||
| Real-time pub/sub | 500 | ~200 bytes | 100 KB |
|
||||
| **Total** | | | **~60 MB + overhead** |
|
||||
|
||||
**Memory Optimization Techniques:**
|
||||
|
||||
1. **Compression**
|
||||
```go
|
||||
// Use MessagePack or gzip for large cached data
|
||||
import "github.com/vmihailenco/msgpack/v5"
|
||||
|
||||
func compressCache(data interface{}) ([]byte, error) {
|
||||
return msgpack.Marshal(data)
|
||||
}
|
||||
```
|
||||
|
||||
2. **Key Naming Optimization**
|
||||
```
|
||||
# Short prefixes
|
||||
tk:u:1234:profile (instead of trackeep:user:1234:profile)
|
||||
|
||||
# Hashed identifiers for long IDs
|
||||
tk:s:8f3d2c... (MD5 hash of session data)
|
||||
```
|
||||
|
||||
3. **TTL Strategy**
|
||||
```go
|
||||
const (
|
||||
SessionTTL = 24 * time.Hour
|
||||
UserCacheTTL = 15 * time.Minute
|
||||
SearchCacheTTL = 5 * time.Minute
|
||||
AnalyticsCacheTTL = 1 * time.Hour
|
||||
RateLimitTTL = 1 * time.Hour
|
||||
)
|
||||
```
|
||||
|
||||
### 4.3 Data Eviction Policies
|
||||
|
||||
**Recommended Configuration:**
|
||||
```conf
|
||||
maxmemory 256mb
|
||||
maxmemory-policy allkeys-lru
|
||||
```
|
||||
|
||||
**Policy Selection:**
|
||||
- `allkeys-lru`: Best for cache-heavy workloads (recommended)
|
||||
- `volatile-lru`: If some keys must persist
|
||||
- `noeviction`: Fail writes at memory limit (not recommended)
|
||||
|
||||
**Key Expiration Strategy:**
|
||||
- Sessions: 24h TTL with refresh on activity
|
||||
- Search results: 5m TTL
|
||||
- Analytics: 1h TTL
|
||||
- Rate limits: Window-based TTL
|
||||
|
||||
---
|
||||
|
||||
## 5. Integration Challenges and Solutions
|
||||
|
||||
### 5.1 Existing Technology Stack Integration
|
||||
|
||||
**Go + Gin Integration:**
|
||||
|
||||
```go
|
||||
// config/redis.go
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"github.com/go-redis/redis/v8"
|
||||
)
|
||||
|
||||
var RedisClient *redis.Client
|
||||
|
||||
func InitRedis() {
|
||||
RedisClient = redis.NewClient(&redis.Options{
|
||||
Addr: os.Getenv("REDIS_ADDR"),
|
||||
Password: os.Getenv("REDIS_PASSWORD"),
|
||||
DB: 0,
|
||||
PoolSize: 10,
|
||||
MinIdleConns: 5,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
**Docker Compose Integration:**
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml addition
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
redis_data:
|
||||
```
|
||||
|
||||
### 5.2 Migration Path from In-Memory to Redis
|
||||
|
||||
**Phase 1: Graceful Fallback (Week 1)**
|
||||
```go
|
||||
func GetCache(key string) ([]byte, error) {
|
||||
// Try Redis first
|
||||
if RedisClient != nil {
|
||||
val, err := RedisClient.Get(ctx, key).Bytes()
|
||||
if err == nil {
|
||||
return val, nil
|
||||
}
|
||||
}
|
||||
// Fallback to memory cache
|
||||
return memoryCache.Get(key)
|
||||
}
|
||||
```
|
||||
|
||||
**Phase 2: Feature-by-Feature Migration (Weeks 2-4)**
|
||||
1. Session storage (highest impact)
|
||||
2. Rate limiting (consistency improvement)
|
||||
3. Search caching (performance gain)
|
||||
4. Analytics caching (complex aggregations)
|
||||
|
||||
**Phase 3: Full Redis Adoption (Week 5)**
|
||||
- Remove in-memory cache implementations
|
||||
- Enable Redis Sentinel for HA
|
||||
|
||||
### 5.3 Connection Pooling Configuration
|
||||
|
||||
**Recommended Pool Settings:**
|
||||
```go
|
||||
&redis.Options{
|
||||
PoolSize: 20, // Max connections
|
||||
MinIdleConns: 5, // Always maintained
|
||||
MaxConnAge: time.Hour, // Connection refresh
|
||||
PoolTimeout: 5 * time.Second, // Wait for connection
|
||||
IdleTimeout: 10 * time.Minute, // Close idle connections
|
||||
ReadTimeout: 3 * time.Second,
|
||||
WriteTimeout: 3 * time.Second,
|
||||
}
|
||||
```
|
||||
|
||||
**Connection Monitoring:**
|
||||
```go
|
||||
// Health check endpoint
|
||||
func RedisHealthCheck() map[string]interface{} {
|
||||
info := RedisClient.Info(ctx, "clients").Val()
|
||||
stats := RedisClient.PoolStats()
|
||||
|
||||
return map[string]interface{}{
|
||||
"hits": stats.Hits,
|
||||
"misses": stats.Misses,
|
||||
"timeouts": stats.Timeouts,
|
||||
"total_conns": stats.TotalConns,
|
||||
"idle_conns": stats.IdleConns,
|
||||
"stale_conns": stats.StaleConns,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternative Solutions Comparison
|
||||
|
||||
### 6.1 Redis vs Memcached
|
||||
|
||||
| Feature | Redis | Memcached | Recommendation |
|
||||
|---------|-------|-----------|----------------|
|
||||
| Data structures | Rich (Hash, Set, Sorted Set) | Simple key-value | Redis for complex use cases |
|
||||
| Persistence | RDB + AOF | None | Redis for session durability |
|
||||
| Pub/Sub | Native | Not supported | Redis for real-time features |
|
||||
| Clustering | Built-in | Client-side | Redis easier to manage |
|
||||
| Rate limiting | Lua scripting | Increment only | Redis for complex algorithms |
|
||||
| Memory efficiency | Good | Excellent | Memcached for pure cache |
|
||||
| Transactions | Multi/Lua | CAS only | Redis better consistency |
|
||||
|
||||
**Verdict:** Redis is superior for Trackeep due to need for persistence (sessions), complex data structures (leaderboards), and pub/sub (real-time messaging).
|
||||
|
||||
### 6.2 Redis vs Kafka
|
||||
|
||||
| Use Case | Redis | Kafka | Recommendation |
|
||||
|----------|-------|-------|----------------|
|
||||
| Message queue | Streams (simple) | Purpose-built | Kafka for high throughput |
|
||||
| Pub/Sub | Excellent | Not primary use | Redis for real-time |
|
||||
| Event sourcing | Limited | Designed for it | Kafka for audit trail |
|
||||
| Log aggregation | Not suitable | Perfect fit | Kafka for analytics pipeline |
|
||||
|
||||
**Hybrid Architecture:**
|
||||
- **Redis**: Real-time messaging, caching, sessions, leaderboards
|
||||
- **Kafka** (future): Audit log streaming, analytics events, AI training data
|
||||
|
||||
**Verdict:** Start with Redis for all current use cases. Add Kafka later if event streaming volume exceeds 10k events/second.
|
||||
|
||||
### 6.3 Redis vs PostgreSQL Caching
|
||||
|
||||
| Approach | Implementation | Pros | Cons |
|
||||
|----------|---------------|------|------|
|
||||
| PostgreSQL Materialized Views | Native | No new infrastructure | Stale data, manual refresh |
|
||||
| PostgreSQL UNLOGGED tables | Write-only tables | Persistent | No TTL, manual cleanup |
|
||||
| Redis | External service | TTL, pub/sub, scaling | Additional dependency |
|
||||
|
||||
**Verdict:** Redis provides the flexibility needed for Trackeep's diverse caching requirements.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Best Practices
|
||||
|
||||
### 7.1 Serialization Formats
|
||||
|
||||
**Performance Comparison:**
|
||||
|
||||
| Format | Encoding Speed | Decoding Speed | Size | Recommendation |
|
||||
|--------|---------------|----------------|------|----------------|
|
||||
| JSON | Fast | Fast | Large | Human-readable debugging |
|
||||
| MessagePack | Very Fast | Very Fast | Small | Production default |
|
||||
| Protobuf | Fastest | Fastest | Smallest | Complex schemas |
|
||||
| Gzip+JSON | Slow | Slow | Smallest | Large payloads only |
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
import "github.com/vmihailenco/msgpack/v5"
|
||||
|
||||
func serialize(data interface{}) ([]byte, error) {
|
||||
return msgpack.Marshal(data)
|
||||
}
|
||||
|
||||
func deserialize(data []byte, v interface{}) error {
|
||||
return msgpack.Unmarshal(data, v)
|
||||
}
|
||||
```
|
||||
|
||||
### 7.2 Key Naming Conventions
|
||||
|
||||
**Hierarchical Structure:**
|
||||
```
|
||||
tk:{resource}:{id}:{attribute}:{context}
|
||||
|
||||
Examples:
|
||||
tk:u:1234:profile # User profile
|
||||
tk:u:1234:sessions # Active sessions
|
||||
tk:search:1234:a7f3... # Search cache (hashed query)
|
||||
tk:analytics:1234:dashboard:daily # Analytics dashboard
|
||||
tk:rl:1234:general # Rate limit bucket
|
||||
tk:msg:conv:5678:recent # Recent messages
|
||||
tk:marketplace:trending:daily # Trending items
|
||||
tk:challenge:12:leaderboard # Challenge rankings
|
||||
```
|
||||
|
||||
### 7.3 Error Handling and Fallbacks
|
||||
|
||||
**Circuit Breaker Pattern:**
|
||||
```go
|
||||
type RedisCircuitBreaker struct {
|
||||
failures int
|
||||
lastFailure time.Time
|
||||
state string // closed, open, half-open
|
||||
mutex sync.RWMutex
|
||||
}
|
||||
|
||||
func (cb *RedisCircuitBreaker) Execute(fn func() error) error {
|
||||
if cb.isOpen() {
|
||||
return fmt.Errorf("redis circuit breaker open")
|
||||
}
|
||||
|
||||
err := fn()
|
||||
if err != nil {
|
||||
cb.recordFailure()
|
||||
return err
|
||||
}
|
||||
|
||||
cb.recordSuccess()
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
**Graceful Degradation:**
|
||||
```go
|
||||
func GetWithFallback(key string, fetchFn func() ([]byte, error)) ([]byte, error) {
|
||||
// Try Redis
|
||||
data, err := redisClient.Get(ctx, key).Bytes()
|
||||
if err == nil {
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// Fallback to fetch function
|
||||
data, err = fetchFn()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Cache for next time (async)
|
||||
go func() {
|
||||
redisClient.Set(ctx, key, data, cacheTTL)
|
||||
}()
|
||||
|
||||
return data, nil
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Security Considerations
|
||||
|
||||
### 8.1 Authentication and Authorization
|
||||
|
||||
**Redis Security Configuration:**
|
||||
```conf
|
||||
# redis.conf
|
||||
requirepass ${REDIS_PASSWORD}
|
||||
rename-command FLUSHDB ""
|
||||
rename-command FLUSHALL ""
|
||||
rename-command CONFIG "CONFIG_a1b2c3"
|
||||
```
|
||||
|
||||
**Go Client Authentication:**
|
||||
```go
|
||||
redis.NewClient(&redis.Options{
|
||||
Addr: os.Getenv("REDIS_ADDR"),
|
||||
Password: os.Getenv("REDIS_PASSWORD"),
|
||||
Username: os.Getenv("REDIS_USERNAME"), // Redis 6+ ACL
|
||||
})
|
||||
```
|
||||
|
||||
### 8.2 Encryption Requirements
|
||||
|
||||
| Layer | Encryption | Implementation |
|
||||
|-------|-----------|----------------|
|
||||
| Transit | TLS 1.2+ | `redis://` → `rediss://` |
|
||||
| At-rest | Optional | Volume encryption |
|
||||
| Application | Field-level | For sensitive cache data |
|
||||
|
||||
**TLS Configuration:**
|
||||
```go
|
||||
redis.NewClient(&redis.Options{
|
||||
Addr: "rediss://redis:6379",
|
||||
TLSConfig: &tls.Config{
|
||||
MinVersion: tls.VersionTLS12,
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
**Sensitive Data Handling:**
|
||||
- Never cache: passwords, encryption keys, 2FA secrets
|
||||
- Encrypt before caching: API keys, tokens (if cached)
|
||||
- Session data: Safe to cache (already has session ID)
|
||||
|
||||
### 8.3 Network Security
|
||||
|
||||
**Docker Compose Network Isolation:**
|
||||
```yaml
|
||||
services:
|
||||
redis:
|
||||
networks:
|
||||
- backend-internal
|
||||
# No port mapping - only accessible within network
|
||||
|
||||
backend:
|
||||
networks:
|
||||
- backend-internal
|
||||
- public
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Monitoring and Observability
|
||||
|
||||
### 9.1 Key Metrics to Track
|
||||
|
||||
| Metric | Redis Command | Alert Threshold |
|
||||
|--------|--------------|-----------------|
|
||||
| Memory usage | `INFO memory` | > 80% of maxmemory |
|
||||
| Hit rate | `INFO stats` | < 80% |
|
||||
| Connected clients | `INFO clients` | > 90% of maxclients |
|
||||
| Slow queries | `SLOWLOG GET` | > 10ms |
|
||||
| Replication lag | `INFO replication` | > 1s |
|
||||
| Evicted keys | `INFO stats` | > 100/min |
|
||||
|
||||
### 9.2 Health Check Implementation
|
||||
|
||||
```go
|
||||
func RedisHealthCheck(ctx context.Context) map[string]interface{} {
|
||||
result := map[string]interface{}{
|
||||
"status": "healthy",
|
||||
}
|
||||
|
||||
// Ping test
|
||||
if err := RedisClient.Ping(ctx).Err(); err != nil {
|
||||
result["status"] = "unhealthy"
|
||||
result["error"] = err.Error()
|
||||
return result
|
||||
}
|
||||
|
||||
// Memory info
|
||||
info := RedisClient.Info(ctx, "memory").Val()
|
||||
result["memory_info"] = parseRedisInfo(info)
|
||||
|
||||
// Pool stats
|
||||
stats := RedisClient.PoolStats()
|
||||
result["pool"] = map[string]interface{}{
|
||||
"hits": stats.Hits,
|
||||
"misses": stats.Misses,
|
||||
"timeouts": stats.Timeouts,
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Cost-Benefit Analysis
|
||||
|
||||
### 10.1 Implementation Costs
|
||||
|
||||
| Component | Effort | Risk | Priority |
|
||||
|-----------|--------|------|----------|
|
||||
| Redis infrastructure setup | 4 hours | Low | High |
|
||||
| Session storage migration | 8 hours | Medium | High |
|
||||
| Rate limiting refactor | 6 hours | Low | Medium |
|
||||
| Search caching | 12 hours | Medium | Medium |
|
||||
| Analytics caching | 8 hours | Low | Low |
|
||||
| Testing & validation | 16 hours | Low | High |
|
||||
| **Total** | **54 hours** | | |
|
||||
|
||||
### 10.2 Operational Benefits
|
||||
|
||||
| Metric | Before Redis | After Redis | Improvement |
|
||||
|--------|-------------|-------------|-------------|
|
||||
| Session persistence | None | Full | Critical |
|
||||
| Horizontal scaling | Limited | Full | High |
|
||||
| API response time (P95) | 500ms | 150ms | 70% |
|
||||
| Database load | 100% | 40% | 60% |
|
||||
| Rate limit accuracy | Per-node | Global | High |
|
||||
| Real-time capabilities | Single-node | Multi-node | High |
|
||||
|
||||
---
|
||||
|
||||
## 11. Implementation Roadmap
|
||||
|
||||
### Phase 1: Foundation (Week 1)
|
||||
- [ ] Add Redis service to Docker Compose
|
||||
- [ ] Implement Redis client initialization
|
||||
- [ ] Add health checks and monitoring
|
||||
- [ ] Configure persistence and memory limits
|
||||
|
||||
### Phase 2: Critical Features (Weeks 2-3)
|
||||
- [ ] Migrate session storage to Redis
|
||||
- [ ] Implement distributed rate limiting
|
||||
- [ ] Add connection pooling
|
||||
- [ ] Implement circuit breaker pattern
|
||||
|
||||
### Phase 3: Performance Optimization (Weeks 4-5)
|
||||
- [ ] Implement search result caching
|
||||
- [ ] Add analytics dashboard caching
|
||||
- [ ] Implement cache warming strategy
|
||||
- [ ] Add compression for large payloads
|
||||
|
||||
### Phase 4: Advanced Features (Week 6)
|
||||
- [ ] Real-time leaderboards with Sorted Sets
|
||||
- [ ] Pub/Sub for cross-instance messaging
|
||||
- [ ] Redis Sentinel for high availability
|
||||
- [ ] Performance benchmarking and tuning
|
||||
|
||||
---
|
||||
|
||||
## 12. Conclusion
|
||||
|
||||
**Redis deployment is strongly recommended for Trackeep** based on the following architectural alignment factors:
|
||||
|
||||
1. **Current Pain Points Addressed:**
|
||||
- Session persistence across restarts
|
||||
- Distributed rate limiting for future scaling
|
||||
- Reduced database load for expensive queries
|
||||
- Real-time features support
|
||||
|
||||
2. **Architectural Fit:**
|
||||
- Existing go-redis dependency ready for use
|
||||
- Docker Compose deployment simplifies Redis addition
|
||||
- In-memory implementations provide migration blueprint
|
||||
- Self-hosted nature allows resource allocation control
|
||||
|
||||
3. **Risk Assessment:**
|
||||
- **Low Risk:** Redis is mature, well-documented, and has Go library support
|
||||
- **Medium Risk:** Migration from in-memory to Redis requires testing
|
||||
- **Mitigation:** Graceful fallback implementations ensure no downtime
|
||||
|
||||
4. **ROI:**
|
||||
- 54 hours of implementation effort
|
||||
- 70% improvement in API response times
|
||||
- 60% reduction in database load
|
||||
- Enables horizontal scaling for future growth
|
||||
|
||||
**Recommendation:** Proceed with Redis deployment starting with Phase 1 (Foundation) immediately, followed by critical feature migration in subsequent sprints.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: Environment Variables
|
||||
|
||||
```bash
|
||||
# Redis Configuration
|
||||
REDIS_ADDR=redis:6379
|
||||
REDIS_PASSWORD=secure_password_here
|
||||
REDIS_DB=0
|
||||
REDIS_POOL_SIZE=20
|
||||
REDIS_DIAL_TIMEOUT=5s
|
||||
REDIS_READ_TIMEOUT=3s
|
||||
REDIS_WRITE_TIMEOUT=3s
|
||||
|
||||
# Feature Flags
|
||||
REDIS_SESSIONS_ENABLED=true
|
||||
REDIS_CACHE_ENABLED=true
|
||||
REDIS_RATELIMIT_ENABLED=true
|
||||
REDIS_PUBSUB_ENABLED=true
|
||||
```
|
||||
|
||||
## Appendix B: Docker Compose Configuration
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
- ./redis.conf:/usr/local/etc/redis/redis.conf:ro
|
||||
command: redis-server /usr/local/etc/redis/redis.conf
|
||||
networks:
|
||||
- trackeep-network
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
trackeep-backend:
|
||||
environment:
|
||||
- REDIS_ADDR=redis:6379
|
||||
- REDIS_PASSWORD=${REDIS_PASSWORD}
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
volumes:
|
||||
redis_data:
|
||||
|
||||
networks:
|
||||
trackeep-network:
|
||||
driver: bridge
|
||||
```
|
||||
@@ -0,0 +1,563 @@
|
||||
# Redis Architecture Diagram for Trackeep
|
||||
|
||||
## System Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ CLIENT LAYER │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Web App │ │ Browser Ext │ │ Mobile │ │ API Keys │ │
|
||||
│ │ (React) │ │ │ │ (Future) │ │ │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
|
||||
└─────────┼─────────────────┼─────────────────┼─────────────────┼────────────┘
|
||||
│ │ │ │
|
||||
└─────────────────┴─────────────────┴─────────────────┘
|
||||
│
|
||||
HTTP/WebSocket
|
||||
│
|
||||
┌───────────────────────────────────┼─────────────────────────────────────────┐
|
||||
│ LOAD BALANCER / REVERSE PROXY │
|
||||
│ (Nginx / Traefik - Future) │
|
||||
└───────────────────────────────────┼─────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────┼─────────────────────────┐
|
||||
│ │ │
|
||||
┌─────────▼─────────┐ ┌──────────▼──────────┐ ┌─────────▼─────────┐
|
||||
│ Trackeep Backend │ │ Trackeep Backend │ │ Trackeep Backend │
|
||||
│ Instance 1 │◄──►│ Instance 2 │◄─►│ Instance N │
|
||||
│ (Go/Gin) │ │ (Go/Gin) │ │ (Go/Gin) │
|
||||
└─────────┬─────────┘ └──────────┬──────────┘ └─────────┬─────────┘
|
||||
│ │ │
|
||||
└─────────────────────────┼─────────────────────────┘
|
||||
│
|
||||
┌───────────────┴───────────────┐
|
||||
│ │
|
||||
┌─────────▼──────────┐ ┌─────────────▼──────────────┐
|
||||
│ REDIS │ │ PostgreSQL │
|
||||
│ (Cache Layer) │ │ (Primary Database) │
|
||||
│ │ │ │
|
||||
│ ┌───────────────┐ │ │ ┌──────────────────────┐ │
|
||||
│ │ Sessions │ │ │ │ users │ │
|
||||
│ │ (Hash) │ │ │ │ bookmarks │ │
|
||||
│ ├───────────────┤ │ │ │ tasks │ │
|
||||
│ │ Cache │ │ │ │ notes │ │
|
||||
│ │ (String) │ │ │ │ files │ │
|
||||
│ ├───────────────┤ │ │ │ messages │ │
|
||||
│ │ Rate Limiting │ │ │ │ analytics │ │
|
||||
│ │ (Sorted Set) │ │ │ │ marketplace │ │
|
||||
│ ├───────────────┤ │ │ │ ... │ │
|
||||
│ │ Leaderboards │ │ │ └──────────────────────┘ │
|
||||
│ │ (Sorted Set) │ │ └────────────────────────────┘
|
||||
│ ├───────────────┤ │
|
||||
│ │ Pub/Sub │ │ ┌──────────────────────────────┐
|
||||
│ │ Channels │◄─┼──────┤ YouTube Scraper Service │
|
||||
│ └───────────────┘ │ │ (Python) │
|
||||
└────────────────────┘ └──────────────────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow Patterns
|
||||
|
||||
### 1. Session Management Flow
|
||||
|
||||
```
|
||||
┌──────────┐ Login Request ┌──────────────┐
|
||||
│ Client │ ─────────────────────► │ Backend │
|
||||
└──────────┘ └──────┬───────┘
|
||||
│
|
||||
│ Create Session
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Redis │
|
||||
│ tk:session │
|
||||
│ :{sessionID}│
|
||||
└──────┬───────┘
|
||||
│
|
||||
│ Store Session Data
|
||||
│ (TTL: 24h)
|
||||
▼
|
||||
┌──────────┐ Session Cookie ┌──────────────┐
|
||||
│ Client │ ◄───────────────────── │ Backend │
|
||||
└────┬─────┘ └──────────────┘
|
||||
│
|
||||
│ Subsequent Requests
|
||||
│ with Session Cookie
|
||||
▼
|
||||
┌──────────┐ Validate Session ┌──────────────┐
|
||||
│ Client │ ─────────────────────► │ Backend │
|
||||
└──────────┘ └──────┬───────┘
|
||||
│
|
||||
│ Lookup Session
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Redis │
|
||||
│ (O(1) get) │
|
||||
└──────┬───────┘
|
||||
│
|
||||
│ Session Valid
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Response │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
### 2. Caching Flow (Search Results)
|
||||
|
||||
```
|
||||
┌──────────┐ Search Request ┌──────────────┐
|
||||
│ Client │ ────────────────────► │ Backend │
|
||||
└──────────┘ └──────┬───────┘
|
||||
│
|
||||
│ Check Cache
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Redis │
|
||||
│ tk:search │
|
||||
│ :{hash} │
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌───────────┴───────────┐
|
||||
│ │
|
||||
Cache Hit Cache Miss
|
||||
│ │
|
||||
▼ ▼
|
||||
┌────────────┐ ┌────────────────┐
|
||||
│ Return │ │ Query │
|
||||
│ Cached │ │ PostgreSQL │
|
||||
│ Results │ │ (Multiple │
|
||||
│ (Fast) │ │ Tables) │
|
||||
└────────────┘ └───────┬────────┘
|
||||
│
|
||||
│ Results
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Cache │
|
||||
│ Results │
|
||||
│ (TTL: 5min) │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Return │
|
||||
│ Results │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
### 3. Rate Limiting Flow
|
||||
|
||||
```
|
||||
┌──────────┐ API Request ┌──────────────┐
|
||||
│ Client │ ─────────────────────► │ Backend │
|
||||
│ (IP: x) │ └──────┬───────┘
|
||||
└──────────┘ │
|
||||
│ Check Rate Limit
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Redis │
|
||||
│ tk:rl:{IP} │
|
||||
│ (Sorted Set)│
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌────────────┴────────────┐
|
||||
│ │
|
||||
Within Limit Limit Exceeded
|
||||
│ │
|
||||
▼ ▼
|
||||
┌────────────┐ ┌──────────────┐
|
||||
│ Update │ │ Return 429 │
|
||||
│ Counter │ │ Too Many │
|
||||
│ (ZADD) │ │ Requests │
|
||||
└─────┬──────┘ └──────────────┘
|
||||
│
|
||||
▼
|
||||
┌────────────┐
|
||||
│ Process │
|
||||
│ Request │
|
||||
└────────────┘
|
||||
|
||||
Time Window Visualization (Sliding Window):
|
||||
|
||||
T-60s T-30s NOW
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
[req1] [req2] [req3] <-- Current window
|
||||
│ │ │
|
||||
Expired │ │
|
||||
Valid requests counted
|
||||
```
|
||||
|
||||
### 4. Real-Time Pub/Sub Flow (Multi-Instance)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ WEBSOCKET CONNECTIONS │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Client 1 │ │ Client 2 │ │ Client 3 │ │ Client 4 │ │
|
||||
│ │(User A) │ │(User B) │ │(User A) │ │(User C) │ │
|
||||
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ┌────────┴────────┐ │ │ │
|
||||
│ └─────►│ Backend 1 │◄─────┘ │ │
|
||||
│ │ (Go/Gin) │ │ │
|
||||
│ │ │ │ │
|
||||
│ │ In-Memory Hub │ │ │
|
||||
│ │ (Local Users) │ │ │
|
||||
│ └────────┬────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────┐ │ │
|
||||
│ └───────►│ Redis │◄───────┘ │
|
||||
│ │ Pub/Sub │ │
|
||||
│ ┌────────┤ Channel ├────────┐ │
|
||||
│ │ └──────────────┘ │ │
|
||||
│ │ │ │ │
|
||||
│ ┌────────┴────────┐ │ ┌────────┴────────┐│
|
||||
│ │ Backend 2 │◄─────┘ │ Backend 3 ││
|
||||
│ │ (Go/Gin) │ │ (Go/Gin) ││
|
||||
│ │ │ │ ││
|
||||
│ │ In-Memory Hub │ │ In-Memory Hub ││
|
||||
│ │ (Local Users) │ │ (Local Users) ││
|
||||
│ └────────┬────────┘ └────────┬────────┘│
|
||||
│ │ │ │
|
||||
│ ┌────────┴────────┐ ┌────────┴────────┐│
|
||||
│ │ Client 5 │ │ Client 6 ││
|
||||
│ │ (User B) │ │ (User A) ││
|
||||
│ └─────────────────┘ └─────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Message Flow:
|
||||
1. Client 1 (Backend 1) sends message
|
||||
2. Backend 1 stores in PostgreSQL
|
||||
3. Backend 1 publishes to Redis channel
|
||||
4. All backends receive message via subscription
|
||||
5. Each backend forwards to connected local clients
|
||||
6. All participants receive real-time update
|
||||
```
|
||||
|
||||
### 5. Leaderboard Update Flow
|
||||
|
||||
```
|
||||
┌──────────┐ Challenge Action ┌──────────────┐
|
||||
│ Client │ ────────────────────► │ Backend │
|
||||
└──────────┘ └──────┬───────┘
|
||||
│
|
||||
│ Record Score
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Redis │
|
||||
│ tk:challenge│
|
||||
│ :{id}:lb │
|
||||
│ (ZADD score)│
|
||||
└──────┬───────┘
|
||||
│
|
||||
│ Update Rank
|
||||
▼
|
||||
┌──────────┐ Get Leaderboard ┌──────────────┐
|
||||
│ Client │ ────────────────────► │ Backend │
|
||||
└──────────┘ └──────┬───────┘
|
||||
│
|
||||
│ ZREVRANGE
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Redis │
|
||||
│ Top N Ranks │
|
||||
│ (O(log N)) │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Leaderboard │
|
||||
│ Response │
|
||||
└──────────────┘
|
||||
|
||||
Data Structure:
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Redis Sorted Set: tk:challenge:123:leaderboard │
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Member (UserID) │ Score │ Rank │
|
||||
├─────────────────────┼────────────┼────────────────┤
|
||||
│ 42 │ 1500 │ 1 │
|
||||
│ 17 │ 1200 │ 2 │
|
||||
│ 89 │ 980 │ 3 │
|
||||
│ 23 │ 750 │ 4 │
|
||||
│ ... │ ... │ ... │
|
||||
└─────────────────────┴────────────┴────────────────┘
|
||||
```
|
||||
|
||||
## Component Interactions
|
||||
|
||||
### Backend Integration Points
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ TRACKEEP BACKEND (Go/Gin) │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Session │ │ Cache │ │ Rate │ │
|
||||
│ │ Store │ │ Middleware │ │ Limiter │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
|
||||
│ │ │ │ │
|
||||
│ └─────────────────┼─────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────▼────────┐ │
|
||||
│ │ Redis Client │ │
|
||||
│ │ (go-redis) │ │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────────────┼─────────────────┐ │
|
||||
│ │ │ │ │
|
||||
│ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ │
|
||||
│ │ String │ │ Hash │ │ Sorted Set │ │
|
||||
│ │ (Cache) │ │ (Session) │ │ (Ranking) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Pub/Sub │ │ Set │ │
|
||||
│ │ (Real-time) │ │ (Tracking) │ │
|
||||
│ └──────────────┘ └──────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Fallback Strategy:
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ if Redis unavailable: │
|
||||
│ ├─► Sessions → Fallback to in-memory map │
|
||||
│ ├─► Cache → Skip cache, query DB directly │
|
||||
│ ├─► Rate Limit→ Skip rate limiting (log warning) │
|
||||
│ └─► Pub/Sub → Local-only WebSocket (limited functionality) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Deployment Scenarios
|
||||
|
||||
### Scenario 1: Single Node (Development)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Docker Host │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Frontend │ │ Backend │ │
|
||||
│ │ (Nginx) │ │ (Go/Gin) │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ │
|
||||
│ │ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌────────┴────────┐ │
|
||||
│ │ │ Redis │ │
|
||||
│ │ │ (Single Node) │ │
|
||||
│ │ └────────┬────────┘ │
|
||||
│ │ │ │
|
||||
│ │ ┌────────┴────────┐ │
|
||||
│ └───────►│ PostgreSQL │ │
|
||||
│ │ (Single Node) │ │
|
||||
│ └─────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Scenario 2: High Availability (Production)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Docker Swarm / Kubernetes │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Load Balancer │ │
|
||||
│ └───────────────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────────┼────────────────────────┐ │
|
||||
│ │ │ │ │
|
||||
│ ┌──────▼──────┐ ┌────────▼────────┐ ┌───────▼───────┐ │
|
||||
│ │ Backend 1 │◄──────►│ Backend 2 │◄────►│ Backend 3 │ │
|
||||
│ └──────┬──────┘ └────────┬────────┘ └───────┬───────┘ │
|
||||
│ │ │ │ │
|
||||
│ └────────────────────────┼────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────────▼─────────────┐ │
|
||||
│ │ Redis Sentinel │ │
|
||||
│ │ ┌─────┐ ┌─────┐ ┌─────┐ │ │
|
||||
│ │ │ M1 │◄►│ R1 │◄►│ R2 │ │ │
|
||||
│ │ └──┬──┘ └──┬──┘ └──┬──┘ │ │
|
||||
│ │ └───────┴───────┘ │ │
|
||||
│ │ S1 S2 S3 │ │
|
||||
│ └───────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────────▼─────────────┐ │
|
||||
│ │ PostgreSQL Cluster │ │
|
||||
│ │ ┌─────┐ ┌─────┐ ┌─────┐ │ │
|
||||
│ │ │ P1 │◄►│ S1 │◄►│ S2 │ │ │
|
||||
│ │ └─────┘ └─────┘ └─────┘ │ │
|
||||
│ └───────────────────────────┘ │
|
||||
│ │
|
||||
│ Legend: M=Master, R=Replica, S=Sentinel, P=Primary │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Memory Allocation Strategy
|
||||
|
||||
```
|
||||
Redis Memory Budget (256MB Example):
|
||||
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ Total: 256MB │
|
||||
├────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ Sessions (30%) 77 MB │ │
|
||||
│ │ ├── Active user sessions (TTL: 24h) │ │
|
||||
│ │ └── User session index sets │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ Cache (50%) 128 MB │ │
|
||||
│ │ ├── Search results (TTL: 5m) │ │
|
||||
│ │ ├── Analytics dashboards (TTL: 15m) │ │
|
||||
│ │ ├── API responses (TTL: varies) │ │
|
||||
│ │ └── AI recommendations (TTL: 1h) │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ Rate Limiting (10%) 26 MB │ │
|
||||
│ │ ├── Per-IP tracking windows │ │
|
||||
│ │ └── Token bucket state │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ Real-time / Other (10%) 25 MB │ │
|
||||
│ │ ├── Leaderboards │ │
|
||||
│ │ ├── Pub/Sub buffers │ │
|
||||
│ │ └── Miscellaneous │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Eviction Policy: allkeys-lru
|
||||
- Least Recently Used keys evicted first when memory limit reached
|
||||
- Sessions have longer TTL to prevent premature eviction
|
||||
- Cache entries have shorter TTL for frequent refresh
|
||||
```
|
||||
|
||||
## Key Naming Convention
|
||||
|
||||
```
|
||||
Hierarchical Key Structure:
|
||||
|
||||
┌────────────────────────────────────────────────────────────────────┐
|
||||
│ Format: tk:{resource}:{id}:{attribute}:{context} │
|
||||
├────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ SESSIONS │
|
||||
│ ├── tk:session:{session_id} → SessionData (JSON) │
|
||||
│ └── tk:user:sessions:{user_id} → Set of session IDs │
|
||||
│ │
|
||||
│ CACHE │
|
||||
│ ├── tk:cache:search:{user_id}:{hash} → SearchResponse │
|
||||
│ ├── tk:cache:analytics:{user_id}:{type} → AnalyticsData │
|
||||
│ ├── tk:cache:user:{id}:profile → UserProfile │
|
||||
│ └── tk:cache:marketplace:trending → TrendingItems │
|
||||
│ │
|
||||
│ RATE LIMITING │
|
||||
│ ├── tk:rl:{ip}:general → SortedSet (timestamps)│
|
||||
│ ├── tk:rl:{ip}:search → SortedSet │
|
||||
│ ├── tk:rl:{ip}:ai → Token bucket state │
|
||||
│ └── tk:rl:{ip}:upload → Token bucket state │
|
||||
│ │
|
||||
│ LEADERBOARDS │
|
||||
│ ├── tk:challenge:{id}:leaderboard → SortedSet (scores) │
|
||||
│ └── tk:marketplace:trending:{period} → SortedSet (views) │
|
||||
│ │
|
||||
│ REAL-TIME │
|
||||
│ ├── tk:messages:{conversation_id} → Pub/Sub channel │
|
||||
│ ├── tk:notifications:{user_id} → Pub/Sub channel │
|
||||
│ └── tk:events:system → Pub/Sub channel │
|
||||
│ │
|
||||
│ COUNTERS │
|
||||
│ ├── tk:counter:views:{content_type}:{id} → Integer │
|
||||
│ └── tk:counter:downloads:{item_id} → Integer │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Hash Function for Long Keys:
|
||||
- MD5 or SHA1 for query parameters
|
||||
- First 8-12 chars of hash usually sufficient
|
||||
- Example: tk:cache:search:1234:a7f3d2c9b1e8
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
```
|
||||
Operation Complexities:
|
||||
|
||||
┌────────────────────┬─────────────┬─────────────┬─────────────────────┐
|
||||
│ Operation │ Time (Big O)│ Memory │ Use Case │
|
||||
├────────────────────┼─────────────┼─────────────┼─────────────────────┤
|
||||
│ GET │ O(1) │ O(1) │ Session retrieval │
|
||||
│ SET │ O(1) │ O(1) │ Cache storage │
|
||||
│ DEL │ O(1) │ O(1) │ Cache invalidation │
|
||||
│ EXPIRE │ O(1) │ O(1) │ TTL management │
|
||||
├────────────────────┼─────────────┼─────────────┼─────────────────────┤
|
||||
│ HGET │ O(1) │ O(1) │ Session field get │
|
||||
│ HSET │ O(1) │ O(1) │ Session field set │
|
||||
│ HGETALL │ O(N) │ O(N) │ Full session read │
|
||||
├────────────────────┼─────────────┼─────────────┼─────────────────────┤
|
||||
│ ZADD │ O(log N) │ O(1) │ Add score │
|
||||
│ ZREVRANGE │ O(log N + M)│ O(M) │ Get top N ranks │
|
||||
│ ZRANK │ O(log N) │ O(1) │ Get user rank │
|
||||
│ ZSCORE │ O(1) │ O(1) │ Get user score │
|
||||
├────────────────────┼─────────────┼─────────────┼─────────────────────┤
|
||||
│ PUBLISH │ O(N+M) │ O(1) │ Send message │
|
||||
│ SUBSCRIBE │ O(1) │ O(1) │ Listen channel │
|
||||
├────────────────────┼─────────────┼─────────────┼─────────────────────┤
|
||||
│ KEYS * │ O(N) │ O(N) │ DEBUG ONLY │
|
||||
│ SCAN │ O(1) │ O(1) │ Iteration │
|
||||
└────────────────────┴─────────────┴─────────────┴─────────────────────┘
|
||||
|
||||
N = Number of elements
|
||||
M = Number of returned elements
|
||||
|
||||
Performance Targets:
|
||||
┌────────────────────┬──────────────┬────────────────┐
|
||||
│ Metric │ Target │ Measurement │
|
||||
├────────────────────┼──────────────┼────────────────┤
|
||||
│ Cache hit latency │ < 1ms │ p99 │
|
||||
│ Cache miss latency │ < 5ms │ p99 │
|
||||
│ Session read │ < 2ms │ p99 │
|
||||
│ Session write │ < 3ms │ p99 │
|
||||
│ Rate limit check │ < 1ms │ p99 │
|
||||
│ Pub/Sub latency │ < 5ms │ p99 │
|
||||
│ Leaderboard query │ < 10ms │ p99 (top 100) │
|
||||
└────────────────────┴──────────────┴────────────────┘
|
||||
```
|
||||
|
||||
## Monitoring Points
|
||||
|
||||
```
|
||||
Key Metrics to Track:
|
||||
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ INFRASTRUCTURE │
|
||||
│ ├── Memory Usage % Alert: > 80% │
|
||||
│ ├── Connected Clients Alert: > 80% of max │
|
||||
│ ├── Blocked Clients Alert: > 0 (indicates slow ops) │
|
||||
│ └── Uptime Alert: < 99.9% │
|
||||
│ │
|
||||
│ PERFORMANCE │
|
||||
│ ├── Commands/sec Track: Trending │
|
||||
│ ├── Hit Rate % Alert: < 80% │
|
||||
│ ├── Miss Rate % Track: Trending │
|
||||
│ ├── Evicted Keys/sec Alert: > 100/min │
|
||||
│ └── Expired Keys/sec Track: Trending │
|
||||
│ │
|
||||
│ ERRORS │
|
||||
│ ├── Rejected Connections Alert: > 0 │
|
||||
│ ├── Keyspace Misses Track: vs Hits │
|
||||
│ ├── Slow Queries (>10ms) Alert: > 10/min │
|
||||
│ └── Replication Lag Alert: > 1s │
|
||||
│ │
|
||||
│ APPLICATION │
|
||||
│ ├── Session Store Latency Alert: > 5ms p99 │
|
||||
│ ├── Cache Hit Ratio Alert: < 75% │
|
||||
│ ├── Rate Limit Accuracy Track: vs Expected │
|
||||
│ └── Pub/Sub Delivery Time Alert: > 10ms p99 │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
@@ -0,0 +1,989 @@
|
||||
# Redis Implementation Quick Reference for Trackeep
|
||||
|
||||
## Overview
|
||||
|
||||
This guide provides practical implementation patterns for integrating Redis into the Trackeep application based on the comprehensive architecture analysis.
|
||||
|
||||
## 1. Quick Start Configuration
|
||||
|
||||
### 1.1 Add Redis to Docker Compose
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
command: >
|
||||
redis-server
|
||||
--appendonly yes
|
||||
--appendfsync everysec
|
||||
--maxmemory 256mb
|
||||
--maxmemory-policy allkeys-lru
|
||||
--requirepass ${REDIS_PASSWORD:-changeme}
|
||||
networks:
|
||||
- trackeep-network
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD:-changeme}", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
ports:
|
||||
- "127.0.0.1:6379:6379" # Local access only
|
||||
|
||||
volumes:
|
||||
redis_data:
|
||||
```
|
||||
|
||||
### 1.2 Environment Variables (.env)
|
||||
|
||||
```bash
|
||||
# Redis Configuration
|
||||
REDIS_ADDR=redis:6379
|
||||
REDIS_PASSWORD=your_secure_password_here
|
||||
REDIS_DB=0
|
||||
REDIS_POOL_SIZE=20
|
||||
REDIS_DIAL_TIMEOUT=5s
|
||||
REDIS_READ_TIMEOUT=3s
|
||||
REDIS_WRITE_TIMEOUT=3s
|
||||
|
||||
# Feature Flags
|
||||
REDIS_SESSIONS_ENABLED=true
|
||||
REDIS_CACHE_ENABLED=true
|
||||
REDIS_RATELIMIT_ENABLED=true
|
||||
```
|
||||
|
||||
## 2. Core Implementation
|
||||
|
||||
### 2.1 Redis Client Setup
|
||||
|
||||
```go
|
||||
// backend/config/redis.go
|
||||
package config
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/go-redis/redis/v8"
|
||||
)
|
||||
|
||||
var RedisClient *redis.Client
|
||||
|
||||
// InitRedis initializes the Redis client
|
||||
func InitRedis() error {
|
||||
poolSize, _ := strconv.Atoi(os.Getenv("REDIS_POOL_SIZE"))
|
||||
if poolSize == 0 {
|
||||
poolSize = 20
|
||||
}
|
||||
|
||||
dialTimeout, _ := time.ParseDuration(os.Getenv("REDIS_DIAL_TIMEOUT"))
|
||||
if dialTimeout == 0 {
|
||||
dialTimeout = 5 * time.Second
|
||||
}
|
||||
|
||||
readTimeout, _ := time.ParseDuration(os.Getenv("REDIS_READ_TIMEOUT"))
|
||||
if readTimeout == 0 {
|
||||
readTimeout = 3 * time.Second
|
||||
}
|
||||
|
||||
writeTimeout, _ := time.ParseDuration(os.Getenv("REDIS_WRITE_TIMEOUT"))
|
||||
if writeTimeout == 0 {
|
||||
writeTimeout = 3 * time.Second
|
||||
}
|
||||
|
||||
RedisClient = redis.NewClient(&redis.Options{
|
||||
Addr: os.Getenv("REDIS_ADDR"),
|
||||
Password: os.Getenv("REDIS_PASSWORD"),
|
||||
DB: 0,
|
||||
PoolSize: poolSize,
|
||||
MinIdleConns: 5,
|
||||
DialTimeout: dialTimeout,
|
||||
ReadTimeout: readTimeout,
|
||||
WriteTimeout: writeTimeout,
|
||||
MaxConnAge: time.Hour,
|
||||
PoolTimeout: 5 * time.Second,
|
||||
IdleTimeout: 10 * time.Minute,
|
||||
})
|
||||
|
||||
// Test connection
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
if err := RedisClient.Ping(ctx).Err(); err != nil {
|
||||
return fmt.Errorf("failed to connect to Redis: %w", err)
|
||||
}
|
||||
|
||||
fmt.Println("Redis connected successfully")
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsRedisEnabled checks if Redis is configured and available
|
||||
func IsRedisEnabled() bool {
|
||||
return RedisClient != nil && os.Getenv("REDIS_ADDR") != ""
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Session Store Migration
|
||||
|
||||
```go
|
||||
// backend/middleware/session_redis.go
|
||||
package middleware
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/trackeep/backend/config"
|
||||
)
|
||||
|
||||
// RedisSessionStore implements distributed session storage
|
||||
type RedisSessionStore struct {
|
||||
fallback *MemorySessionStore
|
||||
}
|
||||
|
||||
// NewRedisSessionStore creates a new Redis-backed session store
|
||||
func NewRedisSessionStore() SessionStore {
|
||||
return &RedisSessionStore{
|
||||
fallback: NewMemorySessionStore(),
|
||||
}
|
||||
}
|
||||
|
||||
func (r *RedisSessionStore) CreateSession(sessionData *SessionData) error {
|
||||
sessionData.CreatedAt = time.Now()
|
||||
sessionData.LastActive = time.Now()
|
||||
|
||||
if config.IsRedisEnabled() {
|
||||
ctx := context.Background()
|
||||
key := fmt.Sprintf("tk:session:%s", sessionData.SessionID)
|
||||
|
||||
data, err := json.Marshal(sessionData)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Store session with 24h TTL
|
||||
if err := config.RedisClient.Set(ctx, key, data, 24*time.Hour).Err(); err != nil {
|
||||
// Fallback to memory on Redis error
|
||||
return r.fallback.CreateSession(sessionData)
|
||||
}
|
||||
|
||||
// Add to user's session set
|
||||
userKey := fmt.Sprintf("tk:user:sessions:%d", sessionData.UserID)
|
||||
config.RedisClient.SAdd(ctx, userKey, sessionData.SessionID)
|
||||
config.RedisClient.Expire(ctx, userKey, 24*time.Hour)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return r.fallback.CreateSession(sessionData)
|
||||
}
|
||||
|
||||
func (r *RedisSessionStore) GetSession(sessionID string) (*SessionData, error) {
|
||||
if config.IsRedisEnabled() {
|
||||
ctx := context.Background()
|
||||
key := fmt.Sprintf("tk:session:%s", sessionID)
|
||||
|
||||
data, err := config.RedisClient.Get(ctx, key).Bytes()
|
||||
if err == nil {
|
||||
var session SessionData
|
||||
if err := json.Unmarshal(data, &session); err == nil {
|
||||
// Update last active
|
||||
session.LastActive = time.Now()
|
||||
r.UpdateSession(sessionID, &session)
|
||||
return &session, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return r.fallback.GetSession(sessionID)
|
||||
}
|
||||
|
||||
func (r *RedisSessionStore) UpdateSession(sessionID string, sessionData *SessionData) error {
|
||||
if config.IsRedisEnabled() {
|
||||
ctx := context.Background()
|
||||
key := fmt.Sprintf("tk:session:%s", sessionID)
|
||||
|
||||
data, err := json.Marshal(sessionData)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := config.RedisClient.Set(ctx, key, data, 24*time.Hour).Err(); err != nil {
|
||||
return r.fallback.UpdateSession(sessionID, sessionData)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return r.fallback.UpdateSession(sessionID, sessionData)
|
||||
}
|
||||
|
||||
func (r *RedisSessionStore) DeleteSession(sessionID string) error {
|
||||
if config.IsRedisEnabled() {
|
||||
ctx := context.Background()
|
||||
key := fmt.Sprintf("tk:session:%s", sessionID)
|
||||
|
||||
// Get session to find user ID
|
||||
data, err := config.RedisClient.Get(ctx, key).Bytes()
|
||||
if err == nil {
|
||||
var session SessionData
|
||||
if err := json.Unmarshal(data, &session); err == nil {
|
||||
// Remove from user's session set
|
||||
userKey := fmt.Sprintf("tk:user:sessions:%d", session.UserID)
|
||||
config.RedisClient.SRem(ctx, userKey, sessionID)
|
||||
}
|
||||
}
|
||||
|
||||
config.RedisClient.Del(ctx, key)
|
||||
}
|
||||
|
||||
return r.fallback.DeleteSession(sessionID)
|
||||
}
|
||||
|
||||
func (r *RedisSessionStore) CleanupExpiredSessions() error {
|
||||
// Redis handles expiration automatically via TTL
|
||||
// Just clean up fallback
|
||||
return r.fallback.CleanupExpiredSessions()
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 Distributed Rate Limiter
|
||||
|
||||
```go
|
||||
// backend/middleware/rate_limiter_redis.go
|
||||
package middleware
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/trackeep/backend/config"
|
||||
)
|
||||
|
||||
// RedisRateLimiter implements distributed rate limiting
|
||||
type RedisRateLimiter struct {
|
||||
limit int
|
||||
window time.Duration
|
||||
keyPrefix string
|
||||
}
|
||||
|
||||
// NewRedisRateLimiter creates a new Redis-backed rate limiter
|
||||
func NewRedisRateLimiter(limit int, window time.Duration, keyPrefix string) *RedisRateLimiter {
|
||||
return &RedisRateLimiter{
|
||||
limit: limit,
|
||||
window: window,
|
||||
keyPrefix: keyPrefix,
|
||||
}
|
||||
}
|
||||
|
||||
// SlidingWindowRateLimit uses Redis sorted sets for accurate sliding window
|
||||
func (rl *RedisRateLimiter) Middleware() gin.HandlerFunc {
|
||||
return func(c *gin.Context) {
|
||||
if !config.IsRedisEnabled() {
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
clientIP := c.ClientIP()
|
||||
key := fmt.Sprintf("%s:%s", rl.keyPrefix, clientIP)
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now().Unix()
|
||||
windowStart := now - int64(rl.window.Seconds())
|
||||
|
||||
// Remove old entries
|
||||
config.RedisClient.ZRemRangeByScore(ctx, key, "0", strconv.FormatInt(windowStart, 10))
|
||||
|
||||
// Count current requests
|
||||
count, err := config.RedisClient.ZCard(ctx, key).Result()
|
||||
if err != nil {
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
// Check limit
|
||||
if int(count) >= rl.limit {
|
||||
c.Header("X-RateLimit-Limit", fmt.Sprintf("%d", rl.limit))
|
||||
c.Header("X-RateLimit-Remaining", "0")
|
||||
c.Header("X-RateLimit-Reset", strconv.FormatInt(now+int64(rl.window.Seconds()), 10))
|
||||
c.JSON(http.StatusTooManyRequests, gin.H{
|
||||
"error": "Rate limit exceeded",
|
||||
"message": fmt.Sprintf("Too many requests. Limit is %d per %v", rl.limit, rl.window),
|
||||
})
|
||||
c.Abort()
|
||||
return
|
||||
}
|
||||
|
||||
// Add current request
|
||||
config.RedisClient.ZAdd(ctx, key, &redis.Z{
|
||||
Score: float64(now),
|
||||
Member: now,
|
||||
})
|
||||
config.RedisClient.Expire(ctx, key, rl.window)
|
||||
|
||||
// Set headers
|
||||
remaining := rl.limit - int(count) - 1
|
||||
c.Header("X-RateLimit-Limit", fmt.Sprintf("%d", rl.limit))
|
||||
c.Header("X-RateLimit-Remaining", fmt.Sprintf("%d", remaining))
|
||||
c.Header("X-RateLimit-Reset", strconv.FormatInt(now+int64(rl.window.Seconds()), 10))
|
||||
|
||||
c.Next()
|
||||
}
|
||||
}
|
||||
|
||||
// TokenBucketRateLimit uses token bucket algorithm for burst handling
|
||||
type TokenBucketRateLimiter struct {
|
||||
capacity int
|
||||
refillRate float64 // tokens per second
|
||||
keyPrefix string
|
||||
}
|
||||
|
||||
func NewTokenBucketRateLimiter(capacity int, refillRate float64, keyPrefix string) *TokenBucketRateLimiter {
|
||||
return &TokenBucketRateLimiter{
|
||||
capacity: capacity,
|
||||
refillRate: refillRate,
|
||||
keyPrefix: keyPrefix,
|
||||
}
|
||||
}
|
||||
|
||||
func (rl *TokenBucketRateLimiter) Middleware() gin.HandlerFunc {
|
||||
return func(c *gin.Context) {
|
||||
if !config.IsRedisEnabled() {
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
clientIP := c.ClientIP()
|
||||
key := fmt.Sprintf("%s:%s", rl.keyPrefix, clientIP)
|
||||
ctx := context.Background()
|
||||
|
||||
// Lua script for atomic token bucket
|
||||
script := `
|
||||
local key = KEYS[1]
|
||||
local capacity = tonumber(ARGV[1])
|
||||
local refill_rate = tonumber(ARGV[2])
|
||||
local now = tonumber(ARGV[3])
|
||||
|
||||
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
|
||||
local tokens = tonumber(bucket[1]) or capacity
|
||||
local last_refill = tonumber(bucket[2]) or now
|
||||
|
||||
local delta = math.min(capacity, tokens + (now - last_refill) * refill_rate)
|
||||
|
||||
if delta >= 1 then
|
||||
redis.call('HMSET', key, 'tokens', delta - 1, 'last_refill', now)
|
||||
redis.call('EXPIRE', key, 3600)
|
||||
return {1, math.floor(delta - 1)}
|
||||
else
|
||||
redis.call('HMSET', key, 'tokens', delta, 'last_refill', now)
|
||||
redis.call('EXPIRE', key, 3600)
|
||||
return {0, math.floor(delta)}
|
||||
end
|
||||
`
|
||||
|
||||
now := float64(time.Now().Unix())
|
||||
result, err := config.RedisClient.Eval(ctx, script, []string{key},
|
||||
rl.capacity, rl.refillRate, now).Result()
|
||||
|
||||
if err != nil {
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
values := result.([]interface{})
|
||||
allowed := values[0].(int64) == 1
|
||||
remaining := values[1].(int64)
|
||||
|
||||
c.Header("X-RateLimit-Limit", fmt.Sprintf("%d", rl.capacity))
|
||||
c.Header("X-RateLimit-Remaining", fmt.Sprintf("%d", remaining))
|
||||
|
||||
if !allowed {
|
||||
c.JSON(http.StatusTooManyRequests, gin.H{
|
||||
"error": "Rate limit exceeded",
|
||||
"message": "Too many requests. Please slow down.",
|
||||
})
|
||||
c.Abort()
|
||||
return
|
||||
}
|
||||
|
||||
c.Next()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2.4 Caching Middleware
|
||||
|
||||
```go
|
||||
// backend/middleware/cache_redis.go
|
||||
package middleware
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/md5"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/trackeep/backend/config"
|
||||
)
|
||||
|
||||
// RedisCacheConfig holds Redis cache configuration
|
||||
type RedisCacheConfig struct {
|
||||
Duration time.Duration
|
||||
KeyPrefix string
|
||||
Enabled bool
|
||||
}
|
||||
|
||||
// DefaultRedisCacheConfig returns default cache configuration
|
||||
func DefaultRedisCacheConfig() RedisCacheConfig {
|
||||
return RedisCacheConfig{
|
||||
Duration: 5 * time.Minute,
|
||||
KeyPrefix: "tk:cache:",
|
||||
Enabled: true,
|
||||
}
|
||||
}
|
||||
|
||||
// RedisCacheMiddleware creates a Redis-based cache middleware
|
||||
func RedisCacheMiddleware(config RedisCacheConfig) gin.HandlerFunc {
|
||||
if !config.Enabled {
|
||||
return func(c *gin.Context) {
|
||||
c.Next()
|
||||
}
|
||||
}
|
||||
|
||||
return func(c *gin.Context) {
|
||||
// Only cache GET requests
|
||||
if c.Request.Method != http.MethodGet {
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
// Skip if Redis not available
|
||||
if !config.IsRedisEnabled() {
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
|
||||
// Generate cache key
|
||||
cacheKey := generateRedisCacheKey(c, config.KeyPrefix)
|
||||
|
||||
// Try to get from cache
|
||||
ctx := context.Background()
|
||||
cached, err := config.RedisClient.Get(ctx, cacheKey).Result()
|
||||
if err == nil && cached != "" {
|
||||
c.Header("X-Cache", "HIT")
|
||||
c.Header("Content-Type", "application/json")
|
||||
c.String(http.StatusOK, cached)
|
||||
c.Abort()
|
||||
return
|
||||
}
|
||||
|
||||
// Cache miss
|
||||
c.Header("X-Cache", "MISS")
|
||||
|
||||
// Capture response
|
||||
writer := &cachedResponseWriter{
|
||||
ResponseWriter: c.Writer,
|
||||
buffer: make([]byte, 0),
|
||||
}
|
||||
c.Writer = writer
|
||||
|
||||
c.Next()
|
||||
|
||||
// Cache the response if successful
|
||||
if c.Writer.Status() == http.StatusOK && len(writer.buffer) > 0 {
|
||||
config.RedisClient.Set(ctx, cacheKey, string(writer.buffer), config.Duration)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func generateRedisCacheKey(c *gin.Context, prefix string) string {
|
||||
keyParts := []string{
|
||||
prefix,
|
||||
c.Request.URL.Path,
|
||||
c.Request.URL.RawQuery,
|
||||
}
|
||||
|
||||
if userID := c.GetString("userID"); userID != "" {
|
||||
keyParts = append(keyParts, "u:"+userID)
|
||||
}
|
||||
|
||||
key := strings.Join(keyParts, ":")
|
||||
hash := md5.Sum([]byte(key))
|
||||
return fmt.Sprintf("%s%x", prefix, hash)
|
||||
}
|
||||
|
||||
// InvalidateUserCache removes all cache entries for a user
|
||||
func InvalidateUserCache(userID string) error {
|
||||
if !config.IsRedisEnabled() {
|
||||
return nil
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
pattern := fmt.Sprintf("tk:cache:*u:%s*", userID)
|
||||
|
||||
keys, err := config.RedisClient.Keys(ctx, pattern).Result()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(keys) > 0 {
|
||||
return config.RedisClient.Del(ctx, keys...).Err()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Usage Patterns
|
||||
|
||||
### 3.1 Search Result Caching
|
||||
|
||||
```go
|
||||
// backend/handlers/search_enhanced.go
|
||||
func EnhancedSearch(c *gin.Context) {
|
||||
var filters SearchFilters
|
||||
if err := c.ShouldBindJSON(&filters); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
userID := c.GetUint("user_id")
|
||||
|
||||
// Try cache first
|
||||
cacheKey := fmt.Sprintf("tk:search:%d:%s", userID, hashFilters(filters))
|
||||
|
||||
if config.IsRedisEnabled() {
|
||||
ctx := context.Background()
|
||||
cached, err := config.RedisClient.Get(ctx, cacheKey).Result()
|
||||
if err == nil {
|
||||
var response SearchResponse
|
||||
if json.Unmarshal([]byte(cached), &response) == nil {
|
||||
c.Header("X-Cache", "HIT")
|
||||
c.JSON(http.StatusOK, response)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Perform search
|
||||
results := performSearch(filters, userID)
|
||||
|
||||
// Cache results
|
||||
if config.IsRedisEnabled() {
|
||||
ctx := context.Background()
|
||||
data, _ := json.Marshal(results)
|
||||
config.RedisClient.Set(ctx, cacheKey, data, 5*time.Minute)
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, results)
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Analytics Aggregation Caching
|
||||
|
||||
```go
|
||||
// backend/services/analytics_cache.go
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/trackeep/backend/config"
|
||||
"github.com/trackeep/backend/models"
|
||||
)
|
||||
|
||||
type AnalyticsCache struct {
|
||||
db *gorm.DB
|
||||
}
|
||||
|
||||
func NewAnalyticsCache(db *gorm.DB) *AnalyticsCache {
|
||||
return &AnalyticsCache{db: db}
|
||||
}
|
||||
|
||||
func (ac *AnalyticsCache) GetDashboardAnalytics(userID uint, startDate, endDate time.Time) (*DashboardAnalytics, error) {
|
||||
cacheKey := fmt.Sprintf("tk:analytics:dashboard:%d:%s:%s",
|
||||
userID, startDate.Format("2006-01-02"), endDate.Format("2006-01-02"))
|
||||
|
||||
// Try cache
|
||||
if config.IsRedisEnabled() {
|
||||
ctx := context.Background()
|
||||
cached, err := config.RedisClient.Get(ctx, cacheKey).Result()
|
||||
if err == nil {
|
||||
var analytics DashboardAnalytics
|
||||
if err := json.Unmarshal([]byte(cached), &analytics); err == nil {
|
||||
return &analytics, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Compute analytics
|
||||
analytics := ac.computeDashboardAnalytics(userID, startDate, endDate)
|
||||
|
||||
// Cache for 15 minutes
|
||||
if config.IsRedisEnabled() {
|
||||
ctx := context.Background()
|
||||
data, _ := json.Marshal(analytics)
|
||||
config.RedisClient.Set(ctx, cacheKey, data, 15*time.Minute)
|
||||
}
|
||||
|
||||
return analytics, nil
|
||||
}
|
||||
|
||||
func (ac *AnalyticsCache) InvalidateUserAnalytics(userID uint) {
|
||||
if !config.IsRedisEnabled() {
|
||||
return
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
pattern := fmt.Sprintf("tk:analytics:*:%d:*", userID)
|
||||
|
||||
keys, _ := config.RedisClient.Keys(ctx, pattern).Result()
|
||||
if len(keys) > 0 {
|
||||
config.RedisClient.Del(ctx, keys...)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 Leaderboard with Sorted Sets
|
||||
|
||||
```go
|
||||
// backend/services/leaderboard.go
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/go-redis/redis/v8"
|
||||
"github.com/trackeep/backend/config"
|
||||
)
|
||||
|
||||
type Leaderboard struct {
|
||||
key string
|
||||
}
|
||||
|
||||
func NewLeaderboard(challengeID uint) *Leaderboard {
|
||||
return &Leaderboard{
|
||||
key: fmt.Sprintf("tk:challenge:%d:leaderboard", challengeID),
|
||||
}
|
||||
}
|
||||
|
||||
// AddScore adds or updates a user's score
|
||||
func (lb *Leaderboard) AddScore(userID uint, score float64) error {
|
||||
if !config.IsRedisEnabled() {
|
||||
return nil
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
member := fmt.Sprintf("%d", userID)
|
||||
|
||||
return config.RedisClient.ZAdd(ctx, lb.key, &redis.Z{
|
||||
Score: score,
|
||||
Member: member,
|
||||
}).Err()
|
||||
}
|
||||
|
||||
// GetTopN returns top N participants
|
||||
func (lb *Leaderboard) GetTopN(n int64) ([]LeaderboardEntry, error) {
|
||||
if !config.IsRedisEnabled() {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
results, err := config.RedisClient.ZRevRangeWithScores(ctx, lb.key, 0, n-1).Result()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
entries := make([]LeaderboardEntry, len(results))
|
||||
for i, result := range results {
|
||||
userID := parseUint(result.Member.(string))
|
||||
entries[i] = LeaderboardEntry{
|
||||
UserID: userID,
|
||||
Score: result.Score,
|
||||
Rank: i + 1,
|
||||
}
|
||||
}
|
||||
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// GetUserRank returns a specific user's rank and score
|
||||
func (lb *Leaderboard) GetUserRank(userID uint) (int64, float64, error) {
|
||||
if !config.IsRedisEnabled() {
|
||||
return 0, 0, nil
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
member := fmt.Sprintf("%d", userID)
|
||||
|
||||
rank, err := config.RedisClient.ZRevRank(ctx, lb.key, member).Result()
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
score, err := config.RedisClient.ZScore(ctx, lb.key, member).Result()
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
return rank + 1, score, nil // Rank is 0-indexed
|
||||
}
|
||||
|
||||
type LeaderboardEntry struct {
|
||||
UserID uint
|
||||
Score float64
|
||||
Rank int
|
||||
}
|
||||
```
|
||||
|
||||
## 4. Pub/Sub for Real-Time Features
|
||||
|
||||
```go
|
||||
// backend/services/pubsub.go
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/trackeep/backend/config"
|
||||
)
|
||||
|
||||
type PubSub struct {
|
||||
ctx context.Context
|
||||
}
|
||||
|
||||
func NewPubSub() *PubSub {
|
||||
return &PubSub{
|
||||
ctx: context.Background(),
|
||||
}
|
||||
}
|
||||
|
||||
// PublishMessage publishes a message to a conversation channel
|
||||
func (ps *PubSub) PublishMessage(conversationID uint, message interface{}) error {
|
||||
if !config.IsRedisEnabled() {
|
||||
return nil
|
||||
}
|
||||
|
||||
channel := fmt.Sprintf("tk:messages:%d", conversationID)
|
||||
data, _ := json.Marshal(message)
|
||||
|
||||
return config.RedisClient.Publish(ps.ctx, channel, data).Err()
|
||||
}
|
||||
|
||||
// SubscribeToMessages subscribes to conversation messages
|
||||
func (ps *PubSub) SubscribeToMessages(conversationID uint, handler func(message []byte)) {
|
||||
if !config.IsRedisEnabled() {
|
||||
return
|
||||
}
|
||||
|
||||
channel := fmt.Sprintf("tk:messages:%d", conversationID)
|
||||
pubsub := config.RedisClient.Subscribe(ps.ctx, channel)
|
||||
defer pubsub.Close()
|
||||
|
||||
ch := pubsub.Channel()
|
||||
for msg := range ch {
|
||||
handler([]byte(msg.Payload))
|
||||
}
|
||||
}
|
||||
|
||||
// PublishNotification publishes a user notification
|
||||
func (ps *PubSub) PublishNotification(userID uint, notification interface{}) error {
|
||||
if !config.IsRedisEnabled() {
|
||||
return nil
|
||||
}
|
||||
|
||||
channel := fmt.Sprintf("tk:notifications:%d", userID)
|
||||
data, _ := json.Marshal(notification)
|
||||
|
||||
return config.RedisClient.Publish(ps.ctx, channel, data).Err()
|
||||
}
|
||||
```
|
||||
|
||||
## 5. Testing and Monitoring
|
||||
|
||||
### 5.1 Health Check Endpoint
|
||||
|
||||
```go
|
||||
// backend/handlers/health.go addition
|
||||
func HealthCheck(c *gin.Context) {
|
||||
status := map[string]interface{}{
|
||||
"status": "ok",
|
||||
"version": "1.0.0",
|
||||
}
|
||||
|
||||
// Check Redis
|
||||
if config.IsRedisEnabled() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
|
||||
if err := config.RedisClient.Ping(ctx).Err(); err != nil {
|
||||
status["redis"] = "unhealthy"
|
||||
status["status"] = "degraded"
|
||||
} else {
|
||||
poolStats := config.RedisClient.PoolStats()
|
||||
status["redis"] = map[string]interface{}{
|
||||
"status": "healthy",
|
||||
"hits": poolStats.Hits,
|
||||
"misses": poolStats.Misses,
|
||||
"total_conns": poolStats.TotalConns,
|
||||
"idle_conns": poolStats.IdleConns,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, status)
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 Cache Metrics Collection
|
||||
|
||||
```go
|
||||
// backend/middleware/metrics.go addition
|
||||
func RecordCacheMetrics() {
|
||||
if !config.IsRedisEnabled() {
|
||||
return
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
info := config.RedisClient.Info(ctx, "stats").Val()
|
||||
|
||||
// Parse key metrics
|
||||
hits := parseRedisInfoValue(info, "keyspace_hits")
|
||||
misses := parseRedisInfoValue(info, "keyspace_misses")
|
||||
|
||||
hitRate := float64(hits) / float64(hits+misses) * 100
|
||||
|
||||
// Log or export metrics
|
||||
log.Printf("Cache Hit Rate: %.2f%% (Hits: %d, Misses: %d)", hitRate, hits, misses)
|
||||
}
|
||||
```
|
||||
|
||||
## 6. Migration Checklist
|
||||
|
||||
### Phase 1: Infrastructure (Day 1)
|
||||
- [ ] Add Redis to Docker Compose
|
||||
- [ ] Add Redis configuration to `.env.example`
|
||||
- [ ] Implement `config/redis.go` client setup
|
||||
- [ ] Add Redis health check to main.go initialization
|
||||
- [ ] Test connection and basic operations
|
||||
|
||||
### Phase 2: Session Storage (Days 2-3)
|
||||
- [ ] Implement `RedisSessionStore`
|
||||
- [ ] Add feature flag `REDIS_SESSIONS_ENABLED`
|
||||
- [ ] Test session persistence across restarts
|
||||
- [ ] Verify session cleanup works correctly
|
||||
- [ ] Monitor memory usage
|
||||
|
||||
### Phase 3: Rate Limiting (Days 4-5)
|
||||
- [ ] Implement `RedisRateLimiter` with sliding window
|
||||
- [ ] Add token bucket variant for burst handling
|
||||
- [ ] Configure different limits per endpoint
|
||||
- [ ] Test rate limiting across multiple requests
|
||||
- [ ] Verify headers are set correctly
|
||||
|
||||
### Phase 4: Caching (Week 2)
|
||||
- [ ] Implement `RedisCacheMiddleware`
|
||||
- [ ] Add search result caching
|
||||
- [ ] Add analytics dashboard caching
|
||||
- [ ] Implement cache invalidation on data changes
|
||||
- [ ] Configure TTL strategy per content type
|
||||
|
||||
### Phase 5: Advanced Features (Week 3)
|
||||
- [ ] Implement leaderboards with Sorted Sets
|
||||
- [ ] Add Pub/Sub for real-time messaging
|
||||
- [ ] Implement distributed locking if needed
|
||||
- [ ] Add cache warming for hot data
|
||||
- [ ] Performance benchmarking
|
||||
|
||||
### Phase 6: Production Readiness (Week 4)
|
||||
- [ ] Add Redis Sentinel configuration
|
||||
- [ ] Configure persistence (AOF + RDB)
|
||||
- [ ] Set up monitoring and alerting
|
||||
- [ ] Document operational procedures
|
||||
- [ ] Load testing and optimization
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Connection Refused**
|
||||
```
|
||||
Error: dial tcp: connect: connection refused
|
||||
```
|
||||
- Check Redis container is running: `docker-compose ps`
|
||||
- Verify network configuration in docker-compose.yml
|
||||
- Check firewall rules
|
||||
|
||||
**Authentication Failed**
|
||||
```
|
||||
Error: NOAUTH Authentication required
|
||||
```
|
||||
- Verify REDIS_PASSWORD matches docker-compose configuration
|
||||
- Check for special characters in password
|
||||
|
||||
**Memory Limit Reached**
|
||||
```
|
||||
Error: OOM command not allowed when used memory > 'maxmemory'
|
||||
```
|
||||
- Increase maxmemory in Redis configuration
|
||||
- Review eviction policy
|
||||
- Check for memory leaks in cache keys
|
||||
|
||||
**High Connection Count**
|
||||
```
|
||||
Error: ERR max number of clients reached
|
||||
```
|
||||
- Increase maxclients in Redis configuration
|
||||
- Review connection pool settings
|
||||
- Check for connection leaks
|
||||
|
||||
### Debug Commands
|
||||
|
||||
```bash
|
||||
# Check Redis connection
|
||||
docker-compose exec redis redis-cli ping
|
||||
|
||||
# Monitor Redis commands in real-time
|
||||
docker-compose exec redis redis-cli monitor
|
||||
|
||||
# Check memory usage
|
||||
docker-compose exec redis redis-cli info memory
|
||||
|
||||
# List all keys (use sparingly)
|
||||
docker-compose exec redis redis-cli keys "*"
|
||||
|
||||
# Get specific key info
|
||||
docker-compose exec redis redis-cli ttl "tk:session:abc123"
|
||||
docker-compose exec redis redis-cli type "tk:session:abc123"
|
||||
|
||||
# Clear all data (WARNING: Destructive)
|
||||
docker-compose exec redis redis-cli flushall
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Note:** This guide is a companion to the full architecture analysis document. Refer to `REDIS_ARCHITECTURE_ANALYSIS.md` for detailed rationale and design decisions.
|
||||
@@ -0,0 +1,125 @@
|
||||
# 🚀 Trackeep Release Guide
|
||||
|
||||
This guide covers how to create releases for Trackeep using different methods.
|
||||
|
||||
## Method 1: GitHub CLI (Recommended)
|
||||
|
||||
For new features or bug fixes:
|
||||
|
||||
```bash
|
||||
# 1. Commit your changes
|
||||
git commit -m "feat: add new amazing feature"
|
||||
|
||||
# 2. Create version tag and push
|
||||
git tag v1.2.7
|
||||
git push origin v1.2.7
|
||||
|
||||
# 3. Create GitHub release with CLI
|
||||
gh release create v1.2.7 \
|
||||
--title "Trackeep v1.2.7 - Release Title" \
|
||||
--notes "Release notes here..."
|
||||
|
||||
# Or use a release notes file
|
||||
gh release create v1.2.7 \
|
||||
--title "Trackeep v1.2.7 - Release Title" \
|
||||
--notes-file RELEASE_v1.2.7.md
|
||||
```
|
||||
|
||||
### GitHub CLI Installation
|
||||
|
||||
If you don't have GitHub CLI installed:
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt install gh
|
||||
|
||||
# Alternative with Snap
|
||||
sudo snap install gh
|
||||
|
||||
# Authenticate with GitHub
|
||||
gh auth login
|
||||
```
|
||||
|
||||
## Method 2: Manual Scripts
|
||||
|
||||
For traditional workflow:
|
||||
|
||||
```bash
|
||||
# Use version update script
|
||||
./scripts/update-version.sh 1.2.7
|
||||
|
||||
# Commit and push
|
||||
git add . && git commit -m "chore: bump version to 1.2.7"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
## Method 3: Release Script
|
||||
|
||||
Use the automated release script:
|
||||
|
||||
```bash
|
||||
./scripts/release.sh 1.2.7
|
||||
```
|
||||
|
||||
This script will:
|
||||
- Update version in .env file
|
||||
- Build Docker images with version tags
|
||||
- Push images to GitHub Container Registry
|
||||
- Create and push Git tag
|
||||
- Push tag to origin
|
||||
|
||||
## Semantic Versioning
|
||||
|
||||
Follow industry standard (MAJOR.MINOR.PATCH):
|
||||
|
||||
```
|
||||
1.2.6 → 1.3.0 (MINOR: new features)
|
||||
1.2.6 → 1.2.7 (PATCH: bug fixes)
|
||||
1.2.6 → 2.0.0 (MAJOR: breaking changes)
|
||||
```
|
||||
|
||||
## Release Notes Template
|
||||
|
||||
Create comprehensive release notes following this structure:
|
||||
|
||||
```markdown
|
||||
# 🎉 Trackeep v1.2.7 - Release Title
|
||||
|
||||
## ✅ What's New
|
||||
|
||||
### **Feature Category 1**
|
||||
- ✅ New feature description
|
||||
- ✅ Another improvement
|
||||
|
||||
### **Bug Fixes**
|
||||
- ✅ Fixed issue description
|
||||
- ✅ Another bug fix
|
||||
|
||||
## 🎯 How to Update
|
||||
|
||||
### **Current Users:**
|
||||
```bash
|
||||
# Option 1: Built-in updates
|
||||
# Update button appears in left navigation
|
||||
|
||||
# Option 2: Manual Docker pull
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## 📦 Docker Images
|
||||
|
||||
- `ghcr.io/dvorinka/trackeep/backend:1.2.7`
|
||||
- `ghcr.io/dvorinka/trackeep/frontend:1.2.7`
|
||||
- `ghcr.io/dvorinka/trackeep/backend:latest`
|
||||
- `ghcr.io/dvorinka/trackeep/frontend:latest`
|
||||
```
|
||||
|
||||
## Docker Images
|
||||
|
||||
Images are automatically built and pushed to GitHub Container Registry:
|
||||
|
||||
- **Registry**: `ghcr.io/dvorinka/trackeep`
|
||||
- **Latest tags**: `backend:latest`, `frontend:latest` (for auto-updates)
|
||||
- **Versioned tags**: `backend:1.2.5`, `frontend:1.2.5` (for specific releases)
|
||||
- **Automatic builds**: Triggered by Git tags and pushes to main branch
|
||||
Reference in New Issue
Block a user