A deep dive into designing, implementing, and optimizing security logging infrastructure for modern threat detection
Press enter or click to view image in full size
Introduction
A while ago, I was called in to investigate a sophisticated breach at a mid-sized fintech company. What started as routine anomaly detection quickly revealed a months-long campaign involving credential harvesting, lateral movement, and data exfiltration. The attack itself was concerning, but what kept me awake at night was something else entirely: we were flying blind for most of it.
Their AWS environment generated hundreds of gigabytes of logs daily, yet critical security events were scattered across dozens of services, stored in incompatible formats, and often aged out before analysis. When we finally pieced together the timeline, it became clear that the attacker’s initial foothold (T1078 — Valid Accounts) could have been detected on day one if proper logging architecture had been in place.
This investigation reinforced a fundamental truth I’ve learned over seven years in cyber security: you can’t defend what you can’t see, and you can’t see what you don’t log properly.
Today, I want to share the lessons learned from that incident and dozens of others like it. We’ll build a comprehensive security logging architecture that can detect, contain, and investigate threats at cloud scale — all while keeping costs manageable and maintaining operational efficiency.
Why Security Logging is Your First Line of Defense
During that fintech investigation, I discovered the attacker had been present for 127 days. They moved laterally through seventeen EC2 instances, accessed three S3 buckets containing customer PII, and established persistence through multiple techniques including scheduled tasks (T1053.005) and account manipulation (T1098). Yet for 120 of those days, we had no visibility into their activities.
The reality is stark: without comprehensive logging, you’re not doing security — you’re playing security theater.
The Detection Gap
Consider this timeline from the investigation:
Day 1: Initial compromise via stolen credentials (T1078.004) Day 15: Lateral movement to production systems (T1021.001) Day 43: Data discovery and collection (T1083, T1119) Day 89: Exfiltration begins (T1041) Day 127: Discovery by external threat intel feed
Each of these phases generated log events that could have triggered alerts:
Day 1: Unusual login patterns from new geographic locations
Day 15: Abnormal process execution and network connections
Day 43: Mass file enumeration and database queries
Day 89: Large data transfers to external IPs
But those logs were either not collected, not correlated, or had already been purged by retention policies designed for cost optimization rather than security effectiveness.
Modern Threat Landscape Demands Modern Logging
Today’s attackers operate with patience and sophistication. They understand that most organizations have visibility gaps and exploit them ruthlessly. A solid logging architecture serves multiple critical functions:
1. Real-time threat detection— Identifying active threats as they unfold
2. Historical analysis — Understanding attack timelines and attribution
3. Compliance — Meeting regulatory requirements for audit trails
4. Operational intelligence — Supporting incident response and forensics
5. Threat hunting — Enabling proactive searches for unknown threats
Understanding AWS Logging Data Types
Before diving into architecture, let’s categorize the types of security-relevant data AWS generates. I’ve learned to think about logs in four distinct categories:
1. Administrative Logs
These capture management plane activities — the “who did what” of your AWS environment.
AWS CloudTrail is your primary source here. Every API call generates a detailed record:
This single log entry tells us exactly who assumed what role, when, and from where — critical for detecting privilege escalation (T1548) and unauthorized access attempts.
2. Data Access Logs
These track access to your data stores and are crucial for detecting data exfiltration.
S3 Access Logs provide granular visibility into bucket operations:
These capture application-level events and user activities.
CloudWatch Logs from EC2 instances, Lambda functions, and containers provide deep visibility into application behavior. During my investigation, application logs revealed the attacker’s reconnaissance activities:
2024-01-15 10:25:33 INFO [RequestHandler] User: compromised-user accessed endpoint: /api/customers/list 2024-01-15 10:25:45 WARN [DatabaseHandler] Large result set returned: 15,847 records for query: SELECT * FROM customers 2024-01-15 10:26:12 INFO [RequestHandler] User: compromised-user accessed endpoint: /api/customers/export
4. Security Service Logs
AWS security services generate their own specialized logs.
GuardDuty findings provide machine learning-driven threat detection:
Let me walk you through the critical log sources I configure in every AWS environment. Each serves a specific purpose in our overall detection strategy:
Purpose: Automated threat detection and SIEM integration
MITRE Coverage: Multi-tactic coverage through ML-driven detection
Architectural Design: Building for Scale and Security
Based on my experience with environments ranging from startups to Fortune 500 companies, I’ve developed a reference architecture that balances security effectiveness with operational practicality.
High-Level Architecture Overview
Press enter or click to view image in full size
System Components Deep Dive
Log Collectors
The collector layer is where logs enter our architecture. I’ve learned that having multiple collection methods is crucial for resilience. Here are the tools I recommend based on different use cases:
The aggregation layer normalizes and enriches logs before storage. I prefer Kinesis Data Firehose for this role because of its managed nature and built-in transformations.
Tool Options:
Amazon Kinesis Data Firehose — Managed AWS service with built-in transformations
Logstash — Part of Elastic Stack, powerful parsing and transformation
Fluentd — Data collector with rich plugin ecosystem
Vector — High-performance observability data pipeline
Apache NiFi — Visual dataflow programming for complex routing
Fluent Bit — Lightweight log processor and forwarder
The alerting layer transforms log events into actionable security notifications, enabling rapid response to threats. I’ve learned that effective alerting requires both automated detection and intelligent noise reduction.
Tool Options:
Amazon SNS — Simple notification service for basic alerting
PagerDuty — Incident management with escalation policies
Slack/Microsoft Teams — Collaborative alerting for security teams
Splunk SOAR — Security orchestration and automated response
Elasticsearch Watcher — Built-in alerting for Elasticsearch clusters
Prometheus Alertmanager— Alert handling for metrics-based systems
n8n — Low-code workflow automation for custom alerting and integrations
def lambda_handler(event, context): """ Intelligent alert processing and enrichment """ for record in event['Records']: message = json.loads(record['Sns']['Message'])
After analyzing hundreds of incidents, I’ve developed a strategic approach to logging that focuses on detection efficacy rather than volume.
Priority-Based Log Collection
Critical (Real-time processing):
Authentication events (CloudTrail IAM actions)
Privilege escalation attempts
Data access to sensitive resources
Network connections to known bad IPs
High (Near real-time, < 5 minutes):
Administrative actions
Security group changes
S3 bucket policy modifications
GuardDuty findings
Medium (Batch processing, < 1 hour):
VPC Flow Logs
Application logs
Database audit logs
DNS queries
Low (Daily processing):
Configuration snapshots
Billing logs
Resource inventory
Detection Rule Framework
I organize detection rules around the MITRE ATT&CK framework. Here’s an example rule for detecting credential dumping (T1003):
-- Detect potential credential dumping via suspicious process execution SELECT timestamp, instance_id, user_name, process_name, command_line, parent_process, COUNT(*) as event_count FROM aws_cloudwatch_logs WHERE timestamp >= NOW() - INTERVAL '1 HOUR' AND ( process_name LIKE '%mimikatz%' OR process_name LIKE '%procdump%' OR command_line LIKE '%lsass%' OR command_line LIKE '%sam%' OR command_line LIKE '%ntds.dit%' ) AND NOT ( -- Exclude known good processes user_name = 'backup-service' AND process_name = 'ntdsutil.exe' ) GROUP BY timestamp, instance_id, user_name, process_name, command_line, parent_process HAVING COUNT(*) >= 3 ORDER BY timestamp DESC;
Lateral Movement Detection (T1021.001 — Remote Desktop Protocol):
-- Detect unusual RDP connections indicating lateral movement WITH rdp_sessions AS ( SELECT source_ip, destination_ip, timestamp, ROW_NUMBER() OVER ( PARTITION BY source_ip ORDER BY timestamp ) as session_sequence FROM vpc_flow_logs WHERE destination_port = 3389 AND action = 'ACCEPT' AND timestamp >= NOW() - INTERVAL '24 HOURS' ), lateral_movement AS ( SELECT r1.source_ip, r1.destination_ip as first_target, r2.destination_ip as second_target, r1.timestamp as first_connection, r2.timestamp as second_connection, TIMESTAMPDIFF(MINUTE, r1.timestamp, r2.timestamp) as time_diff FROM rdp_sessions r1 JOIN rdp_sessions r2 ON r1.source_ip = r2.source_ip WHERE r1.session_sequence = 1 AND r2.session_sequence = 2 AND r1.destination_ip != r2.destination_ip AND TIMESTAMPDIFF(MINUTE, r1.timestamp, r2.timestamp) <= 60 ) SELECT source_ip, first_target, second_target, first_connection, second_connection, time_diff FROM lateral_movement ORDER BY first_connection DESC;
Data Exfiltration Detection (T1041 — Exfiltration Over C2 Channel):
-- Detect large data transfers to external IPs WITH baseline_traffic AS ( SELECT source_ip, AVG(bytes) as avg_bytes, STDDEV(bytes) as stddev_bytes FROM vpc_flow_logs WHERE timestamp >= NOW() - INTERVAL '30 DAYS' AND timestamp < NOW() - INTERVAL '1 DAY' GROUP BY source_ip ), recent_traffic AS ( SELECT source_ip, destination_ip, SUM(bytes) as total_bytes, COUNT(*) as connection_count, timestamp FROM vpc_flow_logs WHERE timestamp >= NOW() - INTERVAL '1 HOUR' AND action = 'ACCEPT' -- External IPs only AND NOT ( destination_ip LIKE '10.%' OR destination_ip LIKE '172.16.%' OR destination_ip LIKE '172.17.%' OR destination_ip LIKE '172.18.%' OR destination_ip LIKE '172.19.%' OR destination_ip LIKE '172.20.%' OR destination_ip LIKE '172.21.%' OR destination_ip LIKE '172.22.%' OR destination_ip LIKE '172.23.%' OR destination_ip LIKE '172.24.%' OR destination_ip LIKE '172.25.%' OR destination_ip LIKE '172.26.%' OR destination_ip LIKE '172.27.%' OR destination_ip LIKE '172.28.%' OR destination_ip LIKE '172.29.%' OR destination_ip LIKE '172.30.%' OR destination_ip LIKE '172.31.%' OR destination_ip LIKE '192.168.%' ) GROUP BY source_ip, destination_ip, timestamp ) SELECT rt.source_ip, rt.destination_ip, rt.total_bytes, rt.connection_count, rt.timestamp, bt.avg_bytes, (rt.total_bytes - bt.avg_bytes) / NULLIF(bt.stddev_bytes, 0) as z_score FROM recent_traffic rt LEFT JOIN baseline_traffic bt ON rt.source_ip = bt.source_ip WHERE rt.total_bytes > 100000000 -- 100MB threshold AND ( bt.avg_bytes IS NULL OR -- New source IP (rt.total_bytes - bt.avg_bytes) / NULLIF(bt.stddev_bytes, 0) > 3 -- 3 sigma deviation ) ORDER BY rt.total_bytes DESC;
Cost Optimization in Storage
During my time managing security logging for a large e-commerce platform, we learned that uncontrolled log growth could quickly consume entire security budgets. Here’s how we reduced costs by 60% while improving detection capabilities:
Intelligent Log Sampling
Not all logs need to be stored at 100% volume. I implement sampling based on risk and value:
import random import json from datetime import datetime
def is_suspicious_ip(ip_address): """Check if IP is on threat intelligence feeds""" # Implementation would check against threat intel APIs suspicious_networks = [ '203.0.113.0/24', # Documentation range used for examples '198.51.100.0/24' # Another documentation range ] return any(ip_in_network(ip_address, network) for network in suspicious_networks)
After seven years and hundreds of investigations, here are the critical lessons that separate effective security logging from mere compliance theater:
Lesson 1: Log What Matters, Not Everything
Early in my career, I made the mistake of thinking “more logs = better security.” This led to a many logging environment that was completely unmanageable. The signal-to-noise ratio was so poor that real threats were drowning in benign events.
What changed my approach: During a critical incident, we had logs showing an attacker’s presence, but our SIEM was generating ~500 alerts per day. The real alerts were buried in false positives.
Key insight: Focus on logs that directly support your detection use cases. If you can’t articulate how a log type supports threat detection, incident response, or compliance requirements, don’t collect it.
Lesson 2: Context is King
Raw logs are just data points. The magic happens when you correlate them with context:
-- Example: Enriching authentication logs with user context SELECT ct.eventTime, ct.sourceIPAddress, ct.userIdentity.userName, ct.userIdentity.arn, gd.finding_type, threat_intel.reputation, user_baseline.normal_locations, CASE WHEN ct.sourceIPAddress NOT IN (user_baseline.normal_locations) THEN 'anomalous_location' ELSE 'normal_location' END as location_assessment FROM cloudtrail_events ct LEFT JOIN guardduty_findings gd ON ct.sourceIPAddress = gd.remote_ip LEFT JOIN threat_intelligence threat_intel ON ct.sourceIPAddress = threat_intel.ip LEFT JOIN user_behavioral_baseline user_baseline ON ct.userIdentity.userName = user_baseline.username WHERE ct.eventName IN ('ConsoleLogin', 'AssumeRole') AND ct.eventTime >= NOW() - INTERVAL '1 HOUR' ORDER BY ct.eventTime DESC;
test_events = [ event for event in response['events'] if test_event_id in event['message'] ]
if not test_events: self.alert_logging_failure("CloudTrail test event not found") return False
return True
Conclusion: Building Security That Scales
As I wrap up this deep dive into AWS security logging architecture, I want to return to that fintech investigation I mentioned at the beginning. The 127-day dwell time wasn’t just a security failure — it was an architecture failure.
We had all the pieces: CloudTrail was enabled, VPC Flow Logs were configured, GuardDuty was running. But these systems were operating in isolation, generating massive volumes of data that no human could meaningfully analyze. We had logs, but we didn’t have visibility.
The architecture I’ve outlined here isn’t just about collecting more data — it’s about collecting the right data, processing it intelligently, and ensuring it serves your actual security needs. Every component, from the initial collectors to the final storage tiers, should support your detection and response capabilities.
Key Takeaways for Implementation
Start with Use Cases: Before configuring any log source, define exactly what threats you’re trying to detect. Map your logging strategy to MITRE ATT&CK techniques relevant to your environment.
Build for Scale: Your logging architecture will need to handle 10x growth. Use managed services like Kinesis and MSK that can scale automatically.
Optimize for Cost: Implement intelligent sampling, compression, and lifecycle policies from day one. Monitor costs as closely as you monitor security events.
Test Continuously: Log quality degrades over time. Build automated testing into your architecture to catch failures before they impact investigations.
Plan for the Long Term: Consider not just current storage needs, but long-term compliance requirements and the eventual need to search historical data during major incidents.
The Path Forward
Modern threat actors are patient, sophisticated, and persistent. They count on organizations having visibility gaps and exploit them ruthlessly. By implementing comprehensive, scalable security logging, we deny them the darkness they need to operate.
The architecture I’ve shared here has been battle-tested across organizations from startups to Fortune 500 companies. It’s not perfect — no architecture is — but it provides the visibility foundation that effective security programs require.
Your logging infrastructure is your security team’s eyes and ears. Invest on it accordingly.
This post represents my personal experiences and opinions. All examples use dummy data.
