Overview
CloudWatch and CloudTrail answer different questions and are commonly conflated because both involve logging. CloudWatch is the operational-observability platform, metrics, logs, dashboards, and alarms describing what a system is actually doing. CloudTrail is the audit trail of AWS account activity, a record of API calls and console actions describing who did what. Every CloudTrail trail logs management events (control-plane actions) by default, but data events (object- or invocation-level activity on the resources themselves) are opt-in and cost extra, a gap that only tends to surface during an actual investigation that needed them.
Quick Reference
| Component | Answers | Logged by default |
|---|---|---|
| CloudWatch Metrics | What is the system doing | Yes, emitted automatically by applicable AWS services |
| CloudWatch Alarms | Is a threshold breached | No, explicitly created, not present by default |
| CloudWatch Logs | What did the application/service emit | Depends on whether logging is configured |
| CloudTrail management events | Who changed what configuration/control-plane state | Yes |
| CloudTrail data events | Who read/wrote a specific object or invoked a specific resource | No, opt-in, additional cost |
Syntax
aws cloudwatch put-metric-alarm \
--alarm-name high-error-rate \
--comparison-operator GreaterThanThreshold \
--threshold 0.05 --evaluation-periods 2 \
--metrics '[
{"Id": "errors", "MetricStat": {"Metric": {"Namespace": "AWS/Lambda", "MetricName": "Errors", "Dimensions": [{"Name": "FunctionName", "Value": "process-orders"}]}, "Period": 60, "Stat": "Sum"}, "ReturnData": false},
{"Id": "invocations", "MetricStat": {"Metric": {"Namespace": "AWS/Lambda", "MetricName": "Invocations", "Dimensions": [{"Name": "FunctionName", "Value": "process-orders"}]}, "Period": 60, "Stat": "Sum"}, "ReturnData": false},
{"Id": "error_rate", "Expression": "errors / invocations", "Label": "ErrorRate", "ReturnData": true}
]'Examples
-- CloudWatch Logs Insights - find the slowest 5% of requests to
-- a specific endpoint in the last hour, something no alarm or
-- pre-built dashboard was set up to answer.
fields @timestamp, @message
| filter @message like /POST \/api\/orders/
| stats pct(duration, 95) as p95 by bin(5m)CloudTrail data events are opt-in and cost extra
Enabling a trail logs management events by default, creating a role, changing a security group, but not object-level S3 access or individual Lambda invocations. If an investigation needs "who read this exact object," data events have to have been explicitly enabled beforehand, there's no retroactive way to recover that visibility.
Visual Diagram
Common Mistakes
- Confusing CloudTrail's audit trail with CloudWatch's operational data, expecting CloudTrail to explain a performance problem, or CloudWatch to explain who changed a security group.
- Assuming CloudTrail captures object-level S3 access or per-invocation Lambda activity out of the box, when only management events are on by default.
- Alarming only on CPU utilization when the real bottleneck is latency, error rate, or queue depth, custom metrics included.
- Ingesting high-volume debug-level application logs into CloudWatch Logs with no retention policy, driving up ingestion and storage cost indefinitely.
Performance
- CloudWatch metric resolution is configurable, standard 1-minute or high-resolution down to 1 second, higher resolution costs more and should be reserved for genuinely latency-sensitive alarms.
- CloudWatch Logs Insights query cost and speed scale with the volume of log data scanned; a tight time-range filter is the single biggest lever on both.
- CloudTrail log delivery to S3 has real, non-trivial delivery latency, it is an audit trail for after-the-fact investigation and compliance, not a real-time alerting mechanism.
Best Practices
- Use CloudTrail for "who did what" (compliance, audit, security investigation) and CloudWatch for "what is the system doing" (operational health); don't expect one to answer the other's questions.
- Enable CloudTrail data events explicitly for any resource where object- or invocation-level audit genuinely matters, S3 buckets and Lambda functions handling sensitive data are the common case.
- Alarm on the metric that reflects the real bottleneck, latency, error rate, queue depth, not CPU utilization by default.
- Set explicit CloudWatch Logs retention policies per log group instead of leaving the default indefinite retention in place.