CloudWatch Logs

CloudWatch Logs Overview

CloudWatch Logs enables you to centralize the logs from all your systems, applications, and AWS services that you use, in a single, highly scalable service. You can then view them, search them for specific error codes or patterns, filter them based on specific fields, or archive them to long-term storage.

Core Concepts

Log Events

A log event is a record of some activity recorded by the application or resource being monitored. It consists of a timestamp and a raw message.

Log Streams

A log stream is a sequence of log events that share the same source. For example, a log stream could be associated with an EC2 instance or a Lambda function.

Log Groups

Log groups define a group of log streams that share the same retention, monitoring, and access control settings. You can define log retention policies to automatically delete old log events.

Metric Filters

You can use metric filters to extract metric observations from ingested events and transform them into data points for a CloudWatch metric. This is a powerful way to generate metrics from application logs (e.g., counting the number of "ERROR" occurrences).

Subscription Filters

Subscription filters provide access to a real-time feed of log events and can deliver them to other services such as Amazon Kinesis, AWS Lambda, or Amazon OpenSearch Service for custom processing, analysis, or loading to other systems.

Log Sources

CloudWatch Logs can collect logs from a wide variety of sources:

AWS Services Natively integrates with services like Lambda, API Gateway, VPC Flow Logs, Route 53, and more.

EC2 Instances Use the CloudWatch Agent to push system logs (e.g., `/var/log/syslog`) and application logs to CloudWatch.

On-Premises Servers The CloudWatch Agent can also be installed on on-premises servers to centralize logs from your hybrid environment.

Custom Applications Use the AWS SDKs to publish log events directly from your applications.

CloudWatch Logs Insights

CloudWatch Logs Insights is a powerful interactive query service that enables you to explore, analyze, and visualize your logs. You can run complex queries to efficiently search log data and identify operational issues.

Example Logs Insights Query

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() as errorCount by bin(5m)
| sort @timestamp desc

This query finds all log messages containing the word "ERROR", counts them in 5-minute intervals, and displays the results in descending order by time.

Key Takeaways

1

Logs are organized into Log Groups, which contain Log Streams, which in turn contain Log Events.

2

The CloudWatch Agent is the primary method for getting logs from EC2 and on-premises servers into CloudWatch.

3

Metric Filters allow you to turn log data into actionable CloudWatch Metrics (e.g., for alarming).

4

Subscription Filters enable real-time streaming of logs to other AWS services like Lambda or Kinesis.

5

CloudWatch Logs Insights provides a powerful, ad-hoc query language for deep analysis of your log data.

6

Set retention policies on Log Groups to manage costs and comply with data retention requirements.