Observability and Monitoring Amazon EC2 Instances with Amazon CloudWatch
Optimize your Amazon EC2 Performance
What is Amazon EC2?
Amazon EC2 (Elastic Compute Cloud) is a cloud service that allows anyone to rent servers located at different data centers around the world. This eliminates the time and upfront cost of installing servers locally. This is done so that developers can focus on development and deployment rather than on the server infrastructure and networking essentials.
What is Observability?
Observability helps you understand what's going on in a system, it helps users to understand the current state of the system/infrastructure/instance based on the external output so that users can take the required proactive action.
How does Observability help?
Let's say you run a real estate startup and one of the goals is for users to complete their KYC and make payment. Let's assume you noticed a drop in total KYC completed and a reduction in sales after your last website upgrade. You have identified that the payment gateway takes time to load up the payment modal and this in turn leads to poor customer experience, which leads to potential customers abandoning their payment. You also noticed that your 3rd Party KYC service takes time to return data. You only noticed these changes after 3 months of the website upgrade.
In the future, you want to be proactive to find out inconsistencies and inefficiencies when you scale your web app with certain architectural changes in your upgrades so that you wouldn't lose customers in the long run.
Well, good news!! Observability lets you do these things, it helps you get a good grasp of what's going on in a system and it gives you the reasons why they happen so that you and your development team can correctly prioritize the right bugs to be fixed or the right features to be developed first.
What is Monitoring?
Monitoring helps you see focused and well-selected metrics for a particular application or service. It helps in taking note of individual metrics eg cpu_time_active or memory available which enables the system to be observable.
It doesn't provide a comprehensive insight into the overall health of the system.
Observability helps us see the system from a bird's eye view while monitoring gives us tiny details about that system.
Why is Monitoring and Observability of a system important?
As you scale and add more features the more complex your system would be, and these services would need to communicate with each other.
How does the metrics of a group of services affect the efficiency of another service that interacts with that EC2 instance? A good understanding of the overall health of your system would enable you to make the right decisions and to take them fast.
Questions to ask when coming up with a good monitoring strategy
What are your goals for monitoring?
What resources will you monitor?
How often will you monitor these resources?
What monitoring tools will you use?
Who will perform the monitoring tasks?
Who should be notified when something goes wrong?
Steps to take for optimized EC2 performance
Measure your EC2 performance under different parameters, load conductions and different durations.
Store a history of the data from monitoring the system.
Compare present data with data from the previous week, month, and quarter to help you identify peak performance times, anomalies and patterns.
Come up with methods to address performance anomalies.
Some items you can monitor in your EC2 instance
CPU utilization
Network Utilization
Disk Performance
Disk Read/writes
Disk Space Utilzation
How to setup Amazon CloudWatch to Observe and monitor your EC2 instances
CloudWatch collects and converts data from your EC2 instance into readable, graphical near real-time metrics. Cloudwatch records and displays data for about 15 months so that you can have historical insight into how your service is performing.
Amazon EC2 console displays the data from CloudWatch in a graphical format. Based on your needs and how deep the insights you want to get, you can get the data from Amazon CloudWatch instead of the EC2 console.
How to turn on detailed monitoring for instances
Your instance is already set to basic monitoring, but you also have the option to enable detailed monitoring.
It's important to note that with basic monitoring data is available automatically in 5-minute periods and there is no charge, but with detailed monitoring, data is available within 1-minute periods and you are charged per metric that is sent to CloudWatch.
Required IAM permissions
To get detailed monitoring for an instance, the current user must be able to use MonitorInstances API action, based on the permission set.
Enable detailed monitoring
You can enable detailed monitoring on an instance as you launch it or after the instance is running or stopped.
When using the console, take the following steps
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
In the navigation pane, choose Instances.
Select the instance and choose Actions, Monitoring, and Manage detailed monitoring.
On the Detailed monitoring detail page, for Detailed monitoring, select the Enable check box.
Choose Save.
When using AWS CLI
Use the following monitor-instances command to enable detailed monitoring for the specified instances.
aws ec2 monitor-instances --instance-ids i-973kj9984398puh4
To enable detailed monitoring when launching an instance
Use the run-instances command with the --monitoring flag to enable detailed monitoring.
aws ec2 run-instances --image-id ami-3286970 --monitoring
Conclusion
Observability isn't a marathon, it's a journey, your plans and strategy should change and improve over time based on the historical data and insight you've gotten. Your Observability insights should be used to improve your systems and align them to your business needs and requirements. The priority should be on supporting the business needs.
Developing an observable system early on might require some time but it's a worthy investment in the long run.
If you found this article helpful please like or leave a comment, share it with your friends/network who might need this, and connect with me on Linkedin, Twitter and Dev.to.