AWS Resource tracker

In this article, we will build a shell script project and I will describe how things work in this project.

Problem statement

Let's assume you are working for an organization as a DevOps engineer and that organization is using AWS as its cloud provider. Your manager assigned the task of giving the report of all the resources used by their staff/employees every day at some particular time.

What we will be building

Write a shell script that appends the report to a file.
Automating the shell script to report daily.

Tools / Tech Stacks

Shell / Bash scripting (I used ubuntu as my Linux distribution)
Aws-cli
Cronjob (for automation)

Prerequisites

A Linux machine,

[ Here I used a virtual machine on top of the Windows using a VMware workstation ]
Aws-cli installed and configured to your account,

[ I have used my account for this demo, for getting started checkout AWS Documentation. ]
Cronjob installed in your distribution,

[ Many distributions come in handy by a pre-installing cronjob, check if its service is running or not by using,
```
  systemctl status cron
```
If it is active cron is running properly otherwise start your cron service
```
  service cron start
```
You may need to have root access for that use sudo.]

Writing a shell script

For this script, I have created a separate directory to store the script and report that we generate, and created a shell script file name tracker.sh .

Making a directory and changing to that directory:

mkdir aws-demo && cd aws-demo

Creating a script file named tracker with .sh extension which specifies shell script.

touch tracker.sh

After the creation of the file, check whether your AWS is configured or not.

cat ~/.aws/credentials

If this shows you aws_access_key_id and aws_secret_access_key , then you have configured otherwise configure it.

Now let's start writing the script file, for that I will be using the vim editor but you can use any of the other editors. Open the file by using vim tracker.sh

Every bash script should start with the hash-bang #!/bin/bash , this is used to invoke the specified shell that executes the commands which are in the script.

After that, you can have metadata like Author, Date, Version, and About the script.

If you want to debug your script like want to know what is happening while executing the script you can use set -x which enables debugging feature for your script file

Here, you are reporting to your manager every day which means at times you may confuse about data so that we can mention the date and time in the first line of the script. With `date` , we will get the date, time, and week in our output.

Since we need to generate the report, so the output should be in file format.

I will append every output line to the Report.txt file by using >> .

Now we will implement the actual task of listing all the AWS resources. Since there were many resources an organization uses and that too varies from org to org. Here I will take only 4 services those were ec2 instances,s3 buckets, lambda functions, and IAM users.

Their listing commands will be,

EC2 instances: aws ec2 describe-instances
S3 buckets: aws s3 ls
Lambda functions: aws lambda list-functions
IAM users: aws iam list-users

Since this IAM users list command gives us some personal information so we will trim that output with JSON parsing( jq ) and give the output only the UserName.

Command after trimming will be: aws iam list-user | jq '.Users[].UserName' . We used '[]' since 'Users' is a list and '.' is the object denotation.

Script is,

#!/bin/bash
##################################################
# Author: Shiva Abhishek                         #
# Date:                                          #
# Version:v1                                     #
# This script reports the AWS report usage       #
# like EC2 instances,s3 buckets,lambda funtions, #
# IAM users.                                     #
##################################################
#Printing Date and time
echo "On" `date` "report is" >> ./Report.txt

echo "List of s3 buckets" >> ./Report.txt
aws s3 ls >> ./Report.txt

echo "List of EC2 instances" >> ./Report.txt
aws ec2 describe-instances >> ./Report.txt

echo "List of Lambda-Functions" >> ./Report.txt
aws lambda list-functions >> ./Report.txt

echo "List of IAM users" >> ./Report.txt
aws iam list-users | jq '.Users[].UserName' >> ./Report.txt

After saving this file, make it executable by changing its permissions chmod +700 tracker.sh .

Now execute it with ./tracker.sh command, then it runs the script and appends the output to the Report.txt file since we appended the output to that file.

Since I have used my personal account that is the reason there are no reports i.e, there are no ec3 instances running, s3 buckets and lambda functions are active, and I am the only user(I have created it) it is showing my UserName.

Integrating with Cronjob

Basically, cronjob is for automating scripts.

It has the basic syntax/format. What we have to do is we have to mention the on what time/date/day we require that script to run and the path to that script.

It basically contains 5 parameters for automation time and 1 is for a path to the script which denotes in this way,

[Minutes] [Hours] [Days] [Month] [Weeday] [path to the script]
    *        *       *      *       *      /path

Here we need to type the command crontab -e which opens the cronjob file in edit mode, there we have to give the parameter and path to the file.

I wrote it to that cronjob and gave the automating at 7 PM/19 IST in hour parameter and gave everything remaining as an asterisk(*) that means irrespective of any minute/day/month/weekday run this script at 7 PM IST.

We can see the list of corn jobs using crontab -l .

Hence every day at 7 PM IST, this script will run and the output is appended to Report.txt.

Just for demo purposes for the sake of showing the example, I assigned the crontab to automatically give the report for every hour. So that I have edited the crontab file and made it like in the below image,

Result is

Hope you learned something new from this blog, feedback is appreciated.

Happy learning😊.

Thank you.