Sep 24, 2019

Some Command Line Tools for AWS

I’ve been working a lot in AWS lately and while I like the tech stack, going through the console to do stuff is really annoying to a command-line junkie such as myself. I could cook up my own scripts via the aws-cli or using boto3 and Python. But luckily others have already done that for me. Here are a couple of tools I use everyday.

awslogs

If I want to check the logs for my lambda function, I used to log in to the console, go to CloudWatch, find the my lambda (which is made difficult because the search bar doesn’t do substrings), click on my lambda group, and then sort by time, because it doesn’t necessarily sort by time, and then click on the link for the logs. That is exhausting.

Now I use awslogs, which does a lot of the work for me. awslogs groups will list all the existing CloudWatch groups and then awslogs get <group name> will display the latest entries. Of course you pipe it to grep or ag and search for messages. Or you can use --search in the awslogs command. You can also set a start time and/or an end time for the log messages.

This command doesn’t seem to honor the standard AWS_PROFILE environment variable, but you can use --profile <name> at the end of the awslogs command (it doesn’t seem to work anywhere else in the command, unfortunately).

s3cmd

I think of s3cmd as a little Swiss Army knife for all things S3. It does syncing, copying, tree-traversals. It even knows when to use multi-part uploads. Most of my work is in S3 lately, and I use this all the time.

My biggest problem with s3cmd is it doesn’t use AWS’s standard configs but it’s own config file. That makes switching between profiles (something I do a lot) tricky. I ended up making a wrapper script that reads ~/.aws/credentials and takes the given profile name and executes the S3 call. You can see the script here. Usage is:

s3ctl.py <profile_name> <s3cmd command>

so if you are putting fancy-file into s3://my-fancy-bucket/some-folder using your development AWS profile, it’s:

s3ctl.py development put fancy-file  s3://my-fancy-bucket/some-folder 

About the Author

Mike Hostetler profile.

Mike Hostetler

Principal Technologist

Mike has almost 20 years of experience in technology. He started in networking and Unix administration, and grew into technical support and QA testing. But he has always done some development on the side and decided a few years ago to pursue it full-time. His history of working with users gives Mike a unique perspective on writing software.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Android Development for iOS Developers
Android development has greatly improved since the early days. Maybe you tried it out when Android development was done in Eclipse, emulators were slow and buggy, and Java was the required language. Things have changed […]
Add a custom object to your Liquibase diff
Adding a custom object to your liquibase diff is a pretty simple two step process. Create an implementation of DatabaseObject Create an implementation of SnapshotGenerator In my case I wanted to add tracking of Stored […]
Keeping Secrets Out of Terraform State
There are many instances where you will want to create resources via Terraform with secrets that you just don’t want anyone to see. These could be IAM credentials, certificates, RDS DB credentials, etc. One problem […]
Validating Terraform Plans using Open Policy Agent
When developing infrastructure as code using terraform, it can be difficult to test and validate changes without executing the code against a real environment. The feedback loop between writing a line of code and understanding […]