Introduction

AWS provides you with the most powerful cloud tools on the planet. All of them are accessible via UI on the AWS website (AWS Console). However to unlock the full potential of AWS you need to master usage of the AWS Command Line tools. And once you master aws command, you will be rewarded with ability to truly automate and script all operations of your cloud infrastructure and achieve truly scalable and elastic cloud architecture.

In this guide we will look in particular at managing your DynamoDB using AWS CLI. We will learn how to perform all core operations on DynamoDB using command line:

  • create table
  • delete table
  • update table throughput (increase/decrease read/write capacity)
  • put items
  • get items

Prerequisites

AWS CLI is a command line utility written in Python. This is why you have to have Python (version 2.6.5 and above, including 3.x) and pip installed on your system.

Windows users who do not want to spend time installing Python/pip, can simply install windows MSI installer for the AWS CLI Tools as described below

Installation

To install install AWS CLI from your terminal:

# Installing 
$ pip install awscli  

# if you get exception you may try to prefix command with sudo
$ sudo pip install awscli

# Checking version
$ aws --version
aws-cli/1.14.32 Python/2.7.12 Linux/4.4.0-112-generic botocore/1.8.36


Windows users who do not have Python/pip (or do not want to mess with installation/configuration of Python) can install AWS CLI using MSI-Installer. The installer comes already bundled with Python and dependencies and after installation only takes 48Mb of disk space!!! You can download appropriate version for your OS: 64bit or 32bit.

Authentication: Generating Access Credentials

Each time you run AWS CLI command like:

$ aws dynamodb create-table --table-name myTable
# or 
$ aws ec2 start-instances --instance-ids i-0e5cd205c4a3296ea

this command will be forwarded to the AWS API, but when it’s forwarded it needs to be executed on behalf of a particular user (or role), this is why before we are able to execute commands we need provide AWS CLI with access credentials.

To generate access credentials we need to complete three steps:

  • create new user

  • attach to the user policy named AdminstratorAccess

  • generate access key and secret

For simplicity of this tutorial, we attach to our user a AdminstratorAccess policy provides second highest level of access to your account resources (where only root account has higher access).

In this video I will show How to add new Admin user via AWS Console using IAM

Authentication: Provide AWS CLI with Credentials

After we have got your credentials, in my example these are :

Key: AKIAI7HZAZFVBOSI4WKA

Secret: 370XpWIegEt2umRu3fA1Kq+R0GMtl7jjVxLYVeYU

From the first sight, credentials look like bunch of random characters, however for your convenience access keys always start with AK..... whereas secret is just a random sequence of bytes encoded with base64.

To configure AWS CLI to use those credentials by default run aws configure and when requested provide key and secret:

$ aws configure
AWS Access Key ID [None]: AKIAI7HZAZFVBOSI4WKA
AWS Secret Access Key [None]: 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
Default region name [None]: eu-west-1
Default output format [None]: json

You will be also asked to configure two more parameters:

   
Default region name Remember, that whenever you are using AWS Console (Web User Interface), your current region is always indicated in the top-right corner, next to your account name.
This means that whenever you execute a command (eg. dynamodb list-tables or ec2 start-instances it is sent to particular datacenter in a particular region ).
In this tutorial, I will use data-center closest to me: eu-west-1 but you check out complete list of AWS Regions
Default output format We will indicate that we want to use JSON format by setting value of json (lowercase!!!)

Running your first command

After successfully configuring default credentials, default region and default output format you can finally run your first command.

Let’s start by listing DynamoDB tables available on your account:

#
# if you have some tables in your region 
#
$ aws dynamodb list-tables
{
    "TableNames": [
        "images2"
    ]
}

#
# however if you don't have any tables, you will see empty output
#
$ aws dynamodb list-tables
{
    "TableNames": []
}

The resulting output of the command only lists tables within your default region. If you want to list tables within other region, you can force alternative region via command line switch: aws --region us-west-1 dynamodb list-tables

Troubleshooting

Most of the time problems with AWS CLI occur either due to the following reasons:

  • outdated version of AWS CLI

  • wrong version of Python/dependencies

  • incorrectly configured credentials.

You can find some methods and strategies to diagnose and fix those errors in article Troubleshooting AWS CLI

Now let’s use it

Listing tables

$ aws dynamodb list-tables
{
    "TableNames": [
        "images-blog", 
        "images2"
    ]
}

Creating table

AWS CLI syntax is annoyingly verbose, and noisy. This is why let me start explanation with the “pseudo- command”

#!/bin/bash
aws dynamodb create-table \
  --columns Name=photo_id,Type=S \          # create with single column, of type STRING
  --table-name images-blog \              
  --keys Name=photo_id,Type=HASH \          # this will be the key for the key-value store 											        # WARNING: this is NOT similar as adding
  				 					        # regular INDEX or UNIQUE in SQL
  --provisioned-throughput Read=1,Write=1   # you need to think of throughput from step 1

The “pseudocommand” above is pretty self explanatory.

Before I show you the final command, I want you to take a look at yet another “pseudocmmand”, but this time I made it resemble real command even more. The comments show two changes

#!/bin/bash
aws dynamodb create-table \
  --attribute-definitions Name=photo_id,Type=S \  # --columns  >>> --attribute-definitions
  --table-name images-blog \
  --key-schema Name=photo_id,Type=HASH \		  # --keys     >>> --key-schema
  --provisioned-throughput Read=1,Write=1

As you can see command line parameter to define columns is called --attribute-definitions and one to define keys is called --key-schema. This is a bit verbose, but still readable.

Now let’s take a look at the real command:

#!/bin/bash
aws dynamodb create-table \
  --attribute-definitions AttributeName=photo_id,AttributeType=S \
  --table-name images-blog \
  --key-schema AttributeName=photo_id,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

There’re three changes:

  • AttributeName and AttributeType
  • KeyType and AttributeType
  • ReadCapacityUnits

Deleting Table

$ aws dynamodb delete-table --table-name images-blog
{
    "TableDescription": {
        "TableArn": "arn:aws:dynamodb:eu-west-1:326625054455:table/images-blog", 
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0, 
            "WriteCapacityUnits": 1, 
            "ReadCapacityUnits": 1
        }, 
        "TableSizeBytes": 0, 
        "TableName": "images-blog", 
        "TableStatus": "DELETING", 
        "TableId": "3d915874-a472-463e-bbb0-38f0d5b365a8", 
        "ItemCount": 0
    }
}

Changing table provisioned throughput

Well sometimes if you have created table with some capacity units, you may want to resize. You can achive this with the update-table subcommand:

$ aws dynamodb update-table \
    --table-name images2 \
    --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

YOU CAN DOWNSCALE TABLE THROUGHPUT ONLY UP TO 9 TIMES A DAY

You can increase table throughput any amount of times per day, but you can decrease table throughput only about 9 times a day.

INCREASING THROUGHPUT CAN INCREASE YOUR BILL DRASTICALLY

Keep in mind that DynamoDB is favored by companies who spend six-seven digit figures on AWS bills monthly and support sometimes millions of requests per second. This means that DynamoDB can become extremely powerful at extremely high cost.

In the table below you can see the estimated monthly cost of running DynamoDB depending on the provisioned throughput.

When you see 10 Read Capacity units - that simply means 10 read requests/sec. From the images below you can see that 1000 read requests/sec will cost you $656 p/m and if you happen to set capacity to 10,000 requests/sec this will cost you over $6,500. So keep an eye on your provisioned throughput and be sure to set up SNS alarms to inform you in case you go over your budget.

Where 10 RCU means 10 reads/sec, 10 WCU actually means 5 writes/sec. In this case AWS follows common sense logic that reads are “cheaper” in terms of CPU or IO-time rather than writes.


ddb-provisioned-capacity-10-10


ddb-provisioned-capacity-1000-1000


ddb-provisioned-capacity-10000-10000


Getting help on commands

If you want to list all the commands available at aws dynamodb namespace, run:

# Opens man page with list of all commands and description (even on Windows!)
aws dynamodb help

If you don’t want full blown help page, but just a shortlist of available commands, just type some characters after aws dynamo <some-characters-here> and you will get command shortlist as shown below:

$ aws dynamodb a
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument operation: Invalid choice, valid choices are:

batch-get-item                           | batch-write-item                        
create-backup                            | create-global-table                     
create-table                             | delete-backup                           
delete-item                              | delete-table                            
describe-backup                          | describe-continuous-backups             
describe-global-table                    | describe-limits                         
describe-table                           | describe-time-to-live                   
get-item                                 | list-backups                            
list-global-tables                       | list-tables                             
list-tags-of-resource                    | put-item                                
query                                    | restore-table-from-backup               
scan                                     | tag-resource                            
untag-resource                           | update-global-table                     
update-item                              | update-table                            
update-time-to-live                      | wait                                    
help                                    

Making command output easier to read

Throughout this article, whenever we were running aws command the results of the command were displayed in JSON format. This is consequence of setting default output to json during aws configure step.

However, often to make sense of the results, you will want to force particular output format. This can be easily done with the --output switch. Here are some examples:

Text output

Text output is least verbose, often easy to read and perfect for parsing using awk

#
# simple
#
$ aws --output=text dynamodb list-tables 
TABLENAMES	images2

#
# verbose
#
$ aws --output=text ec2 describe-regions
REGIONS ec2.ap-south-1.amazonaws.com    ap-south-1
REGIONS ec2.eu-west-3.amazonaws.com     eu-west-3
REGIONS ec2.eu-west-2.amazonaws.com     eu-west-2
REGIONS ec2.eu-west-1.amazonaws.com     eu-west-1
REGIONS ec2.eu-central-1.amazonaws.com  eu-central-1
...
REGIONS ec2.us-east-1.amazonaws.com     us-east-1
REGIONS ec2.us-east-2.amazonaws.com     us-east-2
REGIONS ec2.us-west-1.amazonaws.com     us-west-1
REGIONS ec2.us-west-2.amazonaws.com     us-west-2

Table output

Another output format supported by the AWS CLI is called table.

The table output format promises to be the most-human readable format of all and something what you will use every day. In practice, however, I found that this format is least useful.

The main challenge reading output of aws command is handling very verbose commands. It is not unusual for a command to generate 2-3 screens of output with multiple levels of nesting. You can filter text output using grep, you can filter json output using jq or built-in jmespath, but you CANNOT filter table format :(

Nevertheless we will look at some examples of using table format below:

#
# even output for trivial command looks verbose as `table` 
#
$ aws --output=table dynamodb list-tables 
---------------
| ListTables  |
+-------------+
||TableNames ||
|+-----------+|
||  images2  ||
|+-----------+|

#
# trivial command, looks pretty, but eats vertical screen-real estate
#
$aws --output=table ec2 describe-regions
----------------------------------------------------------
|                     DescribeRegions                    |
+--------------------------------------------------------+
||                        Regions                       ||
|+-----------------------------------+------------------+|
||             Endpoint              |   RegionName     ||
|+-----------------------------------+------------------+|
||  ec2.ap-south-1.amazonaws.com     |  ap-south-1      ||
||  ec2.eu-west-3.amazonaws.com      |  eu-west-3       ||
||  ec2.eu-west-2.amazonaws.com      |  eu-west-2       ||
||  ec2.eu-west-1.amazonaws.com      |  eu-west-1       ||
||  ec2.ap-northeast-2.amazonaws.com |  ap-northeast-2  ||
||  ...                              |  ...             ||
||  ec2.us-east-1.amazonaws.com      |  us-east-1       ||
||  ec2.us-east-2.amazonaws.com      |  us-east-2       ||
||  ec2.us-west-1.amazonaws.com      |  us-west-1       ||
||  ec2.us-west-2.amazonaws.com      |  us-west-2       ||
|+-----------------------------------+------------------+|

JSON Output

One of the most flexible and useful formats is JSON, and you can force this format by setting output value to --output=json

$ aws --output=json dynamodb list-tables 
{
    "TableNames": [
        "images2"
    ]
}