AWS SDK API 相关

AWS SDK API 相关
SDK安装 SDK示例 API签名示例 API调用示例
2018/11/22 Chenxin
2019/06/28 更新
与阿里云比较

请问aws有没有类似阿里云的api参考帮助问题,类似以下方式: https://api.aliyun.com/?spm=a2c1g.8271268.10000.2.3f04df25RQ8QeL 可以直接给出部分参数,在线调试.并自动生成对应的SDK代码,方便直接修改.
答：目前AWS没有这种在线生成代码的工具，但通常每种SDK都会提供示例代码。

任意服务的认证和签名调用是不是都一样,有参考的么?
目前AWS服务都通过signature-version-4进行签名和认证.
这里有一个python版本的签名示例：
https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
aws签名知识参考(包含了上面的链接)
https://docs.aws.amazon.com/zh_cn/general/latest/gr/signing_aws_api_requests.html

派生签名密钥 是什么意思?

Version 到哪里去找呢
在每个API(比如EC2,或者cloudwatch)的api页面的左上角,会显示当前版本.
Amazon Elastic Compute Cloud API Reference (API Version 2016-11-15)
Amazon CloudWatch API Reference (API Version 2010-08-01)
有些操作是需要指定API版本的.比如EC2:
request_parameters = 'Action=DescribeInstances&Version=2013-10-15' 或者
request_parameters = 'Action=DescribeInstances&Version=2016-11-15' 当前都适用(2018/11/26).
而cloudwatch,如果不指定Version,会默认指到2009版本,该版本不支持DescribeAlarms这个参数.
Amazon CloudWatch API Reference (API Version 2010-08-01) 则 request_parameters = 'Action=DescribeAlarms&Version=2010-08-01'

API的调用方式说明
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Query-Requests.html GET请求的大致结构
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html POST结构的请求.是个统计数据,目前不需要
https://docs.aws.amazon.com/zh_cn/AmazonCloudWatch/latest/APIReference/making-api-requests.html cloudwatch大部分请求需要post方式
https://docs.aws.amazon.com/zh_cn/general/latest/gr/sigv4-signed-request-examples.html#sig-v4-examples-post POST签名

SDK调用cloudwatch对应参数的方式说明
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/cw-example-metrics.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.html#CloudWatch.Client.get_metric_data

SDK boto3 boto 认证配置
20181011 Chenxin
boto3官方文档 https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
boto2貌似就是boto,是2006年开发的版本.很多服务功能都没有.不过目前貌似还是有很多人使用.

安装

windows下方式(linux略)
安装python2.7,3.6
如果是API调用,aws目前更多的是使用的python2.x(尤其以2.7版本为重)版本.但最好还是使用3.X版本的python.
SDK的,貌似3.x版本支持的更好.
后安装的3.6.2版本,故系统环境变量中为3.6的版本.

安装pycharm
参见<pycharm安装>笔记.安装和破解说明

安装boto3
打开cmd命令,或者powershell,执行 pip install boto3
需要进入到对应的python安装目录下的,如C:Program Files (x86)Python36-32Scripts 下有pip.exe进行安装.
支持 Python 2 和 3. Boto3 可在 Python 版本 2.6.5+、2.7 和 3.3+ 中提供原生支持.

配置认证

windows下配置aws的认证文件
参考: http://blog.51cto.com/daibaiyang119/2146091?source=drh
powershell创建文件夹 C:Userschenxin>mkdir .aws
2个配置文件 config 和 credentials
PS C:Userschenxin.aws> cat .config
[default]
region=ap-southeast-1
PS C:Userschenxin.aws> cat .credentials
[default]
aws_access_key_id = xxx
aws_secret_access_key = xxxxxx

linux下配置aws的认证文件
CLI指令: aws configure

思考:
单个账号,对应的多个地区的EC2实例,如何通过1个脚本获取.可以做到,使用boto3.client的模块.
多个账号,对应的多个地区呢.

创建boto3的python虚拟环境(linux下.windows下的请参考pycharm笔记)
sudo pip install virtualenv # 安装虚拟环境工具
mkdir -p ~/.virtualenvs # 为虚拟环境准备文件夹
virtualenv ~/.virtualenvs/awspython # 创建虚拟环境
source ~/.virtualenvs/awspython/bin/activate # 进入虚拟环境
pip install boto # 虚拟环境下安装boto
mkdir ~/.aws
cd ~/.aws # 之后通过aws 的 cli 来创建对应的账号名,地区,密钥.

boto方式 示例-获取实例信息

boto3的client方式

import boto3
from pprint import pprint

ec2client = boto3.client('ec2',region_name = 'ap-southeast-2') # 获取悉尼的实例信息

print(dir(ec2client)) # boto3.client()有极其多的方法和属性可以使用.比如创建网关,网络接口等等.describe_instances()是其中1个.

response = ec2client.describe_instances() # 该region的所有ec2实例信息

print("response: ",response) # 是个字典.包含Reservations,ResponseMetadata这2个键

pprint(response) #格式化输出,方便查看

print("response['Reservations']: ",response['Reservations']) # 是个列表.Reservations键对应的值,是个列表.列表里面是字典.在Reservations键对应的值里的所有实例(也是嵌套的数据结构).包括key有Groups,Instances,OwnerId,ReservationId这4个.

print("dir(response['Reservations']): ",dir(response['Reservations'])) # 这是个类,不同于上面的实例对象.包含了属性和方法

for reservation in response["Reservations"]: # response["Reservations"]是个列表.字符 "Reservations" 从哪里知道的?见上
# print(reservation) # 是个字典,包括的key有Groups,Instances,OwnerId,ReservationId这4个
# print(reservation["Instances"]) #Instances这个key对应的值,也就是个列表,里面嵌套着单个实例信息的字典
# print(dir(reservation["Instances"])) #列表,属性和方法
for instance in reservation["Instances"]:
# print("instance: ",instance) #从列表中取字典,仍为单个实例信息.去掉了最外层的列表
print("instance["InstanceId"]: ",instance["InstanceId"]) #InstanceId为字典中的一个key
print("instance["Tags"]: ",instance["Tags"]) #h获取所有标签信息
instanceTags = instance["Tags"]
for instanceUerTags in instanceTags:
# print(instanceUerTags["Value"]) # 获取所有tag键值
if instanceUerTags["Key"] == "Name": #仅获取key值为Name的字典,取另外的key为Value对应的值
print("Server-name: ",instanceUerTags["Value"])
try:
print(instance["PublicIpAddress"]) #获取公网IP
except:
print("PublicIpAddress: None")
print(instance["PrivateIpAddress"]) #获取内网IP
print()

老的boto方式(不推荐)

import boto
from boto import ec2

#设置地区

conn = boto.ec2.connect_to_region('ap-southeast-1')

# 获取instance id

instanceStatus = conn.get_all_instance_status()

print(instanceStatus)

for instanceid in instanceStatus:

print(instanceid)

# 获取公网ip

WanAddress = conn.get_all_addresses()

print(WanAddress)

for ip in WanAddress:

print(ip)

# 或者这种方式获取公网IP地址

for instance in conn.get_only_instances():

print(instance.ip_address)

def getAllInstances():
instances=[]
conn = ec2.connect_to_region('ap-southeast-1')
#conn=ec2.connect_to_region('cn-north-1')
for instance in conn.get_only_instances():
# 看看instance都有哪些属性
print(dir(instance))
ins={}
ins['instance_id']=instance.id
# ins['name']=instance.tags.get('Name','').encode('gb2312')
ins['name']=instance.tags.get('Name','')
ins['tags']=instance.tags.get('SOOMI','')
ins['instance_type']=instance.instance_type
ins['state']=instance.state
ins['placement']=instance.placement
ins['private_ip']=instance.private_ip_address
ins['public_ip']=instance.ip_address
ins['vpc_id']=instance.vpc_id
ins['subnet_id']=instance.subnet_id
ins['image_id']=instance.image_id
ins['virtualization_type']=instance.virtualization_type
ins['launch_time']=instance.launch_time
ins['root_device_name']=instance.root_device_name
instances.append(ins)
return instances

if name == 'main':
instances = getAllInstances()
for i in instances:
# print(i['name'],i['public_ip'],i['private_ip'],i['instance_id'])
print(i)

API 签名 认证
aws 区域 与 终端节点
https://docs.aws.amazon.com/zh_cn/general/latest/gr/rande.html

python3 版本 signature-api-get-and-header-auth-py3.py

!/usr/bin/env python

-- coding:utf-8 --

"""

File Name: signature-api-get-and-header-auth-py3
Description:
Author: 陈信
date: 2018/11/23 16:18

"""

https://docs.aws.amazon.com/zh_cn/general/latest/gr/sigv4-signed-request-examples.html

文档中3种API签名方式.1结合使用 GET 和 Authorization 标头 + 2使用 POST + 3在查询字符串中结合使用 GET 和身份验证信息.本例中是第1种

本例适用于python3.X版本

This version makes a GET request and passes the signature

in the Authorization header.

import sys, os, base64, datetime, hashlib, hmac
import requests

import ConfigParser

import configparser

************* REQUEST VALUES *************

method = 'GET'
service = 'sns'
host = 'sns.us-east-1.amazonaws.com'
region = 'us-east-1'
endpoint = 'http://sns.us-east-1.amazonaws.com'

发邮件; %3A为冒号;参数要按照字母顺序排列

request_parameters = '
Action=Publish&Message=testMessage222
&Subject=testSubject111
&TopicArn=arn%3Aaws%3Asns%3Aus-east-1%3A615624949551%3ANotifyMe
'

Key derivation functions. See:

http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python

def sign(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
kRegion = sign(kDate, regionName)
kService = sign(kRegion, serviceName)
kSigning = sign(kService, 'aws4_request')
return kSigning

cf = configparser.ConfigParser()
cf.read("signature.conf")
access_key = cf.get("signature", "key-id")
secret_key = cf.get("signature", "key")

if access_key is None or secret_key is None:
print('No access key is available.')
sys.exit()

Create a date for headers and the credential string

t = datetime.datetime.utcnow()
amzdate = t.strftime('%Y%m%dT%H%M%SZ')
datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope

************* TASK 1: CREATE A CANONICAL REQUEST *************

Step 1 is to define the verb (GET, POST, etc.)--already done.

Step 2: Create canonical URI--the part of the URI from domain to query string (use '/' if no path)

canonical_uri = '/'

Step 3: Create the canonical query string. In this example (a GET request),

request parameters are in the query string. Query string values must

be URL-encoded (space=%20). The parameters must be sorted by name.

For this example, the query string is pre-formatted in the request_parameters variable.

canonical_querystring = request_parameters

Step 4: Create the canonical headers and signed headers. Header names

must be trimmed and lowercase, and sorted in code point order from low to high.

Note that there is a trailing .

canonical_headers = 'host:' + host + ' ' + 'x-amz-date:' + amzdate + ' '

Step 5: Create the list of signed headers. This lists the headers

in the canonical_headers list, delimited with ";" and in alpha order.

Note: The request can include any headers; canonical_headers and

signed_headers lists those that you want to be included in the

hash of the request. "Host" and "x-amz-date" are always required.

signed_headers = 'host;x-amz-date'

Step 6: Create payload hash (hash of the request body content). For GET

requests, the payload is an empty string ("").

payload_hash = hashlib.sha256(''.encode('UTF-8')).hexdigest()

Step 7: Combine elements to create canonical request

canonical_request = method + ' ' + canonical_uri + ' ' + canonical_querystring + ' ' + canonical_headers + ' ' + signed_headers + ' ' + payload_hash

************* TASK 2: CREATE THE STRING TO SIGN*************

Match the algorithm to the hashing algorithm you use, either SHA-1 or SHA-256 (recommended)

algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + ' ' + amzdate + ' ' + credential_scope + ' ' + hashlib.sha256(canonical_request.encode('UTF-8')).hexdigest()

************* TASK 3: CALCULATE THE SIGNATURE *************

Create the signing key using the function defined above.

signing_key = getSignatureKey(secret_key, datestamp, region, service)

Sign the string_to_sign using the signing_key

signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()

************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************

The signing information can be either in a query string value or in

a header named Authorization. This code shows how to use a header.

Create authorization header and add to request headers

authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' + 'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature

The request can include any headers, but MUST include "host", "x-amz-date",

and (for this scenario) "Authorization". "host" and "x-amz-date" must

be included in the canonical_headers and signed_headers, as noted

earlier. Order here is not significant.

Python note: The 'host' header is added automatically by the Python 'requests' library.

headers = {'x-amz-date': amzdate, 'Authorization': authorization_header}

************* SEND THE REQUEST *************

request_url = endpoint + '?' + canonical_querystring

print(' BEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + request_url)
r = requests.get(request_url, headers=headers)

print(' RESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d ' % r.status_code)
print(r.text)

python2 版本 signature-api-get-and-header-auth-py2.py

!/usr/bin/env python

-- coding:utf-8 --

"""

File Name: signature-api-get-and-header-auth-py2.py
Description: 一个aws api的签名示例
Author: 陈信
date: 2018/11/23 12:12

"""

https://docs.aws.amazon.com/zh_cn/general/latest/gr/sigv4-signed-request-examples.html

文档中3中API签名方式.结合使用 GET 和 Authorization 标头 + 使用 POST + 在查询字符串中结合使用 GET 和身份验证信息

本例中是第一种

本例适用于python2.X版本

This version makes a GET request and passes the signature

in the Authorization header.

import sys, os, base64, datetime, hashlib, hmac
import requests
import ConfigParser

************* REQUEST VALUES *************

method = 'GET'
service = 'sns'
host = 'sns.us-east-1.amazonaws.com'
region = 'us-east-1'
endpoint = 'http://sns.us-east-1.amazonaws.com'

发邮件; %3A为冒号;参数要按照字母顺序排列

request_parameters = '
Action=Publish&Message=testMessage222
&Subject=testSubject111
&TopicArn=arn%3Aaws%3Asns%3Aus-east-1%3A615624949551%3ANotifyMe
'

Key derivation functions. See:

http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python

def sign(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
kRegion = sign(kDate, regionName)
kService = sign(kRegion, serviceName)
kSigning = sign(kService, 'aws4_request')
return kSigning

cf = ConfigParser.ConfigParser()
cf.read("signature.conf")
access_key = cf.get("signature", "key-id")
secret_key = cf.get("signature", "key")

if access_key is None or secret_key is None:
print 'No access key is available.'
sys.exit()

Create a date for headers and the credential string

t = datetime.datetime.utcnow()
amzdate = t.strftime('%Y%m%dT%H%M%SZ')
datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope

************* TASK 1: CREATE A CANONICAL REQUEST *************

Step 1 is to define the verb (GET, POST, etc.)--already done.

Step 2: Create canonical URI--the part of the URI from domain to query string (use '/' if no path)

canonical_uri = '/'

Step 3: Create the canonical query string. In this example (a GET request),

request parameters are in the query string. Query string values must

be URL-encoded (space=%20). The parameters must be sorted by name.

For this example, the query string is pre-formatted in the request_parameters variable.

canonical_querystring = request_parameters

Step 4: Create the canonical headers and signed headers. Header names

must be trimmed and lowercase, and sorted in code point order from low to high.

Note that there is a trailing .

canonical_headers = 'host:' + host + ' ' + 'x-amz-date:' + amzdate + ' '

Step 5: Create the list of signed headers. This lists the headers

in the canonical_headers list, delimited with ";" and in alpha order.

Note: The request can include any headers; canonical_headers and

signed_headers lists those that you want to be included in the

hash of the request. "Host" and "x-amz-date" are always required.

signed_headers = 'host;x-amz-date'

Step 6: Create payload hash (hash of the request body content). For GET

requests, the payload is an empty string ("").

payload_hash = hashlib.sha256('').hexdigest()

Step 7: Combine elements to create canonical request

canonical_request = method + ' ' + canonical_uri + ' ' + canonical_querystring + ' ' + canonical_headers + ' ' + signed_headers + ' ' + payload_hash

************* TASK 2: CREATE THE STRING TO SIGN*************

Match the algorithm to the hashing algorithm you use, either SHA-1 or

SHA-256 (recommended)

algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + ' ' + amzdate + ' ' + credential_scope + ' ' + hashlib.sha256(canonical_request).hexdigest()

************* TASK 3: CALCULATE THE SIGNATURE *************

Create the signing key using the function defined above.

signing_key = getSignatureKey(secret_key, datestamp, region, service)

Sign the string_to_sign using the signing_key

signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()

************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************

The signing information can be either in a query string value or in

a header named Authorization. This code shows how to use a header.

Create authorization header and add to request headers

authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' + 'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature

The request can include any headers, but MUST include "host", "x-amz-date",

and (for this scenario) "Authorization". "host" and "x-amz-date" must

be included in the canonical_headers and signed_headers, as noted

earlier. Order here is not significant.

Python note: The 'host' header is added automatically by the Python 'requests' library.

headers = {'x-amz-date': amzdate, 'Authorization': authorization_header}

************* SEND THE REQUEST *************

request_url = endpoint + '?' + canonical_querystring

print ' BEGIN REQUEST++++++++++++++++++++++++++++++++++++'
print 'Request URL = ' + request_url
r = requests.get(request_url, headers=headers)

print ' RESPONSE++++++++++++++++++++++++++++++++++++'
print 'Response code: %d ' % r.status_code
print r.text

API的调用
已cloudwatch的当前监控报警指标为例

本例适用于python3.X版本

import sys, os, base64, datetime, hashlib, hmac
import requests
import configparser
from xml.etree import ElementTree

# ************* REQUEST VALUES *************

AWS end_point : https://docs.aws.amazon.com/zh_cn/general/latest/gr/rande.html

method = 'GET'
service = 'monitoring'

region_char = 'us-east-1'

region_char = 'ap-southeast-1'

region_char = 'ap-northeast-1'

region_char = 'eu-central-1'

region_char = 'sa-east-1'

host = 'monitoring.' + region_char + '.amazonaws.com'
region = region_char
endpoint = 'http://monitoring.' + region_char + '.amazonaws.com'

%3A为冒号;参数要按照字母顺序排列

request_parameters = 'Action=ListMetrics&Version=2010-08-01' # 返回可以监控的各个指标,包括EC2,S3,EBS等

request_parameters = 'Action=ListDashboards&Version=2010-08-01' # 返回为空

request_parameters = 'Action=DescribeAlarms&Version=2010-08-01' # 返回当前已经创建的所有监控值

request_parameters = 'Action=DescribeAlarms&NextToken=xbzj-taihe-bei-vpc-01-sub-01-game-004-mem&Version=2010-08-01' # 返回当前已经创建的所有监控值

request_parameters = 'Action=DescribeAlarms&NextToken=xbzj-taihe-bei-vpc-01-sub-01-log-to-aliyun-002&Version=2010-08-01' # 返回当前已经创建的所有监控值

Key derivation functions. See:

http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python

def sign(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
kRegion = sign(kDate, regionName)
kService = sign(kRegion, serviceName)
kSigning = sign(kService, 'aws4_request')
return kSigning

cf = configparser.ConfigParser()
cf.read("signature_Taihe.conf")
access_key = cf.get("signature", "key-id")
secret_key = cf.get("signature", "key")

if access_key is None or secret_key is None:
print('No access key is available.')
sys.exit()

Create a date for headers and the credential string

t = datetime.datetime.utcnow()
amzdate = t.strftime('%Y%m%dT%H%M%SZ')
datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope

************* TASK 1: CREATE A CANONICAL REQUEST *************

canonical_uri = '/'
canonical_querystring = request_parameters
canonical_headers = 'host:' + host + ' ' + 'x-amz-date:' + amzdate + ' '
signed_headers = 'host;x-amz-date'
payload_hash = hashlib.sha256(''.encode('UTF-8')).hexdigest()
canonical_request = method + ' ' + canonical_uri + ' ' + canonical_querystring + ' ' + canonical_headers + ' ' + signed_headers + ' ' + payload_hash

************* TASK 2: CREATE THE STRING TO SIGN*************

algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + ' ' + amzdate + ' ' + credential_scope + ' ' + hashlib.sha256(canonical_request.encode('UTF-8')).hexdigest()

************* TASK 3: CALCULATE THE SIGNATURE *************

signing_key = getSignatureKey(secret_key, datestamp, region, service)
signature = hmac.new(signing_key, string_to_sign.encode('utf-8'), hashlib.sha256).hexdigest()

************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************

authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' + 'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature
headers = {'x-amz-date': amzdate, 'Authorization': authorization_header}

************* SEND THE REQUEST *************

request_url = endpoint + '?' + canonical_querystring

print(' BEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + request_url)
r = requests.get(request_url, headers=headers)

print(' RESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d ' % r.status_code)
print(r.text)

print(type(r))

print(type(r.text))

print('++++++++++++++++++++++++++++++++++++++++++')

root = ElementTree.fromstring(r.content)
print('------member------')
for member in root.iter('{http://monitoring.amazonaws.com/doc/2010-08-01/}member'):
for sub_element_member in member.iter():
if sub_element_member.tag == "{http://monitoring.amazonaws.com/doc/2010-08-01/}MetricName":
if (sub_element_member.text == 'DiskSpaceUtilization') or (sub_element_member.text == 'MemoryUtilization'):
print(sub_element_member.text)
else:
break
if sub_element_member.tag == "{http://monitoring.amazonaws.com/doc/2010-08-01/}StateReason":
print(sub_element_member.text)
if sub_element_member.tag == "{http://monitoring.amazonaws.com/doc/2010-08-01/}AlarmArn":
print(sub_element_member.text)
print()