跳至主要內容
翻譯頁面

本頁面由英文在 AI 協助下翻譯而成。若兩者有出入,請以英文版本為準。

AWS 全球資源清單

在本指南中,我們將示範如何使用 StackQL(一款可使用 SQL 語法查詢及部署雲端基礎架構與資源的強大開發工具),針對指定 AWS 帳戶中所有區域、所有服務的所有資源產生完整的資源清單。

測試環境:內嵌式 SQL 後端 postgres SQL 後端 macos linux powershell

運作方式​

StackQL 實作了可以非同步處理多個查詢的 SQL 引擎,例如:

SELECT region, COUNT(*) as num_functions
FROM aws.lambda.functions
WHERE region IN (
'us-east-1','us-east-2','us-west-1','us-west-2',
'ap-south-1','ap-northeast-3','ap-northeast-2',
'ap-southeast-1','ap-southeast-2','ap-northeast-1',
'ca-central-1','eu-central-1','eu-west-1',
'eu-west-2','eu-west-3','eu-north-1','sa-east-1')
GROUP BY region;

上述查詢實際上是 17 個平行執行的查詢(每個區域一個查詢),結果會經過彙總並以報表呈現。使用 pystackql(StackQL 的 Python 包裝程式庫),我們可以針對指定服務所支援的所有區域,逐一處理所有服務中的所有資源。結果是一份報表,列出每種資源在每個區域中已部署的執行個體數量,如下方範例所示:

AWS 資源清單範例

必要條件​

  • pystackql,可使用 pip install pystackql 安裝
  • 已附加 ReadOnlyAccess 政策或更高權限的 AWS 使用者或角色(可使用 aws sts assume-role 擔任角色)
  • 已設定 AWS_ACCESS_KEY_ID 與 AWS_SECRET_ACCESS_KEY 環境變數(可使用 AWS CloudShell 自動完成)

程式碼​

下列程式碼會執行以下動作:

  1. 定義所有 AWS 區域的清單(regions),以及每項服務支援的區域清單(supported_regions);如需推導 supported_regions 的詳細資訊,請參閱 AWS Service Support by Region at a Glance
  2. 如果尚未安裝 aws 與 awscc StackQL Provider,則提取這兩個 Provider。
  3. 逐一處理每項服務(使用 SHOW SERVICES),再針對每項服務逐一處理每個資源(使用 SHOW RESOURCES)
  4. 以非同步方式跨所有區域選取每個資源的所有執行個體(使用 SELECT),依區域將資料分組,並計算每個區域中每個資源的執行個體數量

程式碼如下:

from regions import supported_regions, regions
from resources import excluded_resources, resource_filters
import time
from pystackql import StackQL
import pandas as pd

stackql = StackQL(output='pandas', page_limit=-1, execution_concurrency_limit=-1)

def pull_stackql_providers():
providers_df = stackql.execute("SHOW PROVIDERS")
installed = providers_df['name'].values if not providers_df.empty else []
for provider in ('aws', 'awscc'):
if provider not in installed:
print(f"{provider} not installed. Installing {provider}")
res = stackql.executeStmt(f"REGISTRY PULL {provider}")
print(res['message'])

def get_s3_buckets():
# special case for s3 buckets - use the awscc provider's list view, which
# returns each bucket alongside its region in a single query across all regions
regions_in = ", ".join([f"'{region}'" for region in regions])
buckets_df = stackql.execute(f"""
SELECT region, bucket_name
FROM awscc.s3.buckets_list_only
WHERE region IN ({regions_in});
""")
# Group by region and count the total resources (buckets) in each region
grouped = buckets_df.groupby('region').size().reset_index(name='total_resources')
grouped['svc'] = 's3'
grouped['res'] = 'buckets'
output_df = grouped[['svc', 'res', 'region', 'total_resources']]
return output_df

def main():
start_time = time.time()
pull_stackql_providers()
results_df = pd.DataFrame(columns=['svc', 'res', 'region', 'total_resources'])
# get all enumerable aws services
services_df = stackql.execute("SHOW SERVICES IN aws")
for svcIx, svcRow in services_df.iterrows():
service = svcRow['name']
if service in supported_regions:
# check all enumerable aws resources in the service against supported regions
resources_df = stackql.execute(f"SHOW RESOURCES IN aws.{service}")
plural_resources = [resource for resource in resources_df['name'].tolist() if resource.endswith('s')]
if plural_resources:
for resource in plural_resources:
print(f"Checking {resource} in {service}...")
if service == 's3' and resource == 'buckets':
print(f"Checking location for s3 buckets...")
resource_df = get_s3_buckets()
else:
# should we skip this service?
if service in excluded_resources and resource in excluded_resources[service]:
continue
# check if the resource is global or regional
if 'global' in supported_regions[service]:
regions_in = ", ".join([f"'{region}'" for region in regions])
else:
regions_in = ", ".join([f"'{region}'" for region in supported_regions[service]])

# check if there are additional where clauses for this resource
where_clause = resource_filters.get(service, {}).get(resource, '')

resource_df = stackql.execute(f"""
SELECT '{service}' as svc, '{resource}' as res, region, COUNT(*) as total_resources
FROM aws.{service}.{resource}
WHERE region IN ({regions_in}){where_clause}
GROUP BY svc, res, region
""")

if not resource_df.empty:
# Append resource_df to the results_df
if not 'error' in resource_df.columns:
results_df = pd.concat([results_df, resource_df], ignore_index=True)
end_time = time.time()
elapsed_time = end_time - start_time
print(f"Elapsed time: {elapsed_time:.2f} seconds")
results_df.to_csv('all_aws_resources.csv', index=False)

if __name__ == "__main__":
main()