跳到主要内容
翻译页面

本页面由 AI 辅助从英文翻译而来。如两者存在差异，以英文版本为准。

AWS 全球资源清单

在本指南中，我们将演示如何使用 StackQL 生成指定 AWS 账号内 所有区域、所有服务 中 所有资源 的完整清单。StackQL 是一款功能强大的开发工具，可使用 SQL 语法查询和部署云基础设施与资源。

测试环境：嵌入式 SQL 后端 postgres SQL 后端 macos linux powershell

工作原理​

StackQL 实现了一个可以异步处理多个查询的 SQL 引擎，例如：

SELECT region, COUNT(*) as num_functions
FROM aws.lambda.functions
WHERE region IN (
	'us-east-1','us-east-2','us-west-1','us-west-2',
	'ap-south-1','ap-northeast-3','ap-northeast-2',
	'ap-southeast-1','ap-southeast-2','ap-northeast-1',
	'ca-central-1','eu-central-1','eu-west-1',
	'eu-west-2','eu-west-3','eu-north-1','sa-east-1')
GROUP BY region;

上面的查询实际上是并行运行的 17 个查询（每个区域一个查询），其结果经聚合后以报告的形式呈现。使用 StackQL 的 Python 封装库 pystackql，我们可以遍历 所有服务 中的 所有资源，覆盖相应服务支持的 所有区域。最终得到一份报告，列出每种资源在每个区域中部署的实例数量，如下例所示：

AWS 资源清单示例

前提条件​

  • pystackql，可使用 pip install pystackql 安装
  • 一个附加了 ReadOnlyAccess 或更高权限策略的 AWS 用户或角色（可使用 sts assume-role 担任）
  • 已设置 AWS_ACCESS_KEY_ID 和 AWS_SECRET_ACCESS_KEY 环境变量（可通过 AWS Cloud Shell 自动完成）

代码​

以下代码执行如下操作：

  1. 定义所有 AWS 区域的列表（regions）以及每个服务所支持区域的列表（supported_regions），有关如何得出 supported_regions 的更多信息，请参阅 AWS Service Support by Region at a Glance
  2. 拉取 aws 和 aws StackQL Provider（如果尚未安装）。
  3. 遍历每个服务（使用 SHOW SERVICES），然后针对每个服务遍历每个资源（使用 SHOW RESOURCES）
  4. 使用 SELECT 异步查询所有区域中每个资源的全部实例，按区域对数据分组，并统计每个区域中每个资源的实例数量

代码如下……

from regions import supported_regions, regions
from resources import excluded_resources, resource_filters
import time
from pystackql import StackQL
import pandas as pd

stackql = StackQL(output='pandas', page_limit=-1, execution_concurrency_limit=-1)

def pull_stackql_providers():
    providers_df = stackql.execute("SHOW PROVIDERS")
    installed = providers_df['name'].values if not providers_df.empty else []
    for provider in ('aws', 'awscc'):
        if provider not in installed:
            print(f"{provider} not installed. Installing {provider}")
            res = stackql.executeStmt(f"REGISTRY PULL {provider}")
            print(res['message'])

def get_s3_buckets():
    # special case for s3 buckets - use the awscc provider's list view, which
    # returns each bucket alongside its region in a single query across all regions
    regions_in = ", ".join([f"'{region}'" for region in regions])
    buckets_df = stackql.execute(f"""
        SELECT region, bucket_name
        FROM awscc.s3.buckets_list_only
        WHERE region IN ({regions_in});
    """)
    # Group by region and count the total resources (buckets) in each region
    grouped = buckets_df.groupby('region').size().reset_index(name='total_resources')
    grouped['svc'] = 's3'
    grouped['res'] = 'buckets'
    output_df = grouped[['svc', 'res', 'region', 'total_resources']]
    return output_df

def main():
    start_time = time.time()
    pull_stackql_providers()    
    results_df = pd.DataFrame(columns=['svc', 'res', 'region', 'total_resources'])
    # get all enumerable aws services
    services_df = stackql.execute("SHOW SERVICES IN aws")
    for svcIx, svcRow in services_df.iterrows():
        service = svcRow['name']
        if service in supported_regions:
            # check all enumerable aws resources in the service against supported regions
            resources_df = stackql.execute(f"SHOW RESOURCES IN aws.{service}")
            plural_resources = [resource for resource in resources_df['name'].tolist() if resource.endswith('s')]
            if plural_resources:
                for resource in plural_resources:
                    print(f"Checking {resource} in {service}...")
                    if service == 's3' and resource == 'buckets':
                        print(f"Checking location for s3 buckets...")
                        resource_df = get_s3_buckets()
                    else:
                        # should we skip this service?
                        if service in excluded_resources and resource in excluded_resources[service]:
                            continue
                        # check if the resource is global or regional
                        if 'global' in supported_regions[service]:
                            regions_in = ", ".join([f"'{region}'" for region in regions])
                        else:
                            regions_in = ", ".join([f"'{region}'" for region in supported_regions[service]])      
                        
                        # check if there are additional where clauses for this resource
                        where_clause = resource_filters.get(service, {}).get(resource, '')
                        
                        resource_df = stackql.execute(f"""
                            SELECT '{service}' as svc, '{resource}' as res, region, COUNT(*) as total_resources  
                            FROM aws.{service}.{resource} 
                            WHERE region IN ({regions_in}){where_clause}
                            GROUP BY svc, res, region
                        """)

                    if not resource_df.empty:
                        # Append resource_df to the results_df
                        if not 'error' in resource_df.columns:
                            results_df = pd.concat([results_df, resource_df], ignore_index=True)
    end_time = time.time()
    elapsed_time = end_time - start_time
    print(f"Elapsed time: {elapsed_time:.2f} seconds")
    results_df.to_csv('all_aws_resources.csv', index=False)

if __name__ == "__main__":
    main()