Skip to main content

33 posts tagged with "stackql"

View All Tags

Auditing three clouds without writing three scripts

· 6 min read
Nirmal Chhodvadiya
Cloud Consultant and AWS Community Builder

If you've audited buckets in more than one cloud, you already know what this is like. Three consoles. Three CLIs. Three auth patterns. And a bit of glue code to pull the outputs together every time someone asks a question that spans all three.

The question is usually simple. "Which buckets are public across all our clouds?" You end up running three separate scripts, then reconciling three different output shapes just to give one answer.

This tutorial doesn't make that faster. It replaces it. One query, one view, all three clouds.

What StackQL is, in three sentences​

StackQL is SQL for cloud APIs. You write a normal SQL query, and it makes the API calls to AWS, GCP, Azure, or dozens of other providers to get the answer back as rows and columns.

The important part for this tutorial: it queries the live provider APIs on every run. There's no local database, no cache, no sync job to schedule. What you see is the current state of your cloud, right now.

It exists because someone got tired of writing the same audit logic three times.

What you'll need​

You'll need Docker installed and read-only credentials for whichever clouds you want to audit. All three are optional. If you leave a cloud's credentials blank, it gets skipped.

For AWS, GCP, and Azure, the audit needs a service account or role that can list storage buckets and read their configuration. The audit-action repo has the exact permissions per provider here.

Run the audit​

The tutorial folder has three files: a docker-compose file, an .env.audit.example template, and a README. Grab the folder from docs/tutorials/preview-bucket-01/ in the stackql/stackql repo.

1. Copy the env template

cp ./audit/.env.audit.example ./audit/.env.audit

Then open .env.audit and fill in credentials for whichever clouds you want. The variables are named for what they are: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, GOOGLE_CREDENTIALS (the service account JSON, on one line), and GOOGLE_ORG_ID.

If you're only auditing one or two clouds, leave the others blank.

2. Pull the StackQL Docker image

docker compose -f docker-compose.bucket.audit.yaml pull

The tutorial pins to stackql/stackql:v0.10.601. The audit uses views under stackql_preview.* which are still evolving, so pinning matters for reproducibility.

3. Run the audit

docker compose -f docker-compose.bucket.audit.yaml run --rm stackql

On a modest account this takes about 30 seconds. Output goes straight to your terminal as a table.

What the output tells you​

When the audit finishes, you get a single table printed to your terminal. Something like this (bucket names are placeholders):

Terminal screenshot showing StackQL audit output as a table with columns for provider, bucket name, encryption class, public flag, and HTTPS enforcement. Rows include AWS, GCP, and Azure buckets with a mix of provider-managed and customer-managed encryption, and one public bucket flagged per AWS and GCP.

Example output from the audit. Bucket configuration across three cloud providers in one normalised table.

The point isn't the specific findings. It's the shape.

Three different resource types (S3 buckets, GCS buckets, and Azure Storage Accounts) normalised into one table with the same columns. Encryption class, public flag, HTTPS enforcement. Side by side, for every cloud you provided credentials for.

A few patterns to notice in the table:

  • Public buckets across providers. Filter by public = true and you have the answer to your security team's question, in every cloud, in one place.
  • Encryption side by side. The encryption_class column shows which buckets use provider-managed keys versus customer-managed. Different providers have different naming conventions internally; StackQL normalises them to the same two categories.
  • HTTPS enforcement in one column. Whether it's S3 bucket policies, GCS uniform access, or Azure secure transfer, same column, same values, no translation needed.

This is what SQL for cloud APIs actually delivers. Not just SQL as a query language, but a uniform data model across providers, so you can ask one question and get one answer.

Performance and scope​

StackQL runs each provider's query in parallel using the credentials you provided. There's no local database being populated and no background sync; every run hits the live APIs and returns what's there right now.

On the test account this took about 30 seconds. The query is scoped to one AWS region and one GCP organization, which keeps the API footprint bounded and reflects the current preview default. Streaming output and broader audits (across regions, deeper across resource types) are in flight from the StackQL team.

What this covers today​

Today, this audit covers storage buckets across AWS, GCP, and Azure. One region at a time for AWS, one organization at a time for GCP. Entitlements, IAM, and other resource types are on the roadmap as separate audits under stackql_preview.* and will get their own tutorials as they land.

Where to go next​

github.com/stackql/stackql is the flagship repo. If this tutorial was worth reading, give it a star. It's genuinely the thing that tells the team to build more tutorials like this one, and it helps other engineers with the same three-console problem find the project.

A few natural next steps from here:

  • Run it against your own accounts. The audit works exactly the same way against real credentials. You can start with one cloud and add the others by adding their env vars.
  • Move it into CI. The stackql-audit-action repo has ready-to-use GitHub Actions workflows for all-clouds, single-cloud, deep-audit, and OIDC-authenticated variants. Drop one into your repo, add secrets, and you have a scheduled cross-cloud audit running on every push.
  • Ask questions. The StackQL community Slack is the fastest way. GitHub issues on the flagship repo also work.

More audits are on the way, including entitlements, deeper checks, and larger estates. If there's a specific one you'd find useful, opening an issue is the fastest way to get it on the roadmap.

From finding to fix: cloud auto-remediation with AI and StackQL

· 7 min read
Kieran Rimmer
Technologist and Cloud Consultant

A cloud audit tells you what is wrong. The work starts when you have to fix it. Most tooling stops at the findings list and hands a spreadsheet to an engineer, and the findings sit there until someone has a quiet afternoon.

This post walks through the other half: a remediation loop that turns each finding into a reviewable pull request, verifies live state before it changes anything, and applies the fix on merge. It runs entirely in GitHub Actions, authenticates with OIDC, and uses StackQL to talk to cloud control planes. The repo is public at stackql-labs/stackql-ai-remediation, and the example throughout is FinOps waste (unattached disks, idle IPs, zero-VM projects), though the shape is the same for posture and security checks.

Autonomous Edge Defense with AI Agents and SQL

· 6 min read
Jeffrey Aven
Technologist and Cloud Consultant

Edge defense is a natural fit for an agentic loop: traffic patterns shift constantly, rate-limit thresholds need to follow, and every adjustment should leave an auditable trail. The hard part is usually the plumbing - one API for analytics, another for the rate-limit control plane, another for the durable log.

edgepilot collapses all of that to SQL. Two Claude agents - a recon agent and an action agent - observe a live Cloudflare zone, tighten its rate-limit rule when traffic warrants, and write a decision record to a Confluent Kafka topic. Neither agent knows anything about Cloudflare's GraphQL Analytics API, Cloudflare's rulesets engine, or Confluent's Kafka REST proxy. They know SQL. The StackQL MCP server does the rest.

Run StackQL Queries from the Databricks Web Terminal

· 2 min read
Jeffrey Aven
Technologist and Cloud Consultant

If you have access to a Databricks workspace, you can run StackQL queries directly from the Databricks Web Terminal using your Databricks identity.

How It Works​

Download the latest release of stackql, then run the convenience script included (similar scripts are included for other cloud provider terminals - e.g. AWS Cloud Shell).

curl -L https://bit.ly/stackql-zip -O && unzip stackql-zip
sh stackql-databricks-shell.sh

Example Queries​

Here are the sample queries run in the video, just change the deployment_name for your workspace.

User entitlements

SELECT
  deployment_name,
  id,
  userName,
  displayName,
  entitlement
FROM databricks_workspace.iam.vw_user_entitlements
WHERE deployment_name = 'dbc-74aa95f7-8c7e';

All workspace settings

SELECT * FROM
  databricks_workspace.settings.vw_all_settings
WHERE deployment_name = 'dbc-74aa95f7-8c7e';

Tag policies filtered by key prefix

SELECT
  tag_key as key,
  description
FROM databricks_workspace.tags.tag_policies
WHERE deployment_name = 'dbc-74aa95f7-8c7e'
AND tag_key LIKE 'class%';

Catalog count by type

SELECT
  catalog_type,
  COUNT(*) as num_catalogs
FROM databricks_workspace.catalog.catalogs
WHERE deployment_name = 'dbc-74aa95f7-8c7e'
GROUP BY catalog_type;

Provider Coverage​

The databricks_workspace provider covers workspace related services, the databricks_account provider covers account-level operations including provisioning, billing, and account IAM.

The web terminal flow covers workspace-scoped queries using the token of the logged-in user. For account-level queries (provisioning, billing, account IAM), you need a Databricks service principal with account admin rights and OAuth2 credentials:

export DATABRICKS_ACCOUNT_ID="your-account-id"
export DATABRICKS_CLIENT_ID="your-client-id"
export DATABRICKS_CLIENT_SECRET="your-client-secret"

These are the same variables used by the Databricks CLI and Terraform provider, so if you already have those configured the auth story is identical.

Get Started​

Full provider documentation:

Visti StackQL on GitHub.

(Quickly) Identify Old Node Runtimes in AWS Lambda

· 3 min read
Jeffrey Aven
Technologist and Cloud Consultant

Have you been sent one of these?

[Action Required] AWS Lambda end of support for Node.js 18 [AWS Account: 824123456789] [EU-CENTRAL-1]

If you are like me and manage AWS accounts with numerous Lambda functions potentially deployed across multiple regions, you need to identify affected resources, in this case, Lambda node runtimes, which will be discontinued later this year.  

With stackql this task is easy...

  1. Open AWS cloud shell in your AWS account (any region - it doesn't matter)
  2. Download stackql
curl -L https://bit.ly/stackql-zip -O && unzip stackql-zip
  1. Open an authenticated stackql command shell
sh stackql-aws-cloud-shell.sh
  1. Run some analytic queries using stackql; here are some examples...

🔍 List all functions and runtimes across regions​

Run a stackql query to get the details about functions, runtimes, etc, deployed at any given time across one or more AWS regions.  You can include all 25 AWS regions; each query will be performed asynchronously - speeding up the results.

select 
  function_name, 
  region,
  runtime 
FROM aws.lambda.functions 
WHERE region IN ('us-east-1', 'eu-west-1');

📊 Group by runtime and region​

Perform an analytic query like a group by aggregate query such as...

select 
  runtime, 
  region, 
  count(*) as num_functions 
FROM aws.lambda.functions 
WHERE region IN ('us-east-1', 'eu-west-1', 'ap-southeast-2')
GROUP BY runtime, region;
tip

You can easily visualise this data using a notebook; see stackql-codespaces-notebook or stackql-jupyter-demo.

Using StackQL you can:

  • Quickly spot functions running on runtimes like nodejs18.x that are approaching end of support.
  • Plan your upgrades region-by-region with confidence.

⭐ us on GitHub and join our community!