stackql-deploy takes that pattern and turns it into a framework. Define your resources once as SQL-anchored files, declare them in a manifest, and let the framework run the exists check, the create-or-update, the state check, and the exports as a single build. Same underlying pattern, now something a team can operate at scale.

This tutorial walks through the tool end to end. What it is, how a project is shaped, how a single resource is defined, and how the whole thing runs against a real Google Cloud project.

What stackql-deploy is​

stackql-deploy is a model-driven deployment framework built on top of stackql. You describe cloud resources as SQL query files (each with a few named blocks called anchors), collect them in a manifest, and run a single command to converge your cloud state to what the manifest declares. It's inspired by dbt in shape: a project directory, models as files, environments as configurations, and a small CLI that operates on the whole thing.

Three properties are worth naming up front.

Declarative. You describe what you want to exist. The tool figures out whether to create, update, or leave alone based on live queries against the cloud API, not a cached state file.

Idempotent by design. Every resource has an exists check and an optional state check. The tool skips resources that already match, updates what has drifted, and creates what is missing. Running build twice in a row produces the same converged state, but only the first run actually mutates anything.

Provider-agnostic through the stackql query layer. Because everything under the hood is stackql SQL against real cloud APIs, the same tool can deploy AWS, GCP, Azure, Snowflake, Databricks, or any other provider stackql supports. One framework, one query grammar, many clouds.

How a project is shaped​

A stackql-deploy project is a directory with a manifest at the root and one .iql file per resource. Scaffolding a new project uses the init command:

stackql-deploy init example-stack --provider google

This creates a directory shaped like this:

example-stack/
├── stackql_manifest.yml
└── resources/
    └── example_vpc.iql

The manifest declares the providers, the global variables, and the resources the stack contains. Here is a minimal one for a single VPC network in GCP:

version: 1
name: "example-stack"
description: description for "example-stack"
providers:
  - google
globals:
  - name: project
    value: "{{ MY_PROJECT_NAME }}"
  - name: region
    value: australia-southeast1
resources:
  - name: example_vpc
    props:
      - name: vpc_name
        value: "{{ stack_name }}-{{ stack_env }}-vpc"
    exports:
      - vpc_name
      - vpc_link

The globals block holds values that apply across every resource. The template syntax reads environment variables or values passed at build time. The resources block lists each thing to deploy, with per-resource properties that get substituted into the SQL below.

How a resource is defined​

Each .iql file holds one resource. Inside the file, named blocks (anchors) declare the different query shapes the framework runs at different points in the lifecycle:

Read the anchors in order. exists is a live query that answers whether the resource is already there. create is the mutation to run if it isn't. statecheck runs after create to confirm the resource actually reached the desired configuration, with retries and delays for slow-to-propagate cloud APIs. exports captures values to reuse in other resources within the same stack. delete is the reverse mutation for teardown.

A note on the INSERT syntax. The columns split between path parameters (project stays plain since it maps to the URL path) and body fields (data__name, data__autoCreateSubnetworks, data__routingConfig get the data__ prefix). This is method-specific rather than universal across providers, and you can confirm the exact contract for any resource with:

DESCRIBE METHOD google.compute.networks.insert;

This is the same query-before-mutation loop Tutorial 2 walked through by hand, formalized into a resource-file structure. The framework runs it for every resource in the manifest, in dependency order, with proper handling for the states each resource can be in.

Running a dry-run build​

Install stackql-deploy via cargo:

cargo install stackql-deploy

Then, from the parent directory of your stack, run build in dry-run mode with a stack name, an environment name, and your project ID:

stackql-deploy build example-stack dev --dry-run -e MY_PROJECT_NAME=your-project-id

The framework loads the manifest, resolves the globals with your environment variables, then walks each resource in order. In dry-run mode, it renders every query it would execute against the cloud API and reports what it would do, without actually mutating anything. All the template variables are resolved: {{ project }} becomes your project ID, {{ vpc_name }} becomes example-stack-dev-vpc, and the routingConfig JSON is inlined into the INSERT.

Dry-run is the first thing to reach for. It gives you the full execution plan without touching your cloud, which makes it the natural pattern for pull-request checks and pre-deploy previews.

Running for real​

Once the dry-run output looks right, drop the flag to execute:

stackql-deploy build example-stack dev -e MY_PROJECT_NAME=your-project-id

The framework runs the same sequence, but this time the exists check hits the cloud API, and if the resource is missing, the create INSERT is sent for real. statecheck retries until GCP reports the resource has converged to the desired configuration, then exports capture the live values:

Read that output top to bottom. exists returned no match, so create ran. Right after create, the framework re-ran the exists check and confirmed the VPC was there. statecheck ran once and passed. exports captured vpc_name and vpc_link. Total time: under 4 seconds for a single-resource stack. The same pattern scales to a stack with dozens of resources across multiple providers.

For test-only runs (checks without mutations, useful in CI):

stackql-deploy test example-stack dev -e MY_PROJECT_NAME=your-project-id

And to reverse the whole stack:

stackql-deploy teardown example-stack dev -e MY_PROJECT_NAME=your-project-id

What this enables​

The obvious use is provisioning: describe a stack once, deploy it across dev, sit, and prod environments by changing one flag. But the deeper value is what the anchor structure enables.

Because every resource has an exists check and a state check, running build against an already-deployed stack is a policy convergence run. If someone changed a firewall rule out of band, the next build detects the drift via statecheck and re-applies the correct configuration. Nothing else on the stack gets touched. That's continuous convergence without needing a separate drift-detection tool.

Because everything is SQL and the query grammar is the same across every provider, extending a stack from GCP-only to multi-cloud is a matter of adding new .iql files and provider entries in the manifest. No new tool to learn, no separate DSL, no provider-specific templating conventions.

And because the framework is CI-friendly (test for validation without mutation, dry-run for previews, exit codes for pass/fail), it slots into whatever pipeline you already use to ship infrastructure changes.

Where this fits in the series​

Tutorial 1 introduced stackql for querying live cloud state across multiple clouds with SQL. Tutorial 2 showed how the same query engine lets agents read live state before mutating anything, one query at a time. The Actions guide put the audit pattern in CI. This tutorial shows the framework that operationalizes the whole pattern: not one query at a time, but a manifest-driven declaration of an entire cloud stack, with idempotence, environment scoping, and continuous convergence built in.

If the pattern makes sense to you, star the stackql repository so more developers find it. Full documentation, more manifest examples, and provider-specific guides are at stackql-deploy.io.

Next in this series: authoring your own multi-cloud stack from scratch, using stackql-deploy against AWS and GCP together.

Tutorial 1 in this series walked through auditing three clouds with stackql from your terminal. Tutorial 2 showed how the same query engine lets an agent read live state before mutating anything. This guide takes the same audit pattern and puts it in your CI, so the check runs every day, or on every pull request, or every hour, without anyone remembering to run it.

The tool is stackql-audit-action, a GitHub Action that runs an opinionated set of security checks against your cloud accounts and renders findings inline on the workflow run page. Setup is copy-paste. First results in under two minutes.

What it checks​

The action ships with seven built-in checks for GCP:

• SSH (22/tcp) open to the internet — HIGH
• RDP (3389/tcp) open to the internet — HIGH
• Cloud SQL instances reachable on a public IP — HIGH
• Compute instances with public IPs — MEDIUM
• Compute instances using the default service account — MEDIUM
• Storage buckets without uniform bucket-level access — MEDIUM
• Default VPC network exists — MEDIUM

Similar built-in check sets exist for AWS and Azure. A provider only runs if its credentials are supplied, so you can start with GCP and add the others by dropping in their secrets. This walkthrough covers GCP end to end.

Setup​

You need three things:

• A GitHub repository
• A GCP project you're authorized to audit
• A Google Cloud service account JSON key with read access to Compute, Cloud SQL, Storage, and IAM

Store the service account key as a GitHub secret. In your repository, go to Settings → Secrets and variables → Actions → New repository secret. Name the secret GCP_SA_JSON and paste the full JSON key as the value.

Never commit credentials to the repo. Never share a key JSON on Slack or email. Rotate the key if you ever paste one somewhere by accident.

Add the workflow​

Create .github/workflows/cloud-audit.yml with this content:

name: GCP Audit

on:
  workflow_dispatch:
    inputs:
      project-id:
        description: GCP project ID to audit
        required: true
      fail-on-severity:
        description: Severity threshold to fail on (CRITICAL/HIGH/MEDIUM/LOW/NONE)
        required: false
        default: HIGH

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: stackql/stackql-audit-action@v0.1
        with:
          project-id: ${{ inputs.project-id }}
          gcp-credentials: ${{ secrets.GCP_SA_JSON }}
          fail-on-severity: ${{ inputs.fail-on-severity }}

Two things worth noting. The action is pinned to @v0.1 for reproducibility. If you use @main, tomorrow's audit runs against tomorrow's version, which can change without warning. Pin to a real tag or a commit SHA. The fail-on-severity input controls when the workflow fails. For a first run, set it to NONE so the workflow completes even if findings surface. Once you've reviewed the output and confirmed the checks match your expectations, raise it to HIGH or MEDIUM as an enforcement gate.

Run the audit​

Commit and push. Then in your repository, go to Actions → GCP Audit → Run workflow. Fill in the project ID and start the run. First run takes about 20-30 seconds.

When the run completes, click into it and scroll down. The audit summary renders inline as a markdown table on the workflow run page:

Every check reports its status. Green checkmarks mean the check ran and found nothing. Yellow (MEDIUM) or red (HIGH) markers mean findings surfaced, with the affected resource names listed in a table and a short remediation note underneath.

In this test run, the audit surfaced nine medium-severity findings across three categories: one Compute instance using the default service account, one Compute instance with a public IP, and seven buckets without uniform bucket-level access. The other four checks ran and returned no findings. That distinction matters. A check running cleanly is signal too.

Making it continuous​

Manual triggers are for testing. For continuous audit, change the trigger to run on a schedule or on pull requests:

on:
  schedule:
    - cron: '0 9 * * *'   # every day at 09:00 UTC
  pull_request:
    branches: [main]
  workflow_dispatch:      # keep manual trigger as a fallback
    inputs:
      project-id:
        description: GCP project ID to audit
        required: true

The scheduled run catches drift that happens outside your PR flow. The pull-request run catches policy violations before they merge. Together they cover both the intentional and the accidental changes that can widen your attack surface between security reviews.

Extending it with your own checks​

The seven built-in checks are a starting point, not the full set your organization cares about. The action lets you point at your own directory of check definitions, one YAML file per check:

id: my-org-firewall-check
name: Allow only known source ranges
severity: HIGH
description: Catch firewall rules with sourceRanges outside our corp CIDRs.
remediation: Restrict to known CIDRs or migrate to IAP.
query: |
  SELECT name, network, sourceRanges
  FROM google.compute.firewalls
  WHERE project = '${PROJECT_ID}'
    AND direction = 'INGRESS'
columns: [name, network, sourceRanges]

A query returning zero rows means no findings. Any rows returned become finding rows. Every check is a small YAML file, so extending the audit to cover your organization's specific policies is one file per rule. See the queries/ directory in the action repository for the built-in checks as reference implementations.

Where this fits​

Tutorial 1 in this series introduced the audit pattern manually: SELECT across three clouds, spot the risky resources, know what to fix. Tutorial 2 introduced query-before-mutation: read live state before an agent acts, apply a policy gate, converge safely. This guide connects the two. The audit runs continuously in CI. When something drifts, the query-before-mutation pattern is how an agent or an operator brings it back into policy.

If the pattern makes sense to you, star the stackql-audit-action repository so more developers can find it. And if you have suggestions for checks that belong in the built-in set, the queries/ directory takes pull requests.

Next in this series: authoring your own check library with stackql-audit-action, and running the same pattern against AWS and Azure alongside GCP.

Infrastructure-as-code was designed around human-speed changes: write a plan, review it, apply it, and use state snapshots to know what was managed before. That works best when writers are few and changes are infrequent.

Agents change those assumptions. They run continuously and multiple actors can touch the same resources at the same time. Drift is no longer exceptional. It is unavoidable and relentless.

Canonical IaC tools like Terraform still matter because they represent intent. What's weakened is their ability to reflect current reality. Agents rarely own the whole picture. They usually operate in narrow scopes, working on a slice of infrastructure that other agents, other tools, or humans are also touching. In that world, an agent will regularly encounter infrastructure that was mutated out of band, outside whatever IaC system nominally manages it.

Query-before-mutation handles this case. Read live cloud state, compare it with policy, apply a bounded policy gate, mutate only what is out of policy, then verify. Each run starts from reality rather than a cached view.

The demo takes about ten minutes and uses Google Cloud Storage bucket encryption, but the pattern is provider-agnostic.

Query before mutation​

Correctness comes from querying live state before acting. Safety comes from something separate: policy gates that bound what a single run is allowed to do. Location locks, resource caps, budget ceilings, allowlists. Two patterns, two jobs. Agentic query-before-mutation complements centralised IaC approaches leveraging snapshots of state. Policy gates replace the review meeting.

The combined loop is correct by default because it reads reality, and safe by default because it cannot do more than the gate allows. Agents can run it continuously. Humans can run it manually. The output converges either way.

stackql implements this pattern. Any resource on any supported cloud can be read with SQL and mutated with SQL where the provider exposes mutation methods. Kubernetes solved reconciliation inside the cluster, but nothing reconciles across your cloud. That gap is where this pattern belongs. The demo uses Google Cloud Storage because the mutation surface is clean, but the shape is the same for AWS S3, Vault secrets, GitHub org settings, or anything else an agent needs to reason about.

Setup​

You need three things:

• Docker (to run stackql without installing anything else)
• A Google Cloud project with credentials that can read buckets and update encryption config, plus an existing Cloud KMS key you want to enforce
• About ten minutes

Clone the tutorial folder from the stackql repo:

git clone https://github.com/stackql/stackql.git
cd stackql/docs/tutorials/query-before-mutation-01

Copy the sample environment file and add your service account credentials:

cp .env.sample .env
# Then edit .env to set:
# GOOGLE_CREDENTIALS=<service-account-json-on-one-line>

Bring up an interactive stackql shell:

docker compose run --rm --entrypoint bash stackql
# then inside the container:
stackql shell

Pull the Google provider from the registry. This only needs to happen once per shell session:

REGISTRY PULL google v26.07.00432;

You're ready.

Query the live state​

Ask the GCS API which buckets exist and what encryption is currently applied to each.

SELECT name, location, encryption 
FROM google.storage.buckets 
WHERE project = 'your-project-id';

Expected output on a project with a mix of encryption configurations:

| name                       | location | encryption                                    |
| demo-app-bucket1           | US       | null                                          |
| demo-app-bucket2           | US       | null                                          |
| stackql-demo-src-bucket    | US       | null                                          |
| stackql-encrypted-bucket-1 | US       | {"defaultKmsKeyName":"projects/.../keys/..."} |

Three buckets show encryption as null, meaning Google-managed encryption keys (GMEK), the default. One already uses a customer-managed key. Under a policy that requires customer-managed keys, three buckets are out of policy.

A state snapshot can tell you what Terraform previously configured. This query tells you what GCS reports now, regardless of who created or changed the bucket. If ten agents and two humans are all touching this project at various points during the day, the query still returns the current truth every time you run it. That's the property the pattern is built on.

Apply a policy gate​

Three buckets don't match policy. Before mutating, the agent needs to prove it's allowed to. That's what the policy gate does.

A gate is anything that bounds what a single run of the agent can do. Scope by project or location, cap the number of resources changed, or enforce an explicit allowlist. The shape depends on how much freedom the agent has and how much you're willing to lose in one bad run.

For this demo, two gates cover most of the risk. The first is a location lock. The agent should only ever act on buckets in the location it was configured for. This is a WHERE clause on every query and mutation:

WHERE location = 'US'

Trivial when the project's buckets are in one location. In production, agents move between locations based on task, and touching a bucket in the wrong region can mean acting on resources in an environment the agent has no business in. One clause, but it's the difference between a bounded action and an unbounded one.

The second gate is a count cap. Before running a mutation across multiple buckets, ask the API how many the mutation would touch:

SELECT COUNT(*) 
FROM google.storage.buckets 
WHERE project = 'your-project-id' 
AND location = 'US' 
AND encryption IS NULL;

If the query returns 20 buckets and the agent's policy expects to be operating in a project with 3 to 5 non-compliant buckets, something has changed. Maybe an account merger, maybe a script created buckets out of band, maybe the credentials point to the wrong project entirely. The agent shouldn't proceed. It should log the discrepancy and wait for a human to look.

Gates like these aren't clever. They're routine, boring, and easy to skip. They're also the reason a query-before-mutation loop can run without a human review meeting attached to every apply. The gate carries the weight the meeting used to.

Mutate to converge​

The gates passed. For each bucket without a customer-managed key, apply the policy:

UPDATE google.storage.buckets 
SET data__encryption = '{"defaultKmsKeyName":"projects/your-project-id/locations/us/keyRings/your-ring/cryptoKeys/your-key"}' 
WHERE bucket = 'demo-app-bucket1';

Two things worth noting. The data__encryption prefix is how stackql passes fields into the request body, distinguishing them from URL parameters. The WHERE clause uses bucket rather than name, because bucket is the identifier the underlying patch method requires.

Expected output:

The operation was despatched successfully

In an agent loop, this runs as a bounded iteration over the non-compliant buckets returned by the earlier query, applying the same UPDATE to each one.

Verify convergence​

A successful mutation is not enough. Query the bucket again to verify:

SELECT name, encryption 
FROM google.storage.buckets 
WHERE bucket = 'demo-app-bucket1';

Expected output:

| name             | encryption                                                                    |
| demo-app-bucket1 | {"defaultKmsKeyName":"projects/your-project-id/locations/us/keyRings/..."}    |

The bucket converged from GMEK to the required customer-managed key.

Run the loop again and something interesting happens: the mutation is a no-op. The SELECT finds no drift, the count check passes trivially, no UPDATE fires. That's idempotence by design. The agent can run every minute, every hour, every day, and it does the same right thing whether the target is already compliant or has just drifted. No state snapshot needed to remember what it did last time. Every run starts from live reality.

The same pattern from an agent​

Everything above is you typing SQL into a shell. The same pattern maps directly to an agent through stackql's MCP server, listed in the Claude directory at claude.ai/directory/ant.dir.gh.stackql.stackql.

Install it, add your GCP credentials, and ask Claude in plain English:

List all GCS buckets in the project stackql-demo, and for each 
show whether encryption is a customer-managed key or the 
Google-managed default.

Claude uses the stackql MCP server to run the same underlying SELECT you saw earlier, then formats the result into a readable table. The query, gate, mutate, and verify loop runs the same way an agent runs it, just as reasoning steps in a conversation instead of shell commands.

Notice what the agent adds that the raw shell output doesn't: it separates CMEK buckets from Google-managed default ones, calls out which KMS key each CMEK bucket uses, and flags a caveat you didn't ask about (the difference between bucket-default encryption and per-object encryption). That's the value of running the pattern through an agent rather than a script. Same query, same result, plus the reasoning layer on top.

The shell walkthrough above was for you to see the pattern happen line by line. The MCP integration is how the pattern runs end to end in an actual agent's workflow.

Where this goes next​

Query before mutation is one instance of a broader pattern. Six patterns, really, that hold together as a working model for how agents should operate on infrastructure. Query before mutation is one. Idempotent assertions, bi-temporal views, agents as event sources, universal interfaces, and policy gates are the other five. Together they replace the assumptions the plan-review-apply model was built on.

The full argument is in Jeff Aven's talk from AI Engineer Melbourne, "Treating Infrastructure as Data: Building an AI-Native Control Plane." If the pattern in this tutorial made sense to you, the talk is the map for what else fits with it.

If this tutorial was useful, star the stackql repo. Every star helps other developers and agents find the pattern, and helps the project climb the MCP registry where more agents can discover it.

Next in this series: idempotent assertions with stackql-deploy. Same shape, different pattern, applied to declarative infrastructure that runs continuously without breaking on repeat.

The question is usually simple. "Which buckets are public across all our clouds?" You end up running three separate scripts, then reconciling three different output shapes just to give one answer.

This tutorial doesn't make that faster. It replaces it. One query, one view, all three clouds.

What StackQL is, in three sentences​

StackQL is SQL for cloud APIs. You write a normal SQL query, and it makes the API calls to AWS, GCP, Azure, or dozens of other providers to get the answer back as rows and columns.

The important part for this tutorial: it queries the live provider APIs on every run. There's no local database, no cache, no sync job to schedule. What you see is the current state of your cloud, right now.

It exists because someone got tired of writing the same audit logic three times.

What you'll need​

You'll need Docker installed and read-only credentials for whichever clouds you want to audit. All three are optional. If you leave a cloud's credentials blank, it gets skipped.

For AWS, GCP, and Azure, the audit needs a service account or role that can list storage buckets and read their configuration. The audit-action repo has the exact permissions per provider here.

Run the audit​

The tutorial folder has three files: a docker-compose file, an .env.audit.example template, and a README. Grab the folder from docs/tutorials/preview-bucket-01/ in the stackql/stackql repo.

1. Copy the env template

cp ./audit/.env.audit.example ./audit/.env.audit

Then open .env.audit and fill in credentials for whichever clouds you want. The variables are named for what they are: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, GOOGLE_CREDENTIALS (the service account JSON, on one line), and GOOGLE_ORG_ID.

If you're only auditing one or two clouds, leave the others blank.

2. Pull the StackQL Docker image

docker compose -f docker-compose.bucket.audit.yaml pull

The tutorial pins to stackql/stackql:v0.10.601. The audit uses views under stackql_preview.* which are still evolving, so pinning matters for reproducibility.

3. Run the audit

docker compose -f docker-compose.bucket.audit.yaml run --rm stackql

On a modest account this takes about 30 seconds. Output goes straight to your terminal as a table.

What the output tells you​

When the audit finishes, you get a single table printed to your terminal. Something like this (bucket names are placeholders):

Example output from the audit. Bucket configuration across three cloud providers in one normalised table.

The point isn't the specific findings. It's the shape.

Three different resource types (S3 buckets, GCS buckets, and Azure Storage Accounts) normalised into one table with the same columns. Encryption class, public flag, HTTPS enforcement. Side by side, for every cloud you provided credentials for.

A few patterns to notice in the table:

• Public buckets across providers. Filter by public = true and you have the answer to your security team's question, in every cloud, in one place.
• Encryption side by side. The encryption_class column shows which buckets use provider-managed keys versus customer-managed. Different providers have different naming conventions internally; StackQL normalises them to the same two categories.
• HTTPS enforcement in one column. Whether it's S3 bucket policies, GCS uniform access, or Azure secure transfer, same column, same values, no translation needed.

This is what SQL for cloud APIs actually delivers. Not just SQL as a query language, but a uniform data model across providers, so you can ask one question and get one answer.

Performance and scope​

StackQL runs each provider's query in parallel using the credentials you provided. There's no local database being populated and no background sync; every run hits the live APIs and returns what's there right now.

On the test account this took about 30 seconds. The query is scoped to one AWS region and one GCP organization, which keeps the API footprint bounded and reflects the current preview default. Streaming output and broader audits (across regions, deeper across resource types) are in flight from the StackQL team.

What this covers today​

Today, this audit covers storage buckets across AWS, GCP, and Azure. One region at a time for AWS, one organization at a time for GCP. Entitlements, IAM, and other resource types are on the roadmap as separate audits under stackql_preview.* and will get their own tutorials as they land.

Where to go next​

github.com/stackql/stackql is the flagship repo. If this tutorial was worth reading, give it a star. It's genuinely the thing that tells the team to build more tutorials like this one, and it helps other engineers with the same three-console problem find the project.

A few natural next steps from here:

• Run it against your own accounts. The audit works exactly the same way against real credentials. You can start with one cloud and add the others by adding their env vars.
• Move it into CI. The stackql-audit-action repo has ready-to-use GitHub Actions workflows for all-clouds, single-cloud, deep-audit, and OIDC-authenticated variants. Drop one into your repo, add secrets, and you have a scheduled cross-cloud audit running on every push.
• Ask questions. The StackQL community Slack is the fastest way. GitHub issues on the flagship repo also work.

More audits are on the way, including entitlements, deeper checks, and larger estates. If there's a specific one you'd find useful, opening an issue is the fastest way to get it on the roadmap.

This post walks through the other half: a remediation loop that turns each finding into a reviewable pull request, verifies live state before it changes anything, and applies the fix on merge. It runs entirely in GitHub Actions, authenticates with OIDC, and uses StackQL to talk to cloud control planes. The repo is public at stackql-labs/stackql-ai-remediation, and the example throughout is FinOps waste (unattached disks, idle IPs, zero-VM projects), though the shape is the same for posture and security checks.

The loop​

The pipeline is deliberately small:

1. A scheduled or dispatched audit runs against the cloud control plane and writes findings as structured data.
2. For each finding, the loop opens one pull request carrying the exact proposed change.
3. A preflight status check queries live reality before anything can merge.
4. On merge, the change is applied through the vendor CLI.
5. A post-apply check confirms the resource is actually gone.

What is deliberately absent matters as much as what is present. No agents running in the estate. No long-lived cloud keys. No log pipeline, no inventory sync, no external scanner appliance. The data path is GitHub Actions to StackQL to the cloud control-plane APIs, and the audit trail is the pull request history.

A finding is just data​

The audit emits each finding as JSON, and that JSON is the contract the rest of the loop reads. A single FinOps finding looks like this:

{
  "run_id": "27901134862",
  "provider": "aws",
  "check_id": "aws-finops-unattached-volume",
  "check_name": "Unattached EBS volumes",
  "severity": "LOW",
  "category": "waste",
  "region": "ap-southeast-2",
  "suggested_remediation": {
    "type": "delete",
    "tool": "stackql",
    "preflight_query": "SELECT VolumeId FROM aws.ec2.volumes WHERE region = 'ap-southeast-2' AND VolumeId = 'vol-0a1b2c3d4e5f' AND State = 'available'",
    "sql_query": "DELETE FROM aws.ec2.volumes WHERE region = 'ap-southeast-2' AND VolumeId = 'vol-0a1b2c3d4e5f'",
    "description": "Delete the unattached EBS volume (snapshot first if its data may be needed)."
  },
  "fields": {
    "volumeId": "vol-0a1b2c3d4e5f",
    "size": "20",
    "status": "available",
    "estimated_monthly_usd": 1.92
  }
}

The finding already carries its own remediation: a preflight query and the SQL that would fix it, both fully substituted with concrete values. The rest of the loop is mechanical from here.

One pull request per finding​

generate_proposals.py reads the findings and writes one directory per finding:

remediations/proposed/<run-id>/<n>-<check-id>-<resource-id>/
  finding.json      # verbatim copy of the finding
  preflight.sql     # the live-state check, pass = >=1 row
  remediation.sql   # the canonical fix (applied via CLI on merge)
  rationale.md      # deterministic explanation from the finding fields

The SQL comes straight from the audit's suggested_remediation block, already templated per check type, so the proposal is reproducible and reviewable rather than generated fresh each run. Where a model does help is a separate, optional step: an agent reads the batch and flags the captain's-call risks - the changes a human should look at twice - so reviewers know where to spend attention. Judgment and mutation are kept apart on purpose.

Query before mutation​

The preflight is the part worth dwelling on:

-- pass criterion: returns >= 1 row
SELECT VolumeId
FROM aws.ec2.volumes
WHERE region = 'ap-southeast-2'
  AND VolumeId = 'vol-0a1b2c3d4e5f'
  AND State = 'available';

This runs as a required status check on the pull request. It is a live query against the cloud API, not a read of a cached state file, and it asserts that the resource is still in the state the finding assumed at the moment of merge, not at the moment of audit. Findings go stale. Between the audit run and someone clicking merge, the volume may have been re-attached, deleted by someone else, or moved. A state file would not know. The API does. If the preflight returns zero rows, the check fails and the change cannot land.

This is the single most important property of the loop: the gap between "what the audit saw" and "what is true now" is closed by a live query, every time, automatically.

The fix is a SQL statement​

StackQL treats the cloud control plane as a queryable and mutable surface. You SELECT to read it and DELETE or INSERT to change it, with the same grammar across providers:

DELETE FROM aws.ec2.volumes
WHERE region = 'ap-southeast-2'
  AND VolumeId = 'vol-0a1b2c3d4e5f';

The statement is idempotent by shape. Run it twice and the end state is the same, which is exactly the property you want when a pipeline can retry. On merge, the apply workflow executes the remediation through the vendor CLI; remediation.sql is kept as the canonical, human-readable record of what was applied.

Policy gates instead of a review bottleneck​

Branch protection is where the safety model lives. The rule on main is:

• Required approvals: 0
• Require status checks to pass before merging: on, so the preflight is enforced

The preflight check is the gate, not a human sign-off. That sounds aggressive until you see what it replaces: instead of one person eyeballing a diff, the merge is bounded by a machine-enforced envelope. Low-severity waste can auto-merge once the preflight is green; you keep a human approver in the path for higher-severity classes by raising the required-approvals count on those. The same pattern extends to budget ceilings, region locks, and change-rate caps. Policy is the loop's safety mechanism, expressed as checks rather than as a meeting.

Onboarding without handing over keys​

The loop authenticates with OIDC and federated identity only - AWS AssumeRoleWithWebIdentity, GCP Workload Identity Federation, Azure federated credentials. There are no static cloud keys in the repo.

Bootstrap stays inside the cloud admin's own console session. The onboarding templates under cicd/onboarding/ give you a one-click path per cloud: a CloudFormation "Launch Stack" URL for AWS, a gcloud script for GCP, an ARM "Deploy to Azure" template for Azure. The admin clicks, authenticates in their own console, and the only thing that comes back to the pipeline is a role ARN or a workload-identity provider. Nothing long-lived crosses a boundary. It is the same integration model SaaS security tools already use.

Run it​

End to end:

1. Clone stackql-labs/stackql-ai-remediation.
2. Set branch protection on main: required approvals 0, required status check for the preflight.
3. Run the onboarding template for each cloud you want to cover to create the OIDC role or binding.
4. Store the resulting role ARNs and WIF provider as repo secrets.
5. Dispatch the audit workflow, or let the schedule run it.

Findings upload as workflow artifacts, proposals generate, and one pull request opens per finding. Merge a green one and watch the apply and post-check complete. A representative cross-cloud sweep here finished in a couple of minutes and surfaced a handful of unattached volumes, each as its own PR with the saving attached.

Why this shape​

The properties that make this worth running are all consequences of one decision: reach the control plane as data. Findings are data, the safety check is a query, the fix is a statement, and the record is the pull request. No new dashboard, no new vendor, no agent footprint in the estate. It is boring and inspectable, which is what you want from something allowed to change production.

The same loop runs for security posture and access findings by swapping the checks. That is the next post.

edgepilot collapses all of that to SQL. Two Claude agents - a recon agent and an action agent - observe a live Cloudflare zone, tighten its rate-limit rule when traffic warrants, and write a decision record to a Confluent Kafka topic. Neither agent knows anything about Cloudflare's GraphQL Analytics API, Cloudflare's rulesets engine, or Confluent's Kafka REST proxy. They know SQL. The StackQL MCP server does the rest.

The setup​

The demo runs against a throwaway Cloudflare zone (stackql.xyz) and a freshly provisioned Confluent Cloud Kafka cluster. A small load generator drives synthetic traffic - some normal browsers, some AI-crawler user agents - at the zone so the analytics tables have something real to report.

Both agents speak to a single MCP server - StackQL running in stdio mode - and that server fans out to whichever provider the query targets. The agents never see API URLs, auth headers, GraphQL query strings, or Kafka REST endpoints. They see tables.

The recon agent​

The recon agent runs three SELECTs. Notice that nothing in these queries hints that one is a managed cluster API, one is a GraphQL analytics endpoint, and one is a rulesets PUT/GET pair.

🔍 Find the active Kafka cluster​

SELECT
  id                                                                              AS kafka_cluster_id,
  JSON_EXTRACT(spec, '$.region')                                                  AS region,
  LOWER(JSON_EXTRACT(spec, '$.cloud'))                                            AS cloud_provider,
  SPLIT_PART(SPLIT_PART(JSON_EXTRACT(spec, '$.http_endpoint'), '//', 2), '.', 1)  AS kafka_endpoint_id
FROM confluent.managed_kafka_clusters.clusters
WHERE environment = 'env-xxxxx';

|------------------|-----------|----------------|-------------------|
| kafka_cluster_id | region    | cloud_provider | kafka_endpoint_id |
|------------------|-----------|----------------|-------------------|
| lkc-abc123       | us-east-1 | aws            | pkc-xyz9w         |
|------------------|-----------|----------------|-------------------|

📊 Read live zone analytics​

This one talks to Cloudflare's GraphQL Analytics API under the hood. The agent sees a table.

SELECT datetime, client_country_name, edge_response_status,
       client_request_http_method_name, requests, bytes
FROM cloudflare.zones.http_requests_adaptive_groups
WHERE zone_tag = '<zone_id>'
AND   since    = '2026-06-04T09:30:00Z'
AND   until    = '2026-06-04T10:00:00Z';

|----------------------|---------------------|----------------------|---------------------------------|----------|--------|
| datetime             | client_country_name | edge_response_status | client_request_http_method_name | requests | bytes  |
|----------------------|---------------------|----------------------|---------------------------------|----------|--------|
| 2026-06-04T09:31:00Z | United States       | 200                  | GET                             | 412      | 184320 |
|----------------------|---------------------|----------------------|---------------------------------|----------|--------|
| 2026-06-04T09:31:00Z | Germany             | 200                  | GET                             | 87       | 39424  |
|----------------------|---------------------|----------------------|---------------------------------|----------|--------|
| 2026-06-04T09:32:00Z | United States       | 429                  | GET                             | 23       | 9728   |
|----------------------|---------------------|----------------------|---------------------------------|----------|--------|
| 2026-06-04T09:34:00Z | Singapore           | 200                  | POST                            | 19       | 8442   |
|----------------------|---------------------|----------------------|---------------------------------|----------|--------|

🛡️ Inspect the current rate-limit rule​

SELECT id,
       JSON_EXTRACT(rules, '$[0].id')                            AS rule_id,
       JSON_EXTRACT(rules, '$[0].description')                   AS description,
       JSON_EXTRACT(rules, '$[0].ratelimit.requests_per_period') AS threshold,
       JSON_EXTRACT(rules, '$[0].ratelimit.period')              AS period
FROM cloudflare.rulesets.phases
WHERE zone_id       = '<zone_id>'
AND   ruleset_phase = 'http_ratelimit';

|----------------------------------|----------------------------------|----------------------------------------|-----------|--------|
| id                               | rule_id                          | description                            | threshold | period |
|----------------------------------|----------------------------------|----------------------------------------|-----------|--------|
| 1a2b3c4d5e6f7890abcdef1234567890 | 9f8e7d6c5b4a3210fedcba9876543210 | edgepilot demo rule (managed by sqld)  | 100       | 10     |
|----------------------------------|----------------------------------|----------------------------------------|-----------|--------|

The recon agent hands these results to the action agent as a short plain-text report.

The action agent​

The action agent does two things: tighten the rate limit on Cloudflare, then write a decision record to Confluent Kafka. Both are mutations, expressed as standard SQL.

✏️ Tighten the Cloudflare rate limit​

StackQL implements REPLACE against Cloudflare's modern rulesets PUT endpoint. The agent doesn't have to know that under the covers this is a full-document replace of every rule in the http_ratelimit phase entrypoint.

REPLACE cloudflare.rulesets.phases
SET rules = '[
  {
    "action": "block",
    "ratelimit": {
      "characteristics": ["ip.src", "cf.colo.id"],
      "period": 10,
      "requests_per_period": 30,
      "mitigation_timeout": 10
    },
    "expression": "len(http.request.uri.path) gt 0",
    "description": "edgepilot demo rule (managed by stackql-deploy)",
    "enabled": true
  }
]'
WHERE zone_id       = '<zone_id>'
AND   ruleset_phase = 'http_ratelimit';

📨 Log the decision to Kafka​

And the action agent publishes a record to Confluent Kafka - again, as plain SQL. No producer client, no AVRO schema registry plumbing, no REST endpoint to look up. StackQL resolves the cluster's data-plane endpoint from the cluster id and posts the record.

INSERT INTO kafka.kafka.records(
  cluster_id, kafka_endpoint_id, region, cloud_provider,
  topic_name, key, value
)
SELECT
  'lkc-abc123',
  'pkc-xyz9w',
  'us-east-1',
  'aws',
  'edge-autopilot-decisions',
  '{"type":"STRING","data":"edgepilot.action"}',
  '{"type":"JSON","data":{"agent":"action","action":"tighten_rate_limit","threshold":30,"reasoning":"elevated bot traffic, dropping threshold from 100 to 30","timestamp":"2026-06-04T10:00:00Z"}}';

The record appears live in the Confluent Cloud UI -> Topics -> edge-autopilot-decisions -> Messages tab.

Why this works as an agentic loop​

Edge defense has three properties that make it well-suited to agents, and three properties that usually make it painful to wire up. SQL-over-MCP collapses the painful side:

• The observe step (analytics), the act step (rate-limit control plane), and the audit step (durable log) normally each have their own SDK, auth model, and request shape. Here they're the same shape: a query.
• LLMs are reliable SQL generators - far more reliable than they are at chaining three provider-specific SDK calls correctly. Constrain the schema and the agent will stay on the rails.
• Adding a new signal source (WAF events, R2 logs, Workers analytics) or a new action surface (a different CDN, a different broker) is a registry pull, not a refactor of the agent loop.

The whole loop - observe, decide, act, audit - takes about 30 to 60 seconds to run.

Try wiring StackQL into your own agent loop:

• Pick any provider in the StackQL registry and pull it.
• Point an MCP-aware client (Claude Desktop, Cline, or your own SDK loop) at the StackQL MCP server.
• Let the model discover the schema via SHOW METHODS and DESCRIBE - no fine-tuning, no provider-specific prompting.

⭐ us on GitHub and join our community!

How It Works​

Download the latest release of stackql, then run the convenience script included (similar scripts are included for other cloud provider terminals - e.g. AWS Cloud Shell).

curl -L https://bit.ly/stackql-zip -O && unzip stackql-zip
sh stackql-databricks-shell.sh

Example Queries​

Here are the sample queries run in the video, just change the deployment_name for your workspace.

User entitlements

SELECT
  deployment_name,
  id,
  userName,
  displayName,
  entitlement
FROM databricks_workspace.iam.vw_user_entitlements
WHERE deployment_name = 'dbc-74aa95f7-8c7e';

All workspace settings

SELECT * FROM
  databricks_workspace.settings.vw_all_settings
WHERE deployment_name = 'dbc-74aa95f7-8c7e';

Tag policies filtered by key prefix

SELECT
  tag_key as key,
  description
FROM databricks_workspace.tags.tag_policies
WHERE deployment_name = 'dbc-74aa95f7-8c7e'
AND tag_key LIKE 'class%';

Catalog count by type

SELECT
  catalog_type,
  COUNT(*) as num_catalogs
FROM databricks_workspace.catalog.catalogs
WHERE deployment_name = 'dbc-74aa95f7-8c7e'
GROUP BY catalog_type;

Provider Coverage​

The databricks_workspace provider covers workspace related services, the databricks_account provider covers account-level operations including provisioning, billing, and account IAM.

The web terminal flow covers workspace-scoped queries using the token of the logged-in user. For account-level queries (provisioning, billing, account IAM), you need a Databricks service principal with account admin rights and OAuth2 credentials:

export DATABRICKS_ACCOUNT_ID="your-account-id"
export DATABRICKS_CLIENT_ID="your-client-id"
export DATABRICKS_CLIENT_SECRET="your-client-secret"

These are the same variables used by the Databricks CLI and Terraform provider, so if you already have those configured the auth story is identical.

Get Started​

Full provider documentation:

• Databricks Workspace Provider for StackQL
• Databricks Account Provider for StackQL

Visti StackQL on GitHub.

[Action Required] AWS Lambda end of support for Node.js 18 [AWS Account: 824123456789] [EU-CENTRAL-1]

If you are like me and manage AWS accounts with numerous Lambda functions potentially deployed across multiple regions, you need to identify affected resources, in this case, Lambda node runtimes, which will be discontinued later this year.  

With stackql this task is easy...

1. Open AWS cloud shell in your AWS account (any region - it doesn't matter)
2. Download stackql

curl -L https://bit.ly/stackql-zip -O && unzip stackql-zip

3. Open an authenticated stackql command shell

sh stackql-aws-cloud-shell.sh

4. Run some analytic queries using stackql; here are some examples...

🔍 List all functions and runtimes across regions​

Run a stackql query to get the details about functions, runtimes, etc, deployed at any given time across one or more AWS regions.  You can include all 25 AWS regions; each query will be performed asynchronously - speeding up the results.

select 
  function_name, 
  region,
  runtime 
FROM aws.lambda.functions 
WHERE region IN ('us-east-1', 'eu-west-1');

|----------------------------------------------|---------------|------------|
| function_name                                | region        | runtime    |
|----------------------------------------------|---------------|------------|
| order-processing-handler                     | us-east-1     | nodejs16.x |
|----------------------------------------------|---------------|------------|
| payment-service-lambda                       | us-east-1     | nodejs18.x |
|----------------------------------------------|---------------|------------|
| user-notifications-service                   | us-east-1     | python3.8  |
|----------------------------------------------|---------------|------------|
| generate-invoice-pdf                         | eu-west-1     | nodejs18.x |
|----------------------------------------------|---------------|------------|
| realtime-data-stream-processor               | eu-west-1     | java11     |
|----------------------------------------------|---------------|------------|
| chat-message-router                          | eu-west-1     | nodejs20.x |
|----------------------------------------------|---------------|------------|
| scheduled-maintenance-checker                | ap-southeast-2| python3.11 |
|----------------------------------------------|---------------|------------|
| s3-image-thumbnail-generator                 | ap-southeast-2| nodejs16.x |
|----------------------------------------------|---------------|------------|
| email-delivery-status-updater                | ap-southeast-2| python3.8  |
|----------------------------------------------|---------------|------------|

📊 Group by runtime and region​

Perform an analytic query like a group by aggregate query such as...

select 
  runtime, 
  region, 
  count(*) as num_functions 
FROM aws.lambda.functions 
WHERE region IN ('us-east-1', 'eu-west-1', 'ap-southeast-2')
GROUP BY runtime, region;

|--------------|----------------|---------------|
| runtime      | region         | num_functions |
|--------------|----------------|---------------|
| nodejs16.x   | us-east-1      | 102           |
|--------------|----------------|---------------|
| nodejs18.x   | us-east-1      | 98            |
|--------------|----------------|---------------|
| python3.8    | us-east-1      | 138           |
|--------------|----------------|---------------|
| nodejs18.x   | eu-west-1      | 40            |
|--------------|----------------|---------------|
| java11       | eu-west-1      | 113           |
|--------------|----------------|---------------|
| nodejs20.x   | eu-west-1      | 120           |
|--------------|----------------|---------------|
| python3.11   | ap-southeast-2 | 384           |
|--------------|----------------|---------------|
| nodejs16.x   | ap-southeast-2 | 88            |
|--------------|----------------|---------------|
| python3.8    | ap-southeast-2 | 18            |
|--------------|----------------|---------------|

Using StackQL you can:

• Quickly spot functions running on runtimes like nodejs18.x that are approaching end of support.
• Plan your upgrades region-by-region with confidence.

⭐ us on GitHub and join our community!

Microsoft Azure is retiring TLS 1.0 and 1.1 for its services, requiring customers to transition to TLS 1.2 or later to ensure uninterrupted connectivity. If you have workloads still using older TLS versions, you’ll need to update them.

Using StackQL to Identify Non-Compliant Resources​

With StackQL, you can quickly identify resources in your Azure environment that are still using older TLS versions. This article shows how to leverage StackQL queries to check various Azure services for compliance.

Prerequisites​

1. Pull the latest StackQL provider for Azure using REGISTRY PULL azure.
2. Authenticate with Azure using StackQL by setting up your credentials as environment variables (or using your existing az login system/session authentication).

Queries to Run​

Below are example queries you can use to identify resources affected by the TLS 1.2 requirement (use your subscriptionId of course):

1. Check Application Gateway Configurations​

Azure Application Gateways may support older TLS versions. Run the following query to get their configurations:

SELECT
    id,
    name,
    JSON_EXTRACT(properties, '$.sslPolicy') as ssl_policy,
	JSON_EXTRACT(properties, '$.defaultPredefinedSslPolicy') as default_predefined_ssl_policy
FROM
    azure.network.application_gateways
WHERE
	subscription_id = '123e4567-e89b-12d3-a456-426614174000'
    AND ssl_policy IS NOT NULL
    AND JSON_EXTRACT(properties, '$.sslPolicy') NOT LIKE '%TLS12%';

This query lists all Application Gateways configured with TLS versions lower than 1.2.

2. Inspect App Service Configurations​

If you use Azure App Services (Web Apps), check their TLS configurations with this query:

SELECT
    id,
    name,
	JSON_EXTRACT(properties, '$.httpsOnly') as https_only,
	JSON_EXTRACT(properties, '$.siteConfig.minTlsVersion') as min_tls_version
FROM
    azure.web.web_apps
WHERE
	subscription_id = '123e4567-e89b-12d3-a456-426614174000'
	AND JSON_EXTRACT(properties, '$.siteConfig.minTlsVersion') < '1.2';

This returns all web apps that allow connections using TLS versions older than 1.2.

3. Check SQL Server Instances​

Azure SQL Databases and SQL Managed Instances may also have TLS configurations that need checking:

SELECT
	location,
	fullyQualifiedDomainName,
	minimalTlsVersion,
	state
FROM 
	azure.sql.servers
WHERE 
	subscription_id = '123e4567-e89b-12d3-a456-426614174000'
	AND minimalTlsVersion < '1.2';

This shows all SQL servers with a minimal TLS version set below 1.2.

We’d love to hear your feedback. ⭐ us on GitHub and let us know how StackQL helps you manage your Azure resources!

This guide demonstrates how to use StackQL to enumerate and delete all default VPCs and their associated resources in all AWS regions.  

What you need​

All you need to do is to install the pystackql package using:

pip install pystackql

Deleting resources will require a privileged AWS IAM user. You can do this from a terminal with the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables set (and optionally AWS_SESSION_TOKEN set if you are using sts assume-role).  

You can also use the StackQL Cloud Shell Scripts, from within AWS Cloud Shell, to run authenticated StackQL queries using:

sh stackql-aws-cloud-shell.sh

How it works​

This example uses the awscc (AWS Cloud Control) provider. Cloud Control resources are read in two steps: a _list_only view returns the resource identifiers in a region, and the resource view returns the full properties for a single resource selected by its Identifier.

Default VPCs in AWS regions have a CIDR block of 172.31.0.0/16, before deleting anything, the program ensures it is not in use by listing the network interfaces in the region and checking whether any belong to the VPC:

SELECT id
FROM awscc.ec2.network_interfaces_list_only
WHERE region = '{region}'

If any resources are using the VPC (such as EC2, RDS, ELB/ALB, VPC attached Lambda functions, etc), it will skip these VPCs.  Once determined that the VPC is not in use, all of the resources associated with the VPC must be deleted before the VPC itself can be deleted; this includes:

• External Routes (awscc.ec2.routes)
• Internet Gateways (awscc.ec2.internet_gateways)
• NACLs (awscc.ec2.network_acls)
• Subnets (awscc.ec2.subnets)
• and then finally, the VPC (awscc.ec2.vpcs)

The default route table and default security group are deleted automatically when deleting the VPC

This is done by discovering the resources using StackQL SELECT queries and deleting them using StackQL DELETE queries, like:

DELETE FROM awscc.ec2.network_acls
WHERE data__Identifier = '{nacl_id}'
AND region = '{region}'

Complete code​

The following Python program uses StackQL to list and delete default VPCs and their associated resources (subnets, route tables, internet gateways, security groups, and network ACLs) across all AWS regions.

from pystackql import StackQL
stackql = StackQL()
stackql.executeStmt("REGISTRY PULL awscc")

def ensure_one_or_zero(resource_list, resource_name, region):
    if len(resource_list) > 1:
        raise RuntimeError(f"ah snap! multiple default {resource_name} resources found in {region}")
    elif len(resource_list) == 0:
        print(f"/* no default {resource_name} found in {region} */\n")
        return False
    return True

def get_resource(region, resource, identifier):
    # fetch the full properties of a single Cloud Control resource by its identifier
    rows = stackql.execute(f"""
        SELECT *
        FROM awscc.ec2.{resource}
        WHERE region = '{region}'
        AND Identifier = '{identifier}'
    """)
    return rows[0] if len(rows) else None

def find_in_vpc(region, resource, id_col, vpc_id):
    # list all identifiers for a resource in a region, then keep those attached to the target VPC
    matches = []
    for row in stackql.execute(f"SELECT {id_col} FROM awscc.ec2.{resource}_list_only WHERE region = '{region}'"):
        detail = get_resource(region, resource, row[id_col])
        if detail and detail.get('vpc_id') == vpc_id:
            matches.append(detail)
    return matches

regions = [
    'us-east-1', 'us-east-2', 'us-west-1', 'us-west-2',
    'eu-central-1', 'eu-central-2', 'eu-west-1', 'eu-west-2', 'eu-west-3', 'eu-north-1', 'eu-south-1',
    'ap-east-1', 'ap-south-1', 'ap-south-2', 'ap-northeast-1', 'ap-northeast-2', 'ap-northeast-3',
    'ap-southeast-1', 'ap-southeast-2', 'ap-southeast-3', 'ap-southeast-4', 'af-south-1',
    'ca-central-1',  'me-south-1', 'me-central-1', 'sa-east-1'
]

for region in regions:
    # find the default VPC (CIDR 172.31.0.0/16, untagged) by listing VPCs then inspecting each
    default_vpcs = []
    for row in stackql.execute(f"SELECT vpc_id FROM awscc.ec2.vpcs_list_only WHERE region = '{region}'"):
        vpc = get_resource(region, 'vpcs', row['vpc_id'])
        if vpc and vpc.get('cidr_block') == '172.31.0.0/16' and not vpc.get('tags'):
            default_vpcs.append(vpc)
    if not ensure_one_or_zero(default_vpcs, 'VPC', region): continue
    vpc_id = default_vpcs[0]['vpc_id']

    # check if the VPC is in use
    network_interfaces = find_in_vpc(region, 'network_interfaces', 'id', vpc_id)
    if len(network_interfaces) > 0:
        print(f"/* skipping deletion of default VPC ({vpc_id}) in {region} because it is in use */\n")
        continue

    print(f"/* deleting resources for default VPC ({vpc_id}) in {region} */\n")

    # get the default route table for the VPC
    route_tables = find_in_vpc(region, 'route_tables', 'route_table_id', vpc_id)
    ensure_one_or_zero(route_tables, 'route table', region)
    route_table_id = route_tables[0]['route_table_id']

    # get the internet gateway from the default (0.0.0.0/0) route
    default_route = get_resource(region, 'routes', f'{route_table_id}|0.0.0.0/0')
    inet_gateway_id = default_route['gateway_id'] if default_route else None

    # delete the default route
    print(f"/* deleting default VPC routes in route table ({route_table_id}) in {region} */")
    print(f"""
DELETE FROM awscc.ec2.routes
WHERE data__Identifier = '{route_table_id}|0.0.0.0/0'
AND region = '{region}';
    """)

    # detach the internet gateway
    print(f"/* detaching default VPC internet gateway ({inet_gateway_id}) in {region} */")
    print(f"""
DELETE FROM awscc.ec2.vpc_gateway_attachments
WHERE data__Identifier = 'IGW|{vpc_id}'
AND region = '{region}';
    """)

    # delete the internet gateway
    print(f"/* deleting default VPC internet gateway ({inet_gateway_id}) in {region} */")
    print(f"""
DELETE FROM awscc.ec2.internet_gateways
WHERE data__Identifier = '{inet_gateway_id}'
AND region = '{region}';
    """)

    # delete the network ACL
    nacls = find_in_vpc(region, 'network_acls', 'id', vpc_id)
    ensure_one_or_zero(nacls, 'network acl', region)
    nacl_id = nacls[0]['id']
    print(f"/* deleting default VPC NACL ({nacl_id}) in {region} */")
    print(f"""
DELETE FROM awscc.ec2.network_acls
WHERE data__Identifier = '{nacl_id}'
AND region = '{region}';
    """)

    # delete the subnets
    subnets = find_in_vpc(region, 'subnets', 'subnet_id', vpc_id)
    for subnet in subnets:
        print(f"/* deleting default VPC subnet ({subnet['subnet_id']}) in {region} */")
        print(f"""
DELETE FROM awscc.ec2.subnets
WHERE data__Identifier = '{subnet['subnet_id']}'
AND region = '{region}';
        """)

    # delete the VPC
    print(f"/* deleting default VPC ({vpc_id}) in {region} */")
    print(f"""
DELETE FROM awscc.ec2.vpcs
WHERE data__Identifier = '{vpc_id}'
AND region = '{region}';
    """)

run the program using:

python3 delete-all-default-vpcs.py > delete_default_vpcs.iql

to generate a StackQL script you can run as a batch using stackql exec or as individual statements in the stackql shell.

Conclusion​

Deleting default VPCs can help improve your AWS security posture by removing potentially noncompliant network configurations. This program leverages StackQL to automate the process, ensuring consistency across all regions.

Let us know what you think! ⭐ us on GitHub.

SELECT 
	name, 
	source_range,
	ip_protocol,
	allowed_ports,
	direction
FROM (
	SELECT 
	name,
	source_ranges.value as source_range,  
	JSON_EXTRACT(allowed.value, '$.IPProtocol') as ip_protocol,
	JSON_EXTRACT(allowed.value, '$.ports') as allowed_ports,
	direction
	FROM google.compute.firewalls, json_each(sourceRanges) as source_ranges, json_each(allowed) as allowed
	WHERE project = 'stackql-k8s-the-hard-way-demo' 
	AND network = 'https://www.googleapis.com/compute/v1/projects/stackql-k8s-the-hard-way-demo/global/networks/kubernetes-the-hard-way-dev-vpc'
) t 
WHERE 
source_range = '0.0.0.0/0' 
and direction = 'INGRESS';

This query provides a comprehensive list of all ingress firewall rules that apply to any IP address (0.0.0.0/0) within the specified Google Cloud project and network. The results include the firewall rule name, the source IP range, the protocol, the allowed ports, and the direction of the traffic, an example is shown below:

|-----------------------------------------------|--------------|-------------|---------------|-----------|                                                                                         
|                     name                      | source_range | ip_protocol | allowed_ports | direction |                                                                                         
|-----------------------------------------------|--------------|-------------|---------------|-----------|                                                                                         
| default-allow-icmp                            | 0.0.0.0/0    | icmp        | null          | INGRESS   |                                                                                         
|-----------------------------------------------|--------------|-------------|---------------|-----------|                                                                                         
| default-allow-rdp                             | 0.0.0.0/0    | tcp         | ["3389"]      | INGRESS   |                                                                                         
|-----------------------------------------------|--------------|-------------|---------------|-----------|                                                                                         
| default-allow-ssh                             | 0.0.0.0/0    | tcp         | ["22"]        | INGRESS   |                                                                                         
|-----------------------------------------------|--------------|-------------|---------------|-----------|                                                                                         
| kubernetes-the-hard-way-dev-allow-external-fw | 0.0.0.0/0    | tcp         | ["22"]        | INGRESS   |                                                                                         
|-----------------------------------------------|--------------|-------------|---------------|-----------|                                                                                         
| kubernetes-the-hard-way-dev-allow-external-fw | 0.0.0.0/0    | tcp         | ["6443"]      | INGRESS   |                                                                                         
|-----------------------------------------------|--------------|-------------|---------------|-----------|                                                                                         
| kubernetes-the-hard-way-dev-allow-external-fw | 0.0.0.0/0    | icmp        | null          | INGRESS   |                                                                                         
|-----------------------------------------------|--------------|-------------|---------------|-----------| 

You can use this query to help quickly identify potential security vulnerabilities. Regularly auditing these rules ensures that your cloud environment remains secure and that only the necessary ports and protocols are open to the internet.

Give us your feedback! ⭐ us here!

StackQL allows you to query and interact with your cloud and SaaS assets using a simple SQL framework. Use cases include CSPM, asset inventory and analysis, finops and more, as well as IaC and sysops (lifecycle management).

Using stackql and the aws provider (AWS Cloud Control provider for stackql), here's how you can query your entire AWS estate in real time (globally) and generate a simple report like this...

Check out the code at AWS Global Inventory!

Visit us and give us a ⭐ on GitHub

StackQL allows you to query and interact with your cloud and SaaS assets using a simple SQL framework. Use cases include CSPM, asset inventory and analysis, finops and more, as well as our IaC and ops (lifecycle management).

The three major cloud providers all offer a built-in Linux shell for executing commands using their respective CLIs; in some cases, they come with tools like terraform pre-installed. They are pre-authorized with your credentials in the cloud console for the user you authenticated with.

Now you can easily use stackql - a unified analytics and IaC dev tool - in all major cloud providers' built-in shells, using cloud shell scripts packaged with the stackql Linux binary (available from v0.5.587 onwards).

StackQL is particularly useful for asynchronously querying across regions in AWS, projects in Google, or resource groups in Azure, which is challenging to do via the CLIs. For example:

SELECT region, COUNT(*) as num_functions
FROM aws.lambda.functions
WHERE region IN (
	'us-east-1','us-east-2','us-west-1','us-west-2',
	'ap-south-1','ap-northeast-3','ap-northeast-2',
	'ap-southeast-1','ap-southeast-2','ap-northeast-1',
	'ca-central-1','eu-central-1','eu-west-1',
	'eu-west-2','eu-west-3','eu-north-1','sa-east-1')
GROUP BY region;

Additionally, you could authenticate to another provider from one cloud shell simultaneously and run multi-cloud inventory commands. For example:

SELECT 
  name, 
  SPLIT_PART(machineType, '/', -1) as instance_type,
  'google' as provider
  FROM google.compute.instances 
  WHERE project IN ('myproject1','myproject2')
UNION
SELECT
  instanceId as name,
  instanceType as instance_type,
  'aws' as provider
  FROM aws.ec2.instances 
  WHERE region IN (
	'us-east-1','us-east-2','us-west-1','us-west-2',
	'ap-south-1','ap-northeast-3','ap-northeast-2',
	'ap-southeast-1','ap-southeast-2','ap-northeast-1',
	'ca-central-1','eu-central-1','eu-west-1',
	'eu-west-2','eu-west-3','eu-north-1','sa-east-1');

Getting Started​

To get started with StackQL in your preferred cloud shell environment, download the StackQL package using the following command:

curl -L https://bit.ly/stackql-zip -O \
&& unzip stackql-zip

This command downloads the StackQL package, unzips it, and sets the appropriate permissions. From there, you can use our tailored scripts for AWS, Google Cloud, or Azure to integrate StackQL seamlessly into your cloud shell environment.

Using StackQL in the AWS Cloud Shell​

Run the stackql-aws-cloud-shell.sh as follows to use the StackQL command shell within the AWS cloud shell:

sh stackql-aws-cloud-shell.sh

An example is shown here:

You can also run stackql exec commands using the stackql-aws-cloud-shell.sh script; for instance, this command will write a CSV file for the results of a query that could be downloaded from the Cloud Shell.

sh stackql-aws-cloud-shell.sh exec \
--output csv --outfile instances.csv \
"SELECT region, instanceType FROM aws.ec2.instances WHERE region IN ('us-east-1')"

Additionally, you can supply an IAM role using the --role-arn argument to assume another role for your query or mutation operation, an example is shown here:

sh stackql-aws-cloud-shell.sh \
--role-arn arn:aws:iam::824532806693:role/SecurityReviewerRole exec \
--infile query.iql \
--output csv --outfile output.csv

Using StackQL in the Azure Cloud Shell​

Run the stackql-azure-cloud-shell.sh as follows to open a StackQL command shell from the Azure Cloud Shell:

sh stackql-azure-cloud-shell.sh

An example is shown here:

Similar to the AWS script, you can also invoke stackql exec as well, an example is shown here:

sh stackql-azure-cloud-shell.sh exec \
--output csv --outfile instances_by_location.csv \
"SELECT location, COUNT(*) as num_instances FROM azure.compute.virtual_machines WHERE resource_group_name =  'stackql-ops-cicd-dev-01' AND subscription_id = '631d1c6d-2a65-43e7-93c2-688bfe4e1468' GROUP BY location"

Using StackQL in the Google Cloud Shell​

Run the stackql-google-cloud-shell.sh as shown below to launch a StackQL command shell from within the google cloud shell:

sh stackql-google-cloud-shell.sh

An example is shown here:

As with the other two providers, you can run exec commands following the example below:

sh stackql-google-cloud-shell.sh exec \
--output csv --outfile instances.csv \
"SELECT name, status FROM google.compute.instances WHERE project = 'stackql-demo'"

Please give us your feedback! Star us at github.com/stackql.

You can leverage the powerful combination of StackQL and PowerBI to create comprehensive dashboard interfaces. These dashboards are perfect for reporting on various aspects such as cloud security, inventory, and configuration.

Quick Start Guide​

Set Up StackQL Server​

To get started, you can run a StackQL server container on port 7432. Use the following project for easy setup: StackQL Server on GitHub.

Create an ODBC Connection​

Next, set up an ODBC connection using the PostgreSQL ODBC driver. You can download the latest driver from the PostgreSQL ODBC Driver Versions. Install this driver on your local machine to proceed.

Integrating with PowerBI​

Once your ODBC connection is ready, you can move on to PowerBI. Here’s how you can integrate StackQL queries into PowerBI:

1. Create Data Sources in PowerBI: For each StackQL query that you want to visualize, create a new data source in PowerBI.

2. Test Queries Locally: Before integrating with PowerBI, you can test your StackQL queries locally using psql. For example:

   $ psql -h localhost -p 7432 -U stackql -d stackql
   psql (14.9 (Ubuntu 14.9-0ubuntu0.22.04.1), server 0.0.0)
   Type "help" for help.
   
   stackql=> select name, stargazers FROM
   (select name, stargazers_count as stargazers
   from github.repos.repos 
   where org = 'stackql' 
   and visibility = 'public'
   order by stargazers_count desc) t
   limit 3;
                  name                | stargazers 
   -----------------------------------+------------
    stackql                           | 179
    stackql-provider-registry         | 21
    google-discovery-to-openapi       | 18
   (3 rows)
   

3. Visualize with PowerBI: With your named data sources created, you can now visualize the result sets in PowerBI. Use various visualization tools like bar charts, pie charts, and line charts to create rich and insightful dashboards.

Let us know your thoughts! Visit us and give us a ⭐ on GitHub

Most AWS services and resources are regionally scoped, meaning the UI, CLI, SDKs, and all other methods of querying the aws provider give you a regional view (us-east-1 or ap-southeast-2, for instance). Many customer AWS estates span multiple regions - for multinational organizations, for example, or organizations with numerous dispersed locations within the US.

Sure, you could write custom scripts wrapping the CLI or SDKs - which would require development effort (not reusable for other providers); or get an abstract view with tools like AWS Config or Systems Manager, which requires these services to be enabled and configured (not flexible and not extendible to other providers). In either case:

1. You can't write and run customized queries and generate custom reports - as you can do in SQL
2. Any solutions you build will have to be rebuilt entirely for other providers

Using the latest (AWS provider for StackQL - which leverages the AWS Cloud Control API) and the executeQueriesAsync method in the pystackql Python package, I've put together an example here which runs a query to bring back attributes from all AWS Lambda functions deployed across 17 different AWS regions asynchronously. Results can be returned as a list of Python dictionaries or a Pandas dataframe. I am doing the former here, which took less than 10s.

from pystackql import StackQL
from pprint import pprint
from asyncio import run
stackql = StackQL()
stackql.executeStmt("REGISTRY PULL aws") # not required if the aws provider is already installed

async def stackql_async_queries(queries):
    return await stackql.executeQueriesAsync(queries)

regions= ["us-east-1","us-east-2","us-west-1","us-west-2","ap-south-1","ap-northeast-3","ap-northeast-2","ap-southeast-1",
         "ap-southeast-2","ap-northeast-1","ca-central-1","eu-central-1","eu-west-1","eu-west-2","eu-west-3","eu-north-1",
          "sa-east-1"]

# list functions from all regions asynchronously
get_fns = [
    f"""
    SELECT region, FunctionName AS function_name
    FROM aws.lambda.functions
    WHERE region = '{region}'
    """
    for region in regions
]

functions = run(stackql_async_queries(get_fns))

# get function details for all functions across all regions asynchronously
get_fn_details = [
    f"""
    SELECT 
    JSON_EXTRACT(Configuration, '$.FunctionName') AS function_name,
    region,
    JSON_EXTRACT(Configuration, '$.FunctionArn') AS arn,
    JSON_EXTRACT(Configuration, '$.Description') AS description,
    JSON_EXTRACT(Configuration, '$.Architectures') AS architectures,
    JSON_EXTRACT(Configuration, '$.MemorySize') AS memory_size,
    JSON_EXTRACT(Configuration, '$.Runtime') AS runtime
    FROM aws.lambda.functions
    WHERE region = '{function['region']}'
    AND data__Identifier = '{function['function_name']}'
    """
    for function in functions
]

function_details = run(stackql_async_queries(get_fn_details))
pprint(function_details)

which returns...

[{'architectures': '["x86_64"]',
  'arn': 'arn:aws:lambda:us-east-1:824532806693:function:stackql-helloworld-fn',
  'description': '',
  'function_name': 'stackql-helloworld-fn',
  'memory_size': '128',
  'region': 'us-east-1',
  'runtime': 'nodejs18.x'},
 {'architectures': '["x86_64"]',
  'arn': 'arn:aws:lambda:us-east-2:824532806693:function:stackql-helloworld-fn',
  'description': '',
  'function_name': 'stackql-helloworld-fn',
  'memory_size': '128',
  'region': 'us-east-2',
  'runtime': 'nodejs18.x'},
 {'architectures': '["x86_64"]',
  'arn': 'arn:aws:lambda:us-west-1:824532806693:function:stackql-helloworld-fn',
  'description': '',
  'function_name': 'stackql-helloworld-fn',
  'memory_size': '128',
  'region': 'us-west-1',
  'runtime': 'nodejs18.x'},
...

You could customize the StackQL query to run specific reports and visualize the results in a Jupyter notebook, for example:

• Functions by runtimes
• Function by memory size
• Functions by tags
• etc...

You could do something similar for other hyperscalars, for example, GCP, which scopes resources by projects, or Azure, which scopes resources by resource groups.

Let us know your thoughts! Visit us and give us a ⭐ on GitHub

Codespaces and the power of IPython and notebooks provide a quick and easy way to use StackQL to analyze and report on your cloud estate and resource configuration. No software necessary!

Setting Up StackQL in Codespaces​

See the stackql-codespaces-notebook repository as an example.

With the example devcontainer.json configuration file shown below, you can use the stackql/stackql-codespaces-base image, which includes stackql and the pystackql package (which provides the IPython magic extension used to run stackql queries and return Pandas dataframes for inspection or visualization).

{
  "image": "stackql/stackql-codespaces-base",
  "containerEnv": {
    "STACKQL_GITHUB_PASSWORD": "${secrets:STACKQL_GITHUB_PASSWORD}",
    "STACKQL_GITHUB_USERNAME": "${secrets:STACKQL_GITHUB_USERNAME}"
  },
  "hostRequirements": {
    "cpus": 2
  },  
  "customizations": {
    "vscode": {
      "extensions": [
        "ms-toolsai.jupyter",
        "ms-python.python"
      ]
    }
  }
}

Note that the devcontainer configuration includes the essential extensions for Jupyter and Python; you can also optionally specify host requirements for the Codespaces machine.

Provider Authentication​

The environment variables required to authenticate to your specific provider or providers can be supplied using Codespaces secrets and passed securely to the Codespaces container using the containerEnv object in the example above. For specifics about variables required for different providers, see the provider documentation for your provider in the StackQL Provider Registry.

Launching the Codespaces Environment​

Given the devcontainer configuration shown above in the root of your repository at .devcontainer/devcontainer.json, you can launch codespaces from <your_repo_slug>/codespaces for example https://github.com/stackql/stackql-codespaces-notebook/codespaces. You can start a codespaces environment on any branch of your repo.

Running queries and visualizing output​

When the environment is provisioned (usually takes a minute or two), you can use the StackQL magic extension and the %%stackql magic decorator to seamlessly run stackql queries, including variable substitution, for example.

%load_ext pystackql.magic

(loads the Stackql magic extension, making the %%stackql decorator available)

region = 'us-east-1'

(set some notebook variables for reusability)

%%stackql
SELECT instanceType, COUNT(*) as num_instances 
FROM aws.ec2.instances
WHERE region = '$region'
GROUP BY instanceType

(run a query)

_.plot(kind='pie', y='num_instances', labels=_['instanceType'], title='Instances by Type', autopct='%1.1f%%')

(visualize the results - using matplotlib, plotly, or any other visualization package)

heres an example:

Using the pystackql package, you can also run asynchronous queries such as querying assets across AWS regions, Azure resource groups (or subscriptions) or Google projects in one statement, for example:

# get multiple regions asynchronously
regions = ["ap-southeast-2", "us-east-1"]

queries = [
    f"""
    SELECT '{region}' as region, instanceType, COUNT(*) as num_instances
    FROM aws.ec2.instances
    WHERE region = '{region}'
    GROUP BY instanceType
    """
    for region in regions
]

instances_df = await stackql.executeQueriesAsync(queries)

Visit us and give us a ⭐ on GitHub

Visit us and give us a ⭐ on GitHub

Push tags​

In my case, I merged a PR to the main branch for an updated GitHub action in a repo called stackql-exec, then I pushed a tag with an updated semver:

git tag v1.2.1
git push origin v1.2.1

Pull the StackQL GitHub provider​

From the StackQL shell or via the exec command, pull the latest GitHub provider for StackQL using:

REGISTRY PULL github;

Generate a template for creating a release (optional)​

StackQL allows you to create resource templates for creating resources such as ec2 instances, google cloud storage buckets, or github releases. This is done using the SHOW INSERT INTO command as seen here:

stackql exec --output text "SHOW INSERT INTO github.repos.releases"

The template will also generate an optional jsonnet variable block to define reusable or externally sourced variables (e.g., from environment variables).

Run your INSERT statement​

In this case, I have just used literals to keep it simple...

INSERT INTO github.repos.releases(
  owner,
  repo,
  data__body,
  data__draft,
  data__generate_release_notes,
  data__name,
  data__prerelease,
  data__tag_name,
  data__target_commitish
)
SELECT
  'stackql',
  'stackql-exec',
  'Updated authentication and added support for external variables',
   false,
   false,
  'v1.2.1',
   false,
  'v1.2.1',
  'main'
;

SELECT FROM github.repos.releases (optional)​

You can inspect the releases created for your repo, including the one you just created, using a simple SELECT statement as shown here...

select 
name,
tag_name,
target_commitish,
created_at
from github.repos.releases where owner = 'stackql' and repo = 'stackql-exec';

You will see output like the following:

|--------------------------------|-------------|------------------|----------------------|                                              
|              name              |  tag_name   | target_commitish |      created_at      |                                              
|--------------------------------|-------------|------------------|----------------------|                                              
| v1.2.1                         | v1.2.1      | main             | 2023-09-03T05:44:40Z |                                              
|--------------------------------|-------------|------------------|----------------------|                                              
| v1.2.0                         | v1.2.0      | main             | 2023-09-03T05:20:42Z |                                              
|--------------------------------|-------------|------------------|----------------------|                                              
| Fix composite action issue     | v1.0.1      | main             | 2023-03-01T09:14:20Z |                                              
| with path                      |             |                  |                      |                                              
|--------------------------------|-------------|------------------|----------------------|                                              
| Production release with        | v1.0.0      | main             | 2023-02-25T23:07:32Z |                                              
| outputs                        |             |                  |                      |                                              
|--------------------------------|-------------|------------------|----------------------|                                              
| Initial release                | v1.0.0-beta | main             | 2023-02-05T12:50:01Z |                                              
|--------------------------------|-------------|------------------|----------------------|              

More information about the GitHub Provider for StackQL can be found here.

Enjoy!

The googleadmin StackQL provider is now available, which allows you to query, provision, or manage Google Workspace users, groups, devices, and more using StackQL. The googleadmin provider can be used with the google provider or other cloud providers to generate entitlements reports (or user access reviews) where Google Workspace identites are used in identity federation or IAM bindings.

The full documentation on how to use a Google service account for authentication to the googleadmin provider is available here. Information about the directory resources available and their fields and methods, is available in the StackQL Provider Registry Docs.

Simple Query​

A simple query using the googleadmin provider is shown here:

SELECT
 primaryEmail, 
 lastLoginTime 
FROM 
 googleadmin.directory.users 
WHERE domain = 'stackql.io'
AND primaryEmail = 'javen@stackql.io';

which would return the following results...

|------------------|--------------------------|                                                                                                                                                   
|   primaryEmail   |      lastLoginTime       |                                                                                                                                                   
|------------------|--------------------------|                                                                                                                                                   
| javen@stackql.io | 2023-07-08T23:30:31.000Z |                                                                                                                                                   
|------------------|--------------------------|  

Example Query Using Built-In Functions​

Here is an example using built-in functions in StackQL (more information about built-in functions is available in the StackQL docs):

SELECT
 primaryEmail,
 json_extract(name, '$.fullName') as full_name, 
 lastLoginTime 
FROM 
 googleadmin.directory.users 
WHERE domain = 'stackql.io'
AND primaryEmail = 'javen@stackql.io';

which would return results like this...

|------------------|--------------|--------------------------|                                                                                                                                    
|   primaryEmail   |  full_name   |      lastLoginTime       |                                                                                                                                    
|------------------|--------------|--------------------------|                                                                                                                                    
| javen@stackql.io | Jeffrey Aven | 2023-07-08T23:30:31.000Z |                                                                                                                                    
|------------------|--------------|--------------------------|  

Example Query Using Aggregate Functions​

Here is an example of a summary query that could be useful:

SELECT
  isAdmin,
  COUNT(*) as num_admins
FROM 
 googleadmin.directory.users 
WHERE domain = 'stackql.io'
GROUP BY isAdmin

results in...

|---------|------------|                                                                                                                                                                          
| isAdmin | num_admins |                                                                                                                                                                          
|---------|------------|                                                                                                                                                                          
| false   |          9 |                                                                                                                                                                          
|---------|------------|                                                                                                                                                                          
| true    |          2 |                                                                                                                                                                          
|---------|------------|  

Entitlements Report Using a LEFT JOIN with the google provider​

Using the LEFT OUTER JOIN capability with StackQL, you can generate entitlements or user access management reports that span across Google Workspace as an Identity Provider (IdP) and a Google Cloud resource (including Organizations, Folders, Projects, and resources), such as:

SELECT 
  split_part(json_extract(iam.members,'$[0]'), ':', 2) as member, 
  iam.role as role,
  users.lastLoginTime
FROM google.cloudresourcemanager.organizations_iam_bindings iam
LEFT OUTER JOIN googleadmin.directory.users users
ON split_part(json_extract(iam.members,'$[0]'), ':', 2) = users.primaryEmail
WHERE users.domain = 'stackql.io' 
AND iam.organizationsId = 141318256085
AND users.primaryEmail = 'javen@stackql.io';

which would return...

|------------------|------------------------------|--------------------------|                                                                                                                    
|      member      |             role             |      lastLoginTime       |                                                                                                                    
|------------------|------------------------------|--------------------------|                                                                                                                    
| javen@stackql.io | roles/bigquery.resourceAdmin | 2023-07-08T23:30:31.000Z |                                                                                                                    
|------------------|------------------------------|--------------------------|                                                                                                                    
| javen@stackql.io | roles/logging.admin          | 2023-07-08T23:30:31.000Z |                                                                                                                    
|------------------|------------------------------|--------------------------|

Let us know what you think!

This quick start guide outlines how to create a superset + stackql dashboard on your laptop using docker desktop, helm, and kubernetes. We certainly do not want to go into depth on superset, a third-party application, so this guide is terse.

Supplying secrets​

In this example, we use:

• AWS environment variable credentials as per the AWS CLI.
• Azure environment variables, as per the documentation for the golang SDK.
• digitalocean personal access token.
• github personal access token.
• google Service Account JSON key.

All of the associated principals must be granted access using provider-specific access controls.

NOTE keep all of these values secret and certainly do not commit into source control. We have supplied examples for numerous providers, and we suggest that you configure only what you need.

Create a file helm/stackql-dashboards/secrets/secret-values.yaml, containing the following, replacing placeholders:

stackql:
  extraSecretEnv:
    AWS_ACCESS_KEY_ID: '<your aws access key id>'
    AWS_SECRET_ACCESS_KEY: '<your aws secret key>'
    AZURE_CLIENT_ID: '<your azure client id>'
    AZURE_CLIENT_SECRET: '<your azure client secret>'
    AZURE_TENANT_ID: '<your azure tenant id>'
    DIGITALOCEAN_TOKEN: '<your digitalocean token>'
    STACKQL_GITHUB_TOKEN: '<your github personal access token>'
    GOOGLE_APPLICATION_CREDENTIALS: '/opt/stackql/config/google-credentials.json'
  extraSecrets:
    google-credentials.json: |
      <full google json key>

superset:
  init:
    adminUser:
      password: 'mypassword'

Expand templates and deploy locally​

Here we will set up and expose a local dashboard using the local kubernetes cluster supplied with docker desktop.

These steps assume that your kubectl config is pointed at your local cluster (depending on your version of docker, something like kubectl config use-context docker-desktop should do the trick) and that you execute from the root directory of the stackql-cloud repository. We will let the system dynamically assign a local port.

helm dependency update  helm/stackql-dashboards

helm template --release-name v1 --namespace default --set superset.service.type=NodePort --set superset.service.nodePort.http="" -f helm/stackql-dashboards/secrets/secret-values.yaml helm/stackql-dashboards > helm/stackql-dashboards/out/stackql-demo-dashboards.yaml

kubectl apply -f helm/stackql-dashboards/out/stackql-demo-dashboards.yaml

Log into and set up superset​

Allow a minute or so for init actions to complete.

First, inspect the output of kubectl get svc and note the host port for the service v1-superset. In my case, I see (redacted):

$ kubectl get svc | grep NodePort      
v1-superset                  NodePort    ...    ...        8088:31930/TCP   ...

So, my local port is 31390 on this occasion. Hereafter let us refer to this port as <SUPERSET_LOCAL_PORT>.

Go to your browser address bar and punch in http://localhost:<SUPERSET_LOCAL_PORT>. Log in using admin / mypassword (or other if you reconfigured), and then you can begin using superset.

From the top RHS Settings dropdown, select Database Connections. Then, select the + DATABASE button (just below Settings) and do the following (the password does not matter in this context, add anything you want):

Press "CONNECT"

Press "FINISH"

NOTE: we have enabled DML here so that meta queries like show and describe will work. You certainly do not have to do this if you don't want to.

Experiment​

Here we present a simple GCP scenario; you can follow the same pattern to create many charts and populate a dashboard...

Navigate to SQL > SQL Lab and then input the below, substituting <your gcp project> for whatever google project your service account can access:

select name, guestCpus from google.compute.machine_types where project = '<your gcp project>' and zone = 'australia-southeast1-a';

Press "RUN SELECTION"

A table of results should appear.

Press "Save" > "Save Dataset"

Give it whatever name you want.

You can click the option to create a chart immediately or navigate to your chart via the Charts menu item.

Once inside the UI for your new dataset, do something like this (we will leave it to your creativity)...

...and then...

Click on "SAVE & GO TO NEW DASHBOARD", and you have your first dashboard + stackql!