![\"stackql](\"https://stackql.io/assets/images/stackql-mcp-browse-connectors-b71dd6365c48b10d58c5f8da6b09bcba.png\")
StackQL v0.11 is out. The StackQL MCP server: now includes the current Model Context Protocol revision, 2026-07-28, alongside every earlier revision it already supported, and the audit log that records what an agent did can be written as OpenTelemetry log records instead of the bespoke JSONL format. Both are available today through every install channel.
2026-07-28 is the largest revision of the protocol since it launched. Relevant features for the StackQL MCP server incldue:
initialize / notifications/initialized handshake is gone. Every request carries the protocol version and the client's capabilities in its _meta block, and a client discovers a server with a single server/discover call.Mcp-Session-Id header, and tools/list, prompts/list and resources/list must return the same answer on every connection.input_required result, and the client retries the call with the answer attached (multi round-trip requests, SEP-2322).The StackQL MCP server negotiates per client, so a mixed fleet works against one server:
\n| Revision | Lifecycle | Approval prompt in safe / delete_safe mode |
|---|---|---|
2026-07-28 | No handshake; version and capabilities in _meta on every request | input_required result, retried with inputResponses |
2025-11-25, 2025-06-18 | initialize handshake | Server-initiated elicitation/create request |
2025-03-26, 2024-11-05 | as above | as above |
The gated-write flow is the headline feature of the server, so it was the acceptance test for this work: a mutation in safe mode still stops for approval, on both revisions, over both transports, and the audit record still says whether the user accepted, declined or dismissed the prompt. Nothing changes in how you run the server or in the SQL an agent writes.
The stdio transport serves every revision on one process. A current client's first request is served without a handshake, and an older client's initialize still works. If you use Claude Desktop, the npm or PyPI launchers, or the Docker image, there is nothing to configure.
stateless optionOver HTTP the protocol removed sessions, and the two models cannot share one listener. By default the server keeps the stateful, session-per-client model, which serves revisions up to 2025-11-25. A 2026-07-28 client learns that from server/discover and negotiates down, so existing HTTP integrations keep their sessions and their approval prompts unchanged.
To serve 2026-07-28 natively over HTTP, set stateless on the server:
stackql mcp --mcp.server.type=http \\
--mcp.config '{\"server\": {\"transport\": \"http\", \"address\": \"127.0.0.1:9992\", \"stateless\": true} }'
A sessionless server issues no Mcp-Session-Id, keeps the list endpoints connection-invariant, and runs the approval round trip through input_required. It still accepts an older client's initialize and serves reads to it, but it cannot retain the elicitation capability that client declared at handshake time, so an older client cannot approve gated writes on a sessionless server. Pick stateless for current-revision hosts, and leave the default for a fleet that still includes older clients. The protocol revision support section of the docs has the full matrix, alongside the server modes it interacts with.
Every tool call the server handles writes one audit record: the tool, the server mode, the gate decision, the verbatim SQL, the duration and any error. Until now that record was a line of StackQL-specific JSON, which meant a custom parser between the file and whatever you wanted to do with it.
\nThis release adds a second format. --mcp.log.format=otel writes the same records as OpenTelemetry log records, in the OTLP/JSON encoding, one export payload per line:
stackql mcp --mcp.server.type=stdio --mcp.log.format=otel
The default, jsonl, is unchanged byte for byte, and the format can also be set in mcp.config as \"audit\": {\"format\": \"otel\"}. The destination is the same rotating file either way. The log format reference lists every option and attribute.
A SELECT run through the server produces a record like this (pretty-printed; the file holds it on one line):
{
\"resourceLogs\": [{
\"resource\": {\"attributes\": [
{\"key\": \"service.name\", \"value\": {\"stringValue\": \"stackql\"}},
{\"key\": \"service.version\", \"value\": {\"stringValue\": \"0.10.623\"}}
]},
\"scopeLogs\": [{
\"scope\": {\"name\": \"github.com/stackql/stackql/pkg/mcp_server/audit\", \"version\": \"1.0.0\"},
\"logRecords\": [{
\"timeUnixNano\": \"1788652800123456789\",
\"observedTimeUnixNano\": \"1788652800123987654\",
\"severityNumber\": 9,
\"severityText\": \"INFO\",
\"body\": {\"stringValue\": \"execute_tool run_select_query\"},
\"attributes\": [
{\"key\": \"gen_ai.operation.name\", \"value\": {\"stringValue\": \"execute_tool\"}},
{\"key\": \"mcp.method.name\", \"value\": {\"stringValue\": \"tools/call\"}},
{\"key\": \"gen_ai.tool.name\", \"value\": {\"stringValue\": \"run_select_query\"}},
{\"key\": \"gen_ai.tool.call.id\", \"value\": {\"stringValue\": \"3f1c9a6b2e8d4a70\"}},
{\"key\": \"mcp.protocol.version\", \"value\": {\"stringValue\": \"2026-07-28\"}},
{\"key\": \"stackql.mode\", \"value\": {\"stringValue\": \"safe\"}},
{\"key\": \"stackql.decision\", \"value\": {\"stringValue\": \"allow\"}},
{\"key\": \"stackql.query\", \"value\": {\"stringValue\": \"select name, id from google.storage.buckets where project = 'stackql-demo'\"}},
{\"key\": \"stackql.query_class\", \"value\": {\"stringValue\": \"select\"}},
{\"key\": \"stackql.duration_ms\", \"value\": {\"intValue\": \"412\"}},
{\"key\": \"stackql.rows_returned\", \"value\": {\"intValue\": \"5\"}}
],
\"traceId\": \"16c62eeb8bb8c6e30917626336d61829\",
\"spanId\": \"ff6b2aa55252ac0e\"
}],
\"schemaUrl\": \"https://opentelemetry.io/schemas/1.44.0\"
}],
\"schemaUrl\": \"https://opentelemetry.io/schemas/1.44.0\"
}]
}
The attribute names follow the OpenTelemetry GenAI and MCP semantic conventions: gen_ai.operation.name is execute_tool, gen_ai.tool.name is the MCP tool, gen_ai.tool.call.id ties records of one call together, and mcp.method.name, mcp.protocol.version and mcp.session.id describe the protocol exchange. StackQL's own facts sit under a stackql. prefix: the verbatim stackql.query, its stackql.query_class, the stackql.mode and stackql.decision of the gate, stackql.duration_ms, stackql.rows_returned, and stackql.provider for metadata tools. A refused or failed call is an ERROR record with error.type and error.message.
A call that went through the approval gate produces a second record for the elicitation decision, sharing the same gen_ai.tool.call.id and trace id, so an accepted or declined write is visible as two correlated events.
The design inlcudes:
\n1.0.0) names the schema, the schemaUrl pins the stable conventions, and a test in the repository asserts the exact attribute list. When it changes, the scope version changes with it.If the calling agent propagates W3C trace context in params._meta (a traceparent key, as the MCP conventions describe), the records carry that trace and span. When it does not, the server generates one trace id per MCP session, so every record from one agent session still correlates in your backend.
The line format is exactly what the OpenTelemetry Collector's otlp_json_file receiver reads, so the stream reaches an OTLP pipeline with no transform processor. This configuration was verified against otelcol-contrib 0.160.0:
receivers:
otlp_json_file:
include: [/var/log/stackql-mcp.log]
exporters:
debug: {}
service:
pipelines:
logs:
receivers: [otlp_json_file]
exporters: [debug]
Swap the debug exporter for otlphttp, or for whichever backend you run, and the agent's activity lands next to the rest of your telemetry with the same resource, scope and trace semantics as everything else. Sinks and dashboards on top of the stream are the next step; this release makes the stream standards-shaped.
The OpenTelemetry encoding is not tied to the MCP server. It is implemented as a decorator in the generic pkg/sink package that wraps any record sink: a payload can describe its own log records, and any other JSON-shaped record is decorated automatically, with its top-level fields as typed attributes and the record as the body. The MCP audit is the first consumer; other StackQL log and result channels can adopt the same option without new plumbing.
stateless, modes and the audit log: stackql mcp--mcp.log.format⭐ Star us on GitHub and let us know what your agents build.
", "url": "https://stackql.io/blog/product/stackql-mcp-2026-07-28-and-opentelemetry", "title": "StackQL update: MCP protocol revision 2026-07-28 and OpenTelemetry output", "summary": "StackQL v0.11 brings the MCP server up to protocol revision 2026-07-28, and adds an OpenTelemetry (OTLP/JSON) output format for the agent audit log so any OTel collector can consume it without a custom parser.", "date_modified": "2026-09-05T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "mcp", "model context protocol", "opentelemetry", "ai agents", "ai" ] }, { "id": "https://stackql.io/blog/product/stackql-mcp-anthropic-directory", "content_html": "The StackQL MCP server has been reviewed by Anthropic and is now listed in the Anthropic MCP Directory. StackQL is a member of the Claude Partner Network, and the directory listing makes the MCP server discoverable and installable directly from within Claude Desktop - no manual bundle download, no custom connector configuration.
\nIn Claude Desktop, go to Settings -> Connectors -> Add -> Browse Connectors and search for \"stackql\":
\n
You can also view the listing directly at claude.ai/directory/stackql.
\nInstallation is one click.
\n![\"stackql](\"https://stackql.io/assets/images/stackql-mcp-install-connector-ee96b611297d647ca7f77346b06d0572.png\")
The listing is backed by the signed MCPB bundle published with each StackQL MCP release on GitHub, so the version you install from the directory is the same artifact you would get from a GitHub release - reviewed once, then kept current automatically as new releases are published.
\nThe StackQL MCP server exposes cloud and SaaS provider APIs to Claude as data sources accessed via SQL. Claude can discover providers, inspect resource schemas, and run queries against live control planes and data planes - AWS, Google, Azure, GitHub, Databricks, Snowflake, Confluent, Cloudflare, Okta, EntraID and others - using the same grammar across all of them.
\nThat makes the loop in a Claude conversation look like this: ask a question about your estate, Claude formulates a SELECT against the relevant provider, the query runs against the live API, and the answer reflects reality at the moment you asked - not a cached inventory or a state file. For provisioning and remediation workflows, INSERT and DELETE follow the same pattern.
Documentation for the StackQL MCP server is available at StackQL MCP Tools, StackQL MCP Server Usage and StackQL MCP Server Installation. Credential setup per provider, is available via the StackQL registry.
", "url": "https://stackql.io/blog/product/stackql-mcp-anthropic-directory", "title": "StackQL MCP server now available in the Anthropic MCP Directory", "summary": "The StackQL MCP server is listed in the Anthropic MCP Directory. It is now discoverable and installable directly from Claude Desktop, with future releases flowing through automatically from GitHub.", "date_modified": "2026-07-07T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "mcp", "anthropic", "claude" ] }, { "id": "https://stackql.io/blog/product/run-the-stackql-mcp-server-anywhere", "content_html": "The StackQL MCP server is now available through every runtime an agent is likely to live in: prebuilt Claude Desktop bundles, npm, PyPI, Docker, a GitHub Action, and the Official MCP Registry. It is the same server in each case - one binary, pulled and launched the way your environment prefers.
\nStackQL exposes cloud and SaaS providers - AWS, Google Cloud, Azure, GitHub, Kubernetes, Snowflake, Databricks and more - as a single SQL surface. The MCP server puts that surface in front of an AI agent: the agent discovers providers, services, resources and methods, then runs SELECT queries to read state and (when you allow it) INSERT / UPDATE / DELETE to change it. Reads and writes are gated by a server mode and recorded to an audit log, so \"what the agent did\" is always answerable.
For background on the protocol itself, see the original StackQL MCP Server Now Available post and the MCP command reference.
\nEvery channel runs the same stackql binary. Pick the one that matches your client:
| Channel | Get it | Best for |
|---|---|---|
| Claude Desktop bundle | stackql-mcp-<platform>.mcpb from the release page | One-click install, no separate StackQL on PATH |
| npm | npx -y @stackql/mcp-server | Node environments, no global install |
| PyPI | uvx stackql-mcp-server or pip install stackql-mcp-server | Python environments |
| Docker | docker run -i --rm stackql/stackql-mcp | Containerised / isolated runtimes (amd64 + arm64) |
| GitHub Action | stackql/setup-stackql-mcp@v1 | CI and agentic workflows |
| MCP Registry | io.github.stackql/stackql-mcp | Directory-driven discovery and install |
A typical stdio client config is three lines. For npx:
\n{ \"mcpServers\": { \"stackql\": { \"command\": \"npx\", \"args\": [\"-y\", \"@stackql/mcp-server\"] } } }
Swap npx for uvx stackql-mcp-server or docker run -i --rm stackql/stackql-mcp and you have the Python or Docker form. The npm and PyPI launchers download the signed stackql binary on first run, verify its checksum, and share a single cache. The full matrix - including the manual claude_desktop_config.json form for an existing binary - is in Installing the MCP server.
Letting an agent touch your cloud is a trust decision, so the supply chain is built to be checkable:
\nstackql binary is Authenticode-signed (Windows) and Apple-notarised (macOS)..mcpb bundle ships with a published SHA-256 checksum on the release page.On top of the supply chain, the server defaults to mode: safe - reads run freely, mutations and lifecycle operations need approval through the MCP elicitation flow. Pin read_only for inventory agents that should never write, or full_access for trusted automation. See Server modes.
The GitHub Action is where the multi-vector story pays off. stackql/setup-stackql-mcp@v1 installs the binary and writes an MCP config (defaulting to read_only), and anthropics/claude-code-action consumes it through claude_args. The result is an agent that audits your AWS account on every run and files an issue with the SQL it used as evidence:
- id: stackql
uses: stackql/setup-stackql-mcp@v1
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
with:
mode: read_only
- uses: anthropics/claude-code-action@v1
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: |
Using the stackql tools, audit our AWS account for: S3 buckets without
encryption or with public access, security groups open to 0.0.0.0/0 on
sensitive ports, and IAM users without MFA. Open a GitHub issue
\"Cloud audit <date>\" summarising findings WITH the SQL you ran as
evidence. If nothing is found, do not open an issue.
claude_args: |
--mcp-config ${{ steps.stackql.outputs.mcp-config-file }}
--allowedTools 'mcp__stackql__*'
Because the config is pinned to read_only, the audit can read everything and change nothing - the safety contract is enforced by the server, not by trust in the prompt. The action README has more recipes, including cost estimates on a pull request and a credential-free GitHub inventory.
Under the hood the agent works the StackQL hierarchy with the same tools whatever the runtime. Pulling the GitHub provider and listing its services looks like this:
\n> pull_provider {\"provider\": \"github\"}
github provider, version 'v26.05.00393' successfully installed
> list_services {\"provider\": \"github\"}
actions, activity, apps, billing, checks, code_scanning, codespaces,
copilot, dependabot, gists, git, issues, orgs, packages, projects,
pulls, repos, search, secret_scanning, teams, users, ...
From there the agent can call list_resources and list_methods to discover the required WHERE parameters, then run_select_query to answer a question like \"how many public repositories does the stackql org have?\" - all without anyone hand-writing SQL.
⭐ Star us on GitHub and tell us what your agents build.
", "url": "https://stackql.io/blog/product/run-the-stackql-mcp-server-anywhere", "title": "Run the StackQL MCP Server Anywhere Your Agent Does", "summary": "The StackQL MCP server now ships via npm, PyPI, Docker, GitHub Actions, prebuilt Claude Desktop bundles, and the Official MCP Registry - one server, every runtime, signed and SHA-pinned.", "date_modified": "2026-06-15T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "mcp", "model context protocol", "ai agents", "claude", "infrastructure-as-code", "ai" ] }, { "id": "https://stackql.io/blog/product/stackql-deploy-2-0-rust-rewrite", "content_html": "stackql-deploy 2.0 is a full rewrite in Rust. The Python package (stackql-deploy on PyPi) is archived at 1.9.4. CLI interface and stack file format are unchanged - no migration required.
The move to Rust was primarily about distribution and operational simplicity. Rust also brings stronger guarantees around performance and memory safety. Running everything in-process without Foreign Function Interface (FFI) boundaries simplifies the architecture while maintaining predictable resource usage.
\nThe most significant functional change in 2.0 is that stackql-deploy now runs the StackQL engine as an embedded in-process server over a local postgres wire protocol connection rather than shelling out to the StackQL binary as an external process.
There is nothing to start, stop, or configure. The server is lifecycle-managed by stackql-deploy itself and binds to localhost only - no port is exposed on the network, no inbound firewall rules needed in CI.
The previous model spawned a new StackQL process per operation. The embedded server keeps a persistent connection for the duration of a deployment run. For stacks with many resources, the reduction in process spawn overhead is noticeable - particularly on Windows where process creation is expensive.
\nIn addition to added the architectural change to use the embedded server, several other workflow improvements were added including:
\n/*+ exists */ queries: When an exists query returns a named field (e.g. vpc_id) instead of count, the value is captured as a resource-scoped variable (this.vpc_id) and made available to all subsequent queries for that resource (e.g. statecheck, exports). This eliminates the need for redundant lookups to resolve resource identifiers between query stages.RETURNING payloads from DML operations: INSERT, UPDATE, and DELETE statements can include a RETURNING clause. Fields from the response can be mapped to resource-scoped variables via return_vals in the manifest, keyed by operation (create, update, delete). This allows identifiers assigned by the provider during creation to be used immediately without a round-trip query.to_aws_tag_filters filter, which converts global_tags to the AWS Resource Groups Tagging API TagFilters format, and type-preserving YAML-to-JSON serialization that maintains string types through the rendering pipeline.The canonical install URL detects your OS and redirects to the latest release asset automatically. You can also download directly from your browser at get-stackql-deploy.io.
curl -L https://get-stackql-deploy.io | tar xzf -
Use the PowerShell script below or download directly from your browser at get-stackql-deploy.io.
Invoke-WebRequest https://get-stackql-deploy.io -OutFile stackql-deploy.zip
Expand-Archive stackql-deploy.zip -DestinationPath .
Pre-built binaries are attached to every release on the releases page. A SHA256SUMS file is included for verification.
| Platform | Asset |
|---|---|
| Linux x86_64 | stackql-deploy-linux-x86_64.tar.gz |
| Linux arm64 | stackql-deploy-linux-arm64.tar.gz |
| macOS Universal (Apple Silicon + Intel) | stackql-deploy-macos-universal.tar.gz |
| Windows x86_64 | stackql-deploy-windows-x86_64.zip |
Use the stackql/stackql-deploy-action to run build or test commands in your CI pipeline. Provider credentials are passed via environment variables sourced from GitHub Actions Secrets.
Deploy a stack:
jobs:
deploy:
runs-on: ubuntu-latest
env:
GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}
steps:
- uses: actions/checkout@v4
- uses: stackql/stackql-deploy-action@v2
with:
command: 'build'
stack_dir: 'examples/my-stack'
stack_env: 'prod'
env_vars: 'GOOGLE_PROJECT=my-project'
Deploy and capture outputs:
- name: Deploy Stack
id: stackql-deploy
uses: stackql/stackql-deploy-action@v2
with:
command: 'build'
stack_dir: 'examples/my-stack'
stack_env: 'prod'
output_file: 'deployment-outputs.json'
env_vars: 'GOOGLE_PROJECT=my-project'
- name: Use outputs
run: |
echo '${{ steps.stackql-deploy.outputs.deployment_outputs }}' | jq .
See the stackql-deploy-action repo or the GitHub Action Marketplace for the full input/output reference.
If you have the Rust toolchain installed (rustup is the recommended way to get it on any platform), cargo install builds and installs the binary directly from crates.io.
cargo install stackql-deploy
The CLI interface is unchanged from the Python version:
\n# deploy a stack
stackql-deploy build my-stack prod \\
--e GOOGLE_PROJECT=${GOOGLE_PROJECT}
# test a stack
stackql-deploy test my-stack prod \\
--e GOOGLE_PROJECT=${GOOGLE_PROJECT}
# tear down a stack
stackql-deploy teardown my-stack prod \\
--e GOOGLE_PROJECT=${GOOGLE_PROJECT}
# dry run
stackql-deploy build my-stack prod \\
--e GOOGLE_PROJECT=${GOOGLE_PROJECT} \\
--dry-run
Stack files and stackql_manifest.yml structure are unaffected - no migration work needed.
stackql-deploy 1.9.4 on PyPi is the final Python release. The Python source repository is archived. If you have pip install stackql-deploy in any scripts or CI pipelines, replace it with one of the install methods above. The 1.9.4 package remains on PyPi and installable, but will not receive updates.
![\"AAIF](\"https://stackql.io/img/aaif_memberbadge_silver.svg\")
\n![\"Linux](\"https://stackql.io/img/LF_MemberLevel_silver.svg\")
\nStackQL Studios has joined the Linux Foundation and the Agentic AI Foundation (AAIF) as a Silver Member.
\nAs agentic AI moves from experimentation toward production workloads, we believe the infrastructure layer matters. Open, interoperable standards will be essential for AI agents to work reliably across cloud providers, data platforms, and enterprise systems.
\nStackQL provides a SQL-based interface for querying, provisioning, and managing cloud infrastructure across providers. Our work on the StackQL MCP Server brings this capability to AI agents through the Model Context Protocol, enabling agents to interact with cloud resources using natural language while maintaining the governance and auditability that production systems require.
\nJoining AAIF aligns with our long-standing commitment to open-source infrastructure tooling. We look forward to contributing to the foundation's work on MCP, agent runtimes, and the broader ecosystem of standards that will shape how AI agents interact with the systems they manage.
\nMore information about the Agentic AI Foundation is available at aaif.io.
", "url": "https://stackql.io/blog/product/stackql-joins-linux-foundation-agentic-ai-foundation", "title": "StackQL Joins the Linux Foundation and Agentic AI Foundation", "summary": "StackQL Studios joins the Linux Foundation and Agentic AI Foundation as a Silver Member.", "date_modified": "2026-01-01T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "open-source", "linux-foundation", "agentic-ai", "mcp" ] }, { "id": "https://stackql.io/blog/product/window-functions-and-ctes-now-available", "content_html": "Window functions and Common Table Expressions (CTEs) are now generally available in StackQL. These features work with both the embedded SQLite backend and PostgreSQL backend.
\nWindow functions allow you to perform calculations across sets of rows related to the current row. Supported functions include:
\nROW_NUMBER(), RANK(), DENSE_RANK(), NTILE()LAG(), LEAD(), FIRST_VALUE(), LAST_VALUE(), NTH_VALUE()PERCENT_RANK(), CUME_DIST()SUM(), COUNT(), AVG(), etc. with OVER clauseSELECT
login,
contributions,
DENSE_RANK() OVER (ORDER BY contributions DESC) as rank
FROM github.repos.contributors
WHERE owner = 'stackql' AND repo = 'stackql';
SELECT
login,
contributions,
SUM(contributions) OVER (ORDER BY contributions DESC) as running_total,
ROUND(100.0 * contributions / SUM(contributions) OVER (), 2) as pct_of_total
FROM github.repos.contributors
WHERE owner = 'stackql' AND repo = 'stackql';
CTEs let you define temporary named result sets using the WITH clause. This simplifies complex queries by breaking them into logical components.
WITH all_contributors AS (
SELECT login, contributions
FROM github.repos.contributors
WHERE owner = 'stackql' AND repo = 'stackql'
UNION ALL
SELECT login, contributions
FROM github.repos.contributors
WHERE owner = 'stackql' AND repo = 'stackql-deploy'
)
SELECT
DENSE_RANK() OVER (ORDER BY SUM(contributions) DESC) as rank,
login,
SUM(contributions) as total_contributions
FROM all_contributors
GROUP BY login
ORDER BY total_contributions DESC;
Full documentation is available:
\n\nLet us know your thoughts! Visit us and give us a star on GitHub.
", "url": "https://stackql.io/blog/product/window-functions-and-ctes-now-available", "title": "Window Functions and CTEs Now Available in StackQL", "summary": "Window functions and Common Table Expressions (CTEs) are now available in StackQL for advanced analytics across cloud and SaaS resources.", "date_modified": "2025-12-04T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "analytics", "window functions", "cte", "sql" ] }, { "id": "https://stackql.io/blog/product/stackql-provider-development-skill-for-claude", "content_html": "We've published a Claude Skill for StackQL provider development. It provides Claude with the context needed to help you build providers using the any-sdk library (interface used by StackQL to interact with the cloud providers).
\nThe skill covers the full provider development workflow:
\nprovider.yaml schema and service document layoutx-stackQL-* extensions including x-stackQL-resources, x-stackQL-config, x-stackQL-objectKey, and othersSELECT, INSERT, UPDATE, and DELETEA Claude Skill is a markdown file that provides Claude with specialized knowledge for a particular domain or task. When you add a skill to your project, Claude can reference it during conversations to give more accurate, context-aware responses.
To use this skill:
Once added, Claude will automatically apply this knowledge when you're working on StackQL provider development.
Grab the skill from the any-sdk repository:
\nDownload stackql-provider-development.md
\nWith the skill loaded, you can ask Claude things like:
\nClaude will have the full context of StackQL's OpenAPI extensions and provider patterns to work from.
\nHere's a sample interaction after loading the skill:
\nYou: I have an API endpoint GET /users that returns {\"data\": {\"users\": [...]}}.
How do I set up the resource?
Claude: For that response structure, you'd configure the method like this:
methods:
list:
operation:
$ref: '#/paths/~1users/get'
response:
mediaType: application/json
openAPIDocKey: '200'
objectKey: $.data.users
The objectKey uses JSONPath to extract the users array from the nested response.
⭐ Star us on GitHub
", "url": "https://stackql.io/blog/product/stackql-provider-development-skill-for-claude", "title": "StackQL Provider Development Skill for Claude", "summary": "A Claude Skill for developing StackQL providers, covering OpenAPI extensions, resource definitions, SQL verb mappings, and more.", "date_modified": "2025-11-28T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "claude", "ai", "provider development" ] }, { "id": "https://stackql.io/blog/product/markdownkv-output-format-available-in-pystackql", "content_html": "pystackql now includes a markdownkv output format optimized for LLM processing of control plane and data plane data from cloud providers.
Recent research from ImprovingAgents.com tested 11 data formats to determine which ones LLMs parse most accurately. Using 1,000 synthetic employee records and 1,000 randomized queries, they measured how well different formats preserved data integrity through LLM processing.
\nThe results:
\n| Format | Accuracy | 95% CI |
|---|---|---|
| Markdown-KV | 60.7% | 57.6%–63.7% |
| JSON | 52.3% | 49.2%–55.4% |
| Markdown Tables | 51.9% | 48.8%–55.0% |
| JSONL | 45.0% | 41.9%–48.1% |
| CSV | 44.3% | 41.2%–47.4% |
Markdown-KV showed a 37% improvement over CSV and 16 percentage points over JSON. The tradeoff: it uses approximately 2.7x more tokens than CSV.
\nMarkdown-KV uses hierarchical markdown headers with code blocks for key-value pairs:
\n# Query Results
## Record 1
id: i-1234567890abcdef0
name: prod-web-01
region: us-east-1
instance_type: t3.large
state: running
## Record 2
id: i-0987654321fedcba0
name: staging-web-01
region: us-west-2
instance_type: t3.medium
state: stopped
The format combines clear hierarchy, explicit key-value pairs, and readability for both humans and LLMs.
\nfrom pystackql import StackQL
stackql = StackQL()
# Query with Markdown-KV output
result = stackql.execute(
\"\"\"
SELECT instanceId, instanceType, state, availabilityZone
FROM aws.ec2.instances
WHERE region = 'us-east-1'
\"\"\",
output='markdownkv'
)
# Use with LLMs
response = llm_client.complete(
f\"Identify instances that should be stopped:\\n\\n{result}\"
)
Works in server mode too:
\nstackql = StackQL(server_mode=True)
result = stackql.execute(
\"SELECT name, region, encryption FROM google.storage.buckets WHERE project = 'my-project'\",
output='markdownkv'
)
Markdown-KV is useful when:
\nThe token cost is a real tradeoff, but infrastructure queries typically return targeted result sets, not massive datasets. When you're asking an LLM to analyze your production environment, accuracy matters.
\nUpdate pystackql:
pip install --upgrade pystackql
Add output='markdownkv' to your execute calls or in the StackQL object instantiation:
result = stackql.execute(query, output='markdownkv')
The Markdown-KV output format is available in pystackql v3.8.2 and later.
⭐ Star us on GitHub and join our community!
", "url": "https://stackql.io/blog/product/markdownkv-output-format-available-in-pystackql", "title": "Markdown-KV Output Format Available in pystackql", "summary": "New output format in pystackql optimizes cloud infrastructure data for LLM processing based on research showing accuracy improvement over CSV.", "date_modified": "2025-11-10T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "pystackql", "llm", "ai", "infrastructure-as-code" ] }, { "id": "https://stackql.io/blog/product/stackql-mcp-server-now-available", "content_html": "StackQL now supports the Model Context Protocol (MCP). This integration enables AI agents and assistants to query and manage cloud infrastructure across multiple providers using natural language.
\nThe Model Context Protocol is an open standard that enables AI applications to securely connect to external data sources and tools. By running StackQL as an MCP server, AI agents like Claude, ChatGPT, and other LLM-based assistants can interact with your cloud infrastructure using StackQL's powerful SQL-based query capabilities.
\nCombining MCP with StackQL creates a powerful interface for AI-assisted infrastructure management:
\nStackQL's MCP server supports three flexible deployment modes to suit different architectural requirements:
\nPerfect for development and AI agent integration:
\nstackql mcp \\
--mcp.server.type=http \\
--mcp.config '{\"server\": {\"transport\": \"http\", \"address\": \"127.0.0.1:9912\"}}'
Run both MCP and PostgreSQL wire protocol simultaneously with high-performance in-memory communication:
\nstackql srv \\
--mcp.server.type=http \\
--mcp.config '{\"server\": {\"transport\": \"http\", \"address\": \"127.0.0.1:9912\"}}' \\
--pgsrv.port 5665
This mode is ideal when you need both AI agent access and traditional database client connectivity.
\nFor production environments requiring distributed deployments and encrypted connections:
\nstackql srv \\
--mcp.server.type=reverse_proxy \\
--mcp.config '{\"server\": {\"tls_cert_file\": \"/path/to/cert.pem\", \"tls_key_file\": \"/path/to/key.pem\", \"transport\": \"http\", \"address\": \"127.0.0.1:9004\"}, \"backend\": {\"dsn\": \"postgres://stackql:stackql@127.0.0.1:5446?default_query_exec_mode=simple_protocol\"}}' \\
--pgsrv.port 5446
When running as an MCP server, StackQL exposes several tools that AI agents can invoke:
\n| Tool | Description |
|---|---|
greet | Test connectivity with the MCP server |
list_providers | List all available StackQL providers |
list_services | List services for a specific provider |
list_resources | List resources within a provider service |
list_methods | List available methods for a resource |
query_v2 | Execute StackQL queries |
To integrate StackQL with Claude Desktop, add this configuration to your MCP settings file (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):
{
\"mcpServers\": {
\"stackql\": {
\"command\": \"stackql\",
\"args\": [
\"mcp\",
\"--mcp.server.type=http\",
\"--mcp.config\",
\"{\\\"server\\\": {\\\"transport\\\": \\\"http\\\", \\\"address\\\": \\\"127.0.0.1:9912\\\"}}\"
]
}
}
}
Once configured, you can ask your AI assistant questions like:
\nThe AI agent will use StackQL's MCP server to execute the appropriate queries and return structured results.
\nHere's how an AI agent interacts with StackQL via MCP:
\n# AI agent lists available providers
Tool: list_providers
Response: [\"google\", \"aws\", \"azure\", \"github\", ...]
# AI agent explores a provider's services
Tool: list_services
Args: {\"provider\": \"google\"}
Response: [\"compute\", \"storage\", \"cloudresourcemanager\", ...]
# AI agent executes a query
Tool: query_v2
Args: {\"sql\": \"SELECT name, status FROM google.compute.instances WHERE project = 'my-project' AND zone = 'us-east1-a'\"}
Response: [{\"name\": \"instance-1\", \"status\": \"RUNNING\"}, ...]
Download StackQL version 0.9.250 or later from stackql.io/install
\nSet up provider authentication:
\nexport GOOGLE_CREDENTIALS=$(cat /path/to/credentials.json)
export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
stackql mcp \\
--mcp.server.type=http \\
--mcp.config '{\"server\": {\"transport\": \"http\", \"address\": \"127.0.0.1:9912\"}}'
For comprehensive documentation on configuring and using the MCP server, including:
\nVisit the MCP command documentation.
\nWe're actively developing additional MCP capabilities and welcome your feedback. Future enhancements may include:
\nThe MCP server feature is available now in StackQL 0.9.250. We'd love to hear about your experiences integrating StackQL with AI agents. Share your use cases, provide feedback, or contribute to the project on GitHub.
\n⭐ Star us on GitHub and join our community!
", "url": "https://stackql.io/blog/product/stackql-mcp-server-now-available", "title": "StackQL MCP Server Now Available", "summary": "Enable AI agents to query and manage cloud infrastructure using the Model Context Protocol.", "date_modified": "2025-11-02T00:00:00.000Z", "author": { "name": "Kieran Rimmer", "url": "https://www.linkedin.com/in/kieranrimmer/" }, "tags": [ "stackql", "mcp", "model context protocol", "ai agents", "claude", "infrastructure-as-code", "ai" ] }, { "id": "https://stackql.io/blog/product/stackql-deploy-docs-site-live", "content_html": "The stackql-deploy docs site is now available, offering a comprehensive guide to using stackql-deploy for your cloud resource deployments and tests. The site includes detailed documentation, examples, and best practices to help you get started quickly and effectively.
stackql-deploy is a declarative, stateless (and state file-less) infrastructure-as-code and test framework, driven by stackql queries. stackql-deploy is capable of provisioning, updating, de-provisioning and testing cloud and SaaS stacks across all cloud and SaaS providers.
![\"stackql-deploy-github-actions-screenshot\"](\"https://stackql.io/assets/images/stackql-deploy-github-actions-screenshot-e8012484620647fb48c612fed13c1400.png\")
Let us know what you think! ⭐ us on GitHub.
", "url": "https://stackql.io/blog/product/stackql-deploy-docs-site-live", "title": "stackql-deploy Docs Site Live", "summary": "Use stackql-deploy to deploy and test cloud resource deployments.", "date_modified": "2024-08-24T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "devops", "infrastructure", "github actions", "cloud security", "CI/CD", "iac", "infrastructure-as-code", "dbt" ] }, { "id": "https://stackql.io/blog/product/linux-arm64-build-available", "content_html": "StackQL is now available for ARM64-based Linux systems. To download the ARM64 binary, please visit our downloads page, where you can find the appropriate version for your system.
\nLet us know what you think! ⭐ us on GitHub.
", "url": "https://stackql.io/blog/product/linux-arm64-build-available", "title": "Linux arm64 StackQL Binary Available", "summary": "ARM64 architecture build available for StackQL.", "date_modified": "2024-08-20T00:00:00.000Z", "author": { "name": "Kieran Rimmer", "url": "https://www.linkedin.com/in/kieranrimmer/" }, "tags": [ "stackql", "linux", "arm", "arm64", "devops", "infrastructure", "cloud security", "CI/CD", "iac", "infrastructure-as-code" ] }, { "id": "https://stackql.io/blog/product/stackql-deploy-github-action-available", "content_html": "stackql-deploy is now available in the GitHub Actions Marketplace.
stackql-deploy is a declarative, stateless (and state file-less) infrastructure-as-code and test framework, driven by stackql queries. stackql-deploy is capable of provisioning, updating, de-provisioning and testing cloud and SaaS stacks across all cloud and SaaS providers.
Given this example stackql-deploy stack definition in a GitHub repo, you would simply add the following to your GitHub Actions workflow:
...
jobs:
stackql-actions-test:
name: StackQL Actions Test
runs-on: ubuntu-latest
env:
GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Deploy a Stack
uses: stackql/setup-deploy@v1.0.1
with:
command: build
stack-dir: examples/k8s-the-hard-way
stack-env: dev
env-vars: GOOGLE_PROJECT=stackql-k8s-the-hard-way-demo
Example output is shown here:
\n
Let us know what you think! ⭐ us on GitHub.
", "url": "https://stackql.io/blog/product/stackql-deploy-github-action-available", "title": "GitHub Action available for stackql-deploy", "summary": "Use stackql-deploy in a GitHub Actions workflow to provision, update and test infrastructure.", "date_modified": "2024-08-10T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "devops", "infrastructure", "github actions", "cloud security", "CI/CD", "iac", "infrastructure-as-code", "dbt" ] }, { "id": "https://stackql.io/blog/product/introducing-model-driven-iac-with-stackql-deploy", "content_html": "stackql-deploy is a multi-cloud resource provisioning framework using stackql. It is inspired by dbt (data build tool), which manages data transformation workflows in analytics engineering by treating SQL scripts as models that can be built, tested, and materialized incrementally. With StackQL, you can create a similar framework for cloud and SaaS provisioning. The goal is to treat infrastructure-as-code (IaC) queries as models that can be deployed, managed, and interconnected.
This ELT/model-based framework for IaC allows you to provision, test, update, and tear down multi-cloud stacks, similar to how dbt manages data transformation projects, with the benefits of version control, peer review, and automation. This approach enables you to deploy complex, dependent infrastructure components in a reliable and repeatable manner.
\nStackQL simplifies the interaction with cloud resources by using SQL-like syntax, making it easier to define and execute complex cloud management operations. Resources are provisioned with INSERT statements, and tests are structured around SELECT statements.
Features include:
\nstackql-deployTo get started with stackql-deploy, run the following:
pip install stackql-deploy
stackql-deploy will automatically download the latest release of stackql using the pystackql Python package. You can then use the info command to get runtime information:
$ stackql-deploy info
stackql-deploy version: 1.1.0
pystackql version : 3.6.1
stackql version : v0.5.612
stackql binary path : /home/javen/.local/stackql
platform : Linux x86_64 (Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35), Python 3.10.12
A stackql-deploy project is a directory with declarative SQL definitions to provision, de-provision, or test resources in a stack. The key components and their definitions are listed here:
stackql_manifest.yml : The manifest file for your project, defining resources and properties in your stack.stackql_resources directory : Contains StackQL queries to provision and de-provision resources in your stack.stackql_tests directory : Contains StackQL queries to test the desired state for resources in your stack.Use the init command to create a starter project directory:
stackql-deploy init activity_monitor
You will now have a directory named activity_monitor with stackql_resources and stackql_tests directories and a sample stackql_manifest.yml file, which will help you to get started.
The general syntax for stackql-deploy is described here:
stackql-deploy [OPTIONS] COMMAND [ARGS]...
Commands include:
\nbuild: Create or update resources based on the defined stack.teardown: Remove or decommission resources that were previously deployed.test: Execute test queries to verify the current state of resources against the expected state.info: Display the version information of the stackql-deploy tool and current configuration settings.init: Initialize a new project structure for StackQL deployments.Optional global options (for all commands) include:
\n--custom-registry TEXT: Specify a custom registry URL for StackQL. This URL will be used by all commands for registry interactions.--download-dir TEXT: Define a download directory for StackQL where all files will be stored.--help: Show the help message and exit.Options for build, test, and teardown include:
--on-failure [rollback|ignore|error]: Define the action to be taken if the operation fails. Options include rollback, ignore, or treat as an error.--dry-run: Perform a simulation of the operation without making any actual changes.-e <TEXT TEXT>...: Specify additional environment variables in key-value pairs.--env-file TEXT: Path to a file containing environment variables.--log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]: Set the logging level to control the verbosity of logs during execution.Using the activity_monitor stack we created previously using the init command, we can start defining a stack and defining the associated queries; here is the manifest file:
version: 1
name: activity_monitor
description: oss activity monitor stack
providers:
- azure
globals:
- name: subscription_id
description: azure subscription id
value: \"{{ vars.AZURE_SUBSCRIPTION_ID }}\"
- name: location
value: eastus
- name: resource_group_name_base
value: \"activity-monitor\"
resources:
- name: monitor_resource_group
description: azure resource group for activity monitor
props:
- name: resource_group_name
description: azure resource group name
value: \"{{ globals.resource_group_name_base }}-{{ globals.stack_env }}\"
\t# more resources would go here...
\n\n\n
globals.stack_envis a variable referencing the user-specified environment label.
Our stackql_resources directory must contain a .iql file (StackQL query file) with the same name as each resource defined in the resources key in the manifest file. Here is an example for stackql_resources/monitor_resource_group.iql:
/*+ createorupdate */
INSERT INTO azure.resource.resource_groups(
resource_group_name,
subscription_id,
data__location
)
SELECT
'{{ resource_group_name }}',
'{{ subscription_id }}',
'{{ location }}'
/*+ delete */
DELETE FROM azure.resource.resource_groups
WHERE resource_group_name = '{{ resource_group_name }}' AND subscription_id = '{{ subscription_id }}'
Similarly, our stackql_tests directory must contain a .iql file (StackQL query file) with the same name as each resource defined in the stack. Here is an example for stackql_tests/monitor_resource_group.iql:
/*+ preflight */
SELECT COUNT(*) as count FROM azure.resource.resource_groups
WHERE subscription_id = '{{ subscription_id }}'
AND resource_group_name = '{{ resource_group_name }}'
/*+ postdeploy, retries=2, retry_delay=2 */
SELECT COUNT(*) as count FROM azure.resource.resource_groups
WHERE subscription_id = '{{ subscription_id }}'
AND resource_group_name = '{{ resource_group_name }}'
AND location = '{{ location }}'
AND JSON_EXTRACT(properties, '$.provisioningState') = 'Succeeded'
Now we can build, test, and teardown our example stack using these commands (starting with a dry-run, which will render the target queries without executing them):
# stackql-deploy build|test|teardown {stack_name} {stack_env} [{options}]
stackql-deploy build example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000 --dry-run
stackql-deploy build example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000
stackql-deploy test example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000
stackql-deploy teardown example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000
Give us your feedback! ⭐ us here!
", "url": "https://stackql.io/blog/product/introducing-model-driven-iac-with-stackql-deploy", "title": "Introducing model-driven IaC with stackql-deploy", "summary": "Inventory all resources across all regions in an AWS account.", "date_modified": "2024-04-21T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "iac", "infrastructure-as-code", "dbt" ] }, { "id": "https://stackql.io/blog/product/builtin-parallel-query-execution-in-stackql", "content_html": "stackql is a dev tool that allows you to query and manage cloud and SaaS resources using SQL, which developers and analysts can use for CSPM, assurance, user access management reporting, IaC, XOps and more.
An exciting new feature to cap off 2023! Parallel query execution in StackQL. With the latest release of StackQL, parameters in WHERE IN clauses are fetched asyncronously.
This query for example, queries lambda functions across 17 AWS regions in under 1.5 sec, technically these are 17 parallel queries to 17 different endpoints.
\nSELECT region, function_name
FROM aws.lambda.functions
WHERE region IN ( 'us-east-1','us-east-2','us-west-1','us-west-2','ap-south-1','ap-northeast-3','ap-northeast-2','ap-southeast-1','ap-southeast-2','ap-northeast-1','ca-central-1','eu-central-1','eu-west-1','eu-west-2','eu-west-3','eu-north-1','sa-east-1'
)
You could do something similar for other hyperscalars, for example querying resources across projects in GCP asynchronously, or querying across resource groups in Azure asynchronously.
\nThis capability was previously available using the pystackql package, as discussed in the Query Resources Across AWS Regions Asynchronously blog post, but is now available natively in the StackQL query optimizer.
You just need to add --execution.concurrency.limit=-1 to your stackql exec or stackql shell commands or when starting a StackQL Server using stackql srv. More query optimizations coming! Happy New Year! 🎉 🎉 🎉
Let us know your thoughts! Visit us and give us a ⭐ on GitHub
", "url": "https://stackql.io/blog/product/builtin-parallel-query-execution-in-stackql", "title": "Builtin Parallel Query Execution in StackQL", "summary": "stackql is a dev tool that allows you to query and manage cloud and SaaS resources using SQL, which developers and analysts can use for CSPM, assurance, user access management reporting, IaC, XOps and more.", "date_modified": "2023-12-30T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "inventory", "cpsm", "analytics" ] }, { "id": "https://stackql.io/blog/product/introducing-materialized-views-with-stackql", "content_html": "Materialized Views are now available in StackQL. Materialized Views can be used to improve performance for dependent or repetetive queries within StackQL provisioning or analytics routines.
\nUnlike standard views that provide a virtual representation of data, a Materialized View physically stores the result set of a query. This implies that the data is pre-computed and stored, which can lead to performance gains as the data doesn't need to be fetched from the underlying resource(s) every time it is queried.
\nPerformance Boost: With data already stored and readily available, Materialized Views can substantially reduce StackQL query execution time, especially for complex and frequently-run queries.
\nData Consistency: Since Materialized Views provide a snapshot of the data at a specific point in time, it ensures consistent data is returned every time it is accessed until it is refreshed.
\nFlexibility: You have the flexibility to refresh the Materialized View as needed usign the REFRESH MATERIALIZED VIEW lifecycle operation in StackQL. This is particularly useful when working with rapidly changing data.
Here's a step-by-step guide on how you to use this new feature in StackQL:
\nCREATE MATERIALIZED VIEW vw_ec2_instance_types AS
SELECT
memoryInfo,
hypervisor,
autoRecoverySupported,
instanceType,
SPLIT_PART(processorInfo, '\\n', 3) as processorArch,
currentGeneration,
freeTierEligible,
hibernationSupported,
SPLIT_PART(vCpuInfo, '\\n', 2) as vCPUs,
bareMetal,
burstablePerformanceSupported,
dedicatedHostsSupported
FROM aws.ec2.instance_types
WHERE region = 'us-east-1';
REFRESH MATERIALIZED VIEW vw_ec2_instance_types;
SELECT
i.instanceId,
i.instanceType,
it.vCPUs,
it.memoryInfo
FROM aws.ec2.instances i
INNER JOIN vw_ec2_instance_types it
ON i.instanceType = it.instanceType
WHERE i.region = 'us-east-1';
DROP MATERIALIZED VIEW vw_ec2_instance_types;
More information on Materialized Views in StackQL can be found here.
", "url": "https://stackql.io/blog/product/introducing-materialized-views-with-stackql", "title": "Introducing Materialized Views with StackQL", "summary": "Materialized Views are now available in StackQL. Materialized Views can be used to improve performance for dependent or repetetive queries within StackQL provisioning or analytics routines.", "date_modified": "2023-09-29T00:00:00.000Z", "author": { "name": "Kieran Rimmer", "url": "https://www.linkedin.com/in/kieranrimmer/" }, "tags": [ "stackql", "analytics" ] }, { "id": "https://stackql.io/blog/product/introducing-table-valued-functions-in-stackql", "content_html": "Many provider query responses include columns which are arrays, the iam policy related resources in google are a classic example of this. for example, this query:
\nselect *
from google.cloudresourcemanager.projects_iam_policies
where projectsId = 'stackql';
produces..
\n|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
| condition | members | role |
|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
| null | [\"serviceAccount:1234567890-compute@developer.gserviceaccount.com\",\"serviceAccount:1234567890@cloudservices.gserviceaccount.com\",\"serviceAccount:stackql@appspot.gserviceaccount.com\"] | roles/editor |
|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
| null | [\"serviceAccount:1234567890-compute@developer.gserviceaccount.com\",\"serviceAccount:1234567890@cloudservices.gserviceaccount.com\",\"serviceAccount:stackql@appspot.gserviceaccount.com\"] | roles/editor |
|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
| null | [\"serviceAccount:1234567890-compute@developer.gserviceaccount.com\",\"serviceAccount:1234567890@cloudservices.gserviceaccount.com\",\"serviceAccount:stackql@appspot.gserviceaccount.com\"] | roles/editor |
|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
What you want to do is unnest each member in members for each role binding (and condition if applicable)
\nEnter the table valued function json_each.
The json_each function accepts a field (optionally with a json path expression) and returns a table object with fields that can be projected in your result set, for example (querying the same underlying resource as above), this...
select
iam.role,
SPLIT_PART(json_each.value, ':', 1) as member_type,
SPLIT_PART(json_each.value, ':', 2) as member
from google.cloudresourcemanager.projects_iam_policies iam, json_each(members)
where projectsId = 'stackql';
now provides something much more useful from an analytic perspective:
\n|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| role | member_type | member |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/appengine.serviceAgent | serviceAccount | service-1234567890@gcp-gae-service.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/cloudbuild.builds.builder | serviceAccount | 1234567890@cloudbuild.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/cloudbuild.serviceAgent | serviceAccount | service-1234567890@gcp-sa-cloudbuild.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/compute.serviceAgent | serviceAccount | service-1234567890@compute-system.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/container.serviceAgent | serviceAccount | service-1234567890@container-engine-robot.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/containerregistry.ServiceAgent | serviceAccount | service-1234567890@containerregistry.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/dataflow.serviceAgent | serviceAccount | service-1234567890@dataflow-service-producer-prod.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/dataproc.serviceAgent | serviceAccount | service-1234567890@dataproc-accounts.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/editor | serviceAccount | 1234567890-compute@developer.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/editor | serviceAccount | 1234567890@cloudservices.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/editor | serviceAccount | stackql@appspot.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/firebaserules.system | serviceAccount | service-1234567890@firebase-rules.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/firestore.serviceAgent | serviceAccount | service-1234567890@gcp-sa-firestore.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/owner | serviceAccount | stackql-provisioner@stackql.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/owner | serviceAccount | t1-804@stackql.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/owner | user | javen@stackql.io |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/owner | user | krimmer@stackql.io |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/pubsub.serviceAgent | serviceAccount | service-1234567890@gcp-sa-pubsub.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
| roles/viewer | serviceAccount | testing-admin@stackql.iam.gserviceaccount.com |
|--------------------------------------|----------------|-----------------------------------------------------------------------------|
__json_each__is available from version 0.5.418 or stackql onwards, this function can also be used in StackQL GitHub Actions such as stackql-exec or stackql-assert and in Python and Pandas using pystackql.
stackql is a dev tool that allows you to query, manage, and perform analytics against cloud and SaaS resources in real time using SQL, which developers and analysts can use for CSPM, assurance, user access management reporting, IaC, XOps and more.
We are pleased to announce the addition of support for OUTER JOIN operations in StackQL queries. This is a significant addition to the language, and we are excited to see what our users will do with it!
An OUTER JOIN is a type of JOIN operation that returns all records from one table (or StackQL resource) and only those records from a second table or resource where the joined fields are equal (i.e. the JOIN condition is met). If there is no match, the missing side of the JOIN is filled with NULL values.
OUTER JOIN operations are important because they allow you to combine data from two or more resources (within a StackQL provider or across StackQL providers), even when there is no match between the two resources. This is a common scenario when performing analytics and reporting on user access management (for example between an IdP (like Okta) and a resource provider like AWS or Google).
OUTER JOIN operations in StackQLIf you wanted to find all users in your AWS account that have not logged in to their account in the last 20 days, and compare that to the same information for users in your Google Workspace account, you could use an OUTER JOIN operation to do this. The following query:
select
aws_users.UserName as aws_user_name
,aws_users.PasswordLastUsed as aws_last_Login_time
,CASE
WHEN aws_users.PasswordLastUsed = '' then 'false'
WHEN ( strftime('%Y-%m-%d %H:%M:%SZ', aws_users.PasswordLastUsed) > ( datetime('now', '-20 days' ) ) ) then 'true'
else 'false' end as aws_is_active
,json_extract(google_users.name, '$.fullName') as google_user_name
,google_users.lastLoginTime as google_last_Login_time
,CASE
WHEN google_users.lastLoginTime is null then 'false'
WHEN google_users.lastLoginTime = '' then 'false'
WHEN ( strftime('%Y-%m-%d %H:%M:%SZ', google_users.lastLoginTime) > ( datetime('now', '-20 days' ) ) ) then 'true'
else 'false' end as google_is_active
from
aws.iam.users aws_users
LEFT OUTER JOIN
googleadmin.directory.users google_users
ON lower(substr(aws_users.UserName, 1, 5)) = lower(substr(json_extract(google_users.name, '$.fullName'), 1, 5))
WHERE aws_users.region = 'us-east-1' AND google_users.domain = 'stackql.io'
;
would produce a result like this:
\n|------------------------|----------------------|---------------|------------------|--------------------------|------------------|
| aws_user_name | aws_last_Login_time | aws_is_active | google_user_name | google_last_Login_time | google_is_active |
|------------------------|----------------------|---------------|------------------|--------------------------|------------------|
| demo-stackql-cicd-user | null | false | null | null | false |
|------------------------|----------------------|---------------|------------------|--------------------------|------------------|
| github_actions | null | false | null | null | false |
|------------------------|----------------------|---------------|------------------|--------------------------|------------------|
| jeffrey.aven | 2023-06-30T04:29:14Z | true | null | null | false |
|------------------------|----------------------|---------------|------------------|--------------------------|------------------|
| kieran.rimmer | 2023-06-03T08:40:49Z | false | Kieran Rimmer | 2023-06-23T06:01:46.000Z | true |
|------------------------|----------------------|---------------|------------------|--------------------------|------------------|
| ... | ... | ... | ... | ... | ... |
|------------------------|----------------------|---------------|------------------|--------------------------|------------------|
Currently only LEFT OUTER JOIN sppport is available, but we will be adding support for RIGHT OUTER JOIN and FULL OUTER JOIN in the near future. Stay tuned!
stackql is a dev tool that allows you to query and manage cloud and SaaS resources using SQL, which developers and analysts can use for CSPM, assurance, user access management reporting, IaC, XOps and more.
pystackql is now available in PyPI, and documentation for the package is available at Read the Docs. pystackql can be used with pandas, matplotlib, plotly, jupyter and more to run queries and visualize results.
The latest version of stackql for any platform can be installed using:
\npip install pystackql
here is a complete example...
\n![\"pystackql](\"https://stackql.io/assets/files/pystackql-081e3bd0624a56ab046a8d2db418dd1f.gif\")
from pystackql import StackQL
import pandas as pd
provider_auth = {
\"aws\": {
\"credentialsenvvar\": \"AWS_SECRET_ACCESS_KEY\",
\"keyIDenvvar\": \"AWS_ACCESS_KEY_ID\",
\"type\": \"aws_signing_v4\"
}
}
regions = [\"ap-southeast-2\", \"us-east-1\"]
stackql = StackQL(auth=provider_auth)
query = \"\"\"
SELECT '%s' as region, instanceType, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = '%s'
GROUP BY instanceType
UNION
SELECT '%s' as region, instanceType, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = '%s'
GROUP BY instanceType
\"\"\" % (regions[0], regions[0], regions[1], regions[1])
res = stackql.execute(query)
df = pd.read_json(res)
print(df)
which returns a pandas DataFrame like the following:
instanceType num_instances region
0 t2.medium 2 ap-southeast-2
1 t2.micro 7 ap-southeast-2
2 t2.small 4 ap-southeast-2
3 t2.micro 6 us-east-1
\n\nSource code for
\npystackqlcan be found atstackql/pystackql
Let us know what you think!
", "url": "https://stackql.io/blog/product/stackql-pypi-package-available", "title": "StackQL PyPI Package is Now Available", "summary": "This how-to article demonstrates how to use stackql with Python using the pystackql package", "date_modified": "2023-02-23T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "python" ] }, { "id": "https://stackql.io/blog/product/setup-stackql-github-action", "content_html": "stackql is a dev tool that allows you to query and manage cloud and SaaS resources using SQL, which developers and analysts can use for CSPM, assurance, user access management reporting, IaC, XOps and more.
The setup-stackql action is now available in the GitHub Marketplace. setup-stackql downloads and installs the latest stackql binary in your Actions runner. Different runner operating systems are fully tested and supported, including ubuntu-latest, windows-latest and darwin-latest.
Embedding stackql in an actions workflow can be done for assurance, compliance, and provisioning. An example workflow is shown here:
name: 'Setup StackQL Example'
on:
pull_request:
defaults:
run:
shell: bash
jobs:
stackql-test-matrix:
name: Stackql local run on ubuntu-latest with wrapper
runs-on: 'ubuntu-latest'
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Stackql
uses: ./
with:
use_wrapper: true
- name: Validate Stackql Version
run: |
stackql --version
- name: Prep Google Creds (bash)
run: | ## use the secret to create json file
sudo echo ${{ secrets.GOOGLE_CREDS }} | base64 -d > sa-key.json
- name: Use Google Provider
run: |
stackql exec \"SELECT name, status
FROM google.compute.instances
WHERE project = 'stackql-demo' AND zone = 'australia-southeast1-a';\" --auth='{ \"google\": { \"type\": \"service_account\", \"credentialsfilepath\": \"sa-key.json\" }}'
the log outputs showing...
\n|-----------------------------------------------------|------------|
| name | status |
|-----------------------------------------------------|------------|
| stackql-demo-001 | TERMINATED |
|-----------------------------------------------------|------------|
| stackql-demo-002 | RUNNING |
|-----------------------------------------------------|------------|
| stackql-demo-003 | RUNNING |
|-----------------------------------------------------|------------|
| stackql-demo-004 | RUNNING |
|-----------------------------------------------------|------------|
\n\nadditional output types available through the
\n--outputargument includejsonandcsv
Source code for the action can be found at stackql/setup-stackql
much more to come, stay tuned!
", "url": "https://stackql.io/blog/product/setup-stackql-github-action", "title": "Using the setup-stackql Github Action", "summary": "This is a how-to article demonstrating the use of setup-stackql github action to run stackql with github actions", "date_modified": "2023-02-05T00:00:00.000Z", "author": { "name": "Yuncheng Yang", "url": "https://www.linkedin.com/in/yuncheng-fabio-yang/" }, "tags": [ "stackql", "devops", "infrastructure", "github actions", "cloud security", "CI/CD" ] }, { "id": "https://stackql.io/blog/product/stackql-middleware-and-playground-available", "content_html": "Proud to announce the release of the first version of our middleware server and playground for StackQL.
\n\n\nStackQL allows you to query and interact with cloud services and APIs using SQL grammar and an ORM which is a direct reflection of a provider API, no database is required or implemented
\n
Our middleware solution allows you to use StackQL as a query language to interact with APIs, much like GraphQL - however, the query DSL is SQL, providing a friendlier, more data-centric experience for developers. As shown in the example below, developers can POST queries to a /stackql endpoint; the queries are parsed and executed by a StackQL runner via the middleware server.
As the StackQL middleware API server and runner are stateless, the solution is horizontally scalable. With cache and authorization to be implemented, the solution provides a flexible, robust, scalable, performant back end for applications.
\nFurthermore, using SQL semantics, developers can JOIN data between API providers and perform projections, filtering, aggregation, windowing operations, and more on simple or complex data types.
\nThe StackQL Playground is a TypeScript app that connects to a StackQL Middleware Server, which provides access to backend APIs using SQL. Features of the playground include...
\nThe StackQL Playground allows you to explore and query providers, services, resources, fields, and methods.
\nUsing the Playground you can submit queries to the /stackql endpoint of the StackQL Middleware Server.
Query results can be sorted or filtered in the grid result set in the StackQL Playground app, JSON results can also be viewed or copied from the JSON results tab.
\nUsers can save results to local CSV or JSON files as well.
Furthermore, after modeling a query you can export the TypeScript types using the Get Types button.
\nHuge thanks to Yuncheng Yang for the work he put in on this!
\nYou can find the complete code here to launch an environment using docker-compose, which includes the StackQL Middleware Server, a StackQL runner (runs the queries on the back end), and the StackQL Playground app.
Let us know what you think.
", "url": "https://stackql.io/blog/product/stackql-middleware-and-playground-available", "title": "StackQL Middleware and Playground Available", "summary": "The StackQL Middleware Server along with the StackQL Playground App allow users to explore and query resources from API providers using SQL.", "date_modified": "2022-11-13T00:00:00.000Z", "author": { "name": "Jeffrey Aven", "url": "https://www.linkedin.com/in/jeffreyaven/" }, "tags": [ "stackql", "stackql middleware", "stackql playground", "middleware", "stackql provider registry", "multicloud", "asset management", "cloud security" ] } ] }