Skip to main content

49 posts tagged with "stackql"

View All Tags

New Okta Provider Released

· 6 min read
Technologist and Cloud Consultant

The latest okta provider for StackQL is available now (v25.09.00341), featuring a comprehensive reorganization of services and resources that better aligns with Okta's API structure. This update improves discoverability, logical grouping, and overall usability when working with Okta resources through StackQL.

What's Changed​

The updated Okta provider features a more granular and logical organization of services that mirrors Okta's API architecture more closely. Here is a summary of the services included in the latest okta provider:

ServiceDescription
agentpoolsManages agent pools for on-premises integration and deployment
api_tokensHandles API token creation, management, and access control
appsManages Okta application integration, configuration, and assignments
attack_protectionConfigures security controls to prevent various attack vectors
authenticatorsManages authentication methods, factors, and settings
authorizationserversControls OAuth 2.0 and OIDC authorization servers and policies
behaviorsConfigures end-user behavioral patterns for risk-based authentication
brandsManages customization of Okta UI appearance and branding
captchasConfigures CAPTCHA settings to prevent automated attacks
device_accessControls access policies based on device attributes
device_assurancesManages device trust requirements and verification
device_integrationsConfigures integrations with MDM and endpoint security tools
device_posture_checksVerifies security posture of devices accessing resources
devicesManages device enrollment, lifecycle, and inventory
directoriesHandles directory service connections and synchronization
domainsManages custom domain configuration for Okta tenant
email_domainsControls email domains for user provisioning and validation
email_serversConfigures email server settings for notifications
eventhooksManages webhook subscriptions for Okta event notifications
featuresControls feature flags and preview feature management
first_party_app_settingsConfigures settings for Okta-developed applications
groupsManages group creation, membership, and rule configuration
hook_keysHandles encryption keys for securing hook communications
iamProvides core identity and access management functionality
identity_sourcesManages sources of identity data for user provisioning
idpsConfigures external identity providers for federation
inlinehooksManages customization points within Okta workflows
integrationsConfigures third-party service integrations
logsProvides access to system logs and audit events
logstreamsManages streaming of log data to external systems
mappingsConfigures attribute mappings between systems
metaProvides metadata about the Okta API and environment
oauth2Manages OAuth 2.0 clients, scopes, and tokens
okta_personal_settingsControls user-specific preferences and settings
orgManages organization-level settings and configurations
orgsHandles multi-org deployments and org relationships
policiesConfigures authentication, password, and access policies
principal_rate_limitsManages API rate limits for specific principals
privileged_accessControls privileged access management settings
push_providersConfigures push notification delivery services
rate_limit_settingsManages global API rate limit configurations
realm_assignmentsMaps users and groups to authentication realms
realmsConfigures authentication realms for different user populations
riskManages risk-based authentication settings and policies
rolesControls administrative role assignments and permissions
securityProvides security settings and configurations
security_events_providersManages integrations with security event sources
sessionsControls user session policies and management
ssfConfigures server-side functions for customization
templatesManages templates for emails, notifications, and forms
threatsControls threat intelligence settings and configurations
trustedoriginsManages CORS and redirect configurations
usersHandles user lifecycle, profiles, and credentials
webauthn_registrationManages WebAuthn/FIDO2 credential registration
zonesConfigures network zones for policy application
note

The improved organization including service and resource naming and mapping will introduce changes to queries against the previous provider version (v23.03.00121), you can pin the previous version in stackql-deploy or via registry pull in the interim while you make necessary query modifications.

Enhanced Documentation​

The new Okta Provider Docs provide comprehensive documentation on how to use the new okta provider including ready-to-use SQL examples for each resource and method. A standout feature is the copy-paste functionality for all SQL queries, making it incredibly easy to:

  1. Compose Infrastructure-as-Code workflows: Each method documentation includes working SQL examples that can be directly copied into your deployment scripts or CI/CD pipelines. Simply click the copy button next to any example to get production-ready SQL code.

  2. Build analytics dashboards: Create sophisticated cross-service queries by combining examples from different resources. The documentation's consistent query formatting makes it simple to join related data across multiple Okta services.

  3. Develop governance reports: Copy baseline queries and customize them for your specific compliance needs. The pre-formatted SQL provides the perfect starting point for custom reporting.

Getting Started​

To start using the updated okta provider, simply pull the latest version from stackql shell or stackql registry command:

registry pull okta;

Then you can begin querying your Okta resources with SQL:

-- Get all users and their group assignments
SELECT
u.id,
JSON_EXTRACT(u.profile, '$.email') as email,
u.status,
JSON_EXTRACT(g.profile, '$.name') as group_name
FROM
okta.user.users u
JOIN
okta.group.users gu ON u.id = gu.user_id
JOIN
okta.group.groups g ON gu.group_id = g.id
WHERE subdomain = 'my-company';

Use Cases for the Okta Provider​

  1. Infrastructure as Code: Manage your Okta resources alongside other cloud providers in a unified IaC approach, see stackql-deploy.

  2. Cost Optimization: Identify unused resources and opportunities for cost savings.

  3. Security and Compliance: Audit account roles, permissions, and access patterns to ensure compliance with security policies.

  4. Performance Monitoring: Track query performance, warehouse utilization, and identify optimization opportunities.

  5. Cross-Provider Orchestration: Build workflows that span Okta and other resource providers, enabling sophisticated data and infrastructure pipelines.

  6. Automated Reporting: Create automated reports on Okta usage, performance, and costs.

⭐ us on GitHub and join our community!

Updated Google Providers for StackQL Available

· 2 min read
Technologist and Cloud Consultant

The latest versions of the Google-related providers for StackQL: google, googleadmin, googleworkspace, and firebase are available now. These updates include the latest services, resources and methods available from Google.

What's New​

The latest release introduces several new services to the google provider, expanding your ability to manage and query Google Cloud resources:

  • API Hub: Centrally manage and discover APIs across your organization
  • Area Insights: Access location-based insights and analytics
  • Cloud Location Finder: Identify optimal Google Cloud regions for your workloads
  • Gemini Cloud Assist: Leverage Google's AI assistant for cloud operations
  • Managed Kafka: Work with Google's fully-managed Apache Kafka service
  • Observability: Enhanced monitoring and observability services
  • Parallel Store: Interact with Google's high-performance storage solution
  • Parameter Manager: Manage configuration parameters across services
  • SaaS Service Management: Tools for managing SaaS offerings on Google Cloud
  • Secure Source Manager: Google's secure, fully-managed source control service
  • Security Posture: Assess and improve your cloud security posture
  • Storage Batch Operations: Perform batch operations on Cloud Storage resources

Enhanced Documentation​

We've also released enhanced user documentation to help you get the most out of these providers. Check out our comprehensive docs:

Getting Started​

To start using these updated providers, simply pull the latest version from stackql shell or stackql registry command:

registry pull google;
registry pull googleadmin;
registry pull googleworkspace;
registry pull firebase;

Then you can begin querying your Google resources with SQL:

SELECT name, region, status
FROM google.compute.instances
WHERE project = 'my-project';

Use Cases for the Google Provider​

The Google provider for StackQL opens up numerous possibilities:

  1. Infrastructure as Code: Manage your Google resources alongside other cloud providers in a unified IaC approach, see stackql-deploy.

  2. Cost Optimization: Identify unused resources and opportunities for cost savings.

  3. Security and Compliance: Audit account roles, permissions, and access patterns to ensure compliance with security policies.

  4. Performance Monitoring: Track query performance, warehouse utilization, and identify optimization opportunities.

  5. Cross-Provider Orchestration: Build workflows that span Google and other cloud providers, enabling sophisticated data and infrastructure pipelines.

  6. Automated Reporting: Create automated reports on Google usage, performance, and costs.

⭐ us on GitHub and join our community!

Snowflake Provider for StackQL Released

· 3 min read
Technologist and Cloud Consultant

We're excited to announce the release of the Snowflake provider for StackQL! This new provider enables you to query and interact with your Snowflake resources using familiar SQL syntax, bridging the gap between data analytics and infrastructure management.

The Snowflake provider for StackQL gives you the ability to:

  • Query Snowflake metadata and statistics using SQL
  • Monitor warehouse, database, and query performance
  • Analyze resource usage and optimize costs
  • Integrate Snowflake management with your existing cloud infrastructure
  • Build cross-provider workflows and automation

Full documentation for the Snowflake provider is available here.

Getting Started​

Getting started is as easy as...

REGISTRY PULL snowflake;

Example Queries​

Let's explore some powerful examples of what you can do with the Snowflake provider for StackQL.

Analyzing Warehouses​

SELECT
size,
count(*) as num_warehouses
FROM
snowflake.warehouse.warehouses
WHERE
endpoint = 'OKXVNMC-VH34026'
GROUP BY
size;

Other fields for the warehouses resource include : name, warehouse_type, state, scaling_policy, auto_suspend, auto_resume, resource_monitor, enable_query_acceleration, query_acceleration_max_scale_factor, max_concurrency_level, owner, warehouse_credit_limit, target_statement_size and more.

Table Analysis​

SELECT
name,
bytes,
data_retention_time_in_days,
table_type
FROM
snowflake.table.tables
WHERE
database_name = 'SNOWFLAKE_SAMPLE_DATA'
AND schema_name = 'TPCH_SF10'
AND endpoint = 'OKXVNMC-VH34026'
ORDER BY
bytes DESC;

Other Services and Resources​

Other notable resources which can be provisioned, managed or queried using the snowflake provider for stackql include:

alerts, api_integrations, catalog_integrations, compute_pools, databases, database_roles, dynamic_tables, event_tables, external_volumes, functions, grants, iceberg_tables, image_repositories, network_policies, notebooks, notification_integrations, pipes, procedures, roles, schemas, stages, streams, tasks, users, user_defined_functions, views, and more!

Use Cases for the Snowflake Provider​

The Snowflake provider for StackQL opens up numerous possibilities:

  1. Infrastructure as Code: Manage your Snowflake resources alongside other cloud providers in a unified IaC approach, see stackql-deploy.

  2. Cost Optimization: Identify unused resources, inefficient warehouses, and opportunities for cost savings.

  3. Security and Compliance: Audit account roles, permissions, and access patterns to ensure compliance with security policies.

  4. Performance Monitoring: Track query performance, warehouse utilization, and identify optimization opportunities.

  5. Cross-Provider Orchestration: Build workflows that span Snowflake and other cloud providers, enabling sophisticated data and infrastructure pipelines.

  6. Automated Reporting: Create automated reports on Snowflake usage, performance, and costs.

⭐ us on GitHub and join our community!

New AWS Provider Available (Jan 2025)

· 2 min read
Technologist and Cloud Consultant
info

To get started with the aws provider for stackql, pull the provider from the registry as follows:  

registry pull aws;

for more detailed provider documentation, see here.

Happy New Year 🎉. The latest AWS provider for StackQL is now available.  The StackQL AWS Provider by the numbers:

  • 230 services
  • 3174 resources
  • 3917 methods

with additional new support for the following services:

  • amazonmq - Managed message broker service for Apache ActiveMQ and RabbitMQ that simplifies setup and operation of open-source message brokers on AWS.
  • applicationsignals - CloudWatch Application Signals automatically provides a correlated view of application performance that includes real user monitoring data and canaries.
  • apptest - AWS mainframe modernization ppplication Testing
  • connectcampaignsv2 - Amazon Connect Outbound Campaigns V2
  • invoicing - Deploy and query invoice units allowing you separate AWS account costs and configures your invoice for each business entity
  • launchwizard - Easily size, configure, and deploy third party applications on AWS
  • pcaconnectorscep - AWS Private CA Connector for SCEP
  • pcs - AWS Parallel Computing Service, easily run HPC workloads at virtually any scale
  • rbin - Recycle Bin is a resource recovery feature that enables you to restore accidentally deleted snapshots and EBS-backed AMIs.
  • s3tables - Amazon S3 Tables enabling Tabular Data Storage At Scale
  • ssmquicksetup - AWS Systems Manager Quick Setup

And 150 new resources with some notable additions including:

  • aws.apigateway.domain_name_access_associations
  • aws.appconfig.deployments, aws.appconfig.deployment_strategies
  • aws.batch.job_definitions
  • aws.bedrock.flows, aws.bedrock.prompts
  • aws.chatbot.custom_actions
  • aws.cloudformation.guard_hooks, aws.cloudformation.lambda_hooks
  • aws.cloudfront.anycast_ip_lists
  • aws.cloudtrail.dashboards, aws.cloudwatch.dashboards
  • aws.codepipeline.pipelines
  • aws.cognito.user_pool_identity_providers
  • aws.ec2.security_group_vpc_associations, aws.ec2.vpc_block_public_access_exclusions, aws.ec2.vpc_block_public_access_options
  • aws.glue.crawlers, aws.glue.databases, aws.glue.jobs, aws.glue.triggers
  • aws.guardduty.malware_protection_plans
  • aws.iot.commands
  • aws.memorydb.multi_region_clusters
  • aws.rds.db_shard_groups
  • aws.redshift.integrations
  • aws.sagemaker.clusters, aws.sagemaker.endpoints
  • aws.secretsmanager.resource_policies, aws.secretsmanager.rotation_schedules, aws.secretsmanager.secret_target_attachments
  • aws.workspaces.workspaces_pools
  • aws.wisdom.ai_agents, aws.wisdom.ai_prompts, aws.wisdom.ai_guardrails, aws.wisdom.message_templates
  • and much more!

⭐ us on GitHub and join our community!

Databricks Provider for StackQL Available

· 3 min read
Technologist and Cloud Consultant

We are pleased to announce the release of the Databricks provider for StackQL today.  The Databricks provider is two different providers, databricks_account and databricks_workspace.

info

Check out the registry docs at databricks_account or databricks_workspace.

To get started, pull the providers from the registry as follows:  

registry pull databricks_account;
registry pull databricks_workspace;

databricks_account provider​

The databricks_account provider is used for account-level operations, including provisioning or managing users, groups, unity catalog metastores, workspaces, and account-level cloud resources used by workspaces (such as networking resources).  Services include:

stackql  >>show services in databricks_account;
|----------------------------|---------------|--------------------------------|
|             id             |     name      |             title              |
|----------------------------|---------------|--------------------------------|
| billing:v00.00.00000       | billing       | Account Billing                |
|----------------------------|---------------|--------------------------------|
| iam:v00.00.00000           | iam           | Identity and Access Management |
|----------------------------|---------------|--------------------------------|
| logging:v00.00.00000       | logging       | Log Delivery                   |
|----------------------------|---------------|--------------------------------|
| oauth:v00.00.00000         | oauth         | OAuth Integrations             |
|----------------------------|---------------|--------------------------------|
| provisioning:v00.00.00000  | provisioning  | Account Provisioning           |
|----------------------------|---------------|--------------------------------|
| settings:v00.00.00000      | settings      | Account Settings               |
|----------------------------|---------------|--------------------------------|
| unity_catalog:v00.00.00000 | unity_catalog | Unity Catalog                  |
|----------------------------|---------------|--------------------------------|

Some example databricks_account queries are shown here:

stackql  >>select *  from  databricks_account.iam.account_users where account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce' and active = true;
|--------|--------------|-------------------------------------------------------------|------------|------------------|---------------------------------------------|---------------------------------------------|------------------|
| active | displayName  |                           emails                            | externalId |        id        |                    name                     |                    roles        
|     userName     |
|--------|--------------|-------------------------------------------------------------|------------|------------------|---------------------------------------------|---------------------------------------------|------------------|
| true   | Jeffrey Aven | [{"primary":true,"type":"work","value":"javen@stackql.io"}] | null       | 5728205706991489 | {"familyName":"Aven","givenName":"Jeffrey"} | [{"type":"direct","value":"account_admin"}] | javen@stackql.io |
|--------|--------------|-------------------------------------------------------------|------------|------------------|---------------------------------------------|---------------------------------------------|------------------|

or..

stackql  >>SELECT applicationId,  displayName
stackql  >>FROM databricks_account.iam.account_service_principals, JSON_EACH(roles)
stackql  >>WHERE account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce'
stackql  >>AND JSON_EXTRACT(json_each.value, '$.value') = 'account_admin';
|--------------------------------------|-------------|
|            applicationId             | displayName |
|--------------------------------------|-------------|
| 0b7b23de-3e7d-4432-812c-cf517e079a22 | stackql     |
|--------------------------------------|-------------|

or..

stackql  >>select
stackql  >>workspace_id,
stackql  >>workspace_name,
stackql  >>deployment_name,
stackql  >>workspace_status,
stackql  >>pricing_tier,
stackql  >>aws_region,
stackql  >>credentials_id,
stackql  >>storage_configuration_id
stackql  >>from
stackql  >>databricks_account.provisioning.workspaces where account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce';
|------------------|----------------|-------------------|------------------|--------------|------------|--------------------------------------|--------------------------------------|
|   workspace_id   | workspace_name |  deployment_name  | workspace_status | pricing_tier | aws_region |            credentials_id            |       storage_configuration_id       |
|------------------|----------------|-------------------|------------------|--------------|------------|--------------------------------------|--------------------------------------|
| 1583879855205171 | stackql-test   | dbc-ddbc0f51-c9cf | RUNNING          | PREMIUM      | us-west-2  | dcacd875-c782-46ea-9d3e-8307975d758a | e52e029f-24bb-4a75-99c3-7796c202dd89 |
|------------------|----------------|-------------------|------------------|--------------|------------|--------------------------------------|--------------------------------------|

databricks_workspace provider​

The databricks_workspace provider is used for workspace-level operations, such as provisioning and managing clusters, dashboards, and workflow jobs (including delta live table pipelines).  Services include:  

stackql  >>show services in databricks_workspace;
|------------------------------|-----------------|-----------------|
|              id              |      name       |      title      |
|------------------------------|-----------------|-----------------|
| apps:v24.12.00279            | apps            | Apps            |
|------------------------------|-----------------|-----------------|
| cleanrooms:v24.12.00279      | cleanrooms      | Cleanrooms      |
|------------------------------|-----------------|-----------------|
| compute:v24.12.00279         | compute         | Compute         |
|------------------------------|-----------------|-----------------|
| dbsql:v24.12.00279           | dbsql           | Dbsql           |
|------------------------------|-----------------|-----------------|
| deltalivetables:v24.12.00279 | deltalivetables | Deltalivetables |
|------------------------------|-----------------|-----------------|
| deltasharing:v24.12.00279    | deltasharing    | Deltasharing    |
|------------------------------|-----------------|-----------------|
| filemanagement:v24.12.00279  | filemanagement  | Filemanagement  |
|------------------------------|-----------------|-----------------|
| iam:v24.12.00279             | iam             | Iam             |
|------------------------------|-----------------|-----------------|
| lakeview:v24.12.00279        | lakeview        | Lakeview        |
|------------------------------|-----------------|-----------------|
| machinelearning:v24.12.00279 | machinelearning | Machinelearning |
|------------------------------|-----------------|-----------------|
| marketplace:v24.12.00279     | marketplace     | Marketplace     |
|------------------------------|-----------------|-----------------|
| realtimeserving:v24.12.00279 | realtimeserving | Realtimeserving |
|------------------------------|-----------------|-----------------|
| repos:v24.12.00279           | repos           | Repos           |
|------------------------------|-----------------|-----------------|
| secrets:v24.12.00279         | secrets         | Secrets         |
|------------------------------|-----------------|-----------------|
| unitycatalog:v24.12.00279    | unitycatalog    | Unitycatalog    |
|------------------------------|-----------------|-----------------|
| vectorsearch:v24.12.00279    | vectorsearch    | Vectorsearch    |
|------------------------------|-----------------|-----------------|
| workflows:v24.12.00279       | workflows       | Workflows       |
|------------------------------|-----------------|-----------------|
| workspace:v24.12.00279       | workspace       | Workspace       |
|------------------------------|-----------------|-----------------|

An example query could be:

stackql  >>select
stackql  >>cluster_id,
stackql  >>aws_attributes,
stackql  >>node_type_id,
stackql  >>state
stackql  >>from
stackql  >>databricks_workspace.compute.clusters
stackql  >>where deployment_name = 'dbc-ddbc0f51-c9cf';
|----------------------|---------------------------------------------------------------------------------------------------------|--------------|------------|
|      cluster_id      |                                             aws_attributes                                              | node_type_id |   state    |
|----------------------|---------------------------------------------------------------------------------------------------------|--------------|------------|
| 1218-233957-q9v9oi86 | {"availability":"SPOT_WITH_FALLBACK","first_on_demand":1,"spot_bid_price_percent":100,"zone_id":"auto"} | m5d.large    | TERMINATED |
|----------------------|---------------------------------------------------------------------------------------------------------|--------------|------------|

To use either provider, set the following environment variables (either locally or as secrets in your preferred CI tool):

  • DATABRICKS_ACCOUNT_ID - a uuid representing your Databricks account id, you can get this from the Databricks UI
  • DATABRICKS_CLIENT_ID - obtained after creating a service principal through the Databricks UI
  • DATABRICKS_CLIENT_SECRET - obtained after creating a service principal secret through the Databricks UI, using the "Generate Secret" function

These are the same variables that Terraform, the Databricks SDKs, and CLI use.  

stackql-deploy examples coming soon, stay tuned!  

⭐ us on GitHub and join our community!