Skip to main content

StackQL Linode provider Released

ยท 2 min read
Technologist and Cloud Consultant
info

stackql is a dev tool that allows you to query and manage cloud and SaaS resources using SQL, which developers and analysts can use for CSPM, assurance, user access management reporting, IaC, XOps and more.

The StackQL Linode provider is now available. Using the StackQL Linode provider you can create, query, and manage Linodes (instances), Volumes, NodeBalancers, Firewalls, StackScripts, Databases, Kubernetes Clusters, Object Storage Buckets, and much more.

You can use the StackQL Linode provider with other StackQL providers (such as aws, google, azure, digitalocean, and more) to perform multi-cloud CSPM, inventory queries, or multi-provider stack deployments. Documentation for the Linode provider is available at StackQL Linode provider docs.

Here is an example of creating a Linode (a VM instance), passing variables from a jsonnet config file as well as CI secrets (GitHub Actions Secrets, GitLab CI Secrets, etc.):

INSERT INTO linode.instances.linodes(
data__authorized_keys,
data__authorized_users,
data__root_pass,
data__image,
data__label,
data__region,
data__type
)
SELECT
'[ "{{ .authorized_key }}" ]',
'[ "{{ .authorized_user }}" ]',
'{{ .root_pass }}',
'{{ .image }}',
'{{ .label }}',
'{{ .region }}',
'{{ .type }}'
;

Querying objects in Linode can be done using SELECT statements, such as:

select id,
label,
region,
JSON_EXTRACT(specs, '$.vcpus') as vcpus,
JSON_EXTRACT(specs, '$.memory') as memory,
JSON_EXTRACT(specs, '$.disk') as disk,
status
from linode.instances.linodes;

Which would return:

|----------|-----------|--------------|-------|--------|-------|---------|
| id | label | region | vcpus | memory | disk | status |
|----------|-----------|--------------|-------|--------|-------|---------|
| 46063573 | my-linode | ap-southeast | 1 | 1024 | 25600 | running |
|----------|-----------|--------------|-------|--------|-------|---------|

Summary or aggregate queries such as GROUP BY -> COUNT or SUM are fully supported with StackQL, as are JOIN and UNION operations (including cross-provider JOIN operations).

StackQL supported outputs include table, csv (using a comma or user-specified delimiter), and json.

StackQL can be accessed through the interactive shell stackql shell as well as noninteractive access using stackql exec and server-based access using stackql srv - where you can use any Postgres wire protocol client to run StackQL queries. GitHub actions, Jupyter notebooks, and Superset dashboards are other options for using StackQL.

Digital Ocean provider for StackQL Available

ยท 3 min read
Technologist and Cloud Consultant
info

stackql is a dev tool that allows you to query and manage cloud and SaaS resources using SQL, which developers and analysts can use for CSPM, assurance, user access management reporting, IaC, XOps and more.

The Digital Ocean provider is now available for StackQL. You can use StackQL to provision, manage or report on Droplets, Apps, Functions, Databases, Volumes, Spaces, and more.

To use the Digital Ocean provider, generate a Personal Access Token from the Digital Ocean Control Panel under the API section. Export the value of the token created to a variable named DIGITALOCEAN_TOKEN (on your local system or as a CI secret). You can then run queries against the Digital Ocean provider using StackQL.

The following example demonstrates the creation of a Droplet in Digital Ocean.

INSERT INTO digitalocean.droplets.droplets (
data__name,
data__region,
data__size,
data__image,
data__backups,
data__ipv6,
data__monitoring,
data__tags
)
SELECT
'droplet-1.example.com',
'nyc3',
's-1vcpu-1gb',
'ubuntu-20-04-x64',
true,
true,
true,
'["env:prod", "web"]';

You can use jsonnet as a configuration, templating language with StackQL to provide variables or parameters to IaC operations in StackQL; this can be done using the --data flag in the stackql exec command as follows:

./stackql exec --infile create_droplets.iql --iqldata vars.jsonnet

The code for create_droplets.iql and vars.jsonnet is shown here:

{{range $index, $element := .droplets}}
INSERT INTO digitalocean.droplets.droplets (
data__name,
data__region,
data__size,
data__image,
data__backups,
data__ipv6,
data__monitoring,
data__tags
)
SELECT
'droplet-{{$index}}.stackql.io',
'nyc3',
'{{.size}}',
'ubuntu-20-04-x64',
true,
true,
true,
'["env:prod", "web"]';
{{end}}

StackQL is a unified SQL-based framework that can be used for analytics and reporting as well as provisioning, de-provisioning, and lifecycle opertaions. As a native multi-cloud solution, StackQL can analyze and report across assets across multiple different providers; an example is shown here:

SELECT
name,
JSON_EXTRACT(region, '$.name') as region,
JSON_EXTRACT(size, '$.slug') as size,
'digitalocean' as provider
FROM digitalocean.droplets.droplets
UNION
SELECT
instanceId as name,
'us-east-1' as region,
instanceType as size,
'aws' as provider
FROM aws.ec2.instances
WHERE region = 'us-east-1';

Digital Ocean and multi-cloud queries can be visualized using BI tools or notebooks; examples of using StackQL with Jupyter can be found here.

More information about the Digital Ocean provider for StackQL can be found here.

Sumologic Provider for StackQL Now Available

ยท 2 min read
Technologist and Cloud Consultant

The StackQL Sumologic provider is now available in the public StackQL Provider Registry. Docs are available at sumologic provider docs.

StackQL is an intelligent API client which uses SQL as a front-end language. StackQL can be used for querying cloud and SaaS providers, as well as provisioning and lifecycle operations.

The StackQL Sumo provider can query, create, update and delete Sumologic collectors and sources, view and manage ingest budgets, health events, dashboards, user and account access and activity, and more.

Some example queries include:

SELECT id, name FROM sumologic.collectors.collectors WHERE region = 'au';

or using built-in functions to simplify and format query outputs, such as:

SELECT alive, datetime(lastSeenAlive/1000, 'unixepoch') AS lastSeenAliveUtc,
datetime(lastSeenAlive/1000, 'unixepoch', 'localtime') AS lastSeenAliveLocal
FROM sumologic.collectors.collectors
WHERE region = 'au' AND id = 116208196;

another example...

SELECT id, email,
firstName || ' ' || lastName AS fullName,
isMfaEnabled,
lastLoginTimestamp,
round(julianday('now') - julianday(lastLoginTimestamp), 0) as daysSinceLastLogin
FROM sumologic.users.users WHERE region = 'au';

An example using StackQL with the Sumologic provider to query users and roles and join the results to get a list of users and their roles:

SELECT u.email as email, r.name AS role
FROM sumologic.users.users u
JOIN sumologic.roles.roles r
ON JSON_EXTRACT(u.roleIds, '$[0]') = r.id
WHERE u.region = 'au' AND r.region = 'au';

An example using StackQL and Jupyter is shown here (see stackql/stackql-jupyter-demo):

Use StackQL and Jupyter to query SumoLogic

StackQL can also be used to provision objects in Sumologic, the following query can be used to create a collector for instance:

INSERT INTO sumologic.collectors.collectors(region, data__collector)
SELECT 'au',
'{ "collectorType":"Hosted", "name":"My Hosted Collector", "description":"An example Hosted Collector", "category":"HTTP Collection" }';

Let us know what you think!

New Functions and Azure Provider Available

ยท One min read
Technologist and Cloud Consultant

A new version of the Azure provider for StackQL plus additional built-in functions are available now.

Version 0.3.0 of the Azure provider for StackQL is available now. This update includes support for extended resource properties, along with support for Hybrid Azure Kubernetes Services. The Azure provider allows you to query across your Azure estate for cloud security posture, asset inventory, analysis and reporting, finops, sysops, and more - all using a natural query language (SQL) and a natural object mapping.

In addition, we have announced the release of several new built-in functions, including SPLIT_PART() - to split a string by a delimiter and extract a single element, additional unicode functions, and expanded regular expression support, including REGEXP_REPLACE() and more.

An example StackQL query using the split_part() function with the azure v0.3.0 provider is shown here:

SELECT name,
split_part(id, '/', 3) as subscription,
split_part(id, '/', 5) as resource_group,
json_extract(properties, '$.hardwareProfile.vmSize') as vm_size
FROM azure.compute.virtual_machines WHERE resource_group_name = 'stackql-ops-cicd-dev-01' AND subscription_id = '242c6a2d-16f9-4912-90f6-59b1cf85509d';

You can find more information on the latest Azure provider here.

StackQL Provider for AWS Released

ยท 4 min read
Technologist and Cloud Consultant

Pleased to announce the initial release of the AWS provider for StackQL.

StackQL allows you to query, provision, and manage cloud and SaaS resources using a simple, SQL-based framework.

The initial release of the AWS provider covers EC2, S3, and the Cloud Control API - with support for other services to be released soon. The documentation for the StackQL AWS provider is available here.

Follow the steps below to get started querying AWS in the StackQL interactive command shell:

Authenticate and Connectโ€‹

Connect to an authenticated shell using the syntax shown below:

# AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID should be set as environment variables
AUTH="{ \"aws\": { \"type\": \"aws_signing_v4\", \"credentialsenvvar\": \"AWS_SECRET_ACCESS_KEY\", \"keyID\": \"${AWS_ACCESS_KEY_ID}\" }}"
stackql shell --auth="${AUTH}"

Download the AWS providerโ€‹

Download the AWS provider from the StackQL Provider Registry:

REGISTRY PULL aws v0.1.3;

Explore the AWS providerโ€‹

Explore the AWS provider using StackQL metacommands (such as SHOW and DESCRIBE), for example...

Show available servicesโ€‹

Show the services available in the StackQL AWS provider:

SHOW SERVICES IN aws;

Show available resourcesโ€‹

Show the resources available in the AWS EC2 service (filtered by a fuzzy match on instances):

SHOW RESOURCES IN aws.ec2 LIKE '%instances%';

Show 'selectable' fieldsโ€‹

Show the 'selectable' fields available in a resource:

DESCRIBE EXTENDED aws.ec2.instances;

Show operations availableโ€‹

Show the available operations on a resource:

SHOW EXTENDED METHODS IN aws.ec2.instances;

Run some queriesโ€‹

Now that you've identified the available resources and fields let's run some queries!

Instances by region (across multiple regions)โ€‹

SELECT 'N. Virginia' as region, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'us-east-1'
UNION
SELECT 'N. California' as region, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'us-west-1'
UNION
SELECT 'Sydney' as region, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'ap-southeast-2';

Instances grouped by instanceTypeโ€‹

SELECT instanceType, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'ap-southeast-2'
GROUP BY instanceType;

Instances grouped by instanceStateโ€‹

SELECT instanceState, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'ap-southeast-2'
GROUP BY instanceState;

Enjoy!