For more background on using StackQL with GitHub Actions see StackQL GitHub Actions Tutorial

- name: setup StackQL
  uses: stackql/setup-stackql@v1.1.0
  with:
    use_wrapper: true

- name: get changed files
  env:
    STACKQL_GITHUB_USERNAME: ${{ secrets.STACKQL_GITHUB_USERNAME }}
    STACKQL_GITHUB_PASSWORD: ${{ secrets.STACKQL_GITHUB_PASSWORD }}
  shell: bash
  run: |
    ORG=$(echo "$GITHUB_REPOSITORY" | cut -d '/' -f1)
    REPO=$(echo "$GITHUB_REPOSITORY" | cut -d '/' -f2)
    QUERY="WITH changed_files AS (SELECT json_each.value AS file FROM github.repos.commits, JSON_EACH(files) WHERE owner = '${ORG}' AND repo = '${REPO}' AND ref = '${GITHUB_SHA}') SELECT JSON_EXTRACT(file, '$.filename') AS filename FROM changed_files"
    echo "pulling github provider"
    stackql exec "REGISTRY PULL github"
    echo "running query: ${QUERY}"
    stackql --output json -f changed_files.txt exec "${QUERY}"

changed_files.txt looks like this...

[{"filename":"src/app.ts"},{"filename":"src/mod.ts"},...]

You could then do something with the changed files in a further step like:

- name: Do something with changed files
  run: |
    while IFS="" read -r filename || [ -n "$filename" ]
    do
      echo "processing ${filename}..."
      #
      # do something interesting here...
      #
    done < <(jq -r '.[] | .filename' changed_files.txt)

Each element of the files array returned by github.repos.commits has other interesting fields which could be projected (via JSON_EXTRACT) and used for actioning or reporting. You can inspect the files structure using:

DESCRIBE EXTENDED github.repos.commits;

Fields available within each files array element include:

• status - one of added, removed, modified, renamed, copied, changed or unchanged
• filename - filename which has changed
• previous_filename - previous filename if the filename had changed in the commit
• additions - the number of additions in each file
• changes - the number of changes to each file
• deletions - the number of deletions in each file
• patch - git diff output for each file
• blob_url - the blob url for the file
• raw_url - the raw url for the file
• contents_url - the contents url for the file
• sha - The sha for each individual file

Read More​

Check out a full demo on using StackQL with GitHub Actions​

• stackql-actions-demo

Check the GitHub Repos​

• setup-stackql
• stackql-exec
• stackql-assert

We provide a set of useful actions that allow you use StackQL with GitHub Actions Workflow. These actions are available in the GitHub Marketplace.

setup-stackql​

This action installs the stackql CLI on Actions runners. It is used if you want to perform custom operations using StackQL queries.

Using the setup-stackql action allows you to perform StackQL CLI operations in the subsequent steps of a GitHub Actions job. The setup-stackql action installs the stackql CLI on the runner, making it available to all subsequent steps. This means you can use StackQL queries and commands directly in your GitHub Action Workflow Steps.

stackql-exec​

This action executes a StackQL query within an Actions workflow.

The stackql-exec action is a wrapper around setup-stackql that allows you to pass your StackQL query and authentication information as inputs. This makes it easy to execute StackQL queries directly within your GitHub Action Workflow Step.

stackql-assert​

This action is used to test assertions against the results of a StackQL query. You can use this action to validate the state of a resource after an IaC or lifecycle operation has been performed or to validate the system (e.g., CSPM or compliance queries).

The stackql-assert action is also a wrapper around setup-stackql that allows you to pass your StackQL query and authentication information as inputs. This makes it easy to test assertions against the results of your StackQL queries directly within your GitHub Action Workflow Step.

Use setup-stackql to run StackQL queries in your GitHub Workflow Job​

To run StackQL query with GitHub Actions, you need to install the stackql CLI on the runner. This can be done using the setup-stackql action. Once installed, the stackql CLI is available to all subsequent steps in the job via StackQL.

- name: setup StackQL
  uses: stackql/setup-stackql@v1.1.0
  with:
    use_wrapper: true

Run StackQL query​

After using the setup-stackql action, you can run the stackql CLI as a shell command in subsequent steps of the job. For example, you can run a dry run of a StackQL query as a shell command like this:

- name: Prep Google Creds (bash)
      if: ${{ matrix.os != 'windows-latest' }}
      run: | ## use the secret to create json file
        sudo echo ${{ secrets.GOOGLE_CREDS }} | base64 -d > sa-key.json

    - name: Use Google Provider
      run: | 
        stackql exec -i ./examples/google-example.iql --auth='{ "google": { "type": "service_account",  "credentialsfilepath": "sa-key.json" }}'

You can check more example here.

Use stackql-exec to run StackQL queries in a single step​

stackql-exec action provides a simpler and more streamlined way to executestackql CLI queries in your GitHub Workflow. You can create a step like this:

- name: stop running instances using stackql-exec
  uses: stackql/stackql-exec@v1.0.1
  with:
    auth_obj_path: './stackql/auth.json'
    query_file_path: './stackql/scripts/stop-instances.iql'

The auth_obj_path parameter specifies the path to the authentication object that contains the credentials required to authenticate with the data source. The query_file_path parameter specifies the path to the StackQL query file that you want to execute.

Once you have specified the authentication object and query file, you can run the StackQL query by executing the stackql-exec action in a single step in your GitHub Workflow.

You can check more examples here.

Validating resource state with stackql-assert​

stackql-assert is another useful action that StackQL provides, allowing you to test assertions against the results of a StackQL query.

This action is particularly helpful for validating the state of a resource after an infrastructure as code (IaC) or lifecycle operation has been performed.

You can run an assert step like this example:

- name: check terraform deployment using stackql-assert - should fail
  uses: stackql/stackql-assert@v1.0.2
  with:
    auth_obj_path: './stackql/auth.json'
    test_query_file_path: './stackql/scripts/check-terraform-instances.iql'
    expected_results_str: '[{"name":"terraform-test-1","name":"terraform-test-2"}]'

Authentication​

setup-stackql​

if you use basic token, you can setup authentication with the following example

- name: Use GitHub Provider
      run: |
        stackql exec -i ./examples/github-example.iql --auth='{ "github": { "type": "basic", "credentialsenvvar": "STACKQL_GITHUB_CREDS" } }'
      env: 
        STACKQL_GITHUB_CREDS: ${{  secrets.STACKQL_GITHUB_CREDS }}

Meanwhile if you are using a credential file, you can create the file via this example bash command

    - name: Prep Google Creds (bash)
      run: | ## use the secret to create json file
        sudo echo ${{ secrets.GOOGLE_CREDS }} | base64 -d > sa-key.json

    - name: Use Google Provider
      run: | 
        stackql exec "SELECT name, status
        FROM google.compute.instances 
        WHERE project = 'stackql-demo' AND zone = 'australia-southeast1-a';" --auth='{ "google": { "type": "service_account",  "credentialsfilepath": "sa-key.json" }}'

stackql-exec and stackql-assert​

For stackql-exec and stackql-assert, you can use the same authentication method as setup-stackql, except the authentication object is passed via the auth_obj_path or auth_strparameter

• auth_obj_path parameter to specify the path to the authentication object that contains the credentials required to authenticate with the data source.
• auth_str parameter to specify the authentication object as a string.

Example Authentication Object file (auth.json):​

{
  "github": {
    "type": "basic",
    "credentialsenvvar": "STACKQL_GITHUB_CREDS"
  }
}

Example Usage of auth_obj_path:​

  - name: Prep Google Creds (bash)
  if: ${{ matrix.os != 'windows-latest' }}
  shell: bash
  run: | ## use the base64 encoded secret to create json file
    sudo echo ${{ secrets.GOOGLE_CREDS }} | base64 -d > sa-key.json
    
  - name: stop running instances using stackql-exec
    uses: stackql/stackql-exec@v1.0.1
    with:
      auth_obj_path: './stackql/auth.json'
      query_file_path: './stackql/scripts/stop-instances.iql'

  - name: check if we have 4 instances using stackql-assert
    uses: stackql/stackql-assert@v1.0.2
    with:
      auth_obj_path: './stackql/auth.json'
      test_query_file_path: './stackql/scripts/check-instances.iql'
      expected_rows: 4

Example Usage of auth_str:​

  - name: Prep Google Creds (bash)
    if: ${{ matrix.os != 'windows-latest' }}
    shell: bash
    run: | ## use the base64 encoded secret to create json file
      sudo echo ${{ secrets.GOOGLE_CREDS }} | base64 -d > sa-key.json
      
  - name: exec github example
    uses: ./
    with:
      auth_str: '{ "github": { "type": "basic", "credentialsenvvar": "STACKQL_GITHUB_CREDS" } }'
      query: "REGISTRY PULL github v23.01.00104;
              SHOW PROVIDERS;
              select total_private_repos
              from github.orgs.orgs
              where org = 'stackql';"
    env: 
      STACKQL_GITHUB_CREDS: ${{ secrets.STACKQL_GITHUB_CREDS }}

  - name: Use test query string and expected rows, with auth string
    uses: ./
    with:
      auth_str: '{ "google": { "type": "service_account",  "credentialsfilepath": "sa-key.json" }}'
      test_query_file_path: './.github/workflows/workflow_scripts/google-example.iql'
      expected_rows: 1

Read More​

Check the full demo​

• stackql-actions-demo

Check the GitHub Repos​

• setup-stackql
• stackql-exec
• stackql-assert

This exercise will show you how to run a real-time query across your AWS and Google cloud environments. You may do this for inventory analysis, security analysis, or any other reason you can think of. We will use stackql to query the state of your cloud resources across your AWS and Google environments. You can also use stackql to provision, de-provision or manage resources across different cloud and SaaS providers.

The steps we will take are:

1. Prepare your environment for stackql usage.
2. Use stackql to provision some resources in cloud. optional
3. Use stackql to query resources present in the cloud.
4. Use stackql to tear down resources created in step (2), if any. Important: you must destroy any resources created through this exercise, or you will incur ongoing charges.

Preparation​

For this exercise, credentials with privileges against google and aws are required. It is outside the scope of this document to go into great detail on the various topics and options relevant to this. Instead, the below steps provide both: (i) reference to vendor documentation and (ii) suggestions for workarounds to get yourself going.

for old hands​

All the materials required for this exercise are:

1. A current stackql executable.
2. A Google Service Account Key JSON file, where the corresponding Service Account possesses permissions sufficient to create, interrogate and delete compute block storage.
3. AWS credentials stored in the traditional AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, where the corresponding Service Account possesses permissions sufficient to create, interrogate and delete ec2 block storage.

step by step​

First, please do the following:

1. Download and install stackql from our website.
2. For google:
   • (i) Create and download a Google Service Account Key as per Google documentation. Remember the location of your key file.
   • (ii) You will need to grant the Service Account at least read, list, create, and delete privileges. For more information about google iam and Service Accounts in particular, please consult the documentation. For this exercise, grant your service account the roles/compute.storageAdmin role would be adequate.
3. For AWS:
   • (i) Create and download AWS user credentials as per AWS documentation. We will require long-lived credentials. In keeping with vendor advice, we strongly recommend against using root user credentials. We have created a dedicated CICD user for this exercise.
   • (ii) Set up the AWS CLI environment variables as per the documentation.
   • (iii) The user will need create / read / delete privileges against ec2 volumes. This can be done though the AWS IAM console in various ways. For example, one can use groups and permission policies. Adding your user to a group with AmazonEC2FullAccess will certainly work, although lesser privileges may be adequate.

Then, create some shell variables:

# you will need to edit the file path as appropriate

GOOGLE_DOWNLOADED_KEY_FILE_PATH="/path/to/your/downloaded/key.json"

AWS_AUTH_FRAGMENT='{ "type": "aws_signing_v4", "credentialsenvvar": "AWS_SECRET_ACCESS_KEY", "keyIDenvvar": "AWS_ACCESS_KEY_ID" }'

GOOGLE_AUTH_FRAGMENT='{ "credentialsfilepath": "'"${GOOGLE_DOWNLOADED_KEY_FILE_PATH}"'", "type": "service_account" }'

export STACKQL_AUTH_CTX='{ "aws":  '"${AWS_AUTH_FRAGMENT}"', "google": '"${GOOGLE_AUTH_FRAGMENT}"'  }'

$GOOGLE_DOWNLOADED_KEY_FILE_PATH = "C:\path\to\your\downloaded\key.json"

$AWS_AUTH_FRAGMENT = '{ "type": "aws_signing_v4", "credentialsenvvar": "AWS_SECRET_ACCESS_KEY", "keyIDenvvar": "AWS_ACCESS_KEY_ID" }'

$GOOGLE_AUTH_FRAGMENT = '{ "credentialsfilepath": "' + $GOOGLE_DOWNLOADED_KEY_FILE_PATH + '", "type": "service_account" }'

$env:STACKQL_AUTH_CTX = '{ "aws": ' + $AWS_AUTH_FRAGMENT + ', "google": ' + $GOOGLE_AUTH_FRAGMENT + ' }'

Start a stackql shell session​

To start an interactive shell session, in the same shell you setup your envrioment variables, run:

stackql --auth="${STACKQL_AUTH_CTX}" shell

You can exit at any time with ctrl + C.

Setup and meta queries to get started​

StackQL providers are installed from the StackQL Provider Registry using the REGISTRY command. StackQL supports meta queries such as SHOW and DESCRIBE which can be used to explore the available services, resources, fields, and operations available in a given cloud or SaaS provider.

-- see available providers
registry pull list;

-- pull the required providers
registry pull google;

registry pull aws;

-- some the installed providers
show providers;

-- some meta queries
show services in google;

show resources in google.compute;

describe google.compute.disks;

Create block storage (optional)​

You will need to replace the items in <ANGLE_BRACKETS>.

-- create a google volume, await and verify creation completes successfully
insert /*+ AWAIT */ into google.compute.disks(
  project, 
  zone, 
  data__name, 
  data__sizeGb
) 
select 
  '<YOUR_GCP_PROJECT>', 
  'australia-southeast1-a', 
  'my-stackql-demo-disk-01', 
  '10' ;

-- create an aws volume, operation despatched on a BEST EFFORT basis
insert into aws.ec2.volumes(
  AvailabilityZone, 
  Size, 
  region) 
select 
  'ap-southeast-2a', 
  10, 
  'ap-southeast-2';

Interrogate cloud block storage​

-- query one resource from google
select 
 name, 
 split_part(split_part(type, '/', 11), '-', 2) as type, 
 status, 
 sizeGb as size 
from google.compute.disks 
 where project = '<YOUR_GCP_PROJECT>' 
 and zone = 'australia-southeast1-a';

-- query the equivalent from aws
select 
 volumeId as name, 
 volumeType as type, 
 status, 
 size 
from aws.ec2.volumes 
 where region = 'ap-southeast-2';

-- union the equivalent resources across clouds
select 
 'google' as vendor, 
 name, 
 split_part(split_part(type, '/', 11), '-', 2) as type, 
 status, 
 sizeGb as size 
from google.compute.disks 
 where project = '<YOUR_GCP_PROJECT>' 
 and zone = 'australia-southeast1-a'
union
select 
 'aws' as vendor, 
 volumeId as name, 
 volumeType as type, 
 status, 
 size 
from aws.ec2.volumes 
 where region = 'ap-southeast-2';

-- create a view for convenience
create view dual_cloud_block_storage as
select 
 'google' as vendor, 
 name, 
 split_part(split_part(type, '/', 11), '-', 2) as type, 
 status, 
 sizeGb as size 
from google.compute.disks 
 where project = '<YOUR_GCP_PROJECT>' 
 and zone = 'australia-southeast1-a'
union
select 
 'aws' as vendor, 
 volumeId as name, 
 volumeType as type, 
 status, 
 size 
from aws.ec2.volumes 
 where region = 'ap-southeast-2';

-- select from the newly created view, with ordering
select * from dual_cloud_block_storage order by name desc;

Delete block storage (if required)​

This will only work if the disks are deletable. For example, aws.ec2.volumes must have status = available; you can check this with the view we created above.

/* delete a google volume, await and verify creation completes successfully.
One at a time only... */
delete /*+ AWAIT */ from google.compute.disks
 where project = '<YOUR_GCP_PROJECT>' 
 and zone = 'australia-southeast1-a' 
 and disk = 'my-stackql-demo-disk-01';

-- delete an aws volume, operation despatched on a BEST EFFORT basis
delete from aws.ec2.volumes 
 where VolumeId = 'vol-049ee07b31aff451a'  
 and region = 'ap-southeast-2';

Verify the cleanup was successful​

select * from dual_cloud_block_storage order by name desc;

That's it for the scripted demo!

Get involved​

StackQL is an intelligent API client which uses SQL as a front-end language with support for multi-cloud and SaaS provider environments, you can find more information at github.com/stackql/stackql

StackQL can provide valuable insights into your cloud and SaaS estates, whether for security posture management, cross-cloud entitlements reporting, cost optimization, or asset/inventory management.

As an interactive analysis tool, Jupyter notebooks can leverage StackQL to provide sources for cloud and SaaS provider data.

We've recently added magic function support for running StackQL queries in Jupyter notebooks, making the integration between StackQL and Jupyter more seamless. StackQL magic and be used on a line in a cell or the entire cell itself, as shown here:

The stackql-jupyter-demo Docker image is available from Docker Hub. You can find instructions to run using the Docker Hub image and instructions to run using docker-compose at github.com/stackql/stackql-jupyter-demo.

Give us your feedback!

StackQL allows you to query and interact with your cloud and SaaS assets using a simple SQL framework

Understanding entitlements across a GCP org with a complex hierarchy is a challenge. I have taken and data-centric approach to this in this article.

Prerequisites include setting up a Jupyter environment with StackQL (done here using Docker): stackql-jupyter-demo. You will also need a service account and associated key with the roles/iam.securityReviewer role.

I've broken the notebook bits down to explain...

Setup​

This step includes importing the required libraries (pandas etc.) and instantiating a StackQL client with the service account creds you created before. You will supply your root node here using the org_id and org_name variables.

Next we will create some helper functions; these will help us enumerate nodes in the GCP org resource hierarchy and fetch and unnest IAM policies.

Get all nodes in the resource hierarchy​

Create a dataframe containing all nodes in the resource hierarchy, including the root node (the organization), each folder with its subfolders, and projects. The functions used will search each folder in the hierarchy to find its subfolders and projects using a depth-first search approach.

Inspecting the output, it looks like this:

Create a dataset including each node and its associated IAM policies​

This step will fetch all of the policies applied at each node in the data structure we created in the previous step.

The IAM policies response from SELECT role, members FROM google.cloudresourcemanager.project_iam_policies ... presents some challenges as members is a nested list which we need to unnest (or explode) along with the associated role and conditions (if they exist).

This bit of massaging will give us a SQL-friendly model we can use for analysis and join with another data source (such as a list of identities from an identity provider).

Inspecting the Final Output

We can now peek at the final data set, which looks like this:

What's next? You could now join this with data from your IdP, or other SaaS services to correlate entitlements across your entire estate. You could also drill into specific service accounts, users, or groups. Queries are run in real-time, so you can refresh the data by simply rerunning the cells.

Welcome your feedback by getting in touch or raising issues at stackql/stackql or stackql/stackql-provider-registry, give us some ⭐️ love while you are there!

Enjoy!