Filtering and formatting fun with gcloud, GCP’s command line interface

Posted by Salmaan Rashid, Technical Account Manager, Google Cloud Platform


The gcloud command line tool is your gateway to manage and interact with Google Cloud Platform. Being a command line tool, you're probably already thinking of using system tools like cat|sed|awk|grep|cut to extract out all the info gcloud offers. In fact, gcloud itself offers a variety of options that will help you avoid having to use those commands. In this article, we describe a couple of options you can use to automatically parse and format the results. We’ll also show you a how to chain these commands together in a bash or powershell script to extract the embedded data.

We’re going to demonstrate three gcloud features which you can extend and combine in a variety of ways:

• filters to return a subset of the result
• format to change how that data is rendered
• projections to apply transforms or logic directly to the data returned

Format

Let's start off by formatting a simple command that you are already familiar with that lists the projects to which you have access:

1. gcloud projects list

PROJECT_ID            NAME          PROJECT_NUMBER
canvas-syntax-130823  scesproject2  346904393285
windy-bearing-129522  scesproject1  222844913538

Now let’s see the raw out output of this command by asking for the raw JSON format of the response:

2. gcloud projects list --format="json"

[
  {
    "createTime": "2016-04-28T22:33:12.274Z",
    "labels": {
      "env": "test",
      "version": "alpha"
    },
    "lifecycleState": "ACTIVE",
    "name": "scesproject1",
    "parent": {
      "id": "297814986428",
      "type": "organization"
    },
    "projectId": "windy-bearing-129522",
    "projectNumber": "222844913538"
  },
  {
    "createTime": "2016-05-11T03:08:13.359Z",
    "labels": {
      "env": "test",
      "version": "beta"
    },
    "lifecycleState": "ACTIVE",
    "name": "scesproject2",
    "parent": {
      "id": "297814986428",
      "type": "organization"
    },
    "projectId": "canvas-syntax-130823",
    "projectNumber": "346904393285"
  }
]

Seeing the raw JSON now lets us select the resources we're interested in and the formats we'd like. Let's display the same response in a formatted box sorted by createdTime and only select certain properties to display:

3. gcloud projects list --format="table[box,title='My Project List'](createTime:sort=1,name,projectNumber,projectId:label=ProjectID,parent.id:label=Parent)"

┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                        My Project List                                         │
├──────────────────────────┬──────────────┬────────────────┬──────────────────────┬──────────────┤
│       CREATE_TIME        │     NAME     │ PROJECT_NUMBER │      ProjectID       │    Parent    │
├──────────────────────────┼──────────────┼────────────────┼──────────────────────┼──────────────┤
│ 2016-04-28T22:33:12.274Z │ scesproject1 │ 222844913538   │ windy-bearing-129522 │ 297814986428 │
│ 2016-05-11T03:08:13.359Z │ scesproject2 │ 346904393285   │ canvas-syntax-130823 │ 297814986428 │
└──────────────────────────┴──────────────┴────────────────┴──────────────────────┴──────────────┘

Tip: you can derive the JSON path value for a property by using --format='flattened' flag.

Say you don't want a formatted box, just a table without a border with a simple display of the date property in the format year-month-day:

4. gcloud projects list --format="table(createTime.date('%Y-%m-%d'),name,projectNumber,projectId)"

CREATE_TIME  NAME          PROJECT_NUMBER  PROJECT_ID
2016-05-11   scesproject2  346904393285    canvas-syntax-130823
2016-04-28   scesproject1  222844913538    windy-bearing-129522

Now let's do some more complex formatting. To see this, list out the Compute Engine zones and peek at the JSON:

5. gcloud compute zones list --format="json"

{
    "creationTimestamp": "2014-05-30T18:35:16.514-07:00",
    "description": "us-central1-a",
    "id": "2000",
    "kind": "compute#zone",
    "name": "us-central1-a",
    "region": "us-central1",
    "selfLink": "https://www.googleapis.com/compute/v1/projects/windy-bearing-129522/
     zones/us-central1-a",
    "status": "UP"
  },

Note the selfLink. It's the fully qualified name that you'd like to parse. gcloud can help here too by giving you functions to select the JSON value and then extract and parse it. Let’s grab the last part of the URL segment of selfLink by using the selfLink.scope() function:

6. gcloud compute zones list --format="value(selfLink.scope())"

us-central1-a

Alternatively, you can extract the value using .basename():

7. gcloud compute zones list --format="value(selfLink.basename())"

us-central1-a

Suppose you want to extract part of the selfLink starting from the /projects path:

8. gcloud compute zones list --format="value(selfLink.scope(projects))"

windy-bearing-129522/zones/us-central1-a

Some GCP objects have multi-valued resources and we often need to enumerate them. For example, consider listing out all scopes enabled for a given GCE instance:

9. gcloud compute instances list --format="json"


"serviceAccounts": [
      {
        "email": "1071284184436-compute@developer.gserviceaccount.com",
        "scopes": [
          "https://www.googleapis.com/auth/devstorage.read_only",
          "https://www.googleapis.com/auth/logging.write",
          "https://www.googleapis.com/auth/monitoring.write",
          "https://www.googleapis.com/auth/cloud.useraccounts.readonly"
        ]
      }
    ],

What we actually want to do here is flatten the multi-valued resources:

10. gcloud compute instances list --format="flattened(name,serviceAccounts[].email,serviceAccounts[].scopes[].basename())"

name:                         instance-1
serviceAccounts[0].email:     1071284184436-compute@developer.gserviceaccount.com
serviceAccounts[0].scopes[0]: devstorage.read_only
serviceAccounts[0].scopes[1]: logging.write
serviceAccounts[0].scopes[2]: monitoring.write
serviceAccounts[0].scopes[3]: cloud.useraccounts.readonly

Or flatten multi-values to a separate line per value:



11. gcloud compute instances list --filter=name:instance-1 --flatten="serviceAccounts[].scopes[]" --format="csv(name,id,serviceAccounts.email,serviceAccounts.scopes.basename())"

name,id,email,scopes
instance-1,763097360168409044,1071284184436-compute@developer.gserviceaccount.com,
devstorage.read_only
instance-1,763097360168409044,1071284184436-compute@developer.gserviceaccount.com,
logging.write
instance-1,763097360168409044,1071284184436-compute@developer.gserviceaccount.com,
monitoring.write
instance-1,763097360168409044,1071284184436-compute@developer.gserviceaccount.com,
servicecontrol
instance-1,763097360168409044,1071284184436-compute@developer.gserviceaccount.com,
service.management

Here is the same information in an easy-to-read, structured format:

12. gcloud compute instances list --filter=name:instance-1 --format="table[box,no-heading](name,id,serviceAccounts:format='table[box,no-heading](email,scopes:format=\"table[box,no-heading](.)\")')"

┌────────────┬────────────────────┐
│ instance-1 │ 763097360168409044 │
└────────────┴────────────────────┘
    ┌────────────────────────────────────────────────────┐
    │ 1071284184431-compute@developer.gserviceaccount.com│
    └────────────────────────────────────────────────────┘
        ┌──────────────────────────────────────────────────────┐
        │ https://www.googleapis.com/auth/devstorage.read_only │
        │ https://www.googleapis.com/auth/logging.write        │
        │ https://www.googleapis.com/auth/monitoring.write     │
        │ https://www.googleapis.com/auth/servicecontrol       │
        │ https://www.googleapis.com/auth/service.management   │
        └──────────────────────────────────────────────────────┘

The final formatting example parses a multi-valued resource to display the service account keys with the service account for the following raw output:

13. gcloud beta iam service-accounts keys list --iam-account svc-2-429@mineral-minutia-820.iam.gserviceaccount.com --project mineral-minutia-820 --format="json"

[
  {
    "name": "projects/mineral-minutia-820/serviceAccounts/svc-2-429@mineral
-minutia-820.iam.gserviceaccount.com/keys/
04bd2d56d0cc5746b125d17f95d4b0dd654accca",
    "validAfterTime": "2016-03-11T05:30:04.000Z",
    "validBeforeTime": "2026-03-09T05:30:04.000Z"
  },
  {
    "name": "projects/mineral-minutia-820/serviceAccounts/svc-2-
429@mineral-minutia-820.iam.gserviceaccount.com/keys/
1deb44e2f54328fc7bb316e5a87315e3314f114f",
    "validAfterTime": "2016-01-02T18:54:26.000Z",
    "validBeforeTime": "2025-12-30T18:54:26.000Z"
  },
....
]

So use .scope() to extract just the serviceAccount part, then grab the first '/' delimited part with segment(0):

14. gcloud beta iam service-accounts keys list --iam-account svc-2-429@mineral-minutia-820.iam.gserviceaccount.com --project mineral-minutia-820 --format="table(name.scope(serviceAccounts).segment(0):label='service Account',name.scope(keys):label='keyID',validAfterTime)"

(click to enlarge)

Filters

Let's talk about filters. Filters allow you to select only the resources to which you want to apply formatting.For example, suppose you labeled your resource (projects, VM's, etc.) with a specific name, and you want to list only those projects where the labels match specific values (e.g. label.env='test' and label.version=alpha):

15. gcloud projects list --format="json" --filter="labels.env=test AND labels.version=alpha"

[
  {
    "createTime": "2016-04-28T22:33:12.274Z",
    "labels": {
      "env": "test",
      "version": "alpha"
    },
    "lifecycleState": "ACTIVE",
    "name": "scesproject1",
    "parent": {
      "id": "297814986428",
      "type": "organization"
    },
    "projectId": "windy-bearing-129522",
    "projectNumber": "222844913538"
  }
]

You can also apply projections on keys. In the example below, the filter is applied on the createTime key after the date formatting is set:

16. gcloud projects list --format="table(projectNumber,projectId,createTime)" --filter="createTime.date('%Y-%m-%d', Z)='2016-0511'"

PROJECT_NUMBER  PROJECT_ID            CREATE_TIME
346904393285    canvas-syntax-130823  2016-05-11T03:08:13.359Z

Notice the filter selected above actually references a JSON structure (labels.env=test).You can of course use that and combine it in any number of ways.

Projection transforms

Projection transforms allow you to alter the value rendered directly. We already showed several examples above (e.g., .extract(), .scope(), .basename(), .segment()). To note, one interesting capability of transforms is that you can combine and chain them together with .map() and and apply them to multi-valued data.

For example, the following applies conditional projection to the parent.id key such that if the parent.id key exists, the output is "YES" and otherwise its "NO". This is a quick way to see which of your projects meets a specific criteria (in this case, is it part of the Organization Node)

17. gcloud projects list --format="table(projectId,parent.id.yesno(yes="YES", no=”NO”):label='Has Parent':sort=2)"

PROJECT_ID                Has Parent
mineral-minutia-820       NO
fabled-ray-104117         YES
rk-test-0506              YES
user2proj1                YES
user2project2             YES

18. gcloud compute instances list --format="flattened(name,serviceAccounts[].email,serviceAccounts[].scopes.map().scope())"

name:                         instance-1
serviceAccounts[0].email:     1071284184436-compute@developer.gserviceaccount.com
serviceAccounts[0].scopes[0]: devstorage.read_only
serviceAccounts[0].scopes[1]: logging.write
serviceAccounts[0].scopes[2]: monitoring.write
serviceAccounts[0].scopes[3]: cloud.useraccounts.readonly

Scripts

Finally, let's see how we can combine gcloud commands into a script that will help us easily extract embedded information. In the following example, we list all the keys associated with all your projects’ service accounts.To do this, we first need to enumerate all the projects, then for each project, get all of its service accounts.Finally, for each service account, we list all the keys created against it. This is basically a nested loop to iterate over:

As a bash script:

#!/bin/bash
for project in  $(gcloud projects list --format="value(projectId)")
do
  echo "ProjectId:  $project"
  for robot in $(gcloud beta iam service-accounts list --project $project --format="value(email)")
   do
     echo "    -> Robot $robot"
     for key in $(gcloud beta iam service-accounts keys list --iam-account $robot --project $project --format="value(name.basename())")
        do
          echo "        $key" 
     done
   done
done

Or as Windows PowerShell:

foreach ($project in gcloud projects list --format="value(projectId)")
{
  Write-Host "ProjectId: $project"
  foreach ($robot in  gcloud beta iam service-accounts list --project $project --format="value(email)")
  {
      Write-Host "    -> Robot $robot"
      foreach ($key in gcloud beta iam service-accounts keys list --iam-account $robot --project $project --format="value(name.basename())")
      {
        Write-Host "        $key" 
      }
  }
}

You'll also often need to parse response fields into arrays for processing. The following example parses the service account information associated with an instance into an array for easy manipulation. Notice the serviceAccounts[].scope field is multi-valued within the csv and delimited by a semicolon since we defined "separator=;". That is, each response line from the gcloud command below will be in the form name,id,email,scope_1;scope_2;scope_3. The script below essentially parses the response from example 12 above:

#!/bin/bash
for scopesInfo in $(
    gcloud compute instances list --filter=name:instance-1 \
        --format="csv[no-heading](name,id,serviceAccounts[].email.list(),
                      serviceAccounts[].scopes[].map().list(separator=;))")
do
      IFS=',' read -r -a scopesInfoArray<<< "$scopesInfo"
      NAME="${scopesInfoArray[0]}"
      ID="${scopesInfoArray[1]}"
      EMAIL="${scopesInfoArray[2]}"
      SCOPES_LIST="${scopesInfoArray[3]}"

      echo "NAME: $NAME, ID: $ID, EMAIL: $EMAIL"
      echo ""
      IFS=';' read -r -a scopeListArray<<< "$SCOPES_LIST"
      for SCOPE in  "${scopeListArray[@]}"
      do
        echo "  SCOPE: $SCOPE"
      done
done

Hopefully, this has given you ideas for how to effectively filter and format gcloud command output. You can apply these techniques and extend them to any gcloud response — just look at the raw response, think about what you want to do, and then format away!