TL;DR if you want to skip all this and just get to the good stuff, click here to download and run dq.

---

As is the case with any seasoned DevOps engineer, I have a set of tools in my kit that I simply cannot live without. If I had to distill them into a top-5 list it would be the following:

1. Bash scripting for portability.
2. jq to filter JSON objects from RESTful APIs.
3. Docker ( or more recently Podman ) for managing containers.
4. AWS CLI as I cannot stand dealing with the web console.
5. curl ( with wget as a reasonable fallback ) to interact with various APIs.
6. Terraform for most of my IaC needs.

It wouldn’t be an exaggeration to say I literally use these tools every day. Specifically, the first 3 items. Furthermore, to say that manually sorting through Docker containers is a practice in tedium would be a massive understatement. Because of this, not a day passes where I am not grateful to have found jq.

In this case, my favourite duo is Docker and jq... which is what this article is about. Let’s learn some fun filtering patterns to make your life a bit easier if you’re stuck in a terminal all day like myself.

Disclaimer: Yes, this is 6 tools, but i listed Terraform as a bonus item because ❤️.

Setting Up

First and foremost, you’re going to need to install jq in your test environment. If you’re on most common Linux distributions:

apt-get install jq

Or, on MacOS:

brew update && brew install jq

I’m assuming that if you’re reading this you already have Docker installed in your test environment. If not, head out to their website and read their installation documentation. It tends to vary wildly depending on your OS.

Let’s spin up some containers we can play around with. I use redis for stuff like this:

for i in {1..5}; do docker run -it --rm -d redis; done                                                                  ⏎ ✹

Environmental Variables

Sourceable Blocks

There may be times I wish to collate all environmental variables associated with a set of containers and stick them a .env file to source later. Here we create a block for all the redis containers we just spun up:

docker inspect $(docker ps -q) | jq -r '
    .[].Config.Env
  | flatten 
  | .[]
'

This one will generate a lot of duplicates as all our test containers are effectively clones. What if we want a distinct set of instead? We can use the unique filter:

docker inspect $(docker ps -q) | jq -r '
    [
      .[].Config.Env
    ] 
  | flatten 
  | unique 
  | .[]
'

What if I want a set from specific containers?

docker inspect $(docker ps -q) | jq -r '
    [
        .[] 
      | select([.Id] | inside([
          "747c2c92855f88a8e9aa9709dcdf3a01f1677e70bf4c0bf0e520eb38ac502876",
          "2cf70261ef62bc36d19aba02f8ef7b8d7aabfcb1e9f4593a919baa17f80aca5b"
        ]))
      | .Config.Env
    ] 
  | flatten 
  | unique 
  | .[]
'

How about filtering by a specific ip address for the default bridge network? Using a different network? Just replace bridge with the alternate network name.

docker inspect $(docker ps -q) | jq -r '
    .[] 
  | select(.NetworkSettings.Networks.bridge.IPAddress == "172.17.0.2") 
  | .Config.Env 
  | .[]
'

Multiple ip addresses?

docker inspect $(docker ps -q) | jq -r '
    .[] 
  | select([.NetworkSettings.Networks.bridge.IPAddress] 
  | inside([
      "172.17.0.2",
      "172.17.0.3"
    ])) 
  | .Config.Env 
  | .[]
'

As Flags Instead

Here’s a strange one I’ve had to do. What if we want to recreate a list of --env flags we could pass to other docker run ... commands?

docker inspect $(docker ps -q) | jq -r '
    [
        [
          .[].Config.Env
        ] 
      | flatten 
      | unique 
      | "--env=\(.[])"
    ] 
  | join(" ")
'

You’ll get a string that looks similar to the following.

--env=GOSU_VERSION=1.14 --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=REDIS_DOWNLOAD_SHA=f0e65fda74c44a3dd4fa9d512d4d4d833dd0939c934e946a5c622a630d057f2f --env=REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-7.0.4.tar.gz --env=REDIS_VERSION=7.0.4

Filtering By Relative Time

I’ve had to use these several times for things like garbage collecting and cleaning up long-running containers. This uses a bit of Bash to help generate the relative timestamps. We use the date command for this, but if you’re on a darwin-based OS you’ll need to install gdate first via Homebrew with:

brew install coreutils

Substitute the following calls to date with gdate below if on MacOS.

Newer Than …

Filtering out recent containers with relative time is fairly straight-forward. The following gives us all containers created in the last hour as JSON output from docker inspect ...:

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) > $threshold)
'

Perhaps you just want the names of the containers instead:

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) > $threshold)
  | .Name[1:]
'

Docker likes to put a / at the beginning of each generated container pet name, so we use .Name[1:] to snip it off.

What about just returning the associated ids?

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) > $threshold)
  | .Id
'

Or, ip addresses:

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) > $threshold)
  | .NetworkSettings.Networks.bridge.IPAddress
'

Older Than???

The same filters apply, but you’re just flipping the compare operator from > to <:

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) < $threshold)
'

Other Common Patterns

There have been times where I need a block of container ip addresses so I can dynamically update some Nginx upstreams somewhere. With jq it’s as easy as:

docker inspect $(docker ps -q) | jq -r '
  .[].NetworkSettings.Networks.bridge.IPAddress
'

The output would look something like this:

172.17.0.6
172.17.0.5
172.17.0.4
172.17.0.3
172.17.0.2

Perhaps, I want to find a specific container id by it’s ip address:

docker inspect $(docker ps -q) | jq -r '
    .[]
  | select(.NetworkSettings.Networks.bridge.IPAddress == "172.17.0.3")
  | .Id
'

Or, just give me the names of all running containers:

docker inspect $(docker ps -q) | jq -r '
  .[].Name[1:]
'

Which would give you something like:

zealous_hertz
fervent_dijkstra
focused_galileo
zealous_poincare
naughty_wescoff

Cleaning Up …

We’re considerate people, so let’s clean up after ourselves by killing all containers created using the redis image:

docker kill $(docker inspect $(docker ps -q) | jq -r '
    .[] 
  | select(.Config.Image == "redis") 
  | .Id
')

Keep in mind this will kill all redis containers.

In Closing …

Having a toolbox filled with utilities you can mix and match together is great if you spend most of your day working in a terminal on a glowing rectangle. Hopefully, I’ve made a strong enough case to for Docker and jq as a great combo.

But, Wait! There’s More!

Why copy and paste these commands in your terminal, when you could just download and install a handy little Bash script to do it all for you? dq has all the above built-in as well as heaps more bells and whistles.

Here are a few command examples if you’re interested:

dq older-than 4 months --only-return ips --network pebkac
dq newer-than 2 fortnights --only-return ids
dq filter '.[].Name[1:]'
dq find-by-ip-address 172.17.0.2 --only-return names --network foo

Give it a try here: