First, a Small Rant#

Before you call me a Luddite, let’s be clear here, I don’t have an issue with the technology itself. I’m an engineer. I think it all has its uses, but force-feeding it into almost every aspect of our lives with no regard to any social and environmental impacts with frequently no chance of being able to easily opt-out does not sit right with me one bit.

We’ve seen this happen over the last decade with touchscreens, IoT, blockchain, NFTs, Web3 the list goes on. All promoted by the endless parade of interchangeable, nameless faceless talentless hacks and disingenuous grifters who only care about making “BIG LINE GO UP”.

It’s all so tiring, so perhaps you can understand why I’d immediately recoil in disgust and go my own way with an alternative I can control.

Or, to put it in even simpler terms:

The “Stack”#

I suppose it’s technically a “stack”, but it seems a bit funny to use the term when referencing a blog setup. Just as with my Plant🌱Smart project, the goal is to automate as much as possible while keeping maintenance requirements and hard- and soft-dollar costs as low as possible. For this, I need only use a handful of tools to keep this space operational.

As far as the question over “cost”, outside of time spent, it’s about ~$100 AUD per year for the domain name. This blog is a very small and inconsequential part of this domain, so the overall cost is absorbed by “other stuff”.

So, the greatest cost is… time? I guess 🤷

Obsidian#

While I won’t be going into what Obsidian is and how it works, I will say it’s been a personal boon in how I keep notes and track of ideas. I have a lot of fleeting / ephemeral thoughts I that would otherwise lose instantly if I didn’t immediately jot them down using this app.

So, it makes perfect sense for me to use my personal vault to store all my blog content. The fact Obsidian content nothing more than Markdown files makes working with them using other bits of tech that much easier.

Oh, it’s free btw.

Hugo#

This is probably one of the more popular options when it comes to static site generators. It’s easy to install, is incredibly extensible and has loads of documentation to help you along. The template syntax sits upon Golang’s templating engine, so if you have a strong background as a Go developer like myself, you’ll be right at home.

However, knowing Go isn’t a hard requirement though it will make getting on top of things quite a bit easier.

What I enjoyed while learning about this project was I got to control every aspect of how I wanted my blog to look and function. Building the current design from the ground up with Hugo was incredibly fun.

Also, free. Very cool.

Cloudflare#

I already used Cloudflare to manage this domain’s DNS settings, so it only made sense to publish my static site to Cloudflare Pages over, say, Vercel or Github Pages. There’s nothing spectacular going on here outside of me just wanting to keep these two things in the same place.

That being said, I have no issues with migrating else where if the need should ever arise. Though, I doubt this blog will hit the 20k file limit of Cloudflare Pages any time soon.

Again, free for the purposes of this article. If you don’t have a dedicated personal domain, you will be provided a subdomain attached to the pages.dev apex.

The Good Stuff#

How is this all put together for day-to-day usage? Luckily, the Cloudflare stuff runs on auto-pilot, so it’s pretty much set-and-forget. I suppose you could say the same about the other components, but they’re the parts I touch the most.

I have a single vault in Obsidian. It’s where all my thoughts go. Stored within is a top-level Blog/ directory. Care to hazard a guess as to what it may contain?

Hugo has the concept of page bundles where you can group all resources associated with your blog articles within a single directory. You’ll notice above all directories mirror the blog posts hosted on this site. In my case, we can consider the directory names to be the human-friendly slugs you see in your address bar that point to the associated content. Within these directories, you’d see any other resource I may link to; source code, images, etc… The best thing about this is I only have to perform a relative reference to these resources. If I want to link to image-1.png, I only have to reference it as ![](/blog/my-blog-publishing-setup/image-1.png) without worrying about specifying an absolute path; very nice.

When I want to write a new article, I first create a new directory here. I may already have the title of the article in mind, but it can be in flux until I decide to publish it. In the new directory, I create a file called index.md. This file contains all the content of the associated article written using Markdown.

I then use an Obsidian template dedicated to new blog posts and apply it to the new file. This ensures I have all front matter properties ready to go. This makes configuring my posts in the editor that much easier.

All that’s left is writing my articles! Well, not really. I need a way to synchronise these files with my local Hugo instance. As you can see here, the content mirrors what I have stored in my Obsidian vault.

This can be easily addressed with 2 small apps:

1. fswatch which monitors a specified directory for changes and allows you to trigger subsequent commands for specific file system events. Basically, I want to watch for any changes in my Obsidian vaults Blog/ directory.
2. rsync to dynamically keep the target Hugo directory in sync with the Obsidian source directory.

I, of course, have this placed in a script that I have running in the background while I write. This is a great help during the drafting phase as I have Hugo dynamically processing and displaying these changes locally. This allows me to easily proof my work visually as I carry on writing.

Here’s the full script:

#!/usr/bin/env bash

SOURCE=${SOURCE:="${1}"}
export SOURCE

DESTINATION=${DESTINATION:="${2}"}
export DESTINATION

fswatch -o "${SOURCE}" | while read -r event; do 
    rsync -av --delete "${SOURCE}" "${DESTINATION}";
done

I keep this running whenever I’m writing a new article. Obsidian saves in almost realtime, so as I type the changes almost immediately sync across to the local Hugo server so I can review my changes.

You effectively run it as:

$ ./sync.sh /path/to/obsidian/blog/ /path/to/hugo/content/

For rsync to work as intended and keep the destination directory completely in sync with the source directory, remember to add a trailing slash to both directory arguments.

No Auto-Commit?#

Nope. Not here. There are a few other peeps with blogs out there that “commit on change”, but I prefer to have a bit more control over what actually gets published. For that, I simply use plain-old git commands to publish my new article:

$ git add -A
$ git commit -m 'some banal new bullshit screed'
$ git push origin main

Cloudflare monitors the associated repository, picks up the change and the auto-magic-ally builds and pushes everything to the public eye.

With this flow, I can see my changes locally in near realtime and push to “production” only when I’m happy with the results.

In Closing …#

I’m stoked to have my very own place again that I have complete control over. I’ve thoroughly enjoyed creating this new blog design and I may even open-source it one day.

That said, I’ve made a very loose commitment to post a new article every week this year of our lord 2025 and I’m hoping this will make things heaps simpler. I only want to worry about what I’m going to write and just get the content out there.

I sincerely hope I can keep this self-imposed weekly commitment. 😬👍

I have no idea if time is actually passing. Obviously, I can eyeball timestamps if available, but there are times when so many lines zip through the buffer that it’s easy to lose track or even go cross-eyed.

It can be tricky to notice that lines are still being tailed if the output doesn’t drastically change in some meaningful or noticeable way. Something I like to is intercept each line and then output some kind of divider whenever n lines have been added to the buffer.

It’s as simple as:

#!/usr/bin/env bash

lines=0
while IFS= read -r line; do
	echo "${line}"
	((lines++))

	if ((lines % 5 == 0)); then
		echo "----------"
	fi
done

Effectively, all this does is:

1. Intercept each line of output being piped into the script.
2. Keep a running tally of lines we’ve intercepted so far; $lines++.
3. If the current tally is divisible by n, or in this case 5, spit out an additional line containing a divider.
4. Keep doing this forever until the process is terminated.

If you were to save this in a file named as divider.sh and set it to execute with something like chmod a+x you could test it by doing something like:

$ while true; do echo "emitting a noop"; sleep 1; done | ./divider.sh

And you’ll see something like the following:

emitting a noop
emitting a noop
emitting a noop
emitting a noop
emitting a noop
----------
emitting a noop
emitting a noop
emitting a noop
emitting a noop
emitting a noop
----------

This is pretty dumb, but now you can see that output is still being placed in your terminal buffer. I’ve set it to every 5 lines, but you can update the script to make that configurable. You could also play a sound when the script places a divider in the buffer using something like tput bel.

Anyway… Enjoy.

Before we begin#

Today, I went ahead and made my Instagram profile private. I have no desire to delete my posts just yet, but decided to go ahead and unfollow everyone as a start. There are several ways to do this:

• API calls using an SDK, which requires a developer account.
• Manually deleting “handraulically” via the native app or browser, but that can quickly become tedious if you have more than a hundred or so follows.
• Random “GreaseMonkey” scripts off the web that may, or may not, work properly if at all.
• Reverse-engineering XHR calls from your browser and writing a simple Bash script to automate the process.

If you’ve read some of my previous blog posts, you probably can guess which one I’ll be going with.

Requirements#

This guide assumes you have some programming or scripting knowledge.

You will need the following on your machine:

• A shell terminal with Bash, or ZSH, support.
• curl to make API calls from the command line.
• jq to easily parse JSON-based responses from said API calls. This can easily be installed using Homebrew or some other supported package manager.
• A modern browser that supports a developer tools console to intercept XHR calls.

It’s worth mentioning that I am using a MacBook with the Chrome browser, but other webkit-based browsers should support similar functionality.

Fire up the browser!#

Open up your browser, or new tab, and navigate to your Instagram profile. Open your browser’s developer console and navigate to the “Network” tab. Clear out whatever requests are currently listed and then filter by “Fetch/XHR”.

Your following list#

We need to intercept an XHR request that returns a JSON body containing some of your followers. Click the “following” link and monitor the resulting requests that pop in the list on the “Network” tab.

You should see something like following/count=12... in the list. Right-click that item and select “Copy as cURL”. Paste the new value in your clipboard into a scratch file for reference later.

Be aware this command will contain all the headers required to make requests from your terminal on your behalf. Do NOT share these details with anyone. I haven’t checked how long until the tokens within the auth headers expire, so assume they are long(ish)-lived and treat them accordingly.

An unfollow request#

Next, we need to intercept an XHR request to unfollow someone. Going back to your following list, select a random user and unfollow them while once again keeping an eye on the “Network” tab.

You’re going to see something like the following; an XHR event making a POST request to “destroy” your chosen follower. Once again, you’ll right-click and “Copy and cURL”. Paste this into your scratch file as well for future reference.

Putting it all together#

Now that you have your two types of requests, we’ll need to script the following steps:

• Execute a curl request to fetch a JSON body containing the accounts you follow, one page of results at a time.
• Use jq to extract their Instagram IDs.
• Store the results in a variable named… drumroll, pls… $ids.
• Iterate through each $id.
• Execute a curl request to unfollow each $id.

Here is the meat of the script. I have included only the necessary headers required to successfully make these requests on your Instagram account’s behalf. You’ll need to swap out the ... with your own values.

You may also notice that I’ve change the count=... parameter to 100. It may be able to go higher, but I haven’t tested it.

ids=$(
  curl 'https://www.instagram.com/api/v1/friendships/.../following/?count=100&hl=en' \
    -H 'cookie: ...'           \
    -H 'x-asbd-id: ...'        \
    -H 'x-csrftoken: ...'      \
    -H 'x-ig-app-id: ...'      \
    -H 'x-ig-www-claim: ...'   \
    -H 'x-requested-with: ...' \
    -H 'x-web-session-id: ...' | jq -r '.users[].id'
)

while IFS= read -r id; do
  echo -n "Unfollowing user '${id}'"

  curl -s -o /dev/null "https://www.instagram.com/api/v1/friendships/destroy/${id}/?hl=en" \
    -H 'cookie: ...'           \
    -H 'x-asbd-id: ...'        \
    -H 'x-csrftoken: ...'      \
    -H 'x-ig-app-id: ...'      \
    -H 'x-ig-www-claim: ...'   \
    -H 'x-requested-with: ...' \
    -H 'x-web-session-id: ...' \
    --data-raw "container_module=profile&nav_chain=PolarisProfilePostsTabRoot%3AprofilePage%3A1%3Avia_cold_start%2CPolarisProfilePostsTabRoot%3AprofilePage%3A2%3Aunexpected&user_id=${id}"

  echo ' ... done!'
done <<< "${ids}"

echo "Finished"

You can save this as a file and make it executable, but I just paste this bad boy into your terminal. The above script will remove 100 accounts at a time. Just keep executing the script until you no longer see results like the following:

Unfollowing user '111'... done
Unfollowing user '222'... done
Unfollowing user '333'... done
Unfollowing user '444'... done
Unfollowing user '555'... done
Unfollowing user '666'... done
Unfollowing user '777'... done
Unfollowing user '888'... done
Finished

Caveats, gotchas & other things#

I’ve only had a few hundred accounts that I followed, so I didn’t feel the need to make this a bullet-proof solution. There’s quite a bit missing and things can go wrong. Here are some things to consider off the top of my head :

• The session tokens will expire eventually, so you will have to refer back to your “Network” tab in your developer tools console to get new ones.
• As this example only unfollows a static amount at a time, re-executing the script may become tedious for very large follow numbers. Perhaps, you can modify this script to add some basic pagination.
• There is no error handling here, so keep an eye out for potential rate limiting or expired token issues with the Instagram API.
• Constantly updating session header values can be made a bit simpler by resorting to variables instead.

In closing …#

Cleaning out old, unused or unwanted social media accounts can be liberating. I’ve recently gutted my Twitter presence and I haven’t been on Facebook in years, though I still do use Messenger to keep in touch with a handful of people. I’d prefer using Signal for this, but trying to convince other people to install yet another messaging app is like pulling teeth.

Anyway, I actually enjoy writing these kinds of posts. What other social networks should I cover? LinkedIn? Threads? What are some ways you can improve and build on the above? Lemme know in the comment section below.

Let’s leave out all doubt regarding our god-tier levels of snarkiness.

Today, we’re going to write a laughably-simple Bash script that lets our coworkers know exactly what we think. Along the way, you’ll learn a little bit more about Bash such as:

1. Setting some default flags.
2. Reading text from stdin.
3. Parameter expansion.
4. Looping through each character of a string.
5. Creating random numbers.
6. Swapping character types using the tr command.
7. And, finally, making your long-suffering coworkers’ eyes roll.

Here We Go!#

First, I’m going to just paste the entire script here and then we’ll go through all the important bits line-by-line:

#!/usr/bin/env bash

set -eo pipefail

read text

for (( i=0; i < "${#text}"; i++ )); do
  if [[ $(( "${RANDOM}" % 2 )) -eq 0 ]]; then
    echo -n "${text:${i}:1}" | tr '[:lower:]' '[:upper:]'
  else
    echo -n "${text:${i}:1}" | tr '[:upper:]' '[:lower:]'
  fi
done

echo

Alright, let’s take it from the top. All Bash scripts should start with a “shebang line”. This tells your terminal which environments and runtimes your script should run under. There are more traditional versions of this — #!/bin/bash, for instance — but, our way is considered to be the most portable overall.

#!/usr/bin/env bash

Next, we setup some environmental flags. I use at least the following for all of my personal scripts:

set -eo pipefail

Here is what they do:

1. set -e: Instructs Bash to immediately exit if any command has a non-zero exit status. This is how it works in most languages, but with Bash, it just keeps on trying to execute subsequent commands. This is generally acceptable on the command line, but not in a script. If we encounter an error, we want to exit immediately.
2. set -o pipefail: This tells Bash not to mask errors that may appear in a pipeline of commands. We want any failed command’s exit code in a pipeline to bubble up to the script itself and then exit with that code.

Speaking of pipelines, we want to be able to pipe text to this script so we can do something like the following:

echo "wilhelm, i asked you to patch the server." | ./spongebob
wiLHELm, I AsKEd yOu TO PatCh The seRvEr.

Let’s halt the script and wait for user input and then assign that input to variable text:

read text

Now that we have some text, we need to start randomly-swapping between upper and lower case characters. In order to do that, we need to know the number of characters in our string so we can build a nice loop:

for (( i=0; i < "${#text}"; i++ )); do
  # ... sweet code goes here 
done

We could do something in a sub-shell here like $(echo "${text}" | wc -c) to get the character count, but why do that when we could get the same result with "${#text}". This lovely bit of “shell parameter expansion” helps us avoid sub-commands and sub-shells.

Next, we want to be able to randomly swap the capitalisation of each character in the string to get the appropriate effect. A character is either upper- or lower-case, so minimum we need only to swap randomly between 2 values; 0 and 1. We can get this effect in Bash by using the internal ${RANDOM} function like so with the modulo ( or “mod” ) operator:

if [[ $(( "${RANDOM}" % 2 )) -eq 0 ]]; then
  # ... do something
else
  # ... do the opposite
fi

We’re now at the point where we want to modify the character associated with the for loop’s current iteration, but how do we get it from the text variable? Once again we use some parameter expansion in the form of ${text:offset:length}. We have the value for “offset” already; it’s ${i}. We only want a single character returned, so we use 1 for “length”.

echo -n "${text:${i}:1}"

This spits out the current character for each iteration of our loop. The -n in the echo statement simply stops Bash from adding a newline to the end of the result. Otherwise, you’d get a line per character as output.

Finally, we want to do case swapping. For this, we pipe the output of the above echo command into the tr command. Within the above if statement, if our random number equals 0, let’s swap a lowercase [:lower:] character with an uppercase [:upper:]:

echo -n "${text:${i}:1}" | tr '[:lower:]' '[:upper:]'

And, then, we do the opposite for any other result:

echo -n "${text:${i}:1}" | tr '[:upper:]' '[:lower:]'

You’ll notice a final echo command at the bottom of the script. Thanks to the final echo -n ... command from the previous for loop, you may find your results prepended to your command prompt. This ensures a newline makes it to the end of your 🧽 output.

Testing Time#

Save the script as spongebob and make it executable with chmod a+x spongebob. That should be it! Here are a few of my results:

$ echo "abandon all hope, ye who enter here." | ./spongebob
abaNDON ALl Hope, yE wHO EnTer heRE.
$ echo "Wilhelm, that last deployment failed. Could you roll it back, please?" | ./spongebob
WilhelM, THAT LASt dEployMeNT fAILED. coUlD YOu RoLL It BaCk, PlEaSE?
$ echo "Wilhelm, I am your manager. Please, stop mocking me." | ./spongebob
wIlHElM, i aM yOUr MANagER. PlEasE, sTOP mOCKINg mE.
$ echo "Wilhelm, should we use Kubernetes for this?" | ./spongebob
WilHELM, should We Use KuBerNeTEs for THiS?

In Conclusion…#

Ok, obviously this was all a clever ploy to get you to learn a few more Bash things. Definitely do not use this new knowledge to frustrate and annoy your coworkers. Please, be considerate of other people’s mental well being. I mean, what I meant to say was:

There are any number of ways you could change this script. Instead of liberal use of echo, you could just build a string assigned to a variable and spit that out at the end. Instead of using else you could use continue to skip the final if fallback. Try a few and see what changes.

For the purposes of this article I felt the above sequence of commands was clear enough for most people to follow along.

I hope you learned something!

1. Bash for portability.
2. jq to filter JSON objects from RESTful APIs.
3. AWS CLI as I cannot stand dealing with the web console.
4. curl ( with wget as a reasonable fallback ) to interact with various APIs.
5. Terraform for most of my IaC needs.

It wouldn’t be an exaggeration to say I literally use these tools every day. Specifically, the first 3 items. Furthermore, to say that manually sorting through Docker containers is a practice in tedium would be a massive understatement. Because of this, not a day passes where I am not grateful to have found jq.

In this case, my favourite duo is Docker and jq… which is what this article is about. Let’s learn some fun filtering patterns to make your life a bit easier if you’re stuck in a terminal all day like myself.

Setting Up#

First and foremost, you’re going to need to install jq in your test environment. If you’re on most common Linux distributions:

apt-get install jq

Or, on MacOS:

brew update && brew install jq

I’m assuming that if you’re reading this you already have Docker installed in your test environment. If not, head out to their website and read their installation documentation. It tends to vary wildly depending on your OS.

Let’s spin up some containers we can play around with. I use redis for stuff like this:

for i in {1..5}; do docker run -it --rm -d redis; done

Environmental Variables#

Sourceable Blocks#

There may be times I wish to collate all environmental variables associated with a set of containers and stick them a .env file to source later. Here we create a block for all the redis containers we just spun up:

docker inspect $(docker ps -q) | jq -r '
    .[].Config.Env
  | flatten 
  | .[]
'

This one will generate a lot of duplicates as all our test containers are effectively clones. What if we want a distinct set of instead? We can use the unique filter:

docker inspect $(docker ps -q) | jq -r '
	[
	  .[].Config.Env
	] 
  | flatten 
  | unique 
  | .[]
'

What if I want a set from specific containers?

docker inspect $(docker ps -q) | jq -r '
    [
	    .[] 
      | select([.Id] | inside([
	      "747c2c92855f88a8e9aa9709dcdf3a01f1677e70bf4c0bf0e520eb38ac502876",
	      "2cf70261ef62bc36d19aba02f8ef7b8d7aabfcb1e9f4593a919baa17f80aca5b"
	    ]))
      | .Config.Env
	] 
  | flatten 
  | unique 
  | .[]
'

How about filtering by a specific ip address for the default bridge network? Using a different network? Just replace bridge with the alternate network name.

docker inspect $(docker ps -q) | jq -r '
    .[] 
  | select(.NetworkSettings.Networks.bridge.IPAddress == "172.17.0.2") 
  | .Config.Env 
  | .[]
'

Multiple ip addresses?

docker inspect $(docker ps -q) | jq -r '
    .[] 
  | select([.NetworkSettings.Networks.bridge.IPAddress] 
  | inside([
      "172.17.0.2",
      "172.17.0.3"
    ])) 
  | .Config.Env 
  | .[]
'

As Flags Instead#

Here’s a strange one I’ve had to do. What if we want to recreate a list of --env flags we could pass to other docker run ... commands?

docker inspect $(docker ps -q) | jq -r '
    [
        [
          .[].Config.Env
        ] 
      | flatten 
      | unique 
      | "--env=\(.[])"
    ] 
  | join(" ")
'

You’ll get a string that looks similar to the following.

--env=GOSU_VERSION=1.14 --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=REDIS_DOWNLOAD_SHA=f0e65fda74c44a3dd4fa9d512d4d4d833dd0939c934e946a5c622a630d057f2f --env=REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-7.0.4.tar.gz --env=REDIS_VERSION=7.0.4

Filtering By Relative Time#

I’ve had to use these several times for things like garbage collecting and cleaning up long-running containers. This uses a bit of Bash to help generate the relative timestamps. We use the date command for this, but if you’re on a darwin-based OS you’ll need to install gdate first via Homebrew with:

brew install coreutils

Substitute the following calls to date with gdate below if on MacOS.

Newer Than …#

Filtering out recent containers with relative time is fairly straight-forward. The following gives us all containers created in the last hour as JSON output from docker inspect ...:

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) > $threshold)
'

Perhaps you just want the names of the containers instead:

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) > $threshold)
  | .Name[1:]
'

Docker likes to put a / at the beginning of each generated container pet name, so we use .Name[1:] to snip it off.

What about just returning the associated ids?

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) > $threshold)
  | .Id
'

Or, ip addresses:

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) > $threshold)
  | .NetworkSettings.Networks.bridge.IPAddress
'

Older Than???#

The same filters apply, but you’re just flipping the compare operator from > to <:

threshold=$(date -d "1 hour ago" +%s)
docker inspect $(docker ps -q) | jq --argjson threshold "${threshold}" -r '
    .[] 
  | select((.State.StartedAt | split(".")[0] | "\(.)Z" | fromdate) < $threshold)
'

Other Common Patterns#

There have been times where I need a block of container ip addresses so I can dynamically update some Nginx upstreams somewhere. With jq it’s as easy as:

docker inspect $(docker ps -q) | jq -r '
  .[].NetworkSettings.Networks.bridge.IPAddress
'

The output would look something like this:

172.17.0.6
172.17.0.5
172.17.0.4
172.17.0.3
172.17.0.2

Perhaps, I want to find a specific container id by it’s ip address:

docker inspect $(docker ps -q) | jq -r '
	.[]
  | select(.NetworkSettings.Networks.bridge.IPAddress == "172.17.0.3")
  | .Id
'

Or, just give me the names of all running containers:

docker inspect $(docker ps -q) | jq -r '
  .[].Name[1:]
'

Which would give you something like:

zealous_hertz
fervent_dijkstra
focused_galileo
zealous_poincare
naughty_wescoff

Cleaning Up …#

We’re considerate people, so let’s clean up after ourselves by killing all containers created using the redis image:

docker kill $(docker inspect $(docker ps -q) | jq -r '
	.[] 
  | select(.Config.Image == "redis") 
  | .Id
')

Keep in mind this will kill all redis containers.

In Closing …#

Having a toolbox filled with utilities you can mix and match together is great if you spend most of your day working in a terminal on a glowing rectangle. Hopefully, I’ve made a strong enough case to for Docker and jq as a great combo.

But, Wait! There’s More!#

Why copy and paste these commands in your terminal, when you could just download and install a handy little Bash script to do it all for you? dq has all the above built-in as well as heaps more bells and whistles.

Here are a few command examples if you’re interested:

dq older-than 4 months --only-return ips --network pebkac
dq newer-than 2 fortnights --only-return ids
dq filter '.[].Name[1:]'
dq find-by-ip-address 172.17.0.2 --only-return names --network foo

Gathering Requirements#

Since we’re doing some destructive operations, we’ll need to make sure our Slack user has the appropriate permissions to manage emoji; adding & deleting. This means using a different section of the Slack interface which uses an entirely seperate set of API endpoints.

You’ll still be using the following, which can be found using instructions from the previous article:

• An API request token.
• A session cookie.
• A workspace, or team, id.

In addition, you will need your workspaces’s subdomain, or URL. This can easily be found within the Slack app itself:

Testing Your Findings#

This is an incredibly straight-forward process as it’s quite similar to what we already know. The primary difference is this is a multipart/form-data upload. So, instead of shipping of a JSON payload, it’s a form with associated fields.

With the information gathered above, you can upload directly to Slack using a simple cURL command like so:

curl -s --compressed "https://<domain>.slack.com/api/emoji.add" \
  -H 'content-type: multipart/form-data' \
  -H "cookie: d=<cookie>;" \
  -F "token=<token>" \
  -F "name=<name>" \
  -F mode=data \
  -F "image=@<local-emoji-path>"

Replace the following:

• <domain> is your workspace’s, or team’s, private URL.
• <cookie> is your session cookie value.
• <token> is your request token.
• <name> will be the named reference of your new emoji.
• <path> is the relative, or absolute, local path of your emoji file to upload.

The JSON response to this request will have the following structure if all goes well:

{ "ok": true }

And, if something went wrong:

{ 
	"ok": false,
	"error": "err_code"
}

That’s it! Now that we know how to upload from the command line.

The Implementation#

If you’re only uploading a handful of emojis, it might make sense to just do it via Slack’s UI. However, this can get a bit tedious if you have dozens, or even hundreds, to upload. We can automate things even further by:

• Gathering all supported images from a specific source directory.
• Iterating through our findings and bulk-upload them all in one go.

So, let’s do just that. But, first, let’s set up some environmental variables so we can easily configure out script:

: ${SLACK_COOKIE:=}
: ${SLACK_TOKEN:=}
: ${SLACK_DOMAIN:=}

Slack only supports png, gif and jpg image formats, so lets see what we can find from our present working directory:

files=$(find . -iname \*.gif -o -iname \*.png -o -iname \*.png -maxdepth 1)
[[ "${files}" == "" ]] && exit 0

Exit if we can’t find any results. No need to continue if we haven’t a thing to upload.

Next, we just iterate through our findings using read and some more Bash parameter expansion to determine what will be the new emoji’s reference name:

echo "${files}" | while read -r path; do 
  name=$(basename "${path%.*}")
done

We can easily parse the response bodies with jq for some error checking and we’re good to go:

if [[ $(echo "${result}" | jq -r ".ok") == false ]]; then
  echo "${result}" | jq -r ".error"
  exit 1
fi

We now have all the information we need to upload each file in bulk. Put it all together and you’ve got a working bulk emoji uploader:

#!/usr/bin/env bash

set -eo pipefail

: ${SLACK_COOKIE:=}
: ${SLACK_TOKEN:=}
: ${SLACK_DOMAIN:=}

files=$(find . -iname \*.gif -o -iname \*.png -o -iname \*.png -maxdepth 1)
[[ "${files}" == "" ]] && exit 1

echo "${files}" | while read -r path; do 
  name=$(basename "${path%.*}")

  result=$(
    curl -s --compressed "https://${SLACK_DOMAIN}.slack.com/api/emoji.add" \
      -H 'content-type: multipart/form-data' \
      -H "cookie: d=${SLACK_COOKIE};" \
      -F "token=${SLACK_TOKEN}" \
      -F "name=${name}" \
      -F mode=data \
      -F "image=@${path}" 
  )

  if [[ $(echo "${result}" | jq -r ".ok") == false ]]; then
    echo "${result}" | jq -r ".error"
    exit 1
  fi

  echo "uploaded :${name}:!"
done

There you go! To test it out yourself, save this as an executable script and make sure your present working directory has some supported images in it. Pass through the your environmental variables when executing and you’re off:

$ SLACK_DOMAIN=*** SLACK_COOKIE=*** SLACK_TOKEN=*** ./upload.sh
uploaded :stonks:!
uploaded :boop:!
uploaded :derp:!
uploaded :booyah:!
uploaded :merp-flakes:!

Nice! 😊

Once Again, Something Better!#

While perfectly functional, there’s not a lot of flexibility. No error checking, filtering or confirmation checks. If you’re looking for something a bit more fleshed out, look no further!

The Final Piece …#

So far, we’ve gone over how to download and upload large sets of emoji, but what if you want to nuke them from orbit? The final article in this series will cover how to bulk delete while using advanced filters to pin-point specific sets of emoji you wish to remove.

Hope you’ve learned something useful!

But, when it comes time to part ways it’s only understandable to want to gather your stuff for the next place to start the process again. Over the years one can collect and curate quite a nice hoard of stupid images.

I’ve been using Slack prolifically for the past few years. Unfortunately, there is no simple way of exporting your precious collection outside of the old “right-click and save” method. This is effective for smaller collections of a couple dozen or so, but becomes impractical when there are hundreds or even thousands.

And, of course, I’m not alone. There have been a non-zero number of times when someone left the company only to hit me up later asking for an emoji archive.

Yes, this really happens and something must be done about it!

What follows is an unnecessarily long dive into a little bit of reverse engineering one of Slack’s API endpoints, subverting it for our own use and writing a stupid little tool to help ourselves ( and others ) automate this process for the future.

Some Investigative Work#

There just has to be a straightforward way we can mostly automate this. So, I did some cursory digging around the Chrome Developer Tools console and found this endpoint in the “Network” tab:

https://edgeapi.slack.com/cache/T0XXXX/emojis/list?fp=97

The T0XXXX will be your workspace, or team, id. Make note of it, you’ll need it later.

If you intercept one of these requests and take a peak at the “Response” tab in the “Network” panel, you’ll see a JSON response with a structure similar to:

{
	"ok": true,
	"next_marker": "a-custom-emoji",
	"results": [
		{
			"name": "another-custom-emoji",
			"value": "https://emoji.slack-edge.com/T0XXXX/another-custom-emoji/xxxxxxx.png"
		}
		... more emojis ...
	]
}

• ok tells you whether the request was successful or not. If it returns false there will be an additional field named error that should clue you into what went wrong.
• next_marker tells you where in the list of custom emoji the next page of results should start. We use this as the value of marker in subsequent request payloads ( see below ). This is effectively how you page through large lists of emoji.
• results should be obvious, but it contains an array of objects with name and value fields.
  • name is the alias of the emoji.
  • value is the direct CDN URL to the emoji.

You may see other fields, but next_marker and value are the only fields we really care about.

Next, If you take a look at the “Payload” tab in the “Network” panel for the same request, you’ll see something similar to:

{
	"token": "xoxc-xxxx-xxxx-xxxx",
	"count": 100,
	"marker": "some-other-emoji"
}

• token is the API request token the Slack client uses to make the emoji list request on your behalf.
• count is the amount of emojis to return as a page.
• marker tells the API where in the emoji list to start paging.

The final piece of the puzzle is getting the value of your session cookie. You can grab this by clicking the “Request Headers” section under the “Headers” tab. You should see a header labeled cookie.

There’s going to be a fairly large block of text to parse through, but you’re looking for the cookie named d. Save everything between the d= and the closing ;. You’re going to need this to authenticate with the Slack API itself.

Testing Your Findings#

We now understand the structure of our payload. We also have the endpoint to hit as well as our tokens and session cookie to authenticate. Piecing it all together, you can now use cURL to hit the API directly from your terminal:

$ curl --silent https://edgeapi.slack.com/cache/<team>/emojis/list?fp=97
  -H 'cookie: d=<cookie>;' \
  --data-raw '{"token":"<token>","count":10}'

Replace the following:

• <team> is your workspace, or team, id.
• <cookie> is your session cookie value.
• <token> is your request token.

If everything worked out, you should have successful API response like the one above. If the response has root field named marker_next, make note of the associated value to help page through subsequent results. Keep doing this in a loop until your response no longer returns a marker_next field.

Once that happens, you’re at the end of the list.

With a little bit of help from jq to parse and filter the JSON responses, we can grab a simple list of URLs.

$ <previous_curl_command> | jq -r '.results[].value'
https://emoji.slack-edge.com/T0XXXX/no/060fca9aa2581a93.png
https://emoji.slack-edge.com/T0XXXX/nods/c7fe54342ab5b8b6.gif
https://emoji.slack-edge.com/T0XXXX/nods-back/fbd1b5384cdd0905.gif
https://emoji.slack-edge.com/T0XXXX/nomnomnom/612cfc74c785010d.gif
https://emoji.slack-edge.com/T0XXXX/octocat/627964d7c9.png
https://emoji.slack-edge.com/T0XXXX/ohyou/6a352df984d9c076.gif
https://emoji.slack-edge.com/T0XXXX/one-sec-cooking/e749627cb088859d.png
https://emoji.slack-edge.com/T0XXXX/oof/db94710445cbb206.png
https://emoji.slack-edge.com/T0XXXX/petrol/35ed3db238f795fc.png
https://emoji.slack-edge.com/T0XXXX/piggy/b7762ee8cd.png

We now understand how to collect everything we need, so let’s automate this a bit more.

The Implementation#

We’re going to write a little script that’ll download all the custom emojis from the target Slack workspace. It’s going to perform the following steps:

1. Define a starting JSON payload.
2. Create a run loop.
3. With each loop we make a request using the API endpoint.
4. Do some error checking.
5. Iterate through the results and save them to disk.
6. If the results contain a next_marker update the JSON payload with a marker field and continue the run loop.
7. Keep going until the last request no longer provides a next_marker.

First, we want to define some environmental variables so we can easily configure the script:

: ${SLACK_WORKSPACE_ID:=}
: ${SLACK_COOKIE:=}
: ${SLACK_TOKEN:=}

Now we define our JSON payload object. We use jq here to keep things easy. Not only is it great for parsing and filtering JSON in Bash, it also makes building and manipulating objects simple:

payload=$(
  jq -nc \
    --arg     token "${SLACK_TOKEN}" \
    --argjson count "100" \
    '{
      token: $token, 
      count: $count
    }'
)

Next, is the run loop where we make our API requests. This passes through the payload object we just defined while also targeting our workspace:

while true; do
  result=$(
    curl -s --compressed "https://edgeapi.slack.com/cache/${SLACK_WORKSPACE_ID}/emojis/list?fp=97" \
      -H "cookie: d=${SLACK_COOKIE};" \
      --data-raw "${payload}" 
  )
done

We obviously want to do some simple error checking so we know what we did wrong if the requests fail. Grab the error, spit it out and terminate the script.

if [[ $(echo "${result}" | jq -r ".ok") == false ]]; then
  echo "${result}" | jq -r ".error"
  exit 1
fi

We now start iterating through any of the results we got from the initial request. We really only care about the value field as the URL gives us all the pieces we need to give our emojis a human-readable filename.

Here we treat the resulting URL as a standard file path and use dirname paired with basename to get the name of the parent directory of the file. The parent directory holds the actual name of the emoji, while the file itself is only a random hash. We use Bash’s parameter expansion feature to construct our filename.

Finally, we download the remote file to the current directory:

echo "${result}"  | jq -r '.results[].value' | while read -r url; do 
name=$(basename $(dirname "${url}"))
curl -s -o "${name}.${url##*.}" "${url}"
done

We need to check if there are more results to page through, so let’s look for the marker_next field. If we can’t find one, then mission accomplished; all done. However, if we do get a value, we take the previous payload we constructed and add the marker field.

The next loop will pick this change up and pass it on through to the API request telling Slack to give us the next page of precious emojis.

marker=$(echo "${result}" | jq -r ".next_marker")
[[ "${marker}" == "null" ]] && exit 0

payload=$(echo "${payload}" | jq --arg marker "${marker}" '. + { marker: $marker }')

The Finished Product#

#!/usr/bin/env bash

set -eo pipefail

: ${SLACK_WORKSPACE_ID:=}
: ${SLACK_COOKIE:=}
: ${SLACK_TOKEN:=}

payload=$(
  jq -nc \
    --arg     token "${SLACK_TOKEN}" \
    --argjson count "100" \
    '{
      token: $token, 
      count: $count
    }'
)

while true; do
  result=$(
    curl -s --compressed "https://edgeapi.slack.com/cache/${SLACK_WORKSPACE_ID}/emojis/list?fp=97" \
      -H "cookie: d=${SLACK_COOKIE};" \
      --data-raw "${payload}" 
  )

  if [[ $(echo "${result}" | jq -r ".ok") == false ]]; then
    echo "${result}" | jq -r ".error"
    exit 1
  fi

  echo "${result}"  | jq -r '.results[].value' | while read -r url; do 
    name=$(basename $(dirname "${url}"))
    curl -s -o "${name}.${url##*.}" "${url}"
  done 

  marker=$(echo "${result}" | jq -r ".next_marker")
  [[ "${marker}" == "null" ]] && exit 0

  payload=$(echo "${payload}" | jq --arg marker "${marker}" '. + { marker: $marker }')
done

This is pretty much it. A barebones script that downloads all findings to its present working directory. To test it out yourself, make the script executable, define the environmental variables and fire it off like so:

$ SLACK_WORKSPACE_ID=*** SLACK_COOKIE=*** SLACK_TOKEN=*** ./fetch.sh

An Even Better Version!#

I know I just made you read through all this, but you also could’ve just downloaded the tool from here. It’s got heaps more bells and whistles ( if you’re into that sort of thing):

The Bitter Irony#

In typical bored engineer fashion, I’ve spent more time writing this article, documenting and publishing the code than I ever would have personally just “handraulically” doing this from time to time.

Hopefully, you’ve learned something new! 🤞

Bonus Relevant XKCD Image#

source: xkcd: automation

But, good or bad, far too many of them will pass over your resume for the wildest reasons. We’re not here to go over all of them. No, today we’re going to address the somewhat recent trend of glancing at your public Github profile page and grading you based on how many green squares show up in your participation graph.

What’s your point?#

Like basing your performance on LoC, using “commits made” as a recruiting metric is beyond useless. There can be any number of reasons why your graph isn’t as populated as they’d like:

1. You work on private repositories and you haven’t configured your profile to show anonymised private contributions.
2. You have an alternate account from your personal one to keep work and private life separate. Or, maybe it’s just company policy.
3. You’ve signed an NDA and can’t publish your contributions publicly.
4. You set personal boundaries for your work-life balance and don’t care to stare at glowing rectangles in your free time.
5. Or, maybe you’ve just had a slow year? 🤷

Personally, I have thousands of commits that will never see the light of day for some of the ☝️ points. If I’m out job hunting, I would hate to be passed over because I don’t tick some mundane box on an arbitrary checklist.

What to do?#

Well, we light up that graph like it’s a switch board, of course. Make this information utterly useless. What do they care if your profile says you’ve had 2,345 commits in the last week? Maybe you did. If it’s 🟩 , they cross it off their checklist and move on to something else.

Ok, show me.#

Crack your knuckles and get ready to do some coding. For this little project, we’ll be doing some simple Bash scripting. I’m currently working on MacOS, but this code should be fairly portable on other Linux-based distributions.

You will, naturally, need a Github account. Let’s start there.

Configure Your Participation Graph#

Did you know you can configure your participation graph to display anonymised private commit statistics? Ensure that setting is ✅ and move on to the next step.

New Private Repository#

Let’s create a new empty repository entitled graph-participation and make sure it’s private.

Checkout Local Working Copy#

Follow the post-setup instructions Github gives you to check out a local working copy of your new 🔒 repository!

echo "Let's goooooooo!" >> README.md
git init
git add README.md
git commit -m "The first and final legitimate commit in this repository ..."
git branch -M main
git remote add origin git@github.com:<account>/github-participation.git
git push -u origin main

Of course, replace <account> with the name of your Github account.

Writing Some Code#

Here comes the best part! What, you didn’t think you’d walk away from this article without writing a bit of code, did you?

This script is super-simple. There’s no need for elegance here as we’re probably only ever going to run this code once or twice. I’ll show you the full snippet and then walk you through the details.

start=2021-05-01
stop=2021-05-09
max=20
while [[ "${start}" != "${stop}" ]]; do 
  start=$(gdate -I -d "${start} + 1 day")
  for ((run=1; run <= $((RANDOM % max + 1)); run++)); do
    openssl rand -hex 32 > touch.txt
    git add -A
    export GIT_AUTHOR_DATE="${start}T00:01"
    export GIT_COMMITTER_DATE="${GIT_AUTHOR_DATE}"
    git commit -m "Touch #${run} for ${GIT_AUTHOR_DATE}"
  done
done

First things first, we only need to know three things to get started:

1. start: The day of our first “commit” in the format of YYYY-MM-DD.
2. stop: The day of our last “commit” in the format of YYYY-MM-DD.
3. max: We don’t want a single commit per day. Otherwise, the graph will be all one colour and not at all random enough to look legitimate. So, to account for this, we randomly create between 1 and 20 commits per day.

We start a while loop that keeps going until our start and stop strings are no longer different. With each iteration, we add 1 day to our start variable. Eventually, this date will increment until both start and stop strings are equal, thus ending the loop.

For each day iteration, we create another loop that randomly-iterates anywhere between 1 and max times. Within each nested iteration we replace the content of the file named touch.txt with a random value making it a unique commit.

We then stage this change and assign a date value to both the GIT_AUTHOR_DATE and GIT_COMMITTER_DATE environmental variables. These are what allow you to easily back date a commit.

Finally, we commit this unique change locally. This keeps happening for every day until the script stops.

Write this to an executable file, or just paste it into your terminal of choice and hit enter. If you see the following kind of log output, you know you’re doing the right thing:

[main 2ebb5b9] Touch #1 2021-05-07T00:01
 1 file changed, 1 insertion(+), 1 deletion(-)
[main e104b0a] Touch #2 2021-05-07T00:01
 1 file changed, 1 insertion(+), 1 deletion(-)
[main 873fa39] Touch #3 2021-05-07T00:01
 1 file changed, 1 insertion(+), 1 deletion(-)
[main 9def564] Touch #4 2021-05-07T00:01
 1 file changed, 1 insertion(+), 1 deletion(-)
[main 1b83187] Touch #1 2021-05-08T00:01
 1 file changed, 1 insertion(+), 1 deletion(-)
[main b9f00db] Touch #2 2021-05-08T00:01
 1 file changed, 1 insertion(+), 1 deletion(-)

Now, you simply have to push all your changes up to your repository on Github!

git push origin main

All you now have to do is sit back and wait for everything to update.

We’ve gone from barren landscape:

To a verdant garden paradise:

And that’s it! 🎉

If you’ve made a mistake, simply delete your private repository and try again with a fresh one.

Any final thoughts?#

Job hunting is hard enough. Especially with more and more companies becoming increasingly risk-averse in their interviewing and recruiting processes. It’s my hope that this little bit of harmless mischief will make things a bit easier for you!