A blue question mark on a pink background

Dark Theme | Category: Code, Educational

Solving Common Bash Problems

Hello Gentlefriends and welcome to a roundup of three common Bash problems and one less common problem that was very challenging but ultimately so useful that I decided to include it. How do I know these problems are common, you may ask? Because of the number of posts on Stack Overflow and Stack Exchange about them, as well as the number of times I have encountered them in my own work. In fact, if I wanted to call myself out, this blog post could be titled “A Reference Guide for Solving the Bash Problems I Keep Running Into”. Here they are:

  1. You need to use a variable in a cURL command, but Bash is using the name of the variable rather than the value.
  2. You’re using an if statement to compare two values and getting a “binary operator expected” error.
  3. You have an array saved in a variable, but when you try to work with it, Bash is treating the contents of the array like one big element.
  4. Translating an array created by jq into an array Bash can use.

The Problem: You need to use a variable in a cURL command

Let’s say you’re writing a script to query an API for data, and one of the filters the API accepts is a date range. You want to be able to pass two dates as arguments when you run your script. In our example code, we’re going to query the Atlassian API for all of the DevOps tickets created in the past week:

If we run our script, though, we’re going to get an error message like this:

That’s because the variable wasn’t expanded – that is, instead of the start and end date we entered when running our script, the cURL command literally contained ${START} and ${END}.

The Solution: Double Quotes

Fortunately, this one has an easy solution! Bash variables are only expanded inside of double quotes, so all you need to do is swap out those single quotes for doubles in your cURL command:

Since we’re using an IDE (VS Code with the Kimbie Dark theme) we can see the difference in color that indicates Bash is now treating our variables as variables. This same approach would also apply if we were using variables directly in the URL portion of our cURL command. Note that we also had to change from double quotes to single quotes around ‘DevOps’ (or we could have escaped the double quotes) so that Bash wouldn’t read them as the end of one string and the beginning of a new one on either side of the word DevOps.

 

The Problem: binary operator expected error

Let’s say you need to compare two lists of email addresses so that you can identify and do something with the emails that appear on only one list (for example, maybe you have a list of approved users, and you need to identify and remove users who gained access to a system without documented approval). You have this very simple script:

When you run it, however, you get:

Note: you get this error three times because you’re looping through once for each of the three email addresses in the EMAILS array, and the script isn’t set up to exit in the event of an error, so it just moves on and gets the same error every loop.

The Solution: [[ ]]

To resolve this error, add another layer of square brackets around your if statement, like this:

Here’s why: [[ ]] is actually its own syntactical element, different from [ ], and as such it has some additional features. You can think of it kind of like the way different HTML5 tags come with different styling by default: per the way the language was written, there are some additional features that are not immediately apparent to the naked eye. The top response to this Stack Overflow question does an excellent job of laying out those different features. Keep in mind that [[ ]] is specific to Bash (and just a select few other shell languages), so if portability is important to the solution you’re writing you would need to take a different approach to array comparison than this example script.

 

The Problem: Your variable is an array, but Bash is treating the contents like one big element

I’m not going to lie – this one gets me every time. Let’s say you have an array. In this case it’s hard-coded for visual ease, but maybe in the real world you created it dynamically and you want to echo it out to make sure every element you expect to be inside actually made it inside. For the purpose of this example, we’re also echoing out the length of the array, in the event that you have a gigantic array and want to test based on length as opposed to individual elements. Our code example and output looks like this:

As you can see, only the first element of the array was echoed, and the length we asked for is applying only to that first element (Garfield, which has 8 characters) rather than the array itself, which has 4 elements. Let’s also check what happens when we try to access Felix (element 2) specifically:

Okay, he is definitely in there. So what gives?

The Solution: [@]

In order to access the array as a whole, you need to add [@] to the end of the variable name inside of the curly braces, like so:

Now we can see that we’re getting the output we expect: the name of each cool cat, and the number of cool cats in our array.

I know what you’re thinking – what the actual f[@]ck is the deal with that? And if you aren’t thinking that, congratulations, you’re way less uptight than I am. This functionality, while it may seem like an unnecessary additional step in the moment, actually gives us more flexibility with how arrays can be handled in Bash, particularly in regards to IFS characters such as spaces, tabs, and newlines. Embrace the [@], memorize the [@], eat sleep and breathe the [@] until it becomes part of your soul. Otherwise you might spend a significant chunk of time troubleshooting a busted array that isn’t actually busted at all, like… this person I know… once did (okay, it was me, and it was more than once).

 

The Problem: translating a jq array into a Bash array

This may beĀ  a niche problem, but between the amount of time I spent working on it and the immense feeling of triumph I felt when I figured it out, I feel very strongly that it deserves a place on this list. If you’re not familiar, jq is to Bash what jelly is to peanut butter: you can enjoy them separately, but they are more powerful and delightful as a team. jq is like a stream editor for processing JSON, so if you have a JSON file and you need to pull out some data and do things with it in Bash, you can use jq. When I first started working with jq, he gave me serious regex vibes and I was like “Ew, get away from me”, but then our mutual friend Bash was like “I know he looks a little crazy, but just trust me, you’ll love him once you get to know him.” I always trust in Bash, and now here we are, the three best friends that anyone could have.

Anyway, let’s say you have a JSON file that contains some user data and you want to put the email addresses in an array so you can work with them in Bash. Here is what our example JSON looks like:

Conventional jq wisdom dictates that you can output an array of email addresses (and then echo the array and the length of the array) like so:

However, when we run our file, we see some unexpected output (and if we try to work with this array, we’ll get some funky behavior, too):

You may already recognize the problem based on what the array output looks like, but in case you’re still scratching your head, here’s what’s wrong: Bash arrays don’t use quotes, and they use spaces – not commas – as the delimiter (aka the character that separates one array element from another). So while that array is something JavaScript would be perfectly happy with, Bash isn’t having it.

The Solution

When I took this problem to my more experienced colleague, he offered a solution using readarray that works on Windows but doesn’t work on Mac. So before you Windows people scream that readarray is way prettier than what I’m about to show you – I know. I know that my Mac-and-Windows-friendly solution isn’t as sleek and sexy, but it’s effective and portable, which I chose to prioritize. Here it is:

Use the -r flag to strip the styling off the jq input, take out those square brackets, and then jam that puppy inside a Bash array:

Now when we run our script that echoes the EMAILS array and the length, we get:

You can also tidy it up to be one line:

Why does this work? We didn’t even use our [@] trick! It works because the jq statement produces a piece of data in a form that Bash expects to see between parentheses.

 

I hope this post saves you some time and energy the next time you encounter these common Bash problems in your own scripts. Bash is a beautiful, powerful, quirky language and I truly love learning more about it and sharing those lessons with you. Eventually I will crawl out of my comfort zone and share more educational content on other languages, so if Bash isn’t your thing I hope you will stay tuned and be patient with me while I argue with the voices in my head that tell me I have nothing valuable to contribute because I haven’t been doing this for 20 years. Happy scripting, Gentlefriends!

</ XOXO>

Enjoy my content and want to show your appreciation? You can share this post, pay it forward by teaching someone else, or buy me a coffee!

[Photo credit: Towfiqu barbhuiya via Unsplash]

Back to the Blog