We used the pipeline in our past posts. However, we did look into what they are and what they do. If you have had experience with Linux (or Bash in general), there is a good chance you know that the pipeline simply passes the output of a certain command to the command to its right.

In this article, we discuss all about pipelines:

Understanding the pipeline

Metaphorically speaking, think of a cmdlet as a machine. The machine accepts raw material as its input (values passed to parameters), which is then processed, and the processed product (output) is sent out of the machine. Think of this product as collecting just outside of the machine, right at the outlet.

Now imagine you got yourself another machine. This machine accepts the product sent out by the aforementioned machine as its raw material to process. To enable automated feeding of the second machine from the first, you bring a pipe and connect the outlet of the first machine into (one of) the inlet(s) of the new machine. Now, you send raw material into the first machine, which processes it and creates something. Instead of the product falling out of the outlet onto the floor, it is directly carried into the next machine through the pipeline, and the product gets further processed by the second machine to give us what we want.

In PowerShell terms, this “product” is the output object. This object is carried through the pipeline into the next cmdlet. Of course, we must ensure that, just like with any machine, the output of the preceding machine is compatible with the succeeding one.

To use the power of PowerShell, we should be able to move objects (not text) from one cmdlet to another, and then to another, and then to another, and so on…

Piping is a very powerful concept. Technically speaking, when you run any command, the output of the command is sent to the host by default, because the console (or host) is the default final destination. That in turn, would mean that every output would be simply text shown on the screen. Pretty much like CMD. That’s not helpful. That’s not utilising the power of PowerShell.

To use the power of PowerShell, we should be able to move objects (not text) from one cmdlet to another, and then to another, and then to another, and so on, until we get what we desire. This is done using the pipeline operator: |. The pipe essentially redirects the output of the first cmdlet to the input of the next cmdlet. (More about output redirection is covered in Understanding Streams in PowerShell). This process can go on as long as the next cmdlet accepts the current output as input.

Let us look at a little demonstration to understand this further:

Get-ChildItem -Path 'D:\Logs'

This gives us output in four columns, namely Mode, LastWriteTime, Length and Name. Let’s imagine that we’re listing out the logs present in a certain log folder. Our manager has asked for the list of log files present at a location, emailed to her.

Copy-paste this data into an email; the twenty thousand entries? A better option would be to use a comma-separated values (CSV) sheet:

Get-ChildItem -Path 'D:\Logs' | Export-Csv -Path 'C:\Users\MyName\Documents\FolderExport.csv'

How did I know the location had 20,000 entries? Count the number of objects output by Get-ChildItem.

Get-ChildItem -Path 'D:\Logs' | Measure-Object

You should see a count of contents in the output. The Measure-Object cmdlet picked up the output of Get-ChildItem -Path 'D:\Logs' and counted the objects in it.

Using the pipeline to select properties

Now we know how the pipeline works. Well, sort of. The concept is still sinking in. To reinforce the understanding, let us use the pipeline to only select a few properties from an output.

Change in scenario: The directory we picked contains files from the last twenty years. There’s an auditor sitting right next to you and your manager and is asking you for a list of files along with the year they were created in. (Recall: CreationTime.) We just need two columns. So we select only the two properties that we need.

Get-ChildItem -Path 'D:\Logs'

The output:

  • Is not in the order we need it in
  • Contains two columns we don’t really need

So, to meet our requirement, we extend the command a little, like so:

Get-ChildItem -Path 'D:\Logs' | Select-Object Name, CreationTime

Remember that the column names should be the same as the ones in the output. So don’t use Last Write Time or Last Modified.

If you remember aliases, there’s a convenience alias for Select-Object, called select:

The cases don’t matter here, so you could also say Select or even lastwritetime for that matter. The reason for the Pascal Case is so that we can easily distinguish one word from another in a concatenated phrase.

Get-ChildItem -Path 'D:\Logs' | select Name, CreationTime

Pipe another command to this and place all this content in a CSV file:

Get-ChildItem -Path 'D:\Logs' | Select-Object Name, CreationTime | Export-Csv -Path 'D:\Logs\Logs.csv'

Filtering the output based on a parameter

PowerShell, like we saw in one of our earlier posts, is a lot like natural language—the language we speak on a day-to-day basis. In natural language, we use pronouns. Pronouns are nothing but a common reference, such as, ‘Ryan stood in the queue at the counter. He checked his wallet to ensure he had enough money for a balcony ticket.’ If we think of it, a pronoun is nothing but reference to the current subject—Ryan, in this case.

PowerShell has a similar concept: the $_ (or $PSItem) automatic variable. It is a virtual container that holds the current object in the pipeline. It can also be thought of as the “pronoun variable”, if you will.

Change in scenario: When the auditor opens the CSV, he realises that it has over 20,000 rows, the modified date starting somewhere in 1997! All that he’s concerned about, though, are those files that were last edited in or after 2015, and he’s not ready to filter the content using Excel.

Get-ChildItem -Path 'D:\Logs' | Select-Object Name, LastWriteTime

From the post about objects, we also learnt to select just the Year component of the LastWriteTime attribute, like so:

(Get-ChildItem).LastWriteTime.Year

Let’s combine these cmdlets to get us a list of files that were modified in or after 2015. But let’s first understand how to create filters. Think about it in natural language. How do we filter content in plain English?

‘List out the names of the places where there’s no rainfall at all.’

“Where”, “which”, “who”, etc. are all filters in English. PowerShell uses only Where-Object for filtration, such as, ‘List out all the files where the “modified date” object is greater than or equal to 2015.’

Add the concept of piping cmdlets and of $_, and you have a nice filter ready.

Get-ChildItem -Path 'D:\Logs' | Select-Object Name, LastWriteTime | Where-Object {$_.LastWriteTime.Year -ge 2015}

In plain English, it would sound something like, ‘Get the child objects of the path specified, select the name and the last-modified of only those objects whose modified year is greater than or equal to 2015.’

Notice how we used $_.LastWriteTime.Year in place of (Get-ChildItem -Path 'D:\Logs').LastWriteTime.Year. $_ simply replaced (Get-ChildItem -Path 'D:\Logs')[$i]. Something like, (TheCurrentObject).LastWriteTime.Year. Also, note that we must not use the entire cmdlet in place of the placeholder while filtering. Ever. Because the output could be of multiple rows, while logically, we filter content one-by-one.

Update: If you are using PowerShell 3 or higher (which is most probably the case in 2018), you do not need $_ to filter based on a single property. You can rewrite the command as:

Get-ChildItem -Path 'D:\Logs' | Where-Object LastWriteTime -ge (Get-Date '1 January 2015')

Correct, the property cannot be exploded further, like LastWriteTime.Year in this case. With PowerShell 3, Where-Object supports the Property parameter, as opposed to it supporting only the FilterScript parameter. (FilterScript is what you write within the curly braces.) This is useful in situations such as:

Get-ChildItem | Where-Object Name -like 'EMP*'

Also, now, $_ is also called $PSItem, to make it less confusing.

Let’s add another pipe (because, well, we can), and export the content to a more friendly CSV:

Get-ChildItem -Path 'D:\Logs' |
 Select-Object Name, LastWriteTime |
  Where-Object {$_.LastWriteTime.Year -ge 2015} |
   Export-Csv -Path 'D:\Logs\CurrentLogs.csv' -NoTypeInformation

Line can be broken at the pipeline; PowerShell will see the entire block above as a single command. Indentation is not mandatory; I use it to aid legibility.

NoTypeInformation: If you opened the CSV file before running this command, you would’ve seen some content in the first row of the first column that was not really relevant to the situation. It was the object type information PowerShell added to the CSV. Adding the NoTypeInformation parameter to Export-Csv tells the cmdlet that we’re not interested in knowing the object type. -NoTypeInformation is no longer necessary in PowerShell Core; it’s the default behaviour.

This is how we get work done using just one-liner commands—without the need to have an entire script. However, this powerful feature can turn into a point of confusion if care is not taken.

How PowerShell outputs content

Like we all know a little too well by now, PowerShell outputs objects. The formatting engine in PowerShell makes this output look like tables. Simple tables, which contain data in simple rows and columns. Like we’ve read about tables, each row is a complete record, and each column looks like an attribute. So when we did a Get-ChildItem, all the content that was output was a PowerShell object, that contained several objects within itself, which in turn contained several objects in themselves and so on.

As I mentioned in The object-oriented model, PowerShell does not output all of the columns all the time.

In cases of outputs of cmdlets such as Get-AdUser -Filter {Name -like '*Doe*'}, the output would be a list instead of a table. But then, what if you wanted the content in the table format?

Formatting the output

You could change the way the object is shown to you. In my opinion, a table is usually more space-efficient, considering the screen real estate. But there are situations when you want to get the items in the list format You could convert the output to a list like so:

Get-ChildItem | Select-Object Mode, LastWriteTime, Length, Name | Format-List

And here’s how you force the output of a cmdlet to be a table:

Get-AdUser -Filter {Name -like '*Doe*'} | Format-Table

If content overflows from columns, add the -Autosize switch, like so:

Get-AdUser -Filter {Name -like '*Doe*'} | Format-Table -Autosize

Nifty? Here’s the catch.

Only in the end

One very important point to remember is that the Format commands rip the objects, and output just the text. So the output of Format-Table or Format-List are not objects, but plain text. It takes away the soul of the output and lets the body remain. So remember that Format commands should be used only in the end. You cannot pipe the content to another non-text-manipulating cmdlet. So you cannot do something like:

Get-AdUser -Filter {Name -like '*Doe*'} | Format-Table | Remove-AdUser

If you would like to inspect the output of, say, Format-Table, use the Get-Member cmdlet.

Get-AdUser -Filter {Name -like '*Doe*'} | Format-Table | Get-Member

Wrapping up

In this article, we tried to understand chaining commands using the pipeline. We leveraged the pipeline to select only those properties that we needed, and also, to filter the output based on a certain criterion.

Remember, that using Select-Object modifies the output object to contain only those properties that you selected. If the cmdlet that follows Select-Object needs a property that you did not select with Select-Object, add that property to Select-Object. For example, the following will not work as expected:

Get-ChildItem | Select-Object Name, LastWriteTime | Where-Object { $PSItem.CreationTime.Year -gt 2015 }

You could handle this in either of the two following ways:

# Adding the necessary object to `Select-Object`:

Get-ChildItem | Select-Object Name, LastWriteTime, CreationTime | Where-Object { $PSItem.CreationTime.Year -gt 2015 }

# Moving `Select-Object` to the end

Get-ChildItem | Where-Object { $PSItem.CreationTime.Year -gt 2015 } | Select-Object Name, LastWriteTime

Your choice should depend on the situation. If you are dealing with, say, three million records, you might want to use the first approach, because the filtration would happen on a smaller object. The running time would see a significant reduction.