2.1.19 Reddest clumps, cutouts and parallelization ¶

As a final step, let’s go back to the original clumps-based color measurement we generated in Working with catalogs (estimating colors). We will find the objects with the strongest color and make a cutout to inspect them visually and finally, we will see how they are located on the image. With the command below, we will select the reddest objects (those with a color larger than 1.5):

$ asttable cat/mags-with-color.fits --range=F105W-F160W,1.5,inf

You can see how many they are by piping it to wc -l:

$ asttable cat/mags-with-color.fits --range=F105W-F160W,1.5,inf | wc -l

Let’s crop the F160W image around each of these objects, but we first need a unique identifier for them. We will define this identifier using the object and clump labels (with an underscore between them) and feed the output of the command above to AWK to generate a catalog. Note that since we are making a plain text table, we will define the necessary (for the string-type first column) metadata manually (see Gnuastro text table format).

$ echo "# Column 1: ID [name, str10] Object ID" > cat/reddest.txt
$ asttable cat/mags-with-color.fits --range=F105W-F160W,1.5,inf \
           | awk '{printf("%d_%-10d %f %f\n", $1, $2, $3, $4)}' \
           >> cat/reddest.txt

Let’s see how these objects are positioned over the dataset. DS9 has the “Region”s concept for this purpose. And you build such regions easily from a table using Gnuastro’s astscript-ds9-region installed script, using the command below:

$ astscript-ds9-region cat/reddest.txt -c2,3 --mode=wcs \
           --command="ds9 flat-ir/xdf-f160w.fits -zscale"

We can now feed cat/reddest.txt into Gnuastro’s Crop program to get separate postage stamps for each object. To keep things clean, we will make a directory called crop-red and ask Crop to save the crops in this directory. We will also add a -f160w.fits suffix to the crops (to remind us which filter they came from). The width of the crops will be 15 arc-seconds (or 15/3600 degrees, which is the units of the WCS).

$ mkdir crop-red
$ astcrop flat-ir/xdf-f160w.fits --mode=wcs --namecol=ID \
          --catalog=cat/reddest.txt --width=15/3600,15/3600  \
          --suffix=-f160w.fits --output=crop-red

Like the MakeProfiles command in Aperture photometry, if you look at the order of the crops, you will notice that the crops are not made in order! This is because each crop is independent of the rest, therefore crops are done in parallel, and parallel operations are asynchronous. So the order can differ in each run, but the final output is the same! In the command above, you can change f160w to f105w to make the crops in both filters. You can see all the cropped FITS files in the crop-red directory with this command:

$ astscript-fits-view crop-red/*.fits

To view the crops more easily (not having to open ds9 for each image), you can convert the FITS crops into the JPEG format with a shell loop like below.

$ cd crop-red
$ for f in *.fits; do \
    astconvertt $f --fluxlow=-0.001 --fluxhigh=0.005 --invert -ojpg; \
  done
$ cd ..
$ ls crop-red/

You can now use your general graphic user interface image viewer to flip through the images more easily, or import them into your papers/reports.

The for loop above to convert the images will do the job in series: each file is converted only after the previous one is complete. But like the crops, each JPEG image is independent, so let’s parallelize it. In other words, we want to run more than one instance of the command at any moment. To do that, we will use Make. Make is a very wonderful pipeline management system, and the most common and powerful implementation is GNU Make, which has a complete manual just like this one. We cannot go into the details of Make here, for a hands-on video tutorial, see this video introduction. To do the process above in Make, please copy the contents below into a plain-text file called Makefile. Just replace the __[TAB]__ part at the start of the line with a single ‘TAB’ button on your keyboard.

jpgs=$(subst .fits,.jpg,$(wildcard *.fits))
all: $(jpgs)
$(jpgs): %.jpg: %.fits
__[TAB]__astconvertt $< --fluxlow=-0.001 --fluxhigh=0.005 \
__[TAB]__            --invert -o$ 

Now that the Makefile is ready, you can run Make on 12 threads using the commands below. Feel free to replace the 12 with any number of threads you have on your system (you can find out by running the nproc command on GNU/Linux operating systems):

$ make -j12

Did you notice how much faster this one was? When possible, it is always very helpful to do your analysis in parallel. You can build very complex workflows with Make, for example, see Akhlaghi 2021 so it is worth spending some time to master.