---

6.2.4.11 Dimensionality changing operators ¶

Through these operators you can change the dimensions of the output through certain statistics on the dimensions that should be removed. For example, let’s assume you have a 3D data cube that has 300 by 300 pixels in the RA and Dec dimensions (first two dimensions), and 3600 slices along the wavelength (third dimension), so the whole cube is \(300\times300\times3600\) voxels (volume elements). To create a narrow-band image that only contains 100 slices around a certain wavelength, you can crop that section (using Crop), giving you a \(300\times300\times100\) cube. You can now use the collapse-sum operator below to “collapse” all the 100 slices into one 2D image that has \(300\times300\) pixels. Every pixel in this 2D image will have the flux of the sum of the 100 slices.

to-1d

Convert the input operand into a 1D array; irrespective of the number of dimensions it has. This operator only takes a single operand (the input array) and just updates the metadata. Therefore it does not change the layout of the array contents in memory and is very fast.

If no further operation is requested on the 1D array, recall that Arithmetic will write a 1D array as a table column by default. In case you want the output to be saved as a 1D image, or to see it on the standard output, please use the --onedasimage or --onedonstdout options respectively (see Invoking Arithmetic).

This operator is useful in scenarios where after some operations on a 2D image or 3D cube, the dimensionality is no longer relevant for you and you just care about the values. In the example below, we will first make a simple 2D image from a plain-text file, then convert it to a 1D array:

## Contents of 'a.txt' to start with.
$ cat a.txt
# Image 1: DEMO [counts, uint8] An example image
1 2 3
4 5 6
7 8 9

## Convert the text image into a FITS image.
$ astconvertt a.txt -o a.fits

## Convert it into a table column (1D):
$ astarithmetic a.fits to-1d -o table.fits

## Convert it into a 1D image:
$ astarithmetic a.fits to-1d -o table.fits --onedasimage

A more real-world example would be the following: assume you want to “flatten” two images into a single 1D array (as commonly done in convolutional neural networks, or CNNs¹⁶⁰). First, we show the contents of a new \(2\times2\) image in plain-text image, then convert it to a 2D FITS image (b.fits). We will then use arithmetic to make both a.fits (from the example above) and b.fits into a 1D array and stitch them together into a single 1D image with one call to Arithmetic. For a description of the stitch operator, see below (same section).

## Contents of 'b.txt':
$ cat b.txt
# Image 1: DEMO [counts, uint8] An example image
10 11
12 13

## Convert the text image into a FITS image.
$ astconvertt b.txt -o b.fits

# Flatten the two images into a single 1D image:
$ astarithmetic a.fits to-1d b.fits to-1d 2 1 stitch -g1 \
                --onedonstdout --quiet
1
2
3
4
5
6
7
8
9
10
11
12
13

stitch

Stitch (connect) any number of given images together along the given dimension. The output has the same number of dimensions as the input, but the number of pixels along the requested dimension will be different from the inputs. The stitch operator takes at least three operands:

• The first popped operand (placed just before stitch) is the direction (dimension) that the images should be stitched along. The first FITS dimension is along the horizontal, therefore a value of 1 will stitch them horizontally. Similarly, giving a value of 2 will result in a vertical stitch.
• The second popped operand is the number of images that should be stitched.
• Depending on the value given to the second popped operand, stitch will pop the given number of datasets from the stack and stitch them along the given dimension. The popped images have to have the same number of pixels along the other dimension. The order of the stitching is defined by how they are placed in the command-line, not how they are popped (after being popped, they are placed in a list in the same order).

For example, in the commands below, we will first crop out fixed sized regions of \(100\times300\) pixels of a larger image (large.fits) first. In the first call of Arithmetic below, we will stitch the bottom set of crops together along the first (horizontal) axis. In the second Arithmetic call, we will stitch all 6 along both dimensions.

## Crop the fixed-size regions of a larger image ('-O' is the
## short form of the '--mode' option).
$ astcrop large.fits -Oimg --section=1:100,1:300     -oa.fits
$ astcrop large.fits -Oimg --section=101:200,1:300   -ob.fits
$ astcrop large.fits -Oimg --section=201:300,1:300   -oc.fits
$ astcrop large.fits -Oimg --section=1:100,301:600   -od.fits
$ astcrop large.fits -Oimg --section=101:200,301:600 -oe.fits
$ astcrop large.fits -Oimg --section=201:300,301:600 -of.fits

## Stitch the bottom three crops into one image.
$ astarithmetic a.fits b.fits c.fits 3 1 stitch -obottom.fits

# Stitch all the 6 crops along both dimensions
$ astarithmetic a.fits b.fits c.fits 3 1 stitch \
                d.fits e.fits f.fits 3 1 stitch \
                2 2 stitch -g1 -oall.fits

The start of the last command is like the one before it (stitching the bottom three crops along the first FITS dimension, producing a \(300\times300\) image). Later in the same command, we then stitch the top three crops horizontally (again, into a \(300\times300\) image) This leaves the the two \(300\times300\) images on the stack (see Reverse polish notation). We finally stitch those two along the second (vertical) dimension. This operator is therefore useful in scenarios like placing the CCD amplifiers into one image.

trim

Trim all blank elements from the outer edges of the input operand (it only takes a single operand). For example see the commands below using Table’s Column arithmetic:

$  cat table.txt
nan
nan
nan
3
4
nan
5
6
nan

$ asttable table.txt -Y -c'arith $1 trim'
3.000000
4.000000
nan
5.000000
6.000000

Similarly, on 2D images or 3D cubes, all outer rows/columns or slices that are fully blank get “trim”ed with this operator. This is therefore a very useful operator for extracting a certain feature within your dataset.

For example, let’s assume that you have set NoiseChisel and Segment on an image to extract all clumps and objects. With the command below on Segment’s output, you will have a smaller image that only contains the sky-subtracted input pixels corresponding to object 263.

$ astarithmetic seg.fits -hINPUT-NO-SKY seg.fits -hOBJECTS \
                263 ne nan where trim --output=obj-263.fits

add-dimension-slow

Build a higher-dimensional dataset from all the input datasets stacked after one another (along the slowest dimension). The first popped operand has to be a single number. It is used by the operator to know how many operands it should pop from the stack (and the size of the output in the new dimension). The rest of the operands must have the same size and numerical data type. This operator currently only works for 2D input operands, please contact us if you want inputs to have different dimensions.

The output’s WCS (which should have a different dimensionality compared to the inputs) can be read from another file with the --wcsfile option. If no file is specified for the WCS, the first dataset’s WCS will be used, you can later add/change the necessary WCS keywords with the FITS keyword modification features of the Fits program (see Fits).

If your datasets do not have the same type, you can use the type transformation operators of Arithmetic that are discussed below. Just beware of overflow if you are transforming to a smaller type, see Numeric data types.

For example, let’s assume you have 3 two-dimensional images a.fits, b.fits and c.fits (each with \(200\times100\) pixels). You can construct a 3D data cube with \(200\times100\times3\) voxels (volume-pixels) using the command below:

$ astarithmetic a.fits b.fits c.fits 3 add-dimension-slow

add-dimension-fast

Similar to add-dimension-slow but along the fastest dimension. This operator currently only works for 1D input operands, please contact us if you want inputs to have different dimensions.

For example, let’s assume you have 3 one-dimensional datasets, each with 100 elements. With this operator, you can construct a \(3\times100\) pixel FITS image that has 3 pixels along the horizontal and 5 pixels along the vertical.

collapse-sum

Collapse the given dataset (second popped operand), by summing all elements along the first popped operand (a dimension in FITS standard: counting from one, from fastest dimension). The returned dataset has one dimension less compared to the input.

The output will have a double-precision floating point type irrespective of the input dataset’s type. Doing the operation in double-precision (64-bit) floating point will help the collapse (summation) be affected less by floating point errors. But afterwards, single-precision floating points are usually enough in real (noisy) datasets. So depending on the type of the input and its nature, it is recommended to use one of the type conversion operators on the returned dataset.

If any WCS is present, the returned dataset will also lack the respective dimension in its WCS matrix. Therefore, when the WCS is important for later processing, be sure that the input is aligned with the respective axes: all non-diagonal elements in the WCS matrix are zero.

One common application of this operator is the creation of pseudo broad-band or narrow-band 2D images from 3D data cubes. For example, integral field unit (IFU) data products that have two spatial dimensions (first two FITS dimensions) and one spectral dimension (third FITS dimension). The command below will collapse the whole third dimension into a 2D array the size of the first two dimensions, and then convert the output to single-precision floating point (as discussed above).

$ astarithmetic cube.fits 3 collapse-sum float32

collapse-min

collapse-max

collapse-mean

collapse-median

Similar to collapse-sum, but the returned dataset will be the desired statistic along the collapsed dimension, not the sum.

collapse-madclip-fill-mad

collapse-madclip-fill-std

collapse-madclip-fill-mean

collapse-madclip-fill-median

collapse-madclip-fill-number

Collapse the input dataset (fourth popped operand) along the FITS dimension given as the first popped operand by calculating the desired statistic after median absolute deviation (MAD) filled re-clipping. The MAD-clipping parameters (namely, the multiple of sigma and termination criteria) are read as the third and second popped operands respectively.

This is the most robust method to reject outliers; for more on filled re-clipping and its advantages, see Contiguous outliers. For a more general tutorial on rejecting outliers, see Clipping outliers. If you have not done this tutorial yet, we recommend you to take an hour or so and go through that tutorial for optimal understanding and results.

When more than 95\(\%\) of the area of an operand is masked, the full operand will be masked. This is necessary in like this: one of your inputs has many outliers (for example it is much more noisy than the rest or its sky level has not been subtracted properly). Because this operator fills holes between outlying pixels, most of the area of the input will be masked, but the thin edges (where there are no “holes”) will remain, causing different statistics in those thin edges of that input in your final stack. Through this mask coverage fraction (which is currently hard-coded¹⁶¹), we ensure that such thin edges do not cause artifacts in the final stack.

For example, with the command below, the pixels of the input 2 dimensional image.fits will be collapsed to a single dimension output. The first popped operand is 2, so it will collapse all the pixels that are vertically on top of each other. Such that the output will have the same number of pixels as the horizontal axis of the input. During the collapsing, all pixels that are more than \(3\sigma\) (third popped operand) are rejected, and the clipping will continue until the standard deviation changes less than \(0.2\) between clips. Finally the counter operator is used to have a two-column table with the first one being a simple counter starting from one (see Size and position operators).

$ astarithmetic image.fits 3 0.2 2 collapse-sigclip-mean \
                counter --output=collapsed-vertical.fits

Printing output of collapse in plain-text: the default datatype of collapse-sigclip-mean is 32-bit floating point. This is sufficient for any observed astronomical data. However, if you request a plain-text output, or decide to print/view the output as plain-text on the standard output, the full set of decimals may not be printed in some situations. This can lead to apparently discrete values in the output of this operator when viewed in plain-text! The FITS format is always superior (since it stores the value in binary, therefore not having the problem above). But if you are forced to save the output in plain-text, use the float64 operator after this to change the type to 64-bit floating point (which will print more decimals).

collapse-madclip-mad

collapse-madclip-std

collapse-madclip-mean

collapse-madclip-median

collapse-madclip-number

Collapse the input dataset (fourth popped operand) along the FITS dimension given as the first popped operand by calculating the desired statistic after median absolute deviation (MAD) clipping. This operator is called similarly to the collapse-madclip-fill-* operators, see the description there for more.

collapse-sigclip-fill-mad

collapse-sigclip-fill-std

collapse-sigclip-fill-mean

collapse-sigclip-fill-median

collapse-sigclip-fill-number

Collapse the input dataset (fourth popped operand) along the FITS dimension given as the first popped operand by calculating the desired statistic after filled \(\sigma\) re-clipping. This operator is called similarly to the collapse-madclip-fill-* operators, see the description there for more.

collapse-sigclip-mad

collapse-sigclip-std

collapse-sigclip-mean

collapse-sigclip-median

collapse-sigclip-number

Collapse the input dataset (fourth popped operand) along the FITS dimension given as the first popped operand by calculating the desired statistic after \(\sigma\)-clipping. This operator is called similarly to the collapse-madclip-fill-* operators, see the description there for more.