Thoughts on Tensors in DL

Tensor Relationships

1. deep learning tensors ≠ math / physics tensors

• In deep learning, tensors are used as multidimensional arrays for data handling, unlike in math and physics, where they describe relationships between vectors and scalars.

Tensor Concepts:

Rank: how many indecies are need to refere to a specific data point within the tensor. (number of axes / dimensions) Axes: specific dimension of a tensor. Length of Axes: how many indecies are in this dimension, the elements of the last Axes are always numbers the others contain an n-dimensional array. Shape: Length of each Axis

1>>> import torch
2
3>>> dd = [
4        [1, 2, 3],
5        [4, 5, 6],
6        [7, 8, 9]
7    ]
8>>> t = torch.tensor(dd)
9
10>>> print(t)
11tensor([[1, 2, 3],
12        [4, 5, 6],
13        [7, 8, 9]])
14
15>>> print(type(t))
16<class 'torch.Tensor'>
17
18>>> print(t.shape)
19torch.Size([3, 3]) # pytorch size = tensor shape
20>>> print(t.shape)
21

Shape incodes all relevent information

Tensor Manipulation Functions

Reshaping:

• changes shape but not underlying data points
• reshapes must have enough positions so that every element fits

1>>> reshaped_t = t.reshape(1,9)
2>>> print(reshaped_t)
3    tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9]])
4
5>>> print(reshaped_t.shape)
6    torch.Size([1, 9])
7

Squeeze:

• squeeze removes dimensions of size 1 from a tensor.
• used to eliminate unnecessary dimensions that do not contribute to the data structure's complexity.

1>>> t = torch.tensor([[[1, 2, 3], [4, 5, 6]]])
2>>> print(t.shape)
3    torch.Size([1, 2, 3])
4
5>>> squeezed_t = t.squeeze()
6>>> print(squeezed_t.shape)
7    torch.Size([2, 3])
8

Flatten:

• flatten converts a tensor into a one-dimensional tensor, effectively collapsing all its dimensions.
• useful for preparing data for input into fully connected layers in neural networks.

1>>> t = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
2>>> flattened_t = t.flatten()
3>>> print(flattened_t)
4    tensor([1, 2, 3, 4, 5, 6, 7, 8, 9])
5>>> print(flattened_t.shape)
6    torch.Size([9])
7

Low-Level workings

Multidimensional arrays aka. tensors are reshaped and stored as one dimensional arrays in RAM.

The position of each data point is calculated using strides, which determine how many steps are needed to move along each axis.

The position of any element is calculated as:

$\text{position} = \sum_{i=0}^{n-1} \text{index}_i \times \text{stride}_i$ i.e. $\text{position} = \text{row} \times \text{stride for rows} + \text{column} \times \text{stride for columns}$

For example, a grayscale image = (100,100) array, to find element (5,3): $\text{position} = 5 \times 100 + 3 \times 1 = 503$

For example, an RGB image = (3, 100,100) array, to find element (2,6,4): $\text{position} = 2 \times (100 \times 100) + 6 \times 100 + 4 \times 1 = 10604$