Introduction to Tensors in PyTorch

Tensors are the fundamental building blocks of PyTorch, serving as multi-dimensional arrays that can represent data and model parameters. Understanding how to work with tensors is crucial for anyone looking to excel in deep learning with PyTorch.

Creating Tensors

Let's start by exploring different ways to create tensors:

import torch

# Create a tensor from a list
tensor_1 = torch.tensor([1, 2, 3, 4])

# Create a tensor with specific data type
tensor_2 = torch.tensor([1.0, 2.0, 3.0], dtype=torch.float32)

# Create a tensor with all ones
tensor_3 = torch.ones(3, 3)

# Create a tensor with all zeros
tensor_4 = torch.zeros(2, 4)

# Create a tensor with random values
tensor_5 = torch.rand(3, 3)

Tensor Attributes

Tensors have several important attributes:

# Shape of the tensor
print(tensor_1.shape)  # Output: torch.Size([4])

# Data type of the tensor
print(tensor_2.dtype)  # Output: torch.float32

# Device on which the tensor is stored (CPU or GPU)
print(tensor_3.device)  # Output: cpu

Tensor Operations

PyTorch provides a wide range of operations for manipulating tensors:

Arithmetic Operations

a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])

# Addition
c = a + b  # or torch.add(a, b)

# Subtraction
d = b - a  # or torch.sub(b, a)

# Multiplication (element-wise)
e = a * b  # or torch.mul(a, b)

# Division
f = b / a  # or torch.div(b, a)

Matrix Operations

m1 = torch.tensor([[1, 2], [3, 4]])
m2 = torch.tensor([[5, 6], [7, 8]])

# Matrix multiplication
result = torch.matmul(m1, m2)  # or m1 @ m2

# Transpose
transposed = m1.t()

Reshaping and Resizing Tensors

Changing the shape of tensors is a common operation in deep learning:

original = torch.tensor([1, 2, 3, 4, 5, 6])

# Reshape to 2x3 matrix
reshaped = original.reshape(2, 3)

# View as 3x2 matrix (shares memory with original)
viewed = original.view(3, 2)

# Squeeze removes dimensions of size 1
squeezed = torch.tensor([[[1], [2], [3]]]).squeeze()

# Unsqueeze adds a dimension of size 1
unsqueezed = original.unsqueeze(0)  # Adds dimension at index 0

Indexing and Slicing

Accessing specific elements or subsets of tensors is crucial:

tensor = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

# Get a single element
element = tensor[1, 2]  # Value: 6

# Get a row
row = tensor[1, :]  # Values: [4, 5, 6]

# Get a column
column = tensor[:, 1]  # Values: [2, 5, 8]

# Slicing
slice = tensor[0:2, 1:3]  # Values: [[2, 3], [5, 6]]

Advanced Operations

Broadcasting

PyTorch can automatically broadcast tensors of different shapes during operations:

a = torch.tensor([1, 2, 3])
b = torch.tensor([[1], [2], [3]])

c = a + b  # Broadcasting happens automatically

Gather and Scatter

These operations allow for more complex indexing:

# Gather
source = torch.tensor([[1, 2], [3, 4], [5, 6]])
index = torch.tensor([[0, 0], [1, 0]])
result = torch.gather(source, 0, index)  # Values: [[1, 2], [3, 2]]

# Scatter
destination = torch.zeros(3, 2)
torch.scatter(destination, 0, index, source)

Conclusion

Tensor operations and manipulation are at the heart of PyTorch. By mastering these concepts, you'll be well-equipped to build and optimize complex neural networks. Practice these operations regularly, and you'll find yourself becoming more proficient in PyTorch development.