Machine Learning & Deep Learning & Reinforcement Learning

Posted on 2026-07-27 Edited on 2026-07-30 In tech Views:

Theory of Machine Learning

0. Reference

Learning Path Reference: Machine Learning Path

Main Story (Parallelism)

Stanford CS229 (Andrew Ng)
[!NOTE]
1. Download / Save the notes of this course, read repeatedly.
2. Solve the “Problem Sets”.
Dive into Deep Learning

Dive into Deep Learning — Dive into Deep Learning 1.0.3 documentation

Combination:

When learn some knowledge points in CS229, find them in the d2l, and implement in PyTorch.

Reference Books:

Machine Learning, ISBN 9787302423287 (西瓜书)
Deep Learning : Adaptive Computation and Machine Learning series, ISBN 9780262035613 (花书)

1. Machine Learning

2. Deep Learning

2.1 Preparation

It is planned to study with Dive into Deep Learning.

Dive into Deep Learning — Dive into Deep Learning 1.0.3 documentation

2.1.1 Install MiniConda

2.1.1.1 Install for Windows

I choose to install manually, so I download the graphical installer:

Download Success | Anaconda

Check the 1st, 3rd item:

- Create shortcuts (supported packages only).
- Register Miniconda3 as the system Python 3.14 Allows other programs, such as VSCode, PyCharm, etc. to automatically detectMiniconda3 as the primary Python 3. 14 on the system.
- Clear the package cache upon completion Recommended. Recovers some disk space without harming functionality.

2.1.1.2 Configuration for GitBash

1	/d/softwares/MiniConda/Scripts/conda init

After initiation, the terminal displays:

1	==> For changes to take effect, close and re-open your current shell. <==

However, when I restart the GitBash, the bash can’t find the command conda, while the PowerShell can find it:

GitBash:

1
2
3

test@□□ѩ□□□□ MINGW64 /e/Computer Science System Building
$ conda
bash: conda: command not found

PowerShell:

1 2	(base) PS E:\Computer Science System Building> conda --version conda 26.5.3

This because the conda init for windows writes some configuration for PowerShell only.

So we need to write the configuration manually:

echo '
  # >>> conda initialize >>>
  if [ -f "/d/softwares/MiniConda/etc/profile.d/conda.sh" ]; then
      . "/d/softwares/MiniConda/etc/profile.d/conda.sh"
  fi
  # <<< conda initialize <<<
  ' >> ~/.bashrc

Then restart the GitBash:

1
2
3

test@□□ѩ□□□□ MINGW64 /e/Computer Science System Building
$ conda --version
conda 26.5.3

2.1.2.3 Initialize a New Env

Accept new terms:

1
2
3

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/msys2

Create new env:

1	conda create --name d2l python=3.9 -y

Switch to new env:

1	conda activate d2l

2.1.2 Install a Deep Learning Framework

Determine to install PyTorch with CUDA version 12.6 (remember to switch the env to d2l) :

1	python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

Check:

1
2
3

$ python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"
True
NVIDIA GeForce RTX 4070 Laptop GPU

2.1.2.1 Ideal Installation of d2l package

Install d2l package:

1	python -m pip install d2l

If there’s an error when you install d2l due to the high version of python (like python 13.x):

$ python -m pip install d2l
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1010)')': /simple/d2l/
Collecting d2l
  Using cached d2l-1.0.3-py3-none-any.whl.metadata (556 bytes)
Collecting jupyter==1.0.0 (from d2l)
  Using cached jupyter-1.0.0-py2.py3-none-any.whl.metadata (995 bytes)
Collecting numpy==1.23.5 (from d2l)
  Using cached numpy-1.23.5.tar.gz (10.7 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "D:\softwares\MiniConda\envs\d2l\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 389, in <module>
          main()
        File "D:\softwares\MiniConda\envs\d2l\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 373, in main
          json_out["return_val"] = hook(**hook_input["kwargs"])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "D:\softwares\MiniConda\envs\d2l\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 137, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "D:\softwares\MiniConda\envs\d2l\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 70, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "D:\softwares\MiniConda\envs\d2l\Lib\importlib\__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 999, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "C:\Users\test\AppData\Local\Temp\pip-build-env-b_gnacsy\overlay\Lib\site-packages\setuptools\__init__.py", line 16, in <module>
          import setuptools.version
        File "C:\Users\test\AppData\Local\Temp\pip-build-env-b_gnacsy\overlay\Lib\site-packages\setuptools\version.py", line 1, in <module>
          import pkg_resources
        File "C:\Users\test\AppData\Local\Temp\pip-build-env-b_gnacsy\overlay\Lib\site-packages\pkg_resources\__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed to build 'numpy' when getting requirements to build wheel

2.1.2.2 Actual Installation

I run those commands to avoid:

1 2	python -m pip install d2l --no-deps python -m pip install jupyter numpy matplotlib scipy pandas

there are some warnings that can be ignored:

Installing collected packages: webencodings, pure-eval, fastjsonschema, widgetsnbextension, websocket-client, webcolors, wcwidth, urllib3, uri-template, tzdata, traitlets, tornado, tinycss2, soupsieve, six, send2trash, rpds-py, rfc3986-validator, pyzmq, pyyaml, pywinpty, python-json-logger, pyparsing, pygments, pycparser, psutil, prometheus-client, platformdirs, parso, pandocfilters, nest-asyncio2, mistune, lark, kiwisolver, jupyterlab_widgets, jupyterlab-pygments, jsonpointer, json5, idna, h11, fqdn, fonttools, executing, defusedxml, decorator, debugpy, cycler, contourpy, comm, colorama, charset_normalizer, certifi, bleach, babel, attrs, async-lru, asttokens, terminado, stack_data, rfc3987-syntax, rfc3339-validator, requests, referencing, python-dateutil, prompt_toolkit, matplotlib-inline, jupyter-core, jedi, ipython-pygments-lexers, httpcore, cffi, beautifulsoup4, anyio, pandas, matplotlib, jupyter-server-terminals, jupyter-client, jupyter-builder, jsonschema-specifications, ipython, httpx, arrow, argon2-cffi-bindings, jsonschema, isoduration, ipywidgets, ipykernel, argon2-cffi, nbformat, jupyter-console, nbclient, jupyter-events, nbconvert, jupyter-server, notebook-shim, jupyterlab-server, jupyter-lsp, jupyterlab, notebook, jupyter
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
d2l 1.0.3 requires scipy==1.10.1, but you have scipy 1.18.0 which is incompatible.
d2l 1.0.3 requires jupyter==1.0.0, but you have jupyter 1.1.1 which is incompatible.
d2l 1.0.3 requires matplotlib==3.7.2, but you have matplotlib 3.11.1 which is incompatible.
d2l 1.0.3 requires matplotlib-inline==0.1.6, but you have matplotlib-inline 0.2.2 which is incompatible.
d2l 1.0.3 requires numpy==1.23.5, but you have numpy 2.5.1 which is incompatible.
d2l 1.0.3 requires pandas==2.0.3, but you have pandas 3.0.5 which is incompatible.
d2l 1.0.3 requires requests==2.31.0, but you have requests 2.34.2 which is incompatible.

2.1.3 Open the Notebook

1
2
3

mkdir d2l-en && cd d2l-en
curl https://d2l.ai/d2l-en-1.0.3.zip -o d2l-en.zip
unzip d2l-en.zip && rm d2l-en.zip

Each time you should activate the env and run :

1
2
3

conda activate d2l
cd pytorch
jupyter notebook

Then, a progress will start on your port: http://localhost:8888 You can an run the code for each section of the book.
Whenever you open a new command line window, you will need to execute conda activate d2l to activate the runtime environment before running the D2L notebooks, or updating your packages (either the deep learning framework or the d2l package). To exit the environment, run conda deactivate.

2.1.4 Notational Conventions

Numerical Objects

x: a scalar
x: a vector
X: a matrix
X: a general tensor
I: the identity matrix (of some given dimension), i.e., a square matrix with 1 on all diagonal entries and 0 on all off-diagonals
x_i, [x]_i: the i^th element of vector x
x_ij, x_i, j, [X]_ij, [X]_i, j: the element of matrix X at row i and column j.

Set Theory

𝒳: a set
ℤ: the set of integers
ℤ⁺: the set of positive integers
ℝ: the set of real numbers
ℝⁿ: the set of n-dimensional vectors of real numbers
ℝ^a × b: The set of matrices of real numbers with a rows and b columns
|𝒳|: cardinality (number of elements) of set 𝒳
𝒜 ∪ ℬ: union of sets 𝒜 and ℬ
𝒜 ∩ ℬ: intersection of sets 𝒜 and ℬ
𝒜 \ ℬ: set subtraction of ℬ from 𝒜 (contains only those elements of 𝒜 that do not belong to ℬ)

Functions and Operators - f(⋅): a function - log (⋅): the natural logarithm (base e) - log₂(⋅): logarithm to base 2 - exp (⋅): the exponential function - 1(⋅): the indicator function; evaluates to 1 if the boolean argument is true, and 0 otherwise - 1_𝒳(z): the set-membership indicator function; evaluates to 1 if the element z belongs to the set 𝒳 and 0 otherwise - (⋅)^⊤: transpose of a vector or a matrix - X⁻¹: inverse of matrix X - ⊙: Hadamard (elementwise) product - [⋅, ⋅]: concatenation - ∥ ⋅ ∥_p: ℓ_p norm - ∥⋅∥: ℓ₂ norm - ⟨x, y⟩: inner (dot) product of vectors x and y - ∑: summation over a collection of elements - ∏: product over a collection of elements - $\stackrel{\text{def}}{=}$: an equality asserted as a definition of the symbol on the left-hand side

Calculus - $\frac{dy}{dx}$: derivative of y with respect to x - $\frac{\partial y}{\partial x}$: partial derivative of y with respect to x - ∇_xy: gradient of y with respect to x - ∫_a^bf(x) dx: definite integral of f from a to b with respect to x - ∫f(x) dx: indefinite integral of f with respect to x

Probability and Information Theory - X: a random variable - P: a probability distribution - X ∼ P: the random variable X follows distribution P - P(X = x): the probability assigned to the event where random variable X takes value x - P(X ∣ Y): the conditional probability distribution of X given Y - p(⋅): a probability density function (PDF) associated with distribution P - E[X]: expectation of a random variable X - X ⟂ Y: random variables X and Y are independent - X ⟂ Y ∣ Z: random variables X and Y are conditionally independent given Z - σ_X: standard deviation of random variable X - Var(X): variance of random variable X, equal to σ_X² - Cov(X, Y): covariance of random variables X and Y - ρ(X, Y): the Pearson correlation coefficient between X and Y, equals $\frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}$ - H(X): entropy of random variable X - D_KL(P∥Q): the KL-divergence (or relative entropy) from distribution Q to distribution P

2.2 Preliminaries

To prepare for your dive into deep learning, we will need a few survival skills:

techniques for storing and manipulating data;
libraries for ingesting and preprocessing data from a variety of sources;
knowledge of the basic linear algebraic operations that we apply to high-dimensional data elements;
just enough calculus to determine which direction to adjust each parameter in order to decrease the loss function;
the ability to automatically compute derivatives so that you can forget much of the calculus you just learned;
some basic fluency in probability, our primary language for reasoning under uncertainty.
some aptitude for finding answers in the official documentation when you get stuck.

2.2.1 Data Manipulation

Generally, there are two important things we need to do with data:

acquire them
process them once they are inside the computer

There is no point in acquiring data without some way to store it, so to start, let’s get our hands dirty with n-dimensional arrays, which we also call tensors.

If you already know the NumPy scientific computing package, this will be a breeze. For all modern deep learning frameworks, the tensor class (ndarray in MXNet, Tensor in PyTorch and TensorFlow) resembles NumPy’s ndarray, with a few killer features added.

First, the tensor class supports automatic differentiation.

Second, it leverages GPUs to accelerate numerical computation, whereas NumPy only runs on CPUs.

These properties make neural networks both easy to code and fast to run.

2.2.1.1 Start with Tensor

A tensor represents a (possibly multidimensional) array of numerical values.

In the one-dimensional case, i.e., when only one axis is needed for the data, a tensor is called a vector. With two axes, a tensor is called a matrix.

With K > 2 axes, we drop the specialized names and just refer to the object as a

$ K^{th}-order$ tensor.

PyTorch provides a variety of functions for creating new tensors prepopulated with values.

For example, by invoking arange(n), we can create a vector of evenly spaced values, starting at 0 (included) and ending at n (not included). By default, the interval size is 1.

Unless otherwise specified, new tensors are stored in main memory and designated for CPU-based computation.

import torch

x = torch.arange(12, dtype=torch.float32)
print(x)

1	tensor([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11.])

Each of these values is called an element of the tensor. The tensor x contains 12 elements. We can inspect the total number of elements in a tensor via its numel method.

1	print("nums of elements is:", x.numel())

1	nums of elements is: 12

We can access a tensor’s shape (the length along each axis) by inspecting its shape attribute. Because we are dealing with a vector here, the shape contains just a single element and is identical to the size.

1	print("shape of x is:", x.shape)

1	shape of x is: torch.Size([12])

We can change the shape of a tensor without altering its size or values, by invoking reshape.

For example, we can transform our vector x whose shape is (12,) to a matrix X with shape (3, 4).

This new tensor retains all elements but reconfigures them into a matrix. Notice that the elements of our vector are laid out one row at a time and thus x[3] == X[0, 3] (3rd element in x equals to x[row 0, column 3]).

print ("reshape x to 3 rows and 4 columns")
reshaped_x = x.reshape(3, 4)
print("reshaped x is:", reshaped_x)
print("x[3] = ", x[3])
print("reshaped_x[0, 3] = ", reshaped_x[0, 3])

1 2	x[3] = tensor(3.) reshaped_x[0, 3] = tensor(3.)