Data Analytics and AI
Forward and Backward Propagation
Yizhe Zhang · May 10, 2021

Forward propagation for layer l

Procedure

$$\begin{align} z^{[l]}&=w^{[l]}\cdot a^{[l-1]} +b^{[l]}\\a^{[l]}&=g^{[l]}(z^{[l]})\end{align}$$

Backward propagation for layer l

Procedure

$$\begin{align} dz^{[l]} &=da^{[l]}\times g^{[l]'}(z^{[l]})\\dw^{[l]}&=dz^{[l]}\cdot a^{[l-1]}\\db^{[l]}&=dz^{[l]}\\da^{[l-1]}&=W^{[l]^T}\cdot dz^{[l]} \end{align}$$

For backward propagation, you can use the below equation to calculator the $da$ when you meet the Classification problem.

$$da^{[L]}=-\frac{y}{a}+\frac{1-y}{1-a}$$

If you meet other NNs problem, you can get the $da^{[l]}$ by derivation of Loss Function ($L(\hat y, y)$)

$$dZ^{[L]}=A^{[L]}-Y$$