Correct Answer: Option 1 only

Let's understand this step by step. The key concept here is scaling invariance of the ReLU activation function.

Step 1: Understand the original network

We are given three hidden neurons:

$h_{11}=a(b_{11}+w_{11}x)$$h_{11}=a(b_{11}+w_{11}x)$

$h_{12}=a(b_{12}+w_{12}x)$$h_{12}=a(b_{12}+w_{12}x)$

$h_{13}=a(b_{13}+w_{13}x)$$h_{13}=a(b_{13}+w_{13}x)$

The final output is:

$y=b_{21}+w_{21}{h}_{11}+w_{22}{h}_{12}+w_{23}{h}_{13}$$y=b_{21}+w_{21}h_{11}+w_{22}h_{12}+w_{23}h_{13}$

Step 2: Apply the transformation

The question says:

• Multiply $b_{11}$$b_{11}$ by $\alpha$$\alpha$.

• Multiply $w_{11}$$w_{11}$ by $\alpha$$\alpha$.

• Divide $w_{21}$$w_{21}$ by $\alpha$$\alpha$.

• Everything else remains unchanged.

Therefore:

$b_{11}^{\prime }=\alpha b_{11}$$b_{11}'=\alpha b_{11}$

$w_{11}^{\prime }=\alpha w_{11}$$w_{11}'=\alpha w_{11}$

$w_{21}^{\prime }=\frac{w_{21}}{\alpha }$$w_{21}'=\frac{w_{21}}{\alpha}$

Now, the new hidden neuron output becomes:

$h_{11}^{\prime }=a(\alpha b_{11}+\alpha w_{11}x)$$h_{11}'=a(\alpha b_{11}+\alpha w_{11}x)$

Taking $\alpha$$\alpha$ common:

$\boxed{{\displaystyle h_{11}^{\prime }=a(\alpha (b_{11}+w_{11}x))}}$$\boxed{h_{11}'=a(\alpha(b_{11}+w_{11}x))}$

This is the important equation.

Step 3: Check each option

Option 1: ReLU when $\alpha >0$$\alpha>0$

TRUE

Recall:

$ReLU(z)=max(0,z)$$ReLU(z)=\max(0,z)$

An important property of ReLU is:

$\boxed{{\displaystyle ReLU(\alpha z)=\alpha ReLU(z),\alpha >0}}$$\boxed{ReLU(\alpha z)=\alpha ReLU(z),\quad \alpha>0}$

Let's substitute:

$h_{11}^{\prime }=ReLU(\alpha (b_{11}+w_{11}x))$$h_{11}'=ReLU(\alpha(b_{11}+w_{11}x))$

Since $\alpha >0$$\alpha>0$:

$h_{11}^{\prime }=\alpha ReLU(b_{11}+w_{11}x)$$h_{11}'=\alpha ReLU(b_{11}+w_{11}x)$

Therefore:

$h_{11}^{\prime }=\alpha h_{11}$$h_{11}'=\alpha h_{11}$

Now consider its contribution to the final output:

$w_{21}^{\prime }{h}_{11}^{\prime }$$w_{21}'h_{11}'$

Substitute the transformed values:

$=\frac{w_{21}}{\alpha }\times \alpha h_{11}$$=\frac{w_{21}}{\alpha}\times \alpha h_{11}$

Cancel $\alpha$$\alpha$:

$\boxed{{\displaystyle w_{21}^{\prime }{h}_{11}^{\prime }=w_{21}{h}_{11}}}$$\boxed{w_{21}'h_{11}'=w_{21}h_{11}}$

The contribution remains unchanged.

Hence, the entire network output remains unchanged.

Option 1 is TRUE.

Option 2: ReLU when $\alpha <0$$\alpha<0$

FALSE

Remember:

$ReLU(\alpha z)\ne \alpha ReLU(z)$$ReLU(\alpha z)\neq \alpha ReLU(z)$

when $\alpha <0$$\alpha<0$, in general.

Let's take an example:

Suppose:

$z=2,\alpha =-2$$z=2,\quad \alpha=-2$

Original:

$ReLU(2)=2$$ReLU(2)=2$

After transformation:

$ReLU(-2\times 2)=ReLU(-4)=0$$ReLU(-2\times2)=ReLU(-4)=0$

But:

$-2\times ReLU(2)=-4$$-2\times ReLU(2)=-4$

Clearly:

$0\ne -4$$0\neq -4$

Therefore, the contribution does not remain unchanged.

Option 2 is FALSE.

Option 3: Sigmoid when $\alpha >0$$\alpha>0$

FALSE

Recall the sigmoid function:

$\sigma (z)=\frac{1}{1+e^{-z}}$$\sigma(z)=\frac{1}{1+e^{-z}}$

Unlike ReLU, sigmoid does not satisfy:

$\sigma (\alpha z)=\alpha \sigma (z)$$\sigma(\alpha z)=\alpha\sigma(z)$

Let's take an example:

$z=2,\alpha =2$$z=2,\quad \alpha=2$

Original:

$\sigma (2)\approx 0.881$$\sigma(2)\approx0.881$

After transformation:

$\sigma (4)\approx 0.982$$\sigma(4)\approx0.982$

But:

$2\sigma (2)\approx 1.762$$2\sigma(2)\approx1.762$

Clearly, these are not equal.

Therefore, the contribution to the output changes.

Option 3 is FALSE.

Option 4: Contribution remains unchanged for all $\alpha \ne 0$$\alpha\neq0$

FALSE

This statement says the contribution remains unchanged for every non-zero value of alpha.

But we already established:

• For ReLU and $\alpha >0$$\alpha>0$, contribution remains unchanged.

• For ReLU and $\alpha <0$$\alpha<0$, it does not necessarily remain unchanged.

• For sigmoid, it does not generally remain unchanged.

Therefore, the statement is false.

Final answer

Exam trick: Remember this property

The entire question revolves around one property:

$\boxed{{\displaystyle ReLU(\alpha x)=\alpha ReLU(x),\alpha >0}}$$\boxed{ReLU(\alpha x)=\alpha ReLU(x),\quad \alpha>0}$

This is called positive homogeneity.

Sigmoid does not have this property.

Remember: Whenever you see a question where weights entering a ReLU neuron are multiplied by a positive constant and outgoing weights are divided by the same constant, think about whether the network output remains invariant.