math calculator

AI Activation Functions & Derivatives Calculator

Calculate and visualize neural network activation functions like Sigmoid, ReLU, and Softmax along with their derivatives instantly.

Formula Verified by Dr. Elena Rostova, PhD

Quick actions: Save inputs, view saved runs, print, share, or export results

Saved Runs

Inputs

Activation Function

Input x

Results

f(1)

0.731059

f′(1) Derivative0.196612

Function Curve

f(1) = 0.7311 · f′(1) = 0.1966

Calculation Steps

Apply the Sigmoid function: \[f(x) = \frac{1}{1 + e^{-x}} = \frac{1}{1 + e^{-1}} = \frac{1}{1 + 0.3679} = 0.731059\]

Calculate the derivative: \[f'(x) = \sigma(x)(1 - \sigma(x)) = 0.7311 \times (1 - 0.7311) = 0.196612\]

Tip

Save this calculator to quickly check activation values, derivatives, and softmax probabilities while working through a model by hand.

Formula

Activation Function Formulas

\sigma(x) = \frac{1}{1+e^{-x}}, \quad \text{ReLU}(x) = \max(0, x), \quad \text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}

Activation functions introduce non-linearity into neural networks. The choice of function affects training speed, gradient flow, and model capacity.

Step by step

How the calculation works

1Select a function type and input x to compute its output value and derivative.
2The curve plot shows the function shape with your x value highlighted in blue.
3Use the Softmax tab to distribute a logit vector into a probability distribution.

Related guides

Learn more about this calculation

Guide

Scientific Notation Guide: Powers of Ten Made Practical

Learn how to convert, compare, and calculate with powers of ten using worked examples and the live DTC scientific calculator.

Site Guide

How to Use Do The Calculation Calculators: A Practical Step-by-Step Guide

Learn the fastest reliable workflow for using Do The Calculation calculators, reading results, checking formulas, and using save, print, share, and export actions correctly.

Bayes' Theorem

Bayes' Theorem Calculator

Compute posterior probability distributions used in probabilistic models.

Scientific

Scientific Calculator

Compute exponentials, logarithms, and hyperbolic functions.

Results

f(1)

0.731059

f′(1) Derivative0.196612

Function Curve

f(1) = 0.7311 · f′(1) = 0.1966

Calculation Steps

Apply the Sigmoid function: \[f(x) = \frac{1}{1 + e^{-x}} = \frac{1}{1 + e^{-1}} = \frac{1}{1 + 0.3679} = 0.731059\]

Calculate the derivative: \[f'(x) = \sigma(x)(1 - \sigma(x)) = 0.7311 \times (1 - 0.7311) = 0.196612\]

How to understand and use this calculator

Why Neural Networks Need Non-Linear Activations

An activation function is what a neuron applies to its weighted input before passing a value to the next layer. Without a non-linear activation, stacking any number of layers would still collapse mathematically into a single linear transformation — depth would add no representational power at all. The non-linearity is what lets a network approximate the curved, complex decision boundaries that real data requires.

Which function to use is a real design trade-off, not a fixed rule: it affects how well gradients survive backpropagation through many layers, how expensive the function is to compute at scale, and how it interacts with the network's weight initialization scheme.

How to Use This Calculator

Pick a function — Sigmoid, Tanh, ReLU, Leaky ReLU, or Softmax — then enter an input value x. For Leaky ReLU, also set the leak coefficient α (default 0.01). The calculator returns the function's output f(x), its derivative f'(x) at that point, and a curve plot showing where x sits. Switch to Softmax to enter a comma-separated list of logits instead and see the normalized probability distribution.

Worked Example: Sigmoid Activation at x = 1.0

Input: x = 1.0, function = Sigmoid.

f(1.0) = 1/(1 + e^(-1.0)) = 1/(1 + 0.367879) = 0.731059

f'(1.0) = f(1.0)(1 - f(1.0)) = 0.731059 × 0.268941 = 0.196612

The same network's output layer, with logits z = [2.0, 1.0, 0.1] run through Softmax: e^(2.0) = 7.389056, e^(1.0) = 2.718282, e^(0.1) = 1.105171, summing to 11.212509. Dividing each exponential by that sum gives probabilities of 65.90%, 24.24%, and 9.86% — the highest logit dominates the distribution, which is exactly the effect Softmax is designed to produce.

The Five Functions This Calculator Computes

Sigmoid f(x) = 1/(1+e^(-x)): squashes any real input into (0, 1). Used for binary-classification output layers, but its derivative maxes out at 0.25 and shrinks toward zero at the extremes — stacked across many layers, that repeated multiplication is the classic cause of vanishing gradients.

Tanh f(x) = tanh(x): the zero-centered cousin of Sigmoid, squashing to (-1, 1). Zero-centered outputs tend to keep the next layer's gradients better balanced, but Tanh still saturates (and its gradient still shrinks) for large |x|.

ReLU f(x) = max(0, x): the default for hidden layers in most modern networks. Its derivative is exactly 1 for any positive input — no shrinking, no saturation — which is why deep ReLU networks train far more reliably than deep Sigmoid/Tanh ones. The trade-off is the dead ReLU problem: a neuron that's driven to a negative input produces a zero output *and* a zero gradient, so it stops learning entirely.

Leaky ReLU f(x) = x for x>0, α x for x ≤ 0: patches the dead-ReLU problem with a small non-zero slope (typically α = 0.01) on the negative side, so a neuron that drifts negative still carries a gradient and can recover during training.

Softmax P(y_i) = e^(z_i)/(Σ_j e^(z_j)): not a per-neuron activation but an output-layer function that turns a vector of raw logits into a probability distribution that sums to 1 — the standard choice for multi-class classification, almost always paired with categorical cross-entropy loss.

Vanishing and Exploding Gradients

Backpropagation computes each weight's gradient via the chain rule, multiplying an activation function's derivative at every layer it passes through. If that derivative is consistently below 1 — the norm for Sigmoid and Tanh once inputs move away from zero — the product shrinks exponentially with depth, and early layers stop receiving any meaningful update. That's the vanishing gradient problem, and it's the main reason Sigmoid and Tanh have been displaced by ReLU-family functions in hidden layers of deep networks.

The opposite failure, exploding gradients, happens when derivatives compound to values well above 1, usually from large initial weights, producing destabilizing updates and eventually NaN losses. Weight initialization schemes matched to the activation function — He initialization for ReLU-family functions, Xavier/Glorot for Tanh or Sigmoid — exist specifically to keep the variance of activations (and their gradients) stable layer to layer.

Beyond This Calculator: GELU, Swish, and Other Variants

This calculator covers the five functions above, which is enough to work any Sigmoid, Tanh, ReLU, Leaky ReLU, or Softmax problem by hand. Modern large-scale architectures — Transformers in particular — often use smoother variants instead: GELU f(x) = x · Φ(x) (weighting each input by its percentile under the standard normal CDF Φ) is the default in most Transformer-based language models, and Swish f(x) = x · σ(β x) is common in vision architectures like EfficientNet. Both are smooth and non-monotonic, which avoids the sharp kink at zero that plain ReLU has — useful in architectures sensitive to gradient noise, like the self-attention layers in a Transformer. PReLU (a learnable Leaky ReLU slope), ELU, and SELU are further ReLU variants aimed at the same dead-neuron problem from different angles. None of these are computed by this tool today, but the same derivative and gradient-flow logic above applies to all of them.

Related Calculators

Activation functions sit at the intersection of calculus and linear algebra: use the derivative solver to check gradients of other functions by hand, or the matrix solver for the linear-algebra operations that feed each neuron before activation. For general exponentials and logarithms outside a neural-network context, the logarithm calculator handles those directly.

Related concepts

Backpropagation

The chain-rule algorithm neural networks use to compute how much each weight contributed to the error, layer by layer.

Weight initialization

Schemes like He and Xavier initialization that set a network's starting weights to keep activation variance stable across layers.

Vanishing/exploding gradients

The failure modes where repeated multiplication of small or large derivatives across layers shrinks or blows up backpropagated gradients.

Frequently Asked Questions

What is the mathematical definition of the Sigmoid function?+

Sigmoid is f(x) = 1 / (1 + e^-x). It squashes any real-valued input into a bounded (0, 1) range, historically used for binary probability estimation.

How does the dead ReLU problem occur in training?+

It happens when a neuron's weights are updated such that it consistently receives negative input. Standard ReLU outputs exactly zero for any negative value, so the neuron's gradient is also zero — it stops updating and stays "dead" for the rest of training.

Does this calculator support GELU or Swish?+

No — this calculator computes Sigmoid, Tanh, ReLU, Leaky ReLU, and Softmax. GELU and Swish are common in modern Transformer and vision architectures, but aren't in this tool's function list yet; see the section above for their formulas.

What causes the vanishing gradient problem in deep networks?+

It arises when activation functions with derivatives smaller than 1 — like Sigmoid or Tanh — are stacked across many layers. Backpropagation multiplies those derivatives together, so the gradient shrinks exponentially with depth until early layers get essentially no update.

When should I use Tanh instead of Sigmoid?+

Prefer Tanh in hidden layers when you want zero-centered outputs (-1 to 1) rather than Sigmoid's (0, 1) range — zero-centering tends to keep the next layer's gradient updates more balanced and can speed up convergence.

What is the mathematical basis for Softmax in output layers?+

Softmax exponentiates each logit and divides by the sum of all the exponentials, so every output is positive and the whole vector sums to exactly 1 — a valid probability distribution over mutually exclusive classes.

How does weight initialization interact with activation functions?+

Initialization scale should match the activation: He initialization is calibrated for ReLU-family functions, while Xavier/Glorot initialization is calibrated for symmetric functions like Tanh or Sigmoid. Mismatching them can reintroduce vanishing or exploding gradients even with a good activation choice.

How does Leaky ReLU prevent zero-gradient saturation?+

It replaces the flat zero output in ReLU's negative domain with a small linear slope, f(x) = αx for x ≤ 0 (α is typically 0.01). That keeps a small non-zero gradient flowing backward even for neurons with negative input, so they can still recover during training.

Why is ReLU cheaper to compute than Sigmoid, Tanh, GELU, or Swish?+

ReLU is a single comparison (max(0, x)) with no exponential involved. Sigmoid, Tanh, GELU, and Swish all require computing e^x, which costs meaningfully more compute and memory bandwidth at the scale of billions of neurons — part of why ReLU stays the default for latency-sensitive or edge deployments.

What's the difference between an activation function and a loss function?+

An activation function transforms a neuron's or layer's output during the forward pass. A loss function compares the network's final output to the true label to produce a single number to minimize. Softmax (activation) and categorical cross-entropy (loss) are typically paired, but they measure different things.

How to understand and use this calculator

Why Neural Networks Need Non-Linear Activations

How to Use This Calculator

Worked Example: Sigmoid Activation at x = 1.0

Input: x = 1.0, function = Sigmoid.

f(1.0) = 1/(1 + e^(-1.0)) = 1/(1 + 0.367879) = 0.731059

f'(1.0) = f(1.0)(1 - f(1.0)) = 0.731059 × 0.268941 = 0.196612

The Five Functions This Calculator Computes

Vanishing and Exploding Gradients

Beyond This Calculator: GELU, Swish, and Other Variants

Related Calculators

Related concepts

Backpropagation

The chain-rule algorithm neural networks use to compute how much each weight contributed to the error, layer by layer.

Weight initialization

Schemes like He and Xavier initialization that set a network's starting weights to keep activation variance stable across layers.

Vanishing/exploding gradients

The failure modes where repeated multiplication of small or large derivatives across layers shrinks or blows up backpropagated gradients.

Frequently Asked Questions

What is the mathematical definition of the Sigmoid function?+

Sigmoid is f(x) = 1 / (1 + e^-x). It squashes any real-valued input into a bounded (0, 1) range, historically used for binary probability estimation.

How does the dead ReLU problem occur in training?+

Does this calculator support GELU or Swish?+

What causes the vanishing gradient problem in deep networks?+

When should I use Tanh instead of Sigmoid?+

What is the mathematical basis for Softmax in output layers?+

How does weight initialization interact with activation functions?+

How does Leaky ReLU prevent zero-gradient saturation?+

Why is ReLU cheaper to compute than Sigmoid, Tanh, GELU, or Swish?+

What's the difference between an activation function and a loss function?+

AI Activation Functions & Derivatives Calculator

Inputs

Results

Function Curve

Calculation Steps

Activation Function Formulas

How the calculation works

Learn more about this calculation

Scientific Notation Guide: Powers of Ten Made Practical

How to Use Do The Calculation Calculators: A Practical Step-by-Step Guide

Bayes' Theorem Calculator

Scientific Calculator

AI Activation Functions & Derivatives Calculator

Inputs

Results

Function Curve

Calculation Steps

Key output metrics

Function Value vs. Derivative

How to understand and use this calculator

Why Neural Networks Need Non-Linear Activations

How to Use This Calculator

Worked Example: Sigmoid Activation at x = 1.0

The Five Functions This Calculator Computes

Vanishing and Exploding Gradients

Beyond This Calculator: GELU, Swish, and Other Variants

Related Calculators

Related concepts

Frequently Asked Questions

Activation Function Formulas

How the calculation works

Learn more about this calculation

Scientific Notation Guide: Powers of Ten Made Practical

How to Use Do The Calculation Calculators: A Practical Step-by-Step Guide

Bayes' Theorem Calculator

Scientific Calculator

Key output metrics

Function Value vs. Derivative

How to understand and use this calculator

Why Neural Networks Need Non-Linear Activations

How to Use This Calculator

Worked Example: Sigmoid Activation at x = 1.0

The Five Functions This Calculator Computes

Vanishing and Exploding Gradients

Beyond This Calculator: GELU, Swish, and Other Variants

Related Calculators

Related concepts

Frequently Asked Questions