激活函数之ReLU/softplus介绍及C++实现

softplus函数(softplus function)：ζ(x)=ln(1+exp(x)).

softplus函数可以用来产生正态分布的β和σ参数，因为它的范围是(0,∞)。当处理包含sigmoid函数的表达式时它也经常出现。softplus函数名字来源于它是另外一个函数的平滑(或”软化”)形式，这个函数是x⁺=max(0,x)。softplus 是对 ReLU 的平滑逼近的解析函数形式。

softplus函数别设计成正部函数(positive part function)的平滑版本，这个正部函数是指x⁺=max{0,x}。与正部函数相对的是负部函数(negative part function)x^-=max{0, -x}。为了获得类似负部函数的一个平滑函数，我们可以使用ζ(-x)。就像x可以用它的正部和负部通过等式x⁺-x^-=x恢复一样，我们也可以用同样的方式对ζ(x)和ζ(-x)进行操作，就像下式中那样：ζ(x) -ζ(-x)=x.

Rectifier:In the context of artificial neural networks, the rectifier is an activation function defined as:

f(x)=max(0,x)

where x is the input to a neuron. This activation function was first introduced to a dynamical network by Hahnloser et al. in a 2000 paper in Nature. It has been used in convolutional networks more effectively than the widely used logistic sigmoid (which is inspired by probability theory; see logistic regression) and its more practical counterpart, the hyperbolic tangent. The rectifier is, as of 2015, the most popular activation function for deep neural networks.

A unit employing the rectifier is also called a rectified linear unit (ReLU).

A smooth approximation to the rectifier is the analytic function: f(x)=ln(1+e^x), which is called the softplus function. The derivative of softplus is: f’(x)=e^x/(e^x+1)=1/(1+e^-x), i.e. the logistic function.

Rectified linear units(ReLU) find applications in computer vision and speech recognition using deep neural nets.

Noisy ReLUs: Rectified linear units can be extended to include Gaussian noise, making them noisy ReLUs, giving: f(x)=max(0, x+Y), with Y∽N(0, σ(x)). Noisy ReLUs have been used with some success in restricted Boltzmann machines for computer vision tasks.

Leaky ReLUs：allow a small, non-zero gradient when the unit is not active：

Parametric ReLUs take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural network parameters:

Note that for a≤1, this is equivalent to: f(x)=max(x, ax), and thus has a relation to "maxout" networks.

ELUs：Exponential linear units try to make the mean activations closer to zero which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs：

a is a hyper-parameter to be tuned and a≥0 is a constraint.

以上内容摘自：《深度学习中文版》和维基百科

以下是C++测试code：

#include "funset.hpp"
#include <math.h>
#include <iostream>
#include <string>
#include <vector>
#include <opencv2/opencv.hpp>
#include "common.hpp"// ========================= Activation Function: ELUs ========================
template<typename _Tp>
int activation_function_ELUs(const _Tp* src, _Tp* dst, int length, _Tp a = 1.)
{if (a < 0) {fprintf(stderr, "a is a hyper-parameter to be tuned and a>=0 is a constraint\n");return -1;}for (int i = 0; i < length; ++i) {dst[i] = src[i] >= (_Tp)0. ? src[i] : (a * (exp(src[i]) - (_Tp)1.));}return 0;
}// ========================= Activation Function: Leaky_ReLUs =================
template<typename _Tp>
int activation_function_Leaky_ReLUs(const _Tp* src, _Tp* dst, int length)
{for (int i = 0; i < length; ++i) {dst[i] = src[i] > (_Tp)0. ? src[i] : (_Tp)0.01 * src[i];}return 0;
}// ========================= Activation Function: ReLU =======================
template<typename _Tp>
int activation_function_ReLU(const _Tp* src, _Tp* dst, int length)
{for (int i = 0; i < length; ++i) {dst[i] = std::max((_Tp)0., src[i]);}return 0;
}// ========================= Activation Function: softplus ===================
template<typename _Tp>
int activation_function_softplus(const _Tp* src, _Tp* dst, int length)
{for (int i = 0; i < length; ++i) {dst[i] = log((_Tp)1. + exp(src[i]));}return 0;
}int test_activation_function()
{std::vector<double> src{ 1.23f, 4.14f, -3.23f, -1.23f, 5.21f, 0.234f, -0.78f, 6.23f };int length = src.size();std::vector<double> dst(length);fprintf(stderr, "source vector: \n");fbc::print_matrix(src);fprintf(stderr, "calculate activation function:\n");fprintf(stderr, "type: sigmoid result: \n");fbc::activation_function_sigmoid(src.data(), dst.data(), length);fbc::print_matrix(dst);fprintf(stderr, "type: sigmoid fast result: \n");fbc::activation_function_sigmoid_fast(src.data(), dst.data(), length);fbc::print_matrix(dst);fprintf(stderr, "type: softplus result: \n");fbc::activation_function_softplus(src.data(), dst.data(), length);fbc::print_matrix(dst);fprintf(stderr, "type: ReLU result: \n");fbc::activation_function_ReLU(src.data(), dst.data(), length);fbc::print_matrix(dst);fprintf(stderr, "type: Leaky ReLUs result: \n");fbc::activation_function_Leaky_ReLUs(src.data(), dst.data(), length);fbc::print_matrix(dst);fprintf(stderr, "type: Leaky ELUs result: \n");fbc::activation_function_ELUs(src.data(), dst.data(), length);fbc::print_matrix(dst);return 0;
}

GitHub： https://github.com/fengbingchun/NN_Test

激活函数之ReLU/softplus介绍及C++实现相关推荐

激活函数、Sigmoid激活函数、tanh激活函数、ReLU激活函数、Leaky ReLU激活函数、Parametric ReLU激活函数详细介绍及其原理详解
相关文章梯度下降算法.随机梯度下降算法.动量随机梯度下降算法.AdaGrad算法.RMSProp算法.Adam算法详细介绍及其原理详解反向传播算法和计算图详细介绍及其原理详解激活函数.Sigmo ...
SMU激活函数 | 超越ReLU、GELU、Leaky ReLU让ShuffleNetv2提升6.22%
点击上方"计算机视觉工坊",选择"星标" 干货第一时间送达作者丨ChaucerG 来源丨集智书童选择一个好的激活函数可以对提高网络性能产生重要的影响.Han ...
花书+吴恩达深度学习（二）非线性激活函数（ReLU, maxout, sigmoid, tanh）
目录 0. 前言 1. ReLU 整流线性单元 2. 绝对值整流线性单元 3. 渗漏整流线性单元 4. 参数化整流线性单元 5. maxout 单元 6. logistic sigmoid 单元 7. ...
激活函数总结RELU，Leaky RELU
ReLU 修正线性单元(Rectified linear unit,ReLU)是神经网络中最常用的激活函数. ReLu激活函数的优点是: 1,相比Sigmoid/tanh函数,使用梯度下降(GD)法时 ...
常见的激活函数 sigmod Relu tanh LeakyRelu及复现代码
目录 1.激活函数的作用 2.sigmod 3.Relu 4.tanh 5.LeakyRelu 1.激活函数的作用关于神经网络中的激活函数的作用,通常都是这样解释:如果不使用激活函数的话,神经网络的 ...
激活函数之ReLU函数
0 前言激活函数的引入是为了增加神经网络模型的非线性,没有激活函数每层就相当于矩阵相乘.每一层输出都是上层的输入的线性函数,无论神经网络多少层,输出都是输入的线性组合,就是最原始的感知机加入激活函 ...
22个激活函数，ReLU、RReLU、LeakyReLU、PReLU、Sofplus、ELU、CELU、SELU、GELU、ReLU6、Sigmoid、Tanh、Softsign、Hardtanh等
转自:https://www.pianshen.com/article/33331174884/ 1.22.Linear常用激活函数 1.22.1.ReLU torch.nn.ReLU() 1.22. ...
激活函数（relu，prelu，elu，+BN）对比on cifar10
最近做了个对比实验,通过修改激活函数观察对图片分类准确率的影响,现记录如下: 一．理论基础 1.1激活函数 1.2 elu论文(FAST AND ACCURATE DEEP NETWORK LEARN ...
机器学习入门（12）— 激活函数层 ReLU、Sigmoid 层的实现
1. ReLU 函数层激活函数 ReLU(Rectified Linear Unit)由下式(5.7)表示. 通过式(5.7),可以求出 y 关于 x 的导数,如式(5.8)所示. 在式(5.8)中 ...

激活函数之ReLU/softplus介绍及C++实现

激活函数之ReLU/softplus介绍及C++实现相关推荐

最新文章

热门文章