Eicholtz asked . 2021-07-13

Am I computing cross entropy incorrectly?

I am working on a neural network and would like to use cross entropy as my error function. I noticed from a previous question that MATLAB added this functionality starting with R2013b. I decided to test the crossentropy function by running the simple example provided in the documentation. The code is reprinted below for convenience:

[x,t] = iris_dataset;
net = patternnet(10);
net = train(net,x,t);
y = net(x);
perf = crossentropy(net,t,y)

When I run this code, I get perf = 0.0367. To verify this result, I ran the code:

ce = -mean(sum(t.*log(y)+(1-t).*log(1-y)))

which resulted in ce = 0.1100. Why are perf and ce unequal? Do I have an error in my calculation?

neural network , cross entropy , error function

Expert Answer

Prashant Kumar answered . 2024-12-21 00:44:46

If c classes are mutually exclusive, classifier target probability values should be the certain probability values of 0 or 1 and must sum to 1.

If the corresponding classifier uses a softmax output transfer function, output estimates are bounded by the open range (0,1) and sum to 1.

If classes are not mutually exclusive (e.g., tall, dark ,handsome ), 0 or 1 classifier target probability values do not have to sum to 1.

If the corresponding classifier uses a logsig output transfer function, output estimates are bounded by the open range (0,1) but are not constrained to have a unit sum.

A useful performance function is the crossentropy between outputs and targets.

For mutually exclusive targets and a softmax output, the corresponding form for crossentropy is

 Xent1 = -sum( t.*log(y))

For non-mutually exclusive targets and a logsig output, the corresponding form for crossentropy is

 Xent2 = -sum( t.*log(y)) + (1-t).*log(1-y))

For your example I get

 clear all, clc
 [ x, t ] = iris_dataset;
 [ O N  ] = size(t)   % [ 3 150 ]
 minmax0  = repmat([0 1],3,1)
 checkt1  = max(abs( minmax(t)- minmax0))%[0 0]
 checkt2  = max(abs(sum(t)-ones(1,N))) % 0
 net      = patternnet(10);
 rng(0)
 [ net tr y ] = train(net,x,t);
 checky1      = max(abs( minmax(y)- minmax0))
 % checky1    = [ 2.4214e-4  1.8807e-3 ]
 checky2      = max(abs(sum(y)-ones(1,N))) % 2.2204e-16

 perf  = crossentropy(net,t,y)                  % 0.033005
 Xent1 = mean(-sum(t.*log(y)))                  % 0.049552
 Xent3 = mean(-sum((1-t).*log(1-y)))            % 0.049464
 Xent2 = mean(-sum(t.*log(y)+ (1-t).*log(1-y))) % 0.099015

Unfortunately, none of the following gives a formula

 help crossentropy
 doc crossentropy
 type crossentropy

and the example in the website documentation incorrectly uses Xent2 which is only valid for nonexclusive classes.

If you search on crossentropy in the comp.ai.neural-nets newsgroup, you should find many posts on the topic.

Bottom Line: Xent2 is the correct answer. However, your calculation of crossentropy and Xent3 are not too far from mine. If you use rng(0) they should match.

Not satisfied with the answer ?? ASK NOW

Recent Matlab Projects

Classification of Covid and Non-Covid Lungs CT-Scan using Deep Learning with MATLAB

Matlab simulation on Wind Energy system

Frequently Asked Questions

MATLAB offers tools for real-time AI applications, including Simulink for modeling and simulation. It can be used for developing algorithms and control systems for autonomous vehicles, robots, and other real-time AI systems.

MATLAB Online™ provides access to MATLAB® from your web browser. With MATLAB Online, your files are stored on MATLAB Drive™ and are available wherever you go. MATLAB Drive Connector synchronizes your files between your computers and MATLAB Online, providing offline access and eliminating the need to manually upload or download files. You can also run your files from the convenience of your smartphone or tablet by connecting to MathWorks® Cloud through the MATLAB Mobile™ app.

Yes, MATLAB provides tools and frameworks for deep learning, including the Deep Learning Toolbox. You can use MATLAB for tasks like building and training neural networks, image classification, and natural language processing.

MATLAB and Python are both popular choices for AI development. MATLAB is known for its ease of use in mathematical computations and its extensive toolbox for AI and machine learning. Python, on the other hand, has a vast ecosystem of libraries like TensorFlow and PyTorch. The choice depends on your preferences and project requirements.

You can find support, discussion forums, and a community of MATLAB users on the MATLAB website, Matlansolutions forums, and other AI-related online communities. Remember that MATLAB's capabilities in AI and machine learning continue to evolve, so staying updated with the latest features and resources is essential for effective AI development using MATLAB.

Without any hesitation the answer to this question is NO. The service we offer is 100% legal, legitimate and won't make you a cheater. Read and discover exactly what an essay writing service is and how when used correctly, is a valuable teaching aid and no more akin to cheating than a tutor's 'model essay' or the many published essay guides available from your local book shop. You should use the work as a reference and should not hand over the exact copy of it.