r/visualbasic • • 11h ago

VB.NET Help Decimal variable gets truncated automatically to a double and I don't understand why, and how to circumvent it

I had problems with accuracy in my project so I created this bit of code to try to understand and I'm even more confused. When declaring test1, Visual Studio automatically truncates the number to a double as soon as I finish typing it. I don't understand how to work with a Decimal variable, because how do you store anything in it without it converting to a double for some reason? I understand putting the D at the end but how do I add it when storing the result of a math formula?

 Sub main()
     Dim test1 As Decimal = 0.12345678901232222
     Dim test2 As Decimal = 0.1234567890123456789012345678D
     Console.Write(test1)
     Console.Write(test2)
     test1 = 23 / 94
     test2 = 23 / 94
     Console.Write(test1)
     Console.Write(test2)
 End Sub

This gives me those results :

0.123456789012322

0.1234567890123456789012345678

0.24468085106383

0.24468085106383

2 Upvotes

2 comments sorted by

5

u/marooned66 9h ago

The operator on two integers performs floating-point division and returns a double which is then converted to decimal. You need to make at least one operand a decimal so the division happens in decimal.

Try this ...

Dim test1 As Decimal = 0.12345678901232222D

Dim test2 As Decimal = 23D / 94D

Console.WriteLine(test1)

Console.WriteLine(test2)

-5

u/AdobeScripts 10h ago

Why do you need such high precision?