Calculation error with long

Posted on May 17, 2011 at 14:20

The L suffix means long (32-bit), not long long (64-bit)

This works in Keil, other combinations result in ''integer operation result is out of range''

                long long a = (long long)-1154800 * (long long)3289;

Probably handled by the preprocessor, at the very least it should be folded by the optimizer.

The LL suffix also works.

                long a = -1154800 * 3289;

                long long b = -1154800 * 3289;

                long long c = -1154800L * 3289L;

                long long d = (long long)-1154800 * (long long)3289;

                long long e = -1154800LL * 3289LL;

                printf(''a %ld\n'',a);

                printf(''b %lf\n'',(double)b);

                printf(''c %lf\n'',(double)c);

                printf(''d %lf\n'',(double)d);

                printf(''e %lf\n'',(double)e);

a 496830096                                                                    

b 496830096.000000                                                             

c 496830096.000000                                                             

d -3798137200.000000                                                           

e -3798137200.000000                                                           

Posted on May 17, 2011 at 14:20

Appears to function as expected with GNU 4.3.0, both d) and e) create the correct value.

Using built-in specs.

Target: arm-eabi

Configured with: ../gcc-4.3.0/configure --with-pkgversion='WinARM March 2008' --with-bugurl=http://en.mikrocontroller.net/forum/17 --target=arm-eabi --prefix=/c/WinARM --disable-nls --disable-shared --disable-threads --with-gcc --with-gnu-ld --with-gnu-as --enable-languages=c,c++ --enable-interwork --enable-multilib --disable-libssp --disable-libstdcxx-pch --disable-libmudflap --disable-win32-registry --disable-libgomp --with-dwarf2 --program-prefix=arm-eabi- --with-newlib --disable-newlib-supplied-syscalls --disable-libunwind-exceptions --disable-newlib-atexit-dynamic-alloc --with-gmp=/c/winarms/build/gmp_mpfr --with-mpfr=/c/winarms/build/gmp_mpfr -v

Thread model: single

gcc version 4.3.0 (WinARM March 2008)

void write32(long addr, long x)

{

  *((long *)addr) = x;

}

void write64(long addr, long long x)

{

  *((long long *)addr) = x;

}

{

        long a = -1154800 * 3289;

        long long b = -1154800 * 3289;

        long long c = -1154800L * 3289L;

        long long d = (long long)-1154800 * (long long)3289;

        long long e = -1154800LL * 3289LL;

        write32(0x20002000,a);

        write64(0x20002010,b);

        write64(0x20002020,c);

        write64(0x20002030,d);

        write64(0x20002040,e);

}

000001E8 A013            add    r0, pc, #76    ; (adr r0, $000238=$1D9D0690)

000001EA C803            ldmia    r0!, {r0, r1}   ; Load incorrect 64-bit

000001EC 4B18            ldr    r3, [pc, #96]    ; ($000250=$20002000)

000001EE 4A19            ldr    r2, [pc, #100]    ; ($000254=$1D9D0690) Load incorrect 32-bit

000001F0 601A            str    r2, [r3, #0]

000001F2 3310            adds    r3, #16

000001F4 E883 0003        stmia.w    r3, {r0, r1}

000001F8 3310            adds    r3, #16

000001FA E883 0003        stmia.w    r3, {r0, r1}

000001FE A010            add    r0, pc, #64    ; (adr r0, $000240=$1D9D0690)

00000200 C803            ldmia    r0!, {r0, r1}   ; Load correct 64-bit

00000202 3310            adds    r3, #16

00000204 E883 0003        stmia.w    r3, {r0, r1}

00000208 3310            adds    r3, #16

0000020A E883 0003        stmia.w    r3, {r0, r1}

00000238 1D9D0690        dd    $1D9D0690       ; 64-bit WRONG

0000023C 00000000           dd      $00000000

00000240 1D9D0690        dd    $1D9D0690       ; 64-bit CORRECT

00000244 FFFFFFFF        dd      $FFFFFFFF

00000248 2000000A        dd    $2000000A

0000024C 40011000        dd    $40011000

00000250 20002000        dd    $20002000

00000254 1D9D0690        dd    $1D9D0690       ; 32-bit WRONG

00000258 000AFFFF        dd    $000AFFFF

Posted on May 17, 2011 at 14:20

Ok, your POINT structure should use signed integers (int) as some of your computation generate negative numbers as POINT values are subtracted from each other, and unless casted will be handled as (unsigned int). This was specifically important with the computation of divisor when I tried casting to (double).

The coefficients seem to be correctly computed.

In the computation of x and y, you need to use 64-bit temporary results, x and y can remain as 32-bit signed long.

x = (((long long)matrix.An * xAnalogReading) + ((long long)matrix.Bn * yAnalogReading) + matrix.Cn) / matrix.Divider;

y = (((long long)matrix.Dn * xAnalogReading) + ((long long)matrix.En * yAnalogReading) + matrix.Fn) / matrix.Divider;