Calculation error with long

[tkjmail2]

Posted on January 05, 2011 at 22:35

Calculation error with long

[Tesla DeLorean]

Posted on May 17, 2011 at 14:20

The L suffix means long (32-bit), not long long (64-bit)

This works in Keil, other combinations result in ''integer operation result is out of range''                 long long a = (long long)-1154800 * (long long)3289; Probably handled by the preprocessor, at the very least it should be folded by the optimizer. The LL suffix also works.                 long a = -1154800 * 3289;                 long long b = -1154800 * 3289;                 long long c = -1154800L * 3289L;                 long long d = (long long)-1154800 * (long long)3289;                 long long e = -1154800LL * 3289LL;                 printf(''a %ld\n'',a);                 printf(''b %lf\n'',(double)b);                 printf(''c %lf\n'',(double)c);                 printf(''d %lf\n'',(double)d);                 printf(''e %lf\n'',(double)e); a 496830096                                                                     b 496830096.000000                                                              c 496830096.000000                                                              d -3798137200.000000                                                            e -3798137200.000000                                                           

[tkjmail2]

Posted on May 17, 2011 at 14:20

Hi Clive.

I've tried the ''long long'' method, but it didn't work.

long long e = -1154800LL * 3289LL;

This still resulted in the 496... number!

Thomas

[Tesla DeLorean]

Posted on May 17, 2011 at 14:20

Appears to function as expected with GNU 4.3.0, both d) and e) create the correct value.

Using built-in specs. Target: arm-eabi Configured with: ../gcc-4.3.0/configure --with-pkgversion='WinARM March 2008' --with-bugurl=http://en.mikrocontroller.net/forum/17 --target=arm-eabi --prefix=/c/WinARM --disable-nls --disable-shared --disable-threads --with-gcc --with-gnu-ld --with-gnu-as --enable-languages=c,c++ --enable-interwork --enable-multilib --disable-libssp --disable-libstdcxx-pch --disable-libmudflap --disable-win32-registry --disable-libgomp --with-dwarf2 --program-prefix=arm-eabi- --with-newlib --disable-newlib-supplied-syscalls --disable-libunwind-exceptions --disable-newlib-atexit-dynamic-alloc --with-gmp=/c/winarms/build/gmp_mpfr --with-mpfr=/c/winarms/build/gmp_mpfr -v Thread model: single gcc version 4.3.0 (WinARM March 2008) void write32(long addr, long x) {   *((long *)addr) = x; } void write64(long addr, long long x) {   *((long long *)addr) = x; } {         long a = -1154800 * 3289;         long long b = -1154800 * 3289;         long long c = -1154800L * 3289L;         long long d = (long long)-1154800 * (long long)3289;         long long e = -1154800LL * 3289LL;         write32(0x20002000,a);         write64(0x20002010,b);         write64(0x20002020,c);         write64(0x20002030,d);         write64(0x20002040,e); } 000001E8 A013            add    r0, pc, #76    ; (adr r0, $000238=$1D9D0690) 000001EA C803            ldmia    r0!, {r0, r1}   ; Load incorrect 64-bit 000001EC 4B18            ldr    r3, [pc, #96]    ; ($000250=$20002000) 000001EE 4A19            ldr    r2, [pc, #100]    ; ($000254=$1D9D0690) Load incorrect 32-bit 000001F0 601A            str    r2, [r3, #0] 000001F2 3310            adds    r3, #16 000001F4 E883 0003        stmia.w    r3, {r0, r1} 000001F8 3310            adds    r3, #16 000001FA E883 0003        stmia.w    r3, {r0, r1} 000001FE A010            add    r0, pc, #64    ; (adr r0, $000240=$1D9D0690) 00000200 C803            ldmia    r0!, {r0, r1}   ; Load correct 64-bit 00000202 3310            adds    r3, #16 00000204 E883 0003        stmia.w    r3, {r0, r1} 00000208 3310            adds    r3, #16 0000020A E883 0003        stmia.w    r3, {r0, r1} 00000238 1D9D0690        dd    $1D9D0690       ; 64-bit WRONG 0000023C 00000000           dd      $00000000 00000240 1D9D0690        dd    $1D9D0690       ; 64-bit CORRECT 00000244 FFFFFFFF        dd      $FFFFFFFF 00000248 2000000A        dd    $2000000A 0000024C 40011000        dd    $40011000 00000250 20002000        dd    $20002000 00000254 1D9D0690        dd    $1D9D0690       ; 32-bit WRONG 00000258 000AFFFF        dd    $000AFFFF

[tkjmail2]

Posted on May 17, 2011 at 14:20

Hi.

Now the calculation is correct too, when I just type in numbers. Though Ride7 and the RLINK debugger still shows the decimal representation wrong, as it can't show 64-bit numbers. But if I look in the RAM, the 8MSB's are F's - so the number is: 0xFFFFFFFF1D9D0690, which is correct.

Though if I just change my calibration matrix variables to long long, the code doesn't work at all, as the calculated x and y is over the screen limit - which shouldn't occour, but it occours on calculation errors.

The calibration matrix code and more can be seen here: 

http://pastebin.com/ARy27c5P

The analog readings go from 0 to 4096. The calibration code works fine on another screen with 320x240 resolution, but on the 800x480 resolution it doesn't work!

Best Regards

Thomas Jespersen

[Tesla DeLorean]

Posted on May 17, 2011 at 14:20

Do you have some real world examples for touchptr readings? And some test analogue readings.

[Tesla DeLorean]

Posted on May 17, 2011 at 14:20

Ok, your POINT structure should use signed integers (int) as some of your computation generate negative numbers as POINT values are subtracted from each other, and unless casted will be handled as (unsigned int). This was specifically important with the computation of divisor when I tried casting to (double).

The coefficients seem to be correctly computed. In the computation of x and y, you need to use 64-bit temporary results, x and y can remain as 32-bit signed long. x = (((long long)matrix.An * xAnalogReading) + ((long long)matrix.Bn * yAnalogReading) + matrix.Cn) / matrix.Divider; y = (((long long)matrix.Dn * xAnalogReading) + ((long long)matrix.En * yAnalogReading) + matrix.Fn) / matrix.Divider;