Assembly Language Programming Tips, Tricks, and Traps

High level languages are easy to use, simple to debug, and (usually) powerful enough to write most programs in, but when the last bit of speed is required, or a few undocumented APIs are needed to finish the work, assembly language comes into its own. I don't (as yet) have a C compiler capable of producing OS/2 code, so I tend to do more in pure assembly than other people might; as a result I have burnt more fingers than perhaps I would have when working through a compiler.

From the burnt fingers, as is usual when fingers are burnt, knowledge was gained. In the hope that someone else is as crazy as I am, I hereby publish this list. This is released to the world, for anyone to use, copy, distribute, and plin plon platty ploo, on condition that all copies be maintained intact, including the version number and link below. This document is copyright, and may not be distributed in modified form without the author's permission. However, any changes which anyone can suggest, particularly answers to the questions asked below, will be looked into carefully and probably included for the next version.

This version of ALPTTT is dated 20020216. The latest version can always be found at http://www.kepl.com.au/esstu/alpttt.html.

Most of these tips apply to 32-bit programs. They may also apply to 16-bit programs, and may not. The reason for this emphasis on 32-bit code is that my docs are mostly about 16-bit programming, so I didn't burn my fingers there :-).

• General
  • Always ensure that your stack has sufficient space, and is never corrupted! So if you start a function with PUSH EBP/MOV EBP,ESP you must finish it with POP EBP (possibly preceded by MOV ESP,EBP) at EVERY LOCATION WHICH CAN RETURN. It's all too easy to add an extra RET without a POP!
    
  • If you're struggling to get a number (eg return code from an API) displayed but the program keeps crashing, try using DosExit to return that number to the operating system, and then display that number with PROMPT $R [$P] or similar.
• Calling APIs
  • This /really/ got me! In 16-bit programs, linking with OS2286.LIB, API functions are declared FAR PASCAL - parameters pushed in order, and the called function cleans up the stack. This was what my docs said. But in 32-bit programs, APIs use the C convention - parameters pushed in *reverse* order, and the calling routine cleans up the stack. This is a more robust technique when a program pushes the wrong number of parameters - the stack will be cleaned up based on the number pushed, not the number expected. See tip 1.a!
  • Be careful of which parameters should be passed by _value_ and which by _reference_. Passing a pointer to DosBeep will result in an odd sound!
• PM considerations
  • In the WindowProcedure (the one that gets the hwnd, msg, mp1, mp2 parameters), the default message handling routine must return the value that WinDefWindowProc returns. This applies to the default WM_COMMAND handler, if you have one, too. If this isn't done the window produces odd behaviour when using a menu bar (I'm not sure what's happening, but it's not good). The same applies to WM_TIMER - flashing cursors, autorepeating scroll bars and such are implemented with WM_TIMER messages.
• Use of S2Macros.ASM
  • Warning: S2Macros is at ALPHA stage. It may produce incorrect code, but so far it's worked for me. Also, the macros (and the number of parameters each one uses) may change before everything's stable at 1.0.
  • S2Macros (obtainable from The Esstu Pack, http://www.kepl.com.au/esstu/s2macros.html) requires NASM. Using the file with any other assembler (especially MASM and compatibles) will require considerable modifications. Also, use of a version of NASM earlier than 0.98.22 (my version) may not work.
  • Most useful, of course, is the callos2 macro. This macro pushes the parameters, calls the API, and cleans up the stack. As an added bonus it declares the API extern, so you don't need to worry about listing all your externs at the top of the program. (This relies on NASM allowing a function to be declared extern twice, and also allowing externs to be declared anywhere. These features may not be available in other assemblers.) To use callos2, simply use a line such as 'callos2 DosBeep,[Freq],Dur' which will declare and call DosBeep, passing it two arguments: the contents of the memory location Freq (grabbing a dword from that location), and the value of the token Dur, which should be declared with equ (or similar). Dur will be pushed first, as required by the API. Both will be pushed as dwords, and then 8 bytes will be discarded from the stack.
  • For the C programmer, the switch/case/default/endswitch set of macros will be easy to use. But since there's _no_ optimization done, it's up to the programmer to make intelligent choices about what to switch on - I strongly recommend the use of AL, AX, or EAX as there are compact comparison commands for these. Note: switch commands can be nested, which is useful for Window Procedures in PM programs (eg for WM_COMMAND), by the use of NASM's context stack. Note: As yet the break command is not implemented nor required. This may change.
  • func is a very simple macro - it simply declares a function global and exported, and makes the label, followed by PUSH EBP/MOV EBP,ESP. Handy for OS/2-callable functions. func was my first attempt to make a useful macro.
• Interfacing with REXX
  • The default RXSTRING for the return value has only 256 bytes (+/- a bit) available. If you expect your return string to exceed this length, either allocate a new buffer with DosAllocMem, or use the address of a fixed buffer in your data area. (Note: This is only what *APPEARS* to work! If anyone has certain information about this - and the docs only say to use DosAllocMem - can you please tell me??)
  • VX-REXX provides very good support for lower-level (C and assembly) programming techniques. Interfacing VX-REXX and assembly code is very simple, especially if the REXX code does most of the work.
    • The HWND property of all objects is simply the characters '?HWND' prefixed onto the OS/2 Window Handle, expressed in hexadecimal (with lowercase letters). A REXX program can easily parse off the prefix, translate the HWND into decimal or ascii, and pass it to some other routine.
    • Standard OS/2 PM window messages (WM_*) can be trapped and handled as VX-REXX events. Standard messages can also be sent from a VX-REXX program to any window on the screen.
  • A puzzle - can anyone help me? The RexxStart function, which calls a REXX program, doesn't return until REXX finishes. The docs say that only the thread which called RexxStart can use RexxVariablePool to set/get variables in the REXX program's memory. Is it possible for the calling program to "tinker with" the REXX without being explicitly called (eg as an external function)?