accessing decorated function names in a DLL

[alaa-alawi]

Hi,
I'm trying to interface with FANN library and create wrapper classes for it, using 'External Interfacing' topic under the Programming cookbook from the (really very useful) help file.

the FANN library functions are not consistent, some are decorated and some are not. trying to access the decorated functions like

<stdcall: void '_fann_train@12' FANNStruct* float* float*>

gives a memory access error/fault
the undecorated functions seems to work without errors.

i tried to search the given documentation and search the net but could not found a solution.

I assume I'm missing something simple and got confused midway. is there a way to call these functions or identify and resolve the issue?

thanks in advance for the support

[blairmcg]

I'll give a fairly comprehensive answer to this, as it is a topic that might be generally useful to others in future.

The function names are essentially passive, so the only really failure mode possible with those is that you get the exported name wrong and Dolphin can't find it. That isn't the case here, as you've got the exported name right (e.g. by running dumpbin/exports on the DLL). The "decorated" names that you are seeing are just the conventions for stdcall exports, i.e. leading underscore and @<args size> suffix. From the exports I see there are a few functions without the stdcall decorations. The reason for that is that these are "varargs" functions (i.e. they accept a variable number of arguments), and so are not actually compatible with the stdcall calling convention, they have to be cdecl. The cdecl functions exported by the C compiler will have undecorated names, e.g. fann_create_sparse

FANN_EXTERNAL struct fann *FANN_API fann_create_sparse(float connection_rate, unsigned int num_layers, ...)

Getting the calling convention wrong is another common error, and may result in Dolphin crashing altogether (stack faults are generally not recoverable). Also not the case here. Assuming you built the fannfloat DLL in Win32 configuration from the VS2010 solution in libfann/fann, then I can see that stdcall is the correct convention for all the exported functions aside from the 3x varargs functions and one data export.

Some possible reasons for memory faults are:

1. Passing the wrong types of arguments. Here you'd have to pass an instance of the custom FANNStruct structure that you've defined, and probably two FLOAT structure objects. The first of these would need to be a 'buffer' instance (i.e. holding a 4-byte ByteArray in its bytes instance variable to hold the float value), the 2nd would need to be a pointer instance (i.e. holding an ExternalAddress at its bytes) to receiver the output value.
2. Your FANNStruct may be incorrectly defined, having some mistake in the type, order or completeness of the fields, or having the wrong padding.
3. You FANNStruct instance is incorrectly initialized, e.g. some internal pointer is not set when it needs to be.

For the first, a correct invocation might look something like this:

ann := FANNStruct new.
"Initialize fields of `ann` appropriately here"
input := FLOAT new value: 1.23; yourself.
desired_output := FLOAT newPointer.
FANNLibrary default fann_train: ann input: input desired_output: desired_output.

For the 2nd, the structure layout can be tricky for large structures. I often build IDL and type libraries for large APIs so that I can get an auto-generated first level API wrapper to avoid errors hand building the projection myself, but that in itself can be a big investment of time and you have to be familiar with MIDL, so this may not be feasible. What I'd do as a quick check is to first compare the byteSize reported for the FANNStruct you have defined with that reported by the C compiler. To do this I added the following line to the top of one of the simple_train_float project's main function:

	printf("sizeof(fann)=%d\n", sizeof(struct fann));

I then ran it, and it printed sizeof(fann)=276. If FANNStruct byteSize is not also 276, then memory faults/corruption are very likely. Even if it is correct, the field definitions/offsets may still be wrong. Dolphin will use the default Windows structure packing algorithm start looking from ExternalStructure class>>#paddedSizeFor: for more details. It looks to me like the project does not specify custom packing with either the compiler command line option /Zp or any #pragma pack directives. Therefore the layout calculated by Dolphin should be correct if you got the field definitions right. To verify this, I'd use a similar technique as with the overall size, and insert a bunch more printfs in the test program. I've pasted my test code and output below at the end of my response (because it is long). You can repeat this technique for any nested structures, e.g. fann_layer to verify that the layout is the same as the C compiler expects.

You can then compare the offsets against the field offsets in the structure (inspect it, or look at the accessor methods). You can always modify the defineTemplate methods for the structures to actually specify the offsets and total size. This is what the type library analyzer will do, although for hand-coded definitions it does have the disadvantage that if you need to change the definition at all, e.g. to use the fanndouble library, then you won't benefit from the auto-calculation of the layout (assuming that is correct).

For the 3rd issue you may be able to refer to existing examples to cross check that you are setting things up correctly. I often try and transcribe an existing sample so that I know what I am doing should work if I can just translate it correctly. However, creating a projection to any reasonably complex library can require abandoning treating the library like a black box, and instead dropping down into a low-level debugger and debugging through the library code. You at least have access to the source, and can build the library easily, so this is straightforward. Having verified all the structure layouts are correct, and that you are passing the correct argument types, if memory faults still occur then your next step is to run under the VisualStudio debugger. In the debug configuration of the FANN dll you are using, go into the the project configuration properties in VS and change the Debugging/Command to the filename of the dolphin launcher exe you are using. Then F5 and Dolphin should launch. Note that you will find Dolphin will occasionally generate memory faults internally to the VM because it uses reserved but uncommitted pages to manage dynamic growth of heap and stacks. You should start to recognise where these occur, and can just ignore them (i.e continue execution). If you now start running your faulting code, you should be able to catch the error and try and work back from there. If the fault fires in the FANN code you will hopefully be able to see what is not correctly set up. If the fault is due to corruption of Dolphin's heap, then building and run a debug VM may help. It may not be easy.

Lastly, bear in mind that Dolphin's Float class is actually a double in C terms. There is no native equivalent of float, so you'll be constantly converting back and forth, and aside from the perf cost of this there is also the "excess" precision of Dolphin's native floating point values. It may be worth using the double version of the FANN library. Or at least that may simplify the language binding work a bit, e.g. you can pass a Dolphin Float, or its address, directly to APIs expecting double or double* arguments, which is both simpler and avoids the costs of allocating FLOAT structure and copying values in and out. However there may be other disadvantages. It's difficult to judge what would be the best choice without actually doing the work to understand the trade offs.

HTH

Blair

---

  printf("sizeof(fann)=%d\n", sizeof(struct fann));

  printf("offsetof(struct fann, errno_f) = %d\n", offsetof(struct fann, errno_f));
  printf("offsetof(struct fann, error_log) = %d\n", offsetof(struct fann, error_log));
  printf("offsetof(struct fann, errstr) = %d\n", offsetof(struct fann, errstr));
  printf("offsetof(struct fann, learning_rate) = %d\n", offsetof(struct fann, learning_rate));
  printf("offsetof(struct fann, learning_momentum) = %d\n", offsetof(struct fann, learning_momentum));
  printf("offsetof(struct fann, connection_rate) = %d\n", offsetof(struct fann, connection_rate));
  printf("offsetof(struct fann, network_type) = %d\n", offsetof(struct fann, network_type));
  printf("offsetof(struct fann, first_layer) = %d\n", offsetof(struct fann, first_layer));
  printf("offsetof(struct fann, last_layer) = %d\n", offsetof(struct fann, last_layer));
  printf("offsetof(struct fann, total_neurons) = %d\n", offsetof(struct fann, total_neurons));
  printf("offsetof(struct fann, num_input) = %d\n", offsetof(struct fann, num_input));
  printf("offsetof(struct fann, num_output) = %d\n", offsetof(struct fann, num_output));
  printf("offsetof(struct fann, weights) = %d\n", offsetof(struct fann, weights));
  printf("offsetof(struct fann, connections) = %d\n", offsetof(struct fann, connections));
  printf("offsetof(struct fann, train_errors) = %d\n", offsetof(struct fann, train_errors));
  printf("offsetof(struct fann, training_algorithm) = %d\n", offsetof(struct fann, training_algorithm));
  printf("offsetof(struct fann, total_connections) = %d\n", offsetof(struct fann, total_connections));
  printf("offsetof(struct fann, output) = %d\n", offsetof(struct fann, output));
  printf("offsetof(struct fann, num_MSE) = %d\n", offsetof(struct fann, num_MSE));
  printf("offsetof(struct fann, MSE_value) = %d\n", offsetof(struct fann, MSE_value));
  printf("offsetof(struct fann, num_bit_fail) = %d\n", offsetof(struct fann, num_bit_fail));
  printf("offsetof(struct fann, bit_fail_limit) = %d\n", offsetof(struct fann, bit_fail_limit));
  printf("offsetof(struct fann, train_error_function) = %d\n",
         offsetof(struct fann, train_error_function));
  printf("offsetof(struct fann, train_stop_function) = %d\n", offsetof(struct fann, train_stop_function));
  printf("offsetof(struct fann, callback) = %d\n", offsetof(struct fann, callback));
  printf("offsetof(struct fann, user_data) = %d\n", offsetof(struct fann, user_data));
  printf("offsetof(struct fann, cascade_output_change_fraction) = %d\n",
         offsetof(struct fann, cascade_output_change_fraction));
  printf("offsetof(struct fann, cascade_output_stagnation_epochs) = %d\n",
         offsetof(struct fann, cascade_output_stagnation_epochs));
  printf("offsetof(struct fann, cascade_candidate_change_fraction) = %d\n",
         offsetof(struct fann, cascade_candidate_change_fraction));
  printf("offsetof(struct fann, cascade_candidate_stagnation_epochs) = %d\n",
         offsetof(struct fann, cascade_candidate_stagnation_epochs));
  printf("offsetof(struct fann, cascade_best_candidate) = %d\n",
         offsetof(struct fann, cascade_best_candidate));
  printf("offsetof(struct fann, cascade_candidate_limit) = %d\n",
         offsetof(struct fann, cascade_candidate_limit));
  printf("offsetof(struct fann, cascade_weight_multiplier) = %d\n",
         offsetof(struct fann, cascade_weight_multiplier));
  printf("offsetof(struct fann, cascade_max_out_epochs) = %d\n",
         offsetof(struct fann, cascade_max_out_epochs));
  printf("offsetof(struct fann, cascade_max_cand_epochs) = %d\n",
         offsetof(struct fann, cascade_max_cand_epochs));
  printf("offsetof(struct fann, cascade_min_out_epochs) = %d\n",
         offsetof(struct fann, cascade_min_out_epochs));
  printf("offsetof(struct fann, cascade_min_cand_epochs) = %d\n",
         offsetof(struct fann, cascade_min_cand_epochs));
  printf("offsetof(struct fann, cascade_activation_functions) = %d\n",
         offsetof(struct fann, cascade_activation_functions));
  printf("offsetof(struct fann, cascade_activation_functions_count) = %d\n",
         offsetof(struct fann, cascade_activation_functions_count));
  printf("offsetof(struct fann, cascade_activation_steepnesses) = %d\n",
         offsetof(struct fann, cascade_activation_steepnesses));
  printf("offsetof(struct fann, cascade_activation_steepnesses_count) = %d\n",
         offsetof(struct fann, cascade_activation_steepnesses_count));
  printf("offsetof(struct fann, cascade_num_candidate_groups) = %d\n",
         offsetof(struct fann, cascade_num_candidate_groups));
  printf("offsetof(struct fann, cascade_candidate_scores) = %d\n",
         offsetof(struct fann, cascade_candidate_scores));
  printf("offsetof(struct fann, total_neurons_allocated) = %d\n",
         offsetof(struct fann, total_neurons_allocated));
  printf("offsetof(struct fann, total_connections_allocated) = %d\n",
         offsetof(struct fann, total_connections_allocated));
  printf("offsetof(struct fann, quickprop_decay) = %d\n", offsetof(struct fann, quickprop_decay));
  printf("offsetof(struct fann, quickprop_mu) = %d\n", offsetof(struct fann, quickprop_mu));
  printf("offsetof(struct fann, rprop_increase_factor) = %d\n",
         offsetof(struct fann, rprop_increase_factor));
  printf("offsetof(struct fann, rprop_decrease_factor) = %d\n",
         offsetof(struct fann, rprop_decrease_factor));
  printf("offsetof(struct fann, rprop_delta_min) = %d\n", offsetof(struct fann, rprop_delta_min));
  printf("offsetof(struct fann, rprop_delta_max) = %d\n", offsetof(struct fann, rprop_delta_max));
  printf("offsetof(struct fann, rprop_delta_zero) = %d\n", offsetof(struct fann, rprop_delta_zero));
  printf("offsetof(struct fann, sarprop_weight_decay_shift) = %d\n",
         offsetof(struct fann, sarprop_weight_decay_shift));
  printf("offsetof(struct fann, sarprop_step_error_threshold_factor) = %d\n",
         offsetof(struct fann, sarprop_step_error_threshold_factor));
  printf("offsetof(struct fann, sarprop_step_error_shift) = %d\n",
         offsetof(struct fann, sarprop_step_error_shift));
  printf("offsetof(struct fann, sarprop_temperature) = %d\n", offsetof(struct fann, sarprop_temperature));
  printf("offsetof(struct fann, sarprop_epoch) = %d\n", offsetof(struct fann, sarprop_epoch));
  printf("offsetof(struct fann, train_slopes) = %d\n", offsetof(struct fann, train_slopes));
  printf("offsetof(struct fann, prev_steps) = %d\n", offsetof(struct fann, prev_steps));
  printf("offsetof(struct fann, prev_train_slopes) = %d\n", offsetof(struct fann, prev_train_slopes));
  printf("offsetof(struct fann, prev_weights_deltas) = %d\n", offsetof(struct fann, prev_weights_deltas));
  printf("offsetof(struct fann, scale_mean_in) = %d\n", offsetof(struct fann, scale_mean_in));
  printf("offsetof(struct fann, scale_deviation_in) = %d\n", offsetof(struct fann, scale_deviation_in));
  printf("offsetof(struct fann, scale_new_min_in) = %d\n", offsetof(struct fann, scale_new_min_in));
  printf("offsetof(struct fann, scale_factor_in) = %d\n", offsetof(struct fann, scale_factor_in));
  printf("offsetof(struct fann, scale_mean_out) = %d\n", offsetof(struct fann, scale_mean_out));
  printf("offsetof(struct fann, scale_deviation_out) = %d\n", offsetof(struct fann, scale_deviation_out));
  printf("offsetof(struct fann, scale_new_min_out) = %d\n", offsetof(struct fann, scale_new_min_out));
  printf("offsetof(struct fann, scale_factor_out) = %d\n", offsetof(struct fann, scale_factor_out));

Which gave me this output:

sizeof(fann)=276
offsetof(struct fann, errno_f) = 0
offsetof(struct fann, error_log) = 4
offsetof(struct fann, errstr) = 8
offsetof(struct fann, learning_rate) = 12
offsetof(struct fann, learning_momentum) = 16
offsetof(struct fann, connection_rate) = 20
offsetof(struct fann, network_type) = 24
offsetof(struct fann, first_layer) = 28
offsetof(struct fann, last_layer) = 32
offsetof(struct fann, total_neurons) = 36
offsetof(struct fann, num_input) = 40
offsetof(struct fann, num_output) = 44
offsetof(struct fann, weights) = 48
offsetof(struct fann, connections) = 52
offsetof(struct fann, train_errors) = 56
offsetof(struct fann, training_algorithm) = 60
offsetof(struct fann, total_connections) = 64
offsetof(struct fann, output) = 68
offsetof(struct fann, num_MSE) = 72
offsetof(struct fann, MSE_value) = 76
offsetof(struct fann, num_bit_fail) = 80
offsetof(struct fann, bit_fail_limit) = 84
offsetof(struct fann, train_error_function) = 88
offsetof(struct fann, train_stop_function) = 92
offsetof(struct fann, callback) = 96
offsetof(struct fann, user_data) = 100
offsetof(struct fann, cascade_output_change_fraction) = 104
offsetof(struct fann, cascade_output_stagnation_epochs) = 108
offsetof(struct fann, cascade_candidate_change_fraction) = 112
offsetof(struct fann, cascade_candidate_stagnation_epochs) = 116
offsetof(struct fann, cascade_best_candidate) = 120
offsetof(struct fann, cascade_candidate_limit) = 124
offsetof(struct fann, cascade_weight_multiplier) = 128
offsetof(struct fann, cascade_max_out_epochs) = 132
offsetof(struct fann, cascade_max_cand_epochs) = 136
offsetof(struct fann, cascade_min_out_epochs) = 140
offsetof(struct fann, cascade_min_cand_epochs) = 144
offsetof(struct fann, cascade_activation_functions) = 148
offsetof(struct fann, cascade_activation_functions_count) = 152
offsetof(struct fann, cascade_activation_steepnesses) = 156
offsetof(struct fann, cascade_activation_steepnesses_count) = 160
offsetof(struct fann, cascade_num_candidate_groups) = 164
offsetof(struct fann, cascade_candidate_scores) = 168
offsetof(struct fann, total_neurons_allocated) = 172
offsetof(struct fann, total_connections_allocated) = 176
offsetof(struct fann, quickprop_decay) = 180
offsetof(struct fann, quickprop_mu) = 184
offsetof(struct fann, rprop_increase_factor) = 188
offsetof(struct fann, rprop_decrease_factor) = 192
offsetof(struct fann, rprop_delta_min) = 196
offsetof(struct fann, rprop_delta_max) = 200
offsetof(struct fann, rprop_delta_zero) = 204
offsetof(struct fann, sarprop_weight_decay_shift) = 208
offsetof(struct fann, sarprop_step_error_threshold_factor) = 212
offsetof(struct fann, sarprop_step_error_shift) = 216
offsetof(struct fann, sarprop_temperature) = 220
offsetof(struct fann, sarprop_epoch) = 224
offsetof(struct fann, train_slopes) = 228
offsetof(struct fann, prev_steps) = 232
offsetof(struct fann, prev_train_slopes) = 236
offsetof(struct fann, prev_weights_deltas) = 240
offsetof(struct fann, scale_mean_in) = 244
offsetof(struct fann, scale_deviation_in) = 248
offsetof(struct fann, scale_new_min_in) = 252
offsetof(struct fann, scale_factor_in) = 256
offsetof(struct fann, scale_mean_out) = 260
offsetof(struct fann, scale_deviation_out) = 264
offsetof(struct fann, scale_new_min_out) = 268
offsetof(struct fann, scale_factor_out) = 272

[alaa-alawi]

Thanks for the answer. it is dense with information. really appreciate it. I'll keep digging until I find the issue and report back.
Another question, is their any disadvantage of using the VS memory layout facility for structures as it gives the same result of printf + sizeof?

[blairmcg]

Using the VS memory layout would be less effort, but it's new(ish) and I just like the old school certainty of correctness of using results from the actual compiler. Very likely the VS provided layout is correct, but I don't know how that feature is implemented so I don't completely trust it.