Plugin Interface

[Ron]

Dear plugin developers,

As you might know I'm currently working on Girder 3.2, more specifically the plugin mechanism. The current way of doing things with plugins is getting old and messy, updating it is prooving to be a major headache with lots of traps and problems. I am in favour of completely redesigning the structure. This redesign will be in such a way that you can easily modify your current plugins to the new architecture.

The major goal of the redesign is 1 plugin type that can serve as an event generator and receiver ( hardware plugin / event plugin ). For the major part this will only mean that your plugins will need a few function name changes. For the plugins that are available in source code form I offer to do this myself ( maybe with a little help of the original author ;-) )

However I do not want to do this if there is a strong opposition to this from the developers themselfs.

I designed the base of the current mechanism three years ago, but I never envisioned that Girder would become this popular and multifunctional. I have been able to keep it going with some minor updates but I feel the time has come to throw out the old crap and give it a fresh shine.

I hope that you feel the same and want to spend a little time on updating your plugins!

The first draft is here
http://www.girder.nl/devel32/api32.php

The new girder.h is here
http://www.girder.nl/devel/girder.h

[Ron]

Okay here is a first draft

(removed)

If you have better names or suggestions share them here.

[Ron]

Ah of course we need a learn_event export, that slipped my mind, I'll update the data above.

I think we are pretty much stuck with device numbers, I am going to include a list in Girder so girder has a text description of a missing plugin in the future. What is the drawback of plugin numbers ? ( aside from a limited number ( well limited ;-) )

I'm not sure we should do away with most exports except for info_plugin, I use them to determine what kind of plugin it is. e.g. I use it to determine if it should appear on the plugin list for the commands. Of course there are other ways to determine this... not sure about this, let me sleep about that :-)

About the gui thing indeed this one doesn't need the pointer to the command structure, as long as it has been updated before the GUI is called of course.

[Ron]

As a side note, if there are binary only plugins that I/we cannot retrace the author of I could think about a wrapper .dll to load the old plugins.

A second tool that we also need is a gml devicenumber actionid translator.

[Ron]

Okay here is draft 2 of the API. I have not done any coding with this so anyone feel free to suggest changes.
(removed)

[Ron]

Actually trigger command should not allow for the payload. The payload is something that travels through girder along with an event. The payload persists until the event has ended, thus if a plugin uses trigger_command from the correct thread Girder will still have the original payload that came along with the event that triggered the plugin. ( is this clear ?)

If you want to send a modified payload the plugin will have to send an event.

I was going to assign new plugin ids because we now have overlapping hardware and action id's. Thus we need a tool that translates our existing .gml files to the new numbers. Preventing people from having to relearn all the events affected and the action plugins that are affected.

[Ron]

The conversion is coming along pretty fine, the UIR plugin is already working. One thing though, the keyboard plugin, or any plugin that uses Hooks is gonna be problematic :( the hook is called into the context of the foreground application, so the plugin cannot directly call Girder through send_event . No fair! I really want to kick the windows messages out of Girder, lets see how I can work around this.

[Ron]

Ah but suggestion are always welcome :-)

About the context problem, I was indeed leaning towards fixing this inside the plugin instead of from within Girder.

I've got the new system implemented for 99% now. Its working _very_ smoothly ( al least for completely new code i'm getting no crashes, I'm amazed :P ) I started around 1pm today its now 11 pm, that is a 10 hour straight coding session :roll: The new code feels and looks good, I'm going to make a few more modifications to the naming of the API tomorrow but overall I'm pretty happy with things as they are. I can also report that porting the plugins is a breeze. I hope to release a new snapshot/alpha release later this weekend.

[Ron]

I've updated the API draft. If anyone thinks his favourite function/feature is missing speak up now.

(removed)

[Ron]

Thanks its corrected

Draft 6 of the API is here

(removed)

the sparkling new header file is here

(removed)

[Ron]

Hi Vincent, welcome to the forum!

I'm still working on the API documentation so I'm afraid this is all we have right now. I can send you the converted UIR plugin to serve as an example.

You ask about the definition of info_plugin, seems that I named it incorrectly somewhere in my posts. It should have been gir_info. The declaration is in the api.html. If your plugin works without it in the old API simply return GIR_TRUE when Girder calls this function. It will contain messages about powermanagement etc.

send_event works as follows

send_event("myirstringhere", NULL,0,YOURDEVICENUM);

It replaces the Postmessage in the old API definition. Don't worry about the payload.

Feel free to ask! Thats why the forum is here

[Ron]

Either way is fine by me. So that only leaves the speed factor. If I implement this in Girder we loose 1 function call but gain 1 conditional operation. Seems like it doesn't matter at all. I'll make it optional. I cannot think of any other problems aside from aesthetics.

[Ron]

I would agree if these where global variables, but every plugin has their own copy of these structures. So it would only be Girder and the plugin that might conflict. I basically declared the API stable, so I'll need to declare API 2 already :( unless vlinders reads this and doesn't mind yet another change.

I'm looking into the critical section and the passing of handles from delphi to c++... should work just fine i guess.

out of curiosity, what is the fundamental difference between a mutex and a critical section ? Okay found it :


What’s the difference between a critical section and a mutex?
A mutex is a OS kernel object, and can thus be used across process boundaries. A critical section is limited to the process in which it was created

[Ron]

I'll put it on the email.

[Ron]

I'll move gir_info to the optional exports too. Most plugins won't care about the stuff that goes over that line.

[Ron]

Hmm, no that would not affect your plugin in the least. Just be sure to update the girder.h at some point. If you ever decide to use the command structure you might get a nasty surprise.

[Ron]

I need to think about this new revelation. I didn't realize there were so many copies of these structures floating around. You can ignore my previous comments as the ramblings of the un-informed.

Ah but how could you know, the last time we talked about this was before the new plugin interface, since then a lot has changed including this.

Thanks for the explanation btw!

The Critical_section stuff is implemented :-)

I've made major changes to Girder today, I'm now able to start and stop the VCL ( borlands MFC ) at my wish, thus getting yet another step closer to the service implementation. (REALLY close now, the next step is actually adding the service code)

Girder now runs without the UI loaded!!!! whoooweeee. I'll try to release a new version tomorrow. If I don't get any crashes that is :-)

[Ron]

Its implemented as you suggested. The command structure now includes (starting alpha 15) a critical_section variable.

Btw, who needed this NT-service stuff, windows wasn't designed to support Girder type of services (not to mention all the plugin problems). pfff. I have it more or less working but I'm not happy about it. I'll see if it stays in.

[Ron]

Even if it horribly breaks your osdmenu ( which it will ) ?

[Ron]

I know Mark, if I'm going to keep it in Girder it will be with the express understanding that the plugin authors are not to be bother with it. Its just too much trouble for the benefits.

[Ron]

Thanks, glad you like it, I'm not quite satisfied, but it's a good start. I've updated the info on the run_parser part.

[Ron]

Yeah Pass the plugin number, Girder needs it. Nothing changed otherwise in this command.

[Mark F]

I look forward to it!

It sounds like the basic architecture (multiple entrypoints in a .dll) will remain the same but some of the overlapping function between a HW and Action plugin will be removed or refined.

As a side note, if you want Girder to run as a service (in the future), you might want to think about any needed changes to the UI portion of the plugin interface as part of this change.

[Mark F]

I'll need to think about this some more but at first glance this looks like a great start.

A couple of observations:

I think you need to work on some groupings. For example, even though the start_plugin() and stop_plugin() interfaces are optional, if I include start_plugin(), I also MUST include stop_plugin().

Also I question the need to pass a command pointer through the command_gui_plugin() interface UNLESS this could be a different command pointer than the one passed through the command_changed_plugin() interface. These should be grouped (both supplied or neither) and when the user clicks on a different command in the tree, all the relevent plugins get the changed notification. Any that are not showing their GUI will save the pointer and return and any that are showing their GUI can save the pointer and update the fields accordingly.

I believe we still need an optional "learn event" interface. It is difficult to get some hardware to emit a specific event when you really need it.

Upon further review:

I was hoping we could move away from plugin or device numbers being compiled into each plugin. I don't have a suggestion for a better identification mechanism though. :(

Upon even further review: :)

The info_plugin() interface looks REALLY interesting. I was wondering if all the other interfaces would be needed with such a generic interface available? The plugin would be an enormous switch statement (like most Windows programs) but all of the other interfaces could be message handlers.

I have to stop babbling and get back to work now. ;)

[Mark F]

The only current drawback to plugin numbers is they need to be administered. You have to hand them out for every new plugin.

I always thought moving to something along the lines of GUIDs would make this easier. Runtime mapping functions (in the APIfunctions) could make using these monsterous things a bit more managable.


// this
int devicenum_plugin(); // the device number
// could become this
GUID* deviceguid_plugin(); // the device GUID

// in APIfunctions

// map a GUID to an int
// if the mapping already exists, return the previous mapping
// if the mapping doesn't exist, add a new GUID mapping
int GUIDToGirder(GUID* pGUID);

// return the GUID this int represents
// if it exists, returns a mapping
// if it doesn't exist, returns NULL
GUID* GirderToGUID(int i);

Plugins would continue to use INTs as the plugin ID but they would need to get the mapping from Girder (probably during the open_plugin() call) using the above GUIDToGirder() interface every time they are opened.

The .GML file could contain the GUID and plugin name string in the command OR it could have an INT in the command and a mapping table/section somewhere else in the file.

When the .GML file is loaded or imported, Girder would reconcile the mappings with any it already has and add any new mappings it doesn't have. This would allow the "Softplugin XXXX not found" message to have the name of the plugin it needs even if you have never heard of it.

I'm just thinking out loud. I'm pretty sure this would cause other problems that I haven't thought about.

Leaving the device numbers as INTs will also work. :)

The single entry point interface was a silly idea. It WOULD be more trouble than it is worth and would probably be slower than the current interfaces. :(

[Mark F]

I had forgotten about trigger_command(). :) I think trigger_command() needs to allow payload data (optionally NULL) to be passed with the command.

One more new APIFunction suggestion:


// use this to communicate directly between plugins
int other_plugin(int pluginID, int message, int wparam, int lparam)
// this interface calls the plugin with ID pluginID at the info_plugin() entry point.


I really like the wrapper .dll idea.


A second tool that we also need is a gml devicenumber actionid translator.
I'm a bit slow this morning, what does this mean?

[Mark F]

Aaaaah! Now I see. Thanks for the gentle clarification. ;)

[Mark F]

You didn't ask for suggestions, opinions or help but I can't seem to keep my hands off the keyboard. :)


the hook is called into the context of the foreground application, so the plugin cannot directly call Girder through send_event.
I think what you are describing is a PLUGIN problem. It would be up to the plugin developer to handle the case where they are in the wrong context to call Girder.

Since there is more than one plugin that needs this type of mechanism, maybe the compatibility/wrapper plugin could be used for this? It will have a Windows message queue/pump and a shared memory segment for the G3.0 plugins. That will allow cross context boundary messaging.

[Mark F]

A clarification, not a change to function:

Under "Type Definitions and structures" is
typedef int (WINAPI *t_send_event) (PCHAR eventstring, void *payload, int len, int device);

Under "The Support functions API (Feedback)" is
int send_event(char *eventstring, void *payload, int len);

The second is missing the device. :)

I'll start porting Logger, Popup, Serial and UserEvent tonight.

[Mark F]

If a plugin wants to send an event to Girder, it must export the gir_compare() interface. This seems wasteful and is not obvious.

I'd guess that 99.99% :) of plugins have events that are deterministic. This means all input plugins must export an interface so the .01% can support non-deterministic events.

Why not have a default gir_compare() routine in Girder (strcmp()) and only use the plugin's routine when it exists?

[Mark F]

I'm sorry this is so late but I'm becoming concerned with the way commands are updated in the tree. I know we just went through a round of fixes that ended with two command structures (one for the execution path and one for the display path) but the structure for the display path still bothers me.

There is nothing that stops multiple threads from accessing the exact same memory at the exact same time. I think we need to add a critical section structure to the command structure. The processing in a plugin could work like this:


// save function structure
m_Girder_fn = *api_functions;
...

// save command data pointer
m_curPointer = command;
...

// update command data via the pointer
EnterCriticalSection(&m_curPointer->CriticalSection);
// update plugin specific data
m_curPointer->actiontype = PLUGIN_ID;
...
// tell Girder about the change
m_Girder_fn.set_command(m_curPointer);
// leave the critical section
LeaveCriticalSection(&m_curPointer->CriticalSection);


A plugin or Girder shouldn't block while holding a critical section.
A plugin or Girder should aquire the critical section before updating anything in the structure.

I use a mutex to make my shared data areas thread safe, it make sense for Girder to do the same.

Thoughts?

[Mark F]

I need to think about this new revelation. I didn't realize there were so many copies of these structures floating around. You can ignore my previous comments as the ramblings of the un-informed. :)


How would I pass the mutex/criticalsection ? A name or a handle ?
That is the beauty of a critical section AKA a RAM semaphore. You don't open one, you initialize it. It is a data structure that exists in the data area YOU define for it. To add one for controlling access to a shared memory structure, it is easiest to add it to the shared memory structure.

For example:

typedef struct s_command {
CRITICAL_SECTION critsec;
PCHAR name;
int actiontype;
int actionsubtype;
PCHAR svalue1;
PCHAR svalue2;
PCHAR svalue3;
int bvalue1;
int bvalue2;
int bvalue3;
int ivalue1;
int ivalue2;
int ivalue3;
int lvalue1;
int lvalue2;
int lvalue3;
void *binary;
int size;
} t_command, *p_command;

t_command MyCommand;

Girder would use InitializeCriticalSection(&MyCommand.critsec) before passing a pointer (p) to the command structure to a plugin. The plugin or Girder can then use the critical section (EnterCriticalSection(&p->critsec) and LeaveCriticalSection(&p->critsec)) without doing anything further to prepare the semaphore.

I will be replacing the mutex semaphores in my plugins with these as I don't need to share the semaphores across process address spaces.

EDIT: This is what happens when I start a post and go to meetings for 3 hours and then finish the post. :) You already have your answers.

[Mark F]

It sounds like you are making good progress. :)

When you said
The Critical_section stuff is implementeddoes this mean we should prepare for this to be in the command structure or you re-did the mutexes that Girder uses internally? Sorry, I'm a bit slow this morning.

I re-compiled my plugins with the new definitions of true and false and ran a very quick test this morning on 3.2a14. All worked. They need much more extensive testing but so far so good.

[Mark F]

Yeah, it's going to be BAD.

From the MS support site (http://support.microsoft.com/default.aspx?scid=kb;EN-US;q238721)

Problems Using Microsoft Foundation Class Library Within Services

The Microsoft Foundation Class (MFC) library loads and initializes the Comctl32.dll file when an MFC dialog is initialized; this occurs even when no common control is used within the dialog. In addition, the MFC library itself uses global atoms to store data for subclassing windows. For these reasons, you should not use MFC within a service. For additional information, click the article number below to view the article in the Microsoft Knowledge Base:
Q164166 (http://support.microsoft.com/default.aspx?scid=kb;en-us;Q164166) PRB: Assert in Wincore.cpp with MFC in a Windows NT Service

All of my plugins use MFC. :(

[Mark F]

When you get a chance, could you update the doc with some information about the run_parser() interface? Specifically, the return code(s) and value(s) in the error_value variable. Thanks. :)

By the way, I *really* like the on-line document (http://www.girder.nl/devel32/api32.php).

[Mark F]

Thanks Ron. :)

[mattwire]

If improvements are necessary to the plugin structure then better to do it now rather than be stuck with the old forever!
Best to do it with a major release of course (ie 3.2).
Also, I agree with Mark about implementing any changes needed for service processes now.

[mattwire]

Congratulations Ron for separating the UI!
I didn't suggest it but I will find it very useful to have girder as a service. :)

[mattwire]

yep! I'm gonna rewrite most of it anyway...

[Snakebite]

Hi,

did the target_enum function change? Before, the first parameter was the command and now I should pass in my plugin id-number? But the target specification is still unique to the command, right?

[vlinders]

This is my first post here, sorry if it seems silly.

Where is the documentation about the new 3.2 API.
The api.html page is a little too terse for me.


It also adds a new export info_plugin()

Err... where is the declaration of this function ?

Ditto for open_plugin...

How works the send_event ?

[vlinders]

Ron, you caught me just in time before I changed my plugin :roll:
Take your time to finalize the API, I'll make all the changes at once.

Changing subject, is it possible to have source code for TaskSwitch ?

[vlinders]

@Ron, by the way, my plugin is not using the command structure, so the impact should ntot be very big on me, right ?