First off, I am developing custom c functions for PostgreSQL 8.3 in
Windows (sorry... not my choice). It appears that there is some
underlying problem in the server that is causing a runtime error to
crash the server when I use palloc. For example... I run the
following:

PG_FUNCTION_INFO_V1(add_one);
Datum
add_one(PG_FUNCTION_ARGS)
{
int32 arg = PG_GETARG_INT32(0);
PG_RETURN_INT32(arg + 1);
}

and it works just fine. NOTE: this is directly from the tutorial
directory. Now, when I add a palloc statement in there it crashes:


PG_FUNCTION_INFO_V1(add_one);
Datum
add_one(PG_FUNCTION_ARGS)
{
int32 arg = PG_GETARG_INT32(0);
palloc(sizeof(Point));
PG_RETURN_INT32(arg + 1);
}

Running "SELECT add_one(1);" crashes the server. The logs show this:

2008-05-03 21:29:08 MDT LOG: server process (PID 3008) was terminated
by exception 0xC0000005
2008-05-03 21:29:08 MDT HINT: See C include file "ntstatus.h" for a
description of the hexadecimal value.
2008-05-03 21:29:08 MDT LOG: terminating any other active server processes
2008-05-03 21:29:08 MDT LOG: all server processes terminated; reinitializing
2008-05-03 21:29:09 MDT FATAL: pre-existing shared memory block is still in use
2008-05-03 21:29:09 MDT HINT: Check if there are any old server
processes still running, and terminate them.

I also get the exact same error when I try to run either the copytext
or concat_text functions from the same funcs_new.c file in the
tutorial directory. This essentially means that I cannot write any
UDFs that require memory allocation or text parameters. Heron seems to
be experiencing the same thing. Is this a bug in 8.3? Can anyone help?

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Nathan Thatcher wrote:
> First off, I am developing custom c functions for PostgreSQL 8.3 in
> Windows (sorry... not my choice). It appears that there is some
> underlying problem in the server that is causing a runtime error to
> crash the server when I use palloc.

[Assuming you're using the stock 8.3.1 binaries and MSVC++8.0 or the
Express Edition of the same]:

Are you 100% certain you are using the same compiler and version that
PostgreSQL was built with, and that you have linked to the *exact* same
runtime library that the backend was linked to? These issues seem to be
a major cause of all sorts of fun problems especially for apps that
originated in the happier, saner UNIX world.

PostgreSQL probably doesn't put in the absurd effort required to avoid
freeing memory in a different DLL to where it was allocated[*]. Having
origins in nice sane UNIX environments where everybody just uses the
same compiler and runtime library it probably does not, this is quite
understandable. However, if I'm right about that (I have not verified
it) then you MUST link to the exact same runtime library that the
backend was linked to or things WILL break. Using the same version of
the same compiler would also be a good idea.

You'll also have to use a dynamic version of the runtime. Linking to a
static version will not work properly, because each DLL will have its
own private memory allocator.

Use Dependency Walker (depends.exe, from http://dependencywalker.com/)
to examine the names and resolved paths of the linked libraries. If you
are linking to a different runtime to the backend, or to multiple
runtimes, things are highly likely to explode.

The stock 8.3.1 win32 binaries appear to be linked to a copy of
MSVCR80.DLL in the WinSxS library tree. That means that you must link
only to MSCVR80.DLL (via the import library msvcrt.lib), which is the
multi-threaded non-debug dynamic runtime for Visual C++ 8. Make sure
you're using the same copy in the same WinSxS location, not a subtly
different version (say, from a VC++8.0 beta).

If your library/extension is linked to any other runtimes, like
msvcr80d.dll or msvcm80.dll, things will probably break.

If your library links to LIBCMT.LIB or LIBCMTD.LIB (note: these may not
appear in dependency walker as they're static libraries; check your
linker command line) things will CERTAINLY break.

That goes for MinGW, too. Do not expect a shared library compiled by
MinGW to load and run happily in an executable built with MSVC++ unless
all the public interfaces are very careful about how they handle memory
allocation, ownership and deallocation. Either build Pg with MinGW too
(if that's supported or even possible) or preferably just get the Visual
C++ Express Edition 8.0 from MS and use that to build your extension.

BTW: Rather than specifying the import library for the C library
directly as a linker argument, use the /MD flag in the linker options or
set it in the project file. If you are not using VC++ 8.0 it's not as
simple as just using /MD ; you might be able to fudge it with
/NODEFAULTLIB but I really do not know.

See:
http://msdn.microsoft.com/en-us/libr...yh(VS.80).aspx
for some information on the surprising variety of runtime libraries
provided by MSVC++ 8.0 alone.


> I also get the exact same error when I try to run either the copytext
> or concat_text functions from the same funcs_new.c file in the
> tutorial directory. This essentially means that I cannot write any
> UDFs that require memory allocation or text parameters. Heron seems to
> be experiencing the same thing. Is this a bug in 8.3? Can anyone help?

Most text handling involves memory allocations and deallocations. I
would not be at all surprised if your problem was mismatched compilers
or runtimes.



* Yes, there's more to it than just avoiding freeing memory in a
different DLL than it was allocated in. File descriptors, for example,
need similar treatment, as do all sorts of other weird little corners
that aren't worth enumerating.

--
Craig Ringer

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Nathan Thatcher wrote:
> First off, I am developing custom c functions for PostgreSQL 8.3 in
> Windows (sorry... not my choice). It appears that there is some
> underlying problem in the server that is causing a runtime error to
> crash the server when I use palloc. For example... I run the
> following:
>
Sorry, I forgot to to ask in my previous post:

Can you send me a complete, compileable example that illustrates the
problem, along with instructions on how to reproduce it? If possible,
send a compiled binary and any dependencies too. I'm curious to see if
this is in fact shared library issues.

--
Craig Ringer

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Craig Ringer wrote:
> Can you send me a complete, compileable example that illustrates the
> problem, along with instructions on how to reproduce it? If possible,
> send a compiled binary and any dependencies too. I'm curious to see if
> this is in fact shared library issues.

I'll send you a .zip file separately with the actual code, but here are
the steps I took:

- Install Postgres 8.3.1
- Install MinGW 5.1.4
- Install the gettext snapshot from MinGW's downloads page under "snapshots"

Then, unzip the zip file I'll send you. Make sure MinGW's bin directory
is in your windows PATH environment variable. In a command prompt, go
to your unzipped directory and run mingw32-make. The makefile I've
included will put the .dll in Postgres' lib/plugins directory (assuming
you installed postgres in its default location - modify the makefile if
that is not the case).

Once the .dll is ready, I created the function in Postgres using the
CREATE FUNCTION command given in the documentation, modified slightly:

CREATE FUNCTION copytext(text) RETURNS text
AS '$libdir/plugins/mylib.dll', 'copytext'
LANGUAGE C STRICT;

Then I used the following SELECT statement:

SELECT copytext(colname) FROM tablename;

where "colname" is a text column in table "tablename". This statement
crashes the database server.

I read somewhere on Postgres' website that the win32 binaries were built
with MinGW - and in fact they must be, since MSVC++ does not have some
of the unix headers needed to build Postgres.

Your theory could very well be true - especially if the issue is with a
different version of gettext. MinGW's website's snapshot is from 2006
or something, so it wouldn't surprise me if Postgres uses a newer version.

A zip file is attached to a separate e-mail to you. Anyone else who
would like a copy may have one, just let me know, but it's basically
just the copytext example from the tutorials pasted into a .c file, and
a makefile.

- Dan

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Dan "Heron" Myers wrote:
> Craig Ringer wrote:
> > Can you send me a complete, compileable example that illustrates
> > the problem, along with instructions on how to reproduce it? If
> > possible, send a compiled binary and any dependencies too. I'm
> > curious to see if this is in fact shared library issues.
>
> I'll send you a .zip file separately with the actual code, but here
> are the steps I took:
>
> - Install Postgres 8.3.1
> - Install MinGW 5.1.4
> - Install the gettext snapshot from MinGW's downloads page under
> "snapshots"

This may be your problem. We have previously seen a lot of issues with
different versions of gettext on Windows. They do a lot of things
in ways that "you are not supposed to on Windows"...

Could you try a build without NLS?


> I read somewhere on Postgres' website that the win32 binaries were
> built with MinGW - and in fact they must be, since MSVC++ does not
> have some of the unix headers needed to build Postgres.

No, as of 8.3 they are build with MSVC.


//Magnus

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Dan "Heron" Myers wrote:
> Craig Ringer wrote:
> > Can you send me a complete, compileable example that illustrates the
> > problem, along with instructions on how to reproduce it? If possible,
> > send a compiled binary and any dependencies too. I'm curious to see if
> > this is in fact shared library issues.
>

> Then I used the following SELECT statement:
>
> SELECT copytext(colname) FROM tablename;
>
> where "colname" is a text column in table "tablename". This statement
> crashes the database server.

I'm certainly seeing a server crash with your DLL.

I've built the example with Visual C++ 9 (I had a to make a couple of
changes, such as defining BUILDING_DLL and exporting the `copytext'
function with PGDLLIMPORT) and I get a backend crash with that too. Of
course, that's also using a different runtime, and it's far from
impossible that I'm doing something wrong in the build process.

I'm having some trouble getting the example building under Visual Studio
Express 2005 ( I had to install it to test, and it's *much* uglier than
2008 ) so I haven't been able to test it properly with a matching runtime.

Looking at Pg's headers, though, I'm going to say that my first guess
about mismatched runtimes is probably wrong. That's often the cause of
plugin/extension problems on win32, but I'm not sure I see what would be
causing a mismatched runtime related failure here. Not with memory
handling, anyway. palloc() is a macro, but it's implemented using
exported functions from the backend, so allocations are actually being
performed with the backend's runtime. No memory appears to be being
freed in the example code, only being handed over to the backend to be
released later.

I'll see if it works when built with VC++ Express 2005 if I can get it
to behave (ie if/when I figure out what stupid thing I've done). I'm not
too sure that'll actually be the issue though.

> Your theory could very well be true - especially if the issue is with a
> different version of gettext. MinGW's website's snapshot is from 2006
> or something, so it wouldn't surprise me if Postgres uses a newer version.

I assume you needed gettext because PostgreSQL's headers expose Pg's use
of libintl to users of the Pg headers, so you needed libintl.h?

As the standard win32 build of Pg is built with gettext, but does not
ship the headers for the version of gettext used, it's a bit tricky to
build external C functions etc. It'd be nice to bundle those headers or
avoid including libintl.h and friends from Pg headers that're likely to
be needed for external C functions.

Ideally only .c sources would include libintl.h where required (but I
know how practical *that* sometimes turns out to be -
*cough*Freetype*cough*).

--
Craig Ringer

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Magnus Hagander wrote:
> Dan "Heron" Myers wrote:

>> I read somewhere on Postgres' website that the win32 binaries were
>> built with MinGW - and in fact they must be, since MSVC++ does not
>> have some of the unix headers needed to build Postgres.
>
> No, as of 8.3 they are build with MSVC.

In case it's of any help, I was able to build the test with VC++ 8 and
confirmed that it does still crash at the palloc() call even when the
exact same runtime is used by both the test DLL and by postgres.exe
(according to depends.exe).

Anyway, the birds are chirping and I have to work way too soon...

--
Craig Ringer

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

So what options does that leave all of us who need to compile and run
our custom C functions in Windows?

On Sun, May 4, 2008 at 4:47 PM, Craig Ringer
<craig@postnewspapers.com.au> wrote:
> Magnus Hagander wrote:
>
>
> > Dan "Heron" Myers wrote:
> >
>
>
> >
> > > I read somewhere on Postgres' website that the win32 binaries were
> > > built with MinGW - and in fact they must be, since MSVC++ does not
> > > have some of the unix headers needed to build Postgres.
> > >
> >
> > No, as of 8.3 they are build with MSVC.
> >
>
> In case it's of any help, I was able to build the test with VC++ 8 and
> confirmed that it does still crash at the palloc() call even when the exact
> same runtime is used by both the test DLL and by postgres.exe (according to
> depends.exe).
>
> Anyway, the birds are chirping and I have to work way too soon...
>
> --
> Craig Ringer
>
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Nathan Thatcher wrote:
> So what options does that leave all of us who need to compile and run
> our custom C functions in Windows?

Debug further :-)

Because it's certainly possible to do it. IIRC, PostGIS for 8.3 is
build with MingW and it works fine. And other extensions are known
working when built with MSVC Express. So it should work in principle,
we just need to figure out exactly what the issue is in this specific
case.

I'm still thinking gettext could be the issue here.

//Magnus

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Nathan Thatcher wrote:
> So what options does that leave all of us who need to compile and run
> our custom C functions in Windows?

After a bit of sleep and with the advantage of a now-working brain I've
got it working, at least with 8.3 .

The problem was with DLL linkage, especially the CurrentMemoryContext
global variable. I was defining BUILDING_DLL and using the PGDLLIMPORT
macro to export the copytext() function - but this was of course causing
everything in the Pg headers to be declared __declspec(dllexport)
instead of __declspec(dllimport) . AFAIK this is survivable for
functions, which will use a slower call thunk instead, but not for
exported variables.

As a result, MemoryContextAlloc(CurrentMemoryContext, (sz)) was being
called with a nonsensical pointer for CurrentMemoryContext and crashing.


To build the C function DLL correctly it correctly do not define the
BUILDING_DLL macro. Instead, define your own DLL export macros like:

#if defined(_MSC_VER) || defined(__MINGW32__)
#define COPYTEXT_EXPORT __declspec (dllexport)
#else
#define COPYTEXT_EXPORT
#endif

then declare your function as:


COPYTEXT_EXPORT Datum copytext2(PG_FUNCTION_ARGS) {
// blah blah
}


It should now be exported in the DLL's interface correctly AND have
correct access to Pg's exported functions and variables.

I did notice that I'm getting some warnings about inconsistent DLL
linkage for Pg_magic_func and pg_finfo_copytext . These really need to
be using __declspec(dllexport) rather than using __declspec(dllimport)
via PGDLLIMPORT .


Maybe it's worth providing a PGMODULEEXPORT macro for PG_MODULE_MAGIC,
PG_FUNCTION_INFO_V1, and for module writers to use to export their
functions in the DLL interface. A module author would have to ensure
that BUILDING_MODULE was defined.

Here's how the PG_MODULE_MAGIC and PG_FUNCTION_INFO_V1 macros would look:

/* This might want to go somewhere other than fmgr.h, like
* pg_config_os.h alongside the definition of PGDLLIMPORT
*/
#if defined(_MSC_VER) || defined(__MINGW32__)
#if defined(BUILDING_MODULE)
#define PGMODULEEXPORT __declspec (dllexport)
#else
// Never actually used
#define PGMODULEEXPORT __declspec (dllimport)
#endif
#else
#define PGMODULEEXPORT
#endif

#define PG_MODULE_MAGIC \
PGMODULEEXPORT Pg_magic_struct * \
PG_MAGIC_FUNCTION_NAME(void) \
{ \
static const Pg_magic_struct Pg_magic_data = PG_MODULE_MAGIC_DATA; \
return &Pg_magic_data; \
} \
extern int no_such_variable

#define PG_FUNCTION_INFO_V1(funcname) \
PGMODULEEXPORT const Pg_finfo_record * \
CppConcat(pg_finfo_,funcname) (void) \
{ \
static const Pg_finfo_record my_finfo = { 1 }; \
return &my_finfo; \
} \
extern int no_such_variable



I've tested this definition and it produces a DLL that links correctly
and does so without the warnings of inconsistent DLL linkage produced by
the original versions (which declared the function __declspec(dllimport)
then defined it).

Anyway, the C function examples need some changes to work correctly on
win32. I've attached updated versions. These redefine the
PG_MODULE_MAGIC and PG_FUNCTION_INFO_V1 macros, but if you omit that
code the module will still built, just with warnings, and will still work.

--
Craig Ringer

/*
* PostgreSQL example C functions.
*
* This file must be built as a shared library or dll and
* placed into the PostgreSQL `lib' directory. On Windows
* it must link to postgres.lib .
*
* postgresql/include/server must be on your header search path.
* With MSVC++ on win32 so must postgresql/include/server/port/win32_msvc .
* With MinGW use postgresql/include/server/port/win32 .
*/

#if defined(_MSC_VER) || defined(__MINGW32__)
#ifndef _USE_32BIT_TIME_T
#define _USE_32BIT_TIME_T
#endif
#endif

/* BUILDING_DLL causes the declarations in Pg's headers to be declared
* __declspec(dllexport) which will break DLL linkage. */
#ifdef BUILDING_DLL
#error Do not define BUILDING_DLL when building extension libraries
#endif

/* Ensure that Pg_module_function and friends are declared __declspec(dllexport) */
#ifndef BUILDING_MODULE
#define BUILDING_MODULE
#endif

#include "postgres.h"
#include <string.h>
#include "fmgr.h"
#include "utils/geo_decls.h"

/*--------------- BEGIN REDEFINITION OF PG MACROS -------------------
*
* These rewritten versions of PG_MODULE_MAGIC and PG_FUNCTION_INFO_V1
* declare the module functions as __declspec(dllexport) when building
* a module. They also provide PGMODULEEXPORT for exporting functions
* in user DLLs.
*/
#undef PG_MODULE_MAGIC
#undef PG_FUNCTION_INFO_V1

/* This might want to go somewhere other than fmgr.h, like
* pg_config_os.h alongside the definition of PGDLLIMPORT
*/
#if defined(_MSC_VER) || defined(__MINGW32__)
#if defined(BUILDING_MODULE)
#define PGMODULEEXPORT __declspec (dllexport)
#else
// Never actually used
#define PGMODULEEXPORT __declspec (dllimport)
#endif
#else
#define PGMODULEEXPORT
#endif

#define PG_MODULE_MAGIC \
PGMODULEEXPORT Pg_magic_struct * \
PG_MAGIC_FUNCTION_NAME(void) \
{ \
static const Pg_magic_struct Pg_magic_data = PG_MODULE_MAGIC_DATA; \
return &Pg_magic_data; \
} \
extern int no_such_variable

#define PG_FUNCTION_INFO_V1(funcname) \
PGMODULEEXPORT const Pg_finfo_record * \
CppConcat(pg_finfo_,funcname) (void) \
{ \
static const Pg_finfo_record my_finfo = { 1 }; \
return &my_finfo; \
} \
extern int no_such_variable

/*--------------- END REDEFINITION OF PG MACROS -------------------*/


PG_MODULE_MAGIC;

PG_FUNCTION_INFO_V1(add_one);

PGMODULEEXPORT Datum
add_one(PG_FUNCTION_ARGS)
{
int32 arg = PG_GETARG_INT32(0);

PG_RETURN_INT32(arg + 1);
}

/* by reference, fixed length */

PG_FUNCTION_INFO_V1(add_one_float8);

PGMODULEEXPORT Datum
add_one_float8(PG_FUNCTION_ARGS)
{
/* The macros for FLOAT8 hide its pass-by-reference nature. */
float8 arg = PG_GETARG_FLOAT8(0);

PG_RETURN_FLOAT8(arg + 1.0);
}

PG_FUNCTION_INFO_V1(makepoint);

PGMODULEEXPORT Datum
makepoint(PG_FUNCTION_ARGS)
{
/* Here, the pass-by-reference nature of Point is not hidden. */
Point *pointx = PG_GETARG_POINT_P(0);
Point *pointy = PG_GETARG_POINT_P(1);
Point *new_point = (Point *) palloc(sizeof(Point));

new_point->x = pointx->x;
new_point->y = pointy->y;

PG_RETURN_POINT_P(new_point);
}

/* by reference, variable length */

PG_FUNCTION_INFO_V1(copytext);

PGMODULEEXPORT Datum
copytext(PG_FUNCTION_ARGS)
{
text *t = PG_GETARG_TEXT_P(0);
/*
* VARSIZE is the total size of the struct in bytes.
*/
text *new_t = (text *) palloc(VARSIZE(t));
SET_VARSIZE(new_t, VARSIZE(t));
/*
* VARDATA is a pointer to the data region of the struct.
*/
memcpy((void *) VARDATA(new_t), /* destination */
(void *) VARDATA(t), /* source */
VARSIZE(t) - VARHDRSZ); /* how many bytes */
PG_RETURN_TEXT_P(new_t);
}

PG_FUNCTION_INFO_V1(concat_text);

PGMODULEEXPORT Datum
concat_text(PG_FUNCTION_ARGS)
{
text *arg1 = PG_GETARG_TEXT_P(0);
text *arg2 = PG_GETARG_TEXT_P(1);
int32 new_text_size = VARSIZE(arg1) + VARSIZE(arg2) - VARHDRSZ;
text *new_text = (text *) palloc(new_text_size);

SET_VARSIZE(new_text, new_text_size);
memcpy(VARDATA(new_text), VARDATA(arg1), VARSIZE(arg1) - VARHDRSZ);
memcpy(VARDATA(new_text) + (VARSIZE(arg1) - VARHDRSZ),
VARDATA(arg2), VARSIZE(arg2) - VARHDRSZ);
PG_RETURN_TEXT_P(new_text);
}



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general