gmane.comp.gis.gdal.devel

Re: Heads-up re: poppler/pdf on Windows

Even Rouault — 2012-05-25T21:34:23

Selon Joaquim Luis <jluis< at >ualg.pt>:


0.18.X and 0.20.X are indeed supported, but 0.19.X not. Odd numbered versionss
of poppler are developement versions, whereas even ones are considered stables.
I initially supported 0.19 in the hope of being reading for 0.20 release, but
there was a API break later in the 0.19 development, and because it is
sufficiently enough difficult to autodetect the version of poppler, I decided it
was not worth the effort to go on supporting it and only support 0.20 instead.

(In case someone would want to blame poppler devs, don't. The GDAL PDF driver
uses an internal low-level API of poppler for its needs, so we are bound to
adapt to the changes that sometimes happen there)

Re: Heads-up re: poppler/pdf on Windows

Even Rouault — 2012-05-25T21:33:19

Selon Joaquim Luis <jluis< at >ualg.pt>:


0.18.X and 0.20.X are indeed supported, but 0.19.X not. Odd numbered versionss
of poppler are developement versions, whereas even ones are considered stables.
I initially supported 0.19 in the hope of being reading for 0.20 release, but
there was a API break later in the 0.19 development, and because it is
sufficiently enough difficult to autodetect the version of poppler, I decided it
was not worth the effort to go on supporting it and only support 0.20 instead.

(In case someone would want to blame poppler devs, don't. The GDAL PDF driver
uses an internal low-level API of poppler for its needs, so we are bound to
adapt to the changes that sometimes happen there)

Re: Heads-up re: poppler/pdf on Windows

Even Rouault — 2012-05-25T21:16:23

Selon Jeff McKenna <jmckenna< at >gatewaygeomatics.com>:


Hi Jeff

Or use trunk or latest state of 1.9 branch ;-) . 1.9 branch has been updated
with poppler 0.20 support just a bit after 1.9.1, but it will be in 1.9.2. See
http://trac.osgeo.org/gdal/ticket/4668 if you need to patch. For 0.20 support
you need to uncomment all the POPPLER_ lines in nmake.opt (including the new
POPPLER_0_20_OR_LATER = YES)

0.18 should work too. At least I've tested it successfully on Linux.

Best regards,

Even

Re: georeference and project a jpeg

jdmorgan — 2012-05-25T16:44:42

I am sending this because I just realized that I posted w/ the wrong 
subject so that it gets filed correctly in the archive:

This is a great listserv!

Chaitanya using a vrt worked so thank for that tip. Also, it must be the
case as Jukka says that img isn't holding the control points. But as
suggested a GeoTiff (and VRT) work fine.

Thanks so much for your help.

Cheers,
Derek


On 5/25/2012 3:00 AM, Chaitanya kumar CH wrote:

Re: Heads-up re: poppler/pdf on Windows

Joaquim Luis — 2012-05-25T14:27:12

Hmm, I have 0.19.0 (it was the SVN version) working but I have restrain 
to rebuild GDAL since I saw the commits saying that one would have to 
update to 0.20 (or so I (mis)understood the commit messages meaning).

Joaquim

Re: gdal-dev Digest, Vol 96, Issue 55

jdmorgan — 2012-05-25T14:09:08

This is a great listserv!

Chaitanya using a vrt worked so thank for that tip. Also, it must be the 
case as Jukka says that img isn't holding the control points. But as 
suggested a GeoTiff (and VRT) work fine.

Thanks so much for your help.

Cheers,
Derek

On 5/25/2012 7:25 AM, gdal-dev-request< at >lists.osgeo.org wrote:
>> â€œThere is no affine transformation and no GCPs.â€

Heads-up re: poppler/pdf on Windows

Jeff McKenna — 2012-05-25T13:40:06

Hopefully this helps someone else down the road...

- the latest version of Poppler that I am able to get working with GDAL
(1.9.1 now) on MSVC (2008) is 0.16.4
- 0.20.0 or git master gives errors
- example of errors: http://pastebin.com/4YSnSJkn

So, stick with 0.16.4 and save yourself a day of testing newer versions.

Have a good weekend everyone.

-jeff

Re: ogr2ogr shapefile to mapinfo tab: no SRS.

Frans Knibbe | Geodan — 2012-05-25T11:25:17

I am trying to understand what is happing with GDAL ticket 481 
(http://trac.osgeo.org/gdal/ticket/481). If I understand correctly, the 
old way was to guess the right Mapinfo SRS definition by string 
matching. This should have been replaced by first trying to match the 
EPSG code. Assuming that all patches have been applied in the version of 
ogr2ogr that I am using, shouldn't just specifying 'a_srs EPSG:28992' be 
sufficient for finding the right Mapinfo SRS specification? In that case 
there should not be a need to try to interpret the prj file, right?

Regards,
Frans


On 2012-05-25 12:23, Luca Sigfrido Percich wrote:

Re: ogr2ogr shapefile to mapinfo tab: no SRS.

Frans Knibbe | Geodan — 2012-05-25T11:11:24

Hello Sig,

I have just tried that. Only with the ESRI WKT, because the OGC WKT 
already had the space between 'Bessel' and '1841'. This change neither 
had the desired effect.

Greetings,
Frans

On 2012-05-25 12:23, Luca Sigfrido Percich wrote:

Re: ogr2ogr shapefile to mapinfo tab: no SRS.

Luca Sigfrido Percich — 2012-05-25T10:23:22

Frans,

try to replace also 
SPHEROID["Bessel_1841" 
with 
SPHEROID["Bessel 1841" in your prj file.

I'm still following the idea that the problem lies with unmatched names
in mitab_spatialref.cpp.

Sig


Il giorno gio, 24/05/2012 alle 14.42 +0200, Frans Knibbe | Geodan ha
scritto:


_____________
PRIVACY
Le informazioni contenute in questo messaggio sono riservate e confidenziali. Il loro utilizzo e' consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora Lei non fosse la persona a cui il presente messaggio è destinato, La invitiamo ad eliminarlo dal Suo Sistema e a distruggere le varie copie o stampe, dandone gentilmente comunicazione all’indirizzo mail del mittente. Ogni utilizzo improprio e' contrario ai principi del D.lgs 196/03 e alla legislazione europea (Direttiva 2002/58/CE).

PRIVACY
Le informazioni contenute in questo messaggio sono riservate e confidenziali. Il loro utilizzo e' consentito esclusivamente al destinatario del messaggio, per

Re: ogr2ogr shapefile to mapinfo tab: no SRS.

Frans Knibbe | Geodan — 2012-05-25T09:55:30

Hello Luca,

Thank you for your suggestion, but it does not seem to work. I tried in 
in different ways: editing the prj file of the shapefile and specifying 
an alternative prj file with the a_srs option. I also tried both ESRI 
WKT and OGC WKT. And I also tried 'DATUM["D_Netherlands_Bessel"' in the 
ESRI WKT case, because it was 'DATUM["D_Amersfoort"' originally. In all 
cases, the SRS in the output is still NonEarth.

Regards,
Frans


On 2012-05-24 19:19, Luca Sigfrido Percich wrote:

Re: georeference and project a jpeg

Jukka Rahkonen — 2012-05-25T08:54:23

     I am attempting to georeference and project a
       jpeg image
       using only gdal tool.  I
       have georeferenced
       the jpeg using the following command:  
     gdal_translate
       -a_srs EPSG:102719 -of HFA -gcp 0.0 0.0
       -82.5586187839508 35.59414007259327 -gcp 0.0 410.0
       -82.55858659744263
       35.5937998255945 -gcp 520.0 0.0 -82.55741715431213
       35.594340730401775 -gcp
       520.0 410.0 -82.55728840827942 35.5940266570877 
NCDCLogo.jpg NCDCLogo.img 
.....
     Derek



I made a test with some holiday picture jpeg and this way it works for me.

Step one
========
gdal_translate -of GTiff -gcp 0 0 -82.5586187839508 35.59414007259327 
-gcp 0 410
-82.55858659744263 35.5937998255945 -gcp 520 0 -82.55741715431213
35.594340730401775 -gcp 520 410 -82.55728840827942 35.5940266570877
 test.jpg
test_temp.tif

Step two
========
gdalwarp -s_srs epsg:102719 -t_srs epsg:102719 test_temp.tif test_final.tif

It seems that
1. Imagine format is not right for holding ground control poi

Re: georeference and project a jpeg

Chaitanya kumar CH — 2012-05-25T07:00:56

Derek,

Please provide the gdalinfo output of the img file.

Also, try this:
Create a vrt file instead of a HFA using the same gdal_translate command.
See if gdalwarp works with the vrt file.

On Fri, May 25, 2012 at 7:17 AM, jdmorgan <jdmorgan< at >unca.edu> wrote:

georeference and project a jpeg

jdmorgan — 2012-05-25T01:47:24

Hello,

I am attempting to georeference and project a jpeg image using only gdal 
tool.I have georeferenced the jpeg using the following command:

gdal_translate -a_srs EPSG:102719-of HFA -gcp 0.0 0.0 -82.5586187839508 
35.59414007259327 -gcp 0.0 410.0 -82.55858659744263 35.5937998255945 
-gcp 520.0 0.0 -82.55741715431213 35.594340730401775 -gcp 520.0 410.0 
-82.55728840827942 35.5940266570877NCDCLogo.jpg NCDCLogo.img

The command run fine and produces a img file however it does not line up 
with similarly project layer per the gcp's.When I check the projection 
information the file shows to be unprojected. Further, when I attempt to 
project the resulting raster with gdalwarp I get the following errors:

"There is no affine transformation and no GCPs."

Any pointers here would be greatly appreciated.

Thanks,

Derek

_______________________________________________
gdal-dev mailing list
gdal-dev< at >lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: memory leak in GRIB reader (with Python bindings)

Even Rouault — 2012-05-24T21:52:35

Le jeudi 24 mai 2012 23:47:14, Chris Barker a écrit :

Yes, there's a config option, GRIB_CACHEMAX, that you can set if necessary 
(value in megabytes, and applies for each dataset). I didn't advertize it in 
the doc for now, as I think there are virtually no use case where you would 
need to alter it. (unless you open at a dozains of huge grib files...)

Re: memory leak in GRIB reader (with Python bindings)

Chris Barker — 2012-05-24T21:47:14

On Thu, May 24, 2012 at 2:40 PM, Even Rouault
<even.rouault< at >mines-paris.org> wrote:

seems reasonable -- is there an API to change that threshold at run
time? Not I think it's needed...


it sounds good to me.

meanwhile, closing the dataset every once in a while (in my case,
every 28 bands read) works great -- topping out at around 180MB memory
use.

Thanks for the hint and the fix.


IIUC, GRIB compresses, so yes, it does make sense to read a whole band at once.


understandable, but sub-optimum!

Thanks,
  -Chris

Re: memory leak in GRIB reader (with Python bindings)

Even Rouault — 2012-05-24T21:40:37

Ok, see http://trac.osgeo.org/gdal/ticket/4682 for a fix. Basically, the 
current caching strategy is maintained (cache all bands that have been 
accessed), until a threshold is reached (arbitrarly set to 100 MB by default). 
When the threshold is reached, then we only cache one band at a time. That 
could be made smarter, but I think this is good enough for now.


I'm not a specialist of the GRIB API, but from what I see, it only returns the 
data for a full band, and not for partial reads. So for example, if you 
accessed one line at a time, and that GDAL didn't do any caching, it would 
mean that GDAL would have to decode the whole band each time. Pretty slow !


The previous strategy didn't cache all the bands, but each band once you have 
you read it once. Obviously, if you read all the bands, then at the end, it 
has cached all the bands ;-) The rationale behind this was that it was the 
best strategy to minimize the number of lines of codes in the GRIB driver ;-)

Re: memory leak in GRIB reader (with Python bindings)

Chris Barker — 2012-05-24T21:18:04

Even,

Thanks so much!


I just tested reading only a subset of teh band (because I don't need
the whole thing), and it used exactly the same amount of memory --
which fits this data model.


maybe -- but what is GDAL policy usually? It doesn't read the data
until you ask for it, and I would have expected to keep copy myself if
want to use it again.


yup.


That would be great -- I can see caching a full band, so you could
pull out pieces efficiently, but caching the entier thing seems like a
bad idea. Even in the single bad case, I'd expect the user to pull the
whole thing if s/he wanted that.


actually, the code already does that (I'm using this tro translate to
another file format, and break it up into smaller chunks). I hadn't
thought to close data set in between -- that will be easy to do.

Thanks for the very fast reply!

-Chris

Re: memory leak in GRIB reader (with Python bindings)

Even Rouault — 2012-05-24T21:06:21

Chris,

ok I reproduce your issue.

The GRIB driver actually caches all the raster data from a band the first type 
you access it, and never releases it. This is to speed-up successive RasterIO 
operations on a band, which is a nice feature generally. But if you iterate 
over all the bands, it means that GDAL will end up allocating (number_of_bands 
* x_size * y_size * sizeof(double) ) bytes. In your case : 1129 * 720 * 360 * 
8 = 2.3 GB indeed.

I'm going to try to find a fix where GDAL wouldn't cache more than XXX bytes 
from a dataset to avoid this situation.

In the meantime, you can perhaps try reworking your algorithm to iterate on a 
limited number of bands (let's say 100) at a time. "At a time" means that 
between each iteration you close and re-open the dataset. (GDAL will recover 
nicely the memory it has cached at dataset closing). Or more simply close and 
re-open each time you process a band (opening time on your dataset doesn't 
seem to be so slow).

Best regards,

Even

Le jeudi 24 mai 2012 21

memory leak in GRIB reader (with Python bindings)

Chris Barker — 2012-05-24T19:55:15

Hi folks,

I"m finding what appears to be a memory leak, using the GRIB reader,
with the python bindings.

What I'm trying to do is read the data one band at a time, then throw
it away and read the next band -- there are 1129 bands in the file at
hand, and I can't hold it all in memory (32 bit still...)

However, when I do this, memory use just keeps climbing.

Should the memory be freed? I would expect so.

I'm using RasterBand.ReadAsArray()

Is this a leak? or is supposed to keep it around in memory?

Either way, is there a way to force it to release that memory (I"m
already doing and exlicite del and gc.collect call, so I dont think
it's a python reference counting issue)

I've enclosed a simpel test script -- watch the memory climb.

The data file is to big (186MB) to enclose here, you can get it here:

http://nomads.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120522/18/time_grib_01/ocnu5.01.2012052218.daily.grb2

If you want to give this a try.


(note -- Grib giving some pretty good compression

Re: ogr2ogr shapefile to mapinfo tab: no SRS.

Luca Sigfrido Percich — 2012-05-24T17:19:59

Hi Frans,

Il giorno gio, 24/05/2012 alle 14.42 +0200, Frans Knibbe | Geodan ha
scritto:


I checked in my c:\program files\Mapinfo\Professional\mapinfow.prj, and
this CRS is listed as follows:

"--- Netherlands Coordinate Systems ---"
"Netherlands National System\p28992", 20, 109, 7, 5.387638889,
52.156160556, 0.9999079, 155000, 463000

so MapInfo knows that the EPSG code 28992 is associated with it, and
this should be matched in mitab_spatialref.cpp as

Projection Type = 20 (SRS_PT_STEREOGRAPHIC)
Datum = 109 (Netherlands_Bessel)
Units = 7 (SRS_UL_METER)

SRS_PP_CENTRAL_MERIDIAN = 5.387638889
SRS_PP_LATITUDE_OF_ORIGIN = 52.156160556
SRS_PP_SCALE_FACTOR = 0.9999079
SRS_PP_FALSE_EASTING = 155000
SRS_PP_FALSE_NORTHING = 463000

Can you try to replace 
DATUM["Amersfoort",
with
DATUM["Netherlands_Bessel"
in the prj file and see if it works?

Sig