#mapnik log: Friday 26, February 2010

2010 | 02

previous | next
00:17:05 *** chad_burt has quit (Quit: Leaving...)
00:59:23 *** StormTide has quit (Quit: Leaving)
01:30:03 *** cgs_bob has quit (Ping timeout: 265 seconds)
02:14:33 *** cgs_bob (~bob@122.135-78-65.ftth.swbr.surewest.net) has joined #mapnik
02:18:07 *** dkb (~dkb@66-219-8-179.ip.gvtel.com) has joined #mapnik
02:18:10 *** dkb has parted #mapnik (None)
02:18:25 *** dkb (~dkb@66-219-8-179.ip.gvtel.com) has joined #mapnik
02:33:31 *** springmeyer has quit (Quit: springmeyer)
02:38:27 *** tcarobruce has quit (Quit: tcarobruce)
02:52:28 *** shoe has quit (Read error: Connection reset by peer)
02:52:32 *** shoe (~user@ip24-250-63-6.ri.ri.cox.net) has joined #mapnik
03:09:54 *** Ldp__ has quit ()
05:03:26 *** shoe has quit (Read error: Connection reset by peer)
05:03:37 *** shoe (~user@ip24-250-63-6.ri.ri.cox.net) has joined #mapnik
05:21:05 *** dkb has quit (Ping timeout: 265 seconds)
05:21:12 *** cgs_bob_ (~bob@122.135-78-65.ftth.swbr.surewest.net) has joined #mapnik
05:21:13 *** dkb (~dkb@66-219-8-179.ip.gvtel.com) has joined #mapnik
05:22:03 *** cgs_bob has quit (Ping timeout: 265 seconds)
05:28:53 *** dkb has quit (Quit: Leaving.)
05:45:07 *** malthe (~user@196.0.26.44) has joined #mapnik
06:32:36 *** malthe_ (~user@196.0.26.43) has joined #mapnik
06:33:21 *** malthe has quit (Ping timeout: 252 seconds)
06:42:46 *** HounD (~HounD@unics1.grfc.ru) has joined #mapnik
06:47:10 *** HounD has quit (Ping timeout: 264 seconds)
06:48:33 *** HounD (~HounD@unics1.grfc.ru) has joined #mapnik
08:17:03 *** harobed_ has quit (Quit: Ex-Chat)
08:17:12 *** harobed (~stephane@pda57-1-82-231-115-1.fbx.proxad.net) has joined #mapnik
09:18:58 *** HounD has quit (Ping timeout: 264 seconds)
09:19:05 *** HounD (~HounD@unics1.grfc.ru) has joined #mapnik
10:26:53 *** Genscher (~Richard@dslb-188-099-095-052.pools.arcor-ip.net) has joined #mapnik
11:27:56 *** ser (~ser@sergiusz.pawlowicz.name) has joined #mapnik
11:32:49 *** ser has quit (Ping timeout: 264 seconds)
11:43:00 *** ser (~ser@sergiusz.pawlowicz.name) has joined #mapnik
11:47:08 *** ser has quit (Ping timeout: 246 seconds)
11:55:32 *** ser (~ser@sergiusz.pawlowicz.name) has joined #mapnik
11:59:20 *** gavinf has quit (Quit: gavinf)
11:59:52 *** gavinf (~gavinf@196.211.2.133) has joined #mapnik
12:15:17 *** gavinf has quit (Quit: gavinf)
12:31:31 *** gavinf (~gavinf@196.211.119.210) has joined #mapnik
12:34:26 *** Ldp__ (~thid@osm.xs4all.nl) has joined #mapnik
12:45:54 *** Phurl_ has quit (Read error: Operation timed out)
12:48:02 *** ajturner (~ajturner@pool-72-66-109-70.washdc.fios.verizon.net) has joined #mapnik
12:48:03 *** Phurl_ (~mdupont@2001:0:53aa:64c:2069:172f:ae2d:1b81) has joined #mapnik
12:51:32 *** Phurl_ has quit (Read error: Operation timed out)
12:55:39 *** Phurl_ (~mdupont@2001:0:53aa:64c:2069:172f:ae2d:1b81) has joined #mapnik
13:02:09 *** ajturner has quit (Quit: ajturner)
13:07:27 *** malthe_ has quit (Ping timeout: 276 seconds)
13:16:13 *** luneff (~yury@93.178.82.56) has joined #mapnik
13:34:55 *** dkb (~dkb@66-219-8-179.ip.gvtel.com) has joined #mapnik
13:34:58 *** dkb has quit (Client Quit)
13:35:00 *** dkb (~dkb@66-219-8-179.ip.gvtel.com) has joined #mapnik
13:58:35 *** kredik_ (~chatzilla@ANice-551-1-138-77.w92-143.abo.wanadoo.fr) has joined #mapnik
13:59:16 *** kredik has quit (Ping timeout: 276 seconds)
14:00:46 *** kredik_ has quit (Client Quit)
14:01:18 *** kredik (~chatzilla@ANice-551-1-138-77.w92-143.abo.wanadoo.fr) has joined #mapnik
14:32:02 *** HounD has quit (Ping timeout: 258 seconds)
16:11:15 *** kredik has quit (Quit: ChatZilla 0.9.86 [Firefox 3.5.8/20100202165920])
16:25:10 *** chad_burt (~chad_burt@mm-01.msi.ucsb.edu) has joined #mapnik
16:29:38 *** mishok13 has quit (Quit: Leaving)
16:36:29 *** mperry (~mperry@63.227.17.10) has joined #mapnik
16:39:00 *** StormTide (~Kevin@2002:186c:64c0:0:21d:60ff:fe5e:cf66) has joined #mapnik
16:46:55 *** gavinf has quit (Read error: Connection reset by peer)
17:00:40 *** Genscher has quit (Quit: Leaving)
17:02:09 <dkb> I don't see Dane here but was wondering if someone else would be able to assist with getting  quantumnik working on my system?  I'm using 0.34 plugin, 1.40 QGIS.  When selecting loading "Mapnik xml file" I select my osm-local.xml file but get error  /Users/dkb/Development/Weather/MapnikOSM/world_boundaries/shoreline_300 does not exist (encountered during parsing of layer 'world').  I have the shoreline_300 shape files at that location.
17:03:34 *** shoe has quit (Read error: Connection reset by peer)
17:03:45 *** shoe (~user@ip24-250-63-6.ri.ri.cox.net) has joined #mapnik
17:12:38 *** cgs_bob_ has quit (Ping timeout: 246 seconds)
17:31:46 *** mperry has quit (Ping timeout: 248 seconds)
18:04:18 *** xreal (~mweber@mailgw.FB12.Uni-Dortmund.DE) has joined #mapnik
18:05:30 *** xreal has parted #mapnik (None)
18:05:38 *** xreal (~mweber@mailgw.FB12.Uni-Dortmund.DE) has joined #mapnik
18:08:12 <xreal> PSQL-question: Could anyone please explain me, how that's possible? http://paste.pocoo.org/show/183114/
18:09:38 <xreal> Oh got ... I think, I've seen my failure already.
18:09:57 <xreal> should be 53.01
18:10:05 <dodobas> yello
18:26:48 *** tcarobruce (~tcarobruc@adsl-75-10-247-30.dsl.pltn13.sbcglobal.net) has joined #mapnik
18:28:15 *** harobed has quit (Quit: Ex-Chat)
18:44:52 *** gavinf (~gavinf@196.211.2.133) has joined #mapnik
18:46:51 *** cgs_bob_ (~bob@149.sub-75-211-20.myvzw.com) has joined #mapnik
18:53:18 *** D3f0 (~D3f0@190.177.67.113) has joined #mapnik
19:04:45 *** luneff has quit (Quit: Leaving)
19:53:54 *** jburgess has quit (Remote host closed the connection)
20:04:26 *** D3f0 has quit (Quit: Saliendo)
20:31:33 *** jburgess (~jburgess@15.92.187.81.in-addr.arpa) has joined #mapnik
20:48:20 *** luneff (~yury@185.53.117.87.donpac.ru) has joined #mapnik
20:52:12 *** springmeyer (~springmey@75-151-97-182-Washington.hfc.comcastbusiness.net) has joined #mapnik
20:52:12 *** springmeyer has quit (Excess Flood)
20:52:46 *** springmeyer (~springmey@75-151-97-182-Washington.hfc.comcastbusiness.net) has joined #mapnik
21:13:48 <dkb> springmeyer: do you have time for a quantumnik issue?
21:14:22 <springmeyer> depends on what it is :)
21:14:33 <dkb>  I'm using 0.34 plugin, 1.40 QGIS.  When selecting loading "Mapnik xml file" I select my osm-local.xml file but get error  /Users/dkb/Development/Weather/MapnikOSM/world_boundaries/shoreline_300 does not exist (encountered during parsing of layer 'world').  I have the shoreline_300 shape files at that location.
21:15:44 <springmeyer> are you accidentally pointing at the folder rather than the actual shapefile?
21:15:51 <springmeyer> what does this give?
21:15:53 <springmeyer> file /Users/dkb/Development/Weather/MapnikOSM/world_boundaries/shoreline_300
21:16:37 <dkb> there is a directory named that
21:17:14 <springmeyer> is the shapefile this?
21:17:25 <springmeyer> '/Users/dkb/Development/Weather/MapnikOSM/world_boundaries/shoreline_300/shoreline_300.shp'
21:17:26 <springmeyer> ?
21:17:36 <springmeyer> if so you should put that in your Mapnik XML
21:18:45 <dkb> ok, I will check that, thanks
21:22:10 <springmeyer> dkb: so I assume you've tried rendering something outside of Quantumnik? and it worked without this error?
21:22:18 <dkb> yes
21:22:24 <springmeyer> what Mapnik version?
21:22:27 <dkb> 0.7
21:22:30 <springmeyer> odd
21:23:01 <springmeyer> perhaps a permissions problem?
21:23:15 <dkb> whatever I used to autogenerate the xml file put the values in like that (without the .shp)
21:23:38 <springmeyer> its fine without the .shp
21:23:48 <springmeyer> it works with or without
21:28:18 <dkb> this is odd, I added ".shp" to the file name in the xml and it got rid of the error but now is complaining about the processed_p file (which does not have the .shp specified in the xml)
21:29:09 <dkb> when I removed the ".shp" from the shoreline_300 name in the xml after trying to load the xml with quantumnik it didn't complain
21:36:05 <dkb> File "/Users/dkb/.qgis//python/plugins/quantumnik/quantumnik.py", line 311, in load_xml
21:36:06 <dkb>     mapnik.load_map(self.mapnik_map,str(mapfile))
21:36:06 <dkb> RuntimeError: /Users/dkb/Development/Weather/MapnikOSM/world_boundaries/processed_p does not exist (encountered during parsing of layer 'coast-poly')
21:36:47 <dkb> even though process_p.shp is sitting right next to the shoreline_300.shp file
21:37:12 <dkb> OS X 10.6.2, permissions same
21:42:00 <springmeyer> can you dpaste.com the output of:
21:42:12 <springmeyer> $ ls -l /Users/dkb/Development/Weather/MapnikOSM/world_boundaries/
21:42:17 <springmeyer> and
21:42:25 <springmeyer> ls -l /Users/dkb/Development/Weather/MapnikOSM/world_boundaries/processed_p
21:43:48 <dkb> I found it
21:44:39 <dkb> for some reason the shoreline_300 was specified in the correct location but the processed_p and builtup_area were in different location.
21:45:14 <dkb> fixing the path in the XML then loaded the mapnik rendered map in QGIS :)
21:46:51 <springmeyer> good
21:47:23 <springmeyer> okay, so to recap, there was no problem with Mapnik or Quantumnik, your paths were just wrong?
21:47:42 <springmeyer> the behavior should be exactly the same whether you are loading via Qnik or not...
21:50:08 <dkb> yes, Quantumnik was correct in reporting the errors, for some reason the osm-local.xml file did not have correct path for the processed_p and builtup_area
21:51:35 <dkb> there was also a directory "shoreline_300" which I think confused loading the shoreline_300.shp file located at same directory level
22:02:03 <dkb> That is BTW pretty awesome having mapnik render right in QGIS
22:06:23 <nikq> Mapnik Trac: Ticket #502 (Map buffer_size not accounted for within GDAL plugin scaling logic) updated | http://trac.mapnik.org/ticket/502#comment:3
22:06:27 <springmeyer> dkb: okay thanks for clarifying
22:07:40 <springmeyer> dkb: glas you think it is pretty awesome
22:08:04 <springmeyer> notice that if you hit ctrl-r it should re-load the XML (and represent any changes you made) without resetting the zoom
22:08:29 <springmeyer> so you can hone in on one spot/zoom level and edit and be able to get a re-render pretty quickly
22:11:39 <springmeyer> dkb: do you know you?
22:13:02 <springmeyer> gah, I meant: do I know you?
22:13:15 <dkb> no
22:13:55 <dkb> tried the re-load xml, that is cool
22:14:14 <dkb> I'm in Minnesota
22:14:14 <springmeyer> k
22:14:25 <springmeyer> brrrr :)
22:14:54 <dkb> so haven't been to Washington state for some time
22:16:38 <shoe> I just rendered my first maps yesterday, so be gentle :)  but I'm seeing funny grey spots: http://c.tile.mine.nu/default/9/130/211.png
22:16:43 <dkb> Quantumnik definitely taxes a 2GHz mac mini
22:16:54 <shoe> any idea what causes that?
22:17:44 <springmeyer> shoe: first guess would be a problem with the "coastline" shapefiles
22:18:14 <springmeyer> you might try-redownloading if you have not fetched them recently
22:18:32 <Ldp__> shoe: is that off the east coast of the US?
22:18:36 <shoe> yes
22:18:41 <jburgess> the last set of coastline shapefiles had some errors. If you hold on a moment I'll update the ones on the openstreetmap.org server with some better ones
22:18:46 <Ldp__> definately a shapefile issue
22:19:05 <Ldp__> jburgess: mine?
22:19:15 <jburgess> Ldp__: yes :)
22:19:28 <Ldp__> jburgess: I'll mail TomH now for an errol account
22:19:45 <springmeyer> dkb: if you actually notice Quantumnik taking up resources then that is a bug, but I imagine what you are seeing is the CPU strain of Mapnik rending OSM data live (without caching)
22:21:34 <shoe> speaking of those coastline shapefiles, I wondered if there was any advantage to importing them with shp2pgsql, or if it's better to just leave them as .shp
22:21:45 <jburgess> shoe: try getting the new shoreline_300 file from: SecBlade: Occasional RPC timeouts while under load
22:21:59 <jburgess> oops make that: http://tile.openstreetmap.org/shoreline_300.tar.bz2
22:22:09 <dkb> springmeyer: I see both Qgis and postgres processes sharing taking up all the CPU when zooming in/out
22:22:23 <jburgess> those shapefiles should fix the zoom 0 - 10 tiles
22:22:43 <springmeyer> dkb: hmm, okay
22:23:08 <shoe> jburgess: thanks, I'll try it out.
22:23:13 <jburgess> shoe: to fix the zoom11+ you need to update this one http://tile.openstreetmap.org/processed_p.tar.bz2
22:23:38 <jburgess> but the second is much larger so you might want to download and try with just the first one initially
22:23:51 <shoe> In general, how often are these updated?
22:24:43 *** filbertkm (~chatzilla@40.sub-75-226-173.myvzw.com) has joined #mapnik
22:24:56 <jburgess> shoe: it varies, normally about once per month but I got some new ones from Ldp__ because of the errors you pointed out
22:25:50 <Ldp__> jburgess: I generated these with the latest osm2coast.c patch that I checked in, and max nodes per way = 2000
22:26:06 <jburgess> shoe: we did some benchmarks of mapnik reading the shapefile directly vs reading from posgis and it really made little or no difference to the performance
22:26:41 <jburgess> shoe: mapnik is pretty efficient at reading the shapefile provided it has the .index file
22:27:01 <shoe> awesome.  thanks a bunch for the new coastlines.
22:27:18 <jburgess> Ldp__: ok, so I can revert my local changes which increased the node count
22:28:08 <Ldp__> jburgess: basically what is in SVN right now + the one modified tool for the low zoom version
22:28:46 <Ldp__> there were a few notices along the way. I'll log those next time
22:31:54 <shoe> since there are people around... I'll try to get more advice... :)  for my first try I used osm2pgsql, and I realize now that I used an old schema (default.style) because I had to disable mapnik rules that used newer columns...
22:32:55 <shoe> q1: can I use osm2pgsql again but use the newer schema?
22:33:03 <jburgess> that happens a lot
22:33:24 <shoe> I mean, do I need to drop the tables first, or will osm2pgsql take care of it?
22:33:40 <jburgess> osm2pgsql will take care of removing the data if you run it again
22:33:49 <shoe> oh nice.
22:34:13 <jburgess> unless you try to use the --append mode ... but you probably won't because that is for applying diffs on top of the existing data
22:34:48 <shoe> oh, I didn't know osm2pgsql could do --append.  I thought that was an osmosis-only feature.
22:35:18 <jburgess> you probably want to use osmosis too, for fetching and performing some pre-processing on the diffs
22:35:32 <jburgess> but to apply them to the DB you use the append option in osm2pgsql
22:36:18 <shoe> ah, I see.
22:36:31 <jburgess> but in order to run the append mode you need use the --slim option when doing your initial data import
22:36:50 <shoe> oh shoot.  I didn't the first time.
22:37:00 <jburgess> this will create some extra tables which are used for the subsequent updates (but are not needed for rendering)
22:37:24 <jburgess> it does make the DB significantly larger and slower so it is a tradeoff.
22:37:38 <shoe> can I run with --slim on my second full import?
22:37:58 <shoe> (and then be able to --append?)
22:38:14 <jburgess> if you want, you might need an extra postgres setup step if you have not already installed the "intarray" feature
22:38:54 <shoe> /usr/share/postgresql/8.3/contrib/_int.sql, right?
22:38:55 <jburgess> if you add --slim to the osm2pgsql command line then it will tell you if the intarray support is missing and that you can't use it for diffs
22:39:05 <jburgess> right _int.sql
22:40:24 <dkb> I grabbed the new shoreline_300 file and just did a cmd-R in QGIS, and before my eyes all those ocean blobs disappeared! :)
22:43:07 <shoe> this might be slightly off-topic, but... I'm also importing SRTM data. If I do "shp2pgsql -a -I -g way $file contours" on the same .shp file twice, does it duplicate data in the database?
22:43:32 <shoe> or is there some unique primary key used?
22:45:10 <jburgess> I would guess that with the '-a' it will add the data. I think there is a primary but it is an auto-increment type one so the new data won't clash
22:47:27 <shoe> hmm.. yeah, I was worried about that.
22:52:36 <dkb> with the new shoreline file, off the southeast coast of North Carolina, at 33.955891,-77.952805 you can see where a large bay is filled in with straight lines, was that like that before?
22:52:55 *** xreal has quit ()
22:54:42 <Ldp__> dkb: a direct link would be handy
22:55:00 <shoe> dkb: yey!  my blobs are gone, too.
22:55:50 <jburgess> shoe: I just did a test by importing a shpefile twise, using -a on the second one and it has duplicated all the data
22:56:39 <jburgess> dkb: do you mean this straight line? http://www.openstreetmap.org/browse/way/16151163
22:57:17 <jburgess> If so then that accurately matches the data, this view is better http://www.openstreetmap.org/?way=16151163
22:57:19 <dkb> yes, by Zekes island there
22:58:02 <jburgess> the blue highlight indicates the way which defines that bit of coastline
22:58:40 <dkb> on satellite it looks like all open water: http://maps.google.com/maps?q=33.955891,-77.952805&num=1&sll=37.09024,-95.712891&sspn=85.587375,131.132816&gl=us&ie=UTF8&ll=33.93567,-77.977524&spn=0.20765,0.261269&t=h&z=12
22:58:41 <jburgess> feel free to correct it using the yahoo imagery
22:59:07 <jburgess> that bit of way is marked with source=PGS. It can be really inaccurate
22:59:17 <jburgess> but was better than nothing
22:59:30 <dkb> where does one edit shoreline data?
22:59:55 <jburgess> it is generated from all the ways which have the tag natural=coastline in OSM
23:00:30 <jburgess> but a special processing step is needed to join them all together to build the enormous polygons required for all the land
23:00:44 <shoe> jburgess: shp2pgsql needs a --undo option :)
23:01:19 <jburgess> If you know the count of features you imported then you should be able to delete that range of gid's
23:01:25 <dkb> ok, wasn't aware you were able to edit those..  you can use the yahoo images for this?
23:01:42 <jburgess> "shpinfo shapefile-name" to find out how many features you just added
23:02:13 <jburgess> dkb: yes, zoom in a bit more, select the 'edit' option and potlatch should bring up the yahoo imagery
23:02:42 <jburgess> then find the coastline way (should be obvious since it will be the one following the existing land/sea boundary
23:03:03 <jburgess> and start adding nodes and refining the positions to make it match the yahoo pictures
23:03:08 <shoe> Is "rm 3/0/0/0/0/0.meta" a valid way to force re-rendering of zoom 3 in a mod_tile setup?
23:03:41 <jburgess> shoe: yes, that will remove the zoom 3 tiles and the next time you view then it'll will get rendered again
23:03:41 <Ldp__> rf -rf 3
23:03:45 <dkb> jburgess: ok thanks
23:04:11 <shoe> Ldp__: oh yeah, tab completion and all.
23:04:12 <jburgess> alternatively, if you right click and get one of the tile URLs, if you append "/dirty" on the end it will force a re-render of the relevant meta tile
23:04:28 <jburgess> you can append "/status" on to the URL to find out when it was last rendered
23:06:50 <shoe> hm. still have blobs at some zooms :(  maybe I need to restart renderd.
23:07:16 <jburgess> e.g http://b.tile.openstreetmap.org/13/2321/3273.png/status
23:07:31 <jburgess> you might have some tiles cached in your browser too
23:07:43 <shoe> yeah, it re-renders with blobs, and I cleared browser cache.
23:08:01 <jburgess> which zoom level?
23:08:08 <shoe> 4
23:08:25 <jburgess> how does it compare with http://www.osm.org/ ?
23:09:05 <jburgess> it would be worth checking the timestamps to make sure you have rendered them since updating the shapefile
23:09:15 <jburgess> or just delete all the low zoom tiles
23:10:49 <jburgess> It looks like there might still be one bad block at the southern tip of Florida but I don't see any other obvious errors
23:11:24 <dkb> Where would appropriate discussion/assistance be located if I was given access to my counties GIS data that was all done with GPS and includes correct street names?
23:11:46 <shoe> oh man,  Chrome is lying to me about clearing the browser cache.
23:12:05 <Ldp__> jburgess: I saw that one too, but chalked it up to an actual coastline error
23:12:35 <Ldp__> dkb: the OSM imports mailing list would be a good start
23:13:03 <jburgess> Ldp__: I think you are correct - there is one bit which is completely broken off the southern tip of Florida
23:14:04 <Ldp__> http://coastline.openstreetmap.nl/?zoom=8&lat=26.64454&lon=-80.02441&layers=B00T
23:14:08 <jburgess> http://coastline.openstreetmap.nl/?zoom=15&lat=25.02638&lon=-80.49528&layers=B00T
23:14:12 <Ldp__> few errors there
23:14:28 <Ldp__> ooh, shiny
23:15:43 <jburgess> I looked at fixing it but the data is a strange mix of boundaries and coastline which are the same in some places and different in others.
23:17:28 <jburgess> my previous experience with the coastchecker is that disconnected or overlapping ways are the biggest triggers for land/sea problems.
23:21:32 <Ldp__> there was a time when the coastline was relatively free of major errors, but we've done a few steps back, it seems
23:25:50 <jburgess> I looked at some of the places where there were errors and the existing PGS data was of poor quality.
23:26:31 <jburgess> Someone was probably trying to improve it by deleting the existing way and adding a more detailed one, but perhaps not joining up the two ends with the previous data
23:27:23 <shoe> ok. apparently, clearing browser cache while having resource tracking enabled in developer pane will 1) not clear the cache, and 2) make chrome very angry.
23:27:35 <shoe> (all that to say, yeah, the blobs really are gone.)
23:29:02 <shoe> however, there are similar blobs off the coast of India and China.
23:34:07 *** filbertkm has quit (Quit: ChatZilla 0.9.86 [Firefox 3.5.8/20100214235838])
23:47:35 *** luneff has quit (Quit: Leaving)
23:53:59 *** dkb has quit (Quit: Leaving.)
23:55:45 *** D3f0 (~D3f0@190.177.67.113) has joined #mapnik