gmane.comp.gis.openstreetmap.devel

Re: [OSM-dev] New database server

Paweł Paprota — 2013-05-21T17:50:44

Reluctant +1 from me though I am sure this is not a popular view ("we 
don't want authority" and all that)... I can only say from my own 
experience what I was/am trying to do with OWL/history tab/whatever you 
call it - in itself not a trivial task but on the other hand the 
challenges with the overall architecture (both logical and physical) are 
at least at the same level (if not above!) of complexity.

And what I can tell you is that people in OSM (I mean admins here) are 
very supportive and open to changes. I love this in this project that I 
did something and I had/have the opportunity to get it into production. 
Sadly, I simply don't have enough time to finish it to the extent that 
it is acceptable (to me, not to mention others/admins) to consider it 
for production use.

So I think your statement is 100% true but I think the main problem is 
that there are no people or/and no money to do this. Think if you were 
the person that would need/want to do this work. It most definitely *is* 
a full time

Re: [OSM-dev] New database server

Ian Dees — 2013-05-21T15:12:57

Can someone add more information about when and how the OWG meets? The wiki
page is pretty bare on how one might help:

http://www.osmfoundation.org/wiki/Operations_Working_Group


On Tue, May 21, 2013 at 10:07 AM, Simon Poole <simon< at >poole.ch> wrote:

_______________________________________________
dev mailing list
dev< at >openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] New database server

Simon Poole — 2013-05-21T15:07:00

I normally stay out of the tech bike-shedding discussions, however I do
want to point out 

- we are aeons away from requiring and running cutting/bleeding edge
hardware (and having to pay for such)

- in the grand scheme of things we are not spending a lot of money on
hardware (on the one hand our sys admins and the OWG are very frugal and
on the other see the first point)

- the amount of money we spend is a lot of money for the foundation, at
least relative to our other spending, however it is extremely unlikely
that we could away with spending less regardless of implementation
(distributed, 3rd party cloud etc etc etc).

- our current setup is fairly straightforward, fancier schemes are very
likely to be more error prone with the associated costs (manpower)

All this said, I would recommend that anybody who actually wants to help
should participate in the OWG and help with the other tech tasks that we
have in abundance.

Simon

Re: [OSM-dev] New database server

Frederik Ramm — 2013-05-21T14:43:42

Hi,

On 05/21/13 16:08, Serge Wroclawski wrote:

Knee-jerk call for authority? Never worked well.


It is not something that someone can do *on their own*, and I think that 
is not so bad. To make a change in OSM you need to work together with 
others, make an argument for your vision over a longer time, get buy-in 
from a larger group of people, and eventually things will move. I 
wouldn't want to sacrifice that careful, evolutionary process in 
exchange for a visionary leader where we all just do what he says.


I don't think we want or need authority figures who are somehow exempt 
from having to *convince* us that their idea is good.

Bye
Frederik

Re: [OSM-dev] New database server

Richard Fairhurst — 2013-05-21T14:41:38

Define "can".

The project has plenty of people capable of doing this.

But IME the main barrier between "capable of doing" and "actually doing" is
the amount of shit you have to suffer from armchair experts whenever you try
to "actually do" anything in OSM.

Richard





--
View this message in context: http://gis.19327.n5.nabble.com/New-database-server-tp5761947p5762058.html
Sent from the Developer Discussion mailing list archive at Nabble.com.

Re: [OSM-dev] New database server

Serge Wroclawski — 2013-05-21T14:08:48

Jason,

You're not at all wrong about the issues with the server design.

This is something that's been well known and understood for several years:

   As the project grows, the cost of scaling on a single system will
not scale accordingly.

What I mean by that is, that it's not a linerar cost to buy a single
machine with linear scaling.

So if you are growing, it makes more economical (and technical) sense
to scale away, rather than building up.

What would this mean in the context of OSM?

It might mean something like moving the GPX data off the main
database. Or maybe having historical data on a slower database than
the current data.

It also includes things like aggressive caching and uausing tiled map
calls (something that Ian and I worked on, and Ian has a new
implementation of).

And there's room for more optimizations even then, but just these
would make an impact.

So why doesn't this happen? Frankly, because I think the project
doesn't have anyone who can act in the kind of technical leadership
ro

Re: [OSM-dev] New database server

Andy Allan — 2013-05-21T09:49:08


Indeed, it might not be the best way to go, and any thoughts and
brainpower applied to the problem are always very welcome.

Of course, at OWG we *do* think it's the best approach, given all the
trade-offs involved, and it's a decision that hasn't been undertaken
lightly.


It's worth bearing in mind that we don't have any places on this
year's GSoC, and even if OSGeo decided to go for this project on our
behalf, it could be months before we even have any idea whether or not
the implementation is feasible. I don't mean to be negative, but
weighing up the "here and now" requirements against a hypothetical
alternative at some point in the future is one of these trade-offs
that we routinely have to make at OWG.


Well, we could certainly spend the money on small edge servers, but
it's not clear to me why you think that would make the central server
less expensive. I think this proposal may be worthwhile but it's
somewhat orthogonal to the goals of the new server.

At the moment we have two osm-database-class

Re: [OSM-dev] New database server

Florian Lohoff — 2013-05-21T07:24:27

Without beeing involved in the server issues my guess is that not the
amount of API calls but the working set is the problem here.

We are talking about a multi terabyte working set. Grabbing data out of
this working set is a very tedious task. You might want to grab 10 bytes
in the front - 20 in the middle and 60 at the end. For this you need to
walk through multiple gigabytes of indexes and move your disk heads like
1750 times.

Not looking at SSDs the number of spindles is the concurrency you can
get from this. The more heads - the more concurrent accesses.

SSDs will help accelerate indexes but today are not a solution for the
full database comparing € or $.

Flo

[OSM-dev] New database server

Jason Remillard — 2013-05-21T01:32:30

Hi,

I could not find a discussion on the new database server.

http://wiki.openstreetmap.org/wiki/New_server_and_fund_raising_drive_2013

The server that we are planning on purchasing is monster. Very
complicated and expensive. I am concerned that this might not be the
best way to go.

We have a google summer of code proposal to write an edge proxy server
for the OSM API. I don't know if the project will be accepted, but it
has got me thinking about the approach and our funding drive. The idea
is that each front facing server has a local snapshot copy of the OSM
database to service all of the read only calls. These edge servers
could be geographically distributed. It would just leave the central
database server to deal with write requests, history requests, and
diffs ( anything that can't be handled with a snapshot database
schema). This would allow the site to scale more incrementally, and
potentially scale to larger loads than putting all of our eggs into
two monster servers. For the money we planning on

Re: [OSM-dev] SotM ads for front page

Andrew — 2013-05-18T13:06:30


No CWG recordshave been posted since February. Is there a chance we could
have some?

--
Andrew

Re: [OSM-dev] Overpass DEV url?

Frederik Ramm — 2013-05-18T11:13:35

Hi,

On 18.05.2013 10:46, amrit karmacharya wrote:

Sounds better than my way - it seems that JOSM even checks referential 
integrity when you do that. Just be sure to change the API URL between 
downloading and uploading.

Bye
Frederik

Re: [OSM-dev] Overpass DEV url?

amrit karmacharya — 2013-05-18T08:46:33

Thanks for the detailed suggestion,

the problem was that JOSM could not connect/validate dev url *"
http://api06.dev.openstreetmap.org/api", *i tried it now and its been
validated.

My query:
Instead of using text editor, why don't i

1) Select the objects i want to upload in JOSM
2) Make a new layer
3) paste the objects there
4) upload to dev server

from my previous experiences this will give new id's to objects and mark
them as new objects.

but i don't know how this will affect the ref=* and relations. BTW i am not
using relations now, but i will in near future.


On Sat, May 18, 2013 at 2:09 PM, Frederik Ramm <frederik< at >remote.org> wrote:

_______________________________________________
dev mailing list
dev< at >openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Overpass DEV url?

Frederik Ramm — 2013-05-18T08:24:55

Hi,

On 18.05.2013 06:39, amrit karmacharya wrote:

A general rule for discussion in technical communities is: "Does not 
work" is not a sufficient description of a problem.

This is how you use JOSM to upload test data to the dev API:

* Download data from OSM (either with JOSM or through export tab)
* Open XML file in text editor
* Remove all changeset="...." attributes
* Change all object IDs to negative - e.g. in vi you would use the 
following replacement operations

s/ id="/ id="-/
s/ ref="/ ref="-/

(if XML file was saved from JOSM, take care to use ' not ")

* make sure you have no dangling references (e.g. relations pointing to 
objects not in your file) - if you don't need relations the easiest 
thing is to simply remove them all from your file

* open file in JOSM
* IMPORTANT - change JOSM settings to point to the dev API!
* upload

Not that uploading data into the dev API will *not* give you a rendered 
map with that data, *nor* make the data accessible via XAPI or Overpass. 
Also, the data you u

Re: [OSM-dev] Overpass DEV url?

Paul Norman — 2013-05-18T06:51:02

using the iD dev mode, but the process is highly time consuming

Looking at what you want to do in your original message, there is not an
easy way to test a mechanical edit based on overpass/xapi when you want to
test the entire toolchain. Basically, you need to load up the region you're
interested in into an apidb database and set up a rails port instance. 

You probably want to either figure out a way to verify the parts separately
and rely on well-tested upload tools or take a small area from the main API,
upload it to the dev API (converting the XML file as needed), downloading
the area from the dev API with map? and loading that small area into your
own overpass instance.

Re: [OSM-dev] Overpass DEV url?

amrit karmacharya — 2013-05-18T04:39:08

How can we enter a small bunch of data into dev server?

Using josm didnot work, and there are no data in my area to test. I tried
using the iD dev mode, but the process is highly time consuming


On Sat, May 18, 2013 at 12:45 AM, Tom Hughes <tom< at >compton.nu> wrote:

_______________________________________________
dev mailing list
dev< at >openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Overpass DEV url?

Tom Hughes — 2013-05-17T19:00:12


The dev site is completely separate, with a separate database, so only 
has whatever small amounts of data people have chosen to enter.

Tom

[OSM-dev] Overpass DEV url?

Tac Tacelosky — 2013-05-17T18:45:38

I'm getting a bunch of data from the Overpass API, and about to test
updating it.  I don't want to test on the live server, so I'm using
http://api06.dev.openstreetmap.org/.

But the id's don't seem to work -- that is, the ID's that come back from
overpass aren't  in (or are the same) as on the dev site.

Is there either an Overpass dev site I can use (to get the right id's), or
is there another solution to testing that I'm not seeing?

Thanks,

Tac
_______________________________________________
dev mailing list
dev< at >openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Using Google Street View Player with my own panoramicimages

Tac Tacelosky — 2013-05-17T16:36:09

There must be some situations where the streetview player comes up in
Flash, but the tool that Google provides is javascript.  An older version
of our product supported a Flash-based view that had a different tiling
system, which required us to render all the tiles twice.

Except for the OSM community, everyone want to see it in the Google street
view player.  It's fast and easy.  Still, I'd like to offer something that
can be used to update the OSM database, and am trying to find a solution.

One of our developers is trying to change the viewer to use Leaflet for the
panoramic tiles.  Unfortunately, it doesn't appear that Leaflet supports
scaled zooming, so images would always be a exact zoom levels, not
fractional zoom levels.

We may end up trying KRPano as well, but I don't know if it gives us all
the events we need.

Thanks,

Tac


On Fri, May 17, 2013 at 12:16 PM, Rob Nickerson
<rob.j.nickerson< at >gmail.com>wrote:

_______________________________________________
dev mailing list
dev< at >openstreetmap.org
http

Re: [OSM-dev] Using Google Street View Player with my own panoramicimages

Rob Nickerson — 2013-05-17T16:16:16

I'm no legal expert but "Products" and "Content" is defined by Google:

"By downloading, installing, or using the Google Earth software, accessing
or using the Google Maps service (together, the "Products" or "Services"),
or accessing or using any of the content available within the Products..."

To me, the viewer is a "product" and therefore is bound by the rules in the
terms of use.

In any case, isn't the StreetView viewer Flash? Firefox claims it is when I
force quit the plugin-container process. If this is the case then why not
use KRPano?

Regards,
Rob
_______________________________________________
dev mailing list
dev< at >openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Ads for front page - Proposed solutions

Rob Nickerson — 2013-05-17T16:01:48

I don't comply with this: I clearly don't want to be spammed by Emails.

Hi Michael,

That was incredibly selective quoting of my original text and makes me
wonder why I bothered putting my ideas down in writing. As noted any email
list would be "opt-in" only; something I addressed as an issue in the
sentence following the part you have quoted. Furthermore due to the
decreasing popularity (and visibility) of RSS, some OSM users may actually
like a monthly email delivered to them.

I don't want to be overwhelmed by irrelevant banners, and hence I tried to
propose some alternatives. I'm sorry you didn't like them.

Rob
_______________________________________________
dev mailing list
dev< at >openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Using Google Street View Player with my own panoramicimages

andrzej zaborowski — 2013-05-17T12:43:37

Hi,

On 17 May 2013 13:33, Tac Tacelosky <tacman< at >gmail.com> wrote:

Nick Whitelegg's OpenTrailView
(http://wiki.openstreetmap.org/wiki/OpenTrailView) had developed a
simple javascript library for browsing free panoramas but it's not as
smooth as Google's.  Writing one using either canvas of WebGL
shouldn't be too difficult.

As for Google's library, you could check the headers of the javasciprt
sources for a license information. If there's nothing there and the
Google Maps TOU say nothing (as you argument) then you can't really
assume any license other than "all rights reserved" and should ask
Google about it.

Cheers