gmane.network.freenet.devel

Re: Priorities for 22 June to 1 October

Matthew Toseland — 2013-05-19T19:16:26

Just to clarify this last point: I'm not anti-police. I'm not pro-criminality. The point is that breaking Freenet is easy, and if our police can do it, so can evil corporations (not all corporations are evil), and forces of oppression in unpleasant countries. There is a market in exploits; these things are likely for sale in the right places, or at least they will be as soon as Freenet is big enough for there to be any demand.
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Priorities for 22 June to 1 October

Matthew Toseland — 2013-05-19T19:13:11

I will be able to work full time on Freenet for four months after my exams are over. After that, it's iffy.

Obviously the first priority has to be to make sure that other people can do releases.

Beyond that, IMHO I should focus primarily on sorting out darknet. Darknet should be easy, fast, and secure. It should coexist with opennet, and automatically detect when opennet is unnecessary. And then we can implement tunnels, probably based on Pisces.

This is a fairly limited amount of time, and IMHO darknet is *the key issue* for Freenet. The fact is, right now using Freenet in darknet mode is slow, insecure and above all inconvenient. Using Freenet in opennet mode is slow to bootstrap (darknet invites would actually solve this problem), and even more insecure. IMHO the major police support agencies probably already have tools to trace opennet users, they just avoid showing them in court. The academic literature shows tunneling makes much more sense on darknet than on opennet; peer selection for DHTs is an un

Re: Pisces redux

Matthew Toseland — 2013-05-19T12:35:15

Caveat: We don't want to bias tunnel setup too much. If e.g. the middle hop is too close to the originator, we lose anonymity. If it's too close to the end that's bad too (see the paper). So it's complicated...
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: Understanding the current state of Freenetdevelopment for file sharing/searching project

Matthew Toseland — 2013-05-16T23:09:14

Basically the layering is:

Storage layer: SSKs (1kB, named by pubkey and filename), CHKs (32kB, named by content hash)

Splitfiles/client layer: A file is uploaded to freenet to a URI (CHK or SSK). The file can be any size. If it doesn't fit into one block, we split it up into a pyramid consisting of one block (a CHK or an SSK), containing metadata (a list of keys) for the next layer. The next layer may be the final data, or it may be metadata for another layer. And so on. Each layer other than the top block has redundancy.

Btree nodes: Each btree node is a file (a single CHK block or a splitfile). The nodes are always inserted as CHKs.

Btree: Top node, inserted as a CHK like the other nodes. Note that these trees can be updated incrementally, not necessarily by the person who originally uploaded them.

Library index: The top USK points to the two trees and has some metadata e.g. the size of the index. We have two trees inside a library index at the top level, and IIRC we use a tree within each term as w

Understanding the current state of Freenetdevelopment for file sharing/searching project

leuchtkaefer — 2013-05-16T16:47:14


Hello everybody,

I am planning to implement file sharing/search improvements/new features. As an initial step I immersed myself in all different sources of info about Freenet to understand better all part of the projects related to the distributed search. I more or less read lot of stuff listed in a guide that I started writing in case someone is in the same situation as me in the near future. I can uploaded my guide to the list if somebody is interested on it.


HOWEVER it is still difficult for a newbie in Freenet to understand the basic implemented data structures. It is also somehow confusing know what things have been done but are not in the official deployment.

For instance, I really need to understand:


According to doc in source Library an INDEX has metadata and two BTree.Index
 |-- metadata
 |-- ttab: BTreeMap<String, BTreeSet<TermEntry>> 
 |-- utab: BTreeMap<URIKey, BTreeMap<FreenetURI, URIEntry>>

Does each p2p node construct one index? It is this index the actual structure in official Fre

Re: Please review #5679

Matthew Toseland — 2013-05-16T17:57:40

Yes.

Ugh. Well, if the API is limited, we just have to do what's possible...

Yes, 4 spaces indent and 100 char width should be fine. I'm sure this was documented but I can't find where. :(

Right, unless the plugin itself has some way to determine the MTU other than querying the interfaces. The JSTUN plugin should just run JSTUN, it should not query the interfaces if it doesn't need to.
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Freenet 0.7.5 build 1442

Matthew Toseland — 2013-05-14T19:00:03

Freenet 0.7.5 build 1442 is now available. Please let me know if you have any trouble upgrading; your Freenet node should update itself automatically. This build will be mandatory on the 21st of May. Major changes:
- Process connection setup messages on a separate thread to the one that handles packets from already connected peers. This should make connections more stable, especially on seednodes, ARM nodes and when there are CPU usage problems.
- Increase the maximum number of locations that a peer can tell us about to 110. This does NOT actually increase the maximum number of opennet peers yet, that will be in 1443.
- Update update.sh and update.cmd via the auto-updater.
- Minor optimisations in auto-update.
- Write a backup of the old main jar to freenet.jar.bak when updating on Linux (on Windows, we alternate between freenet.jar and freenet.jar.new).
- Paranoid code to try to prevent the problems in 1439 from happening again.
- Increase the startup timeout for waiting for entropy to 1 hour, explain the s

Re: Please review #5679

2013-05-14T11:32:30

Hey Toad, 

Thanks for your points.
Matthew Toseland <toad-EI5O+8PHWbL67tp2J140zg< at >public.gmane.org> Okay, one or two things to deal with first:
I'm sorry about the link. My recollection was that I even tested the
link and it was pointing to the right commit. Maybe github doesn't do
permanent link.

So, If I'm posting a branch instead of a patch, I don't need to do
that, Freeneters are going to take care of that, Correct?

Yes that was odd for me too. My understanding is that the interface get
max MTU independent of which ip protocol is going to be used. Then when
the connection is establish the MTU will be different based on if it is
a IPv6 or IPv4 connection and who is on the way. 

Further more, the MTU is the maximum of any MTU (where it is IPv6 or
IPv4). So it can't be minimum of both MTU (unless they are
equal). Because the way to do MTU discovery is to set MTU equal to the 
interface's MTU and decrease till eventually it passes through the
connection at that time you know you found the MTU of that spec

Re: Please review #5679

Matthew Toseland — 2013-05-12T12:37:26

Okay, one or two things to deal with first:
- It would be better to just say which branch you want us to merge. There is a button for this on github, otherwise post a link. (The link you posted in the first commit was wrong)
- You should always use git diff -u --ignore-space-change
- Calling reportMTU inside the loop is a bit odd, but I guess it's unavoidable? I believe IPv4 and IPv6 on the same interface is how it will generally work, in which case the JVM should report whichever MTU is lower? This should be mentioned in the comments anyway.
- Freenet typically uses a style of 4 space tabs and 100 char lines IIRC. So you are chopping off some lines unnecessary. Hmmm, I thought there was a page for this on the wiki? Apparently not ... :|
- Once this is deployed we should update the plugin as well...
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Please review #5679

2013-05-12T11:37:28

Please review:

https://bugs.freenetproject.org/file_download.php?file_id=311&type=bug

Thanks,
Vmon

vmonmoonshine-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org writes:

Re: Progress on #5679

Matthew Toseland — 2013-05-12T11:11:35

I think that might be the wrong branch?
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Pisces redux

Matthew Toseland — 2013-05-11T20:16:00

Some work on Pisces:

TUNNEL LENGTH:

Pisces supports setting up standard 2-hop tunnels, with anonymity close to that of a full 25-hop tunnel. The catch is, this means the nodes need to reveal themselves, to connect directly. So there may be a tradeoff between invisibility, security and performance: Long tunnels mean poor performance, which means we probably end up using more of them, which means less security. A partial solution is to use some of the hops: Friend-of-a-friend connections are planned, we could use these to skip a hop at each stage, giving 12-13 hops. We could expand this to FOAFOAF (i.e. 2 hops), and cut it to 8 hops. Or we could treat visible nodes differently to invisible nodes: If a node has opennet enabled (for example), it can safely return its IP address in a tunnel setup request.

Related issue: Should the opennet peer limit be completely separate from darknet connections? Will this encourage people to get bogus darknet connections?

WHANAU/UNDERLYING DHT:

Pisces requires a reliable D

Progress on #5679

2013-05-11T12:58:29

Still need to ignore local ips but that shouldn't be hard. I'll add that
shortly. Also it calls updateMTU more frequently. I don't know if it is
a deal breaker.

Anyway, I ran it and at least I was able to browse normally so I haven't
broke anything.

https://github.com/vmon/fred-staging/commit/1b108d0c0210731c5424af0e36415a867004b5c1

Cheers,
Vmon

FlogHelper anonymity breach and Freenet 0.7.5build 1441

Matthew Toseland — 2013-05-08T15:02:28

Freenet 0.7.5 build 1441 is now available. Please upgrade, the changes are minimal but include a critical security fix for FlogHelper. If you have used FlogHelper and have published flogs with the "safe backups" option enabled, your hostname is visible to anyone who downloads flog.db4o. Sorry about this, we should have been more careful (it's an information leak inside db4o...). A more detailed warning will show up if you use it. Note that this may not affect you even if you do have a FlogHelper flog with backups turned on, if it hasn't been updated recently and the backup file is large enough not to be in the main container (in which case it may have fallen out).

Changes:
- Update the SSL certificate for the last resort update scripts.
- Catch more db4o errors on startup.
- Library v27.
- FlogHelper v29.
- Fix an NPE in non-persistent USK inserts.

ANONYMITY BREACH: Your flog "${title}" has leaked your hostname "${hostname}" for your WoT identity ${author}, in the backup file "flog.db4o". That means all yo

Re: New darknet routing paper may be important

Matthew Toseland — 2013-05-07T12:47:59

Not so far. Sounds like you have some ideas you might want to share with them, if you don't have time to publish them yourself?

I can talk to them if appropriate, maybe it's easier for somebody who speaks the same language? Certainly there are big issues to consider with both algorithms.
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: New darknet routing paper may be important

Oskar Sandberg — 2013-05-07T10:42:07

Have the authors of these papers talked to you guys? Presumably they are
working on this stuff because they want to contribute, it seems strange
that you get the papers after they are published.
On May 6, 2013 6:03 PM, "Matthew Toseland" <toad-EI5O+8PHWbJeeLb3ft/vUmD2FQJk+8+b< at >public.gmane.org>
wrote:

_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Issue 5679

2013-05-07T09:32:55

Hey Toad,

I can simply directly call getMTU() inside Node.getMinimumMTU or I can
replace the pluginDetectedIP.mtu with getMTU() calls in
NodeIPDetector.processDetectedIPs. 

Could you please tell me which one do you think is the
better/cleaner/more harmonious way to do it? 

Thanks,
Vmon

Interplanetary Internet

Robert Hailey — 2013-05-06T17:06:39

FYI.

http://www.wired.com/wiredscience/2013/05/vint-cerf-interplanetary-internet/

Apart from the "cell phone" analogy, seems like much of the "web" could be used across planets using freenet.

--
Robert Hailey

_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: New darknet routing paper may be important

Matthew Toseland — 2013-05-06T15:03:18

Thanks for your thorough and readable explanation. I hope Michael can do something with this given you don't have time.

There are lots of ways to find the size of the network, most of them can be fooled though. For example we have network size stats based on a tag/release principle (i.e. probes return an ID and uptime; correlate between multiple runs); IIRC that takes quite a few samples though, so it may be impractical to run on every node. Anything based on probes is vulnerable at each hop.

Also, the "probe for the closest node location to a random location" query is unreliable in my experience; sometimes we get bad answers, where it is much further away from the target than expected. Possibly the probe requests that had this problem were implemented badly; I haven't tried it with the more recent implementation. It doesn't matter if we use several keys anyway...

Michael's simulations of your original proposal so far have been inconclusive AFAIK, though some of them I think were simulating the wrong alg

Re: New darknet routing paper may be important

Oskar Sandberg — 2013-05-06T14:00:21

On Sat, May 4, 2013 at 4:07 PM, Matthew Toseland
<toad-EI5O+8PHWbJeeLb3ft/vUmD2FQJk+8+b< at >public.gmane.org>wrote:


This doesn't mean calculate a single value, it means that you can draw your
position randomly from a known distribution (a distribution is something
saying e.g. "you have 1/2 chance of getting value 1, 1/4 of getting value
2, 1/8 of value 3, etc.." for the set of possible values) rather than
having to simulate a random process to chose your position.

Taking a step back:

Basically, the type of random processes that we are dealing with here are
called "Markov chains", which means that what happens in the next step
depends only on the current state (the current position) not on the
history. Under certain conditions Markov chains have the property that when
run for a long time, they gradually forget in what state they started, and
you have a fixed probability of the process being in each possible state
(regardless of what state they started in). This is what I referred to as
the "stationary distrib

Re: Reconsidering peer limits...

Matthew Toseland — 2013-05-06T13:22:38


It affects performance directly, it also results in data being cached in the wrong places (and maybe stored too?), so it affects data persistence too.

No, one person with a patched build can have only a limited impact. Lots of people with patched builds can have a larger impact.

I am strongly opposed to a hierarchical topology because it is easy to attack. However, aspects of it will likely emerge automatically on opennet, and there are lots of easy attacks you can do on opennet with relatively small resources.

So it probably makes sense just to increase the peer limit to 100 for now, but with the same bandwidth requirements (slow nodes with lots of peers are bad). And maybe file a bug for looking at the aristocracy issue.

Hmmm, how much bandwidth does 100 peers correspond to anyway? We should calibrate it to something sensible... 100 peers is approx 800kB/sec, so about 8Mbps upstream. Running this much bandwidth 24x7 is problematic for most users but certainly not all users...

Note that the reason we

gmane.network.freenet.devel

Re: Priorities for 22 June to 1 October

Priorities for 22 June to 1 October

Re: Pisces redux

Re: Understanding the current state of Freenetdevelopment for file sharing/searching project

Understanding the current state of Freenetdevelopment for file sharing/searching project

Re: Please review #5679

Freenet 0.7.5 build 1442

Re: Please review #5679

Re: Please review #5679

Please review #5679

Re: Progress on #5679

Pisces redux

Progress on #5679

***FlogHelper anonymity breach*** and Freenet 0.7.5build 1441

Re: New darknet routing paper may be important

Re: New darknet routing paper may be important

Issue 5679

Interplanetary Internet

Re: New darknet routing paper may be important

Re: New darknet routing paper may be important

Re: Reconsidering peer limits...

FlogHelper anonymity breach and Freenet 0.7.5build 1441