gmane.network.freenet.devel

Transport Plugins Update

Chetan Hosmani — 2012-05-23T17:00:40

Hello everyone,

I have been a bit busy last week, and haven't committed much code.
Here are the following refactor changes I have implemented and pushed
to "refactor-1" branch-

1. Added a separate list for transport managers. So we basically have
a list which will contain a manager for every mode. Presently it has
two - opennet and darknet.

2. NodeCrypto functions in the following way-
a. Transport managers are created in the node irrespective of whether
opennet or darknet is running.
b. NodeCrypto on creation for every mode accesses its respective
transport manager to get a list of transports.
c. It runs the method handleNewTransport to initialise a PacketMangler
and handle each transport plugin.
d. If the list of transports is empty, then it will function using the
inherent UDPSocketHandler transport and run normally.
d. When a plugin is loaded (on registering with TransportManager), the
manager checks if the corresponding mode (opennet/darknet) exists and
notifies the NodeCrypto by calling the handleNewTransport method.

3. UDPSocketHandler now extends PacketTransportPlugin, and I have made
changes so that it looks like a plugin. I have built it and checked.
It runs fine.
It also registers in the transport manager.

4. Presently working on replacing the Session keys in the PeerNode
object. There are three - current, previous and unverified. I am not
yet well acquainted how these work.
But the idea is to have a separate connection object PeerConnection that -
a. Has a unique PeerNode
b. Has a transport associated
c. Has the corresponding Peer (detected Peer) object associated
(different transports are on different ports, and we can keep it open
to use different addresses itself)
d. A list of session keys. It need not be three. The code can become
more generic.

So PeerNode has a list of PeerConnection objects, for every active
transport. The PeerNode will decide the transport to use using these
objects.


The official coding period has started. So after a few more additions
I will merge them to "Transports" branch and begin working on a new
branch "PacketTransports".
Also awaiting some code review, after nextgens becomes free.

For the coming week-

1. Work and finish handleNewTransport
2. Finish the PeerConnection object
3. List the changes needed for multiple transport support in PeerNode
(implementing PeerConnection list)
4. Figure out PeerNode code to use different transports

Later on I will mainly work on NewPacketFormat which needs to be
changed so that they are more generic for Packet Transports. Hope to
have toad's and zidel's help then.

ArneBab - Flog is coming too. Sorry :)

Regards,
Chetan Hosmani

Statistics Project Update #4

Steve Dougherty — 2012-05-19T21:46:12

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've finished a proof of concept for Metropolis-Hastings corrected
probes in Fred. If you are so inclined, please offer feedback on the
patches. [1] Metropolis-Hastings correction was described previously,
[2] and the probes are exposed over FCP. [3] An originating node starts
a probe request by specifying the type of result they're looking for,
and optionally a value for hopsToLive, which currently defaults to its
maximum of 50 if not specified. We will in practice initially use
something closer to 30. One can disable responding to any number of
result types on the Core settings page.

Possible types are:

BANDWIDTH - outgoing bandwidth limit.
BUILD - Freenet build number. (For instance, currently 1407.) Useful
        for measuring update propagation.
IDENTIFIER - endpoint identifier. Useful for improving estimates of
             network size and churn.
LINK_LENGTHS - differences in location between endpoint and each of
               endpoint's peers.
STORE_SIZE - size of datastore.
UPTIME - session uptime and the percentage of the past 48 hours during
         which the endpoint was up.

+/- up to 1% noise is added to bandwidth, link lengths, store size, and
uptime information in an attempt to make the information less
identifiable but still useful for analysis.

Bandwidth, link lengths, store size, and uptime are intended to improve
knowledge of the network so that any problems will hopefully become
apparent, or at the very least allow more detailed simulations which
more accurately reflect actual network conditions. This is with the
goal of implementing changes to improve network performance and
reliability.

Possible outcomes of the probes as implemented are:
    -non-propagated timeout
    -non-propagated disconnection
    -endpoint refusal for the result type
    -result from endpoint

Based on feedback in #freenet I've built a TODO list:

1. Access MHProbe from the callbacks, which are inner classes, via
MHProbe.this. This will mean no need to pass it in to use MHProbe as a
ByteCounter. Also because the callbacks are inner classes there is no
need for the horrible hack that is making pendingProbes public static.
2. If FOAF data for a peer is not available, ignore that peer.
3. Implement error messages so that resources can be freed quickly in
the event on an error:
    - next in chain disconnected
    - timeout
    - type unrecognized
    - FOAF disabled
4. If FOAF is completely disabled on a node, rather than ignore every
peer, leading to HTL decrementing until a local response, respond with
a "FOAF disabled" error message as above.
5. Add a node configuration option for a random (not based on noderef
like current probe and swap request identifier) identifier which is set
to a random value when at its default of -1.
6. Use a Gaussian distribution for randomNoise() because many tests
assume Gaussian noise, so why not?
7. Quantize things: instead of just adding noise, report bandwidth in
as 64-bit integer KiB and store size as 64-bit integer GiB.
8. Instead of separate identifier and uptime requests, allow requesting:
    - [ identifier, quantized noisy 7-day uptime percentage ]
    - [ noisy 7-day uptime percentage ]
    - [ noisy 48-hour uptime percentage ]
9. Set timeout as constant1 * HTL + constant2 to account for
probabilistic decrement at HTL = 1.

Next week I plan to address the TODO list and any more issues that come
up in the patch set, then submit it for merging into Fred once it has
been approved. After that, I will work on FCP library support for the
new probes, and update pyProbe to support gathering and analyzing the
new information. In the event that I finish these things I can work on
making the network simulator more flexible. Our target deployment date
is Friday the 25th May or the day after.

Thanks,
operhiem1

[1] https://github.com/thynix/fred-staging/tree/newProbes
[2] https://emu.freenetproject.org/pipermail/devl/2012-April/036363.html
[3] https://wiki.freenetproject.org/User:Operhiem1/MHProbeFCP
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJPuBSaAAoJECLJP19KqmFurkUQAJZbvpgctvsZxgppkRIW/ZOY
meNolQJCutEUHl8VFSdut/6ahRwJ3gfdCgOiF/GHcbEkZqm4S+Jna58Sz6E9WBSa
MfhjQ1dULutiE1vmXN/5v/vNwFYXlATAk2XK35IrUqbCorq7NZHQhSyn61j6fLdB
cGIBOV8GCRHd2rhFu4+oP9tECyoBbL7B9OqjI3AHPEuDEyvFVfuxfk6HDcOam+4Z
dzeecRUMU4T1wquOZ9l6sDTqe8m+RRuR++WHObQ/pI/n1s848HWTM0qs3Dzj3pIU
GNcgX4dbgH8yz3veInur2i6FSkM41IdwEbV3OHro+peOIxVmCJ3c6cEbkEYLQW/R
1mICM7+6CLNkzjrxSwiPNrhVYGGKrYy7F4YSDj2GWwwfwd5aQaxzCM5CO9+VaHWj
IL0b7TvRb2wiNnJWhKlnzNxosZMfQmgGGPskeIImcXUnKPuwy/SfGhEdNifVcsY9
nCF56vGQLmEK/PPdX4CjhWRHO20l2hle5qltapd0VZ9Fyq/pcxw9SiGEW6soBlnw
1gmyxLiWe5PSSic5QD/xtNLWXnPiqnt9gwyaJu0htfZxhB57usrfzivxJGOZbenK
t+QsBGTh8rv0qDHLu1MYKdz0zkGzMSNjNI5dAUs4xKptJ5wXeMsc8K8Ho+Ho7XcF
fxh2AN9VYZSUzTPRAb2L
=rwqn
-----END PGP SIGNATURE-----

A better FAQ

Ian Clarke — 2012-05-18T16:31:28

A friend of mine has created this service: http://helpjuice.com/

It's basically a very smart FAQ that is capable of learning.  Our current
FAQ, I think, is somewhat lackluster and intimidating.  My friend has
offered to let us use HelpJuice for free - thoughts?  We should be able to
integrate it into our site reasonably seamlessly.

Ian.

Transport Plugins (Update)

Chetan Hosmani — 2012-05-15T07:13:33

Hello,

Sorry for the previous very lengthy update. I'll keep this one short.

After several discussions the basic idea of how I'll implement
transports is now clear.
I have begun writing some basic plugin architecture. The same has been
committed to the branch "refactor-1". I intend to finish, by the end
of this week, refactoring portions of code from NodeCrypto. The main
objective is to remove the "dependence on a single socket object"

Currently I have submitted code on the following new classes-

https://github.com/chetanhosmani/fred-staging/tree/refactor-1

1. TransportManager - This will now contain a list of available
transports (packet transports and stream transports). Transports
register here to let the node know of its availability.
2. TransportPlugin - A base class for the transports
3. StreamTransportPlugin - Base class for stream based transports. It
mainly supports a "connect" method and a "listen" method.
4. PacketTransportPlugin - This is based on PacketSocketHandler itself.

One issue I had was regarding opennet and darknet modes. So I have
extended the plugins so that these methods know the mode of operation
through a boolean. I am not sure if this is okay, though it is a bit
crude. The plugin will provide separate sockets for opennet and
darknet modes. The node side classes only need to mention this through
the "isOpennet" variable.

For the coming week the objective is to remove any "dependence on a
single socket object". So I am working on it in the following way-

Create TransportManager class for a node. Will list all transports
irrespective of darknet and opennet mode. So this means there will be
no issue when opennet is enabled later on.
This will be followed by changing-
NodeCrypto to handle new transports
PeerNode to use new transports

I am sorry I have not been able to update the flog. I will do some
short updates there too.

Regards,
Chetan





toad - Need specific advice on this part.
Scenario 1 - transport plugin added after initialization of nodecrypto
(say darknet).
Scenario 2 - Opennet enabled after transport plugin is loaded.

So transportmanager can be the class in between.

1. To handle new transports plugins coming in-
Transport registers, TransportManager calls on NodeCrypto (maybe
handleNewTransport(...) ) to handle the new transport for opennet and
darknet modes

2. To handle new mode (opennet) after transport plugins are loaded
The new nodecrypto checks for available transports and uses them at
initialization.

So I suppose it is okay to retain UDPSocketHandler as a default when
transports are not yet loaded? I hope this is okay with you?

Also as I mentioned here and on IRC, about using a boolean to
distinguish between opennet and darknet, which the plugin takes care
of. Is that also fine with you?
Do let me know :)

Lantern

Ian Clarke — 2012-05-13T04:29:14

Lantern is a great example of how good a Java desktop app *can* look, to
try it:

$ git clone git://github.com/getlantern/lantern.git
$ cd lantern
$ ./run.bash

Imagine if Freenet looked that good...

Ian.

Statistics Project Update #3

Steve Dougherty — 2012-05-12T04:23:25

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm still working on the initial implementation of MH-corrected probes
in Fred. I'm pretty close to having minimal functionality working, and
next week I plan to finish up the initial implementation. However, I'm
not happy with the current design, so if anyone would like to take a
look at the bleeding edge of development and possibly offer feedback,
the code is available. [1] This branch is quite rough; before submitting
a pull request or a formal request for review I will rewrite the history
onto another branch with each commit making a reasonable unit of change
along with descriptive commit messages. Which is to say this branch does
not have those properties.

The probes do not have a concept of acknowledgement or two-stage
(propagated) timeouts. The possible outcomes of a probe request are
a timeout, a node participating in the probe being disconnected, (also
not propagated - only occurs for local disconnection) or a set of
results. The possible results are:

- -Swap identifier - useful for network population estimates
- -Statistics on uptime: session, 48-hour percentage, and 7-day percentage
- -Output bandwidth limit
- -Datastore size limit
- -Link lengths
- -Freenet build
- -Whether a key is found at the endpoint
- -The number of nodes along the trace back on which a key is found

Results like link lengths, bandwidth limit, or store size will have a
small amount of random noise added in an attempt to make them less
identifiable. The originating node requests exactly one type of result.
As the request is routed the HTL is probabilistically decremented, with
a very low probability of decrementing at HTL = 1. This is intended to
make it more difficult to interrogate directly connected nodes by
sending them probe requests with HTL = 1. Each node has a hard limit of
how many probes it can be waiting on at any point in time, and will not
respond to requests beyond that limit. Timeout is proportional to HTL.

The probes are exposed over FCP. A tentative design for the request and
names for responses is available on the wiki. [2] Nodes will be able to
disable any number of these response types, and refusal to respond to a
probe is (intended to be) indistinguishable from a timeout.

Comments and suggestions are very welcome!

Thanks,
operhiem1

[1] https://github.com/thynix/fred-staging/tree/newProbesDev
[2] https://wiki.freenetproject.org/User:Operhiem1/MHProbeFCP
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJPreW2AAoJECLJP19KqmFuSYcP/jU2MI/jujZQVDfBVWbNW7xJ
M+qh0EPdQ5yWBSj5H10LNLW4Ea29KNTmsI7ttnR9GPAHYeeOZ8F8AVZYB0eQd3jX
IRecJiwWm2N48OcgMtEnpTbaaLYX9a70a/PLXQWZldX8aOwkbVefFn0DXF0D+j5u
HgCDLuBa26tO5XwaGgmxkhLyJNfO/R94MezWUNaxF2HiwlP3zajUFDGRo/hZSC89
FNrqJYlAimfogli0RPatTNoJ1fG4YhQiyeQyxChCvooH+6FIHFzgI/gGVNAKam9i
xDyk0cew6eQBp1L6aPzyzpgw/jkK0oXWB/jyaqLvRZM2aYY+jWCPrq1Z6jXzGnq8
Ps8VLkpNYGoegobiQCtf9kJXnrJ1UjN7wRky2Cvi6pwoDkulHS5HSv82lSSjeqK4
aWOkMfjQXFNjGCZj1CwlD2sgFhRkozc4HPkqe3ub8yVkshBto1W6xpApvQc7iI2F
5mVOLn+uaat9glq9OrYtwMB7nMJVyFFp9MGDRLk58z3vhk28kj6qIQmDIzAh6urc
tDVgAtcrIckYN1j3WPnSJqN4ev+8lH49tSkfef3xnAIiEo5lqvT80xZi7PQoKKn3
5WW7KWGBlKoqMVWZlkKysB8Ks9xF5EzvCQ6//B2uZdwFQm2Xa0CBtNvqEMNTXuWI
8uTWVYqYdvB7Qub3HZD0
=P41g
-----END PGP SIGNATURE-----

Transport Plugins: Update (GSoC 2012)

Chetan Hosmani — 2012-05-05T12:14:43

Hello,

This is an update on the first stage of my project. I have been
working out the details regarding modifications in the core to
simplify the implementation of the transport plugins. This refactoring
stage will involve some minor modifications and some major ones. These
are based on the discussions on #freenet and #freenet-chat. However I
would like to avoid doing changes that would bring about major
differences in the way things function at present.

I have planned to finish refactoring by May 21 (official coding begins
for GSoC). I intend to do that in 3 stages. I have also simultaneously
discussed the entire implementation of transport plugins here. I will
review the feedback and this report, to evaluate what constitutes the
refactoring. After those changes are made I can work on implementing
the needed core level changes, which mostly involve design for
streams.

1) Cryptography and Node

This will involve mainly the classes NodeCrypto and Node. I intend to
make NodeCrypto function purely for keys and not do any network
related task. Presently it also handles binding to the port, selection
of ports for Darknet mode and Opennet mode.
I am considering moving these portions to TransportManager and
PortManager classes.

TransportManager - A registry of registered transports with other
information. Initially I ll dump the refactor changes of UDP into it.
Later on, once UDP is ready as a plugin, along with TCP, it ll be a
generic class for any transport. There will be two objects of this -
one each for opennet and darknet modes.  NodeCrypto e.g. queries
TransportManager to find all the transports' addresses (as strings) to
put in the noderef.

Now UDPSocketHandler is equivalent to a UDP transport plugin, only
it's currently part of the core of freenet. So for the moment
TransportManager can communicate with UDPSocketHandler. That I believe
will be an intermediate step before actually converting
UDPSokcetHandler to a plugin.

PortManager - A registry of all active ports used by freenet - 1) List
of ports on fproxy page 2) List of ports for UPnP
Must contain - 1)Port number 2)Socket(TCP/UDP) 3)Forwarded
4)Description 5)Transport reference
However PortManager is second priority, and will not be addressed if
time does not permit.

Also a major refactoring change would be regarding the fact that
objects no longer take a socket or a port for granted. All queries
must go through the TransportManager. I would need help at certain
places to figure out how this can achieved. For e.g.
NodeCrypto/Node/some other classes assume that an unique darknet port
exists. toad suggested we use a debugging identifier here(some default
number, etc). That makes it simpler for me too.

2) Messages and Packets

These part have been discussed on #freenet(and #freenet-chat) and here
are some issues and conclusions-

a. The existing architecture supports only packet based transports.
The first step would be to draw a clear distinction between
packet-based(connectionless sockets) and stream-based(connection-based
sockets). Thus we will need separate classes for streams(with a base
class for streams and packets), that will handle Message-in-flight and
Packets-in-flight, tracking them, allowing for transports to be
switched midway. This will be handled differently in streams. For
Packets the existing classes can be used, however with modifications
to address the following.

b. For streams we need two features - To handle messages as a whole
entity, and acknowledge once an entire message has been received; and
To support partial messages, if for e.g. we have just switched from
UDP to TCP or we are sending a really large message on TCP.

c. NewPacketFormat currently 1) takes a byte[] from UdpSocketHandler
and decodes it 2) creates and sends a packet. Now the idea is to
create NewStreamFormat that handles streams, and also provide a pair
of streams StreamConnectionHandler for the plugin to write/read.
NewPacketFormat handles the list of message-in-flight. A new object
must be used for this purpose while NewPacketFormat performs lesser
tasks. Both NewPacketFormat and NewStreamFormat can communicate with
it.

Question: MessageWrapper tracks a message-in-flight. But is it really
necessary to create a base class for MessageWrapper and create
StreamMessageWrapper too. Or would it be simple to make necessary
modifications(if there are any) to MessageWrapper.
Please explain the purpose of doing so, as I have very little
understanding, as of now, of the object Message. Also how it will be
treated differently for streams against packets.

d. Another issue is regarding FNPPacketMangler. This is an important
aspect of freenet, as it deals with the handshake between two nodes,
and the exchange of cryptographic keys. So the issue being whether per
NodeCrypto we have an object or per transport we have an object, of
the FNPPacketMangler type. And assuming these are packet based
transports.

Question: How can the equivalent of FNPPacketMangler be created for
stream based connections. I don't think this was discussed at all or
in detail on the IRC. Some suggestions here would be helpful. My
knowledge is that the JFK protocol works for packet based transports,
specifically UDP. How would this function in case of streams?

e. Another thing that needs clarification is regarding who decides
when to send the packets. Since stenography is all about the timing
and the format of the packet, the plugin must have control. However
that would make some other transport plugins difficult to write. toad
suggested the following. To quote him- "PacketSender constantly loops
over peers to send packets to them. We'll probably have two separate
mechanisms for packet-based - one where we are just wrapping packets
with different headers, so the node decides when to send a packet, and
one where the plugin decides when to send a packet and demands data to
fill it from the node. The latter is "constant bitrate", or really
accurate fakery; it's a long term project, as is how to fill up the
packets when there's nothing to send." This sounds acceptable, however
when I get there I will discuss how this should be implemented. But I
believe that it will not affect the initial stages of the project.

Question: How does the same work for streams? Firstly the transport
must have its own flow control mechanism to decide when to send. Also
there must be a way to let the node know this. I am guessing
NewStreamFormat handle this? It must also create
StreamConnectionHandler for the plugin to read/write. Must also do
encryption using existing BlockCiphers? It must convert the encrypted
data into a stream? I suppose I can draw parallels to NewPacketFormat
and do it.

3) Plugins, streams, transports

I will need to create a base class for transports - TransportPlugin.
This is designed as per the developer's requirements. However the
major idea is that it would open sockets, and listen for connections.
For a packet-based plugin as mentioned it is called by PacketSender to
send data. If it wishes to for stenographic reasons, can send dummy
data by itself. Also it will handle all necessary parsing and encoding
for whichever protocol it is trying to mimic. It creates its own
threads to copy data from freenet and send it.
For a stream-based plugin, it runs its own loop of getting/writing
data from/to StreamConnectionHandler(node-side), encoding/decoding,
and sending/receiving to its socket. It must have its own flow control
mechanism.

To be discussed-

One major point I haven't discussed is how multiple transports work.
For e.g. a node might be able to communicate using several transports,
while its peer might not be able to do so. It had been the case that
for datagrams one needed a single port that sent messages to all its
clients. But with multiple transports, the node needs a record of
active transports for each peer. Apart from this we need to discuss
how the messages are distributed between transports. This also means
some transports might be slow, or the user prefers one of the
transports. Although I wish to address it within the timeline of GSoC,
I am not sure how it will turn out. But for the moment my idea is to
be able to choose the needed transport, although the framework to use
them simultaneously or switch from one to another will be supported.
If I find that the logic to implement selection of transports is easy,
then I can implement it. But I assume that needs some usage analysis
as well. Based on that we can comment.
My objective is to atleast get a functional freenet that will atleast
figure one way to connect to a peer, and this can be different for
different peers.
I need your opinions on this. But atleast some modification of
PeerNode to keep track of which transport works is a must.


This is a haphazard report, that has questions, issues as well as
conclusions. Based on feedback and suggestions, for the next two weeks
I wish to work on refactoring changes to support the above. Also
thanks to toad for making it very simple for me, and the fact that
over a month, all his design implementation ideas mentioned on email
and irc are consistent and complementary to each other.

I will work on a separate branch and I am not sure how it follows from
now on, as I submit code. The present idea is to create a branch for
every task, to be completed every week. Comments and suggestions are a
must!

Regards,
Chetan Hosmani

Statistics Project Update #2

Steve Dougherty — 2012-05-05T05:43:01

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've completed an initial run of simulation work on probes. The code is
available, [1] as well as the simulation results from which the plots
were generated. [2] The point of immediate interest though is the plots
themselves, [3] which show the predicted network coverage of different
probe routing techniques on networks with ideal degree distribution
(more on this later) and that following the degree distribution of
Freenet as measured. [4] Link lengths and locations do not factor into
this simulation because probes take only degree into account and are not
seeking any given destination; their goal is only to average out to
distribute endpoints uniformly throughout the network.

The ideal network distribution has each node add a fixed number of
remote connections without regard for the number of connections it or
the nodes it's connecting to have. I don't know whether this or having
each node have the same number of total connections is the ideal. The
results of the simulation did not appear to greatly change with
network size, as shown by the consistent behavior between the 12,000
and 45,000 node versions of the MH-corrected degree-conforming
simulation. [5]

As expected, the plots suggest that using Metropolis-Hastings correction
will be an immense improvement in endpoint uniformity over the current
uniform random routing, but specifically suggest that an HTL of around
20 hops is close enough to a baseline uniform endpoint probability to be
a good starting point. I've noticed that these CDFs aren't a very good
format for demonstrating closeness of distributions, given overlapping
lines, but I don't understand the Kolmogorov–Smirnov test yet, so I'm
planning to just use these results as a guideline and begin implementing
the new probes next week.

The gnuplot scripts to generate the degree and link length distribution
plots are part of pyProbe, [6] and GNU parallel [7] is used in test.sh
to run simulations in parallel. In the simulator source there are
scripts from an earlier effort to plot coverage as percentages, but that
was even less clear than the CDFs.

Comments and suggestions are very welcome!

Thanks,
operhiem1

[1] https://github.com/Thynix/routing-simulator/tree/dev
[2] http://asksteved.com/plot-source.tar.xz
[3] http://imgur.com/a/Z8SBS#2
[4] http://i.imgur.com/ehfBP.png
[5] http://i.imgur.com/rtRIB.png
[6] https://github.com/Thynix/pyProbe
[7] http://www.gnu.org/software/parallel/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJPpL3dAAoJECLJP19KqmFuNn0QAMsA4nzk6AfPf8pIqrmoEW8U
2jcc7L3KnUkCIgvh9FyhJkZ9Fm42zCoqgxXmyavM9T18ZO52eYaNMaSfkA5FWltk
iBElymF7ZCGd3ERX9XPirbXGDeMbpNsFbVHFoJbqKzb94MrnSUivLsVQz0Nl1KOJ
g1yfYdA4RK3ywYIvwS7nWkIIrxhuik/Jzjaq5cuqY2L6i3DgiM9gjYweyJLpzt6r
k/mRNOuKTI0MSdqMWclBFXOEOzTg/vZKZSvvslpZRwt0Opp+nK9VKBMVzvqiqUpr
G9EEke4vPqU8OdWffxqu3nF5ZXlr4aB3mWw6B7zimE+7C3Wvk3oQHxxv/p/PqD96
GQ/sUbkFERSv/SnMDCuz8BVoPNihTyohvRJmeW92P2KpFCJ7Ynsx1uC6XLKDQVIO
Qxds7EUKkdEQaEbNYRKMkzx9qzOszRZlcvLElX2Fgw15KvTMKmMDb/7t1DpbBysY
tl7JnkYW6crq3nvBpWu3JFmSOYERhEzzKxkRsE76DVzkBz35AYOb1ZTLx06mEgP4
F8HFs31Ra8LNlVCoN5jEHW3WhUIVkVtx8zauXGOtjJuY4ePhEXS9TvXOKAbvxMiA
d/Nu78MORKBdq1repSMIcCLUl1Ya0AT0BEugvJ4KyKPScl0JL0GPOiFBG8Dr01GZ
pQtYR4VDpcLlPzPkq1xj
=apcO
-----END PGP SIGNATURE-----

Freemail 0.1.1 and 0.2 released

Martin Nyhus — 2012-05-01T18:11:37

After being in development for nearly a year Freemail 0.2 has been released.
The two main features in 0.2 is the new identity management which uses the Web
of Trust plugin, and a webmail client. This work was mostly done as part of
Google Summer of Code 2011. Due to the identity management changes this
version is incompatible with the earlier releases, but to ease the migration
the two versions can be used at the same time without any issues.

Version 0.1.1 fixes a number of bugs including one potential security issue
that could enable an attacker with access to the Freenet log files to fetch
emails from Freenet after they should have been inaccessible.

Both versions will be available through Freenet after the next update, and from
CHK< at >xXc1H2eh-1OtCoE4Z0-G4~5SfOFpTWa6YWOP~D4PyuA,dKRuAVWFlHd8ZNvFsIJj1Hzzuyoc7EfoVIXYgz-rfsg,AAIC--8/Freemail-0.1.1.jar
CHK< at >S0AnW~5CXyh3pLkzJHbRm5xj8kf26GyLcT3Zuf1UAsU,pt-TGV39s1ucWNTkZKMdo1dQyaxzaUsSjXlYtBAqfvc,AAIC--8/Freemail-0.2.jar

I am very interested in feedback about the reliability of Freemail, any bugs I
have missed and performance problems. This can be reported through the mailing
list, the bug tracker and on Freenet.


Detailed changelog for 0.1.1 (also included in 0.2):
  o Security fixes:
    - Log message fetch and insert keys at debug instead of normal/error. If a
      collision occurred the new slot would be logged at error, which
      would break the forward secrecy of the slot system until the log was
      deleted. This would enable an attacker with access to the log files to
      retrieve messages from Freenet.

  o Bugfixes:
    - Folders deleted using a mail client are now deleted properly
    - Fixes a crash that could occur if a mail client connected while Freemail
      was shutting down
    - The startup message now shows the correct licence (GPL)
    - Fixes a bug where certain email addresses would cause received messages to
      be empty
    - Fixes a race condition which could lead to Freemail hanging
    - Don't delete CC headers from a message before sending
    - Always print a log message when Freemail isn't connected to the node
    - IMAP: Remove extra space that was printed in a fetch response without a range
    - IMAP: Fix error message when the end of a range was invalid
    - IMAP: Handle strange sequence number ranges
    - IMAP: Remove \* from permanent flags since they were not stored
    - IMAP: Fix append with two or more flags
    - IMAP: Reply with error if the append length couldn't be parsed
    - Fix various locking issues
    - Don't log the recently failed fetch result as an error

  o Improvements:
    - Improve the explanations on the create account page
    - Only resend the RTS once per two days instead of once per message in the
      outbox per two days, reducing resource usage for unacked messages
    - Send messages in the order they will be received, improving performance
      when sending a large amount of messages
    - Alternate between sending and receiving, stopping sending/receiving a large
      number of messages from blocking other operations

  o Build improvements:
    - Compile for Java 1.6
    - Include git describe output in version
    - Enable warnings when building
    - Make Ant and Eclipse output files to the same location (build/)

  o Code changes:
    - Add unit tests for various classes (mostly IMAP)
    - Improve errors returned/thrown by HighLevelFCPClient
    - Add type parameters to all code
    - Add missing < at >Override annotations
    - Throw AssertionError in some cases that should be impossible
    - Use constants for config file keys
    - Respond to interrupts in the FCP code
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Yet Another GSoC Greetings

Pouyan Zaxar — 2012-04-28T09:25:44

Hi everyone!

I have been accepted for GSoC 2012 and I'm going to work on
reimplementation of FProxy using Apache Wicket. The main objective is to
separate structure (HTML) and design (CSS) of the web interface. Finally
it makes it possible for anyone even without prior Java knowledge to
write their own themes for Freenet.

Since a complete make-over is going to be done, this would also be a
good chance new features/functionalities to improve UI functionality:
suggestions more than welcome.

infinity0 is mentoring this project.

Cheerio
pausb
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Statistics Project Update #1

Steve Dougherty — 2012-04-28T02:34:32

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Here's an update on my progress on the statistics project for the
first week:

The current probes are biased towards better-connected nodes: at each
hop they choose a random peer to pass the request to - this is a random
walk. However, because better-connected nodes by definition have more
connections, the requests will be passed to them more often and they
will be over-represented in results. To address this, the new probes I
will implement will use Metropolis-Hastings correction: unlike the
uniform random walk which always uses the random peer it picks, it is
less likely to pick a well-connected node, and more likely to pick a
poorly-connected node. As there are more chances to pick a
well-connected node than a poorly-connected one, this balances out to
a uniform probability to pick any given node.

I'm starting from evanbd's network simulator,[1] which is able to
generate networks based off theoretical models and perform some routing
simulations. It can now also reproduce a given degree (number of peers)
distribution which allows simulating the network as it is measured to be
in addition to purely theoretical models.

Currently I'm working on plotting the distribution of this routing
strategy with different limits on hops before returning information -
Hops To Live - HTL. This is to get a good number to start with in the
implementation of these probes. I will also refactor the simulator and
make it easier to configure: currently values are hard-coded and
changing the simulation parameters means recompiling.

My goal is to have this area of simulation done and have begun planning
if not implementing the new probe requests by the end of next week.

Thanks,
operhiem1

[1]
USK< at >gjw6StjZOZ4OAG-pqOxIp5Nk11udQZOrozD4jld42Ac,BYyqgAtc9p0JGbJ~18XU6mtO9ChnBZdf~ttCn48FV7s,AQACAAE/flog/29/200911.xhtml
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJPm1czAAoJECLJP19KqmFuZ2MP/1VRYipBzRKOQDjkKSIl0dq6
+FmJg/lPwBltn/gDXMRB0+vY1Msdbd/ydXx4JEqtJzeHUC/bqreLxMov2EqzYZAl
0kLmuNUp7Cn9h7PlhAVpQ5t7BMLbJF5UikDLHz3vUfAE4bKgDuNwiy9Z0Uf+zqr/
esRD0qWn24dACBOA4rRAkb0b+14UgIKgMj3ohMRkpK2NgHuB4OUmGMOQIf/9h/Vb
/FwruM6RdiUUd0g+ldKhzpfflqahKt30xjHCQeNvMZRx7N0OMJfnBUTbCP+ogaD5
aM/BoXoKo1WUCjMKUN8vFvby1BF+4zolywyIhxUqrQv76yjoGvpI/K3qdsH5YWUG
AZJeVbbQdV959S1waAU4gji2iEKVzhretZNBMQApY421WG1c0C8MUtNOY37zZ3iO
q/Nxlck9vVDTNgXAs3vzm7VbuKeeyEfqHe+imIYhiqYjfqUQteSgO70T2gpMrw6f
ojbs2ohtM7sXLPp8P6Lf57VcHEsmSUJkWTua7ycdPXGHmdo/MLqFLW3UVYsWzy5P
c67y6yjvxVVqJfbu38FQ/mTqgOGTduU1568BYApBO9bp6/b+2jkZcfcIsL8apGM2
vvjtxxxCfoESHobwH59NoSezslGxHBddEpWcDl2ggY6NgkzH72iAtjF4a4GRkGJM
ju5xKSZ40OCpLNUCA/M0
=cnM8
-----END PGP SIGNATURE-----

GSoC 2012 Greetings

Steve Dougherty — 2012-04-25T17:00:35

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've been accepted to work on a network probing and statistics project.
We know very little about Freenet's network health and how to improve
it, and I hope to fill those gaps in knowledge with both simulation and
measurement. Specifically, I plan to:

* Build simulators to establish good parameters for network probing,
  reproduce the current network state after figuring out what that is,
  then make changes to improve that behavior in simulation.
* Depreciate the current probe requests in favor of those in bug #3568.
  [1] These report more useful information in a more privacy-conscious
  way, and will be exposed over FCP.
* Write a tool to run networks probes and present plots of the results.

evanbd is mentoring me on this project. On Fridays I will give updates
to the development mailing list, FMS, and Sone.

Thanks,
operhiem1

[1] https://bugs.freenetproject.org/view.php?id=3568
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJPmC2rAAoJECLJP19KqmFubyEQAK6a0xbKt3wIFuv5mEg0pDiJ
FTqN9FoT450Wzw245Yn7ESt+hjno1MQXURjgoyzXWcDAD4isq9vSBwvTiDzDAB+9
HtFI0xocLyNr0Dj61rQh2vUbbHUC6lzw7PIi7vN6M/WJopKyWxScv2V/6PxINl2x
UN5tno/2idrVZ0drISk/FH7QoXMBBnMDTmJUrm83ws5ObOh5gZVIfisjsYs+PSxm
LSZehZDlqDwUFP8fQ9Nb0WnC63JaISQWUDIC22JZwoYZxddLnJRqSsEL87mjHgp5
C4l/DghijWkUiIHbLmknzGcwISG9wZWVknNRqR1rhs3YHB29r7DgYlbiuE0l58xd
4JW0UApNCuovgd6vZ7EfWXebk2tKLH5yBR0ReB+nJ0ETYDBF2hL5MxnXt9pKHGsG
0TV6PDv4vmMUamhnrRX2oGZH23CnlE8X+4/yUn+p+z2ViYViEztoSCJtECn15AFS
FV+ywQA3eq/Irs5qKtBeoRyvqeXQnTMZ7pYLzUynanQn4YCLa2vFor71/PHKU8e5
Q5zzKCHRXtXtSrJM0c06DNxoyC0lTqxQNnkhhPeZR26jGOL+rqdykwVXbVQ4Eugk
P3VNyQBPNyhuiATe7I5qg3pp6nd4LLdnRVIgnPgG4iKbSa+3mTKkqhAcYzG7daox
6Ks7bDcoOMIgCDye0uhJ
=c5cX
-----END PGP SIGNATURE-----

GSoC, please reply today or Thursday

xor — 2012-04-18T19:26:55

[19:32] <p0s> toad_: whats the situation of GSoC? any applications for wot/ft 
related projects? 
[19:33] <p0s> toad_: deadline for mentor applications is friday, so i should 
apply today/tomorrow if i'm needed
[20:01] <toad_> p0s: i have no idea, ask nextgens / sanity

Improving build security against back-doors: Bytecodeverification

Matthew Toseland — 2012-04-10T19:01:52

We need a script that downloads the latest released jar, and fetches the corresponding git tag, compiles the code, and compares it to what has been released. Nextgens had a script doing something similar for a while to check indenting changes; Java compilation to bytecode is deterministic, but you can't just compare the jar's, you need to break out the class files and then compare them. Whoever runs this (hopefully more than one person) would need to have the same setup that builds are generated on. When I release a build, I compile on my system, which is Debian stable. The script could be totally automated with a little work (and would have to be adjusted for releases by other people, but this is easily checked by who signed the tag).

Anyone want to write such a script? Nextgens do you have the old whitespace change checker script still?

I suspect we could get suitable volunteers fairly easily.

IMHO it is important to have third party verification (with said third parties not being connected to FPI and ideally some of them not being traceable). For all we know my computer is backdoored and it's releasing patched builds with surveillance addons already! And future laws, in the UK and elsewhere, may compel developers to do this themselves, secretly.

This should be relatively easy to implement, and should put a lot of people's minds at rest. So anyone want to develop such a script?
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Promiscuous opennet

Juiceman — 2012-04-10T18:32:20

Should we do something about the amount of announcements/refs handed
out?  Seems a good way to harvest opennet (known problem, but perhaps
we could slow it down...) or perhaps monopolize/monitor the key-space
somehow?

opennetSizeEstimateSession: 18985 nodes
opennetSizeEstimate24h:        12435 nodes
opennetSizeEstimate48h:        14855 nodes
nodeUptimeSession:               4d23h


Seed stats
IP                    Connected  Announced  Accepted  Completed  Sent
refs    Version
24.126.77.67        1766        352            352            352
      520            1406
202.134.203.35     1785        279            279            279
     1812          1377
213.134.163.3       1792        275            275            275
      1819          1388
68.227.101.68       1793        293            293            293
      1747          1389
62.107.42.7           1803        257            257            257
        1774          1389
67.61.122.246        1840        267            267            267
       1809         1397
82.66.7.51             1843        247            247            247
         1400          1397
78.105.55.206        1846        290            290            290
       2053          1388
68.144.7.7             1853        438            438            438
         148            1406
80.217.175.104      1868        314            314            314
      2053          1388
67.169.243.33        1919        317            317            317
       416            1406
208.76.88.92          1939        327            327            327
        1980          1376
98.21.244.113        1951        317            317            317
       2238          1376
50.4.40.146            1979        308            308            308
         2106          1385
71.204.143.231       2041        292            292            292
       1629          1385
81.56.65.62            2043        278            278            278
         2209          1384
88.174.162.112       2071        297            297            297
       2104          1378
88.90.64.247          2110        278            278            278
        1658          1397
88.172.49.91          2120        305            305            305
        1730          1372
69.228.173.126       2146        318            318            318
       2247          1364


--
I may disagree with what you have to say, but I shall defend, to the
death, your right to say it. - Voltaire
Those who would give up Liberty, to purchase temporary Safety, deserve
neither Liberty nor Safety. - Ben Franklin
_______________________________________________
Devl mailing list
Devl< at >freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

new external Java updater

Ximin Luo — 2012-04-09T16:51:59

This is one way to allow us to deploy the new split freenet-ext.jar. It's not
the simplest way, but it has certain other advantages, described below.

## Rationale

Unify existing platform-specific and distribution-specific installers and
freenet.jar's updater functionality into a single platform-indepedent
installer+updater that can handle a greater variety of installation layouts.

Most of the existing installers do very similar work, but have slight
differences in how they handle installation layouts and updates, etc. Some work
is platform-specific, such as setting registry keys or cron jobs, but the large
bulk of the work can be unified.

## Overview of new design

Along with freenet.jar, have a update-freenet.jar. This jar will take care of
both the initial installation, and any subsequent updates.

- It will be runnable via java -jar, or double-clicking in windows. (It should
read from stdin after execution so that windows users can see the output.)
- It must be self-contained and include any third-party libraries it depends on.
- It will replace all of the following:
  - 1run.sh
  - update.sh
  - update.cmd
  - all the helper *.jars used as part of the pre-install process

The general steps:

- attempt to download a "distribution" of freenet via HTTP into a $TMPDIR
  - if $TMPDIR already exists, this is skipped
  - if --only-local is given, this is skipped
- verify $TMPDIR using signatures and checksums hard-coded into the updater
- install components and update config files (expanded below)
- if a new wrapper was installed, launch a new one with the same {cwd,env,args}
and stop the current wrapper. this probably needs to be done from a separate
process. (if this turns out to be a problem, it's a problem for all updaters
not just this design.)

### Installing from scratch

Anything not listed above will need to be added to a platform-specific
installer, as is the case currently. The installer can then call
update-freenet.jar at an appropriate time during the overall process.

### Updating

The "updater" functionality within freenet.jar will be reduced to merely
downloading a new distribution from freenet, placing it in $TMPDIR (as
described above), then restarting itself. The update-freenet.jar will pick up
the changes and install them, then re-launch freenet.

This is accomplished by setting wrapper.java.command to a script that does the
equivalent of

if [ -d "$TMPDIR" ]; then
$JAVA -jar $FREENET_UPDATER --only-local
# the wrapper will automatically restart freenet
else
$JAVA "$< at >"
fi

This should be possible even with windows cmd.exe, but if not we can easily
write a small .exe to do it.

Env vars can be set in wrapper.conf - see [1]. Having $FREENET_UPDATER as an
env var allows us to update the updater in the same way as other components,
which would be necessary if e.g. we change certificates / keys.

[1] http://wrapper.tanukisoftware.com/doc/english/props-envvars.html#definition

## Design of update-freenet.jar

update-freenet.jar works on a "distribution" which is downloaded to a $TMPDIR
first. The distribution contains a manifest file with the following information

<stem> <version> <type>
- stem is something like "freenet-ext-$V.jar"
- version is any string, where ascii sort order determines age
- type is one of "jar", "lib", "bin", "wrapper-bin", "plugin-jar". each type
has an installation policy (expanded later)

Locally, we also need to store the following information in a "register" config
file:

<stem> <version> <fullpath> <in-use>
- fullpath: as installed on the filesystem; the basename must match
- in-use: whether in use by the current set of config files

This lists all the files installed by update-freenet.jar. When files are no
longer in-use, they can be deleted. To ensure safety, this file should be
updated in the same step as e.g. wrapper.conf, and likewise backed-up.

### Update algorithm

- download distro
  the HTTP downloader (part of update-freenet.jar) might choose to skip files
if they already exist in the register, but to keep things simple and improve
retention, freenet.jar should probably simply download the entire distro

- touch .update_in_progress
- for each (s, v, t) in $TMPDIR/manifest
    if register.inUse(s, v) { continue; }
    Policy p = getPolicy(t);
    File t = p.getInstallTargetLocation(s, v)
    if (!f.exists()) {
      register.add(s, v, false);
      cp $TMPDIR/(s, v) to f;
    } else if register.containsButNotInUse(s, v) {
      // incomple install
      cp $TMPDIR/(s, v) to f;
    } else {
      // already exists on system, possibly as a system library
    }
- for each config "f"
  if (no backup) { backup } else { restore backup /* start from scratch to be
safe */ }
  after that, rewrite "f" to point to newly-installed files
  - note: this INCLUDES the register, since we need to update the "in-use"
flag; but adding entries for newly-installed components is done above
- for each config "f"
  - remove the backup, or move it to an archive folder so it doesn't interfere
with future logic
- rm .update_in_progress

(Of course, there needs to be

### Installation policy

getInstallTargetLocation(s, v) {
  - usually this will be simply register.getPath(s, any-version).dirname()
  - or register.getPath(any-s-of-same-type, any-version).dirname()
  - it should only fall back to a hard-coded default ONLY IF ***all other
options have been exhausted***

### Migrating from the existing install scheme

We can migrate to the new scheme by bundling update-freenet.jar and the
platform-specific stub scripts into freenet.jar itself.

When freenet starts, if it doesn't detect the new scheme (e.g. if wrapper.conf
doesn't already define FREENET_UPDATER), then it can write those resources to
the appropriate places (e.g. the same directory as wrapper.conf).

We also need to initialise the register using existing system information. We
can obtain each <fullpath> as follows:

1. wrapper pid: WrapperManager.getWrapperPID()
2. wrapper bin: via syscall from wrapper pid
3. wrapper.conf: in command line of wrapper proc, via syscall from wrapper pid
4. wrapper lib: wrapper.conf
5. freenet.ini: wrapper.conf
6. freenet jars: wrapper.conf
7. plugin jars: freenet.ini

In this specific case, we can reverse the <stem> <version> from <fullpath>
because we also know what files are currently distributed.

After all of this, the next time freenet restarts, the new updater will be in
effect.

## End

I think that covers everything, or hopefully it gives enough info to
extrapolate solutions for everything else.

X

freenet-ext.jar splitting and updater: minimumrequirements

Matthew Toseland — 2012-04-06T19:16:46

As I see it, the minimum functionality we need to split freenet-ext.jar is:
- A manifest file in freenet-ext.jar listing each, for other file that the new freenet-ext.jar depends on:

New SSL certificate

Florent Daigniere — 2012-04-06T07:52:04

Hi,

We are in the process of deploying a new SSL certificate accross our different websites,
 here is its fingerprint:

SN: 11:21:0A:32:B3:66:76:7C:CE:E6:03:83:CB:0B:79:96:0E:D8
MD5 Fingerprint=E9:77:77:7E:92:32:5A:13:2F:C6:D1:20:21:C8:7D:B5
SHA1 Fingerprint=D9:1C:04:A4:68:FC:1A:FC:94:89:3E:1A:43:BE:0B:62:45:1E:97:41

Certificate chain
 0 s:/C=US/OU=Domain Control Validated/CN=*.freenetproject.org
   i:/O=AlphaSSL/CN=AlphaSSL CA - G2
 1 s:/O=AlphaSSL/CN=AlphaSSL CA - G2
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
 2 s:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA

Florent

bigint-gnu-gmp and bigint7

Matthew Toseland — 2012-04-05T14:29:23

What is the purpose of the bigint-gnu-gmp branch? And bigint7?
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Space stripping

Matthew Toseland — 2012-04-05T14:24:47

Re 2d011ed5bd2f59b0a1a90383566b2f62378245b8 (origin/accept-linewrapped-keys):
IMHO we should only try stripping spaces after we have failed to parse without stripping them. So it should be an option on decode(), and FreenetURI should try with and without.
_______________________________________________
Devl mailing list
Devl-RdDMkVZAZeuJnvDnx1genB2eb7JE58TQ< at >public.gmane.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

GSoC 2012 Transport Plugin

Chetan Hosmani — 2012-04-04T06:15:26

Hello,

With only three days left for the deadline of submission of proposals
I was hoping I could get some suggestions on my proposal.
This is my only proposal. I am also working on another proposal to
Freenet itself (implementing JCA).
Any suggestions will be really helpful.

The proposal can viewed publicly here-
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/chetanhosmani/1

Apart from adding references and links, this is the final version.
Please leave comments.

Thank you
Chetan