> Notably this project was conceived by a backroom decision to dump the original Freenet development team's work,
This is a false narrative, from the Freenet FAQ[1]:
Why was Freenet rearchitected and rebranded?
In 2019, Ian began developing a successor to the original Freenet, internally named “Locutus.” This redesign was a ground-up reimagining, incorporating lessons learned from the original Freenet and addressing modern challenges. The original Freenet, although groundbreaking, was built for an earlier era.
This isn’t the first time Freenet has undergone significant changes. Around 2005, we transitioned from version 0.5 to 0.7, which was a complete rewrite introducing “friend-to-friend” networking.
In March 2023, the original Freenet (developed from 2005 onwards) was spun off into an independent project called “Hyphanet” under its existing maintainers. Concurrently, “Locutus” was rebranded as “Freenet,” also known as “Freenet 2023,” to signal this new direction and focus. The rearchitected Freenet is faster, more flexible, and better equipped to offer a robust, decentralized alternative to the increasingly centralized web.
To ease the transition the old freenetproject.org domain was redirected to hyphanet’s website, while the recently acquired freenet.org domain was used for the new architecture.
It is important to note that the maintainers of the original Freenet did not agree with the decision to rearchitect and rebrand. However, as the architect of the Freenet Project, and after over a year of debate, Ian felt this was the necessary path forward to ensure the project’s continued relevance and success in a world far different than when he designed the previous architecture.
> The new "Freenet" does not have anonymity as a design goal anymore,
Because the new Freenet will have a menu of anonymity options rather than committing to a one-size-fits-all approach, while also addressing the issue of illegal content[2].
As the FAQ explains, the existing maintainers didn't agree with my decision, but I stand by it - particularly in light of the fact that we now have a working decentralized group chat on the new Freenet, something that the old architecture could never have supported.
Whether or not it was the right decision will be determined by the outcomes, which so far are promising, because we have a working network that does things that the old architecture could never do.
I'm very glad to hear that—the anonymity of the original Freenet has led to it being a very unsavory place that was more well known for CSAM then anything positive or useful. As an outsider, it sounds like this new direction is the right choice for Freenet to try and attract new users and fulfill the team's original goals.
The new Freenet will support the creation of anonymity systems as services on top of it, which is much better architecturally than tying the platform to one approach to anonymity as I did when I designed the original Freenet.
We will also have a decentralized reputation system that will protect people from being exposed to unsavory or illegal content, a common criticism of the old Freenet architecture.
So he forked the project and went his own way. I am not sure I see the issue here. This is how we do open source on the internet. You don't have to join him, but he also has the right to go his own way too.
Looks very interesting! I stumbled across your webpage a few months ago while looking into the state of peer-to-peer. Glad to see p2p projects are still active.
For values that don't have a natural merge function (or where you don't want to bother writing one), would it make sense to sync update logs instead? That is:
- The synced value is a history of client updates, sorted in some eventually consistent order (e.g. by hybrid logical clocks). Merging takes the union of the update sets.
- The user-visible value is the result of processing these updates in order, using arbitrary contract code.
This is overkill for simple last-writer-wins values, but it lets you support fairly general data types & arbitrary update functions, including ones that preserve application-specific invariants.
The Automerge CRDT library works like this already [1][2], but it only allows specific updates to JSON data. Sharing code via your contracts solves the hard part of generalizing that to arbitrary data & updates.
> For values that don't have a natural merge function (or where you don't want to bother writing one), would it make sense to sync update logs instead?
Yes, in fact you can implement this within the current framework, for example with our group chat River, each room state maintains a list of the N most recent messages sorted by (approximate) timestamp.
The idea is that you can adapt the merge logic to the needs of the specific application, and I think a time ordered event log will be a common pattern.
I think better approach for "ghost keys" would be requiring X amount of crypto to be sent to 0x0 (burning). Current implementation (requiring donation to freenet) basically gives freenet foundation infinite reputation (including any other potential project that would accept ghost keys as identity), kinda breaking the decentralization aspect
Ghost keys will ultimately be just one of a menu of options for bootstrapping reputation in a decentralized reputation system. They have the advantage of simplicity, anonymity, and helping to fund the project, but as you correctly point out - they are centralized.
A cryptocurrency-based solution like you suggest will undoubtedly be one of a menu of reputation bootstrapping options that will develop over time.
Very interesting. Beyond ideological motivation, I’m curious what the long-term incentive is for someone to run a peer.
For example, if Freenet were to reach scale, it could eventually need some kind of economic primitive around it. Something similar to how Filecoin handles decentralized storage, but for app state. One way to do this could be paying peers to keep app state available, serve it reliably, etc. and prove they are doing so.
Why would you want to exchange private messages with anonymous counterparties?
- You participate in a secret cabal, and don't want participants' identities be visible to each other.
- You're a journalist, and want to give your informants in sensitive matters, or from oppressed countries, a way to securely interact with you, so that you won't be technically capable of reporting their identities.
- You're selling illegal goods or services.
I'd say that in the first two cases I would consider running a separate copy of the network, because it apparently involves one supernode, and I might want to control the supernode (or maybe not).
Our intention is that Freenet will only consume surplus resources, but we plan to build a reputation system that could have a notion of "karma" that is earned by providing resources to the network. This karma could be used to gate access to resources, for example like a VIP chat room on River.
So there are a lot of possibilities but for now users are motivated by a desire to see the network succeed, and that seems to be a sufficient motivator at our current scale.
Seems reasonable to build a cryptocurrency around this. The network could pay the cryptocurrency out to users dedicating resources. Have you thought about that?
You'd still need to solve the double-spend problem, because while contract state on Freenet will usually synchronize within a few seconds, it isn't guaranteed to converge to a single globally consistent state.
Freenet's approach works well for things like group chat, where temporary inconsistency is mostly just an irritation, but for a cryptocurrency it is fatal.
I'm not saying you couldn't build a cryptocurrency on Freenet, but you'd still need a solution to that problem.
Neat. I've been wanting to see WASM-defined network behavior like this for a while (yay arbitrary consistency algorithms!), I'll have to explore it in more detail :)
(the main thing I've been wanting to try: rather than graphql, send a WASM blob along with your request to a server, and just run it to filter fields in the response / pipeline requests / define "fail if any err / pair errors with requests" for concurrent requests. arguably you could even have it control callee-internal retries.)
Can you tell me about the old Freenet? I've read up on Wikipedia and it seems to be very much in the line with the 90s/2000s p2p file sharing software. Except that you can store stuff on other people's computers and it's encrypted?
Which then led to people storing Bad Stuff, and this is somehow addressed in the new version? (I also read some stuff about friends and trust in the previous one, but haven't looked into properly.)
I think understanding the old one and the issues it ran into would be helpful for understanding the context, and the motivations for the changes.
Or to put it very bluntly: what is this, why should I care, and why not just use the old one?
> We've developed a unique (AFAIK) solution to the consistency problem, every contract must define a "merge" operation for the contract's associated state. This operation must be commutative, meaning that you can merge multiple states in any order and you'll get the same end result.
Where can I learn more about this? How is this different from CRDTs/CmRDTs?
It's very closely related, you can view Freenet contract state as a CmRDT, where the details of the merge operation are specified in the webassembly contract.
It looks a lot like a CvRDT (i.e. a state-based CRDT).
They describe it as a commutative monoid, which means it has associativity and commutativity. CvRDTs also need idempotence, so they can handle duplicate data. Either they are idempotent too (which would make it semilattice-like), or the network protocol handles the deduplication outside of the data itself.
Letting the payload/application define the merge operation is clever. I assume it would mean contracts could opt in to idempotency if it doesn't already exist.
The other bit Freenet has added is doing all this with DHT routing and subscriptions, rather than a more basic peer mesh. This is very different to a blockchain and means it probably isn't suited for anything transactional.
tombstones are sorta the default answer here (i.e. at simplest, you keep all data forever so you can merge correctly, but you hide anything where you've seen a tombstone after it).
but "makes sense" and ways to optimize that can change massively with context. e.g. for a chat app, as soon as you see "deleted message X", you can reasonably drop all past and future changes to X because they won't be shown. if you do that with "deleted chars 87..93" in a text editor, past-edits that you receive in the future might affect the behavior (it might add chars before those, changing what that range means), so you can't simply forget those chars (e.g. an easy option is to replay all events that occur after an event syncs, but that means retaining all events forever). the semantics you choose and what you do with the data affect your outcomes a lot.
tbh this is one of the reasons I like the idea of a WASM-defined algorithm. no one algorithm will be "best" for all data, and the storage/computation/transmission savings can be extreme.
It depends on the nature of the data, for example in our group chat app River[1] it stores the most recent messages - deleting the oldest to make room for the newest as necessary. Banning a user will remove them from the members list along with their messages, and recently banned users are stored in a banned list (like a tombstone).
So there is no one approach to this, rather you design the approach based on the application, and since contracts are just webassembly they are extremely flexible.
For a basic CRDT set, merge rules have to have some kind of temporality basis in the messages such that commutativity is preserved. usually it's a timestamp, sometimes it's an unforgeable value like a hash, e.g.
A: { "prev_hash": null, "content": "foobar" }
B: { "prev_hash": "<hash of A>", "content": "foobarbaz" }
C: { "prev_hash": "<hash of B>", "content": "foobaz" }
and when played out of order, it's guaranteed to resolve to foobaz eventually or immediately, depending on when messages are received
when you encounter the scenario of a fork, there's usually a fork resolution rule, e.g.
D: { "prev_hash": "<hash of B>", "content": "foobazbar" }
to resolve C vs D, sort lexicographically, choose direction of sort order and pick first
When you have non-continuous data due to messages dropping, e.g. you have B and perhaps an E that builds on C, you can either use the same lexicographic rule, or make the hash basis a combination of timestamp and hash, so you get temporality and lineage.
As for deletes, you have either the single set approach of simply making the message content empty and that _is_ the delete, or you have the 2-phase sets, where there exists an add set and a delete set.
Quite a few ways to approach it, but commutativity can be readily preserved.
This is awesome. I rotated some ideas like this in my head a while ago but never had the motivation to put it together. Happy to see more types of protocols like this.
Very interesting. I have been working on something quite some time, where something like this would play a very crucial role, but i never got around to really thinking about how to implement everything. And as I have still a lot of work to do on my project, that would utilize something like freenet, i am very eager to dive into your work. Just wanted to write this as some form as appreciation for your work.
I wonder though, what is your idea of a future, where freenet plays an important role in most peoples lives?
Great work it seems, so far. I will yet have to really look through it all. Congratulations on this.
> I wonder though, what is your idea of a future, where freenet plays an important role in most peoples lives?
While I realize this is wildly ambitious, my goal is that Freenet could ultimately replace the world wide web and the client-server architecture more generally which I view as inherently concentrating power in the hands of a few (which it has done).
That project had nothing to do with the freenet that ran after 2006 either. It's not the first ground up rewrite with major breaking changes using the same name.
Big fan of this project. Three years ago, I interviewed Ian Clarke about his upcoming Freenet rewrite. He's the original "OG" of decentralized content networks. We go into detail regarding its architecture on the podcast:
In my early days of technology tinkering when I was young I was always interested in security, and one day I stumbled upon freenet, and my world changed.
It was amazing and led me to get far more acquainted with the cyberpunk scene. It was this alternative separate internet from what the rest of the world saw with all of the good and bad that brought.
I've been meaning to set it up again and get back into it. I will say for everyone pining for the Internet of yesteryear freenet is it. Go and explore it it is everything the 90's Internet was like, super slow, crazy unhinged nerds all over the place random collections of links, crazy.
Thanks for all you've done Ian
Edit:
Btw what is the best way to support the project and get involved?
If you're in a position to support the project financially you're more than welcome to donate[1], we're a 501c3 non-profit and all funds go to support development.
If you're a developer and are interested in building on Freenet I suggest starting with https://freenet.org/build/manual/tutorial/, you can also join our Matrix room[2], or install Freenet[3] and chat with us on River[3], our decentralized group chat.
in favor of a rewrite from different developers, without asking anyone on the original team.
It was an ivory tower decision which was announced on the mailing list without prior discussion.
The old team did not agree, yet it was forced through by a decision of the "board".
The "board" was a group of people which had not been active on the project for over a decade.
https://www.mail-archive.com/devl@freenetproject.org/msg5526...
The funding of the existing, original "Freenet" was repurposed for the new one of course.
The new "Freenet" does not have anonymity as a design goal anymore,
while the old one continues to exist and is maintained under its new name "Hyphanet" at:
https://www.hyphanet.org/
This is a false narrative, from the Freenet FAQ[1]:
Why was Freenet rearchitected and rebranded?
In 2019, Ian began developing a successor to the original Freenet, internally named “Locutus.” This redesign was a ground-up reimagining, incorporating lessons learned from the original Freenet and addressing modern challenges. The original Freenet, although groundbreaking, was built for an earlier era.
This isn’t the first time Freenet has undergone significant changes. Around 2005, we transitioned from version 0.5 to 0.7, which was a complete rewrite introducing “friend-to-friend” networking.
In March 2023, the original Freenet (developed from 2005 onwards) was spun off into an independent project called “Hyphanet” under its existing maintainers. Concurrently, “Locutus” was rebranded as “Freenet,” also known as “Freenet 2023,” to signal this new direction and focus. The rearchitected Freenet is faster, more flexible, and better equipped to offer a robust, decentralized alternative to the increasingly centralized web.
To ease the transition the old freenetproject.org domain was redirected to hyphanet’s website, while the recently acquired freenet.org domain was used for the new architecture.
It is important to note that the maintainers of the original Freenet did not agree with the decision to rearchitect and rebrand. However, as the architect of the Freenet Project, and after over a year of debate, Ian felt this was the necessary path forward to ensure the project’s continued relevance and success in a world far different than when he designed the previous architecture.
> The new "Freenet" does not have anonymity as a design goal anymore,
Because the new Freenet will have a menu of anonymity options rather than committing to a one-size-fits-all approach, while also addressing the issue of illegal content[2].
[1] https://freenet.org/about/faq/#why-was-freenet-rearchitected...
[2] https://freenet.org/about/faq/#how-does-freenet-handle-harmf...
There was no "year of debate".
You came to the mailing list and announced it for the first time as a finalized decision already,
without any prior debate with the original team.
The "board" you cited as the body which allegedly discussed it did neither join the mailing list discussion,
nor were you willing to hand out their contact info.
It's all public for anyone to see on the mailing list archive:
https://www.mail-archive.com/devl@freenetproject.org/msg5526...
https://www.mail-archive.com/devl@freenetproject.org/
Incorrect, I raised the issue with the lead maintainer over a year prior to that announcement.
> You came to the mailing list and declared it as a finalized decision.
As the project's architect I'm entitled to make decisions about the project's future direction.
> It's all public for anyone to see on the mailing list archive:
> https://www.mail-archive.com/devl@freenetproject.org/msg5526...
I stand by every word I said in that mailing list thread.
The previous lead maintainer, Steve, voiced their frustration with your decision here:
https://www.mail-archive.com/devl@freenetproject.org/msg5526...
To which you sent a brash reply, which sounds like you don't know Steve's position in the community:
https://www.mail-archive.com/devl@freenetproject.org/msg5526...
To which the current lead maintainer, Arne, said he agrees with the sentiment of Steve:
https://www.mail-archive.com/devl@freenetproject.org/msg5526...
So if you discussed this with Arne for a year, then why does he agree with the frustration of Steve?
And even if the discussion with Arne happened, it still was a backroom decision:
Two people are not representative on a project with plenty of developers and an active community.
> As the project's architect I'm entitled to make decisions about the project's future direction.
A sense of entitlement is not a leadership quality.
A leadership quality would be to admit a mistake:
That repurposing the name was not only bad for the original project,
but also for the new one (because these discussions will haunt it forever),
and to then rename the new project to a fresh name which no other software used before.
Whether or not it was the right decision will be determined by the outcomes, which so far are promising, because we have a working network that does things that the old architecture could never do.
https://www.mail-archive.com/devl@freenetproject.org/msg5526...
I'm sorry, but nothing following that even comes close to proving that it's a false narrative. Quite the opposite actually.
And even without agreeing on whether people should be anonymous on the Internet,
it could be agreed that replacing a software which guards against a certain threat model (repressions) with one which does not,
without changing the name, is not exactly a wise decision.
We will also have a decentralized reputation system that will protect people from being exposed to unsavory or illegal content, a common criticism of the old Freenet architecture.
This isn't even the first time we did a ground-up redesign/rewrite of the Freenet codebase, we did this in 2008 with the 0.7 release.
I integrated "Fair Tunes", which tried to pay musicians for mp3 files, long before any label was selling mp3's.
[1] https://sourceforge.net/projects/snarfzilla/
I also would like to see an emphasis on local-first approaches.
This experiment, in the spirit of UNIX, composes git and text files to form a social network:
https://github.com/dharmatech/9social
Video demo: https://youtu.be/q6qVnlCjcAI
OK, time for inception... 9social on freenet-git
¯ \ _ ( ツ ) _ / ¯
- The synced value is a history of client updates, sorted in some eventually consistent order (e.g. by hybrid logical clocks). Merging takes the union of the update sets.
- The user-visible value is the result of processing these updates in order, using arbitrary contract code.
This is overkill for simple last-writer-wins values, but it lets you support fairly general data types & arbitrary update functions, including ones that preserve application-specific invariants.
The Automerge CRDT library works like this already [1][2], but it only allows specific updates to JSON data. Sharing code via your contracts solves the hard part of generalizing that to arbitrary data & updates.
[1] https://automerge.org/
[2] https://arxiv.org/abs/1805.04263
Yes, in fact you can implement this within the current framework, for example with our group chat River, each room state maintains a list of the N most recent messages sorted by (approximate) timestamp.
The idea is that you can adapt the merge logic to the needs of the specific application, and I think a time ordered event log will be a common pattern.
A cryptocurrency-based solution like you suggest will undoubtedly be one of a menu of reputation bootstrapping options that will develop over time.
For example, if Freenet were to reach scale, it could eventually need some kind of economic primitive around it. Something similar to how Filecoin handles decentralized storage, but for app state. One way to do this could be paying peers to keep app state available, serve it reliably, etc. and prove they are doing so.
- You participate in a secret cabal, and don't want participants' identities be visible to each other.
- You're a journalist, and want to give your informants in sensitive matters, or from oppressed countries, a way to securely interact with you, so that you won't be technically capable of reporting their identities.
- You're selling illegal goods or services.
I'd say that in the first two cases I would consider running a separate copy of the network, because it apparently involves one supernode, and I might want to control the supernode (or maybe not).
So there are a lot of possibilities but for now users are motivated by a desire to see the network succeed, and that seems to be a sufficient motivator at our current scale.
Freenet's approach works well for things like group chat, where temporary inconsistency is mostly just an irritation, but for a cryptocurrency it is fatal.
I'm not saying you couldn't build a cryptocurrency on Freenet, but you'd still need a solution to that problem.
(the main thing I've been wanting to try: rather than graphql, send a WASM blob along with your request to a server, and just run it to filter fields in the response / pipeline requests / define "fail if any err / pair errors with requests" for concurrent requests. arguably you could even have it control callee-internal retries.)
Which then led to people storing Bad Stuff, and this is somehow addressed in the new version? (I also read some stuff about friends and trust in the previous one, but haven't looked into properly.)
I think understanding the old one and the issues it ran into would be helpful for understanding the context, and the motivations for the changes.
Or to put it very bluntly: what is this, why should I care, and why not just use the old one?
Thanks
> We've developed a unique (AFAIK) solution to the consistency problem, every contract must define a "merge" operation for the contract's associated state. This operation must be commutative, meaning that you can merge multiple states in any order and you'll get the same end result.
Where can I learn more about this? How is this different from CRDTs/CmRDTs?
Thank you!
> Where can I learn more about this?
If you don't mind watching a video I gave this talk back in March that should be fairly comprehensive: https://youtu.be/3SxNBz1VTE0?si=R4ifrsfEUJfvjDPx
If you would prefer an article I recommend: https://freenet.org/about/news/summary-delta-sync/
> How is this different from CRDTs/CmRDTs?
It's very closely related, you can view Freenet contract state as a CmRDT, where the details of the merge operation are specified in the webassembly contract.
They describe it as a commutative monoid, which means it has associativity and commutativity. CvRDTs also need idempotence, so they can handle duplicate data. Either they are idempotent too (which would make it semilattice-like), or the network protocol handles the deduplication outside of the data itself.
Letting the payload/application define the merge operation is clever. I assume it would mean contracts could opt in to idempotency if it doesn't already exist.
The other bit Freenet has added is doing all this with DHT routing and subscriptions, rather than a more basic peer mesh. This is very different to a blockchain and means it probably isn't suited for anything transactional.
but "makes sense" and ways to optimize that can change massively with context. e.g. for a chat app, as soon as you see "deleted message X", you can reasonably drop all past and future changes to X because they won't be shown. if you do that with "deleted chars 87..93" in a text editor, past-edits that you receive in the future might affect the behavior (it might add chars before those, changing what that range means), so you can't simply forget those chars (e.g. an easy option is to replay all events that occur after an event syncs, but that means retaining all events forever). the semantics you choose and what you do with the data affect your outcomes a lot.
tbh this is one of the reasons I like the idea of a WASM-defined algorithm. no one algorithm will be "best" for all data, and the storage/computation/transmission savings can be extreme.
So there is no one approach to this, rather you design the approach based on the application, and since contracts are just webassembly they are extremely flexible.
[1] https://github.com/freenet/river
and when played out of order, it's guaranteed to resolve to foobaz eventually or immediately, depending on when messages are received
when you encounter the scenario of a fork, there's usually a fork resolution rule, e.g. D: { "prev_hash": "<hash of B>", "content": "foobazbar" }
to resolve C vs D, sort lexicographically, choose direction of sort order and pick first
When you have non-continuous data due to messages dropping, e.g. you have B and perhaps an E that builds on C, you can either use the same lexicographic rule, or make the hash basis a combination of timestamp and hash, so you get temporality and lineage.
As for deletes, you have either the single set approach of simply making the message content empty and that _is_ the delete, or you have the 2-phase sets, where there exists an add set and a delete set.
Quite a few ways to approach it, but commutativity can be readily preserved.
I wonder though, what is your idea of a future, where freenet plays an important role in most peoples lives?
Great work it seems, so far. I will yet have to really look through it all. Congratulations on this.
> I wonder though, what is your idea of a future, where freenet plays an important role in most peoples lives?
While I realize this is wildly ambitious, my goal is that Freenet could ultimately replace the world wide web and the client-server architecture more generally which I view as inherently concentrating power in the hands of a few (which it has done).
Think of it as going back to the Internet's decentralized roots.
https://www.youtube.com/watch?v=JWrRqUkJpMQ
It was amazing and led me to get far more acquainted with the cyberpunk scene. It was this alternative separate internet from what the rest of the world saw with all of the good and bad that brought.
I've been meaning to set it up again and get back into it. I will say for everyone pining for the Internet of yesteryear freenet is it. Go and explore it it is everything the 90's Internet was like, super slow, crazy unhinged nerds all over the place random collections of links, crazy.
Thanks for all you've done Ian
Edit: Btw what is the best way to support the project and get involved?
If you're in a position to support the project financially you're more than welcome to donate[1], we're a 501c3 non-profit and all funds go to support development.
If you're a developer and are interested in building on Freenet I suggest starting with https://freenet.org/build/manual/tutorial/, you can also join our Matrix room[2], or install Freenet[3] and chat with us on River[3], our decentralized group chat.
[1] https://freenet.org/donate/
[2] https://matrix.to/#/#freenet-locutus:matrix.org
[3] https://freenet.org/quickstart/
https://news.ycombinator.com/item?id=40469711 - Ian Clarke explains the next generation of Freenet [video] (2023)