Very welcome and useful initiative. I deeply hate the fact that it's a next.js app (these apps simply should not be an option for the web unless HTML has somehow disappeared without a trace and humanity can't remember that it has ever existed) and not an HTML site with some JS where needed / useful. I suppose I understand the appeal and the convenience from a development point of view.
Open-sourcing the code is commendable and I'm happy to see this.
> Do you know of any additional countries public donation datasets that are actually machine-readable (CSV/JSON/API)?
It was also a side project to start tinkering with next. I found it had a nice DX and allowed me to quickly build it.
Now I think it might also be possible to just use something like Astro as I don't need many dynamic SSR features.
> Now I think it might also be possible to just use something like Astro as I don't need many dynamic SSR features.
You're doing a very useful service to the community, which is absolutely praise-worthy; please don't listen too much to my moaning about next.js. As long as it does the job, you're gold.
I built this tool over the years primarily for myself.
I had some questions about political finance flows that were impossible to answer just by staring at the raw tables published by governments.
I needed a way to visualize the insights for my own curiosity. Since the result was useful, I polished it up and published it so everyone else can explore the data too.
The Architecture (No Database):
To keep it fast and low-maintenance, there is no running database.
- backend: The scrapers are run locally and pre-generate the entire dataset into static JSON files
- frontend: The client loads these JSON chunks on demand. Many pages are staticly pre-rendered on build.
- stack: It’s a next.js app, hosted via OpenNext on Cloudflare Workers.
Moving Next.js to the edge (using OpenNext) combined with a purely static data model means the site is cheap to host and scales instantly, even when a big new dataset drops.
The "Open Data" reality:
While building the scrapers to answer my questions, I encountered some data quirks. A few favorites:
- The Greek Belgian: One 2021 EU foundation document randomly used Greek unicode characters (ΒΕ) instead of ASCII (BE) for the Belgium country code.
- The Encoding: All Austrian CSVs were standard UTF-8, except the 2022 file which was randomly encoded in Mac OS Roman (Apple Macintosh).
- The Vanishing Files: The Croatian Electoral Commission’s database has 404 errors for specific 2019/2020 party donation lists that used to be there. They made them available again after being informed about this issue.
I report these quirks upstream so the next developer doesn't have to suffer. I keep a log of the weirdest ones here: https://donation.watch/en/fun.
I’ve finally cleaned up the code a bit and open-sourced the whole thing (AGPL-3.0 for code, CC BY 4.0 for the data).
I'm looking to add more countries. Do you know of any additional countries public donation datasets that are actually machine-readable (CSV/JSON/API)?
I've already investigated a few that are currently too messy (e.g., scanned PDFs or bad formats), listed here: https://donation.watch/en/other-countries.
I'd like to integrate the US dataset in the future. The issue was that it's really big and I still have to figure out a good way to filter uninteresting donations out.
Open-sourcing the code is commendable and I'm happy to see this.
> Do you know of any additional countries public donation datasets that are actually machine-readable (CSV/JSON/API)?
I recently came across https://banipartide.ro/ but I'm not entirely sure whether they made all their data machine-readable. I found this: https://db.banipartide.ro/banipartide/Venituri+anuale+din+ra...
I'll write them to ask if they're ok with me using their data.
I've previously found https://finantarepartide.ro/partidul-pro-romania-pro-romania... which should be the official romanian source. When attempting to integrate them I found some missing links tho and they never replied to my mail :D
Regarding next:
It was also a side project to start tinkering with next. I found it had a nice DX and allowed me to quickly build it. Now I think it might also be possible to just use something like Astro as I don't need many dynamic SSR features.
You're doing a very useful service to the community, which is absolutely praise-worthy; please don't listen too much to my moaning about next.js. As long as it does the job, you're gold.
I’m the creator of donation.watch.
I built this tool over the years primarily for myself. I had some questions about political finance flows that were impossible to answer just by staring at the raw tables published by governments.
I needed a way to visualize the insights for my own curiosity. Since the result was useful, I polished it up and published it so everyone else can explore the data too.
The Architecture (No Database): To keep it fast and low-maintenance, there is no running database.
- backend: The scrapers are run locally and pre-generate the entire dataset into static JSON files
- frontend: The client loads these JSON chunks on demand. Many pages are staticly pre-rendered on build.
- stack: It’s a next.js app, hosted via OpenNext on Cloudflare Workers.
Moving Next.js to the edge (using OpenNext) combined with a purely static data model means the site is cheap to host and scales instantly, even when a big new dataset drops.
The "Open Data" reality:
While building the scrapers to answer my questions, I encountered some data quirks. A few favorites:
- The Greek Belgian: One 2021 EU foundation document randomly used Greek unicode characters (ΒΕ) instead of ASCII (BE) for the Belgium country code.
- The Encoding: All Austrian CSVs were standard UTF-8, except the 2022 file which was randomly encoded in Mac OS Roman (Apple Macintosh).
- The Vanishing Files: The Croatian Electoral Commission’s database has 404 errors for specific 2019/2020 party donation lists that used to be there. They made them available again after being informed about this issue.
I report these quirks upstream so the next developer doesn't have to suffer. I keep a log of the weirdest ones here: https://donation.watch/en/fun.
I’ve finally cleaned up the code a bit and open-sourced the whole thing (AGPL-3.0 for code, CC BY 4.0 for the data).
Repo: https://github.com/donationwatch/donationwatch
Site: https://donation.watch
I'm looking to add more countries. Do you know of any additional countries public donation datasets that are actually machine-readable (CSV/JSON/API)?
I've already investigated a few that are currently too messy (e.g., scanned PDFs or bad formats), listed here: https://donation.watch/en/other-countries. I'd like to integrate the US dataset in the future. The issue was that it's really big and I still have to figure out a good way to filter uninteresting donations out.