Arcology is my site generator Project currently written in FastAPI and backed by my org-mode wiki set up on top of org-roam. The Goal for the project is to be able to publish subsections of my notes and code and knowledge across multiple websites, and to be able to publish a feed of new and updated pages within those subsections over open protocols like RSS and ActivityPub.
What's in a Name?
Arcology is an architectural and cultural school of design, a portmanteau of Ecology and Architecture, a philosophy of design principles for densely populated, ecologically low-impact human habitats. In Sci-Fi they're closed-system environments where human needs are met without harming the wider environment, through technical, social, and automated means. A solar plant and a vertical farm feeding thousands of residents, with countless acres of land left to wild outside the Arcology. In the real world there's a prototype desert commune in northern Arizona.
My Arcology is a technical and cultural school of design, of building systems which mesh cleanly with those outside it and the user(s) within it. It's a self-sustaining system choosing to remain unreliant on proprietary and liberty-hostile systems. This is a Privacy-focused self-hosted document publishing platform, a Hypermedia application exposing the inner working of my knowledge archive, link archive, and inner thinking. It's part of A Modern User Agent built for myself for now, and for the family and community I will have in a decade.
Vision and Goals
I am working towards a Minimum Viable Product for this tool as a system to publish any of my org-documents to the web for sharing among friends and family and my technical peers. It's a dynamic site written in Python, leveraging a metadata of my org-roam Knowledge Base and the Knowledge Base itself to build my websites.
I intend to slowly improve the org-mode caching system through my Arroyo System Cache project; I am finding it to be a compelling system badly in need of polish and presentation. org-roam provide a sqlite3 database, Arroyo gives it a re-normalized schema, and my Arcology server process does nothing but1 read this database and the org-mode documents on the file-system to provide a dynamically updating website and eventually social features like RSS and ActivityPub.
I am trying to fully develop this as an Literate Programming application, and so far the results have been mixed. See Arcology FastAPI Introduction.
Arcology Static Site Generator
Arcology started as an Emacs Lisp library which produced a static site where individual org-mode headings would be exported using a custom templating system to provide a site that was built around Microformat metadata specifications and the informal protocols around the Indieweb community.
This is the software which generate[s/d/ing] https://notes.whatthefuck.computer and it worked well enough, but it's failed to scale with my current system, and especially with EXWM having a long syncronous process in my Emacs process was a nonstarter – I stopped publishing my site because of how costly it was, and that must change.
I was also butting up against the organisational bad habits that I had picked up with regard to how my org-mode documents were organized and how I handled my personal Knowledge Base. I explored and began to realize that what I wanted was less of outline-style heirarchical organization of thought, and I was more interested in non-heirarchical structures like graphs and links.
Arcology Rails was the first attempt at a "dynamic thinking environment", a set of CRUD applications that could help me think, and remember. It was an experiment in building a simple graph structure without namespacing, and building intelligence on top of that.
I had visions of building dynamic applications on top of three components:
- a content addressed store of files
- a graph-database
- a lens-based programming environment on top of them
I am very far from being interested in building this sort of plumbing in the style and systems I am investing in. If those exist some day in a decentralized, equitable, tactile option, I would love to see them.
A brief exploration of TiddlyWiki
TiddlyWiki is a very cool programmable notebook, and I spent some time working with it in January and February 2020. Very promising thing, and it lead me to the value of embedded interlinks over metadata. But I wanted a lot of things that it couldn't give me, and so I went back to the drawing board on a static site generator built on top of org-mode.
Arcology Gen was another attempt at a smarter, more streamlined static site generator, but this time outside the Emacs process. This is where I started to realize that I need a cache database, something queryable, and something persistent. I constructed some very simple org-mode parsers, and that was interesting for publishing a directory, but I needed metadata extraction to power indices and feeds.
In early 2020, org-roam gave me that database, and with some extensions, I could build the website. Around this time, Tor and I started working on a Home Mesh project in Elixir, and co-hosting that with my web-site seems like a reasonable cross-over. I wanted to use Ecto to build the site, and it may some day provide static HTML, but for now it's a dynamic Elixir application, a mashup of Phoenix and Ecto and some custom code to glue it to an Emacs environment
Now in early 2021 I have a working minimum viable prototype of the Arcology at https://dev.arcology.garden and I must evaluate the next steps. Jethro of org-roam is currently evaluating a 2.0 refactor and simplification and the vision of it doesn't quite line up with what I want even if it's for the good of the software and the ecosystem. Alas, I'll have to modify the Arcology Link Routers to handle these opaque ID links, and I think i can, but this major version bump lays bare the cost of my dependencies.
The Literate Programming Elixir Phoenix project is worth evaluating in its own right, still: Is developing entire services as an org-mode meta application viable or for that matter fun? How does the interwoven format aid long-term maintenance, refactoring and development? I do intend to explore this at some point (open threads)
- An honest evaluation of the Arcology MVP
- An honest evaluation of Literate Programming the Arcology MVP
As of [2021-09-25] I haven't written those things, and have struggled to find the energy and will to work on the Elixir version of arcology. Work has pushed me towards picking up FastAPI and now I am using that, hoping that I can get a small custom web app put together as I found that I wasn't using the dynamic high-scale, interesting things which Phoenix and Elixir provide.
The FastAPI version is going to be a lightweight implementation of the ideas I fleshed out in the Elixir implementation. It's still literate-ly programmed. Flask would probably be good enough, but having the database stuff integated with SQLModel is going to be nice.
At this point I have boiled down the "Arcology as dynamic site" down to a set of essential functions and ideas which could lbe implemented in any language, and I intend to explore at least a Rust implementation of it next. However, this system is lithe and fun to work with.
org-mode documents are the canonical datastore
I value plain-text formats and org-mode provides enough structurally significant syntax that I can extract any information that I need from the text, and re-encode that without a huge loss of fidelity.
org-roam provides a SQLite3 metadata cache
Parsing org-mode outside of Emacs is a fools-game, frankly. see also, what alphapapa calls this Greenspun's eleventh rule.
With my org-roam: #+KEYWORD caching in org-roam-db extensions, I can cache arbitrary keywords and then query them in other processes. Everyone makes big warning sounds and scary noises when people start talking about using sqlite3 in production, but this is the perfect usecase for it: I only have read concerns outside of org-roam. SQLite is not a toy database
SQLModel provides an interface to the metadata
FastAPI provides some nice quality of life and a
high-performance HTTP server
NEXT [#A] HEAD support with opengraph and the social embed shit, rss feeds for site and page…
NEXT [#A] Work on Arcology once a week
DONE [#A] sitemap web like Arcology Elixir had
DONE [#A] page metadata for dynamic robots.txt
Disallow / by default, Allow tagged pages
NEXT [#A] 404 page with stats for missing links clicked
NEXT [#B] update Data Architecture above to serve as a "tour of the arcology" type of thing
DONE Base feature set
NEXT test coverage
DONE [#B] feed generator
WAITING [#B] webmention server
NEXT [#B] iNaturalist observations PESOS importer
A page for each plant, a heading for each observation.
similar PESOS My Instagram posts in to blog
dogsheep's iNaturalist to Sqlite is half the work
NEXT [#B] [2020-10-27 Tue 22:31] Arcology should make sure that pages in the Archive are saved in the Internet Archive
NEXT [#B] make Pandoc HTML generation able to optionally bundle media
it'd be nice to resize the images and strip exif data and whatnot, but maybe i just need to do that on "my" side…
WAITING [#C] ActivityPub server
WAITING [#C] calendar generator
WAITING Trailblazer Mode
feed2toot pulls feed list from a JSON API
endpoint dynamically rather than needing a deploy
dynamically add feeds for event pages and have them go to Fediverse automagically
Let's be honest; it's doing quite a bit more. It's caching pages and metadata, it's providing a Sitemap, it's providing high-quality Routing, and it allows me to build features a theoretical Arcology Anti-Viral Technology, I would love it if external links and references could be automatically committed to the Internet Archive↩︎