Building a Personal Internet from Markdown Files

2023-12-28


Briefly

Note-taking apps like Obsidian have helped me link my digital libraries together, but don't go all the way to meeting my archiving needs. I jerry-rigged my own solution, which I'm calling Folderbase.


All the information I've ever committed to electronic storage consists of text files, images, audio and video. Computers have been able to handle these file formats for decades, and yet I find myself needing to use multiple separate applications to manage them all.

Information Silo

Image Libraries

OSX Photos

OSX Photos

Samsung Gallery

Samsung Gallery

Pinterest

Pinterest

Eagle.design

Eagle

Notes / Journaling

Obsidian

Obsidian

Sublime Text

Sublime Text

MacDown

MacDown

Apple Notes

Apple Notes

General File Management

OSX Finder

MacOS Finder

Windows Explorer

Windows Explorer

Simple File Explorer

Simple File Explorer

Most apps focus on managing only one of these filetypes well, so my information ends up getting siloed inside a particular app.

It doesn't have to be like this! We already have a universal software standard for viewing documents: the web browser. By using a lightweight markdown-based CMS on the server and making some client-side tweaks on the browser, I was certain I could string together a better file management solution.

And so I came up with Folderbase: a universal 'view layer' for my folders of text notes and multimedia assets that:

  • can be "installed" as a web app to run in full-screen
  • doesn't require a build step to compile
  • works offline
  • runs on my laptop, phone and e-reader on the same codebase

How It Works

On the back end is a PHP library that loops through the folders, gets metadata for the .md Markdown files, and compiles them to HTML. The front-end input is handled by any Markdown-file compatible text editor, which is pointed at a folder whose content structure looks like this:

```
|-- folderbase
|  |-- content
|  |    |-- note1.md
|  |    |-- note2.md
|  |-- assets
|  |    |-- img1.jpg
|  |    |-- audio1.wav
```
```
note1.md
========
---
title: Note 1
template: page
---
Content for Note 1 (text, img, audio, video)
```

The output is an installable PWA: a read-only knowledgebase that's fully portable*, self-hostable and works completely offline. This is a tool I made due to my mounting frustrations around wrangling my personal information in a way that made it easy to reference and build upon.


How I'm using it

Organizing different types of media is hard. Trying to corral them all into 1 off-the-shelf app that can handle multiple use cases was impossible. I had to scale down the scope of my ambitions into something more feasible:

Image Moodboards

I use Pinterest because the user experience of curating images is good, but Pinterest's insistence on making the app a 'social', online-only experience doesn't suit my use case. Also, as an entirely ad-funded app, Pinterest doesn't make money unless I view my collections inside their ad delivery system convenient mobile app.

Offline Reading

I wanted to be able to get rid of read-it-later apps like Instapaper, Pocket and Readwise. I have a browser extension that can copy a web page to a local Markdown file, so it's just a matter of setting up the CSS to render it nicely on the web.

Frequently Referenced Notes

We've all got them: grocery lists, cooking preparation checklists, exercise routines. Opening up Obsidian or another PKM app to view these requires a full app load and time to locate the note itself. Folderbase on the other hand launches almost instantaneously, and no page is more than 3 taps away.

Internet Whitelist

I used to open my browser only when I was looking for something. In the social media + smartphone era, I found myself going straight to Reddit or Twitter to see an immediate feed of content.

An internet whitelist changes that. Instead of mindless scrolling, I can pick where I want to go based on some rough categories: text-only news sites, small-internet personal websites and music websites. Yes, I could just go back to browser bookmarks, but the UX of organzing a bunch of haphazardly bookmarked sites that I remembered to hit ⌥ + D on makes that a hassle compared to simply editing a text file.


The Inspiration

I watched a conference talk about Hypercard (1987-2004), a precursor to the web browser and PowerPoint. Hypercard was a user-friendly software that came pre-installed with the Mac and let you create cards with data in them (text, image, audio), and link them together like a browser does with hyperlinks.

Hypercard Home screen

Hypercard home screen (source: Low End Mac)

It had a scripting language underneath (similar to how Excel had VBA underneath to write custom functions), so users could graduate from making manually hyperlinked mini-databases, into something with a front-end that could input data and display it. One of the comments on the video jumped out at me:

Youtube comment: I would kill to have something like this on my iPhone – instead of paying for shabby flashcard- and to-do apps. How a pocket computer that you cannot program yourself is considered a "smart" phone is beyond me.

It is crazy that what people call "a computer in your pocket" can't actually write any programs the way a real computer can.


Choosing a Featureset

I didn't want to spend any more time trying to find the 'the perfect note-taking app'. There were too many out there already, and they all had some kind of gotcha that caused them to be incompatible with the core requirements I had:

Offline-first

Folderbase is a repository of personal notes and documents that are unlikely to be shared with others, so you shouldn't need an Internet connection to access it. To launch it offline, you can move your folders to the root of a local server app running PHP 7.3+ (e.g. the htdocs folder for KSWeb on Android) and and navigate to the server address, (e.g. localhost:8080/folderbase/). If you want to put it online, you can follow the exact same steps; the only difference is that you'd be plopping the folder into a remote webserver root via SFTP or Git.

Separate app for text input

I use two apps for note-taking: Obsidian and Markor. Every now and then, I'll edit MD files in Sublime Text. All 3 apps work well for me, so I point them all to the main folder in which my text notes sit. The Folderbase web app is exclusively a 'view layer' for data sitting in my folders, with CRUD operations handled by a Markdown notes app (add/remove/edit text files) and a file manager (add/remove/edit image/media files).

File and Folder-based

Text files in one folder, images and other non-text assets in another. No database allowed. This structure offers maximum portability and eliminates vendor lock-in, as it doesn't require any kind of app scaffolding to navigate. If all you have are the raw files on a USB drive, your content will still be intact.

Even Stephen Ango, the CEO of Obsidian (defacto leader of offline note apps) advocates for a system that's based on files, and free of apps:

> File over app is a philosophy: if you want to create digital artifacts that last, they must be files you can control, in formats that are easy to retrieve and read. Use tools that give you this freedom. > @kepano

Fast Navigation

My files are organized in rough accordance with the Johnny Decimal system: all data stored in a maximum of 10 folders, each of which can have 10 subfolders, for a total of 100 folders. From a navigation standpoint, this means I can reach any file in the system with a maximum of 3 clicks.

This was inspired by reading about the design process of the Palm Pilot PDAs of the '90s, the earliest offline-first digital assistant. I prefer to navigate this way instead of using a search bar, which would the right UX option if I had thousands of files to look through. Having a spatial awareness of where my data sits gives me a feeling of comfort.

Better browse-ability

As much as I like Obsidian, its UI is designed for writing information and not really for referencing that information later. From a document browsing standpoint it resembles Sublime Text or Notepad++; a directory column on the left, and a wide editor/reading view on the right. I wanted something that resembled a 'desktop', or 'homepage', as offered in other (non-Markdown) tools like Notion.

Obsidian UI Filetree

Obsidian Workspace UI

Notion UI

Notion Workspace homepage (Source)

Free and Open-Source Software (FOSS)

Folderbase is a personal, non-commercial tool that uses open-source libraries, so making the source code available was a no-brainer. The bigger reason for making it open-source is to allow anyone to extend it as they see fit. In general, FOSS makes it such that if the software stops being maintained (or stops meeting my needs), I can downgrade to an older version that still works, or better yet, I can implement fixes directly to the source code. This is what happened for me with Hotglue, a 10-year old WYSIWIG web editor whose runtime code needed some deprecated functions to be commented out in order to run on post-2015 versions of PHP.

Web over Native App

I wanted to be able to write the code once and have it work on any operating system with a browser - my personal machine and phone (MacOS + Android) as well as my work laptop and phone (Windows 10, iPhone). If Linux is in the mix one day, it'll work there too.

Using web tools (HTML, CSS, JS, PHP) and using an app-ish browser window as the view layer is the only sane way to do this. Writing OS applications with native tools would require me to learn how to do that, 3 times over (C+ for Windows, Swift for Mac/iOS, Kotlin for Android). Even then, on platforms like iOS, a native app's source code would not be able to be distributed as open source


Approach

There are static site generators that can convert Markdown files into a browsable, hyperlinked website.

However, most SSGs are Javascript-based and deploying them required knowledge of NodeJS and a build system, as well as specific syntax for SSG options like Ghost, Hugo or Jekyll. That was a no-go for me as it would be a net-new thing to learn on top of all the work that would be required to actually build the app. Dropping a folder into the root of a webserver and firing up localhost:8888 on the browser was as complicated as I wanted the deployment to be.

Obsidian has a Dataviews query language that handle some of my complaints. And I could have tried to learn it, but the more I looked into it, I realized I would be learning a made-up query syntax, the knowledge for which would be useless outside of Obsidian. Why do that when I could just design a layout with the universal language of HTML and CSS?

I settled on using a little-known PHP-based markdown CMS called Pico and got to work.


UI/UX

The goal here was to design a system that provided a low-friction way of getting my many notes into a permanent, structured digital format:

Custom Layouts

If I wanted to create a Pinterest-style moodboard by dragging some images into an Obsidian file, it will immediately resemble a stodgy Word document. But here's how it looks in Folderbase, with nothing other than a <div class="moodboard"></div> block enclosing the images.

Creating a Moodboard with drag-and-drop

Install as Web App

I can't stand looking at browser chrome when using an app; it takes away vertical screen real estate and fills it with the distracting icons for all the other tabs I've left open! Mercifully, Chrome, Safari and Edge browsers all offer the ability to "install" a web app to your desktop. Once 'installed', the application launches in full screen, without an address bar or browser chrome, making it look and feel more like a native app (especially with the HTMX transitions). This requires a manifest.json file with links to an app icon image, as well as mobile and desktop preview images.

If your browser is Chrome, installing as a web app means that when you launch the app from your app drawer, it first loads a Chrome window, then a separate headless window containing the app. That's a bit awkward UX-wise; Safari and Microsoft Edge don't do this fortunately.

install as web app dialog message

"Install Web App" dialog on Chrome

Low-friction input with Obsidian

Obsidian's excellent out-of-the-box editing functionality, coupled with some CMS-friendly formatting options makes it a great 'edit layer' for Folderbase. For example, with a couple of setting changes, you can drag images into Obsidian and have them be saved in the 'assets' folder directly, with Markdown-friendly image links that display nicely in a web browser. This is a lot better UX-wise than saving to a generic 'download' folder, moving the file to 'assets' and then painstakingly adding a Markdown link to the file

Page Layout

The writing experience in most Markdown apps is fine; the reading experience however is not. Obsidian for example shows your document in a boring top-down view with narrow margins, pretty much exactly in the layout you used while wroting it. If you use a paid service like Obsidian Publish to put it on the web, it will use the same display theme as in the app -- boring! There's so much available screen real estate, but the Markdown format confines you to single-column layouts.

Not on Folderbase though. With a simple code block I can change the page layout into multi-column grid that supports 4 columns on desktop, and folds down to 2 on mobile:


<div class="grid-4">

<div markdown="1"> // Column 1 start

Column Content (text/image/audio/video)

</div> // Column 1 end

</div>

4-column grid design

4-column grid layout on mobile

The same applies for CSS columns (separate from CSS Grid). By enclosing paragraphs within a code block like `

Page Transitions with HTMX

When clicking through to a new page, there was an unsightly white flash in the browser as the server switched from displaying Page A to Page B. A proper web application would have a system of views and URL routing that would smoothly load in the content of a new page without refreshing the entire browser viewport.

For those building framework-less non SPA sites, there are ways around this. You could use client-side solutions like smoothstate.js, BarbaJS, or the SWUP library. Of these, I'd only ever gotten SWUP to work, and even then it didn't have the smooth effect I expected.

HTMX to the rescue. I'd taken a Javascript class some years ago and had learned about AJAX/XHR requests, a way to load content into a web page without a full page refresh. Unfortunately, the amount of code required for a single XHR request was a hassle to type out. HTMX wraps all that up into a single property hx-get, which can pull in the URI of the linked page via a Pico object page.url. I added in an optional attribute: hx-replace-url. This ensured that page changes would update the browser history, and that the breadcrumb navigation I had set up would reflect the correct values for "parent page" and "child page" when internal links were clicked.

I am not a developer, so building this out required a lot of trial and error, in addition to hunting through closed or abandoned tickets in the Pico CMS Github.


Deployment

A PHP-based application needs a server to run, so Folderbase doesn't work unless a local web server is active on your computer. You can open the app from the app drawer like normal, but if there's no local server app running, you'll see a blank screen. Some types of 'web-to-application' wrappers like Electron can be configured to launch a local server at runtime. But Electron is for Javascript-based apps, and mine is PHP-based.

There are some wrapper packages that are PHP-compatible, but to limit the scope of the project, I kept it as is. As this is personal software, this isn't a big deal for me. I run MacOS and Android, and both operating systems offer web server apps for local development (MAMP and KSWeb (or Termux), respectively). Folderbase works on both. In future I may try to use MacOS Shortcuts to create a script and shortcut that handles the sequence of events (1. launch server app 2. launch Folderbase) with a single click.


Final Thoughts

It feels good to make my own tools. It's an ability that feels like it'll become only more relevant as the software I depend on may not always work the same way. For example, Simple Mobile Tools, a company that makes truly great, uncomplicated open-source Android apps (Gallery, Voice Recorder, Calculator etc), was recently acquired by an adware company. So its future app updates are likely to have anti-user features like greater telemetry, advertisements and require elevated device permissions (geolocation, microphone etc).

The biggest takeaway from building Folderbase is that developing an app that references locally stored data is hard. An off-the-shelf flat-file CMS provided the basic framework, but I still had to tweak a lot of Twig loops and filters to get the information architecture right. On top of that, I had to learn the CSS Grid quite well to get the layout I wanted.

In the 1980s, the home computer industry was so new that most computers came with a programming environment that was easily accessible to the end-user. There was so little software out there, users were expected to build their own little programs.

Today, we have the opposite problem; there's too much third-party software, each with its own 'lock-in-the-user' philosophy. The hundred or so notes in my Apple Notes app can be synced via Apple iCloud, but cannot be exported to a plaintext format*. Meanwhile the ability to write and execute code on your own device seems to get hidden behind more and more byzantine menus, permissions and scary security dialogs with each passing OS upgrade.

Perhaps programming can't be as simple as writing a cell reference formula in Microsoft Excel. But should it really be harder than firing up a Hypercard-style app, or MS Access/Filemaker?


References and Further Reading: