As a coder, sharpening my tools is part of my nature. Obsidian is the Visual Studio Code of knowledge workers. Both are ushering in a new era of efficiency with the adoption of large language model LLM-powered extensions, revolutionizing how we manage and manipulate information in natural language or code.

In this post, I introduce Obsidian, compare it to Visual Studio Code, and highlight two notable LLM-powered Obsidian plugins namely TextGenerator and Copilot. I follow on with their respective synergies, and the ability to use both of them with online-proprietary and local-open LLMs. I end with the prospect of more multi-modal Obsidian-shaped extendable interfaces to leverage LLMs.

Knowledge Tasks in Obsidian

LLM-powered tooling improved the productivity of my day-to-day research. The task categories include understanding concepts faster, introspection of my writing/thoughts, hardening arguments, brainstorming on options, extracting certain subjects for in-depth analysis, and compressing information. For most of the above, I use Obsidian as my interface.

For the ones who haven't heard of Obsidian yet: Obsidian is a personal knowledge management (PKM) tool, some also call it a 'second brain'. Obsidian allows the interlinking of notes which over time results in a graph of connected knowledge. The notes are plaintext files stored locally in the Markdown format. This ensures continuity/access to the data if you're offline (for instance on a boat) or even without Obsidian. The interface is intuitive, and general enough that it allows for many different use cases. If you haven't tried it yet, please take a stab at it- It's free for personal usage. All in all, it's a beautiful piece of software.

Open Extensions

Visual Studio Code has become the go-to IDE for many software developers and one important reason might be its vast extension library, featuring among many more syntax extensions for the most obscure languages, integration with essential toolchains e.g. deployment, and code linting or manipulation. The latter has become surprisingly good with Github's CoPilot or with plugins using OpenAI GPT-3/4 in the background (experiment last year).

Analogous to Visual Studio Code, Obsidian features extendability via Community Plugins. The power/flexibility that this gives to a code literate is unparalleled. There are many helpful plugins that I use on my day-to-day, however, here I'd like to focus on the LLM-powered plugins TextGenerator and Copilot (not to be confused with GitHub's CoPilot) and use cases, such as the ones described at the beginning of the article.

Plugins for Online and Local LLMs

The first plugin, I tried is nhaouari's TextGenerator. The most basic feature allows you to write prompts inline or select existing text to be evaluated as a prompt using various models. The plugin allows extendability with a prompt template system allowing users to create reusable prompts that can parse Markdown and Obsidian syntax, see documentation. If the underlying model supports it, it even allows multi-modal prompt input such as PDFs, audio, and images. Supported are proprietary models from OpenAI, Anthropic, and Google but also- and this is great for privacy, offline-first, and independence: TextGenerator features local open-source LLM integration with Ollama. Ollama is a user-friendly application to download and run open-source LLMs such as Mistral, Nous Research Hermes and many others. And even better: you don't even need a beefy GPU: e.g. Hermes 13B runs fast on a MacBook Pro M1 Pro.

The second plugin I want to highlight in this introductory post is Logan Yang's Obsidian Copilot. One set up with OpenAI or local LLM backend, you have a chat side pane to prompt generally or to prompt specifically towards the current note. Compared to TextGenerator, this option is more suitable for on-demand prompting and prototyping of prompts/questions. You can even save the chat log as a note to do further introspection. Therefore, using both in combination is ideal: first, determine the most effective approach and then create a reusable prompt template for TextGenerator.

Conclusion

The integration of LLM-powered extensions within open platforms such as Obsidian and Visual Studio Code represents a significant leap forward in how we manipulate information in natural language or code. Importantly, these new LLM enablers come in two different flavors: online-proprietary and local-open. While general proprietary LLMs are leading the general purpose benchmarks, the scores of open general LLMs caught up fast. Also- open LLMs allow for crowd-sourced fine-tuning and specialization, as seen on HuggingFace, which is similar to the extension/plugin architecture of Obsidian and Visual Studio Code. I'm personally excited to see more extendable interfaces to make use of the coming plethora of (specialized) open LLMs.

To end with some ideas one could think of a collaborative/multi-user photo-editing such as Photoshop or Gimp, DAW such as Ableton, a whiteboard such as Miro or Mural, or even a "Blender" (VR editing) that allows the easy creation of template prompts (for inserting, manipulating or commenting) based on online-proprietary or crowd-sourced local-open models.