Project Spotlight: KOI Pond
Project Spotlight: KOI Pond
Metagov is partnering with BlockScience to integrate an experimental knowledge management infrastructure—known as Knowledge Organization Infrastructure (or KOI)—into its knowledge ecosystem.
Welcome to another Metagov Project Spotlight
In this spotlight, Metagovernor and KOI Pond contributor Brooke Ann Coco introduces KOI Pond, an initiative that unites the complementary expertise of BlockScience—a complex systems engineering, research and development, and analytics firm with a history of architecting information infrastructures—and Metagov’s range of self-governance research and experiments developed in its laboratory for digital governance.
The partnership of these two organizations will facilitate experimental research into the self-governance of Knowledge Organization Infrastructures by online communities in support of Metagov’s mission of “cultivat[ing] tools, practices, and communities that enable self-governance in the digital age.”
This spotlight introduces the project, details the history of its development, and includes information about how to participate in the project—in particular our Koimmunity Cove Calls, a community call kicking off today, June 13th at 8-9pm UTC, where we will demo, discuss, and steer the KOI Pond project.
Introduction
Since its inception in 2019, Metagov has evolved from a small group of researchers into an expansive network of researchers, practitioners, and activators of digital governance. Together, they collaborate around a shared vision: "working toward a governance layer for the internet that is empowering, creative, interconnected, and accountable." As Metagov has expanded, its research activities have flourished, creating a knowledge commons that serves as an open resource for the benefit of all.
While Metagov has celebrated this growth, this expansion has not come without challenges. As the organization's community and research outputs have become more varied, Metagov has had to grapple with issues associated with scaling.1 These include defining its shifting identity, establishing clear boundaries for its geographically dispersed network, and organizing its burgeoning knowledge base. To this end, several community initiatives are actively working to address these challenges. These efforts include the launch of Metagov's redesigned website, along with revamped mission, vision, and value statements developed through co-design workshops and community feedback. Metagov has also been experimenting with various approaches to community membership as well as improving orientation materials and processes to better assimilate new and prospective members.
In this post, we are excited to introduce another such initiative aimed at addressing these challenges - KOI Pond.
The Origin Story: BlockScience’s KMS
Frustrated by the fragmented state of digital data management and its contribution to problematic attention economies, BlockScience built a locally aggregated search tool that could retrieve contextually relevant data when and where it was needed. Known internally as the Knowledge Management System (KMS), the in-house tool integrated all of BlockScience’s internal non-proprietary data to streamline knowledge retrieval within the organization. However, despite its utility, the KMS saw minimal adoption within BlockScience. Although the KMS could locate relevant information within the organization’s ecosystem, it introduced yet another platform that team members had to incorporate into their daily routines.
As large language models (LLMs) had increasingly captivated the global zeitgeist, one team member off-handedly suggested integrating an LLM interface to enhance the system's usability. Within days, the interface was operational, sparking excitement throughout the organization. Team members flocked to the tool, eager to engage with the newly dynamic knowledge system.
This was a powerful concept. But why?
Like most algorithmic systems, LLMs are designed with scalability in mind. They are trained on massive data sets to identify syntactic patterns in human language. Utilizing this knowledge, LLMs perform advanced calculations to determine the most probable sequence of words in response to given prompts. Therefore, despite their extensive knowledge base, LLMs lack semantic nuance. As a result, they require a well-defined context to generate useful, rather than generic, responses. This explains the surge in educational resources focused on prompt engineering. For this reason, some experts, such as Ethan Mollick, suggest treating these tools as if they were enthusiastic interns - confident, hard-working, and eager to please, yet needing clear expectations, close guidance, and frequent verification of their work.
The introduction of an LLM as an interface to BlockScience's KMS effectively addressed the challenge of providing an LLM with a definitive context space. As one of KMS’s lead developers highlighted, “LLMs as an interface are only as good as the underlying data structure” (Orion Reed in Nabben 2023, p16). With direct access to the KMS, the LLM was capable of delivering highly contextual information tailored to BlockScience’s knowledge environment (as pictured below). Michael Zargham, Metagov Research Director and founder/CEO of BlockScience further observed, “The introduction of [an] LLM interface… brought the information in the KMS to the user instead of making the user go to the data.” This user-friendly interface transformed the KMS from an intellectually stimulating but operationally limited initiative into one of BlockScience's most exciting and engaging projects. Ultimately, the team opted to allocate internal resources to integrate the system more permanently into BlockScience's technical infrastructure.
Discussions and Applications of LLMs in Metagov
Meanwhile, as BlockScience was building out its KMS, discussions surrounding LLMs were becoming more prevalent within Metagov. Notably, Metagovernors within the #attentionecon Slack channel exchanged examples of where LLMs were being used in governance. These conversations helped inform this case study on Valory’s Governatooorr. An expanded version of this case study was later included in the working group’s forthcoming article entitled “Online Governance Surfaces and Attention Economies”. Others had begun incorporating LLMs into Metagov's D20 Governance project, in which Metagovernors built an interactive environment where communities could actively learn about basic governance concepts then engage with them in a playful, hands-on manner.
Faced with the ongoing challenge of defining Metagov’s cultural and organizational identity, another Metagovernor proposed the use of LLMs as an ethnographic tool to help address this identity crisis. Subsequent discussions about how to employ an LLM in this way continued to unfold in the #govbase-labs Slack channel, where Metagovernors began to brainstorm potential research designs. Early discussions considered fine-tuning a foundational LLM on Metagov Slack data. Recognizing the relevance of BlockScience’s concurrent KMS project, Michael Zargham encouraged key KMS developers to engage in these discussions. He further initiated the idea of a potential collaboration between Metagov and BlockScience on this research as an expansion of our past collaborations which have included work on tools for participatory digital ethnography and theorizing computer aided ethnography.
Metagov’s Partnership with BlockScience
In line with their core values, BlockScience eventually decided to open-source the technical architecture of its internal KMS-GPT system. However, originally developed for in-house use, the infrastructure requires retrofitting to enhance accessibility and adaptability for diverse use cases. Therefore, before the system could be fit for external use, BlockScience needed to identify a test case sufficiently different from their own operations to refine the architecture for broader application. Due to its expertise in digital self-governance and its explicit interest in experimenting with LLMs, Metagov was a logical choice for this collaboration. In exchange, BlockScience's KMS-GPT infrastructure seemed poised to help Metagov manage the dynamic, distributed knowledge environment it had been grappling with.
To reflect the broadening scope of BlockScience's KMS-GPT infrastructure, the improved architecture was renamed Knowledge Organization Infrastructure (KOI), while Metagov's experimentation with the infrastructure became known as KOI Pond.
Indeed, Metagov acts as an effective foil to BlockScience. As a firm, BlockScience operates with a more centralized corporate governance structure, where decision-making is often constrained by business policy. It is characterized by a smaller, technically proficient team that benefits from tight feedback loops on its projects. In contrast, Metagov, a nonprofit, has a more distributed operational structure. Metagov has a larger and more diffuse membership with varying levels of technical proficiency. Because many Metagov-affiliated research projects involve a mix of volunteers and contractors, the feedback loops on these initiatives are comparatively looser.
Furthermore, BlockScience and Metagov operate within vastly divergent knowledge environments and use cases. As a small, private engineering and R&D firm, BlockScience leverages its KMS-GPT system to enhance the organization of non-client confidential data for internal use. Consequently, external data access and participatory governance tools were deprioritized. By contrast, Metagov handles a wide spectrum of data that ranges from public to private. As an open community, Metagov must be much more vigilant in attending to the governance of its knowledge objects. For this reason, Metagov appeared to be an ideal candidate to pilot the Reference Identifier (RID) system, a new feature set that BlockScience began developing in mid-2023. Integration of the RID system to the KOI architecture is anticipated to improve the flexibility of community data curation by enabling knowledge objects to be organized dynamically without the constraints of a fixed data structure. In affording greater plasticity, the RID system allows the KOI to be tailored to specific operational contexts.
Metagov’s KOI
Over time, a working group took shape from the weekly Govbase Labs calls to discuss the logistical, analytical, and operational details of implementing a KOI into Metagov's ecosystem. During the initial stages of the project, as lead developer Luke Miller prepared the architecture for integration, discussions centered on formulating research questions, enhancing our understanding of the system's functions and capabilities, designing research experiments, and addressing ethical considerations. As Metagov's KOI has become more operational, discussions have increasingly been directed towards funding and community engagement.
See these images for a preview of the system in development:
____
Want to join the conversation? Join Metagov's Slack and visit:
#koi-pond to participate in project discussion
#koi-tank to participate in infrastructure testing
#govbase-labs to join our weekly meetings (meeting information is in a pinned post to the channel)
____
We are also planning an ethnographic component to this research to examine how the Metagov community navigates the adoption, integration, and maintenance of its KOI. Of particular interest to this research are the digital governance tools and collective governance frameworks that the community employs to better align the KOI with its desired outcomes. Brooke Ann Coco, a fellow Metagovernor, is leading this portion of the research. The findings will contribute to her doctoral dissertation, which is being completed under the guidance of Professor Ellie Rennie and Distinguished Professor Jean Burgess. The ethnographic research has received ethics approval after successfully passing standard ethical review processes. More information about the ethnographic component of this project will be posted in future project updates.
Get Involved!
Now that Metagov’s KOI is nearly ready for user testing, we welcome the broader community to play a more active role in the project. We invite the community to attend our Koimmunity Cove Calls, a bi-weekly KOI community call where we will discuss and steer the KOI Pond project.
The inaugural Koimmunity Cove Call is scheduled for June 13, 2024 8-9pm UTC. This hour-long session will include an introductory overview of the proposed research experiments and current project documentation, a brief demonstration of its capabilities, a discussion of short-term goals and future aspirations, and a Q&A session. During this session, Meatgovernors can also sign up to receive updates on future contribution opportunities. The call will be recorded for those who are unable to attend.
Subsequent Koimmunity Cove Calls will be 30 minutes in duration every second and fourth Thursday of the month at 8:30-9pm UTC (unless stated otherwise). One of the early objectives of these sessions will be to establish community practices that encourage active involvement from the broader Metagov community. Activities may involve tasks such as prompting our KOI in #koi-tank, responding to polls about which data to have observed next, or participatory workshops to evaluate the quality of Metagov’s KOI. These practices are intended to be light touch actions to build practices around KOI, while enhancing collaboration and communication between system engineers, research directors, project contributors, and the broader community.
- Brooke Ann Coco
Brooke is a contributor to KOI Pond — see the full list of contributors on the project webpage.
To date, Metagov has been affiliated with over 30 different research projects, while the Metagov Slack engages over 280 active monthly users.
 | A guest post by
|