Innovation at the boundaries

When watching the Dark’s Philosophy talk on the darklang website I was really intrigued by a phrase they mentioned: “innovating at the boundaries”. They concluded that there were three pillars of the development workflow: editor, language, and infrastructure. They also discussed the large amount of accidental complexity here, especially in programming languages and infrastructure. It’s mostly attributed to incrementalism, where new functionality and abstractions slowly build upon what was already there.

Eventually, this results in a hot mess that we know very well in software.

Our current technology is the result of incremental improvements which has resulted in a lot of innovation, but has come at a high cost in terms of complexity. If you focus in on web development in particular, our tooling is a stratification of abstractions over decades. Dark seeks to change that by starting from scratch.

He mentions in his talk that they’re not seeking the reinvention of every wheel, but spending time innovating at the boundaries of the pillars in order to improve the developer experience. The editor itself is an abstraction over darklang’s AST which it uses not only for autocompletion, but surfacing live production data. Editing code is automatically deployed behind feature flags, with stopgaps in place for data migration.

I think that’s why this approach resonates so well with me, and is seemingly something I’ve sought out in lots of projects without realizing it. The boundaries are where you can have a high impact without as much of an investment of time and money. As they state, there’s typically lots of low-hanging fruit at these intersections like the editor and language.

In Dark in Action we see how creating a schema and querying the database provisioned it for you. It’s a subtle feature but is really powerful. Darklang understands the intent of the developer and knows that if you’re writing code that interacts with the database you probably want one provisioned.

This sounds ambitious

Understandably rewriting everything isn’t a sound product strategy, so they’re building an abstraction over current technologies that focuses on the boundaries. This becomes “deployless” because users don’t have to think about their infrastructure outside of the editor itself. It also allows the Dark team to focus on the most essential aspects of their workflow first: their three pillars.

Applying these concepts to Gatsby

It’s interesting to think about how some of these ideas can be applied to building for the frontend. It’s applicable to some of the thinking we’ve been doing with our work at Gatsby, particularly in regards to a Gatsby page building experience.

Gatsby itself has three pillars of development: Sourcing data, querying data, and rendering components.

Sourcing data is a pretty solved problem in Gatsby. For most popular services there’s typically already a plugin that’s straightforward to setup. Data is sourced and added to the GraphQL data layer. You can even apply additional plugins to transform data or add new nodes.

Querying data is also a solved in Gatsby since GraphQL and the GraphiQL explorer make the experience solid. Gatsby’s instance of GraphiQL will even generate the code for queries which you can cut and paste into your project.

Building and rendering components can be improved with a component builder that abstracts away React/Theme UI code, but generally speaking this is solved because of React and its rich ecosystem. Theme UI itself is an innovation at this layer which can be built upon. Ultimately I envision a page editor experience that allows users to edit components in a “No Code” way where production-ready React code is the data format.

This leaves me wondering: where can we innovate at these boundaries?

Between sourcing data and querying

Once data is sourced we need to be able to query it in order to pass to components and programmatically create pages. I see Gatsby themes as an important innovation that sits between sourcing data and querying it.

A key innovation here will be abstracting away the data source and building more core themes for data types. Ideally, being able to handle multiple providers for the same data type will be key, too. This would allow users to leave old posts in a CMS and switch to something new or even use the filesystem.

Between components and querying data

I think this is the most straightforward boundary which would have a high impact, data needs to be queried and then components render the data. A great workflow for building pages is allowing for users to essentially explore their data and then initialize components based on their production data. From there they can edit their blog listing with actual blog post information and then change their blog post page as well.

With Theme UI/MDX components it becomes possible to edit the components that render your content at a more granular level, too. Change the paragraph font size, tweak your syntax highlighting colors, add a border to block quotes, and tweak your YouTube embeds.

Between sourcing data and components

At first look it might not seem like there’s much you can do in terms of innovation between data sourcing and components, but I think there are ways we can improve this. When we detect new types of data we could prompt the user to instantiate a “template” for a listing and item page.

We often think of data in terms of collections and when we infer new data we can automate the templating workflow and scaffold out basic queries that a user can modify. Ultimately it’s sort of a repetitive task (though one you don’t have to do often) which we’d need to solve for an amazing Gatsby page building experience anyway.

Conclusion

Dark is definitely interesting technology and I’m excited to see what comes of it. I didn’t think that watching their videos would inspire so many thoughts but I’m glad it did.

Innovating at the boundaries seems a sound strategy for making impactful products with as small an investment as possible. I’m excited to see where we can take this at Gatsby to improve the end user experience, especially for those new to React/JavaScript/GraphQL.