Adding Spans to Neopolitan's Flags (and Attribute Keys)

Visions of Code Dancing in My Head

I had another code dream^previously. This time, it was about flags.

Where it Started

Neopolitan^neo files consist of two things: blocks and spans.

Blocks are wrappers. Spans are content. Each can have metadata associated with them in the form of flags and attributes.

Here's an empty block with one of each:

{
  "attrs": {
    "class": [
      {
        "category": "text",
        "content": "example-attr",
        "kind": "text"
      }
    ]
  },
  "category": "basic",
  "children": [],
  "end_block": null,
  "flags": ["example-flag"],
  "kind": "title"
}

figure 1

Attributes (aka attrs) have a single key (class in figure 1) and an array of spans for the value. All span types are allowed. This provides a way to use attributes as full subsections of content.

Flags consist of an array strings (e.g. example-flag in figure 1).

Historically, attribute keys and individual flags could use any text character unless they needed to be escaped^escaped. The reason is reproducibility. The AST^ast would have to keep track of escaped characters in order to be able to reproduce a valid copy of the source^reproduce.

The Clunky Way To Avoid

Using an array of spans for the flags and attribute keys would solve the reproducibility problem.

For example, flags in code spans^code need to have the backtick character escaped. That means you couldn't reproduce the source file if all you had was this:

"flags": ["example`flag"],

figure 2

But, this would work:

"flags": [
  {
    "category": "text",
    "content": "example",
    "kind": "text"
  },
  {
    "category": "escaped-character",
    "content": "`",
    "kind": "escaped-span"
  },
  {
    "category": "text",
    "content": "flag",
    "kind": "text"
  }
]

figure 3

Unfortunately, that would make working with the flags in the output templates harder. Doing something like this would no longer be possible:

[! if "example-flag" in flags !]
  Do Something
[! endif !]

figure 4

You'd have to assemble the spans from the flag into a single string every time you wanted to check against them^complicated.

Enter Sandman

This problem was turning over in my dreams. Forbidding escape characters was the best compromise I had come up with to balance usability against ensuring the source is reproducible.

Effective as it is, I didn't like it. It keeps usability forefront by limiting what you can actually do with the format.

My subconscious didn't like it either. After chewing on it, it floated a solution to me during the hazy state between dreams and daylight:

Use the flags as keys that hold a value with their spans

This:

"flags": ["example`flag"],

figure 5

Turns into this:

"flags": {
  "example`flag": [
    {
      "category": "text",
      "content": "example",
      "kind": "text"
    },
    {
      "category": "escaped-character",
      "content": "`",
      "kind": "escaped-span"
    },
    {
      "category": "text",
      "content": "flag",
      "kind": "text"
    }
  ]
}

figure 6

Doing the conditional check from figure 4 works the same way. Only now, we can reassemble the source file by using the set of spans associated with it.

Friction Count Zero

My efforts in deigning Neopolitan are all about reducing friction. Aiming for the smoothest possible way to make web pages that offer the full power of HTML, CSS, and JavaScript.

I find the rough spots by using the tools. Adding these spans brings the things that feel like they are causing unnecessary friction to zero^format. Next step is to implement it and see if reality matches expectation.

If it does, it'll finalize Neopolitan version 2.0.

-a

-- end of line --

Endnotes

The last examples only shows adding the solution for flags. A similar approach will be used for attribute keys. I just need to do a little more thinking about exactly how I want to slot things in.

These example don't have the ok/error structure described in Using Rust's 'Result' Approach For Neopolitan's Parser. That'll be added as well. Doing so also means having the explicit ability to point to characters inside a flag or attribute key if parsing fails on them.

Footnotes

previously ⤴

Using Rust's 'Result' Approach For Neopolitan's Parser

neo ⤴

Neopolitan - Like Markdown on Steroids

The plain-text format this post is all about. Version 2.0 is getting closer every day.

escaped ⤴

Escaping a character means adding a \ in front of it. It's necessary when you want to use a character (or set of characters) that would otherwise cause something else to happen.

For example, code shorthands begin and end with pairs of backticks. If you want to a pair of backticks in a shorthand's attributes they have to be escaped. Otherwise, they would end up closing the shorthand. You'd end up with weird formatting or possibly the parser would fail all together.

ast ⤴

Abstract Syntax Trees (AST)

The Neopolitan parser reads the files we humans make and turns them into data that apps can use to build websites. The formal name for that type of data is Abstract Syntax Tree.

reproduce ⤴

As much as I focus on usability, being able to reproduce the file from the AST is a higher requirement.

Being reproducible means you can:

Parse a source file into and AST
Run the AST through a process to remake the source file
Parse the newly created source file and have it match the AST from the first step.

The source files from step 1 and 2 may not be identical. The biggest difference being the length of lines of text. The AST treats spaces and newlines the same. It also collapses multiple spaces in text spans into a single one.

The key is that none of the differences change the way the HTML output renders.

Using this approach opens up the ability to make auto formatters which are a nice quality of life improvement.

(It's possible to reproduce source files exactly by using a Parse tree (aka Concrete Syntax Tree but that's more complicated for no practical benefit.)

code ⤴

Code shorthands look like ``this``

complicated ⤴

Most likely you'd end up with a macro helper to do something like:

[! if "example-flag" in marco.flag_spans(flags) !]
  Do Something
[! endif !]

I used that in a prior iteration. It works, but it adds complexity and friction.

format ⤴

I'm speaking here about setting Neopolitan up to provide ways to reduce the friction of working with it as much as possible. It's still up to the tools that use the format to do a good job.