Website​

The miti.today website is my attempt at making a minimal calendar that gets me the Nepali date without any fluff or ambition.

My requirements for the website were simple:

1. build it in a day
2. hosted for free
3. Loads fast

So I came up with a system where I host a static website on Cloudflare Pages. And every 24 hours at 12AM Nepal time, a cloudflare worker triggers the deployment again, updating the page.

An invocation a day is well within the free tier, and something that even if I had to pay for, it would be in cents/day.

I use a table of months in each day to calculate what date today should be:

const daysInYears = [
  ... 
  [31, 32, 31, 32, 31, 30, 30, 30, 29, 29, 30, 30], // 2080
  [31, 32, 31, 32, 31, 30, 30, 30, 29, 30, 29, 31], // 2081
  ...
];

const startingTime = new Date("2018-04-14T00:00:00+05:45").getTime() / 1000;
let daysSince = (seconds - startingTimestamp) / 86400;

let year = 0,
month = 0,
day = 0,
firstDayOfMonth = 0;

let _daySum = 0;
const daysInYearsFlat = daysInYears.flat();
for (let i = 0; i < daysInYearsFlat.length; i++) {
    _daySum += daysInYearsFlat[i];
    if (_daySum >= daysSince) {
        _daySum -= daysInYearsFlat[i]; // revert
        year = Math.floor(i / 12) + 2075;
        month = (i % 12) + 1;
        day = Math.floor(daysSince - _daySum) + 1;
        firstDayOfMonth = (_daySum % 7) - 1;
        break;
    }
}

The code is crude but works fine.

For a while I considered using the pancanga perl script by Professors M. YANO and M FUSHIMI (demo here). It is interesting and and something I hope to study in detail in future, but ultimately I felt it was too complex for my current use.

Once I have the date and the calendar, I generate the HTML "by hand".

 calendar.forEach((week) => {
    week.forEach((day) => {
      let $day;
      if (day === "-") {
        $day = $(`<div class='day'></div>`);
      } else {
        $day = $(
          `<div class='day${
            dateArray[2] === parseInt(day) ? " toDay" : ""
          }'>${day}</div>`
        );
      }
      $calendar.append($day);
    });
  });

  const html = $(".calendar").prop("outerHTML");

  const monthEnHtml = `<div class="monthEn">${
    monthsEn[dateArray[1] - 1]
  }</div>`;

I'm using Cheerio to work with HTML easier. The generated HTML gets injected into a template using mustache.

It was important to me that the HTML generated be a single page, with no extra HTTP requests once it reaches the browser. Thus:

1. I embedded the font in the HTML (and makes up for more than 90% of the page weight)
2. The favicon is a SVG that has today's date in Nepali.
3. The CSS is inlined.
4. There is no JS.

Apps​

The miti.today website was useful. It is something I reference everyday.

But over time, I began to wonder if a widget would be more useful. I decided to write a simple widget for iOS and see if it would be more useful. I used SwiftUI to make it, it was quite pleasant.

It turned out to beuseful. Some of my friends asked for an Android version. So I used Compose with Glance to make it. It wasn't as pleasant.

The Android app is in internal testing in Play Store, and I'm waiting for my Apple developer account to be approved. I'll update this page with links after they are generally available. The apps will be listed at app.miti.today.

Your real server is behind a reverse proxy. But all ingress traffic comes through a different gated proxy not in your control that locks everything behind it with a Basic Auth.

This is quite a pickle. Because both the basic auth that Nginx is expecting and the bearer token that our server is expecting uses the same Authentication HTTP header.

Basic auth is cool because it is a simple way to password protect a system. The browser itself handles asking the user for password, saving it in memory and making sure all requests have the basic auth header. The HTML/JS code doesn't even have to know if it's password protected.

But it's also inflexible and less secure. Just logging out is a dance. Bearer token based authentication is flexible, can be made much more secure, and allows for a much better user and developer experience.

Eating the Pickle​

It would make sense that, for certain applications (for example in-house tools), all incoming traffic is protected by default with a basic auth. But the more complex application-specific authorization is handled with a proper JWT token.

In such cases, we can get by with some JavaScript and Caddy gymnastics. And an extra header: X-Custom-Authorization (arbitrarily named).

Caddy Configuration​

Here Caddy is doing the fun things. It sets the value of the Authorization header to whatever is in the X-Custom-Authorization header before any request goes to the upstream server.

example.com {
  reverse_proxy localhost:3000{
    header_up Authorization {http.request.header.X-Custom-Authorization}
  }
}

What's going on in the client browser?​

We also need some code on the browser to make sure that:

1. the Authorization header doesn't exist. If the browser notices that we have manually set the Authorization header, it won't overwrite it with the basic auth credentials. Since asking for, saving, and using the basic auth is handled by the browser, we must allow browser to handle this auth.

2. before any request goes to the browser, it is intercepted and it's outgoing bearer Authorization header is renamed to X-Custom-Authorization, leaving the Authorization free for the browser to use.

Axios Interceptor​

If we had complete control of the client code, we'd just make sure all our requests went through a middleware before passing through. For example, as an Axios interceptor:

axios.interceptors.request.use(
  (config) => {
    config.headers["X-Custom-Authorization"] = config.headers["Authorization"];
    delete config.headers["Authorization"];
    return config;
  },
  (error) => {
    return Promise.reject(error);
  }
);

Pocketbase​

Or as a PocketBase before send hook:

pb.beforeSend = function (url, options) {
  if (options.headers["Authorization"]) {
    options.headers = Object.assign({}, options.headers, {
      "X-Custom-Authorization": options.headers["Authorization"],
    });
    delete options.headers.Authorization;
  }

  return { url, options };
};

Chrome Extension​

However, we might not always have access to all parts of the client JS code. If its for an internal tool, then a simple chrome extension might suffice:

chrome.webRequest.onBeforeSendHeaders.addListener(
  function (details) {
    if (details.frameType === "outermost_frame") {
      for (var i = 0; i < details.requestHeaders.length; ++i) {
        if (details.requestHeaders[i].name === "Authorization") {
          details.requestHeaders[i].name = "X-Custom-Authorization";
          break;
        }
      }
    }
    return { requestHeaders: details.requestHeaders };
  },
  { urls: ["<all_urls>"] },
  ["blocking", "requestHeaders"]
);

{
  "manifest_version": 2,
  "name": "Outbound Header Modifier",
  "version": "1.0.0",
  "permissions": ["webRequest", "webRequestBlocking", "https://example.com/*"],
  "background": {
    "scripts": ["background.js"],
    "persistent": true
  }
}

Conclusion​

Whatever the method, what we're doing is making sure that the right layers in the stack see the appropriate information.

I finally got the Playdate. This little yellow machine is nice to look at and quite fun to play with, but it doesn't yet have the kind of games I like to play. No problem, I'm writing them myself.

Meanwhile, I have 3d printed a stand for it, and it sits on my desk acting as a unique clock.

Now that I have a stand and a really cool memory LCD display which barely needs power to display static images, I can display anything on it while it sits on my desk.

One of my pet peeves has been that computers and mobile phones display date in AD, but daily life in Nepal needs me to know the date in BS. And I can't just whip out my phone to check the Nepali date without resorting to some external app or some ugly widget. The external app (or website) is almost always riddled with ads, horoscopes, religious messaging, and news. They load slow and the important information is is buried in a maze of irrelevant information.

To remedy these minor annoyances, I have done two important things:

1. I have established a website miti.today for times when I'm already using a browser and I just need to check the date really fast. It just displays the Nepali date today in large font in the center of the page, and barely anything else. I am also working on patro.today which will be an extremly simple calendar website.

2. I wrote a simple wallpaper utility for the Mac. It edits in the time and Nepali date information into a selected wallpaper and changes it every minute. The code was a lot more involved than I expected it to be, but I use it on my monitors has been useful. niravcodes/clock_on_wallpaper_mac

Now that I have the Playdate I'm free to add a third:

• Write software for the Playdate that takes as little power as possible while it displays the current time, date, and a wallpaper of my selection.

So I went ahead and wrote it.

The really awesome robot wallpaper I'm using is titled The Nerd Desk by Marcelo Vaz.

So a friend was building a device which had an array of 6 seven-segment-displays. We were thinking about some cool idle and loading animations we might be able to program into it.

I didn't ask for the specifics but it was clear that the display unit had an arrangement of one shift register (probably a 74HC595) per digit. The 74HC595 is a nice little chip that can be fed one byte serially, and it outputs that byte in parallel. It also latches onto that byte for however long we ask it to, and it can be daisy-chained. So it is a cheap way to drive a huge amount of LED lights with a few micro-controller pins.

He was hand-writing the bytes. Each 1 on the byte turns on some LED, and each 0 turns it off (0b01001111 or 0x4f to display the digit '3'). I realized I could quickly hack together some tool to make it easier and more fun to animate.

It took me about an hour to write the tool but I polished it up a little for this article.

One Seven Segment display​

First I need to display and manipulate a single digit.

The Model​

These kinds of systems are best described by some form of Model-View-Presenter separation.

Which is just a fancy way of saying: keep the representation of the LED matrix separate from the visualization of it.

If we assume the display to be labelled like below:

It can be described with a single object:

let model = { a: 0, b: 0, c: 0, d: 0, e: 0, f: 0, g: 0, h: 0 };

This model can theoretically represent all the states of the digit display. For example, to display one, this model would be:

{ a: 0, b: 1, c: 1, d: 0, e: 0, f: 0, g: 0, h: 0 };

Rendering​

Now we need to find some way to display this model visually. It could have been a picture or a canvas. For this project, it is a bunch of <div> elements positioned with some CSS:

function render(model) {
  const container = document.createElement("div");
  container.className = "sevenseg";

  Object.entries(model).forEach(([key, value]) => {
    const el = document.createElement("div");
    el.style.backgroundColor = value ? ON_COLOR : OFF_COLOR;
    el.className = "seg seg-" + key;
    container.appendChild(el);
  });
  return container;
}

The render method simply takes any model object and faithfully generates a visual representation of it. Notice that it is a pure function. It has no side effects. It doesn't affect the program in any way other than taking an model as an input, doing some processing on it, and returning an output.

This separation is an incredible thing. We have cleverly avoided all kinds of complications by having a pure render function that just renders whatever the system state is describing.

Now, changing the system model is a simple matter of changing the model and calling the render function again.

function render(model) {
  const container = document.createElement("div");
  container.className = "sevenseg";

  Object.entries(model).forEach(([key, value]) => {
    const el = document.createElement("div");
    el.style.backgroundColor = value ? ON_COLOR : OFF_COLOR;
    el.className = "seg seg-" + key;
    el.onclick = () => present({ ...model, [key]: !model[key] });
    container.appendChild(el);
  });
  return container;
}

function present(model) {
  document.body.innerHTML = "";
  const view = render(model);
  document.body.appendChild(view);
}

Note that we have added an el.onclick function to each of the segment elements. Each time a segment is clicked on, it toggles its associated key in the model, and calls render again (via the present function). The present function simply makes sure that each rendering overwrites the last one.

Serialization​

Serialization here means just converting the digit as it is being viewed into the final byte representation that will be programmed into the display firmware.

The really cool thing about the Model-View separation is that one model can have multiple views. Conceptually, a serialized byte representation of the system model can also be considered a view. So it can be "rendered" as before, but onto a different "medium".

const KEY_ORDER = ["h", "g", "f", "e", "d", "c", "b", "a"];

function serialize(model) {
  let binaryString = "";
  for (const key of KEY_ORDER) {
    if (model.hasOwnProperty(key)) {
      const binaryValue = model[key] ? "1" : "0";
      binaryString += binaryValue;
    } else {
      binaryString += "0";
    }
  }
  // Convert binary string to hexadecimal
  const hexString = parseInt(binaryString, 2).toString(16).padStart(2, "0");
  return "0x" + hexString;
}

The KEY_ORDER constant is introduced because what binary bits map onto what LEDs is hardware dependent.

Multiple displays​

An easy way to support multiple displays is to compartmentalize all the data pertaining to a single display inside a class.

const ON_COLOR = "red";
const OFF_COLOR = "black";
const KEY_ORDER = ["h", "g", "f", "e", "d", "c", "b", "a"];
const EMPTY = { a: 0, b: 0, c: 0, d: 0, e: 0, f: 0, g: 0, h: 0 };

class SevenSeg {
  constructor(root) {
    this.root = $(root);
    this.model = { ...EMPTY };
    this.render();
  }

  render() {
    const container = document.createElement("div");
    container.className = "sevenseg";
    Object.entries(this.model).forEach(([key, value]) => {
      const el = document.createElement("div");
      el.style.backgroundColor = value ? ON_COLOR : OFF_COLOR;
      el.className = "seg seg-" + key;
      el.onclick = (() => {
        this.model[key] = !this.model[key];
        this.render();
      }).bind(this);
      container.appendChild(el);
    });
    this.root.innerHTML = "";
    this.root.appendChild(container);
  }

  serialize() {
    let binaryString = "";
    for (const key of KEY_ORDER) {
      if (this.model.hasOwnProperty(key)) {
        const binaryValue = this.model[key] ? "1" : "0";
        binaryString += binaryValue;
      } else {
        binaryString += "0";
      }
    }
    // Convert binary string to hexadecimal
    const hexString = parseInt(binaryString, 2).toString(16).padStart(2, "0");
    return "0x" + hexString;
  }
}

.container { display: flex; }

.sevenseg { width: 80px; position: relative; transform: skew(-7deg); margin-left: 20px; }
.segGroup { width: 120px; height: 190px; }

.seg { position: absolute; }
.sevenseg > .seg-a { top: 0px; left: 0px; height: 15px; width: 70px; }
.sevenseg > .seg-b { top: 0px; right: 0px; height: 70px; width: 15px; }
.sevenseg > .seg-c { top: 70px; right: 0px; height: 70px; width: 15px; }
.sevenseg > .seg-d { top: 140px; right: 0px; width: 70px; height: 15px; }
.sevenseg > .seg-e { top: 70px; left: 0px; height: 70px; width: 15px; }
.sevenseg > .seg-f { top: 0px; left: 0px; height: 70px; width: 15px; }
.sevenseg > .seg-g { top: 70px; right: 0px; width: 70px; height: 15px; }
.sevenseg > .seg-h { top: 140px; left: 90px; height: 15px; width: 15px; }

Now we can just create a new SevenSeg() per display.

const NUM_DISPLAYS = 6;
const $ = (x) => document.getElementById(x);
let frames = [];

function createNewSegmentGroup(frameIdentifier, frame = []) {
  const div = document.createElement("div");
  div.id = `container${frameIdentifier}`;
  div.className = "container";
  $("container").appendChild(div);
  let display = [];
  for (let j = 1; j <= NUM_DISPLAYS; j++) {
    const segContainer = document.createElement("div");
    segContainer.id = `c${frameIdentifier}a${j}`;
    segContainer.className = "segGroup";
    div.appendChild(segContainer);
    let seg = new SevenSeg(segContainer.id);
    if (frame[j - 1] !== undefined) seg.deserialize(frame[j - 1]);
    display.push(seg);
  }
  frames.push(display);
}

Why not React?​

For simple and quick-use application like this, the abstractions of React feel clunky and unnecessary. Remember that React came into the scene to make complex web-apps simpler to manage. It didn't invent interactive web pages. And in any case, without an intuitive grasp of the system model, React wouldn't have been much help.

CLICK and even the cyanbox universe stops and the silence starts; silence of the eyes and the shadows looming and the shadows looming and the shadows moving about in the absolute darkness; these shadows, like rainclouds, like whispered questions, shadows of an old friend; these shadows like a tree in the forest and the forest unwhispering unmoving planted firmly in the ancient and everlasting night.

this ancient night that I have inherited and that I have suffered like a fever all this time, is all I have yet I keep it at bay with the overbright incandescent lights.

can't sleep won't sleep
nirav unsleeping in this non-night

In the dark will it start?
the slow death the red dream
just the soul; nowhere;
content and wandering
in the dark forest of the night

or will it start the thoughtengine sputtering and smokevomiting the ugly mechanical and noisy discordant rising from the gutter truly an abomination this thoughtengine; spewing and oozing from rusted pipes leaking and smelly too and it just comes to be in the ancient night the ancestral night,

pristine, with fireflies decorated,
pure, like the sound of a stream,
the silent nightforest where 
my soul would be content if not for the

unwieldy and ginormous, infested with flies this thoughtengine thoughtmuttering thoughtsputtering thick acidlike illwords strung in illsentences and singing them cacophonic in the language of sorrows and fears; making sin of the sacred; lustful eyes froglegs the thoughtengine looks at my soul and corrupts my being.

If you've ever had to use Nepal Stock Exchange's Trade Management System (often called just TMS), you probably hate it. It is at best an incompetently made software with many glaring issues, hosted over a woefully underpowered infrastructure that cannot even handle the most predictable of traffic spikes. On more than one occasion I've oversold or undersold shares because it's UI was out of sync with it's database.

The TMS's login captcha is the epitome of this incompetence and negligence.

If you didn't know, a captcha is supposed to be a challenge that only humans can solve in a reasonable time. A malicious computer program which, if unopposed, could execute hundreds of thousands of requests every second testing various username:password combos or security vulnerabilities, or just plain overloading the system so that it becomes unresponsive to legitimate users, is blocked by the captcha.

With that definition, however, the TMS's "captcha" actually turns out to be an anti-captcha, the exact opposite. It appears as a challenge to the user, who has to manually type it out every time. But it is completely transparent to any half-decent computer script. In fact, when you hit the login button, the captcha is not even sent to the server. It's a purely cosmetic effect. It's like a gate that you have to painstakingly unlock every time, but every burglar can pass through it as if it wasn't even there.

The TMS captcha implementation​

Even a cursory inspection will show that the implementation of the TMS captcha system is not an actual captcha. Any OCR like Google Lens can read it readily, making it very cheap to overcome.

But you don't have to go that far. The captcha's random text is generated in the browser by a simple JavaScript code, and it's laid over an static image. The css user-select property is set to none so you can't just copy the text from over the image and paste it in the captcha field.

Which means it should be accessible via the DOM:

Unsurprisingly, it is.

Now, if you can read the text from the DOM, it's not a big deal to write it back to DOM:

document.getElementById("captchaEnter").value =
  document.getElementById("randomfield").value;

This piece of code will autofill the captcha input.

A problem arises​

An interesting problem that emerges is that when you autofill the captcha input via JavaScript then and hit Login, the website still says "Incorrect Captcha" even when the captcha is actually correct.

But how could that be? Could the TMS's system be more well thought out than I expected?

You wish! Turns out, they're using a frontend framework (Angular.js). Angular.js internally maintains a state, with the values of all inputs. That state gets altered only when the corresponding input's input event is fired. Otherwise the value change is not registered. But that's easy to fire. Here:

const $ = (_) => document.getElementById(_);
$("captchaEnter").value = $("randomfield").value;
$("captchaEnter").dispatchEvent(new Event("input")); // Fire the event to trigger angular state change

And with that, the captcha auto-filler works.

Making a chrome extension​

Now that I've come this far, I might as well make a chrome extension. so that I don't have to deal with this again. Chrome extensions are pretty easy to make. It takes like 10 minutes to make something this simple. Here's the relevant part of the code:

1. manifest.json

{
  "name": "TMS Captcha Autofiller",
  "description": "This extention will autofill TMS captchas. Made by nirav.com.np",
  "version": "0.5",
  "manifest_version": 3,
  "content_scripts": [
    {
      "matches": ["https://*.nepsetms.com.np/login"],
      "js": ["contentScript.js"]
    }
  ]
}

2. contentScript.js

{
  const $ = (_) => document.getElementById(_); //aliasing a long fn call
  function l() {
    $("captchaEnter").value = $("randomfield").value;
    $("captchaEnter").dispatchEvent(new Event("input")); // Fire the event to trigger angular state change
  }
  window.onload = l;
}

Done. Put these two files in a folder and import it from the chrome extension page (chrome://extensions).

A slightly more decorated version is published in my Github: github.com/niravcodes/TMSCaptchaAutoFiller.

You can go there and install that version instead.

Update: The extension is now available in Google Extension store. So you can download the app from official Google channel. Download Here

Request to the TMS devs

Please change the font in the captcha field from sans serif to a serif one if that's the least you do. Thousands of people needlessly confuse capital i and small L everyday.

Update

The websites have been fixed and the captcha font has been changed to a legible one. Thanks devs.

This is how it turned out finally:

To play this game in your browser, click here.

Before starting, the question was how I would simulate the motion of an octopus in the ocean. So I started with a prototype in javascript. After some trial and error, and some primitive physics, I had the rough behavior of the octopus down.

It worked as I'd envisioned but looked rather drab. So I found a placeholder animation on Dribbble by Narubalie and put it in. At this stage I had to write an animation-player for the game, because HTML5 canvas can't directly play a gif.

Next I wrote an AABB collision detection system (meaning non-rotating rectangular bounding boxes can now detect when they overlap). So now I could finally make a little island for the octopus to land on.

Next up is obviously an angry enemy angler fish. I found a cool animation on Tumblr made by Blabberf, and coded it's motion.

Now I wanted the octopus to be able to explore the ocean outside of the rectangular box it's confined in. So I wrote a camera system that will follow the octopus wherever it goes. I was also experimenting with different speeds at this time so it might seem jumpy in the video.

Then I wrote the level management system and made new components (text, static image, etc) to build the UI of the game with. I found a nice octopus icon made by FreePik from flaticon.com for the title screen.

At some point, I realized that I could decouple the game code from the game engine code, so I did. Then, just for kicks, I made a Flappy Bird clone using only the Engine code, which was rather fun.

Now I wanted to animate my own main character instead of using some random image on the internet. So I studied a lot of videos on octopus motion, and got to work on Piskel app, which is a nice browser-based pixel art software. After laboring over the motion for days, this is what I came up with:

You'll notice that this octopus is far more exaggerated in motion than the previous one, which is intentional, and I'm rather proud of this.

Next, I made the animation of the octopus crawling on the floor.

You might notice that the sprites are lacking colors. That's because I'm not a good artist, and colors are difficult.

Next I coded a quick background parallax system and a minimal emitter system to emit bubbles convincingly. This is the end result.

Then I burned out. And now I can't bring myself to even look at the code. Maybe you can, though. So check out the code on GitHub.

Conclusion​

I worked on this project without much planning, or design. So it was interesting to see the abstractions almost form themselves, and to see the architecture emerge spontaneously from beneath the insignificant lines of code as I typed them.

Right in front of my eyes, the coordinate-based motion wrapped itself in a physics interface and neatly exposed methods like applyForce (magnitude, angle). The Spritesheet animation system amalgamated with Motion and Collider to from a high level Character object for my benefit. Components organised themselves into categories like drawables and managers, which allowed for other interesting experiments like duck-typing-aided composition.

Not to mean that it is perfection, of course. Far from it. On the engine side, the API is painful, there's a lot of vestigial code, scenes are not isolated (and colliders leak through by default), there is no obvious way to handle instantiation of multiple objects, and there's a lot of cohesion among components. On the game side, the controls are terrible, the world is incomplete, and moving around is clumsy at best. The only consolation is that my next game will be easier to make because of this exercise.

Like the wind she'll
blow out the candles I've lit
and like the wind she'll
whisper in my ear

I'm waiting for that
whisper in my ear ...

Because I've picked up this strange habit of forgetting to exist at times. True, I wake up in flesh and blood, and I walk through the air as it parts for me. Yet between my step out of one room and into the other, I find myself dissolving into thin air: much like a whisper that never finds its ear.

I keep on waiting
for a quick kiss;
a peck on the cheek will do.
or a red slap
right on that same cheek.

Because I've picked up this strange habit of wallowing in ennui; of basking in the dull glow of the half moon and SCREAMING as a phantasm that never existed yet I'm right here and phantasmal and SCREAMING

I keep on waiting
on the terrace and barefoot

I keep on waiting
on the terrace at 11 pm

I keep on waiting
on the terrace with the power out

I keep on waiting
on the terrace where I'm the only man in the universe

It's 11 pm, served with a light drizzle and a garnish of absolute darkness here on this terrace where I stand so lonely I don't exist at all

And here the stars have descended into the gourds amongst whom I stand
And here they'll wander down to the kitchen garden twinkling as ever
And here they'll land upon the bushes: a constellation of berries
And here they'll ripen into Fireflies,
one presumes,
to flicker for a little while just
to go out like a candle

And among these Fireflies, I wait.
I wait for that barefoot wind and her subdued whisper
to put me out for a little while
so I can wake up in flesh and blood
and remain in flesh and blood for the rest of the day

Because I've picked up this strange habit of being everywhen:
in the vast array of nameless days,
in the spectrum between the Full Moon and the Crescent,
during the afternoons and the evenings,
and all moments in between.

And so my abode is smeared across days of the week and stretched across the months
And it might eventually settle itself across the years and decades
like a dead cat curled up but rotten.

In my abode the time-stream flows like endless words, relentless words,
words that never stop and words that never stay

So I wait,
for that flowery giggle
which will punctuate these words
into sentences that I can live
one at a time

So I wait
for the kisses
which will punctuate these
Sundays and Wednesdays,
and the untold number of Fridays,
into hours that I can live
one at a time

Featured photo by Mike Lewinski on Unsplash

Meet Flappy Millennial. He's an average guy in his mid 20's and the future looks bleak for him. With the threats of climate change, mass-extinctions, rising authoritarianism, economic depression, unprecedented wildfires and so much more ever-looming, nihilism has become his background music.

But worry not. For he has an antidote—well, not so much an antidote as a sedative. He has his phone! Come help this Flappy Millennial ignore reality while he scrolls through Instagram or something. You should play this game not because it's good or anything, but simply so you too can pretend that things are okay for a while.

Play Game

Browse the source code on GitHub at niravcodes/flappy-millennial.

How it came to be​

I began writing a different game for HTML5 canvas a month or so ago. As I was programming the game, I realised that a lot of my code was game agnostic. The abstractions and modules that had organically emerged could be packaged up and used to make other games. So I separated the game logic from the lower level management code, and lo! I had created my first game engine essentially as a side effect of refactoring.

Over time, I began to notice some problems with the design of the game engine API, and to investigate further, I asked some of my friends to try using it to make a game and report issues. But apparently they had their own work to do and "ain't nobody got time for fun and games, we're doing Serious Work". So like a neglected grandpa weaving his own straw mat while his granddaughters make TikTok videos, I began making Flappy Millennial.

But why another Flappy Bird clone?​

Because I'm unoriginal.

But also because Flappy Bird has an engaging gameplay with a remarkably simple game mechanic. It's a child's play to write a clone and for some reason even the worst clones turn out to be fun. To prove the assertion I wrote yet another flappy bird clone right here. Turns out just 60 lines of code and about an hour or two is enough to make a fun flappy clone. Click the square below to focus the game.

Once you've played this to your heart's delight, check out the JavaScript it took to make it work:

<style>
  #c:focus {
    outline: dashed 4px #000;
  }
</style>
<link
  href="https://fonts.googleapis.com/css2?family=Poiret+One&display=swap"
  rel="stylesheet"
/>
<canvas id="c" tabindex="1"></canvas>
<script>
  let snd = new Audio(
    "data:audio/wav;base64,//uQRAAAAWMSLwUIYAAsYkXgoQwAEaYLWfkWgAI0wWs/ItAAAGDgYtAgAyN+QWaAAihwMWm4G8QQRDiMcCBcH3Cc+CDv/7xA4Tvh9Rz/y8QADBwMWgQAZG/ILNAARQ4GLTcDeIIIhxGOBAuD7hOfBB3/94gcJ3w+o5/5eIAIAAAVwWgQAVQ2ORaIQwEMAJiDg95G4nQL7mQVWI6GwRcfsZAcsKkJvxgxEjzFUgfHoSQ9Qq7KNwqHwuB13MA4a1q/DmBrHgPcmjiGoh//EwC5nGPEmS4RcfkVKOhJf+WOgoxJclFz3kgn//dBA+ya1GhurNn8zb//9NNutNuhz31f////9vt///z+IdAEAAAK4LQIAKobHItEIYCGAExBwe8jcToF9zIKrEdDYIuP2MgOWFSE34wYiR5iqQPj0JIeoVdlG4VD4XA67mAcNa1fhzA1jwHuTRxDUQ//iYBczjHiTJcIuPyKlHQkv/LHQUYkuSi57yQT//uggfZNajQ3Vmz+Zt//+mm3Wm3Q576v////+32///5/EOgAAADVghQAAAAA//uQZAUAB1WI0PZugAAAAAoQwAAAEk3nRd2qAAAAACiDgAAAAAAABCqEEQRLCgwpBGMlJkIz8jKhGvj4k6jzRnqasNKIeoh5gI7BJaC1A1AoNBjJgbyApVS4IDlZgDU5WUAxEKDNmmALHzZp0Fkz1FMTmGFl1FMEyodIavcCAUHDWrKAIA4aa2oCgILEBupZgHvAhEBcZ6joQBxS76AgccrFlczBvKLC0QI2cBoCFvfTDAo7eoOQInqDPBtvrDEZBNYN5xwNwxQRfw8ZQ5wQVLvO8OYU+mHvFLlDh05Mdg7BT6YrRPpCBznMB2r//xKJjyyOh+cImr2/4doscwD6neZjuZR4AgAABYAAAABy1xcdQtxYBYYZdifkUDgzzXaXn98Z0oi9ILU5mBjFANmRwlVJ3/6jYDAmxaiDG3/6xjQQCCKkRb/6kg/wW+kSJ5//rLobkLSiKmqP/0ikJuDaSaSf/6JiLYLEYnW/+kXg1WRVJL/9EmQ1YZIsv/6Qzwy5qk7/+tEU0nkls3/zIUMPKNX/6yZLf+kFgAfgGyLFAUwY//uQZAUABcd5UiNPVXAAAApAAAAAE0VZQKw9ISAAACgAAAAAVQIygIElVrFkBS+Jhi+EAuu+lKAkYUEIsmEAEoMeDmCETMvfSHTGkF5RWH7kz/ESHWPAq/kcCRhqBtMdokPdM7vil7RG98A2sc7zO6ZvTdM7pmOUAZTnJW+NXxqmd41dqJ6mLTXxrPpnV8avaIf5SvL7pndPvPpndJR9Kuu8fePvuiuhorgWjp7Mf/PRjxcFCPDkW31srioCExivv9lcwKEaHsf/7ow2Fl1T/9RkXgEhYElAoCLFtMArxwivDJJ+bR1HTKJdlEoTELCIqgEwVGSQ+hIm0NbK8WXcTEI0UPoa2NbG4y2K00JEWbZavJXkYaqo9CRHS55FcZTjKEk3NKoCYUnSQ0rWxrZbFKbKIhOKPZe1cJKzZSaQrIyULHDZmV5K4xySsDRKWOruanGtjLJXFEmwaIbDLX0hIPBUQPVFVkQkDoUNfSoDgQGKPekoxeGzA4DUvnn4bxzcZrtJyipKfPNy5w+9lnXwgqsiyHNeSVpemw4bWb9psYeq//uQZBoABQt4yMVxYAIAAAkQoAAAHvYpL5m6AAgAACXDAAAAD59jblTirQe9upFsmZbpMudy7Lz1X1DYsxOOSWpfPqNX2WqktK0DMvuGwlbNj44TleLPQ+Gsfb+GOWOKJoIrWb3cIMeeON6lz2umTqMXV8Mj30yWPpjoSa9ujK8SyeJP5y5mOW1D6hvLepeveEAEDo0mgCRClOEgANv3B9a6fikgUSu/DmAMATrGx7nng5p5iimPNZsfQLYB2sDLIkzRKZOHGAaUyDcpFBSLG9MCQALgAIgQs2YunOszLSAyQYPVC2YdGGeHD2dTdJk1pAHGAWDjnkcLKFymS3RQZTInzySoBwMG0QueC3gMsCEYxUqlrcxK6k1LQQcsmyYeQPdC2YfuGPASCBkcVMQQqpVJshui1tkXQJQV0OXGAZMXSOEEBRirXbVRQW7ugq7IM7rPWSZyDlM3IuNEkxzCOJ0ny2ThNkyRai1b6ev//3dzNGzNb//4uAvHT5sURcZCFcuKLhOFs8mLAAEAt4UWAAIABAAAAAB4qbHo0tIjVkUU//uQZAwABfSFz3ZqQAAAAAngwAAAE1HjMp2qAAAAACZDgAAAD5UkTE1UgZEUExqYynN1qZvqIOREEFmBcJQkwdxiFtw0qEOkGYfRDifBui9MQg4QAHAqWtAWHoCxu1Yf4VfWLPIM2mHDFsbQEVGwyqQoQcwnfHeIkNt9YnkiaS1oizycqJrx4KOQjahZxWbcZgztj2c49nKmkId44S71j0c8eV9yDK6uPRzx5X18eDvjvQ6yKo9ZSS6l//8elePK/Lf//IInrOF/FvDoADYAGBMGb7FtErm5MXMlmPAJQVgWta7Zx2go+8xJ0UiCb8LHHdftWyLJE0QIAIsI+UbXu67dZMjmgDGCGl1H+vpF4NSDckSIkk7Vd+sxEhBQMRU8j/12UIRhzSaUdQ+rQU5kGeFxm+hb1oh6pWWmv3uvmReDl0UnvtapVaIzo1jZbf/pD6ElLqSX+rUmOQNpJFa/r+sa4e/pBlAABoAAAAA3CUgShLdGIxsY7AUABPRrgCABdDuQ5GC7DqPQCgbbJUAoRSUj+NIEig0YfyWUho1VBBBA//uQZB4ABZx5zfMakeAAAAmwAAAAF5F3P0w9GtAAACfAAAAAwLhMDmAYWMgVEG1U0FIGCBgXBXAtfMH10000EEEEEECUBYln03TTTdNBDZopopYvrTTdNa325mImNg3TTPV9q3pmY0xoO6bv3r00y+IDGid/9aaaZTGMuj9mpu9Mpio1dXrr5HERTZSmqU36A3CumzN/9Robv/Xx4v9ijkSRSNLQhAWumap82WRSBUqXStV/YcS+XVLnSS+WLDroqArFkMEsAS+eWmrUzrO0oEmE40RlMZ5+ODIkAyKAGUwZ3mVKmcamcJnMW26MRPgUw6j+LkhyHGVGYjSUUKNpuJUQoOIAyDvEyG8S5yfK6dhZc0Tx1KI/gviKL6qvvFs1+bWtaz58uUNnryq6kt5RzOCkPWlVqVX2a/EEBUdU1KrXLf40GoiiFXK///qpoiDXrOgqDR38JB0bw7SoL+ZB9o1RCkQjQ2CBYZKd/+VJxZRRZlqSkKiws0WFxUyCwsKiMy7hUVFhIaCrNQsKkTIsLivwKKigsj8XYlwt/WKi2N4d//uQRCSAAjURNIHpMZBGYiaQPSYyAAABLAAAAAAAACWAAAAApUF/Mg+0aohSIRobBAsMlO//Kk4soosy1JSFRYWaLC4qZBYWFRGZdwqKiwkNBVmoWFSJkWFxX4FFRQWR+LsS4W/rFRb/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////VEFHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAU291bmRib3kuZGUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMjAwNGh0dHA6Ly93d3cuc291bmRib3kuZGUAAAAAAAAAACU="
  );
  let pnt = (r = 0.2) => {
    snd.playbackRate = r;
    snd.play();
  };
  let C = document.querySelector("#c");
  let W = (C.width = H = C.height = 350);
  let CC = C.getContext("2d");
  colH = S = P = 1;
  let pipe1, pipe2;
  {
    let rH = () => (Math.random() * 100 + 50) | 0;
    let pipe = (off) => {
      let x = off,
        h = rH();
      return (rst) => {
        if (rst) {
          x = off;
          colH = 0;
          return;
        }
        if ((x = x - 2) < -50) {
          x = W;
          h = rH();
        }
        x > 70 && x < 100 ? (colH = h) : 0;
        if (x == 20) {
          colH = 0;
          ++S;
          pnt(1);
        }
        CC.fillRect(x, 0, 50, h);
        CC.fillRect(x, H, 50, h - H + 140);
      };
    };
    pipe1 = pipe(W + 100);
    pipe2 = pipe(W + 300);
  }
  let y = (h = w = 30);
  (window.vy = 0), (ay = 0.4);
  let LP = (window.onload = () => {
    CC.fillStyle = "#235789";
    CC.fillRect(0, 0, W, H);
    if (P) {
      pt("Press any key to play", H / 2);
      pt("Score: " + S, H / 2 + 40);
      pipe1(1);
      pipe2(1);
      return;
    }
    CC.fillStyle = "#f1d302";
    y += vy += ay;
    CC.fillRect(70, y, w, h);
    if (colH && (y < colH || y + h > colH + 140)) {
      y = 30;
      P = 1;
      pnt();
    }
    y > H + h ? (P = 1) & (vy = 0) & (y = 30) & pnt() : 0;
    pipe1();
    pipe2();
    pt(S, 40);
    requestAnimationFrame(LP);
  });
  CC.font = "24px 'Poiret One'";
  var pt = (t, h) => {
    CC.fillStyle = "#fff";
    CC.fillText(t, (W - CC.measureText(t).width) / 2, h);
  };
  c.ontouchstart = c.onkeydown = (e) =>
    P ? (P = 0) & (S = 1) & LP() : (vy = -8);
</script>

That's about 60 lines of javascript which makes a playable Flappy Bird game, complete with touch screen support and sound effects.

Now, that should have been the end of it. A game in 60 lines of code is good enough. But something in me was begging to take it even further. And as if to tempt me yet more, the power went out and with it the WiFi. What is a young man in this lockdown to do without the umbilical cord of the Internet forever pumping numbness into his thoughts?

So then I went ahead and made it even smaller.

<canvas id="c"/><script>C=document.querySelector('#c');W=C.width=H=C.height=350
colH=S=P=1;y=h=w=30;vy=0,ay=.4;for($ in CC=C.getContext('2d'))CC[$[0]+($[6]||'')]
=CC[$];rH=_=>Math.random()*99+50;pp=(off)=>{ let x=off,ph=rH();return r=>{r?(x=off)
&(colH=0):0;(x=x-2)<-50?(x=W)&(ph=rH()|0):0;(x>70&&x<100)?colH=ph:0;
x==20?(colH=0)&++S&B(400):0;CC.fc(x,0,50,ph);CC.fc(x,H,50,ph-H+140);}}
p1=pp(W+100);p2=pp(W+300);LP=onload=_=> {CC.fillStyle="#258";CC.fc(0,0,W,H)
if(P){pt("Score "+S,H/2);p1(1);p2(1);return};CC.fillStyle="#fd0"
y+=vy+=ay;CC.fc(70,y,w,h);colH&&(y<colH||y+h>colH+140)?(y=30)&
(P=1)&B():0;y>H+h?(P=1)&(vy=0)&(y=30)&(B()):0;p1();p2();pt(S,40);
requestAnimationFrame(LP)};CC.font="24px serif"
pt=(t,h)=>{CC.fillStyle="#fff";CC.fx(t,(W-CC.me(t).width)/2,h)}
c.ontouchstart=onkeydown=(e)=>P?(P=0)&(S=1)&LP():vy=-8;
A=new AudioContext;B=_=>{o=A.createOscillator();o.frequency.value=_||999
o.start();o.connect(A.destination);setTimeout(_=>o.stop(),99)}</script>

Yeah, that's right. It is 15 lines of handwritten HTML and javascript code that will make a flappy bird game in your browser. Pretty cool, huh? But line count is not a valid metric for this code. Here's a proper one: this code is less than 970 bytes. That's about a 100 times smaller than the code size of flappy millennial. And it still works! Of course, at that point I was left wondering why my actual Flappy Millennial code is so huge.

How?​

The main game logic is in the js/game.js file. It begins by importing a bunch of classes and functions from the library, including Character, Spritesheet, Collider, and StaticText. These classes abstract the mechanism of animating on the canvas and checking for collisions among other things. These things are very helpful because the canvas API, while flexible, is very low level. For example, if you wanted to play an animation using the bare canvas API, you'd have to convert your GIF into a sequence of images and manually draw each image while also taking care of the timing and positioning. Things like that don't scale well as the game gets bigger. With my library, you can simply say Character.runAnimation() and the rest is taken care of for you. The game is gradually implemented using these primitives and very simple javascript. A casual skim of the code in js/game.js on GitHub should be enough to get most implementation details.

I made most of the art for the game myself, including our titular character Flappy Millennial. It was made using Piskel which is a great pixel art animation software. It isn't as feature-rich as I'd like but works well right in the browser and I love it for that.

This is the animation for our Flappy Millennial. I labored over it for a couple of hours and I feel like it's turned out okay. But if I had to start again, I'd definitely make the wings not look so stiff and cardboard-like. The beautiful background art that I'm using was made by Vicente Nitti.

Build system

I've acquired the habit of writing ES6 javaScript by default because I do so much work with React, Vue and NodeJS. That's generally a good thing, but it also means that if I want my javascript code to work on most browsers, I'll have to use external tooling. That was the case with Flappy Millennial.

So I had to put together a set of tools to convert my modern javascript dialect to standard javascript understood by most browsers. Without this process, my code wouldn't work on current mobile browsers and slightly outdated or extended support editions of desktop browsers. It isn't a particularly difficult thing to do, because these tool makers have already put in all the effort.

First we pass our javascript through Babel with plugins that convert to ES5 javascript. Then output is then passed to Browserify, which reads all the imports and requires, and puts all the code together in a single, huge file. That file is finally passed to Google closure compiler, which minifies the code and in our case, creates a file that's half the size of the unminified code but with the exact behavior.

I could have used something like Gulp to create this pipeline, but this time it isn't too complicated so a simple bash script did the trick.

#! /bin/bash
echo "Rebuilding Game"
rm -rf dist
mkdir -p dist/js
cp index_dist.html dist/index.html
cp favicon.ico dist/favicon.ico
echo "Copying Assets"
cp -r assets dist/
echo "Transpiling with Babel"
npx babel js/library -d dist/js/library
npx babel js/game.js -o dist/js/game.js
npx babel js/LOGSCORE.js -o dist/js/LOGSCORE.js
echo "Browserifying"
npx browserify dist/js/game.js -o dist/js/game_browserify.js
echo "Cleaning up"
rm -rf dist/js/library
echo "Minifying"
npx google-closure-compiler --js=dist/js/game_browserify.js --js_output_file=dist/js/game.js --jscomp_off=checkVars
rm dist/js/LOGSCORE.js
# rm dist/js/game_browserify.js
echo "Done! (ignore the warnings)"

Azure and Analytics

One of the tangential goals of this project was to set up a simple analytics system that logs what score you make and when you make it. With that data I can look for interesting patterns like how many times the average player plays the game before rage quitting, or the global high score, or if players come back to play a second time.

Side note: You don't have to worry about privacy. The data I collect is totally anonymous and ridiculously harmless. I'll eventually publish interesting findings from the data once I have enough of it.

So anyway, to make that system, I chose to use Azure Functions. Azure Function (and its equivalent from other cloud providers) is an awesome service that makes backend development, deployment and scaling so much easier. I've fallen in love with it and all the possibilities it represents.

A cloud function is essentially some code that triggers when some event happens (say a HTTP request, or a database modification), does some processing, and creates some output (a HTTP response, a database entry modification). In my case, every time the player dies, his score is sent to a cloud function that does some verification and records the score in a CosmosDB database. All in all, it took about 10 lines of code and some basic configuration. To do the same thing with a virtual machine, I'd have to instantiate a new VM and a new disk to go with it, configure the firewall, install all the software, scp local files to the VM, npm install and wait, provision https certificates, and just so much more.

But was it a smooth ride? Nope.

I actually have a lot of bad things to say about Azure's user experience and I'm very tempted to say them. In fact there are a bunch of paragraphs just sitting here in the drafts waiting to be unleashed. But they'll have to wait a lot longer. I need to explore Azure a lot more to either articulate my opinions properly or to be proven wrong.

First few hours​

This is the data collected in the first few hours after I published the game on GitHub. So far the highscore is 27. Think you can do better? Go play the game at nirav.com.np/flappy-millennial and prove it. I won't know who you are, but I'll know you're a badass.

I've been working on and off on a Nepali programming language with my friends for the last few months. It's called मनसा (IAST: mansā) and I think it's ready for an alpha release. If you'd like to try the language out, visit mnsa.cc - the official website. You can play around with the language right in the browser without having to download anything, not even a Devanagari keyboard layout.

This post is a collection of random things I want to say about the language, including how the idea came about, the interesting things I learnt making the project, and the problems faced.

How it all started​

In the sixth semester, we are required to submit a team project by the end of the term. Most teams in my class were making boring things, like hotel management systems, or polling websites. If you get to chose your own project, why not do something exciting and new? We settled on making a programming language. "Why not in Nepali?", I said, and after a long debate, we settled on building a Nepali programming language.

In the beginning, we thought a Nepali programming language wouldn't have any practical use at all. We thought that all this project would ever be is a novelty, a fun thing to play with and forget about, something like Brainfuck or Befunge. But as we continued to look into things to include in our proposal, we realized that a Nepali programming language might have a few very niche but very practical uses.

After some more deliberation, we decided that it had to be a compiled language, because that was the most difficult to make in terms of code, and the "coolest". It was exciting and somewhat uncertain because we don't have Compiler Construction classes in our syllabus, so we had basically zero idea on how we'd proceed. And we didn't know how to work with Nepali characters in the program. It was a great learning experience.

The language​

The language is actually very tentative at this point and we're still refining it. You should be able to get the general idea of the syntax by the small example code below. If you don't, wait for the official documentation to be finished. It's in the works.

We have made a few changes to the conventional structure of a program statement to make it more expressive. For example, in function calls, the name of the function comes after the parameters, as verb comes at the end of sentence in Nepali. So "print Something" becomes "something लेख". We similarly modified other constructs such as conditionals and loops to fit into the Nepali grammar structure more. Technically, it is a very small change but it really does help make the programs more readable in Nepali. It was surprising how fluid the language becomes by changes so simple.

The compiler​

The compiler was written in C++ completely from scratch. I didn't want to use lexer or parser generators because these tools hide the interesting and complicated details. That is a good thing if your goal is to make good, maintainable compilers but, as a student, I wanted to explore all the gory details and learn everything I could. I should mention here that the series of lectures by Alex Aiken is an amazing resource, and if you're planning on doing compilers too, you can at least skim through the videos to see the big picture.

I used Visual Studio Code to write C++ code this time. I actually started coding using plain old vim as always, but eventually it became so unmanageable with files all over the place that I had to find a more manageable tool. I went with vscode, and I'm glad I did because vscode is incredibly well made. And the c/c++ extension is so good. It apparently makes an AST for my code on the fly and checks for all kinds of errors. I had never before used these sophisticated tools for writing C++. I generally used vanilla sublime with vim extention, or just command line vim. But really it's so nice. It's like discovering geysers for the first time when you've been showering with cold water all your life (strangely specific metaphor? sorry).

When writing the compiler, I used many new C++17 features. I was drawn into modern C++ first because of constexpr which allows you to make functions that are evaluated at compile time. I wanted to build a fast but manageable lexical analyser using a composition of constexpr functions but I realised that I was trying to prematurely optimize again, so I controlled myself. Maybe in version 0.2.

I also used C++17 lambda functions to declare nested functions to eliminate repeated actions that are only relevant inside a specific function. Here's an example:

auto makeDeclNode = [&](ast::astType a) {
   stmt.reset(new ast::Ast);
   astNode declaration;
   if (declaration = declList()) {
      stmt->type = a;
      stmt->left = std::move(declaration);
   } else
      errQuit("declarations");
};

if (accept(lexer::tokenType::_int)) {
   makeDeclNode(ast::astType::_intDecl);
} else if (accept(lexer::tokenType::_str)) {
   makeDeclNode(ast::astType::_strDecl);
} else if (accept(lexer::tokenType::_bool)) {
   makeDeclNode(ast::astType::_boolDecl);
}

Maybe it's because I've coding a lot in javascript these days, but nested functions feel like an important language feature to have. मनसा has nested function support too.

I also tried to use C++ smart pointers everywhere. They make memory management so pleasant and take away most of the issues I used to deal when using raw pointers. But I did find the lack of good observer pointer idiom annoying and dangerous. Maybe they'll make it better by C++20, but hopefully I'll already be using Rust by then.

In v0.1, I have only implemented the lexer, parser, semantic analyzer and a rudimentary code generator with emits C++ code. In the new versions, I will progressively add new modules like optimizer, and redo some old ones.

1. The Lexical Analyser is a simple finite state machine which iterates over each each UTF8 encoded character and converts them into tokens. I learnt a lot about Unicode while making the lexer. I am interested in world languages and scripts, so reading the Unicode documentation and other articles was fascinating to me. If you're interested in languages too, I cannot recommend at least skimming through the Unicode Standard enough. If you're in love with writing scripts like I am, do read this article on Smashing Magazine. I have written about Unicode in another blog post so maybe also check that out.

   The encoding of Unicode characters is also a fascinating topic. I find UTF-8 particularly beautiful. And as it turns out, it was designed by Ken and Rob from Bell Labs. I'm a big fan of the Bell Labs people.

2. I wrote the Parser using the Recursive Descent algorithm. It's crazy how simple yet powerful Recursive Descent is. I learnt it using only the Wikipedia article, wrote my first parser in a weekend, and it works like magic. It's just so elegant. In the current code base, the parser takes up the most volume at about a thousand lines. But I do believe that I should have mixed in some Pratt parsing to parse the operators, because recursive descent has to make a lot of function calls even for trivial tasks, which makes it inefficient. Oh well, maybe next time.

   The Parser makes an AST. Because the AST is vital, I wrote some routines to generate actual pictures from the AST. I've detailed the process in another blog post but this is what the AST's visualization looks like:

   

3. The Semantic Analyser is all about recursively getting to all nodes in the AST and checking their types and making sure everything is according to rules.

4. I had to write my own Symbol Table, which was awesome because I got to play with STL containers like pairs and unordered maps. Once you discover these tools, you can never go back to hand-coding data structures (which I'll admit was stupid, but I'm slowly fighting my not-invented-here syndrome).

5. The actual Code Generator is not done yet. In it's place is a simple function which recursively navigates the AST and generates C++ code. For variable names, I just base64 encoded all Devanagari identifiers and replaced the illegal characters in base64 with underscore. The generated code looks like this:

int main(){
   auto v4KSo4KSv4KS_pX_pCksuCkvuCkh_pCkqATT1 = [&]() {
      cout << "\n";
   };
   int v4KSG4KSH = 100;
   string v4KSV4KWB4KSw4KS_p = "नमस्ते";
   while  (v4KSG4KSH > 0) {
      cout << v4KSV4KWB4KSw4KS_p;
      v4KSo4KSv4KS_pX_pCksuCkvuCkh_pCkqATT1();
      v4KSG4KSH= (v4KSG4KSH - 10) ;
   };
}

The Website​

Making the website was like a pleasant stroll in the morning sun. It was not particularly challenging. I used Adobe XD to make a prototype and then designed it with vanilla CSS. I decided that unless there's an online compiler a website for a programming language is no use. I also fell in love with Haskell's online shell. But making it was going to be difficult.

I used vanilla javascript, socket.io library and Codemirror to make the compiler part. Learning to write the syntax highlighting was somewhat complicated so I just modified a preexisting one. I wanted anybody with a computer or a mobile phone to be able to go in, type code and try it out. So I also made an in-browser keyboard layout switcher which reads input keys and converts them to Nepali based on this keyboard layout, which we also made. I have yet to make the keyboard work on the phone, but I'm so busy these days that I don't yet have the time.

Then I made a nice keyboard layout switching drawer that I love to play with.

I apologize for the low quality GIF. I had to run the screen recording through so many online converters that all compressed the original in some form or another that now it looks like it came straight from early 2000s.

The backend part is also interesting, but it still needs lots of work. I basically instantiated a linux server on Azure and ran a simple NodeJS program on it which listens to socket.io requests for programs in मनसा, compiles the program, runs it in a sandbox, and pipes the input and output to the web terminal. I have put some primitive rate limiting and protections in the code but at some point I intend to use recaptcha or something to that effect to guard the API. I might also look to use cloud functions instead of running a full linux box all the time. I wrote a blog post documenting a part of the process here.

The IDE​

I've hated ElectronJS with a passion I only reserve for a few things in life (like the Cursed Child) The hate is a result of a few factors. I use a 6 year old Dell laptop as my computer which is so dilapidated at this point that it stutters when I have lots of chrome tabs and VScode running at the same time. So I appreciate parsimony from the developer's side. I'm also kind of annoyed at this developer-first culture we developers are promoting, piggybacking simply on the fact that computers are getting faster and with more memory than before.

But then the night before a major demonstration of the project I was in such a rush because I had so many things to make and refine, that I broke all my rules and made a deal with the devil itself. Oh mighty devil, oh accomplice of the photon, oh sinful chimera of chromium and nodejs, oh gods of chaos and asynchronicity, I implore you.

Anyway, the results were astounding. In about three hours, I had a perfectly working syntax-highlighting enabled (thanks again, CodeMirror), beautiful IDE for demonstration. It could compile and run the code and upload it to Arduino too (with some avrdude magic, of course). ElectronJS is incredibly fast and simple (for the developer, at least). Even though the reasons for hating ElectronJS are still valid, I got a new perspective by actually working on it, and I've added it into my toolbox. Thank you, ElectronJS, for taking me back to my childhood days of Visual Basic and RAD.

The Arduino interface​

मनसा can also write code for Arduino. The idea came when I was trying to teach my cousin (who's 12) this programming language. I realized that it's difficult to get a kid to relate to the world inside the computer. For example, when teaching the concept of looping for the first time, it's way easier to show him a led blinking 10 times and explain the code than to show him ten instances of 'Hello World'.

So I added in some features to the language, mainly the external mechanism which allows you to write C code inside मनसा functions. The resulting system is kind of hacky but surprisingly flexible. The embedded C code does most of the register twiddling and bit flipping, and exposes high level functions to मनसा. That code is written as library. Then, using the import mechanism of the language, you can import these functions and access them like normal मनसा functions. I will put in a video demonstration when I have the time.

Random, interesting observations​

1. It's nice to have a small code diary, if you will, to jot down all complex ideas that you think up while coding. I made a diary.md file on the root of the source tree to keep a chronological log of my thoughts, snippets of codes, things I had to do next, of bugs and other things in the file.
2. C++11 range based loops are awesome. I know about these before.
3. Makefiles get very sloppy very fast and annoying to maintain. I must learn to use more sophisticated build system soon. Maybe Cmake? The build system should be able to build on both Linux and on WSL, install all required programs if they don't exist, and only compile the libraries that actually changed.
4. Most of the 3000+ lines of code I've written is worthless. I read somewhere lines of code are not investments, rather they are loans. The more you add to the codebase, the more expensive you make it to maintain it. It takes more time for a prospective contributor to make sense of the whole project. The more lines you write, the more bugs you hide. I would have been far far better using a parser generator and making the language interpreted instead of compiled. That would reduce complexity of the system and let me focus on the big picture and on the applications of the language.
5. At some point, you really need a second, vertical monitor.
6. Compiling larger code bases takes far more time than you expect.

Data structures help structure and organize data effectively, and provide several abstracted operations on the data. They are elegant and convenient, and computer scientists love to use them. Implementing these data structures in the computer require the programmer to flatten the structure into a one-dimensional model using pointers or references because the memory is actually arranged as a one-dimensional run of data elements.

It is often necessary to inspect the data structure in debugging and program verification phases. But some data structures happen to be particularly graphical, in that they have multiple associations and elaborate hierarchies or layers. Think of a Rope or a Graph. Due to many associations between nodes, it's not possible to properly display the graph on the terminal.

I recently ran into such problem, and found an interesting solution. In this post, I will try to document it. I've written mainly about the Abstract Syntax Tree in this post, but it should be trivial to translate it to any other deterministically traversable data structure.

I'm currently working on writing a compiler for my own Nepali programming language. The front-end part of a compiler converts the program code into a data structure called AST. AST, or Abstract Syntax Tree, is a tree-based representation of the source code. It's much more suitable to further transformations in the semantic analysis, optimization and code generation phases.

This picture of Abstract Syntax Tree taken from Wikipedia page on AST

But AST is also an incredibly complex data structure. It has to store a bunch of information and maintain a tree structure. It is not immediately obvious how one might present it on the terminal screen. Several ideas to exist, but they don't scale well. Two solutions that I thought up were:

1. Print in the terminal
2. Unparse the AST back to source code.

Unparse the AST back to source code​

The AST is a lossless transformation of the source code as far as intent is concerned. So it should be possible to go the other direction and retrieve the source code back from the AST. If the AST doesn't have any errors, then it's unparsing should result in code that's nearly identical to the original source code. This step is also essential when trying to verify compiler correctness.

I'm going to have to explore this option further in the future. But because this method doesn't allow me to directly observe the AST at several stages of the compilation process, I didn't use this.

Print in the terminal​

I had written a compiler for my minor project in my 6^th semester but the language it accepted was very basic. So, back then, I got away with simply printing the tree in the terminal. For example, this code

generates this AST:

The AST is printed in the terminal rotated 90° counter-clockwise such that the root of the tree is at the bottom-left. This was served me well at the time. Frankly, for a language that didn't even have an else if clause, there's are limited ways to complicate the AST other than by writing longer code.

But the language I'm working on right now is extensive. And so the tree it creates is immense. I had to come up with some other way to analyse the AST.

Enter Graphviz​

Brian Kernighan has given an amazing talk on successful language design, where he also talked about Pic, a language he designed for specifying graphs. I was immediately taken by the idea because this was a much better way to make graphs than the clumsy touchpad.

Well, Brian's Pic is antiquated now, but Graphviz does the same kind of thing. Graphviz uses the DOT graph specification language. DOT is extremely simple. It literally takes 5 minutes to be able to make nearly any graph you want. For example, consider the DOT code below

digraph any_other_name{
    Brahma->Ram
    Brahma->Hari
    Ram->Tom
    Hari->Dick
    Hari->Harry
}

It produces the Graph:

If you're interested, you should try some online Graphviz software to get the hang of it. I used this one but there are a lot of choices one web search away.

I can only imagine how many hours this thing could have saved me the previous semester when I was stuck making elaborate graphs for my minor project report and presentation.

Generating the DOT​

Generating the DOT for Graphviz is generally trivial. For linked list, a recursive function such as the one below suffices (for c++):

void printElement(listElement* el, int nodeNumber){
  string DOT;
  DOT = "n" + to_string(nodeNumber) + " [label=";
  DOT+= el->name+"\"]\n";

  if (el->right){
    DOT += "n" + to_string(nodeNumber) + "->";
    DOT += "n" + to_string(nodeNumber+1) + "\n";
    printElement(el->right, nodeNumber+1);
  }
  cout << DOT;
}

It is a little more complex for the AST. Because trees are recursive structures, it's easy to write elegant recursive functions that process the trees. In this case, the genDOT(astNode,bool) function helps recursively descend the AST tree, and print out it's nodes.

int rootNum = 0;
void Parser::genDOT() {
  cout << "digraph G{\n";
  cout << "graph[fontname=Rajdhani, color="#242038"]\n";
  cout << "node[fontname=Rajdhani, color="#242038", shape=square]\n";
  cout << "edge[fontname=Rajdhani, color="#242038"]\n";
  genDOT(root, false);
  cout << "}\n";
}

void Parser::genDOT(astNode &n, bool isLeft) {
  std::string print;

  // DOT for current node
  print = "n" + to_string(rootNum) + " [label=< <B>" +   translateTypeStr(n->type) + "</B>";
  if (translateDataStr(n) != "")
    print += "<BR/>" + translateDataStr(n) + " >";
  else
    print += " >";
  if (isLeft)
    print += ", color="#FF595E", fontcolor="#FF595E"";
  else
    print += ", color="#274690", fontcolor="#274690"";

  if (n->type == ast::astType::numericConstant ||
      n->type == ast::astType::stringConstant ||
      n->type == ast::astType::boolConstant) {
    print += ", shape=egg";
  }
  print += "]\n";
  int rootNumBackup = rootNum;
  rootNum++;
  if (n->type == ast::astType::_if) {
    if (n->cond) {
      print += "n" + to_string(rootNumBackup) + "->n" +
        to_string(rootNum) + " [label=condition]\n";
      genDOT(n->cond);
    }
    rootNum += 1;
    if (n->_if) {
      print += "n" + to_string(rootNumBackup) + "->n" +
        to_string(rootNum) + " [label=if]\n";
      genDOT(n->_if);
    }
    rootNum += 1;
    if (n->_elseIf) {
      print += "n" + to_string(rootNumBackup) + "->n" +
        to_string(rootNum) + " [label=elseif]\n";
      genDOT(n->_elseIf);
    }
    rootNum += 1;
    if (n->_else) {
      print += "n" + to_string(rootNumBackup) + "->n" +
        to_string(rootNum) + " [label=else]\n";
      genDOT(n->_else);
    }
    rootNum += 1;
  } else if (n->type == ast::astType::_while) {
    if (n->cond) {
      print += "n" + to_string(rootNumBackup) + "->n" +
        to_string(rootNum) + " [label=condition]\n";
      genDOT(n->cond);
    }
    rootNum += 1;
    if (n->_if) {
      print += "n" + to_string(rootNumBackup) + "->n" +
        to_string(rootNum) + " [label=do]\n";
      genDOT(n->_if);
    }
    rootNum += 1;
  } else {
    if (n->left || n->right) {
      if (n->left) {
        print += "n" + to_string(rootNumBackup) + "->n" +
          to_string(rootNum) + " [color="#FF595E"]\n";
        genDOT(n->left);
      } else {
        print += "n" + to_string(rootNum) + " [label=x]\n";
        print += "n" + to_string(rootNumBackup) + "->n" +
          to_string(rootNum) + "\n";
      }
      rootNum += 1;
      if (n->right) {
        print += "n" + to_string(rootNumBackup) + "->n" +
          to_string(rootNum) + " [color="#274690"]\n";
        genDOT(n->right, false);
      } else {
        print += "n" + to_string(rootNum) + " [label=x]\n";
        print += "n" + to_string(rootNumBackup) + "->n" +
          to_string(rootNum) + "\n";
      }
    }
  }
  cout << print << endl;
}

Of course, this is not the complete code. It only covers if and while loops, but not function bodies and other blocks. But this is the essence. Funnily enough, I spent less time writing code to actually generate the graph than I spent trying to make it look cooler. But given how much time I'm going to have to stare at and navigate these graphs, I think the it was worth the effort.

The rootNum keeps track of the parent node at each recursion. The parent node needs to point to all it's the children, and rootNum helps keep track of the that. The genDOT() function is simply the public interface which first prints out the necessary DOT boilerplate and then calls the main genDOT(astNode,bool) function. The genDOT(astnode,bool) function recursively prints out all the DOT code. Here's the kind of DOT code it produces:

digraph G{
    graph[fontname=Rajdhani, color="#242038"]
    node[fontname=Rajdhani, color="#242038", shape=square]
    edge[fontname=Rajdhani, color="#242038"]
    n3 [label=< <B>Declare Var</B><BR/>नाम >, color="#FF595E", fontcolor="#FF595E"]
    n2 [label=< <B>chain</B> >, color="#FF595E", fontcolor="#FF595E"]
    n2->n3 [color="#FF595E"]
    n5 [label=x]
    n2->n5
    n1 [label=< <B>Declare String </B> >, color="#FF595E", fontcolor="#FF595E"]
    n1->n2 [color="#FF595E"]
    n6 [label=x]
    n1->n6
    n10 [label=< <B>dataEl</B><BR/>नाम >, color="#FF595E", fontcolor="#FF595E"]
    n12 [label=< <B>string</B><BR/>नीरब >, color="#274690", fontcolor="#274690", shape=egg]
    n9 [label=< <B>=</B> >, color="#FF595E", fontcolor="#FF595E"]
    n9->n10 [color="#FF595E"]
    n9->n12 [color="#274690"]
    n17 [label=< <B>string</B><BR/>नमस्ते मालिक >, color="#FF595E", fontcolor="#FF595E", shape=egg]
    n16 [label=< <B>chain</B> >, color="#FF595E", fontcolor="#FF595E"]
    n16->n17 [color="#FF595E"]
    n19 [label=x]
    n16->n19
    n15 [label=< <B>FunctionCall</B><BR/>लेख >, color="#FF595E", fontcolor="#FF595E"]
    n15->n16 [color="#FF595E"]
    n20 [label=x]
    n15->n20
    n14 [label=< <B>Statement<BR/>3</B> >, color="#FF595E", fontcolor="#FF595E"]
    n14->n15 [color="#FF595E"]
    n21 [label=x]
    n14->n21
    n8 [label=< <B>If</B> >, color="#FF595E", fontcolor="#FF595E"]
    n8->n9 [label=condition]
    n8->n14 [label=if]
    n7 [label=< <B>Statement<BR/>2</B> >, color="#274690", fontcolor="#274690"]
    n7->n8 [color="#FF595E"]
    n25 [label=x]
    n7->n25
    n0 [label=< <B>Statement<BR/>1</B> >, color="#274690", fontcolor="#274690"]
    n0->n1 [color="#FF595E"]
    n0->n7 [color="#274690"]
}

After the code was working, I wrote a Makefile that saves the output, generates a PNG image of the graph and opens that image in the feh image viewer.

test_ast: parser/parser.C parser/parser.h tests/parser/parser.C lexer/lexer.C lexer/token.C lexer/characters.C
    $(CC) $(CFLAGS) parser/parser.C parser/ast.C parser/helper.C tests/parser/parser.C lexer/characters.C lexer/token.C lexer/lexer.C file_handler/file_handler.C -o tests/parser/parser.out
    @echo "Starting Parser Test"
    @echo
    @tests/parser/parser.out > .graph
    dot -Tpng < .graph > .png
    feh .png
    @echo

This makefile simply redirects the DOT output from STDOUT to a file, and uses the graphviz program dot to generate a image that can be opened with feh.

Results​

After the set up, the whole thing just fit into my workflow.

Here's an example graph generated from the makefile.

So the next time you're trying to debug or inspect a complex data structure or a tangle of pointers, try taking some time out to write code which generates DOT so that you can inspect the graph. I tell you, a picture is worth a thousand words. The AST below is worth much more:

In the classroom, the teacher drones on and on in his maroon colored t-shirt that doesn’t suit him. His empty vibrations ripple through out the room, bouncing off the glass windows shut tight against the summer breeze, mixing with the clickety-clack of the ceiling fan to create a subtle dissonance, and stretching indefinitely and interminably in time and in space.

The only consolation, the only medicine is a certain girl who is facing a certain degrees away from that maroon-T-shirt-wearing teacher; looking into the window-pane with a certain temperament; certainly thinking of beautiful thoughts about who knows what. Yes, that girl between two other girls in the second bench. We will call her Sabina because she is also the unintentional witch who ensnared my lonely heart with no intention of ever doing so, not knowing that that intentlessness was the strongest incantation.

That girl, with her pretty red cap and her long neck, has set up an irrevocable residence in the little part of my heart where I plant roses and other flowers. It is of little consequence that she probably keeps me in a humdrum study table of her heart in a small unimportant box where she keeps her useless pencils-that is if she keeps me at all. When I dream of her, I find myself in an incomprehensible universe where I am without ground or sky, the angry stars murmur and roar; where she is the absolute and unequivocal queen of the cosmos such that when she walks the space ripples in her feet and inside her incomprehensible home is the absolute pool of swirling rainbows and sweet water where she lounges with her grandmother. Where she fills me up with the eternal promise of fragrant flowers of all colors and emotions. It is of little consequence that most likely she doesn’t even dream of me.

Yes her. We will daydream about her today. We will first pretend that we are secret artists and that we possess all the curves that exist in the universe. We will then pretend to draw her profile with the expert application of those curves which flow gracefully out of the pencil oddly sharpened with a knife and tucked in our ears when not in use, while in reality we are just absentmindedly writing her name and circling it repeatedly with a dazed, upward gaze. We will, in our reverie, first draw her nose. The small inward curve beneath the brow-ridge, followed by a beautiful nose - made with two curves: the first subtly bumps upward and falls, then the next smoothly turns from the tip to the philtrum. Then we draw the lips gently, as though we are carefully tracing it’s outlines without daring touch the flesh. We then continue on to draw her chin, then with one fluid line we conjure her jaw, then her small, pretty ear. We continue on in this fashion, drawing her hair, and her shirt’s collar and what not. The result is a beautiful picture of a beautiful girl, which is, naturally, to be expected seeing how we are secret artists who possess all the curves in the universe.

After the picture is drawn, we will imagine tearing it off our notebook and keep it inside our shirt right next to our beating heart that pumps red hot blood through our body and skips a beat when it hears her name.

Now, because this is a daydream, and we are the sole masters of the events that unfold in this contrived sphere, we will engineer the circumstances so that in the recess, as we enter the classroom engrossed in our secret artistic thoughts, she happens to be walking out perhaps agitated by a similarly beautiful preoccupation, such that the scene resolves itself in a mild collision, a whiff of the intertwining fragrances of her shampoo, of her hair, and of her skin; a little bit of that summer breeze; and my own unwittingly committed blunder that allows that little piece of paper to find a way away from my heart and out of the shirt, where it traces a flourish in that thin summer breeze to land right next to her feet.

By pure chance she looks down and notices the prettiest curves of the universe which we had employed to trace the profile of the prettiest girl in the universe. She realizes the lines, plucked from a rose for her ears, borrowed from a butterfly for her nose, and taken from the celestial darkness with an astral sprinkling for her eyes - could only have been drawn by someone in love. But because we are ourselves secret artists and because we like for it to stay that way, we have no choice but to grab her by the wrist, and guide her through between the various bodies and obstacles to a private spot in the staircase which leads to the terrace but which is currently blocked, thus forming a sort of staircasey cul-de-sac. There we look into her radiant eyes and say that we like her very much and so on and so forth until she looks right into our eyes too and smiles that bemused smile that says she has understood everything that hasn’t yet been said and says something back but we can’t hear it because now the maroon-T-shirt-wearing teacher has stopped producing those turbulent vibrations to focus his ugly energy into spotting the inattentive students and remarking on their inattentiveness. With our head back in the classroom, we realize that the notebook page is covered with various instances of her name in various sizes and with various flourishes, so we must first take care of that.

Then we borrow a book from the guy behind us and try to make sense of whatever the hell is being taught.

The Nepali language gets very little representation on the internet. Take, for example, the Nepali Wikipedia which has about 33 thousand articles. The Esperanto Wikipedia boasts 8 times that number (at around two hundred thousand articles), which is kind of sad, because Esperanto is an artificial language created by one person in the 19th century. It is spoken by a meager 2 million people worldwide. Compare this to the Nepali language, which has more than 25 million speakers.

It's as they say, “whoever controls the media controls the mind”. And the media of the 21st century, the internet, is so desperately out of Nepali's hands that, not only do we read the news, articles, the weather, and novels in English, but we go on to caption our photos of warm, intimate moments in English. Our pop culture is stuffed full of references to Hindi media. My little brother, who is not even 10, will sniff my phone from a mile away and hide out in some corner to watch Motu Patlu on YouTube. He might not be as fluent in Hindi as he is in Nepali. But in his little world, Hindi is the language that superheroes speak. And Nepali? Only his annoying brother.

I'm not saying that I'm any better. If anything, I'm far worse. I'm one of those people who, after being told a phone number, says, "Okay, repeat that. But in English". Ask me to translate this very post to Nepali and watch how fast I run away. My friends are, frankly, no better.

The problems are much worse for other languages of Nepal. The Newari Language is listed by UNESCO as being “definitely endangered”. The Kusunda language has speakers in single digit numbers, which is particularly unfortunate because Kusunda happens to be a language isolate, so once it’s gone, we’ll lose any chance of accurately reconstructing it. Half of the 123 languages of Nepal are endangered.

In this post, I will talk about Unicode for Nepali language in hopes that I can shed some technical light on Unicode in Nepali context for the uninitiated. The idea is to let interested people know of the tools and ideas in Unicode, specially in context of the Nepali language to encourage adoption and usage.

Some background​

Note: The numbers starting with ‘0x’ and 'U+' are Hexadecimal numbers. 'U+' prefix additionally implies that the number following it is a Unicode code point.

Computers fundamentally think in numbers. To a computer, a picture is a 2d matrix of color intensity at each pixel as recorded by the camera. An audio recording is, similarly, a sequence of amplitudes of a sound wave recorded several thousand times each second. It goes without saying that text is also read by the computer as a sequence of numbers.

Of course, what the numbers mean to the computer is entirely dependent on the program that reads the data. In case of a picture, the numbers are used to vary the brightness of color elements on the monitor. For an audio, the data sequence is used to proportionally adjust the voltage across the speaker coil, which causes the membrane to vibrate and create sound. For text, these numbers will be used to do a wide variety of task. If you’re typing, numbers corresponding to the alphabets you press will be stored in the memory. These numbers will be used to look up characters inside the font file, which stores a visual glyph for each displayable number. That will be drawn on the screen (after many more steps).

So, it seems that for the computer system to support text at all, characters will have to have a standardized mapping to numbers. If the font file and the keyboard don’t agree on what the number 65 is, then the wrong character will be displayed on the monitor.

ASCII is the standard mapping of these numbers to characters. It assigns the numbers from 0 to 127 to English alphabets and symbols. Because the early development of computers was centralized in the US and UK, ascii encoding became the primary method of text representation on computers.

Computers almost universally store data in blocks of 8 bits (a byte) which can represent 256 numbers (28). ASCII only uses half of that (128 characters). So, with ASCII, half of the representational space of a byte goes unused. There have been several extensions to ASCII which make use of the unused space from 128 to 255.

So how do you type Nepali in ASCII? Well, you don't. ASCII is, by its very definition, American. In the early ‘90s, Bureau of Indian Standards came up with its very own encoding scheme called ISCII. ISCII is an extension of ASCII, so it retains all characters from ASCII in their rightful place. From 128 to 255, it stores characters used in the languages of India. Interestingly, ISCII actually unified a bunch of similar scripts like Devanagari, Tamil, Bengali, and Oriya. So, a number 164 (0xA4) in ISCII can mean all of the following: अ, அ, অ, ଅ and more. The idea was that, since all these scripts are descendants of the same Brahmi script, they behave similarly and represent similar sounds, which is true, so all four of the above letters represent the same /ʌ/ sound.

To switch between scripts in ISCII, you just change the fonts rendering the numbers. If a Devanagari font is selected, computer displays Devanagari characters. If a Tamil font was selected, it displays the same text in Tamil. The equivalent for English language would be to do assign numbers to the 26 English alphabets, and switch cases by changing fonts. The phonetic information stays the same but the representing characters seem different. ISCII used a special ATR byte to specify which script the fragment of text was in. ISCII is important historically, because Unicode specification for Devanagari and other Indic scripts is based almost entirely on ISCII. The ATR byte was not adopted by Unicode because font attributes are not a part of Unicode.

In Nepal, however, the government didn't initiate any efforts. What I think happened instead was that people or groups (like Muni Shakya) independently assigned the 256 numbers to various characters and ligatures. Which character a specific number represented was completely up to the font you chose. That must have been messy. Fonts are meant to change how the text looks, not what it represents!

The table below shows the character mapping of two Nepali fonts from the late ‘90s. You’ll notice how same field of the tables doesn’t encode the same character. Both learning to type in these fonts and changing the fonts must have been an inconvenience.

So far, I have talked exclusively about the representation of text. Typing was a different beast entirely. Suppose that you were using the Annapurna font. Since keyboards come in the standard QWERTY layout, if you pressed the A key (which is internally ASCII 65), an अ would appear. But on changing the font to Sabdatara and it would change to a द्ब. The image below demonstrates how, typing कलम in Sabdatara font, and then changing it to Annapurna results in garbage.

But thankfully, over time these mappings stabilized to the what is now known as the Traditional or Remington's keyboard layout. Generally, if someone over 30 tells you that they can type in Nepali, what they probably mean is that they can type in the traditional keyboard layout. It is what you type if you use the Sagarmatha or the Preeti font.

Unicode​

Naturally, India was not alone in wanting an encoding for its scripts. The late ‘80s and the ‘90s saw the rise of encoding schemes such as PASCII, VSCII, JIS X, many ISO standards, and much more, each developed by a particular country or a group of countries to encode their language. All this was fine and dandy for the systems back then. But then came the Internet.

The Internet pretty much broke everything. Computers all over the world were expected to understand each other, but they spoke in different encodings. If you didn’t encode your data in the same format as the receiving computer was set to, your text would render as garbage. The problem was that a number in one computer in one country represented something entirely different in another computer in another country.

Unicode was designed from the ground up to support all scripts of the world. It currently has 1,37,994 characters from 150 writing scripts. Unicode puts each character its own unique identifying number. For example, the number 2325 (U+0915) uniquely identifies the Devanagari letter क, the number 70658 (U+11402) identifies the Newari letter 𑐂, the number 65 (0x41) represents the English capital letter A and the number 24859 (0x611b) the unified Chinese, Japanese and Korean ideograph 愛. These numbers mean the same thing no matter where in the world you are or what computer you are using. These numbers are also immutable, meaning that, Unicode won’t ever change what a number means, hence maintaining backward compatibility.

This table lists all Nepali Characters in the Unicode Devanagari block with their codepoint numbers.

Characters of the same script live in the same block. A block is essentially a range of numbers. For example, Devanagari characters live in the U+0900 to U+097F block. Newa is in the U+1900 to U+194F block. The Limbu writing system, Sirijanga Script, lives in U+1900 to U+194F block.

The advantage of separating language into blocks, is that each block can get its own kind of treatment from the software. We know that different scripts behave differently. In Devanagari, some characters go above the previous character, some under. Some change the character itself. In Tibetan, diacritics stack up vertically. A single Urdu character, when typed, can change how the entire word looks. Not to mention that scripts have different directions of reading. Chinese is written from top to bottom. Urdu is written from right to left. All these quirks can be handled by software based on what block the characters come from.

The Devanagari block is shown below. It was taken from Wikipedia.

Now, if you look at the table above, you will notice a lot of characters missing. Where is क्ष? Or द्य? Or all the half forms like क्‍ , ग्‍ , and the rest? Well, because Unicode was based on ISCII, and since Hindi considers क्ष, त्र and ज्ञ to be compound characters, they aren’t allocated separate code points. That doesn’t mean you cannot type it, though. You just have to input the constituent characters and the text-rendering system will take care of the rest. क्‌ष makes क्ष, त्‌र makes त्र and ज्‌ञ makes ज्ञ automatically.

Unicode Specification on how a software should handle compound characters in Indic scripts.

So then, what if you want to type out क्‌ष explicitly? Unicode provides a character called Zero Width Non-Joiner (ZWNJ) (zero width because it doesn’t take up any space in the text). But it prevents the Halant ्‌ from changing characters to their half forms. Similarly, there is a Zero Width Joiner which explicitly asks for the half form to be displayed, even if there isn’t a character next to it.

Now, if you look at the table for Sabdatara and Annapurna font above, you’ll notice that the half forms are explicitly encoded. That is probably an artifact from the printing press era. When typesetting for the printing press, or when using a Linotype, a क followed by a ्‌ wouldn’t magically change into क्‍. You would have to select the correct type blocks manually. That kind of thinking could have carried over into the font design process.

But computers are smarter than that. They will easily substitute the glyphs for you. The only thing to keep in mind is to use your ZWJs and the ZWNJs when you want the other forms.

Unicode intends to depict the underlying characters rather than the renderings. Think of it this way: प्राप्त is the rendering and प + ्‌ + र + ा‌ + प + ्‌ + त is its underlying character composition. Unicode delegates the rendering to the display system. This model of text encoding is called the Virama-based model (virama meaning halanta), and is not unique to Devanagari.

The Unicode 12.0 specification has an insightful image to clarify further.

UTF​

Unicode is mostly concerned with assigning abstract numbers to characters, and describing their properties. Unicode Transformation Format, or UTF is the implementation of the Unicode specification. UTFs are the algorithms which convert these abstract numbers into bits and bytes that are eventually stored in the memory, processed or transmitted.

Only three Unicode Transformation Formats are in widespread use: UTF-8, UTF-16 and UTF-32. The numbers in their name signify the number of bits they use. So UTF-8 uses 8 bits (one byte), UTF-16 uses 2 bytes and UTF-32 uses 4 bytes. The figure below will try to demonstrate how UTF8 works. It’s okay if it ends up confusing you instead. Unless you are writing multilingual computer programs, it is not generally necessary to know how these encoding schemes work. Just remember that most all of internet uses utf-8 because it is the most efficient encoding.

Nepali Input

Unless you can already type in the traditional layout, I recommend that you try out the Phonetic layout by Nepali Language Technology Kendra. I’m sure any reasonably dedicated person can learn this layout in a day.

As it turns out, it is a fairly simple exercise to create a new keyboard layout from scratch. I was able to make myself a modified version of the Phonetic layout in less than an hour. To make your own layout, download and install the Microsoft Keyboard Layout Creator tool.

Nepal Language Technology Kendra website also provides keyboard layouts for Nepali Keyboard layouts for Linux. On Android phones, you can install either Hamro Keyboard or Google Indic keyboard, both of which are excellent. On iPhones and iMac, simply activating the required input method in Settings should suffice.

Fonts​

Google Fonts has a usable selection of fonts for Devanagari. To filter the fonts by the language, find the Language dropdown and select Devanagari (even though it isn’t a language).

Featured photo of Linotype by werner moser from FreeImages.com

So, I started to write a web app to make the program easily accessible. The web app lets user write code in their browser, then compiles and executes the program on the server, and allows the user to send input from the browser to the server as it executes.

My first instinct was to use something like AWS Lambda to compile and run each process as a cloud function, but then I looked into the deep abyss of my wallet and found myself lost in the darkness.

Another idea was to forego the cloud altogether. Compiling code into assembly can be done in any under-powered Virtual Private Server. I can write an implementation of a simple virtual machine in JavaScript, then I can add a new backend to my compiler to generate code for the virtual machine. Then I can embed the JS virtual machine in the webpage, and when the user hits compile, all server has to do is compile the language into machine code for the virtual machine and send that back to the client. Execution becomes client-side headache. Something like (a slightly saner version of) Brainfuck could be perfect for this kind of application. It's relatively simple to make a Brainfuck Virtual machine.

Anyway, I decided that AWS Lambda is too wasteful for my needs. Virtual Machine on Webpage idea is going to significantly increase code maintenance related tasks in the future. I tried to find some other way of executing user's programs on the server.

Issues with executing unsafe binaries on server​

But huge security issues emerge by allowing user-generated executables to run on your server. Just to name a few:

1. The executable can start an expensive infinite loop (for example: listing all prime numbers above 10^5), which makes the system unbearably slow for other processes.
2. It can generate and store a huge amount of data which either completely fills up the RAM, the storage system, or both, causing the system to slow down or crash, and making it physically impossible for other users to store and run their executables.
3. It can overwrite important files in the server (for example, the very node.js program responsible for compiling and executing user programs) and infect all clients with malicious payloads.
4. It can utilise the program bugs or kernel bugs to elevate it's status to root to install stealthy rootkits or bitcoin mining software to use our servers for their benefit.

This is just the tip of the iceberg. So many other malicious attacks are possible depending on the system and infrastructure arrangement. The system designer has to be very careful in setting up the system where untrusted and potentially unsafe executables are run, without causing significant lags for genuine users.

Solutions and preventive measures​

This was the first time I had to design a system like this, so I had to research quite a bit. In this post, I highlight some results of my research. First, lets see how the above issues can be dealt with in a general way.

1. To prevent expensive infinite loops, we can limit the maximum CPU percentage allowed to each user process and kill it after a fixed time.
2. To prevent memory overuse, simply limit the maximum memory allowed per process.
3. Either completely block file writes, or have a sandboxing file layer which redirects all file accesses of the process to a safe temporary directory. Or run the process on a virtual file system in a separate mount namespace.
4. Use redundant security measures and tightly constrain the execution environment using whitelists to minimize the kernel attack surface.
5. Execute the user process in a sandbox, which essentially manages most of the above concerns and provides more isolation.

These are, of course, just the general guidelines. In my case, because i. the users actually can't directly craft the machine code that runs in the server and ii. I have full control over and knowledge of the assembly code that is being generated, things are a little easier security-wise. But this will eventually change as I add more features and as more contributors join. So taking some time to tighten the security is more future-proof.

Features the Linux Kernel provides​

I'll start with the features already provided by a relatively recent Linux kernel by default.

1. Users and Groups: It's obvious, but by simply ensuring that there's no case in which the unsafe binary will be run as root and that no program started with sudo executes the unsafe binary, we cut off a big chunk of attacks. But let's take it a step further. Make a new user account with limited privileges and run the unsafe binary as that user. Set up permissions accordingly so that the new user account cannot read what need not be read. Add the user only to groups which it absolutely requires.
2. Namespaces: Namespace is a feature of Linux kernel that allows you to isolate a process from other processes in certain respects. For example, if you run the untrusted executable in a separate process namespace, the process is not able to see or interact with any other process running in the system. It literally becomes the init process in it's view. Namespaces allow containers like dockers to fake isolated systems without the overhead of the full Virtual Machine.
3. Control groups: Control groups (also called cgroups) allow you to allocate resources and set limits on CPU time and memory, among other things. This, when used in conjunction with namespaces allows for effective containerization of apps.
4. Capabilities: Capabilities in Linux allows selective provisioning of root privileges. If you really have to allow the untrusted program to do things that only root is able to, then capabilities allows you to allocate only the required root-only operations to the running process
5. Seccomp-bpf: Seccomp-bpf stands for Secure Computing mode-Berkeley Packet Filter (although no one calls it this). Seccomp by itself blocks all syscalls except four (exit(), sigreturn(), read() and write() on already open file descriptors) so unsafe compute-bound processes can be run without many risks, as almost 99% of the syscalls are blocked by the kernel. BPF is an addon to seccomp which allows you to block any syscalls you want. strace is a linux tool that allows you to trace syscalls made by a process.

Because these features are provided natively by the Linux kernel, we can apply them using corresponding syscalls and parameters and make with a relatively robust sandbox. I toyed with the idea of making my own restricted micro-sandboxing program (and I really wanted to) but decided not to because I was already juggling more things than I like to.

There are programs which use these kernel features and more to sandbox applications for us. I had expected there to be many, specially in this age of cloud computing and lambda functions.

nsjail: I couldn't get it to compile because of some strange protobuf dependency error. It doesn't help that the GitHub readme doesn't have any build steps or the versions of dependencies required. Which is a shame because this was almost exactly what I was looking for: a lightweight application sandbox. I might look into it some more later. This is the official site for nsjail.

mbox: It doesn't seem to work in 2019. The example usages in GitHub fail to do any kind of blocking. The -n still doesn't block internet access. I'm guessing that it relies on some old kernel specific features. It was last updated 3 or 4 years ago on github. Also, reading the author's paper and ycombinator comments, I got the feeling that he's much more proud of his filesystem layering work than his sandboxing work. Also, Mbox seems more like some academic/proof-of-concept work. Doesn't fit my requirements. This is the official site for Mbox.

Docker: Heavy and not built for my use case. It is also not as security-focused, and apparently it can be broken out of. At any rate, I'm not gonna be instantiating full containers for executing a sub-megabyte programs. Although Docker running small distros like Puppy linux is an interesting idea. This is the official site for Docker.

systemd-nspawn is really interesting. It needs a full chown-able filesystem I didn't use it this time, but I'm definitely going to use this in some future project.

minijail: This little software is apparently used by Google to sandbox chromium programs. It is not as feature rich as nsjail so I ended up using this in conjunction with cgroups and some other programs to isolate the unsafe binary.

https://en.wikipedia.org/wiki/OS-level_virtualisation#Implementations

PS Mosh is really cool substitute to SSH, specially when you're using vim to code directly on the server. Plus, the fact that I don't have to restart SSH connection every time I wake my laptop is such a convenience.

Resources

1. https://en.wikipedia.org/wiki/OS-level_virtualisation#Implementations
   This page has a list of many sandboxing tools that don't use full VMs.
2. https://chromium.googlesource.com/chromiumos/docs/+/master/sandboxing.md
   Explains how minijail can be used for security. If you're using Minijail, also check out the embedded video.
3. https://people.csail.mit.edu/nickolai/papers/kim-mbox.pdf
   The author's paper explaining Mbox.
4. https://blogs.rdoproject.org/2015/08/hands-on-linux-sandbox-with-namespaces-and-cgroups/
   https://www.toptal.com/linux/separation-anxiety-isolating-your-system-with-linux-namespaces
   Use these two links if you're going baremetal. Also try finding pages in chromium.googlesource.com about sandboxing. They have done a lot of work in that area. A representative page is below: https://chromium.googlesource.com/chromium/src/+/master/docs/design/sandbox.md
5. http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/34913.pdf
   This is the paper explaining Native Client, a sandboxing system developed by Chrome developers to allow execution of C/C++ programs in the web browser with near-native speed.
6. https://blog.golang.org/playground
   The Go language website has an online code compiler similar to ours. In this blog post, they describe how they used Native Client (see resource 5) and other ideas to provide the safe online compilation and execution service to users. Particularly interesting is their decision to disallow any interactive input to favour caching results and to reduce CPU time per program.
   They also seem to be using a separate branch of compiler to generate Native Client executable for executables. I had contemplated on whether I should add an extra backend in my compiler which generates secure code at compile time. But I decided that the maintenance burden was not worth it.

I was asked to animate a dot-matrix display for the robotics club recently. They wanted something that said "Robotics Club" to hang over their door. We had some old P10(1r) DMDs which I had worked on in the past to make a little scoreboard for a robot football match. And, even though I am not very good at it, I really love animating things. So I decided to give it a shot. I ended up writing an animation software for the DMD in JavaScript which is unfortunately only as functional as a flipbook. But I had fun animating little person carrying a balloon, ugly gear trains and a little pixel Wall-E blinking.

Yeah, that's a little Wall-E looking at you. After thinking long and hard about it, I decided that Wall-E is the best robot to look over us all while we do our robotics stuff. Wall-E was also, thanks to his angular design and distinctly recognizable shape, much better suited to a low resolution display than some other things I wanted to do.

Another thing I animated was a little man who carries a balloon and strolls into the DMD where "Robotics Club" is written and just stands there idly for a while, then goes away. I thought it was a cool thing to do, but no one else agreed. So I reluctantly scrapped that idea. I also wanted to animate some other things, like a gear train driven by a hamster, and GIR from Invader Zim (remember that show?). But of course, a monochrome 32x32 matrix doesn't give you much space to play with. I was also limited by time. I'm working almost full time on our minor project, and I also have to attend my classes. For the next version of this screen, I'm planning a much cooler animation, that uses the whole 32x32 screen and the whole 32KB of the AVR program space, and maybe also some form of compression. In this post, I document all the major portions of the project, while also trying to be helpful to people designing DMDs.

I made all the electronics for the project last semester with Bhuwan. We made the circuit board by hand, doing everything from hack-sawing the PCBs, etching the copper traces with a fumigating solution of FeCl₃, drilling the boards and soldering the components. That process, while manual, was very enjoyable. The resulting circuit looks nice and is so robust that from the time it was built, there has never been any hardware bug in the system. I mean, sure, being a MCU based digital circuit doesn't leave much room for bugs. But still, considering the many many many hardware bugs we encountered while doing the robot MIMO (we ended up making 6 iterations for the circuit board), this one was a walk in the park.

The board is straightforward. It's an Atmega32A with it's support circuitry (oscillators, decoupling capacitors and such) and a bunch of headers: one for connecting the In-System Programmer, one for connecting to a control remote, and one for the Dot-Matrix display.

The display itself (P10(1r)-V70) is a matrix of LEDs all connected to a bunch of 74HC595 shift registers in series. The DMD takes the following inputs:

1. Output Enable (OE)
2. Row Selectors (A and B)
3. A data line (D)
4. A clock line (CLK)
5. A Latch line (SCLK)
6. A ground

The data line D inputs the data serially, clocked by the CLK line. On the rising edge of the SCLK pin, the data shifted in is displayed on the LEDs. So far, it is exactly as if we were using the 74HC595 directly. But the row selectors complicate things slightly. The people who designed this board have done something really cool. Perhaps to reduce part count, or to keep the board layout simple, or maybe because of fanout issues, they have connected 4 consecutive LEDs to the same set of shift registers. Which rows of the matrix you are addressing depends on the inputs to the selector pins. That is, for A = LOW, B = LOW, the first of the four rows is selected; for A = LOW, B = HIGH, the second; and so on. I will talk more about this further below.

The Animation Software

For various reasons, I had to write the animation software myself. The last time, I had written an animation program on Python using Tkinter. Actually, 'written' is a bad verb to describe that process. I had jerrybuilt the whole system in an hour. And over the course of the next week, I had haphazardly stuck various appendages and functions which made the code very difficult to navigate. On top of that, the program itself was very ugly.

This time, I decided to go with a HTML canvas based approach because I didn't want to download a GUI designer for python, and I sure as heck wasn't going to sit around and waste my precious time learning Tkinter's packs and grids and what not. So anyway, I surprised myself by jerrybuilding the entire system in a few hours, complete with animation frames, onion skinning and the undo feature. The canvas API was so simple, and JavaScript so forgiving, that I didn't even run into any time consuming bug. And the most interesting part? The complete HTML file (with JS and CSS in the same document) is 500 lines of code! It is still bad code, and is very inefficient in many respects. But because it is a means to an end, I allowed myself to take shortcuts. The result is the screenshot below.

It was so fun to make animations in my own animation program. The only seriously desirable feature that this program lacks is the select and the move feature. The old python version actually had that feature. But I was already pressed for time so I didn't implement it.

The software can be accessed at nirav.com.np/Animator-inator. Feel free to check it out.

The Firmware

I used the same firmware I had written a semester ago. The code itself is straightforward. But I remember it took me a long time to write it, because I didn't have a lot of documentation to consult. So I had to sit there, with the DMD board and it's various pins and wires, and I had to prod this and poke that to figure out how it worked, rather like a puzzle. Thankfully I had worked with the shift registers before, so it took a lot less work than it otherwise would have. This piece of code below is pretty much the only part of the code that counts.

// Slightly altered from original
// for readablity
void clock_selected_lines(char selector) {
  for (int i = 0; i < (WIDTH / 8); i++)
    for (int j = 3; j >= 0; j--)
      clockbyte(display[j * 4 + selector][i]);
  setselector(selector);
  sendpulse(LATCH);
}

int main() {
  init();
  OCR0 = 0xff;
  TIMSK = 2;
  TCCR0 = 0X0b;
  sei();

  int frame = 0;
  while (1) {
    cli();
    disableoutput();
    assign_to_display(frame++);
    enableoutput();
    sei();

    if (frame >= framecount) frame = 0;
    _delay_ms(10);
  }
}