Typolino – Cache Control

To improve the UX it is important to serve content as fast as possible. The Firebase hosting is pretty clever, but the images we serve from the storage has a default cache configuration that disallows caching the content. We can control the cache-control header when uploading the images.

   bucket.file(imagePath).createWriteStream({
            metadata: {
               cacheControl: 'public, max-age=604800' 
            }
   }
            

This will allow caching the content for 7 days. Sill – accessing the images is quite slow as we first need to get the actual download URL. I wonder whether it would make sense to store the actual URL in the DB so that we can save the additional lookup call. Also we could add some PWA features to preload the images.

Typolino – Upload Lesson

Finally I find some time to write about the upload utility. Maybe we can add an online editor later, but to get started quickly I decided to use a pure node application running on my local device using the Firebase Admin SDK.

download the private key – and NEVER share it!

Now we are already connected to our DB and storage and can just upload and manipulate the data as we need. Each lesson will be store in a folder containing the actual lesson structure and all the media files.

each lesson is in a separate folder

And this is an example of a single lesson:

The lesson.json file has the same content as we will upload to the DB.

{
  "name": "Blumen",
  "description": "Lerne die Blumen kennen",

  "difficulty": "EASY",
  "words": [
    {
      "word": "Rose",
      "sentence": "",
      "imageId": "rose.jpg",
      "audioId": "rose.mp4"
    },
    {
      "word": "Tulpe",
      "sentence": ".",
      "imageId": "tulpe.jpg",
      "audioId": "tulpe.mp4"
    },
    {
      "word": "Lilie",
      "sentence": "",
      "imageId": "lilie.jpg",
      "audioId": "lilie.mp4"
    },
    {
      "word": "Seerose",
      "sentence": "",
      "imageId": "seerose.jpg",
      "audioId": "seerose.mp4"
    },
    {
      "word": "Sonnenblume",
      "sentence": "",
      "imageId": "sonnenblume.jpg",
      "audioId": "sonnenblume.mp4"
    }
  ]
}

Note I just configured the audioId, but we are not using it yet. So.. you can just ignore it for now.

Upload Script

The upload script is really simple and can be used like:

node upload.js <folder>

It will look for the lesson.json file, create a lesson document in DB and upload all non JSON files to the storage. It expects you have no typos in the configuration – but it’s not that hard to fix it and just upload again.

We will discuss this topic later and (hopefully) even improve more. As with many cloud services you have a pay-as-you-go model (OpEx) it is important to minimize these costs as much as possible. So having to download multi MB images and then just render them in a small 640px * 480px resolution doesn’t make much sense. I love to tune things as much as possible, I feel being back at the times where resources where sparse and finding creative ways to save some money is really cool. 🙂 One aspect where we definitley need to look into later on is caching.

Long story short: we transform every image before uploading to fit our target resolution. For this I found a cool library called Jimp.

(async() => { 
    const admin = require("firebase-admin");
    const fs = require("fs");
    const Jimp = require("jimp");

    const arguments = process.argv;
    const lessonFolder = arguments[2]; // find a better way e.g. oclif?

    console.log(`You would like to uplaod folder ${lessonFolder}`);

    const serviceAccount = require("C:/Dev/typolino-firebase-adminsdk-4dg10-b2047407d9.json");
    const app = admin.initializeApp({
    credential: admin.credential.cert(serviceAccount),
    databaseURL: "https://typolino.firebaseio.com",
    });

    // upload the data
    const lesson = require(`./${lessonFolder}/lesson.json`);
    const firestore = app.firestore();
    const storage = app.storage();
    const bucket = storage.bucket("gs://typolino.appspot.com/");

    // create basic structure
    firestore.collection("lessons").doc(lessonFolder).set(lesson);

    // upload all files
    fs.readdirSync(lessonFolder)
    .filter((file) => !file.endsWith(".json"))
    .forEach(async (file) => {
        const imagePath = `${lessonFolder}/${file}`;
        const image = await Jimp.read(imagePath);
        await image.resize(640, 480);
        await image.quality(80);        
        bucket.file(imagePath).createWriteStream().end(await image.getBufferAsync("image/jpeg"));                
    });
})();

Not much tuning yet, but at least we don’t upload super big files. That’s all! For now it is good enough and does the job.

Typolino – Adding Speech

If we want to add sound to Typolino we need to consider the proper format. I’m not an expert but https://caniuse.com is usually very good at answering these kind of questions and also Wikipedia provides a good overview. I don’t care too much about the lossyness of the format as my audio capture device anyhow is pretty cheap. So let’s try with mp3.

To record the audio I usually use Audacity. The tool is great and full with features. To make it a bit more complicated to code let’s try to keep one audio file and just jump to the correct location for every character. The map needs to be confiugred and only one audio resource needs to be managed. This would be a similar approach to what we use to do with CSS sprites. Having zero to no experience I would say: just try it. 🙂

Recording

Recording was quite straight forward. Just had to find a moment of (almost) silence. Audacity is really cool – they have so many built in effects. I was only interested in the one to remove silence and the noise suppression. I think especially noise suppression cleans the curves quite a bit and helps in keeping the file size smaller. But really – not an expert. 🙂

The exported mp3 file is roughly 200k. Not sure what else we can do to improve – but for now I’m quite happy. Without noise suppression it was around 260k.

Storing

I keep the file as an asset. For this I just put it into the assets folder.

Seek and Play

For our audio sprite we need a service. It seems to be a bit cumbersome to use the native API so I found this Angular *wrapper* which looks quite promising:
https://github.com/chrisguttandin/angular-audio-context

Installation is simple:

 npm install --save angular-audio-context 

…and it looks way too complex. I don’t want to deal with all the details of the web audio API. But as I’m writing as I’m coding sometimes you change your mind. Not sure how I found this one (but it was not obvious to me at least), but it looks so much easier and seems to offer exactly what I need:
https://howlerjs.com/

It even supports Audio sprites, which.. kind of answers my initial question. 🙂 So let’s try this one.

 npm install --save howler 
import { Howl, Howler } from 'howler';
import { Injectable } from '@angular/core';

@Injectable({
  providedIn: 'root',
})
export class AudioService {
  private sound: Howl;

  constructor() {
    this.sound = new Howl({
      src: ['assets/alphabet.mp3'],
      sprite: {
        A: [0, 380],
        B: [389, 450],
        C: [830, 530],
        D: [1360, 420],
        E: [1805, 460],
        F: [2280, 360],
        G: [2650, 500],
        H: [3115, 490],
        I: [3615, 460],
        J: [4065, 360],
        K: [4440, 470],
        L: [4920, 510],
        M: [5430, 480],
        N: [5920, 530],
        O: [6460, 490],
        P: [6960, 460],
        Q: [7427, 510],
        R: [7937, 465],
        S: [8413, 557],
        T: [8982, 495],
        U: [9467, 470],
        V: [9950, 470],
        W: [10439, 528],
        X: [10972, 560],
        Y: [11548, 770],
        Z: [12380, 365],
      },
    });
  }

  play(letter: string) {
    this.sound.play(letter.toUpperCase());
  }
}

Conclusion

It works, not sure how well it will work in the wild. But it was interesting to implement this functionality. It took a bit of time in setting up the sprite but howlerjs just worked out of the box. Just for the sake of completeness: very first I was thinking of using speech synthesis API but I wasn’t so happy with the results.

Typolino – Improve image loading

Right now when the browser loads an image in a Typolino session you may experience some flickering. The word is rendered before the image is loaded. That makes sense, but it’s just plain ugly. One way to solve this issue is to show a placeholder image – or to wait with the whole content as long as the image is loading. Personally I think a placeholder is better as we don’t really know how long we would have to wait for the image. The technique works particulary well in our case as the image has a predictable size.

Placeholder

Instead of an actual image we can also use a dvi with some PrimeNg spinner / icon. This is the class for the image. Note the images that we use will all have a fixed resolution of 640px * 480px.

.lesson__img {
  object-fit: cover;
  max-width: 640px;
  width: 100%;
  background-color: #dbdbdb;
  padding: 1rem;
}

So the only thing we need to do is finding out when the image is loaded and then switch between the placeholder and the actual image. For this we can use the hidden attribute on the element.

<!-- Image -->
  <div class="ui-g ui-fluid">
    <div class="ui-g-12">
      <img
        class="lesson__img"
        [hidden]="!imageLoaded"
        *ngIf="imageUrl$ | async; let imageUrl"
        [src]="imageUrl"
        (load)="loaded()"
      />
      <img
        [hidden]="imageLoaded"
        class="lesson__img"
        src="assets/img_placeholder.jpg"
      />
    </div>
  </div>

I tend to put event these simple state changes into a function of the component:

loaded() {
    this.imageLoaded = true;
  }

Result

The result is as expected. But I really wonder if there is a way to mimic this resizing behavior with pure CSS and a simple DIV. I have to think about this a bit more, but the combination of max-width and some way to keep the aspect ratio (view height / view width??) doesn’t sum up correctly in my mind right now. Anyone any good ideas?

Typolino – Training

So we are finally at the most important component. The first PoC technically worked, but no one would be happy to use the tool as is. So we have to apply the same styles and create a more interactive experience.

Keyboard Events

The key functionality is to type the word you see, letter by letter. For the kids this should be visualized in a way they don’t get lost easily. I decided to not provide a classical input field but just register for keyboard events directly. Altough the application so far is designed to be responsive, I never had in mind to make it usably on the mobile. But on the other hand it would be really, really nice to have it on the go (for instance in the car). Let’s think about this later.

Adding a host listener works quite well to caputre all events.

@HostListener('window:keyup', ['$event'])
  keyUp(event: KeyboardEvent) {
    const newWord = (this.userWord + event.key).toUpperCase();
    if (this.currentWord.startsWith(newWord)) {
      this.userWord = newWord;
      this.checkWord();
    } else {
      this.millisSpent += 2_000; // penalty
      this.highscoreMissed = this.millisSpent > this.lesson.bestTime;
    }
}

The checkWord function is just checking for the completion of the word and initiates the next word – or in case it was the last word returns to the overview screen.

  checkWord() {
    if (this.userWord === this.currentWord) {
      this.messageService.add({
        key: 'lesson',
        severity: 'success',
        summary: 'Super',
        detail: 'Du hast es drauf!',
      });

      timer(1_000).pipe(first()).subscribe(this.next.bind(this));
    }
  }

The next operation is actually the most complicated part as we need to store the highscore, save the completion status etc.

next(): void {
    if (this.lesson.words.length > this.index + 1) {
      this.index++;
      this.setupWord(this.lesson.words[this.index]);
      this.messageService.clear();
    } else {
      this.timerSubscription.unsubscribe(); // stop timer
      this.lessonService
        .completeLesson(this.lesson, this.millisSpent)
        .subscribe(() => {
          this.navigationService.showLessons();
        });
    }
  }

And here the complete function. I’ll have to think this through again to make it easier to test. With RxJS you can really mess up a lot. I haven’t used transactions and I’d like to split it up into smaller pieces that can be tested individually. This article explains quite well what we should watch out for.

 completeLesson(lesson: Lesson, millisSpent: number): Observable<any> {
    return this.fireAuth.user.pipe(
      switchMap((user) => {
        return this.firestore
          .collection(`user-data/${user.uid}/lessons`)
          .doc(lesson.id)
          .get();
      }),
      switchMap((doc) => {
        if (!doc.exists || millisSpent < doc.data().bestTime) {
          return doc.ref
            .set({
              bestTime: millisSpent,
            })
            .then(() => doc);
        } else {
          return of(doc);
        }
      }),
      switchMap((doc: DocumentData) => {
        return doc.ref
          .collection('results')
          .add({
            timeStamp: new Date(),
            millisSpent: millisSpent,
          })
          .then(() => null);
      })
    );
  }

Timer

As soon as the user types the first key a timer is started. As long as it is your first run or you are below you last highscore it should be displayed green – else it switches to red.

With RxJS we get a powerful tool to implement such features.

  @HostListener('window:keyup', ['$event'])
  keyUp(event: KeyboardEvent) {
    if (!this.timerSubscription) {
      this.timerSubscription = timer(1_000, 1_000).subscribe((time) => {
        this.millisSpent += 1_000;
        this.highscoreMissed = this.millisSpent > this.lesson.bestTime;
      });
    }

…and cleanup

 ngOnDestroy() {
    if (this.timerSubscription) {
      this.timerSubscription.unsubscribe();
    }
  }

The rendering is simple using the built in Angular pipe.

  <!-- Timer -->
  <div
    *ngIf="timerSubscription"
    class="lesson__timer"
    [ngClass]="{ 'lesson__timer--highscore_missed': highscoreMissed }"
  >
    <h2>{{ millisSpent | date: "mm:ss" }}</h2>
  </div>

I don’t really like to check for the timerSubscription, but why add an extra attribute? Maybe subject to refactoring. But other than that I think it should work.

What next

Now we are coming to the fun part soon: the tuning and refactoring. I really want to explore how to further improve the application in terms of code quality, performance and usability.

But first: we need a simple way to upload lessons. For this I’d like to use the admin SDK and a super simple way to just upload images and lessons.

Typolino – LESS / Responsive Helper

I like LESS. It is simple and has some cool features. I usually try to follow BEM when it comes to define classes and styles – but mixins are great.
To extend the responsive functionality I created a set of media query helpers that can be added to any class:

.sm(@rules) {
  @media only screen and (max-width: 40em) {
    @rules();
  }
}

.md(@rules) {
  @media only screen and (min-width: 40.063em) {
    @rules();
  }
}

.lg(@rules) {
  @media only screen and (min-width: 64.063em) {
    @rules();
  }
}

.xl(@rules) {
  @media only screen and (min-width: 90.063em) {
    @rules();
  }
}

The widths match the PrimeNG grid, I just took the values from there. Now we can just add special rules for each media query.

body {
  background-color: #202040;
  color: white;
  text-transform: uppercase;
  font-family: "Baloo Paaji 2", cursive;
  font-family: "Roboto", sans-serif;
  letter-spacing: 0.5rem;
  word-spacing: 1rem;
  padding-left: 10%;
  padding-right: 10%;

  .sm({
    padding-left: 0;
    padding-right: 0;
  });

  .md({
    padding-left: 10%;
    padding-right: 10%;
  });

  .lg({
    padding-left: 20%;
    padding-right: 20%;
  });
}

Typolino – Lesson selection

Same pattern also on the lesson selection screen. Here I used the card component to render each individual lesson. We can still improve this page a lot I think – but for a first release it should do the job.

The upper case text, the huge font and the spacing make it a bit difficult to render all the information. Hiding the overflow plus using ellipsis is a typical approach but it doesn’t really help the user.

Besides I haven’t changed the logic by any means. It’s just listing the lessons. The actual training part will be a bit more work. 🙂

Typolino – Registration

Let’s apply the same pattern to style the registration form. I don’t really like to have so many options at once – the UI feels overloaded. But also being a technical demonstration of what we built so far I’d like to keep all the options. Why not have a default one and let the user show more only if requested? Let’s keep the whole page as simple as possible.

Show only the standard option
Extend the options view

PrimeNg

I have added PrimeNg quite early. The reason is that I really think this is a quite complete widget set. It proved very helpful so far and it has a nice password component that I’d like to use to indicate the passoword strength.

<!-- Password-->
<div class="ui-g ui-fluid">
  <div class="ui-g-12">
    <input
      [(ngModel)]="password"
      type="password"
      pPassword 
      placeholder="Passwort"
    />
  </div>
</div>

Code

I’ll publish the complete code at end to Github and only focus on some interesting aspects on the blog.

Typolino – Add Style

So far the app really looks crappy. But to be honest I follow the very same pattern in my job. Sometimes it’s hard to convince people that it will look better later on. One can waste so much time in making the stuff nice and shiny (and if you only focus on a small piece of the software there might be no chance to establish common patterns – and the end of the story is proliferation everywhere).

Login screen

The login screen is quite easy and a good playground to see which pattern we should follow for Typolino. First I thought I just hack it in, but somehow that’s impossible. Let’s try to make it clean and responsive. As we already added PrimeNg and it comes with a built in grid system – let’s just use that one.

It is actually quite weird how much noise you produce in HTML to style it properly. It just explodes! Therefore I like to add comments in HTML so that I can easily identify the parts later on. HTML is one of the rare cases where I really like comments.

<div class="ui-g ui-fluid">
  <!-- Title -->
  <div class="ui-g-12">
    <h2>Login</h2>
  </div>
</div>

<!-- E-Mail -->
<div class="ui-g ui-fluid">
  <div class="ui-g-12">
    <input [(ngModel)]="email" type="text" pInputText placeholder="E-Mail" />
  </div>
</div>

<!-- Password -->
<div class="ui-g ui-fluid">
  <div class="ui-g-12">
    <input
      [(ngModel)]="password"
      type="password"
      pInputText
      placeholder="Passwort"
    />
  </div>
</div>

<!-- Login button -->
<div class="ui-g ui-fluid">
  <div class="ui-g-12">
    <button
      (click)="login()"
      pButton
      type="button"
      label="Login"
      icon="pi pi-lock"
    ></button>
  </div>
</div>

<!-- Register button -->
<div class="ui-g ui-fluid">
  <div class="ui-g-12">
    <button
      (click)="register()"
      pButton
      class="ui-button-secondary"
      type="button"
      label="Registrieren"
      icon="pi pi-user-plus"
    ></button>
  </div>
</div>

I’m not sure if this is a bug with the formatter I use in VSC. But aren’t input tags supposed to be self closing and are forbidden to be closed? Anyway, it works.

Global Styles

To get some common colors and layout we should add some styles to the body. These styles should be added to the styles.less file.

  • I choose the colors from a list of dark mode colors.
  • The font should be really easy for kids to read
  • The extra letter spacing and word spacing should help with readability.
body {
  background-color: #202040;
  color: white;
  text-transform: uppercase;
  font-family: "Baloo Paaji 2", cursive;
  font-family: "Roboto", sans-serif;
  letter-spacing: 0.5rem;
  word-spacing: 1rem;
  padding-left: 20%;
  padding-right: 20%;
}

Result

And this is how the login page looks like.