Your Site's RSS Feed

Test Apache Age

Mon, 04 May 2026 00:00:06 GMT

a docker compose file to get you start with apache age in postgres and a viewer

version: '3.7'

services:
  age:
    container_name: age-db
    image: apache/age
    restart: always
    environment:
      POSTGRES_USER: age
      POSTGRES_PASSWORD: age
      POSTGRES_DB: age
    ports:
      - "5432:5432"
    networks:
      - age-network

  age-viewer:
    container_name: age-viewer
    image: skaiworldwide/agviewer
    depends_on:
      - age
    ports:
      - "3000:3000"
    networks:
      - age-network

networks:
  age-network:
    driver: bridge

Nextjs - basic setup and

Fri, 08 Nov 2024 00:00:01 GMT

install nest JS Cli

npm install -g @nestjs/cli

create new project

nest new nestjs-task-management

npm run start:dev

goto localhost:3000

create a module

nest g module tasks

create a controller

nest g controller tasks

create a service

nest g service tasks

VALIDATORS

npm i class-validator class-transformer

ORM

npm i typeorm @nestjs/typorm pg

Git diff patch

Wed, 18 Dec 2019 00:00:00 GMT

Today wanted to get a diff of the current branch ignoring diffs on the target branch and create a backup patch.

git diff TARGETBRANCH…CURRENTBRANCH > FILE.patch

this seemed to work. replace TARGET BRANCH with the branch that has the commits you want to ignore. Replace CURRENTBRANCH with the branch you have the commits you want to backup and the FILE will be what you want to name the patch

Gitlab Runner Docker Hub Push

Fri, 29 Nov 2019 00:00:00 GMT

To push to your docker hub registry. first change the gitlab-runner password.

sudo su
passwd gitlab-runner

generate a token on docker hub Account Setings> New Access Token.

su gitlab-runner

and following the instruction docker hub e.g

docker login --username rjgallac

then enter the password token you generated on git hub

Undo Git Merge

Tue, 26 Nov 2019 00:01:00 GMT

If you’re in the middle of a merge

git merge --abort

If you’ve finished and want to undo

git reset --hard

Git diff two branches

Tue, 26 Nov 2019 00:00:00 GMT

This is an invaluable tool to quickly check before you merge two branches

git diff branch1   branch2

Local gitlab and Runners

Sat, 23 Nov 2019 00:00:00 GMT

In an attempt to learn gitlab and how to use runners I thought I would try and setup the environments on my laptop so I could experiment.

To set up a local gitlab is quite simple(you’ll need docker install docker). Open a command prompt and enter the following

sudo docker run --detach \
  --hostname gitlab.example.com \
  --publish 443:443 --publish 80:80 --publish 22:22 \
  --name gitlab \
  --restart always \
  --volume /srv/gitlab/config:/etc/gitlab \
  --volume /srv/gitlab/logs:/var/log/gitlab \
  --volume /srv/gitlab/data:/var/opt/gitlab \
  gitlab/gitlab-ce:latest

Make sure you add gitlab.example.com to your host file so you can browse to http://gitlab.example.com/

You’ll then need to create a runner locally. Enter the following on the command prompt.

sudo gitlab-runner register

You’ll be asked a series of questions. You’ll get your token from gitlab> admin> runners.

I created a shell runner and a docker runner.

The last question will ask you to give it a name. This will be how you reference it from your gitlab-ci.yml file.

Another thing I had to do was configure the gitlab toml file.

sudo nano /etc/gitlab-runner/config.toml

to get docker runners working I had to add extra host to the runners.docker section. Also note I up’d the conncurrent to 3 which was previously 1.

concurrent = 3
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "rob-Inspiron-5520"
  url = "http://gitlab.example.com/"
  token = "sY6dhj9QssARP8UWv4Q5"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

[[runners]]
  name = "rob-Inspiron-5520"
  url = "http://gitlab.example.com/"
  token = "vqpBor_Psc_zRCQ3arbY"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "node:13.1.0-alpine3.10"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    extra_hosts = ["gitlab.example.com:192.168.0.19"]

I created three very simple projects to experiment with running concurrently.

Two were simple shell scripts that slept for a numbers of seconds and the third was a node project to exercise the

The simple ones looks like this

stages:
  - testshell

testshell:
  stage: testshell
  script:
    - whoami
    - sleep 30
  tags: [robshell]

The node ones like this

stages:
  - testshell
  - test
cache:
  paths:
    - node_modules/

testshell:
  stage: testshell
  script:
    - whoami
    - sleep 3
  tags: [robshell]

testing:
  image: node:13.1.0-alpine3.10
  stage: test
  script: 
    - whoami
    - npm i
    - npm test
  tags: [robsdocker]

Docker Volume Permissions

Fri, 22 Nov 2019 00:00:00 GMT

I’ve struggled recently with outputing results in a docker container. The files get root permissions and gitlab fails to clean up. I found this little trick to give the files the user if of the current user.

-u `id -u`

e.g

docker run --rm -u `id -u` -v  `pwd`/data/:/var/data:z -w="/var/app" --net=host dockerimage:5  commandToRun

anything saved to the /va/data folder inside folder will get the owner of whoever ran it. The group stays as root.

Watch Server Script

Thu, 21 Nov 2019 00:00:00 GMT

A little script to ping/poll a server until it up before launching into the next stage of the pipeline build.

#!/bin/bash
while ! (wget https://REPLACE_SERVER_NAME/ --no-check-certificate -q -O /dev/null)
do
        echo "sleeping";
        sleep 2;
done

Locale dates in javascript

Wed, 06 Nov 2019 00:00:00 GMT

Seemed to spend a long time on this. Trying to display UTC date from backend in locale time

---- Using plain old javascript

newdate= new Date("2019-01-20T19:58:01");
document.write(newdate)
document.write(new Date(Date.UTC(newdate.getYear(), newdate.getMonth(), newdate.getDate(), newdate.getHours(), newdate.getMinutes(), newdate.getSeconds())))

---- Using momentjs

const momentUTC = "2019-01-20T18:18:01";
document.write("LOCAL: " + moment.utc(momentUTC).local().format("YYYY-MM-DD HH:mm:ss"))

Blog moved

Thu, 17 Oct 2019 00:00:00 GMT

Took the decision to move my wordpress blog to gatsby.

Was as simple as running.

  gatsby new my-blog https://github.com/gatsbyjs/gatsby-starter-blog

Reasons included hopefully saving on hosting fees. I’m using netlify as a free host for now but as its a static site now it can site anywhere i.e github pages.. I should be able to update it via github GUI. Hopefully it’ll be more secure i.e. no database involved, no wordpress involved.

Jest toHaveBeenCalledWith in order multiple times

Thu, 19 Sep 2019 00:00:06 GMT

Spent some time looking for a way to confirm that my service was called multiple time in the corrcet order. Tried doing a expect over multlpe lines like so.

    expect(asdf).toHaveBeenCalledWith(1);
    expect(asdf).toHaveBeenCalledWith(1);

but this did not work

however this did work

    const spy = spyOn(myService, “myMethod”);
    expect(spy.mock.calls).toEqual([[result1], [result2]]);

Angular observable service unit testing

Wed, 18 Sep 2019 00:00:06 GMT

Struggled today to write a unit test for an observable. The old jasmine (done) =>{done()} wasn’t doing it for me tried various other methods around the web and nothing worked… hoever this combination did work for me.

    import { TestBed, tick, fakeAsync, flush } from "@angular/core/testing";
    import { ToDoService } from "./todo.service";

    ......

    it("test todo service", fakeAsync(() => {
        const toDo = {"name":"test};
        toDoService.addToDo(toDo);
        tick(10000);
        toDoService.todos.subscribe(data => {
        expect(data.length).toBe(1);
        });
        flush();
    }));

The important parts are the fakeAsync, tick(1000) and flush();

java reduce

Tue, 02 Jul 2019 00:00:06 GMT

    tasks.stream().map(t -> t.getName()).collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));

A quick way of taking an array of objects and reducing it to a list of string and the count of accurances i.e. a group by.

flowable – a new bpm

Tue, 02 Jul 2019 00:00:05 GMT

Seems like a nice port of activiti https://www.flowable.org/documentation.html. I have tried it and of the little examples I’ve created it seems a little easier than activiti. A nice little addition is to start up an entire stack which includes a great little modeler

    docker run --name flowable -p 8080:8080 flowable/all-in-one

Custom fields missing from wordpress

Wed, 26 Jun 2019 00:00:05 GMT

click on the three dots, then options

Oh My Zsh -installing on ubuntu

Thu, 06 Jun 2019 00:00:05 GMT

This is a great command line improvement for me for linux especially if you’re using git as it’ll show you quite clearly which branch you’re on.

    apt install zsh
    chsh -s /bin/zsh root
    wget https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh -O - | zsh
    cp ~/.oh-my-zsh/templates/zshrc.zsh-template ~/.zshrc
    source ~/.zshrc
    nano ~/.zshrc

on git bash you may needs to install powerfonts and change some git bash settings to UTF-8 if you are not seeing the icons and a bit of gibberish but it should work if you’re in x-win. If you run this it’ll tell you something up with your fonts and character encoding.

    echo "\ue0b0 \u00b1 \ue0a0 \u27a6 \u2718 \u26a1 \u2699"

Youll probably have to do something like the following to fix

https://github.com/powerline/fonts ./install

Git – throw away all local commits and reset back to origin

Wed, 05 Jun 2019 00:00:05 GMT

    git reset --hard origin/branch-name

Virtualbox

Wed, 05 Jun 2019 00:00:04 GMT

My current style of working is use my IDE in windows and connect to docker containers running in virtual box(As I’ve not had much joy with docker in windows). To control the virtual box I’ll ssh into it so I can see what the docker containers are doing etc.To start a vm in headless mode run the following from windows command line

    vboxmanage startvm  "ubuntu dev" --type headless

this will start up the virtual box machine. I can then ssh into it from windows and do docker-compose up. Use tmux to give you more windows in one terminal. This way the only window open is potentially your IDE and a bash shell.

You could get an even cleaner desktop by running the terminal from your idea but tmux in a small terminal window in your IDE isn’t a great experience.

docker host ip

Tue, 04 Jun 2019 00:00:04 GMT

to get the host machine ip in a docker file

    DOCKER_HOST=$(ip -4 addr show docker0 | grep -Po 'inet \K[\d.]+')

which can then be used with your docker container or in docker compose

    --add-host=frontendapp:${DOCKER_HOST}

git stash

Tue, 04 Jun 2019 00:00:03 GMT

stash commands

    git stash list
    git stash pop 0
    git stash show 0
    git stash show -p 0

git file history

Tue, 04 Jun 2019 00:00:02 GMT

to view the changes to a file

git log -p

Angular karma test and selecting DOM elements for testing

Fri, 24 May 2019 00:00:02 GMT

using css selectors you can interrogate the DOM.

  const liElements = fixture.nativeElement.querySelectorAll(".delete-product");
  expect(liElements.length).toBe(0);

  const addProduct = fixture.nativeElement.querySelector("#add-product");
  expect(addProduct).toBeTruthy();

Angular Karma test

Fri, 24 May 2019 00:00:01 GMT

How to test and mock services in karma and spy on service using jasmine

import { TestBed, async, ComponentFixture } from '@angular/core/testing';
import { AppComponent } from './app.component';
import {TestService} from './test.service';
import { DebugElement } from '@angular/core';


describe('AppComponent', () => {
  let testService: TestService;
  let fixture: ComponentFixture<AppComponent>;
  let debugElement: DebugElement;

  beforeEach(async(() => {
    TestBed.configureTestingModule({
      declarations: [
        AppComponent
      ],
    }).compileComponents();
    fixture = TestBed.createComponent(AppComponent);
    debugElement = fixture.debugElement;
    testService = debugElement.injector.get(TestService);

  }));



  it('should render title in a h1 tag', () => {
    const fixture = TestBed.createComponent(AppComponent);
    fixture.detectChanges();
    const compiled = fixture.debugElement.nativeElement;
    expect(compiled.querySelector('h1').textContent).toContain('Welcome to angest!');
    expect(compiled.querySelector('h3').textContent).toContain('hi');
  });

  it('should rend', () => {
    spyOn(testService, 'getHi').and.returnValue("hi ho");
    const fixture = TestBed.createComponent(AppComponent);
    fixture.detectChanges();
    const compiled = fixture.debugElement.nativeElement;
    expect(compiled.querySelector('h3').textContent).toContain('hi ho');
  });


});

time taken in javascript

Thu, 23 May 2019 00:00:01 GMT

    const start = performance.now();
    const end = performance.now();
    this.searchTimeTakenInSeconds = ((end - start) / 1000).toFixed(2);

Delete a local branch from git

Tue, 21 May 2019 00:00:01 GMT

This will delete a branch locally

    git branch -d the_local_branch

This should be safe and will warn you if it hasnt been merged. Use uppercase -D to force delete

Check is a branch is already merged into master

Tue, 21 May 2019 00:00:00 GMT

To list all branches already merged

    git branch --merged master

to list all branches not merged into master

    git branch --no-merged

mocha and freecodecamp test

Wed, 15 May 2019 00:00:01 GMT

On the final module. Not a lot of instruction on how to run the test but you should be able to run them by

    ./node_modules/.bin/mocha --ui tdd ./tests/1_unit-tests.js

Turn a cold observable into a shared hot in Angular 6

Tue, 14 May 2019 00:00:01 GMT

Rather than have multiple backend calls a cold http service call can be turned into a hot shared multiplex service using the following

    public doGetSharedProductsService() {
        return this.ProductService.getProducts().pipe(share());
    }

Git pull another branch without switching

Tue, 14 May 2019 00:00:00 GMT

Something I’ve been doing for a long time is stashing a change, swicthing to the develop branch, pulling , switching back to my feature branch, unstashing and then merging. No need to switch, you can just pull in develop without switching.

    git pull origin develop:develop

Git undo

Sun, 12 May 2019 00:00:00 GMT

    git reset –soft HEAD~1

This will undo the last commit leaving the files you committed in your local workspace

    git reset --soft HEAD~1

Levenshtein Distance

Fri, 08 Mar 2019 00:00:00 GMT

Needed for a recent code wars.

See https://en.wikipedia.org/wiki/Levenshtein_distance for explanation.

The java code

public class LevenshteinDistance {

    public int calc(String s1, int s1_length, String s2, int s2_length) {
        int cost;

        if(s1_length == 0) return s2_length;
        if(s2_length == 0) return s1_length;

        if(s1.charAt(s1_length-1) == s2.charAt(s2_length-1)){
            cost = 0;
        } else {
            cost = 1;
        }
        return min(
                calc(s1, s1_length-1, s2, s2_length) +1,
                min(
                calc(s1, s1_length, s2, s2_length -1) +1,
                calc(s1, s1_length-1, s2, s2_length -1) + cost)

        );
    }
}

@Test
public void test(){
    String s1 = "saturday";
    String s2 = "sunday";
    LevenshteinDistance levenshteinDistance = new LevenshteinDistance();
    int calc = levenshteinDistance.calc(s1, s1.length(), s2, s2.length());
    System.out.println(calc);
    assertEquals(3, calc);
}

@Test
public void test2(){
    String s1 = "test";
    String s2 = "test2";
    LevenshteinDistance levenshteinDistance = new LevenshteinDistance();
    int calc = levenshteinDistance.calc(s1, s1.length(), s2, s2.length());
    assertEquals(1, calc);
}

PHP + Composer

Fri, 14 Dec 2018 00:00:01 GMT

    curl -sS https://getcomposer.org/installer | sudo php — –install-dir=/usr/local/bin –filename=composer

then

    composer install

Reset WordPress pass

Fri, 14 Dec 2018 00:00:00 GMT

update wp_users set user_pass = “5f4dcc3b5aa765d61d8327deb882cf99” where user_login = “username_here”;

linux – cpu and memory usage

Fri, 01 Jun 2018 00:00:00 GMT

I was looking to see how much ram and cpu a process was using. Running a war on a tomcat instance on my hosting was using over a constant 80%.

After looking into a new micro framework micronaut I looked at the ps command and htop but counldn’t isolate the process usage. However the following helped me easily watch the usage.

    watch ps -p [PID] -o %cpu,%mem,cmd

installing micronaut

Thu, 31 May 2018 00:00:00 GMT

Then trying to install micronaut and spending a lot of time trying to work out why I can’t build it I stumble upon sdkman

    curl -s "https://get.sdkman.io" | bash

and then

    sdk install micronaut

Keycloak

Sun, 20 May 2018 00:00:02 GMT

I’ve been recommended keycloak as a single sign on solution. Thought I would have a look into it this weekend. Found a simple tutorial here for implementing it in Spring boot first of all using freemarker and locking down the the tomcat container using keycloak and then using spring security. I should have stopped there.

I then tried to freestyle and use javascript to connect to the rest endpoints and I was met with my old friend CORS and access-control-allow-origin error on the front end. I spent about 30 mins getting keycloak working and as soon as I seen the CORS error I knew I was in for a very long night. About five hours later messing about with CORS in keycloak and spring boot and the front end I realise I was trying to use the same client for locking down both the front end and the backend. I release I can make another client for the ReST backend and set its access type to bearer only. And I would have another client thats access type is public for the angular front end.

nexus and gradle caching dependencies

Fri, 04 May 2018 00:00:02 GMT

Spent some time wrestling with gradle and a SNAPSHOT dependency. I would push a jar to nexus and try to pull it but it was still using the old dependency.

Every attempt to update index etc in nexus wouldn’t work. I tried going into .gradle/caches/modules/files and deleting all the file for my repo however even that wouldn’t work.

I couldn’t get this to work for me thinking maybe gradle had something to do with it.

configurations.all {
    resolutionStrategy {
        cacheDynamicVersionsFor 0, 'seconds'
        cacheChangingModulesFor 0, 'seconds'
    }

What did work in the end was that I noticied there was a .gradle/caches/modules/metadata folder. When I deleted my repo from here and refreshed my dependencies it worked.

Fetch API – ajax call without libraries

Fri, 04 May 2018 00:00:01 GMT

Putting this here as it took me a while to track it down.

https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

e.g

getAll () {
    console.log("GET ALL")
    return fetch('http://127.0.0.1:8084/customer/', {
        method: 'get',
        headers: {
          'Accept': 'application/json',
          'Content-Type': 'application/json'
        }
      }).then(function(response) {
        return response.json();
      }).then(function(data) {
        return data;
      });
}

or a post

addCustomer(customer){
  fetch('http://127.0.0.1:8084/customer/', {
    method: 'post',
    headers: {
      'Accept': 'application/json',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify(customer)
  }).then(function(response) {
    return response.json();
  }).then(function(data) {
  });
}

Thoughtworks tech radar and impostor syndrome

Sun, 15 Apr 2018 00:00:01 GMT

One site I regularly visit is tech radar to see if there are any new advances in programming that I should be looking at. Usually a lot of it is over my head but sometimes I’ll come across something that I will look into further and sometimes even being aware of these terms has proved useful. I recommend you visit every now and then https://www.thoughtworks.com.

A lot of the time it can bring out my impostor syndrome and I think I’m missing out on a lot of good nice tech because I just don’t have the time to keep up. I do my best in my own time to research and improve myself but I see things move into the archive section of the radar that completely slipped by me. Wish I had more time to try out some of these but I already feel like I am sacrificing a lot of family time just to do what I already do.

I consider myself a fullstack developer and although my work has mostly revolved around Java/Spring and angular I feel that I’m spread so thin that I know nothing and I am inferior to all of my peers. I know you can’t know everything and people have their own strengths, I know it can be exciting to live in a rapidly changing environment but I wish things would slow down for 5 minutes and not have a new framework released that I don’t have time to learn.

Versioning jars

Sun, 15 Apr 2018 00:00:00 GMT

After struggling for a long time trying to get snapshots and release into their respective places in nexus I thought I’d share my experience. I’m working on a small test for bundling a client API with a server. This seemed like an ideal time to improve my knowledge of nexus.

The client API contains the shared code such as DTOs that the ReST server and calling clients will use. The idea is that you could pass a JAR that contains the DTOs and the client and a user could simply Autowire in the client and call your service by invoking

productClient.getOne(1L);

I started off creating using a local repo using the maven plugin in gradle.

apply plugin: 'maven'
uploadArchives {
    repositories {
        flatDir {
           dirs '../repos'
        }
    }
}

This works well, in the gradle tasks you should have a task called jar which when called creates the JAR and a task called uploadArchives which will put it in your local repo. You can then reference the JARs from your other projects like so

 compile files('../repos/productapi-0.1.0.jar')

This works great when working on your own. I can see a benefit of this when wanting to make changes locally without having to push to git before you can test with the server locally and risk breaking changes.

I then decided to create a local docker instance of nexus and start pushing to that. The maven plugin had me covered just by changing the flatdir to the following.

mavenDeployer {
    repository(url: 'http://localhost:8081/nexus/content/repositories/snapshots') {
        authentication(userName: "admin", password: "admin123")
    }
    pom.version = project.version
    pom.artifactId = project.artifactId
    pom.groupId = project.groupId
}

I wanted to also improve my understanding of versioning which has always been a bit of a mystery to me. At this point I was happily pushing snapshots but what about releases and whats the difference anyway. I always thought SNAPSHOTS went something like 0.0.1-SNAPSHOT 0.0.2-SNAPSHOT 0.0.3-SNAPSHOT and then when you were happy you’d release i.e. 0.0.4-RELEASE. Turns out, not so. It looks like it goes 0.0.1-SNAPSHOT and you can have lots of iterations of this until your happy at which point when you release you have a 0.0.1-RELEASE. An automated process will then create a 0.0.2-SNAPSHOT for you.

So it was at this point I decided to look in nexus and work out how the hell I get a release file into the release part of nexus. By using a second plugin called maven release https://github.com/researchgate/gradle-release

plugins {
    id 'net.researchgate.release' version '2.6.0'
}

This gives you a nice little release task which when run you’ll see create a new JAR with the SNAPSHOT portion in the filename, updates the version in your gradle.properties and increments the version number. But it wasnt pushing the release build to nexus, I could see it locally but not in nexus. A lot of googling later I stumbled upon this snippet

 repository(url: "http://localhost:8081/nexus/content/repositories/${project.version.endsWith('-SNAPSHOT') ? 'snapshots' : 'releases'}") {

which looked very hacky to me and I couldn’t get to work.

however much googling later I stumbled upon this

uploadArchives {
    repositories {
        mavenDeployer {
            snapshotRepository(url: 'http://localhost:8081/nexus/content/repositories/snapshots') {
                authentication(userName: "admin", password: "admin123")
            }
            repository(url: 'http://localhost:8081/nexus/content/repositories/releases') {
                authentication(userName: "admin", password: "admin123")
            }
            pom.version = project.version
            pom.artifactId = project.artifactId
            pom.groupId = project.groupId
        }
    }
}

which looked and felt better however still didn’t work. It still wasn’t pushing to release. Lots of googling later I was still none the wiser until I took a break and realised that the release plugin was unaware of the maven upload plugin and I should be plugging the uploader into the release plugin. My thinking was it was doing this

CREATE RELEASE -> CREATE SNAPSHOT -> UPLOAD

when I wanted

CREATE RELEASE -> UPLOAD -> CREATE SNAPSHOT

so I tried

afterReleaseBuild.dependsOn uploadArchives

and it worked. Funny how one line makes a difference.

I then decided to get Jenkins involved and to let jenkins build the api and push to nexus. I did spend some time looking at the jenkins nexus plugin but I felt that that would be tying myself to installing plugins and I wanted as much as possible to live in my code(pipeline as code) and let gradle do the heavy lifting.

Code visualization

Fri, 06 Apr 2018 00:00:03 GMT

On my quest to find something to visualise java code I found these useful tools for intellij

http://findtheflow.io

https://plugins.jetbrains.com/plugin/7324-code-iris

https://github.com/Vanco/SequencePlugin

Intellij updates

Fri, 06 Apr 2018 00:00:02 GMT

I frequently find myself with a duff shortcut on my desktop when I update intellij on ubuntu.

Edit this /.local/share/applications/jetbrains-idea-ce.desktop

and update to the location of the latest update

enter non running docker

Fri, 30 Mar 2018 00:00:02 GMT

add this command to set off an infinite loop giving you time to bash into the container and check the folders etc

    -c “while true; do sleep 2; date; done”

play with docker and open faas

Fri, 30 Mar 2018 00:00:01 GMT

A nice little way to test open faas as well is to use play with docker

https://labs.play-with-docker.com

start a new session, create an instance and run this

    docker swarm init --advertise-addr eth0 && \
    git clone https://github.com/openfaas/faas && \
    cd faas && \
    git checkout 0.7.1 && \
    ./deploy_stack.sh && \
    docker service ls

play with docker and docker swarm

Fri, 30 Mar 2018 00:00:00 GMT

Reacquainted myself with docker swarm and followed the offical tutorial here https://docs.docker.com/get-started/. Also following along with this https://www.youtube.com/watch?v=bGYAkFGkUws&t=306s reminded me of the online sandbox https://labs.play-with-docker.com/. I managed to reproduce the stack from my local on the online sandbox is seconds. The only issue was with the redis data folder which I fixed by simply mounting the data folder inside the container. The youtube video above talks about using ngrok but the play-with-docker site publishes these ports anyway so didnt see the point.

openfaas and localstack

Thu, 29 Mar 2018 00:00:00 GMT

I’ve only dabbled with these on my local machine but the following let you play about with serverless on you machine. Something I really want to get into but just haven’t had the time to do as yet.

https://github.com/openfaas/faas

https://github.com/localstack/localstack

groking and ELK

Tue, 27 Mar 2018 00:00:00 GMT

Run into a bit of trouble with logstash and it not processing ERROR level messages from spring boot. Countless searches turned up nothing. By slowly building up the grok expression it turned out the problem was a rouge space. INFO and DEBUG have two spaces whilst ERROR and WARN have one. So the end expression is so

    "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})%{SPACE}%{LOGLEVEL:level} %{NUMBER:pid} --- \[(?<thread>[A-Za-z0-9-]+)\] [A-Za-z0-9.]*\.(?<class>[A-Za-z0-9#_]+)\s*:\s+(?<logmessage>.*)"

Yak Shaving

Sat, 24 Mar 2018 00:00:00 GMT

Talk about shaving a yak. I created a tool to analysis spring bean dependencies. It worked fine on a simple test app I created and produced a nice Directed graph. Ok, so I wanted to try it on a semi real app I created however it wasn’t working. I realised it the actuator /beans endpoints had a different formatted json output which I realise must be a version difference of spring. So I thought I’ll just rewrite the node app that outputs the json in the required format for D3js. Ok, 30 mins later I think instead of this I’ll upgrade my semi real app to spring 2. 3 hrs later after writing the app so its uses the new repo methods getById , the native queries needed changing from using mappers to projections, updating flyway, updating my tests, fixing caching.

I forget what the hell I was trying to do in the first place.

redis spring boot caching

Wed, 21 Mar 2018 00:00:00 GMT

spent a long time investigating jedis unexpected end of stream issue. I’m using docker and it turns out spring boot couldn’t see it. Solved it by adding –net=host for the mean time.

jmeter etags and 304 not modified

Sat, 03 Mar 2018 00:00:04 GMT

When testing 304 not modified etag using jmeter you need to add http cache manager and tick use cache control/expires headers.

This did kind of work for me however it looked like to was only doing it for every second request. The solution – update jemeter. Looks like there may be an issue in 3.1 of jmeter. I upgraded to version 4 of jmeter and alls good

REST

Sat, 03 Mar 2018 00:00:03 GMT

Made a small realisation this morning. I’ve read up a lot of REST and the background etc. As you know ReST stands for Representational State Transfer. I’ve read about the four levels or what is called the Richard Maturity Model

level 0 – being the swamp of Pox (no order and all chaos) level 1 – being resources ( /customer/, /order/1 etc) level 2 – being using verbs correctly (POST, PUT, GET etc) level 3 – being HATEOAS(hypertext and the Engine of Application State)

The big realisation I made as I code my latest example app was that as I was adding caching and eTags to my spring boot app I was adding the most important element of ReST. Understanding the levels above was important but more important is to know what it means to be an application at ReST.

An application at ReSt should not be making unnecessary calls to the database and this is where the cache comes in. An application at Rest should not be returning large payloads to the front end when its not necessary and this is where eTags come in. As long as you are doing GETs which are meant to be idempotent i.e. the dont change anything there should be little to no activity on the backend. The application is at ReST.

As soon as a POST is made which are not idempotent ie they do change something then something will be removed from the cache, you will see a database hit , evict and update cache but after that the system should be back at ReST.

There’s still a lot for me to learn in the area but hopefully getting there

Jest code coverage

Sat, 03 Mar 2018 00:00:02 GMT

I was experiencing a lot of frustration with coverage and jest this weekend. I would write tests but the coverage was not going up. Turns out theres a cache.

I tried adding

    –no-cache

and it would but as soon as I removed the flag I’d be back to square one. There is another option

    –clearCache

which worked a treat

docker volumes , backing up and restore

Sun, 11 Feb 2018 00:00:02 GMT

Its useful to be able to restore a database to an original state for integration/end to end testing.

Create a volume using the -v command which link the /var/lib/mysql folder in the container to my local folder /home/bob/dockerdata/mysql e.g.

    docker run -v /home/bob/dockerdata/mysql_sivtest:/var/lib/mysql -p3306:3306 –name sivtest_mysql -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=sivtest -d mysql

backup the data by navigating to /home/bob/docker/data and run

    sudo tar -cjf mysql_sivtest_archive mysql_sivtest

to restore the data you’ll have to stop the container , restore the data and start the container.

    docker stop sivtest_mysql
    sudo tar -xvf mysql_sivtest_archive
    docker start sivtest_mysql

This is working very well for me

vuejs and axios

Sun, 11 Feb 2018 00:00:01 GMT

When trying to mock ajax calls I tried moxios however I couldn’t get it to work. I had more success with axios mock adapter https://github.com/ctimmerm/axios-mock-adapter

to mock out a call

    import the modules

    import MockAdapter from 'axios-mock-adapter';
    import axios from 'axios'
    and an example would look like

    it('Company mounted 1 company', (done) =>{
        mock = new MockAdapter(axios); 
        mock.onGet('http://127.0.0.1:8082/company/').reply(200, [{"id":1,"name":"company name"}]);
        const wrapper = mount(Companies)
        setTimeout(() => {
        expect(wrapper.findAll('table tr.result').length).toEqual(1)
        done()       
        },0)
    })

The set timeout is important as I believe it gives the async call time and the done() is also important as the test will wait until you tell it you’re done.

vuejs and unit tests

Sun, 11 Feb 2018 00:00:00 GMT

Was getting a lot of issues doing unit tests getting errors around vuejs trying to pull in the router and other components. Discovered vue-test-utils which greatly simplifies a lot of the mocking https://vue-test-utils.vuejs.org/en/

You simply your components as follows

    const wrapper = mount(Companies)

You can print the html like so

    console.log(wrapper.html())

and expect like so

    expect(wrapper.html()).toContain('<div><h1>HOME</h1> <p>Welcome to Your Vue.js App</p></div>')

it also has some handy utils that simplify click events

    wrapper.find('#addcompany').trigger('click')

Router params can be mocked like so

    const $route = {
        params: { providerid: '1' },
    }

    const wrapper = mount(Benefits, {
        mocks: {$route}
    })

This library has been instrumental in getting my vuejs unit tests working

Docker Registry

Wed, 17 Jan 2018 00:00:00 GMT

mkdir registry_certs

openssl req -newkey rsa:4096 -nodes -sha256 \
-keyout registry_certs/domain.key -x509 -days 356 \
-out registry_certs/domain.cert

put domain.cert here on all clients
/etc/docker/certs.d//ca.cert

docker run -d -p 5000:5000 \
-v `pwd`/registry_certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.cert \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
–restart=always –name registry registry:2

docker pull hello-world
docker tag hello-world dockreg:5000/hello
docker push dockreg:5000/hello
docker tag hello-world dockreg:5000/hello:1
docker push dockreg:5000/hello:1

https://yourserver.com:5000/v2/_catalog https://dockreg:5000/v2/hello/tags/list

SOLR search POC

Sun, 17 Dec 2017 00:00:00 GMT

I wanted to set up a quick test to see if I could pass in a user created search lucene search string that would mean users could write their own queries. Seemed simple enough.

I tried using some guides on the web to get me started with spring data solr which was the first hurdle. None of the tutorials I found worked, possibly because of dependency issues. I ended up using spring initializr and specified web and solr. https://start.spring.io/

Next hurdle, I couldn’t see any ways for using spring data CRUD repo to create the query from a passed in string. The CRUD is fine for hard wired queries but not for what I Wanted. Figured I have to go a level down and use the solrclient.

Next issue was converting the solr results into POJOs. This is done via getBeans but I had issues with SOLR and its automatic schema creation. It guesses my strings were all multivalue. So I figured out a way of editing the schema manually and setting the multivalue to false and it works.

heres my repo https://github.com/rjgallac/solrtest

Portainer

Fri, 15 Dec 2017 00:00:00 GMT

A nice UI for docker.

docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer

http://127.0.0.1:9000/

SASS and vuejs

Sun, 10 Dec 2017 00:00:00 GMT

add this to build/webpack.base.conf.js

{
test: /\.s[a|c]ss$/,
loader: 'style!css!sass'
}

and in your component import yout scss

<style lang="scss" >
@import './component1.scss';
</style>

However I’m strongly considering the benefits of SASS when using vue components. SASS lets you break up your stylesheets into smaller composable chunks which is pretty much out of the box with vue. Variable substitution is still beneficial, there might be a way to do this in vue too. Time for some investigating me thinks

angular and docker

Fri, 08 Dec 2017 00:00:00 GMT

issuing the following three commands sets up angular through a docker container and serves it on http://localhost:4200/

docker run -it --rm -v "$PWD":/app -w /app node:9-alpine npm install @angular/cli
docker run -it --rm -v "$PWD":/app -w /app node:9-alpine node_modules/.bin/ng new my-app --directory ./
docker run -it --rm -p4200:4200 -v "$PWD":/app -w /app node:9-alpine node_modules/.bin/ng serve --host 0.0.0.0 --open

and lastly react through docker

Fri, 08 Dec 2017 00:00:00 GMT

docker run -it –rm -v “$PWD”:/app -w /app node:9-alpine npm install create-react-app
docker run -it –rm -v “$PWD”:/app -w /app node:9-alpine node_modules/.bin/create-react-app react-app
docker run -it –rm -v “$PWD/react-app”:/app -w /app node:9-alpine npm start

Closing leaves no existence of node/npm..

The nice thing about these in a development environment is ensuring everybody is using the same version of node/npm, the docker container ensures the project is built on the same platform i.e. linux for consistency, its pretty lightweight and I can through it away when finished.

Node , NPM and docker

Thu, 07 Dec 2017 00:00:00 GMT

A follow up post. Using the offical node image to install npm packages and run an express app.

docker run -it —rm -v “$PWD”:/app -w /app node:9-alpine npm init The -it takes you to the terminal to fill in the package info. The –rm will remove the container. Once complete you should see a package.json in the local directory.

docker run —rm -v “$PWD”:/app -w /app node:9-alpine npm install express body-parser cookie-parser multer —save The above will install the packages in the local directory.

docker run —rm -p8081:8081 -v “$PWD”:/app -w /app node:9-alpine node index.js The above will run the app and remove the container when finished.

The following is the index.js var express = require(‘express’); var app = express(); app.get(’/’, function (req, res) { res.send(‘Hello World’); }) var server = app.listen(8081, function () { var host = server.address().address var port = server.address().port console.log(“Example app listening at http://%s:%s”, host, port) })

Node and docker

Thu, 07 Dec 2017 00:00:00 GMT

I don’t tend to install node on my laptop as I don’t want to clutter my machine with a lot of different software. I like to try stuff out save it to github and forget about it for a few months. Docker lets me do this.

for example to run a quick node script

docker run -p8080:8080 -v "$PWD":/app node:9-alpine node app/console.js

console.js

console.log("Hello World!");

Another example – running a webserver.

docker run -p8080:8080 -v "$PWD":/app node:9-alpine node app/webserver.js

webserver.js

http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/html'});
res.end('Hello World!');
}).listen(8080);

Another test – docker , webpack and vuejs

Thu, 07 Dec 2017 00:00:00 GMT

I wondered if I could use docker to prepare, install and run a webpack server inside docker containers. Again without installing node/npm on my machine.

docker run -it —rm -v “$PWD”:/app -w /app node:9-alpine npm install vue-cli This will initialise the project in the current folder. You should see prompts and fill in the detail. docker run -it —rm -v “$PWD”:/app -w /app node:9-alpine node_modules/.bin/vue init webpack . This will create a default vue project for you in the current folder using the awesome vue cli..

docker run -it —rm -v “$PWD”:/app -w /app node:9-alpine npm i This will install the packages. You should see npm pull down all the related packages specified in the package.json

docker run -it —rm -p8080:8080 -v “$PWD”:/app -w /app node:9-alpine npm run start And this will run up the app on port 8080. Just go to localhost:8080

N.B, you might need to edit package.json and add –host 0.0.0.0 as an option to the webpack-dev-server command

as an added bonus you might want to build the project and package it up into a nginx docker image for running as the webpack dev server is not production ready. Simple create the following Dockerfile FROM nginx:1.13-alpine COPY dist /usr/share/nginx/html then docker build -t nodevue . then docker run —rm -p8080:80 nodevue then browser to localhost:8080 when you close there should be no evidence that node or nginx was ever on your machine 🙂

Camel and activemq

Sat, 28 Jan 2017 00:00:00 GMT

Camel does not seem to want to pick up the broker url when specified in spring boot properties file.

instead you have to add it to the url like so

activemq:queue:people.processpeople?trustAllPackages=true&brokerURL=“+activemqBrokerUrl

in addition it doesnt pick up the trustAllPackages properties so specify that there as well

Activemq Topics and Queues

Sat, 28 Jan 2017 00:00:00 GMT

When working on a quick proof of concept I was going to try to programmatically get camel to multicast to all listening microservices when I stumbled upon the subtle but important difference between queues and topics. Queues will distribute messages to one of the listening services whereas with topics messages will get distributed to all listening services.

https://github.com/rjgallac/robamq

Docker – clean up old images

Tue, 01 Nov 2016 00:00:00 GMT

If you do a > docker images

you might notice a lot of images with the name of none. I believe these are a load of out of date images. You can clean these up by executing the following on the command line

docker rmi $(docker images -f “dangling=true” -q)

docker install – linux

Tue, 01 Nov 2016 00:00:00 GMT

tall – linux Docker can be a pain to install. You can follow lots of install instructions and apt-get install or just follow execute this.

wget -qO- https://get.docker.com/ | sh also, to install docker-compose

sudo apt-get -y install python-pip sudo pip install docker-compose most of this taken from here

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-compose-on-ubuntu-14-04

I do recommend the digital ocean tutorials.

Freeeeeee certs from lets encrypt

Mon, 14 Dec 2015 22:12:03 GMT

Not tried this but intend to very soon. Get your free SSL certificates from lets encrypt. Seems like a good deal to me and something I see popping up in my RSS feeds very often. https://letsencrypt.org/.