Simplify Complexities

Testing the Production

An interesting read – https://techbeacon.com/app-dev-testing/test-production-yes-you-can-you-should

SQL Essential Queries

Find number of rows between two dates

SELECT COUNT(customer_id) FROM customer_table where created_at BETWEEN '2021-01-07 00:00:00' AND '2021-01-14 00:00:00';

List all columns of a table
```
show columns from customer_table
```
Get all tables
```
show tables
```
Get table schema
```
show create table customer_table;
```

How to add Bash script to cloudbuild deployment, job?

version 1

steps:- name: 'bash'  args: ['echo', 'I am running a bash command']

version 2

steps:
- name: 'gcr.io/cloud-builders/gcloud'  
- entrypoint: 'bash'  
- args:  
 - '-eEuo'  
 - 'pipefail'  
 - '-c'  
 - |-
   if (( $(date '+%-e') % 2 )); then  
       echo "today is an odd day"
    else
       echo "today is an odd day, with an even number"
    fi

- name: another example
        image: gcr.io/ubuntu:latest
        command: ["/bin/sh"]
        args: 
        - '-c'
        - |
          DB_PATH="mysql://${MASTER_DSN}"
          echo "hello world!"
        imagePullPolicy: Always    
        env:
        - name: MASTER_DSN

Another example

credit

steps:  
- name: 'gcr.io/cloud-builders/curl'  
entrypoint: 'bash'  
args:  
- '-c'  
- |  
  PET="$$(curl -s https://pets.doingdevops.com/pet_flaky --max-time 10)"  
  while [ "$$PET" == "ERROR" ] ; do  
      echo "Error: API failed to respond with a pet! Try again..."  
      PET="$$(curl -s https://pets.doingdevops.com/pet_flaky --max-time 10)"  
  done
    
  echo "Success! $${PET}"

References

How Golang DB migration tool work?

migrate

Database migrations written in Go. Use as CLI or import as library.

Migrate reads migrations from sources and applies them in correct order to a database.

The migration tool uses a schema_migrations table. One table exists for one DB.

mysql> select * from schema_migrations;
+---------+-------+
| version | dirty |
+---------+-------+
|      33 |     0 |
+---------+-------+
1 row in set (0.00 sec)

The command to apply all migrations to a DB would check-in schema_migrations table and pick migrations after the version (for the current case – 34 and onwards).

migration up

migration down 33

The down migration executes the down file for version 33 and updates the schema_migrations table.

Code

Migration files are read from 1..N if no version is specified.

migrate/migrate.go

// read reads either up or down migrations from source `from` to `to`.
// Each migration is then written to the ret channel.
// If an error occurs during reading, that error is written to the ret channel, too.
// Once read is done reading it will close the ret channel.
func (m *Migrate) read(from int, to int, ret chan<- interface{})

Troubleshooting

no migration found for version
Explanation: The migration version was updated in schema_registry marked dirty but the migration files are deleted/ missing. So maybe the previous migration was done with N and then Nth files were deleted. Now up or down would not work as the schema_registry has version N.
So the solution is either to bring back Nth migrations or force the schema_regsitry version to N-1.

migrate force N-1

error: Dirty database version 32. Fix and force version.
Explanation: The 33rd migration file was corrupted or had an error.

mysql> select * from schema_migrations;
+---------+-------+
| version | dirty |
+---------+-------+
|      32 |     1 |
+---------+-------+
1 row in set (0.01 sec)

Manually undo the changes of version 33 (refer to 33 down migration file) and run migrate force 32

References

Written with StackEdit.

Tests Timing Out? Check Go Tests Parallelism

Go Tests Parallelism

The command go test ./... lists all packages in the current directory, builds tests executable for each package, and runs them. The run part is interesting.

By default, the number of packages executed in parallel is equal to the value of GOMAXPROCS variable. On my laptop, it is 12 which is the number of CPU cores available.

The other parallelism is inside a package. If a test uses t.Parallel, go test -parallel <num of tests> determines the parallelism.

Why is it important to tune the degree of parallelism

Too many tests running in parallel would time out/crash for want of resources or it could expose a bug. There might be a leakage in the socket, buffer, and so on. If your machine is busy and you see tests timing out often, do try to tune the parallelism.

Tip

Always use go test ./... because it caches tests result for a package. So you save time, CPU in upcoming runs given your changes are not spreading to all packages.

References

go help test
go help testflag
go help build

Credit to StackOverflow
GOMAXPROCS

package main

import (
    "runtime"
    "fmt"
)

func getGOMAXPROCS() int {
    return runtime.GOMAXPROCS(0)
}

func main() {
    fmt.Printf("GOMAXPROCS is %d\n", getGOMAXPROCS())
}

Written with StackEdit.

What is SemVer or Semantic Versioning?

In the Linux world, kernel releases follow MAJOR.MINOR.PATCH version format.

Semver is a similar concept. It advises to follow a simple set of rules:

Major version is changed for incompatible changes.
Minor versions are changed for new functionality
Patch version is changed after bug fixes.

Versions are always incremented and reset to zero:

Patch version is set to zero if the Minor version is changed
Patch and Minor versions are set to zero if the Major version is changed.

References

https://semver.org/

golang: NewReader vs NewBufferString

NewReader

func bytes.NewReader(b []byte) *bytes.Reader

bytes.NewReader on pkg.go.dev

NewReader returns a new Reader reading from b.

NewBufferString

func bytes.NewBufferString(s string) *bytes.Buffer

bytes.NewBufferString on pkg.go.dev

NewBufferString creates and initializes a new Buffer using string s as its initial contents. It is intended to prepare a buffer to read an existing string.

Difference

// A Reader implements the io.Reader, io.ReaderAt, io.WriterTo, io.Seeker,
// io.ByteScanner, and io.RuneScanner interfaces by reading from
// a byte slice.
// Unlike a Buffer, a Reader is read-only and supports seeking.
// The zero value for Reader operates like a Reader of an empty slice.

The bufferString creates a mutable buffer.

go: cannot find main module, but found .git config to create a module there, run: go mod

go: cannot find main module, but found .git/config to create a module there, run: go mod

This problem occurred in a monorepo opened in VSCode. The repo has many services placed in isolated directories with respective go.mod files.

I realized from various sources that the problem is from two places:

Move the mono repo under $GOPATH/src.

The ~/.zshrc or ~/.bashrc must be as follows:

export GOPATH=$HOME/go/
export GOROOT=/usr/local/go

export PATH="$GOROOT/bin:$PATH"
export PATH="$PATH:$GOPATH/bin"

Do not add any other variable that affects gopls such as GO111MODULE=on.

Written with StackEdit.

References

How to Add Multiple Directories in GOPATH with VSCode?

Since go versions keep moving up, maintaining GOPATH inside a Go installation path is a pain.

export GOPATH=/home/a-user/go/1.14.9

ls /home/a-user/go/1.14.0/src 
bitbucket.org github.com    go.uber.org   golang.org    gopkg.in

Adding multiple directories in GOPATH

Create a directory e.g. for work
```
mkdir $HOME/mywork
```
GOPATH assumes a structure in the directory, so create three directories inside.
```
ls $HOME/mywork
bin pkg src
```
You will do all your personal development work inside $HOME/mywork/src.
```
ls $HOME/mywork/src
project-01
```

Append .bashrc or .zshrc (append is a must, it won’t work otherwise because GOPATH is preset by goenv)

# keep it at the end of the file
export GOENV_ROOT="$HOME/.goenv"
export PATH="$GOENV_ROOT/bin:$PATH"
eval "$(goenv init -)"

export PATH="$GOROOT/bin:$PATH"
export PATH="$PATH:$GOPATH/bin"
export PATH="$PATH:$HOME/bin"

# add personal directory
export GOPATH="$GOPATH:$HOME/mywork/"

Open the directory $HOME/mywork/src/project-01 in VSCode workspace.
If you have the VSCode extension for Go installed, restart VSCode.

If go to implementation doesn’t work, ensure that go is installed properly with src directory.

ls $HOME/go/1.14.0/ 
bin pkg src

That’s all folks!

References

https://www.digitalocean.com/community/tutorials/understanding-the-gopath
go help gopath

VSCode Plugins for Golang Development

This is a minimal list of Go development extensions in my opinion.

Share this:

Share this:

References

Share this:

migrate

Code

Troubleshooting

References

Share this:

Go Tests Parallelism

Why is it important to tune the degree of parallelism

Tip

References

Share this:

References

Share this:

NewReader

NewBufferString

Difference

Share this:

References

Share this:

Adding multiple directories in GOPATH

References

Share this:

Share this: