Taogen's Blog

Stay hungry stay foolish.

Frontend responsibilities

  • Layout and style of web pages.
  • Page redirection.
  • Event handling.
  • Form validation and submission.
  • Call API and render the data. Note data shouldn’t be converted and formatted in the frontend.

Backend responsibilities

  • Design data model.
  • Validation and conversion of parameters for HTTP requests.
  • Business logic processing.
  • Build response data with the correct structure and format.

Event Handling

  1. .click(handler) - event handling for specified elements
$("your_selector").click(function(event) {
// do something
});
  1. .on( events [, selector ] [, data ], handler ) - event handling for dynamic elements
$(document).on("click","your_selector", function (event) {
// do something
});

if you know the particular node you’re adding dynamic elements to, you could specify it instead of the document.

$("parent_selector").on("click","your_selector", function (event) {
// do something
});

Passing data to the handler

$(document).on("click", "#your_div", {name: "Jack"}, handler);

function handler(event){
console.log(event.data.name);
}

Deprecated API

As of jQuery 3.0, .bind() and .delegate() have been deprecated. It was superseded by the .on() method for attaching event handlers to a document since jQuery 1.7, so its use was already discouraged.

Get the element that triggered the event

$("your_selector").click(function(event) {
// get the element
console.log(event.target);
console.log(this);

// get the element id
console.log(this.id);
console.log(event.target.id);
console.log($(this).attr('id'));

// Get jQuery object by element
console.log($(this).html());
console.log($(event.target).html());
});

Get the element that triggered the event

  • event.target
  • this

Note: event.target equals this, and equals document.getElementById("your_selector")

Get the element id

  • this.id
  • event.target.id
  • $(this).attr('id')

Get jQuery object by element

  • $(this)
  • $(event.target)

jQuery Events

Form Events

  • .blur(handler)
  • .change(handler)
  • .focus(handler)
  • .focusin(handler)
  • .focusout(handler)
  • .select(handler)
  • .submit(handler)

Keyboard Events

  • .keydown(handler)
  • .keypress(handler)
  • .keyup(handler)

Mouse Events

  • .click(handler)
  • .contextmenu(handler). The contextmenu event is sent to an element when the right button of the mouse is clicked on it, but before the context menu is displayed.
  • .dblclick(handler). The dblclick event is sent to an element when the element is double-clicked.
  • .hover(handlerIn, handlerOut)
  • .mousedown(handler)
  • .mouseenter(handler)
  • .mouseleave(handler)
  • .mousemove(handler)
  • .mouseout(handler)
  • .mouseover(handler)
  • .mouseup(handler)
  • .toggle(handler, handler). Bind two or more handlers to the matched elements, to be executed on alternate clicks.

Kill Long Running Queries

Query information_schema.innodb_trx

  • trx_mysql_thread_id: thread ID
SELECT trx_mysql_thread_id, trx_state, trx_started, trx_query
FROM information_schema.innodb_trx
where trx_state = "RUNNING" and trx_query like "%SELECT%"
ORDER BY `trx_started`;

kill {trx_mysql_thread_id};

-- To check again by query information_schema.innodb_trx.
-- Sometimes need kill two times.

After killed the query thread. The client receive a error message 2013 - Lost connection to server during query.

Query information_schema.processlist

  • id: thread ID
  • time: cost time in seconds
  • state: Sending Data, executing
SELECT * 
FROM information_schema.processlist
WHERE
info like '%SELECT%'
order by `time` desc;

kill {ID};

-- To check again by query information_schema.processlist.
-- Sometimes need kill two times.

Kill Locked SQL Statements

An Example of Locked SQL Statements

Create a table for test

CREATE TABLE `t_user` (
`id` int NOT NULL AUTO_INCREMENT,
`name` varchar(100) DEFAULT NULL,
`age` int DEFAULT NULL,
PRIMARY KEY (`id`)
);
INSERT INTO `t_user` (`id`, `name`, `age`) VALUES (1, 'Jack', 20);
INSERT INTO `t_user` (`id`, `name`, `age`) VALUES (2, 'Tom', 30);
INSERT INTO `t_user` (`id`, `name`, `age`) VALUES (3, 'John', 22);

Executing the SQL statement 1 to lock the table

SET autocommit = 0;  
START TRANSACTION;
update t_user set age = 2;

Executing the SQL statement 2, which will wait for the lock.

-- Temporarily set the lock wait timeout to 10 minutes. By default, it is 50 seconds. We need a longer timeout to find out the locked SQL statements.
SET SESSION innodb_lock_wait_timeout = 600;
update t_user set age = 3;

If waiting for lock is timeout (by default, it is 50 seconds), SQL statement 2 will receive a error message

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

After finished the lock test, you can COMMIT or ROLLBACK the transaction of SQL statement 1 and set autocommit to 1.

COMMIT;
ROLLBACK;
SET autocommit = 1;

Get thread IDs and SQL statements of lock-holding and lock-waiting executing SQLs

Query whether some SQL statement threads are waiting for lock

SELECT *
FROM `information_schema`.`innodb_trx`
where trx_state = "LOCK WAIT"
ORDER BY `trx_started`;
  • trx_state: LOCK WAIT
  • trx_started: 2022-10-21 14:13:38
  • trx_mysql_thread_id: 17 (thread ID)
  • trx_query: the executing SQL statement
  • trx_requested_lock_id: 1207867061312:1021:4:2:1207832760632
  • trx_wait_started: 2022-10-21 14:13:38

Query whether some SQL statement treads are running with lock

SELECT *
FROM `information_schema`.`innodb_trx`
where trx_state = "RUNNING" and trx_tables_locked > 0 and trx_rows_locked > 0
ORDER BY `trx_started`;
  • trx_state: RUNNING
  • trx_started: 2022-10-21 14:09:57
  • trx_mysql_thread_id: 16 (thread ID)
  • trx_query: the executing SQL statement
  • trx_tables_locked: 1
  • trx_lock_structs: 2
  • trx_rows_locked: 3
  • trx_row_modified: 2

Get More Locked Information

Query what table is locked

show open tables where in_use > 0;

Query Lock Information

Get transaction IDs and real Thread IDs of lock-holding and lock-waiting executing SQL.

SHOW ENGINE INNODB STATUS;

To find “TRANSACTION xxx, ACTIVE xxx sec” in the result text

lock-holding transaction Information

---TRANSACTION 580438, ACTIVE 1862 sec
2 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 2
MySQL thread id 16, OS thread handle 20276, query id 278 localhost ::1 root

lock-waiting transaction Information

---TRANSACTION 580444, ACTIVE 36 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 1136, 1 row lock(s)
MySQL thread id 17, OS thread handle 16228, query id 454 localhost ::1 root updating
update t_user set age = 3
------- TRX HAS BEEN WAITING 36 SEC FOR THIS LOCK TO BE GRANTED

There are only one lock-holding transaction and one lock-waiting transaction. So we can guess that thread 16 block thread 17, or that transaction 580438 block transaction 580444.

Check lock dependency - what blocks what

MySQL 5.x

SELECT * FROM INFORMATION_SCHEMA.INNODB_LOCKS;

MySQL 8

Query lock dependency

SELECT * FROM performance_schema.data_lock_waits;
  • BLOCKING_ENGINE_TRANSACTION_ID: lock-holding transaction ID
  • REQUESTING_ENGINE_TRANSACTION_ID: lock-waiting transaction ID

The result is BLOCKING_ENGINE_TRANSACTION_ID blocked REQUESTING_ENGINE_TRANSACTION_ID. We can confirm that transaction 580438 blocked transaction 580444. You can get the real thread IDs from the result of SHOW ENGINE INNODB STATUS;. Therefore, we can confirm that thread 16 blocked thread 17.

Query lock-holding and lock-waiting transaction information

SELECT * FROM performance_schema.data_locks;
  • ENGINE_TRANSATION_ID: transation_id in SHOW ENGINE INNODB STATUS;
  • OBJECT_NAME: table name
  • LOCK_STATUS: “WATING”/“GRANT”

Kill the Locked Tread

kill {thread_ID};
kill 16;

References

Hardware Information

Hardware Information

sudo lshw
sudo lshw -short
sudo lshw -html > lshw.html

CPU

CPU Information

CPU Information

lscpu

CPU Architecture

arch

CPU Usage

vmstat

echo "CPU Usage: "$[100-$(vmstat 1 2|tail -1|awk '{print $15}')]"%"

/proc/stat

grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print "CPU Usage: " usage "%"}'
cat /proc/stat |grep cpu |tail -1|awk '{print ($5*100)/($2+$3+$4+$5+$6+$7+$8+$9+$10)}'|awk '{print "CPU Usage: " 100-$1 "%"}'

top

top -bn2 | grep '%Cpu' | tail -1 | grep -P '(....|...) id,'|awk '{print "CPU Usage: " 100-$8 "%"}'

Disk

Disk Information

Block Devices Information

lsblk
lsblk -a

Disk Usage

df -h

Folder Disk Space Usage

# all subdirectories size and total size
du -h <folder_name>
# -s total size of a directory
du -sh <folder_name>
# -a all files size, subdirectories size and total size
du -ah <folder_name>
# -c add total usage to the last line
du -ch <folder_name>

File Disk Space Usage

ls -lh .
du -ah <folder_name>

Memory

Memory Information

sudo dmidecode -t memory

Memory Usage

free -h
# the percentage of memory in use of user processes
free | grep Mem | awk '{print $3/$2 * 100.0 "%"}'
# the real percentage of memory in use included OS memory. available / total memory.
# -m: Display the amount of memory in megabytes.
# N: your server total memory in GB.
free -m | grep Mem | awk '{print (N * 1024 - $7)/(N * 1024) * 100.0 "%"}'

Network

Network Traffic

Total network traffic

nload
speedometer -t eth0
bmon

traffic by socket

iftop
iftop -F 192.168.0.1/16

traffic by process ID (PID)

nethogs

Network Speed

speedtest-cli

# install speedtest-cli
sudo apt install speedtest-cli
# or
sudo yum install speedtest-cli

# run speed test
speedtest-cli
speedtest-cli --simple
# or
speedtest
speedtest --simple

IP Address

LAN/private IP address

ifconfig
# or
hostname -I
# or
ip route get 1.2.3.4 | awk '{print $7}'

Public IP address

curl ifconfig.me
curl ipinfo.io/ip

Public IP Information

curl ipinfo.io

Check Server Open Ports from Local

nmap

Nmap adapts its techniques to use the best available methods using the current privilege level, unless you explicitly request something different. The things that Nmap needs root (or sudo) privilege for on Linux are: Sniffing network traffic with libpcap. Sending raw network traffic.

# fast scan top 100 open parts (-F)
sudo nmap --min-hostgroup 100 -sS -n -T4 -F <Target_IP>

# fast scan top 100 open parts (-F) when ping is disabled. Add -Pn.
sudo nmap --min-hostgroup 100 -sS -n -T4 -F -Pn <Target_IP>

# fast scan top 1000 ports (-top-ports)
sudo nmap --min-hostgroup 100 -sS -n -T4 -top-ports 1000 <Target_IP>

# fast scan a range of ports 20-80
sudo nmap --min-hostgroup 100 -sS -n -T4 -p20-80 <Target_IP>

# fast scan some specific ports 80,8080
sudo nmap --min-hostgroup 100 -sS -n -T4 -p80,8080 <Target_IP>

# scan ports are listening for TCP connections
sudo nmap -sT -p- <ip>

# scan for UDP ports use -sU instead of -sT
sudo nmap -sU -p- <ip>
  • Scan method
    • -sS: (TCP SYN scan) - SYN scan is the default and most popular scan option for good reasons. It can be performed quickly, scanning thousands of ports per second on a fast network not hampered by restrictive firewalls. It is also relatively unobtrusive and stealthy since it never completes TCP connections.
    • -sT: (TCP connect scan)
    • -sU: (UDP scans)
  • Faster scan
    • -n: (No DNS resolution) - Tells Nmap to never do reverse DNS resolution on the active IP addresses it finds. Since DNS can be slow even with Nmap’s built-in parallel stub resolver, this option can slash scanning times.
    • -T: Set a timing template
      • -T4: prohibits the dynamic scan delay from exceeding 10 ms for TCP ports. Note that a faster speed can be less accurate if either the connection or the computer at the other end can’t handle it, and is more likely to trigger firewalls or IDSs.
      • -T5: prohibits the dynamic scan delay from exceeding 5 ms for TCP ports.
    • --min-hostgroup numhosts: (Adjust parallel scan group sizes) Nmap has the ability to port scan or version scan multiple hosts in parallel.
  • Speicify ports
    • -F: (Fast (limited port) scan) Scan fewer ports than the default scan. Normally Nmap scans the most common 1,000 ports for each scanned protocol. With -F, this is reduced to 100.
    • –top-ports [number]: to scan the top [number] most common ports.
    • -p-: to scan 65535 TCP ports. Scanning all ports is too slow.
    • -p<from>-<to>: to scan a range of ports.
    • -p<port1>,<port2>: to scan specific ports.
    • -p<from>-<to>,<port1>,<port2>: to scan multiple ports.
  • Other
    • -Pn: (No ping) This option skips the host discovery stage altogether. When ping is disabled on target server, we need add -Pn to skip ping.

States of nmap

  • Accessible states
    • open: An application is actively accepting TCP connections, UDP datagrams or SCTP associations on this port.
    • closed: A closed port is accessible (it receives and responds to Nmap probe packets), but there is no application listening on it.
    • unfiltered: The unfiltered state means that a port is accessible, but Nmap is unable to determine whether it is open or closed.
  • Inaccessible states
    • filtered: Nmap cannot determine whether the port is open because packet filtering prevents its probes from reaching the port. The filtering could be from a dedicated firewall device, router rules, or host-based firewall software. These ports frustrate attackers because they provide so little information.
    • open|filtered: Nmap places ports in this state when it is unable to determine whether a port is open or filtered.
    • closed|filtered: This state is used when Nmap is unable to determine whether a port is closed or filtered. It is only used for the IP ID idle scan.

Operating System Information

Operating System

Linux Distro name and version

cat /etc/os-release
cat /etc/*-release
# or
lsb_release -a
# or
hostnamectl

Linux kernel version

uname -a
uname -r
uname -mrs
# or
cat /proc/version

System hostname and related settings

hostnamectl

Start date and time of operating system

uptime -s
uptime
# start time of the pid=1 proccess
ps -p 1 -o lstart

Environment Variables

Environment variables

env
# or
printenv

PATH

echo -e ${PATH//:/\\n}

Processes and Port

Processes and Port Management

View Processes

top
ps -ef
ps aux

View listening ports

lsof

# lsof
sudo lsof -i -P -n | grep LISTEN
sudo lsof -i -P -n | grep 'IPv4.*LISTEN'
  • -i: select IPv4 or IPv6 files
  • -P: no port names
  • -n: no host names
COMMAND       PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
redis-ser 108626 root 6u IPv4 342240 0t0 TCP 127.0.0.1:6379 (LISTEN)
java 1997058 root 433u IPv6 9877486 0t0 TCP *:58080 (LISTEN)
  • NAME = *:[port]: expose port on all network.
  • NAME = [127.0.0.1 or localhost]:[port]: expose port on local network.
# netstat
sudo netstat -tulpn | grep LISTEN
# ss
sudo ss -tulpn | grep LISTEN
# nmap
sudo nmap -sTU -O IP-address-Here

Kill a Process

kill <PID>
kill -9 <PID>

Kills a process by searching for its name

pkill -9 -f YOUR_PROCESS_NAME
# or
pgrep -f YOUR_PROCESS_NAME | xargs kill -9
# or
ps -ef | grep YOUR_PROCESS_NAME | awk '{print $2}' | head -1 | xargs kill -9
# or
ps -ef | grep YOUR_PROCESS_NAME | awk '{print $2}' | tail -1 | xargs kill -9

Kill a process by port

lsof -t -i:port | xargs kill

Process Information

Process start time

ps -p <pid> -o lstart,etime

Process basic information

ps -p <pid> -o pid,cmd,lstart,etime,pcpu,pmem,rss,thcount
  • lstart: accurate start time. e.g. Thu Nov 14 13:42:17 2019
  • start: start time of today or date. e.g. 13:42:17 or Nov 14
  • etime: elapsed time since the process was started, in the form [[DD-]hh:]mm:ss.
  • etimes: elapsed time since the process was started, in seconds
  • pid: process ID.
  • cmd: simple name of executable
  • pcpu: %CPU
  • pmem: %MEM
  • rss: memory use in bytes
  • thcount: thread count

Software

Architecture of software

cat `which {your_software}` | file -

References

[1] 10 Commands to Collect System and Hardware Info in Linux

[2] bash shell configuration files

[3] Configuration Files in Linux

MySQL Full-Text Index

MySQL 5.7.6 supports full-text index for Chinese, Japanese, and Korean (CJK).

The built-in MySQL full-text parser uses the white space between words as a delimiter to determine where words begin and end, which is a limitation when working with ideographic languages that do not use word delimiters. To address this limitation, MySQL provides an ngram full-text parser that supports Chinese, Japanese, and Korean (CJK). The ngram full-text parser is supported for use with InnoDB and MyISAM.

Create a Full-Text Index

Creating a FULLTEXT Index that Uses the ngram Parser

CREATE FULLTEXT INDEX content_fulltext ON table_name(column1, column2,...) with parser ngram;

Full-Text Searches

Natural Language Full-Text Searches

SELECT COUNT(*) FROM articles
WHERE MATCH (title,body) AGAINST ('database' IN NATURAL LANGUAGE MODE);
-- or
SELECT COUNT(*) FROM articles
WHERE MATCH (title,body) AGAINST ('database');

the columns named in the MATCH() function (title and body) are the same as those named in the definition of the article table’s FULLTEXT index. To search the title or body separately, you would create separate FULLTEXT indexes for each column.

Boolean Full-Text Searches

-- retrieves all the rows that contain the word “MySQL” but that do not contain the word “YourSQL”
SELECT * FROM articles WHERE MATCH (title,body)
AGAINST ('+MySQL -YourSQL' IN BOOLEAN MODE);
  • + stands for AND
  • - stands for NOT
  • [no operator] implies OR. The word is optional, but the rows that contain it are rated higher.
  • “”. A phrase that is enclosed within double quote (") characters matches only rows that contain the phrase literally, as it was typed.

References

[1] ngram Full-Text Parser - MySQL 5.7 Documentation

[2] MySQL 全文索引

[3] MySql5.6全文索引 及 5.7 中文索引插件ngram

A column should represent only one dimension/concept/meaning

Why:

  1. It creates confusion. It makes your code complex and hard to understand.

For example, status column should only represent enable or disable. If you want to represent if it’s formal, you can add a column formal.

Multiple groups of the same structure columns should be put into another separate table

Why:

  1. It may cause business logic process to become more complex.
  2. low scalability. It’s hard to add a new type of group of columns and add new columns for each group.

How:

report (id, name, s1_content, s1_status, s1_submit_time, s2_content, s2_status, s2_submit_time,...)

=>

report (id, name)

report_section (id, report_id, content, status, submit_time)

Moving TEXT/BLOB column to a separate table

Why

If a table contains string columns such as name and address, but many queries do not retrieve those columns, consider splitting the string columns into a separate table and using join queries with a foreign key when necessary. When MySQL retrieves any value from a row, it reads a data block containing all the columns of that row (and possibly other adjacent rows). Keeping each row small, with only the most frequently used columns, allows more rows to fit in each data block. Such compact tables reduce disk I/O and memory usage for common queries.

MySQL you are discouraged from keeping TEXT data (and BLOB, as written elsewhere) in tables frequently searched.

References

[1] MySQL Documentation Chapter 8 Optimization

[2] MySQL Table with TEXT column - Stack Overflow

Basic Selectors

Select All

$("*")

Select by element id

$("#id")  

Select by element class name

$(".className")  

Select by element tag name

$("div")  

Select by multiple condition union

$("selector1,selector2,...,selectorN ")

Select by tag name and class name

$('div.my-class')

Hierarchical Selectors

Select Children and Descendant

$("form > input") // all direct children
$("table td") // select all descendant

Select After

$("label + input") // select all input after label

Select all Siblings

$("#prev ~ div") // select all siblings

Selector Filters

Filter by Position

$("tr:first")
$("tr:last")
$("tr:even")
$("tr:odd")
$("tr:eq(1)")
$("tr:gt(1)") // greater than
$("tr:lt(3)") // less than

Filter by Content

// :contains
$("div:contains(hello)") // select all div contains text 'hello'

// :empty
$("td:empty") // select all td text is empty

// :has()
$("div:has(p)") // select all div has <p> element

// :parent()
$("div :parent(p)") // select all div contained by <p> element

Filter by Attribute

// [attr]
$("div[id]") // all <div> have id attr

// [attr="value"]
$("div[id=div1]")

// [attr!="value"] not equal
$("div[id!=div1]")

// [attr^="value"] attr value start with
$("div[id^=test]")] // all div, 'id' attr value start with 'test'

// [att|="value"] attr value equals or start with
$("div[id|=test]")]

// [attr$="value"] attr value end with
$("div[id$=test]") // all div, 'id' attr value end with 'test'

// [attr*="value"] attr value contains
$("div[id*=test]") // all div, 'id' attr value contains 'test'

// [name~="value"] attr value split with whitespace. splited values contains
$( "input[name~='man']" )

Filter by attribute and class name

$('.myclass[reference="12345"]')
$("input[reference=12345].myclass")

Filter by Visibility

// :hidden
$("input:hidden")

// :visible
$("input:visible")

Select Child

// :nth-child
$("ul li:nth-child(2)") // match ul child li second

// :first-child
$("ul li:first-child")

// :last-child
$("ul li:last-child")

// :only-child
$("ul li:only-child") // match ul only contain one li child

Select Form

Filter by element name

:button	//Selects all button elements and elements of type button.
:input //Selects all input, textarea, select and button elements.

Filter by input element type

:checkbox
:file
:image
:text
:password
:radio
:reset
:submit

Filter by status of form element

:checked 	//Matches all elements that are checked or selected.
:selected
$("option:selected[value=1]")
:focus //Selects element if it is currently focused.
:disabled
:enabled //Selects all elements that are enabled.

:Not

// :not()
$("input:not(:checked)")
$("#form1 input:not('#id')")

Others

:header // select all header elements, such as <H1>,<H2>
:animated

Filter Methods

Filter Methods

filter("p") // get all <p> elements in jquery dom set
$("li").has("span") //get all li has child element 'span'
not("p, .name") // get all not <p> and class=name elements in jquery dom set

Position Method

first()
last()
eq("") // index begin with 0

Hierarchical Methods

Select parent or ancestors

parent() // one parent
parents() // all ancestors
$("span").parents("ul"); // all ancestors
$("span").parentsUntil("div") // returns all ancestor elements between two given arguments

Select children or descendants

children("") 
children()
find(""); // find descendants
// the find() is similar to the children() method,but childern() only find children not descendants.

Select siblings

siblings() 
siblings("") // get other dom of same level, don't contain self
next()
nextAll()
nextUntil()
prev()
prevAll()
prevUntil()

References

[1] Selectors - jQuery Documentation

Common SQL Optimization

Select only need columns

Convert a complex SQL statement to multiple simple SQL statements

Convert queries to use indexes. Using indexes operators: =, >, <, >=, <=, BETWEEN, IN. Not using indexes operators: LIKE, <>, NOT IN. Using MySQL built-in functions on column make query cannot use index.

NOT IN

Using LEFT JOIN or NOT EXISTS to replace NOT IN

A NOT IN query example

select * 
from table_a
where table_a.id is not in (
select aid from table_b where table_b.status = 1)
-- or
select *
from table_a
where table_a.id is not in (1,2,3, ...)

Use NOT EXISTS

select * 
from table_a
where not exists (
select * from table_b where table_a.id = table_b.aid and table_b.status = 1
)

Use LEFT JOIN

select *
from table_a
left join table_b on table_a.id=table_b.aid and table_b.status = 1
where table_b.id is null;

LIKE

Using Full-text search MATCH ... AGAINST... or search engine replace LIKE

OR

Using IN, or UNION, or AND NOT to replace OR

ORs are notoriously bad performers, because it splinters the execution path. The UNION alleviates that splintering, and the combines the two results sets. That said, IN is preferable to ORs because though being logically the same, the execution of IN is generally more optimized.

The OR query example 1:

select *
from {table_name}
where name = "Tom" or name = "Jack"

Using IN

select *
from {table_name}
where name in ("Tom", "Jack")

The OR query example 2:

select *
from {table_name}
where name like "T%" or name = "J%"

Using UNION

select *
from {table_name}
where name like "T%"
union
select *
from {table_name}
where name like "J%"

The OR query example 3:

select *
from {table_name}
where {col1} = "a" or {col2} = "b"

Using AND NOT

select *
from {table_name}
where {col1} = "a" and {col2} != "b";
select *
from {table_name}
where {col2} = "b";

Built-in Functions

Avoid using functions in predicates

Use BETWEEN to replace DATE(), DATE_FORMAT()

A DATE_FORMAT example:

DATE_FORMAT(`date`, "%Y-%m") = "2022-07"
YEAR(`date`) = "2022" and MONTH(`date`) = "7"

Use BETWEEN

`date` BETWEEN '2022-07-01' and '2022-07-31'

Convert function to use on right side

WHERE f(g(h(column))) > value
WHERE column > H(G(F(value)))

Page Query

Basic Optimization

  • Query only the necessary columns instead of select *
  • Query part of the TEXT column by SUBSTR(column, start, length) if it could be.

High Offset Page Query Improvement

Late row lookups: Inner join ids instead of select *

  • Inner join is first to find offset + size rows id, then find all rows data by id set
  • select * is just to find offset + size rows data.

select *

select *
from {table_name} as a
where ...
limit {x}, 10

Inner join ids

select *
from {table_name} as a
inner join (
select id
from {table_name}
where ...
order by id
limit {x}, 10
) as b on a.id=b.id;

In most cases, the inner join has a 5x performance improvement, but in some cases there is no difference in performance.

The performance gap is more evident for large row-width tables and queries with large offset.

Next Page Query Improvement

Specify start row id for next page query

first page query

select *
from {table_name} as a
inner join (
select id
from {table_name}
where ...
order by id
limit x, 10
) as b on a.id=b.id;

next page query

select *
from {table_name} as a
inner join (
select id
from {table_name}
where id > {last_page_max_id} and ...
order by id
limit 10
) as b on a.id=b.id;

Cache Page Query

  • Caching total row number for the same condition queries.
  • Caching every page start row id value and end row id value for the same condition queries.

References

General

Optimizing SELECT Statements - MySQL 8.0 Reference Manual

High Offset

MySQL ORDER BY / LIMIT performance: late row lookups

OR

Optimize MySQL COUNT (*) query

jdbcType

The jdbcType is a property for mapping Java object field values to SQL column values.

You need to specify the jdbcType when passing null values for parameters. Some databases need to know the value’s type even if the value itself is NULL. The JDBC specification itself that requires the type to be specified.

  • preparedStatement.setNull(int parameterIndex, int sqlType)

Most of the times you don’t need to specify the jdbcType as MyBatis is smart enough to figure out the type from the objects you are working with. But if you send your parameters to the MyBatis statement inside a HashMap, for example, and one of the parameters is null, MyBatis won’t be able to determine the type of the parameter by looking at the HashMap because the HashMap is just a generic container and null itself carries no type information. At that point it would be o good idea to provide the jdbcType so that switching the database implementation later on does not cause any issues with null values.

JDBC Types

  • BIT, BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT
  • FLOAT, REAL, DOUBLE, NUMERIC, DECIMAL
  • CHAR, VARCHAR, NVARCHAR, LONGVARCHAR, NCHAR
  • BINARY, VARBINARY, LONGVARBINARY, BLOB, CLOB, NCLOB
  • TIMESTAMP, DATE, TIME
  • OTHER, UNDEFINED, NULL, CURSOR, ARRAY

MySQL and Java DB currently do not support the ARRAY SQL data type.

Update default mappings of Java values to column values

Convert java.util.Date objects to date strings

The java.util.Date objects will default convert to datetime strings (TIMESTAMP JDBC type) like 2022-08-18 11:12:05.545. If you want to convert a Date object value to a date string (DATE JDBC type) like 2022-08-18, you can specify the jdbcType to DATE.

javaType

The javaType is a property for mapping result set column values to Java field values.

MyBatis can usually figure out the type if you’re mapping to a JavaBean. However, if you are mapping to a HashMap, then you should specify the javaType explicitly to ensure the desired behavior.

typeHandler

typeHandler can convert column values to Java values and verse vice.

Convert between the Java Enum type value and the column value

<resultMap id="BaseResultMap" type="com.demo.Task">
...
<result column="category" typeHandler="com.demo.typehandler.CategoryEnumTypeHandler"
property="category"/>
</resultMap>
#{category,typeHandler=com.demo.typehandler.CategoryEnumTypeHandler}
public class TaskCategoryEnumTypeHandler extends BaseTypeHandler<TaskTypeCategory> {

@Override
public void setNonNullParameter(PreparedStatement preparedStatement, int i, TaskTypeCategory taskTypeCategory, JdbcType jdbcType) throws SQLException {
preparedStatement.setString(i, taskTypeCategory.getName());
}

@Override
public TaskTypeCategory getNullableResult(ResultSet resultSet, String columnName) throws SQLException {
return TaskTypeCategory.fromDatabaseValue(resultSet.getString(columnName));
}

@Override
public TaskTypeCategory getNullableResult(ResultSet resultSet, int columnIndex) throws SQLException {
return TaskTypeCategory.fromDatabaseValue(resultSet.getString(columnIndex));
}

@Override
public TaskTypeCategory getNullableResult(CallableStatement callableStatement, int columnIndex) throws SQLException {
return TaskTypeCategory.fromDatabaseValue(callableStatement.getString(columnIndex));
}
}
public enum TaskTypeCategory {

DEVELOPMENT("development", "研发"),
OPERATION("operation", "运维"),
;

private String value;

private String name;

TaskTypeCategory(String name, String value) {
this.name = name;
this.value = value;
}

public String getName() {
return name;
}

/**
* @JsonValue Deserializing JSON to Enum
*/
@JsonValue
public String getValue() {
return value;
}

public static TaskTypeCategory fromColumnValue(String columnValue) {
if (databaseValue == null) {
return null;
}
TaskTypeCategory taskTypeCategory = Arrays.asList(TaskTypeCategory.values())
.stream()
.filter(item -> item.getName().equals(databaseValue))
.findFirst()
.get();
return taskTypeCategory;
}
}
public class Task {
...
private TaskTypeCategory category;
...
}

References

[1] Mapper XML Files - MyBatis

[2] Is jdbcType necessary in a MyBatis mapper?

Nested Object

The POJO (a plain Java object)

@Data
public class User {
private Integer id;
private String name;
private Address address;

@Data
public static class Address {
private String province;
private String city;
private String area;
}
}

Receive from Request URL Query String

Frontend

id=xxx&name=xxx&address.province=xxx&address.city=xxx&address.area=xxx

Code example

let json = {
id: 1,
name: "Tom",
address: {
province: "A1",
city: "A1-1",
area: "A1-2-2"
}
}
function serializeTwoLevelsNestedJson(json) {
return Object.keys(json).map(function(key) {
if (typeof json[key] === "object" && !Array.isArray(json[key])) {
return Object.keys(json[key]).map(function(subJsonKey) {
return encodeURIComponent(key) +'.' + encodeURIComponent(subJsonKey)+ '=' + encodeURIComponent(json[key][subJsonKey]);
}).join('&');
} else {
return encodeURIComponent(key) +'=' + encodeURIComponent(json[key])
}
}).join('&');
}

Backend

@RequestMapping("/test")
public String test(User user) {
}

Receive from Form Data

Frontend

  • id
  • name
  • address.province
  • address.city
  • address.area

Backend

@RequestMapping("/test")
public String test(User user) {
}

Receive from Request Body JSON

Frontend

{
"id": 1,
"name": "Jack",
"address": {
"province": "AAA",
"city": "BBB"
}
}

Backend

@RequestMapping("/test")
public String test(@RequestBody User user) {
}

Nested Object List

The POJO (a plain Java object)

@Data
public class User {
private Integer id;
private String name;
private List<Address> addressList;

@Data
public static class Address {
private String province;
private String city;
private String area;
}
}

Receive from Request URL Query String

Frontend

This type of query strings looks like id=xxx&name=xxx&addressList[0].province&addressList[0].city&addressList[0].area, but it actually is

id=1&name=ABC&addressList%5B0%5D.province=AA&addressList%5B0%5D.city=BB&addressList%5B0%5D.area=CC

You need to Encode square brackets [ and ] by the encodeURIComponent(str) method.

Code example

let json = {id: 1, name: "Tom", addressList: [
{
province: "A1",
city: "A1-1",
area: "A1-2-2"
},
{
province: "B1",
city: "B1-1",
area: "B1-2-2"
}
]};
function serializeTwoLevelsNestedJson(json) {
return Object.keys(json).map(function(key) {
if (typeof json[key] === "object") {
if (Array.isArray(json[key])) {
return json[key].map((arrayItem, index) => {
return Object.keys(arrayItem).map(function(arrayItemKey) {
return encodeURIComponent(key) +encodeURIComponent('[') + index + encodeURIComponent('].') + encodeURIComponent(arrayItemKey) + '=' + encodeURIComponent(arrayItem[arrayItemKey]);
}).join('&')
}).join('&');
} else {
return Object.keys(json[key]).map(function(subJsonKey) {
return encodeURIComponent(key) +'.' + encodeURIComponent(subJsonKey)+ '=' + encodeURIComponent(json[key][subJsonKey]);
}).join('&');
}
}
else {
return encodeURIComponent(key) +'=' + encodeURIComponent(json[key])
}
}).join('&');
}

Backend

@RequestMapping("/test")
public String test(User user) {
}

Receive from Form Data

Frontend

  • id
  • name
  • addressList[0].province
  • addressList[0].city
  • addressList[0].area
  • addressList[1].province

Backend

@RequestMapping("/test")
public String test(User user) {
}

Receive from Request Body JSON

Frontend

{
"id": 1,
"name": "Jack",
"addressList": [
{
"province": "AAA",
"city": "BBB",
"area": "CCC"
},
...
]
}

Backend

@RequestMapping("/test")
public String test(@RequestBody User user) {
}
0%