Taogen's Blog

Common text processing commands on Linux

• grep
• sed
• awk
• tr
• sort
• wc

Filter

Filter lines

echo -e "Foo\nBar" | grep "Fo"

Insert

Insert lines

Insert to the specific line

echo -e 'Foo\nBar' | sed '2i\the new line\' # Foo\nthe new line\nBar

Add line to beginning and end

echo -e 'Foo\nBar' | sed '1 i\First line'
echo -e 'Foo\nBar' | sed '$aEnd line'

Insert lines after match pattern

echo -e 'Foo\nBar' | sed '/Foo/a NewLine1\nNewLine2'
echo -e 'Foo\nBar' | sed '/Foo/r add.txt'

Insert text to the beginning and end

Insert text to the beginning of each line

echo 'foo' | sed 's/^/BeginText/'

Insert text to the end of each line

echo 'foo' | sed 's/$/EndText/'

Insert text to the begining and end of each line

echo 'foo' | sed 's/^/BeginText/' | sed 's/$/EndText/'

Insert a new line to the end of each line

echo -e 'Foo\nBar'| sed 's/$/\r\n/'

Replace

Replace first

echo "old names, old books" | sed 's/old/new/'
# or
echo "old names, old books" | sed '0,/old/{s/old/new/}'

Replace all

echo "old names, old books" | sed 's/old/new/g'

Remove

Remove matched lines

echo -e "Foo\nBar" | sed '/Foo/d'

Remove empty line

echo -e "Foo\n \nBar" | sed '/^\s*$/d'
# or
echo -e "Foo\n \nBar" | sed '/^[[:space:]]*$/d'

Remove comment /**/ or //

# reomve lines start with / or *
sed '/^ *[*/]/d' 

Remove n lines after a pattern

# including the line with the pattern
echo -e "Line1\nLine2\nLine3\nLine4" | sed '/Line1/,+2d' # Line4

# excluding the line with the pattern
echo -e "Line1\nLine2\nLine3\nLine4" | sed '/Line1/{n;N;d}' # Line1\nLine4

Remove all lines between two patterns

# including the line with the pattern
sed '/pattern1/,/pattern2/d;'
echo -e "Foo\nAAA\nBBB\nBar\nCCC" | sed '/Foo/,/Bar/d' # CCC

# excluding the line with the pattern
sed '/pattern1/,/pattern2/{//!d;};'
echo -e "Foo\nAAA\nBBB\nBar\nCCC" | sed '/Foo/,/Bar/{//!d;}' # Foo\nBar\nCCC

Substring

Get substring by index

cut -c start-end1,start_end2
printf 'hello,world' | cut -c 1-5 # hello

Split and get fields

cut -d DELIMITER -f field_number1,field_number2
print 'hello,world' | cut -d ',' -f 2 # world

Find String

Find String by Pattern

echo -e 'Hello Java developer!\nHello Web developer!' | sed 's/Hello \(.*\) developer!/\1/'

Join

Join lines

echo -e "Foo\nBar" | tr '\n' ' '

Split

Split to multiple lines

echo "Foo Bar" | tr '[:space:]' '[\n*]'

Deduplication

# sort and deduplication
echo -e "1\n3\n2\n1" | sort -u

Sort

echo -e "1\n3\n2\n1" | sort

Count

Count lines

echo -e "1\n3\n2\n1" | wc -l # 4

Count matched lines

echo -e "1\n3\n2\n1" | grep -c "1" # 2

Count matched number of string

echo "hello world" | grep -o -i "o" | wc -l # 2

Format

To Upper/Lower Case

# to upper case
echo "hello WORLD" | tr a-z A-Z
# to lower case
echo "hello WORLD" | tr A-Z a-z

Format JSON string

echo '{"name":"Jack","age":18}' | jq .
echo '{"name":"Jack","age":18}' | jq .name

Crypto

Encode

Base64

Base64 Encode

printf 'hello' | base64
# or
echo -n 'hello' | base64

• -n: do not output the trailing newline

Base64 Decode

printf 'hello' | base64 | base64 -d

URL encode

URL encode

printf '你好' | jq -sRr @uri
# or
echo -n '你好' | jq -sRr @uri

• -n: do not output the trailing newline

Hash

md5

md5 -r /path/to/file

printf 'hello' | md5sum

echo -n 'hello' | md5sum
# or
echo -n 'hello' | md5

• -n: do not output the trailing newline

md5sum on linux, md5 on macOS

sha

shasum -a 256 /path/to/file

printf 'hello' | shasum -a 256

Examples

Wrap in double quotes and join with comma

echo 'hello
world' | sed 's/^/"/' | sed 's/$/"/' | tr '\n' ','

SSH, or secure shell, is an encrypted protocol used to manage and communicate with servers. You can connect to your server via SSH. There are a few different ways to login an SSH server. Public key authentication is one of the SSH authentication methods. It allows you to access a server via SSH without a password.

Creating SSH keys

List supported algorithms of SSH keys on your client and server:

$ ssh -Q key

Output

ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
ssh-rsa
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
ssh-rsa-cert-v01@openssh.com
ssh-dss-cert-v01@openssh.com
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521-cert-v01@openssh.com

Choose an algorithm that supports both your client and server for generating an SSH key pair.

$ ssh-keygen -t {your_algorithm} -C "{your_comment}"

We recommend using the ed25519 algorithm to generate your SSH key. The Ed25519 was introduced on OpenSSH version 6.5. It’s using elliptic curve cryptography that offers better security with faster performance compared to DSA, ECDSA, or RSA. The RSA is even considered not safe if it’s generated with a key smaller than 2048-bit length.

$ ssh-keygen -t ed25519 -C "mac_taogenjia@gmail.com"

Output

Enter file in which to save the key (/Users/taogen/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in {filename}
Your public key has been saved in {filename}.pub

You can specify your SSH key’s filename. If you don’t want to change the filename, by default the private key filename is id_{algorithm} and the public key filename is id_{algorithm}.pub .

For security reasons, it’s best to set a passphrase for your SSH keys.

Coping the SSH public key to your server

Copying Your Public Key Using ssh-copy-id

The simplest way to copy your public key to an existing server is to use a utility called ssh-copy-id.

ssh-copy-id -i public_key_filepath username@remote_host
# or use a specific SSH port
ssh-copy-id -i public_key_filepath -p ssh_port username@remote_host

For example

ssh-copy-id -i ~/.ssh/id_ed25519_remote_1.pub -p 38157 root@xxx.xx.xxx.xxx

Copying Your public key using SSH

cat ~/.ssh/id_rsa.pub | ssh username@remote_host "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

• cat ~/.ssh/id_rsa.pub: Output the file.
• mkdir -p ~/.ssh: Creating the ~/.ssh directory if it doesn’t exist.
• cat >> ~/.ssh/authorized_keys: append the standard output of the previous command of the pipeline to the file ~/.ssh/authorized_keys on the remote host.

Configuring SSH

If there are multiple SSH keys on your local system, you need to configure which destination server uses which SSH key. For example, there is an SSH key for GitHub and another SSH key for a remote server.

Creating the SSH configuration file ~/.ssh/config if it doesn’t exist.

vim ~/.ssh/config

Add the config like the folowing content

# GitHub
Host github.com
  User git
  Port 22
  Hostname github.com
  IdentityFile "~/.ssh/{your_private_key}"
  TCPKeepAlive yes
  IdentitiesOnly yes

# Remote server
Host {remote_server_ip_address}
  User {username_for_ssh}
  Port {remote_server_ssh_port}
  IdentityFile "~/.ssh/{your_private_key}"
  TCPKeepAlive yes
  IdentitiesOnly yes

SSH login with the SSH private key

If you have copied your SSH public key to the server, SSH login will automatically use your private key. Otherwise, you will need to enter the password of the remote server’s user to login.

$ ssh username@remote_host
# or use a specific port
$ ssh -p ssh_port username@remote_host

Disabling password authentication on your server

Using password-based authentication exposes your server to brute-force attacks. You can disable password authentication by updating the configuration file /etc/ssh/sshd_config.

Before disabling password authentication, make sure that you either have SSH key-based authentication configured for the root account on this server, or preferably, that you have SSH key-based authentication configured for an account on this server with sudo access.

sudo vim /etc/ssh/sshd_config

Uncomment the following line by removing # at the beginning of the line:

PasswordAuthentication no

Save and close the file when you are finished. To actually implement the changes we just made, you must restart the service.

sudo systemctl restart ssh

References

[1] How To Configure SSH Key-Based Authentication on a Linux Server

[2] Upgrade Your SSH Key to Ed25519

Configuring DNS

Ensure your Linux server’s DNS configuration file /etc/resolv.conf contains some DNS servers.

If there are no DNS servers in the /etc/resolv.conf, you must add some DNS servers to the file.

vim /etc/resolv.conf

Add the following content to the file /etc/resolv.conf

nameserver 192.168.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4

You can try to restart your system networking to check whether the problem of being unable to ping domain names is resolved. See the section “Restart Networking” of this post.

Configuring Default Route Gateway

You need to check your route table and check if the destination host 0.0.0.0 is routed to the default gateway IP (e.g. 192.168.0.1). If not you need to update the gateway IP.

Get Default Gateway IP

$ ip r | grep default

default via 192.168.0.1 dev eth0 proto dhcp metric 100

Some computers might have multiple default gateways. The gateway with lowest Metric is the first to be searched and used as the default gateway.

My server’s default gateway IP is 192.168.0.1.

Check the Route Table

Print the route table:

$ sudo route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1   0.0.0.0         UG    0      0        0 eth0
...

• Destination: The destination network or destination host.
• Gateway: The gateway address or ’*’ if none set.
• Genmask: The netmask for the destination net: 255.255.255.255 for a host destination and 0.0.0.0 for the default route.

What Is The Meaning of 0.0.0.0 In Routing Table?

Each network host has a default route for each network card. This will create a 0.0.0.0 route for such card. The address 0.0.0.0 generally means “any address”. If a packet destination doesn’t match an individual address in the table, it must match a 0.0.0.0 gateway address. In other words, default gateway is always pointed by 0.0.0.0

Update the 0.0.0.0 Route to the Default Gateway

Update the destination host 0.0.0.0 route the the default gateway IP e.g 192.168.0.1.

ip command

You can temporarily update the 0.0.0.0 route gateway by the ip command.

Add the default route:

ip route add default via 192.168.0.1

You can also delete the default route:

ip route delete default

route command

You can temporarily update the 0.0.0.0 route gateway by the route command.

Add the default route:

route add default gw 192.168.0.1

You can also delete the default route:

route del default gw 192.168.0.1

Update configuration file

You can permanently update the 0.0.0.0 route gateway by in system configuration file.

CentOS/RHEL

vim /etc/sysconfig/network

Add the following content to the file /etc/sysconfig/network

NETWORKING=yes
GATEWAY=192.168.0.1

Debian/Ubuntu

vim /etc/network/interfaces

Find network interface and add the following option

... 
gateway 192.168.0.1
...

Restart Networking

After update the gateway configuration file, you need restart the networking.

Restart the networking on CentOS/RHEL

sudo systemctl restart NetworkManager.service
# or
sudo systemctl stop NetworkManager.service
sudo systemctl start NetworkManager.service

Restart the networking on Debian/Ubuntu

sudo /etc/init.d/networking restart

Appendixes

Some public DNS servers

# OpenDNS
208.67.222.222
208.67.220.220

# Cloudflare
1.1.1.1
1.0.0.1

# Google
8.8.8.8
8.8.4.4

# Quad9
9.9.9.9

References

[1] Understanding Routing Table

[2] What Is The Meaning of 0.0.0.0 In Routing Table?

介绍麒麟操作系统

银河麒麟（NeoKylin）是由中国麒麟软件有限公司基于Linux开发的商业操作系统。其社区版为Ubuntu Kylin。中标麒麟（NeoKylin）与银河麒麟同为中国麒麟软件有限公司基于Linux开发的商业操作系统。

银河麒麟高级服务器操作系统V10（Kylin Linux Advanced Server V10 (Tercel)）是针对企业级关键业务，适应虚拟化、云计算、大数据、工业互联网时代对主机系统可靠性、安全性、性能、扩展性和实时性等需求，依据CMMI5级标准研制的提供内生本质安全、云原生支持、自主平台深入优化、高性能、易管理的新一代自主服务器操作系统。银河麒麟系统采用同源构建支持六款自主CPU平台(飞腾、鲲鹏、龙芯、申威、海光、兆芯等国产CPU)，所有组件基于同一套源代码构建。

查看操作系统信息

# 查看 Linux 系统发行版
$ cat /etc/os-release

NAME="Kylin Linux Advanced Server"
VERSION="V10 (Tercel)"
ID="kylin"
VERSION_ID="V10"
PRETTY_NAME="Kylin Linux Advanced Server V10 (Tercel)"
ANSI_COLOR="0;31"

# 查看 CPU 架构
$ lscpu

Architecture:                    aarch64
CPU op-mode(s):                  64-bit
Model name:                      Kunpeng-920
...

Architecture:                    aarch64
CPU op-mode(s):                  64-bit
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       HiSilicon
Model:                           0
Model name:                      Kunpeng-920
Stepping:                        0x1
BogoMIPS:                        200.00
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm ssbs

配置 yum 源仓库

使用中标软件的 yum 源仓库

# 备份系统原来的 yum 配置
mv /etc/yum.repos.d/kylin_aarch64.repo /etc/yum.repos.d/kylin_aarch64.repo.bak
# 创建新的 yum 源配置文件
vi /etc/yum.repos.d/kylin_aarch64.repo 

将下面的内容添加到 /etc/yum.repos.d/kylin_aarch64.repo 文件中

[ks10-adv-os]
name=Kylin-Linux-Advanced-Server-os
baseurl=http://update.cs2c.com.cn:8080/NS/V10/V10SP1/os/adv/lic/base/aarch64/
gpgcheck=0
enable=1

baseurl格式为 http://update.cs2c.com.cn:8080/NS/V10/{版本}/os/adv/lic/base/{架构}/

这里的版本选择的是 V10SP1，架构选的是 aarch64（鲲鹏）。

# 清除原有 yum 缓存和生成新的缓存
$ sudo yum clean all && sudo yum makecache
# 测试
$ sudo yum update
$ sudo yum install java-1.8.0-openjdk
Is this ok [y/N]: N

介绍 CTyunOS

天翼云操作系统 CTyunOS 是基于欧拉开源操作系统（openEuler，简称“欧拉”）开发的，欧拉操作系统是华为基于 CentOS 开发的。2019年9月，华为宣布 EulerOS 开源，开源名称为 openEuler（开源欧拉）。EulerOS 支持 AArch64（鲲鹏）处理器、容器虚拟化技术，是一个面向企业级的通用服务器架构平台。华为的鸿蒙是手机手表的操作系统，面向C端。而欧拉是电脑服务器的操作系统，是面向B端。

CTyunOS 操作系统基于 openEuler 20.03 LTS 版本自主研发的。CTyunOS 3 是 CTyunOS 在2023年4月发布的最新版本，上游采用了openEuler社区发布的长期稳定版本 openEuler 22.03 LTS SP1作为基线，针对云计算、云原生场景进行了深度的开发，目前在天翼云上已经可以开通该操作系统的弹性云主机。

查看操作系统信息

# 查看 Linux 系统发行版
$ cat /etc/os-release

NAME="ctyunos"
VERSION="2.0.1"
ID="ctyunos"
VERSION_ID="2.0.1"
PRETTY_NAME="ctyunos 2.0.1"
ANSI_COLOR="0;31"

# 查看 CPU 架构
$ lscpu

Architecture:                    aarch64
CPU op-mode(s):                  64-bit
Model name:                      Kunpeng-920
...

Architecture:                    aarch64
CPU op-mode(s):                  64-bit
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       HiSilicon
Model:                           0
Model name:                      Kunpeng-920
Stepping:                        0x1
BogoMIPS:                        200.00
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm ssbs

配置 yum 源仓库

# 备份系统原来的 yum 配置
$ mv ctyunos.repo ctyunos.repo.bak
# 下载华为云的 openeuler yum 源配置文件
$ curl -O https://repo.huaweicloud.com/repository/conf/openeuler_aarch64.repo
# 移动到系统的 yum 源配置目录
$ mv openeuler_aarch64.repo /etc/yum.repos.d
# 清除原有 yum 缓存和生成新的缓存
$ sudo yum clean all && yum makecache
# 测试
$ sudo yum update
$ sudo yum install java-1.8.0-openjdk
Is this ok [y/N]: N

华为云 Linux 软件包仓库配置文件目录：https://repo.huaweicloud.com/repository/conf/

References

[1] Configuring Yum and Yum Repositories - RedHat

[2] 配置OpenEuler的网络yum源

[3] CentOS配置yum仓库的三种方法

File Operations

Archive Files

zip

# Create an achive file
zip -r achive.zip dir_name

# List files of the achive file
unzip -l achive.zip

# Extract files
unzip achive.zip
unzip achive.zip -d /target/filepath

tar

# Create an achive file
tar -czvf achive.tar.gz dir_name
tar -cjvf achive.tar.bz2 dir_name
tar -cvf achive.tar dir_name

# List files of the achive file
tar -tf achive.tar.gz

# Extract files
tar -xvf achive.tar.gz
tar -xvf achive.tar.gz -C /target/filepath

Get information of file

Real path

# Print real path of file
realpath path/to/file

Architecture of software

# Architecture of software
cat `which {your_software}` | file -

# file type
file path/to/file

Files and Directories

Directories

• /lib: It is a place for the essential standard libraries. Think of libraries required for your system to run. If something in /bin or /sbin needs a library that library is likely in /lib.
• /usr
  • /usr/bin: Executables binary files. E.g. java, mvn, git, apt, kill.
  • /usr/local: To keep self-compiled or third-party programs.
  • /usr/sbin: This directory contains programs for administering a system, meant to be run by ‘root’. Like ‘/sbin’, it’s not part of a user’s $PATH. Examples of included binaries here are chroot, useradd, in.tftpd and pppconfig.
  • /usr/share: This directory contains ‘shareable’, architecture-independent files (docs, icons, fonts etc).
  • /usr/lib: the /usr directory in general is as it sounds, a user based directory. Here you will find things used by the users on the system. So if you install an application that needs libraries they might go to /usr/lib. If a binary in /usr/bin or /usr/sbin needs a library it will likely be in /usr/lib.
• /var: variable data files. log files.
  • /var/lib: the /var directory is the writable counterpart to the /usr directory which is often required to be read-only. So /var/lib would have a similar purpose as /usr/lib but with the ability to write to them.

Configuration Files

Bash Configuration Files

Most global config files are located in the /etc directory

Important Global Config Files

Important User-Specific Config Files

System Settings

System Time

Time

# show date time
date

# date time format
date '+%Y-%m-%d'
date '+%Y-%m-%d %H:%M:%S'
date '+%Y-%m-%d_%H-%M-%S'

# update time and date from the internet
timedatectl set-ntp true

Timezone

# list timezones
timedatectl list-timezones

# set timezone
timedatectl set-timezone Asia/Shanghai

# show time settings
timedatectl status

hostname

The hostname is used to distinguish devices within a local network. It’s the machine’s human-friendly name. In addition, computers can be found by others through the hostname, which enables data exchange within a network, for example. Hostnames are used on the internet as part of the fully qualified domain name.

you can configure a computer’s hostname:

# setting
$ hostnamectl set-hostname server1.example.com
# verify the setting
$ less /etc/hostname
# query your computer's hostname
$ hostname

hosts

The /etc/hosts file contains the Internet Protocol (IP) host names and addresses for the local host and other hosts in the Internet network. This file is used to resolve a name into an address (that is, to translate a host name into its Internet address).

sudo vim /etc/hosts

Add Environment Variables

/etc/profile

Add environment variables

cp /etc/profile "/etc/profile.bak.$(date '+%Y-%m-%d_%H-%M-%S')"
echo "export name=value" >> /etc/profile
cat /etc/profile
source /etc/profile

Add to path

cp /etc/profile /etc/profile.bak.$(date '+%Y-%m-%d_%H-%M-%S')
echo 'export PATH=$PATH:/usr/local/mysql/bin' >> /etc/profile
source /etc/profile

System Log

# print OOM killer log
dmesg -T | egrep -i 'killed process'

Upload File

scp

SCP will always overwrite existing files. Thus, in the case of a clean upload SCP should be slightly faster as it doesn’t have to wait for the server on the target system to compare files.

# transfer a file
scp local_file remoteuser@remote_ip_address:/remote_dir
# transfer multiple files
scp local_file1 local_file2 remoteuser@remote_ip_address:/remote_dir

# transfer a directory
scp -r local_dir remoteuser@remote_ip_address:/remote_dir

# transfer a file from remote host to local
scp remoteuser@remote_ip_address:/remote_file local_dir

# Transfer Files Between Two Remote Systems 
scp remoteuser@remote_ip_address:/remote_file remoteuser@remote_ip_address:/remote_file

• -P SSH_port

rsync over ssh

In the case of a synchronization of files that change, like log files or list of source files in a repository, rsync is faster.

Copy a File from a Local Server to a Remote Server with SSH

rsync -avzhe ssh backup.tar.gz root@192.168.0.141:/backups/
# Show Progress While Transferring Data with Rsync
rsync -avzhe ssh --progress backup.tar.gz root@192.168.0.141:/backups/

Copy a File from a Remote Server to a Local Server with SSH

rsync -avzhe ssh root@192.168.0.141:/root/backup.tar.gz /tmp

sftp

sftp [username]@[remote hostname or IP address]

# download file to the local system's Home directory. 
get [path to file]
# change directory
get [path to file] [path to directory]
# change filename
get [path to file] [new file name]

# upload the local system's Home directory's file to the remote server's current directory
put [path to file]
# change directory
put [path to file] [path to directory]
# change filename
put [path to file] [new file name]

Application Data

logging file path: /var/log/{application_name}

upload file path: /data/{application_name}/upload

application build and running file path: /var/java/{application_name}, /var/html/{application_name}

References

• How to configure a hostname on a Linux system
• What is a Hostname?
• hosts File Format for TCP/IP

Reverse Proxy

HTTP

http {
    ...
    server {
        listen 80;
        server_name myserver.com;
        # The default root is /usr/share/nginx/www, /usr/share/nginx/html or /var/www/html
        root /var/www/your_domain/html;

        # Staic files
        location / {
            # redefine the root
            root   /var/www/your_domain/html;
            try_files $uri $uri/index.html =404;
            # if not found redirect to home page $root/index.html
            # try_files $uri $uri/ /index.html;
        }

        # API
        location  /api/ {
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # Mapping http(s)://my-domain.com/api/xxx to http://localhost:8080/api/xxx.
            proxy_pass http://localhost:8080/api/;
            # Mapping http(s)://my-domain.com/api/xxx to https://localhost:8080/xxx.
            # proxy_pass http://localhost:8080/;
        }

        # Cache js, css, image, etc.
    }
    ...
}

• Response static files: {root path}/requestURI
• Proxy requests: {proxy_pass path}/requestURI

Passing Request Headers

By default, NGINX redefines two header fields in proxied requests, “Host” and “Connection”, and eliminates the header fields whose values are empty strings. “Host” is set to the $proxy_host variable, and “Connection” is set to close.

HTTPS

http {
    # reuse SSL session parameters to avoid SSL handshakes for parallel and subsequent connections.
    # or "ssl_session_cache builtin:1000 shared:SSL:10m;"
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 10m;
    # enabling keepalive connections to send several requests via one connection and the second is to reuse SSL session parameters to avoid SSL handshakes for parallel and subsequent connections.
    keepalive_timeout   70;
	
    server {
        listen 443 ssl;
        server_name myproject.com;
        
        ssl_certificate /etc/ssl/projectName/projectName.com.pem;
        ssl_certificate_key /etc/ssl/projectName/projectName.com.key;
        
        # Additional SSL configuration (if required)
        ssl_protocols TLSv1.3 TLSv1.2 TLSv1.1 TLSv1;
        ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;

        # same with HTTP
        location / {
            ...
        }
    }
    ...
}

HTTP to HTTPS

http {
    ...
    server {
        listen 80;
        server_name myserver.com;
        return 301 https://myserver.com$request_uri;
    }
    ...
}

Optimization

Caching

Cache static files

When you build static assets with versioning/hashing mechanisms, adding a version/hash to the filename (cache-busting filenames) or query string is a good way to manage caching. In such a case, you can add a long max-age value and immutable because the content will never change.

http {
    server {
        ...
        location /static {
            root   /var/www/your_domain/staic;
            # To disable access log off for saving disk I/O 
            access_log off;
            # or "expires max";
            expires 7d;
            # build static assets with versioning/hashing mechanisms
            add_header Cache-Control "public, immutable";
            # revalidate
            # add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }
        ...
    }
}

http {
    server {
        ...
        location ~* \.(js|css|png|jpg|jpeg|gif|svg|ico|woff2|woff|ttf|eot|pdf|txt)$ {
            root   /var/www/your_domain/staic;
            # To disable access log off for not hitting the I/O limit
            access_log off;
            # or "expires max";
            expires 7d;
            # build static assets with versioning/hashing mechanisms
            add_header Cache-Control "public, immutable";
            # revalidate
            # add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }
        ...
    }
}

After setting cache, the following headers will be present in the response headers:

Cache-Control: max-age=604800, public, must-revalidate, proxy-revalidate

Warning: Cache-Control "public, immutable" for JS, CSS may cause program exceptions after update program.

Compression

Enable gzip

http {
    ...
    server {
        ...
        # gzip can be set in `http {}` or `server {}`
        # --with-http_gzip_static_module
        gzip on;
        # By default, NGINX compresses responses only with MIME type text/html. To compress responses with other MIME types, include the gzip_types directive and list the additional types.
        gzip_types text/css application/json text/xml application/javascript;
        # To specify the minimum length of the response to compress, use the gzip_min_length directive. The default is 20 bytes
        gzip_min_length 200;
        # Sets the number and size of buffers used to compress a response.
        gzip_buffers 32 4k;
        # Sets a gzip compression level of a response. Acceptable values are in the range from 1 to 9.
        gzip_comp_level 6;
        gzip_vary on;
        ...
    }
}

After enable gzip, the following headers will be present in the response headers:

content-encoding: gzip

Enable HTTP/2

The ngx_http_v2_module module (1.9.5) provides support for HTTP/2. This module is not built by default, it should be enabled with the --with-http_v2_module configuration parameter.

To enable HTTP/2, you must first enable SSL/TLS on your website. HTTP/2 requires the use of SSL/TLS encryption, which provides a secure connection between the web server and the client’s browser.

http {
    server {
        # enable http2
        listen 443 ssl http2;
        
        ssl_certificate server.crt;
        ssl_certificate_key server.key;
    }
}

Connection Handling

Keepalive Connections

http {
    keepalive_timeout 65;
}

Keepalive connections reduce the overhead of repeatedly establishing new connections for multiple requests from the same client (e.g., a web browser loading multiple assets from a single page). This saves time and resources associated with TCP handshake and SSL/TLS negotiation.

TCP Optimizations

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
}

• sendfile → efficient file transfer.
• tcp_nopush → optimizes packet sending.
• tcp_nodelay → reduces latency for small packets.

When sendfile on; and tcp_nopush on; are used together in Nginx for serving static files, Nginx will initially buffer data to send full TCP packets.

However, for the very last packet(s) of a file, which may not be full, Nginx will dynamically disable tcp_nopelay (effectively removing TCP_CORK) and enable tcp_nodelay to ensure that these remaining partial packets are sent immediately without delay, thus completing the file transfer quickly.

SSL/TLS Optimization

http {
    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 1d;

    ssl_protocols TLSv1.3 TLSv1.2 TLSv1.1 TLSv1;
    ssl_prefer_server_ciphers on;
}

• ssl_session_cache and ssl_session_timeout: Reuse SSL sessions → fewer handshakes.
• ssl_protocols: TLS 1.3 is faster than TLS 1.2.

Worker Processes and Connections

http {
    worker_processes auto;
}

• worker_processes: This directive determines the number of Nginx worker processes that will handle incoming requests. A common practice is to set this to auto, which automatically sets the number of worker processes to match the number of CPU cores on your server. This ensures that each core is utilized efficiently.

Logging

Disabling Access Logs

While access logs are valuable for monitoring and debugging, they can consume significant CPU and disk resources on high-traffic sites. If you don’t need detailed logging for every request, you can either buffer the logs or disable them entirely to reduce overhead.

To disable access log for static files

http {
    server {
        ...
        location ~* \.(js|css|png|jpg|jpeg|gif|svg|ico|woff2|woff|ttf|eot|pdf|txt)$ {
            # To disable access log off for saving disk I/O 
            access_log off;
        }
        ...
    }
}

Or disable all access logs

http {
    # To disable access log off for saving disk I/O 
    access_log off;
}

Buffering Logs (Optional)

Instead of writing to the log file for every single request, you can configure Nginx to buffer log data and write it in larger chunks. This reduces the number of disk I/O operations and can improve performance.

Benefits of Log Buffering:

• Reduced I/O Operations: Fewer, larger writes to disk instead of many small writes, improving performance.
• Lower CPU Consumption: Less overhead associated with managing individual log entries.
• Improved Disk Lifespan: Reduced wear and tear on storage devices.

Considerations:

• Data Latency: Buffered log entries are not immediately written to disk, which can introduce a slight delay in log availability for real-time analysis.
• Memory Usage: Buffering consumes a small amount of memory per worker process.
• Out-of-Order Entries: In rare cases, if multiple worker processes are writing to the same log file with buffering enabled, log entries might appear slightly out of order if not flushed simultaneously. However, most log analysis systems can handle this by sorting based on timestamps.

Settings

Timeout

http {
    ...
    # proxy_connect_timeout default 60s
    proxy_connect_timeout 180s;
    # proxy_send_timeout default 60s
    proxy_send_timeout 180s;
    # proxy_read_timeout default 60s
    proxy_read_timeout 180s;
    ...
}

Upload File Size

http {
    ...
    # client_max_body_size default 1M
    client_max_body_size 100M;
    ...
}

Enable CORS for API

Enable CORS for specified sites

http {
    ...

    map "$http_origin" $cors {
      default '';
      "~^https?://localhost(:\d+)?$" "$http_origin";
      "~^https?://10.0.0\.\d{1,2}(:\d+)?$" "$http_origin";
      "~^https?://example\.com$" "$http_origin";
    }

    server {
        # API
        location  /api/ {
            ...
            
            # Attach CORS headers only if it's a valid origin ($cors should not be empty)
            if ($cors != "") {
                # using $cors for specified sites or using $http_origin for any sites.
                proxy_hide_header 'Access-Control-Allow-Origin';
                add_header 'Access-Control-Allow-Origin' '$cors' always;
                proxy_hide_header 'Access-Control-Allow-Credentials';
                add_header 'Access-Control-Allow-Credentials' true always;
                proxy_hide_header 'Access-Control-Allow-Methods';
                add_header 'Access-Control-Allow-Methods' 'POST, GET, DELETE, PUT, PATCH' always;
                proxy_hide_header 'Access-Control-Allow-Headers';
                add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With' always;
            }
            # Check if it's a preflight request and "cache" it for 20 days
            if ($request_method = 'OPTIONS') {
                add_header 'Access-Control-Allow-Origin' '$cors' always;
                add_header 'Access-Control-Allow-Credentials' true always;
                add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With' always;
                add_header 'Access-Control-Max-Age' 1728000;
                add_header 'Content-Type' 'text/plain charset=UTF-8';
                add_header 'Content-Length' 0;
                return 204;
            }
            
            ...
        }
    }
}

Enable CORS for all sites

http {
    ...

    server {
        # API
        location  /api/ {
            ...
            
            proxy_hide_header 'Access-Control-Allow-Origin';
            add_header 'Access-Control-Allow-Origin' '$http_origin' always;
            proxy_hide_header 'Access-Control-Allow-Credentials';
            add_header 'Access-Control-Allow-Credentials' true always;
            proxy_hide_header 'Access-Control-Allow-Methods';
            add_header 'Access-Control-Allow-Methods' 'POST, GET, DELETE, PUT, PATCH' always;
            proxy_hide_header 'Access-Control-Allow-Headers';
            add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With' always;

            # Check if it's a preflight request and "cache" it for 20 days
            if ($request_method = 'OPTIONS') {
                add_header 'Access-Control-Allow-Origin' '$http_origin' always;
                add_header 'Access-Control-Allow-Credentials' true always;
                add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With' always;
                add_header 'Access-Control-Max-Age' 1728000;
                add_header 'Content-Type' 'text/plain charset=UTF-8';
                add_header 'Content-Length' 0;
                return 204;
            }
            
            ...
        }
    }
}

Load Balancing

http {
    ...
    upstream backend-server {
        server xxx.xxx.xxx.xxx:8080 max_fails=1 fail_timeout=300s;
        server xxx.xxx.xxx.xxx:8080 max_fails=1 fail_timeout=300s;
        ...
    }

    server {
        ...
        location /api/ {
            proxy_set_header    X-Real-IP        $remote_addr;
            proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
            proxy_pass http://backend-server/;
        }
    }
}

Test the Nginx Configuration is Updated

Adding the following config to the Nginx configuration file. You can verify if the configuration is updated by updating the return status code (e.g. 403 Forbidden, 406 Not Acceptable, 423 Locked) of the /test location and visiting the test URL http://yourDomain/testConfig.

location /testConfig {
    # 403 Forbidden, 406 Not Acceptable, 423 Locked
    return 403;
}

Appendixes

Embedded Variables

• $proxy_host: name and port of a proxied server as specified in the proxy_pass directive;
• $proxy_add_x_forwarded_for: the “X-Forwarded-For” client request header field with the $remote_addr variable appended to it, separated by a comma. If the “X-Forwarded-For” field is not present in the client request header, the $proxy_add_x_forwarded_for variable is equal to the $remote_addr variable.
• $host: In this order of precedence: host name from the request line, or host name from the “Host” request header field, or the server name matching a request.
• $remote_addr: Client address

Build Nginx From Source

# Download Nginx source code
wget http://nginx.org/download/nginx-{latest-stable-version}.tar.gz

You can know the latest version of Nginx by visiting the Nginx download page.

tar -zxvf nginx-{latest-stable-version}.tar.gz

cd nginx-{latest-stable-version}

# Configuring Nginx
./configure \
--with-pcre \
--with-http_ssl_module \
--with-http_image_filter_module=dynamic \
--modules-path=/etc/nginx/modules \
--with-http_v2_module \
--with-stream=dynamic \
--with-http_addition_module \
--with-http_mp4_module \
--with-http_gzip_static_module

More configuration

Common errors when running ./configure

1. ./configure: error: the HTTP rewrite module requires the PCRE library.

Solution

sudo apt update && apt upgrade -y && apt autoremove && apt autoclean
apt-get install libpcre3 libpcre3-dev
# or
sudo yum update -y && yum upgrade && yum autoremove
yum install libpcre3 libpcre3-dev
yum -y install pcre-devel openssl openssl-devel

2. ./configure: error: the HTTP image filter module requires the GD library. You can either do not enable the module or install the libraries.

Solution

sudo apt update && apt upgrade -y && apt autoremove && apt autoclean
apt-get install gd-devel
# or
sudo yum update -y && yum upgrade && yum autoremove
yum install gd-devel -y

3. ./configure: error: C compiler cc is not found

Solution

sudo apt update && apt upgrade -y && apt autoremove && apt autoclean
sudo apt-get install linux-kernel-headers build-essential -y
# or
sudo yum update -y && yum upgrade && yum autoremove
sudo yum install gcc -y

Successful output of configure

Configuration summary
  + using system PCRE2 library
  + using system OpenSSL library
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx modules path: "/etc/nginx/modules"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

You can add the following parameters to specify paths:

--prefix=/var/www/html \
--sbin-path=/usr/sbin/nginx \
--modules-path=/etc/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--pid-path=/var/run/nginx.pid \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--lock-path=/var/lock/nginx.lock \

# Build nginx
$ make
$ sudo make install

Successful output of build

make -f objs/Makefile install
make[1]: 进入目录“/root/nginx-1.26.1”
...
# make[1]: 离开目录“/root/nginx-1.26.1”

# Start nginx
$ cd /usr/local/nginx/sbin
$ ./nginx -V
$ ./nginx
# Verify
$ curl http://localhost

Rebuild Nginx source

# Just remove the nginx binary file. Or completely remove nginx `sudo apt-get purge nginx` or `yum remove package`
cd /usr/local/nginx/sbin
mv nginx nginx.bak

# configure
tar -zxvf nginx-{latest-stable-version}.tar.gz
cd nginx-{latest-stable-version}
# Configuring Nginx
./configure ...

# Build and install nginx
$ make
$ sudo make install

References

[1] Configuring HTTPS servers - Nginx

[2] Alphabetical index of variables - Nginx

[3] Serving Static Content - Nginx

[4] NGINX Reverse Proxy - Nginx

Web Servers

Nginx

Error Information

Status Code: 504

Response:

<html>
    <head><title>504 Gateway Time-out</title></head>
    <body bgcolor="white">
        <center><h1>504 Gateway Time-out</h1></center>
        <hr><center>nginx</center>
    </body>
</html>

Default Timeout

The default timeout for Nginx is 60 seconds.

Settings

Update proxy timeout to 180 seconds:

http {
    proxy_connect_timeout 180s;
    proxy_send_timeout 180s;
    proxy_read_timeout 180s;
    ...
}

Java HTTP Client

Spring RestTemplate

Default Timeout

The default timeout is infinite.

By default RestTemplate uses SimpleClientHttpRequestFactory and that in turn uses HttpURLConnection.

By default the timeout for HttpURLConnection is 0 - ie infinite, unless it has been set by these properties :

-Dsun.net.client.defaultConnectTimeout=TimeoutInMiliSec 
-Dsun.net.client.defaultReadTimeout=TimeoutInMiliSec 

Settings

@Bean
public RestTemplate restTemplate() {
    SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
    // Time to establish a connection to the server from the client-side. Set to 20s.
    factory.setConnectTimeout(20000);
    // Time to finish reading data from the socket. Set to 300s.
    factory.setReadTimeout(300000);
    return new RestTemplate(factory);
}

JavaScript HTTP Client

axios

Default Timeout

The default timeout is 0 (no timeout).

Settings

const instance = axios.create({
    baseURL: 'https://some-domain.com/api/',
    // `timeout` specifies the number of milliseconds before the request times out.
    // If the request takes longer than `timeout`, the request will be aborted.
    timeout: 60000,
    ...
});

Plugins

Browser plugins

Chrome extensions

• Chrome extension documentation
• chrome-extensions-samples - GitHub

IDE plugins

Web, Mobile and Desktop Application

Management System

Website

Website categories

1. E-commerce: Websites that facilitate online buying and selling of goods and services, such as Amazon or eBay.
   1. Shopping mall
2. Social Networking: Websites that connect people and allow them to interact and share information, such as Facebook or LinkedIn.
   1. IM
   2. Forum/BBS
3. News and Media: Websites that provide news articles, videos, and other multimedia content, such as CNN or BBC.
4. Blogs and Personal Websites: Websites where individuals or organizations publish articles and personal opinions, such as WordPress or Blogger.
5. Educational: Websites that provide information, resources, and learning materials for educational purposes, such as Khan Academy or Coursera.
6. Entertainment: Websites that offer various forms of entertainment, such as games, videos, music, or movies, such as Netflix or YouTube.
7. Government and Nonprofit: Websites belonging to government institutions or nonprofit organizations, providing information, services, and resources, such as whitehouse.gov or Red Cross.
8. Business and Corporate: Websites representing businesses and corporations, providing information about products, services, and company details, such as Apple or Coca-Cola.
9. Sports: Websites dedicated to sports news, scores, analysis, and related information, such as ESPN or NBA.
10. Travel and Tourism: Websites that provide information and services related to travel planning, accommodations, and tourist attractions, such as TripAdvisor or Booking.com.

Mobile Software

Desktop Software

• Instant message. E.g. Telegram.
• Email client. E.g. Mozilla Thunderbird.
• Web browser. E.g. Google Chrome.
• Office software. E.g. Microsoft Office, Typora, XMind.
• Note-taking software. E.g. Notion, Evernote.
• PDF reader. E.g. SumatraPDF.
• File processing. E.g. 7-Zip
• Media player. E.g. VLC.
• Media processing. E.g. FFmpeg, HandBrake, GIMP.
• Flashcard app. E.g. anki.
• Stream Media. E.g. Spotify.
• HTTP proxy. E.g. V2rayN.

Libraries, Tools, Services

Libraries

• General-purpose libraries for programming language. E.g. Apache Commons Lang.
• File processing. E.g. Apache POI.
• Data parser. E.g. org.json.
• Chart, Report, Graph.
• Logging.
• Testing.
• HTTP Client.

Developer Tools

• Editor
• IDE
• Service Client.

Services

• Web servers. E.g. Nginx, Apache Tomcat.
• Databases. E.g. MySQL.
• Cache. E.g. Redis.
• Search engines. E.g. Elasticsearch.
• Deliver software / contioner. E.g. Docker.
• Other services. E.g. Gotenberg, Aliyun services (media, ai).

Operating Systems

Programming Languages

Apache PDFBox is a Java tool for working with PDF documents. In this post, I will introduce how to use Apache PDFBox to handle PDF files. The code examples in this post are based on pdfbox v2.0.29.

<dependency>
    <groupId>org.apache.pdfbox</groupId>
    <artifactId>pdfbox</artifactId>
    <version>2.0.29</version>
</dependency>

Extract Text

Extract all page text

String inputFilePath = "your/pdf/filepath";
// Load PDF document
PDDocument document = PDDocument.load(new File(inputFilePath));
// Create PDFTextStripper instance
PDFTextStripper pdfStripper = new PDFTextStripper();
// Extract text from PDF
String text = pdfStripper.getText(document);
// Print extracted text
System.out.println(text);
// Close the document
document.close();

Extract page by page

String inputFilePath = "your/pdf/filepath";
// Load the PDF document
PDDocument document = PDDocument.load(new File(inputFilePath));
// Create an instance of PDFTextStripper
PDFTextStripper stripper = new PDFTextStripper();
// Iterate through each page and extract the text
for (int pageNumber = 1; pageNumber <= document.getNumberOfPages(); pageNumber++) {
    stripper.setStartPage(pageNumber);
    stripper.setEndPage(pageNumber);

    String text = stripper.getText(document);
    System.out.println("Page " + pageNumber + ":");
    System.out.println(text);
}
// Close the PDF document
document.close();

Split and Merge

Split

private static void splitPdf(String inputFilePath, String outputDir) throws IOException {
    File file = new File(inputFilePath);
    // Load the PDF document
    PDDocument document = PDDocument.load(file);
    // Create a PDF splitter object
    Splitter splitter = new Splitter();
    // Split the document
    List<PDDocument> splitDocuments = splitter.split(document);
    // Get an iterator for the split documents
    Iterator<PDDocument> iterator = splitDocuments.iterator();
    // Iterate through the split documents and save them
    int i = 1;
    while (iterator.hasNext()) {
        PDDocument splitDocument = iterator.next();
        String outputFilePath = new StringBuilder().append(outputDir)
            .append(File.separator)
            .append(file.getName().replaceAll("[.](pdf|PDF)", ""))
            .append("_split_")
            .append(i)
            .append(".pdf")
            .toString();
        splitDocument.save(outputFilePath);
        splitDocument.close();
        i++;
    }
    // Close the source document
    document.close();
    System.out.println("PDF split successfully!");
}

Merge PDF files

private static void mergePdfFiles(List<String> inputFilePaths, String outputFilePath) throws IOException {
    PDFMergerUtility merger = new PDFMergerUtility();
    // Add as many files as you need
    for (String inputFilePath : inputFilePaths) {
        merger.addSource(new File(inputFilePath));
    }
    merger.setDestinationFileName(outputFilePath);
    merger.mergeDocuments();
    System.out.println("PDF files merged successfully!");
}

Insert and remove pages

Insert pages

public static void insertPage(String sourceFile, String targetFile, int pageIndex) throws IOException {
    // Load the existing PDF document
    PDDocument sourceDoc = PDDocument.load(new File(sourceFile));
    Integer sourcePageCount = sourceDoc.getNumberOfPages();
    // Validate the requested page index
    if (pageIndex < 0 || pageIndex > sourcePageCount) {
        throw new IllegalArgumentException("Invalid page index");
    }
    // Create a new blank page
    PDPage newPage = new PDPage();
    // Insert the new page at the requested index
    if (sourcePageCount.equals(pageIndex)) {
        sourceDoc.getPages().add(newPage);
    } else {
        sourceDoc.getPages().insertBefore(newPage, sourceDoc.getPages().get(pageIndex));
    }
    // Save the modified PDF document to a target file
    sourceDoc.save(targetFile);
    // Close the source and target documents
    sourceDoc.close();
}

Remove pages

private static void removePage(String inputFilePath, String outputFilePath, int pageIndex) throws IOException {

    PDDocument sourceDoc = PDDocument.load(new File(inputFilePath));
    Integer sourcePageCount = sourceDoc.getNumberOfPages();
    // Validate the requested page index
    if (pageIndex < 0 || pageIndex >= sourcePageCount) {
        throw new IllegalArgumentException("Invalid page index");
    }
    sourceDoc.getPages().remove(pageIndex);
    sourceDoc.save(outputFilePath);
    sourceDoc.close();
}

private static void removePage2(String inputFilePath, String outputFilePath, int pageIndex) throws IOException {
    PDDocument sourceDoc = PDDocument.load(new File(inputFilePath));
    Integer sourcePageCount = sourceDoc.getNumberOfPages();
    // Validate the requested page index
    if (pageIndex < 0 || pageIndex >= sourcePageCount) {
        throw new IllegalArgumentException("Invalid page index");
    }
    Splitter splitter = new Splitter();
    List<PDDocument> pages = splitter.split(sourceDoc);
    pages.remove(pageIndex);
    PDDocument outputDocument = new PDDocument();
    for (PDDocument page : pages) {
        outputDocument.addPage(page.getPage(0));
    }
    outputDocument.save(outputFilePath);
    sourceDoc.close();
    outputDocument.close();
}

Encryption

Encrypt

public static void encryptPdf(String inputFilePath, String outputFilePath, String password) throws IOException {
    PDDocument doc = PDDocument.load(new File(inputFilePath));

    AccessPermission ap = new AccessPermission();
    // disable printing, 
    ap.setCanPrint(false);
    //disable copying
    ap.setCanExtractContent(false);
    //Disable other things if needed...

    // Owner password (to open the file with all permissions)
    // User password (to open the file but with restricted permissions)
    StandardProtectionPolicy spp = new StandardProtectionPolicy(password, password, ap);
    // Define the length of the encryption key.
    // Possible values are 40, 128 or 256.
    int keyLength = 256;
    spp.setEncryptionKeyLength(keyLength);

    //Apply protection
    doc.protect(spp);

    doc.save(outputFilePath);
    doc.close();
}

Update password

public static void updatePdfPassword(String inputFilePath, String outputFilePath,
                                      String oldPassword, String newPassword) throws IOException {
    PDDocument doc = PDDocument.load(new File(inputFilePath), oldPassword);

    AccessPermission ap = new AccessPermission();
    // disable printing,
    ap.setCanPrint(false);
    //disable copying
    ap.setCanExtractContent(false);
    //Disable other things if needed...

    // Owner password (to open the file with all permissions)
    // User password (to open the file but with restricted permissions)
    StandardProtectionPolicy spp = new StandardProtectionPolicy(newPassword, newPassword, ap);
    // Define the length of the encryption key.
    // Possible values are 40, 128 or 256.
    int keyLength = 256;
    spp.setEncryptionKeyLength(keyLength);

    //Apply protection
    doc.protect(spp);

    doc.save(outputFilePath);
    doc.close();
}

Remove password

public static void removePdfPassword(String inputFilePath, String outputFilePath,
                                      String password) throws IOException {
    PDDocument doc = PDDocument.load(new File(inputFilePath), password);
    // Set the document access permissions
    doc.setAllSecurityToBeRemoved(true);
    // Save the unprotected PDF document
    doc.save(outputFilePath);
    // Close the document
    doc.close();
}

Convert to Image

PDF to Image

public static void pdfToImage(String pdfFilePath, String imageFileDir) throws IOException {
    File file = new File(pdfFilePath);
    PDDocument document = PDDocument.load(file);
    // Create PDFRenderer object to render each page as an image
    PDFRenderer pdfRenderer = new PDFRenderer(document);
    // Iterate over all the pages and convert each page to an image
    for (int pageIndex = 0; pageIndex < document.getNumberOfPages(); pageIndex++) {
        // Render the page as an image
        // 100 DPI: general-quality
        // 300 DPI: high-quality
        // 600 DPI: pristine-quality
        BufferedImage image = pdfRenderer.renderImageWithDPI(pageIndex, 300);
        // Save the image to a file
        String imageFilePath = new StringBuilder()
            .append(imageFileDir)
            .append(File.separator)
            .append(file.getName().replaceAll("[.](pdf|PDF)", ""))
            .append("_")
            .append(pageIndex + 1)
            .append(".png")
            .toString();
        ImageIO.write(image, "PNG", new File(imageFilePath));
    }
    // Close the document
    document.close();
}

Image to PDF

private static void imageToPdf(String imagePath, String pdfPath) throws IOException {
    try (PDDocument doc = new PDDocument()) {
        PDPage page = new PDPage();
        doc.addPage(page);
        // createFromFile is the easiest way with an image file
        // if you already have the image in a BufferedImage,
        // call LosslessFactory.createFromImage() instead
        PDImageXObject pdImage = PDImageXObject.createFromFile(imagePath, doc);
        // draw the image at full size at (x=0, y=0)
        try (PDPageContentStream contents = new PDPageContentStream(doc, page)) {
            // to draw the image at PDF width
            int scaledWidth = 600;
            if (pdImage.getWidth() < 600) {
                scaledWidth = pdImage.getWidth();
            }
            contents.drawImage(pdImage, 0, 0, scaledWidth, pdImage.getHeight() * scaledWidth / pdImage.getWidth());
        }
        doc.save(pdfPath);
    }
}

Create PDFs

String outputFilePath = "output/pdf/filepath";

PDDocument document = new PDDocument();
PDPage page = new PDPage(PDRectangle.A4);
document.addPage(page);
// Create content stream to draw on the page
PDPageContentStream contentStream = new PDPageContentStream(document, page);
contentStream.setFont(PDType1Font.HELVETICA, 12);
// Insert text
contentStream.beginText();
contentStream.newLineAtOffset(100, 700);
contentStream.showText("Hello, World!");
contentStream.endText();
// Load the image
String imageFilePath = "C:\\Users\\Taogen\\Pictures\\icon.jpg";
PDImageXObject image = PDImageXObject.createFromFile(imageFilePath, document);
// Set the scale and position of the image on the page
float scale = 0.5f; // adjust the scale as needed
float x = 100; // x-coordinate of the image
float y = 500; // y-coordinate of the image
// Draw the image on the page
contentStream.drawImage(image, x, y, image.getWidth() * scale, image.getHeight() * scale);
contentStream.close();
document.save(outputFilePath);
document.close();

Compress (TODO)

Watermark (Todo)