Taogen's Blog

Stay hungry stay foolish.

Query

QueryWrapper

QueryWrapper

// usage 1
QueryWrapper<SysUser> queryWrapper = new QueryWrapper<SysUser>()
.eq("user_id", userId)
.like(name != null, "name", name);

// usage 2
QueryWrapper<SysUser> queryWrapper = Wrappers.query(sysUser)
.eq("user_id", userId);

LambdaQueryWrapper

// usage 1
LambdaQueryWrapper<SysUser> lambdaQueryWrapper = new LambdaQueryWrapper<SysUser>()
.eq(SysUser::getUserId, userId)
.like(name != null, SysUser::getName, name);

// usage 2
LambdaQueryWrapper<SysUser> lambdaQueryWrapper = Wrappers.lambdaQuery(sysUser)
.eq(SysUser::getUserId, userId);

// usage 3
LambdaQueryWrapper<SysUser> lambdaQueryWrapper = Wrappers.lambdaQuery(SysUser.class)
.eq(SysUser::getUserId, userId);

Convert QueryWrapper to LambdaQueryWrapper

LambdaQueryWrapper<SysUser> lambdaQueryWrapper = new QueryWrapper<SysUser>()
.eq("user_id", userId)
.lambda()
.eq(SysUser::getType, type);

Basic Query

An example

List<SysUser> sysUserList = userService.list(
new QueryWrapper<SysUser>()
.select("name")
.eq("type", type)
.orderByDesc("id")
.last("limit 10"));

from

new QueryWrapper<Entity>()

select

wrapper.select("column 1", "column 2")

where

wrapper.eq("type", 1)
.like("title", "test")
.in("user_id", userIds)
.ge("age", 18)
.apply("find_in_set({0}, type)", type);
.apply("date(gmt_create)=date({0})", new Date())
.apply("date_format(gmt_modified,'%Y-%m-%d') = date({0})", new Date())

and…or

// where type = 1 or type = 2
wrapper.and(wrapper -> wrapper.eq("type", 1)
.or().eq("type", 2))
// find_in_set(?, category_id) or find_in_set(?, category_id)
queryWrapper.and(wrapper -> {
for (int i = 0; i < categoryIds.size(); i++) {
if (i > 0) {
wrapper.or();
}
wrapper.apply("find_in_set({0}, category_id)", categoryIds.get(i));
}
return wrapper;
});
// where status = 0 and (type = 1 or (type > 10 and type < 20))
wrapper.eq("status", 0)
.and(wrapper -> wrapper.eq("type", 1)
.or(wrapper2 ->
wrapper2.gt("type", 10)
.lt("type", 20));

order by

wrapper.orderByDesc("id")

limit

wrapper.last("limit 10")

Query Conditions

find_in_set

wrapper.apply("find_in_set({0}, type)", type);

Date functions

date equal

wrapper.apply("date(gmt_create)=date({0})", new Date())
.apply("date_format(gmt_modified,'%Y-%m-%d') = date({0})", new Date())

Recent 7 days

wrapper.apply("create_time > DATE_SUB(NOW(), INTERVAL 7 DAY)"));

Page Query

  1. Request query string parameters

http://xxx.xxx/?current=1&size=10&orders[0].column=pubtime&orders[0].asc=true

  • current
  • size
  • orders[].column
  • orders[].asc
  1. Page Entity

com.baomidou.mybatisplus.extension.plugins.pagination.Page

Fields for Request parameters

  • long size
  • long current
  • List<OrderItem> orders
    • String column
    • boolean asc

Fields for response

  • List<T> records
  • long total
  1. Page Query Methods

MyBatis Plus page query method

IPage<T> selectPage(IPage<T> page, Wrapper<T> queryWrapper)

Custom page query in Mapper XML

IPage<MyResult> myPageQuery(@Param("page") Page page, @Param("param") MyParam param);
  • Pass com.baomidou.mybatisplus.extension.plugins.pagination.Page object as a parameter.
  • Using the IPage class as the return type.
  • Don’t need to add limit start, size in mapper XML. The SQL is query all rows. But MyBatis Plus automatically add limit at the end of SQL. If you want to query all, update to List<MyResult> queryAll(@Param("param") MyParam param);

One Column Query

List<Object> names = baseMapper.selectObjs(
new LambdaQueryWrapper<User>()
.select(User::getName)
.eq(User::getType, 1));
if (names == null) {
return Collections.emptySet();
} else {
return names.stream()
.map(Object::toString)
.collect(Collectors.toSet());
}

Aggregation Query

Aggregation Query methods

  • selectMaps()
List<Map<String, Object>> mapList = userMapper.selectMaps(
new QueryWrapper<SysUser>()
.select("type as name, count(*) as typeCount")
.groupBy("type"));
  • selectObjs()
List<Object> mapList = userMapper.selectObjs(
new QueryWrapper<SysUser>()
.select("sum(num) as typeTotal")
.groupBy("type"));
  • selectCount()
userMapper.selectCount(queryWrapper);

select

queryWrapper.select("count(*) as typeCount");
queryWrapper.select("submit_content_type as name, count(*) as value")

group by

queryWrapper.groupBy("type"));

having

queryWrapper.having("COUNT(*) > 10"))

Others

Non-query fields

Use @TableField(exist = false)

@TableField(exist = false)
private IdName creator;

Use @TableName(excludeProperty={})

@TableName(value = "my_table_name", excludeProperty = {"creator"})
public class User {
private IdName creator;
}

Using @TableField(condition = SqlCondition.LIKE)

@TableField(condition = SqlCondition.LIKE)  
private String name;

Using @TableField(whereStrategy = FieldStrategy.NOT_EMPTY)

  • IGNORED: 不判断
  • NOT_NULL: 非NULL判断
  • NOT_EMPTY: 非空判断

Using MySQL keyword as a column name

@TableField(value = "`type`")
private Integer type;

Using another entity name

@Alias("UserV2")
public class User {
}

DML

update it to null when it value is null

@TableField(fill = FieldFill.UPDATE)
private String name;

logical delete

// Note that the field for logical deletion cannot be modified by update() method
@TableLogic(value="0",delval="1")
private Boolean delFlag;

Update

userService.update(new UpdateWrapper<User>()
.set("name", "updateName")
.eq("id", 1));

MyBatis Plus Configurations

mybatis-plus:
mapper-locations: classpath:mybatis/mapper/*.xml
type-aliases-package: com.example.entity,com.example.modules.**.entity
global-config:
db-config:
where-strategy: not_empty
logic-delete-field: deleteFlag
logic-delete-value: 1
logic-not-delete-value: 0

More configurations reference MyBatis Plus使用配置

Background

Running the Spring Boot project occurs an error.

Error Info

Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'spring.datasource.druid.initialSize' in value "${spring.datasource.druid.initialSize}"
at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:178)
... 97 common frames omitted

Solutions

1. Check that the property you need exists in a configuration file. Find out in which configuration file your injected property is.

2. Check whether the property spring.profiles.active value is correct in application.yml. Make sure the active profile is in your project.

3. Check whether your injected property exists in application.yml (or application.properties) and the active profile application-xxx.yml (or application-xxx.properties).

Reasons

The value of the property spring.profiles.active in my application.yml is spring.profiles.active=pro, but I don’t have the application-pro.yml file. The property’s value spring.profiles.active=pro should update to prod.

Background

When I do an insert operation to save the item data to MySQL, occurs an error.

Error Info

Error updating database.  Cause: java.sql.SQLException: Data truncated for column 'xxx' at row 1

Solutions

Check whether the column type is right.

Check whether the length of the field value over the length of the column.

Reasons

The length of the field value over the length of the column.

My passed field value is a set type value, but the column type is enum. The column type should update to set.

Background

When I use the FreeMarker to generate text files, occurs an error: Failed to “?eval” string.

<#assign dataFromConfigJson = columnMapping.formComponentDataFromConfig?eval>

Error Info

FreeMarker template error:
Failed to "?eval" string with this error:

---begin-message---
Syntax error in ?eval-ed string in line 1, column 175:
Encountered "}", but was expecting one of:
<STRING_LITERAL>
<RAW_STRING>
"false"
"true"
<INTEGER>
<DECIMAL>
"."
".."
<DOT_DOT_LESS>
"..*"
"?"
"??"
"+"
"-"
"!"
","
"["
"]"
"("
"{"
<ID>
<TERMINATING_EXCLAM>
"+"
"-"
---end-message---

The failing expression:
==> columnMapping.formComponentDataFromConfig?eval [in template "templates/demo\\html\\test.ftl" at line 208, column 41]

----
FTL stack trace ("~" means nesting-related):
- Failed at: #assign dataFromConfigJson = columnMa... [in template "templates/demo\\html\\test.ftl" in macro "generateFormItems" at line 208, column 11]
- Reached through: @generateFormItems type="search" [in template "templates/demo\\html\\test.ftl" at line 582, column 7]
----

Solutions

Corrects the JSON string of the field of data that passed to the FreeMarker Template object.

Reasons

The JSON string is invalid.

问题1:扩展显示,外接显示器模糊

解决方案:笔记本和外接显示器的缩放比例保持一致。

Display settings -> Scale and layout

问题2:扩展显示,合上笔记本时,外接显示器变模糊

解决方案:将外接屏幕设为主显示器,并设置“仅显示外接显示器”。

Display settings -> Rearrange your display -> Click 2 for setting extended monitor

Display settings -> Multiple displays -> select “Show only on 2”, and checked “Make this main display”

问题3:笔记本使用HDMI接口连接了显示器后没声音

Solution: 声音设置中选择笔记本的扬声器作为声音输出设备。

Reason: HDMI接口是包含音频+视频两种信号的接口。当电脑的HDMI接口被使用时,系统就会默认从HDMI设备输出声音信号。

在我看来,编程它既是脑力活,也是体力活。脑力活在于如何解决未知的问题,体力活则在于一行一行地敲代码完成已知的问题。

编程中最累人的便是面对未知问题。如:未知的技术栈,未知的系统设计,未知的设计模式和类的继承结构,未知的算法实现,未知的异常等等。遇到未知问题时,我们会注意力保持高度集中,不断的失败,不断的尝试,我们绞尽脑汁想要解决这个问题,同时我们感到沮丧和急躁。然而,长时间的保持紧绷的神经和僵硬的坐姿会严重影响我们的健康状态。作为一名程序员,为了保持良好的身心健康,我们要学会如何与未知相处。

  1. 从源头出发,减少未知

我们不可能去知道所有的技术和知识。但是我们可以把自己经历过的问题,总结一下,下次遇到相似的问题,可以轻松地应对。我们也可以尽量地学习和了解一些你所处领域相关的你没有掌握的技术。

  1. 承认自己遇到了难题,并给与自己信心

每个人掌握着有限的知识,每个人都会遇到难题。遇到难题不能完全代表一个人的能力水平,难题给了我们一次成长的机会。我们尽力去解决问题,最终无论我们能不能解决这个难题,我们都会有所收获。

  1. 给与解决问题充足的时间,让问题慢慢变得明朗

解决未知问题的过程就是把未知慢慢变成已知的过程。我们在解决未知问题的过程中,经过了一段时间的思考之后,我们会发现自己有了点思路或者有了点感觉,慢慢地问题就逐渐变得越来越明朗了。

保持心平气和地去解决未知问题,这是急不来的,我们需要给大脑一些时间去理解和记忆。

Content

Upload

Upload files with Spring framework

@PostMapping(value = "/uploadFile", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public String uploadFile(@RequestParam("file") MultipartFile file, String moduleUri) throws IOException {
InputStream inputStream = file.getInputStream();
String fileName = file.getOriginalFilename();
// to save file
...
}

Upload file with Java Servlet

public class FileUploadByCommonsFileUploadServlet extends HttpServlet {
public void doPost(HttpServletRequest request, HttpServletResponse response) throws Exception {
DiskFileItemFactory diskFileItemFactory = new DiskFileItemFactory();
diskFileItemFactory.setSizeThreshold(MAX_MEMORY_SIZE);
diskFileItemFactory.setRepository(new File(System.getProperty("java.io.tmpdir")));
ServletFileUpload servletFileUpload = new ServletFileUpload(diskFileItemFactory);
servletFileUpload.setFileSizeMax(MAX_FILE_SIZE);
servletFileUpload.setSizeMax(MAX_FILE_SIZE);
List<FileItem> formItems = servletFileUpload.parseRequest(request);
for (FileItem item : formItems) {
if (!item.isFormField()) {
String fileName = new File(item.getName()).getName();
String filePath = new StringBuilder()
.append(UPLOAD_DIR)
.append(File.seperator)
.append(fileName)
.toString()
item.write(new File(filePath));
}
}
}
...
}

Download

Download file with spring framework

Write input streams to the OutPutStream of the HttpServletResponse

@GetMapping(value = "/download" )
public void download(HttpServletResponse response, String fileUri) {
response.setCharacterEncoding("utf-8");
response.setContentType("application/octet-stream");
response.setHeader("Content-Disposition", "attachment;fileName=\"" + URLEncoder.encode(filename, "UTF-8").replace("+", "%20") + "\"");
ServletOutputStream outputStream = response.getOutputStream();
int bufSize = 1024;
byte[] buffer = new byte[bufSize];
int len;
// get filePath by fileUri
...
// get input stream by filePath
...
while ((len = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, len);
}
inputStream.close();
}

Write file Resource to ReponseBody

@GetMapping(value = "/download" )
public ResponseEntity<Resource> download(String fileUri) {
// get filePath by fileUri
//...
String uploadDir = "D:\\upload";
Path path = Paths.get(uploadDir).toAbsolutePath().normalize();
path = path.resolve(fileUri).normalize();
Resource resource = new UrlResource(path.toUri());
String resultFileName = URLEncoder.encode(resource.getFilename().replace("%20", " "), "UTF-8").replace("+", "%20");
return ResponseEntity.ok()
.contentType(MediaType.parseMediaType("application/octet-stream"))
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + resultFileName + "\"")
.body(resource);
}

Setting Response Headers

  • content-type: application/octet-stream
  • content-disposition: attachment; filename=”test.txt”

To encode filename when download file

original file name: “中 文.txt”

filename = resource.getFilename() // 中%20文.txt

browser download filename: -%20‡.txt

1

filename = URLEncoder.encode(resource.getFilename().replace("%20", " "), "UTF-8") // %E4%B8%AD+%E6%96%87.txt

browser download filename: 中+文.txt

2

filename = URLEncoder.encode(resource.getFilename().replace("%20", " "), "UTF-8").replace("+", "%20") // %E4%B8%AD%20%E6%96%87.txt

browser download filename: 中 文.txt

When browser resolve file name, they don’t decode “+” to “ “, so we need replace “+” to “%20”.

The Example of Upload and download files in spring framework

@RestController
@RequestMapping("files")
public class MyFileController {
private static final Logger logger = LogManager.getLogger();

public static final String UPLOAD_DIR = "D:\\My Desktop\\upload";

public static final String DOWNLOAD_PREFIX = "/files/download";

@PostMapping(value = "/uploadFile", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public String uploadFile(@RequestParam("file") MultipartFile file, String moduleUri) throws IOException {
String uploadFileUri = getUploadFileUri(file, moduleUri);
String uploadFilePath = new StringBuilder()
.append(UPLOAD_DIR)
.append(File.separator)
.append(uploadFileUri)
.toString();
ensureDirectoryExist(uploadFilePath);
try (
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(
new FileOutputStream(uploadFilePath));
) {
byte[] buf = new byte[1024];
int len;
InputStream inputStream = file.getInputStream();
while ((len = inputStream.read(buf)) != -1) {
bufferedOutputStream.write(buf, 0, len);
}
}
return new StringBuilder()
.append(DOWNLOAD_PREFIX)
.append("/")
.append(uploadFileUri)
.toString();
}

private String getUploadFileUri(MultipartFile file, String moduleUri) {
String uuid = UUID.randomUUID().toString().replaceAll("-", "");
Calendar calendar = Calendar.getInstance();
StringBuilder dateUri = new StringBuilder()
.append(calendar.get(Calendar.YEAR))
.append("/")
.append(calendar.get(Calendar.MONTH) + 1)
.append("/")
.append(calendar.get(Calendar.DAY_OF_MONTH));
StringBuilder resultFileUri = new StringBuilder()
.append(moduleUri)
.append("/")
.append(dateUri)
.append("/")
.append(uuid)
.append("/")
.append(file.getOriginalFilename());
return resultFileUri.toString();
}

private void ensureDirectoryExist(String filePath) {
filePath = filePath.replace("/", File.separator).replace("\\", File.separator);
String fileDir = filePath.substring(0, filePath.lastIndexOf(File.separator));
logger.debug("fileDir: {}", fileDir);
File dir = new File(fileDir);
if (!dir.exists()) {
dir.mkdirs();
}
}

@GetMapping(value = "download/**")
public void download(HttpServletRequest request, HttpServletResponse response) throws IOException {
String servletPath = request.getServletPath();
String fileUri = servletPath.substring(DOWNLOAD_PREFIX.length());
String filePath = new StringBuilder()
.append(UPLOAD_DIR)
.append(File.separator)
.append(fileUri)
.toString();
try (
BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream(filePath))
) {
ServletOutputStream outputStream = response.getOutputStream();
byte[] buf = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buf)) != -1) {
outputStream.write(buf, 0, len);
}
}
}
}

Read

Read data from disk

java.io.FileInputStream

InputStream is = new FileInputStream("D:\\My Desktop\\test.txt");

org.springframework.core.io.UrlResource

Spring framework Resource Interface

Java’s standard java.net.URL class and standard handlers for various URL prefixes unfortunately are not quite adequate enough for all access to low-level resources. For example, there is no standardized URL implementation that may be used to access a resource that needs to be obtained from the classpath, or relative to a ServletContext. While it is possible to register new handlers for specialized URL prefixes (similar to existing handlers for prefixes such as http:), this is generally quite complicated, and the URL interface still lacks some desirable functionality, such as a method to check for the existence of the resource being pointed to.

String filepath = "D:\\My Desktop\\test.txt";
Path path = Paths.get(filepath).normalize();
//path.toUri().toString(): file:///D:/My%20Desktop/test.txt
Resource resource = new UrlResource(path.toUri());
InputStream is = resource.getInputStream();

java.net.URL

String filepath = "D:\\My Desktop\\test.txt";
Path path = Paths.get(filepath).normalize();
//path.toUri().toString(): file:///D:/My%20Desktop/test.txt
InputStream is = new URL(path.toUri().toString()).openStream();

Read data from Java classpath

org.springframework.core.io.ClassPathResource

// src/main/resources/application.yml
Resource resource = new ClassPathResource("application.yml");
InputStream is = resource.getInputStream();

org.springframework.core.io.ResourceLoader

@Autowired
ResourceLoader resourceLoader;

resourceLoader.getResource("classpath:application.yml");
InputStream is = resource.getInputStream();

Read data from memory

byte[] data = new String("data...").getBytes(StandardCharsets.UTF_8);
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(data);

Read data From HTTP URL

java.net.URL

InputStream is = new URL("http://demo.com/test.txt").openStream();

org.springframework.core.io.UrlResource

Resource resource = new UrlResource("http://demo.com/test.txt");
InputStream is = resource.getInputStream();

java.net.HttpURLConnection

URLConnection urlConnection = new URL("http://demo.com/test.txt").openConnection();
urlConnection.connect();
InputStream is = urlConnection.getInputStream();

Read data from OSS

Reference my another post: Common OSS Java SDK Usage

Write

Write data to disk

byte[] data = new String("data").getBytes(StandardCharsets.UTF_8);
String storeFilePath = "D:/My Workspace/test/test.txt";
try (
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(data);
FileOutputStream fileOutputStream = new FileOutputStream(storeFilePath)
) {
byte[] buf = new byte[1024];
int len = 0;
while ((len = byteArrayInputStream.read(buf)) != -1) {
fileOutputStream.write(buf, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
}

Write data to memory

byte[] data = new String("data").getBytes(StandardCharsets.UTF_8);
byte[] storeDataArray;
try (
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(data);
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
) {
byte[] buf = new byte[1024];
int len = 0;
while ((len = byteArrayInputStream.read(buf)) != -1) {
byteArrayOutputStream.write(buf, 0, len);
}
storeDataArray = byteArrayOutputStream.toByteArray();
System.out.println(new String(storeDataArray, StandardCharsets.UTF_8));
} catch (IOException e) {
e.printStackTrace();
}

Write data to OSS

Reference my another post: Common OSS Java SDK Usage

stream

InputStream is = new FileInputStream("D:\\My Desktop\\test.txt");
new BufferedReader(new InputStreamReader(is)).lines().forEach(System.out::println);

java.nio.file.Files

Path path = Paths.get("D:\\My Desktop\\test.txt");
System.out.println(new String(Files.readAllBytes(path)));

Package files to zip

basic zip

try (
FileOutputStream fileOutputStream = new FileOutputStream("D:/My Workspace/test/test.zip");
ZipOutputStream zipOutputStream = new ZipOutputStream(fileOutputStream)
) {
int fileNumber = 3;
for (int i = 1; i <= fileNumber; i++) {
String childFilename = "file" + i + ".txt";
zipOutputStream.putNextEntry(new ZipEntry(childFilename));
zipOutputStream.write(("data" + i).getBytes(StandardCharsets.UTF_8));
zipOutputStream.closeEntry();
}
zipOutputStream.finish();
} catch (IOException e) {
e.printStackTrace();
}

nested zip

try (
FileOutputStream fileOutputStream = new FileOutputStream("D:/My Workspace/test/test.zip");
ZipOutputStream zipOutputStream = new ZipOutputStream(fileOutputStream)
) {
int fileNumber = 3;
for (int i = 1; i <= fileNumber; i++) {
String childFilename = "file" + i + ".zip";
zipOutputStream.putNextEntry(new ZipEntry(childFilename));
try (
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
ZipOutputStream nestedZipOutputStream = new ZipOutputStream(byteArrayOutputStream);
) {
int nestedNumber = 3;
for (int j = 1; j <= nestedNumber; j++) {
String nestedFileName = new StringBuilder()
.append("file").append(i)
.append("nestFile").append(j)
.append(".txt").toString();
nestedZipOutputStream.putNextEntry(new ZipEntry(nestedFileName));
nestedZipOutputStream.write(("nest file" + j).getBytes(StandardCharsets.UTF_8));
nestedZipOutputStream.closeEntry();
}
nestedZipOutputStream.finish();
zipOutputStream.write(byteArrayOutputStream.toByteArray());
}
zipOutputStream.closeEntry();
}
zipOutputStream.finish();
} catch (IOException e) {
e.printStackTrace();
}

Speed up read or write

Using buffer to decrease calls to the underlying runtime system

  1. Direct Buffering
String inputFilePath = "C:\\Users\\Taogen\\Desktop\\input.txt";
String outputFilePath = "C:\\Users\\Taogen\\Desktop\\output.txt";
try (
FileInputStream fileInputStream = new FileInputStream(inputFilePath);
FileOutputStream fileOutputStream = new FileOutputStream(outputFilePath)
) {
byte[] buf = new byte[2048];
int len = 0;
while ((len = fileInputStream.read(buf)) != -1) {
fileOutputStream.write(buf, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
}
  1. Using buffered Input/output Stream
String inputFilePath = "C:\\Users\\Taogen\\Desktop\\input.txt";
String outputFilePath = "C:\\Users\\Taogen\\Desktop\\output.txt";
try (
FileInputStream fileInputStream = new FileInputStream(inputFilePath);
BufferedInputStream bufferedInputStream = new BufferedInputStream(fileInputStream);
FileOutputStream fileOutputStream = new FileOutputStream(outputFilePath);
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
) {

int b;
while ((b=bufferedInputStream.read()) != -1) {
bufferedOutputStream.write(b);
}
} catch (IOException e) {
e.printStackTrace();
}

BufferedInputStream‘s default buffer size is int DEFAULT_BUFFER_SIZE = 8192;

  1. BufferedInputStream + Buffer array
String inputFilePath = "C:\\Users\\Taogen\\Desktop\\recovery_data.sql";
String outputFilePath = "C:\\Users\\Taogen\\Desktop\\recovery_data2.sql";
try (
FileInputStream fileInputStream = new FileInputStream(inputFilePath);
BufferedInputStream bufferedInputStream = new BufferedInputStream(fileInputStream);
FileOutputStream fileOutputStream = new FileOutputStream(outputFilePath);
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
) {
byte[] buf = new byte[1024];
int len = 0;
while ((len = bufferedInputStream.read(buf)) != -1) {
bufferedOutputStream.write(buf, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
}

Time cost: BufferedInputStream + Direct Buffer array < Direct Buffer array < Using a BufferedInputStream < Read Method.

Using BufferedInputStream rather than direct buffer is probably “right” for most applications. Because your used buffer size of direct buffer may be worse than BufferedInputStream default buffer size in speed.

Will making the buffer bigger make I/O go faster? Java buffers typically are by default 1024 or 2048 bytes long. A buffer larger than this may help speed I/O, but often by only a few percent, say 5 to 10%.

Problems

Max upload file size

Character encoding of file name

URL encode of file name

Appendixes

Temporary Files and Directories

java.io.tmpdir

System.getProperty("java.io.tmpdir")

Windows 10: C:\Users\{user}\AppData\Local\Temp\

Debian: /tmp

Create temporary file

// If you don't specify the file suffix, the default file suffix is ".tmp".
File file = File.createTempFile("temp", null);
System.out.println(file.getAbsolutePath());
file.deleteOnExit();

Common Content-Type (MIME types) Table

Java get file’s mimeType

// 1
String mimeType = Files.probeContentType(file.toPath());
// 2
String mimeType = URLConnection.guessContentTypeFromName(fileName);
// 3
FileNameMap fileNameMap = URLConnection.getFileNameMap();
String mimeType = fileNameMap.getContentTypeFor(file.getName());
// 4
MimetypesFileTypeMap fileTypeMap = new MimetypesFileTypeMap();
String mimeType = fileTypeMap.getContentType(file.getName());
Content-Type(Mime-Type) Kind of document Extension
application/octet-stream Any kind of binary data .bin
text/plain Text, (generally ASCII or ISO 8859-n) .txt
<Images>
image/bmp Windows OS/2 Bitmap Graphics .bmp
image/jpeg JPEG images .jpeg .jpg
image/png Portable Network Graphics .png
image/gif Graphics Interchange Format (GIF) .gif
image/vnd.microsoft.icon Icon format .ico
image/svg+xml Scalable Vector Graphics (SVG) .svg
<media>
audio/mpeg MP3 audio .mp3
video/mp4 MP4 video .mp4
<code>
text/css Cascading Style Sheets (CSS) .css
text/csv Comma-separated values (CSV) .csv
application/json JSON format .json
text/javascript JavaScript, JavaScript module .js .mjs
text/html HyperText Markup Language (HTML) .htm .html
<doc>
application/msword Microsoft Word .doc
application/vnd.openxmlformats-officedocument.wordprocessingml.document Microsoft Word (OpenXML) .docx
application/pdf Adobe Portable Document Format (PDF) .pdf
<archive>
application/gzip GZip Compressed Archive .gz
application/vnd.rar RAR archive .rar
application/x-tar Tape Archive (TAR) .tar
application/zip ZIP archive .zip
application/x-7z-compressed 7-zip archive .7z

Two primary MIME types are important for the role of default types:

  • text/plain is the default value for textual files. A textual file should be human-readable and must not contain binary data.
  • application/octet-stream is the default value for all other cases. An unknown file type should use this type. Browsers pay a particular care when manipulating these files, attempting to safeguard the user to prevent dangerous behaviors.

References

[1] Tuning Java I/O Performance

[2] Common MIME types - MDN Web Docs

MyBatis

MyBatis is a first class persistence framework with support for custom SQL, stored procedures and advanced mappings. MyBatis eliminates almost all of the JDBC code and manual setting of parameters and retrieval of results. MyBatis can use simple XML or Annotations for configuration and map primitives, Map interfaces and Java POJOs (Plain Old Java Objects) to database records.

  1. Add dependencies
  • mybatis
  • mysql-connector-java
<project ...>
<dependencies>
<!-- MyBatis -->
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<version>${latest_version}</version>
</dependency>
<!-- MySQL Connector -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${latest_version}</version>
</dependency>
</dependencies>
</project>
  1. Write the mybatis-config.xml file
  • Configuring the data source.
  • Inject mappers
<configuration>
<environments default="development">
<environment id="development">
<dataSource type="POOLED">
<property name="driver" value="${MYSQL_DRIVER_CLASS}"/>
<property name="url" value="${MYSQL_URL}"/>
<property name="username" value="${MYSQL_USER}"/>
<property name="password" value="${MYSQL_PASSWD}"/>
</dataSource>
</environment>
</environments>
<mappers>
<mapper resource="mybatis/mapper/UserMapper.xml" />
</mappers>
</configuration>
  1. Write maper XML file
<mapper>
...
</mapper>
  1. Usage
  • Load mybatis-config.xml
  • Build SqlSessionFactory object
  • Get SqlSession object
  • Get mapper object
  • Call methods of mapper object
public static void main(String[] args) throws IOException {
String resource = "mybatis/mybatis-config.xml";
InputStream inputStream = Resources.getResourceAsStream(resource);
SqlSessionFactory sqlSessionFactory = new SqlSessionFactoryBuilder().build(inputStream);
try (SqlSession session = sqlSessionFactory.openSession()) {
UserMapper userMapper = session.getMapper(UserMapper.class);
User user = userMapper.selectByPrimaryKey(1);
logger.debug("user is {}", user);
}
}

MyBatis with Spring

MyBatis-Spring integrates MyBatis seamlessly with Spring. This library allows MyBatis to participate in Spring transactions, takes care of building MyBatis mappers and SqlSessions and inject them into other beans, translates MyBatis exceptions into Spring DataAccessExceptions, and finally, it lets you build your application code free of dependencies on MyBatis, Spring or MyBatis-Spring.

  1. Add dependencies
  • spring related
  • mybatis
  • mybatis-spring
  • mysql-connector-java
<!-- Spring -->
...
<!-- mybatis -->
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<version>${mybatis.version}</version>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis-spring</artifactId>
<version>${mybatis.spring.version}</version>
</dependency>
<!-- mysql -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysql.jdbc.version}</version>
</dependency>
  1. Write spring-mybatis.xml configuration file
  • Configuring dataSource
  • Configuring sqlSessionFactory for specifying mapper xml files location.
  • Configuring MapperScannerConfigurer for specifying mapper interface Java files location.
<bean id="dataSource" class="com.alibaba.druid.pool.DruidDataSource"
init-method="init" destroy-method="close">
...
</bean>
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="mapperLocations" value="classpath:mybatis/mapper/*.xml"/>
</bean>
<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
<property name="basePackage" value="com.taogen.example.dao"/>
<property name="sqlSessionFactoryBeanName" value="sqlSessionFactory"/>
</bean>
  1. Write maper XML file
<mapper>
...
</mapper>
  1. Usage
  • Call Dao interface method
@Service
public class UserServiceImpl implements UserService {
@Autowired
private UserDao userDao;

@Override
public List<User> listAllUsers() {
return userDao.listAllUsers();
}
...
}

MyBatis with Spring Boot

The MyBatis-Spring-Boot-Starter help you build quickly MyBatis applications on top of the Spring Boot.

By using this module you will achieve:

  • Build standalone applications
  • Reduce the boilerplate to almost zero
  • Less XML configuration

MyBatis-Spring-Boot-Starter will:

  • Autodetect an existing DataSource
  • Will create and register an instance of a SqlSessionFactory passing that DataSource as an input using the SqlSessionFactoryBean
  • Will create and register an instance of a SqlSessionTemplate got out of the SqlSessionFactory
  • Auto-scan your mappers, link them to the SqlSessionTemplate and register them to Spring context so they can be injected into your beans
  1. Add dependencies
  • mybatis-spring-boot-starter
  • mysql-connector-java
<!-- spring boot -->
...
<!-- data access -->
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
  1. Write Spring Boot configuration file application.yml
  • Configuring data sources
  • Configuring mybatis for specifying mapper xml files location.
spring:
datasource:
url: jdbc:mysql://localhost:3306/my_test?serverTimezone=UTC&useUnicode=true&characterEncoding=utf-8
username: root
password: root
driverClassName: com.mysql.cj.jdbc.Driver
mybatis:
mapper-locations: classpath:mybatis/mapper/*.xml
type-aliases-package: com.taogen.demo.springbootcrud.module.*.entity
  1. Write maper XML file
<mapper>
...
</mapper>
  1. Definition mapper interfaces

The MyBatis-Spring-Boot-Starter will search, by default, for mappers marked with the @Mapper annotation.

@Mapper
public interface UserDao {
}

You may want to specify a custom annotation or a marker interface for scanning. If so, you must use the @MapperScan annotation.

@MapperScan(value = "com.taogen.example.dao")

The value attribute is a alias for the basePackages attribute.

  1. Usage
  • Call Dao interface method
@Service
public class UserServiceImpl implements UserService {
@Autowired
private UserDao userDao;

@Override
public List<User> listAllUsers() {
return userDao.listAllUsers();
}
...
}

MyBatis Plus with Spring Boot

MyBatis-Plus (opens new window)(简称 MP)是一个 MyBatis (opens new window)的增强工具,在 MyBatis 的基础上只做增强不做改变,为简化开发、提高效率而生。

  • 无侵入:只做增强不做改变,引入它不会对现有工程产生影响,如丝般顺滑。
  • 损耗小:启动即会自动注入基本 CURD,性能基本无损耗,直接面向对象操作。
  • 强大的 CRUD 操作:内置通用 Mapper、通用 Service,仅仅通过少量配置即可实现单表大部分 CRUD 操作,更有强大的条件构造器,满足各类使用需求。
  • 支持 Lambda 形式调用:通过 Lambda 表达式,方便的编写各类查询条件,无需再担心字段写错。
  1. Add dependencies
  • mybatis-plus-boot-starter
  • mysql-connector-java
<!-- spring boot -->
...
<!-- data access -->
<dependency>
<groupId>com.baomidou</groupId>
<artifactId>mybatis-plus-boot-starter</artifactId>
<version>3.4.3</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
  1. Write Spring Boot configuration file application.yml
  • Configuring data sources
  • Configuring mybatis for specifying mapper xml files location. (If you just use mybatis plus service and dao method, you needn’t configure this.)
spring:
datasource:
url: jdbc:mysql://localhost:3306/my_test?serverTimezone=UTC&useUnicode=true&characterEncoding=utf-8
username: root
password: root
driverClassName: com.mysql.cj.jdbc.Driver
mybatis:
mapper-locations: classpath:mybatis/mapper/*.xml
type-aliases-package: com.taogen.demo.springbootcrud.module.*.entity
  1. Write maper XML file

(If you just use mybatis plus service and dao method, you needn’t this.)

<mapper>
...
</mapper>
  1. Definition mapper interfaces

The MyBatis-Spring-Boot-Starter will search, by default, for mappers marked with the @Mapper annotation.

import com.baomidou.mybatisplus.core.mapper.BaseMapper;

@Mapper
public interface UserDao extends BaseMapper<User> {
}

You may want to specify a custom annotation or a marker interface for scanning. If so, you must use the @MapperScan annotation.

@MapperScan(value = "com.taogen.example.dao")

The value attribute is a alias for the basePackages attribute.

  1. extends MyBatis Plus class

dao

import com.baomidou.mybatisplus.core.mapper.BaseMapper;

@Mapper
public interface UserDao extends BaseMapper<User> {
}

service

import com.baomidou.mybatisplus.extension.service.IService;

public interface UserService extends IService<User> {
}
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;

@Service
public class UserServiceImpl extends ServiceImpl<UserDao, User> implements UserService {
}
  1. Usage
  • Call MyBatis Plus CRUD methods
QueryWrapper<User> queryWrapper = new QueryWrapper();
List<User> userList = userService.list(queryWrapper);

More MyBatis Plus method reference CRUD接口

References

[1] mybatis

[2] mybatis-spring

[3] mybatis-spring-boot-autoconfigure

[4] MyBatis-Plus 快速开始

Today I try to delete a directory with 485,000+ files on Linux server. It shows the error message below.

/bin/rm: Argument list too long.

If you’re trying to delete a very large number of files in a single directory at one time, you will probably run into this error. The problem is that when you type something like rm -f /data/*. So how do we working with directories with a large number of files? Let talk about it below.

Show first files

To show first n files in the directory instead of to show all files.

ls -U | head -10

Count the number of files

ls -f | wc -l

Remove files

Using for Loop

for f in *.pdf; do echo rm "$f"; done
for f in *.pdf; do rm "$f"; done

Using Find

Using find -delete. (2000 files/second, it is about three time faster rm)

find . -maxdepth 1 -type f -name "$.split-*" -print -delete

Using find and xargs. To find every file and pass them one-by-one to the “rm” command (not recommend, xargs is dangerous*)

find . -type f -exec rm -v {} \;
find . -maxdepth 1 -name "$.split-*" -print0 | xargs -0 rm

View and Edit large files

Open the 4.14 GB sql file

Computer

  • ThinkPad T14s
  • Power mode: Best performance
  • Processor: 10th Generation Intel® Core™ i5-10210U Processor (1.60 GHz, up to 4.20 GHz with Turbo Boost, 4 Cores, 8 Threads, 6 MB Cache)
  • Memory: 16GB
  • Hard Drive: 512 GB PCIe SSD

Result

  • vim: opened about 13 seconds. saved about 20 seconds.
  • sublime: opened 3 min 30 seconds. saved about 13 seconds.
  • less: opened instantly. can’t edit.
  • Intellij IDEA: opened about 3 seconds. can’t edit. and tip “The file is too large. Read-only mode.”
  • emacs: opened about 15 seconds. saving crashed.
  • Nodepad++: Can’t open. Error: file is too big opend by Notepad++.
  • VS code: Can’t open. open crashed

Best overall: vim.

Best for view: less or Intellij IDEA.

Export data from or Import data to remote MySQL server

Export 4.14 GB data from remote MySQL server

Cost time: 3 hours.

best exporting data at night.

0%