Taogen's Blog

Stay hungry stay foolish.

Posted on Edited on In ,

HTML 元素原则上应该保持唯一，但没有硬性要求。

存在多个相同的 id 元素时，CSS 对所有元素都生效。

<style>
    #myId {
        color:red;
    }
</style>
<div id="myId">
    div1
</div>
<div id="myId">
    div2
</div>

然而，JavaScript 和 jQuery 都只获取多个相同的 id 元素的第一个元素。

document.getElementById("myId").innerHTML //div1
$("#myId").html() //div1

Posted on Edited on In

Background

Using List<Integer> ids to receive a form parameter of HTTP request. The size of the parameter ids is over 255.

Error Info

org.springframework.beans.InvalidPropertyException: Invalid property 'ids[256]' of bean class [com.fantasi.manage.console.web.modules.recovery.dto.RecoveryGroupDto]: Invalid list index in property path 'ids[256]'; nested exception is java.lang.IndexOutOfBoundsException: Index: 256, Size: 256.

Solutions

To override init Binder method and configure the limit size which needs so that you can pass those many objects to your controller classes.

@Controller
public class MyController {

    @InitBinder
    public void initBinder(WebDataBinder dataBinder) {
        dataBinder.setAutoGrowCollectionLimit(500);
    }

    ...
}

@InitBinder plays the role to identify the methods which initialize the WebDataBinder.

WebDataBinder is a DataBinder that binds request parameter to JavaBean objects.

Reasons

By default spring only maps only 255 objects to the java bean. Spring developers has done this to prevent OutOfMemory problem.

References

[1] Spring initBinder for Handling Large List of Java Objects

Posted on Edited on In ,

Encode

Base64

String text = "hello 世界";
String encodeText = Base64.getEncoder().encodeToString(text.getBytes(StandardCharsets.UTF_8));
System.out.println(encodeText);
String decodeText = new String(Base64.getDecoder().decode(encodeText), StandardCharsets.UTF_8);
System.out.println(decodeText);

URL Encode

String text = "hello 世界";
String encodeText = URLEncoder.encode(text, StandardCharsets.UTF_8.name());
System.out.println(encodeText);
String decodeText = URLDecoder.decode(encodeText, StandardCharsets.UTF_8.name());
System.out.println(decodeText);

Hash Algorithms

MD5

MD5 (Message Digest 5)

String text = "123456";
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
messageDigest.reset();
messageDigest.update(text.getBytes(StandardCharsets.UTF_8));
byte[] digest = messageDigest.digest();
String hashText = DatatypeConverter.printHexBinary(digest);
System.out.println(hashText);

Note: the MessageDigest is not thread-safe. Consequently, we should use a new instance for every thread.

SHA

SHA (Secure Hash Algorithm)

SHA-256 by MessageDigest

String text = "123456";
MessageDigest messageDigest = MessageDigest.getInstance("SHA-256");
messageDigest.reset();
messageDigest.update(text.getBytes(StandardCharsets.UTF_8));
byte[] digest = messageDigest.digest();
String hashText = DatatypeConverter.printHexBinary(digest);
System.out.println(hashText);

SHA3-256 by MessageDigest (since Java 9)

String text = "123456";
MessageDigest messageDigest = MessageDigest.getInstance("SHA3-256");
messageDigest.reset();
messageDigest.update(text.getBytes(StandardCharsets.UTF_8));
byte[] digest = messageDigest.digest();
String hashText = DatatypeConverter.printHexBinary(digest);
System.out.println(hashText);

Conclusion

  • SHA256 is difficult to handle than MD5 because of its size.
  • SHA256 is less secure than MD5
  • MD5 result in an output of 128 bits whereas SHA256 result output of 256 bits.

Concluding all points, it will be better to use MDA5 if you want to secure your files otherwise you can use SHA256.

Symmetric Encryption Algorithms

DES

DES (data encryption standard, 1976)

DES is Not Secure.

3DES

3DES is Not Secure.

AES

AES (advanced encryption system, 2001)

The AES algorithm has six modes of operation:

  1. ECB (Electronic Code Book) Not Recommend
  2. CBC (Cipher Block Chaining)
  3. CFB (Cipher FeedBack)
  4. OFB (Output FeedBack)
  5. CTR (Counter)
  6. GCM (Galois/Counter Mode)

Don’t use AES Electronic codebook (ECB) Mode. The AES ECB mode, or AES/ECB/PKCS5Padding (in Java) is not semantically secure.

AES encryption best practice: Don’t reuse IV with the same key.

IV

The IV (initial value or initial vector), it is random bytes, typically 12 bytes or 16 bytes. In Java, we can use SecureRandom to generate the random IV.

public static IvParameterSpec generateIv() {
    byte[] iv = new byte[16];
    new SecureRandom().nextBytes(iv);
    return new IvParameterSpec(iv);
}

secret key

The AES secret key, either AES-128 or AES-256. In Java, we can use KeyGenerator to generate the AES secret key.

public static SecretKey getAESKey() throws NoSuchAlgorithmException {
    KeyGenerator keyGen = KeyGenerator.getInstance("AES");
    keyGen.init(256, SecureRandom.getInstanceStrong());
    return keyGen.generateKey();
}

The AES secret key that derived from a given password. In Java, we can use the SecretKeyFactory and PBKDF2WithHmacSHA256 to generate an AES key from a given password.

public static SecretKey getAESKeyFromPassword(char[] password, byte[] salt)
    throws NoSuchAlgorithmException, InvalidKeySpecException {
    SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
    // iterationCount = 65536
    // keyLength = 256
    KeySpec spec = new PBEKeySpec(password, salt, 65536, 256);
    SecretKey secret = new SecretKeySpec(factory.generateSecret(spec).getEncoded(), "AES");
    return secret;
}

salt

We use salt to protect rainbow attacks, and it is also a random byte, we can use the SecureRandom to generate it.

import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.KeySpec;
import java.util.ArrayList;
import java.util.List;

public class CryptoUtils {

    public static byte[] getRandomNonce(int numBytes) {
        byte[] nonce = new byte[numBytes];
        new SecureRandom().nextBytes(nonce);
        return nonce;
    }

    // AES secret key
    public static SecretKey getAESKey(int keysize) throws NoSuchAlgorithmException {
        KeyGenerator keyGen = KeyGenerator.getInstance("AES");
        keyGen.init(keysize, SecureRandom.getInstanceStrong());
        return keyGen.generateKey();
    }

    // Password derived AES 256 bits secret key
    public static SecretKey getAESKeyFromPassword(char[] password, byte[] salt)
        throws NoSuchAlgorithmException, InvalidKeySpecException {
        SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
        // iterationCount = 65536
        // keyLength = 256
        KeySpec spec = new PBEKeySpec(password, salt, 65536, 256);
        SecretKey secret = new SecretKeySpec(factory.generateSecret(spec).getEncoded(), "AES");
        return secret;
    }
    
    // hex representation
    public static String hex(byte[] bytes) {
        StringBuilder result = new StringBuilder();
        for (byte b : bytes) {
            result.append(String.format("%02x", b));
        }
        return result.toString();
    }

    // print hex with block size split
    public static String hexWithBlockSize(byte[] bytes, int blockSize) {
        String hex = hex(bytes);
        // one hex = 2 chars
        blockSize = blockSize * 2;
        // better idea how to print this?
        List<String> result = new ArrayList<>();
        int index = 0;
        while (index < hex.length()) {
            result.add(hex.substring(index, Math.min(index + blockSize, hex.length())));
            index += blockSize;
        }
        return result.toString();
    }
}

CBC Mode

Input

  • algorithm: AES/CBC/PKCS5Padding
  • iv bytes
  • aes key bits
  • secretKey: generate by password
  • Initialization Vector (IV): random bytes

GCM Mode


Asymmetric Encryption Algorithms

RSA

RSA (Rivest-Shamir-Adleman)

public static void getKeyPair() throws NoSuchAlgorithmException {
    KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
    kpg.initialize(2048);
    KeyPair keyPair = kpg.generateKeyPair();
    Key pub = keyPair.getPublic();
    Key pvt = keyPair.getPrivate();
    System.out.println("public key");
    System.out.println(Base64.getEncoder().encodeToString(pub.getEncoded()));
    System.out.println("private key");
    System.out.println(Base64.getEncoder().encodeToString(pvt.getEncoded()));
}

public static String decrypt(String privateKeyBase64String, String cipherText) throws NoSuchAlgorithmException, InvalidKeySpecException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException {
    KeyFactory keyFactory = KeyFactory.getInstance("RSA");
    EncodedKeySpec pkcs8KeySpec = new PKCS8EncodedKeySpec(Base64.getDecoder().decode(privateKeyBase64String.getBytes()));
    PrivateKey privateKey = keyFactory.generatePrivate(pkcs8KeySpec);
    Cipher cipher = Cipher.getInstance(keyFactory.getAlgorithm());
    cipher.init(Cipher.DECRYPT_MODE, privateKey);
    byte[] bytes = cipher.doFinal(Base64.getDecoder().decode(cipherText));
    return new String(bytes);
}

public static String encrypt(String publicKeyBase64String, String plainText) throws NoSuchAlgorithmException, InvalidKeySpecException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException {
    KeyFactory keyFactory = KeyFactory.getInstance("RSA");
    EncodedKeySpec pkcs8KeySpec = new X509EncodedKeySpec(Base64.getDecoder().decode(publicKeyBase64String.getBytes()));
    PublicKey publicKey = keyFactory.generatePublic(pkcs8KeySpec);
    Cipher cipher = Cipher.getInstance(keyFactory.getAlgorithm());
    cipher.init(Cipher.ENCRYPT_MODE, publicKey);
    byte[] bytes = cipher.doFinal(plainText.getBytes());
    return new String(Base64.getEncoder().encode(bytes));
}

ECC

ECC (Elliptic Curve Cryptography)


References

[1] Java AES encryption and decryption

Posted on Edited on In ,

Query

QueryWrapper

QueryWrapper

// usage 1
QueryWrapper<SysUser> queryWrapper = new QueryWrapper<SysUser>()
    .eq("user_id", userId)
    .like(name != null, "name", name);

// usage 2
QueryWrapper<SysUser> queryWrapper = Wrappers.query(sysUser)
    .eq("user_id", userId);

LambdaQueryWrapper

// usage 1
LambdaQueryWrapper<SysUser> lambdaQueryWrapper = new LambdaQueryWrapper<SysUser>()
    .eq(SysUser::getUserId, userId)
    .like(name != null, SysUser::getName, name);

// usage 2
LambdaQueryWrapper<SysUser> lambdaQueryWrapper = Wrappers.lambdaQuery(sysUser)
    .eq(SysUser::getUserId, userId);

// usage 3
LambdaQueryWrapper<SysUser> lambdaQueryWrapper = Wrappers.lambdaQuery(SysUser.class)
    .eq(SysUser::getUserId, userId);

Convert QueryWrapper to LambdaQueryWrapper

LambdaQueryWrapper<SysUser> lambdaQueryWrapper = new QueryWrapper<SysUser>()
    .eq("user_id", userId)
    .lambda()
    .eq(SysUser::getType, type);

Basic Query

An example

List<SysUser> sysUserList = userService.list(
    new QueryWrapper<SysUser>()
    .select("name")
    .eq("type", type)
    .orderByDesc("id")
    .last("limit 10"));

from

new QueryWrapper<Entity>()

select

wrapper.select("column 1", "column 2")

where

wrapper.eq("type", 1)
    .like("title", "test")
    .in("user_id", userIds)
    .ge("age", 18)
    .apply("find_in_set({0}, type)", type);
    .apply("date(gmt_create)=date({0})", new Date())
    .apply("date_format(gmt_modified,'%Y-%m-%d') = date({0})", new Date())

and…or

// where type = 1 or type = 2
wrapper.and(wrapper -> wrapper.eq("type", 1)
                .or().eq("type", 2))
// find_in_set(?, category_id) or find_in_set(?, category_id)
queryWrapper.and(wrapper -> {
	for (int i = 0; i < categoryIds.size(); i++) {
		if (i > 0) {
			wrapper.or();
		}
		wrapper.apply("find_in_set({0}, category_id)", categoryIds.get(i));
	}
	return wrapper;
});
// where status = 0 and (type = 1 or (type > 10 and type < 20))
wrapper.eq("status", 0)
    .and(wrapper -> wrapper.eq("type", 1)
         .or(wrapper2 -> 
             wrapper2.gt("type", 10)
             .lt("type", 20));

order by

wrapper.orderByDesc("id")

limit

wrapper.last("limit 10")

Query Conditions

find_in_set

wrapper.apply("find_in_set({0}, type)", type);

Date functions

date equal

wrapper.apply("date(gmt_create)=date({0})", new Date())
    .apply("date_format(gmt_modified,'%Y-%m-%d') = date({0})", new Date())

Recent 7 days

wrapper.apply("create_time > DATE_SUB(NOW(), INTERVAL 7 DAY)"));

Page Query

  1. Request query string parameters

http://xxx.xxx/?current=1&size=10&orders[0].column=pubtime&orders[0].asc=true

  • current
  • size
  • orders[].column
  • orders[].asc
  1. Page Entity

com.baomidou.mybatisplus.extension.plugins.pagination.Page

Fields for Request parameters

  • long size
  • long current
  • List<OrderItem> orders
    • String column
    • boolean asc

Fields for response

  • List<T> records
  • long total
  1. Page Query Methods

MyBatis Plus page query method

IPage<T> selectPage(IPage<T> page, Wrapper<T> queryWrapper)

Custom page query in Mapper XML

IPage<MyResult> myPageQuery(@Param("page") Page page, @Param("param") MyParam param);
  • Pass com.baomidou.mybatisplus.extension.plugins.pagination.Page object as a parameter.
  • Using the IPage class as the return type.
  • Don’t need to add limit start, size in mapper XML. The SQL is query all rows. But MyBatis Plus automatically add limit at the end of SQL. If you want to query all, update to List<MyResult> queryAll(@Param("param") MyParam param);

One Column Query

List<Object> names = baseMapper.selectObjs(
    new LambdaQueryWrapper<User>()
    .select(User::getName)
    .eq(User::getType, 1));
if (names == null) {
    return Collections.emptySet();
} else {
    return names.stream()
        .map(Object::toString)
        .collect(Collectors.toSet());
}

Aggregation Query

Aggregation Query methods

  • selectMaps()
List<Map<String, Object>> mapList = userMapper.selectMaps(
    new QueryWrapper<SysUser>()
    .select("type as name, count(*) as typeCount")
    .groupBy("type"));
  • selectObjs()
List<Object> mapList = userMapper.selectObjs(
    new QueryWrapper<SysUser>()
    .select("sum(num) as typeTotal")
    .groupBy("type"));
  • selectCount()
userMapper.selectCount(queryWrapper);

select

queryWrapper.select("count(*) as typeCount");
queryWrapper.select("submit_content_type as name, count(*) as value")

group by

queryWrapper.groupBy("type"));

having

queryWrapper.having("COUNT(*) > 10"))

Others

Non-query fields

Use @TableField(exist = false)

public class User {
	@TableField(exist = false)
	private Department dept;
}

Use @TableName(excludeProperty={})

@TableName(value = "my_table_name", excludeProperty = {"dept"})
public class User {
    private Department dept;
}

Columns using like "%xxx%"

Using @TableField(condition = SqlCondition.LIKE)

@TableField(condition = SqlCondition.LIKE)
private String name;

Only perform field query if the value is not null or empty

Using @TableField(whereStrategy = FieldStrategy.NOT_EMPTY)

  • IGNORED: 不判断
  • NOT_NULL: 非NULL判断
  • NOT_EMPTY: 非空判断
@TableField(whereStrategy = FieldStrategy.NOT_EMPTY)
private String name;

Using MySQL keyword as a column name

@TableField(value = "`type`")
private Integer type;

Using another entity name

@Alias("UserV2")
public class User {
	//...
}

DML

Update

Update

userService.update(new UpdateWrapper<User>()
    .set("name", "updateName")
    .eq("id", 1));

update it to null when it value is null

@TableField(fill = FieldFill.UPDATE)
private String name;

Delete

Logical delete

// Note that the field for logical deletion cannot be modified by update() method
@TableLogic(value="0",delval="1")
private Boolean delFlag;

MyBatis Plus Configurations

mybatis-plus:
  mapper-locations: classpath:mybatis/mapper/*.xml
  type-aliases-package: com.example.entity,com.example.modules.**.entity
  global-config:
    db-config:
      where-strategy: not_empty
      logic-delete-field: deleteFlag
      logic-delete-value: 1
      logic-not-delete-value: 0

More configurations reference MyBatis Plus使用配置

Posted on Edited on In

Background

Running the Spring Boot project occurs an error.

Error Info

Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'spring.datasource.druid.initialSize' in value "${spring.datasource.druid.initialSize}"
	at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:178)
	... 97 common frames omitted

Solutions

1. Check that the property you need exists in a configuration file. Find out in which configuration file your injected property is.

2. Check whether the property spring.profiles.active value is correct in application.yml. Make sure the active profile is in your project.

3. Check whether your injected property exists in application.yml (or application.properties) and the active profile application-xxx.yml (or application-xxx.properties).

Reasons

The value of the property spring.profiles.active in my application.yml is spring.profiles.active=pro, but I don’t have the application-pro.yml file. The property’s value spring.profiles.active=pro should update to prod.

Posted on Edited on In

Background

When I do an insert operation to save the item data to MySQL, occurs an error.

Error Info

Error updating database.  Cause: java.sql.SQLException: Data truncated for column 'xxx' at row 1

Solutions

Check whether the column type is right.

Check whether the length of the field value over the length of the column.

Reasons

The length of the field value over the length of the column.

My passed field value is a set type value, but the column type is enum. The column type should update to set.

Posted on Edited on In

Background

When I use the FreeMarker to generate text files, occurs an error: Failed to “?eval” string.

<#assign dataFromConfigJson = columnMapping.formComponentDataFromConfig?eval>

Error Info

FreeMarker template error:
Failed to "?eval" string with this error:

---begin-message---
Syntax error in ?eval-ed string in line 1, column 175:
Encountered "}", but was expecting one of:
    <STRING_LITERAL>
    <RAW_STRING>
    "false"
    "true"
    <INTEGER>
    <DECIMAL>
    "."
    ".."
    <DOT_DOT_LESS>
    "..*"
    "?"
    "??"
    "+"
    "-"
    "!"
    ","
    "["
    "]"
    "("
    "{"
    <ID>
    <TERMINATING_EXCLAM>
    "+"
    "-"
---end-message---

The failing expression:
==> columnMapping.formComponentDataFromConfig?eval  [in template "templates/demo\\html\\test.ftl" at line 208, column 41]

----
FTL stack trace ("~" means nesting-related):
	- Failed at: #assign dataFromConfigJson = columnMa...  [in template "templates/demo\\html\\test.ftl" in macro "generateFormItems" at line 208, column 11]
	- Reached through: @generateFormItems type="search"  [in template "templates/demo\\html\\test.ftl" at line 582, column 7]
----

Solutions

Corrects the JSON string of the field of data that passed to the FreeMarker Template object.

Reasons

The JSON string is invalid.

Posted on Edited on In

问题1：扩展显示，外接显示器模糊

解决方案：笔记本和外接显示器的缩放比例保持一致。

Display settings -> Scale and layout

问题2：扩展显示，合上笔记本时，外接显示器变模糊

解决方案：将外接屏幕设为主显示器，并设置“仅显示外接显示器”。

Display settings -> Rearrange your display -> Click 2 for setting extended monitor

Display settings -> Multiple displays -> select “Show only on 2”, and checked “Make this main display”

问题3：笔记本使用HDMI接口连接了显示器后没声音

Solution: 声音设置中选择笔记本的扬声器作为声音输出设备。

Reason: HDMI接口是包含音频+视频两种信号的接口。当电脑的HDMI接口被使用时，系统就会默认从HDMI设备输出声音信号。

Posted on Edited on In ,

在我看来，编程它既是脑力活，也是体力活。脑力活在于如何解决未知的问题，体力活则在于一行一行地敲代码完成已知的问题。

编程中最累人的便是面对未知问题。如：未知的技术栈，未知的系统设计，未知的设计模式和类的继承结构，未知的算法实现，未知的异常等等。遇到未知问题时，我们会注意力保持高度集中，不断的失败，不断的尝试，我们绞尽脑汁想要解决这个问题，同时我们感到沮丧和急躁。然而，长时间的保持紧绷的神经和僵硬的坐姿会严重影响我们的健康状态。作为一名程序员，为了保持良好的身心健康，我们要学会如何与未知相处。

  1. 从源头出发，减少未知

我们不可能去知道所有的技术和知识。但是我们可以把自己经历过的问题，总结一下，下次遇到相似的问题，可以轻松地应对。我们也可以尽量地学习和了解一些你所处领域相关的你没有掌握的技术。

  1. 承认自己遇到了难题，并给与自己信心

每个人掌握着有限的知识，每个人都会遇到难题。遇到难题不能完全代表一个人的能力水平，难题给了我们一次成长的机会。我们尽力去解决问题，最终无论我们能不能解决这个难题，我们都会有所收获。

  1. 给与解决问题充足的时间，让问题慢慢变得明朗

解决未知问题的过程就是把未知慢慢变成已知的过程。我们在解决未知问题的过程中，经过了一段时间的思考之后，我们会发现自己有了点思路或者有了点感觉，慢慢地问题就逐渐变得越来越明朗了。

保持心平气和地去解决未知问题，这是急不来的，我们需要给大脑一些时间去理解和记忆。

Posted on Edited on In ,

Content

Upload

Upload files with Spring framework

@PostMapping(value = "/uploadFile", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public String uploadFile(@RequestParam("file") MultipartFile file, String moduleUri) throws IOException {
    InputStream inputStream = file.getInputStream();
    String fileName = file.getOriginalFilename();
    // to save file
    ...
}

Upload file with Java Servlet

public class FileUploadByCommonsFileUploadServlet extends HttpServlet {
	public void doPost(HttpServletRequest request, HttpServletResponse response) throws Exception {
        DiskFileItemFactory diskFileItemFactory = new DiskFileItemFactory();
        diskFileItemFactory.setSizeThreshold(MAX_MEMORY_SIZE);
        diskFileItemFactory.setRepository(new File(System.getProperty("java.io.tmpdir")));
        ServletFileUpload servletFileUpload = new ServletFileUpload(diskFileItemFactory);
        servletFileUpload.setFileSizeMax(MAX_FILE_SIZE);
        servletFileUpload.setSizeMax(MAX_FILE_SIZE);
        List<FileItem> formItems = servletFileUpload.parseRequest(request);
        for (FileItem item : formItems) {
            if (!item.isFormField()) {
                String fileName = new File(item.getName()).getName();
                String filePath = new StringBuilder()
                    .append(UPLOAD_DIR)
                    .append(File.seperator)
                    .append(fileName)
                    .toString()
                item.write(new File(filePath));
            }
        }
    }
    ...
}

Download

Download file with spring framework

Write input streams to the OutPutStream of the HttpServletResponse

@GetMapping(value = "/download" )
public void download(HttpServletResponse response, String fileUri) {
    response.setCharacterEncoding("utf-8");
    response.setContentType("application/octet-stream");
    response.setHeader("Content-Disposition", "attachment;fileName=\"" + URLEncoder.encode(filename, "UTF-8").replace("+", "%20") + "\"");
    ServletOutputStream outputStream = response.getOutputStream();
    int bufSize = 1024;
    byte[] buffer = new byte[bufSize];
    int len;
    // get filePath by fileUri
    ...
    // get input stream by filePath
    ...
    while ((len = inputStream.read(buffer)) != -1) {
        outputStream.write(buffer, 0, len);
    }
    inputStream.close();
}

Write file Resource to ReponseBody

@GetMapping(value = "/download" )
public ResponseEntity<Resource> download(String fileUri) {
    // get filePath by fileUri
    //...
    String uploadDir = "D:\\upload";
    Path path = Paths.get(uploadDir).toAbsolutePath().normalize();
    path = path.resolve(fileUri).normalize();
    Resource resource = new UrlResource(path.toUri());
    String resultFileName = URLEncoder.encode(resource.getFilename().replace("%20", " "), "UTF-8").replace("+", "%20");
    return ResponseEntity.ok()
        .contentType(MediaType.parseMediaType("application/octet-stream"))
        .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + resultFileName + "\"")
        .body(resource);
}

Setting Response Headers

  • content-type: application/octet-stream
  • content-disposition: attachment; filename=”test.txt”

To encode filename when download file

original file name: “中 文.txt”

filename = resource.getFilename() // 中%20文.txt

browser download filename: -%20‡.txt

1

filename = URLEncoder.encode(resource.getFilename().replace("%20", " "), "UTF-8") // %E4%B8%AD+%E6%96%87.txt

browser download filename: 中+文.txt

2

filename = URLEncoder.encode(resource.getFilename().replace("%20", " "), "UTF-8").replace("+", "%20") // %E4%B8%AD%20%E6%96%87.txt

browser download filename: 中 文.txt

When browser resolve file name, they don’t decode “+” to “ “, so we need replace “+” to “%20”.

The Example of Upload and download files in spring framework

@RestController
@RequestMapping("files")
public class MyFileController {
    private static final Logger logger = LogManager.getLogger();

    public static final String UPLOAD_DIR = "D:\\My Desktop\\upload";

    public static final String DOWNLOAD_PREFIX = "/files/download";

    @PostMapping(value = "/uploadFile", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
    public String uploadFile(@RequestParam("file") MultipartFile file, String moduleUri) throws IOException {
        String uploadFileUri = getUploadFileUri(file, moduleUri);
        String uploadFilePath = new StringBuilder()
                .append(UPLOAD_DIR)
                .append(File.separator)
                .append(uploadFileUri)
                .toString();
        ensureDirectoryExist(uploadFilePath);
        try (
            BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(
                new FileOutputStream(uploadFilePath));
        ) {
            byte[] buf = new byte[1024];
            int len;
            InputStream inputStream = file.getInputStream();
            while ((len = inputStream.read(buf)) != -1) {
                bufferedOutputStream.write(buf, 0, len);
            }
        }
        return new StringBuilder()
                .append(DOWNLOAD_PREFIX)
                .append("/")
                .append(uploadFileUri)
                .toString();
    }

    private String getUploadFileUri(MultipartFile file, String moduleUri) {
        String uuid = UUID.randomUUID().toString().replaceAll("-", "");
        Calendar calendar = Calendar.getInstance();
        StringBuilder dateUri = new StringBuilder()
                .append(calendar.get(Calendar.YEAR))
                .append("/")
                .append(calendar.get(Calendar.MONTH) + 1)
                .append("/")
                .append(calendar.get(Calendar.DAY_OF_MONTH));
        StringBuilder resultFileUri = new StringBuilder()
                .append(moduleUri)
                .append("/")
                .append(dateUri)
                .append("/")
                .append(uuid)
                .append("/")
                .append(file.getOriginalFilename());
        return resultFileUri.toString();
    }

    private void ensureDirectoryExist(String filePath) {
        filePath = filePath.replace("/", File.separator).replace("\\", File.separator);
        String fileDir = filePath.substring(0, filePath.lastIndexOf(File.separator));
        logger.debug("fileDir: {}", fileDir);
        File dir = new File(fileDir);
        if (!dir.exists()) {
            dir.mkdirs();
        }
    }

    @GetMapping(value = "download/**")
    public void download(HttpServletRequest request, HttpServletResponse response) throws IOException {
        String servletPath = request.getServletPath();
        String fileUri = servletPath.substring(DOWNLOAD_PREFIX.length());
        String filePath = new StringBuilder()
                .append(UPLOAD_DIR)
                .append(File.separator)
                .append(fileUri)
                .toString();
        try (
                BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream(filePath))
        ) {
            ServletOutputStream outputStream = response.getOutputStream();
            byte[] buf = new byte[1024];
            int len;
            while ((len = bufferedInputStream.read(buf)) != -1) {
                outputStream.write(buf, 0, len);
            }
        }
    }
}

Read

Read data from disk

java.io.FileInputStream

InputStream is = new FileInputStream("D:\\My Desktop\\test.txt");

org.springframework.core.io.UrlResource

Spring framework Resource Interface

Java’s standard java.net.URL class and standard handlers for various URL prefixes unfortunately are not quite adequate enough for all access to low-level resources. For example, there is no standardized URL implementation that may be used to access a resource that needs to be obtained from the classpath, or relative to a ServletContext. While it is possible to register new handlers for specialized URL prefixes (similar to existing handlers for prefixes such as http:), this is generally quite complicated, and the URL interface still lacks some desirable functionality, such as a method to check for the existence of the resource being pointed to.

String filepath = "D:\\My Desktop\\test.txt";
Path path = Paths.get(filepath).normalize();
//path.toUri().toString(): file:///D:/My%20Desktop/test.txt
Resource resource = new UrlResource(path.toUri()); 
InputStream is = resource.getInputStream();

java.net.URL

String filepath = "D:\\My Desktop\\test.txt";
Path path = Paths.get(filepath).normalize();
//path.toUri().toString(): file:///D:/My%20Desktop/test.txt
InputStream is = new URL(path.toUri().toString()).openStream();

Read data from Java classpath

org.springframework.core.io.ClassPathResource

// src/main/resources/application.yml
Resource resource = new ClassPathResource("application.yml");
InputStream is = resource.getInputStream();

org.springframework.core.io.ResourceLoader

@Autowired
ResourceLoader resourceLoader;

resourceLoader.getResource("classpath:application.yml");
InputStream is = resource.getInputStream();

Read data from memory

byte[] data = new String("data...").getBytes(StandardCharsets.UTF_8);
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(data);

Read data From HTTP URL

java.net.URL

InputStream is = new URL("http://demo.com/test.txt").openStream();

org.springframework.core.io.UrlResource

Resource resource = new UrlResource("http://demo.com/test.txt");
InputStream is = resource.getInputStream();

java.net.HttpURLConnection

URLConnection urlConnection = new URL("http://demo.com/test.txt").openConnection();
urlConnection.connect();
InputStream is = urlConnection.getInputStream();

Read data from OSS

Reference my another post: Common OSS Java SDK Usage

Write

Write data to disk

byte[] data = new String("data").getBytes(StandardCharsets.UTF_8);
String storeFilePath = "D:/My Workspace/test/test.txt";
try (
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(data);
    FileOutputStream fileOutputStream = new FileOutputStream(storeFilePath)
) {
    byte[] buf = new byte[1024];
    int len = 0;
    while ((len = byteArrayInputStream.read(buf)) != -1) {
        fileOutputStream.write(buf, 0, len);
    }
} catch (IOException e) {
    e.printStackTrace();
}

Write data to memory

byte[] data = new String("data").getBytes(StandardCharsets.UTF_8);
byte[] storeDataArray;
try (
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(data);
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
) {
    byte[] buf = new byte[1024];
    int len = 0;
    while ((len = byteArrayInputStream.read(buf)) != -1) {
        byteArrayOutputStream.write(buf, 0, len);
    }
    storeDataArray = byteArrayOutputStream.toByteArray();
    System.out.println(new String(storeDataArray, StandardCharsets.UTF_8));
} catch (IOException e) {
    e.printStackTrace();
}

Write data to OSS

Reference my another post: Common OSS Java SDK Usage

stream

InputStream is = new FileInputStream("D:\\My Desktop\\test.txt");
new BufferedReader(new InputStreamReader(is)).lines().forEach(System.out::println);

java.nio.file.Files

Path path = Paths.get("D:\\My Desktop\\test.txt");
System.out.println(new String(Files.readAllBytes(path)));

Package files to zip

basic zip

try (
    FileOutputStream fileOutputStream = new FileOutputStream("D:/My Workspace/test/test.zip");
    ZipOutputStream zipOutputStream = new ZipOutputStream(fileOutputStream)
) {
    int fileNumber = 3;
    for (int i = 1; i <= fileNumber; i++) {
        String childFilename = "file" + i + ".txt";
        zipOutputStream.putNextEntry(new ZipEntry(childFilename));
        zipOutputStream.write(("data" + i).getBytes(StandardCharsets.UTF_8));
        zipOutputStream.closeEntry();
    }
    zipOutputStream.finish();
} catch (IOException e) {
    e.printStackTrace();
}

nested zip

try (
    FileOutputStream fileOutputStream = new FileOutputStream("D:/My Workspace/test/test.zip");
    ZipOutputStream zipOutputStream = new ZipOutputStream(fileOutputStream)
) {
    int fileNumber = 3;
    for (int i = 1; i <= fileNumber; i++) {
        String childFilename = "file" + i + ".zip";
        zipOutputStream.putNextEntry(new ZipEntry(childFilename));
        try (
            ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
            ZipOutputStream nestedZipOutputStream = new ZipOutputStream(byteArrayOutputStream);
        ) {
            int nestedNumber = 3;
            for (int j = 1; j <= nestedNumber; j++) {
                String nestedFileName = new StringBuilder()
                    .append("file").append(i)
                    .append("nestFile").append(j)
                    .append(".txt").toString();
                nestedZipOutputStream.putNextEntry(new ZipEntry(nestedFileName));
                nestedZipOutputStream.write(("nest file" + j).getBytes(StandardCharsets.UTF_8));
                nestedZipOutputStream.closeEntry();
            }
            nestedZipOutputStream.finish();
            zipOutputStream.write(byteArrayOutputStream.toByteArray());
        }
        zipOutputStream.closeEntry();
    }
    zipOutputStream.finish();
} catch (IOException e) {
    e.printStackTrace();
}

Speed up read or write

Using buffer to decrease calls to the underlying runtime system

  1. Direct Buffering
String inputFilePath = "C:\\Users\\Taogen\\Desktop\\input.txt";
String outputFilePath = "C:\\Users\\Taogen\\Desktop\\output.txt";
try (
    FileInputStream fileInputStream = new FileInputStream(inputFilePath);
    FileOutputStream fileOutputStream = new FileOutputStream(outputFilePath)
) {
    byte[] buf = new byte[2048];
    int len = 0;
    while ((len = fileInputStream.read(buf)) != -1) {
        fileOutputStream.write(buf, 0, len);
    }
} catch (IOException e) {
    e.printStackTrace();
}
  1. Using buffered Input/output Stream
String inputFilePath = "C:\\Users\\Taogen\\Desktop\\input.txt";
String outputFilePath = "C:\\Users\\Taogen\\Desktop\\output.txt";
try (
    FileInputStream fileInputStream = new FileInputStream(inputFilePath);
    BufferedInputStream bufferedInputStream = new BufferedInputStream(fileInputStream);
    FileOutputStream fileOutputStream = new FileOutputStream(outputFilePath);
    BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
) {

    int b;
    while ((b=bufferedInputStream.read()) != -1) {
        bufferedOutputStream.write(b);
    }
} catch (IOException e) {
    e.printStackTrace();
}

BufferedInputStream‘s default buffer size is int DEFAULT_BUFFER_SIZE = 8192;

  1. BufferedInputStream + Buffer array
String inputFilePath = "C:\\Users\\Taogen\\Desktop\\recovery_data.sql";
String outputFilePath = "C:\\Users\\Taogen\\Desktop\\recovery_data2.sql";
try (
    FileInputStream fileInputStream = new FileInputStream(inputFilePath);
    BufferedInputStream bufferedInputStream = new BufferedInputStream(fileInputStream);
    FileOutputStream fileOutputStream = new FileOutputStream(outputFilePath);
    BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
) {
    byte[] buf = new byte[1024];
    int len = 0;
    while ((len = bufferedInputStream.read(buf)) != -1) {
        bufferedOutputStream.write(buf, 0, len);
    }
} catch (IOException e) {
    e.printStackTrace();
}

Time cost: BufferedInputStream + Direct Buffer array < Direct Buffer array < Using a BufferedInputStream < Read Method.

Using BufferedInputStream rather than direct buffer is probably “right” for most applications. Because your used buffer size of direct buffer may be worse than BufferedInputStream default buffer size in speed.

Will making the buffer bigger make I/O go faster? Java buffers typically are by default 1024 or 2048 bytes long. A buffer larger than this may help speed I/O, but often by only a few percent, say 5 to 10%.

Problems

Max upload file size

Character encoding of file name

URL encode of file name

Appendixes

Temporary Files and Directories

java.io.tmpdir

System.getProperty("java.io.tmpdir")

Windows 10: C:\Users\{user}\AppData\Local\Temp\

Debian: /tmp

Create temporary file

// If you don't specify the file suffix, the default file suffix is ".tmp".
File file = File.createTempFile("temp", null);
System.out.println(file.getAbsolutePath());
file.deleteOnExit();

Common Content-Type (MIME types) Table

Java get file’s mimeType

// 1
String mimeType = Files.probeContentType(file.toPath());
// 2
String mimeType = URLConnection.guessContentTypeFromName(fileName);
// 3
FileNameMap fileNameMap = URLConnection.getFileNameMap();
String mimeType = fileNameMap.getContentTypeFor(file.getName());
// 4
MimetypesFileTypeMap fileTypeMap = new MimetypesFileTypeMap();
String mimeType = fileTypeMap.getContentType(file.getName());
Content-Type(Mime-Type) Kind of document Extension
application/octet-stream Any kind of binary data .bin
text/plain Text, (generally ASCII or ISO 8859-n) .txt
<Images>
image/bmp Windows OS/2 Bitmap Graphics .bmp
image/jpeg JPEG images .jpeg .jpg
image/png Portable Network Graphics .png
image/gif Graphics Interchange Format (GIF) .gif
image/vnd.microsoft.icon Icon format .ico
image/svg+xml Scalable Vector Graphics (SVG) .svg
<media>
audio/mpeg MP3 audio .mp3
video/mp4 MP4 video .mp4
<code>
text/css Cascading Style Sheets (CSS) .css
text/csv Comma-separated values (CSV) .csv
application/json JSON format .json
text/javascript JavaScript, JavaScript module .js .mjs
text/html HyperText Markup Language (HTML) .htm .html
<doc>
application/msword Microsoft Word .doc
application/vnd.openxmlformats-officedocument.wordprocessingml.document Microsoft Word (OpenXML) .docx
application/pdf Adobe Portable Document Format (PDF) .pdf
<archive>
application/gzip GZip Compressed Archive .gz
application/vnd.rar RAR archive .rar
application/x-tar Tape Archive (TAR) .tar
application/zip ZIP archive .zip
application/x-7z-compressed 7-zip archive .7z

Two primary MIME types are important for the role of default types:

  • text/plain is the default value for textual files. A textual file should be human-readable and must not contain binary data.
  • application/octet-stream is the default value for all other cases. An unknown file type should use this type. Browsers pay a particular care when manipulating these files, attempting to safeguard the user to prevent dangerous behaviors.

References

[1] Tuning Java I/O Performance

[2] Common MIME types - MDN Web Docs

0%