Taogen's Blog

Stay hungry stay foolish.

Background

Calling backend API, the status code of the response is 500, but the backend is not throw exceptions. The HTTP response message is “Proxy error: Could not proxy request”.

Error Info

Proxy error: Could not proxy request /captchaImage from localhost:8070 to http://10.0.0.74:8090 (ECONNREFUSED).

Solutions

  1. Make sure the config devServer.proxy.target is correct.

vue.config.js

devServer: {
...
proxy: {
[process.env.VUE_APP_BASE_API]: {
target: `http://localhost:8090`,
...
}
},
}
  1. Make sure you can visit the backend server anonymous request APIs in your PC, e.g. http://localhost:8090/captchaImage.

Reasons

The config devServer.proxy.target has no http prefix. e.g. target: localhost:8090.

Or the port number of backend server is not correct.

JUnit 5 Improvements

  • JUnit 5 leverages features from Java 8 or later, such as lambda functions, making tests more powerful and easier to maintain.
  • JUnit 5 has added some very useful new features for describing, organizing, and executing tests. For instance, tests get better display names and can be organized hierarchically.
  • JUnit 5 is organized into multiple libraries, so only the features you need are imported into your project. With build systems such as Maven and Gradle, including the right libraries is easy.
  • JUnit 5 can use more than one extension at a time, which JUnit 4 could not (only one runner could be used at a time). This means you can easily combine the Spring extension with other extensions (such as your own custom extension).

Differences

Imports

  • org.junit.Test => org.junit.jupiter.api.Test
  • Assert => Assertions
    • org.junit.Assert => org.junit.jupiter.api.Assertions

Annotations

  • @Before => @BeforeEach

    • org.junit.Before => org.junit.jupiter.api.BeforeEach
  • @After => @AfterEach

  • @BeforeClass => @BeforeAll

  • @AfterClass => @AfterAll

  • @Ignore => @Disabled

    • org.junit.Ignore => org.junit.jupiter.api.Disabled
  • @Category => @Tag

  • @RunWith, @Rule, @ClassRule => @ExtendWith and @RegisterExtension

    • org.junit.runner.RunWith => org.junit.jupiter.api.extension.ExtendWith
    • @RunWith(SpringRunner.class) => @ExtendWith(SpringExtension.class)
    • org.springframework.test.context.junit4.SpringRunner => org.springframework.test.context.junit.jupiter.SpringExtension

Assertion Methods

JUnit 5 assertions are now in org.junit.jupiter.api.Assertions. Most of the common assertions, such as assertEquals() and assertNotNull(), look the same as before, but there are a few differences:

  • The error message is now the last argument, for example: assertEquals("my message", 1, 2) is now assertEquals(1, 2, "my message").
  • Most assertions now accept a lambda that constructs the error message, which is called only when the assertion fails.
  • assertTimeout() and assertTimeoutPreemptively() have replaced the @Timeout annotation (there is an @Timeout annotation in JUnit 5, but it works differently than in JUnit 4).
  • There are several new assertions, described below.

Note that you can continue to use assertions from JUnit 4 in a JUnit 5 test if you prefer.

Assumptions

Executes the supplied Executable, but only if the supplied assumption is valid.

JUnit 4

assumeThat("alwaysPasses", 1, is(1)); // passes
foo(); // will execute
assumeThat("alwaysFails", 0, is(1)); // assumption failure! test halts
int x = 1 / 0; // will never execute

JUnit 5

@Test
void testNothingInParticular() throws Exception {
Assumptions.assumingThat("DEV".equals(System.getenv("ENV")), () -> {
assertEquals(...);
});
}

Extending JUnit

JUnit 4

@RunWith(SpringRunner.class) // SpringRunner is an alias for the SpringJUnit4ClassRunner.
//@RunWith(SpringJUnit4ClassRunner.class)
public class MyControllerTest {
// ...
}

JUnit 5

@ExtendWith(SpringExtension.class)
class MyControllerTest {
// ...
}

Expect Exceptions

JUnit 4

@Test(expected = Exception.class)
public void testThrowsException() throws Exception {
// ...
}

JUnit 5

@Test
void testThrowsException() throws Exception {
Assertions.assertThrows(Exception.class, () -> {
//...
});
}

Timeout

JUnit 4

@Test(timeout = 10)
public void testFailWithTimeout() throws InterruptedException {
Thread.sleep(100);
}

JUnit 5

@Test
void testFailWithTimeout() throws InterruptedException {
Assertions.assertTimeout(Duration.ofMillis(10), () -> Thread.sleep(100));
}

Converting a Test to JUnit 5

To convert an existing JUnit 4 test to JUnit 5, use the following steps, which should work for most tests:

  1. Update imports to remove JUnit 4 and add JUnit 5. For instance, update the package name for the @Test annotation, and update both the package and class name for assertions (from Asserts to Assertions). Don’t worry yet if there are compilation errors, because completing the following steps should resolve them.
  2. Globally replace old annotations and class names with new ones. For example, replace all @Before with @BeforeEach and all Asserts with Assertions.
  3. Update assertions; any assertions that provide a message need to have the message argument moved to the end (pay special attention when all three arguments are strings!). Also, update timeouts and expected exceptions (see above for examples).
  4. Update assumptions if you are using them.
  5. Replace any instances of @RunWith, @Rule, or @ClassRule with the appropriate @ExtendWith annotations. You may need to find updated documentation online for the extensions you’re using for examples.

New Features

Display Names

you can add the @DisplayName annotation to classes and methods. The name is used when generating reports, which makes it easier to describe the purpose of tests and track down failures, for example:

@DisplayName("Test MyClass")
class MyClassTest {
@Test
@DisplayName("Verify MyClass.myMethod returns true")
void testMyMethod() throws Exception {
// ...
}
}

Assertion Methods

JUnit 5 introduced some new assertions, such as the following:

assertIterableEquals() performs a deep verification of two iterables using equals().

void assertIterableEquals(Iterable<?> expected, Iterable> actual)

assertLinesMatch() verifies that two lists of strings match; it accepts regular expressions in the expected argument.

void assertLinesMatch(List<String> expectedLines, List<String> actualLines)

assertAll() groups multiple assertions together. Asserts that all supplied executables do not throw exceptions. The added benefit is that all assertions are performed, even if individual assertions fail.

void assertAll(Executable... executables)

assertThrows() and assertDoesNotThrow() have replaced the expected property in the @Test annotation.

<T extends Throwable> T assertThrows(Class<T> expectedType, Executable executable)
void assertDoesNotThrow (Executable executable)

Nested tests

Test suites in JUnit 4 were useful, but nested tests in JUnit 5 are easier to set up and maintain, and they better describe the relationships between test groups.

Parameterized tests

Test parameterization existed in JUnit 4, with built-in libraries such as JUnit4Parameterized or third-party libraries such as JUnitParams. In JUnit 5, parameterized tests are completely built in and adopt some of the best features from JUnit4Parameterized and JUnitParams, for example:

@ParameterizedTest
@ValueSource(strings = {"foo", "bar"})
@NullAndEmptySource
void myParameterizedTest(String arg) {
underTest.performAction(arg);
}

Conditional test execution

JUnit 5 provides the ExecutionCondition extension API to enable or disable a test or container (test class) conditionally. This is like using @Disabled on a test but it can define custom conditions. There are multiple built-in conditions, such as these:

  • @EnabledOnOs and @DisabledOnOs: Enables or disables a test only on specified operating systems
  • @EnabledOnJre and @DisabledOnJre: Specifies the test should be enabled or disabled for particular versions of Java
  • @EnabledIfSystemProperty: Enables a test based on the value of a JVM system property
  • @EnabledIf: Uses scripted logic to enable a test if scripted conditions are met

Test templates

Test templates are not regular tests; they define a set of steps to perform, which can then be executed elsewhere using a specific invocation context. This means that you can define a test template once, and then build a list of invocation contexts at runtime to run that test with. For details and examples, see the documentation.

Dynamic tests

Dynamic tests are like test templates; the tests to run are generated at runtime. However, while test templates are defined with a specific set of steps and run multiple times, dynamic tests use the same invocation context but can execute different logic. One use for dynamic tests would be to stream a list of abstract objects and perform a separate set of assertions for each based on their concrete types. There are good examples in the documentation.

Spring Boot Test With JUnit

Spring Boot Test With JUnit 4

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- Starting with Spring Boot 2.4, JUnit 5’s vintage engine has been removed from spring-boot-starter-test. If we still want to write tests using JUnit 4, we need to add the following Maven dependency -->
<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-core</artifactId>
</exclusion>
</exclusions>
</dependency>
@RunWith(SpringRunner.class)
@SpringBootTest
public class MyServiceTest {
@Autowired
private MyRepository myRepository;

@org.junit.Test
public void test(){
}
}

Spring Boot Test With JUnit 5

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
@ExtendWith(SpringExtension.class)
@SpringBootTest
public class MyServiceTest {
@Autowired
private MyRepository myRepository;

@org.junit.jupiter.api.Test
public void test(){
}
}

Conclusion

Although you probably won’t need to convert your old JUnit 4 tests to JUnit 5 unless you want to use new JUnit 5 features, there are compelling reasons to switch to JUnit 5.

References

Migrating from JUnit 4 to JUnit 5: Important Differences and Benefits

JUnit 5

JUnit 4

Problem Description

Given two integers dividend and divisor, divide two integers without using multiplication, division, and mod operator.

The integer division should truncate toward zero, which means losing its fractional part. For example, 8.345 would be truncated to 8, and -2.7335 would be truncated to -2.

Return the quotient after dividing dividend by divisor.

Note: Assume we are dealing with an environment that could only store integers within the 32-bit signed integer range: [−2^31, 2^31 − 1]. For this problem, if the quotient is strictly greater than 2^31 - 1, then return 2^31 - 1, and if the quotient is strictly less than -2^31, then return -2^31.

Example 1:

Input: dividend = 10, divisor = 3
Output: 3
Explanation: 10/3 = 3.33333.. which is truncated to 3.

Example 2:

Input: dividend = 7, divisor = -3
Output: -2
Explanation: 7/-3 = -2.33333.. which is truncated to -2.

Constraints:

  • -2^31 <= dividend, divisor <= 2^31 - 1
  • divisor != 0

Related Topics

  • Math
  • Bit Manipulation

Analysis

set quotient = 0
n ∈ N
when divisor * 2 ^ n <= dividend < divisor * 2 ^ (n+1)
quotient = quotient + (2 ^ n)
dividend = dividend - (divisor ^ n)
when divisor <= dividend < divisor * 2
quotient = quotient + 1
dividend = dividend - divisor
when dividend < divisor
return quotient

Solution

public int divide(int dividend, int divisor) {
if (dividend == Integer.MIN_VALUE && divisor == -1) return Integer.MAX_VALUE; //Cornor case when -2^31 is divided by -1 will give 2^31 which doesnt exist so overflow

boolean negative = dividend < 0 ^ divisor < 0; //Logical XOR will help in deciding if the results is negative only if any one of them is negative

dividend = Math.abs(dividend);
divisor = Math.abs(divisor);
int quotient = 0, subQuot = 0;

while (dividend - divisor >= 0) {
for (subQuot = 0; dividend - (divisor << subQuot << 1) >= 0; subQuot++);
quotient += 1 << subQuot; //Add to the quotient
dividend -= divisor << subQuot; //Substract from dividend to start over with the remaining
}
return negative ? -quotient : quotient;
}

TODO

Demonstrate dividend - (divisor ^ (subQuot + 1)) >= 0 is always right?

References

[1] Divide Two Integers - Java | 0ms | 100% faster | Obeys all conditions

Apache POI is a Java API for Microsoft Documents processing. It provides pure Java libraries for reading and writing files in Microsoft Office formats, such as Word, PowerPoint and Excel. The code examples in this post are based on Apache POI v5.0.0.

<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>5.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>5.0.0</version>
</dependency>

poi is for Excel 97-2003 workbook.

poi-ooxml is for Excel 2007+ workbook.

Word

Write Data into Word

A basic use example

public static void main(String[] args) throws IOException {
XWPFDocument document = new XWPFDocument();
XWPFParagraph paragraph = document.createParagraph();
XWPFRun run = paragraph.createRun();
run.setText("Hello World");
run.addBreak();
run.setText("Apache POI");
run.setBold(true);
run.setFontSize(20);
String filePath = "C:\\Users\\Taogen\\Desktop\\hello.docx";
document.write(new FileOutputStream(filePath));
}

Alignment

XWPFParagraph paragraph = doc.createParagraph();

paragraph.setAlignment(ParagraphAlignment.CENTER);

Indentation

XWPFParagraph paragraph = document.createParagraph();

Integer indentationFirstLine; // 1/20th point
paragraph.setIndentationFirstLine(indentationFirstLine);
Integer indentationLeft; // 1/20th point
paragraph.setIndentationLeft(indentationLeft);
Integer indentationRight; // 1/20th point
paragraph.setIndentationRight();

Spacing

XWPFParagraph paragraph = document.createParagraph();

Integer spacingBefore; // 1/20th point
paragraph.setSpacingBefore(spacingBefore);
Integer spacingBeforeLines; // 1/100th line
paragraph.setSpacingBeforeLines(spacingBeforeLines);
Integer spacingAfter; // 1/20th point
paragraph.setSpacingAfter(spacingAfter);
Integer spacingAfterLines; // 1/100th line
paragraph.setSpacingAfterLines(spacingAfterLines);
Integer spacingBetween; // 1 line or 1 point, It depends on what LineSpacingRule used
paragraph.setSpacingBetween(spacingBetween, LineSpacingRule.AUTO);
paragraph.setSpacingBetween(spacingBetween, LineSpacingRule.EXACT);

Font

XWPFParagraph paragraph = document.createParagraph();

XWPFRun run = paragraph.createRun();
Integer fontSize; // 1 point
run.setFontSize(fontSize);
String fontFamily = "Calibri";
run.setFontFamily(fontFamily);
String color = "000000"; // RGB string
run.setColor(color);
run.setBold(false);
run.setItalic(false);
UnderlinePatterns underline;
run.setUnderline(underline);
run.setStrikeThrough(false);
Integer characterSpacing; // 1/20th points
run.setCharacterSpacing(characterSpacing)

Text

XWPFParagraph paragraph = document.createParagraph();

XWPFRun run = paragraph.createRun();
String text = "Hello World!";
run.setText(text);

Add newline

run.addBreak();

Insert Images

XWPFParagraph imageParagraph = document.createParagraph();

imageParagraph.setAlignment(ParagraphAlignment.CENTER);
XWPFRun imageRun = imageParagraph.createRun();
Integer textPosition = 100; // 1/2nd points
imageRun.setTextPosition(textPosition);
byte[] imageBytes = IOUtils.toByteArray(bufferedInputStream);
BufferedImage bufferedImage = ImageIO.read(new ByteArrayInputStream(imageBytes));
int imageWidth = bufferedImage.getWidth();
int imageHeight = bufferedImage.getHeight();
double scalePercent = 0.2;
int scaledWidth = (int) (imageWidth * scalePercent);
int scaledHeight = (int) (imageHeight * scalePercent);
String fileName = imageFilePath.substring(imageFilePath.lastIndexOf(File.separator) + 1);
imageRun.addPicture(new ByteArrayInputStream(imageBytes),
getImageFormat(fileName),
fileName,
Units.toEMU(scaledWidth),
Units.toEMU(scaledHeight));
document.write(bufferedOutputStream);

Insert Tables

TODO

insert Charts

TODO

Insert HTML

Excel

Write Data into Excel

A basic use example

public static void main(String[] args) throws IOException {
XSSFWorkbook workbook = new XSSFWorkbook();
XSSFSheet sheet = workbook.createSheet();
XSSFRow row = sheet.createRow(0);
XSSFCell cell = row.createCell(0);
cell.setCellValue("Hello World");
workbook.write(new FileOutputStream("D:\\test.xlsx"));
}

Column

Column width

sheet.setColumnWidth(0, 15 * 256); // unit: 1/256 character

Row

Row height

XSSFRow row = sheet.createRow(rowNum);
row.setHeight((short) (row.getHeight() * 20)); // unit: 1/20 point

Cell

XSSFCell

XSSFCell cell = row.createCell(0);
// value
cell.setCellValue("Hello World");

XSSFCellStyle cellStyle = workbook.createCellStyle();
// align
cellStyle.setAlignment(HorizontalAlignment.CENTER);
cellStyle.setVerticalAlignment(VerticalAlignment.CENTER);
// wrap text
cellStyle.setWrapText(true);
// backgroud color
cellStyle.setFillForegroundColor(IndexedColors.YELLOW.getIndex());
cellStyle.setFillPattern(FillPatternType.SOLID_FOREGROUND);
// custom backgroud color. note: use XSSFCellStyle object.
byte[] rgb = {(byte) 155, (byte) 194, (byte) 230};
cellStyle.setFillForegroundColor(new XSSFColor(rgb, new DefaultIndexedColorMap()));
cellStyle.setFillPattern(FillPatternType.SOLID_FOREGROUND);
// border
cellStyle.setBorderTop(BorderStyle.MEDIUM);
cellStyle.setBorderBottom(BorderStyle.MEDIUM);
cellStyle.setBorderLeft(BorderStyle.MEDIUM);
cellStyle.setBorderRight(BorderStyle.MEDIUM);
cellStyle.setTopBorderColor(IndexedColors.BLUE.getIndex());
cellStyle.setBottomBorderColor(IndexedColors.BLUE.getIndex());

// font
Font font = workbook.createFont();
font.setFontName("仿宋");
font.setFontHeightInPoints((short) 14);
font.setColor(IndexedColors.RED.getIndex());
font.setBold(true);
font.setItalic(true);
font.setUnderline(FontUnderline.SINGLE);
font.setStrikeout(true);
cellStyle.setFont(font);

cell.setCellStyle(cellStyle);

HSSFCell

// custom backgroud color
HSSFPalette palette = workbook.getCustomPalette();
// the palette index, between 0x8 to 0x40 inclusive
short colorIndex = 8;
palette.setColorAtIndex(colorIndex, (byte) 189, (byte) 215, (byte) 238);
cellStyle.setFillForegroundColor(palette.getColor(colorIndex).getIndex());
cellStyle.setFillPattern(FillPatternType.SOLID_FOREGROUND);
XSSFCell cell = sheet.createRow(rowNum).createCell(columnNum);
cell.setCellValue("Open " + url);
XSSFWorkbook workbook = sheet.getWorkbook();
CreationHelper createHelper = workbook.getCreationHelper();
XSSFHyperlink hyperlink = (XSSFHyperlink) createHelper.createHyperlink(HyperlinkType.URL);
hyperlink.setAddress(url);
cell.setHyperlink(hyperlink);
XSSFCellStyle cellStyle = workbook.createCellStyle();
XSSFFont font = workbook.createFont();
font.setUnderline(FontUnderline.SINGLE);
font.setColor(IndexedColors.BLUE.getIndex());
cellStyle.setFont(font);
cell.setCellStyle(cellStyle);

Rich Text String

XSSFCell cell = sheet.createRow(rowNum).createCell(columnNum);
XSSFFont font = new XSSFFont();
font.setColor(IndexedColors.RED.getIndex());
XSSFFont font2 = new XSSFFont();
font2.setColor(IndexedColors.GREEN.getIndex());
RichTextString richTextString = sheet.getWorkbook().getCreationHelper()
.createRichTextString("hello world");
richTextString.applyFont(0, 5, font);
richTextString.applyFont(6, 11, font2);
cell.setCellValue(richTextString);

Merge Cell

sheet.addMergedRegion(new CellRangeAddress(startRow, endRow, startCol, endCol));
sheet.addMergedRegion(CellRangeAddress.valueOf("A1:B3"));

Note: If set cell styles for merged cells, you should set the leftmost or topmost cell.

Cell Utility

get cell

Cell cell = CellUtil.getCell(CellUtil.getRow(rowNum, sheet), columnNum);

set cell style

// align
CellUtil.setAlignment(cell, HorizontalAlignment.LEFT);
CellUtil.setVerticalAlignment(cell, VerticalAlignment.TOP);
// wrap text
CellUtil.setCellStyleProperty(cell, CellUtil.WRAP_TEXT, true);
// font
CellUtil.setFont(cell, font);

Insert Images

InputStream inputStream = new FileInputStream("C:\\Users\\Taogen\\Desktop\\demo.png");
byte[] bytes = IOUtils.toByteArray(inputStream);
int pictureIdx = workbook.addPicture(bytes, Workbook.PICTURE_TYPE_PNG);
inputStream.close();
//Returns an object that handles instantiating concrete classes
CreationHelper helper = workbook.getCreationHelper();
//Creates the top-level drawing patriarch.
Drawing drawing = sheet.createDrawingPatriarch();
//Create an anchor that is attached to the worksheet
ClientAnchor anchor = helper.createClientAnchor();
//set top-left corner for the image
anchor.setCol1(1);
anchor.setRow1(2);
//Creates a picture
Picture picture = drawing.createPicture(anchor, pictureIdx);
//Reset the image size
double scale = 0.2;
picture.resize(scale); // or picture.resize() original size

Read Data from Excel

Workbook workbook = new XSSFWorkbook(new File(filePath));
DataFormatter formatter = new DataFormatter();
Sheet sheet = workbook.getSheetAt(0);
Iterator<Row> iterator = sheet.iterator();
Row row;
while (iterator.hasNext()) {
row = iterator.next();
for (Cell cell : row) {
System.out.print(formatter.formatCellValue(cell) + "\t");
}
System.out.println();
}

Appendixes

Javadocs

Excel (XSSF)

Word (XWPF)

In the SQL world, order is not an inherent property of a set of data. Thus, you get no guarantees from your RDBMS that your data will come back in a certain order – or even in a consistent order – unless you query your data with an ORDER BY clause.

If an <order by clause> is not specified, then the ordering of the rows is implementation-dependent.

Default Orders

MySQL Server v5.6, InnoDB

If select fields only in unique key/index, the default order is ordered by unique key/index.

if select fields only in primary key, the default order is ordered by unique key/index or primary key

if select fields contain a field that is not in the primary key, unique key, and index, the default order is ordered by the primary key.

Suggestions

Do not depend on order when ORDER BY is missing.

Always specify ORDER BY if you want a particular order.

References

[1] What is The Default Sort Order of SELECTS with no ORDER BY Clause?

[2] What is the default order of records for a SELECT statement in MySQL?

[3] SQL: What is the default Order By of queries?

Ways of Exporting Requests

Synchronized Export

If to exported data is not large and the exporting process can be finished in a few seconds, we can just use the synchronized exporting. Send an export request, and then download the organized data file.

Asynchronized Export

If to exported data is too large, exporting will cost a lot of time. So we need to use the asynchronized exporting. Send an export request, view the schedule of the export, wait for the handling of exporting files to be finished, and download the organized data file.

Ways of Export implementations

  1. Write data into Java servlet response output stream.

  2. Write data into file and store in user temporary directory user.dir, and return download file URI. Delete file when download is finished.

  3. Upload data to OSS, and return download file URL.

The Limit of Max Size of Exported Data

Sometimes we need consider to setting the max size of data for exporting.

Fetch External Data

Fetch data from relational databases

Fetch static files with HTTP URLs

Build Exported Files

Exported File Types

  • Office Excel
  • Office Doc
  • Zip

Optimization

Database

  • SQL optimization. 1) Only query required columns. 2) Add index for query.

Cache

  • Cache rarely modified database data to Redis or memory.

Network IO

Fetch Database data

  • Fetch rows with multiple threads.
  • Each time to fetch a limited number of rows. It depends on the data size of a row. E.g. for the small size of a row, you can fetch 500 rows at a time.

Fetch Static Files by URL

  • Fetch files with multiple threads. E.g. 20 threads.
  • Caching files in the temporary directory.
  • Using blocking NIO, non-blocking NIO, or NIO2.

Compression

  • Compress images.
  • Compress text files.
  • Compress binary files.

Disk IO

  • When using traditional blocking IO, reading and writing files should use buffered Input/output wrapper class (E.g. BufferedInputStream) or read from/write to direct buffering array.

    // buffered Input/output wrapper class
    FileInputStream fis = new FileInputStream(filepath);
    BufferedInputStream bis = new BufferedInputStream(fis);
    // direct buffering array
    FileInputStream fis = new FileInputStream(filepath);
    byte buf[] = new byte[2048];
    int len;
    while ((len = fis.read(buf)) != -1) {}
  • (Optional) Using blocking NIO, non-blocking NIO, or NIO2. The Java NIO package offers the possibility to transfer bytes between two Channels without buffering them into the application memory. In single thread environment, traditional IO is better. NIO is used not because it’s faster but because it has better scalability especially there are amounts of clients.

    BufferedReader reader = null;
    BufferedWriter writer = null;
    try {
    Path inputFile = Paths.get(inputPath);
    Path outputFile = Paths.get(outputPath);
    reader = Files.newBufferedReader(inputFile, Charset.defaultCharset());
    writer = Files.newBufferedWriter(outputFile, Charset.defaultCharset());
    String lineFromFile = "";
    while ((lineFromFile = reader.readLine()) != null) {
    writer.append(lineFromFile);
    }
    } catch (IOException e) {
    System.out.println(e);
    } finally {
    reader.close();
    writer.close();
    }
    RandomAccessFile input = new RandomAccessFile(inputPath, "r");
    FileChannel inChannel = input.getChannel();
    RandomAccessFile output = new RandomAccessFile(outputPath, "rw");
    FileChannel outChannel = output.getChannel();
    int bufferSize = 1024;
    if (bufferSize > inChannel.size()) {
    bufferSize = (int) inChannel.size();
    }
    ByteBuffer buff = ByteBuffer.allocate(bufferSize);
    buff.clear();
    while (inChannel.read(buff) != -1) {
    buff.flip();
    outChannel.write(buff);
    buff.compact();
    }
    buff.flip();
    while (buff.hasRemaining()) {
    outChannel.write(buff);
    }
    inChannel.close();
    outChannel.close();

Build Files

  • Build exported files with multiple threads. And then pack them into a zip file.

Notice

Out of memory

Be careful when loading large of data into memory.

Memory leak

Don’t forget close resources. E.g. thread pool, input/output streams and so on.

References

[1] Tuning Java I/O Performance - Oracle

[2] how to choose java nio vs io? - Stack Overflow

[3] NIO Performance Improvement compared to traditional IO in Java - Stack Overflow

[4] FileChannel, ByteBuffer, Memory-Mapped I/O, Locks

关于学习编程技术,我认为,没有最好的方法,只有适合自己当前水平状态的、可以达成自己目标的学习方法。每个人有不同的适应自己的学习方法,每个阶段有不同的学习方法,学习的目的不同使用的方法也不同。简而言之,方法不是固定不变的。

学习方法一般有:

  • 看官方文档。
  • 看书籍。
  • 看博客、教程等技术文章。
  • 看视频教程(倍速)。
  • 看源码。
  • 跟着教程敲代码,写demo,做练习。
  • 在真实的项目中应用该技术。

学习的目的一般有:

  • 入门,学会基本概念和使用。
  • 熟悉,查漏补缺,加强熟练度。
  • 精通,深入理解架构设计和代码实现原理。

学习主要考虑两个问题:质量和速度。有的时候需要优先考虑质量,有的时候则需要优先考虑速度。

(一)入门/了解

入门在于理解,表现为能够使用简单的几句话将这门技术的核心概念描述清楚。

如果是比较急(速度优先),则优先考虑看入门视频教程,或者入门博客教程等。文档和书籍一般都会比较权威、严谨和全面详细,然而缺点就是比较枯燥和繁琐,概念特别多,容易抓不住重点。然而入门视频教程通过通俗易懂的讲解、简单的示例代码、和抓重点内容,能够让你更好地快速入门。

如果不是很着急,可以先尝试看官方文档,如果官方文档写得对新手不太友好,可以尝试找一些初级的书籍看看。如果文档和书籍都看不下去,最后,再去找一些入门视频教程和博客文章去看。如 Spring Security 的官方文档就对新手不太友好。

总之,先找一个可以理解的资料看(官方文档 -> 入门书籍 -> 入门视频教程和博客文章),理解之后还是要看一遍官方文档,官网文档必须得看。官方文档是最重要的一手资料。

另外,入门的话,跟着教程敲 demo 代码也是很有必要的。虽然 demo 代码都比较简单,自己能够看得懂,但是还是要自己敲一遍。至少能够加深一些印象。

注意事项:

  • 入门时尽量不记或者少记笔记,在大量练习使用后,再做笔记。这时候才知道哪些是重要的或经常使用的。

(二)熟悉/熟练

熟悉在于大量练习,表现为对这门技术的大部分的功能用法(如:注解、配置等)和 API 调用可以熟稔于心、信手拈来。

想要熟悉一个技术,首先需要认真地多看几遍权威的资料的,如官方文档和书籍。另外,大量的练习也是必不可少的。站在岸上学不会游泳,可以做一些 toy project 或者在真实的项目中应用。

如果一个技术平时很少用到,那么我们可以通过解决别人遇到的问题来不断地接触到它。比如:Spring Security 只会在项目搭建的时候用到,项目搭建好了,就很少能够接触到它。常见的帮助别人解决相关技术问题的方式:1)Stack Overflow 上有专门的技术标签,标签下有很多其他人在使用技术过程中遇到的问题;2)混技术论坛,偶尔会看到有人提出自己遇到的相关技术问题。

在准备面试的时候,我们通常需要去背很多知识点。因为这些知识点平时用的很少,但面试需要我们流畅地表达出来。想要对某些技术知识点做到脱口而出,要么死记硬背,要么大量使用。所以面试的时候,对于用的少的知识,要去记忆和背诵。虽然短时间能记住很多,但没使用过或使用得很少,复述出来还是不太流畅。

熟练:理解上比较深,使用上比较熟悉、轻松。首先,在一定次数的使用中不断地加深理解和熟悉使用,再结合阅读源码了解其实现原理。

(三)精通

何为精通?这似乎是个很模糊的概念,很难去定义。就算是领域高手也很少说自己精通某个技术。

我认为,精通就是有较深的理解,有丰富的使用经验,能够灵活、恰当、熟练地使用它。

似乎精通没有极限,我们只能在通往精通的道路上不断前进。学无止境,不必为了精通而精通。以目的和兴趣去驱动,不断提高技术水平。

(四)最后

学完不用就会忘,技术还是得需求驱动。比如:没有经历高并发场景的人很难学习分布式/微服务相关技术。学习只是入门了解,要熟练掌握一门技术,还需要大量的使用,在大量的重复使用中不断加深理解、熟悉用法。后面,再进一步阅读源码,深入理解。

知易行难,唯有行动,才能改变。不断努力,不断成长。

References

[1] 学习编程是看文档还是看视频教程

HTML 元素原则上应该保持唯一,但没有硬性要求。

存在多个相同的 id 元素时,CSS 对所有元素都生效。

<style>
#myId {
color:red;
}
</style>
<div id="myId">
div1
</div>
<div id="myId">
div2
</div>

然而,JavaScript 和 jQuery 都只获取多个相同的 id 元素的第一个元素。

document.getElementById("myId").innerHTML //div1
$("#myId").html() //div1

Background

Using List<Integer> ids to receive a form parameter of HTTP request. The size of the parameter ids is over 255.

Error Info

org.springframework.beans.InvalidPropertyException: Invalid property 'ids[256]' of bean class [com.fantasi.manage.console.web.modules.recovery.dto.RecoveryGroupDto]: Invalid list index in property path 'ids[256]'; nested exception is java.lang.IndexOutOfBoundsException: Index: 256, Size: 256. 

Solutions

To override init Binder method and configure the limit size which needs so that you can pass those many objects to your controller classes.

@Controller
public class MyController {

@InitBinder
public void initBinder(WebDataBinder dataBinder) {
dataBinder.setAutoGrowCollectionLimit(500);
}

...
}

@InitBinder plays the role to identify the methods which initialize the WebDataBinder.

WebDataBinder is a DataBinder that binds request parameter to JavaBean objects.

Reasons

By default spring only maps only 255 objects to the java bean. Spring developers has done this to prevent OutOfMemory problem.

References

[1] Spring initBinder for Handling Large List of Java Objects

Encode

Base64

String text = "hello 世界";
String encodeText = Base64.getEncoder().encodeToString(text.getBytes(StandardCharsets.UTF_8));
System.out.println(encodeText);
String decodeText = new String(Base64.getDecoder().decode(encodeText), StandardCharsets.UTF_8);
System.out.println(decodeText);

URL Encode

String text = "hello 世界";
String encodeText = URLEncoder.encode(text, StandardCharsets.UTF_8.name());
System.out.println(encodeText);
String decodeText = URLDecoder.decode(encodeText, StandardCharsets.UTF_8.name());
System.out.println(decodeText);

Hash Algorithms

MD5

MD5 (Message Digest 5)

String text = "123456";
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
messageDigest.reset();
messageDigest.update(text.getBytes(StandardCharsets.UTF_8));
byte[] digest = messageDigest.digest();
String hashText = DatatypeConverter.printHexBinary(digest);
System.out.println(hashText);

Note: the MessageDigest is not thread-safe. Consequently, we should use a new instance for every thread.

SHA

SHA (Secure Hash Algorithm)

SHA-256 by MessageDigest

String text = "123456";
MessageDigest messageDigest = MessageDigest.getInstance("SHA-256");
messageDigest.reset();
messageDigest.update(text.getBytes(StandardCharsets.UTF_8));
byte[] digest = messageDigest.digest();
String hashText = DatatypeConverter.printHexBinary(digest);
System.out.println(hashText);

SHA3-256 by MessageDigest (since Java 9)

String text = "123456";
MessageDigest messageDigest = MessageDigest.getInstance("SHA3-256");
messageDigest.reset();
messageDigest.update(text.getBytes(StandardCharsets.UTF_8));
byte[] digest = messageDigest.digest();
String hashText = DatatypeConverter.printHexBinary(digest);
System.out.println(hashText);

Conclusion

  • SHA256 is difficult to handle than MD5 because of its size.
  • SHA256 is less secure than MD5
  • MD5 result in an output of 128 bits whereas SHA256 result output of 256 bits.

Concluding all points, it will be better to use MDA5 if you want to secure your files otherwise you can use SHA256.

Symmetric Encryption Algorithms

DES

DES (data encryption standard, 1976)

DES is Not Secure.

3DES

3DES is Not Secure.

AES

AES (advanced encryption system, 2001)

The AES algorithm has six modes of operation:

  1. ECB (Electronic Code Book) Not Recommend
  2. CBC (Cipher Block Chaining)
  3. CFB (Cipher FeedBack)
  4. OFB (Output FeedBack)
  5. CTR (Counter)
  6. GCM (Galois/Counter Mode)

Don’t use AES Electronic codebook (ECB) Mode. The AES ECB mode, or AES/ECB/PKCS5Padding (in Java) is not semantically secure.

AES encryption best practice: Don’t reuse IV with the same key.

IV

The IV (initial value or initial vector), it is random bytes, typically 12 bytes or 16 bytes. In Java, we can use SecureRandom to generate the random IV.

public static IvParameterSpec generateIv() {
byte[] iv = new byte[16];
new SecureRandom().nextBytes(iv);
return new IvParameterSpec(iv);
}

secret key

The AES secret key, either AES-128 or AES-256. In Java, we can use KeyGenerator to generate the AES secret key.

public static SecretKey getAESKey() throws NoSuchAlgorithmException {
KeyGenerator keyGen = KeyGenerator.getInstance("AES");
keyGen.init(256, SecureRandom.getInstanceStrong());
return keyGen.generateKey();
}

The AES secret key that derived from a given password. In Java, we can use the SecretKeyFactory and PBKDF2WithHmacSHA256 to generate an AES key from a given password.

public static SecretKey getAESKeyFromPassword(char[] password, byte[] salt)
throws NoSuchAlgorithmException, InvalidKeySpecException {
SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
// iterationCount = 65536
// keyLength = 256
KeySpec spec = new PBEKeySpec(password, salt, 65536, 256);
SecretKey secret = new SecretKeySpec(factory.generateSecret(spec).getEncoded(), "AES");
return secret;
}

salt

We use salt to protect rainbow attacks, and it is also a random byte, we can use the SecureRandom to generate it.

import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.KeySpec;
import java.util.ArrayList;
import java.util.List;

public class CryptoUtils {

public static byte[] getRandomNonce(int numBytes) {
byte[] nonce = new byte[numBytes];
new SecureRandom().nextBytes(nonce);
return nonce;
}

// AES secret key
public static SecretKey getAESKey(int keysize) throws NoSuchAlgorithmException {
KeyGenerator keyGen = KeyGenerator.getInstance("AES");
keyGen.init(keysize, SecureRandom.getInstanceStrong());
return keyGen.generateKey();
}

// Password derived AES 256 bits secret key
public static SecretKey getAESKeyFromPassword(char[] password, byte[] salt)
throws NoSuchAlgorithmException, InvalidKeySpecException {
SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
// iterationCount = 65536
// keyLength = 256
KeySpec spec = new PBEKeySpec(password, salt, 65536, 256);
SecretKey secret = new SecretKeySpec(factory.generateSecret(spec).getEncoded(), "AES");
return secret;
}

// hex representation
public static String hex(byte[] bytes) {
StringBuilder result = new StringBuilder();
for (byte b : bytes) {
result.append(String.format("%02x", b));
}
return result.toString();
}

// print hex with block size split
public static String hexWithBlockSize(byte[] bytes, int blockSize) {
String hex = hex(bytes);
// one hex = 2 chars
blockSize = blockSize * 2;
// better idea how to print this?
List<String> result = new ArrayList<>();
int index = 0;
while (index < hex.length()) {
result.add(hex.substring(index, Math.min(index + blockSize, hex.length())));
index += blockSize;
}
return result.toString();
}
}

CBC Mode

Input

  • algorithm: AES/CBC/PKCS5Padding
  • iv bytes
  • aes key bits
  • secretKey: generate by password
  • Initialization Vector (IV): random bytes

GCM Mode


Asymmetric Encryption Algorithms

RSA

RSA (Rivest-Shamir-Adleman)

public static void getKeyPair() throws NoSuchAlgorithmException {
KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
kpg.initialize(2048);
KeyPair keyPair = kpg.generateKeyPair();
Key pub = keyPair.getPublic();
Key pvt = keyPair.getPrivate();
System.out.println("public key");
System.out.println(Base64.getEncoder().encodeToString(pub.getEncoded()));
System.out.println("private key");
System.out.println(Base64.getEncoder().encodeToString(pvt.getEncoded()));
}

public static String decrypt(String privateKeyBase64String, String cipherText) throws NoSuchAlgorithmException, InvalidKeySpecException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException {
KeyFactory keyFactory = KeyFactory.getInstance("RSA");
EncodedKeySpec pkcs8KeySpec = new PKCS8EncodedKeySpec(Base64.getDecoder().decode(privateKeyBase64String.getBytes()));
PrivateKey privateKey = keyFactory.generatePrivate(pkcs8KeySpec);
Cipher cipher = Cipher.getInstance(keyFactory.getAlgorithm());
cipher.init(Cipher.DECRYPT_MODE, privateKey);
byte[] bytes = cipher.doFinal(Base64.getDecoder().decode(cipherText));
return new String(bytes);
}

public static String encrypt(String publicKeyBase64String, String plainText) throws NoSuchAlgorithmException, InvalidKeySpecException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException {
KeyFactory keyFactory = KeyFactory.getInstance("RSA");
EncodedKeySpec pkcs8KeySpec = new X509EncodedKeySpec(Base64.getDecoder().decode(publicKeyBase64String.getBytes()));
PublicKey publicKey = keyFactory.generatePublic(pkcs8KeySpec);
Cipher cipher = Cipher.getInstance(keyFactory.getAlgorithm());
cipher.init(Cipher.ENCRYPT_MODE, publicKey);
byte[] bytes = cipher.doFinal(plainText.getBytes());
return new String(Base64.getEncoder().encode(bytes));
}

ECC

ECC (Elliptic Curve Cryptography)


References

[1] Java AES encryption and decryption

0%