Web Project Functions: Export Data

Posted on 2021-12-08 Edited on 2024-10-15 In Java Web , General

Ways of Exporting Requests

Synchronized Export

If to exported data is not large and the exporting process can be finished in a few seconds, we can just use the synchronized exporting. Send an export request, and then download the organized data file.

Asynchronized Export

If to exported data is too large, exporting will cost a lot of time. So we need to use the asynchronized exporting. Send an export request, view the schedule of the export, wait for the handling of exporting files to be finished, and download the organized data file.

Ways of Export implementations

Write data into Java servlet response output stream.
Write data into file and store in user temporary directory user.dir, and return download file URI. Delete file when download is finished.
Upload data to OSS, and return download file URL.

The Limit of Max Size of Exported Data

Sometimes we need consider to setting the max size of data for exporting.

Fetch External Data

Fetch data from relational databases

Fetch static files with HTTP URLs

Build Exported Files

Exported File Types

Office Excel
Office Doc
Zip

Optimization

Database

SQL optimization. 1) Only query required columns. 2) Add index for query.

Cache

Cache rarely modified database data to Redis or memory.

Network IO

Fetch Database data

Fetch rows with multiple threads.
Each time to fetch a limited number of rows. It depends on the data size of a row. E.g. for the small size of a row, you can fetch 500 rows at a time.

Fetch Static Files by URL

Fetch files with multiple threads. E.g. 20 threads.
Caching files in the temporary directory.
Using blocking NIO, non-blocking NIO, or NIO2.

Compression

Compress images.
Compress text files.
Compress binary files.

Disk IO

When using traditional blocking IO, reading and writing files should use buffered Input/output wrapper class (E.g. BufferedInputStream) or read from/write to direct buffering array.

// buffered Input/output wrapper class
FileInputStream fis = new FileInputStream(filepath);
BufferedInputStream bis = new BufferedInputStream(fis);

// direct buffering array
FileInputStream fis = new FileInputStream(filepath);
byte buf[] = new byte[2048];
int len;
while ((len = fis.read(buf)) != -1) {}

(Optional) Using blocking NIO, non-blocking NIO, or NIO2. The Java NIO package offers the possibility to transfer bytes between two Channels without buffering them into the application memory. In single thread environment, traditional IO is better. NIO is used not because it’s faster but because it has better scalability especially there are amounts of clients.

BufferedReader reader = null;
BufferedWriter writer = null;
try {
    Path inputFile = Paths.get(inputPath);
    Path outputFile = Paths.get(outputPath);
    reader = Files.newBufferedReader(inputFile, Charset.defaultCharset());
    writer = Files.newBufferedWriter(outputFile, Charset.defaultCharset());
    String lineFromFile = "";
    while ((lineFromFile = reader.readLine()) != null) {
        writer.append(lineFromFile);
    }
} catch (IOException e) {
    System.out.println(e);
} finally {
    reader.close();
    writer.close();
}

RandomAccessFile input = new RandomAccessFile(inputPath, "r");
FileChannel inChannel = input.getChannel();
RandomAccessFile output = new RandomAccessFile(outputPath, "rw");
FileChannel outChannel = output.getChannel();
int bufferSize = 1024;
if (bufferSize > inChannel.size()) {
    bufferSize = (int) inChannel.size();
}
ByteBuffer buff = ByteBuffer.allocate(bufferSize);
buff.clear();
while (inChannel.read(buff) != -1) {
    buff.flip();
    outChannel.write(buff);
    buff.compact();
}
buff.flip();
while (buff.hasRemaining()) {
    outChannel.write(buff);
}
inChannel.close();
outChannel.close();