Taogen's Blog

Stay hungry stay foolish.

Backend

Design

  • Class hierarchy design.

Global Functions

  • Core
    • For user
      • Authentication and authorization(menu, privilege, data scope). Sign in and sign out. Session Expiration Time. User, role, privilege. Data privilege and operation privilege.
      • File upload and download.
      • Scheduling Tasks.
    • For system
      • Exception handling. Error message encapsulation.
      • Logging. Method logging with AOP and annotations.
      • Cache handling.
      • Concurrency and asynchronization.
      • Testing. Unit testing.
      • HTTP clients
  • Others
    • For user
      • File online preview.
      • File conversion.
      • SSO (Single Sign on)
      • Web Socket and browser notification. Push messages to client browser pages. (implemented by WebSocket or Ajax loop call)
      • Workflow.
      • I18n (database value, Java string value, JSP element value)
      • Full-text Search by Elasticserach.
    • For system
      • Filter error request.
      • Enable CORS(Cross-Origin Resource Sharing) on server-side.
      • Prevent duplicate form submission.
      • Using multiple data sources.
      • Performance optimization. For example, MySQL optimization. Indexing, optimizing schema and SQL. JVM optimization.
      • Data import into and export from MySQL and Elasticsearch.
      • Third party API access.
      • HTTPS.
      • Reverse Proxy Server.
      • DevOps. Git Flow, code review, test, package, deployment.
      • Crawler and anti-cralwer.

Module Functions

  • RESTful API. CRUD API + UI pages (single table, left tree right table, one-to-many nested tables)

    • Request parameters to POJO. E.g. spring framework@ModelAttribute, @RequestBody.
    • Data validation. E.g. spring framework@Validated).
    • Type conversion. E.g. spring framework @DateTimeFormat for datetime, @JsonSerialize for enumeration).
    • Data formatting. E.g. spring framework @JsonFormat for datetime, @JsonValue form enumeration).
    • Service transaction.
    • Persistence. CRUD.
  • Table data import from and export to Excel file.

  • Aggregation query (SQL, ES) and UI charts.

  • Unit tests.

Frontend

Basic UI Components

  • Top bar <header>. (Logo, website name, user center link, login/logout)
  • Horizontal navigation bar <nav>.
  • Left vertical sidebar menu <aside>.
  • Breadcrumb.
  • Content area <main>, <section>, <article>. (Search, Table and pagination, Form)
  • Bottom footer <footer>.

Global Functions

  • Copy text.
  • View bigger picture.
  • Prevent duplicate form submission.
  • Redirect to another page.
  • Send CRUD HTTP requests. Submit form.
  • Initialize operation privilege.
  • Load components.
  • File structure. Separated index, edit, view page.
  • Alert success and error message.
  • Send CORS Ajax request with jQuery.
  • Send browser notifications.

Module Functions

  • List page
    • Initialize operation privilege.
    • Initialize index page. Display breadcrumb, load search area elements (input, select, date time picker), display operation buttons row, load table and pagination, additionally, may load left area tree.
    • Index page add, edit, delete, search, import/export buttons bind events.
    • Search. Send request with form serialize or FormData.
    • Delete rows.
  • Edit page
    • Initialize edit page title, form elements, and submit button. Form elements: input, radio, checkbox, select, select picker, file, rich text editor, date time picker, and input tags.
    • Add edit page bind events.
    • Form data validation. Show error messages.
    • Save or update. Submit form, send request with FormData or JSON.
  • Other page
    • Statistical charts.
    • Complex page layout and style.

References

Semantic Elements in HTML

Introduction

cron is a UNIX tool that has been around for a long time, so its scheduling capabilities are powerful and proven.

The software utility cron also known as cron job is a time-based job scheduler in Unix-like computer operating systems. Users that set up and maintain software environments use cron to schedule jobs (commands or shell scripts) to run periodically at fixed times, dates, or intervals. It typically automates system maintenance or administration—though its general-purpose nature makes it useful for things like downloading files from the Internet and downloading email at regular intervals. The origin of the name cron is from the Greek word for time, χρόνος (chronos).

Cron is most suitable for scheduling repetitive tasks. Scheduling one-time tasks can be accomplished using the associated at utility.

Format

A cron expression is a string comprised of 6 or 7 fields separated by white space.

second, minute, hour, day of month, month, day(s) of week

Fields can contain any of the allowed values, along with various combinations of the allowed special characters for that field. The fields are as follows:

Field Name Mandatory Allowed Values Allowed Special Characters
Seconds YES 0-59 , - * /
Minutes YES 0-59 , - * /
Hours YES 0-23 , - * /
Day of month YES 1-31 , - * ? / L W
Month YES 1-12 or JAN-DEC , - * /
Day of week YES 1-7 or SUN-SAT , - * ? / L #
Year NO empty, 1970-2099 , - * /

So cron expressions can be as simple as this: * * * * ? *

or more complex, like this: 0/5 14,18,3-39,52 * ? JAN,MAR,SEP MON-FRI 2002-2010

Special characters

  • * (“all values”) - used to select all values within a field. For example, “*****” in the minute field means “every minute”.
  • ? (“no specific value”) - useful when you need to specify something in one of the two fields in which the character is allowed, but not the other. For example, if I want my trigger to fire on a particular day of the month (say, the 10th), but don’t care what day of the week that happens to be, I would put “10” in the day-of-month field, and “?” in the day-of-week field.

Note: ‘?’ can only be specfied for Day-of-Month or Day-of-Week.

  • - - used to specify ranges. For example, “10-12” in the hour field means “the hours 10, 11 and 12”.
  • , - used to specify additional values. For example, “MON,WED,FRI” in the day-of-week field means “the days Monday, Wednesday, and Friday”.
  • / - used to specify increments. For example, “0/15” in the seconds field means “the seconds 0, 15, 30, and 45”. And “5/15” in the seconds field means “the seconds 5, 20, 35, and 50”. You can also specify ‘/’ after the ‘’ character - in this case ‘’ is equivalent to having ‘0’ before the ‘/’. ‘1/3’ in the day-of-month field means “fire every 3 days starting on the first day of the month”.
  • L (“last”) - has different meaning in each of the two fields in which it is allowed. For example, the value “L” in the day-of-month field means “the last day of the month” - day 31 for January, day 28 for February on non-leap years. If used in the day-of-week field by itself, it simply means “7” or “SAT”. But if used in the day-of-week field after another value, it means “the last xxx day of the month” - for example “6L” means “the last friday of the month”. You can also specify an offset from the last day of the month, such as “L-3” which would mean the third-to-last day of the calendar month. When using the ‘L’ option, it is important not to specify lists, or ranges of values, as you’ll get confusing/unexpected results.
  • W (“weekday”) - used to specify the weekday (Monday-Friday) nearest the given day. As an example, if you were to specify “15W” as the value for the day-of-month field, the meaning is: “the nearest weekday to the 15th of the month”. So if the 15th is a Saturday, the trigger will fire on Friday the 14th. If the 15th is a Sunday, the trigger will fire on Monday the 16th. If the 15th is a Tuesday, then it will fire on Tuesday the 15th. However if you specify “1W” as the value for day-of-month, and the 1st is a Saturday, the trigger will fire on Monday the 3rd, as it will not ‘jump’ over the boundary of a month’s days. The ‘W’ character can only be specified when the day-of-month is a single day, not a range or list of days.

The ‘L’ and ‘W’ characters can also be combined in the day-of-month field to yield ‘LW’, which translates to *“last weekday of the month”*.

  • # - used to specify “the nth” XXX day of the month. For example, the value of “6#3” in the day-of-week field means “the third Friday of the month” (day 6 = Friday and “#3” = the 3rd one in the month). Other examples: “2#1” = the first Monday of the month and “4#5” = the fifth Wednesday of the month. Note that if you specify “#5” and there is not 5 of the given day-of-week in the month, then no firing will occur that month.

The legal characters and the names of months and days of the week are not case sensitive. MON is the same as mon.

Examples

Expression Meaning
0 0 12 * * ? Fire at 12pm (noon) every day
0 15 10 ? * * Fire at 10:15am every day
0 15 10 * * ? Fire at 10:15am every day
0 15 10 * * ? * Fire at 10:15am every day
0 15 10 * * ? 2005 Fire at 10:15am every day during the year 2005
0 * 14 * * ? Fire every minute starting at 2pm and ending at 2:59pm, every day
0 */1 * * * ? Fire every minute starting at 0 second, every day
0 0/5 14 * * ? Fire every 5 minutes starting at 2pm and ending at 2:55pm, every day
0 0/5 14,18 * * ? Fire every 5 minutes starting at 2pm and ending at 2:55pm, AND fire every 5 minutes starting at 6pm and ending at 6:55pm, every day
0 0-5 14 * * ? Fire every minute starting at 2pm and ending at 2:05pm, every day
0 10,44 14 ? 3 WED Fire at 2:10pm and at 2:44pm every Wednesday in the month of March.
0 15 10 ? * MON-FRI Fire at 10:15am every Monday, Tuesday, Wednesday, Thursday and Friday
0 15 10 15 * ? Fire at 10:15am on the 15th day of every month
0 15 10 L * ? Fire at 10:15am on the last day of every month
0 15 10 L-2 * ? Fire at 10:15am on the 2nd-to-last last day of every month
0 15 10 ? * 6L Fire at 10:15am on the last Friday of every month
0 15 10 ? * 6L Fire at 10:15am on the last Friday of every month
0 15 10 ? * 6L 2002-2005 Fire at 10:15am on every last friday of every month during the years 2002, 2003, 2004 and 2005
0 15 10 ? * 6#3 Fire at 10:15am on the third Friday of every month
0 0 12 1/5 * ? Fire at 12pm (noon) every 5 days every month, starting on the first day of the month.
0 11 11 11 11 ? Fire every November 11th at 11:11am.

Generators

Cron Expression Generator - freeformatter

References

[1] cron - Wikipedia

[2] Cron Trigger Tutorial

[3] Scheduling Tasks - Spring

Configuring OS Environments

Configuring Environments on Windows

  1. Add environment variable JAVA_HOME={jdk11_path}.
  2. Add %JAVA_HOME%\bin to the environment variable path.
  3. Verify Java version. $ java -version.
  4. Restart IDE.

Note: After updating the JAVA_HOME environment variable, you need to restart your IDE.

Configuring Maven Project pom.xml

By default your version of Maven might use an old version of the maven-compiler-plugin that is not compatible with Java 9 or later versions. To target Java 9 or later, you should at least use version 3.6.0 of the maven-compiler-plugin and set the maven.compiler.release property to the Java release you are targetting (e.g. 9, 10, 11, 12, etc.). [1]

<properties>
<maven.compiler.release>11</maven.compiler.release>
</properties>

<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
</plugin>
</plugins>
</pluginManagement>
</build>

or

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<release>11</release>
</configuration>
</plugin>

Configuring IDE

Configuring Java 11 for IntelliJ IDEA

  1. Open Project Settings (Ctrl + Shift + Alt + S) –> Project –> Project SDK: 11, Project language level: 11.
  2. Open Settings (Ctrl + Alt + S) –> Build, Execution, Deployment –> Compiler –> Java Compiler –> Project bytecode version: 11

Verification

Running test

$ mvn test

References

[1] Maven in 5 Minutes

CSS Colors

Colors in CSS can be specified by the following methods:

  • Hexadecimal colors
  • Hexadecimal colors with transparency
  • RGB colors
  • RGBA colors
  • HSL colors
  • HSLA colors
  • Predefined/Cross-browser color names
  • With the currentcolor keyword

Hexadecimal Colors

A hexadecimal color is specified with: #RRGGBB, where the RR (red), GG (green) and BB (blue) hexadecimal integers specify the components of the color. All values must be between 00 and FF.

For example, the #0000ff value is rendered as blue, because the blue component is set to its highest value (ff) and the others are set to 00.

#p1 {background-color: #ff0000;}   /* red */
#p2 {background-color: #00ff00;} /* green */
#p3 {background-color: #0000ff;} /* blue */

Hexadecimal Colors With Transparency

A hexadecimal color is specified with: #RRGGBB. To add transparency, add two additional digits between 00 and FF.

#p1a {background-color: #ff000080;}   /* red transparency */
#p2a {background-color: #00ff0080;} /* green transparency */
#p3a {background-color: #0000ff80;} /* blue transparency */

RGB Colors

An RGB color value is specified with the rgb() function, which has the following syntax:

rgb(red, green, blue)

Each parameter (red, green, and blue) defines the intensity of the color and can be an integer between 0 and 255 or a percentage value (from 0% to 100%).

For example, the rgb(0,0,255) value is rendered as blue, because the blue parameter is set to its highest value (255) and the others are set to 0.

Also, the following values define equal color: rgb(0,0,255) and rgb(0%,0%,100%).

#p1 {background-color: rgb(255, 0, 0);}   /* red */
#p2 {background-color: rgb(0, 255, 0);} /* green */
#p3 {background-color: rgb(0, 0, 255);} /* blue */

RGBA Colors

RGBA color values are an extension of RGB color values with an alpha channel - which specifies the opacity of the object.

An RGBA color is specified with the rgba() function, which has the following syntax:

rgba(red, green, blue, alpha)

The alpha parameter is a number between 0.0 (fully transparent) and 1.0 (fully opaque).

#p1 {background-color: rgba(255, 0, 0, 0.3);}   /* red with opacity */
#p2 {background-color: rgba(0, 255, 0, 0.3);} /* green with opacity */
#p3 {background-color: rgba(0, 0, 255, 0.3);} /* blue with opacity */

HSL Colors

HSL stands for hue, saturation, and lightness - and represents a cylindrical-coordinate representation of colors.

HSLA Colors

HSLA color values are an extension of HSL color values with an alpha channel - which specifies the opacity of the object.

Predefined/Cross-browser Color Names

140 color names are predefined in the HTML and CSS color specification.

For example: blue, red, coral, brown, etc:

#p1 {background-color: blue;}
#p2 {background-color: red;}
#p3 {background-color: coral;}

The currentcolor Keyword

The currentcolor keyword refers to the value of the color property of an element.

#myDIV {
color: blue; /* Blue text color */
border: 10px solid currentcolor; /* Blue border color */
}

Color Schemes

  • Achromatic Color Schemes
  • Monochromatic Color Schemes. Monochromatic schemes use different tones from the same angle on the color wheel (the same hue).
  • Analogous Color Schemes. Analogous color schemes are created by using colors that are next to each other on the color wheel.
  • Complementary Color Schemes. Complementary schemes are created by combining colors from opposite sides of the color wheel.
  • Triadic. Triadic schemes are made up of hues equally spaced around color wheel.
  • Compound (aka Split Complementary) Color Scheme. Compound schemes are almost the same as complementary schemes. Instead of using colors that are opposites, it uses colors on both sides of the opposite hue.

Color Psychology

  • Colors can influence human behavior.
  • Colors can influence human perceptions.
  • Colors can influence the taste of food.
  • Colors can influence attractiveness.
RED
Red attracts the human eye

Energy: power, strength, excitement.

Danger: fire, blood, urgency, traffic stop.

Passion: appetite, emotion, love.

GREEN
Green is the most restful color

Nature: Fertility, Freshness, New Growth.

Safety: Good Health, Healing Power, Free Traffic.

Harmony: Peace, Easiness, Calmness.

BLUE
Blue is the most used office color

Sea: Water, Ocean, Depth, Wealth.

Quality: Stability, Conservatism, Productivity, Wisdom, Intelligence.

Sky: Truth, Peace, Heaven.

Yellow
Yellow is a Happy Color

Sunshine: Light, Clarity, Energy, Warmth.

Optimism: Happiness, Positivity, Cheerfulness, Youngfulness.

Black
Black is Authority and Mystery

Authority: Power, Elegance, Formality.

Mystery: Evil, Fear, Death.

Gray
Grey is Conservative

Security
Reliability
Classic Knowledge
Professional Wisdom

White
White is Purity
Cleanliness Neutrality Goodness Innocence

Common Colors

  • Color Names Supported by All Browsers
  • Color Groups
  • Color Shades
  • Fashion Color

Color Names Supported by All Browsers

Color Names Supported by All Browsers

Color Groups

HTML Color Groups

2014 Material Design color palettes

The color system - Material Design

Shades of Gray

Gray Shades HEX RGB
HTML Black #000000 rgb(0,0,0)
#080808 rgb(8,8,8)
#101010 rgb(16,16,16)
#181818 rgb(24,24,24)
#202020 rgb(32,32,32)
#282828 rgb(40,40,40)
#303030 rgb(48,48,48)
#383838 rgb(56,56,56)
#404040 rgb(64,64,64)
#484848 rgb(72,72,72)
#505050 rgb(80,80,80)
#585858 rgb(88,88,88)
#606060 rgb(96,96,96)
#686868 rgb(104,104,104)
#696969 rgb(105,105,105)
#707070 rgb(112,112,112)
#787878 rgb(120,120,120)
HTML Gray #808080 rgb(128,128,128)
#888888 rgb(136,136,136)
#909090 rgb(144,144,144)
#989898 rgb(152,152,152)
#A0A0A0 rgb(160,160,160)
#A8A8A8 rgb(168,168,168)
HTML DarkGray !!! #A9A9A9 rgb(169,169,169)
#B0B0B0 rgb(176,176,176)
#B8B8B8 rgb(184,184,184)
X11 Gray #BEBEBE rgb(190,190,190)
HTML Silver #C0C0C0 rgb(192,192,192)
#C8C8C8 rgb(200,200,200)
#D0D0D0 rgb(208,208,208)
HTML LightGray #D3D3D3 rgb(211,211,211)
#D8D8D8 rgb(216,216,216)
HTML Gainsboro #DCDCDC rgb(220,220,220)
#E0E0E0 rgb(224,224,224)
#E8E8E8 rgb(232,232,232)
#F0F0F0 rgb(240,240,240)
HTML WhiteSmoke #F5F5F5 rgb(245,245,245)
#F8F8F8 rgb(248,248,248)
HTML White #FFFFFF rgb(255,255,255)

Colors for Alerts

Display Dangers

Danger!

Red often indicates a dangerous or negative situation.

HEX: #FFDDDD

Danger!

Red often indicates a dangerous or negative situation.

HEX: #F44336

Display Warnings

Warning!

Yellow often indicates a warning that might need attention.

HEX: #FFFFCC

Warning!

Yellow often indicates a warning that might need attention.

HEX: #FFEB3B

Display Successs

Success!

Green often indicates something successful or positive.

HEX: #DDFFDD

Success!

Green often indicates something successful or positive.

HEX: #4CAF50

Display Information

Info!

Blue often indicates a neutral informative change or action.

HEX: #DDFFFF

Info!

Blue often indicates a neutral informative change or action.

HEX: #2196F3

More alert colors

Danger!

Red often indicates a dangerous or negative situation.

HEX: #E91E63

Danger!

Red often indicates a dangerous or negative situation.

HEX: #FF9800

Danger!

Red often indicates a dangerous or negative situation.

HEX: #FF5722

Danger!

Red often indicates a dangerous or negative situation.

HEX: #607D8B

Danger!

Red often indicates a dangerous or negative situation.

HEX: #FFC107

Fashion Colors

2019 - Living Coral
HEX: #FF6F61
PANTONE 16-1546
2018 - Ultra Violet
HEX: #6B5B95
PANTONE 18-3838
2017 Greenery
Hex #88B04B
RGB(136, 176, 75)
Pantone 15-0343
2016 Rose Quartz
Hex #F7CAC9
RGB(247, 202, 201)
Pantone 13-1520
2016 Serenity
Hex #92A8D1
RGB(146, 168, 209)
Pantone 15-3919
2015 Marsala
Hex #955251
RGB(149, 82, 81)
Pantone 18-1438
2014 Radiand Orchid
Hex #B565A7
RGB(181, 101, 167)
Pantone 18-3224
2013 Emerald
Hex #009B77
RGB(0, 155, 119)
Pantone 17-5641
2012 Tangerine Tango
Hex #DD4124
RGB(221, 65, 36)
Pantone 17-1463
2011 Honeysucle
Hex #D65076
RGB(214, 80, 118)
Pantone 18-2120
2010 Turquoise
Hex #45B8AC
RGB(68, 184, 172)
Pantone 15-5510
2009 Mimosa
Hex #EFC050
RGB(239, 192, 80)
Pantone 14-0848
2008 Blue Izis
Hex #5B5EA6
RGB(91, 94, 166)
Pantone 18-3943
2007 Chili Pepper
Hex #9B2335
RGB(155, 35, 53)
Pantone 19-1557
2006 Sand Dollar
Hex #DFCFBE
RGB(223, 207, 190)
Pantone 13-1106
2005 Blue Turquoise
Hex #55B4B0
RGB(85, 180, 176)
Pantone 15-5217
2004 Tigerlily
Hex #E15D44
RGB(225, 93, 68)
Pantone 17-1456
2003 Aqua Sky
Hex #7FCDCD
RGB(127, 205, 205)
Pantone 14-4811
2002 True Red
Hex #BC243C
RGB(188, 36, 60)
Pantone 19-1664
2001 Fuchsia Rose
Hex #C3447A
RGB(195, 68, 122)
Pantone 17-2031
2000 Cerulean Blue
Hex #98B4D4
RGB(152, 180, 212)
Pantone 15-4020

Appendixes

Online Tools

References

CSS Colors

Color Schemes, Psychology

Common Colors

Basics

Types of Boxes

CSS display types (types of boxes):

  • block
  • inline
  • inline-block.
New Line Horizontal Space Actual Width Height Margin Padding
Block elements Always starts on a new line Always occupy the entire horizontal space. 100% by default. You can specify width. Height of content by default. Can specify height. OK OK
Inline elements Don’t start on a new line Width of content. width of content by default. Can’t specify width. Height of content by default. Can’t specify height. Horizontal margin is OK. But vertical margin don’t work. Horizontal padding is OK. But vertical padding don’t occupy space just overlay.
Inline-Block elements Don’t start on a new line Width of content by default. Or specified width. width of content by default. You can specify width. Same with “Block elements” Same with “Block elements” Same with “Block elements”

Height and Width

Width

  • 宽度一般不设置,用内容本身去撑(动态宽度)。也可以设置固定的宽度,使用百分比宽度。
  • 用 padding/margin left/right 百分比,去设置两个元素之间的水平间距。

Height

  • 高度一般也不设置,用内容本身撑(动态高度)。
  • 使用 padding/margin top/bottom 固定值,去增加高度。

Align

Horizontal Align

Horizontally center a div

Horizontally center a div using margin auto
margin: auto;
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#outerDiv {
background-color: darkgray;
}
#innerDiv {
background-color: green;
width: 200px;
height: 200px;
margin: auto;
margin-top: 40px;
margin-bottom: 40px;
}
</style>
</head>
<body>
<div>123</div>
<div id="outerDiv">
<div id="innerDiv">Content</div>
</div>
<div>123</div>
</body>
</html>
Horizontally center a div using flexbox in the container
display:flex;
flex-direction:row; /* by default */
justify-content:center;
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#outerDiv {
background-color: darkgray;
display:flex;
flex-direction:row; /* by default */
justify-content:center;

}
#innerDiv {
background-color: blue;
width: 200px;
height: 200px;
}
</style>
</head>
<body>
<div id="outerDiv">
<div id="innerDiv">Content</div>
</div>
</body>
</html>
Horizontally center a div using transform
/* container */
position: relative;
/* inner div */
position: absolute;
left: 50%;
transform: translate(-50%, 0); /* or transform: translateX(-50%);*/
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#container {
position: relative;
height: 400px;
background-color: lightgray;
}

#centeredDiv {
position: absolute;
left: 50%;
transform: translate(-50%, 0);
height: 200px;
width: 200px;
background-color: white;
}
</style>
</head>
<body>
<div id="container">
<div id="centeredDiv">
Content
</div>
</div>
</body>
</html>

Horizontally Center Text

Horizontally center text using text align
text-align: center;
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#myDiv {
background-color: green;
text-align: center;
}
</style>
</head>
<body>
<div id="myDiv">Content</div>
</body>
</html>
Horizontally center text using flex
display:flex;
flex-direction:row; /* by default */
justify-content:center;
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#myDiv {
background-color: green;
height: 200px;
display:flex;
flex-direction:row; /* by default */
justify-content:center;
}
</style>
</head>
<body>
<div id="myDiv">Content</div>
</body>
</html>

Align a div to right

Align a div to right using position
position: absolute;
right: 0px;
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#outerDiv {
background-color: darkgray;
position: absolute;
right: 0px;
}
#innerDiv {
background-color: blue;
width: 200px;
height: 200px;
}
</style>
</head>
<body>
<div id="outerDiv">
<div id="innerDiv">Content</div>
</div>
</body>
</html>
Align a div to right using float
float: right;
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#outerDiv {
background-color: darkgray;
}
#innerDiv {
background-color: green;
width: 200px;
height: 200px;
float: right;
}
</style>
</head>
<body>
<div id="outerDiv">
<div id="innerDiv">Content</div>
</div>
</body>
</html>

Align div to left and right

Align a div to left and right using float
<div class="container">
<div class="left-item"></div>
<div class="left-item"></div>
<div class="right-item"></div>
<div class="right-item"></div>
</div>
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
.container {
height: 100px;
background-color: gray;
/* fixing overflow outside by adding scroll bar*/
overflow: auto;
}
.container .left-item {
background-color: green;
width: 20%;
height: 200px;
float: left;
margin-right: 10px;
}
.container .right-item {
background-color: blue;
width: 20%;
height: 200px;
float: right;
margin-left: 10px;
}
</style>
</head>
<body>
<div class="container">
<div class="left-item">1</div>
<div class="left-item">2</div>
<div class="right-item">3</div>
<div class="right-item">4</div>
</div>
</body>
</html>

Note: If an element is taller than the element containing it, and it is floated, it will overflow outside of its container. You can use the “clearfix“ hack to fix this. We can add overflow: auto; to the containing element to fix this problem

Vertical Align

Vertically center a div

Vertically center a div using flexbox in the container

Method 1: using align-items: center

align-items: the direction is across with flex-direction.

display:flex;
flex-direction:row; /* by default*/
align-items:center;

Method 2: using justify-content: center

justify-content: the direction is the same with flex-direction.

display:flex;
flex-direction:column;
justify-content:center;

flex-direction:

  • row (by default, from left to right)
  • row-reverse
  • column (from top to bottom)
  • column-reverse
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#outerDiv {
background-color: darkgray;
height: 300px;
display:flex;
flex-direction:row; /* by default */
align-items:center;
}
#outerDiv2 {
background-color: lightgray;
height: 300px;
display:flex;
flex-direction:column;
justify-content:center;
}
#innerDiv {
background-color: green;
width: 200px;
height: 200px;
}
</style>
</head>
<body>
<div id="outerDiv">
<div id="innerDiv">Content</div>
</div>
<div id="outerDiv2">
<div id="innerDiv">Content</div>
</div>
</body>
</html>
Vertically center a div using transform
/* container */
position: relative;
/* inner div */
position: absolute;
top: 50%;
transform: translate(0, -50%); /* or transform: translateY(-50%);*/
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#container {
position: relative;
height: 400px;
background-color: lightgray;
}

#centeredDiv {
position: absolute;
top: 50%;
transform: translate(0, -50%);
height: 200px;
width: 200px;
background-color: white;
}
</style>
</head>
<body>
<div id="container">
<div id="centeredDiv">
Content
</div>
</div>
</body>
</html>

Vertically center text

Vertically center text using line height

Only center a single line text vertically

height: 150px;
line-height: 150px;
Details
<style type="text/css">
#myDiv {
background-color: green;
width: 200px;
height: 150px;
line-height: 150px;
}
</style>

<div id="myDiv">Content</div>
Vertically center text using vertical-align + table-cell

Center multiple lines text vertically

#innerDiv {
display:table-cell;
vertical-align:middle;
}
Details
<style type="text/css">
#myDiv {
background-color: green;
height:400px;
width:150px;
display:table-cell;
vertical-align:middle;
}
</style>

<div id="myDiv">this is multiple lines. this is multiple lines. this is multiple lines. this is multiple lines. this is multiple lines. this is multiple lines. this is multiple lines. </div>
Vertically center text using padding

Using padding in no height a div.

padding-top: 50px;
padding-bottom: 50px;
Details
<style type="text/css">
#myDiv {
background-color: green;
width: 200px;
padding-top: 100px;
padding-bottom: 100px;
}
</style>

<div id="myDiv">Content</div>
Vertically center text using flex
display:flex;
flex-direction:row; /* by default */
align-items:center;
Details
<style type="text/css">
#myDiv {
background-color: blue;
width: 200px;
height: 200px;
display:flex;
flex-direction:row; /* by default */
align-items:center;
}
</style>

<div id="myDiv">Content</div>
Vertically center text by center a inner div vertically

Horizontal and vertical Align

Horizontally and vertically center a div

Using flex
display:flex;
flex-direction:row; /* by default */
justify-content:center;
align-items: center;
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#outerDiv {
height: 500px;
background-color: darkgray;
display:flex;
flex-direction:row; /* by default */
justify-content:center;
align-items: center;
}
#innerDiv {
background-color: blue;
width: 200px;
height: 200px;
}
</style>
</head>
<body>
<div id="outerDiv">
<div id="innerDiv">Content</div>
</div>
</body>
</html>
Using transform
/* container */
position: relative;
/* inner div */
position: absolute;
left: 50%;
top: 50%;
transform: translate(-50%, -50%);
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#container {
position: relative;
height: 400px;
background-color: lightgray;
}

#centeredDiv {
position: absolute;
left: 50%;
top: 50%;
transform: translate(-50%, -50%);
height: 200px;
width: 200px;
background-color: white;
}
</style>
</head>
<body>
<div id="container">
<div id="centeredDiv">
Content
</div>
</div>
</body>
</html>

Horizontally and vertically center text

Using flex
display:flex;
flex-direction:row; /* by default */
justify-content:center;
align-items: center;
Details
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
#myDiv {
background-color: green;
height: 200px;
display:flex;
flex-direction:row; /* by default */
justify-content:center;
align-items: center;
}
</style>
</head>
<body>
<div id="myDiv">Content</div>
</body>
</html>

Layout

Multiple div in one line

float:left (or right);
display:inline-block;
display:flex;

Dynamic Layout

Inline elements in a block element auto wrapping lines until decrease to the block minimal width:

min-width: {fixed_value}

Break Word

  • word-break: normal; // newline for words
  • word-break: break-all; // newline for characters
  • word-break: keep-all; // don’t newline
  • word-break: break-word; // newline for words, if a single word over the block width then newline for character. Deprecated. Using word-break: normal and overflow-wrap: anywhere replace word-break: break-word.

Common CSS

first-of-type and first-child

  • xxx:first-of-type: match the first of selector occur from a parent.
  • xxx:nth-of-type(1)
  • xxx:last-of-type
  • xxx:nth-last-of-type(1)
  • xxx:first-child : match when the first of child of a parent is the selector.
  • xxx:nth-child(1)
<div class="column">
<div class="row">I am .row:first-of-child.</div>
<div class="row">this is a lines. </div>
<div class="row">this is a lines. </div>
</div>
<div class="column">
<p>I am p:first-of-child</p>
<div class="row">I am .row:first-of-type not .row:first-of-child.</div>
<div class="row">this is a lines. </div>
<div class="row">this is a lines. </div>
</div>

Classic Layout

Multiple columns align left using display: flex;

<style type="text/css">
#content {
display: flex;
flex-direction: row;
}
.column {
background-color: darkgray;
padding: 20px 20px 20px 0px;
}
.column:first-of-type {
padding-left: 20px
}
.row {
background-color: green;
height: 30px;
margin-top: 10px;
}
.row:first-of-type {
margin-top: 0px;
font-weight: bold;
}
</style>

<div id="content">
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
</div>

Multiple columns align left and one column align right using display: flex; and margin-left: auto;

<style type="text/css">
#content {
background-color: darkgray;
width: 100%;
display: flex;
flex-direction: row;
}
.column {
background-color: orange;
padding: 20px 20px 20px 0px;
}
.column:first-of-type {
padding-left: 20px
}
.column:nth-last-of-type(1) {
background-color: red;
margin-left: auto;
}
.row {
background-color: green;
height: 30px;
margin-top: 10px;
}
.row:first-of-type {
margin-top: 0px;
font-weight: bold;
}
</style>

<div id="content">
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
</div>

columns have two directions, some from align left, others align right, using flex-direction: row and flex-direction: row-reverse;

<style type="text/css">
#content {
background-color: darkgray;
width: 100%;
display: flex;
flex-direction: row;
justify-content: space-between;
}
#section1 {
display: flex;
flex-direction: row;
}
#section2 {
display: flex;
flex-direction: row-reverse;
}
.column {
background-color: orange;
padding: 20px 20px 20px 0px;
}
.column:first-of-type {
padding-left: 20px
}
.column:nth-last-of-type(1) {
background-color: red;
}
.row {
background-color: green;
height: 30px;
margin-top: 10px;
}
.row:first-of-type {
margin-top: 0px;
font-weight: bold;
}
</style>

<div id="content">
<div id="section1">
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
</div>
<div id="section2">
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
<div class="column">
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
<div class="row">this is multiple lines. </div>
</div>
</div>
</div>

Common class name

  • row
  • column
  • title
  • content
  • item
  • wrapper
  • menu-bar
  • top, header, footer
  • logo

References

[1] CSS display properties: block, inline, and inline-block — & how to tell the difference

[2] CSS Layout - Horizontal & Vertical Align - w3schools

[3] Aligning Items in a Flex Container

Errors

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

Solutions

Check InnoDB status for locks

SHOW ENGINE InnoDB STATUS;

Check MySQL open tables

SHOW OPEN TABLES WHERE In_use > 0;

Check pending InnoDB transactions

for kill slow query by view query SQL statement (trx_query)

SELECT * FROM `information_schema`.`innodb_trx` ORDER BY `trx_started`; 

Check lock dependency - what blocks what

for kill locked query

SELECT * FROM `information_schema`.`innodb_locks`;

Show running threads

get query thread ID

SHOW PROCESSLIST;
SELECT * 
FROM information_schema.processlist
WHERE user <> 'system user' and
COMMAND != 'Sleep'and
info like '%SELECT%'
order by `time` desc;
Id	`User`	Host	db	Command	`Time`	State	Info
  • Id: thread ID.
  • db: database name.
  • Command: Query, Sleep(the session is idle).
  • Time: the time in seconds that the thread has been in its current state. For a replica SQL thread, the value is the number of seconds between the timestamp of the last replicated event and the real time of the replica host.
  • Info: the statement the thread is executing, or NULL if it is executing no statement.

Kill Threads By Thread ID

Kill a Particular Thread

KILL {processlist_id};

get kill all your MySQL queries

SELECT GROUP_CONCAT(CONCAT('KILL ',id,';') SEPARATOR ' ') 
FROM information_schema.processlist
WHERE user <> 'system user' and
COMMAND != 'Sleep'and
info like '%SELECT%';
KILL 1; KILL 2;
SELECT CONCAT('KILL ',id,';') AS kill_list 
FROM information_schema.processlist
WHERE user <> 'system user' and
COMMAND != 'Sleep'and
info like '%SELECT%';
+------------------------+
| kill_list |
+------------------------+
| KILL 1; |
| KILL 2; |
+------------------------+

References

[1] Fixing “Lock wait timeout exceeded; try restarting transaction” for a ‘stuck” Mysql table?

[2] How To Kill MYSQL Queries

[3] 26.3.23 The INFORMATION_SCHEMA PROCESSLIST Table - MySQL

[4] 13.7.8.4 KILL Statement - MySQL

The web page language for displaying is:

  • Not determined by your operating system display language.
  • Not determined by your browser display language.
  • Not determined by the HTTP header Accept-Language.

The web page language for display is determined by the preferred language of your browser. The preferred language list of your browser is the same as your operating system display language or browser display language. But you can set to add your custom preferred language and move it to the top of the list.

Chrome

Chrome settings –> Advanced –> Languages –> Order languages based on your preference –> Add languages –> Move your preferred language for displaying to the top.

Firefox

Options –> General –> Language and Appearance –> Language –> Choose your preferred language for displaying pages –> Choose –> Add your language –> Move your preferred language for displaying to the top.

本文将介绍微信支付开发相关内容,微信支付有很多种途径,本文主要以 JSAPI 支付为例。接下来将介绍微信支付的基础和 JSAPI 支付开发过程。

微信支付的基础

微信支付产品

  • 付款码支付:付款码支付是用户展示微信钱包内的“刷卡条码/二维码”给商户系统扫描后直接完成支付的模式。主要应用线下面对面收银的场景。
  • Native 支付:Native支付是商户系统按微信支付协议生成支付二维码,用户再用微信“扫一扫”完成支付的模式。该模式适用于PC网站支付、实体店单品或订单支付、媒体广告支付等场景。
  • JSAPI支付:JSAPI支付是用户在微信中打开商户的H5页面,商户在H5页面通过调用微信支付提供的JSAPI接口调起微信支付模块完成支付。
  • APP支付:APP支付又称移动端支付,是商户通过在移动端应用APP中集成开放SDK调起微信支付模块完成支付的模式。
  • H5支付:H5支付主要是在手机、ipad等移动设备中通过浏览器来唤起微信支付的支付产品。
  • 小程序支付:小程序支付是专门被定义使用在小程序中的支付产品。目前在小程序中能且只能使用小程序支付的方式来唤起微信支付。

以上不同的微信支付产品对应不同的微信支付系统的 API。其中,线上使用的支付方式:JSAPI 支付,APP 支付,H5 支付。对于在微信公众号内的微信支付,一般使用 JSAPI 支付方式。

常见术语名词解释

  • 微信公众平台:微信公众平台是微信公众账号申请入口和管理后台。商户可以在公众平台提交基本资料、业务资料、财务资料申请开通微信支付功能。
  • 微信开放平台:微信开放平台是商户APP接入微信支付开放接口的申请入口,通过此平台可申请微信APP支付。
  • 微信商户平台:微信商户平台是微信支付相关的商户功能集合,包括参数配置、支付数据查询与统计、在线退款、代金券或立减优惠运营等功能。
  • 微信支付系统:微信支付系统是指完成微信支付流程中涉及的API接口、后台业务处理系统、账务系统、回调通知等系统的总称。
  • 商户证书:商户证书是微信提供的二进制文件,商户系统发起与微信支付后台服务器通信请求的时候,作为微信支付后台识别商户真实身份的凭据。
  • 签名:商户后台和微信支付后台根据相同的密钥和算法生成一个结果,用于校验双方身份合法性。签名的算法由微信支付制定并公开,常用的签名方式有:MD5、SHA1、SHA256、HMAC等。
  • Openid:用户在公众号内的身份标识,不同公众号拥有不同的openid。商户后台系统通过登录授权、支付通知、查询订单等API可获取到用户的openid。主要用途是判断同一个用户,对用户发送客服消息、模版消息等。企业号用户需要使用企业号userid转openid接口与openid互换接口)将企业成员的userid转换成openid。

申请开通微信支付功能

微信公众号商户可以在微信公众平台提交开通微信支付的申请,第三方APP商户可以在微信开放平台提交申请。申请提交后,微信支付工作人员审核资料无误将开通相应的微信支付权限。微信支付申请审核通过后,商户在申请资料填写的邮箱中收取到由微信支付小助手发送的邮件,此邮件包含开发时需要使用的支付账户信息。

  • APPID:appid是微信公众账号或开放平台APP的唯一标识,在公众平台申请公众账号或者在开放平台申请APP账号后,微信会自动分配对应的appid,用于标识该应用。
  • 微信支付商户号 mch_id:商户申请微信支付后,由微信支付分配的商户收款账号。
  • API 密钥:交易过程生成签名的密钥,仅保留在商户系统和微信支付后台,不会在网络中传播。商户妥善保管该Key,切勿在网络中传输,不能在其他客户端中存储,保证key不会被泄漏。
  • Appsecret:AppSecret是APPID对应的接口密码,用于获取接口调用凭证access_token时使用。在微信支付中,先通过OAuth2.0接口获取用户openid,此openid用于微信内网页支付模式下单接口使用。
  • 商户平台登录帐号。
  • 商户平台登录密码。

开通微信支付功能后,会得到额外三个用于开发微信支付的参数:mch_id, API 密钥, Appsecret。这三个参数在调用微信支付系统的 API 时需要用到。

微信 JSAPI 支付开发

微信微信 JSAPI 支付开发前提

  • 申请商户的微信支付账号。商户在微信公众平台或开放平台提交微信支付申请,申请成功后可获取其支付账号的相关参数用于微信支付开发。
  • 设置支付目录。1)商户最后请求拉起微信支付收银台的页面地址我们称之为“支付目录”,例如:https://www.weixin.com/pay.php。2)商户实际的支付目录(支付页面URL)必须和在微信支付商户平台设置的一致。3)如果支付授权目录设置为顶级域名(例如:https://www.weixin.com/ ),那么只校验顶级域名,不校验后缀。
  • 设置授权域名。在统一下单接口中要求必传用户openid,即需要进行网页授权,获取用户信息。

微信支付系统主要提供的 API 和功能

  • 支付下单、查询订单、支付结果通知(微信回调)
  • 申请退款、查询退款、退款通知(微信回调)

详细的 API 列表可参考微信支付官方文档:微信支付 API 列表

微信 JSAPI 支付开发需要的参数

  • 商户号(mch_id)
  • API 密钥(key):交易过程用于生成签名的密钥,仅保留在商户系统和微信支付后台。
  • API 证书(application_cert.pem):微信支付接口中,涉及资金回滚的接口会使用到 API 证书,包括退款、撤销接口。证书文件主要用于退款时发送 SSL 请求。
  • 当然还包括公众平台的参数(appId,appSecret)。

其中公众平台的 appId 与微信支付系统的 mch_id 将公众号应用与微信支付帐号进行了绑定。

微信 JSAPI 支付功能的实现过程

  • 用户发起支付请求。
  • 公众号后端,生成商户自己的订单数据(商户订单号),保存到数据库订单表。
  • 调用微信统一下单API,得到预付单信息(prepay_id)。下单成功后更新数据库订单表。
  • 生成公众号页面调用 JSAPI 支付接口需要的参数并签名(paySign),将参数返回给前端页面。
  • 公众号前端页面,调用 JSAPI 支付接口请求支付。
  • 微信支付系统验证参数的合法性,若参数合法则弹出微信的确认支付的页面。用户输入支付密码,密码验证通过后会跳转到微信的支付成功页面,并同时发起微信支付的回调。
  • 在支付成功页面点击返回商家,跳转到商户指定的页面。商户系统后端调用微信订单查询 API,查询支付结果,生成商户自己的支付结果页面。
  • 公众号后端,处理微信支付的回调。进行业务处理后,返回处理结果给微信支付系统。

详细业务流程可参考微信支付官方文档:JSAPI 支付业务流程时序图

公众号网页调用 JSSDK 支付接口(调起微信支付)

可以使用微信 JSSDK 的支付 API。也可以使用微信支付封装的JS方法。

1)JSSDK 发起支付请求

wx.ready(function(
// auto execution
wx.chooseWXPay({
timestamp: 0, // 支付签名时间戳,注意微信jssdk中的所有使用timestamp字段均为小写。但最新版的支付后台生成签名使用的timeStamp字段名需大写其中的S字符
nonceStr: '', // 支付签名随机串,不长于 32
package: '', // 统一支付接口返回的prepay_id参数值,提交格式如:prepay_id=\*\*\*)
signType: '', // 签名方式,默认为'SHA1',使用新版支付需传入'MD5'
paySign: '', // 支付签名
success: function (res) {
// 支付成功后的回调函数
}
});
));

2)微信支付平台封装的方法

<script>
function onBridgeReady(){
WeixinJSBridge.invoke(
'getBrandWCPayRequest', {
"appId":"wx2421b1c4370ec43b", //公众号名称,由商户传入
"timeStamp":"1395712654", //时间戳,自1970年以来的秒数
"nonceStr":"e61463f8efa94090b1f366cccfbbb444", //随机串
"package":"prepay_id=u802345jgfjsdfgsdg888",
"signType":"MD5", //微信签名方式:
"paySign":"70EA570631E4BB79628FBCA90534C63FF7FADD89" //微信签名
},
function(res){
if(res.err_msg == "get_brand_wcpay_request:ok" ){
// 使用以上方式判断前端返回,微信团队郑重提示:
//res.err_msg将在用户支付成功后返回ok,但并不保证它绝对可靠。
}
});
}

if (typeof WeixinJSBridge == "undefined"){
if( document.addEventListener ){
document.addEventListener('WeixinJSBridgeReady', onBridgeReady, false);
}else if (document.attachEvent){
document.attachEvent('WeixinJSBridgeReady', onBridgeReady);
document.attachEvent('onWeixinJSBridgeReady', onBridgeReady);
}
}else{
onBridgeReady();
}
</script>

注意事项

微信支付需要在微信APP中调试,不能在微信开发者工具中调试。

References

[1] 微信支付V2版API接口文档

[2] 微信支付开发文档3.0

介绍微信公众平台

微信公众平台是微信团队为微信公众号开发者提供的一个基础平台,平台通过为公众号开发者提供一系列的接口,让公众号能够更好的为微信用户提供服务。

微信公众号是微信 APP 中的一个功能,它是公众号运营者为微信用户提供资讯和服务的接入点,微信用户通过订阅公众号和进入公众号,可以看到公众号提供的服务菜单和推送消息。

介绍微信公众号开发

如何开发一个公众号

  • 开发者需要在微信公众平台注册一个公众号开发者账号。公众号分为服务号和订阅号,不同类型的公众号具备不同的公众平台的接口权限。
  • 在公众平台配置自己的域名和服务器,开发者的服务器与微信公众平台进行对接。
  • 开发者在微信公众平台提供的接口的基础上开发自己的服务,服务分为两种:公众号消息会话和公众号内网页。

公众号用户的识别

微信用户在公众号内的请求会携带一个 OpenID 的参数,来标识一个用户。每个用户对每个公众号都会产生一个唯一的 OpenID。如果需要在多个公众号之间做用户共通,需要将这些公众号和应用绑定到一个开发平台账号下,绑定后,一个用户对多个公众号依然有多个不同的 OpenID,但一个用户只有一个唯一的 UnionID。

公众号开发注意事项

  • 微信公众平台开发是指微信公众号进行业务开发,为移动应用、PC端网站、公众号第三方平台的开发,请前往微信开发平台接入。
  • 在申请到认证公众号之前,可以先通过测试号申请系统,快速申请一个接口测试号,立即开始接口测试开发。
  • 在开发过程中,可以使用接口调试工具来在线调试某些接口。
  • 每个接口都有每日接口调用频次限制,可以在公众平台官网-开发者中心处查看具体频次。
  • 在开发出现问题时,可以通过接口调用的返回码,以及报警排查指引(在微信公众平台官网-开发者中心处可以设置接口报警),开发发现和解决问题。
  • 公众平台以 access_token 为接口调用凭据,来调用接口,所有接口的调用需要先获取 access_token,access_token 在2小时内有效,过期需要重新获取,但1天内获取次数有限,开发者需自行存储。
  • 公众平台接口调用仅支持80端口。

公众号服务介绍

公众号消息会话

  1. 群发消息:公众号可以以一定频次(订阅号为每天1次,服务号为每月4次),向用户群发消息,包括文字消息、图文消息、图片、视频和语音等。
  2. 被动回复消息:在用户给公众号发消息后,公众号可以自动回复一个消息。
  3. 客服消息:在用户给公众号发消息后的48小时内,公众号可以给用户发送不限数量的消息,主要用于客服场景。
  4. 模板消息:用于给用户发送服务通知,如刷卡提醒,服务预约成功通知等。公众号可以用特定内容模板主动向用户发送。

公众号内网页

许多复杂的业务场景,需要通过网页形式来提供服务,如商城,水电缴费等。公众号网页可以通过点击公众号发送的消息,或者点击公众号菜单进入网页。网页需要用到的公众平台的功能:网页授权获取用户基本信息,微信JS-SDK。

微信公众号开发接入

微信公众号接入是指:部署自己的公众号应用程序到自己的服务器上,然后在微信公众平台配置自己的服务器信息,进行 token 验证,token 验证成功,说明微信公众平台能够正常地与你的服务器进行通信。

微信公众号开发接入过程如下:

  1. 编写公众号应用程序(含验证 token 的接口),服务器启动程序。
  2. 在微信公众平台网站填写服务器配置。在微信公众平台的“开发-基本配置-服务器配置-修改配置”页面,填写服务器的信息。
  3. 验证 token。点击“基本配置页面”下方的提交按钮,进行 token 验证。若当前页面出现“token验证失败”的提示,则说明配置的信息有误,或者你服务器的代码有误;若token验证成功,则会自动返回基本配置的主页面。
  4. 启用配置,准备开发。token 验证成功后,点击“启用”按钮,启用配置的服务器。然后使用微信公众平台提供的接口实现自己业务逻辑。

公众号接入成功后,可以进行微信公众平台 API 的调用测试。主要过程为:编写程序,调用程序,查看程序返回结果。

公众号开发基础

公众平台接口域名

微信公众平台有多个域名,开发者可以根据自己的服务器部署情况,选择最佳的接入点(延时更低,稳定性更高)。除此之外,开发者可以将其他接入点用作容灾用途,当网络链路发生故障时,可以考虑选择备用接入点来接入。微信公众平台的域名如下:

  1. 通用域名(api.weixin.qq.com),使用该域名将访问官方指定就近的接入点;
  2. 通用异地容灾域名(api2.weixin.qq.com),当上述域名不可访问时可改访问此域名;
  3. 上海域名(sh.api.weixin.qq.com),使用该域名将访问上海的接入点;
  4. 深圳域名(sz.api.weixin.qq.com),使用该域名将访问深圳的接入点;
  5. 香港域名(hk.api.weixin.qq.com),使用该域名将访问香港的接入点。

获取 Access token

调用微信公众平台接口之前,必须获取 access_token,access_token 时公众号的全局唯一调用凭据,公众号调用公众平台的接口时都需要使用 access_token 作为参数。

access_token 的存储至少需要保留 512 个字符空间,它的有效期为 2 个小时,需定时刷新,重复获取将导致上次获取的 access_token 失效。

access_token 使用说明:

  • 建议公众号开发者使用中控服务器统一获取和刷新access_token,其他业务逻辑服务器所使用的access_token均来自于该中控服务器,不应该各自去刷新,否则容易造成冲突,导致access_token覆盖而影响业务;
  • 目前access_token的有效期通过返回的expire_in来传达,目前是7200秒之内的值。中控服务器需要根据这个有效时间提前去刷新新access_token。在刷新过程中,中控服务器可对外继续输出的老access_token,此时公众平台后台会保证在5分钟内,新老access_token都可用,这保证了第三方业务的平滑过渡;

接口调用请求:

https请求方式: GET 
URL: https://api.weixin.qq.com/cgi-bin/token?grant_type=client_credential&appid=APPID&secret=APPSECRET

返回说明:

成功返回信息如下

{"access_token":"ACCESS_TOKEN","expires_in":7200}

IP 白名单

在 IP 白名单内的IP来源,获取 access_token 接口才可调用成功。

接口测试号申请

微信公众平台的某些高级接口的调用权限需要微信认证后才可以获取。为了帮助开发者快速了解和上手微信公众号开发,熟悉各个接口的调用,可以申请微信公众帐号测试号。

微信公众号测试号和你的正式的公众号不是同一个公众号,它们有不同的配置信息,如 appID, appsecret。微信公众号测试号相当于一个虚拟的公众号,可以让你调用很多接口,当你通过测试号实现了你的业务,最后,可以将你的配置参数改为正式的公众号。

测试号不需要设置 IP 白名单,使用测试号的配置信息去调用微信公众平台的接口是没有 IP 限制的。

链接地址:微信公众号接口测试帐号申请

微信公众平台接口在线调试工具

微信公众平台接口在线调试工具,是为了帮助开发者检测调用公众平台 API 时发送的请求参数是否正确。在“在线调试工具”的页面提交相关信息后,提交请求,可获得公众平台服务器的返回结果。

在线调试工具的主要功能是帮你构造参数,其实其它任何地方也可以调用微信公众平台的接口。

链接地址:微信公众平台接口调试工具

Web开发者工具

微信公众平台为开发者提供“web开发者工具”,用于帮助开发基于微信的网页或者webapp。它是一个桌面应用,通过模拟微信客户端的表现使得开发者可以使用这个工具方便地在 PC 上进行开发和调试。

下载地址:web开发者工具

微信公众平台的接口调用和微信公众号内的网页请求

微信公众号的网页,需要在微信公众号内或者在微信web开发者工具中打开。

微信公众平台的接口调用可以在任何支持 HTTP 请求的地方请求并得到返回结果。但“获取access_token 接口”的请求的客户端 IP 需要 IP 白名单中,才能成功返回。然而,调用其它公众平台接口需要 access_token 作为参数,近似相当于没有在 IP 白名单内的客户端请求无法调用微信公众平台接口。不在 IP白名单的客户端的请求结果如下:

https://api.weixin.qq.com/cgi-bin/token?grant_type=client_credential&appid=xxx&secret=xxx
{"errcode":40164,"errmsg":"invalid ip 58.213.199.30 ipv6 ::ffff:58.213.199.30, not in whitelist hint: [Zhnd8cQNe-4P2DPA]"}

微信公众平台的接口和功能

微信公众平台提供的接口和功能如下:

  • 自定义菜单
  • 消息管理
  • 微信网页开发
  • 素材管理
  • 图文消息留言管理
  • 用户管理
  • 帐号管理
  • 数据统计
  • 微信卡券
  • 微信门店
  • 智能接口
  • 微信设备功能
  • 新版客服功能
  • 对话能力(原导购助手)
  • 微信“一物一码”
  • 微信发票

我们可以简单地将微信公众平台的API接口分为三类:

  • 基础接口(菜单,用户,素材等),为消息服务和网页服务提供基础。
  • 消息服务接口。
  • 网页服务接口。

以上功能的具体的使用步骤和接口调用说明,请参考 微信公众号开发指南

微信网页服务开发

网页授权

网页授权是指:在用户访问公众号网页时获取该微信用户的信息需要先跳转到用户授权页面进行授权。

微信公众号应用需要通过微信用户的 openId 来标识用户,我们需要通过网页授权来获取用户信息,根据用户的 openId 知道是哪个用户访问了公众号网页,然后进行相应的业务处理。

网页授权的过程为:公众号页面重定向跳转到微信用户授权页面,微信回调你的服务器,你获得 code 参数,利用 code 参数获取 JSAPI access_token(与基础 API 的 access_token 不同),利用 access_token 去调用微信 API 获取用户基本信息。

网页授权的实现过程:

  • 在后端定义一个 Filter,拦截所有需要进行网页授权的请求(特定的 URI 前缀),检查请求的 session 中是否存在 openId。如果 openId 不存在,则需要进行网页授权,将未授权的请求重定向到微信授权页面;如果 openId 存在,则不进行任何处理。
  • 微信回调后,你获取到了 code 参数,然后利用 code 参数调用微信 API 获取 access_token 参数,然后利用 access_token 调用微信 API 获取用户基本信息。需要调用两次微信 API。
  • 将微信用户基本保存到 session 中。

JS-SDK使用说明

JS-SDK 功能的页面参考样式:https://www.weixinsxy.com/jssdk/

JS-SDK 接口主要的功能:监听分享事件,暂存音频和视频,调用微信APP的功能(如微信扫一扫、内置地图),微信支付等。

访问测试公众号的网页的条件:

  • 需要在微信开发者工具或者微信APP中访问公众号网页。
  • 需要扫码关注了测试公众号的用户才能访问该公众号网页。
  • 微信公众平台中配置”JS接口安全域名”(公众号调用 JSSDK 的域名),需要备案的域名,而不能是 IP。
  • 微信公众平台中配置“接口权限列表–网页授权获取用户基本信息–授权回调页面域名”(用户授权后回调公众号的域名),需要备案的域名才行。
  • “JS接口安全域名”与“授权回调页面域名”,需要保持一致。
  • (Note: 微信公众平台中配置“Token验证” 的 URL是用于“token 验证”(消息通信是否正常)和“微信公众号的消息服务开发”(消息服务通信的 URL 与 token 验证 的 URL 一致,但消息服务是 HTTP POST 请求),token 验证 URL 与公众号网页服务开发没有关系。消息服务的通信 URL 可以是 IP,可以与网页服务的两个域名不一致)

使用微信 JS-SDK 接口

调用微信 JS-SDK 的 API 接口需要开发者应用提供相关配置和签名参数给 JS-SDK 的配置接口。主要是4个参数:appId,timestamp,nonceStr 和 signature。

获取签名的实现过程:通过普通的 access_token 调用微信 API 获取 jsapi_ticket 参数,生成一个随机字符串(可使用UUID),根据签名算法,将需要4个参数转换为1个签名参数。

JS-SDK 的使用如下:

1)进行权限验证

wx.config({
debug: true, // 开启调试模式,调用的所有api的返回值会在客户端alert出来,若要查看传入的参数,可以在pc端打开,参数信息会通过log打出,仅在pc端时才会打印。
appId: '${wxJsapiConfig.appId}', // 必填,公众号的唯一标识
timestamp: '${wxJsapiConfig.timestamp}', // 必填,生成签名的时间戳
nonceStr: '${wxJsapiConfig.nonceStr}', // 必填,生成签名的随机串
signature: '${wxJsapiConfig.signature}',// 必填,签名
jsApiList: [
'updateAppMessageShareData',
'updateTimelineShareData'
] // 必填,需要使用的JS接口列表
});

2)权限验证成功后,会执行JS-SDK 的 wx.ready() 方法:

wx.ready(function(){
//自定义“分享给朋友”及“分享到QQ”按钮的分享内容
// Auto running
wx.updateAppMessageShareData({
title: 'wxPage-index', // 分享标题
desc: 'Test share to friend.', // 分享描述
link: '${wxJsapiConfig.url}', // 分享链接,该链接域名或路径必须与当前页面对应的公众号JS安全域名一致
imgUrl: '', // 分享图标
success: function () {
// 设置成功
console.log("Set share to friend info successfully!");
}
})

// 拍照或从手机相册中选图接口
// onclick running
document.querySelector('#chooseImage').onclick = function () {
wx.chooseImage({
count: 9, // 默认9
sizeType: ['original', 'compressed'], // 可以指定是原图还是压缩图,默认二者都有
sourceType: ['album', 'camera'], // 可以指定来源是相册还是相机,默认二者都有
success: function (res) {
localIds = res.localIds; // 返回选定照片的本地ID列表,localId可以作为img标签的src属性显示图片
console.log("chooseImage -- localIds first: " + localIds[0]);
}
});
};
});

3)权限验证失败后,会执行 JS-SDK 的 wx.error() 方法。

wx.error(function (res) {
alert(res.errMsg);
});

更多内容请参考官方说明:

微信公众号开发接入示例

下面以 Java 为例,基于 Spring Boot 框架进行示例展示。

1.创建和配置项目

创建一个 Maven 项目,项目名为 wechat-official-accounts

mvn archetype:generate -DgroupId=com.taogen.example -DartifactId=wechat-official-accounts -DarchetypeArtifactId=maven-archetype-webapp -DinteractiveMode=false

配置 pom.xml 文件

<project ...>  
...

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
<!-- custom properties -->
<project.java.version>1.8</project.java.version>
<junit.version>4.12</junit.version>
<log4j.version>2.8.2</log4j.version>
</properties>

<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.6.RELEASE</version>
</parent>

<dependencies>
<!-- start -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- log4j2 -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<!-- spring web -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- unit test -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<sourceDirectory>src/main/java</sourceDirectory>
<testSourceDirectory>src/test/java</testSourceDirectory>
<plugins>
<!-- Package as an executable jar/war. $ mvn package spring-boot:repackage -->
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- maven compile -->
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>${project.java.version}</source>
<target>${project.java.version}</target>
</configuration>
</plugin>
</plugins>
</build>
</project>

配置 log4j2 日志

src/main/resources/log4j2.xml

<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d [%t] %-5level %logger{36} - %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Logger name="com.taogen.example" level="debug" additivity="false">
<AppenderRef ref="Console"/>
</Logger>
<Root level="error">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>

配置 Spring Boot 参数和业务参数

src/main/resources/application.yml

wechat:
token: "my_token"
appId: "..."
appSecret: "..."
server:
port: 80

2.编写验证 token 的 Java 代码

src/main/java/com/taogen/example/App.java

@SpringBootApplication
@RestController
public class App {

private static final Logger logger = LogManager.getLogger();
@Value("${wechat.token}")
private String token;
@Value("${wechat.appId}")
private String appId;
@Value("${wechat.appSecret}")
private String appSecret;

public static void main(String[] args) {
SpringApplication.run(App.class, args);
}

@GetMapping("/hello")
public String hello() {
logger.debug("Access /hello");
return "hello " + new Date();
}

/**
* Validate Token
*
* @param signature 微信加密签名,signature结合了开发者填写的token参数和请求中的timestamp参数、nonce参数。
* @param timestamp 时间戳
* @param nonce 随机数
* @param echostr 随机字符串
* @return 若确认此次GET请求来自微信服务器,请原样返回echostr参数内容
*/
@GetMapping("/wx")
public String validateToken(@RequestParam("signature") String signature,
@RequestParam("timestamp") String timestamp,
@RequestParam("nonce") String nonce,
@RequestParam("echostr") String echostr) {
logger.debug("signature: {}, timestamp: {}, nonce: {}, echostr: {}",
signature, timestamp, nonce, echostr);
// 开发者通过检验signature对请求进行校验(下面有校验方式)。
// 1)将token、timestamp、nonce三个参数进行字典序排序
// 2)将三个参数字符串拼接成一个字符串进行sha1加密
// 3)开发者获得加密后的字符串可与signature对比,标识该请求来源于微信
String sortedParams = getSortedParams(timestamp, nonce);
String encryptedParams = getEncryptedParams(sortedParams);
if (encryptedParams.equals(signature)) {
return echostr;
} else {
return "error";
}
}

private String getSortedParams(String timestamp, String nonce) {
List<String> params = new ArrayList<>();
params.add(this.token);
params.add(timestamp);
params.add(nonce);
Collections.sort(params, Comparator.naturalOrder());
StringBuilder validateString = new StringBuilder();
for (String param : params) {
validateString.append(param);
}
return validateString.toString();
}

private String getEncryptedParams(String sortedParams) {
MessageDigest crypt = null;
try {
crypt = MessageDigest.getInstance("SHA-1");
crypt.reset();
crypt.update(sortedParams.toString().getBytes("UTF-8"));
} catch (NoSuchAlgorithmException | UnsupportedEncodingException e) {
e.printStackTrace();
}

return new BigInteger(1, crypt.digest()).toString(16);
}
}

3.打包项目,部署到服务器,运行项目

打包项目:

mvn package spring-boot:repackage

将项目打包文件上传到服务器,运行以下命令启动项目:

java -jar <project_name>.jar

4.在微信公众平台网站进行 token 验证

  • 访问微信公众平台,注册帐号或扫码登录。

  • 登录后,在左侧的菜单栏选择 “开发” - “基本配置”,点击进入基本配置页面。

  • 进入基本配置页面,在“服务器配置”右侧点击修改配置。填写以下配置,主要是 URL 和 你项目中自定义的 token:

    URL: http://<your_server_ip>/wx
    Token: my_token (项目中自主定义的 token)
    EncodingAESKey: (默认)
    消息加密方式:(默认)
  • 点击修改配置页面的“提交”按钮,若 token 验证成功会自动跳转到基本配置页面;若验证失败会在当前页面弹出错误提示。

Summary

微信基础 API 接口

前提要求

  • 客户端请求的 IP 需要在公众平台的 IP 白名单中。但测试公众号不需要。

不满足前提的错误结果

  • 调用微信 API 接口返回的错误提示:

    {"errcode":40164,"errmsg":"invalid ip <your_ip> ipv6 ::ffff:<your_ip>, not in whitelist hint: [jhCDbOwFE-C.kTAa] rid: 5f2baef3-06a383d9-48b6ee02"}

功能列表

  • 获取 access_token
  • 自定义菜单 (创建、删除、查询等)
  • 素材管理(临时素材、永久素材、图文素材的创建、删除和获取等)
  • 图文消息留言管理
  • 用户管理(用户标签、用户备注名、用户基本信息、用户地理位置等)
  • 帐号管理
  • 数据统计
  • 微信卡券

如何对接

  • 获取 access_token,调用微信 API。

微信消息服务

前提要求

  • token 验证通过。

不满足前提的错误结果

  • 无法收到微信服务器的请求

功能列表

  • 自定义菜单 -事件推送
  • 消息管理 (接收普通消息、接收事件推送、回复消息)
  • 消息管理(群发消息、模板消息、一次性订阅消息)

如何对接

  • 接收来自微信服务器的 token 验证请求,进行 token 验证。
  • 接收来自微信服务器的消息,解析 XML 获取消息内容和用户信息,回复 XML 格式的消息。
  • 获取 access_token,调用微信 API,群发消息给多个用户。

微信网页服务

前提要求

  • 公众号的网页需要在微信开发者工具或者微信APP中访问。
  • 公众号网页的 URL 的域名,JS 安全域名和用户授权回调域名需要一致,且域名已经备案。

不满足前提的错误结果

  • 网页出现下面的提示:

    Oops! Something went wrong:(

功能列表

  • 网页授权
  • 网页开发样式 WeUI
  • JS-SDK (基础接口,分享接口,图像接口,音频接口,智能接口,地理位置)
  • JS-SDK(微信支付)
  • 微信开放标签

如何对接

  • 在网页获取微信用户信息。公众号页面重定向跳转到微信用户授权页面,微信回调你的服务器,获得 code 参数,利用 code 参数获取 JSAPI access_token(与基础 API 的 access_token 不同),利用 access_token 去调用微信 API 获取用户基本信息。
  • 在网页调用微信 JS-SDK 的 API。调用微信 API 获取 jsapi_ticket,根据 jsapi_ticket 构造得到签名,将签名和相关参数返回给前端,前端利用签名和相关参数调用 JS-SDK API。

常见问题

Question: 公众号页面重定到微信用户授权页面(获取 code )时出现错误

微信授权页面错误提示为:Oops! Something went wrong:(

Solution:

确保在微信开发者工具或者微信APP 中打开微信公众号的页面链接,而不是在浏览器打开。

检查配置。微信公众平台的接口权限列表中的“网页授权获取用户信息”行,修改“授权回调页面域名”(不需要加 http:// or https://)

Question: JSAPI 参数错误,无效的签名。

错误提示为:{errMsg: "config:fail,Error: 系统错误,错误码:63002,invalid signature [20200804 16:04:22][]"}

Solution:

检查需要的4个参数(noncestr, jsapi_ticket, timestamp, url)是否不为空,是否正确。

检查签名算法是否正确。

检查访问的页面的 URL 要与传递的 URL 一致。注意传递的 url 是 https://xxx, 访问页面也要是 https://xxx,而不能是 http://xxx

Question: JS-SDK debug 模式没有弹出提示

Solution:

检查 JSSDK 的 js 引用是否正确。如,<script src="https://res.wx.qq.com/open/js/jweixin-1.6.0.js"></script>

查看微信开发者工具的 console,JavaScript 代码是否有语法错误。

Appendixes

开发者规范

接口权限说明

全局返回码说明

References

[1] 微信公众号开发指南 - DOC

[2] 微信公众平台

This post we are going to talk about NIO and NIO.2 in Java. The NIO was introduced in Java 5. The NIO is missing several features, which were subsequently provided by NIO.2.

Introduction

Input and output (I/O) facilities are fundamental parts of operating systems along with computer languages and their libraries.

Java initial suite of I/O APIs and related architecture are known as classic I/O. Because modern operating systems feature newer I/O paradigms, which class I/O doesn’t support, new I/O (NIO) was introduced as part of JDK 1.4 to support them. Because lack of time, Some other NIO features being deferred to JDK 5 and JDK 7.

Classic I/O

JDK 1.0 introduced I/O facilities for accessing the file system, accessing file content randomly, and streaming byte-oriented data between sources and destinations in a sequential manner.

NIO

Modern operating systems offer sophisticated I/O services (such as readiness selection) for improving I/O performance and simplifying I/O. Java Specification Request (JSR) 51 was created to address these capabilities.

JSR 51’s description indicates that it provides APIs for scalable I/O, fast buffered binary and character I/O, regular expressions, and charset conversion. Collectively, these APIs are known as NIO. JDK 1.4 implemented NIO in terms of the following APIs:

  • Buffers
  • Channels
  • Selectors
  • Regular expressions
  • Charsets

Buffers

Buffers are the foundation for NIO operations. Essentially, NIO is all about moving data into and out of buffers.

The process of a classic read operation like the following steps. The operating system issues a command to the disk controller to read a block of bytes from a disk into operating system buffer. Once this operation complete, the operating system copies the buffer contents to the buffer specified by the process when it issued a read() operation.

These are some inefficiency in classic I/O. Copying bytes from the operating system buffer to the process buffer isn’t very efficient. It would be more performance to have the DMA controller copy directly to the process buffer, but there are two problems with this approach:

  • The DMA controller typically cannot communicate directly with the user space in which the JVM process runs.
  • Block-oriented device such as a DMA controller work with fixed-size data blocks.

Because of these problems, the operating system acts as an intermediary, tearing apart and recombining data as if switches between the JVM process and the DMA controller.

The data assembly/disassembly tasks can be made more efficient by letting the JVM process pass a list of buffer addresses to the operating system in a single call. The operating system then fills or drains these buffers in sequence, scattering data to multiple buffers during a reading or gathering data from several buffers during a write operation. This scatter/gather activity reduces the number of system calls.

JDK 1.4’s java.nio.Buffer class abstracts the concept of a JVM process buffer.

Channels

Forcing a CPU to perform I/O tasks and wait for I/O completions is wasteful of this resource. Performance can be improved by offloading these tasks to DMA controllers so that the processor can get on with other work.

A channel serves as conduit for communicating (via the operating system) with a DMA controller to efficiently drain byte buffers to or fill byte buffers form a disk. JDK 1.4’s java.nio.channels.Channel interface, its subinterfaces, and various classes implement the channel architecture.

Selectors

I/O is classified as block-oriented or stream-oriented. Reading from or writing to a file is an example of block-oriented I/O. In contrast, reading from the keyboard or writing to a network connection is an example of stream-oriented I/O.

Stream I/O is often slower than block I/O. Many operating systems allow streams to be configured to operate in nonblocking mode in which a thread continually checks available input without blocking when no input is available. The thread can handle incoming data or perform other tasks until data arrives.

This “polling for available input” activity can be wasteful, especially when the thread needs to monitor many input streams. Modern operating systems can perform this checking efficiently, which is known as readiness selection, and which is often built on top of nonblocking mode. The operating system monitors a collection of streams and returns an indication to the thread of which streams are ready to perform I/O. As a result, a single thread can multiplex many active streams via common code and makes it possible.

JDK 1.4 supports readiness selection by providing selectors, which are instances of the java.nio.channels.Selector class that can examine one or more channels and determine which channels are ready for reading or writing.

NIO.2

JSR 51 specifies that NIO would introduce an improved file system interface, and to support asynchronous I/O and complete socket channel functionality. However, lack of time prevented these features from being included. JSR 203 was subsequently created to address these features, which introduced in JDK 7.

Improved File System Interface

The legacy File class suffers from various problems. For example, the renameTo() method doesn’t work consistently across operating systems. The new file system interface fixes these and other problems.

Asynchronous I/O

Nonblocking mode improves performance by preventing a thread that performs a read or write operation on a channel from blocking until input is available or output has been fully written. However, it doesn’t let an application determine if it can perform an operation without actually performing the operation. Nonblocking mode can’t separate code that checks for stream readiness form the data-processing code without making your code significantly complicated.

Asynchronous I/O overcomes this problem by letting the thread initiate the operation and immediately proceed to other work. The thread specifies some kind of callback function that is invoked when the operation finishes.

Completion of Socket Channel Functionality

JDK 1.4 added the DatagramChannel, ServerSocketChannel, and SocketChannel classes to the java.nio.channels package. However, lack of time prevented these classes form supporting binding and option configuration. Also, channel-based multicast datagrams were not supported. JDK 7 added binding support and option configuration to the aforementioned classes. Also, it introduced a new java.nio.channels.MulticastChannel interface.

Buffers

NIO is based on buffers, whose contents are sent to or received from I/O services via channels. This section introduces to NIO’s buffer classes.

Buffer Classes

A buffer is an object that stores a fixed amount of data to be sent to or received from an I/O service. It sits between an application and a channel that writes the buffered data to the service or reads the data from the service and deposits it into the buffer.

Buffers are not safe for use by multiple concurrent threads. If a buffer is to be used by more than one thread then access to the buffer should be controlled by appropriate synchronization.

Buffers have four basic properties:

  • Capacity. It is the number of elements it contains.
  • Limit. It is the index of the first element that should not be read or written.
  • Position. It is the index of the next element to be read or written. A buffer’s position is never negative and is never greater than its limit.
  • Mark. It’s the index to which its position will be reset when the reset method is invoked.
0 <= mark <= position <= limit <= capacity

The Hierarchy of Buffer Classes

A-java.nio.Buffer
|----A-ByteBuffer
|----A-CharBuffer
|----A-DoubleBuffer
|----A-FloatBuffer
|----A-IntBuffer
|----A-LongBuffer
|----A-ShortBuffer

Methods of Buffer

  • Abstract methods
    • abstract Object array()
    • abstract int arrayOffset()
    • abstract boolean hasArray()
    • abstract boolean isDirect()
    • abstract boolean isReadOnly()
  • Get Buffer’s Property
    • int capacity()
    • int limit()
    • int position()
    • boolean hasRemaining(). Tells whether there are any elements between the current position and the limit.
    • int remaining(). Returns the number of elements between the current position and the limit.
  • Set Buffer’s Property
    • Buffer limit(int newLimit). Sets this buffer’s limit.
    • Buffer position(int newPosition). Sets this buffer’s position.
    • Buffer mark(). Sets this buffer’s mark at its position.
    • Buffer reset(). Resets this buffer’s position to the previously-marked position.
  • Clearing, flipping, and rewinding
    • Buffer clear(). Clears this buffer. It makes a buffer ready for a new sequence of channel-read or relative put operations. The position is set to zero, the limit is set to the capacity, and the mark is discard. Invoke this method before using a sequence of channel-read or put operation to fill this buffer. For example, buf.clear(); in.read(buf);
    • Buffer flip(). Flips this buffer. It makes a buffer ready for a new sequence of channel-write or relative get operations. The limit is set to the current position, the position is set to zero. If the mark is defined then it is discarded. After a sequence of channel-read or put operations, invoke this method to prepare for a sequence of channel-write or relative get operations. For example, buf.put(magic); in.read(buf); buf.flip(); out.write(buf);. This method is often used in conjunction with the compact method when transferring data from one place to another.
    • Buffer rewind(). Rewinds this buffer. It makes a buffer ready for re-reading the data that it already contains. The position is set to zero and the mark is discarded. Invoke this method before a sequence of channel-write or get operations, assuming that the limit has already bean set appropriately. For example, out.write(buf); buf.rewind(); buf.get(array);

Buffers in Depth

Buffer Creation

ByteBuffer and the other primitive-type buffer classes declare various class methods for creating a buffer of that type.

Methods for creating ByteBuffer instances:

  • ByteBuffer allocate(int capacity)
  • ByteBuffer allocateDirect(int capacity)
  • ByteBuffer wrap(byte[] array)
  • ByteBuffer wrap(byte[] array, int offset, int length)

These methods show two ways to create a byte buffer: create the ByteBuffer object and allocate an internal array that stores capacity bytes or create the ByteBuffer object and use the specified array to store these bytes. For example:

ByteBuffer buffer = ByteBuffer.allocate(10);
byte[] bytes = new byte[200];
ByteBuffer buffer2 = ByteBuffer.wrap(bytes);

View Buffers

Buffers can manage data stored in other buffers. View buffers are create a buffer that manages another buffer’s data. View buffers share the same internal array. Each buffer has its own position, limit and mark.

ByteBuffer buffer = ByteBuffer.allocate(10);
ByteBuffer bufferView = buffer.duplicate();

Read-only view buffers

ByteBuffer buffer = ByteBuffer.allocate(10);
ByteBuffer bufferView = buffer.asReadOnlyBuffer();

Buffer Writing and Reading

Methods for ByteBuffer’s writing and reading

  • ByteBuffer put(byte b)
  • ByteBuffer put(int index, byte b)
  • Byte get()
  • Byte get(int index)

Flipping Buffers

buffer.flip() equals buffer.limit(buffer.position()).position(0). A example flipping buffer:

buffer.put(s);
buffer.flip();
while (buffer.hasRemaining()){
System.out.print(buffer.get());
}
buffer.clear();

Compact Buffers

The compact() method moves unwritten buffer data to the beginning of the buffer so that the next read() method call appends read data to the buffer’s data instead of overwriting that data. Do this in case of a partial write.

buf.clear();
while (in.read(buf) != -1){
buf.flip();
out.write(buf);
buf.compact();
}

Byte Ordering

Nonbyte primitive types except for Boolean are composed of several bytes. Each value of one of these multibyte types is stored in a sequence of contiguous memory locations. However, the order of these bytes can differ from operating system to operating system.

For example, consider 32-bit long integer 0x10203040. This value’s four bytes could be stored in memory (from low address to high address) as 10, 20, 30, 40; this arrangement is known as big endian order (the most-significant byte, the “big” end, is stored at the lowest address). Alternatively, these bytes could be stored as 40, 30, 20, 10; this arrangement is known as little endian order (the least-significant byte, the “little” end, is stored at the lowest address).

Java provides the java.nio.ByteOrder class to help you deal with byte-order issues when writing/reading multibyte value to/from a multibyte buffer. ByteOrder declares a ByteOrder nativeOrder() method that returns the operating system’s byte order as a ByteOrder instance. This instance is one of ByteOrder’s BIG_ENDIAN and LITTLE_ENDIAN constants.

  • static ByteOrder BIG_ENDIAN
  • static ByteOrder LITTLE_ENDIAN
  • ByteOrder nativeOrder()

ByteBuffer its default byte order is always big endian, even when the underlying operating system’s byte order is little endian. Because Java’s default byte order is also big endian, which lets classfiles and serialized objects store data consistently across Java virtual machines.

Big endian default can impact performance on little endian operating systems, ByteBuffer also declares a ByteBuffer order(ByteOrder bo) method to change the byte buffer’s byte order.

Although it may seem unusual to change the byte order of a byte buffer, this method is useful because ByteBuffer also declares several convenience methods for writing and reading multibyte values, such as ByteBuffer putInt(int value) and int getInt(). These convenience methods write these values according to the byte buffer’s current byte order.

Direct Byte Buffers

Operating systems can directly access the address space of a process. For example, an operating system could directly access a JVM process’s address space to perform a data transfer operation based on a byte array. However, a JVM might not store the array of bytes contiguously or its garbage collector might move the byte array to another location. Because of these limitations, direct byte buffer were created.

A direct byte buffer is a byte buffer that interacts with channels and native code to perform I/O. The direct byte buffer attempts to store byte elements in a memory area that a channel uses to perform direct (raw) access via native code that tells the operating system to drain or fill the memory area directly.

Direct byte buffers are the most efficient means performing I/O on the JVM. Although you can also pass non-direct byte buffers to channels, a performance problem might arise because non-direct byte buffers are not always to serve as the target of native I/O operations.

Although direct byte buffers are optimal for I/O, a direct byte buffer can be expensive to create because memory extraneous to the JVM’s heap will need to be allocated by the operating system, and setting up/tearing down this memory might take longer than when the buffer way located within the heap.

After your code is working and should you want to experiment with performance optimization, you can easily obtain a direct byte buffer by invoking ByteBuffer’s allocateDirect() method.

Channels

Channels partner with buffers to achieve high-performance I/O.

A Channel is an object that represents an open connection to a hardware device, a file, a network socket, an application component, or another entity that’s capable of performing writes, reads, and other I/O operations. Channels efficiently transfer data between byte buffers and operating system-based I/O service sources or destinations.

Channel classes

The Hierarchy of Channel Classes

I-Channel
|----I-ReadableByteChannel
|--------I-ScatteringByteChannel
|----I-WritableByteChannel
|--------I-GatheringByteChannel
|--------I-ByteChannel
|------------I-SeekableByteChannel
|----I-InterruptibleChannel
|--------A-AbstractInterruptibleChannel
|------------A-FileChannel
|------------A-SelectableChannel
|----------------A-AbstractSelectableChannel
|--------------------A-DatagramChannel
|--------------------A-ServerSocketChannel
|--------------------A-SocketChannel
|--------------------A-Pipe.SinkChannel
|--------------------A-Pipe.SourceChannel
|----I-AsynchronousChannel
|--------I-AsynchronousByteChannel
|--------A-AsynchronousFileChannel
|--------A-AsynchronousServerSocketChannel
|--------A-AsynchronousSocketChannel
|----I-NetworkChannel
|--------I-MulticastChannel

All channels are instances of classes that ultimately implement the java.nio.channels.Channel interface. The Chennel declares the following methods:

  • void close()
  • boolean isOpen()

To support I/O, channel is extended by the WritableByteChannel and ReadableByteChannel interface.

  • WritableByteChannel declares an abstract int write(ByteBuffer buffer) method that writes a sequence of bytes from buffer to the current channel.
  • ReadableByteChannel declares an abstract int read(ByteBuffer buffer) method that reads bytes from current channel into buffer.

A channel whose class implements only WritableByteChannel or ReadableByteChannel is unidirectional.

InterruptibleChannel interface describes a channel that can be asynchronous closed and interrupted.

NIO’s designers chose to shut down a channel when a blocked thread is interrupted because they couldn’t find a way to reliably handle interrupted I/O operations in the same manner across operating systems. The only way to guarantee deterministic behavior was to shut down the channel.

Obtain a channel

There are two ways to obtain a channel:

  • The java.nio.channels package provides a Channels utility class that offers two methods for obtaining channels from streams.
    • WritableByteChannel newChannel(OutputStream outputStream)
    • ReadableByteChannel newChannel(InputStream inputStream)
  • Various classic I/O classes have been retrofitted to support channel creation.
    • java.io.RandomAccessFile‘s FileChannel getChannel() method.
    • java.net.Socket‘s SocketChannel getChannel() method.

For example, obtain channels from standard I/O streams:

ReadableByteChannel src = Channels.newChannel(System.in);
WritableByteChannel dest = Channels.newChannel(System.out);
ByteBuffer buffer = ByteBuffer.allocateDirect(2048);
while (src.read(buffer) != -1){
buffer.flip();
dest.write(buffer);
buffer.compact();
}
buffer.flip();
while (buffer.hasRemaining()){
dest.write(buffer);
}

Channels in Depth

Scatter/Gather I/O

Channels provide the ability to perform a single I/O operation across multiple buffers. This capability is known as scatter/gather I/O (and is also known as vectored I/O).

In the context of a write operation, the contents of several buffers are gathered in sequence and then sent through the channel to a destination. In the context of a read operation, the contents of a channel are scattered to multiple buffers in sequence.

Modern operating systems provide APIs that support vectored I/O to eliminate (or at least reduce) system calls or buffer copies, and hence improve performance.

Java Provides the java.nio.channels.ScatteringByteChannel interface to support scattering and GatheringByteChannel interface to support gathering.

ScattheringByteChannel offers the following methods:

  • long read(ByteBuffer[] buffers, int offset, int length)
  • long read(ByteBuffer[] buffers)

GatheringByteChannel offers the following methods:

  • long write(ByteBuffer[] buffers, int offset, int length)
  • long write(ByteBuffer[] buffers)

File Channels

RandomAccessFile, FileInputStream, and FileOutputStream provide getChannel() method for returning a file channel instance, which describes an open connection to a file.

The abstract java.nio.channels.FileChannel class describes a file channel. This class implements the InterruptibleChannel, ByteChannel, GatheringByteChannel, and ScatteringByteChannel interfaces.

Unlike buffers, which are not thread-safe, file channels are thread-safe.

A file channel maintains a current position into the file, which FileChannel lets you obtain and change.

Methods of FileChannel:

  • void force(boolean metadata)
  • long position()
  • FileChannel position(long newPosition)
  • int read(ByteBuffer buffer)
  • int read(ByteBuffer dst, long position)
  • long size()
  • FileChannel truncate(long size)
  • int write(ByteBuffer buffer)
  • int write(ByteBuffer src, long position)

FileChannel objects support the concept of a current file position, which determines the location where the next data item will be read from or written to.

The following code is FileChannel usage example:

RandomAccessFile raf = new RandomAccessFile("temp.txt", "rw");
FileChannel fc = raf.getChannel();
long pos = fc.position();
System.out.println("Position: " + pos);
System.out.println("Size: " + fc.size());
String msg = "This is a test message.";
ByteBuffer buffer = ByteBuffer.allocateDirect(msg.length() * 2);
buffer.asCharBuffer().put(msg);
fc.write(buffer);
fc.force(true);
System.out.println("Position: " + fc.position());
System.out.println("Size: " + fc.size());
buffer.clear();
fc.position(pos);
fc.read(buffer);
buffer.flip();
while (buffer.hasRemaining()){
System.out.print(buffer.getChar());
}

Locking Files

The ability to lock all or part of a file was an important but missing feature from Java until Java 1.4 arrived. This capability lets a JVM process prevent other processes form accessing all or part of a file until it’s finished with the entire file or part of the file.

Although an entire file can be locked, it’s often desirable to lock a smaller region. For example, a database management system might lock individual table rows.

Locks that are associated with files are known as file locks. Each file lock starts at a certain byte position in the file and has a specific length (in bytes) from this position.

There are two kinds of file locks: exclusive and shared.

There are some important for file locking:

  • When an operating system doesn’t support shared locks, a shared lock request is quietly promoted to a request for an exclusive lock.
  • Locks are applied on a per-file basis. The are not applied on a per-thread or per-channel basis.

FileChannel declares four methods for obtaining exclusive and shared locks:

  • FileLock lock()
  • FileLock lock(long position, long size, boolean shared)
  • FileLock tryLock()
  • FileLock tryLock(long position, long size, boolean shared)

A FileLock instance is associated with a FileChannel instance but the file lock represented by the FileLock instance associates with the underlying file and not with the file channel. Without care, you can run into conflicts (and possibly even a deadlock) when you don’t release a file lock after you’re finished using it. The following code shows the FileLock usage:

FileLock lock = FileChannel.lock();
try {
// interact with the file channel
} catch (IOException ioe){
// handle the exception
} finally {
lock.release();
}

Mapping Files into Memory

FileChannel declares a map() method that lets you create a virtual memory mapping between a region of an open file and a java.nio.MappedByteBuffer instance that wraps itself around this region. This mapping mechanism offers an efficient way to access a file because no time-consuming system calls are needed to perform I/O. The map method is:

MappedByteBuffer map(FileChannel.MapMode mode, long position, long size)

FileChannel.MapMode enumerated type:

  • READ_ONLY
  • READ_WRITE
  • PRIVATE

Changes made to the resulting buffer will eventually be propagated to the file. They might not be made visible to other programs that have mapped the same file.

The specified mapping mode is constrained by the invoking FileChannel object’s access permissions. For example, if the file channel was opened as a read-only channel, and if you request READ_WRITE mode, map() will throw NonWritableChannelException.

Invoke MappedByteBuffer‘s isReadOnly() method to determine whether or not you can modify the mapped file.

The position and size parameters define the start and extent of the mapped region.

There is no unmap() method. Once a mapping is established, it remains until the MappedByteBuffer object is garbage collected (or the application exits).

The following code shows A MappedByteBuffer example:

RandomAccessFile raf = new RandomAccessFile("temp.txt", "rw");
FileChannel fc = raf.getChannel();
long size = fc.size();
System.out.println("Size: " + size);
MappedByteBuffer mbb = fc.map(FileChannel.MapMode.READ_WRITE, 0,
size);
mbb.clear();
while (mbb.remaining() > 0) {
System.out.print((char) mbb.get());
}
System.out.println();
System.out.println();

String msg = "hello";
mbb.clear();
mbb.asCharBuffer().put(msg);
while (mbb.hasRemaining()) {
fc.write(mbb);
}
fc.close();

Transferring Bytes Among Channels

To optimize the common practice of performing bulk transfers, two methods have bean added to FileChnnel that avoid the need for intermediate buffers:

  • long transferFrom(ReadableByteChannel src, long position, long count)
  • long transferTo(long position, long count, WritableByteChannel target)

There is a example of transferring between channels:

try (FileInputStream fis = new FileInputStream("temp.txt")){
FileChannel inChannel = fis.getChannel();
WritableByteChannel outChannel = Channels.newChannel(System.out);
inChannel.transferTo(0, inChannel.size(), outChannel);
} catch (IOException ioe){
System.out.println("I/O error: " + ioe.getMessage());
}

Socket Channels

Socket declares a SocketChannel getChannel() method for returning a socket channel instance, which describes an open connection to a socket. Unlike sockets, socket channels are selectable and can function in nonblocking mode. These capabilities enhance the scalability and flexibility of large applications.

Socket channels are describes by the java.nio.channels package’s abstract ServerSocketChannel, SocketChannel, and DatagramChannel classes. Each class ultimately extends SelectableChannle and InterruptibleChannel, making socket channels selectable and interruptible.

SelecableChannel offers the following methods to enable blocking or nonblocking:

  • SelectableChannel configureBlocking(boolean block)
  • boolean isBlocking()
  • Object blockingLock()

To enable nonblocking sockets:

ServerSocketChannel ssc = ServerSocketChannel.open();
ssc.configureBlocking(false);

The blockingLock() method lets you prevent other threads from changing a socket channel’s blocking/nonblocking status.

ServerSocketChannel ssc = ServerSocketChannel.open();
SocketChannel sc = null;
Object lock = ssc.blockingLock();
synchronized(lock){
boolean blocking = ssc.isBlocking();
ssc.configureBlocking(false);
sc = ssc.accept();
ssc.configureBlocking(blocking)
}

Methods of ServerSocketChannel:

  • static ServerSocketChannel open(). Attempt to open a server-socket channel, which is initially unbound; it must be bound to a specific address via one of its peer socket’s bind() methods before connections can be accepted.
  • ServerSocket socket()
  • SocketChannel accept(). Accept the connection made this channel’s socket. If this channel is nonblocking, it immediately returns null when there are no pending connections or returns a socket channel that represents the connection. Otherwise, when the channel is blocking, it blocks.

An example of ServerSocketChannel as shown below:

ServerSocketChannel ssc = ServerSocketChannel.open();
ssc.socket().bind(new InetSocketAddress(9999));
ssc.configureBlocking(false);
String msg = "Local address: " + ssc.socket().getLocalSocketAddress();
ByteBuffer buffer = ByteBuffer.wrap(msg.getBytes());
while (true){
System.out.print(".");
SocketChannel sc = ssc.accept();
if (sc != null){
System.out.println();
System.out.println("Received connection from " +
sc.socket().getRemoteSocketAddress());
buffer.rewind();
sc.write(buffer);
sc.close();
} else {
try{
Thread.sleep(100);
}catch (InterruptedException ie){
assert false; // shouldn't happen
}
}
}

Methods of SocketChannel:

  • static SocketChannel open()
  • static SocketChannel open(InetSocketAddress remoteAddr)
  • Socket socket()
  • boolean connect(SocketAddress remoteAddr)
  • boolean isConnectionPending()
  • booean finishConnect()
  • boolean isConnected()
SocketChannel sc = SocketChannel.open();
sc.connect(new InetSocketAddress("localhost", 9999));
while (!sc.finishConnect()){
System.out.println("waiting to finish connection");
}
ByteBuffer buffer = ByteBuffer.allocate(200);
buffer.asCharBuffer().put("hello at " + new Date());
// send
sc.write(buffer);
// receive
while (sc.read(buffer) >= 0){
buffer.flip();
while (buffer.hasRemaining()){
System.out.print((char) buffer.get());
}
buffer.clear();
}
sc.close();

Pipes

Pipe describes a pair of channels that implement a unidirectional pipe, which is a conduit for passing data in one direction between two entities, such as two file channels or two socket channels. Pipe is analogous to the java.io.PipedInputStream and PiepedOutputStream.

Pipe declares nested SourceChannel and SinkChannel classes that serve as readable and writable byte channels, respectively. Pipe also declares the following methods:

  • static Pipe open()
  • SourceChannel source()
  • SinkChannel sink()

Pipe can be used to pass data within the same JVM. Pipes are ideal in producer/consumer scenarios because of encapsulation: you can use the same code to write data to files, sockets, or pipes depending on the kind of channel presented to the pipe.

The following code shows a producer/consumer example:

final int BUFSIZE = 10;
final int LIMIT = 3;
final Pipe pipe = Pipe.open();
Thread sender = new Thread(() -> {
WritableByteChannel src = pipe.sink();
ByteBuffer buffer = ByteBuffer.allocate(BUFSIZE);
for (int i = 0; i < LIMIT; i++) {
buffer.clear();
for (int j = 0; j < BUFSIZE; j++) {
double random = Math.random();
buffer.put((byte) (random * 256));
System.out.println("CLIENT Send: " + (byte) (random * 256));
}
buffer.flip();
try {
while (src.write(buffer) > 0) ;
} catch (IOException ioe) {
System.err.println(ioe.getMessage());
}
}
try {
src.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
});
Thread receiver = new Thread(() -> {
ReadableByteChannel dst = pipe.source();
ByteBuffer buffer = ByteBuffer.allocate(BUFSIZE);
try {
while (dst.read(buffer) >= 0) {
buffer.flip();
while (buffer.remaining() > 0) {
System.out.println("SERVER Receive: " + (buffer.get() & 255));
}
buffer.clear();
}
} catch (IOException ioe) {
System.err.println(ioe.getMessage());
}
});
sender.start();
receiver.start();

Selectors

Nonblocking mode doesn’t let an application determine if it can perform an operation without actually performing the operation.

The operating system is instructed to observe a group of streams and return some indication of which streams are ready to perform a specific operation (such as read). This capability lets a thread multiplex a potentially huge number of active streams by using the readiness information provided by the operating system.

Selectors let you achieve readiness selection in a Java context.

What is Selectors

A selector is an object created form a subclass of the abstract java.nio.channels.Selector class. The selector maintains a set of channels that it examines to determine which channels are ready for reading, writing, completing a connection sequence, accepting another connection, or some combination of these tasks. The actual work is delegated to the operating system via POSIX select() or similar system call.

How selectors work

Selectors are used with selectable channels, which are objects whose classes ultimately inherit from the abstract java.nio.channels.SelectableChannel class, which describes a channel that can be multiplexed by a selector.

One or more previously created selectable channels are registered with a selector. Each registration returns an instance of a subclass of the abstract SelectionKey class, which is a token signifying the relationship between one channel and the selector. This key keeps track of two sets of operations: interest set and ready set. The interest set identifies that will be tested for readiness the next time one of the selector’s selection methods is invoked. The ready set identifies the key’s channel has been found to be ready.

How to use selectors

Selector‘s methods

  • abstract void close()
  • abstract boolean isOpen()
  • abstract Set<SelectionKey> keys(). Returns this selector’s key set.
  • static Selector open(). Open a selector.
  • abstract SelectorProvider provider()
  • abstract int select(). Returns the number of channels that have become ready for I/O operations since the last time it was called.
  • abstract int select(long timeout). Set timeout for selector’s selection operation.
  • abstract Set<SelectionKey> selectedKeys(). Returns this selector’s selected-key set.
  • abstract int selectNow(). It’s a nonblocking version of select().
  • abstract Selector wakeup(). If another thread is currently blocked in an invocation of the select() or select(long) methods then that invocation will return immediately.

SelectableChannel‘s methods

  • SelectionKey register(Selector sel, int ops)
  • SelectionKey register(Selector sel, int ops, Object att). The third parameter att (a non-null object) is known as an attachment and is a convenient way of recognizing a given channel or attaching additional information to the channel. It’s stored in the SelectionKey instance returned from this method.

SelectionKey

  • int-based constants to ops
    • OP_ACCEPT. Operation-set bit for socket-accept opertions.
    • OP_CONNECT. Operation-set bit for socket-connect operations.
    • OP_READ. Operation-set bit for read operations.
    • OP_WRITE. Operation-set bit for write operations.
  • abstract SelectableChannel channel()

An application typically operations

  1. Performs a selection operation.
  2. Obtains the selected keys followed by an iterator over the selected keys.
  3. Iterates over these keys and performs channel operations (read, write).

A selection operation is performed by invoking one of Selector’s selection methods. It doesn’t return until at least one channel is selected, until this selector’s wakeup() method is invoked, or until the current thread is interrupted, whichever comes first.

A key presents a relationship between a selectable channel and a selector. This relationship can be terminated by invoking SelectionKey’s cancel() method.

When you’re finished with a selector, call Selector’s close() method. If a thread is currently blocked in one of this selector’s selection methods, it’s interrupted as if by invoking the selector’s wakeup() method. Any uncancelled keys still associated with this selector are invalidated, their channels are deregistered, and any other resources associated with this selector are released.

The following code shows a example of selector usage:

// get channels
ServerSocketChannel channel = ServerSocketChannel.open();
channel.configureBlocking(false);
// register channels to the selector
Selector selector = Selector.open();
SelectionKey key = channel.register(selector, SelectionKey.OP_ACCEPT |
SelectionKey.OP_READ |
SelectionKey.OP_WRITE);
// selection operation
while(true){
int numReadyChannels = selector.select();
if (numReadyChannels == 0){
contine;
}
Set<SelectionKey> selectedKeys = selector.selectedKeys();
Iterator<SelectionKey> keyIterator = selectedKeys.iterator();

while(keyIterator.hasNext){
SelectionKey key = keyIterator.next();
if (key.isAcceptable()){
ServerSocketChannel server = (ServerSocketChannel) key.channel();
SocketChannel client = server.accept();
if (client == null){
continue;
}
client.configureBlocking(false);
client.register(selector, SelectionKey.OP_READ);
} else if (key.isReadable()){
SocketChannel client = (SocketChannel) key.channel();
// Perform work on the socket channel...
} else if (key.isWritable()){
SocketChannel client = (SocketChannel) key.channel();
// Perform work on the socket channel...
}
}
}

Asynchronous I/O

NIO provides multiplexed I/O to facilitate the creation of highly scalable servers. Client code registers a socket channel with a selector to be notified when the channel is ready to start I/O.

NIO.2 provides asynchronous I/O, which lets client code initiate an I/O operation and subsequently notifies the client when then operation is complete. Like multiplexed I/O, asynchronous I/O is also commonly used to facilitate the creation of highly scalable servers.

The main concepts of asynchronous I/O are: asynchronous I/O overview, asynchronous file channels, asynchronous socket channels, and asynchronous channel groups.

Asynchronous I/O Overview

The java.nio.channels.AsynchronousChannel interface describes an asynchronous channel, which is a channel that supports asynchronous I/O operation (reads, writes, and so on).

An asynchronous I/O operation is initiated by calling a method that returns a future or requires a completion handler argument, Like

  • Future<V> operation(...). The Future methods may be called to check if the I/O operation has completed.
  • void opertion(... A attachment, CompletionHandler<V, ? super A> handler). The attachment is used for a stateless CompletionHandler object consume the result of many I/O operations. The handler is invoked to consume the result of the I/O operation when it completes or fails.

CompletionHandler declares the following methods to consume the result of an operation when it completes successfully, and to learn why the operation failed and take appropriate action:

  • void complted(V result, A attachment)
  • void failed(Throwable t, A attachment)

After an asynchronous I/O operation called, the method returns immediately. You then call Future methods or provide code in the CompletionHandler implementation to learn more about the I/O operation status and/or process the I/O operation’s results.

The java.nio.channels.AsynchronousByteChannel interface extends AsynchronousChannel. It offers the following four methods:

  • Future<Integer> read(ByteBuffer dst)
  • <A> void read(ByteBuffer dst, A attachment, CompletionHandler<Integer,? super A> handler)
  • Future<Integer> write(ByteBuffer src)
  • <A> void write(ByteBuffer src, A attachment, CompletionHandler<Integer,? super A> handler)

Asynchronous File Channels

The abstract java.nio.channels.AsynchronousFileChannel class describes an asynchronous channel for reading, writing, and manipulating a file.

This channel is created when a file is opened by invoking one of AsynchronousFileChannel’s open() methods.

The file contains a variable-length sequence of bytes that can be read and written, and whose current size can be queried.

Files are read and written by calling AsynchronousFileChannel’s read() and write() methods. One pair returns a Future and the other pair receives a CompletionHandler as an argument.

An asynchronous file channel doesn’t have a current position within the file. Instead, the file position is passed as an argument to each read() and write() method that initiates asynchronous operations.

AsynchronousFileChannel’s methods

  • asynchronous I/O operations
    • Future<FileLock> lock()
    • <A> void lock(A attachment, CompletionHandler<FileLock, ? super A> handler)
    • abstract Future<Integer> read(ByteBuffer dst, long position)
    • abstract <A> void read(ByteBuffer dst, long position, A attachment, CompletionHandler<Integer, ? super A> handler)
    • abstract Future<Integer> write(ByteBuffer src, long position)
    • abstract <A> void write(ByteBuffer src, long position, A attachment, CompletionHandler<Integer, ? super A> handler)
  • static AsynchronousFileChannel open(Path file, OpenOption... options)
  • static AsynchronousFileChannel open(Path file, Set options, ExecutorService executor, FileAttribute... attrs)
  • abstract void force(boolean metaData). Forces any updates to this channel’s file to be written to the storage device that contains it.
  • abstract AsynchronousFileChannel truncate(long size)
  • FileLock tryLock()
  • abstract FileLock tryLock(long position, long size, boolean shared)

The following code is a example of AsynchronousFileChannel:

public static void main(String[] args){
Path path = Paths.get(args[0]);
AsynchronousFileChannel ch = AsynchronousFileChannel.open(path);
ByteBuffer buf = ByteBuffer.allocate(1024);
Future<Integer> result = ch.read(buf, 0);
while (!result.isDone()){
System.out.println("Sleeping...");
Thread.sleep(500);
}
System.out.println("Finished = " + result.isDone());
System.out.println("Bytes erad = " + result.get());
ch.close();
}
public static void main(String[] args){
Path path = Paths.get(args[0]);
AsynchronousFileChannel ch = AsynchronousFileChannel.open(path);
ByteBuffer buf = ByteBuffer.allocate(1024);
Thread mainThd = Thread.currentThread();
ch.read(buf, 0, null,
new CompletionHandler<Integer, Void>(){
@Override
public void completed(Integer result, Void v){
System.out.println("Bytes read = " + result);
mainThd.interrupt();
}
@Override
public void failed(Throwable t, Void v){
System.out.println("Failure: " + t.toString());
mainThd.interrupt();
}
});
System.out.println("Waiting for completion");
try {
mainThd.join();
} catch (InterruptedException ie){
System.out.println("Terminating")
}
ch.close();
}

Asynchronous Socket Channels

The abstract java.nio.channels.AsynchronousServerSocketChannel class describes an asynchronous channel for stream-oriented listening sockets. Its counterpart channel for connecting sockets is described by the abstract java.nio.channels.AsynchronousSocketChannel class.

public class Server
{
private final static int PORT = 9090;
private final static String HOST = "localhost";

public static void main(String[] args)
{
AsynchronousServerSocketChannel channelServer;
try {
channelServer = AsynchronousServerSocketChannel.open();
channelServer.bind(new InetSocketAddress(HOST, PORT));
System.out.printf("Server listening at %s%n",
channelServer.getLocalAddress());
} catch (IOException ioe) {
System.err.println("Unable to open or bind server socket channel");
return;
}
Attachment att = new Attachment();
att.channelServer = channelServer;
channelServer.accept(att, new ConnectionHandler());
try {
Thread.currentThread().join();
} catch (InterruptedException ie) {
System.out.println("Server terminating");
}
}
}
public class Client
{
private final static Charset CSUTF8 = Charset.forName("UTF-8");
private final static int PORT = 9090;
private final static String HOST = "localhost";

public static void main(String[] args)
{
AsynchronousSocketChannel channel;
try {
channel = AsynchronousSocketChannel.open();
} catch (IOException ioe) {
System.err.println("Unable to open client socket channel");
return;
}

try {
channel.connect(new InetSocketAddress(HOST, PORT)).get();
System.out.printf("Client at %s connected%n",
channel.getLocalAddress());
} catch (ExecutionException | InterruptedException eie) {
System.err.println("Server not responding");
return;
} catch (IOException ioe) {
System.err.println("Unable to obtain client socket channel's local address");
return;
}
Attachment att = new Attachment();
att.channel = channel;
att.isReadMode = false;
att.buffer = ByteBuffer.allocate(2048);
att.mainThd = Thread.currentThread();
byte[] data = "Hello".getBytes(CSUTF8);
att.buffer.put(data);
att.buffer.flip();
channel.write(att.buffer, att, new ReadWriteHandler());
try{
att.mainThd.join();
} catch (InterruptedException ie)
{
System.out.println("Client terminating");
}
}
}

Asynchronous Channel Groups

The abstract java.nio.channels.AsynchronousChannelGroup class describes a grouping of asynchronous channels for the purpose of resource sharing. A group has an associated thread pool to which tasks are submitted, to handle I/O events and to dispatch to completion handlers that consume the results of asynchronous operations performed on the group’s channels.

AsynchronousServerSocketChannel and AsyynchronousSocketChannel belong to gourps. When you create an asynchronous socket channel via the no-argument open() method, the channel is bound to the default group.

AsynchronousFileChannel don’t belong to groups. However, they are associated with a thread pool which tasks are submitted, to handle I/O events and to dispatch to completion handlers that consume the results of I/O operations on the channel.

You can configure the default gourp by initializing the following system properties at JVM startup:

  • java.nio.channels.DefaultThreadPool.threadFactory
  • java.nio.channels.DefaultThreadPool.initialSize

You can define your own channel group. It gives you more control over the threads that are used to service the I/O operations. Furthermore, it provides the mechanisms to shut down threads and to await termination. You can use AsynchronousChannelGroup’s methods to create your own channel group:

  • static AsynchronousChannelGroup withCachedThreadPool(ExecutorService executor, int initialSize)
  • static AsynchronousChannelGroup withFixedThreadPool(int nThreads, ThreadFactory threadFactory)
  • static AsynchronousChannelGroup withThreadPool(ExecutorService executor)

Other methods of AsynchronousChannelGroup:

  • abstract boolean awaitTermination(long timeout, TimeUnit unit)
  • abstract boolean isShutdown()
  • abstract boolean isTerminated().
  • abstract void shutdown(). Initiates an orderly shutdown of the group.
  • abstract void shutdownNow(). Shuts down the group and closes all open channels in the group.
  • AsynchronousChannelProvider provider()

After creating a gourp, you can bind an asynchronous socket channel to the group by calling the following class methods:

  • AsynchronousServerSocketChannel’s open(AsynchronousChannelGroup group)
  • AsynchronousSocketChannel’s open(AsynchronousChannelGroup group)
AsynchronousChannelGroup group = AsynchronousChannelGroup.
withFixedThreadPool(20,Executors.defaultThreadFactory());
AsynchronousServerSocketChannel chServer =
AsynchronousServerSocketChannel.open(group);
AsynchronousSocketChannel chClient =
AsynchronousSocketChannel.open(group);

// After the operation has begun, the channel group is used to control the shutdown.
if (!group.isShutdown())
{
// After the group is shut down, no more channels can be bound to it.
group.shutdown();
}
if (!group.isTerminated())
{
// Forcibly shut down the group. The channel is closed and the
// accept operation aborts.
group.shutdownNow();
}
// The group should be able to terminate; wait for 10 seconds maximum.
group.awaitTermination(10, TimeUnit.SECONDS);

Summary

NIO is nonblocking and selectable, and it provides non-blocking IO and multiplexed I/O to facilitate the creation of highly scalable servers

NIO channels read() and write() operations are nonblocking.

Channels read and write operations are work with buffers. For example, write a buffer data to a channel (to a device), read data from a channel (from a device) into a buffer.

NIO channels are selectable it can be managed with a selector. A selector maintains a set of channels that it examines to determine which channels are ready for reading, writing, completing a connection sequence, accepting another connection, or some combination of these tasks. The actual work is delegated to the operating system via POSIX select() or similar system call.

NIO provides non-blocking IO and multiplex I/O, but it still needs to check the channels’ status with a loop. The NIO.2’s asynchronous IO can get notifies when the operation is complete. Like multiplexed I/O, asynchronous I/O is also commonly used to facilitate the creation of highly scalable servers.

All of IO, non-blocking IO, asynchronous IO are supported by the operating system.

References

[1] Java I/O, NIO and NIO.2 by Jeff Friesen

[2] Java SE 8 API

0%