Unicode equivalence

A few days ago I planned to write a Python script to migrate some music playlist from Mac to PC. The task was just to parse XML file and convert the content to PPL file (just plain text). How hard can that be? Done and done within an hour.

Well, it works for most playlist files. But a few fails to load in PC.

All failed playlist involved some Japanese characters in file name. Text editor did not find anything suspicious. Hence I had to go raw. “od -bc” helped a lot.

Content of one file got this:

343 202 253 343 202 231
カ  **  **    ゙  **  ** 

Content of another file got this:

343 202 254
ガ  **  ** 

It should be same Japanese character “ガ”. Why does it get 2 different unicode representation?

OK, here is the culprit: Unicode equivalence. Certain code point sequence is canonically equivalent (e.g. é = e + ◌́, ガ = カ + ◌゙). The former is NFC (Normalization Form Canonical Composition) and the latter is NFD (Normalization Form Canonical Decomposition).

If we know certain file system mainly uses NFC, a simple solution is to convert file name reference to NFC as well. Python code is simple:

string = unicodedata.normalize('NFC', string)
Advertisements
Posted in Computer and Internet, Operation System and Linux | Tagged | Leave a comment

The Iliad of Homer

The Project Gutenberg EBook of The Iliad of Homer by Homer

Online course The Ancient Greek Hero and book The Ancient Greek Hero in 24 Hours by Gregory Nagy

“Hear me, and judge, ye sisters of the main!
How just a cause has Thetis to complain!
How wretched, were I mortal, were my fate!
How more than wretched in the immortal state!
Sprung from my bed a godlike hero came,
The bravest far that ever bore the name;
Like some fair olive, by my careful hand
He grew, he flourish’d and adorn’d the land
To Troy I sent him: but the fates ordain
He never, never must return again.
So short a space the light of heaven to view,
So short, alas! and fill’d with anguish too!
Hear how his sorrows echo through the shore!
I cannot ease them, but I must deplore;
I go at least to bear a tender part,
And mourn my loved-one with a mother’s heart.”

Iliad XVIII

“Ah think, thou favour’d of the powers divine!
Think of thy father’s age, and pity mine!
In me that father’s reverend image trace,
Those silver hairs, that venerable face;
His trembling limbs, his helpless person, see!
In all my equal, but in misery!
Yet now, perhaps, some turn of human fate
Expels him helpless from his peaceful state;
Think, from some powerful foe thou seest him fly,
And beg protection with a feeble cry.
Yet still one comfort in his soul may rise;
He hears his son still lives to glad his eyes,
And, hearing, still may hope a better day
May send him thee, to chase that foe away.
No comfort to my griefs, no hopes remain,
The best, the bravest, of my sons are slain!
Yet what a race! ere Greece to Ilion came,
The pledge of many a loved and loving dame:
Nineteen one mother bore—Dead, all are dead!
How oft, alas! has wretched Priam bled!
Still one was left their loss to recompense;
His father’s hope, his country’s last defence.
Him too thy rage has slain! beneath thy steel,
Unhappy in his country’s cause he fell!

“For him through hostile camps I bent my way,
For him thus prostrate at thy feet I lay;
Large gifts proportion’d to thy wrath I bear;
O hear the wretched, and the gods revere!
“Think of thy father, and this face behold!
See him in me, as helpless and as old!
Though not so wretched: there he yields to me,
The first of men in sovereign misery!
Thus forced to kneel, thus grovelling to embrace
The scourge and ruin of my realm and race;
Suppliant my children’s murderer to implore,
And kiss those hands yet reeking with their gore!”

These words soft pity in the chief inspire,
Touch’d with the dear remembrance of his sire.
Then with his hand (as prostrate still he lay)
The old man’s cheek he gently turn’d away.
Now each by turns indulged the gush of woe;
And now the mingled tides together flow:
This low on earth, that gently bending o’er;
A father one, and one a son deplore:
But great Achilles different passions rend,
And now his sire he mourns, and now his friend.
The infectious softness through the heroes ran;
One universal solemn shower began;
They bore as heroes, but they felt as man.

Iliad XXIV
Posted in Book | Leave a comment

Notes on Irrational Exuberance by Robert J. Shiller

  • Price level of stock market influences valuation and allocation of future expense/saving/investment.
  • Historically, dividend, not capital gain, is the dominant part of the average return on stocks.
  • Ponzi schemes can be considered as a controlled experiments of the existence of positive feedback in financial market.
  • Anchoring and herd behavior could explain change of market valuation without apparent cause.
  • No movement of U.S. aggregate stock prices beyond the trend growth of prices has ever been subsequently justified by dividend movements.

Posted in Economics | Leave a comment

get some shadowsocks

Bing from Microsoft got blocked in China a few days ago. The incident seems accidental, however I still feel it is time to make some backup plan.

Setup is quite easy. Get client from Shadowsocks. Pick some free server from iShadow. Configure client (easy approach: scan QR). OK, you are free to go. This would be good enough for casual user.

Underlying mechanism of shadowsocks seems simple. Just have a usual proxy (actually, a SOCK5 proxy), then encrypt the content between client and server. Since unlike VPN, shadowsocks is usually decentralized (someone would usually rent a low-end VPS, setup a shadowsocks server, done), it would be harder for GFW to find and block the server.

Posted in Computer and Internet | Leave a comment

长乐林

This gallery contains 4 photos.

Gallery | Leave a comment

齐云山 + 木梨硔

This gallery contains 6 photos.

Gallery | Leave a comment

Run multiple jobs in Flink local environment

Actually there is little reason to do this. Flink local environment is generally used to test and debug, not for scalable stream data processing.

But anyway, this is the way to run multiple jobs in the local environment.

First, start the mini-cluster (with some parameters).

Configuration configuration = new Configuration();
configuration.setLong(TaskManagerOptions.MANAGED_MEMORY_SIZE, -1L);
configuration.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 2);
configuration.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 10);

// start cluster
LocalFlinkMiniCluster exec = new LocalFlinkMiniCluster(configuration, true);
exec.start();
...

Then, create and add jobs to the mini-cluster.

StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
DataStream stream = env.addSource(...);
...
stream.addSink(...);

StreamGraph streamGraph = env.getStreamGraph();
streamGraph.setJobName(name);
JobGraph jobGraph = streamGraph.getJobGraph();
exec.submitJobAndWait(jobGraph, true);

In last step, one can choose submitJobDetached instead.

Posted in Cloud, Computer and Internet | Leave a comment