2024/04/28

uHex

uHex 是一個簡單的 hex editor, 原本是 DOS 下的應用程式,但是也可以在支援 POSIX 標準並且有安裝 NCurses 函式庫開發檔案的環境編譯。 主要用來作為觀看二進位檔案以及進行簡單的搜尋與編輯。

因為是個小工具,所以在 Linux 系統我自己採用自行編譯的方式。

下面就是他的操作按鍵:
ALT+H - Help (F1 works too)
ALT+J - Jump to offset
ALT+F - Find an ASCII or HEX string occurence
ALT+S - Save file, applying all modifications
ALT+U - Undo all modifications (reverts the file from disk)
ESC - Quit uHex

2024/04/26

DOSBos

DOSBos 是用於執行適用 MS-DOS 相容作業系統的模擬器 (主要的執行目標是遊戲軟體)。

下面是在 openSUSE Tumbleweed 安裝的指令:

sudo zypper in dosbox

要注意的是,openSUSE Tumbleweed 目前所使用 DOSBox 版本為 DOSBox Staging, 這是自 DOSBox 衍生的一個開發版本,其設定檔案在家目錄下的 .config/dosbox/dosbox.conf。

在按鍵部份,如果要切換到 full-screen(或者是切換回來),使用下列的按鍵組合: ALT-ENTER
如果要關閉 DOSBos: CTRL-F9
捕抓或者是釋放滑鼠: CTRL-F10

我習慣建立一個 DOSBox 目錄作為 c:

[autoexec]
# Lines in this section will be run at startup.
# You can put your MOUNT lines here.
MOUNT C /home/danilo/DOSBox/C -freesize 10240
c:

下面是其它的設定:
fullresolution = original
output = openglnb
memsize = 64
aspect = stretch
gus = false
(openSUSE Tumbleweed 的預設值 gus 是 true,需要設為 false,不然音訊輸出會不正常)

如果要掛載一個 iso 檔案,下面是一個例子:

imgmount d "/home/danilo/DOSBox/ISO/MyISO.iso" -t iso

2024/04/22

PostgreSQL

PostgreSQL是一個開源的物件型關聯式資料庫管理系統,在類似 BSD 授權與 MIT 授權的 PostgreSQL 授權下發行。

Install on openSUSE Tumbleweed

如果不是首次安裝而是升級,需要使用 pg_dump 執行 dump the databases,先對目前的資料進行備份。

首先是安裝(在這裡是 PostgreSQL 16):

sudo zypper install postgresql16-server postgresql16 postgresql16-devel \
postgresql16-contrib postgresql-devel
如果需要設定 postgres 的密碼,
sudo passwd postgres
先切換到 root,再切換到 postgres,
sudo su postgres
然後 init database:
/usr/bin/initdb /var/lib/pgsql/data
如果要手動開啟服務,使用:
/usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start
關閉:
/usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile stop
或者是使用下列的方式開啟服務:
sudo service postgresql start
關閉服務:
sudo service postgresql stop

如果要開機的時候就啟動服務,使用:
sudo chkconfig postgresql on
如果不要,使用:
sudo chkconfig postgresql off

安裝完成以後,可以使用 psql postgres (使用者登入要使用 postgres)來確定目前的設定檔:
SHOW config_file;
目前的 server 版本:
SHOW server_version;
如果要使用 psql 列出目前的 database,使用 \l 命令:
\l
如果要查詢目前的使用者連線:
SELECT * FROM pg_stat_activity;
Digging into the currently opened database in psql one can use the \d command.
\d

Add user to PostgreSQL

如果要使用指令增加使用者和資料庫(需要切換使用者身份到 postgres):
createuser danilo

使用 PostgreSQL SQL shell (psql) 增加使用者。

1. Add a user called danilo

Type the following command to create a user called danilo with a password called danilo:
template1=# CREATE USER danilo WITH PASSWORD 'danilo';

2. Add a database called danilo

Type the following command:
template1=# CREATE DATABASE danilo WITH OWNER danilo ENCODING 'UTF8';

3. Now grant all privileges on database

template1=# GRANT ALL PRIVILEGES ON DATABASE danilo to danilo;


如果要改使用者密碼,下面是 user danilo 的範例:
template1=# ALTER USER "danilo" WITH PASSWORD 'danilo';

或者是使用下列的方式設定 postgres:
template1=# \password postgres

要讓密碼生效,要修改 /var/lib/pgsql/data (或者是指定的 PostgreSQL DATADIR) 目錄下的 pg_hba.conf。 預設的認證方式為 trust,這表示不會做任何的密碼檢查,如果要開啟密碼檢查, 要把 trust 改為 scram-sha-256 或者是其它的認證方式(例如 md5 或者是 peer)。

如果設為 peer 就只會讓 local 同樣的 username 連線。
在需要讓其它的本機使用者連線的情況下,可以改為 scram-sha-256,下面是一個例子:

# "local" is for Unix domain socket connections only
local   all             all                                     scram-sha-256
# IPv4 local connections:
host    all             all             127.0.0.1/32            scram-sha-256
# IPv6 local connections:
host    all             all             ::1/128                 scram-sha-256

如果想知道目前的 username 與 password,可以使用 psql 查詢 pg_shadow 以後列出來:

psql -Atq -U postgres -d postgres -c "SELECT concat('\"', usename, '\" \"', passwd, '\"') FROM pg_shadow"

如果要查詢目前的使用者列表:

\du

如果要設定某一個使用者為 Superuser:

ALTER USER danilo WITH SUPERUSER;

如果要設定某一個使用者為 No Superuser

ALTER USER danilo WITH NOSUPERUSER;

Configuration

PostgreSQL uses three main configuration files to control overall operations. You can find these files in the initialized data cluster (the folder specified during the initialization process using initdb -d).

如果查詢 source code,我們發現 PostgreSQL 設定來源有14種,如default、environment variable、configuration file、client等,具體如下:
const char *constGucSource_Names[] =
{
        /* PGC_S_DEFAULT */ "default",
        /* PGC_S_DYNAMIC_DEFAULT */ "default",
        /* PGC_S_ENV_VAR */ "environment variable",
        /* PGC_S_FILE */ "configuration file",
        /* PGC_S_ARGV */ "command line",
        /* PGC_S_GLOBAL */ "global",
        /* PGC_S_DATABASE */ "database",
        /* PGC_S_USER */ "user",
        /* PGC_S_DATABASE_USER */ "database user",
        /* PGC_S_CLIENT */ "client",
        /* PGC_S_OVERRIDE */ "override",
        /* PGC_S_INTERACTIVE */ "interactive",
        /* PGC_S_TEST */ "test",
        /* PGC_S_SESSION */ "session"
};

如果無法確定設定的來源,可以使用下列的 SQL 述句查詢:

select name, setting, source from pg_settings;

There are several different types of configuration settings, divided up based on the possible inputs they take

  • Boolean: true, false, on, off
  • Integer: Whole numbers (2112)
  • Float: Decimal values (21.12)
  • Memory / Disk: Integers (2112) or "computer units" (512MB, 2112GB). Avoid integers--you need to know the underlying unit to figure out what they mean.
  • Time: "Time units" aka d,m,s (30s). Sometimes the unit is left out; don't do that
  • Strings: Single quoted text ('pg_log')
  • ENUMs: Strings, but from a specific list ('WARNING', 'ERROR')
  • Lists: A comma separated list of strings ('"$user",public,tsearch2)

下面 14 個項定項目是大多數的使用者可能會需要設定的項目:

  1. listen_address
  2. max_connections
  3. shared_buffers
  4. work_mem
  5. maintenance_work_mem
  6. max_fsm_pages
  7. synchronous_commit
  8. checkpoint_segments
  9. wal_buffers
  10. autovacuum
  11. effective_cache_size
  12. default_statistics_target
  13. constraint_exclusion
  14. log_destination & log settings

之所以需要設定一些項目的理由,在於根據自己機器的狀況設定更好的設定值, 可以讓 PostgreSQL 執行的更好。

Each backend process allocates a local memory area for query processing; each area is divided into several sub-areas – whose sizes are either fixed or variable.

  • work_mem: Executor uses this area for sorting tuples by ORDER BY and DISTINCT operations, and for joining tables by merge-join and hash-join operations.
  • maintenance_work_mem: Some kinds of maintenance operations (e.g., VACUUM, REINDEX) use this area.
  • temp_buffers: Executor uses this area for storing temporary tables.

A shared memory area is allocated by a PostgreSQL server when it starts up. This area is also divided into several fix sized sub-areas.

  • shared buffer pool: PostgreSQL loads pages within tables and indexes from a persistent storage to here, and operates them directly.
  • WAL buffer: To ensure that no data has been lost by server failures, PostgreSQL supports the WAL mechanism. WAL data (also referred to as XLOG records) are transaction log in PostgreSQL; and WAL buffer is a buffering area of the WAL data before writing to a persistent storage.
  • commit log: Commit Log(CLOG) keeps the states of all transactions (e.g., in_progress,committed,aborted) for Concurrency Control (CC) mechanism.

如果要查詢目前的設定,可以使用下面的命令:

SHOW ALL;

如果要查詢單項,以 work_mem 來說:

show work_mem;
或者是查詢 pg_settings (a “system view” can be queried),
SELECT * FROM pg_settings WHERE name = 'work_mem';

All these can be edited with a text editor.

File Purpose
postgresql.conf Controls the listening port, IP, and default query planner settings, memory settings, path settings, and logging settings. Can be queried via pg_settings database view.
pg_hba.conf Controls the authentication models used by PostgreSQL and can be set per user, per database, per IP range, or a combination of all.
pg_indent.conf Controls mapping of an OS user to a PostgreSQL user.

postgresql.conf

The following settings are all located in the postgresql.conf file. Remember that these are default settings; many of these you can choose to override for each session, for each database, or for each user/role.

Option Description
listen_addresses Use ‘*’ to listen on all IPs of the server, ‘localhost’ to listen on just local, or a comma separated list of IPs to listen on. Requires service restart if changed and can only be set globally.
port Defaults to 5432, but can be changed to allow multiple postgresql daemon clusters/versions to coexist using same IP but different ports.
search_path List of default schemas that don’t need schema qualification. First schema is where non-schema qualified objects are created.
constraint_exclusion Options: on, off, or partial. Partial was introduced in 8.4 and is the new default. Allows planner to skip over tables if constraint ensures query conditions cannot be satisfied by the table. Mostly used for table partitioning via table inheritance.
shared_buffers Controls how much memory is allocated to PostgreSQL and shared across all processes. Requires service restart and can only be set globally.

In PostgreSQL 9.4, a new SQL construction ALTER SYSTEM was introduced that allows you to set these settings at the system level without editing the postgresql.conf. For many, you still need to do a service restart and for others at least a:

SELECT pg_reload_conf();

pg_hba.conf

PostgreSQL supports many authentication schemes to control access to the database. The pg_hba.conf file dictates which schemes are used based on the rules found in this file. You can mix and match various authentication schemes at the same time. The rules are applied sequentially such that the first match fitting a connection is the one that is used. This is important to remember because if you have a more restrictive rule above a less restrictive, then the more restrictive is the one that trumps.

The most commonly used authentication schemes are trust (which allows connections without a password) and md5 (which authenticates with md5 encrypted passwords). Others include: reject, crypt, password (this is plain text), krb5, ident (authenticate simply by identity of user in OS), pam, and ldap.


如果我們需要每個 query 的執行時間,可以在 psq 使用下列的命令開啟設定(預設值有可能是關閉):
\timing

Tools

PostgreSQL comes bundled with several tools useful for administration and query writing.

Tool Description
psql Command-line client packaged with PostgreSQL. Good for automating SQL jobs, copying data, outputing simple HTML reports.
createdb, dropdb For creating and dropping a database from the OS shell.
pgAdminIII Popular graphical user interface packaged with PostgreSQL.
pg_restore Command-line tool for restoring compressed or .tar backups.
pg_dump Command-line tool for doing backups. Great for automated backups.
pg_dumpall Command-line tool for dumping all databases into a single backup.
pgAgent A daemon/service that can be downloaded from http://www.pgadmin.org/download/pgagent.php. Used for scheduling SQL jobs and batch shell jobs. Jobs can be added easily and monitored using the PgAdmin III job interface.
pg_basebackup Used for doing filesystem hot backup of db data cluster.
pg_upgrade Used for updating in place from one major version of PostgreSQL to another.

SQL statement

Auto-increment column

PostgreSQL 並不支援標準語法,而是提供一個自己實作的近似語法:
CREATE TABLE t1 (col1 SERIAL PRIMARY KEY);

UPSERT

PostgreSQL 的 UPSERT 語法並未採用標準的 merge,而是使用自己實作的語法。

舉例來說,我們建立下面一個表格:
create table notes (title varchar(255), body text, modified timestamp, primary key(title));
於是可以使用 ON CONFLICT DO UPDATE SET 方式來決定是要 update 或者是 insert(PS. 舉例是用 GOLANG 的常數宣告):
        const insertString = `
              insert into notes (title, body, modified)
              values($1, $2, now()) ON CONFLICT (title)
              DO UPDATE SET (body,modified) = (EXCLUDED.body,now())`

Views

You can create a view over the query, which gives a name to the query that you can refer to like an ordinary table:
CREATE VIEW myview AS
    SELECT city, temp_lo, temp_hi, prcp, date, location
        FROM weather, cities
        WHERE city = name;

SELECT * FROM myview;

Making liberal use of views is a key aspect of good SQL database design. Views allow you to encapsulate the details of the structure of your tables, which might change as your application evolves, behind consistent interfaces.

Foreign Keys

Foreign Keys 是維護你資料的 referential integrity 的方法,當插人一個新資料的時候,確定參考的資料是存在的。

下面是一個使用的範例:
CREATE TABLE cities (
        city     varchar(80) primary key,
        location point
);

CREATE TABLE weather (
        city      varchar(80) references cities(city),
        temp_lo   int,
        temp_hi   int,
        prcp      real,
        date      date
);

Unique Constraints

Unique constraints ensure that the data contained in a column, or a group of columns, is unique among all the rows in the table. The syntax is:

CREATE TABLE products (
product_no integer UNIQUE,
name text,
price numeric
);
when written as a column constraint, and:
CREATE TABLE products (
product_no integer,
name text,
price numeric,
UNIQUE (product_no)
);

Transactions

In PostgreSQL, a transaction is set up by surrounding the SQL commands of the transaction with BEGIN and COMMIT commands.

By default (without BEGIN), PostgreSQL executes transactions in "autocommit" mode, that is, each statement is executed in its own transaction and a commit is implicitly performed at the end of the statement (if execution was successful, otherwise a rollback is done).

Window Functions

A window function performs a calculation across a set of table rows that are somehow related to the current row. This is comparable to the type of calculation that can be done with an aggregate function. But unlike regular aggregate functions, use of a window function does not cause rows to become grouped into a single output row — the rows retain their separate identities. Behind the scenes, the window function is able to access more than just the current row of the query result.

A window function call always contains an OVER clause directly following the window function's name and argument(s). This is what syntactically distinguishes it from a regular function or aggregate function. The OVER clause determines exactly how the rows of the query are split up for processing by the window function. The PARTITION BY list within OVER specifies dividing the rows into groups, or partitions, that share the same values of the PARTITION BY expression(s). For each row, the window function is computed across the rows that fall into the same partition as the current row.

下面是一個範例:
SELECT depname, empno, salary,
       rank() OVER (PARTITION BY depname ORDER BY salary DESC)
FROM empsalary;
結果:
  depname  | empno | salary | rank
-----------+-------+--------+------
 develop   |     8 |   6000 |    1
 develop   |    10 |   5200 |    2
 develop   |    11 |   5200 |    2
 develop   |     9 |   4500 |    4
 develop   |     7 |   4200 |    5
 personnel |     2 |   3900 |    1
 personnel |     5 |   3500 |    2
 sales     |     1 |   5000 |    1
 sales     |     4 |   4800 |    2
 sales     |     3 |   4800 |    2
(10 rows)

As shown here, the rank function produces a numerical rank within the current row's partition for each distinct ORDER BY value, in the order defined by the ORDER BY clause. rank needs no explicit parameter, because its behavior is entirely determined by the OVER clause.

The rows considered by a window function are those of the "virtual table" produced by the query's FROM clause as filtered by its WHERE, GROUP BY, and HAVING clauses if any. For example, a row removed because it does not meet the WHERE condition is not seen by any window function. A query can contain multiple window functions that slice up the data in different ways by means of different OVER clauses, but they all act on the same collection of rows defined by this virtual table.

Inheritance

Inheritance is a concept from object-oriented databases. It opens up interesting new possibilities of database design.

下面是一個使用的範例:
CREATE TABLE cities (
  name       text,
  population real,
  altitude   int     -- (in ft)
);

CREATE TABLE capitals (
  state      char(2)
) INHERITS (cities);
當你查詢的時候,會列出 cities 和 capitals 的結果,如果只要 cities 的部份,下面是一個使用的範例:
SELECT name, altitude
    FROM ONLY cities
    WHERE altitude > 500;

WITH Queries (Common Table Expressions)

WITH 是用來建立暫存的資料集,稱為通用資料表運算式 CTE (common table expression) 的暫存具名結果集。 通用資料表運算式可以包括指向本身的參考。 這稱為遞迴通用資料表運算式。

下面的資料來自於 PostgreSQL 網站

In PostgreSQL, the WITH query provides a way to write auxiliary statements for use in a larger query. It helps in breaking down complicated and large queries into simpler forms, which are easily readable.

The WITH query being CTE query, is particularly useful when subquery is executed multiple times. It is equally helpful in place of temporary tables.

The optional RECURSIVE modifier changes WITH from a mere syntactic convenience into a feature that accomplishes things not otherwise possible in standard SQL. Using RECURSIVE, a WITH query can refer to its own output. A very simple example is this query to sum the integers from 1 through 100:

WITH RECURSIVE t(n) AS (
    VALUES (1)
  UNION ALL
    SELECT n+1 FROM t WHERE n < 100
)
SELECT sum(n) FROM t;

The general form of a recursive WITH query is always a non-recursive term, then UNION (or UNION ALL), then a recursive term, where only the recursive term can contain a reference to the query's own output. Such a query is executed as follows:

  1. Evaluate the non-recursive term. For UNION (but not UNION ALL), discard duplicate rows. Include all remainingrows in the result of the recursive query, and also place them in a temporary working table.

  2. So long as the working table is not empty, repeat these steps:

    1. Evaluate the recursive term, substituting the current contents of the working table for the recursive self-reference. For UNION (but not UNION ALL), discard duplicate rows and rows that duplicate any previous result row. Include all remaining rows in the result of the recursive query, and also place them in a temporary intermediate table.

    2. Replace the contents of the working table with the contents of the intermediate table, then empty the intermediate table.

Recursive queries are typically used to deal with hierarchical or tree-structured data.
WITH RECURSIVE included_parts(sub_part, part, quantity) AS (
    SELECT sub_part, part, quantity FROM parts WHERE part = 'our_product'
  UNION ALL
    SELECT p.sub_part, p.part, p.quantity
    FROM included_parts pr, parts p
    WHERE p.part = pr.sub_part
  )
SELECT sub_part, SUM(quantity) as total_quantity
FROM included_parts
GROUP BY sub_part

Index

下面是一個建立 index 的範例:
CREATE TABLE test1c (
    id integer,
    content varchar COLLATE "x"
);

CREATE INDEX test1c_content_index ON test1c (content);

GIN and GiST Index Types

There are two kinds of indexes that can be used to speed up full text searches. Note that indexes are not mandatory for full text searching, but in cases where a column is searched on a regular basis, an index is usually desirable.

CREATE INDEX name ON table USING GIN (column);
and:
CREATE INDEX name ON table USING GIST (column);

GIN indexes are the preferred text search index type. As inverted indexes, they contain an index entry for each word (lexeme), with a compressed list of matching locations. Multi-word searches can find the first match, then use the index to remove rows that are lacking additional words. GIN indexes store only the words (lexemes) of tsvector values, and not their weight labels. Thus a table row recheck is needed when using a query that involves weights.

A GiST index is lossy, meaning that the index might produce false matches, and it is necessary to check the actual table row to eliminate such false matches. (PostgreSQL does this automatically when needed.) GiST indexes are lossy because each document is represented in the index by a fixed-length signature. The signature is generated by hashing each word into a single bit in an n-bit string, with all these bits OR-ed together to produce an n-bit document signature. When two words hash to the same bit position there will be a false match. If all words in the query have matches (real or false) then the table row must be retrieved to see if the match is correct.

Lossiness causes performance degradation due to unnecessary fetches of table records that turn out to be false matches. Since random access to table records is slow, this limits the usefulness of GiST indexes. The likelihood of false matches depends on several factors, in particular the number of unique words, so using dictionaries to reduce this number is recommended.

Full Text Search

Full text searching in PostgreSQL is based on the match operator @@, which returns true if a tsvector (document) matches a tsquery (query).

下面是一個使用的例子:
SELECT 'a fat cat sat on a mat and ate a fat rat'::tsvector @@ 'cat & rat'::tsquery;
 ?column?
----------
 t

SELECT 'fat & cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::tsvector;
 ?column?
----------
 f
另外的例子:
SELECT to_tsvector('fat cats ate fat rats') @@ to_tsquery('fat & rat');
 ?column?
----------
 t
The @@ operator also supports text input, allowing explicit conversion of a text string to tsvector or tsquery to be skipped in simple cases. The variants available are:
tsvector @@ tsquery
tsquery  @@ tsvector
text @@ tsquery
text @@ text
It is possible to do a full text search without an index. A simple query to print the title of each row that contains the word friend in its body field is:
SELECT title
FROM pgweb
WHERE to_tsvector('english', body) @@ to_tsquery('english', 'friend');
PostgreSQL 可以使用 GIN index 來加快 text search 的速度:
CREATE INDEX pgweb_idx ON pgweb USING GIN (to_tsvector('english', body));

UUID

可以安裝 uuid-ossp 模組來處理 UUID。下面是安裝的方式(需要有 Superuser 的權限):

CREATE EXTENSION "uuid-ossp";

於是就可以使用下列的方式生成 UUID:

select uuid_generate_v4();

或者也可以使用下列的方式生成 UUID:

select (md5(random()::text || clock_timestamp()::text))::uuid;

NoSQL 與 PostgreSQL

PostgreSQL 提供了可選的 HStore key-value store extension, 並且 HStore extension 具有將 key-value store 轉為 JSON 的能力(hstore_to_json function)。

測試 PostgreSQL 是否有支援 hstore(使用 CREATE EXTENSION 語句):
package require tdbc::postgres
tdbc::postgres::connection create db -user postgres -password postgres -port 5432

set statement [db prepare {
     CREATE EXTENSION hstore
}]

$statement foreach row {
 puts $row
}

$statement close
db close
下面是一個使用的範例:
CREATE TABLE products (
  id serial PRIMARY KEY,
  name varchar,
  attributes hstore
);
From here you can insert whatever you want into the attributes column. And then query based on those various keys or values.
INSERT INTO products (name, attributes) VALUES (
 'Geek Love: A Novel',
 'author    => "Katherine Dunn",
  pages     => 368,
  category  => fiction'
);

SELECT name, attributes->'author' as author
FROM products
WHERE attributes->'category' = 'fiction'
The obvious benefit here is flexibility, but where it really shines is being able to leverage various index types. In particular, a GIN or GiST index will index every key and value within the hstore. This way when you filter on something it’ll use the index if it makes sense to the planner within Postgres.

PostgreSQL and JSON

在 PostgreSQL 9.2 (及以後)版本中,查詢結果可以轉換為 JSON data type 後返回。 PostgreSQL 也增加了 PLv8 extension,可以使用 JavaScript 來撰寫 PostgreSQL stored procedure。

That has changed with 9.4, with the introduction of the JSONB data type, which stores JSON data in binary form, such that it is both more compact and more efficient than the textual form. Moreover, the same GIN and GIST indexes that now are able to work so well with HStore data also are able to work well, and quickly, with JSONB data. So you can search for and retrieve text from JSONB documents as easily (or more) as would have been the case with a document database, such as MongoDB.

PostgreSQL provides two native operators -> and ->> to help you query JSON data.
  • The operator -> returns JSON object field by key.
  • The operator ->> returns JSON object field by text.
下面是一個使用的例子:
CREATE TABLE integrations (id UUID, data JSONB);
INSERT INTO integrations VALUES (
  uuid_generate_v4(),
  '{
    "service": "salesforce",
    "id": "AC347D212341XR",
    "email": "craig@citusdata.com",
    "occurred_at": "8/14/16 11:00:00",
    "added": {
      "lead_score": 50
    },
    "updated": {
      "updated_at": "8/14/16 11:00:00"
    }
    }')
下面是一個最簡單的例子:
CREATE TABLE people(
    id serial primary key,
    data jsonb not null
);

INSERT INTO people(data) VALUES ('{
    "name": "Bob",
    "addresses": [
        {
            "street": "Centre",
            "streetnumber": 24,
            "town": "Thornlie",
            "state": "WesternAustralia",
            "country": "Australia"
        },
        {
            "street": "Example",
            "streetnumber": "4/311-313",
            "town": "Auckland",
            "country": "NewZealand"
        }
    ],
    "phonenumbers": [
        {
            "type": "mobile",
            "number": "12345678"
        }
    ]
}');

INSERT INTO people(data) VALUES ('{
  "name": "Fred",
  "phonenumbers": [
    { "type": "mobile", "number": "12345678" }
  ]
}');

INSERT INTO people(data) VALUES ('{
  "name": "John Doe"
}');
下面則是使用 inner join 查詢 phonenumbers 的做法:
select
  p1.id AS person1,
  p2.id AS person2,
  p1.data ->> 'name' AS "p1 name",
  p2.data ->> 'name' AS "p2 name",
  pns1 ->> 'type' AS "type",
  pns1 ->> 'number' AS "number"
from people p1
  inner join people p2
    on (p1.id > p2.id)
  cross join lateral jsonb_array_elements(p1.data -> 'phonenumbers') pns1
  inner join lateral jsonb_array_elements(p2.data -> 'phonenumbers') pns2
    on (pns1 -> 'type' = pns2 -> 'type' AND pns1 -> 'number' = pns2 -> 'number');
The ? operator will tell us if some part of JSON has a top level key:
SELECT *
FROM companies
WHERE data->'company'->'tags' ? 'B2B'
如果要回來的 JSONB 結果比較漂亮,可以使用:
SELECT jsonb_pretty(data)
FROM companies;

PostgreSQL and XML

PostgreSQL 提供了 XML data type and support functions 的功能 (SQL/XML), 並且提供了一定程度的 XPATH 支援(xpath_exists function) 與輸出 XML 文件的能力,因此也可以考慮將 PostgreSQL 作為一個 XML 文件資料庫使用。

下面是一個 PostgreSQL XPath function 處理 XML 字串的例子(要注意例子中在 xpath() 的第三個參數設定了 namespace 的別名 n):
WITH x AS ( SELECT
'<promotions xmlns="http://www.demandware.com/xml/impex/promotion/2008-01-31">
    <campaign campaign-id="2013-1st-semester-jet-giveaways">
        <description>2013 1st Semester Jet Giveaways</description>
        <enabled-flag>true</enabled-flag>
        <start-date>2013-01-01T05:00:00.000Z</start-date>
        <end-date>2013-07-01T04:00:00.000Z</end-date>
        <customer-groups>
            <customer-group group-id="Everyone"/>
        </customer-groups>
    </campaign>
 </promotions>'::xml AS t
)
SELECT xpath('/n:promotions/n:campaign/n:description/text()', t
           , '{{n,http://www.demandware.com/xml/impex/promotion/2008-01-31}}')
FROM   x;

List extensions

如果要列出目前系統上的 PostgreSQL extension name(使用 TDBC):

package require tdbc::postgres
tdbc::postgres::connection create db -user postgres -password postgres -port 5432

set statement [db prepare {
     select extname from pg_extension
}]

$statement foreach row {
 puts $row
}

$statement close
db close

List tables

使用 psql 列出 schema public 下的所有 table:
\dt public.*
You can select the tables from information_schema:
SELECT * FROM information_schema.tables WHERE table_schema = 'public';
或者也可以使用下列的方式:
select * from pg_tables where schemaname='public';

Client

C API (libpq)

The libpq library is the C interface to PostgreSQL. It is a set of library functions that allow client programs to interact with PostgreSQL.

下面是一個使用的範例,例印 PostgreSQL libpq 的版本:
#include <libpq-fe.h>

int main() {

    int lib_ver = PQlibVersion();

    printf("Version of libpq: %d\n", lib_ver);

    return 0;
}
使用下列的方式得到 include/library path:
$ pg_config --includedir
/usr/include/postgresql
$ pg_config --libdir
/usr/lib/postgresql95/lib64
所以可以使用下列的方式編譯:
gcc lib_version.c -I`pg_config --includedir` -L`pg_config --libdir` -lpq -std=c99 -o test

再來是列印 PostgreSQL server 的版本:
#include <stdio.h>
#include <stdlib.h>
#include <libpq-fe.h>

void do_exit(PGconn *conn) {

    PQfinish(conn);
    exit(1);
}

int main() {

    PGconn *conn = PQconnectdb("user=postgres password=postgres dbname=postgres");

    if (PQstatus(conn) == CONNECTION_BAD) {

        fprintf(stderr, "Connection to database failed: %s\n",
            PQerrorMessage(conn));
        do_exit(conn);
    }

    int ver = PQserverVersion(conn);

    printf("Server version: %d\n", ver);

    PQfinish(conn);

    return 0;
}

下面是一個簡單的 query 程式:
#include <stdio.h>
#include <stdlib.h>
#include <libpq-fe.h>

void do_exit(PGconn *conn) {

    PQfinish(conn);
    exit(1);
}

int main() {

    PGconn *conn = PQconnectdb("user=postgres password=postgres dbname=postgres");i

    if (PQstatus(conn) == CONNECTION_BAD) {

        fprintf(stderr, "Connection to database failed: %s\n",
            PQerrorMessage(conn));
        do_exit(conn);
    }

    PGresult *res = PQexec(conn, "SELECT VERSION()");

    if (PQresultStatus(res) != PGRES_TUPLES_OK) {

        printf("No data retrieved\n");
        PQclear(res);
        do_exit(conn);
    }

    printf("%s\n", PQgetvalue(res, 0, 0));

    PQclear(res);
    PQfinish(conn);

    return 0;
}

ODBC (Linux/openSUSE)

下載 PostgreSQL ODBC Driver 編譯並且安裝以後, unixODBC 需要設定二個檔案。/etc/unixODBC/odbc.ini 與 /etc/unixODBC/odbcinst.ini。


首先是 odbcinst.ini 的內容:
[PostgreSQL]
Description     = PostgreSQL driver
Driver          = psqlodbcw.so
FileUsage       = 1
再來是 odbc.ini,
[PostgreSQL]
Description         = PostgreSQL
Driver              = PostgreSQL
Trace               = No
TraceFile           =
Database            = postgres
Servername          = localhost
UserName            = postgres
Password            = postgres
Port                = 5432
Protocol            = 7.4
ReadOnly            = No
RowVersioning       = No
ShowSystemTables    = No
ShowOidColumn       = No
FakeOidIndex        = No
ConnSettings        =

下面就是測試的 Tcl script:
package require tdbc::odbc

set connStr "DSN=PostgreSQL; UID=postgres; PWD=postgres;"
tdbc::odbc::connection create db $connStr

set statement [db prepare {
    SELECT VERSION()
}]

$statement foreach row {
    puts [dict get $row version]
}

$statement close
db close

JDBC

PostgreSQL 提供了 JDBC driver 可以使用。

下面是使用 TDBCJDBC 的範例:
package require tdbc::jdbc

set className    {org.postgresql.Driver}
set url          jdbc:postgresql://localhost:5432/danilo
set username     danilo
set password     danilo

tdbc::jdbc::connection create db $className $url $username $password -readonly 0

set statement [db prepare {select extname, extversion from pg_extension}]
puts "List extension name and version:"
$statement foreach row {
    puts "[dict get $row extname] - [dict get $row extversion]"
}

$statement close

db close

Node.js

我們使用 node-postgres 來連線。

首先建立一個目錄 pg-test,並且在目錄下執行

npm init -y

這會建立一個 package.json 檔案,我們可以對需要修改的部份再進行修改。修改 package.json,加入下列的設定:

  "type": "module",

再來使用 NPM 安裝 node-postgres:

npm install pg --save

建立 index.js,內容如下:

import pg from 'pg';

const config = {
    host: 'localhost',
    user: 'danilo',
    password: 'danilo',
    database: 'danilo',
    port: 5432,
    ssl: false
};

const client = new pg.Client(config);

client.connect(err => {
    if (err) {
        console.log("Connection failed.");
        throw err;
    }
});

const query = 'SELECT VERSION()';

client.query(query)
.then(res => {
   const rows = res.rows;
   console.log(`PostgreSQL version: ${rows[0].version}`);
   client.end();
   process.exit();
}).catch(err => {
   console.log(err);
});

執行的時候使用 node index.js 來執行。

Tcl client side interface

Pgtcl is a Tcl package for client programs to interface with PostgreSQL servers. 建立在 libpq 的基礎之上。

下面是一個最簡單的例子:

package require Pgtcl

set db [pg_connect -conninfo [list host = localhost user = postgres password = postgres dbname = postgres]]

pg_select $db "select version() as versoin" version {
    parray version
}

pg_disconnect $db

和 JDBC 單純的使用 ? 來代表參數不同,Pgtcl/libpq 使用 $1, $2 等來代表要傳遞的參數。


如果要使用共同的資料庫存取介面,建議使用 Tcl 8.6 內建的 TDBC 所提供的 PostgreSQL driver 來與 PostgreSQL server 進行連線。 Pgtcl 只有在單純使用 PostgreSQL 時才需要使用。

要注意的是,TDBC-Postgres 使用 PQprepare 來送出 SQL statement,同時並不指定 type,而是讓 PostgreSQL 來推斷 type, 再使用 PQdescribePrepared 來取得相關的 param 資料。但是 Pgtcl 不支援 PQprepare(只送出 SQL statement 到 server 分析), 所以需要使用 PREPARE statement 才行。

PostgreSQL: Foreign data wrappers

In 2003, a new specification called SQL/MED ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.

在使用 FDW 的情況下,PostgreSQL 具有存取外部資料源的能力,並且可以作為一個 SQL query engine 使用。 也就是在 FDW 的處理範圍內,PostgreSQL 也有讀取 Hadoop/HBase big data store 的能力(看 FDW 所提供的能力而定)。

PostgreSQL FDW 的 Column-Oriented Wrappers 有二個選擇, 一個是 monetdb_fdw, 一個是 cstore_fdw

monetdb_fdw

使用 monetdb_fdw 來進行 PostgreSQL Foreign_data_wrappers 的測試。

如果不從 PostgreSQL source code 一起編譯,使用下列的方式:
make USE_PGXS=1
如果 MonetDB 的 MAPI lib 與 include 目錄不正確,需要修改 Makefile。

下面是 Makefile 修改的範例:
SHLIB_LINK = -L/usr/local/lib64 -lmapi
PG_CPPFLAGS = -I/usr/local/include/monetdb

再來是安裝:
sudo make USE_PGXS=1 install
需要修改 /var/lib/pgsql/data/postgresql.conf,加入下列的設定:
shared_preload_libraries = 'monetdb_fdw'

要注意的是,重新啟動 PostgreSQL 前要確定 monetdb_fdw 可以正確的尋找到 MonetDB MAPI library 的路徑位置, 否則要設定才行,不然 PostgreSQL 會無法正確啟動

下面是一個例子:
export LD_LIBRARY_PATH=/usr/local/lib64:$LD_LIBRARY_PATH
然後重新啟動 PostgreSQL。


下面是使用 psql 來進行是否可以使用的測試:
CREATE EXTENSION monetdb_fdw;
再來是建立 server object:
CREATE SERVER monetdb_server FOREIGN DATA WRAPPER monetdb_fdw;
然後建立與 MonetDB table 對應的 foreign table。假設 MonetDB 有一個 person 的 table, 我們可以對應過來以後,就可以對這個表格進行操作。

下面是一個例子:
CREATE FOREIGN TABLE person (id integer, name varchar(40)) SERVER monetdb_server
OPTIONS (host 'localhost', port '50000', user 'monetdb', passwd 'monetdb', dbname 'demo', table 'person');
然後就可以查詢 MonetDB person 表格:
SELECT * FROM person;

如果要刪除不用的 FDW 並且完整的移除,要先刪除全部的 Foreign table:
DROP FOREIGN TABLE person;
Then drop the monetdb server and extension:
DROP SERVER monetdb_server;
DROP EXTENSION monetdb_fdw;
Then remove monetdb_fdw from shared_preload_libraries in your postgresql.conf:
#shared_preload_libraries = 'monetdb_fdw'
Finally remove the monetdb_fdw files, into the monetdb_fdw development folder, execute:
sudo make USE_PGXS=1 uninstall

MonetDB

MonetDB is an open source column-oriented database management system developed at the Centrum Wiskunde & Informatica (CWI) in the Netherlands. MonetDB is primarily an analytical (OLAP) database, where most operations are SELECT-FROM-WHERE-GROUPBY queries. With that in mind, its transaction management scheme is tuned for reading large chunks of data, rather than writing small chunks of data concurrently at high speed.

MonetDB architecture is represented in three layers, each with its own set of optimizers. The front end is the top layer, providing query interface for SQL, with SciQL and SPARQL interfaces under development. Queries are parsed into domain-specific representations, like relational algebra for SQL, and optimized. The generated logical execution plans are then translated into MonetDB Assembly Language (MAL) instructions, which are passed to the next layer. The middle or back-end layer provides a number of cost-based optimizers for the MAL. The bottom layer is the database kernel, which provides access to the data stored in Binary Association Tables (BATs). Each BAT is a table consisting of an Object-identifier and value columns, representing a single column in the database.

MonetDB internal data representation also relies on the memory addressing ranges of contemporary CPUs using demand paging of memory mapped files, and thus departing from traditional DBMS designs involving complex management of large data stores in limited memory.


安裝後要開始 monetdbd 服務:

sudo service monetdbd start

建立資料庫:

sudo monetdb create demo
sudo monetdb release demo

如果需要刪除舊的資料庫:

sudo monetdb destroy demo

然後使用 mclient 連線:

mclient -u monetdb -d demo

加入一個新的使用者:

CREATE SCHEMA danilo;
CREATE USER "danilo" WITH PASSWORD 'danilo' NAME 'danilo' SCHEMA "danilo";

如果要更新使用者的密碼:

ALTER USER "danilo"  WITH PASSWORD 'danilo';

如果需要自動登入,在家目錄中加入 .monetdb,內容如下:

user=monetdb
password=monetdb
language=sql

那就只要下列的指令就可以登入:

mclient demo

MonetDB Embedded

MonetDB 有提供 Embedded version, 也就是 MonetDB/e, 可以作為 Embedded Database 使用,在一個人單機的資料分析作業上十分有用。

openSUSE 切換輸入法框架

雖然目前 openSUSE 仍沒有切換輸入法框架的工具,但只要設定好 INPUT_METHOD 這個環境變數,就可以覆蓋原先為不同語言所設定的預設值。 例如可以設定 INPUT_METHOD="ibus",下次登入就會使用 ibus 輸入法框架。 而可用的輸入法框架,可以參考 /etc/X11/xim.d(openSUSE Tumbleweed 的位置在 /usr/etc/X11/xim.d)目錄下的設定。

一般而言,這些設定都是在設定 GTK_IM_MODULE, QT_IM_MODULE 與 XMODIFIERS 這幾個環境變數, 並且啟動 input method server。以 fcitx5 為例:

#make sure set these vars before dbus-launch
export LC_CTYPE=$LANG
export XMODIFIERS="@im=fcitx"
export GTK_IM_MODULE=fcitx
export QT_IM_SWITCHER=imsw-multi
export QT_IM_MODULE=fcitx

# FIXME: a little bit of delay is required to avoid race (boo#947576)
(sleep 2; fcitx5 -d) &

# success:
return 0

在 openSUSE leap 42.1 以及之後的環境(包含 openSUSE Tumbleweed), INPUT_METHOD 環境變數可以在 ~/.i18n 中設定。下面是 iBus 的設定範例。

#
# Override system wide input method here
#

export INPUT_METHOD=ibus

然後重新登入就會使用指定的輸入法框架。

ibus 的設定(包含輸入法引擎)使用 gconf 來儲存,可以用來觀察目前有的選項,如果因為某些因素需要透過非設定程式的方式修改, 或者在開發程式的過程中想要刪除自己開發過程中最後不用的設定項目,可以使用 dconf-editor 來修改。 dconf-editor 安裝的方式如下:

sudo zypper in dconf-editor

在 openSUSE Leap 15.5 因為修改 systemd xdg autostart 啟動方式,導致 ibus 在 KDE 無法正確啟動, 按照 Bug 1211977 的暫時解法, 需要修改二個檔案:

/usr/bin/ibus-autostart

# boo#1211977: xdg-autostart-generator cannot launch ibus with -d
if [ "$XDG_CURRENT_DESKTOP" = "KDE" ]; then
    exec ibus-daemon --xim
fi

ibus-daemon --xim -d
exit 0
/etc/xdg/autostart/ibus-autostart.desktop
[Desktop Entry]
Name=IBus
GenericName=IBus Daemon
Comment=Start IBus daemon
Exec=ibus-autostart
Icon=ibus-setup
Terminal=false
Type=Application
StartupNotify=false
NoDisplay=true
#X-GNOME-Autostart-Phase=Applications
#X-GNOME-AutoRestart=false
#X-GNOME-Autostart-Notify=false
X-KDE-autostart-after=panel
X-KDE-StartupNotify=false

如果原本沒有安裝,現在要在 openSUSE Tumbleweed 使用 Fcitx 5 並且安裝一些輸入法(包含酷音輸入法), 使用下列的指令:

sudo zypper install fcitx5 fcitx5-table-extra fcitx5-zhuyin fcitx5-chewing

並且在 ~/.i18n 中設定 :

#
# Override system wide input method here
#

export INPUT_METHOD=fcitx5

openSUSE Tumbleweed 可以使用上述的方式加入 Fcitx 5。

另外,.i18n 這個檔案除了可以用來設定輸入法框架,也可以用來設定 locale。 對於 Fcitx 5 而言,除了 /etc/locale.conf,還可以在設定 INPUT_METHOD 環境變數之前設定 LANG, 即可設定 Fcitx 5 的 locale。 Fcitx5 使用系統設定,而不是桌面系統的設定決定使用何種 locale,如果你二者的設定不同,可以在這裡設定你想要的。

另外,Fcitx 5 有個麻煩的地方,就是目前家目錄設定 .i18n 的方式如果切換到其它輸入法框架,Fcitx 5 會對其它的框架造成干擾。 原因是 Fcitx 5 套件會在 /etc/xdg/autostart 放檔案 org.fcitx.Fcitx5, 即使目前的輸入法框架不是使用 Fcitx 5 一樣會自動啟動,如果有需要使用其它的輸入法框架在檔案 org.fcitx.Fcitx5 加入下列的設定。

Hidden=true

下面的小知識與如何切換輸入法框架無關。如果需要檢查 keyboard event(press and release), 在 X Window 系統下可以使用 xev 指令檢查:

xev -event keyboard

如果是在 Wayland 系統,則可以使用 wev 指令檢查。

2024/04/21

Basic Linux commands

Commands

systemd and sysvinit

systemd 是 Linux 作業系統之下的一套中央化系統及設定管理程式(init),包括有常駐程式、函式庫以及應用軟體, 因為使用了 cgroup 與 fanotify 等 Linux 組件以實現其特性,所以只適用於 Linux。 有不少 Linux distribution 已將預設 init 程式從 SysVinit 改為使用 systemd, 但是目前仍然也有使用 SysVinit 或者是其它 init 程式的 Linux distribution。

Linux kernel 開機後,第一個執行的程式為 /sbin/init,如果使用 systemd 的系統,那麼這個檔案會是指向 systemd 的連結。 如果要檢查目前系統使用的 init 程式:

ps 1

檢查 systemd 是否有在使用:

stat /sbin/init

也可以使用 readlink 檢查:

readlink /sbin/init

如果是使用 systemd 的系統,Linux 可以選擇從 CLI 或者是 GUI 開機,如果要知道目前的選項,使用下面的命令:

sudo systemctl get-default

如果只是要 run-time 改變,改為 CLI:

sudo systemctl isolate multi-user.target

改為 GUI:

sudo systemctl isolate graphical.target

如果是要修改預設值從 CLI 開機:

sudo systemctl set-default multi-user.target

修改預設值從 GUI 開機:

sudo systemctl set-default graphical.target

如果預設為 GUI 環境,使用 ALT + CTRL + F1 ~ F6 可以切換到 console,如果要切回來, 使用指令 w 列出目前登入的使用者與 ttyx 的使用情況,按 ALT + Fx (x=1, 2, 3.., etc) 切回來。


如果是 sysvinit 系統,修改 /etc/inittab 檔案(修改預設值從 CLI 開機,那麼將 5 改為 3):

id:5:initdefault:

Shell

Shell 是使用者與 Linux kernel 溝通的橋樑,使用者輸入命令來執行各種各樣的任務。 目前大多數的 Linux distribution 預設的 Shell 為 GNU Bash

Bash (Bourne Again SHell) 是在 1987 年由布萊恩·福克斯為了 GNU 計劃而編寫。1989年釋出第一個正式版本, 原先是計劃用在 GNU 作業系統上,但能執行於大多數類 Unix 系統的作業系統之上。

下面是我的個人設定,用來設定 Prompt 與一些命令的彩色化設定。

# Prompt
export PS1="\[\e[1;36m\]\u\[\e[0m\]@\h:\[\e[38;5;214m\]\w\[\e[0m\]> "

# Bash setting
export HISTCONTROL=ignoreboth

# alias command
alias ls="ls --color=auto"
alias diff="diff --color"
alias grep="grep --color=auto"

# Colors in Man Pages
export GROFF_NO_SGR=1         # For Konsole and Gnome-terminal
man() {
    LESS_TERMCAP_mb=$'\e[01;31m' \
    LESS_TERMCAP_md=$'\e[01;31m' \
    LESS_TERMCAP_me=$'\e[0m' \
    LESS_TERMCAP_se=$'\e[0m' \
    LESS_TERMCAP_so=$'\e[01;44;33m' \
    LESS_TERMCAP_ue=$'\e[0m' \
    LESS_TERMCAP_us=$'\e[01;32m' \
    command man "$@"
}

bash 可以使用 history 列出之前使用過的命令。
history -c 可以清除記錄的命令。

如果要將工作放入背景執行,在命令的最後加入 & 即可。
如果是要將工作暫停放入背景中,使用 CTRL + Z
承上,那要怎麼知道目前背景有哪些工作? 使用 jobs 可以列出來(-l 還會列出 PID 的號碼,-r:列出正在背景工具的程序, -s:則是列出正在背景中暫停的工作)。
承上,觀察上面的列表,在前面會有代表的號碼。可以使用 fg 或者是 fg %jobnumber 來將背景的工作恢復為前景運行中。
如果需要刪除背景的工作,可以使用 kill,方式為 kill %jobnumber
Bash 也有支援將工作送到背景的指令,使用 bg 或者是 bg %jobnumber

上面描述的都是 bash 產出的子程序,所以 logout 之後背景程序一樣會消失。 如果需要將工作放到系統背景,可以使用 nohup

File system

Linux 採用索引式檔案系統 (indexed allocation),通常會將這兩部份的資料分別存放在不同的區塊,權限與屬性放置到 inode 中, 至於實際資料則放置到 data block 區塊中。 另外,還有一個超級區塊 (superblock) 會記錄整個檔案系統的整體資訊,包括 inode 與 block 的總量、使用量、剩餘量等。

每個 inode 與 block 都有編號,至於這三個資料的意義可以簡略說明如下:

  • superblock:記錄此 filesystem 的整體資訊,包括 inode/block 的總量、使用量、剩餘量, 以及檔案系統的格式與相關資訊等;
  • inode:記錄檔案的屬性,一個檔案佔用一個inode,同時記錄此檔案的資料所在的 block 號碼;
  • block:實際記錄檔案的內容,若檔案太大時,會佔用多個 block 。

由於每個 inode 與 block 都有編號,而每個檔案都會佔用一個 inode ,inode 內則有檔案資料放置的 block 號碼。 因此,如果能夠找到檔案的 inode 的話,那麼自然就會知道這個檔案所放置資料的 block 號碼, 也就能夠讀出該檔案的實際資料。

File Permission

Every file in Unix has the following attributes −

  • Owner permissions − The owner's permissions determine what actions the owner of the file can perform on the file.

  • Group permissions − The group's permissions determine what actions a user, who is a member of the group that a file belongs to, can perform on the file.

  • Other (world) permissions − The permissions for others indicate what action all other users can perform on the file.

The permissions of a file are the first line of defense in the security of a Unix system. The basic building blocks of Unix permissions are the read, write, and execute permissions.

UNIX 系統使用 chmod 修改 permission。
下面是一個使用的例子(要注意 Unix permission 採用八進位計算):

chmod 644 index.html

如果要加上執行權限:

chmod +x index.html

如果要刪除執行權限:

chmod -x index.html

如果要更改擁有者或者是檔案的群組,使用 chown 進行修改。

Superuser

Linux 系統最高權限的管理者帳號為 root(user id 為 0 的帳號),也就是超級使用者(superuser)帳號。 因為 root 帳號權限很高,通常不建議直接使用 root 登入系統進行管理系統的動作。

su 指令可以讓一般使用者取得 root 權限,取得 root 權限的使用者就如同 root 一樣可以對系統進行各種管理動作。 另外,su 除了可以讓一般使用者取得 root 權限之外,也可以取得其他的帳號權限。下面是一個例子:

su danilo

sudo 指令類似 su,也是用來取得 root 或是其他帳號的權限,一般使用者可以在指令前加上 sudo 來以 root 身分執行任何指令。 系統會提示使用者提供 root 密碼。如果驗證成功,便會以 root 身分執行指令。

id -un 指令會列印目前使用者的登入名稱:

sudo id -un

sudo 可以使用 -u 指定要切換的使用者,-g 指定要切換的群組。

sudo 的主要規則組態檔案為 /etc/sudoers(如果是 openSUSE Tumbleweed,那麼使用 /usr//etc/sudoers)。 如果檔案格式錯誤,使用者可能無法順桘進入系統,因此強烈建議使用 visudo 來進行編輯。 visudo 可以防止發生編輯衝突,並會在儲存修改內容之前檢查語法錯誤。

Locale

Locale 設定由三個部分所組成:語言代碼 (Language Code)、地域 (Territory)、編碼 (Encoding)。 Locale 的設定名稱就是由這三個一起組成。

可以執行 locale 查詢目前的設定。

  1. LC_CTYPE: 字元分類及處理方式。
  2. LC_COLLATE: 字元順序與字串比較。
  3. LC_MESSAGES: 程式中用何種語言來顯示訊息。
  4. LC_MONETARY: 貨幣顯式格式。
  5. LC_NUMERIC: 數字顯式格式。
  6. LC_TIME: 日期與時間的顯式格式。
  7. LC_ALL: 一次設定以上所有的類別。
  8. LANG: 也可用來一次設定所有的 locale 環境,優先權低於 LC_ALL。

如果要列出目前支援的語系,使用下列的指令:

locale -a

Linux distribution 各個設定檔案(以及設定檔案位置)可能不同。

語系設定可以分為二個部份,系統預設的語系與桌面環境的設定。openSUSE 系統的設定在 /etc/locale.conf
如果在 KDE 環境下,KDE 會參考家目錄下的 .config/plasma-localerc 設定,因此桌面環境的設定可以不同於系統的設定。

如果要修改系統語系,可以嘗試下列的指令:

localectl set-locale LANG=en_US.utf8

如果想使用修改設定檔的方式,下面是修改 /etc/locale.conf 設定主語系 LANG 變數的例子:

LANG=zh_TW.UTF-8

系統所採用的 locale 設定優先性是 LC_ 環境變數高於 LANG 環境變數。LC_ALL 的優先性則是所有類別中最高的, 不過一般不建議設定 LC_ALL。

How long the system is up

使用 uptime 可以列出系統已經運作的時間。

Get the machine architecture

使用 arch 或者是 uname -m 可以列出系統的硬體架構。

Get the name of the host

使用 hostname 或者是 uname -n 可以列出目前的主機名稱。

Print the user name

使用 whoami 可以取得目前的使用者名稱。

List Current Logged In Users

使用 w 或者是 who command。

Date

使用 date 可以列出目前的時間。

如果只是要印出日期,可以使用:

date +%F

Calendar

使用 cal 可以列出這個月的月曆。

Output strings

如果需要顯示訊息,可以使用 echo
如果需要輸出格式化的訊息,可以使用 printf

printf "%s is %d years old.\n" Orange 19

Display result and output to file

如果需要顯示程式的結果並且儲存到檔案,可以使用 tee。

下面是一個例子:

ls -al | tee result

如果你只想要將結果儲存到檔案但是不顯示到 stdout,

下面是一個例子:

ls -al | tee result > /dev/null

Display file content

如果需要顯示檔案的內容,可以使用 cat。

下面是一個例子:

cat result

Display binary file content

如果使用 cat 觀察一些非文字檔案會顯示亂碼。 這時候可以嘗試使用 od 這個命令來顯示非文字檔檔案的內容,有以下的選項可以使用:

  • -A:選擇以何種進位表示位址偏移,-Ax 為十六進位,-An 表示不顯示位址偏移。
  • -a:具名字元,例如換行字元顯示為 nl。
  • -b:以八進制格式顯示。
  • -c:以 ASCII 字符格式顯示,換行字元會顯示為 \n。
  • -d:以有符號十進制格式顯示。
  • -x:以十六進制格式顯示。

文字檔案換行符號轉換

Linux/Unix-like 系列的作業系統文字檔案的換行字元為 \n; 與 Dos (以及 Windows 平台)的換行字元為 \r\n 不同。

可以使用 dos2unixunix2dos 嘗試換行符號轉換。

語系編碼轉換

可以使用 iconv 嘗試編碼轉換。

iconv -f big5 -t utf8 input_file -o output_file

Count the number of lines, words, and bytes in the file

如果需要計算檔案的行數,字數或者是大小,可以使用 wc。

其中 -l 為計算行數, -w 為計算 word 數, -c 為計算檔案的全部 bytes 數。

下面就是透過 ls 與計算行數的方式來算出目前目錄下的檔案與目錄數目:

ls -al | wc -l

Clears the terminal screen

如果需要清除 termial screen 的內容,可以使用 clear。
或者也可以使用 Ctrl + L 的組合鍵。

Copy and remove files

使用 cp 可以複製檔案或者是目錄。cp -r 可以遞迴地複製子目錄下的檔案。
使用 rm 可以刪除檔案或者是目錄。如果要強制刪除一個目錄以及目錄下的檔案,使用 rm -rf
如果要搬移目錄或者是檔案,使用 mv 這個命令。

建立連結檔

使用 ln 可以建立一個連結檔案。
所謂的硬連結就是使用相同 inode 的連結檔案,ln 指令預設就是建立硬連結(可以使用 ls -i 檢查)。
軟連結(符號連結)則是靠著絕對路徑或相對路徑來指向目標檔案的連結檔,若要使用 ln 指令建立軟連結,可以加上 -s 參數。

Create and remove directory

使用 mkdir 可以建立目錄。
使用 rmdir 可以刪除目錄。

Current working directory

使用 pwd 可以取得目前的目錄路徑。

grep

如果要搜尋檔案的內容,可以使用 grep,如果列出在哪一列 (-n) 以及搜尋各個目錄下的各個檔案 (-r),下面是一個範例:

grep -rn "tcl" .

find

如果要搜尋某些檔案以後殺掉,可以用:

find . -name "FILE-TO-FIND" -exec rm -rf {} \;

或者針對檔案的寫法:

find . -type f -name "FILE-TO-FIND" -exec rm -f {} \; 

如果要找出哪個 c 檔案中有 main 函式並且印出,則可以使用:

find . -type f -name *.c -print -exec grep main {} \;

head and tail

如果沒有指定參數,head 會列出一個檔案前十列的內容,而 tail 會列出一個檔案最後十列的內容。

下面是列出 Aapche Http Server error_log 最後十行的例子。

sudo tail /var/log/apache2/error_log

Compare text files

可以使用 diff 來比較文字檔案間的不同,我通常會使用 -Naur 參數取得更詳細的比較內容。

Compare binary files

可以嘗試使用下列的指令列出差異:

cmp -l file1.bin file2.bin | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}'

(這時候不是以 0,而是以 1 開始計算位置,和一般的慣例略有差異)

tar

tar 可以將多個目錄或檔案打包成一個檔案,而且可以透過 gzip/bzip2 的支援,對打包後的檔案進行壓縮。

壓 縮:tar -jcv -f filename.tar.bz2 要被壓縮的檔案或目錄名稱
查 詢:tar -jtv -f filename.tar.bz2
解壓縮:tar -jxv -f filename.tar.bz2 -C 欲解壓縮的目錄
選項與參數:
-c :建立打包檔案,可搭配 -v 來察看過程中被打包的檔名(filename)
-t :察看打包檔案的內容含有哪些檔名,重點在察看『檔名』就是了;
-x :解打包或解壓縮的功能,可以搭配 -C (大寫) 在特定目錄解開
特別留意的是, -c, -t, -x 不可同時出現在一串指令列中。
-J :透過 xz 的支援進行壓縮/解壓縮:此時檔名最好為 *.tar.xz
-j :透過 bzip2 的支援進行壓縮/解壓縮:此時檔名最好為 *.tar.bz2
-z :透過 gzip 的支援進行壓縮/解壓縮:此時檔名最好為 *.tar.gz
-v :在壓縮/解壓縮的過程中,將正在處理的檔名顯示出來!
-f filename:-f 後面要立刻接要被處理的檔名!建議 -f 單獨寫一個選項囉!
-C 目錄 :這個選項用在解壓縮,若要在特定目錄解壓縮,可以使用這個選項。

Report the file system disk space usage

可以使用 df 這個命令。

df -h

List block devices

可以使用 lsblk 這個命令來列出 block devices。

如果要確認,可以使用 fdisk 來確認:

sudo fdisk -l

List systemd services

如果需要列出目前 systemd 正在執行的 services,可以使用下列的指令:

pstree

工作管理員程式

top (table of processes)是一個工作管理員程式,用於監看目前所有程式的執行狀況, 顯示有關 CPU 和記憶體利用率的資訊。在程式執行後,按 q 可以離開程式。

檢查處理程序狀態

使用者可以使用 ps 指令來尋找正在執行中的處理程序,並顯示這些處理程序的相關資訊。

顯示所有在系統上執行的處理程序:

ps -ef

也可以使用下列的參數 (-aux) 顯示所有在系統上執行的處理程序:

ps -aux

在 Linux 中掛載 ISO 檔案

如果需要在 Linux 中掛載 ISO 檔案來存取資料的話,下面是如何掛載的範例:

# cd /mnt
# mkdir cdrom
# mount /home/danilo/Desktop/openSUSE-11.3-DVD-i586.iso /mnt/cdrom -o loop

在 Linux 中建立 ISO 檔案

使用 mkisofs 建立。下面是一個範例:

mkisofs -o home.iso /home/danilo

Create usb boot disk

  • 下載 live image
  • 使用下列的 command:
dd if=image.iso of=/dev/sdb bs=4M;sync

其中 image.iso 和 /dev/sdb 要換成符合自己環境的參數。

Increase The Maximum Number Of Open Files / File Descriptors (FD)

Use the following command command to display maximum number of open file descriptors:

cat /proc/sys/fs/file-max

The hard limit is the ceiling for the soft limit. The soft limit is what is actually enforced for a session or process. This allows the administrator (or user) to set the hard limit to the maximum usage they wish to allow. Other users and processes can then use the soft limit to self-limit their resource usage to even lower levels if they so desire.

針對各個使用者的設定,如果要查詢目前的情況:

ulimit -Sn
ulimit -Hn

要變更設定,可以編輯 /etc/security/limits.conf,下面是設定範例:

danilo soft nofile 80920
danilo hard nofile 102400

要重開機以後才會生效。

參考資料

2024/04/17

OpenCL

簡介

OpenCL(Open Computing Language,開放計算語言)是一個為異構平台編寫程式的框架, 此異構平台可由 CPU、GPU、DSP、FPGA 或其他類型的處理器與硬體加速器所組成。 Portable Computing Language (PoCL) 則是 OpenCL 的一個自由軟體實作, 可以在機器上沒有 GPU 的情況下使用 OpenCL API 進行運算。

OpenCL 包括一組 API 和一個程式語言。基本的原理是程式透過 OpenCL API 取得 OpenCL 裝置(例如顯示晶片)的相關資料, 並將要在裝置上執行的程式(使用 OpenCL 程式語言撰寫)編繹成適當的格式以後在裝置上執行。

An OpenCL application is split into host code and device kernel code. Execution of an OpenCL program occurs in two parts: kernelsthat execute on one or more OpenCL devices and a host program that executes on the host.

The most commonly used language for programming the kernels that are compiled and executed across the available parallel processors is called OpenCL C. OpenCL C is based on C99 and is defined as part of the OpenCL specification.

The core of the OpenCL execution model is defined by how the kernels execute. OpenCL regards a kernel program as the basic unit of executable code (similar to a C function). Kernels can execute with data or task-parallelism. An OpenCL program is a collection of kernels and functions (similar to dynamic library with run-time linking).

An OpenCL command queue is used by the host application to send kernels and data transfer functions to a device for execution. By enqueueing commands into a command queue, kernels and data transfer functions may execute asynchronously and in parallel with application host code.

The kernels and functions in a command queue can be executed in-order or out-of-order. A compute device may have multiple command queues.


A complete sequence for executing an OpenCL program is:

  1. Query for available OpenCL platforms and devices
  2. Create a context for one or more OpenCL devices in a platform
  3. Create and build programs for OpenCL devices in the context
  4. Select kernels to execute from the programs
  5. Create memory objects for kernels to operate on
  6. Create command queues to execute commands on an OpenCL device
  7. Enqueue data transfer commands into the memory objects, if needed
  8. Enqueue kernels into the command queue for execution
  9. Enqueue commands to transfer data back to the host, if needed

A host is connected to one or more OpenCL compute devices. Each compute device is collection of one or more compute units where each compute unit is composed of one or more processing elements. Processing elements execute code with SIMD (Single Instruction Multiple Data) or SPMD (Single Program Multiple Data) parallelism.


For example, a compute device could be a GPU. Compute units would then correspond to the streaming multiprocessors (SMs) inside the GPU, and processing elements correspond to individual streaming processors (SPs) inside each SM. Processors typically group processing elements into compute units for implementation efficiency through sharing instruction dispatch and memory resources, and increasing local inter-processor communication.

OpenCL's clEnqueueNDRangeKernel command enables a single kernel program to be initiated to operate in parallel across an N-dimensional data structure. Using a two-dimensional image as a example, the size of the image would be the NDRange, and each pixel is called a work-item that a copy of kernel running on a single processing element will operate on.

As we saw in the Platform Model section above, it is common for processors to group processing elements into compute units for execution efficiency. Therefore, when using the clEnqueueNDRangeKernel command, the program specifies a work-group size that represents groups of individual work-items in an NDRange that can be accommodated on a compute unit. Work-items in the same work-group are able to share local memory, synchronize more easily using work-group barriers, and cooperate more efficiently using work-group functions such as async_work_group_copy that are not available between work-items in separate work-groups.


OpenCL has a hierarchy of memory types:

  • Host memory - available to the host CPU
  • Global/Constant memory - available to all compute units in a compute device
  • Local memory - available to all the processing elements in a compute unit
  • Private memory - available to a single processing element

OpenCL memory management is explicit. None of the above memories are automatically synchronized and so the application explicitly moves data between memory types as needed.


在 openSUSE Tumbleweed 上安裝:

sudo zypper in ocl-icd-devel pocl-devel opencl-headers clinfo

OpenCL Installable Client Driver (ICD) allows multiple OpenCL implementations to co-exist; also, it allows applications to select between these implementations at runtime.

Use the clGetPlatformIDs() and clGetPlatformInfo() functions to see the list of available OpenCL implementations, and select the one that is best for your requirements.

執行 clinfo 觀察目前的 OpenCL device 資訊。

下面的程式使用 clGetPlatformIDs 函式取得目前可用的 platform 數目 (編譯指令:gcc test.c `pkg-config --libs --cflags OpenCL`):

#include <stdio.h>

#ifdef __APPLE__
#include <OpenCL/opencl.h>
#else
#include <CL/cl.h>
#endif

int main( void ) {
    // OpenCL related declarations
    cl_int err;
    cl_uint num;

    err = clGetPlatformIDs( 0, NULL, &num );
    printf("%d\n", num);

}

下面是另外一個範例:

#include <stdlib.h>
#include <stdio.h>

#ifdef __APPLE__
#include <OpenCL/opencl.h>
#else
#include <CL/cl.h>
#endif

const char *kernel_code =
    "__kernel void vector_add(__global const int *A, __global const int *B, __global int *C) {"
    "    int i = get_global_id(0);"
    "    C[i] = A[i] + B[i];"
    "}";

int main( void ) {
    // OpenCL related declarations
    cl_int err;
    cl_platform_id platform;
    cl_device_id device;
    cl_context_properties props[3] = { CL_CONTEXT_PLATFORM, 0, 0 };
    cl_context ctx;
    cl_program program;
    cl_command_queue queue;
    cl_kernel kernel;
    int i;

    //
    const size_t N = 1024; // vector size
    size_t global_item_size = N; // Process the entire lists
    size_t local_item_size = 64; // Divide work items into groups of 64

    int *A, *B, *C;
    A = (int*) malloc(N * sizeof(*A));
    B = (int*) malloc(N * sizeof(*B));
    C = (int*) malloc(N * sizeof(*C));
    for (i=0; i<N; i++) {
        A[i] = i;
        B[i] = i + 1;
    }
    cl_mem d_A, d_B, d_C;

    /* Setup OpenCL environment. */
    err = clGetPlatformIDs( 1, &platform, NULL );
    err = clGetDeviceIDs( platform, CL_DEVICE_TYPE_DEFAULT, 1, &device, NULL );

    props[1] = (cl_context_properties)platform;
    ctx = clCreateContext( props, 1, &device, NULL, NULL, &err );
    queue = clCreateCommandQueueWithProperties( ctx, device, 0, &err );
    program = clCreateProgramWithSource(ctx, 1, (const char **) &kernel_code, NULL, &err);
    err = clBuildProgram(program, 0, NULL, NULL, NULL, NULL);
    kernel = clCreateKernel(program, "vector_add", &err);

    // initialize buffer with data
    d_A = clCreateBuffer( ctx, CL_MEM_READ_ONLY, N*sizeof(*A), NULL, &err );
    d_B = clCreateBuffer( ctx, CL_MEM_READ_ONLY, N*sizeof(*B), NULL, &err );
    d_C = clCreateBuffer( ctx, CL_MEM_WRITE_ONLY, N*sizeof(*C), NULL, &err );

    err = clEnqueueWriteBuffer( queue, d_A, CL_TRUE, 0, N*sizeof(*A), A, 0, NULL, NULL );
    err = clEnqueueWriteBuffer( queue, d_B, CL_TRUE, 0, N*sizeof(*B), B, 0, NULL, NULL );

    err = clSetKernelArg(kernel, 0, sizeof(cl_mem), (void *)&d_A);
    err = clSetKernelArg(kernel, 1, sizeof(cl_mem), (void *)&d_B);
    err = clSetKernelArg(kernel, 2, sizeof(cl_mem), (void *)&d_C);

    err = clEnqueueNDRangeKernel(queue, kernel, 1, NULL,
            &global_item_size, &local_item_size, 0, NULL, NULL);

    err = clFinish(queue);

    err = clEnqueueReadBuffer( queue, d_C, CL_TRUE, 0, N*sizeof(*C), C, 0, NULL, NULL );
    err = clFinish(queue);

    for(i = 0; i < N; i++)
        printf("%d + %d = %d\n", A[i], B[i], C[i]);

    err = clFlush(queue);
    err = clFinish(queue);

    /* Release OpenCL memory objects. */
    clReleaseMemObject( d_A );
    clReleaseMemObject( d_B );
    clReleaseMemObject( d_C );
    free(A);
    free(B);
    free(C);
    clReleaseKernel( kernel );
    clReleaseProgram( program );
    clReleaseCommandQueue( queue );
    clReleaseContext( ctx );

    return 0;
}

下面的程式是使用 stb_image 讀取圖檔,測試 image object 功能的程式。

#include <stdio.h>
#include <stdlib.h>

#ifdef __APPLE__
#include <OpenCL/opencl.h>
#else
#include <CL/cl.h>
#endif

#define STB_IMAGE_IMPLEMENTATION
#include "stb_image.h"
#define STB_IMAGE_WRITE_IMPLEMENTATION
#include "stb_image_write.h"

const char *kernel_code =
    "__kernel void PixelAccess(__read_only image2d_t imageIn,__write_only image2d_t imageOut)"
    "{"
    "  sampler_t srcSampler = CLK_NORMALIZED_COORDS_FALSE | "
    "    CLK_ADDRESS_CLAMP_TO_EDGE |"
    "    CLK_FILTER_NEAREST;"
    "  int2 imageCoord = (int2) (get_global_id(0), get_global_id(1));"
    "  uint4 pixel = read_imageui(imageIn, srcSampler, imageCoord);"
    "  write_imageui (imageOut, imageCoord, pixel);"
    "}";


int main( int argc, char *argv[] ) {
    // OpenCL related declarations
    cl_int err;
    cl_platform_id platform;
    cl_device_id device;
    cl_context_properties props[3] = { CL_CONTEXT_PLATFORM, 0, 0 };
    cl_context ctx;
    cl_program program;
    cl_command_queue queue;
    cl_kernel kernel;
    int i;
    int width = 0, height = 0, channel = 0;
    unsigned char *data = NULL;
    const char *filename = NULL;

    if (argc < 2) {
        printf("Please give a filename.\n");
        return 0;
    } else if (argc == 2) {
        filename =  argv[1];
    }

    // Load image data
    data = stbi_load(filename, &width, &height, &channel, 0);
    if(!data) {
        fprintf(stderr, "Open image failed.\n");
        return 0;
    }

    cl_mem myClImageInBuffer;
    cl_mem myClImageOutBuffer;
    cl_sampler sampler;

    cl_image_format format;
    if (channel==4) {
        format.image_channel_order = CL_RGBA;
    } else {
        printf("Not supported image format.\n");
        return 0;
    }
    format.image_channel_data_type = CL_UNSIGNED_INT8;

    err = clGetPlatformIDs( 1, &platform, NULL );
    err = clGetDeviceIDs( platform, CL_DEVICE_TYPE_DEFAULT, 1, &device, NULL );

    cl_bool imageSupport = CL_FALSE;
    clGetDeviceInfo(device, CL_DEVICE_IMAGE_SUPPORT, sizeof(cl_bool),
                    &imageSupport, NULL);

    if (imageSupport != CL_TRUE)
    {
        printf("OpenCL device does not support images.\n");
        return 1;
    }

    props[1] = (cl_context_properties)platform;
    ctx = clCreateContext( props, 1, &device, NULL, NULL, &err );
    queue = clCreateCommandQueueWithProperties( ctx, device, 0, &err );
    program = clCreateProgramWithSource(ctx, 1, (const char **) &kernel_code, NULL, &err);
    err = clBuildProgram(program, 0, NULL, NULL, NULL, NULL);
    kernel = clCreateKernel(program, "PixelAccess", &err);

    //
    // For OpenCL 1.2
    cl_image_desc clImageDesc;
    clImageDesc.image_type = CL_MEM_OBJECT_IMAGE2D;
    clImageDesc.image_width = width;
    clImageDesc.image_height = height;
    clImageDesc.image_row_pitch = 0;
    clImageDesc.image_slice_pitch = 0;
    clImageDesc.num_mip_levels = 0;
    clImageDesc.num_samples = 0;
    clImageDesc.buffer = NULL;

    myClImageInBuffer = clCreateImage(ctx, CL_MEM_READ_ONLY,
                            &format, &clImageDesc, NULL, &err);
    if (!myClImageInBuffer) {
        printf("Create myClImageInBuffer failed.\n");
    }

    myClImageOutBuffer = clCreateImage(ctx, CL_MEM_READ_WRITE,
                            &format, &clImageDesc, NULL, &err);
    if (!myClImageOutBuffer) {
        printf("Create myClImageOutBuffer failed.\n");
    }

    size_t origin[3] = {0, 0, 0};
    size_t region[3] = {width, height, 1};

    err = clEnqueueWriteImage(
            queue, myClImageInBuffer,
            CL_TRUE, origin, region,
            0,
            0, data,
            0, NULL, NULL);

    err = clSetKernelArg(kernel, 0, sizeof(cl_mem), (void *) &myClImageInBuffer);
    err = clSetKernelArg(kernel, 1, sizeof(cl_mem), (void *) &myClImageOutBuffer);

    size_t global_item_size[2] = {width, height};
    size_t local_item_size[2] = {1, 1};

    err = clEnqueueNDRangeKernel(queue, kernel, 2, NULL,
            global_item_size, local_item_size, 0, NULL, NULL);

    err = clFinish(queue);

    unsigned char *data2 = NULL;
    data2 = (unsigned char *) malloc(width * height  * channel);
    err = clEnqueueReadImage( queue,
          myClImageOutBuffer, CL_TRUE,
          origin, region,
          width * sizeof(unsigned char) * 4,
          0, data2,
          0, NULL, NULL);

    err = clFinish(queue);

    stbi_write_png("output.png", width, height, channel, data2, 0);

    free(data2);
    stbi_image_free(data);

    clReleaseMemObject( myClImageInBuffer );
    clReleaseMemObject( myClImageOutBuffer );
    clReleaseKernel( kernel );
    clReleaseProgram( program );
    clReleaseCommandQueue( queue );
    clReleaseContext( ctx );

    return 0;
}

參考連結

Audio library

簡介

如果是輕量的函式庫,可以分為二個部份:

  • 讀取 audio file 並且轉為 raw audio data
  • audio I/O

第一個部份,有二個函式庫可以考慮:

第二個部份,有下面的函式庫可以使用:

如果需要查詢與修改 audio 檔案的 meta-data,可以考慮 TagLib, 因為 TagLib 支援十分廣泛的音訊檔案格式。

libao

LibAO is developed under Xiph umbrella. Xiph is the organization who brought you Ogg/Vorbis, FLAC, Theora and currently they are hammering together next generation video codec Daala.

Opus-audio codec standard is also Xiph project. LibAO rised from Xiph’s need multi-platform audio output library for Vorbis-audio codec.

The libao API makes a distinction between drivers and devices. A driver is a set of functions that allow audio to be played on a particular platform (i.e. Solaris, ESD, etc.). A device is a particular output target that uses a driver. In addition, libao distinguishes between live output drivers, which write audio to playback devices (sound cards, etc.), and file output drivers, which write audio to disk in a particular format.

To use libao in your program, you need to follow these steps:

  • Include the <ao/ao.h> header into your program.
  • Call ao_initialize() to initialize the library. This loads the plugins from disk, reads the libao configuration files, and identifies an appropriate default output driver if none is specified in the configuration files.
  • Call ao_default_driver_id() to get the ID number of the default output driver. This may not be successful if no audio hardware is available, it is in use, or is not in the "standard" configuration. If you want to specify a particular output driver, you may call ao_driver_id() with a string corresponding to the short name of the device (i.e. "oss", "wav", etc.) instead.
  • If you are using the default device, no extra options are needed. However, if you wish to to pass special options to the driver, you will need to:
    • Create an option list pointer of type (ao_option *) and initialize it to NULL.
    • Through successive calls to ao_append_option(), add any driver-specific options you need. Note that the options take the form of key/value pairs where supported keys are listed in the driver documentation.
  • Call ao_open_live() and save the returned device pointer. If you are using a file output driver, you will need to call ao_open_file() instead.
  • Call ao_play() to output each block of audio.
  • Call ao_close() to close the device. Note that this will automatically free the memory that was allocated for the device. Do not attempt to free the device pointer yourself!
  • Call ao_shutdown() to close the library.
下面是一個使用 libao 與 libsndfile 的範例:
#include <ao/ao.h>
#include <signal.h>
#include <sndfile.h>

#define BUFFER_SIZE 8192

int cancel_playback;

void on_cancel_playback(int sig) {
    if (sig != SIGINT) {
        return;
    }

    cancel_playback = 1;
    exit(0);
}

static void clean(ao_device *device, SNDFILE *file) {
    ao_close(device);
    sf_close(file);
    ao_shutdown();
}

int play(const char *filename) {
    ao_device *device;
    ao_sample_format format;
    SF_INFO sfinfo;

    int default_driver;

    short *buffer;

    signal(SIGINT, on_cancel_playback);

    SNDFILE *file = sf_open(filename, SFM_READ, &sfinfo);
    if (file == NULL)
        return -1;

    printf("Samples: %d\n", sfinfo.frames);
    printf("Sample rate: %d\n", sfinfo.samplerate);
    printf("Channels: %d\n", sfinfo.channels);

    ao_initialize();

    default_driver = ao_default_driver_id();

    switch (sfinfo.format & SF_FORMAT_SUBMASK) {
    case SF_FORMAT_PCM_16:
        format.bits = 16;
        break;
    case SF_FORMAT_PCM_24:
        format.bits = 24;
        break;
    case SF_FORMAT_PCM_32:
        format.bits = 32;
        break;
    case SF_FORMAT_PCM_S8:
        format.bits = 8;
        break;
    case SF_FORMAT_PCM_U8:
        format.bits = 8;
        break;
    default:
        format.bits = 16;
        break;
    }

    format.channels = sfinfo.channels;
    format.rate = sfinfo.samplerate;
    format.byte_format = AO_FMT_NATIVE;
    format.matrix = 0;

    device = ao_open_live(default_driver, &format, NULL);

    if (device == NULL) {
        fprintf(stderr, "Error opening device.\n");
        return 1;
    }

    buffer = calloc(BUFFER_SIZE, sizeof(short));

    while (1) {
        int read = sf_read_short(file, buffer, BUFFER_SIZE);

        if (read <= 0) {
            break;
        }

        if (ao_play(device, (char *)buffer, (uint_32)(read * sizeof(short))) ==
            0) {
            printf("ao_play: failed.\n");
            clean(device, file);
            break;
        }

        if (cancel_playback) {
            clean(device, file);
            break;
        }
    }

    clean(device, file);

    return 0;
}

int main(int argc, char *argv[]) {
    if (argc > 1) {
        play(argv[1]);
    }
}
編譯方式:
gcc output.c -lsndfile -lao -o output

再來是 libao 與 libmpg123 的例子:
#include <ao/ao.h>
#include <mpg123.h>

#define BITS 8

int main(int argc, char *argv[]) {
    mpg123_handle *mh;
    unsigned char *buffer;
    size_t buffer_size;
    size_t done;
    int err;

    int driver;
    ao_device *dev;

    ao_sample_format format;
    int channels, encoding;
    long rate;

    if (argc < 2)
        exit(0);

    /* initializations */
    ao_initialize();
    driver = ao_default_driver_id();
    mpg123_init();
    mh = mpg123_new(NULL, &err);
    buffer_size = mpg123_outblock(mh);
    buffer = (unsigned char *)malloc(buffer_size * sizeof(unsigned char));

    /* open the file and get the decoding format */
    mpg123_open(mh, argv[1]);
    mpg123_getformat(mh, &rate, &channels, &encoding);

    /* set the output format and open the output device */
    format.bits = mpg123_encsize(encoding) * BITS;
    format.rate = rate;
    format.channels = channels;
    format.byte_format = AO_FMT_NATIVE;
    format.matrix = 0;
    dev = ao_open_live(driver, &format, NULL);

    /* decode and play */
    while (mpg123_read(mh, buffer, buffer_size, &done) == MPG123_OK)
        ao_play(dev, buffer, done);

    /* clean up */
    free(buffer);
    ao_close(dev);
    mpg123_close(mh);
    mpg123_delete(mh);
    mpg123_exit();
    ao_shutdown();

    return 0;
}

OpenAL

OpenAL (Open Audio Library) is a cross-platform audio application programming interface (API). It is designed for efficient rendering of multichannel three-dimensional positional audio, for creation of a virtual 3D world of sound. Its API style and conventions deliberately resemble those of OpenGL.

The application programmer can specify the location, the speed and the direction of the sources of sounds and of the listener.

Objects

Since OpenAL is about audio, it also introduces new concepts, in particular:
  • the listener object
  • the source object
  • the buffer object

Each of these different objects have properties which can be set with their respective API (al*Object*).

  • context: 要播放聲音的地方,可以想成OpenGL裡面的Window
  • listener: OpenAL 支援3D音效,所以在這裡設定聽者的資料
  • sources: 聲源的資訊
  • buffer: 負責聲源的內容

The very first thing to do is to open a handle to a device. This is done like this:

ALCdevice *device;

device = alcOpenDevice(NULL);
if (!device)
        // handle errors

Prior to attempting an enumeration, Open AL provides an extension querying mechanism which allows you to know whether the runtime Open AL implementation supports a specific extension. In our case, we want to check whether Open AL supports enumerating devices:

ALboolean enumeration;

enumeration = alcIsExtensionPresent(NULL, "ALC_ENUMERATION_EXT");
if (enumeration == AL_FALSE)
        // enumeration not supported
else
        // enumeration supported

If the enumeration extension is supported, we can procede with listing the audio devices. If the enumeration is not supported listing audio devices only returns the default device, which is expected in order not to break any application.

static void list_audio_devices(const ALCchar *devices)
{
        const ALCchar *device = devices, *next = devices + 1;
        size_t len = 0;

        fprintf(stdout, "Devices list:\n");
        fprintf(stdout, "----------\n");
        while (device && *device != '\0' && next && *next != '\0') {
                fprintf(stdout, "%s\n", device);
                len = strlen(device);
                device += (len + 1);
                next += (len + 2);
        }
        fprintf(stdout, "----------\n");
}

list_audio_devices(alcGetString(NULL, ALC_DEVICE_SPECIFIER));

Passsing NULL to alcGetString() indicates that we do not want the device specifier of a particular device, but all of them.


In order to render an audio scene, we need to create and initialize a context for this. We do this by the following calls:

ALCcontext *context;

context = alcCreateContext(device, NULL);
if (!alcMakeContextCurrent(context))
        // failed to make context current
// test for errors here using alGetError();

There is nothing specific for our context, so NULL is specified as argument.

Since there is a default listener, we do not need to explicitely create one because it is already present in our scene. If we want to define some of our listener properties however, we can proceed like this:

ALfloat listenerOri[] = { 0.0f, 0.0f, 1.0f, 0.0f, 1.0f, 0.0f };

alListener3f(AL_POSITION, 0, 0, 1.0f);
// check for errors
alListener3f(AL_VELOCITY, 0, 0, 0);
// check for errors
alListenerfv(AL_ORIENTATION, listenerOri);
// check for errors

In order to playback audio, we must create an audio source objet, this source is actually the "origin" of the audio sound. And as such must be defined in the audio scene. If you combine audio with graphics, most likely quite a lot of your graphics objects will also include an audio source object.

Note that you hold a reference (id) to a source object, you don’t manipulate the source object directly.

ALuint source;


alGenSources((ALuint)1, &source);
// check for errors

alSourcef(source, AL_PITCH, 1);
// check for errors
alSourcef(source, AL_GAIN, 1);
// check for errors
alSource3f(source, AL_POSITION, 0, 0, 0);
// check for errors
alSource3f(source, AL_VELOCITY, 0, 0, 0);
// check for errors
alSourcei(source, AL_LOOPING, AL_FALSE);
// check for errros

The buffer object is the object actually holding the raw audio stream, alone a buffer does not do much but occupying memory, so we will see later on what to do with it. Just like sources, we hold a reference to the buffer object.

ALuint buffer;

alGenBuffers((ALuint)1, &buffer);
// check for errors

下面就是載入 Audio raw data 以後的使用範例:

static inline ALenum to_al_format(short channels, short samples)
{
        bool stereo = (channels > 1);

        switch (samples) {
        case 16:
                if (stereo)
                        return AL_FORMAT_STEREO16;
                else
                        return AL_FORMAT_MONO16;
        case 8:
                if (stereo)
                        return AL_FORMAT_STEREO8;
                else
                        return AL_FORMAT_MONO8;
        default:
                return -1;
        }
}

alBufferData(buffer, to_al_format(wave->channels, wave->bitsPerSample),
                bufferData, wave->dataSize, wave->sampleRate);
// check for errors

In order to actually output something to the playback device, we need to bind the source with its buffer. Obviously you can bind the same buffer to several sources and mix different buffers to the same source. Binding is done like this:

alSourcei(source, AL_BUFFER, buffer);
// check for errors

We now have everything ready to start playing our source.

alSourcePlay(source);
// check for errors

alGetSourcei(source, AL_SOURCE_STATE, &source_state);
// check for errors
while (source_state == AL_PLAYING) {
        alGetSourcei(source, AL_SOURCE_STATE, &source_state);
        // check for errors
}

Obviously each and every single object "generated" must be freed, the following does this for us:

alDeleteSources(1, &source);
alDeleteBuffers(1, &buffer);
device = alcGetContextsDevice(context);
alcMakeContextCurrent(NULL);
alcDestroyContext(context);
alcCloseDevice(device);

問題與解法

Fix jack server is not running or cannot be started message (for openal-soft)

如果你在 OpenSUSE Leap 15.0 使用 OpenAL 時發現會出現 jack server is not running or cannot be started 的訊息, 這是因為 openal-soft 支援 jack,所以會嘗試進行初始化的動作。

解決的方法是在 /etc/openal 下新增加 alsoft.conf,並且加入下面的內容:
[general]
drivers = -jack,

Libao debug message

在 OpenSUSE Tumbleweed 中預設的 driver 是 PulseAudio,但是仍然會嘗試載入 ALSA driver 並且印出錯誤訊息。 如果遇到這個情況想要停止印出錯誤訊息,修改 /etc/libao.conf,加入一行 quiet

default_driver=pulse
quiet

參考連結