I Can Do I.T.

[AWS] How to install awscli(AWS Command Line Interface) version 1 on MacOS

Tue, 03 Mar 2020 00:00:00 +0000

Goal

Install awscli version 1 on MacOS

Introduce

What is AWS CLI

The AWS Command Line Interface (AWS CLI) is an open source tool that enables you to interact with AWS services using commands in your command-line shell.

The difference between version 1 and version 2

Version 2.x

The current, generally available release of the AWS CLI that is intended for use in production environments.
It does include some “breaking” changes from version 1 that might require you to change your scripts so that they continue to operate as you expect.

Version 1.x

The previous version of the AWS CLI that is available for backwards compatiblity.
On January 10th, 2020, AWS CLI version 1, which requires a separate installation of Python to operate, stopped supporting Python versions 2.6 and 3.3. All builds of AWS CLI version 1 released after January 10th, 2020, starting with version 1.17, require Python 2.7, Python 3.4, or a later version to successfully use the AWS CLI.

AWS CLI version 2 and Windows MSI installer version of AWS CLI version 1 are not affected by this change.

Practice

1. Download awscli installer file

$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip"

2. Unzip installer file

$ unzip awscii-bundle.zip

3. Execute installer

$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

4. Check the version of awscli

$ awscli --version

References

[Hive] New features on Hive 3.0

Wed, 12 Feb 2020 00:00:00 +0000

Goal

The new features of Apache Hive 3.0

Practice

What’s new in this release: Apache Hive

Apache Hive 3.0.0 was released on 21 May 2018 and the recent version is 3.1.2(based on 18 Feb 2020).
One of the noticeable features is that Hive 3.0 supports seamless ACID(Atomicity, Consistency, Isolation, Durability) and you have to consider serveral thing about the transactional tables created by a Hive version prior to Hive 3.
Futhermore, several SQL features and useful connectors were added.
And also, it enhance the performance through improved LLAP and Apache Spark.

Hive3 New Features

ACID v2

Originally Hive supported write only by adding partitions or loading new file into existing partitions because HDFS Hive already has not allowed file replaced or edited.
Hive had supported the trasactinons and INSERT, UPDATE, DELETE since 0.13 version was released. But it had several drawbacks like these.
- Transactional tables had to be stored in ORC and and bucketed
- Reading transactional tables was significnatly slower than non-transactional ones
- No support for MERGE or UPSERT funcionality
Seamless INSERT/UPDATE/DELETE opertaions on existing tables and improve the performance just as good as non-ACID
Transactional tables don’t have to be bucketed anymore
Non-ORC format bales supported (INSERT & SELECT only)
Compatibility with cloud storage providers
- eg. Amazon S3
New HiveStreaming API (v2)

LLAP(Live Long And Process) workload management

It consists of a long-lived daemon which replaces direct interactions with the HDFS DataNode, and a tightly integrated DAG-based framework.
Functionality such as caching, pre-fetching, some query processing and access control are moved into the daemon. Small/short queries are largely processed by this daemon directly, while any heavy lifting will be performed in standard YARN containers.
Live Long And Process (LLAP) functionality was added in Hive 2.0 and Hive 3.0 supports new features
- Split your LLAP resources in pools
  - e.g. bi (for Business Analytics) and etl (for Extract Transform and Load)
- Automatically map applications to pools
  - e.g. Tableau to the bi pool
- Create triggers to dynamically move applications from one pool to another
  - e.g. move long running queries to the etl pool
- Activate/deactivate your resource plans based on your users’ needs (one active at a time).

JDBC, Druid and Kafka connectors

JDBC connector(JdbcStorageHandler)
- It is read-only for now.
- It allows you to easily read/import data from any of the supported databases.
  - MySQL, PostgreSQL, Oracle, MSSQL and Apache Derby
Kafka connector
- It can be used to query real-time data from Apache Kafka.
  - You connect to Kafka data from Hive by creating an external table that maps to a Kafka topic.
- It can be used to achieve exactly-once offloading of data to Hive and real-time data transformation.
  - The Hive-Kafka connector supports the following serialization and deserialization formats:
    - JsonSerDe (default)
    - OpenCSVSerde
    - AvroSerDe
  - Exactly-once ingestion from Kafka to Druid can be done through Hive.

Hive Warehouse Connector for Apache Spark

All interactions between Hive and Apache Spark has to go through the Hive Warehouse Connector.
This connector takes advantage of Hive LLAP to allow streaming/ACID interaction between Hive and Spark.
The Spark application will need to access a Hive Server Interactive (with LLAP activated) to read Hive managed tables, but it won’t need it to write to Hive managed tables or read/write Hive external tables.

Execute Query Process
- 1. Driver submits query to HiveServer
- 1. Compile query and return “splits” to Driver
- 1. Execute query on LLAP
- 1. Executor Tasks run for each split
- 1. Tasks read Arrow data from LLAP
- 1. HWC returns ArrowColumnVectors to Spark

SQL features

Constraints and default values
- DEFAULT value for each column is available and it can be used in INSERT and UPDATE statements.
  
  UNIQUE and NOT NULL SQL constraints and **PRIMARY FOREIGN KEY** already have been added in Hive 2.1
Tables information
- Hive 3 allows easy exploration of the whole warehouse with information_schema and sys databases.

Improved performance

Based on a recent TPC-DS benchmark by the MR3 team, Hive LLAP 3.1.0 is the fastest SQL-on-Hadoop system available in HDP 3.0.1.
The benchmark compares all the SQL systems embedded with HDP3 as well as Hive on MR3 (a new execution engine for Hadoop and Kubernetes), by running a set of 99 SQL queries.

System	Total TCP-DS running time (seconds)
HDP 3.0.1 LLAP	5516.89
Hive 3.1.0/MR3	6575.052
HDP 3.0.1 Tez	12227.317
Presto 0.208e	12948.224
Spark 2.3.1	26247.471

Materialized views (MV)

It can improve the performance by serveral ways #### Storage for intermediate results
- Materialized views results are stored, allowing mutualizing computing costs. #### Incremental rebuilding
- Updating an materialized view only computes the data that was inserted in its sources tables since the last update. ##### Query rewriting
- When appropriate, the cost-based optimizer uses existing materialized views to plan queries instead of the sources tables, without the user being aware of it.

Other features

Surrogate Keys
TIMESTAMP WITH TIMEZONE data type
Apache Parquet and text files are supported in LLAP

References

[Data Analysis] SQL Recipe for Data Analysis Tutorial #6 - Aggregating data based on the time series

Thu, 06 Feb 2020 00:00:00 +0000

Goal

Studying SQL for data analysis
Practicing chapter 4th in the book
Studying SQL for Manipulating data

Practice

Book for studying

Title	데이터 분석을 위한 SQL 레시피
Author	Nagato Kasaki, Naoto Tamiya
Translator	윤인성
Publisher	한빛미디어
Chapter	4
Lesson	9

I’m gonna study basic SQL syntax and functions for data analysis
I’m gonna study how to extract data for sales
I’m gonna study how to aggregate data based on the time series
I’m gonna try to load test data on PostgreSQL and execute queries in the book

4.9. Aggregating data based on the time series

Download tutorial SQL file

4.9.1. Aggregate sales amount by date

Create purchase log table

-- DROP TABLE IF EXISTS purchase_log;
-- Create purchase log table
CREATE TABLE purchase_log3 (
    dt              varchar(255)
  , order_id        integer
  , user_id         varchar(255)
  , purchase_amount integer
);

-- Insert sample data
INSERT INTO purchase_log3
VALUES
    ('2020-01-01',  1, 'rhwpvvitou', 13900)
  , ('2020-01-01',  2, 'hqnwoamzic', 10616)
  , ('2020-01-02',  3, 'tzlmqryunr', 21156)
  , ('2020-01-02',  4, 'wkmqqwbyai', 14893)
  , ('2020-01-03',  5, 'ciecbedwbq', 13054)
  , ('2020-01-03',  6, 'svgnbqsagx', 24384)
  , ('2020-01-03',  7, 'dfgqftdocu', 15591)
  , ('2020-01-04',  8, 'sbgqlzkvyn',  3025)
  , ('2020-01-04',  9, 'lbedmngbol', 24215)
  , ('2020-01-04', 10, 'itlvssbsgx',  2059)
  , ('2020-01-05', 11, 'jqcmmguhik',  4235)
  , ('2020-01-05', 12, 'jgotcrfeyn', 28013)
  , ('2020-01-05', 13, 'pgeojzoshx', 16008)
  , ('2020-01-06', 14, 'msjberhxnx',  1980)
  , ('2020-01-06', 15, 'tlhbolohte', 23494)
  , ('2020-01-06', 16, 'gbchhkcotf',  3966)
  , ('2020-01-07', 17, 'zfmbpvpzvu', 28159)
  , ('2020-01-07', 18, 'yauwzpaxtx',  8715)
  , ('2020-01-07', 19, 'uyqboqfgex', 10805)
  , ('2020-01-08', 20, 'hiqdkrzcpq',  3462)
  , ('2020-01-08', 21, 'zosbvlylpv', 13999)
  , ('2020-01-08', 22, 'bwfbchzgnl',  2299)
  , ('2020-01-09', 23, 'zzgauelgrt', 16475)
  , ('2020-01-09', 24, 'qrzfcwecge',  6469)
  , ('2020-01-10', 25, 'njbpsrvvcq', 16584)
  , ('2020-01-10', 26, 'cyxfgumkst', 11339)
;

commit;

-- Select sample data
SELECT * FROM purchase_log3;

Aggregate sales amount and the average purchase amount per day

Imagine that you have to make the below chart

-- Aggregate sales amount and the average purchase amount per day
SELECT dt
     , COUNT(*) AS purchase_count
     , SUM(purchase_amount) AS total_amount
     , AVG(purchase_amount) AS avg_amount
  FROM purchase_log3
 GROUP BY dt
 ORDER BY dt ASC
;

4.9.2. Extract the change followed by date through a moving average

You can easily recognize the increase and decrease chage through the moving average
Imagine that you have to make the below chart

-- Extract the change followed by date through a moving average
SELECT dt 
     , SUM(purchase_amount) AS total_amount
     -- Caculate the recent average amount for 7 days
     -- Get the average amount even if there are less 7 days
     , AVG(SUM(purchase_amount))
         OVER(ORDER BY dt ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)
         AS seven_day_avg
     -- Calculate the recent average amount for 7 days exactly
     -- Ignore the average amount which has less 7 days
     , CASE WHEN 7 = COUNT(*)
                      OVER(ORDER BY dt ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)
              THEN AVG(SUM(purchase_amount))
                    OVER(ORDER BY dt ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)
            END AS seven_day_avg_strict
  FROM purchase_log3
 GROUP BY dt 
 ORDER BY dt ASC
;

4.9.3. Calculate the accumulated sales amount of this month

Imagine that you have to make the below chart

-- Calculate the accumulated sales amount of this month
SELECT dt
     -- Extract year-month
     -- if PostgreSQL, Hive, SparkSQL, Redshift
     , substring(dt, 1, 7) AS year_month
     -- if PostgreSQL, Hive, SparkSQL, BigQuery
     -- , substr(dt, 1, 7) AS year_month
     , SUM(purchase_amount) AS total_amount
     , SUM(SUM(purchase_amount))
         OVER(PARTITION BY substring(dt, 1, 7) 
              ORDER BY dt 
              ROWS UNBOUNDED PRECEDING)
         AS agg_amount
  FROM purchase_log3
 GROUP BY dt
 ORDER BY dt
;

Enhance the readability of query through WITH clause

substring functions were used repeatly and it ruined the code readablity
You can make it better by WITH

In BigData analysis, it is usually more important to make the query distinctly than the performance

-- Make a temporary view indicating daily sales amount
WITH
daily_purchase AS (
  SELECT dt
       -- Extract year, month, date respectively
       -- if PostgreSQL, Hive, SparkSQL, Redshift, use substring
       -- if BigQuery, use substr
       , substring(dt, 1, 4) AS year
       , substring(dt, 6, 2) AS month
       , substring(dt, 9, 2) AS date
       , SUM(purchase_amount) AS purchase_amount
       , COUNT(order_id) AS orders
    FROM purchase_log3
   GROUP BY dt
)
SELECT *
  FROM daily_purchase
 ORDER BY dt 
;

-- Calculate the accumulated sales amount of this month through WITH
WITH
daily_purchase AS (
  SELECT dt
       -- Extract year, month, date respectively
       -- if PostgreSQL, Hive, SparkSQL, Redshift, use substring
       -- if BigQuery, use substr
       , substring(dt, 1, 4) AS year
       , substring(dt, 6, 2) AS month
       , substring(dt, 9, 2) AS date
       , SUM(purchase_amount) AS purchase_amount
       , COUNT(order_id) AS orders
    FROM purchase_log3
   GROUP BY dt
)
SELECT dt
     , concat(year, '-', month) AS year_month
     -- if Redshift, concat or ||
     -- year || '-' || month AS year_month
     , purchase_amount
     , SUM(purchase_amount)
         OVER(PARTITION BY year, month ORDER BY dt ROWS UNBOUNDED PRECEDING)
         AS agg_amount
  FROM daily_purchase 
 ORDER BY dt
;

4.9.4. Calculate the sales per month and compare to the one of last year

Create purchase log table

Insert sample data which contains sale amount from 2018 to 2019

-- DROP TABLE IF EXISTS purchase_log4;
CREATE TABLE purchase_log4 (
    dt              varchar(255)
  , order_id        integer
  , user_id         varchar(255)
  , purchase_amount integer
);

-- Insert sample data
INSERT INTO purchase_log4
VALUES
    ('2018-01-01',    1, 'rhwpvvitou', 13900)
  , ('2018-02-08',   95, 'chtanrqtzj', 28469)
  , ('2018-03-09',  168, 'bcqgtwxdgq', 18899)
  , ('2018-04-11',  250, 'kdjyplrxtk', 12394)
  , ('2018-05-11',  325, 'pgnjnnapsc',  2282)
  , ('2018-06-12',  400, 'iztgctnnlh', 10180)
  , ('2018-07-11',  475, 'eucjmxvjkj',  4027)
  , ('2018-08-10',  550, 'fqwvlvndef',  6243)
  , ('2018-09-10',  625, 'mhwhxfxrxq',  3832)
  , ('2018-10-11',  700, 'wyrgiyvaia',  6716)
  , ('2018-11-10',  775, 'cwpdvmhhwh', 16444)
  , ('2018-12-10',  850, 'eqeaqvixkf', 29199)
  , ('2019-01-09',  925, 'efmclayfnr', 22111)
  , ('2019-02-10', 1000, 'qnebafrkco', 11965)
  , ('2019-03-12', 1075, 'gsvqniykgx', 20215)
  , ('2019-04-12', 1150, 'ayzvjvnocm', 11792)
  , ('2019-05-13', 1225, 'knhevkibbp', 18087)
  , ('2019-06-10', 1291, 'wxhxmzqxuw', 18859)
  , ('2019-07-10', 1366, 'krrcpumtzb', 14919)
  , ('2019-08-08', 1441, 'lpglkecvsl', 12906)
  , ('2019-09-07', 1516, 'mgtlsfgfbj',  5696)
  , ('2019-10-07', 1591, 'trgjscaajt', 13398)
  , ('2019-11-06', 1666, 'ccfbjyeqrb',  6213)
  , ('2019-12-05', 1741, 'onooskbtzp', 26024)
;

commit;

-- Select sample data
SELECT * FROM purchase_log4;

Calculate the sales per month and compare to the one of last year

Imagine that you have to make the below chart

-- Calculate the sales per month and compare to the one of last year
WITH
daily_purchase AS (
  SELECT dt
       -- Extract year, month, date respectively
       -- if PostgreSQL, Hive, SparkSQL, Redshift, use substring
       -- if BigQuery, use substr
       , substring(dt, 1, 4) AS year
       , substring(dt, 6, 2) AS month
       , substring(dt, 9, 2) AS date
       , SUM(purchase_amount) AS purchase_amount
       , COUNT(order_id) AS orders
    FROM purchase_log4
   GROUP BY dt
)
SELECT month
     , SUM(CASE year WHEN '2018' THEN purchase_amount END) AS amount_2018
     , SUM(CASE year WHEN '2019' THEN purchase_amount END) AS amount_2019
     , 100.0 * SUM(CASE year WHEN '2019' THEN purchase_amount END)
       / SUM(CASE year WHEN '2018' THEN purchase_amount END)
         AS rate
  FROM daily_purchase 
 GROUP BY month 
 ORDER BY month
;

4.9.5. Utilize Z chart

Z chart
- A diagram charting values over a period (frequently one year) and showing simultaneously monthly figures (or weekly or daily), cumulative totals and the moving averages.
- It normally takes a Z shape, hence its title and is of use in clarifying the trends present in the data displayed

What factors are needed for Z chart

Monthly figure
- The sum amount of that month
- It can be weekly or daily

Cumulative total
- Cumulative total from the first month of this year to current month

Moving total
- Total amount from 11 month ago to current

How to analyze Z chart

Noticeable point on the cumulative total
- The line for the cumulative total figure will be more straight as the monthly figures are regular
- The lean of the line for the cumulative total figure will become steep as the monthly figures are getting bigger
- The lean of the line for the cumulative total figure will be flatting off as the monthly figures are getting smaller

Noticeable point on the moving total
- The line for the moving total will be straight when the sales amount is regular from last year to this year
- The graph will rise up when there is an increasing trend for sales amount
- The graph will drop when there is a decreasing trend for sales amount
Various shapes of Z chart

How to aggregate data for Z chart throug SQL

--- Z chart for sales in 2019
WITH
daily_purchase AS (
  SELECT dt
       -- Extract year, month, date respectively
       -- if PostgreSQL, Hive, SparkSQL, Redshift, use substring
       -- if BigQuery, use substr
       , substring(dt, 1, 4) AS year
       , substring(dt, 6, 2) AS month
       , SUM(purchase_amount) AS purchase_amount
       , COUNT(order_id) AS orders
    FROM purchase_log4
   GROUP BY dt
)
, monthly_purchase AS (
  -- Monthly figure
  SELECT year 
       , month
       , SUM(purchase_amount) AS amount
    FROM daily_purchase 
   GROUP BY year, month 
)
, calc_index AS (
  SELECT year 
       , month
       , amount
       -- cumulative total
       , SUM(CASE WHEN year = '2019' THEN amount END)
           OVER(ORDER BY year, month ROWS UNBOUNDED PRECEDING)
           AS agg_amount
       -- Moving total
       , SUM(amount)
           OVER(ORDER BY year, month ROWS BETWEEN 11 PRECEDING AND CURRENT ROW)
           AS year_avg_amount
    FROM monthly_purchase
   ORDER BY year, month 
)
-- Select data for 2019
SELECT concat(year, '-', month) AS year_month
     , amount 
     , agg_amount 
     , year_avg_amount 
  FROM calc_index
 WHERE year = '2019'
 ORDER BY year_month
;

4.9.6. The significant point for analyzing sales

You have to turn to data about surrounding factors like purchase count, a unit cost to understand the reason why there are changes on sales

-- Aggregate data related with sales
WITH
daily_purchase AS (
  SELECT dt
       -- Extract year, month, date respectively
       -- if PostgreSQL, Hive, SparkSQL, Redshift, use substring
       -- if BigQuery, use substr
       , substring(dt, 1, 4) AS year
       , substring(dt, 6, 2) AS month
       , SUM(purchase_amount) AS purchase_amount
       , COUNT(order_id) AS orders
    FROM purchase_log4
   GROUP BY dt
)
, monthly_purchase AS (
  -- Monthly figure
  SELECT year 
       , month
       , SUM(orders) AS orders
       , AVG(purchase_amount) AS avg_amount
       , SUM(purchase_amount) AS monthly
    FROM daily_purchase 
   GROUP BY year, month 
)
SELECT concat(year, '-', month) AS year_month
     , orders
     , avg_amount 
     , monthly 
     , SUM(monthly)
         OVER(PARTITION BY year 
              ORDER BY monthly 
              ROWS UNBOUNDED preceding)
         AS agg_amount
     -- sales amount 12 months ago
     , LAG(monthly, 12)
         OVER(ORDER BY year, month)
         -- if SparkSQL
         -- OVER(ORDER BY year, month
         --      ROWS BETWEEN 12 PRECEDING AND 12 PRECEDING)
         AS last_year
     -- The rate compared to sales amount 12m ago
     , 100.0 * monthly / LAG(monthly, 12)
                           OVER(ORDER BY year, month)
         AS rate
  FROM monthly_purchase
 ORDER BY year_month
;

References

데이터 분석을 위한 SQL 레시피 - 한빛미디어
Z chart - Monash University

[Data Analysis] SQL Recipe for Data Analysis Tutorial #5 - Manipulating multiple tables

Wed, 05 Feb 2020 00:00:00 +0000

Goal

Studying SQL for data analysis
Practicing chapter 3rd in the book
Studying SQL for Manipulating data

Practice

Book for studying

Title	데이터 분석을 위한 SQL 레시피
Author	Nagato Kasaki, Naoto Tamiya
Translator	윤인성
Publisher	한빛미디어
Chapter	3
Lesson	8

I’m gonna study basic SQL syntax and functions for data analysis
I’m gonna study how to manipulate multiple tables
I’m gonna study about data-intensive and data-processing
I’m gonna try to load test data on PostgreSQL and execute queries in the book

3.8. Manipulating multiple tables

Download tutorial SQL file

3.8.1. Merge several tables horizontally

Create two user table

One of them is the table that stores information about users in application 1, while another table stores users in application 2

 -- DROP TABLE IF EXISTS app1_mst_users;
-- Create user in app1 table
CREATE TABLE app1_mst_users (
    user_id varchar(255)
  , name    varchar(255)
  , email   varchar(255)
);

-- Insert sample data
INSERT INTO app1_mst_users
VALUES
    ('U001', 'Dorbae', '[email protected]'  )
  , ('U002', 'Chang', '[email protected]')
;

-- DROP TABLE IF EXISTS app2_mst_users;
-- Create user in app2 table
CREATE TABLE app2_mst_users (
    user_id varchar(255)
  , name    varchar(255)
  , phone   varchar(255)
);

-- Insert sample data
INSERT INTO app2_mst_users
VALUES
    ('U001', 'Seong', '080-xxxx-xxxx')
  , ('U002', 'Bae', '070-xxxx-xxxx')
;

commit;

-- Select sample data
SELECT *
  FROM app1_mst_users
;

SELECT *
  FROM app2_mst_users
;

Merge tables

You can merge several table throug UNION ALL
UNION or UNION DISTINCT also merges multiple tables and in this case, it provides the same result. But, it spends more costs than UNION ALL
I’m gonna add app_name column for indentifying the app

-- Merge two tables
SELECT 'app1' AS app_name
     , user_id
     , name
     , email
  FROM app1_mst_users
UNION ALL
SELECT 'app1' AS app_name
     , user_id
     , name
     , phone -- or NULL
  FROM app2_mst_users
;

3.8.2. Merge several tables vertically

Create three tables

-- DROP TABLE IF EXISTS mst_categories;
CREATE TABLE mst_categories (
    category_id integer
  , name        varchar(255)
);

INSERT INTO mst_categories
VALUES
    (1, 'dvd' )
  , (2, 'cd'  )
  , (3, 'book')
;

-- DROP TABLE IF EXISTS category_sales;
CREATE TABLE category_sales (
    category_id integer
  , sales       integer
);

INSERT INTO category_sales
VALUES
    (1, 850000)
  , (2, 500000)
;

-- DROP TABLE IF EXISTS product_sale_ranking;
CREATE TABLE product_sale_ranking (
    category_id integer
  , rank        integer
  , product_id  varchar(255)
  , sales       integer
);

INSERT INTO product_sale_ranking
VALUES
    (1, 1, 'D001', 50000)
  , (1, 2, 'D002', 20000)
  , (1, 3, 'D003', 10000)
  , (2, 1, 'C001', 30000)
  , (2, 2, 'C002', 20000)
  , (2, 3, 'C003', 10000)
;

commit;

SELECT * FROM mst_categories;

SELECT * FROM category_sales;

SELECT * FROM product_sale_ranking;

Merge the tables through JOIN

You might loose some data because there are not matched data

-- Join tables
SELECT m.category_id
     , m.name
     , s.sales
     , r.product_id AS sale_product
  FROM mst_categories AS m
    JOIN category_sales AS s
      ON m.category_id = s.category_id
    JOIN product_sale_ranking AS r
      ON m.category_id = r.category_id
;

Merge the tables through LEFT JOIN

You can maintain the record count of category_sales(basis table)

-- Join tables through LEFT JOIN
SELECT m.category_id
     , m.name
     , s.sales
     , r.product_id AS sale_product
  FROM mst_categories AS m
    LEFT JOIN category_sales AS s
      ON m.category_id = s.category_id
    LEFT JOIN product_sale_ranking AS r
      ON m.category_id = r.category_id
        AND r.rank = 1
;

Merge the tables through Correlated Subquery without JOIN

It is possible to use Correlated Subquery when the Middleware supports it

-- Merge tables through Correlated Subquery
SELECT m.category_id
     , m.name
     -- Calculate sales amount byt categories using Correlated Subquery
     , (SELECT s.sales
          FROM category_sales AS s
         WHERE m.category_id = s.category_id
       ) AS sales
     -- Best sales product by categories
     -- Subquery has to return only one row
     , (SELECT r.product_id
          FROM product_sale_ranking AS r
         WHERE m.category_id = r.category_id
         ORDER BY sales DESC
         LIMIT 1
       ) AS top_sale_product
  FROM mst_categories AS m
;

3.8.3. Convert a conditional flag into 0 or 1

Create user with card number table and purchase log table

-- DROP TABLE IF EXISTS mst_users_with_card_number;
-- Create user with card number table
CREATE TABLE mst_users_with_card_number (
    user_id     varchar(255)
  , card_number varchar(255)
);

-- Insert sample data
INSERT INTO mst_users_with_card_number
VALUES
    ('U001', '1234-xxxx-xxxx-xxxx')
  , ('U002', NULL                 )
  , ('U003', '5678-xxxx-xxxx-xxxx')
;

-- DROP TABLE IF EXISTS purchase_log;
-- Create purchase log table
CREATE TABLE purchase_log2 (
    purchase_id integer
  , user_id     varchar(255)
  , amount      integer
  , stamp       varchar(255)
);

INSERT INTO purchase_log2
VALUES
    (10001, 'U001', 200, '2019-01-30 10:00:00')
  , (10002, 'U001', 500, '2019-02-10 10:00:00')
  , (10003, 'U001', 200, '2019-02-12 10:00:00')
  , (10004, 'U002', 800, '2019-03-01 10:00:00')
  , (10005, 'U002', 400, '2019-03-02 10:00:00')
;

commit;

-- Select sample data
SELECT * FROM mst_users_with_card_number;
SELECT * FROM purchase_log2;

Convert boolean flag about purchase or not into 0/1

You can maintain the same record count of mst_users_with_card_number by doing LEFT JOIN and GROUP BY user_id
And, you can ocnvert a flag into 0/1 using CASE or SIGN function
SIGN returns 0 if argument is 0 and returns 1 about a positive number argument and also returns -1 about a negative number argument

-- Convert boolean flag about purchase or not into 0/1
SELECT m.user_id
     , m.card_number
     , COUNT(p.user_id) AS purchase_count
     -- If a card number was registered, return 1 and return 0 about the opposite case
     , CASE WHEN m.card_number IS NOT NULL THEN 1 
            ELSE 0 
            END AS has_case
     -- If user has been ever purchased, return 1 and return 0 about the opposite case
     , SIGN(COUNT(p.user_id)) AS has_purchased
  FROM mst_users_with_card_number AS m
    LEFT JOIN purchase_log2 AS p
      ON m.user_id = p.user_id
  GROUP BY m.user_id, m.card_number
  ORDER BY m.user_id
;

3.8.4. CTE(Common Table Expression)

It has been introduced since SQL99
It enhances a readability of the code

Create table and insert sample data

--DROP TABLE IF EXISTS product_sales;
-- Create product slaes table
CREATE TABLE product_sales (
    category_name varchar(255)
  , product_id    varchar(255)
  , sales         integer
);

-- Insert sample data
INSERT INTO product_sales
VALUES
    ('dvd' , 'D001', 50000)
  , ('dvd' , 'D002', 20000)
  , ('dvd' , 'D003', 10000)
  , ('cd'  , 'C001', 30000)
  , ('cd'  , 'C002', 20000)
  , ('cd'  , 'C003', 10000)
  , ('book', 'B001', 20000)
  , ('book', 'B002', 15000)
  , ('book', 'B003', 10000)
  , ('book', 'B004',  5000)
;

commit;

-- Select sample data
SELECT * FROM product_sales;

Set name to the table added the ranks by categories

-- Set name to the table added the ranks by categories
WITH
product_sale_ranking AS (
  SELECT category_name
       , product_id
       , sales
       , ROW_NUMBER()
           OVER(PARTITION BY category_name ORDER BY sales DESC)
           AS rank
    FROM product_sales
)
SELECT *
  FROM product_sale_ranking
;

Extract the unique rankings from the ranking list by categories

-- Extract the unique rankings from the ranking list by categories
WITH
product_sale_ranking AS (
  SELECT category_name
       , product_id
       , sales
       , ROW_NUMBER()
           OVER(PARTITION BY category_name ORDER BY sales DESC)
           AS rank
    FROM product_sales
)
, mst_rank AS (
  SELECT DISTINCT rank
    FROM product_sale_ranking 
)
SELECT *
  FROM mst_rank
;

Print the ranking by categories

-- Print the ranking by categories
WITH
product_sale_ranking AS (
  SELECT category_name
       , product_id
       , sales
       , ROW_NUMBER()
           OVER(PARTITION BY category_name ORDER BY sales DESC)
           AS rank
    FROM product_sales
)
, mst_rank AS (
  SELECT DISTINCT rank
    FROM product_sale_ranking 
)
SELECT m.rank
     , r1.product_id AS vd
     , r1.sales AS dvd_sales
     , r2.product_id AS cd
     , r3.product_id AS book
     , r3.sales AS book_sales
  FROM mst_rank AS m
    LEFT JOIN product_sale_ranking AS r1
      ON m.rank = r1.rank
      AND r1.category_name = 'dvd'
    LEFT JOIN product_sale_ranking AS r2
      ON m.rank = r2.rank
      AND r2.category_name = 'cd'
    LEFT JOIN product_sale_ranking AS r3
      ON m.rank = r3.rank
      AND r3.category_name = 'book'
  ORDER BY m.rank
;

3.8.5. Make a resemblance table

It can enhance the productivity performance for test

Make a resemblance table with specific records

-- Make a resemblance table with specific records
WITH
mst_devices AS (
  SELECT 1 AS device_id, 'PC' AS device_name
  UNION ALL
  SELECT 2 AS device_id, 'SP' AS device_name
  UNION ALL
  SELECT 3 AS device_id, 'Application' AS device_name
)
SELECT *
  FROM mst_devices
;

UNION ALL might drop the performance whne there are a lot of records

Make a resemblance table with VALUE clause

In PostgreSQL, it is possible to make a resmeblance table using VALUES

-- Make a resemblance table with VALUE clause
WITH
mst_devices(device_id, device_name) AS (
  VALUES
      (1, 'PC')
    , (2, 'SP')
    , (3, 'Application')
 )
SELECT *
  FROM mst_devices
;

Make a resemblance table through structural table function

You can make a dynamic table with a structural function when it is impossible to use VALUE clause
For example, you can define a table through array function and explode function in Hive

-- Make a resemblance table through array, explode in Hive
WITH
mst_devces AS (
  -- Convert a array into columns
  SELECT d[0] AS device_id 
       , d[1] AS device_name
    FROM (SELECT explode (
                   array(
                       array('1', 'PC')
                     , array('2', 'SP')
                     , array('3', 'Application')
                   )
                 ) d
         ) AS t
)
SELECT *
  FROM mst_devices
;

Use map data type intead of array data type

-- Use map data type intead of array data type
WITH
mst_devces AS (
  SELECT d['device_id'] AS device_id 
       , d['device_name'] AS device_name
    FROM (SELECT explode (
                   array(
                       map('device_id', '1', 'device_name', 'PC')
                     , map('device_id', '2', 'device_name', 'SP')
                     , map('device_id', '3', 'device_name', 'Application')
                   )
                 ) d
         ) AS t
)
SELECT *
  FROM mst_devices
;

The function making sequence numbers

Some middleware support the function whixh makes sequence numbers
PostgreSQL supports generate_series function and BigQuery supports generate_array function

-- Make a resemblance table with sequence numbers
WITH 
series AS (
  -- Make from 1 to 5
  SELECT generate_series(1, 5) AS idx
  -- if BigQuery
  -- SELECT idx FROM unnst(generate_array(1, 5)) AS idx
)
SELECT *
  FROM series
;

Make a sequence numbers using repeat funxtion

Hive and SparkSQL don’t support the function like generate_series
Nevertheless, you cna make sequence numbers using repeat and explode functions

-- Make a resemblance table with sequence numbers through repeat
SELECT ROW_NUMBER()
         OVER(ORDER BY x) AS idx
  FROM 
       -- Make an array through repeat and split
       -- After that, spread it through explode
       (SELECT explode(split(repeat('x', 5 - 1), 'x')) AS x) AS t
;

References

데이터 분석을 위한 SQL 레시피 - 한빛미디어

[Data Analysis] SQL Recipe for Data Analysis Tutorial #4 - Manipulating a single table

Fri, 31 Jan 2020 00:00:00 +0000

Goal

Studying SQL for data analysis
Practicing chapter 3rd in the book
Studying SQL for Manipulating data

Practice

Book for studying

Title	데이터 분석을 위한 SQL 레시피
Author	Nagato Kasaki, Naoto Tamiya
Translator	윤인성
Publisher	한빛미디어
Chapter	3
Lesson	7

I’m gonna study basic SQL syntax and functions for data analysis
I’m gonna study how to manipulate a single table
I’m gonna study about data-intensive and data-processing
I’m gonna try to load test data on PostgreSQL and execute queries in the book

3.7. Manipulating a singlie table

Data-intensive
- SQL supports a lot of functions for data-intensive
- Especially, SQL:2003 standard support Window functions which performs a calculation across a set of table rows that are somehow related to the current row
Data-processing
- Sometimes, you need to convert a table as the form that you want because it is useful in an aggregation

Download tutorial SQL file

3.7.1. Make groups

Data-intensive means that making one value from multiple records like COUNT(returns total record count), SUM(returns the sum of all records)

Create table and insert sample data

--DROP TABLE IF EXISTS review;
-- Create review table
CREATE TABLE review (
    user_id    varchar(255)
  , product_id varchar(255)
  , score      numeric
);

-- Insert sample data
INSERT INTO review
VALUES
    ('U001', 'A001', 4.0)
  , ('U001', 'A002', 5.0)
  , ('U001', 'A003', 5.0)
  , ('U002', 'A001', 3.0)
  , ('U002', 'A002', 3.0)
  , ('U002', 'A003', 4.0)
  , ('U003', 'A001', 5.0)
  , ('U003', 'A002', 4.0)
  , ('U003', 'A003', 4.0)
;

commit;

-- Select sample data
SELECT *
  FROM review
;

Calculate several feature values from all records

| Function | Description | | — | — | | DISTINCT | Removes the duplication | | SUM | Returns the sum | | AVG | Returns the average | | MAX | Returns the maximum value | | MIN | Returns the minimum value |

-- Calculate several feature values from all records
SELECT COUNT(*) AS total_count
     , COUNT(DISTINCT user_id) AS user_count
     , COUNT(DISTINCT product_id) AS product_count
     , SUM(score) AS sum
     , AVG(score) AS avg
     , MAX(score) AS max
     , MIN(score) AS min
  FROM review
;

Calculate serveral feature values from groups repectively

GROUP BY clause does intensivly by the key
I’m gonna make groups by user_id

-- Calculate serveral feature values from groups repectively
SELECT user_id
     , COUNT(*) AS total_count
     , COUNT(DISTINCT product_id) AS product_count
     , SUM(score) AS sum
     , AVG(score) AS avg
     , MAX(score) AS max
     , MIN(score) AS min
  FROM review
 GROUP BY user_id 
;

Handle the value that is applied data-intensive function and the one that is not applied it simultaneously

I’ll calculate the difference between user’s private score(score) and the average score by user(user_avg_score)
**OVER(PARTITION BY )** : group by column_name
OVER() : about all records in a table

-- Calculate user's private score(score) and the average score by user(user_avg_score)
SELECT user_id
     , product_id
     -- User's private score
     , score 
     -- The total average score 
     , AVG(score) OVER() AS avg_score
     -- User's average score
     , AVG(score) OVER(PARTITION BY user_id) AS user_avg_score
     -- The differnce
     , score - AVG(score) OVER(PARTITION BY user_id) AS user_avg_score_diff
  FROM review
;

Create popular product table and insert sample data

-- DROP TABLE IF EXISTS popular_products;
-- Create popular product table
CREATE TABLE popular_products (
    product_id varchar(255)
  , category   varchar(255)
  , score      numeric
);

-- Insert sample data
INSERT INTO popular_products
VALUES
    ('A001', 'action', 94)
  , ('A002', 'action', 81)
  , ('A003', 'action', 78)
  , ('A004', 'action', 64)
  , ('D001', 'drama' , 90)
  , ('D002', 'drama' , 82)
  , ('D003', 'drama' , 78)
  , ('D004', 'drama' , 58)
;
     
commit;

-- Select sample data
SELECT *
  FROM popular_products
;

Ordering in a group

You can order records through ORDER BY
There are several functions to return a sequent order

Function	Description
ROW_NUMBER	Returns a unique sequence number
RANK	Returns the rank number in a group and skips the number if there are same rank values eg. 1, 2, 2, 4, 5
DENSE_RANK	Returns the rank number in a group and deosn’t skip the number if there are same rank values eg. 1, 2, 2, 3, 4

And also, there are functions that return the previous or next record

Function	Description
LAG(column_name)	Returns the previous record in a group
LAG(column_name, n)	Returns the previous Nth record in a group
LEAD(column_name)	Returns the next record in a group
LEAD(column_name, n)	Returns the next Nth record in a group

-- Ordering in a table
SELECT product_id
     , score
     -- Unique rank by score
     , ROW_NUMBER() OVER(ORDER BY score DESC) AS rownum
     -- Skip duplicate rank
     , RANK() OVER(ORDER BY score DESC) AS rank
     -- Don't skip duplicate rank
     , DENSE_RANK() OVER(ORDER BY score DESC) AS dense_rank 
     -- Previous record
     , LAG(product_id) OVER(ORDER BY score DESC) AS lag1
     -- Previous 2nd record
     , LAG(product_id, 2) OVER(ORDER BY score DESC) AS lag2
     -- Next record
     , LEAD(product_id) OVER(ORDER BY score DESC) AS lead1
     -- Next 2nd record
     , LEAD(product_id, 2) OVER(ORDER BY score DESC) AS lead2
  FROM popular_products
 ORDER BY rownum 
;

Combine ordering and intensive functions

There are functions that return frst or last value in a window group like FIRST_VALUE, LAST_VALUE
And you can set the range of a window

-- Combine ordering and intensive functions
SELECT product_id
     , score
     , ROW_NUMBER() OVER(ORDER BY score DESC) 
         AS rownum
     -- Cumulative score from the score of a high rank to current in a window
     , SUM(score) 
         OVER(ORDER BY score DESC
           ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)
         AS cum_score
     -- Average score between previous record, current, and next record
     , AVG(score)
         OVER(ORDER BY score DESC
           ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)
         AS local_avg
     -- Product_id of top ranked record
     , FIRST_VALUE(product_id) 
         OVER(ORDER BY score DESC
           ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
         AS first_value
     -- Product_id of bottom ranked record
     , LAST_VALUE(product_id) 
         OVER(ORDER BY score DESC
           ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
         AS first_value
  FROM popular_products
 ORDER BY rownum 
;

Define a window frame

Define a relative frame based on current record
‘ROWS BETWEEN start AND end’ is the basic clause
You can use CURRENT_ROW(current row), n PRECEDING(n previous row), n FOLLOWING(n next row), UNBOUNDED PROCEDING(all previous rows), UNBOUNDED FOLLOWING(all next rows) keyword in start or end
If you don’t define a window frame, the ranges of default frame are whole rows without ORDER BY or from the first record to current record with ORDER BY
PostgresSQL supports array_agg(column_name) keyword and Hive, SparkSQL support collect_list(column_name) for data-intensive

Redshift supports listagg such as array_agg and collect_list. However, it is unable to used with a window frame

-- Integrate product_id by window frames
SELECT product_id
     , ROW_NUMBER() OVER(ORDER BY score DESC) 
         AS rownum
     -- All records
     , array_agg(product_id)	-- collect_list(product_id)
         OVER(ORDER BY score DESC
           ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
         AS whole_agg
     -- From the first record to current record
     , array_agg(product_id)
         OVER(ORDER BY score DESC
           ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)
         AS cum_agg
     -- From previous record to next record
     , array_agg(product_id)
         OVER(ORDER BY score DESC
           ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)
         AS local_agg
  FROM popular_products
 WHERE category = 'action'
 ORDER BY rownum 
;

Combine PARTITION BY and ORDER BY

-- Combine PARTITION BY and ORDER BY
SELECT category
     , product_id
     , score
     -- Unique rank by category
     , ROW_NUMBER()
         OVER(PARTITION BY category ORDER BY score DESC)
         AS rownum
     -- Rank with duplication by category
     , RANK()
         OVER(PARTITION BY category ORDER BY score DESC)
         AS rank
     -- Rank with duplication and skip duplicate rank
     , DENSE_RANK()
         OVER(PARTITION BY category ORDER BY score DESC)
         AS dense_rank
  FROM popular_products pp 
 ORDER BY category, rownum 
;

Extract top n by category

-- Extract top 2 products by category
SELECT *
  FROM 
    (SELECT category
          , product_id
          , score
          , ROW_NUMBER()
              OVER(PARTITION BY category ORDER BY score DESC)
              AS rank
      FROM popular_products pp 
   ) AS popular_products_with_rank
 WHERE rank  <= 2
 ORDER BY category, rank 
;

-- Extrat top ranked product by category
SELECT DISTINCT category
    -- the top ranked product_id
    , FIRST_VALUE(product_id)
        OVER(PARTITION BY category ORDER BY score DESC
          ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
        AS product_id
  FROM popular_products pp
;

3.7.3. Convert columns into rows

Create table and insert sample data

-- DROP TABLE IF EXISTS daily_kpi;
-- Create KPI(Key Point Indicator) table
CREATE TABLE daily_kpi (
    dt        varchar(255)
  , indicator varchar(255)
  , val       integer
);

-- Insert sample data
INSERT INTO daily_kpi
VALUES
    ('2020-01-01', 'impressions', 1800)
  , ('2020-01-01', 'sessions'   ,  500)
  , ('2020-01-01', 'users'      ,  200)
  , ('2020-01-02', 'impressions', 2000)
  , ('2020-01-02', 'sessions'   ,  700)
  , ('2020-01-02', 'users'      ,  250)
;

commit;

-- Select sample data
SELECT *
  FROM daily_kpi
;

Convert rows into columns

Use GROUP BY and MAX(CASE) because there are no duplicate indicators per dt

-- Convert rows into columns through GROUP_BY and MAX(CASE)
SELECT dt
     , MAX(CASE WHEN indicator = 'impressions' THEN val END)
         AS impressions
     , MAX(CASE WHEN indicator = 'sessions' THEN val END)
         AS sessions
     , MAX(CASE WHEN indicator = 'users' THEN val END)
         AS users
  FROM daily_kpi 
 GROUP BY dt 
 ORDER BY dt 
;

Create purchase log table and insert sampled data

-- DROP TABLE IF EXISTS purchase_detail_log;
-- Create purchase log table
CREATE TABLE purchase_detail_log (
    purchase_id integer
  , product_id  varchar(255)
  , price       integer
);

-- Insert sample data
INSERT INTO purchase_detail_log
VALUES
    (100001, 'A001', 3000)
  , (100001, 'A002', 4000)
  , (100001, 'A003', 2000)
  , (100002, 'D001', 5000)
  , (100002, 'D002', 3000)
  , (100003, 'A001', 3000)
;

commit;

-- Select sample data
SELECT *
  FROM purchase_detail_log
;

Convert rows into 1 row with comma delimiters

PostgreSQL and BigQuery support string_agg(column_name, delimiter) and Redshift also supports listagg(column_name, delimiter) to integrate rows into 1 row with delimiters
Hive and SparkSQL are available to integrate them through collect_list(column_name) and concat_ws(delimiter, array)

-- Convert rows into 1 row with comma delimiters
SELECT purchase_id
     -- if PostgresSQL, BigQuery
     , string_agg(product_id, ',') AS product_ids
     -- if Redshift
     -- , listagg(product_id, ',') AS product_ids
     -- if Hive, SparkSQL
     -- , concat_ws(',', collect_list(product_id)) AS product_ids
     , SUM(price) AS amount
  FROM purchase_detail_log 
 GROUP BY purchase_id 
 ORDER BY purchase_id 
;

3.7.4. Convert rows into columns

It is more complicate than converting columns into rows
However, it is useful to analyze data and there are many cases which need it

Quarterly sales table and insert sample data

This table is already exits because I made it in Tutorial #3 - Manipulating multiple values / 3.6.2. Compare multiple values
If you haven’t this table, execute the query below

-- DROP TABLE IF EXISTS quarterly_sales;
-- Create quarterly sales table
CREATE TABLE quarterly_sales (
    year integer
  , q1   integer
  , q2   integer
  , q3   integer
  , q4   integer
);

-- Insert sample data
INSERT INTO quarterly_sales
VALUES
    (2017, 82000, 83000, 78000, 83000)
  , (2018, 85000, 85000, 80000, 81000)
  , (2019, 92000, 81000, NULL , NULL )
;

commit;

-- Select sample data
SELECT * 
  FROM quarterly_sales
;

Convert rows into columns

You have to make pivot table and do CROSS JOIN with it

-- CROSS JOIN with a pivot table
SELECT q.year
     -- Print labels from Q1 to Q4
     , CASE WHEN p.idx = 1 THEN 'q1'
            WHEN p.idx = 2 THEN 'q2'
            WHEN p.idx = 3 THEN 'q3'
            WHEN p.idx = 4 THEN 'q4'
            END AS quarters
     -- Print sales from Q1 to Q4
     , CASE WHEN p.idx = 1 THEN q.q1
            WHEN p.idx = 2 THEN q.q2
            WHEN p.idx = 3 THEN q.q3
            WHEN p.idx = 4 THEN q.q4
            END AS sales
  FROM quarterly_sales AS q
    CROSS JOIN
      -- Create sequence table followed columns in quaterly_sales
      (SELECT 1 AS idx
       UNION ALL
       SELECT 2 AS idx
       UNION ALL
       SELECT 3 AS idx
       UNION ALL
       SELECT 4 AS idx
      ) AS p
;  

Create purchase log table and insert sample data

-- DROP TABLE IF EXISTS purchase_log;
-- Create purchase_log
CREATE TABLE purchase_log (
    purchase_id integer
  , product_ids varchar(255)
);

-- Insert sample data
INSERT INTO purchase_log
VALUES
    (100001, 'A001,A002,A003')
  , (100002, 'D001,D002')
  , (100003, 'A001')
;

commit;

-- Select sample data
SELECT *
  FROM purchase_log
;

Convert array into rows

Several middlewares support TABLE_FUNCTION whichi returns a table
For example, PostgreSQL and BigQuery have unnest(array) function and Hive and SparkSQL have explode(array) function and they convert an array into a table

-- Convert an array into a table
-- if PostgreSQL
SELECT unnest(ARRAY['A001', 'A002', 'A003']) AS product_id;
-- if BigQuery
-- SELECT * FROM unnest(ARRAY['A001', 'A002', 'A003']) AS product_id;
-- if Hive, SparkSQL
-- SELECT explode(ARRAY['A001', 'A002', 'A003']) AS product_id;

You have to use CROSS JOIN in PostgreSQL or BigQuery and use LATERAL VIEW in Hive or SparkSQL

-- Convert a stirng with delimiters into a table
SELECT purchase_id
     , product_id
  FROM purchase_log AS p
    -- if PostgreSQL
    CROSS JOIN unnest(pg_catalog.string_to_array(product_ids, ',')) AS product_id
    -- if BigQuery
    -- CROSS JOIN unnest(split(product_ids, ',')) AS prouct_id
    -- if Hive, SparkSQL
    -- LATERAL VIEW explode(split(producT_ids, ',') e AS producT_id
;

You can use scalar value and a table function in the same time in PostgreSQL
PostgreSQL supports regexp_split_to_table() which returns a table from an string by the regualar expression

-- Convert a stirng with delimiters into a table withtout cross join
SELECT purchase_id
     , regexp_split_to_table(product_ids, ',') AS product_id
  FROM purchase_log
;

Convert a string into rows in Redshift

Redshift doesn’t support array data type offically
Therefore, it is more complicate for you to do it in Redshift and you have to do several pre processing

-- Calculate the count of products through the count of delimiters in Redshift
SELECT purchase_id
     , product_ids
     -- Remove comman by product_id 
     -- and calculate the count of products through the count of delimiters
     , 1 + char_length(product_ids) - char_length(replace(product_ids, ',', ''))
         AS product_num
  FROM purchase_log
;

-- Convert a string into rows using a pivot table in Redshift
SELECT l.purchase_id
     , l.product_ids
     -- Set a sequence by the count of products
     , p.idx
     , split_part(l.product_ids, ',', p.idx) AS product_id 
  FROM purchase_log AS l
    JOIN
      (SELECT 1 AS idx
       UNION ALL
       SELECT 2 AS idx
       UNION ALL
       SELECT 3 AS idx 
      ) AS p
      -- Join when id of the pivot table is less than the count of products
      ON p.idx <= (1 + char_length(product_ids) - char_length(replace(product_ids, ',', '')))
;

References

데이터 분석을 위한 SQL 레시피 - 한빛미디어
SQL:2003 Wiki

[Data Analysis] SQL Recipe for Data Analysis Tutorial #3 - Manipulating multiple values

Tue, 28 Jan 2020 00:00:00 +0000

Goal

Studying SQL for data analysis
Practicing chapter 3rd in the book
Studying SQL for Manipulating data

Practice

Book for studying

Title	데이터 분석을 위한 SQL 레시피
Author	Nagato Kasaki, Naoto Tamiya
Translator	윤인성
Publisher	한빛미디어
Chapter	3
Lesson	6

I’m gonna study basic SQL syntax and functions for data analysis
I’m gonna study how to manipulate from a multiple values
I’m gonna try to load test data on PostgreSQL and execute queries in the book

3.6. Manipulating multiple values

Download tutorial SQL file

3.6.1. Concatenate strings

Create table and insert sample data

-- DROP TABLE IF EXISTS mst_user_location;
-- Create user location table
CREATE TABLE mst_user_location (
    user_id   varchar(255)
  , pref_name varchar(255)
  , city_name varchar(255)
);

-- Insert sample data
INSERT INTO mst_user_location
VALUES
    ('U001', 'Seoul-si', 'Seodaemun-gu')
  , ('U002', 'Kyeonki-do Suwon-si', 'Paldal-gu'  )
  , ('U003', 'Jeju-do', 'Seoguipo-si')
;

commit;

-- Select sample data
SELECT *
  FROM mst_user_location 
;

Concatenate string data type columns

-- Concatenate pref_name and citiy_name
SELECT user_id
     -- if PostgreSQL, Hive, SparkSQL, BigQuery
     , CONCAT(city_name, ', ', pref_name) AS pref_city
     -- if PostgreSQL, Redshift
     , city_name || ', ' || pref_name AS pref_city2
  FROM mst_user_location 
;

Redshift also supports CONCAT function. However, this function in Redshift allows only 2 arguments.

eg. CONCAT(‘a’, ‘b’) (O)
eg. CONCAT(‘a’, ‘b’, ‘c’) (X)

3.6.2. Compare multiple values

Create table and insert sample data

Data in this table illustrates quarterly sales
Null means that the sales haven’t been confirmed yet

-- DROP TABLE IF EXISTS quarterly_sales;
-- Create quarterly sales table
CREATE TABLE quarterly_sales (
    year integer
  , q1   integer
  , q2   integer
  , q3   integer
  , q4   integer
);

-- Insert sample data
INSERT INTO quarterly_sales
VALUES
    (2017, 82000, 83000, 78000, 83000)
  , (2018, 85000, 85000, 80000, 81000)
  , (2019, 92000, 81000, NULL , NULL )
;

-- Select sample data
SELECT *
  FROM quarterly_sales
;

Find out whether the sales of a quater increased or decreased

Print ‘+’ when sales increased and print ‘-‘ when sales decreased. And also, if there was no change, print ‘ ‘(space)
You can use CASE and also SIGN function is available
SIGN returns 1 from a positive number argument or -1 from a negative number one or 0 from 0 argument

-- Find out the fluctuation of sales quaterly
SELECT year 
     , q1
     , q2
     , CASE WHEN q1 < q2 THEN '+'
            WHEN q1 = q2 THEN ' '
            ELSE '-'
            END AS judge_q1_q2
     , q2 - q1 AS diff_q2_q1
     , SIGN(q2 - q1) AS sign_q2_q1
  FROM quarterly_sales
 ORDER BY year
;

Extract maximum and minimum sales

You can utilize greatest, least functions
These functions are not the standard SQL, but most SQL query engines support them

-- Find out the max/min sales of a year
SELECT year 
     -- Maximum sales
     , greatest(q1, q2, q3, q4) AS greatest_sales
     -- Minimum sales
     , greatest(q1, q2, q3, q4) AS least_sales
  FROM quarterly_sales
 ORDER BY year
;

Calculate the average of a year

You have to consider whether null value is exists or not

-- Calculate the average of a year
SELECT year 
     -- Invalid: It doesn't replace null to 0. The result of an arithmetic operation with null is null
     , (q1 + q2 + q3 + q4) / 4 AS null_average
     -- Invalid: You have to skip q3, q4 in 2019. But, it always divide total value into 4
     , (COALESCE(q1, 0) + COALESCE(q2, 0) + COALESCE(q3, 0) + COALESCE(q4, 0)) / 4 AS invalid_denominator_average
     -- Valid
     , (COALESCE(q1, 0) + COALESCE(q2, 0) + COALESCE(q3, 0) + COALESCE(q4, 0)) 
       / (SIGN(COALESCE(q1, 0)) + SIGN(COALESCE(q2, 0)) + SIGN(COALESCE(q3, 0)) + SIGN(COALESCE(q4, 0))) AS correct_average
  FROM quarterly_sales
 ORDER BY year 
;

3.6.3. Calculate the proportion

Create table and insert sample data

--DROP TABLE IF EXISTS advertising_stats;
-- Create advertising status table
CREATE TABLE advertising_stats (
    dt          varchar(255)
  , ad_id       varchar(255)
  , impressions integer
  , clicks      integer
);

-- Insert sample data
INSERT INTO advertising_stats
VALUES
    ('2020-01-28', '001', 100000,  3000)
  , ('2020-01-28', '002', 120000,  1200)
  , ('2020-01-28', '003', 500000, 10000)
  , ('2020-01-29', '001',      0,     0)
  , ('2020-01-29', '002', 130000,  1400)
  , ('2020-01-29', '003', 620000, 15000)
;

commit;

-- Select sample data
SELECT *
  FROM advertising_stats 
;

Calculate CTR(Clikc Through Rate)

CTR : The frequency of clicks / The frequency of exposure(impression)
clicks and impression column are integer type
To get double presion result, you have to convert integer into double through CAST function
Hive, SparkSQL, Redshift, BigQuery support auto conversion when an integer number is divided by an integer number, while you have to convert type explicitly in PostgreSQL

-- Calculate CTR(Click through rate)
SELECT dt
     , ad_id
     , CAST(clicks AS double precision) / impressions AS ctr
     -- 100.0(double) * clicks(integer) -> double
     , 100.0 * clicks / impressions AS ctr_as_percent
     -- if Hive, SparkSQL, Redshift, BigQuery => Auto conversion 
     -- , clicks / impressions AS ctr
  FROM advertising_stats 
 WHERE dt = '2020-01-28'
 ORDER BY dt, ad_id
;

Be careful to divide by zero

If you try to divide clicks by zero, you’ll face the error like below

To avoid this error, you have to replace 0 with null through CASE or NULLIF function

-- Calculate CTR without error
SELECT dt
     , ad_id
     -- use CASE clause
     , CASE WHEN impressions > 0 THEN 100.0 * clicks / impressions 
            END AS ctr_percent_by_case
     -- if PostgreSQL, SparkSQL, Redshift, BigQuery
     -- The result of arithmetic operation with null is null
     , 100.0 * clicks / NULLIF(impressions , 0) AS ctr_as_percent_by_null
     -- if Hive (Not support NULLIF)
     -- , 100.0 * clicks / CASE WHEN imperssions = 0 THEN NULL ELSE impressions END AS ctr_as_percent_by_null
  FROM advertising_stats 
 ORDER BY dt, ad_id
;

3.6.4. Calculate the distance between two points

Create a dimension table and insert sample data

-- DROP TABLE IF EXISTS location_1d;
-- Create a dimension table
CREATE TABLE location_1d (
    x1 integer
  , x2 integer
);

-- Insert sample data
INSERT INTO location_1d
VALUES
    ( 5 , 10)
  , (10 ,  5)
  , (-2 ,  4)
  , ( 3 ,  3)
  , ( 0 ,  1)
;

commit;

-- Select sample data
SELECT *
  FROM location_1d 
;

Calculate the distance between x1 and x2

You can utilize ABS, POWER, SQRT functions

Function	Description
ABS	Return absolute-value
POWER	Return the square of parameter value
SQRT	Return the square root of parameter value

-- Calculate the distance between x1 and x2
SELECT abs(x1 - x2) AS abs
     , sqrt(power(x1 - x2, 2)) AS rms
  FROM location_1d 
;

Create two dimensions table and insert sample data

-- DROP TABLE IF EXISTS location_2d;
-- Create two dimensions table
CREATE TABLE location_2d (
    x1 integer
  , y1 integer
  , x2 integer
  , y2 integer
);

-- Insert sample data
INSERT INTO location_2d
VALUES
    (0, 0, 2, 2)
  , (3, 5, 1, 2)
  , (5, 3, 2, 1)
;

commit;

-- Select sample data
SELECT *
  FROM location_2d 
;

Calculate the Euclidean distance between (x1, y1) and (x2, y2)

You can calculate it through POWER and SQRT
PostgreSQL supports POINT data type and <–> the distance operation

-- Calculate the Euclidean distance between (x1, y1) and (x2, y2)
SELECT sqrt(power(x1 - x2, 2) + power(y1 - y2, 2)) AS distance 
     , point(x1, y1) <-> point(x2, y2) as dist_point
  FROM location_2d 
;

3.6.5. Calculate Date/Time

Create table and insert sample data

-- DROP TABLE IF EXISTS mst_users_with_dates;
-- Create user table with user's birthday and register datetime
CREATE TABLE mst_users_with_dates (
    user_id        varchar(255)
  , register_stamp varchar(255)
  , birth_date     varchar(255)
);

-- Insert sample data
INSERT INTO mst_users_with_dates
VALUES
    ('U001', '2019-02-20 10:00:00', '2000-02-19')
  , ('U002', '2019-02-21 10:00:00', '2000-03-16')
  , ('U003', '2019-03-01 10:00:00', '2000-04-09')
;

commit;

-- Select sample data
SELECT *
  FROM mst_users_with_dates 
;

Calculate date/time

Hive, SparkSQL don’t support date/time functions. So, you have to convert string data into unixtime and convert it into timestamp data type

-- Calculate date/time
SELECT user_id
     , register_stamp::timestamp AS register_stamp
     , register_stamp::timestamp + '1 hour'::interval AS after_1_hour
     , register_stamp::timestamp - '30 minutes'::interval AS before_30_minutes
     , register_stamp::date AS register_date
     , register_stamp::date + '1 day'::interval AS after_1_day
     , register_stamp::date - '1 month'::interval AS before_1_month
     --
     -- if Hive, SparkSQL (Not support timstamp operation functions)
     -- , CAST(register_stamp AS timestamp) AS register_stamp
     -- , from_unixtime(unix_tiestamp(register_stamp) + 60 * 60) AS after_1_hour
     -- , from_unixtime(unix_tiestamp(register_stamp) - 30 * 60) AS before_30_minutes
     -- , to_date(register_stamp) AS register_date
     -- , date_add(to_date(register_stamp), 1) AS after_1_day
     -- , add_months(to_date(register_stamp), -1) AS before_1_month
     --
     -- if Redshift (dateadd)
     -- , register_stamp::timestamp AS register_stamp
     -- , dateadd(hour, 1, register_stamp::timestamp) AS after_1_hour
     -- , dateadd(minute, -30, register_stamp::timestamp) AS before_30_minutes
     -- , register_stamp::date AS register_date
     -- , dateadd(day, 1, register_stamp::date) AS after_1_day
     -- , dateadd(month, -1, register_stamp::date) AS before_1_month
     --
     -- if BigQuery (timestamp_add, timestamp_sub, date_add, date_sub)
     -- , timestamp(register_stamp) AS register_stamp
     -- , timestamp_add(timestamp(register_stamp), interval 1 hour) AS after_1_hour
     -- , timestamp_sub(timestamp(register_stamp), interval 30 minute) AS before_30_minutes
     -- , date(register_stamp) AS register_date
     -- , date_add(date(register_stamp), interval 1 day) AS after_1_day
     -- , date_sub(date(register_stamp), interval 1 month) AS before_1_month
   FROM mst_users_with_dates
;

Calculate the difference between date/time data

-- Calculate the difference between today and register_stamp
SELECT user_id
     -- if PostgreSQL, Redshift
     , CURRENT_DATE AS today
     , register_stamp::date AS register_date
     , CURRENT_DATE - register_stamp::date AS diff_days
     --
     -- if Hive, SparkSQL (datediff)
     -- , CURRENT_DATE() AS today
     -- , to_date(register_stamp) AS register_date
     -- , datediff(CURRENT_DATE(), to_date(register_stamp)) AS diff_days
     --
     -- if BigQuery (date_diff)
     -- , CURRENT_DATE AS today
     -- , date(timestamp(register_stamp)) AS register_date
     -- , date_diff(CURRENT_DATE, date(timestamp(register_stamp)), day) AS diff_days
  FROM mst_users_with_dates muwd 
;

Calculate user’s age from the birthday

PostgreSQL supports age function and it returns days
- age(arg) -> current age from arg
- arg(arg1, arg2) -> age from arg1 to arg2
  
  Redshift also supports age function, but it is not official
You can get YEAR through EXTRACT function

-- Calculate user's age from the birthday
SELECT user_id
     , CURRENT_DATE AS today
     , register_stamp::date AS register_date
     , birth_date::date AS birth_date
     , EXTRACT(YEAR FROM age(birth_date::date)) AS current_age
     , EXTRACT(YEAR FROM age(register_stamp::date, birth_date::date)) AS register_age
     , age(register_stamp::date, birth_date::date) AS register_age2
  FROM mst_users_with_dates muwd 
;

Calculate year difference

Redshift has datediff and BigQuery has date_diff function for getting the year difference
However, it is not exact to calculate the age because it just returns the year difference
- eg. datediff(year, ‘2019-12-31’, ‘2020-01-01’) returns 1, nevertheless the age is 0

-- Calculate year difference
SELECT user_id
     -- if Redshift
     , CURRENT_DATE AS today
     , register_stamp::date AS register_date
     , birth_date::date AS birth_date
     , datediff(year, birth_date::date, CURRENT_DATE) AS current_age
     , datediff(year, birth_date::date, register_stamp::date) AS register_age
     --
     -- if BigQuery
     -- , CURRENT_DATE AS today
     -- , date(timestamp(register_stamp)) AS register_date 
     -- , date(timestamp(birth_date)) AS birth_date 
     -- , date_diff(CURRENT_DATE, date(tiemstamp(birth_date)), year) AS current_age
     -- , date_diff(date(timestamp(registeR_stamp)), date(timestamp(birth_date)), year) AS register_age
  FROM mst_users_with_dates
;

How to calculate the age exactly without a special function like age

It is possible to get exact age by converting date into integer

-- How to calculate the age exactly without a special function like age
SELECT birth_date_integer 
     , current_date_integer 
     , floor((current_date_integer - birth_date_integer) / 10000) AS age
  FROM (SELECT EXTRACT(YEAR from birth_date::date) * 10000 + EXTRACT(MONTH from birth_date::date) * 100 + EXTRACT(YEAR from birth_date::date) AS birth_date_integer
             , EXTRACT(YEAR from CURRENT_DATE) * 10000 + EXTRACT(MONTH from CURRENT_DATE) * 100 + EXTRACT(YEAR from CURRENT_DATE) AS current_date_integer
          FROM mst_users_with_dates muwd)
;

3.6.6. Handle IP address

There are many log data which has IP addresses
Generally, IP address is stored as string data type. But, there are lots of cases to compare IP addresses

Use IP address data type

PostgreSQL supports inet data type which is useful to compare IP addresses

-- Compare IP addresses
SELECT CAST('127.0.0.1' AS inet) < CAST('127.0.0.2' AS inet) AS lt
     , CAST('127.0.0.1' AS inet) > CAST('192.168.0.1' AS inet) AS gt
;

Find out whether an IP address is contained or not using « or » functions

-- Find out whether an IP address is contained or not
SELECT CAST('127.0.0.1' AS inet) << CAST('127.0.0.0/8' AS inet) AS is_contained;

Handle IP address as integer or string

Convert IP address into integer data type

-- Convert IP address into integer data type
SELECT ip
     -- if PostgreSQL, Redshift (split_part)
     , CAST(split_part(ip, '.', 1) AS integer) AS ip_part_1
     , CAST(split_part(ip, '.', 2) AS integer) AS ip_part_2
     , CAST(split_part(ip, '.', 3) AS integer) AS ip_part_3
     , CAST(split_part(ip, '.', 4) AS integer) AS ip_part_4
     --
     -- if Hive, SparkSQL
     -- , CAST(split(ip, '\\.')[0] AS int) AS ip_part_1
     -- , CAST(split(ip, '\\.')[1] AS int) AS ip_part_2
     -- , CAST(split(ip, '\\.')[2] AS int) AS ip_part_3
     -- , CAST(split(ip, '\\.')[3] AS int) AS ip_part_4
     --
     -- if BigQuery
     -- , CAST(split(ip, '.')[0] AS int) AS ip_part_1
     -- , CAST(split(ip, '.')[1] AS int) AS ip_part_2
     -- , CAST(split(ip, '.')[2] AS int) AS ip_part_3
     -- , CAST(split(ip, '.')[3] AS int) AS ip_part_4
  FROM (SELECT CAST('192.168.0.1' AS text) AS ip) AS t
       -- Other databases
       -- (SELECT '192.168.0.1' AS ip) AS t
;

You can get one integer from IP address
- Multiply 4 parts from IP address and 2^24, 2^16, 2^8, 2^0 respectively

-- Convert IP address into integer
SELECT ip
     -- if PostgreSQL, Redshift (split_part)
     , CAST(split_part(ip, '.', 1) AS integer) * 2^24
       + CAST(split_part(ip, '.', 2) AS integer) * 2^16
       + CAST(split_part(ip, '.', 3) AS integer) * 2^8
       + CAST(split_part(ip, '.', 4) AS integer) * 2^0
      AS ip_integer
     --
     -- if Hive, SparkSQL
     -- , CAST(split(ip, '\\.')[0] AS int) * pow(2, 24)
     --   + CAST(split(ip, '\\.')[1] AS int) * pow(2, 16)
     --   + CAST(split(ip, '\\.')[2] AS int) * pow(2, 8)
     --   + CAST(split(ip, '\\.')[3] AS int) * pow(2, 0)
     --
     -- if BigQuery
     -- , CAST(split(ip, '.')[SAFE_ORDINAL(1)] AS int64) * pow(2, 24)
     --   + CAST(split(ip, '.')[SAFE_ORDINAL(2)] AS int64) * pow(2, 16)
     --   + CAST(split(ip, '.')[SAFE_ORDINAL(3)] AS int64) * pow(2, 8)
     --   + CAST(split(ip, '.')[SAFE_ORDINAL(4)] AS int64) * pow(2, 0)
     --  AS ip_integer
  FROM (SELECT CAST('192.168.0.1' AS text) AS ip) AS t
       -- Other databases
       -- (SELECT '192.168.0.1' AS ip) AS t
;

Convert IP address into string with 0 padding

-- Convert IP address into string with 0 padding
SELECT ip
     -- if PostgreSQL, Redshift (lpad)
     , lpad(split_part(ip, '.', 1), 3, '0')
       || lpad(split_part(ip, '.', 2), 3, '0')
       || lpad(split_part(ip, '.', 3), 3, '0')
       || lpad(split_part(ip, '.', 4), 3, '0')
       AS ip_padding
     --
     -- if Hive, SparkSQL
     -- , CONCAT(lpad(split(ip, '\\.')[0], 3, '0')
     --        , lpad(split(ip, '\\.')[1], 3, '0')
     --        , lpad(split(ip, '\\.')[2], 3, '0')
     --        , lpad(split(ip, '\\.')[3], 3, '0')
     --   ) AS ip_padding
     --
     -- if BigQuery
     -- , CONCAT(lpad(split(ip, '.')[SAFE_ORDINAL(1)], 3, '0')
     --        , lpad(split(ip, '.')[SAFE_ORDINAL(2)], 3, '0')
     --        , lpad(split(ip, '.')[SAFE_ORDINAL(3)], 3, '0')
     --        , lpad(split(ip, '.')[SAFE_ORDINAL(4)], 3, '0')
     --   ) AS ip_padding
  FROM (SELECT CAST('192.168.0.1' AS text) AS ip) AS t
       -- Other databases
       -- (SELECT '192.168.0.1' AS ip) AS t
;

References

데이터 분석을 위한 SQL 레시피 - 한빛미디어

[Data Analysis] SQL Recipe for Data Analysis Tutorial #2 - Manipulating a single value

Mon, 27 Jan 2020 00:00:00 +0000

Goal

Studying SQL for data analysis
Practicing chapter 3rd in the book
Studying SQL for Manipulating data

Practice

Book for studying

Title	데이터 분석을 위한 SQL 레시피
Author	Nagato Kasaki, Naoto Tamiya
Translator	윤인성
Publisher	한빛미디어
Chapter	3
Lesson	5

I’m gonna study basic SQL syntax and functions for data analysis
I’m gonna study how to manipulate from a single value to multiple tables
I’m gonna try to load test data on PostgreSQL and execute queries in the book

3.5. Manipulating a single value

Download tutorial SQL file

3.5.1. Replace the code value with a label

To replace the code value with some label to enhance the readability

Make sample table and data

Create user master table which has 3 colums
register_device column has 3 types of code data
- 1: Desktop
- 2: Smartphone
- 3: Application

-- Create user master table
CREATE TABLE mst_users(
    user_id         varchar(255)
  , register_date   varchar(255)
  , register_device integer
);

-- Insert sample data
INSERT INTO mst_users
VALUES
    ('U001', '2020-08-26', 1)
  , ('U002', '2016-08-26', 2)
  , ('U003', '2016-08-27', 3)
;

commit;

-- Select sample data
SELECT *
  FROM mst_users
;

Execute Query

Use CASE clause

-- Query for replacing code with label
SELECT user_id
     , CASE WHEN register_device = 1 THEN 'Desktop'
            WHEN register_device = 2 THEN 'Smartphone'
            WHEN register_device = 3 THEN 'Application'
            -- ELSE (Default)
            END AS device_name
  FROM mst_users
;

3.5.2. Extract some factors from URL

Create access log table

This table has 3 columns such as referrer and url
There are several databases that have URL data type

-- DROP TABLE IF EXISTS access_log ;
-- Create access log table
CREATE TABLE access_log (
    stamp    varchar(255)
  , referrer text
  , url      text
);

-- Insert sample data
INSERT INTO access_log 
VALUES
    ('2020-01-27 12:02:00', 'https://www.other.com/path1/index.php?k1=v1&k2=v2#Ref1', 'http://www.example.com/video/detail?id=001')
  , ('2020-01-27 12:02:01', 'https://www.other.net/path1/index.php?k1=v1&k2=v2#Ref1', 'http://www.example.com/video#ref'          )
  , ('2020-01-27 12:02:01', 'https://www.other.com/'                                , 'http://www.example.com/book/detail?id=002' )
;

commit;

-- Select sample data
SELECT *
  FROM access_log
;

Extract the domain of a referrer

Furthermore, Hive and BigQuery have the function hadling URL
Unfortunately, you have to extract the domain by using regex expression if you are using other databases

HTTP referer : (a misspelling of referrer[1]) is an optional HTTP header field that identifies the address of the webpage (i.e. the URI or IRI) which is linked to the resource being requested. By checking the referrer, the new webpage can see where the request originated

-- Extract the domain of referrer
SELECT stamp
     , substring(referrer from 'https?://([^/]*)') AS referrer_domain
     -- if Redshift
     -- , regexp_replace(regexp_substr(referrer, 'https?://[^/]*'), (https?://', '') AS referrer_domain
     -- if Hive or SparkSQL
     -- , parse_url(refferer, 'HOST') AS referrer_domain
     -- if BigQuery
     -- , host(referrer) AS referrer_domain
  FROM access_log 
;

Extract the path and query parameters from URL

-- Extract the path and query parameters from URL
SELECT stamp
     , url
     , substring(url from '//[^/]+([^?#]+)') AS path 
     , substring(url from 'id=([^%]*)') AS id
     -- if Redshift
     -- , regexp_replace(regexp_substr(referrer, '//[^/]+([^?#]+)'), '//[^/]+', '') AS path
     -- , regexp_replace(regexp_substr(referrer, 'id=([^%]*)', 'id=', '') AS id
     -- if Hive or SparkSQL
     -- , parse_url(refferer, 'PATH') AS path
     -- , parse_url(refferer, 'QUERY') AS id
     -- if BigQuery
     -- , parse_url(url, '//[^/]+([^?#]+)') AS path 
     -- , parse_url(url, 'id=([^%]*)') AS id
  FROM access_log 
;

3.5.3. Separate string into array

There are many cases that you have to separate string by comma or space or anything else
I’m gonna show path level from URL in access_log table

-- Extract path hierarchy
SELECT stamp
     , url
     , split_part(substring(url from '//[^/]+([^?#]+)'), '/', 2) AS path1
     , split_part(substring(url from '//[^/]+([^?#]+)'), '/', 3) AS path2
     -- if Redshift
     -- , split_part(regexp_replace(regexp_substr(url, '//[^/]+([^?#]+'), '//[^/]+([^?#]+', ''), '/' 2) AS path1
     -- , split_part(regexp_replace(regexp_substr(url, '//[^/]+([^?#]+'), '//[^/]+([^?#]+', ''), '/' 3) AS path2
     -- if Hive or SparkSQL (Index number starts from 0)
     -- , split(parse_url(url, 'PATH'), '/')[1] AS path1
     -- , split(parse_url(url, 'PATH'), '/')[2] AS path2
     -- if BigQuery
     -- , split(regexp_extract(url, '//[^/]+([^?#]+)'), '/')[SAFE_ORDINAL(2)] AS path1
     -- , split(regexp_extract(url, '//[^/]+([^?#]+)'), '/')[SAFE_ORDINAL(3)] AS path2
  FROM access_log 
;

Redshift doesn’t support array data type officially. However, you can get Nth factor from separated string through split_part function

Generally, array index is 1 based, while array index is 0 based in Hive and SparkSQL

In BigQuery, there are several ways to access a specific index

OFFSET: 0 Based & Error occurs when you access the index which exceeds the boundary of the array
ORDINAL: 1 Based & Error occurs when you access the index which exceeds the boundary of the array
SAFE_OFFSET: 0 Based & Return null when you access the index which exceeds the boundary of the array
SAFE_ORDINAL: 1 Based & Return null when you access the index which exceeds the boundary of the array

3.5.4. Handling date and timestamp

Extract current date and current timestamp

-- Extract current date and current timestamp
SELECT 
     -- if PostgreSQL, Hive, BigQuery
       CURRENT_DATE as dt
     , CURRENT_TIMESTAMP as stamp
     -- if Hive, SparkSQL, BigQuery
     --   CURRENT_DATE() as dt
     -- , CURRENT_TIMESTAMP() as stamp
     -- if Redshift
     --   CURRENT_DATE as dt
     -- , GETDATE() as stamp
     -- if PostgreSQL, timestamp without timezone
     , LOCALTIMESTAMP as local_stamp
;

Generally, the functions related to timestamp return timestamp without timezone, while CURRENT_TIMESTAMP in PostgreSQL returns timestamp with timezone

In contrast, LOCALTIMESTAMP in PostgreSQL returns timestamp without timezone

In BigQuery, CURRENT_TIMESTAMP or CURRENT_TIMESTAMP() returns UTC time

Convert string data type into date/timestamp

-- Convert string data type into date/timestamp
SELECT 
     -- if PostgreSQL, Hive, SparkSQL, Redshift, BigQuery
       CAST('2020-01-27' AS date) AS dt
     , CAST('2020-01-27 23:15:10' AS timestamp) AS stamp
     -- if Hive, BigQuery
     --   date('2020-01-27') AS dt
     -- , timestamp('2020-01-27 23:15:10') AS stamp
     -- if PostgreSQL, Hive, SparkSQL, Redshift, BigQuery. * Value must be not column name but constant value
     --   date('2020-01-27') AS dt
     -- , timestamp('2020-01-27 23:15:10') AS stamp
     -- if PostgreSQL, Redshift
     , '2020-01-27'::date AS dt2
     , '2020-01-27 23:15:10'::timestamp AS stamp2
;

Extract a specific field from date/timestamp

-- Extract a specific field from date/timestamp
SELECT stamp
     -- if PostgreSQL, Redshift, BigQuery
     , EXTRACT(YEAR from stamp) AS year
     , EXTRACT(MONTH from stamp) AS month
     , EXTRACT(DAY from stamp) AS day
     , EXTRACT(HOUR from stamp) AS hour
     -- if Hive, SparkSQL
     -- , YEAR(stamp) AS year
     -- , MONTH(stamp) AS month
     -- , DAY(stamp) AS day
     -- , HOUR(stamp) AS hour
  FROM (SELECT CAST('2020-01-27 23:26:39' AS timestamp) AS stamp) AS t
;

3.5.5. Replace null with defaul value

The result of arithmetic operation with NULL

Create sample table and insert sample data

-- DROP TABLE IF EXISTS purchase_log_with_coupon;
-- Create purchage log table
CREATE TABLE purchase_log_with_coupon (
    purchase_id varchar(255)
  , amount      integer
  , coupon      integer
);

-- Insert sample data
INSERT INTO purchase_log_with_coupon
VALUES
    ('10001', 3280, NULL)
  , ('10002', 4650,  500)
  , ('10003', 3870, NULL)
;

commit;

-- Select sample data
SELECT *
  FROM purchase_log_with_coupon 
;

Compare the sum with null to the one with default value

-- Calculate the total sales except for coupon
SELECT purchase_id
     , amount
     , coupon
     , amount - coupon AS discount_amount_invalid
     , amount - COALESCE(coupon, 0) AS discount_amount_valid
  FROM purchase_log_with_coupon 
;

References

데이터 분석을 위한 SQL 레시피 - 한빛미디어
HTTP Referer Wiki

[Data Analysis] SQL Recipe for Data Analysis Tutorial #1 - Setup the environment

Thu, 23 Jan 2020 00:00:00 +0000

Goal

Studying SQL for data analysis
Install PostgreSQL docker for buildiing testing environment

Practice

Book for studying

Title	데이터 분석을 위한 SQL 레시피
Author	Nagato Kasaki, Naoto Tamiya
Translator	윤인성
Publisher	한빛미디어

I’m gonna study SQL syntax and functions for data analysis
I’m gonna try to load test data on PostgreSQL and execute queries in the book
First of all, I have to install PostgreSQL for testing and I’ll build the environment by using docker image
And docker is already installed in my system

1. Install

1.1. Check local docker images

$ docker image ls

1.2. Download docker image

Search docker image on DockerHub

$ docker search [IMAGE]

Download PostgreSQL image

I downloaded 12.1 version

$ docker pull [IMAGE][{:TAG}]
$ docker image ls

2. Run PostgreSQL instance

2.1. Create data directory

I’m gonna set data directory on my local directory not on docker image

$ mkdir [DATA_DIRECTORY]

2.2. Run instance

Options
- Environment variables

Variable	Comment
POSTGRES_PASSWORD	Sets the superuser password for PostgreSQL
POSTGRES_USER	Create the specified user with superuser power and a database with the same name. If it is not specified, then the default user of postgres will be used
POSTGRES_DB	ㅇefine a different name for the default database that is created when the image is first started. If it is not specified, then the value of POSTGRES_USER will be used
PGDATA	The default is /var/lib/postgresql/data, but if the data volume you’re using is a filesystem mountpoint (like with GCE persistent disks), Postgres initdb recommends a subdirectory (for example /var/lib/postgresql/data/pgdata ) be created to contain the data

$ docker run --name [CONTAINER_NAME] -e POSTGRES_PASSWORD=[MY_PASSWORD] -d postgres

Check running process

$ docker ps

3. Connection Test

3.1. Connect PostgreSQL by DB tool

I used DBeaver which is open-source database tool

3.2. Create table test

docker start -i -p 5432:5432/tcp postgres-da

References

데이터 분석을 위한 SQL 레시피 - 한빛미디어
DockerHub PostgreSQL

[MacOS] Macs Fan Control

Tue, 10 Sep 2019 00:00:00 +0000

Goal

What is Macs Fan Control?
How to install

Practice

Macs Fan Control

Monitors and controls fans on Apple computers
Real-time monitoring of fan speed and temperature sensors including 3rd party HDD/SSD (using S.M.A.R.T.)
Set custom RPM value (revolution per minute) or control fan speed in relation to a temperature sensor
Every model of iMac, MacBook (Pro/Air), Mac mini, Mac Pro is supported

1. Install

Download Site

Unzip file

Move app file to Applications directory

2. Run

You can set fixed rate
Also, can set auto speed followed by specific sensor

References

Macs Fan Contorl Site

[MacOS] htop

Tue, 10 Sep 2019 00:00:00 +0000

Goal

What is htop?
How to install

Practice

HTOP

Process manager to monitor system processes
It was developed to replace top which was UNIX basic process manager
It illustrates whole processes which are running, while top displays only several processes that occupy large resources
It enhances visual effect by colors and describes the status of processors, swap, memory and so on
It uses Ncurses libaray and is programmed by C language

1. Install

Install using brew

$ brew install htop

2. Run

$ htop

References

htop wiki

[Java] Jackson JSON 'Unrecognized token ***

Mon, 09 Sep 2019 00:00:00 +0000

Problem

I receive JSON data from Django REST Server
However, it caused ‘Unrecognized token’ error
In my case, this error occured by ‘True’ and ‘None’

2019-09-06 17:20:57.019 28853-28945/*** E/dorbae: Failed to get DB data.
    com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'True': was expecting ('true', 'false' or 'null')
     at [Source: (byte[])"{'name': 'WFS', 'columns': ['time', 'current'], 'values': [['2019-08-24T15:20:31+09:00', 7], ['2019-08-24T11:28:41+09:00', 6], ['2019-08-24T06:29:02+09:00', 4], ['2019-08-24T05:50:00+09:00', 4], ['2019-08-24T02:15:17+09:00', 2], ['2019-08-24T02:15:12+09:00', 2], ['2019-08-23T18:05:16+09:00', 2], ['2019-08-23T17:24:44+09:00', 4], ['2019-08-23T17:22:04+09:00', 2], ['2019-08-23T17:21:59+09:00', 2], ['2019-08-23T06:36:24+09:00', 4], ['2019-08-23T03:45:43+09:00', 2], ['2019-08-23T03:45:38+09:00', 2],"[truncated 1953313 bytes]; line: 1, column: 1953814]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3532)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2627)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1053)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:295)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3091)
        at 
...

2019-09-06 17:30:03.244 29666-29758/*** E/dorbae: com.fasterxml.jackson.databind.JsonMappingException: Unrecognized token 'None': was expecting 'null', 'true', 'false' or NaN
        at [Source: (byte[])"{'name': 'qbsiotseries', 'columns': ['time', 'TEMPERATURE', 'HUMIDITY', 'CO2PPM', 'ILLUMINANCE', 'VALUE00', 'VALUE01', 'VALUE02', 'VALUE03'], 'values': [['2019-08-23T08:00:00Z', None, None, None, None, None, None, None, None], ['2019-08-23T09:00:00Z', None, None, None, None, None, None, None, None], ['2019-08-23T10:00:00Z', None, None, None, None, None, None, None, None], ['2019-08-23T11:00:00Z', None, None, None, None, None, None, None, None], ['2019-08-23T12:00:00Z', None, None, None, None, No"[truncated 25509 bytes]; line: 1, column: 179] (through reference chain: ***.InfluxDBResultSet["values"]->java.lang.Object[][0])
        at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:206)
        at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:21)
        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3091)
        at 
 
 ...

Cause

My architecture

Jaskcon libarary was not able to recognize the tokens such as ‘True’, ‘None’ and my django server didn’t nomalize json data

Solution

1. Nomalize JSON data in Django server

json.dumps method nomalizes JSON data followed by ECMA-404
True -> true & None -> null
Add this code

import json

...

json.dumps(jsonObject)

...

 client = InfluxDBClient(host, port, user, password, db_name)
    result = client.query(query)
    _value = ''

    _series = result.raw.get('series')

    if _series is not None:
        # None -> Null & Object -> Object[]
        _value = json.dumps(_series)
        return HttpResponse(_value, status=200)
    # return HTTP Response
    return HttpResponse(_value, status=404)

2. Skip null data in Android

Add this code

...

ObjectMapper mapper = new ObjectMapper();

...

mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);

...

ObjectMapper mapper = new ObjectMapper();
mapper.configure(JsonParser.Feature.ALLOW_SINGLE_QUOTES, true); // 싱글 쿼테이션 허용
mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
InfluxDBResultSet[] influxDBResultSetArray = mapper.readValue(data, InfluxDBResultSet[].class);

References

[IoT] What is MQ2?

Wed, 04 Sep 2019 00:00:00 +0000

Goal

What is MQ2
How to operate

Practice

What is MQ2

MOS(Metal Oxide Semiconductor) type gas sensor
Can detect LPG, Smoke, Alcohol, Propane, Hydrogen, Methan, Carbon Monoxide(CO)
Detects gas using the change of resistance of the sensing material

Specification


Operating Voltage	5V
Load Resistance	20 KΩ
Heater Resistance	33Ω ± 5%
Heating Consumption	< 800mw
Sensing Resistance	10 KΩ - 60 KΩ
Concentration Scope	~~200~~ 300 - 10,000 ppm)
Preheat Time	Over 24 hours

What is ppm(Parts-per-million)

Describe the amount of gas by volume in the air

The ratio of one gas to another

Internal Structure

MQ2 is enclosed in two layers of fine stainless steel mesh. It called Anti-explosion network
It prevents an explosion by heater element which is inside the sensor
- MQ2 is sensing flammable gases such as LPG, Propane and so on
It leads only gaseous elements to pass inside the chamber

The star-shaped structure
H
- Heats the sensing element and are connected through **Nickel-Chrominum coil(conductive alloy)
A & B
- Output signals to platinum wires
- It is connected to the body of the sensing element
- Conveys small changes in the current

The Tin Dioxide(SnO₂) is the most important material being sensitive towards combustible gases
The ceramic substrate merely increases heating efficiency and ensures the sensor area is heated to a working temperature constantly

How deos it work?

The tin dioxide is heated in air at high temperature

Oxygen is adsorbed on the surface
- Donor electrons in the tin dioxide are attracted toward oxygen
- This situation prevents electric current flow and the voltage goes down

Oxygen decreased as it reacts with gases
- Eletrons are then released into the tin dioxide
- Then, it allows current to flow freely through the snesor and the voltage goes up

i.e. Cencentration of gases ∝ Voltage
- Cencentration of gases ↑ -> Voltage ↑
- Cencentration of gases ↓ -> Voltage ↓

References

How MQ2 Gas/Smoke Sensor Works?

[Eclipse RCP] How to fix encoding problem when builing a progrma

Tue, 03 Sep 2019 00:00:00 +0000

Problem

After building my application, text(Korean) in labels were cracked

Cause

My environment
- Eclipse workspace encoding : UTF-8
- System default encoding : MS949(Windows 10)
When exporting RCP application, it was compiled by default text encoding which was used by the IDE

Solution

At first, I tried to add -Dfile.encoding=UTF8 vm option in [APPLICATION_LAUNCHER_NAME].int file. However, it didn’t work well
Because, text was already cracked when it was compiled

So, I tried to add javacDefaultEncoding.. = UTF-8 option in build.properties file

After that, it did work well

References

Stack Overflow

[Android] Build error : Execution failed for task ':app:transformDexWithInstantRunslicesApkForDebug after upgrading Android Studio

Tue, 13 Aug 2019 00:00:00 +0000

Goal

Solve the problem error related with build

Problem

I upgraded Android Studio 3.2.1 -> 3.4.2

After that, when I built a program, the error occured like below

org.gradle.execution.MultipleBuildFailures: Build completed with 1 failures.
	at org.gradle.initialization.DefaultGradleLauncher$ExecuteTasks.run(DefaultGradleLauncher.java:386)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:301)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:293)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
	at org.gradle.initialization.DefaultGradleLauncher.runTasks(DefaultGradleLauncher.java:247)
	at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:159)
	at org.gradle.initialization.DefaultGradleLauncher.executeTasks(DefaultGradleLauncher.java:134)
	at org.gradle.internal.invocation.GradleBuildController$1.execute(GradleBuildController.java:58)
	at org.gradle.internal.invocation.GradleBuildController$1.execute(GradleBuildController.java:55)
	at org.gradle.internal.invocation.GradleBuildController$3.create(GradleBuildController.java:82)
	at org.gradle.internal.invocation.GradleBuildController$3.create(GradleBuildController.java:75)
	at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:183)
	at org.gradle.internal.work.StopShieldingWorkerLeaseService.withLocks(StopShieldingWorkerLeaseService.java:40)
	at org.gradle.internal.invocation.GradleBuildController.doBuild(GradleBuildController.java:75)
	at org.gradle.internal.invocation.GradleBuildController.run(GradleBuildController.java:55)
	at org.gradle.tooling.internal.provider.runner.ClientProvidedBuildActionRunner.run(ClientProvidedBuildActionRunner.java:55)
	at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
	at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
	at org.gradle.launcher.exec.BuildOutcomeReportingBuildActionRunner.run(BuildOutcomeReportingBuildActionRunner.java:58)
	at org.gradle.tooling.internal.provider.ValidatingBuildActionRunner.run(ValidatingBuildActionRunner.java:32)
	at org.gradle.launcher.exec.BuildCompletionNotifyingBuildActionRunner.run(BuildCompletionNotifyingBuildActionRunner.java:39)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner$3.call(RunAsBuildOperationBuildActionRunner.java:49)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner$3.call(RunAsBuildOperationBuildActionRunner.java:44)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:315)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:305)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:101)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner.run(RunAsBuildOperationBuildActionRunner.java:44)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter$1.transform(InProcessBuildActionExecuter.java:49)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter$1.transform(InProcessBuildActionExecuter.java:46)
	at org.gradle.composite.internal.DefaultRootBuildState.run(DefaultRootBuildState.java:78)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:46)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:31)
	at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:42)
	at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:28)
	at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:78)
	at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:52)
	at org.gradle.tooling.internal.provider.SubscribableBuildActionExecuter.execute(SubscribableBuildActionExecuter.java:59)
	at org.gradle.tooling.internal.provider.SubscribableBuildActionExecuter.execute(SubscribableBuildActionExecuter.java:36)
	at org.gradle.tooling.internal.provider.SessionScopeBuildActionExecuter.execute(SessionScopeBuildActionExecuter.java:68)
	at org.gradle.tooling.internal.provider.SessionScopeBuildActionExecuter.execute(SessionScopeBuildActionExecuter.java:38)
	at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:37)
	at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:26)
	at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:43)
	at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:29)
	at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:60)
	at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:32)
	at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:55)
	at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:41)
	at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:48)
	at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:32)
	at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:67)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
	at org.gradle.util.Swapper.swap(Swapper.java:38)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:62)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:81)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)
	at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:295)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':app:transformDexWithInstantRunSlicesApkForDebug'.
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:95)
	at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:91)
	at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:57)
	at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:119)
	at org.gradle.api.internal.tasks.execution.ResolvePreviousStateExecuter.execute(ResolvePreviousStateExecuter.java:43)
	at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:93)
	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:45)
	at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:94)
	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:56)
	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:55)
	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:67)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:49)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:315)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:305)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:101)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:49)
	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:355)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129)
	... 6 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /Users/dorbae/Desktop/dev/android/dhtcomonitor/app/build/intermediates/instant_run_split_apk_resources/debug/instantRunSplitApkResourcesDebug/out/slice_5/resources_ap
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593)
	at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
	at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:720)
	at com.android.ide.common.internal.WaitableExecutor.waitForTasksWithQuickFail(WaitableExecutor.java:149)
	at com.android.build.gradle.internal.transforms.InstantRunSliceSplitApkBuilder.transform(InstantRunSliceSplitApkBuilder.java:246)
	at com.android.build.gradle.internal.pipeline.TransformTask$2.call(TransformTask.java:239)
	at com.android.build.gradle.internal.pipeline.TransformTask$2.call(TransformTask.java:235)
	at com.android.builder.profile.ThreadRecorder.record(ThreadRecorder.java:102)
	at com.android.build.gradle.internal.pipeline.TransformTask.transform(TransformTask.java:230)
	at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
	at org.gradle.api.internal.project.taskfactory.IncrementalTaskAction.doExecute(IncrementalTaskAction.java:47)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:41)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.run(ExecuteActionsTaskExecuter.java:284)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:301)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:293)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:273)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:258)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$200(ExecuteActionsTaskExecuter.java:67)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:145)
	at org.gradle.internal.execution.impl.steps.ExecuteStep.execute(ExecuteStep.java:49)
	at org.gradle.internal.execution.impl.steps.CancelExecutionStep.execute(CancelExecutionStep.java:34)
	at org.gradle.internal.execution.impl.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:69)
	at org.gradle.internal.execution.impl.steps.TimeoutStep.execute(TimeoutStep.java:49)
	at org.gradle.internal.execution.impl.steps.CatchExceptionStep.execute(CatchExceptionStep.java:33)
	at org.gradle.internal.execution.impl.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)
	at org.gradle.internal.execution.impl.steps.SnapshotOutputStep.execute(SnapshotOutputStep.java:43)
	at org.gradle.internal.execution.impl.steps.SnapshotOutputStep.execute(SnapshotOutputStep.java:29)
	at org.gradle.internal.execution.impl.steps.CacheStep.executeWithoutCache(CacheStep.java:134)
	at org.gradle.internal.execution.impl.steps.CacheStep.lambda$execute$3(CacheStep.java:83)
	at java.util.Optional.orElseGet(Optional.java:267)
	at org.gradle.internal.execution.impl.steps.CacheStep.execute(CacheStep.java:82)
	at org.gradle.internal.execution.impl.steps.CacheStep.execute(CacheStep.java:36)
	at org.gradle.internal.execution.impl.steps.PrepareCachingStep.execute(PrepareCachingStep.java:33)
	at org.gradle.internal.execution.impl.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:38)
	at org.gradle.internal.execution.impl.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:23)
	at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:96)
	at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:89)
	at java.util.Optional.map(Optional.java:215)
	at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:52)
	at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:36)
	at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:34)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:91)
	... 35 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /Users/dorbae/Desktop/dev/android/dhtcomonitor/app/build/intermediates/instant_run_split_apk_resources/debug/instantRunSplitApkResourcesDebug/out/slice_5/resources_ap
	at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1431)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.io.FileNotFoundException: /Users/dorbae/Desktop/dev/android/dhtcomonitor/app/build/intermediates/instant_run_split_apk_resources/debug/instantRunSplitApkResourcesDebug/out/slice_5/resources_ap
	at com.android.build.gradle.internal.transforms.InstantRunSliceSplitApkBuilder.lambda$null$1(InstantRunSliceSplitApkBuilder.java:233)
	at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
	... 4 more

Solution

Cause

Gradle version was too low

Update Gradle version

Open gradle-wrapper.properties files in project directory

Replace distributionUrl with the URL which contains recent Gradle verion and execute synchronization
- distributionUrl=https://services.gradle.org/distributions/gradle-[RECENT_GRADLE_VERSION]-all.zip
  
  Ref. Gradle Releases

References

[Android] Bar chart is not visible, nevertheless values of it are visible in MPAndroidChart

Fri, 09 Aug 2019 00:00:00 +0000

Goal

Solve the problem which bars are not visible in chart

Problem

Environment
- Development
  - Android Studio 3.2.1
  - Compile/Target SDK : 28
  - Minimum SDJ : 17
- Test
  - Xiaomi MI MAX 2
  - Android 7.1.1
- MPAndroidChart 3.1.0
I drew combined graph and the values of bar chart displayed well.
But, the bars didn’t displayed despite of replacing the width of bar trough BarData.setBarWidth(float) method

Solution

Cause

I was using millisecond as the unit of X axis
The bars were too thin to be not visible. It was becuase the library drew bars with 1 millisecond width
So, you have to replace the unit of values in X asis with minute or bigger than minute

My Code

private static final SimpleDateFormat DISP_SDF_DAY_OF_WEEK = new SimpleDateFormat("E", Locale.getDefault());
private static final SimpleDateFormat INFLUXDB_UTC_SDF = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");

...

XAxis xAxis = this.chart.getXAxis();
xAxis.setValueFormatter(new ValueFormatter() {
    @Override
    public String getFormattedValue(float value) {
        //
        // Problem
        //
        return DISP_SDF_DAY_OF_WEEK.format(Float.valueOf(value).longValue() * 1000L);
    }
});

...

BarDataSet barDataSet;
YAxis barYAxis = this.chart.getAxis(YAxis.AxisDependency.LEFT);
barYAxis.setPosition(YAxis.YAxisLabelPosition.OUTSIDE_CHART);
barYAxis.setDrawGridLines(false);
barYAxis.setTextColor(android.R.color.black);
barYAxis.setSpaceTop(15f);
barYAxis.setAxisMinimum(0f);
List<BarEntry> barEntryList = new ArrayList<>();
BarEntry barEntry;

ArrayList<Object> list;
long time;
Object temp;
float mean;
try {
    for (int ll = 0; ll < values.length; ll++) {
        list = (ArrayList<Object>) values[ll];

        //
        // Problem
        //
        time = INFLUXDB_UTC_SDF.parse((String) list.get(0)).getTime() / 1000L;  // eg. 2019-08-07T00:00:00Z

        temp = list.get(1);
        if (temp instanceof java.lang.Double) {
            mean = ((Double) temp).floatValue();
        } else if (temp instanceof java.lang.Integer) {
            mean = ((Integer) temp).floatValue();
        } else {
            mean = (float) temp;
        }
        
        barEntry = new BarEntry(time, (int) list.get(2));
        barEntryList.add(barEntry);
    }
} catch (java.text.ParseException e) {
    Log.e("dorbae", e.getMessage(), e);
}

barDataSet = new BarDataSet(barEntryList, "Accumulation");
barDataSet.setColor(getResources().getColor(R.color.colorPrimary));
barDataSet.setValueTextColor(getResources().getColor(R.color.colorPrimary));
barDataSet.setAxisDependency(YAxis.AxisDependency.LEFT);
barDataSet.setVisible(true);
barDataSet.setDrawIcons(false);

BarData barData = new BarData(barDataSet);
//
// It didn't work well because of other reason
//
barData.setBarWidth(30f);

Legend l = this.chart.getLegend();
l.setDrawInside(true);
l.setVerticalAlignment(Legend.LegendVerticalAlignment.BOTTOM);
l.setHorizontalAlignment(Legend.LegendHorizontalAlignment.RIGHT);
l.setTextColor(Color.BLACK);

CombinedData combinedData = new CombinedData();
combinedData.setData(barData);
combinedData.setData(lineData);
this.chart.setData(combinedData);
this.chart.getAxisLeft().setEnabled(true);
this.chart.setVisibility(View.VISIBLE);
this.chart.invalidate();
this.chart.notifyDataSetChanged();

...

1. Change the unit of x value as hour

MyCode

...

xAxis.setValueFormatter(new ValueFormatter() {
    @Override
    public String getFormattedValue(float value) {
        //
        // Solved 
        //
        return DISP_SDF_DAY_OF_WEEK.format(Float.valueOf(value).longValue() * 1000L * 60L * 60L);
    }
});

...

try {
    for (int ll = 0; ll < values.length; ll++) {
        list = (ArrayList<Object>) values[ll];

        //
        // Solved (second -> hour)
        //
        time = INFLUXDB_UTC_SDF.parse((String) list.get(0)).getTime() / 1000L / 60L / 60L);  // eg. 2019-08-07T00:00:00Z

        ...
    }
} 

...

References

MPAndroidChart GitHub

[ESP32] How to flash a firmware into ESP32(LOLIN D32 PRO ver.2.0.0) on MacOS?

Mon, 05 Aug 2019 00:00:00 +0000

Goal

Flash firmware into ESP32

Intro

Wemos LOLIN D32 PRO
macOS Mojave 10.14.6

Practice

Setup of Toolchain

ESP-IDF will use Python
ESP-IDF essentially contains API (software libraries and source code) for ESP32 and scripts to operate the Toolchain
If, you need to install pip
```
sudo easy_install pip
```

Install pyserial

pip install --user pyserial

Install Cmake & Ninja

Ninja is a small build system with a focus on speed
Ninja’s low-level approach makes it perfect for embedding into more featureful build systems
Ninja is used to build Google Chrome, parts of Android, LLVM, and can be used in many other projects due to CMake’s Ninja backend.
ccache is optional. However, it is strongly recommended for faster builds

brew install cmake ninja ccache

Get ESP-IDF

To build applications for the ESP32, you need the software libraries provided by Espressif in ESP-IDF repository

Create local repository directory and clone

If you execute below commands, ~/esp/esp-idf directory will be created
You can use any directory, but you will need to adjust paths for the commands respectively
Keep in mind that ESP-IDF does not support spaces in paths.

mkdir ~/esp
cd ~/esp
git clone --recursive https://github.com/espressif/esp-idf.git

Set up the tools

Execute ESP-IDF Tools installer

cd ~/esp/esp-idf
sh ./install.sh

Set up the environment variables

The installed tools are not yet added to the PATH environment variable
To make the tools usable from the command line some environment variables must be set
You can just execute script which set up variables or add that script in ./profile or .bash_profile

cd ~/esp/esp-idf
sh ./export.sh

Start an nexample project

Copy an example project to ~/esp directory

cd ~/esp
cp -r $IDF_PATH/examples/get-started/hello_world .

Connect Your Device

Serial ports have the following patterns in their names

OS	Pattern
Windows	likes __COM1__
Linux	starting with __/dev/tty__
macOS	starting with __/dev/cu.__

ls /dev/cu*

Build the project

idf.py build

Flash onto the device

Flash the binaries that you just built onto your ESP32 board by running
-b option is optional and default value is 460800
I set up baud as 115200

The option flash automatically builds and flashes the project, so running idf.py build is not necessary

idf.py -p [PORT_PATH] {-b [BAUD]} flash

Monitor

Check if “hello_world” is indeed running

If you want to stop monitoring, type Ctrl + ]

idf.py -p [PORT_PATH] monitor

If the messages were not normal

Your board is likely using a 26MHz crystal
Most development board designs use 40MHz, so ESP-IDF uses this frequency as a default value

cd ~/esp/hello_world
idf.py menuconfig

Component config -> ESP32-specific

Main XTAL frequency

Change CONFIG_ESP32_XTAL_FREQ_SEL to 26MHz

References

[OpenCV] What is the Mat?

Thu, 01 Aug 2019 00:00:00 +0000

Goal

What is the mat
How it works
How to convert a matrix to a bytes array
The method for storing image data

Intro

I had a problem to diplay the image which was serialized.
So, I need to understand how OpenCV handles the matrix exactly

Practice

Change of way to handle the image

OpenCV 1.0
- C language base
- You have to manage the momory manually
- It use IplImage for handling the image
  
  IplImage : C structure for storing the image in the memory
OpenCV 2.0
- It introduce a new C++ interface
- Nevertheless, it is fully compatible with C
- No need to fiddle with the memory management
- However, a lot of embedded developments support only C
- As using the Mat, you don’t need to allocated and release the memory manually. This is because the Mat manages the memory automatically

Mat Structure

To solve the performance issue by unnecessary copies of potentially large images
OpenCV uses a reference counting system
It copy only header generally and support deep copy as well

General Copy

Mat A = imread(...);

Mat B = Mat(A);

Deep Copy

Copy not only heaer but also matrix

Mat F = A.clone();

Mat G;
A.copyTo(G);

When a matrix object is free in the memory?

I mentioned, OpenCV uses counting references
So, If the count of pointers which indicate the matrix, it’ll be free automaticlly

Example

Mat a = imread(...);

Mat B = Mat(A);
Mat C = Mat(A);

free(A);

free(B);
free(C);

Storing methods

Gray scale

White/Black
Create many shades of gray

RGB

The most common method
Similar to our eyes

HSV/HLS

Decompose colors into hue/Saturation and value/luminance components
More natural way to decribe colors

YCrCb

Used by JPEG image format

The smallest data type of each component is char = 1 byte = 8 bits

value range of signed char is -127 and 127

value range of unsigned char is 0 ~ 255

Can use other types like float(4 bytes or 32 bits) and double(8 bytes or 64 bits) and they can provide ever finer control

However, the size of image will be increased as using them

Creating Mat

Mat is a image container and also a general matrix class, that is, 2-dimentional marices
Mat(row, column, store_method)
- Store method
  - CV_[The number of bits per item] [Signed or Unsigned] __{[Type Prefix]}__C[The channel number]
  eg. CV_8UC3 : Unsigned 8bits(unsigned char) and 3 channles
  
  eg. CV_32UC1 : Unsigned 32bits(usigned float) and 1 channel

How to store N-dimensions

Convert N-dimensions matrices to 1-D aaray

1-D Matrix to 1-D Array

eg. 3x3
Matrix

Array(Bytes Array)

Contiguous

The way to order the data in memory
There are C-contiguous, Fortran-contiguous and so on

N-d Matrices to 1-D Array

eg. RGB(CV_8UC3/) 3-Dimensions
Matrices

Array(Bytes Array)

Mat manages RGB as BGR

References

[MacOS] How to save as a image file from Keynote

Thu, 01 Aug 2019 00:00:00 +0000

Goal

Save images or figures in a Keynote page as image file by Preview.app

Intro

I drew a structure using figures, text boxes and arrows.
They need to be a image file such as .jpeg, .png

Practice

Copy the figures that you want to save

Select figures -> Mouse right click -> Copy
Select figures -> CMD + C

Open Preview.app and paste image from clipboard

Open Preivew.app

File > New from Clipboard

Save as a image file

CMD + S

References

instructionaltechtalk

[Linux] How to remove specific files and directories

Wed, 31 Jul 2019 00:00:00 +0000

Goal

Remove old video, data, log files in my CCTV server

Intro

My CCTV server directory tree


CCTV_HOME
|
|__ bin
|__ cctv
|   |__ cam01
|   |   |__ data
|   |   |   |__ 201905
|   |   |   |__ 201906
|   |   |   |__ 201907
|   |   |   |   |__ 20190701
|   |   |   |   |__ 20190702
|   |   |   |   |__ ...
|   |   |   |   |__ 20190730
|   |   |   |   |__ 20190731
|   |   |   |   |   |   cctv_20190731001231123.avi
|   |   |   |   |   |   cctv_20190731001431145.avi
|   |   |   |   |   |   ...
|__ log
|   |   cctv_20190415212234567.log
|   |   cctv_20190613112834139.log
|   |   cctv_20190711082932345.log
|   |   ...
|__ tmp
|   |   temp_20190415212234567.dat
|   |   temp_20190613112834139.dat
|   |   temp_20190711082932345.dat
|   |   ...

Practice

script

#!/bin/bash

###########################################################################
#
# Remove old files including .avi, .dat, .ini, .log and temporary files
#
###########################################################################
# 1.0.0  2019-07-31  dorbae    Initialize
###########################################################################

# Home directory
MY_CAMERA_HOME=/home/dorbae/cctvhome
# Define how many days you keep the files
KEEP_DAYS=7

# Remove old temporary files which were modified more than a day ago
find ${MY_CAMERA_HOME}/tmp/ -name 'temp_*' -mtime +1 -exec rm -f {} \;

# Remove old log files which were modified more than n days ago
find ${MY_CAMERA_HOME}/log/ -name '*.log' -mtime +${KEEP_DAYS} -exec rm -f {} \;

# Remove old video and data files which were modified more than n days ago
find ${MY_CAMERA_HOME}/cctv/ -name 'cctv_*' -mtime +${KEEP_DAYS} -exec rm -f {} \;

# Remove old YYYYMMDD(daily) directory
# eg. if today is 2019-07-31, remove ${MY_CAMERA_HOME}/cctv/*/data/201907/20190724
OLD_DAILY_DIRECTORY=`date '+%Y%m%d' -d "${KEEP_DAYS} day ago"`
OLD_MONTH_DIRECTORY=`date '+%Y%m' -d "${KEEP_DAYS} day ago"`
rm -rf ${MY_CAMERA_HOME}/cctv/*/data/${OLD_MONTH_DIRECTORY}/${OLD_DAILY_DIRECTORY}

# Remove old YYYYMM(monthly) directory
# Also, remove directory which were created 2 month ago.
# Because I want to remove unnecessary directories certainly if it was not deleted accidently by any reason.

# - option : remove a padding(0)
# eg. %d -> 07, 08, 11 / %-d -> 7, 8, 11
TODAY=`date '+%-d'`
if [ ${TODAY} -gt ${KEEP_DAYS} ]; then
    MONTH_AGO_1_DIRECTORY=`date '+%Y%m' -d "1 month ago"`
    MONTH_AGO_1_YYYY=`date '+%Y' -d "1 month ago"`
    CURRENT_YYYY=`date '+%Y'`
    # I don't know why it works like that exactly
    # When I execute `date '+%Y%m' -d "1 month ago"` command on 31th July 
    # It returns not 06 but 07
    # I think it maybe depends on Timezone
    # So, check the month again manually
    if [ ${CURRENT_YYYY} -ne ${MONTH_AGO_1_YYYY} ]; then
        # Remove YYYYMMDD directory
        rm -rf ${MY_CAMERA_HOME}/cctv/*/data/${MONTH_AGO_1_DIRECTORY}
    fi
    
    # Remove the directory which were created 2 month ago
    MONTH_AGO_2_DIRECTORY=`date '+%Y%m' -d "2 month ago"`
    rm -rf ${MY_CAMERA_HOME}/cctv/*/data/${MONTH_AGO_2_DIRECTORY}
fi

exit 0

[Linux] How to remove unnecessary backup files periodically

Mon, 29 Jul 2019 00:00:00 +0000

Goal

Remove unnecessary MariaDB backup files which were modified more 6 days ago

Intro

I have created the backup file from MariaDB once a week
How to dump the backup file from MariaDB

Practice

1. Create old files

I have to create files which have a similar name pattern and were modified more 6 days ago
You can use __touch__ command

touch -t [[CC]YY]MMDDhhmm[.ss]

$ touch -t 201907150700 backup_mariadb_agriculture_19.07.15.0700.sql
$ touch -t 201907220700 backup_mariadb_agriculture_19.07.22.0700.sql

2. Remove old files

You can find only the files that you want and execute a specific command to them by __find__ command

$ find [DIRECTORY] -name [FILE_NAME] -mtime +[DAYS] -exec rm -f {} \;

$ find ~/ -name 'backup_mariadb_agriculture_*.sql' -mtime +6 -exec rm -f {} \;

3. Register in Cron

3.1. Edit the cron table

$ crontab -e

3.2. Register the schedule

Cron pattern

[MINUTE] [HOUR] [DAY] [MONTH] [DAY OF WEEK] [COMMAND]

I tried to execute backup script at 7 O’clock every day

0 7 * * * find ~/ -name 'backup_mariadb_agriculture_*.sql' -mtime +6 -exec rm -f {} \;

References

[MariaDB/MySQL] How to backup the data weekly

Mon, 29 Jul 2019 00:00:00 +0000

Goal

Creat MariaDB backup shell script
Register in Cron

Intro

I wanted to dump the data which was in MariaDB for stability

Practice

1. Dump shell script

Script

#####################################################################
# Date          Name            Desc
#####################################################################
# 2019-07-29    SeongbaeChang   Initialize
# 2018-07-17    SeongbaeChang   Initialize
#####################################################################
#!/bin/bash

# Set current datetime
TIMESTAMP=`date +"%y.%m.%d.%H%M"`
BACKUP_FILENAME=backup_mariadb_agriculture_${TIMESTAMP}.sql
ROOT_HOME="/root"
MY_USER_HOME="/home/dorbae"

echo "[`date +"%Y-%m-%d %H:%M:%S"`] Start Backup..."
echo "[`date +"%Y-%m-%d %H:%M:%S"`] Backup File Name = ${BACKUP_FILENAME}"
# Backup MariaDB data as SQL format
mysqldump -uagric -pagriculture agriculture > ${ROOT_HOME}/${BACKUP_FILENAME}

echo "[`date +"%Y-%m-%d %H:%M:%S"`] Moving file ${ROOT_HOME} to ${MY_USER_HOME}"
# Move to another user home directory
mv ${ROOT_HOME}/${BACKUP_FILENAME} ${MY_USER_HOME}/${BACKUP_FILENAME}
# Change the owner managing the backup file
chown dorbae:dorbae ${MY_USER_HOME}/${BACKUP_FILENAME}

echo "[`date +"%Y-%m-%d %H:%M:%S"`] Finished Backup."

2. Register in Cron

2.1. Edit the cron table

# crontab -e

2.2. Register the schedule

Cron pattern

[MINUTE] [HOUR] [DAY] [MONTH] [DAY OF WEEK] [COMMAND]

I tried to execute backup script at 7 O’clock every day

3. Result

References

[Android] Cleartext HTTP traffic to X.X.X not permitted

Thu, 11 Jul 2019 00:00:00 +0000

Goal

Solve ‘Cleartext HTTP traffic to X.X.X not permitted’ error

Problem

I replaced the target SDK 27 with 28(Pie 9.0)
After that, when trying to communicate with server by HTTP protocol, this error occured

Solution

Cause

From API level 28(Pie 9.0) it has been default to prevent unencrypted HTTP protocl

AndroidDeveloper

1. Add a Network Security Configuration file

The Network Security Configuration feature uses an XML file where you specify the settings for your app

1.1 Add xml file in res/xml

Create network_security_config.xml file in src/res/xml directory

1.2. Add options

Recommend to allow only specific domains

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <!-- Allow specific domains -->
    <domain-config cleartextTrafficPermitted="true">
        <domain includeSubdomains="true">iot.qubics.kr</domain>
    </domain-config>

    <!-- Allow all domains -->
    <!--
    <base-config cleartextTrafficPermitted="true" />
    -->

</network-security-config>

1.3. Edit AndroidManifest.xml

Add android:networkSecurityConfig=”@xml/network_security_config” option

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="io.dorbae.android">
    <application
       android:networkSecurityConfig="@xml/network_security_config"
...>
        ....
    </application>
</manifest>

2. Edit only AndroidManiofest.xml

Add android:useCleartextTraffic=”true” option

div style=”background: #eeeedd; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;”><table><tr><td><pre style="margin: 0; line-height: 125%">1 2 3 4 5 6 7 8 9</pre></td><td><pre style="margin: 0; line-height: 125%"><?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="io.dorbae.android"> <application android:usesCleartextTraffic="true" …> …. </application> </manifest> </pre></td></tr></table></div>

References

[Django] Download files

Wed, 26 Jun 2019 00:00:00 +0000

Goal

How to impletement the function which support file download in Django

Intro

I wanted to make people download apk file in my REST server

Practice

1. Create App and setup files

1.1. Start app

(venv) $ python manage.py startapp [APP_NAME]

1.2 Setup files to be downloaded

You can put files in anywhere, but I made ‘files’ directory in created app directory. And I put my files in there.
In my case, I created new sub directory name ‘apk’ and put my apk files in there.

2. Create function for download

Code

############################################################
#
# Request for downloading
#
############################################################
# Version   Date        Author          Description
############################################################
# 1.0.0     2019-06-26  dorbae          Initialize
############################################################

from django.http import HttpResponse, Http404
import os

# Django project base directory
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


#
# Download APK file
#
def download_apk(request, file_base_name):
    print('Download ' + file_base_name + '.apk...')
    # Full path of file
    file_path = BASE_DIR + '/download/files/apk/' + file_base_name + '.apk'
    if os.path.exists(file_path):
        with open(file_path, 'rb') as fh:
            response = HttpResponse(fh.read(), content_type="application/force_download")
            response['Content-Disposition'] = 'inline; filename=' + os.path.basename(file_path)
            return response
    # If file is not exists
    raise Http404

3. URL Mapping

3.1. Edit [APP]/urls.py

Code

from django.urls import path

from . import views

urlpatterns = [
    path('apks/<str:file_base_name>/', views.download_apk, name='download_apk'),
]

3.2. Edit [PROJECT]/urls.py

Add URL Pattern

4. Test

References

Stackoverflow

[Django] Setup virtual enviroment

Fri, 21 Jun 2019 00:00:00 +0000

Goal

How to setup Python virtual environment

Intro

I had a plan to deploy my Django project on server
However, there were already a lot of Python programs
So, I wanted to make a special python environment containing packages which will be used in only my project

Practice

1. Install venv

venv : the module used to create and manage virtual environments

# apt-get install python3-venv

2. Setup virtual environment

2.1. Create virtual

$ python3 -m venv [VENV_DIRECTORY]

2.2. Activate virtual environment

$ source [VENV_DIRECTORY/bin/activate

References

https://docs.python.org/ko/3/tutorial/venv.html

[Django] Invalid HTTP_HOST header: '[HOST]'. You may need add '[HOST]' to ALLOWED_HOSTS

Wed, 19 Jun 2019 00:00:00 +0000

Goal

To solve “Invalid HTTP_HOST header: ‘[HOST]’. You may need add ‘[HOST]’ to ALLOWED_HOSTS”’ problem.

Intro

When I connected my Django REST server by domain likes this ‘http://iot.dorbae.kr:8000/~’, this error occured and I wasn’t able to connect the server.

Practice

1. Add allowed host

1.1. Open setting.py file in your djang project

1.2. Add allowed hosts

[‘*’] : Allow all
[‘localhost’] or [‘localhost’, ‘iot.dorbae.kr’, ‘10.12.33.55’] : Allow specific hosts

References

https://programmers.co.kr/learn/questions/1025

[Telephone English] Learning English by phone #032

Tue, 18 Jun 2019 00:00:00 +0000

전화영어 - Day 032 (20190618)

Part 1

Question
- Do you like to smile?
Answer (In my case)
- Yes, I love to smiling. Because it makes people happy or makes more funny. And it is also addiction. So some people smile, Many people smile, other people also smile.
Answer (Correction)
- Yes, I love to smiling. Because it makes people happy. Smiling is contagious which means when you smile, others smile as well.
funny : Gag, Humor에 즐거워하는 것 addiction -> contagious : 전염성의

Question
- When people smile at each other?
Answer (In my case)
- Many people smile when they see small babies and behavior of themWhen they. In my case, I smile when I see my puppies.
smile == laugh

Question
- Do you smile, when others take pictures of you?
Answer (In my case)
- No, I don’t. Because I’m not familliar with the picture. So, other people can smile when taking picture. On the other hand, I don’t smile when other take a picture of me.

Part 2

Question
- Describe an impressive work of art such as painting or sculpture you saw
- You should say:
  - What it is
  - When and where you first saw this work or art
  - What it looks like
  - And explain why you like it
Answer (In my case)
- I’d like to talk about some impressive work of art. When I was young, I went to Paris over 1 month. Because my father was studying in Paris over 1 year. So, I went to there with my family. And I was able to have change to go to museum of Louvre. And then I saw 4 paintings which had the face of men which represented the weather. One of them represented Spring and it was composed of fruit and vegetable which grew up in Spring. It was very fresh painting and I deeply moved by the painter’s idea. I saw many works of art in the museum.
Answer (Correction)
- I’d like to talk about some impressivev work of art. When I was young, I went to Paris and stayed there for over a month because my father was studying there. Because of that, I was able to have a chance to go to the Louvre Museum where I saw a set of paintings which had faces of men representing the four seasons, thus it is called The Four Seasons. Since it depicted the four seasons, each painting had the elements of each season. For instance, the painting about spring had some vegetables and fruits that made up the man’s face while the winner shows the face of a man with a tree consisting of few leaves. It was something new to me so I was really impressed. I think the idea was unique but excellent so it kind of moved me compared to the paintings in that museum.
grew up (X) -> grew(O)

Correction Others

furnitures(X) : 셀 수 없는 명사

[PyCharm] How to disable editor vim mode

Fri, 31 May 2019 00:00:00 +0000

Goal

To set Vim mode in editor disabled.

Intro

I felt very uncomportable when using vim mode in editor. So, I wanted to disable this function.

Practice

1. Prefences > plugins > installed

2. Disable IdeaVim

References

http://egloos.zum.com/geneus/v/3536401

[Python] SELECT data from InfluxDB

Fri, 31 May 2019 00:00:00 +0000

Goal

To select data from InfluxDB

Practice

1. Install InfluxDB module

$ pip install influxdb

2. Select data

Code

############################################################
#
# InfluxDB SELECT sample
#
############################################################
# Version   Date        Author          Description
############################################################
# 1.0.0     2019-05-31  dorbae          Initialize
############################################################

from influxdb import InfluxDBClient

def main():
    """Instantiate a connection to the InfluxDB."""
    host = 'localhost'
    port = 8086
    user = 'root'
    password = 'root'
    dbname = 'iotdb'
    query = 'SELECT mean(*) FROM "iotseries" WHERE time > now() - 1d AND mGroupID = \'sample\'  and mDeviceID = \'2435839\' GROUP BY time(1h);'

    client = InfluxDBClient(host, port, user, password, dbname)

    print("Querying data: " + query)
    result = client.query(query)

    for point in result.get_points():
        print(point)

if __name__ == '__main__':
    main()

Result

References

https://influxdb-python.readthedocs.io/en/latest/examples.html

[Telephone English] Learning English by phone #030

Thu, 30 May 2019 00:00:00 +0000

전화영어 - Day 030 (20190530)

Part 1

Question
- Where do people usually go during festival?
Answer (In my case)
- Many people usally go countryside during festival. Because there are many festivals in countryside such as flower festival, music festival.
Answer (Correction)
- I think most people go to the countryside since most festivals are held there.

Part 2

Question
- Describe a newly built public facility (such as parks, cinemas, etc.) that influences your city
- You should say:
  - Where this facility is
  - When it was built
  - What people can do there
  - What influence this facility has
Answer (In my case)
- I’d like to talk about the park in my city. Many people called it as Yeonam-Park. Because it is in Yeonam-dong. In the past, there was train track which was not used for a long time. And three years ago, the goverment rebuilt the park in there. Nowadays, many people take a rest in there drinking beer, playing music, just talking with their friends. After rebuiling the park, many people can relieve their stress taking a break. And also there are many stores such as food restaraunts, accessory shops. Because due to new park, many people go there.
Answer (Correction)
- I’d like to talk about the park in my city. Many people call it as Yonam Park because it is in Yonamdong and in the past, there was train track in there which was not used for a long time until 3 years ago when the government rebuilt the park in there, nowadays many people take a rest in there, drinking beer or playing music or just talking with their friends. So this park provided people with a place where they can relieve their stress, bond with their friends and spend quality time with their family. Around the park, there are some some stores such as restaurants, convenience stores, boutiques and more which means this park also allowed people to start their own business and help those who are unemployed. So I think this park has a lot of influences to the people and our society.
boutique : 옷, 악세서리 파는 가게

Part 3

Question
- What measures should be taken to deal with damage on public facilities?
Answer (In my case)
- I think if someone damages public facilities, administrator of them or the goverment should take fine about that to him or her. And, the goverment should think about preventing the damages. For example, they should warn to people by setting up warning sign in there.
Answer (Correction)
- I think the administrator of those public facilities should impose the rules and regulations strictly. For example, those offenders should pay some fine or even go to jail. It would also be helpful to set up some warning signs and CCTV cameras to prevent people from damaging public facilities.
impose : 재정하다, 부과하다

Question
- What are the differences between old facilities and new facilities?
Answer (In my case)
- I think there are many differences between old ones and new ones. At first, the old facilities have long history of them. Many feel comportable in there, becuase they are very familiar with many people. On the other hand, the new facilities provide more thinks for people. And there are more facilities in there such as toilets and the places for baby. For example, the new shopping complex usually has the place where babies can play and from it many parents can take a shopping freely. Also there are leisure facilities in there such as cinema, boutique, electronic shop, so on.
Answer (Correction)
- First, old facilities usually have long history so people feel more comfortable when visiting them since they have become very familiar to them. On the other, new facilities offer much more convenience to people because they have a lot of new places, activities and more to offer. For example, they have feeding stations for moms and their babies however, old facilities do not. In addition, new shopping malls have more shops to choose from including international ones, but old shopping malls usually have local stores.
convenience : 컨베니언스(x) 컨비니언스(o)

Question
- Is it popular to give hand-made gift in your country?
Answer (In my case)
- Yes, I think it is very popular in my country. So, some people make a cake or neck warmer for birthday of their friends or family. In my case, I also made chocolate by hand-made for my wife. And chocolate stick is very common and popular hand-made gift in my country.

Question
- Do you think expensive gifts are always more valuable than cheap ones?
Answer (In my case)
- When I was young, I thought the price of present was not important and effort was more precious. But, nowadays, pricey present is better than others such as hand-made present. Above all, I think the best present is which receiver wants to buy.
I think the value of a gift doesn’t depent on its price, instead it depends on the the giver’s sincerity.

when kids make a very simple gift for their parents, parents are usually very touched and thrilled

Correction Others

cherry blossom festival : 벚꽃 축제
I want to go to that festival someday -> I want to join in that festival someday

[Docker] Run MariaDB container and edit configuration

Thu, 30 May 2019 00:00:00 +0000

Goal

To run MariaDB container
To edit DB configuration including memory parameter and connector

Intro

Install MariaDB in IBM Linux using docker
I already received docker image in advance.

Practice

1. Run container

1.1 Check Docker images

# docker images

1.2. Create container from image

# docker run --name [CONTAINER_NAME] -e MYSQL_ROOT_PASSWERD=[PASSWORD] -d [IMAGE_NAME{:TAG}]

# docker ps

–name : Container’s name

-e, –env : Set environment variables

-d, –detach : Run container in background and print container ID

1.3. Interface TTY with container

# docker exec -it [CONTAINER_NAME] /bin/bash

-i, –interactive : Keep STDIN open even if not attached

-t, –tty : Allocate a pseudo-TTY

1.4. Install vim

Update apt-get

(container)# apt-get update

Install vim

(container)# apt-get install vim

1.5 Edit MariaDB configuration

1.6. Restart container

# docker restart [CONTAINER_NAME]

References

pyrasis.com