Skip to content

Restoring to an un-configured appliance fails due to a missing license file - #449

Merged
taz merged 3 commits into
masterfrom
taz/fix-settings-restore-and-attributes
Nov 8, 2018
Merged

Restoring to an un-configured appliance fails due to a missing license file#449
taz merged 3 commits into
masterfrom
taz/fix-settings-restore-and-attributes

Conversation

@taz

@taz taz commented Nov 8, 2018

Copy link
Copy Markdown
Member

When restoring to an un-configured appliance, ghe-restore is unable to progress past the ghe-restore-storage step. The restore of storage data fails because the storage-cluster-restore-routes script on the appliance is unable to connect to the MySQL database.

"keiko_auth_recv: 1045 Access denied for user 'github'@'localhost' (using password: YES)"
ghe-restore verbose output
$ bin/ghe-restore -c -v -f -s /opt/backup-utils/data/20181108T152539 x.x.x.x
Checking for leaked keys in the backup snapshot that is being restored ...
* No leaked keys found
Connect x.x.x.x:122 OK (v2.15.1)
Starting restore of x.x.x.x:122 with backup-utils v2.15.0 from snapshot 20181108T152539
Stopping cron and github-timerd ...
Restoring settings ...
 --> Importing settings data /data/user/common/github.conf...
Error: no license file found (/data/user/common/enterprise.ghl)
Restoring license ...
Restoring management console password ...
Restoring CA certificates ...
 --> Importing custom CA certificates...
 --> Installing CA certificate to /usr/local/share/ca-certificates/Let's_Encrypt_Authority_X3_0A0141420000015385736A0B85ECA708.crt...
 --> Updating CA certificates...
 --> Done.
Restoring UUID ...
Restoring MySQL database ...
 --> Importing MySQL data...
Restoring Redis database ...
 --> Importing redis data...
 --> Starting redis...
Restoring Git repositories and Gists ...
building file list ... done
./
c/
c/nw/
c/nw/c4/
c/nw/c4/ca/
c/nw/c4/ca/42/
c/nw/c4/ca/42/1/
c/nw/c4/ca/42/1/1.git/
c/nw/c4/ca/42/1/1.git/HEAD
c/nw/c4/ca/42/1/1.git/audit_log
c/nw/c4/ca/42/1/1.git/config
c/nw/c4/ca/42/1/1.git/description
c/nw/c4/ca/42/1/1.git/dgit-state
c/nw/c4/ca/42/1/1.git/dgit-state.flock
c/nw/c4/ca/42/1/1.git/hooks -> /data/github/current/lib/git-core/fi-template/hooks
c/nw/c4/ca/42/1/1.git/language-stats.cache
c/nw/c4/ca/42/1/1.git/info/
c/nw/c4/ca/42/1/1.git/info/nwo
c/nw/c4/ca/42/1/1.git/logs/
c/nw/c4/ca/42/1/1.git/logs/HEAD
c/nw/c4/ca/42/1/1.git/logs/refs/
c/nw/c4/ca/42/1/1.git/logs/refs/heads/
c/nw/c4/ca/42/1/1.git/logs/refs/heads/master
c/nw/c4/ca/42/1/1.git/objects/
c/nw/c4/ca/42/1/1.git/objects/c9/
c/nw/c4/ca/42/1/1.git/objects/c9/0692700d1ac0fb4580b3fa1d6e9b6aaccc09a8
c/nw/c4/ca/42/1/1.git/objects/ee/
c/nw/c4/ca/42/1/1.git/objects/ee/0656d545fa9be234f1d27e69ce820fc6fcf359
c/nw/c4/ca/42/1/1.git/objects/f9/
c/nw/c4/ca/42/1/1.git/objects/f9/e261d5a038403639c4bb06387479efa955166f
c/nw/c4/ca/42/1/1.git/objects/info/
c/nw/c4/ca/42/1/1.git/objects/pack/
c/nw/c4/ca/42/1/1.git/refs/
c/nw/c4/ca/42/1/1.git/refs/heads/
c/nw/c4/ca/42/1/1.git/refs/heads/master
c/nw/c4/ca/42/1/1.git/refs/tags/
info/
info/nw-layout
info/svn-v4-upgraded

sent 4192 bytes  received 516 bytes  3138.67 bytes/sec
total size is 2467  speedup is 0.52
Restoring GitHub Pages ...
building file list ... done
./
.dpages-layout
0/
1/
2/
3/
4/
5/
6/
7/

sent 251 bytes  received 96 bytes  694.00 bytes/sec
total size is 0  speedup is 0.00
Restoring SSH authorized keys ...
 --> Importing SSH authorized keys...
Restoring storage data ...
/data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:598:in `initialize': keiko_auth_recv: 1045 Access denied for user 'github'@'localhost' (using password: YES) (Keiko::MysqlError)
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:598:in `new'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:598:in `block in build_keiko'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/lib/github/config/stats.rb:163:in `time'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:582:in `build_keiko'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:526:in `connect'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:711:in `execute_raw'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:669:in `block in execute_with_retry'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:764:in `query_with_retry'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:668:in `execute_with_retry'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:254:in `block in execute'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:757:in `execute_with_scanner'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:253:in `execute'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:651:in `block (2 levels) in select'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:764:in `query_with_retry'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:650:in `block in select'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:757:in `execute_with_scanner'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:649:in `select'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/activerecord-5.2.1.acbe726/lib/active_record/connection_adapters/abstract/database_statements.rb:57:in `select_all'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/activerecord-5.2.1.acbe726/lib/active_record/connection_adapters/abstract/query_cache.rb:101:in `select_all'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/github-ds-0.2.9/lib/github/sql.rb:242:in `block in results'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/github-ds-0.2.9/lib/github/sql.rb:411:in `enforce_timezone'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/github-ds-0.2.9/lib/github/sql.rb:229:in `results'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/lib/github/storage/allocator.rb:81:in `get_hosts'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/script/storage-cluster-restore-routes:25:in `main'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/script/storage-cluster-restore-routes:69:in `<main>'

We first see something going wrong when the message Error: no license file found (/data/user/common/enterprise.ghl) is shown. This corresponds with an early ghe-single-config-apply termination recorded in the ghe-config.log on the appliance:

$ cat /data/user/common/ghe-config.log
Current version information:
RELEASE_VERSION="2.15.1"
RELEASE_PLATFORM="ami"
RELEASE_BUILD_ID="dc163e46e7"
RELEASE_BUILD_DATE="2018-10-26T10:53:58Z"
Error: no license file found (/data/user/common/enterprise.ghl)

 

It appears the error is a result of the license file being restored after ghe-restore-settings imports the new config and executes ghe-single-config-apply --phase-1. As the ghe-single-config-apply run is incomplete, the GitHub environment is not properly populated with SQL credentials.

This PR alters the order of operations to introduce the license file to the appliance before attempting to import and apply the configuration.

I also noted while reproducing this issue that if the hostname is specified as the first argument to ghe-restore, that all subsequent flags are ignored. I've included a fix to work around that use case as well.

/cc @github/backup-utils

@taz taz self-assigned this Nov 8, 2018
@taz
taz requested a review from lildude November 8, 2018 13:24

@lildude lildude left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. LGTM.

@taz
taz merged commit 22349b4 into master Nov 8, 2018
@taz
taz deleted the taz/fix-settings-restore-and-attributes branch November 9, 2018 01:11
@lildude lildude mentioned this pull request Nov 13, 2018
dooleydevin added a commit that referenced this pull request Jul 25, 2023
…t-create-current-encryption-key-backup-3.7.0+

Backport 443 for 3.8: Kyfast create current encryption key backup 3.7.0+
@dooleydevin dooleydevin mentioned this pull request Jul 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants