Issue exporting a project containing a diff > 64 KiB

Hi there,

my situation is as follows. I am running an old from source installation (using mysql) which i have upgraded up to 8.16.9 but can’t upgrade any further because of the underlying linux. So i set up a new gitlab instance from scratch and moved over all projects by exporting and importing them one by one.
For 2 out of ~40 projects i am facing an issue during export which i can’t find any information for:

2017-09-03T10:41:23.217Z 18181 TID-oxivf9ah8 ActionMailer::DeliveryJob JID-7b27961c6a625b8ca68dfcf9 INFO: start
2017-09-03T10:41:23.220Z 18181 TID-oxivf9coo ProjectExportWorker JID-418a4a373723cf23cc7bef97 INFO: fail: 25.509 sec
2017-09-03T10:41:23.229Z 18181 TID-oxivf9coo WARN: {“context”:“Job raised exception”,“job”:{“class”:“ProjectExportWorker”,“args”:[3,1],“retry”:3,“queue”:“project_export”,“jid”:“418a4a373723cf23cc7bef97”,“created_at”:1504435257.7106953,“enqueued_at”:1504435257.7111683,“error_message”:"(): found unexpected end of stream while scanning a quoted scalar at line 2 column 10",“error_class”:“Gitlab::ImportExport::Error”,“failed_at”:1504435283.2192109,“retry_count”:0},“jobstr”:"{“class”:“ProjectExportWorker”,“args”:[3,1],“retry”:3,“queue”:“project_export”,“jid”:“418a4a373723cf23cc7bef97”,“created_at”:1504435257.7106953,“enqueued_at”:1504435257.7111683}"}
2017-09-03T10:41:23.229Z 18181 TID-oxivf9coo WARN: Gitlab::ImportExport::Error: (): found unexpected end of stream while scanning a quoted scalar at line 2 column 10
2017-09-03T10:41:23.229Z 18181 TID-oxivf9coo WARN: /home/git/gitlab/app/services/projects/import_export/export_service.rb:50:in cleanup_and_notify' /home/git/gitlab/app/services/projects/import_export/export_service.rb:16:in save_all’
/home/git/gitlab/app/services/projects/import_export/export_service.rb:6:in execute' /home/git/gitlab/app/workers/project_export_worker.rb:11:in perform’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:158:in execute_job' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:138:in block (4 levels) in process’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq.rb:36:in block in <module:Sidekiq>' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:133:in block (3 levels) in process’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke' /home/git/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:5:in call’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke' /home/git/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in call’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/server/active_record.rb:6:in call’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/server/retry_jobs.rb:74:in call’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/server/logging.rb:11:in block in call’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/logging.rb:32:in with_context' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/server/logging.rb:7:in call’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:132:in block (2 levels) in process' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:174:in stats’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:131:in block in process' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq.rb:35:in block in module:Sidekiq’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:126:in process' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:82:in process_one’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/processor.rb:70:in run' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/util.rb:17:in watchdog’
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread’
2017-09-03T10:41:23.348Z 18181 TID-oxivf9ah8 ActionMailer::DeliveryJob JID-7b27961c6a625b8ca68dfcf9 INFO: done: 0.131 sec

I understand that the error might be due to the parsing of incomplete yaml but my ruby knowledge is near zero and i really need some hints on how to further track down the root cause.

Thanks a lot in advance and best regards

Vinz

How are you trying to do the upgrade?
Was in the same situation - old source installation with MySQL as a database.
The way I did was to migrate all the data from MySQL to Omnibus installation Postgresql of the same version as my old source code installation and then just do the regular upgrade. Gitlab 9.x doesn’t full support MySQL so it’s better to switch over to Postgresql.
Furthermore, Gitlab version up process involves modifying some rows in the database so just copying rows would not suffice.
I’ve been able to upgrade to the latest 9.5.2 this way.

In the meantime i have also been able to do the upgrade (manual installation from source) to stable 9.5. The issue with exporting the 2 projects still persists. Something must be broken and i would really like to track that down before it bites in the future.

Cheers

Vinz

Here is a better stack backtrace:

[
“/usr/local/lib/ruby/2.3.0/psych.rb:379:in parse'", "/usr/local/lib/ruby/2.3.0/psych.rb:379:in parse_stream’”,
“/usr/local/lib/ruby/2.3.0/psych.rb:327:in parse'", "/usr/local/lib/ruby/2.3.0/psych.rb:254:in load’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/coders/yaml_column.rb:26:in load'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/type/serialized.rb:19:in type_cast_from_database’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:104:in type_cast'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:42:in original_value’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:37:in value'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/transform_values.rb:11:in block in transform_values’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/transform_values.rb:10:in each'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/transform_values.rb:10:in transform_values’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_set.rb:18:in to_hash'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_methods.rb:284:in attributes’”,
“/home/git/gitlab/app/models/concerns/cache_markdown_field.rb:115:in attributes'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:100:in serializable_hash’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/serialization.rb:17:in serializable_hash'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in block (2 levels) in serializable_hash’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in map'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in block in serializable_hash’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:158:in block in serializable_add_includes'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:156:in each’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:156:in serializable_add_includes'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:112:in serializable_hash’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/serialization.rb:17:in serializable_hash'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in block (2 levels) in serializable_hash’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in map'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in block in serializable_hash’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:158:in block in serializable_add_includes'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:156:in each’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:156:in serializable_add_includes'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:112:in serializable_hash’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/serialization.rb:17:in serializable_hash'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serializers/json.rb:99:in as_json’”,
“/home/git/gitlab/lib/gitlab/import_export/project_tree_saver.rb:34:in project_json'", "/home/git/gitlab/lib/gitlab/import_export/project_tree_saver.rb:28:in project_json_tree’”,
“/home/git/gitlab/lib/gitlab/import_export/project_tree_saver.rb:18:in save'", "/home/git/gitlab/app/services/projects/import_export/export_service.rb:12:in each’”,
“/home/git/gitlab/app/services/projects/import_export/export_service.rb:12:in all?'", "/home/git/gitlab/app/services/projects/import_export/export_service.rb:12:in save_all’”,
“/home/git/gitlab/app/services/projects/import_export/export_service.rb:6:in execute'", "/home/git/gitlab/app/workers/project_export_worker.rb:11:in perform’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:199:in execute_job'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:170:in block (2 levels) in process’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:128:in block in invoke'", "/home/git/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:5:in call’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:130:in block in invoke'", "/home/git/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in call’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:130:in block in invoke'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/server/active_record.rb:15:in call’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:130:in block in invoke'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.5.3/lib/raven/integrations/sidekiq.rb:7:in call’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:130:in block in invoke'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:133:in invoke’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:169:in block in process'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:141:in block (6 levels) in dispatch’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/job_retry.rb:97:in local'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:140:in block (5 levels) in dispatch’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:36:in block in <module:Sidekiq>'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:136:in block (4 levels) in dispatch’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:215:in stats'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:131:in block (3 levels) in dispatch’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/job_logger.rb:7:in call'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:130:in block (2 levels) in dispatch’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/job_retry.rb:72:in global'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:129:in block in dispatch’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/logging.rb:44:in with_context'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/logging.rb:38:in with_job_hash_context’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:128:in dispatch'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:168:in process’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:85:in process_one'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in run’”,
“/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in watchdog'", "/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in block in safe_thread’”
]

Ok, i was able to track it down further:

In the table ‘merge_request_diff_files’ is a record holding a string in the diff field which is 68 KiB.
That one doesn’t get formatted to yaml properly. The yaml is being truncated in the middle which leads to the exception being thrown when parsing that yaml later on.
I still have no clue how to fix that and where exactly the limit is being hit.
I consider this to be a major issue as diffs > 64 KiB can easily happen e.g. when code is reformatted.