Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 28 out of 60 is failed (threshold=3) [...............................X.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000665288s FAIL (2025-08-07 21:35:18 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000687583s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 29 out of 60 is failed (threshold=3) [.............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001107369s FAIL (2025-08-07 21:35:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001131542s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [................................X..XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000973569s FAIL (2025-08-07 21:35:12 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ext segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Aug 7 21:34:42.716392204 (11h55m17.23149477s) [upload=12355]: Starting upload Aug 7 21:34:42.716466463 (11h55m17.231569048s) [upload=12355]: Done starting upload Aug 7 21:34:42.716631833 (11h55m17.231734402s) [upload=12355]: Uploading segments... Aug 7 21:34:42.716905636 (11h55m17.232008212s) [upload=12355]: Got next segment. Inline: false Aug 7 21:34:42.716950835 (11h55m17.232053401s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:42.716955102 (11h55m17.232057682s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:12.678206111 (11h55m47.193308708s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:35:12.678280249 (11h55m47.193382812s) [upload=12355]: Done uploading segments... Aug 7 21:35:12.678299114 (11h55m47.193401676s) [upload=12355]: Upload finished. err: context deadline exceeded Duration: 30.000998086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 200.831804ms FAIL (2025-08-07 21:34:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 200.853374ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.001396425s FAIL (2025-08-07 21:34:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001428348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001732799s FAIL (2025-08-07 21:34:43 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001748725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [..............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000775408s FAIL (2025-08-07 21:34:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000794961s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.........XX..] duration: 40.089862ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:34:42 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 40.110476ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 40.435002ms FAIL (2025-08-07 21:34:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 40.46431ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001303693s FAIL (2025-08-07 21:34:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001324845s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.................................X..XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000847877s FAIL (2025-08-07 21:34:33 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 2219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Duration: 30.000868786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 27 out of 60 is failed (threshold=3) [................................X.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000971459s FAIL (2025-08-07 21:34:27 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000989347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.00157681s FAIL (2025-08-07 21:34:19 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== =ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.739954757 (11h39m30.255057333s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.742133144 (11h39m30.257235706s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.745676444 (11h39m30.260779013s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.747051082 (11h39m30.262153652s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.754982553 (11h39m30.270085122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.763242025 (11h39m30.278344591s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.767097571 (11h39m30.282200149s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.795229256 (11h39m30.310331824s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces. Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments... Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments... Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments... Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments... Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments... Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments... Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments... Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded Aug 7 21:31:18.972600679 (11h51m53.487703265s) [upload=12304]: Starting upload Aug 7 21:31:18.972700584 (11h51m53.487803182s) [upload=12304]: Done starting upload Aug 7 21:31:18.972946355 (11h51m53.488048923s) [upload=12304]: Uploading segments... Aug 7 21:31:18.973271681 (11h51m53.488374263s) [upload=12304]: Got next segment. Inline: false Aug 7 21:31:18.973342730 (11h51m53.488445311s) [upload=12304]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:18.973348044 (11h51m53.488450626s) [upload=12304]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:18.925211587 (11h52m53.440314165s) [upload=12304]: Next segment err: context deadline exceeded Aug 7 21:32:18.925381198 (11h52m53.440483776s) [upload=12304]: Done uploading segments... Aug 7 21:32:18.925404293 (11h52m53.440506857s) [upload=12304]: Upload finished. err: context deadline exceeded Aug 7 21:32:19.033554133 (11h52m53.5486567s) [upload=12319]: Starting upload Aug 7 21:32:19.033620836 (11h52m53.548723412s) [upload=12319]: Done starting upload Aug 7 21:32:19.033708593 (11h52m53.548811183s) [upload=12319]: Uploading segments... Aug 7 21:32:19.033902998 (11h52m53.549005564s) [upload=12319]: Got next segment. Inline: false Aug 7 21:32:19.033935489 (11h52m53.54903806s) [upload=12319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:19.033947286 (11h52m53.549049873s) [upload=12319]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:18.989442137 (11h53m53.504544716s) [upload=12319]: Next segment err: context deadline exceeded Aug 7 21:33:18.989483135 (11h53m53.504585705s) [upload=12319]: Done uploading segments... Aug 7 21:33:18.989519237 (11h53m53.504621808s) [upload=12319]: Upload finished. err: context deadline exceeded Aug 7 21:33:19.125188732 (11h53m53.640291297s) [upload=12333]: Starting upload Aug 7 21:33:19.125248506 (11h53m53.640351083s) [upload=12333]: Done starting upload Aug 7 21:33:19.125315309 (11h53m53.640417872s) [upload=12333]: Uploading segments... Aug 7 21:33:19.125516847 (11h53m53.640619416s) [upload=12333]: Got next segment. Inline: false Aug 7 21:33:19.125552445 (11h53m53.640655011s) [upload=12333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:19.125557201 (11h53m53.640659768s) [upload=12333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:19.079113014 (11h54m53.594215592s) [upload=12333]: Next segment err: context deadline exceeded Aug 7 21:34:19.079283700 (11h54m53.594386266s) [upload=12333]: Done uploading segments... Aug 7 21:34:19.079328529 (11h54m53.594431106s) [upload=12333]: Upload finished. err: context deadline exceeded Duration: 1m0.001605081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001359924s FAIL (2025-08-07 21:34:11 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001384689s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 27 out of 60 is failed (threshold=3) [...............................X..XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00123829s FAIL (2025-08-07 21:34:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001260339s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 26 out of 60 is failed (threshold=3) [.................................X.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000654127s FAIL (2025-08-07 21:33:57 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 47494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000679899s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.........XX.] duration: 11.422036273s remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:33:53 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 11.42205566s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m0.000702923s FAIL (2025-08-07 21:33:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000721109s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 6 out of 60 is failed (threshold=3) [............................................XXX........XX..X] duration: 2m0.000949345s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:33:50 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.000974297s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..................................X..XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001531284s FAIL (2025-08-07 21:33:46 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Duration: 30.0015576s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001190891s FAIL (2025-08-07 21:33:43 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00123125s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.........XX] duration: 2m0.000819542s remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:33:42 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 2m0.000835631s edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000449804s FAIL (2025-08-07 21:33:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000468905s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 26 out of 60 is failed (threshold=3) [................................X..XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000947201s FAIL (2025-08-07 21:33:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000965847s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.000711953s FAIL (2025-08-07 21:33:18 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== , piece_id=RLJSC76ICPPSAK4ZFZPRD2GQP54VESM4RSXMX4EOZTU5LNYDWTZA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.724853022 (11h39m30.239955588s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.725037274 (11h39m30.240139844s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730399702 (11h39m30.245502269s) [noise=true , address=104.36.82.2:4747 , piece_id=ZRKZEN5KXEHMD3OBPISFWZMLA7H2YNKUB5J57IVQ5AVHA5IJGXGQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730432338 (11h39m30.2455349s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.739954757 (11h39m30.255057333s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.742133144 (11h39m30.257235706s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.745676444 (11h39m30.260779013s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.747051082 (11h39m30.262153652s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.754982553 (11h39m30.270085122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.763242025 (11h39m30.278344591s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.767097571 (11h39m30.282200149s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.795229256 (11h39m30.310331824s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces. Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments... Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments... Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments... Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments... Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments... Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments... Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments... Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded Aug 7 21:31:18.972600679 (11h51m53.487703265s) [upload=12304]: Starting upload Aug 7 21:31:18.972700584 (11h51m53.487803182s) [upload=12304]: Done starting upload Aug 7 21:31:18.972946355 (11h51m53.488048923s) [upload=12304]: Uploading segments... Aug 7 21:31:18.973271681 (11h51m53.488374263s) [upload=12304]: Got next segment. Inline: false Aug 7 21:31:18.973342730 (11h51m53.488445311s) [upload=12304]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:18.973348044 (11h51m53.488450626s) [upload=12304]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:18.925211587 (11h52m53.440314165s) [upload=12304]: Next segment err: context deadline exceeded Aug 7 21:32:18.925381198 (11h52m53.440483776s) [upload=12304]: Done uploading segments... Aug 7 21:32:18.925404293 (11h52m53.440506857s) [upload=12304]: Upload finished. err: context deadline exceeded Aug 7 21:32:19.033554133 (11h52m53.5486567s) [upload=12319]: Starting upload Aug 7 21:32:19.033620836 (11h52m53.548723412s) [upload=12319]: Done starting upload Aug 7 21:32:19.033708593 (11h52m53.548811183s) [upload=12319]: Uploading segments... Aug 7 21:32:19.033902998 (11h52m53.549005564s) [upload=12319]: Got next segment. Inline: false Aug 7 21:32:19.033935489 (11h52m53.54903806s) [upload=12319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:19.033947286 (11h52m53.549049873s) [upload=12319]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:18.989442137 (11h53m53.504544716s) [upload=12319]: Next segment err: context deadline exceeded Aug 7 21:33:18.989483135 (11h53m53.504585705s) [upload=12319]: Done uploading segments... Aug 7 21:33:18.989519237 (11h53m53.504621808s) [upload=12319]: Upload finished. err: context deadline exceeded Duration: 1m0.000725902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...................................X..XXXXXXXXXXXXXXXXXXXXXX] duration: 30.000594947s FAIL (2025-08-07 21:33:07 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Duration: 30.000618101s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000366242s FAIL (2025-08-07 21:33:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000381268s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000925184s FAIL (2025-08-07 21:33:00 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000948942s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [..................................X.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001158563s FAIL (2025-08-07 21:32:58 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001181114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 1m0.001559625s FAIL (2025-08-07 21:32:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001589335s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [.................................X..XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001127722s FAIL (2025-08-07 21:32:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001147242s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 30.000361433s FAIL (2025-08-07 21:32:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000389606s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.001138569s FAIL (2025-08-07 21:32:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001157502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000766748s FAIL (2025-08-07 21:32:23 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000802067s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 1m0.000641325s FAIL (2025-08-07 21:32:18 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 5.640676036 (11h39m30.155778606s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.642585356 (11h39m30.157687924s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.643067815 (11h39m30.158170386s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.703545936 (11h39m30.218648502s) [upload=12100]: Next returned nil segment Aug 7 21:18:55.703561907 (11h39m30.218664489s) [upload=12100]: Waiting for error group managing segments... Aug 7 21:18:55.722257588 (11h39m30.237360157s) [noise=true , address=45.91.116.7:4020 , piece_id=RLJSC76ICPPSAK4ZFZPRD2GQP54VESM4RSXMX4EOZTU5LNYDWTZA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.724853022 (11h39m30.239955588s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.725037274 (11h39m30.240139844s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730399702 (11h39m30.245502269s) [noise=true , address=104.36.82.2:4747 , piece_id=ZRKZEN5KXEHMD3OBPISFWZMLA7H2YNKUB5J57IVQ5AVHA5IJGXGQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730432338 (11h39m30.2455349s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.739954757 (11h39m30.255057333s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.742133144 (11h39m30.257235706s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.745676444 (11h39m30.260779013s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.747051082 (11h39m30.262153652s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.754982553 (11h39m30.270085122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.763242025 (11h39m30.278344591s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.767097571 (11h39m30.282200149s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.795229256 (11h39m30.310331824s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces. Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments... Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments... Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments... Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments... Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments... Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments... Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments... Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded Aug 7 21:31:18.972600679 (11h51m53.487703265s) [upload=12304]: Starting upload Aug 7 21:31:18.972700584 (11h51m53.487803182s) [upload=12304]: Done starting upload Aug 7 21:31:18.972946355 (11h51m53.488048923s) [upload=12304]: Uploading segments... Aug 7 21:31:18.973271681 (11h51m53.488374263s) [upload=12304]: Got next segment. Inline: false Aug 7 21:31:18.973342730 (11h51m53.488445311s) [upload=12304]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:18.973348044 (11h51m53.488450626s) [upload=12304]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:18.925211587 (11h52m53.440314165s) [upload=12304]: Next segment err: context deadline exceeded Aug 7 21:32:18.925381198 (11h52m53.440483776s) [upload=12304]: Done uploading segments... Aug 7 21:32:18.925404293 (11h52m53.440506857s) [upload=12304]: Upload finished. err: context deadline exceeded Duration: 1m0.000662453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 24 out of 60 is failed (threshold=3) [...................................X.XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001012163s FAIL (2025-08-07 21:32:12 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001025253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....................................X..XXXXXXXXXXXXXXXXXXXXX] duration: 30.000658135s FAIL (2025-08-07 21:32:07 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== d=12191]: Next segment err: context deadline exceeded Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments... Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments... Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Duration: 30.000679047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000607889s FAIL (2025-08-07 21:31:52 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000637991s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 1m0.000546806s FAIL (2025-08-07 21:31:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000563992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 23 out of 60 is failed (threshold=3) [....................................X.XXXXXXXXXXXXXXXXXXXXXX] duration: 30.001045346s FAIL (2025-08-07 21:31:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001225878s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.........X] duration: 2m0.00053698s remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:31:42 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 2m0.000564794s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....................................X..XXXXXXXXXXXXXXXXXXXX] duration: 30.001414768s FAIL (2025-08-07 21:31:36 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ad=12177]: Next segment err: context deadline exceeded Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments... Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments... Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments... Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments... Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Duration: 30.001443836s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m0.000859717s FAIL (2025-08-07 21:31:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000873796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 30.000660698s FAIL (2025-08-07 21:31:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000688494s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [..................................X..XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001143138s FAIL (2025-08-07 21:31:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001168204s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXX........XX..] duration: 38.788697911s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:31:26 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 38.788739787s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 1m0.001339348s FAIL (2025-08-07 21:31:18 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== I5D5A, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.623648222 (11h39m30.138750801s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.630084181 (11h39m30.145186749s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.636336554 (11h39m30.151439122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.638431947 (11h39m30.15353452s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.640676036 (11h39m30.155778606s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.642585356 (11h39m30.157687924s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.643067815 (11h39m30.158170386s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.703545936 (11h39m30.218648502s) [upload=12100]: Next returned nil segment Aug 7 21:18:55.703561907 (11h39m30.218664489s) [upload=12100]: Waiting for error group managing segments... Aug 7 21:18:55.722257588 (11h39m30.237360157s) [noise=true , address=45.91.116.7:4020 , piece_id=RLJSC76ICPPSAK4ZFZPRD2GQP54VESM4RSXMX4EOZTU5LNYDWTZA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.724853022 (11h39m30.239955588s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.725037274 (11h39m30.240139844s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730399702 (11h39m30.245502269s) [noise=true , address=104.36.82.2:4747 , piece_id=ZRKZEN5KXEHMD3OBPISFWZMLA7H2YNKUB5J57IVQ5AVHA5IJGXGQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730432338 (11h39m30.2455349s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.739954757 (11h39m30.255057333s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.742133144 (11h39m30.257235706s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.745676444 (11h39m30.260779013s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.747051082 (11h39m30.262153652s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.754982553 (11h39m30.270085122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.763242025 (11h39m30.278344591s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.767097571 (11h39m30.282200149s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.795229256 (11h39m30.310331824s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces. Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments... Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments... Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments... Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments... Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments... Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments... Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments... Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded Duration: 1m0.001372762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......................................X..XXXXXXXXXXXXXXXXXXX] duration: 30.001524543s FAIL (2025-08-07 21:31:05 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== oad=12166]: Next segment err: context deadline exceeded Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments... Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments... Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments... Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments... Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments... Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments... Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Duration: 30.001546892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 30.000834102s FAIL (2025-08-07 21:30:56 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000870602s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.001502033s FAIL (2025-08-07 21:30:55 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001541524s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)