Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 527.783448ms FAIL (2025-07-07 20:52:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 527.806178ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 297.641588ms FAIL (2025-07-07 20:52:14 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 297.695178ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 611.973882ms FAIL (2025-07-07 20:51:39 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 612.003008ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.006057509s FAIL (2025-07-07 20:51:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.006083359s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 641.22812ms FAIL (2025-07-07 20:51:22 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 641.254721ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 640.495504ms FAIL (2025-07-07 20:51:01 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 640.526884ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 249.505876ms FAIL (2025-07-07 20:50:23 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 249.536015ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 617.218829ms FAIL (2025-07-07 20:50:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 617.259351ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 335.891624ms FAIL (2025-07-07 20:49:30 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 335.918856ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 506.418416ms FAIL (2025-07-07 20:49:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 506.485683ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 234.560091ms FAIL (2025-07-07 20:48:46 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 234.586868ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 667.358821ms FAIL (2025-07-07 20:48:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 667.40283ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 278.292185ms FAIL (2025-07-07 20:48:12 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 278.318215ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 499.579739ms FAIL (2025-07-07 20:47:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 499.61622ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 257.246123ms FAIL (2025-07-07 20:47:15 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 257.29653ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 563.256571ms FAIL (2025-07-07 20:47:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 563.291712ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 220.847927ms FAIL (2025-07-07 20:46:15 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 220.873689ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 486.078208ms FAIL (2025-07-07 20:45:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.252333ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 264.398073ms FAIL (2025-07-07 20:45:17 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 264.424257ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 602.687437ms FAIL (2025-07-07 20:44:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 602.701016ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 245.000484ms FAIL (2025-07-07 20:44:12 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 245.030446ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 503.451995ms FAIL (2025-07-07 20:43:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.513198ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 247.679048ms FAIL (2025-07-07 20:43:34 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 247.727329ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 245.686838ms FAIL (2025-07-07 20:42:57 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 245.734061ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 509.958245ms FAIL (2025-07-07 20:42:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 509.978997ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 251.975855ms FAIL (2025-07-07 20:41:54 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 252.005202ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.081606973s FAIL (2025-07-07 20:41:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.081634755s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 254.478027ms FAIL (2025-07-07 20:41:35 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 254.508859ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 304.044899ms FAIL (2025-07-07 20:41:18 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 304.078974ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 491.910238ms FAIL (2025-07-07 20:41:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.935879ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 504.038985ms FAIL (2025-07-07 20:40:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.065982ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 247.852276ms FAIL (2025-07-07 20:40:34 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 247.884069ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 516.070044ms FAIL (2025-07-07 20:40:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.092626ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 463.803255ms FAIL (2025-07-07 20:40:15 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 463.831167ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 714.393405ms FAIL (2025-07-07 20:40:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 714.421436ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 585.037375ms FAIL (2025-07-07 20:39:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 585.059626ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 252.904862ms FAIL (2025-07-07 20:39:04 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 252.930554ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 452.563599ms FAIL (2025-07-07 20:38:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 452.592598ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 257.309965ms FAIL (2025-07-07 20:37:59 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 257.337356ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 259.612729ms FAIL (2025-07-07 20:37:20 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 259.746987ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 569.991379ms FAIL (2025-07-07 20:37:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 570.017518ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 453.650624ms FAIL (2025-07-07 20:36:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.684632ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 276.536914ms FAIL (2025-07-07 20:36:42 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 276.567102ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 229.924856ms FAIL (2025-07-07 20:35:36 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 229.971402ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 549.128386ms FAIL (2025-07-07 20:35:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 549.152052ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00139719s FAIL (2025-07-07 20:35:20 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ng upload Jul 7 20:27:39.636855363 (347h16m6.563150052s) [upload=278131]: Done starting upload Jul 7 20:27:39.636922474 (347h16m6.563217146s) [upload=278131]: Uploading segments... Jul 7 20:27:39.636956546 (347h16m6.563251241s) [upload=278131]: Got next segment. Inline: true Jul 7 20:27:39.636960880 (347h16m6.563255567s) [upload=278131]: Waiting for error group managing segments... Jul 7 20:27:39.636996117 (347h16m6.56329079s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:27:39.637000332 (347h16m6.563295019s) [upload=278131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:27:39.637002887 (347h16m6.563297559s) [upload=278131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:27:39.722613186 (347h16m6.648907874s) [upload=278131]: Done uploading segments... Jul 7 20:27:39.722632626 (347h16m6.648927425s) [upload=278131]: Upload finished. err: <nil> Jul 7 20:28:18.207197318 (347h16m45.133491995s) [upload=278139]: Starting upload Jul 7 20:28:18.207272806 (347h16m45.133567492s) [upload=278139]: Done starting upload Jul 7 20:28:18.207341376 (347h16m45.133636063s) [upload=278139]: Uploading segments... Jul 7 20:28:18.207378374 (347h16m45.13367307s) [upload=278139]: Got next segment. Inline: true Jul 7 20:28:18.207394885 (347h16m45.133689572s) [upload=278139]: Waiting for error group managing segments... Jul 7 20:28:18.207450525 (347h16m45.133745211s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:28:18.207456627 (347h16m45.133751313s) [upload=278139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:28:18.207542805 (347h16m45.133837491s) [upload=278139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:28:18.284154465 (347h16m45.210449156s) [upload=278139]: Done uploading segments... Jul 7 20:28:18.284177436 (347h16m45.210472123s) [upload=278139]: Upload finished. err: <nil> Jul 7 20:29:00.049882670 (347h17m26.976177342s) [upload=278148]: Starting upload Jul 7 20:29:00.049949347 (347h17m26.976244042s) [upload=278148]: Done starting upload Jul 7 20:29:00.050028716 (347h17m26.976323392s) [upload=278148]: Uploading segments... Jul 7 20:29:00.050071742 (347h17m26.976366437s) [upload=278148]: Got next segment. Inline: true Jul 7 20:29:00.050077286 (347h17m26.97637198s) [upload=278148]: Waiting for error group managing segments... Jul 7 20:29:00.050136779 (347h17m26.976431476s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:00.050152728 (347h17m26.976447424s) [upload=278148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:00.050158614 (347h17m26.976453307s) [upload=278148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:00.289880244 (347h17m27.216174931s) [upload=278148]: Done uploading segments... Jul 7 20:29:00.289905877 (347h17m27.216200569s) [upload=278148]: Upload finished. err: <nil> Jul 7 20:29:16.108243601 (347h17m43.034538294s) [upload=278152]: Starting upload Jul 7 20:29:16.108348286 (347h17m43.034642973s) [upload=278152]: Done starting upload Jul 7 20:29:16.108440221 (347h17m43.034734912s) [upload=278152]: Uploading segments... Jul 7 20:29:16.108541904 (347h17m43.034836605s) [upload=278152]: Got next segment. Inline: true Jul 7 20:29:16.108554425 (347h17m43.034849122s) [upload=278152]: Waiting for error group managing segments... Jul 7 20:29:16.108613732 (347h17m43.034908418s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:29:16.108622584 (347h17m43.034917283s) [upload=278152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:29:16.108627777 (347h17m43.034922521s) [upload=278152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:29:16.177692122 (347h17m43.103986812s) [upload=278152]: Done uploading segments... Jul 7 20:29:16.177715711 (347h17m43.104010398s) [upload=278152]: Upload finished. err: <nil> Jul 7 20:30:07.200047120 (347h18m34.126341815s) [upload=278164]: Starting upload Jul 7 20:30:07.200122665 (347h18m34.126417361s) [upload=278164]: Done starting upload Jul 7 20:30:07.200215970 (347h18m34.126510646s) [upload=278164]: Uploading segments... Jul 7 20:30:07.200264325 (347h18m34.126559021s) [upload=278164]: Got next segment. Inline: true Jul 7 20:30:07.200270003 (347h18m34.126564697s) [upload=278164]: Waiting for error group managing segments... Jul 7 20:30:07.200316012 (347h18m34.126610706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:30:07.200322371 (347h18m34.12661706s) [upload=278164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:30:07.200326719 (347h18m34.126621393s) [upload=278164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:30:07.271934580 (347h18m34.198229277s) [upload=278164]: Done uploading segments... Jul 7 20:30:07.271992347 (347h18m34.198287045s) [upload=278164]: Upload finished. err: <nil> Jul 7 20:31:07.342724770 (347h19m34.269019468s) [upload=278177]: Starting upload Jul 7 20:31:07.342823078 (347h19m34.269117776s) [upload=278177]: Done starting upload Jul 7 20:31:07.342930361 (347h19m34.269225041s) [upload=278177]: Uploading segments... Jul 7 20:31:07.342999457 (347h19m34.269294153s) [upload=278177]: Got next segment. Inline: true Jul 7 20:31:07.343006337 (347h19m34.269301028s) [upload=278177]: Waiting for error group managing segments... Jul 7 20:31:07.343102972 (347h19m34.269397662s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:07.343108760 (347h19m34.269403447s) [upload=278177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:07.343114220 (347h19m34.269408907s) [upload=278177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:07.405128288 (347h19m34.331422974s) [upload=278177]: Done uploading segments... Jul 7 20:31:07.405148074 (347h19m34.331442761s) [upload=278177]: Upload finished. err: <nil> Jul 7 20:31:22.595191817 (347h19m49.521486506s) [upload=278182]: Starting upload Jul 7 20:31:22.595271255 (347h19m49.521565951s) [upload=278182]: Done starting upload Jul 7 20:31:22.595389433 (347h19m49.521684106s) [upload=278182]: Uploading segments... Jul 7 20:31:22.595486207 (347h19m49.521780903s) [upload=278182]: Got next segment. Inline: true Jul 7 20:31:22.595495146 (347h19m49.521789839s) [upload=278182]: Waiting for error group managing segments... Jul 7 20:31:22.595598902 (347h19m49.52189359s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:31:22.595607776 (347h19m49.521902471s) [upload=278182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:31:22.595611583 (347h19m49.521906258s) [upload=278182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:31:22.668890007 (347h19m49.595184696s) [upload=278182]: Done uploading segments... Jul 7 20:31:22.668910303 (347h19m49.595204991s) [upload=278182]: Upload finished. err: <nil> Jul 7 20:32:09.078329483 (347h20m36.00462417s) [upload=278191]: Starting upload Jul 7 20:32:09.078386523 (347h20m36.004681211s) [upload=278191]: Done starting upload Jul 7 20:32:09.078497920 (347h20m36.004792593s) [upload=278191]: Uploading segments... Jul 7 20:32:09.078571788 (347h20m36.004866477s) [upload=278191]: Got next segment. Inline: true Jul 7 20:32:09.078577217 (347h20m36.004871903s) [upload=278191]: Waiting for error group managing segments... Jul 7 20:32:09.078633608 (347h20m36.004928283s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:09.078638921 (347h20m36.004933597s) [upload=278191]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:09.078642500 (347h20m36.004937176s) [upload=278191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:09.155298109 (347h20m36.08159281s) [upload=278191]: Done uploading segments... Jul 7 20:32:09.155320753 (347h20m36.081615446s) [upload=278191]: Upload finished. err: <nil> Jul 7 20:32:48.912216516 (347h21m15.838511191s) [upload=278200]: Starting upload Jul 7 20:32:48.912320542 (347h21m15.838615218s) [upload=278200]: Done starting upload Jul 7 20:32:48.912413059 (347h21m15.838707755s) [upload=278200]: Uploading segments... Jul 7 20:32:48.912536235 (347h21m15.838830927s) [upload=278200]: Got next segment. Inline: true Jul 7 20:32:48.912542437 (347h21m15.838837129s) [upload=278200]: Waiting for error group managing segments... Jul 7 20:32:48.912602991 (347h21m15.83889767s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:32:48.912611416 (347h21m15.83890612s) [upload=278200]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:32:48.912617199 (347h21m15.838911904s) [upload=278200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:32:48.970632380 (347h21m15.896927057s) [upload=278200]: Done uploading segments... Jul 7 20:32:48.970653336 (347h21m15.896948133s) [upload=278200]: Upload finished. err: <nil> Jul 7 20:33:36.422935558 (347h22m3.349230234s) [upload=278214]: Starting upload Jul 7 20:33:36.423043066 (347h22m3.349337752s) [upload=278214]: Done starting upload Jul 7 20:33:36.423110686 (347h22m3.349405359s) [upload=278214]: Uploading segments... Jul 7 20:33:36.423151673 (347h22m3.349446415s) [upload=278214]: Got next segment. Inline: true Jul 7 20:33:36.423157427 (347h22m3.349452113s) [upload=278214]: Waiting for error group managing segments... Jul 7 20:33:36.423210227 (347h22m3.349504924s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 7 20:33:36.423217136 (347h22m3.349511831s) [upload=278214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 7 20:33:36.423221377 (347h22m3.349516075s) [upload=278214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 7 20:33:36.496833742 (347h22m3.423128455s) [upload=278214]: Done uploading segments... Jul 7 20:33:36.496854271 (347h22m3.423148945s) [upload=278214]: Upload finished. err: <nil> Duration: 30.001434565s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.683351984s FAIL (2025-07-07 20:34:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.683393327s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 538.791704ms FAIL (2025-07-07 20:34:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.826666ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.110993375s FAIL (2025-07-07 20:34:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.111017547s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 555.206311ms FAIL (2025-07-07 20:33:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 555.810898ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)